scout-ai 1.2.3 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (174) hide show
  1. checksums.yaml +4 -4
  2. data/.vimproject +138 -50
  3. data/README.md +171 -290
  4. data/Rakefile +17 -1
  5. data/VERSION +1 -1
  6. data/doc/Improvements.md +325 -0
  7. data/doc/StartHere.md +110 -0
  8. data/doc/developer/Architecture.md +126 -0
  9. data/doc/developer/Backends.md +199 -0
  10. data/doc/developer/ChatLifecycle.md +183 -0
  11. data/doc/developer/DelegationInternals.md +295 -0
  12. data/doc/developer/DesignPrinciples.md +245 -0
  13. data/doc/developer/PromptProcessing.md +292 -0
  14. data/doc/developer/Provenance.md +317 -0
  15. data/doc/user/BuildingAgents.md +345 -0
  16. data/doc/user/Cookbook.md +333 -0
  17. data/doc/user/CoreConcepts.md +181 -0
  18. data/doc/user/Delegation.md +191 -0
  19. data/doc/user/GettingStarted.md +159 -0
  20. data/doc/user/ManagingContext.md +163 -0
  21. data/doc/user/MultiAgentWorkflows.md +256 -0
  22. data/doc/user/Python.md +159 -0
  23. data/doc/user/RunningInference.md +200 -0
  24. data/doc/user/ToolCalling.md +193 -0
  25. data/doc/user/WritingChats.md +197 -0
  26. data/lib/scout/llm/agent/chat.rb +61 -11
  27. data/lib/scout/llm/agent/delegate.rb +274 -65
  28. data/lib/scout/llm/agent/iterate.rb +2 -2
  29. data/lib/scout/llm/agent/save.rb +273 -0
  30. data/lib/scout/llm/agent/workflow.rb +164 -0
  31. data/lib/scout/llm/agent.rb +86 -61
  32. data/lib/scout/llm/ask.rb +62 -17
  33. data/lib/scout/llm/backends/anthropic.rb +9 -2
  34. data/lib/scout/llm/backends/bedrock.rb +15 -3
  35. data/lib/scout/llm/backends/default.rb +183 -99
  36. data/lib/scout/llm/backends/glm.rb +58 -0
  37. data/lib/scout/llm/backends/huggingface.rb +196 -26
  38. data/lib/scout/llm/backends/ollama.rb +13 -1
  39. data/lib/scout/llm/backends/openai.rb +0 -2
  40. data/lib/scout/llm/backends/openwebui.rb +20 -13
  41. data/lib/scout/llm/backends/relay.rb +22 -22
  42. data/lib/scout/llm/backends/responses.rb +1 -1
  43. data/lib/scout/llm/chat/agent_meta.rb +264 -0
  44. data/lib/scout/llm/chat/annotation.rb +39 -10
  45. data/lib/scout/llm/chat/parse.rb +28 -6
  46. data/lib/scout/llm/chat/persist.rb +25 -0
  47. data/lib/scout/llm/chat/process/clear.rb +41 -6
  48. data/lib/scout/llm/chat/process/files.rb +21 -6
  49. data/lib/scout/llm/chat/process/meta.rb +421 -34
  50. data/lib/scout/llm/chat/process/options.rb +21 -1
  51. data/lib/scout/llm/chat/process/tools.rb +56 -15
  52. data/lib/scout/llm/chat/process.rb +4 -0
  53. data/lib/scout/llm/chat/prompt/shorten_tools.rb +125 -0
  54. data/lib/scout/llm/chat/prompt/shorten_tools_epoch.rb +365 -0
  55. data/lib/scout/llm/chat/prompt.rb +48 -0
  56. data/lib/scout/llm/chat/provenance.rb +775 -0
  57. data/lib/scout/llm/chat/tool_calls.rb +76 -0
  58. data/lib/scout/llm/chat.rb +18 -2
  59. data/lib/scout/llm/embed.rb +11 -3
  60. data/lib/scout/llm/image.rb +86 -0
  61. data/lib/scout/llm/mcp.rb +10 -2
  62. data/lib/scout/llm/rag.rb +3 -3
  63. data/lib/scout/llm/tools/call.rb +160 -11
  64. data/lib/scout/llm/tools/knowledge_base.rb +1 -1
  65. data/lib/scout/llm/tools/workflow.rb +32 -16
  66. data/lib/scout/model/python/huggingface/causal.rb +23 -5
  67. data/lib/scout/model/python/huggingface.rb +2 -1
  68. data/lib/scout-ai.rb +1 -0
  69. data/python/README.md +197 -14
  70. data/python/scout_ai/huggingface/eval.py +245 -34
  71. data/python/tests/test_huggingface_eval.py +58 -0
  72. data/research/ChatAnalyst-required-changes.md +167 -0
  73. data/research/agent-delegation-analysis.md +810 -0
  74. data/research/agent-meta-provenance-integration-plan.md +622 -0
  75. data/research/agent-workflow-analysis.md +1120 -0
  76. data/research/backends-analysis.md +836 -0
  77. data/research/chat-core-analysis.md +946 -0
  78. data/research/chatanalyst-provenance/00-baseline.md +30 -0
  79. data/research/chatanalyst-provenance/01-repo-map.md +60 -0
  80. data/research/chatanalyst-provenance/02-event-reconstruction.md +55 -0
  81. data/research/chatanalyst-provenance/03-duplication-evidence.md +45 -0
  82. data/research/chatanalyst-provenance/04-tooling-root-cause.md +57 -0
  83. data/research/chatanalyst-provenance/05-fix-plan.md +46 -0
  84. data/research/chatanalyst-provenance/07-critic-review.md +25 -0
  85. data/research/chatanalyst-provenance/final-report.md +45 -0
  86. data/research/chatanalyst-provenance/resumption.md +37 -0
  87. data/research/coding-philosophy-analysis.md +928 -0
  88. data/research/commands-analysis.md +947 -0
  89. data/research/multi-agent-patterns-analysis.md +853 -0
  90. data/research/prompt-strategies-analysis.md +630 -0
  91. data/research/prov-verbosity-fix-notes.md +77 -0
  92. data/research/provenance-analysis.md +469 -0
  93. data/research/provenance-navigation-design.md +640 -0
  94. data/research/synthesis-report.md +487 -0
  95. data/research/tools-system-analysis.md +779 -0
  96. data/scout-ai.gemspec +100 -11
  97. data/scout_commands/agent/ask +13 -3
  98. data/scout_commands/agent/kb +2 -0
  99. data/scout_commands/llm/ask +11 -4
  100. data/scout_commands/llm/md +76 -0
  101. data/scout_commands/llm/process_queries +48 -0
  102. data/scout_commands/llm/prov +602 -0
  103. data/scout_commands/llm/word +71 -0
  104. data/scout_commands/workflow/mcp +43 -0
  105. data/share/word/reference.docx +0 -0
  106. data/test/etc/AI/mock.yaml +11 -0
  107. data/test/fixtures/backends/anthropic.json +19 -0
  108. data/test/fixtures/backends/anthropic_tool_use.json +24 -0
  109. data/test/fixtures/backends/bedrock.json +8 -0
  110. data/test/fixtures/backends/bedrock_embedding.json +3 -0
  111. data/test/fixtures/backends/bedrock_tool_use.json +17 -0
  112. data/test/fixtures/backends/ollama.json +16 -0
  113. data/test/fixtures/backends/ollama_tool_call.json +27 -0
  114. data/test/fixtures/backends/openai_chat.json +21 -0
  115. data/test/fixtures/backends/openai_chat_tool_call.json +31 -0
  116. data/test/fixtures/backends/responses.json +33 -0
  117. data/test/fixtures/backends/responses_tool_call.json +28 -0
  118. data/test/integration/README.md +32 -0
  119. data/test/integration/scout/llm/backends/test_endpoints.rb +34 -0
  120. data/test/integration/scout/llm/backends/test_openwebui.rb +61 -0
  121. data/test/integration/scout/llm/backends/test_relay.rb +52 -0
  122. data/test/integration/scout/llm/test_infrastructure.rb +74 -0
  123. data/test/{scout → integration/scout}/llm/test_mcp.rb +1 -1
  124. data/test/integration/scout/llm/tools/test_mcp.rb +42 -0
  125. data/test/integration/scout/model/test_base.rb +91 -0
  126. data/test/scout/llm/agent/test_chat.rb +8 -2
  127. data/test/scout/llm/agent/test_save.rb +413 -0
  128. data/test/scout/llm/agent/test_workflow.rb +110 -0
  129. data/test/scout/llm/backends/test_anthropic.rb +93 -10
  130. data/test/scout/llm/backends/test_bedrock.rb +118 -2
  131. data/test/scout/llm/backends/test_huggingface.rb +137 -42
  132. data/test/scout/llm/backends/test_ollama.rb +70 -20
  133. data/test/scout/llm/backends/test_openwebui.rb +42 -40
  134. data/test/scout/llm/backends/test_relay.rb +4 -2
  135. data/test/scout/llm/chat/agent_meta_fixtures.rb +131 -0
  136. data/test/scout/llm/chat/process/test_meta.rb +518 -0
  137. data/test/scout/llm/chat/process/test_normalize_usage.rb +183 -0
  138. data/test/scout/llm/chat/test_agent_meta.rb +357 -0
  139. data/test/scout/llm/chat/test_agent_meta_provenance.rb +467 -0
  140. data/test/scout/llm/chat/test_agent_meta_tokens.rb +594 -0
  141. data/test/scout/llm/chat/test_parse.rb +70 -15
  142. data/test/scout/llm/chat/test_prov_cli.rb +274 -0
  143. data/test/scout/llm/chat/test_provenance.rb +240 -0
  144. data/test/scout/llm/chat/test_tool_calls.rb +38 -0
  145. data/test/scout/llm/test_agent.rb +13 -36
  146. data/test/scout/llm/test_ask.rb +75 -52
  147. data/test/scout/llm/test_chat.rb +107 -13
  148. data/test/scout/llm/test_embed.rb +48 -0
  149. data/test/scout/llm/test_rag.rb +23 -16
  150. data/test/scout/llm/test_tools.rb +12 -1
  151. data/test/scout/llm/tools/test_knowledge_base.rb +0 -1
  152. data/test/scout/llm/tools/test_mcp.rb +5 -3
  153. data/test/scout/llm/tools/test_workflow.rb +23 -2
  154. data/test/scout/model/python/huggingface/causal/test_next_token.rb +11 -5
  155. data/test/scout/model/python/huggingface/test_causal.rb +9 -3
  156. data/test/scout/model/python/huggingface/test_classification.rb +11 -2
  157. data/test/scout/model/python/test_torch.rb +2 -0
  158. data/test/scout/model/python/torch/test_helpers.rb +4 -0
  159. data/test/scout/model/test_base.rb +4 -2
  160. data/test/support/availability.rb +231 -0
  161. data/test/support/fake_clients.rb +138 -0
  162. data/test/support/fixtures.rb +21 -0
  163. data/test/support/infrastructure_probes.rb +136 -0
  164. data/test/support/mock_backend.rb +215 -0
  165. data/test/test_helper.rb +32 -2
  166. metadata +99 -10
  167. data/doc/Agent.md +0 -327
  168. data/doc/Chat.md +0 -458
  169. data/doc/LLM.md +0 -340
  170. data/doc/RAG.md +0 -129
  171. data/scout_commands/documenter +0 -148
  172. data/test/scout/llm/backends/test_openai.rb +0 -192
  173. data/test/scout/llm/backends/test_responses.rb +0 -238
  174. data/test/scout/llm/test_parse.rb +0 -98
data/doc/Agent.md DELETED
@@ -1,327 +0,0 @@
1
- # Agent
2
-
3
- `LLM::Agent` is the stateful wrapper around `LLM.ask` + `Chat`.
4
-
5
- Use an Agent when you want **one or more ongoing conversations** (state), plus:
6
-
7
- - a consistent place to store default LLM options (`endpoint`, `model`, `backend`, `format`, …)
8
- - automatic tool wiring from:
9
- - a **Workflow** (tasks as function tools)
10
- - a **KnowledgeBase** (databases as function tools)
11
- - a convenient Ruby API for structured outputs (`json_format`, `iterate`, `iterate_dictionary`)
12
- - optional delegation to other agents (multi-agent control loops)
13
-
14
- Related docs:
15
-
16
- - `doc/LLM.md` — `LLM.ask`, endpoints/backends, tool calling
17
- - `doc/Chat.md` — chat file roles/options (including `tool`, `task`, `mcp`, `previous_response_id`)
18
-
19
- ---
20
-
21
- ## 1. Quick start
22
-
23
- ### 1.1 Minimal stateful conversation
24
-
25
- ```ruby
26
- require 'scout-ai'
27
-
28
- agent = LLM::Agent.new(endpoint: :nano)
29
- agent.start_chat.system "You are a helpful assistant"
30
-
31
- agent.start # create a new conversation branch
32
- agent.user "Say hi" # append to current_chat
33
- puts agent.chat # ask + append assistant reply
34
- ```
35
-
36
- ### 1.2 Factory shortcut
37
-
38
- ```ruby
39
- agent = LLM.agent(endpoint: :ollama, model: 'llama3.1')
40
- ```
41
-
42
- ---
43
-
44
- ## 2. Conversation lifecycle
45
-
46
- Agents maintain two chats:
47
-
48
- ### `start_chat`
49
-
50
- The “base” chat.
51
-
52
- - It is where you put messages that should always be present (system policy, examples, shared context).
53
- - It is **not automatically sent** unless you create a current chat from it via `start`.
54
-
55
- ### `start(chat=nil)`
56
-
57
- - `start()` with no argument:
58
- - branches `start_chat` (non-destructive copy)
59
- - stores it as `current_chat`
60
- - `start(chat)` with a `Chat` or `Array`:
61
- - adopts that as the `current_chat`
62
-
63
- ### `current_chat`
64
-
65
- The active conversation.
66
-
67
- ### Common pitfall
68
-
69
- If you call `agent.ask(...)` directly with your own messages array, the Agent will not automatically prepend `start_chat`. The simplest “normal” pattern is:
70
-
71
- ```ruby
72
- agent.start
73
- agent.user "..."
74
- agent.chat
75
- ```
76
-
77
- ---
78
-
79
- ## 3. Agent forwards the Chat DSL
80
-
81
- `LLM::Agent` forwards unknown methods to `current_chat` (via `method_missing`).
82
-
83
- So you can use the chat builder methods directly:
84
-
85
- ```ruby
86
- agent.system "You are a domain expert"
87
- agent.user "Summarize this file"
88
- agent.file "paper.md"
89
- agent.image "figure.png"
90
- agent.pdf "supplement.pdf"
91
- ```
92
-
93
- All the special chat-file roles described in `doc/Chat.md` work the same way from an Agent: `import`, `continue`, `tool`, `task`, `mcp`, etc. These use the `message` builder, which is a more general way to add messages to the chat. These are equivalent:
94
-
95
- ```ruby
96
- agent.pdf "supplement.pdf"
97
- agent.message :pdf, "supplement.pdf"
98
- ```
99
- ---
100
-
101
- ## 4. Tool wiring (Workflow + KnowledgeBase)
102
-
103
- ### 4.1 Workflow tools
104
-
105
- If an Agent has a `workflow`, all exported tasks are exposed as callable tools when the model supports function calling.
106
-
107
- ```ruby
108
- agent = LLM::Agent.new(workflow: 'Baking', endpoint: :nano)
109
- agent.start
110
- agent.user "Bake muffins using the tool"
111
- puts agent.chat
112
- ```
113
-
114
- Internally:
115
-
116
- - `LLM.workflow_tools(workflow)` produces one tool definition per task.
117
- - When the model calls a function, `LLM.process_calls` executes it via `LLM.call_workflow`.
118
-
119
- ### 4.2 KnowledgeBase tools
120
-
121
- If an Agent has a `knowledge_base`, each database is exposed as a callable tool.
122
-
123
- - For a database `brothers`, the model can call `brothers(entities: [...])`.
124
- - If the database has fields, an additional tool `brothers_association_details` is exposed.
125
-
126
- ### 4.3 Tool merging rules (important nuance)
127
-
128
- Tools come from multiple places:
129
-
130
- 1) tools passed to `ask(..., tools: ...)`
131
- 2) tools stored in `agent.other_options[:tools]`
132
- 3) tools auto-exported from `workflow` / `knowledge_base`
133
-
134
- The Agent merges them roughly as:
135
-
136
- - start with `options[:tools]` (or `{}`)
137
- - merge `other_options[:tools]`
138
- - merge workflow and knowledge base tools
139
-
140
- If the same tool name appears multiple times, later merges override earlier ones.
141
-
142
- ---
143
-
144
- ## 5. Asking vs chatting
145
-
146
- ### `ask(messages, options={})`
147
-
148
- Low-level: calls `LLM.ask(...)` with Agent defaults merged.
149
-
150
- - returns a **string** by default
151
- - returns a **message trace** if `return_messages: true`
152
-
153
- ### `chat(options={})`
154
-
155
- High-level “stateful” method:
156
-
157
- - calls `ask(current_chat, return_messages: true)`
158
- - appends returned messages onto `current_chat`
159
- - returns the assistant content (the last assistant message)
160
-
161
- ---
162
-
163
- ## 6. Structured outputs
164
-
165
- ### `json`
166
-
167
- Sets chat format to JSON and parses the response:
168
-
169
- ```ruby
170
- agent.start
171
- agent.user "Return {\"content\": [\"a\",\"b\"]}"
172
- pp agent.json
173
- ```
174
-
175
- If the returned JSON is exactly `{"content": ...}`, the helper returns the inner `content`.
176
-
177
- ### `json_format(schema_hash)`
178
-
179
- Requests a JSON response constrained by a schema (supported best by the Responses backend).
180
-
181
- ```ruby
182
- schema = {
183
- name: 'answer',
184
- type: 'object',
185
- properties: {
186
- judgement: { type: :boolean },
187
- notes: { type: :string, default: "" }
188
- },
189
- required: [:judgement],
190
- additionalProperties: false
191
- }
192
-
193
- agent.start
194
- agent.user "Is this funny?"
195
- pp agent.json_format(schema)
196
- ```
197
-
198
- ---
199
-
200
- ## 7. Iteration helpers (programmatic control loops)
201
-
202
- These helpers are designed for “agentic scripts” where you want the model to produce a list/dictionary and then iterate in Ruby.
203
-
204
- ### `iterate(prompt=nil) { |item| ... }`
205
-
206
- - forces `endpoint :responses`
207
- - requests JSON schema `{content: [string, ...]}`
208
- - yields each item
209
- - resets `format` back to `:text`
210
-
211
- ```ruby
212
- agent = LLM.agent
213
- agent.iterate("List 3 next actions") do |action|
214
- puts "- #{action}"
215
- end
216
- ```
217
-
218
- ### `iterate_dictionary(prompt=nil) { |k,v| ... }`
219
-
220
- - requests a JSON object whose values are strings (`additionalProperties: {type: :string}`)
221
-
222
- ```ruby
223
- agent.iterate_dictionary("Return a dict of tool_name => what it does") do |name, desc|
224
- puts "#{name}: #{desc}"
225
- end
226
- ```
227
-
228
- ---
229
-
230
- ## 8. Delegation (multi-agent wiring)
231
-
232
- `Agent#delegate` registers another agent as a **tool**.
233
-
234
- The tool name becomes:
235
-
236
- ```text
237
- hand_off_to_<name>
238
- ```
239
-
240
- The default schema expects:
241
-
242
- - `message` (required)
243
- - `new_conversation` (optional, default false)
244
-
245
- Example:
246
-
247
- ```ruby
248
- joker = LLM.agent(endpoint: :nano)
249
- joker.start_chat.system "You only answer with knock knock jokes"
250
-
251
- judge = LLM.agent(endpoint: :nano, format: { judgement: :boolean })
252
- judge.start_chat.system "Judge if a joke is funny"
253
-
254
- supervisor = LLM.agent(endpoint: :nano)
255
- supervisor.start_chat.system "Use delegated agents to do the work"
256
-
257
- supervisor.delegate(joker, :joker, "Generate jokes")
258
- supervisor.delegate(judge, :judge, "Evaluate jokes")
259
-
260
- supervisor.start
261
- supervisor.user "Try up to 5 jokes until judged funny"
262
- puts supervisor.chat
263
- ```
264
-
265
- ---
266
-
267
- ## 9. Loading an Agent (agent directories)
268
-
269
- Agents can be loaded by name or from a directory.
270
-
271
- ### `LLM::Agent.load_agent(name)`
272
-
273
- Resolution logic (simplified):
274
-
275
- - if `name` is a path:
276
- - if it is a directory with `agent/*.rb` it loads that script
277
- - otherwise it loads the directory as an agent directory
278
- - otherwise it looks under the standard Scout paths:
279
- - workflows (`Scout.workflows[name]`)
280
- - agent dirs (`Scout.var.Agent[name]`)
281
- - chat dirs (`Scout.chats[name]`)
282
-
283
- ### Agent directory layout
284
-
285
- If you create an agent directory, these files are detected automatically:
286
-
287
- ```text
288
- <agent_dir>/workflow.rb # optional
289
- <agent_dir>/knowledge_base/ # optional
290
- <agent_dir>/start_chat # optional chat file
291
- ```
292
-
293
- If `start_chat` is not present but a workflow exists, the Agent will create a base chat containing an `introduce: <workflow>` message (workflow documentation injected).
294
-
295
- ---
296
-
297
- ## 10. Advanced: workflow-provided `ask` task
298
-
299
- If the Agent’s workflow defines a task named `ask`, `Agent#ask` can delegate the entire LLM interaction to that workflow task:
300
-
301
- - the Agent passes `chat: Chat.print(messages)` as input
302
- - the workflow task can implement a custom control loop, custom tool execution, etc.
303
-
304
- This is an escape hatch for “agent frameworks built as workflows”.
305
-
306
- ---
307
-
308
- ## 11. Error handling
309
-
310
- Set `agent.process_exception` to a Proc to intercept exceptions raised during ask/chat.
311
-
312
- If the Proc returns truthy, the call is retried.
313
-
314
- ---
315
-
316
- ## 12. CLI integration
317
-
318
- Agents are primarily used from:
319
-
320
- ```bash
321
- scout-ai agent ask <agent_name> your question
322
- scout-ai agent ask -c my.chat <agent_name> continue this conversation
323
- ```
324
-
325
- Note how the question does not need to be in quotes.
326
-
327
- See `doc/Chat.md` for chat file roles and `doc/LLM.md` for endpoint configuration.