solid_loop 0.0.4 → 0.0.5

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (56) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +445 -0
  3. data/README.md +305 -4
  4. data/Rakefile +5 -4
  5. data/app/controllers/solid_loop/application_controller.rb +6 -0
  6. data/app/controllers/solid_loop/dashboard_controller.rb +167 -22
  7. data/app/controllers/solid_loop/events_controller.rb +9 -1
  8. data/app/controllers/solid_loop/mcp_sessions_controller.rb +12 -0
  9. data/app/controllers/solid_loop/messages_controller.rb +11 -1
  10. data/app/controllers/solid_loop/tool_calls_controller.rb +32 -0
  11. data/app/helpers/solid_loop/application_helper.rb +4 -1
  12. data/app/helpers/solid_loop/metrics_helper.rb +232 -0
  13. data/app/jobs/solid_loop/janitor_job.rb +24 -0
  14. data/app/jobs/solid_loop/llm_completion_job.rb +2 -2
  15. data/app/models/solid_loop/base.rb +89 -6
  16. data/app/models/solid_loop/loop.rb +22 -0
  17. data/app/models/solid_loop/message.rb +33 -7
  18. data/app/models/solid_loop/tool_call.rb +7 -1
  19. data/app/services/solid_loop/adapters/native.rb +156 -23
  20. data/app/services/solid_loop/dialects/anthropic.rb +43 -10
  21. data/app/services/solid_loop/dialects/gemini.rb +55 -15
  22. data/app/services/solid_loop/dialects/open_ai.rb +17 -5
  23. data/app/services/solid_loop/dialects/reasoning_packer.rb +72 -7
  24. data/app/services/solid_loop/llm_usage_parser/llama.rb +18 -3
  25. data/app/services/solid_loop/mcp_session_initializer.rb +1 -0
  26. data/app/services/solid_loop/middlewares/agent_initialization.rb +1 -1
  27. data/app/services/solid_loop/middlewares/error_handling.rb +5 -1
  28. data/app/services/solid_loop/middlewares/event_logging.rb +3 -3
  29. data/app/services/solid_loop/middlewares/message_building.rb +38 -4
  30. data/app/services/solid_loop/middlewares/response_parsing.rb +19 -3
  31. data/app/views/layouts/solid_loop/admin.html.erb +134 -23
  32. data/app/views/solid_loop/dashboard/index.html.erb +330 -41
  33. data/app/views/solid_loop/events/index.html.erb +17 -1
  34. data/app/views/solid_loop/loops/index.html.erb +40 -3
  35. data/app/views/solid_loop/loops/show.html.erb +1 -1
  36. data/app/views/solid_loop/mcp_sessions/index.html.erb +40 -2
  37. data/app/views/solid_loop/messages/_message.html.erb +26 -35
  38. data/app/views/solid_loop/messages/index.html.erb +16 -2
  39. data/app/views/solid_loop/tool_calls/index.html.erb +18 -3
  40. data/db/migrate/20260819000100_solid_loop_add_retention_indexes.rb +22 -0
  41. data/docs/contributing/coverage.md +8 -8
  42. data/docs/decisions/mcp-server.md +4 -2
  43. data/docs/decisions/reasoning_persistence.md +120 -4
  44. data/docs/guides/dialects.md +1 -1
  45. data/docs/guides/mcp_transports.md +72 -0
  46. data/docs/validation.md +85 -0
  47. data/lib/solid_loop/configuration.rb +93 -0
  48. data/lib/solid_loop/engine.rb +8 -0
  49. data/lib/solid_loop/janitor.rb +110 -0
  50. data/lib/solid_loop/mcp/toolset.rb +119 -31
  51. data/lib/solid_loop/pipeline/builder.rb +31 -10
  52. data/lib/solid_loop/redaction.rb +26 -0
  53. data/lib/solid_loop/version.rb +1 -1
  54. data/lib/solid_loop.rb +47 -0
  55. metadata +7 -2
  56. data/lib/tasks/coverage.rake +0 -206
@@ -1,73 +1,64 @@
1
1
  <div id="sl-message-<%= message.id %>" class="sl-card sl-message sl-message--<%= message.role %>">
2
2
  <div class="sl-message__header">
3
- <strong class="sl-message__role">
4
- <%= message.role %>
5
- </strong>
3
+ <strong class="sl-message__role"><%= message.role %></strong>
6
4
  <% model_name = message.metadata&.dig("model") %>
7
5
  <% if model_name.present? %>
8
- <span class="sl-badge sl-badge--pending" title="Model that served this turn" style="margin-left: 6px; font-weight: normal;">
9
- <%= model_name %>
10
- </span>
6
+ <span class="sl-badge sl-badge--pending sl-message__model" title="Model that served this turn"><%= model_name %></span>
11
7
  <% end %>
12
- <span class="sl-message__time">
13
- <%= message.created_at.strftime("%H:%M:%S") %>
14
- </span>
8
+ <span class="sl-message__time"><%= message.created_at.strftime("%H:%M:%S") %></span>
15
9
  </div>
16
10
 
11
+ <%# Each block below owns its own <pre>. The container must NOT be pre-wrap:
12
+ it would render this template's own indentation and blank lines as content,
13
+ which is what used to indent the first line and open gaps before the footer. %>
17
14
  <div class="sl-json">
18
15
  <% if message.content.present? %>
19
- <%= message.content %>
16
+ <pre class="sl-pre-wrap sl-message__body"><%= message.content %></pre>
20
17
  <% end %>
21
18
 
22
19
  <% if message.tool_calls_raw.present? || message.tool_calls.any? %>
23
20
  <div class="sl-message__tools">
24
- <strong>Tools:</strong>
25
- <div class="sl-mt-10">
26
- <% (message.tool_calls.any? ? message.tool_calls : message.tool_calls_raw).each do |tc| %>
27
- <%
28
- name = tc.is_a?(SolidLoop::ToolCall) ? tc.function_name : tc.dig("function", "name")
29
- tc_id = tc.is_a?(SolidLoop::ToolCall) ? tc.id : nil
30
- %>
31
- <div class="sl-badge <%= tc.is_a?(SolidLoop::ToolCall) && tc.is_success ? 'sl-badge--success' : 'sl-badge--pending' %>" style="margin-bottom: 5px;">
32
- <% if tc_id %>
33
- <%= link_to name, tool_call_path(tc_id), style: "color: inherit; text-decoration: none;" %>
34
- <% else %>
35
- <%= name %>
36
- <% end %>
37
- </div>
38
- <% end %>
39
- </div>
21
+ <span class="sl-message__tools-label">Tools</span>
22
+ <% (message.tool_calls.any? ? message.tool_calls : message.tool_calls_raw).each do |tc| %>
23
+ <%
24
+ name = tc.is_a?(SolidLoop::ToolCall) ? tc.function_name : tc.dig("function", "name")
25
+ tc_id = tc.is_a?(SolidLoop::ToolCall) ? tc.id : nil
26
+ %>
27
+ <span class="sl-badge <%= tc.is_a?(SolidLoop::ToolCall) && tc.is_success ? 'sl-badge--success' : 'sl-badge--pending' %>">
28
+ <% if tc_id %><%= link_to name, tool_call_path(tc_id), class: "sl-badge__link" %><% else %><%= name %><% end %>
29
+ </span>
30
+ <% end %>
40
31
  </div>
41
32
  <% end %>
42
33
 
43
34
  <% if message.reasoning_content.present? %>
44
- <div class="sl-message__reasoning">
45
- <em>Thinking:</em>
35
+ <details class="sl-message__reasoning">
36
+ <summary class="sl-message__reasoning-toggle">Thinking (<%= message.reasoning_content.length %> chars)</summary>
46
37
  <pre class="sl-pre-wrap"><%= message.reasoning_content %></pre>
47
- </div>
38
+ </details>
48
39
  <% end %>
49
40
 
50
41
  <div class="sl-message__footer">
51
42
  <div class="sl-message__footer-item">
52
- Tokens: <span class="sl-message__footer-value">P<%= message.tokens_prompt %>+C<%= message.tokens_completion %></span>
43
+ Tokens <span class="sl-message__footer-value">P<%= message.tokens_prompt %>+C<%= message.tokens_completion %></span>
53
44
  <% if message.tokens_prompt_cached > 0 %>
54
- <span style="color: var(--sl-success); font-size: 0.9em; margin-left: 4px;">(cached <%= message.tokens_prompt_cached %>)</span>
45
+ <span class="sl-message__cached">(cached <%= message.tokens_prompt_cached %>)</span>
55
46
  <% end %>
56
47
  </div>
57
48
  <div class="sl-message__footer-item">
58
- Cost: <span class="sl-message__footer-value"><%= sl_money(message.cost) %></span>
49
+ Cost <span class="sl-message__footer-value"><%= sl_money(message.cost) %></span>
59
50
  </div>
60
51
  <% if message.tps > 0 %>
61
52
  <div class="sl-message__footer-item">
62
- TPS: <span class="sl-message__footer-value"><%= message.tps %></span>
53
+ TPS <span class="sl-message__footer-value"><%= message.tps %></span>
63
54
  </div>
64
55
  <% end %>
65
56
  <% if message.ttft > 0 %>
66
57
  <div class="sl-message__footer-item">
67
- TTFT: <span class="sl-message__footer-value"><%= number_with_precision(message.ttft, precision: 2) %>s</span>
58
+ TTFT <span class="sl-message__footer-value"><%= number_with_precision(message.ttft, precision: 2) %>s</span>
68
59
  </div>
69
60
  <% end %>
70
- <div class="sl-message__footer-item" style="margin-left: auto;">
61
+ <div class="sl-message__footer-item sl-message__footer-item--end">
71
62
  <%= link_to "Details", message_path(message), class: "sl-btn sl-btn--small" %>
72
63
  </div>
73
64
  </div>
@@ -66,8 +66,22 @@
66
66
  %>"><%= message.role.upcase %></span>
67
67
  </td>
68
68
  <td class="sl-table__td"><%= message.metadata&.dig("model").presence || "—" %></td>
69
- <td class="sl-table__td">P<%= message.tokens_prompt %>+C<%= message.tokens_completion %></td>
70
- <td class="sl-table__td"><%= sl_money(message.cost) %></td>
69
+ <td class="sl-table__td">
70
+ <%# `tokens_total` is the persisted figure the loop's budget is checked
71
+ against; the in/out split is the detail, so it goes underneath
72
+ rather than being the only thing shown ("P0+C0" told you nothing).
73
+ A message still being streamed has its completion tokens written by
74
+ `trigger_content_append` but no total yet, so fall back to the sum
75
+ rather than printing "—" next to a non-zero breakdown. %>
76
+ <% total = message.tokens_total.to_i.positive? ? message.tokens_total.to_i : message.tokens_prompt.to_i + message.tokens_completion.to_i %>
77
+ <span class="sl-metric"><%= sl_tokens(total) %></span>
78
+ <% if message.tokens_prompt.to_i.positive? || message.tokens_completion.to_i.positive? %>
79
+ <span class="sl-metric__sub">
80
+ in <%= number_with_delimiter(message.tokens_prompt.to_i) %> · out <%= number_with_delimiter(message.tokens_completion.to_i) %><%= " · cached #{number_with_delimiter(message.tokens_prompt_cached.to_i)}" if message.tokens_prompt_cached.to_i.positive? %>
81
+ </span>
82
+ <% end %>
83
+ </td>
84
+ <td class="sl-table__td"><span class="sl-metric <%= "sl-metric--blank" unless message.cost.to_f.positive? %>"><%= sl_money(message.cost) %></span></td>
71
85
  <td class="sl-table__td"><%= message.created_at.strftime("%Y-%m-%d %H:%M") %></td>
72
86
  <td class="sl-table__td" style="text-align: right;">
73
87
  <%= link_to "View", message_path(message), class: "sl-btn sl-btn--small" %>
@@ -1,3 +1,7 @@
1
+ <div class="sl-page-header">
2
+ <h2>Tool Calls</h2>
3
+ </div>
4
+
1
5
  <div class="sl-card">
2
6
  <%= form_with url: tool_calls_path, method: :get, class: "sl-page-header__actions", style: "justify-content: flex-start; flex-wrap: wrap;" do |f| %>
3
7
  <div style="display: flex; align-items: center; gap: 8px;">
@@ -6,7 +10,11 @@
6
10
  </div>
7
11
  <div style="display: flex; align-items: center; gap: 8px; margin-left: 20px;">
8
12
  <span class="sl-stat-card__label">Function:</span>
9
- <%= f.text_field :function_name, value: params[:function_name], class: "sl-btn sl-btn--small", placeholder: "list_files" %>
13
+ <%= f.select :function_name,
14
+ options_for_select([["All Functions", ""]] + @function_names.map { |name| [name, name] }, params[:function_name]),
15
+ {},
16
+ class: "sl-btn sl-btn--small",
17
+ onchange: "this.form.submit()" %>
10
18
  </div>
11
19
  <div style="display: flex; align-items: center; gap: 8px; margin-left: 20px;">
12
20
  <span class="sl-stat-card__label">Status:</span>
@@ -27,20 +35,27 @@
27
35
  <thead>
28
36
  <tr>
29
37
  <th class="sl-table__th">ID</th>
30
- <th class="sl-table__th">Message ID</th>
38
+ <th class="sl-table__th">Loop</th>
39
+ <th class="sl-table__th">Message</th>
31
40
  <th class="sl-table__th">Function</th>
32
41
  <th class="sl-table__th">Status</th>
42
+ <th class="sl-table__th">Duration</th>
33
43
  <th class="sl-table__th">Created At</th>
34
44
  <th class="sl-table__th">Actions</th>
35
45
  </tr>
36
46
  </thead>
37
47
  <tbody>
38
48
  <% @tool_calls.each do |tool_call| %>
49
+ <% loop_id = @loop_ids_by_message[tool_call.message_id] %>
39
50
  <tr>
40
51
  <td class="sl-table__td"><%= tool_call.id %></td>
52
+ <td class="sl-table__td">
53
+ <%= link_to "##{loop_id}", loop_path(loop_id) if loop_id %>
54
+ </td>
41
55
  <td class="sl-table__td"><%= link_to tool_call.message_id, message_path(tool_call.message_id) if tool_call.message_id %></td>
42
- <td class="sl-table__td"><%= tool_call.function_name %></td>
56
+ <td class="sl-table__td"><strong><%= tool_call.function_name %></strong></td>
43
57
  <td class="sl-table__td"><span class="sl-badge sl-badge--<%= tool_call.error_message.present? ? 'error' : 'success' %>"><%= tool_call.error_message.present? ? 'failed' : 'success' %></span></td>
58
+ <td class="sl-table__td"><%= sl_duration_cell(tool_call.duration, scale: :tool) %></td>
44
59
  <td class="sl-table__td"><%= tool_call.created_at.strftime("%Y-%m-%d %H:%M") %></td>
45
60
  <td class="sl-table__td">
46
61
  <%= link_to "View", tool_call_path(tool_call), class: "sl-btn sl-btn--small" %>
@@ -0,0 +1,22 @@
1
+ class SolidLoopAddRetentionIndexes < ActiveRecord::Migration[7.1]
2
+ # Retention sweeps select by age. Without these, `SolidLoop.prune!` seq-scans
3
+ # exactly the tables it exists to keep small — the more overdue the cleanup,
4
+ # the more expensive the query that performs it.
5
+ #
6
+ # Built CONCURRENTLY (hence no DDL transaction): these tables are append-only
7
+ # hot paths — every LLM turn and every inbound MCP request writes one — and an
8
+ # ACCESS EXCLUSIVE lock for the duration of a plain index build would stall
9
+ # agents on a table that is, by construction, already large by the time an
10
+ # operator reaches for retention.
11
+ disable_ddl_transaction!
12
+
13
+ def change
14
+ add_index :solid_loop_events, :created_at,
15
+ name: "index_solid_loop_events_on_created_at",
16
+ algorithm: :concurrently, if_not_exists: true
17
+
18
+ add_index :solid_loop_mcp_inbound_sessions, :created_at,
19
+ name: "index_solid_loop_mcp_inbound_sessions_on_created_at",
20
+ algorithm: :concurrently, if_not_exists: true
21
+ end
22
+ end
@@ -40,25 +40,25 @@ Use `make ci` when you want the closest local approximation of the full gem vali
40
40
 
41
41
  ```bash
42
42
  # staged changes only
43
- rails 'coverage:report[worst,,staged]'
43
+ bundle exec rake 'coverage:report[worst,,staged]'
44
44
 
45
45
  # all branch changes since main
46
- rails 'coverage:report[worst,,branch]'
46
+ bundle exec rake 'coverage:report[worst,,branch]'
47
47
 
48
48
  # last commit
49
- rails 'coverage:report[worst,,head]'
49
+ bundle exec rake 'coverage:report[worst,,head]'
50
50
  ```
51
51
 
52
52
  ## Explore Weak Spots
53
53
 
54
54
  ```bash
55
- rails 'coverage:report[partial]'
56
- rails 'coverage:report[partial,service]'
57
- rails 'coverage:report[worst,controller]'
58
- rails coverage:report
55
+ bundle exec rake 'coverage:report[partial]'
56
+ bundle exec rake 'coverage:report[partial,service]'
57
+ bundle exec rake 'coverage:report[worst,controller]'
58
+ bundle exec rake coverage:report
59
59
  ```
60
60
 
61
61
  ## Source Of Truth
62
62
 
63
- - Task implementation: `lib/tasks/coverage.rake`
63
+ - Task implementation: `tasks/coverage.rake`
64
64
  - Coverage artifact: `coverage/coverage.json`
@@ -160,8 +160,10 @@ Lifecycle:
160
160
  - `DELETE` with a session header → session **terminated, row kept**
161
161
  (`terminated_at`), `204`. Destroying the row would cascade into its events —
162
162
  a client must not be able to erase its own audit trail with one request.
163
- - No TTL/eviction in 0.0.4; rows are cheap and admin-visible. Retention can
164
- ride the 0.0.5 storage work.
163
+ - No TTL/eviction in 0.0.4; rows are cheap and admin-visible. 0.0.5 adds
164
+ age-based retention: with `config.mcp_inbound_session_retention` set,
165
+ `SolidLoop::Janitor` deletes sessions older than that by `created_at`, and
166
+ their events with them. It stays `nil` (keep everything) by default.
165
167
 
166
168
  ## Observability
167
169
 
@@ -14,23 +14,81 @@ If the reasoning block from Turn N is not sent back to the model in Turn N+1, th
14
14
 
15
15
  `SolidLoop` implements a flexible **Reasoning Protocol** to handle the inconsistent landscape of LLM providers. Instead of a single "hack," we use an **array of strategies** configured per agent/model. This allows combining multiple methods to ensure the model receives its thoughts regardless of how strict or fragmented the provider's API is.
16
16
 
17
+ ### Choosing a default
18
+
19
+ The default is `nil` on `SolidLoop::Base#reasoning_strategies`, which means "ask
20
+ the dialect". `open_ai` answers `["reasoning_content"]`, `anthropic`
21
+ `[:anthropic_signed]`, `gemini` `[:gemini_signed]`.
22
+
23
+ `:xml` was the default for every dialect until 0.0.5, on the reasoning that it
24
+ is the shape no provider can strip. That is true and it is also expensive:
25
+ welded reasoning is part of the assistant message body, so it is billed as
26
+ prompt tokens on every later turn and compounds over a run.
27
+
28
+ Measured with vLLM's `/tokenize` on the Qwen3 chat template, one six-message
29
+ conversation:
30
+
31
+ ```
32
+ <think> welded into content 1438 tokens
33
+ reasoning_content field 108
34
+ reasoning omitted entirely 108
35
+ ```
36
+
37
+ `reasoning_content` and omitting the reasoning altogether cost the same, because
38
+ that template drops the field from history — which is what Qwen3's own guidance
39
+ asks for ("historical model output should only include the final output part").
40
+
41
+ That equality has to be read carefully: the two encodings are not the same
42
+ request written differently. Welding puts the old thoughts back in front of the
43
+ model as ordinary text; the field, on this template, does not. Choosing the
44
+ field is therefore a decision to stop replaying reasoning on this model family,
45
+ which is what its authors recommend — not a free compression. The saving is
46
+ large: a long agent loop can end up spending a large fraction of its context
47
+ window on repacked reasoning. Whether a given model does better with or without
48
+ its old thoughts is a question for a run, not for a token count.
49
+
50
+ This is why the choice belongs to the dialect rather than to one global default.
51
+ Each one now answers with the shape its provider actually has a slot for:
52
+ `"reasoning_content"` on `open_ai`, `:anthropic_signed` on `anthropic`,
53
+ `:gemini_signed` on `gemini`.
54
+
55
+ The two signed shapes carry a provider-issued signature that is verified when
56
+ the reasoning comes back — that is what makes a replayed thought provably the
57
+ model's own instead of ordinary text that has to be re-billed. They are
58
+ therefore all-or-nothing: a message with no captured signature sends **no
59
+ reasoning**, because the alternative is a block the provider rejects. Loops that
60
+ ran before the dialects started storing signatures lose their old thoughts on
61
+ the next turn. That is the intended trade — the alternative was leaving both
62
+ dialects on the shape that costs 13× the tokens. A host that would rather keep
63
+ the text can write `[:anthropic_signed, :xml]`: the `:xml` fallback fires
64
+ exactly when the signed strategy could not.
65
+
66
+ The measurement is template-specific. Another chat template may render
67
+ `reasoning_content` into the prompt instead of dropping it, in which case the
68
+ token saving disappears (the cleaner wire shape does not). Re-run the three-way
69
+ comparison against a new model before assuming it carries over.
70
+
17
71
  ### Supported Strategies:
18
72
 
19
73
  * **`:xml` (The DeepSeek/Reliability Standard):**
20
- Wraps the reasoning content in `<think>...</think>` tags and prepends it to the main `content` string. This is the most robust method as it bypasses API field stripping.
74
+ Wraps the reasoning content in `<think>...</think>` tags and prepends it to the main `content` string. This is the most robust method as it bypasses API field stripping — and the most expensive, since the wrapped text becomes part of the message body and is billed as prompt tokens on every later turn. Reach for it when the provider strips fields, not by default.
21
75
  *Note: In `SolidLoop`, this is a "Smart" strategy. It will automatically suppress itself if a native strategy (like `:gemini_signed`) is also present, avoiding redundant context.*
22
76
  * **`:xml!` (Forced XML):**
23
77
  Same as `:xml`, but **ignores** native strategy suppression. Use this if you want to ensure thoughts are visible in the main context even if the provider supports native fields.
24
78
  * **`:gemini_signed` (The Google Standard):**
25
- Passes both the `thought` text and the `thought_signature` (stored in JSONB metadata) back in specific fields required by Gemini 2.0+ for multi-turn reasoning.
79
+ Sends the thought back as a Gemini part — `{ thought: true, text: "...", thoughtSignature: "..." }`. Note that `thought` is a **boolean flag** on the part and the summary text lives in `text`, the same key an ordinary answer uses; the signature rides on a part too (the thought part, or the `functionCall` part on a tool-calling turn), never on the candidate. Drops the reasoning when no signature was captured.
80
+ * **`:anthropic_signed` (The Anthropic Standard):**
81
+ Sends the thought back as a `{ type: "thinking", thinking: "...", signature: "..." }` content block, first in the assistant message's content array. Anthropic verifies the signature, so this drops the reasoning when none was captured.
26
82
  * **`"field_name"` (Explicit Mapping):**
27
83
  Allows copying the reasoning content into a specific JSON key (e.g., `"reasoning_content"`, `"thought"`, or `"internal_thinking"`) to satisfy various provider requirements (OpenRouter, vLLM, Groq).
28
84
 
29
85
  ### Example Combinations:
30
86
 
31
- * **`[:xml]`**: Maximum reliability for DeepSeek or OpenRouter (when field stripping is suspected).
87
+ * **`nil` (the default)**: Let the dialect choose `"reasoning_content"` on `open_ai`, `:anthropic_signed` on `anthropic`, `:gemini_signed` on `gemini`.
88
+ * **`[:xml]`**: Maximum reliability for DeepSeek or OpenRouter (when field stripping is suspected), at full prompt-token price.
32
89
  * **`["reasoning", "reasoning_content"]`**: Populating multiple fields in parallel — useful when the provider's accepted field name is uncertain.
33
90
  * **`[:gemini_signed, "thought", "reasoning_content"]`**: A comprehensive strategy for Gemini-compatible providers that might also look for OpenAI-style fields.
91
+ * **`[:anthropic_signed, :xml]`**: Signed blocks where a signature exists, welded `<think>` where one does not — the `:xml` fallback fires only when the signed strategy could not.
34
92
  * **`[]` (Empty Array)**: Standard for OpenAI o1/o3 (reasoning is stripped to avoid 400 errors).
35
93
 
36
94
  ## 3. The Reasoning Verifier *(not implemented)*
@@ -45,7 +103,65 @@ To ensure an agent is actually "hearing its own thoughts," a **Reasoning Verific
45
103
 
46
104
  Worth doing — it would give a cheap, repeatable way to verify that `ReasoningPacker` is actually working end-to-end with a given provider/strategy combination, rather than assuming it does.
47
105
 
48
- ## 4. Data Storage
106
+ ## 4. Measuring the Cost
107
+
108
+ `ReasoningPacker.pack_all` reports what each request's prompt is carrying, and
109
+ the adapter stamps it onto the assistant message that request produced:
110
+
111
+ ```ruby
112
+ message.metadata["reasoning_carried"]
113
+ # => { "messages" => 6, "chars" => 84616, "inline_chars" => 84616, "tokens_est" => 21154 }
114
+ ```
115
+
116
+ `chars` is everything sent back in any shape; `inline_chars` is only the part
117
+ welded into `content`, which every provider bills. Without it the cost has to be
118
+ reconstructed from message lengths after the fact, which is how a loop reaches
119
+ its window ceiling without anyone seeing why. codex tracks the same number for
120
+ the same reason (`context_manager/history.rs` sums the tokens of every reasoning
121
+ item before the last user message).
122
+
123
+ ## 5. Why There Is No Trimming Strategy
124
+
125
+ The obvious way to bound a growing context is to stop resending old reasoning —
126
+ keep the newest `n` turns, drop the rest. SolidLoop deliberately offers no such
127
+ strategy, because it is a trap.
128
+
129
+ A prefix cache exists because turn N's prompt is turn N-1's plus a new tail: the
130
+ server re-uses the KV of everything before it and prefills only the tail. A
131
+ sliding retained window rewrites the *middle* of the prompt instead — the
132
+ message that ages out changes, the prompt diverges at its position, and
133
+ everything after it is prefilled again, every turn. Measured on llama.cpp
134
+ (Qwen3.8-27B Q6_K, prefix caching on), an 8-step agent chain:
135
+
136
+ | strategy | prompt | fresh/turn | prefill/turn | cached/turn |
137
+ |---|---|---|---|---|
138
+ | full reasoning | 4667 | 576 | 910 ms | 4091 |
139
+ | newest 2 turns | 1541 | 1503 | 1847 ms | 38 |
140
+ | send none | 499 | 59 | 500 ms | 440 |
141
+
142
+ Trimming is not the problem; *sliding* is. The trimmed run costs twice the
143
+ prefill of the untrimmed history it was meant to be cheaper than, and degrades
144
+ the model's context as well — the worst of both. Both stable rules keep the
145
+ cache: with the full history each reasoning block is prefilled exactly once,
146
+ ever; with none, no block is.
147
+
148
+ Neither reference harness trims per turn. codex keeps reasoning whole and only
149
+ *accounts* for it (`get_non_last_reasoning_items_tokens()` counts the items
150
+ before the last instruction boundary for estimation, and removes nothing); its
151
+ boundaries move only when a user message arrives. opencode compacts on overflow
152
+ (`session/compaction.ts`, `PRUNE_MINIMUM = 20_000`, `PRUNE_PROTECT = 40_000`):
153
+ one pass that summarises the old history into a single assistant message and
154
+ keeps a recent tail within a budget, after which the prompt is monotonically
155
+ extending again. The Qwen3 GGUF chat template encodes the same instinct — its
156
+ `preserve_thinking` flag cuts only before `ns.last_query_index`, a user-turn
157
+ boundary, never per turn.
158
+
159
+ **Compaction is the missing piece**, and it is bigger than a strategy symbol: it
160
+ needs a summariser, a checkpoint message kind, and a rule for when to fire.
161
+ Until it exists, the two supported positions are "carry the reasoning and let
162
+ the prefix cache pay for it" and "send none".
163
+
164
+ ## 6. Data Storage
49
165
 
50
166
  * **Reasoning Text:** Stored in the `reasoning_content` column of `solid_loop_messages`.
51
167
  * **Signatures & Metadata:** Stored in the `metadata` JSONB column. This ensures we can support future tokens like `thought_signature` without schema migrations.
@@ -50,6 +50,6 @@ A Dialect object must implement the following methods to be compatible with the
50
50
 
51
51
  1. **`completion_url(base_url)`**: Returns the specific API endpoint.
52
52
  2. **`render_payload(universal_payload)`**: (Optional) Transforms the payload before sending.
53
- 3. **`apply_reasoning_strategies!(messages, strategies)`**: Formats message history according to the [Reasoning Protocol](./reasoning_persistence.md).
53
+ 3. **`apply_reasoning_strategies!(messages, strategies)`**: Formats message history according to the [Reasoning Protocol](../decisions/reasoning_persistence.md). A `strategies` of `nil` means the agent left the choice to the dialect — substitute your own `default_reasoning_strategies`. Return the carry report from `ReasoningPacker.pack_all`; the adapter stamps it onto the message as `metadata["reasoning_carried"]`.
54
54
  4. **`extract_message_data(raw_json)`**: Extracts content and tools from a streaming chunk or partial response.
55
55
  5. **`normalize_response(raw_json)`**: Converts a full API response into the SolidLoop universal format (including token usage and metadata).
@@ -184,6 +184,78 @@ Semantics:
184
184
  - Toolsets are stateless by default (`session_id` = nil → synthetic). A toolset
185
185
  may override `on_initialize(params, context)` to mint and later receive a
186
186
  real session id.
187
+ - **`serverInfo.version`** defaults to the gem's version, which is rarely what a
188
+ mounted server means — semantically it is the version of *your* MCP server.
189
+ Declare it next to the name:
190
+
191
+ ```ruby
192
+ class SearchTools < SolidLoop::Mcp::Toolset
193
+ server_name "search"
194
+ server_version "1.0.0"
195
+ end
196
+ ```
197
+
198
+ - **`ping`** is answered out of the box with an empty result, as the base
199
+ protocol requires. Unknown methods still get `-32601`.
200
+ - **Error codes** follow the MCP specification, which prescribes **`-32602`**
201
+ (invalid params) for an unknown tool or prompt on `tools/call` / `prompts/get`
202
+ — not `-32601`. `-32601` is reserved for an unknown *method*.
203
+
204
+ ### Dynamic tool and prompt catalogs
205
+
206
+ The `tool` / `prompt` DSL registers at class-load time. A host whose catalog
207
+ lives in the database and changes without a deploy overrides four instance
208
+ methods instead — they default to the DSL and are consulted on every request
209
+ (and by `capabilities`), so a row added a second ago is advertised on the next
210
+ `prompts/list`:
211
+
212
+ | Method | Default | Used by |
213
+ | --- | --- | --- |
214
+ | `available_tools` | `self.class.tools` | `tools/list` |
215
+ | `find_tool(name)` | lookup in `available_tools` | `tools/call` |
216
+ | `available_prompts` | `self.class.prompts` | `prompts/list`, `capabilities` |
217
+ | `find_prompt(name)` | lookup in `available_prompts` | `prompts/get` |
218
+
219
+ ```ruby
220
+ class SearchTools < SolidLoop::Mcp::Toolset
221
+ def available_prompts
222
+ SavedPrompt.active.each_with_object({}) do |row, acc|
223
+ acc[row.name] = PromptDefinition.new(
224
+ name: row.name, description: row.description, arguments: row.arguments,
225
+ handler: ->(args, _ctx) { format(row.body, args) }
226
+ )
227
+ end
228
+ end
229
+ end
230
+ ```
231
+
232
+ Return `{}` for "none", and keep the override **total**: `capabilities` calls
233
+ `available_prompts` during `initialize`, so a raising override fails the
234
+ handshake rather than one request.
235
+
236
+ ### Overriding `deliver`
237
+
238
+ `deliver` is the documented seam for answering a method the gem does not model.
239
+ The envelope builders it needs are **public API** and covered by the
240
+ compatibility promise: `success_envelope(request, result)`,
241
+ `error_envelope(request, code, message)`, `json_boundary(value)` and
242
+ `tool_result(value)`.
243
+
244
+ ```ruby
245
+ def deliver(payload, session_id: nil, context: nil)
246
+ request = json_boundary(payload)
247
+ return super unless request["method"] == "resources/list"
248
+
249
+ SolidLoop::Mcp::Result.new(
250
+ body: success_envelope(request, "resources" => []),
251
+ session_id: session_id, raw_request: payload, raw_response: nil, status: 200
252
+ )
253
+ end
254
+ ```
255
+
256
+ Prefer the dynamic-catalog hooks above when the need is "my prompts/tools come
257
+ from somewhere else" — overriding `deliver` for that intercepts the protocol to
258
+ solve a catalog problem, and has to be re-checked against every gem upgrade.
187
259
  - **Transactions:** the block runs on the caller's thread and shares the job's
188
260
  ActiveRecord connection. That makes a tool atomic with SolidLoop's own writes
189
261
  — and makes `transaction do ... end` inside a tool a *nested* transaction.
@@ -0,0 +1,85 @@
1
+ # Validation
2
+
3
+ What SolidLoop has been run against before a release, and what it has not.
4
+
5
+ A gem that drives other people's inference budgets should be specific about the
6
+ shape of its own evidence, including the gaps. This page is that statement, kept
7
+ current per release.
8
+
9
+ ## The automated suite
10
+
11
+ Unit, integration and request specs run against a PostgreSQL dummy app on Rails
12
+ 7.1, 7.2 and 8.1 (see [Testing](../README.md#testing)). The LLM and MCP peers
13
+ are stubbed by emulators that speak each provider's wire format.
14
+
15
+ The limit of that is worth saying plainly: **an emulator encodes our own reading
16
+ of a provider's wire format.** Where the reading is wrong, the emulator is wrong
17
+ in the same direction and the suite stays green. That is exactly what had
18
+ happened to the `gemini` dialect before 0.0.5 — it had misread thought parts
19
+ since it was written, and every test passed. Emulators catch regressions; they
20
+ cannot catch a shared misconception.
21
+
22
+ ## 0.0.5 — local inference, sustained agentic load
23
+
24
+ Before this release SolidLoop ran as the engine of a private agent application:
25
+ batches of loops in parallel, each one a multi-turn coding task using MCP tools,
26
+ repeated across many runs. The inference side was entirely local:
27
+
28
+ | | |
29
+ | --- | --- |
30
+ | Servers | llama.cpp (`--jinja`, `--kv-unified`) and vLLM |
31
+ | Model | Qwen3 27B — GGUF Q6_K on llama.cpp, NVFP4 on vLLM |
32
+ | Hardware | 2 × NVIDIA RTX 5060 Ti, tensor-parallel |
33
+ | Dialect | `open_ai` (both servers expose an OpenAI-compatible API) |
34
+
35
+ Local OpenAI-compatible servers are a **stricter** test surface than the hosted
36
+ API, which is why this is worth reporting rather than apologising for:
37
+
38
+ - **They are unforgiving about malformed messages.** After a rejected tool call
39
+ SolidLoop was emitting a tool message with `"name": null`. vLLM ignores it;
40
+ llama.cpp rejects the entire request, and because the offending message stays
41
+ in history the loop then fails on every retry rather than recovering. Hosted
42
+ providers had never surfaced it. Fixed in 0.0.5.
43
+ - **Their chat templates are inspectable.** What a dialect sends back can be
44
+ traced to an actual prompt-token count through `/tokenize`, which is how the
45
+ reasoning transport in 0.0.5 was chosen rather than guessed — see
46
+ [reasoning_persistence.md](decisions/reasoning_persistence.md).
47
+ - **KV-cache and slot pressure are real and visible**, so context-budget defects
48
+ present as truncated tool-call arguments and traceable server logs instead of
49
+ an opaque 400.
50
+
51
+ The application's own task results are not part of this claim — those measure
52
+ the model, not the gem. What the runs establish about SolidLoop is narrower:
53
+ loops ran concurrently to completion, leases and the reaper held across worker
54
+ restarts, no turn was silently dropped, and the failures that did occur were
55
+ traced to specific defects that now have specs and changelog entries.
56
+
57
+ ## What has not been tested
58
+
59
+ - **The native `anthropic` and `gemini` dialects have never made a live call.**
60
+ Their payload shapes come from the published API documentation and are covered
61
+ by unit tests and emulators, but no real key has exercised them. 0.0.5 changes
62
+ the default reasoning transport for both to the providers' signed block
63
+ formats, so that untested path is now the default path for those two dialects.
64
+ If you are on `anthropic` or `gemini` and want the pre-0.0.5 behaviour while
65
+ you evaluate, set `reasoning_strategies` to `[:xml]`.
66
+ - **Hosted providers were last exercised in 0.0.4**, through the `open_ai`
67
+ dialect against OpenAI- and OpenRouter-served models in a pilot Rails
68
+ application. 0.0.5 has not repeated that run — its provider coverage is the
69
+ local stand above.
70
+ - **Scale.** SolidLoop has not been run at a size where Postgres or the job
71
+ backend was the bottleneck, nor over horizons measured in days. The durability
72
+ design ([durable_attempt_lease.md](decisions/durable_attempt_lease.md)) is
73
+ exercised by specs and by deliberate mid-run restarts, not by a long
74
+ production soak.
75
+
76
+ ## Reports are welcome
77
+
78
+ Running SolidLoop against a provider, model, or scale that is not on this page
79
+ is a genuine contribution, whether or not it comes with a patch. Useful in a
80
+ report: the dialect and provider, the model, whether reasoning came back and in
81
+ which field, and the relevant wire-log rows — `EventLogging` redacts
82
+ credentials, including the API key Gemini carries in the URL, so those rows are
83
+ safe to paste.
84
+
85
+ See [CONTRIBUTING.md](../CONTRIBUTING.md).