solid_loop 0.0.4 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +445 -0
- data/README.md +305 -4
- data/Rakefile +5 -4
- data/app/controllers/solid_loop/application_controller.rb +6 -0
- data/app/controllers/solid_loop/dashboard_controller.rb +167 -22
- data/app/controllers/solid_loop/events_controller.rb +9 -1
- data/app/controllers/solid_loop/mcp_sessions_controller.rb +12 -0
- data/app/controllers/solid_loop/messages_controller.rb +11 -1
- data/app/controllers/solid_loop/tool_calls_controller.rb +32 -0
- data/app/helpers/solid_loop/application_helper.rb +4 -1
- data/app/helpers/solid_loop/metrics_helper.rb +232 -0
- data/app/jobs/solid_loop/janitor_job.rb +24 -0
- data/app/jobs/solid_loop/llm_completion_job.rb +2 -2
- data/app/models/solid_loop/base.rb +89 -6
- data/app/models/solid_loop/loop.rb +22 -0
- data/app/models/solid_loop/message.rb +33 -7
- data/app/models/solid_loop/tool_call.rb +7 -1
- data/app/services/solid_loop/adapters/native.rb +156 -23
- data/app/services/solid_loop/dialects/anthropic.rb +43 -10
- data/app/services/solid_loop/dialects/gemini.rb +55 -15
- data/app/services/solid_loop/dialects/open_ai.rb +17 -5
- data/app/services/solid_loop/dialects/reasoning_packer.rb +72 -7
- data/app/services/solid_loop/llm_usage_parser/llama.rb +18 -3
- data/app/services/solid_loop/mcp_session_initializer.rb +1 -0
- data/app/services/solid_loop/middlewares/agent_initialization.rb +1 -1
- data/app/services/solid_loop/middlewares/error_handling.rb +5 -1
- data/app/services/solid_loop/middlewares/event_logging.rb +3 -3
- data/app/services/solid_loop/middlewares/message_building.rb +38 -4
- data/app/services/solid_loop/middlewares/response_parsing.rb +19 -3
- data/app/views/layouts/solid_loop/admin.html.erb +134 -23
- data/app/views/solid_loop/dashboard/index.html.erb +330 -41
- data/app/views/solid_loop/events/index.html.erb +17 -1
- data/app/views/solid_loop/loops/index.html.erb +40 -3
- data/app/views/solid_loop/loops/show.html.erb +1 -1
- data/app/views/solid_loop/mcp_sessions/index.html.erb +40 -2
- data/app/views/solid_loop/messages/_message.html.erb +26 -35
- data/app/views/solid_loop/messages/index.html.erb +16 -2
- data/app/views/solid_loop/tool_calls/index.html.erb +18 -3
- data/db/migrate/20260819000100_solid_loop_add_retention_indexes.rb +22 -0
- data/docs/contributing/coverage.md +8 -8
- data/docs/decisions/mcp-server.md +4 -2
- data/docs/decisions/reasoning_persistence.md +120 -4
- data/docs/guides/dialects.md +1 -1
- data/docs/guides/mcp_transports.md +72 -0
- data/docs/validation.md +85 -0
- data/lib/solid_loop/configuration.rb +93 -0
- data/lib/solid_loop/engine.rb +8 -0
- data/lib/solid_loop/janitor.rb +110 -0
- data/lib/solid_loop/mcp/toolset.rb +119 -31
- data/lib/solid_loop/pipeline/builder.rb +31 -10
- data/lib/solid_loop/redaction.rb +26 -0
- data/lib/solid_loop/version.rb +1 -1
- data/lib/solid_loop.rb +47 -0
- metadata +7 -2
- data/lib/tasks/coverage.rake +0 -206
|
@@ -1,73 +1,64 @@
|
|
|
1
1
|
<div id="sl-message-<%= message.id %>" class="sl-card sl-message sl-message--<%= message.role %>">
|
|
2
2
|
<div class="sl-message__header">
|
|
3
|
-
<strong class="sl-message__role">
|
|
4
|
-
<%= message.role %>
|
|
5
|
-
</strong>
|
|
3
|
+
<strong class="sl-message__role"><%= message.role %></strong>
|
|
6
4
|
<% model_name = message.metadata&.dig("model") %>
|
|
7
5
|
<% if model_name.present? %>
|
|
8
|
-
<span class="sl-badge sl-badge--pending" title="Model that served this turn"
|
|
9
|
-
<%= model_name %>
|
|
10
|
-
</span>
|
|
6
|
+
<span class="sl-badge sl-badge--pending sl-message__model" title="Model that served this turn"><%= model_name %></span>
|
|
11
7
|
<% end %>
|
|
12
|
-
<span class="sl-message__time">
|
|
13
|
-
<%= message.created_at.strftime("%H:%M:%S") %>
|
|
14
|
-
</span>
|
|
8
|
+
<span class="sl-message__time"><%= message.created_at.strftime("%H:%M:%S") %></span>
|
|
15
9
|
</div>
|
|
16
10
|
|
|
11
|
+
<%# Each block below owns its own <pre>. The container must NOT be pre-wrap:
|
|
12
|
+
it would render this template's own indentation and blank lines as content,
|
|
13
|
+
which is what used to indent the first line and open gaps before the footer. %>
|
|
17
14
|
<div class="sl-json">
|
|
18
15
|
<% if message.content.present? %>
|
|
19
|
-
|
|
16
|
+
<pre class="sl-pre-wrap sl-message__body"><%= message.content %></pre>
|
|
20
17
|
<% end %>
|
|
21
18
|
|
|
22
19
|
<% if message.tool_calls_raw.present? || message.tool_calls.any? %>
|
|
23
20
|
<div class="sl-message__tools">
|
|
24
|
-
<
|
|
25
|
-
|
|
26
|
-
<%
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
<% else %>
|
|
35
|
-
<%= name %>
|
|
36
|
-
<% end %>
|
|
37
|
-
</div>
|
|
38
|
-
<% end %>
|
|
39
|
-
</div>
|
|
21
|
+
<span class="sl-message__tools-label">Tools</span>
|
|
22
|
+
<% (message.tool_calls.any? ? message.tool_calls : message.tool_calls_raw).each do |tc| %>
|
|
23
|
+
<%
|
|
24
|
+
name = tc.is_a?(SolidLoop::ToolCall) ? tc.function_name : tc.dig("function", "name")
|
|
25
|
+
tc_id = tc.is_a?(SolidLoop::ToolCall) ? tc.id : nil
|
|
26
|
+
%>
|
|
27
|
+
<span class="sl-badge <%= tc.is_a?(SolidLoop::ToolCall) && tc.is_success ? 'sl-badge--success' : 'sl-badge--pending' %>">
|
|
28
|
+
<% if tc_id %><%= link_to name, tool_call_path(tc_id), class: "sl-badge__link" %><% else %><%= name %><% end %>
|
|
29
|
+
</span>
|
|
30
|
+
<% end %>
|
|
40
31
|
</div>
|
|
41
32
|
<% end %>
|
|
42
33
|
|
|
43
34
|
<% if message.reasoning_content.present? %>
|
|
44
|
-
<
|
|
45
|
-
<
|
|
35
|
+
<details class="sl-message__reasoning">
|
|
36
|
+
<summary class="sl-message__reasoning-toggle">Thinking (<%= message.reasoning_content.length %> chars)</summary>
|
|
46
37
|
<pre class="sl-pre-wrap"><%= message.reasoning_content %></pre>
|
|
47
|
-
</
|
|
38
|
+
</details>
|
|
48
39
|
<% end %>
|
|
49
40
|
|
|
50
41
|
<div class="sl-message__footer">
|
|
51
42
|
<div class="sl-message__footer-item">
|
|
52
|
-
Tokens
|
|
43
|
+
Tokens <span class="sl-message__footer-value">P<%= message.tokens_prompt %>+C<%= message.tokens_completion %></span>
|
|
53
44
|
<% if message.tokens_prompt_cached > 0 %>
|
|
54
|
-
<span
|
|
45
|
+
<span class="sl-message__cached">(cached <%= message.tokens_prompt_cached %>)</span>
|
|
55
46
|
<% end %>
|
|
56
47
|
</div>
|
|
57
48
|
<div class="sl-message__footer-item">
|
|
58
|
-
Cost
|
|
49
|
+
Cost <span class="sl-message__footer-value"><%= sl_money(message.cost) %></span>
|
|
59
50
|
</div>
|
|
60
51
|
<% if message.tps > 0 %>
|
|
61
52
|
<div class="sl-message__footer-item">
|
|
62
|
-
TPS
|
|
53
|
+
TPS <span class="sl-message__footer-value"><%= message.tps %></span>
|
|
63
54
|
</div>
|
|
64
55
|
<% end %>
|
|
65
56
|
<% if message.ttft > 0 %>
|
|
66
57
|
<div class="sl-message__footer-item">
|
|
67
|
-
TTFT
|
|
58
|
+
TTFT <span class="sl-message__footer-value"><%= number_with_precision(message.ttft, precision: 2) %>s</span>
|
|
68
59
|
</div>
|
|
69
60
|
<% end %>
|
|
70
|
-
<div class="sl-message__footer-item
|
|
61
|
+
<div class="sl-message__footer-item sl-message__footer-item--end">
|
|
71
62
|
<%= link_to "Details", message_path(message), class: "sl-btn sl-btn--small" %>
|
|
72
63
|
</div>
|
|
73
64
|
</div>
|
|
@@ -66,8 +66,22 @@
|
|
|
66
66
|
%>"><%= message.role.upcase %></span>
|
|
67
67
|
</td>
|
|
68
68
|
<td class="sl-table__td"><%= message.metadata&.dig("model").presence || "—" %></td>
|
|
69
|
-
<td class="sl-table__td">
|
|
70
|
-
|
|
69
|
+
<td class="sl-table__td">
|
|
70
|
+
<%# `tokens_total` is the persisted figure the loop's budget is checked
|
|
71
|
+
against; the in/out split is the detail, so it goes underneath
|
|
72
|
+
rather than being the only thing shown ("P0+C0" told you nothing).
|
|
73
|
+
A message still being streamed has its completion tokens written by
|
|
74
|
+
`trigger_content_append` but no total yet, so fall back to the sum
|
|
75
|
+
rather than printing "—" next to a non-zero breakdown. %>
|
|
76
|
+
<% total = message.tokens_total.to_i.positive? ? message.tokens_total.to_i : message.tokens_prompt.to_i + message.tokens_completion.to_i %>
|
|
77
|
+
<span class="sl-metric"><%= sl_tokens(total) %></span>
|
|
78
|
+
<% if message.tokens_prompt.to_i.positive? || message.tokens_completion.to_i.positive? %>
|
|
79
|
+
<span class="sl-metric__sub">
|
|
80
|
+
in <%= number_with_delimiter(message.tokens_prompt.to_i) %> · out <%= number_with_delimiter(message.tokens_completion.to_i) %><%= " · cached #{number_with_delimiter(message.tokens_prompt_cached.to_i)}" if message.tokens_prompt_cached.to_i.positive? %>
|
|
81
|
+
</span>
|
|
82
|
+
<% end %>
|
|
83
|
+
</td>
|
|
84
|
+
<td class="sl-table__td"><span class="sl-metric <%= "sl-metric--blank" unless message.cost.to_f.positive? %>"><%= sl_money(message.cost) %></span></td>
|
|
71
85
|
<td class="sl-table__td"><%= message.created_at.strftime("%Y-%m-%d %H:%M") %></td>
|
|
72
86
|
<td class="sl-table__td" style="text-align: right;">
|
|
73
87
|
<%= link_to "View", message_path(message), class: "sl-btn sl-btn--small" %>
|
|
@@ -1,3 +1,7 @@
|
|
|
1
|
+
<div class="sl-page-header">
|
|
2
|
+
<h2>Tool Calls</h2>
|
|
3
|
+
</div>
|
|
4
|
+
|
|
1
5
|
<div class="sl-card">
|
|
2
6
|
<%= form_with url: tool_calls_path, method: :get, class: "sl-page-header__actions", style: "justify-content: flex-start; flex-wrap: wrap;" do |f| %>
|
|
3
7
|
<div style="display: flex; align-items: center; gap: 8px;">
|
|
@@ -6,7 +10,11 @@
|
|
|
6
10
|
</div>
|
|
7
11
|
<div style="display: flex; align-items: center; gap: 8px; margin-left: 20px;">
|
|
8
12
|
<span class="sl-stat-card__label">Function:</span>
|
|
9
|
-
<%= f.
|
|
13
|
+
<%= f.select :function_name,
|
|
14
|
+
options_for_select([["All Functions", ""]] + @function_names.map { |name| [name, name] }, params[:function_name]),
|
|
15
|
+
{},
|
|
16
|
+
class: "sl-btn sl-btn--small",
|
|
17
|
+
onchange: "this.form.submit()" %>
|
|
10
18
|
</div>
|
|
11
19
|
<div style="display: flex; align-items: center; gap: 8px; margin-left: 20px;">
|
|
12
20
|
<span class="sl-stat-card__label">Status:</span>
|
|
@@ -27,20 +35,27 @@
|
|
|
27
35
|
<thead>
|
|
28
36
|
<tr>
|
|
29
37
|
<th class="sl-table__th">ID</th>
|
|
30
|
-
<th class="sl-table__th">
|
|
38
|
+
<th class="sl-table__th">Loop</th>
|
|
39
|
+
<th class="sl-table__th">Message</th>
|
|
31
40
|
<th class="sl-table__th">Function</th>
|
|
32
41
|
<th class="sl-table__th">Status</th>
|
|
42
|
+
<th class="sl-table__th">Duration</th>
|
|
33
43
|
<th class="sl-table__th">Created At</th>
|
|
34
44
|
<th class="sl-table__th">Actions</th>
|
|
35
45
|
</tr>
|
|
36
46
|
</thead>
|
|
37
47
|
<tbody>
|
|
38
48
|
<% @tool_calls.each do |tool_call| %>
|
|
49
|
+
<% loop_id = @loop_ids_by_message[tool_call.message_id] %>
|
|
39
50
|
<tr>
|
|
40
51
|
<td class="sl-table__td"><%= tool_call.id %></td>
|
|
52
|
+
<td class="sl-table__td">
|
|
53
|
+
<%= link_to "##{loop_id}", loop_path(loop_id) if loop_id %>
|
|
54
|
+
</td>
|
|
41
55
|
<td class="sl-table__td"><%= link_to tool_call.message_id, message_path(tool_call.message_id) if tool_call.message_id %></td>
|
|
42
|
-
<td class="sl-table__td"><%= tool_call.function_name %></td>
|
|
56
|
+
<td class="sl-table__td"><strong><%= tool_call.function_name %></strong></td>
|
|
43
57
|
<td class="sl-table__td"><span class="sl-badge sl-badge--<%= tool_call.error_message.present? ? 'error' : 'success' %>"><%= tool_call.error_message.present? ? 'failed' : 'success' %></span></td>
|
|
58
|
+
<td class="sl-table__td"><%= sl_duration_cell(tool_call.duration, scale: :tool) %></td>
|
|
44
59
|
<td class="sl-table__td"><%= tool_call.created_at.strftime("%Y-%m-%d %H:%M") %></td>
|
|
45
60
|
<td class="sl-table__td">
|
|
46
61
|
<%= link_to "View", tool_call_path(tool_call), class: "sl-btn sl-btn--small" %>
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
class SolidLoopAddRetentionIndexes < ActiveRecord::Migration[7.1]
|
|
2
|
+
# Retention sweeps select by age. Without these, `SolidLoop.prune!` seq-scans
|
|
3
|
+
# exactly the tables it exists to keep small — the more overdue the cleanup,
|
|
4
|
+
# the more expensive the query that performs it.
|
|
5
|
+
#
|
|
6
|
+
# Built CONCURRENTLY (hence no DDL transaction): these tables are append-only
|
|
7
|
+
# hot paths — every LLM turn and every inbound MCP request writes one — and an
|
|
8
|
+
# ACCESS EXCLUSIVE lock for the duration of a plain index build would stall
|
|
9
|
+
# agents on a table that is, by construction, already large by the time an
|
|
10
|
+
# operator reaches for retention.
|
|
11
|
+
disable_ddl_transaction!
|
|
12
|
+
|
|
13
|
+
def change
|
|
14
|
+
add_index :solid_loop_events, :created_at,
|
|
15
|
+
name: "index_solid_loop_events_on_created_at",
|
|
16
|
+
algorithm: :concurrently, if_not_exists: true
|
|
17
|
+
|
|
18
|
+
add_index :solid_loop_mcp_inbound_sessions, :created_at,
|
|
19
|
+
name: "index_solid_loop_mcp_inbound_sessions_on_created_at",
|
|
20
|
+
algorithm: :concurrently, if_not_exists: true
|
|
21
|
+
end
|
|
22
|
+
end
|
|
@@ -40,25 +40,25 @@ Use `make ci` when you want the closest local approximation of the full gem vali
|
|
|
40
40
|
|
|
41
41
|
```bash
|
|
42
42
|
# staged changes only
|
|
43
|
-
|
|
43
|
+
bundle exec rake 'coverage:report[worst,,staged]'
|
|
44
44
|
|
|
45
45
|
# all branch changes since main
|
|
46
|
-
|
|
46
|
+
bundle exec rake 'coverage:report[worst,,branch]'
|
|
47
47
|
|
|
48
48
|
# last commit
|
|
49
|
-
|
|
49
|
+
bundle exec rake 'coverage:report[worst,,head]'
|
|
50
50
|
```
|
|
51
51
|
|
|
52
52
|
## Explore Weak Spots
|
|
53
53
|
|
|
54
54
|
```bash
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
55
|
+
bundle exec rake 'coverage:report[partial]'
|
|
56
|
+
bundle exec rake 'coverage:report[partial,service]'
|
|
57
|
+
bundle exec rake 'coverage:report[worst,controller]'
|
|
58
|
+
bundle exec rake coverage:report
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
## Source Of Truth
|
|
62
62
|
|
|
63
|
-
- Task implementation: `
|
|
63
|
+
- Task implementation: `tasks/coverage.rake`
|
|
64
64
|
- Coverage artifact: `coverage/coverage.json`
|
|
@@ -160,8 +160,10 @@ Lifecycle:
|
|
|
160
160
|
- `DELETE` with a session header → session **terminated, row kept**
|
|
161
161
|
(`terminated_at`), `204`. Destroying the row would cascade into its events —
|
|
162
162
|
a client must not be able to erase its own audit trail with one request.
|
|
163
|
-
- No TTL/eviction in 0.0.4; rows are cheap and admin-visible.
|
|
164
|
-
|
|
163
|
+
- No TTL/eviction in 0.0.4; rows are cheap and admin-visible. 0.0.5 adds
|
|
164
|
+
age-based retention: with `config.mcp_inbound_session_retention` set,
|
|
165
|
+
`SolidLoop::Janitor` deletes sessions older than that by `created_at`, and
|
|
166
|
+
their events with them. It stays `nil` (keep everything) by default.
|
|
165
167
|
|
|
166
168
|
## Observability
|
|
167
169
|
|
|
@@ -14,23 +14,81 @@ If the reasoning block from Turn N is not sent back to the model in Turn N+1, th
|
|
|
14
14
|
|
|
15
15
|
`SolidLoop` implements a flexible **Reasoning Protocol** to handle the inconsistent landscape of LLM providers. Instead of a single "hack," we use an **array of strategies** configured per agent/model. This allows combining multiple methods to ensure the model receives its thoughts regardless of how strict or fragmented the provider's API is.
|
|
16
16
|
|
|
17
|
+
### Choosing a default
|
|
18
|
+
|
|
19
|
+
The default is `nil` on `SolidLoop::Base#reasoning_strategies`, which means "ask
|
|
20
|
+
the dialect". `open_ai` answers `["reasoning_content"]`, `anthropic`
|
|
21
|
+
`[:anthropic_signed]`, `gemini` `[:gemini_signed]`.
|
|
22
|
+
|
|
23
|
+
`:xml` was the default for every dialect until 0.0.5, on the reasoning that it
|
|
24
|
+
is the shape no provider can strip. That is true and it is also expensive:
|
|
25
|
+
welded reasoning is part of the assistant message body, so it is billed as
|
|
26
|
+
prompt tokens on every later turn and compounds over a run.
|
|
27
|
+
|
|
28
|
+
Measured with vLLM's `/tokenize` on the Qwen3 chat template, one six-message
|
|
29
|
+
conversation:
|
|
30
|
+
|
|
31
|
+
```
|
|
32
|
+
<think> welded into content 1438 tokens
|
|
33
|
+
reasoning_content field 108
|
|
34
|
+
reasoning omitted entirely 108
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
`reasoning_content` and omitting the reasoning altogether cost the same, because
|
|
38
|
+
that template drops the field from history — which is what Qwen3's own guidance
|
|
39
|
+
asks for ("historical model output should only include the final output part").
|
|
40
|
+
|
|
41
|
+
That equality has to be read carefully: the two encodings are not the same
|
|
42
|
+
request written differently. Welding puts the old thoughts back in front of the
|
|
43
|
+
model as ordinary text; the field, on this template, does not. Choosing the
|
|
44
|
+
field is therefore a decision to stop replaying reasoning on this model family,
|
|
45
|
+
which is what its authors recommend — not a free compression. The saving is
|
|
46
|
+
large: a long agent loop can end up spending a large fraction of its context
|
|
47
|
+
window on repacked reasoning. Whether a given model does better with or without
|
|
48
|
+
its old thoughts is a question for a run, not for a token count.
|
|
49
|
+
|
|
50
|
+
This is why the choice belongs to the dialect rather than to one global default.
|
|
51
|
+
Each one now answers with the shape its provider actually has a slot for:
|
|
52
|
+
`"reasoning_content"` on `open_ai`, `:anthropic_signed` on `anthropic`,
|
|
53
|
+
`:gemini_signed` on `gemini`.
|
|
54
|
+
|
|
55
|
+
The two signed shapes carry a provider-issued signature that is verified when
|
|
56
|
+
the reasoning comes back — that is what makes a replayed thought provably the
|
|
57
|
+
model's own instead of ordinary text that has to be re-billed. They are
|
|
58
|
+
therefore all-or-nothing: a message with no captured signature sends **no
|
|
59
|
+
reasoning**, because the alternative is a block the provider rejects. Loops that
|
|
60
|
+
ran before the dialects started storing signatures lose their old thoughts on
|
|
61
|
+
the next turn. That is the intended trade — the alternative was leaving both
|
|
62
|
+
dialects on the shape that costs 13× the tokens. A host that would rather keep
|
|
63
|
+
the text can write `[:anthropic_signed, :xml]`: the `:xml` fallback fires
|
|
64
|
+
exactly when the signed strategy could not.
|
|
65
|
+
|
|
66
|
+
The measurement is template-specific. Another chat template may render
|
|
67
|
+
`reasoning_content` into the prompt instead of dropping it, in which case the
|
|
68
|
+
token saving disappears (the cleaner wire shape does not). Re-run the three-way
|
|
69
|
+
comparison against a new model before assuming it carries over.
|
|
70
|
+
|
|
17
71
|
### Supported Strategies:
|
|
18
72
|
|
|
19
73
|
* **`:xml` (The DeepSeek/Reliability Standard):**
|
|
20
|
-
Wraps the reasoning content in `<think>...</think>` tags and prepends it to the main `content` string. This is the most robust method as it bypasses API field stripping.
|
|
74
|
+
Wraps the reasoning content in `<think>...</think>` tags and prepends it to the main `content` string. This is the most robust method as it bypasses API field stripping — and the most expensive, since the wrapped text becomes part of the message body and is billed as prompt tokens on every later turn. Reach for it when the provider strips fields, not by default.
|
|
21
75
|
*Note: In `SolidLoop`, this is a "Smart" strategy. It will automatically suppress itself if a native strategy (like `:gemini_signed`) is also present, avoiding redundant context.*
|
|
22
76
|
* **`:xml!` (Forced XML):**
|
|
23
77
|
Same as `:xml`, but **ignores** native strategy suppression. Use this if you want to ensure thoughts are visible in the main context even if the provider supports native fields.
|
|
24
78
|
* **`:gemini_signed` (The Google Standard):**
|
|
25
|
-
|
|
79
|
+
Sends the thought back as a Gemini part — `{ thought: true, text: "...", thoughtSignature: "..." }`. Note that `thought` is a **boolean flag** on the part and the summary text lives in `text`, the same key an ordinary answer uses; the signature rides on a part too (the thought part, or the `functionCall` part on a tool-calling turn), never on the candidate. Drops the reasoning when no signature was captured.
|
|
80
|
+
* **`:anthropic_signed` (The Anthropic Standard):**
|
|
81
|
+
Sends the thought back as a `{ type: "thinking", thinking: "...", signature: "..." }` content block, first in the assistant message's content array. Anthropic verifies the signature, so this drops the reasoning when none was captured.
|
|
26
82
|
* **`"field_name"` (Explicit Mapping):**
|
|
27
83
|
Allows copying the reasoning content into a specific JSON key (e.g., `"reasoning_content"`, `"thought"`, or `"internal_thinking"`) to satisfy various provider requirements (OpenRouter, vLLM, Groq).
|
|
28
84
|
|
|
29
85
|
### Example Combinations:
|
|
30
86
|
|
|
31
|
-
* **`
|
|
87
|
+
* **`nil` (the default)**: Let the dialect choose — `"reasoning_content"` on `open_ai`, `:anthropic_signed` on `anthropic`, `:gemini_signed` on `gemini`.
|
|
88
|
+
* **`[:xml]`**: Maximum reliability for DeepSeek or OpenRouter (when field stripping is suspected), at full prompt-token price.
|
|
32
89
|
* **`["reasoning", "reasoning_content"]`**: Populating multiple fields in parallel — useful when the provider's accepted field name is uncertain.
|
|
33
90
|
* **`[:gemini_signed, "thought", "reasoning_content"]`**: A comprehensive strategy for Gemini-compatible providers that might also look for OpenAI-style fields.
|
|
91
|
+
* **`[:anthropic_signed, :xml]`**: Signed blocks where a signature exists, welded `<think>` where one does not — the `:xml` fallback fires only when the signed strategy could not.
|
|
34
92
|
* **`[]` (Empty Array)**: Standard for OpenAI o1/o3 (reasoning is stripped to avoid 400 errors).
|
|
35
93
|
|
|
36
94
|
## 3. The Reasoning Verifier *(not implemented)*
|
|
@@ -45,7 +103,65 @@ To ensure an agent is actually "hearing its own thoughts," a **Reasoning Verific
|
|
|
45
103
|
|
|
46
104
|
Worth doing — it would give a cheap, repeatable way to verify that `ReasoningPacker` is actually working end-to-end with a given provider/strategy combination, rather than assuming it does.
|
|
47
105
|
|
|
48
|
-
## 4.
|
|
106
|
+
## 4. Measuring the Cost
|
|
107
|
+
|
|
108
|
+
`ReasoningPacker.pack_all` reports what each request's prompt is carrying, and
|
|
109
|
+
the adapter stamps it onto the assistant message that request produced:
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
message.metadata["reasoning_carried"]
|
|
113
|
+
# => { "messages" => 6, "chars" => 84616, "inline_chars" => 84616, "tokens_est" => 21154 }
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`chars` is everything sent back in any shape; `inline_chars` is only the part
|
|
117
|
+
welded into `content`, which every provider bills. Without it the cost has to be
|
|
118
|
+
reconstructed from message lengths after the fact, which is how a loop reaches
|
|
119
|
+
its window ceiling without anyone seeing why. codex tracks the same number for
|
|
120
|
+
the same reason (`context_manager/history.rs` sums the tokens of every reasoning
|
|
121
|
+
item before the last user message).
|
|
122
|
+
|
|
123
|
+
## 5. Why There Is No Trimming Strategy
|
|
124
|
+
|
|
125
|
+
The obvious way to bound a growing context is to stop resending old reasoning —
|
|
126
|
+
keep the newest `n` turns, drop the rest. SolidLoop deliberately offers no such
|
|
127
|
+
strategy, because it is a trap.
|
|
128
|
+
|
|
129
|
+
A prefix cache exists because turn N's prompt is turn N-1's plus a new tail: the
|
|
130
|
+
server re-uses the KV of everything before it and prefills only the tail. A
|
|
131
|
+
sliding retained window rewrites the *middle* of the prompt instead — the
|
|
132
|
+
message that ages out changes, the prompt diverges at its position, and
|
|
133
|
+
everything after it is prefilled again, every turn. Measured on llama.cpp
|
|
134
|
+
(Qwen3.8-27B Q6_K, prefix caching on), an 8-step agent chain:
|
|
135
|
+
|
|
136
|
+
| strategy | prompt | fresh/turn | prefill/turn | cached/turn |
|
|
137
|
+
|---|---|---|---|---|
|
|
138
|
+
| full reasoning | 4667 | 576 | 910 ms | 4091 |
|
|
139
|
+
| newest 2 turns | 1541 | 1503 | 1847 ms | 38 |
|
|
140
|
+
| send none | 499 | 59 | 500 ms | 440 |
|
|
141
|
+
|
|
142
|
+
Trimming is not the problem; *sliding* is. The trimmed run costs twice the
|
|
143
|
+
prefill of the untrimmed history it was meant to be cheaper than, and degrades
|
|
144
|
+
the model's context as well — the worst of both. Both stable rules keep the
|
|
145
|
+
cache: with the full history each reasoning block is prefilled exactly once,
|
|
146
|
+
ever; with none, no block is.
|
|
147
|
+
|
|
148
|
+
Neither reference harness trims per turn. codex keeps reasoning whole and only
|
|
149
|
+
*accounts* for it (`get_non_last_reasoning_items_tokens()` counts the items
|
|
150
|
+
before the last instruction boundary for estimation, and removes nothing); its
|
|
151
|
+
boundaries move only when a user message arrives. opencode compacts on overflow
|
|
152
|
+
(`session/compaction.ts`, `PRUNE_MINIMUM = 20_000`, `PRUNE_PROTECT = 40_000`):
|
|
153
|
+
one pass that summarises the old history into a single assistant message and
|
|
154
|
+
keeps a recent tail within a budget, after which the prompt is monotonically
|
|
155
|
+
extending again. The Qwen3 GGUF chat template encodes the same instinct — its
|
|
156
|
+
`preserve_thinking` flag cuts only before `ns.last_query_index`, a user-turn
|
|
157
|
+
boundary, never per turn.
|
|
158
|
+
|
|
159
|
+
**Compaction is the missing piece**, and it is bigger than a strategy symbol: it
|
|
160
|
+
needs a summariser, a checkpoint message kind, and a rule for when to fire.
|
|
161
|
+
Until it exists, the two supported positions are "carry the reasoning and let
|
|
162
|
+
the prefix cache pay for it" and "send none".
|
|
163
|
+
|
|
164
|
+
## 6. Data Storage
|
|
49
165
|
|
|
50
166
|
* **Reasoning Text:** Stored in the `reasoning_content` column of `solid_loop_messages`.
|
|
51
167
|
* **Signatures & Metadata:** Stored in the `metadata` JSONB column. This ensures we can support future tokens like `thought_signature` without schema migrations.
|
data/docs/guides/dialects.md
CHANGED
|
@@ -50,6 +50,6 @@ A Dialect object must implement the following methods to be compatible with the
|
|
|
50
50
|
|
|
51
51
|
1. **`completion_url(base_url)`**: Returns the specific API endpoint.
|
|
52
52
|
2. **`render_payload(universal_payload)`**: (Optional) Transforms the payload before sending.
|
|
53
|
-
3. **`apply_reasoning_strategies!(messages, strategies)`**: Formats message history according to the [Reasoning Protocol](
|
|
53
|
+
3. **`apply_reasoning_strategies!(messages, strategies)`**: Formats message history according to the [Reasoning Protocol](../decisions/reasoning_persistence.md). A `strategies` of `nil` means the agent left the choice to the dialect — substitute your own `default_reasoning_strategies`. Return the carry report from `ReasoningPacker.pack_all`; the adapter stamps it onto the message as `metadata["reasoning_carried"]`.
|
|
54
54
|
4. **`extract_message_data(raw_json)`**: Extracts content and tools from a streaming chunk or partial response.
|
|
55
55
|
5. **`normalize_response(raw_json)`**: Converts a full API response into the SolidLoop universal format (including token usage and metadata).
|
|
@@ -184,6 +184,78 @@ Semantics:
|
|
|
184
184
|
- Toolsets are stateless by default (`session_id` = nil → synthetic). A toolset
|
|
185
185
|
may override `on_initialize(params, context)` to mint and later receive a
|
|
186
186
|
real session id.
|
|
187
|
+
- **`serverInfo.version`** defaults to the gem's version, which is rarely what a
|
|
188
|
+
mounted server means — semantically it is the version of *your* MCP server.
|
|
189
|
+
Declare it next to the name:
|
|
190
|
+
|
|
191
|
+
```ruby
|
|
192
|
+
class SearchTools < SolidLoop::Mcp::Toolset
|
|
193
|
+
server_name "search"
|
|
194
|
+
server_version "1.0.0"
|
|
195
|
+
end
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
- **`ping`** is answered out of the box with an empty result, as the base
|
|
199
|
+
protocol requires. Unknown methods still get `-32601`.
|
|
200
|
+
- **Error codes** follow the MCP specification, which prescribes **`-32602`**
|
|
201
|
+
(invalid params) for an unknown tool or prompt on `tools/call` / `prompts/get`
|
|
202
|
+
— not `-32601`. `-32601` is reserved for an unknown *method*.
|
|
203
|
+
|
|
204
|
+
### Dynamic tool and prompt catalogs
|
|
205
|
+
|
|
206
|
+
The `tool` / `prompt` DSL registers at class-load time. A host whose catalog
|
|
207
|
+
lives in the database and changes without a deploy overrides four instance
|
|
208
|
+
methods instead — they default to the DSL and are consulted on every request
|
|
209
|
+
(and by `capabilities`), so a row added a second ago is advertised on the next
|
|
210
|
+
`prompts/list`:
|
|
211
|
+
|
|
212
|
+
| Method | Default | Used by |
|
|
213
|
+
| --- | --- | --- |
|
|
214
|
+
| `available_tools` | `self.class.tools` | `tools/list` |
|
|
215
|
+
| `find_tool(name)` | lookup in `available_tools` | `tools/call` |
|
|
216
|
+
| `available_prompts` | `self.class.prompts` | `prompts/list`, `capabilities` |
|
|
217
|
+
| `find_prompt(name)` | lookup in `available_prompts` | `prompts/get` |
|
|
218
|
+
|
|
219
|
+
```ruby
|
|
220
|
+
class SearchTools < SolidLoop::Mcp::Toolset
|
|
221
|
+
def available_prompts
|
|
222
|
+
SavedPrompt.active.each_with_object({}) do |row, acc|
|
|
223
|
+
acc[row.name] = PromptDefinition.new(
|
|
224
|
+
name: row.name, description: row.description, arguments: row.arguments,
|
|
225
|
+
handler: ->(args, _ctx) { format(row.body, args) }
|
|
226
|
+
)
|
|
227
|
+
end
|
|
228
|
+
end
|
|
229
|
+
end
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Return `{}` for "none", and keep the override **total**: `capabilities` calls
|
|
233
|
+
`available_prompts` during `initialize`, so a raising override fails the
|
|
234
|
+
handshake rather than one request.
|
|
235
|
+
|
|
236
|
+
### Overriding `deliver`
|
|
237
|
+
|
|
238
|
+
`deliver` is the documented seam for answering a method the gem does not model.
|
|
239
|
+
The envelope builders it needs are **public API** and covered by the
|
|
240
|
+
compatibility promise: `success_envelope(request, result)`,
|
|
241
|
+
`error_envelope(request, code, message)`, `json_boundary(value)` and
|
|
242
|
+
`tool_result(value)`.
|
|
243
|
+
|
|
244
|
+
```ruby
|
|
245
|
+
def deliver(payload, session_id: nil, context: nil)
|
|
246
|
+
request = json_boundary(payload)
|
|
247
|
+
return super unless request["method"] == "resources/list"
|
|
248
|
+
|
|
249
|
+
SolidLoop::Mcp::Result.new(
|
|
250
|
+
body: success_envelope(request, "resources" => []),
|
|
251
|
+
session_id: session_id, raw_request: payload, raw_response: nil, status: 200
|
|
252
|
+
)
|
|
253
|
+
end
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
Prefer the dynamic-catalog hooks above when the need is "my prompts/tools come
|
|
257
|
+
from somewhere else" — overriding `deliver` for that intercepts the protocol to
|
|
258
|
+
solve a catalog problem, and has to be re-checked against every gem upgrade.
|
|
187
259
|
- **Transactions:** the block runs on the caller's thread and shares the job's
|
|
188
260
|
ActiveRecord connection. That makes a tool atomic with SolidLoop's own writes
|
|
189
261
|
— and makes `transaction do ... end` inside a tool a *nested* transaction.
|
data/docs/validation.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
# Validation
|
|
2
|
+
|
|
3
|
+
What SolidLoop has been run against before a release, and what it has not.
|
|
4
|
+
|
|
5
|
+
A gem that drives other people's inference budgets should be specific about the
|
|
6
|
+
shape of its own evidence, including the gaps. This page is that statement, kept
|
|
7
|
+
current per release.
|
|
8
|
+
|
|
9
|
+
## The automated suite
|
|
10
|
+
|
|
11
|
+
Unit, integration and request specs run against a PostgreSQL dummy app on Rails
|
|
12
|
+
7.1, 7.2 and 8.1 (see [Testing](../README.md#testing)). The LLM and MCP peers
|
|
13
|
+
are stubbed by emulators that speak each provider's wire format.
|
|
14
|
+
|
|
15
|
+
The limit of that is worth saying plainly: **an emulator encodes our own reading
|
|
16
|
+
of a provider's wire format.** Where the reading is wrong, the emulator is wrong
|
|
17
|
+
in the same direction and the suite stays green. That is exactly what had
|
|
18
|
+
happened to the `gemini` dialect before 0.0.5 — it had misread thought parts
|
|
19
|
+
since it was written, and every test passed. Emulators catch regressions; they
|
|
20
|
+
cannot catch a shared misconception.
|
|
21
|
+
|
|
22
|
+
## 0.0.5 — local inference, sustained agentic load
|
|
23
|
+
|
|
24
|
+
Before this release SolidLoop ran as the engine of a private agent application:
|
|
25
|
+
batches of loops in parallel, each one a multi-turn coding task using MCP tools,
|
|
26
|
+
repeated across many runs. The inference side was entirely local:
|
|
27
|
+
|
|
28
|
+
| | |
|
|
29
|
+
| --- | --- |
|
|
30
|
+
| Servers | llama.cpp (`--jinja`, `--kv-unified`) and vLLM |
|
|
31
|
+
| Model | Qwen3 27B — GGUF Q6_K on llama.cpp, NVFP4 on vLLM |
|
|
32
|
+
| Hardware | 2 × NVIDIA RTX 5060 Ti, tensor-parallel |
|
|
33
|
+
| Dialect | `open_ai` (both servers expose an OpenAI-compatible API) |
|
|
34
|
+
|
|
35
|
+
Local OpenAI-compatible servers are a **stricter** test surface than the hosted
|
|
36
|
+
API, which is why this is worth reporting rather than apologising for:
|
|
37
|
+
|
|
38
|
+
- **They are unforgiving about malformed messages.** After a rejected tool call
|
|
39
|
+
SolidLoop was emitting a tool message with `"name": null`. vLLM ignores it;
|
|
40
|
+
llama.cpp rejects the entire request, and because the offending message stays
|
|
41
|
+
in history the loop then fails on every retry rather than recovering. Hosted
|
|
42
|
+
providers had never surfaced it. Fixed in 0.0.5.
|
|
43
|
+
- **Their chat templates are inspectable.** What a dialect sends back can be
|
|
44
|
+
traced to an actual prompt-token count through `/tokenize`, which is how the
|
|
45
|
+
reasoning transport in 0.0.5 was chosen rather than guessed — see
|
|
46
|
+
[reasoning_persistence.md](decisions/reasoning_persistence.md).
|
|
47
|
+
- **KV-cache and slot pressure are real and visible**, so context-budget defects
|
|
48
|
+
present as truncated tool-call arguments and traceable server logs instead of
|
|
49
|
+
an opaque 400.
|
|
50
|
+
|
|
51
|
+
The application's own task results are not part of this claim — those measure
|
|
52
|
+
the model, not the gem. What the runs establish about SolidLoop is narrower:
|
|
53
|
+
loops ran concurrently to completion, leases and the reaper held across worker
|
|
54
|
+
restarts, no turn was silently dropped, and the failures that did occur were
|
|
55
|
+
traced to specific defects that now have specs and changelog entries.
|
|
56
|
+
|
|
57
|
+
## What has not been tested
|
|
58
|
+
|
|
59
|
+
- **The native `anthropic` and `gemini` dialects have never made a live call.**
|
|
60
|
+
Their payload shapes come from the published API documentation and are covered
|
|
61
|
+
by unit tests and emulators, but no real key has exercised them. 0.0.5 changes
|
|
62
|
+
the default reasoning transport for both to the providers' signed block
|
|
63
|
+
formats, so that untested path is now the default path for those two dialects.
|
|
64
|
+
If you are on `anthropic` or `gemini` and want the pre-0.0.5 behaviour while
|
|
65
|
+
you evaluate, set `reasoning_strategies` to `[:xml]`.
|
|
66
|
+
- **Hosted providers were last exercised in 0.0.4**, through the `open_ai`
|
|
67
|
+
dialect against OpenAI- and OpenRouter-served models in a pilot Rails
|
|
68
|
+
application. 0.0.5 has not repeated that run — its provider coverage is the
|
|
69
|
+
local stand above.
|
|
70
|
+
- **Scale.** SolidLoop has not been run at a size where Postgres or the job
|
|
71
|
+
backend was the bottleneck, nor over horizons measured in days. The durability
|
|
72
|
+
design ([durable_attempt_lease.md](decisions/durable_attempt_lease.md)) is
|
|
73
|
+
exercised by specs and by deliberate mid-run restarts, not by a long
|
|
74
|
+
production soak.
|
|
75
|
+
|
|
76
|
+
## Reports are welcome
|
|
77
|
+
|
|
78
|
+
Running SolidLoop against a provider, model, or scale that is not on this page
|
|
79
|
+
is a genuine contribution, whether or not it comes with a patch. Useful in a
|
|
80
|
+
report: the dialect and provider, the model, whether reasoning came back and in
|
|
81
|
+
which field, and the relevant wire-log rows — `EventLogging` redacts
|
|
82
|
+
credentials, including the API key Gemini carries in the URL, so those rows are
|
|
83
|
+
safe to paste.
|
|
84
|
+
|
|
85
|
+
See [CONTRIBUTING.md](../CONTRIBUTING.md).
|