solid_loop 0.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +156 -0
- data/CODE_OF_CONDUCT.md +67 -0
- data/CONTRIBUTING.md +82 -0
- data/MIT-LICENSE +21 -0
- data/README.md +483 -0
- data/Rakefile +11 -0
- data/app/assets/javascripts/solid_loop/chart.umd.min.js +14 -0
- data/app/assets/javascripts/solid_loop/chartjs-adapter-date-fns.bundle.min.js +7 -0
- data/app/assets/javascripts/solid_loop/chartkick.min.js +2 -0
- data/app/assets/stylesheets/solid_loop/application.css +15 -0
- data/app/controllers/solid_loop/application_controller.rb +29 -0
- data/app/controllers/solid_loop/dashboard_controller.rb +104 -0
- data/app/controllers/solid_loop/events_controller.rb +12 -0
- data/app/controllers/solid_loop/loops_controller.rb +77 -0
- data/app/controllers/solid_loop/mcp_inbound_sessions_controller.rb +13 -0
- data/app/controllers/solid_loop/mcp_sessions_controller.rb +68 -0
- data/app/controllers/solid_loop/mcp_tools_controller.rb +16 -0
- data/app/controllers/solid_loop/messages_controller.rb +12 -0
- data/app/controllers/solid_loop/tool_calls_controller.rb +12 -0
- data/app/helpers/solid_loop/application_helper.rb +125 -0
- data/app/jobs/solid_loop/application_job.rb +13 -0
- data/app/jobs/solid_loop/llm_completion_job.rb +139 -0
- data/app/jobs/solid_loop/observe_broadcast_job.rb +15 -0
- data/app/jobs/solid_loop/reaper_job.rb +20 -0
- data/app/jobs/solid_loop/tool_execution_job.rb +200 -0
- data/app/models/solid_loop/application_record.rb +5 -0
- data/app/models/solid_loop/base.rb +190 -0
- data/app/models/solid_loop/event.rb +7 -0
- data/app/models/solid_loop/loop.rb +225 -0
- data/app/models/solid_loop/mcp_inbound_session.rb +25 -0
- data/app/models/solid_loop/mcp_session.rb +12 -0
- data/app/models/solid_loop/mcp_tool.rb +26 -0
- data/app/models/solid_loop/message.rb +69 -0
- data/app/models/solid_loop/tool_call.rb +42 -0
- data/app/queries/solid_loop/admin/base_query.rb +23 -0
- data/app/queries/solid_loop/admin/events_query.rb +31 -0
- data/app/queries/solid_loop/admin/loops_query.rb +38 -0
- data/app/queries/solid_loop/admin/mcp_inbound_sessions_query.rb +31 -0
- data/app/queries/solid_loop/admin/mcp_sessions_query.rb +26 -0
- data/app/queries/solid_loop/admin/mcp_tools_query.rb +26 -0
- data/app/queries/solid_loop/admin/messages_query.rb +37 -0
- data/app/queries/solid_loop/admin/tool_calls_query.rb +34 -0
- data/app/services/solid_loop/adapters/native.rb +170 -0
- data/app/services/solid_loop/dialects/anthropic.rb +160 -0
- data/app/services/solid_loop/dialects/gemini.rb +122 -0
- data/app/services/solid_loop/dialects/open_ai.rb +62 -0
- data/app/services/solid_loop/dialects/reasoning_packer.rb +37 -0
- data/app/services/solid_loop/llm_usage_parser/anthropic.rb +22 -0
- data/app/services/solid_loop/llm_usage_parser/gemini.rb +18 -0
- data/app/services/solid_loop/llm_usage_parser/llama.rb +15 -0
- data/app/services/solid_loop/llm_usage_parser/openai.rb +18 -0
- data/app/services/solid_loop/llm_usage_parser.rb +19 -0
- data/app/services/solid_loop/mcp_session_initializer.rb +205 -0
- data/app/services/solid_loop/mcp_tool_execution_service.rb +153 -0
- data/app/services/solid_loop/middlewares/agent_initialization.rb +37 -0
- data/app/services/solid_loop/middlewares/error_handling.rb +107 -0
- data/app/services/solid_loop/middlewares/event_logging.rb +100 -0
- data/app/services/solid_loop/middlewares/message_building.rb +92 -0
- data/app/services/solid_loop/middlewares/network_calling.rb +84 -0
- data/app/services/solid_loop/middlewares/response_parsing.rb +348 -0
- data/app/services/solid_loop/middlewares/tool_call_xml_parser.rb +117 -0
- data/app/services/solid_loop/sse_stream_aggregator.rb +105 -0
- data/app/services/solid_loop/tool_failure_reconciler.rb +122 -0
- data/app/services/solid_loop/tool_middlewares/agent_initialization.rb +17 -0
- data/app/services/solid_loop/tool_middlewares/error_handling.rb +66 -0
- data/app/services/solid_loop/tool_middlewares/event_logging.rb +27 -0
- data/app/services/solid_loop/tool_middlewares/response_creation.rb +110 -0
- data/app/services/solid_loop/tool_middlewares/tool_execution.rb +206 -0
- data/app/views/layouts/solid_loop/admin.html.erb +416 -0
- data/app/views/layouts/solid_loop/application.html.erb +17 -0
- data/app/views/solid_loop/dashboard/index.html.erb +184 -0
- data/app/views/solid_loop/events/index.html.erb +47 -0
- data/app/views/solid_loop/events/show.html.erb +76 -0
- data/app/views/solid_loop/loops/index.html.erb +83 -0
- data/app/views/solid_loop/loops/show.html.erb +148 -0
- data/app/views/solid_loop/mcp_inbound_sessions/index.html.erb +53 -0
- data/app/views/solid_loop/mcp_inbound_sessions/show.html.erb +78 -0
- data/app/views/solid_loop/mcp_sessions/_tool_output.html.erb +11 -0
- data/app/views/solid_loop/mcp_sessions/index.html.erb +46 -0
- data/app/views/solid_loop/mcp_sessions/inspector.html.erb +94 -0
- data/app/views/solid_loop/mcp_sessions/show.html.erb +142 -0
- data/app/views/solid_loop/mcp_tools/index.html.erb +44 -0
- data/app/views/solid_loop/mcp_tools/show.html.erb +69 -0
- data/app/views/solid_loop/messages/_message.html.erb +75 -0
- data/app/views/solid_loop/messages/index.html.erb +80 -0
- data/app/views/solid_loop/messages/show.html.erb +121 -0
- data/app/views/solid_loop/shared/_pagination.html.erb +21 -0
- data/app/views/solid_loop/tool_calls/index.html.erb +53 -0
- data/app/views/solid_loop/tool_calls/show.html.erb +59 -0
- data/config/routes.rb +27 -0
- data/db/migrate/20260408000100_solid_loop_init.rb +136 -0
- data/db/migrate/20260709000100_solid_loop_create_mcp_inbound_sessions.rb +16 -0
- data/db/migrate/20260713000100_solid_loop_add_execution_guards.rb +6 -0
- data/db/migrate/20260714000100_solid_loop_add_message_steering.rb +17 -0
- data/db/migrate/20260714000200_solid_loop_add_message_conversation_order.rb +14 -0
- data/db/migrate/20260715000100_solid_loop_add_mcp_inbound_session_principal_key.rb +43 -0
- data/db/migrate/20260715000200_solid_loop_add_llm_lease.rb +40 -0
- data/db/migrate/20260715000300_solid_loop_add_tool_lease.rb +33 -0
- data/db/migrate/20260715000400_solid_loop_add_lease_running_check.rb +32 -0
- data/db/migrate/20260715000500_solid_loop_add_tool_lease_pair_check.rb +35 -0
- data/docs/contributing/coverage.md +64 -0
- data/docs/decisions/durable_attempt_lease.md +364 -0
- data/docs/decisions/mcp-only-tooling.md +135 -0
- data/docs/decisions/mcp-server.md +223 -0
- data/docs/decisions/reasoning_persistence.md +51 -0
- data/docs/decisions/ruby_llm_rejected.md +35 -0
- data/docs/guides/dialects.md +55 -0
- data/docs/guides/llm_middlewares.md +294 -0
- data/docs/guides/mcp_transports.md +374 -0
- data/docs/guides/tool_middlewares.md +148 -0
- data/lib/solid_loop/configuration.rb +175 -0
- data/lib/solid_loop/engine.rb +14 -0
- data/lib/solid_loop/lease_heartbeat.rb +94 -0
- data/lib/solid_loop/lease_renewer.rb +347 -0
- data/lib/solid_loop/llm_metrics.rb +16 -0
- data/lib/solid_loop/mcp/call_context.rb +19 -0
- data/lib/solid_loop/mcp/client_factory.rb +61 -0
- data/lib/solid_loop/mcp/http_transport.rb +52 -0
- data/lib/solid_loop/mcp/principal.rb +82 -0
- data/lib/solid_loop/mcp/result.rb +13 -0
- data/lib/solid_loop/mcp/server.rb +246 -0
- data/lib/solid_loop/mcp/stdio_transport.rb +224 -0
- data/lib/solid_loop/mcp/toolset.rb +347 -0
- data/lib/solid_loop/mcp/transport.rb +20 -0
- data/lib/solid_loop/mcp.rb +25 -0
- data/lib/solid_loop/mcp_client.rb +176 -0
- data/lib/solid_loop/pipeline/builder.rb +38 -0
- data/lib/solid_loop/pipeline/context.rb +61 -0
- data/lib/solid_loop/pipeline/tool_context.rb +53 -0
- data/lib/solid_loop/pipeline.rb +40 -0
- data/lib/solid_loop/reaper.rb +313 -0
- data/lib/solid_loop/version.rb +3 -0
- data/lib/solid_loop.rb +94 -0
- data/lib/tasks/coverage.rake +206 -0
- data/lib/tasks/solid_loop_tasks.rake +4 -0
- metadata +228 -0
|
@@ -0,0 +1,364 @@
|
|
|
1
|
+
# Durable lease + reaper
|
|
2
|
+
|
|
3
|
+
Status: accepted design.
|
|
4
|
+
|
|
5
|
+
This document describes the durable-lease design that makes SolidLoop's LLM turns
|
|
6
|
+
and tool invocations crash/restart-safe: how a worker claims a unit of work, how
|
|
7
|
+
its lease is kept alive while it runs, how a periodic reaper recovers work whose
|
|
8
|
+
worker died, and the guarantees (and their limits) that result.
|
|
9
|
+
|
|
10
|
+
## Context
|
|
11
|
+
|
|
12
|
+
- **Breaking DB changes are acceptable.** A consumer can reset its dev DB and
|
|
13
|
+
hand-fix records, so there is **no online migration, no backfill, no quiesced
|
|
14
|
+
rollout, no recovery task** — the rollout story is "drop, recreate, reset."
|
|
15
|
+
- Durable, crash/restart-safe resumes are a release requirement: without them,
|
|
16
|
+
the claim "durable, crash/restart resumes" is false.
|
|
17
|
+
- Bar: **correct + as simple to build and maintain as possible**, not
|
|
18
|
+
zero-downtime migration of a live fleet.
|
|
19
|
+
|
|
20
|
+
## What the existing code already provides
|
|
21
|
+
|
|
22
|
+
- `LlmCompletionJob.perform(loop_id)` **already CAS-claims**:
|
|
23
|
+
`UPDATE ... SET status=running, execution_token=uuid WHERE status=queued`.
|
|
24
|
+
- `solid_loop_tool_calls.executed_at` is **already the tool checkpoint**;
|
|
25
|
+
`ToolExecutionJob(tool_call_id, execution_token)` already carries the fence.
|
|
26
|
+
- The stable ToolCall idempotency key already propagates via `CallContext` /
|
|
27
|
+
MCP `_meta`.
|
|
28
|
+
- The streaming shell is a plain message row (`status: "processing"`,
|
|
29
|
+
`is_hidden` default false); `MessageBuilding` assembles `where(is_hidden:false)`,
|
|
30
|
+
so hiding a mid-stream partial needs **zero new message columns**.
|
|
31
|
+
|
|
32
|
+
## Design
|
|
33
|
+
|
|
34
|
+
**Schema: 3 new columns, 2 CHECKs, 2 indexes. No new tables or models.**
|
|
35
|
+
|
|
36
|
+
```sql
|
|
37
|
+
ALTER TABLE solid_loop_loops
|
|
38
|
+
ADD COLUMN lease_expires_at timestamptz; -- non-NULL iff an LLM turn is in flight
|
|
39
|
+
ALTER TABLE solid_loop_loops
|
|
40
|
+
ADD CONSTRAINT running_requires_token
|
|
41
|
+
CHECK (status <> 'running' OR execution_token IS NOT NULL);
|
|
42
|
+
ALTER TABLE solid_loop_loops
|
|
43
|
+
ADD CONSTRAINT lease_running_only
|
|
44
|
+
CHECK (lease_expires_at IS NULL OR status = 'running');
|
|
45
|
+
|
|
46
|
+
ALTER TABLE solid_loop_tool_calls
|
|
47
|
+
ADD COLUMN lease_token varchar,
|
|
48
|
+
ADD COLUMN leased_until timestamptz;
|
|
49
|
+
ALTER TABLE solid_loop_tool_calls
|
|
50
|
+
ADD CONSTRAINT tool_lease_pair
|
|
51
|
+
CHECK ((lease_token IS NULL) = (leased_until IS NULL));
|
|
52
|
+
|
|
53
|
+
-- messages: NO new columns (status 'processing' + is_hidden = the shell protocol)
|
|
54
|
+
-- indexes: loops(status, lease_expires_at); tool_calls(leased_until) WHERE executed_at IS NULL
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### State model
|
|
58
|
+
|
|
59
|
+
- **Loop** (existing statuses): `queued` = durable LLM work intent; `running` +
|
|
60
|
+
`lease_expires_at` set = LLM HTTP turn in flight; `running` + `lease_expires_at`
|
|
61
|
+
NULL = tool phase (the LLM commit clears it when emitting tools).
|
|
62
|
+
`execution_token` is BOTH the generation fence AND the LLM lease token —
|
|
63
|
+
rotated at every claim/resume, cleared at pause/stop/reclaim.
|
|
64
|
+
- **ToolCall**: `executed_at IS NULL` = durable intent; live lease = `lease_token`
|
|
65
|
+
set and `leased_until > now`; `executed_at` set = terminal (the existing CAS
|
|
66
|
+
`WHERE executed_at IS NULL` stays the exactly-once-in-DB guard). Two tokens
|
|
67
|
+
exist ONLY because reclaiming one dead parallel tool must not fence healthy
|
|
68
|
+
siblings — the one place a second token is load-bearing (parallel tools are the
|
|
69
|
+
shipped default: `Base#sequential_tool_calls? => false`).
|
|
70
|
+
- **Shell message** (born-hidden protocol): created `status:"processing",
|
|
71
|
+
is_hidden:true`; the ONLY path that flips it visible is the canonical LLM commit
|
|
72
|
+
txn (which also fence-checks); reclaim/pause/stop/error mark any `processing`
|
|
73
|
+
shell for the loop `failed` and leave it hidden (partial content kept for
|
|
74
|
+
admin/audit). `MessageBuilding`'s existing `is_hidden:false` filter already
|
|
75
|
+
excludes it. Force-stop must not mark partial output `success`.
|
|
76
|
+
|
|
77
|
+
### Claim
|
|
78
|
+
|
|
79
|
+
LLM: today's CAS plus `lease_expires_at = now + ttl`. Tool:
|
|
80
|
+
`UPDATE tool_calls SET lease_token=uuid, leased_until=now+ttl
|
|
81
|
+
WHERE id=? AND executed_at IS NULL AND (lease_token IS NULL OR leased_until < now)`
|
|
82
|
+
after verifying `loop.execution_token` matches the job arg; 0 rows ⇒ no-op.
|
|
83
|
+
|
|
84
|
+
**The lease is derived PER CLAIM so `lease > client timeout + margin` holds BY
|
|
85
|
+
CONSTRUCTION** for both LLM and tool — a live worker can never hold an expired
|
|
86
|
+
lease *for the duration of a single inactivity window* (its HTTP/tool client
|
|
87
|
+
raises first). LLM: `read_timeout (600s default) + lease_margin`, from the agent's
|
|
88
|
+
own provider. Tool: `max(default_tool_timeout, slowest resolved HTTP MCP client
|
|
89
|
+
timeout) + lease_margin` — the HTTP timeout (`mcp_config[:timeout] || 60`,
|
|
90
|
+
operator-settable well above the default) MUST be folded in, or a *healthy*
|
|
91
|
+
long-running HTTP tool could be reclaimed → duplicate invocation on a live call.
|
|
92
|
+
Residual (documented in `Configuration`): an in-process/custom tool exposes no
|
|
93
|
+
client timeout to derive from, so `default_tool_timeout` is its ceiling; one that
|
|
94
|
+
legitimately runs longer MAY be reclaimed (a duplicate invocation — harmless: the
|
|
95
|
+
both-token fence yields exactly one canonical result and tools carry the stable
|
|
96
|
+
`solid_loop:tool_call:#{id}` idempotency key), so operators must set
|
|
97
|
+
`default_tool_timeout` above their slowest in-process tool.
|
|
98
|
+
|
|
99
|
+
### Host resume contract — revoke the tool lease before re-enqueue
|
|
100
|
+
|
|
101
|
+
Any host-side `paused→running` (or `failed→running`) transition that re-enqueues a
|
|
102
|
+
`ToolExecutionJob` for an unexecuted tool_call MUST FIRST revoke that call's live
|
|
103
|
+
per-tool lease (`lease_token`/`leased_until` → NULL), or the re-enqueued job's
|
|
104
|
+
self-claim CAS (the `lease_token IS NULL OR leased_until < now` predicate above)
|
|
105
|
+
matches zero rows and no-ops SILENTLY — the loop then strands until the stale
|
|
106
|
+
lease's TTL expires and the reaper (case 2) repairs it (up to `tool_lease_duration
|
|
107
|
+
+ reaper interval`). `SolidLoop::Base#resume!` is the canonical example: it rotates
|
|
108
|
+
a fresh `execution_token` AND `UPDATE tool_calls SET lease_token=NULL,
|
|
109
|
+
leased_until=NULL WHERE id IN (...) AND executed_at IS NULL` in the same locked
|
|
110
|
+
txn BEFORE enqueuing (`app/models/solid_loop/base.rb`, the pending-tool-calls
|
|
111
|
+
branch). Hosts that hand-roll their own resume must mirror BOTH steps.
|
|
112
|
+
Observability: the claim no-op logs a `Rails.logger.warn` (`ToolExecutionJob:
|
|
113
|
+
tool lease claim lost …`) with the tool_call/loop ids and the surviving
|
|
114
|
+
lease_token/leased_until, so the otherwise-invisible strand is diagnosable.
|
|
115
|
+
|
|
116
|
+
### LLM heartbeat
|
|
117
|
+
|
|
118
|
+
The claim-time lease derivation above is NOT sufficient on its own for the LLM
|
|
119
|
+
path. `read_timeout` is an **inactivity** timeout (the silence between two
|
|
120
|
+
consecutive socket reads), NOT a bound on total request duration. A healthy model
|
|
121
|
+
can stream for tens of minutes with **every inter-chunk gap** well under
|
|
122
|
+
`read_timeout`, so a static `read_timeout + margin` lease would expire on a LIVE
|
|
123
|
+
worker mid-stream and the reaper (case 1) would reclaim a live turn — a duplicate
|
|
124
|
+
provider call plus a discarded result. Pre-call **MCP session initialization**
|
|
125
|
+
(before the first chunk) is likewise unbounded by `read_timeout`.
|
|
126
|
+
|
|
127
|
+
So `LeaseHeartbeat` (`lib/solid_loop/lease_heartbeat.rb`) renews
|
|
128
|
+
`lease_expires_at` for the ENTIRE claimed LLM job — MCP init + the whole stream
|
|
129
|
+
(incl. a silent prefill) + finalization — on a DEDICATED pooled DB connection
|
|
130
|
+
through the Rails executor, at ~`ttl/4` cadence. Every renew is a CONDITIONAL
|
|
131
|
+
`UPDATE ... WHERE id=loop_id AND status='running' AND execution_token=<ours>`, so
|
|
132
|
+
it can NEVER renew a lease it no longer owns; a 0-row renew sets a thread-safe
|
|
133
|
+
LOST-LEASE flag. The main path calls `heartbeat.check!` (raising
|
|
134
|
+
`SolidLoop::LostLease`) BEFORE any canonical write/checkpoint (in
|
|
135
|
+
`ResponseParsing`), so a worker whose lease was lost commits nothing; the job
|
|
136
|
+
swallows `LostLease` (recovery belongs to whoever now owns the generation). The
|
|
137
|
+
heartbeat stops in an `ensure` before the canonical completion, joining its thread
|
|
138
|
+
and returning its connection (no pool leak, no thread left running in tests).
|
|
139
|
+
Correctness still ultimately rests on the commit-time fence check under the row
|
|
140
|
+
lock; the heartbeat's job is only to keep a HEALTHY turn from being falsely
|
|
141
|
+
reclaimed. The tool path keeps the static per-claim lease (a tool invocation is a
|
|
142
|
+
single bounded client call, so its lease derivation IS sufficient — no tool
|
|
143
|
+
heartbeat).
|
|
144
|
+
|
|
145
|
+
### The process-wide lease renewer
|
|
146
|
+
|
|
147
|
+
The heartbeat is a thin registration handle over a SINGLE process-wide
|
|
148
|
+
`SolidLoop::LeaseRenewer` (`lib/solid_loop/lease_renewer.rb`), not one thread per
|
|
149
|
+
turn. Each in-flight `LlmCompletionJob` registers its
|
|
150
|
+
`(loop_id, execution_token, lease_duration, max_duration)` on start and
|
|
151
|
+
deregisters when the turn ends; the renewer renews ALL registered leases from ONE
|
|
152
|
+
long-lived connection checked out once, so renewal is **O(1) in connections, not
|
|
153
|
+
O(concurrency)**. A per-turn renewer thread each checking out its own connection
|
|
154
|
+
per tick would livelock at full worker-pool occupancy (pool == concurrency, the
|
|
155
|
+
standard shape): all N connections held by busy workers, so every heartbeat blocks
|
|
156
|
+
in `with_connection`, treats `checkout_timeout` as a lost lease, yields its turn
|
|
157
|
+
uncommitted, and the reaper requeues into the same saturated pool. The single
|
|
158
|
+
renewer avoids this entirely. Deployment requirement: size the DB pool as
|
|
159
|
+
`worker_concurrency + 1` (documented in the README).
|
|
160
|
+
|
|
161
|
+
`LeaseHeartbeat` keeps the public surface (`run`/`start`/`stop`/`lost?`/`check!`)
|
|
162
|
+
and the ownership contract (conditional renew keyed on loop+token,
|
|
163
|
+
`lease_expires_at IS NOT NULL` guard, per-registration lost flag,
|
|
164
|
+
stop-before-commit). The renewer is created via a class-level constant mutex
|
|
165
|
+
(`LeaseRenewer::MUTEX`) wrapping BOTH `@instance ||= new` and the reset-to-nil
|
|
166
|
+
swap, so concurrent first-turns can't each construct a distinct renewer (which
|
|
167
|
+
would each start a thread contending for the one process-lifetime connection —
|
|
168
|
+
a livelock). The blocking `stop!` (thread join) runs OUTSIDE the lock so a
|
|
169
|
+
concurrent `instance` never waits on a join.
|
|
170
|
+
|
|
171
|
+
**Leak ceiling.** Leak safety does not depend on the job's `ensure` ever running.
|
|
172
|
+
A registration that never deregisters (a job that skipped its `ensure`) would
|
|
173
|
+
otherwise be renewed forever, so the reaper's case-1 reclaim could never fire.
|
|
174
|
+
Each registration carries a hard ceiling tied to the **owning agent's own legal
|
|
175
|
+
turn bound** — `max_duration + config.lease_leak_grace` (default `2h + 5min`).
|
|
176
|
+
A turn longer than `max_duration` is illegitimate by the agent's own contract, so
|
|
177
|
+
a legitimate turn always finishes before the ceiling (never dropped), while a
|
|
178
|
+
genuine leak self-expires shortly after `max_duration` → its lease lapses → the
|
|
179
|
+
reaper reclaims it. `max_duration` is registered alongside the other fields (the
|
|
180
|
+
`LlmCompletionJob` already resolves the agent; a fallback
|
|
181
|
+
`config.default_max_duration` covers out-of-band callers). A ceiling tied to
|
|
182
|
+
`lease_duration × K` instead would be wrong: a healthy turn stays registered for
|
|
183
|
+
its whole life, so any legitimate stream longer than K lease-widths (e.g. a small
|
|
184
|
+
`read_timeout` with steady chunks, 30 min vs a ~20-min K=20 ceiling) would be
|
|
185
|
+
dropped → its lease would lapse → the reaper would reclaim a live turn → the
|
|
186
|
+
replacement would hit the same ceiling → no long turn could ever complete.
|
|
187
|
+
|
|
188
|
+
The renewer intentionally runs for the whole process lifetime; its single
|
|
189
|
+
connection is released at process exit (no shutdown hook needed — nothing to
|
|
190
|
+
drain between turns). Each registration's `lost` flag is a
|
|
191
|
+
`Concurrent::AtomicBoolean` so the renewer-thread write and main-path read carry a
|
|
192
|
+
memory barrier on every Ruby engine, not only MRI+GVL.
|
|
193
|
+
|
|
194
|
+
### Reclaim
|
|
195
|
+
|
|
196
|
+
One idempotent `SolidLoop.reap!` service on a host-installed recurrence (30–60s).
|
|
197
|
+
Each case is a short row-locked txn that re-checks expiry under lock (two racing
|
|
198
|
+
reapers ⇒ one wins):
|
|
199
|
+
|
|
200
|
+
1. `running`, `lease_expires_at < now` → dead LLM turn: fail+hide the processing
|
|
201
|
+
shell, rotate the token away, loop → `queued`, re-enqueue
|
|
202
|
+
`LlmCompletionJob(loop_id)`. If the loop is paused/terminal/frozen, just clean
|
|
203
|
+
the shell.
|
|
204
|
+
2. `running`, lease NULL, an unresolved tool_call with expired/absent lease →
|
|
205
|
+
re-enqueue `ToolExecutionJob(id, current execution_token)` (claim CAS dedupes;
|
|
206
|
+
healthy siblings untouched).
|
|
207
|
+
3. `running`, lease NULL, zero unresolved tools → dropped-enqueue repair: loop →
|
|
208
|
+
`queued`, enqueue the next turn.
|
|
209
|
+
4. `queued` older than a threshold → re-enqueue (repairs a lost queue row).
|
|
210
|
+
|
|
211
|
+
A backstop sweep (`reap_orphaned_shells`) over paused/terminal loops fails+hides
|
|
212
|
+
any orphaned `processing` shell (cases 1–4 only scan `running`). A gem-side health
|
|
213
|
+
check reports the last successful reap.
|
|
214
|
+
|
|
215
|
+
### Failure reconciliation
|
|
216
|
+
|
|
217
|
+
Both jobs wrap EVERYTHING — agent `constantize`, provider resolution, MCP init,
|
|
218
|
+
payload build, parse, callbacks — in `rescue StandardError`; if the fence still
|
|
219
|
+
matches under lock: shell → failed+hidden, loop → failed (or paused by error
|
|
220
|
+
class), token + lease cleared. A PRE-lease error takes a no-lease,
|
|
221
|
+
generation-fenced failure path (the loop → failed under the token, no tool_call
|
|
222
|
+
write). Hard kills fall through to the reaper. A renamed/missing agent class or a
|
|
223
|
+
raising config hook now fails the loop VISIBLY instead of stranding it. The two
|
|
224
|
+
job reconcilers share a `ToolFailureReconciler` so their semantics cannot drift.
|
|
225
|
+
|
|
226
|
+
### Steering
|
|
227
|
+
|
|
228
|
+
First act of an LLM turn after claim, in one committed row-locked txn: lock
|
|
229
|
+
pending steering rows `FOR UPDATE` in ID order; flip to delivered with
|
|
230
|
+
delivery-time `conversation_order`; commit. Payload builds from canonical history
|
|
231
|
+
afterward. Cancel locks the same row — no pluck/find gap. A retried turn re-reads
|
|
232
|
+
history (already-delivered steering included) and may drain newly-arrived
|
|
233
|
+
steering; both are fine WITHOUT exact-bytes retry. The loop-completion txn checks
|
|
234
|
+
pending steering and goes `queued` instead of `completed`.
|
|
235
|
+
|
|
236
|
+
### Canonical writes under the row lock
|
|
237
|
+
|
|
238
|
+
The tool canonical writes are LOOP-ROW-LOCKED, not `reload`-fenced. Lock order is
|
|
239
|
+
**loop row → tool_call → message** (deterministic ascending id). `ToolExecution`
|
|
240
|
+
holds the loop lock across both-token validation AND the
|
|
241
|
+
`executed_at`/result/`duration_tools` write; `ResponseCreation` holds the SAME
|
|
242
|
+
loop lock across validation, the tool response `message.create!`, the
|
|
243
|
+
completion-gate transition, and the successor enqueue. A `reload` is NOT a fence;
|
|
244
|
+
the row lock freezes `loop.execution_token` + `tool_call.lease_token` between
|
|
245
|
+
validation and mutation, so a stale/revoked worker writes NOTHING. The tool BODY
|
|
246
|
+
runs OUTSIDE the loop lock (a long tool must never serialize pause/stop/resume
|
|
247
|
+
behind the loop lock). `mcp_session_id` is noncanonical telemetry folded into the
|
|
248
|
+
FENCED canonical write (no separate unfenced update), so a stale worker cannot
|
|
249
|
+
overwrite it.
|
|
250
|
+
|
|
251
|
+
## Load-bearing — must NOT be dropped
|
|
252
|
+
|
|
253
|
+
- The single **canonical-commit transaction** with a fence check under row locks
|
|
254
|
+
(all durable writes happen here or nowhere).
|
|
255
|
+
- **CAS claim** (duplicate job delivery must no-op).
|
|
256
|
+
- **Transactional enqueue with verified `perform_later`** + reaper repair of lost
|
|
257
|
+
queue rows (the "queued forever" failure mode). `SolidLoop.enqueue!` provides
|
|
258
|
+
the verified-enqueue half.
|
|
259
|
+
- **Per-`tool_call` lease columns** (parallel is the shipped default).
|
|
260
|
+
- The **job-level reconciliation wrapper** (~30 lines).
|
|
261
|
+
- **Steering drain as a committed, row-locked txn before payload build** with
|
|
262
|
+
delivery-time `conversation_order`.
|
|
263
|
+
|
|
264
|
+
## Delivery contract (README)
|
|
265
|
+
|
|
266
|
+
Exactly-once (in SolidLoop's DB): canonical assistant message + loop counters per
|
|
267
|
+
logical LLM turn; canonical tool response + ToolCall checkpoint per ToolCall.
|
|
268
|
+
At-least-once: pending job delivery (claim CAS makes duplicates harmless); LLM
|
|
269
|
+
provider request; tool invocation/effect (unless the handler dedupes the stable
|
|
270
|
+
ToolCall key atomically with its effect). Best-effort: streaming UI chunks.
|
|
271
|
+
|
|
272
|
+
## Guarantees and their limits
|
|
273
|
+
|
|
274
|
+
1. Wider duplicate-call window: any death between provider-send and canonical
|
|
275
|
+
commit re-sends the whole request (possible duplicate side effect). This window
|
|
276
|
+
is narrowed, not closed.
|
|
277
|
+
2. Reclaim latency up to `ttl + reaper interval` (~13 min worst case at a 600s
|
|
278
|
+
read timeout) instead of heartbeat-scale. "Permanently stranded" is fully
|
|
279
|
+
closed.
|
|
280
|
+
3. No exact-request-bytes retry ⇒ no provider idempotency keys offered (near-
|
|
281
|
+
useless anyway).
|
|
282
|
+
4. No per-acquisition audit rows (logs + an optional `reclaim_count`).
|
|
283
|
+
5. Mid-stream partial text hidden by default (hosts opt into showing processing
|
|
284
|
+
shells — a one-line host change).
|
|
285
|
+
6. Migration is **additive**, not destructive: the new columns are nullable, so
|
|
286
|
+
old terminal loops sit at `NULL` (correct — they are not in flight). The only
|
|
287
|
+
cleanup is any dangling `running` loop with no `execution_token` (would trip
|
|
288
|
+
the CHECK) — mark it `failed`, and the reaper heals such rows anyway. A
|
|
289
|
+
consumer's own domain tables are untouched. "Breaking" here means we do not
|
|
290
|
+
write careful online-rollout code and the release notes declare incompatibility
|
|
291
|
+
with the prior release — not that data is destroyed.
|
|
292
|
+
7. Reaper case 2's "exactly one re-enqueue" holds for the UNEXECUTED sub-case
|
|
293
|
+
(the CAS-claimed lease fences the losing reaper). The dropped-CONTINUATION
|
|
294
|
+
sub-case (a call `executed_at` set but its canonical response/successor never
|
|
295
|
+
committed) carries no lease to CAS, so two racing reapers may both re-enqueue
|
|
296
|
+
it — harmless: `ResponseCreation`'s `response_message` idempotency check plus
|
|
297
|
+
the completion-gate `transition_status` fence make the duplicate a no-op and
|
|
298
|
+
the next LLM turn still fires exactly once. (At-least-once job delivery,
|
|
299
|
+
claim-CAS-dedupes, per the delivery contract.)
|
|
300
|
+
|
|
301
|
+
"Two racing reapers ⇒ one acts" is LITERALLY true for cases 1, 2-unexecuted, 3,
|
|
302
|
+
AND 4. Cases 1 / 2-unexecuted / 3 resolve to one winner under the row lock
|
|
303
|
+
(expiry re-check / CAS-claimed lease / under-lock re-confirmation). Case 4
|
|
304
|
+
(`queued` past threshold) BUMPS `updated_at` inside the same row-locked txn
|
|
305
|
+
BEFORE the enqueue, so the losing reaper re-reads `updated_at < threshold` as
|
|
306
|
+
false and does not re-enqueue. The ONE remaining duplicate-but-harmless case is
|
|
307
|
+
**case-2-CONTINUATION** (above): it carries no lease/marker to CAS, so a
|
|
308
|
+
duplicate `ToolExecutionJob` delivery is possible but idempotent (CAS-deduped
|
|
309
|
+
at ResponseCreation).
|
|
310
|
+
|
|
311
|
+
## Known limitations
|
|
312
|
+
|
|
313
|
+
- Continuation failure after a tool checkpoint surfaces late (the reaper
|
|
314
|
+
redelivers the checkpointed call; the tool body is not repeated).
|
|
315
|
+
- At-least-once external tool effects for slow in-process/custom tools reclaimed
|
|
316
|
+
past `default_tool_timeout` (the canonical DB result is still exactly one).
|
|
317
|
+
- Tool-lease derivation bounds configured HTTP MCP calls only; opaque custom
|
|
318
|
+
transports and cumulative setup can exceed the derived lease.
|
|
319
|
+
- `SolidLoop.last_reaped_at` is process-local, not fleet-wide.
|
|
320
|
+
- Mixed old/new workers across the lease upgrade are unsupported (drain or deploy
|
|
321
|
+
atomically).
|
|
322
|
+
- Lease timestamps are Rails UTC `:datetime`.
|
|
323
|
+
- Reaper scans/indexes are adequate but not scale-tuned.
|
|
324
|
+
- Middleware that REPLACES (rather than wraps) a canonical finalizer forfeits the
|
|
325
|
+
durability guarantees unless it reproduces the fences itself.
|
|
326
|
+
|
|
327
|
+
## Configuration / tunables
|
|
328
|
+
|
|
329
|
+
```ruby
|
|
330
|
+
SolidLoop.configure do |c|
|
|
331
|
+
c.config.lease_margin # slack (s) added on top of a client timeout when deriving a lease
|
|
332
|
+
c.config.default_read_timeout # fallback read_timeout when an agent's provider omits one; single source for the derived LLM lease and the HTTP client fallback
|
|
333
|
+
c.config.default_tool_timeout # ceiling for an in-process/custom tool's lease
|
|
334
|
+
c.config.lease_leak_grace # grace (s) on top of an agent's max_duration → the renewer's leak ceiling; > 0
|
|
335
|
+
c.config.default_max_duration # fallback max_duration (s) sizing the leak ceiling when a registration omits one
|
|
336
|
+
end
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
The invariant `lease_margin > 0` guarantees `lease > read_timeout` (LLM) and
|
|
340
|
+
`lease > tool timeout` (tools) per claim; the heartbeat then keeps a healthy LLM
|
|
341
|
+
turn's lease alive for its full duration. `Adapters::Native`'s fallback read
|
|
342
|
+
timeout reads `default_read_timeout` (the SAME value the LLM lease derives from),
|
|
343
|
+
so the derived lease can never be shorter than the client timeout. `ReaperJob`
|
|
344
|
+
uses the `solid_loop_maintenance` queue; hosts allowlisting queues must consume
|
|
345
|
+
the COMPLETE set (`solid_loop_llm`, `solid_loop_tool`, `solid_loop_maintenance`).
|
|
346
|
+
`Reaper::Result` is a diagnostic return value, not a precise stable API;
|
|
347
|
+
`cleaned_shell_loops` counts LOOPS, not shell rows.
|
|
348
|
+
|
|
349
|
+
## Test focus
|
|
350
|
+
|
|
351
|
+
The suite is a set of deterministic scenarios using a failpoint +
|
|
352
|
+
`SimulatedHardCrash < Exception`, on real Postgres, non-transactional where locks
|
|
353
|
+
must contend: kill after LLM claim; kill mid-stream (shell failed+hidden, absent
|
|
354
|
+
from retry payload, assert TWO HTTP calls = the documented at-least-once
|
|
355
|
+
boundary); redelivery after commit no-ops; renamed agent / provider
|
|
356
|
+
`NoMethodError` → loop failed + token cleared; pause mid-stream → late commit
|
|
357
|
+
fenced, resume clean; pause during long tool → late tool write fenced; tool kill
|
|
358
|
+
after side effect pre-`executed_at` → reinvoked once, one canonical result (stable
|
|
359
|
+
key); two parallel tools, one dies → sibling unaffected; cancel-vs-drain on two
|
|
360
|
+
connections; two reapers race one expired lease; tool-phase loop (lease NULL)
|
|
361
|
+
never falsely reclaimed + case-3 gap repair; steering delivered exactly once
|
|
362
|
+
across a retried turn; heartbeat renewal under full pool saturation; renewer
|
|
363
|
+
singleton creation race; the leak ceiling keeps a healthy long turn alive but
|
|
364
|
+
drops a leaked one.
|
|
@@ -0,0 +1,135 @@
|
|
|
1
|
+
# Decision: MCP-Only Tooling (removal of native Ruby tools)
|
|
2
|
+
|
|
3
|
+
> **Status: implemented in 0.0.3.**
|
|
4
|
+
|
|
5
|
+
**Date:** 2026-07-08
|
|
6
|
+
**Companion:** [mcp_transports.md](../guides/mcp_transports.md) — reference documentation for the new entities.
|
|
7
|
+
|
|
8
|
+
## Decision
|
|
9
|
+
|
|
10
|
+
SolidLoop drops the native Ruby tool path entirely. **Every tool is an MCP tool.**
|
|
11
|
+
MCP becomes the single tool *interface*; transports are pluggable *implementations*
|
|
12
|
+
behind a small port (`SolidLoop::Mcp::Transport`):
|
|
13
|
+
|
|
14
|
+
| Transport | What it is | Ships with the gem |
|
|
15
|
+
|---|---|---|
|
|
16
|
+
| `Mcp::HttpTransport` | Streamable-HTTP MCP (today's `McpClient` Faraday path) | yes (reference adapter) |
|
|
17
|
+
| `Mcp::Toolset` | In-process MCP server over the host app's service layer — the new home for what used to be native Ruby tools | yes (base class + DSL) |
|
|
18
|
+
| `Mcp::StdioTransport` | Ephemeral child-process MCP server, spawn-per-call | yes, with a restricted contract |
|
|
19
|
+
| custom | Anything implementing the port | user-provided |
|
|
20
|
+
|
|
21
|
+
Users group their tools into domain toolsets ("search", "weather", "files" …) —
|
|
22
|
+
each toolset is a logical MCP server. Transports can be mixed per agent: one domain
|
|
23
|
+
external over HTTP, another in-process, another stdio.
|
|
24
|
+
|
|
25
|
+
## Why
|
|
26
|
+
|
|
27
|
+
1. **One execution path.** Today `ToolExecution` middleware has two resolution
|
|
28
|
+
branches (Ruby class match by `FUNCTION_NAME`/class name, then `mcp_tools`
|
|
29
|
+
lookup), `MessageBuilding` merges two schema sources, and the unknown-tool
|
|
30
|
+
message unions both lists. All of that collapses to one path: `loop.mcp_tools`.
|
|
31
|
+
2. **Uniform observability.** Every tool call becomes a JSON-RPC envelope with a
|
|
32
|
+
wire log and an admin event — native calls today bypass the MCP logging.
|
|
33
|
+
3. **Uniform discovery.** Schemas always come from `tools/list`; no more
|
|
34
|
+
hand-written `to_tool_spec` on ad-hoc classes.
|
|
35
|
+
4. **Ports & adapters.** The transport port makes "local MCP" a first-class
|
|
36
|
+
concept instead of a special case, and lets users bring their own transport.
|
|
37
|
+
5. **Timing.** At decision time, 0.0.1 and 0.0.2 were internal milestones;
|
|
38
|
+
0.0.3 became the first public release. The native-tool API could therefore
|
|
39
|
+
be removed cleanly, with no deprecation cycle and no external users to
|
|
40
|
+
migrate.
|
|
41
|
+
|
|
42
|
+
## What is removed / changed (blast radius)
|
|
43
|
+
|
|
44
|
+
Removed:
|
|
45
|
+
- `SolidLoop::Base#tools` (`attr_accessor :tools`, `app/models/solid_loop/base.rb`).
|
|
46
|
+
- The "Try Ruby Tool" branch in `ToolMiddlewares::ToolExecution#call`
|
|
47
|
+
(constantize + `FUNCTION_NAME`/class-name matching) and the native half of
|
|
48
|
+
`build_unknown_tool_message`.
|
|
49
|
+
- The native merge in `Middlewares::MessageBuilding`
|
|
50
|
+
(`env.agent.tools.map { .to_tool_spec }`); payload tools become
|
|
51
|
+
`env.loop.mcp_tools.ordered.map(&:to_tool_spec)` only.
|
|
52
|
+
- The `to_tool_spec` / `FUNCTION_NAME` conventions for user tool classes
|
|
53
|
+
(dummy app `CheckTaskTool`, `RaisingTool` are rewritten as a test toolset).
|
|
54
|
+
|
|
55
|
+
Changed:
|
|
56
|
+
- `McpClient` delegates all I/O to a `Transport` (see companion doc); the current
|
|
57
|
+
Faraday code moves verbatim into `Mcp::HttpTransport`.
|
|
58
|
+
- `McpSessionInitializer` / `McpToolExecutionService` stop reading Faraday
|
|
59
|
+
internals (`resp&.env&.method`, response objects) and consume the
|
|
60
|
+
transport-agnostic `Mcp::Result` instead. Wire logs keep working for every
|
|
61
|
+
transport — this preserves the observability requirement from
|
|
62
|
+
`docs/decisions/ruby_llm_rejected.md`.
|
|
63
|
+
- `Base#mcps` config accepts new entry forms (`toolset:`, `transport:`, `prefix:`)
|
|
64
|
+
alongside today's `url:` form. See companion doc.
|
|
65
|
+
- `McpSession` gains `transport_kind` (`http` / `local` / `stdio` / `custom`).
|
|
66
|
+
Nothing is published yet, so the column goes directly into the init
|
|
67
|
+
migration (`20260408000100_solid_loop_init.rb`).
|
|
68
|
+
- Unknown-tool feedback message lists `loop.mcp_tools.pluck(:name)` only.
|
|
69
|
+
|
|
70
|
+
Unchanged:
|
|
71
|
+
- `McpSession` / `McpTool` persistence, per-loop session rows, the admin UI,
|
|
72
|
+
the MCP Inspector, `tools:` whitelists and `required_tools` validation,
|
|
73
|
+
the `mcp://<server>/<tool>` metadata URL — all apply uniformly to every
|
|
74
|
+
transport.
|
|
75
|
+
|
|
76
|
+
## Migration guide (native tool → toolset)
|
|
77
|
+
|
|
78
|
+
Before (0.0.2):
|
|
79
|
+
|
|
80
|
+
```ruby
|
|
81
|
+
class CheckTaskTool
|
|
82
|
+
FUNCTION_NAME = "check_task"
|
|
83
|
+
def self.to_tool_spec = { type: "function", function: { name: "check_task", ... } }
|
|
84
|
+
def call(tool_call) = "Task verified: #{tool_call.arguments["path"]}"
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
class MyAgent < SolidLoop::Base
|
|
88
|
+
def tools = [CheckTaskTool]
|
|
89
|
+
end
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
After:
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
class TaskTools < SolidLoop::Mcp::Toolset
|
|
96
|
+
server_name "tasks"
|
|
97
|
+
|
|
98
|
+
tool "check_task",
|
|
99
|
+
description: "Verifies the task completed successfully",
|
|
100
|
+
input_schema: { type: "object",
|
|
101
|
+
properties: { path: { type: "string" } } } do |args, ctx|
|
|
102
|
+
"Task verified: #{args["path"]}"
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
class MyAgent < SolidLoop::Base
|
|
107
|
+
def mcps = [ { name: :tasks, toolset: TaskTools } ]
|
|
108
|
+
end
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## Known trade-offs
|
|
112
|
+
|
|
113
|
+
- **Ergonomics tax.** The 80% case "expose one Ruby method as a tool" gets a
|
|
114
|
+
toolset + explicit JSON schema instead of a bare class. The toolset DSL is the
|
|
115
|
+
make-or-break piece; it must stay one-screen simple. Mitigation: schema
|
|
116
|
+
helpers and a generator are on the table if the raw DSL feels heavy.
|
|
117
|
+
- **stdio does not fit the durable-async core cleanly.** A stdio server's
|
|
118
|
+
session state lives in a child process owned by whichever worker spawned it;
|
|
119
|
+
the next tool call runs in a different job, possibly on a different machine.
|
|
120
|
+
`StdioTransport` therefore has a restricted contract: spawn-per-call,
|
|
121
|
+
stateless servers only, no cross-round server memory. Stateful stdio servers
|
|
122
|
+
(running shells, browser sessions, REPLs) are explicitly unsupported.
|
|
123
|
+
- **Tool-name collisions** become likely with several servers per agent; solved
|
|
124
|
+
with per-entry `prefix:` + hard validation at session init (companion doc).
|
|
125
|
+
|
|
126
|
+
## Out of scope (future work, deliberately not designed here)
|
|
127
|
+
|
|
128
|
+
- Per-tool execution policy (`auto` / `requires_approval`) for HITL gating —
|
|
129
|
+
orthogonal layer on top of `ToolExecution`; the toolset/port design leaves
|
|
130
|
+
room for it but does not define it.
|
|
131
|
+
- Server-initiated MCP notifications / streaming `tools/call` over local and
|
|
132
|
+
stdio transports; first iteration is synchronous request/response only.
|
|
133
|
+
|
|
134
|
+
Serving toolsets over HTTP and consuming MCP resources/prompts were subsequently
|
|
135
|
+
implemented in 0.0.4; see [mcp-server.md](mcp-server.md) for the settled design.
|