solid_loop 0.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (137) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +156 -0
  3. data/CODE_OF_CONDUCT.md +67 -0
  4. data/CONTRIBUTING.md +82 -0
  5. data/MIT-LICENSE +21 -0
  6. data/README.md +483 -0
  7. data/Rakefile +11 -0
  8. data/app/assets/javascripts/solid_loop/chart.umd.min.js +14 -0
  9. data/app/assets/javascripts/solid_loop/chartjs-adapter-date-fns.bundle.min.js +7 -0
  10. data/app/assets/javascripts/solid_loop/chartkick.min.js +2 -0
  11. data/app/assets/stylesheets/solid_loop/application.css +15 -0
  12. data/app/controllers/solid_loop/application_controller.rb +29 -0
  13. data/app/controllers/solid_loop/dashboard_controller.rb +104 -0
  14. data/app/controllers/solid_loop/events_controller.rb +12 -0
  15. data/app/controllers/solid_loop/loops_controller.rb +77 -0
  16. data/app/controllers/solid_loop/mcp_inbound_sessions_controller.rb +13 -0
  17. data/app/controllers/solid_loop/mcp_sessions_controller.rb +68 -0
  18. data/app/controllers/solid_loop/mcp_tools_controller.rb +16 -0
  19. data/app/controllers/solid_loop/messages_controller.rb +12 -0
  20. data/app/controllers/solid_loop/tool_calls_controller.rb +12 -0
  21. data/app/helpers/solid_loop/application_helper.rb +125 -0
  22. data/app/jobs/solid_loop/application_job.rb +13 -0
  23. data/app/jobs/solid_loop/llm_completion_job.rb +139 -0
  24. data/app/jobs/solid_loop/observe_broadcast_job.rb +15 -0
  25. data/app/jobs/solid_loop/reaper_job.rb +20 -0
  26. data/app/jobs/solid_loop/tool_execution_job.rb +200 -0
  27. data/app/models/solid_loop/application_record.rb +5 -0
  28. data/app/models/solid_loop/base.rb +190 -0
  29. data/app/models/solid_loop/event.rb +7 -0
  30. data/app/models/solid_loop/loop.rb +225 -0
  31. data/app/models/solid_loop/mcp_inbound_session.rb +25 -0
  32. data/app/models/solid_loop/mcp_session.rb +12 -0
  33. data/app/models/solid_loop/mcp_tool.rb +26 -0
  34. data/app/models/solid_loop/message.rb +69 -0
  35. data/app/models/solid_loop/tool_call.rb +42 -0
  36. data/app/queries/solid_loop/admin/base_query.rb +23 -0
  37. data/app/queries/solid_loop/admin/events_query.rb +31 -0
  38. data/app/queries/solid_loop/admin/loops_query.rb +38 -0
  39. data/app/queries/solid_loop/admin/mcp_inbound_sessions_query.rb +31 -0
  40. data/app/queries/solid_loop/admin/mcp_sessions_query.rb +26 -0
  41. data/app/queries/solid_loop/admin/mcp_tools_query.rb +26 -0
  42. data/app/queries/solid_loop/admin/messages_query.rb +37 -0
  43. data/app/queries/solid_loop/admin/tool_calls_query.rb +34 -0
  44. data/app/services/solid_loop/adapters/native.rb +170 -0
  45. data/app/services/solid_loop/dialects/anthropic.rb +160 -0
  46. data/app/services/solid_loop/dialects/gemini.rb +122 -0
  47. data/app/services/solid_loop/dialects/open_ai.rb +62 -0
  48. data/app/services/solid_loop/dialects/reasoning_packer.rb +37 -0
  49. data/app/services/solid_loop/llm_usage_parser/anthropic.rb +22 -0
  50. data/app/services/solid_loop/llm_usage_parser/gemini.rb +18 -0
  51. data/app/services/solid_loop/llm_usage_parser/llama.rb +15 -0
  52. data/app/services/solid_loop/llm_usage_parser/openai.rb +18 -0
  53. data/app/services/solid_loop/llm_usage_parser.rb +19 -0
  54. data/app/services/solid_loop/mcp_session_initializer.rb +205 -0
  55. data/app/services/solid_loop/mcp_tool_execution_service.rb +153 -0
  56. data/app/services/solid_loop/middlewares/agent_initialization.rb +37 -0
  57. data/app/services/solid_loop/middlewares/error_handling.rb +107 -0
  58. data/app/services/solid_loop/middlewares/event_logging.rb +100 -0
  59. data/app/services/solid_loop/middlewares/message_building.rb +92 -0
  60. data/app/services/solid_loop/middlewares/network_calling.rb +84 -0
  61. data/app/services/solid_loop/middlewares/response_parsing.rb +348 -0
  62. data/app/services/solid_loop/middlewares/tool_call_xml_parser.rb +117 -0
  63. data/app/services/solid_loop/sse_stream_aggregator.rb +105 -0
  64. data/app/services/solid_loop/tool_failure_reconciler.rb +122 -0
  65. data/app/services/solid_loop/tool_middlewares/agent_initialization.rb +17 -0
  66. data/app/services/solid_loop/tool_middlewares/error_handling.rb +66 -0
  67. data/app/services/solid_loop/tool_middlewares/event_logging.rb +27 -0
  68. data/app/services/solid_loop/tool_middlewares/response_creation.rb +110 -0
  69. data/app/services/solid_loop/tool_middlewares/tool_execution.rb +206 -0
  70. data/app/views/layouts/solid_loop/admin.html.erb +416 -0
  71. data/app/views/layouts/solid_loop/application.html.erb +17 -0
  72. data/app/views/solid_loop/dashboard/index.html.erb +184 -0
  73. data/app/views/solid_loop/events/index.html.erb +47 -0
  74. data/app/views/solid_loop/events/show.html.erb +76 -0
  75. data/app/views/solid_loop/loops/index.html.erb +83 -0
  76. data/app/views/solid_loop/loops/show.html.erb +148 -0
  77. data/app/views/solid_loop/mcp_inbound_sessions/index.html.erb +53 -0
  78. data/app/views/solid_loop/mcp_inbound_sessions/show.html.erb +78 -0
  79. data/app/views/solid_loop/mcp_sessions/_tool_output.html.erb +11 -0
  80. data/app/views/solid_loop/mcp_sessions/index.html.erb +46 -0
  81. data/app/views/solid_loop/mcp_sessions/inspector.html.erb +94 -0
  82. data/app/views/solid_loop/mcp_sessions/show.html.erb +142 -0
  83. data/app/views/solid_loop/mcp_tools/index.html.erb +44 -0
  84. data/app/views/solid_loop/mcp_tools/show.html.erb +69 -0
  85. data/app/views/solid_loop/messages/_message.html.erb +75 -0
  86. data/app/views/solid_loop/messages/index.html.erb +80 -0
  87. data/app/views/solid_loop/messages/show.html.erb +121 -0
  88. data/app/views/solid_loop/shared/_pagination.html.erb +21 -0
  89. data/app/views/solid_loop/tool_calls/index.html.erb +53 -0
  90. data/app/views/solid_loop/tool_calls/show.html.erb +59 -0
  91. data/config/routes.rb +27 -0
  92. data/db/migrate/20260408000100_solid_loop_init.rb +136 -0
  93. data/db/migrate/20260709000100_solid_loop_create_mcp_inbound_sessions.rb +16 -0
  94. data/db/migrate/20260713000100_solid_loop_add_execution_guards.rb +6 -0
  95. data/db/migrate/20260714000100_solid_loop_add_message_steering.rb +17 -0
  96. data/db/migrate/20260714000200_solid_loop_add_message_conversation_order.rb +14 -0
  97. data/db/migrate/20260715000100_solid_loop_add_mcp_inbound_session_principal_key.rb +43 -0
  98. data/db/migrate/20260715000200_solid_loop_add_llm_lease.rb +40 -0
  99. data/db/migrate/20260715000300_solid_loop_add_tool_lease.rb +33 -0
  100. data/db/migrate/20260715000400_solid_loop_add_lease_running_check.rb +32 -0
  101. data/db/migrate/20260715000500_solid_loop_add_tool_lease_pair_check.rb +35 -0
  102. data/docs/contributing/coverage.md +64 -0
  103. data/docs/decisions/durable_attempt_lease.md +364 -0
  104. data/docs/decisions/mcp-only-tooling.md +135 -0
  105. data/docs/decisions/mcp-server.md +223 -0
  106. data/docs/decisions/reasoning_persistence.md +51 -0
  107. data/docs/decisions/ruby_llm_rejected.md +35 -0
  108. data/docs/guides/dialects.md +55 -0
  109. data/docs/guides/llm_middlewares.md +294 -0
  110. data/docs/guides/mcp_transports.md +374 -0
  111. data/docs/guides/tool_middlewares.md +148 -0
  112. data/lib/solid_loop/configuration.rb +175 -0
  113. data/lib/solid_loop/engine.rb +14 -0
  114. data/lib/solid_loop/lease_heartbeat.rb +94 -0
  115. data/lib/solid_loop/lease_renewer.rb +347 -0
  116. data/lib/solid_loop/llm_metrics.rb +16 -0
  117. data/lib/solid_loop/mcp/call_context.rb +19 -0
  118. data/lib/solid_loop/mcp/client_factory.rb +61 -0
  119. data/lib/solid_loop/mcp/http_transport.rb +52 -0
  120. data/lib/solid_loop/mcp/principal.rb +82 -0
  121. data/lib/solid_loop/mcp/result.rb +13 -0
  122. data/lib/solid_loop/mcp/server.rb +246 -0
  123. data/lib/solid_loop/mcp/stdio_transport.rb +224 -0
  124. data/lib/solid_loop/mcp/toolset.rb +347 -0
  125. data/lib/solid_loop/mcp/transport.rb +20 -0
  126. data/lib/solid_loop/mcp.rb +25 -0
  127. data/lib/solid_loop/mcp_client.rb +176 -0
  128. data/lib/solid_loop/pipeline/builder.rb +38 -0
  129. data/lib/solid_loop/pipeline/context.rb +61 -0
  130. data/lib/solid_loop/pipeline/tool_context.rb +53 -0
  131. data/lib/solid_loop/pipeline.rb +40 -0
  132. data/lib/solid_loop/reaper.rb +313 -0
  133. data/lib/solid_loop/version.rb +3 -0
  134. data/lib/solid_loop.rb +94 -0
  135. data/lib/tasks/coverage.rake +206 -0
  136. data/lib/tasks/solid_loop_tasks.rake +4 -0
  137. metadata +228 -0
@@ -0,0 +1,364 @@
1
+ # Durable lease + reaper
2
+
3
+ Status: accepted design.
4
+
5
+ This document describes the durable-lease design that makes SolidLoop's LLM turns
6
+ and tool invocations crash/restart-safe: how a worker claims a unit of work, how
7
+ its lease is kept alive while it runs, how a periodic reaper recovers work whose
8
+ worker died, and the guarantees (and their limits) that result.
9
+
10
+ ## Context
11
+
12
+ - **Breaking DB changes are acceptable.** A consumer can reset its dev DB and
13
+ hand-fix records, so there is **no online migration, no backfill, no quiesced
14
+ rollout, no recovery task** — the rollout story is "drop, recreate, reset."
15
+ - Durable, crash/restart-safe resumes are a release requirement: without them,
16
+ the claim "durable, crash/restart resumes" is false.
17
+ - Bar: **correct + as simple to build and maintain as possible**, not
18
+ zero-downtime migration of a live fleet.
19
+
20
+ ## What the existing code already provides
21
+
22
+ - `LlmCompletionJob.perform(loop_id)` **already CAS-claims**:
23
+ `UPDATE ... SET status=running, execution_token=uuid WHERE status=queued`.
24
+ - `solid_loop_tool_calls.executed_at` is **already the tool checkpoint**;
25
+ `ToolExecutionJob(tool_call_id, execution_token)` already carries the fence.
26
+ - The stable ToolCall idempotency key already propagates via `CallContext` /
27
+ MCP `_meta`.
28
+ - The streaming shell is a plain message row (`status: "processing"`,
29
+ `is_hidden` default false); `MessageBuilding` assembles `where(is_hidden:false)`,
30
+ so hiding a mid-stream partial needs **zero new message columns**.
31
+
32
+ ## Design
33
+
34
+ **Schema: 3 new columns, 2 CHECKs, 2 indexes. No new tables or models.**
35
+
36
+ ```sql
37
+ ALTER TABLE solid_loop_loops
38
+ ADD COLUMN lease_expires_at timestamptz; -- non-NULL iff an LLM turn is in flight
39
+ ALTER TABLE solid_loop_loops
40
+ ADD CONSTRAINT running_requires_token
41
+ CHECK (status <> 'running' OR execution_token IS NOT NULL);
42
+ ALTER TABLE solid_loop_loops
43
+ ADD CONSTRAINT lease_running_only
44
+ CHECK (lease_expires_at IS NULL OR status = 'running');
45
+
46
+ ALTER TABLE solid_loop_tool_calls
47
+ ADD COLUMN lease_token varchar,
48
+ ADD COLUMN leased_until timestamptz;
49
+ ALTER TABLE solid_loop_tool_calls
50
+ ADD CONSTRAINT tool_lease_pair
51
+ CHECK ((lease_token IS NULL) = (leased_until IS NULL));
52
+
53
+ -- messages: NO new columns (status 'processing' + is_hidden = the shell protocol)
54
+ -- indexes: loops(status, lease_expires_at); tool_calls(leased_until) WHERE executed_at IS NULL
55
+ ```
56
+
57
+ ### State model
58
+
59
+ - **Loop** (existing statuses): `queued` = durable LLM work intent; `running` +
60
+ `lease_expires_at` set = LLM HTTP turn in flight; `running` + `lease_expires_at`
61
+ NULL = tool phase (the LLM commit clears it when emitting tools).
62
+ `execution_token` is BOTH the generation fence AND the LLM lease token —
63
+ rotated at every claim/resume, cleared at pause/stop/reclaim.
64
+ - **ToolCall**: `executed_at IS NULL` = durable intent; live lease = `lease_token`
65
+ set and `leased_until > now`; `executed_at` set = terminal (the existing CAS
66
+ `WHERE executed_at IS NULL` stays the exactly-once-in-DB guard). Two tokens
67
+ exist ONLY because reclaiming one dead parallel tool must not fence healthy
68
+ siblings — the one place a second token is load-bearing (parallel tools are the
69
+ shipped default: `Base#sequential_tool_calls? => false`).
70
+ - **Shell message** (born-hidden protocol): created `status:"processing",
71
+ is_hidden:true`; the ONLY path that flips it visible is the canonical LLM commit
72
+ txn (which also fence-checks); reclaim/pause/stop/error mark any `processing`
73
+ shell for the loop `failed` and leave it hidden (partial content kept for
74
+ admin/audit). `MessageBuilding`'s existing `is_hidden:false` filter already
75
+ excludes it. Force-stop must not mark partial output `success`.
76
+
77
+ ### Claim
78
+
79
+ LLM: today's CAS plus `lease_expires_at = now + ttl`. Tool:
80
+ `UPDATE tool_calls SET lease_token=uuid, leased_until=now+ttl
81
+ WHERE id=? AND executed_at IS NULL AND (lease_token IS NULL OR leased_until < now)`
82
+ after verifying `loop.execution_token` matches the job arg; 0 rows ⇒ no-op.
83
+
84
+ **The lease is derived PER CLAIM so `lease > client timeout + margin` holds BY
85
+ CONSTRUCTION** for both LLM and tool — a live worker can never hold an expired
86
+ lease *for the duration of a single inactivity window* (its HTTP/tool client
87
+ raises first). LLM: `read_timeout (600s default) + lease_margin`, from the agent's
88
+ own provider. Tool: `max(default_tool_timeout, slowest resolved HTTP MCP client
89
+ timeout) + lease_margin` — the HTTP timeout (`mcp_config[:timeout] || 60`,
90
+ operator-settable well above the default) MUST be folded in, or a *healthy*
91
+ long-running HTTP tool could be reclaimed → duplicate invocation on a live call.
92
+ Residual (documented in `Configuration`): an in-process/custom tool exposes no
93
+ client timeout to derive from, so `default_tool_timeout` is its ceiling; one that
94
+ legitimately runs longer MAY be reclaimed (a duplicate invocation — harmless: the
95
+ both-token fence yields exactly one canonical result and tools carry the stable
96
+ `solid_loop:tool_call:#{id}` idempotency key), so operators must set
97
+ `default_tool_timeout` above their slowest in-process tool.
98
+
99
+ ### Host resume contract — revoke the tool lease before re-enqueue
100
+
101
+ Any host-side `paused→running` (or `failed→running`) transition that re-enqueues a
102
+ `ToolExecutionJob` for an unexecuted tool_call MUST FIRST revoke that call's live
103
+ per-tool lease (`lease_token`/`leased_until` → NULL), or the re-enqueued job's
104
+ self-claim CAS (the `lease_token IS NULL OR leased_until < now` predicate above)
105
+ matches zero rows and no-ops SILENTLY — the loop then strands until the stale
106
+ lease's TTL expires and the reaper (case 2) repairs it (up to `tool_lease_duration
107
+ + reaper interval`). `SolidLoop::Base#resume!` is the canonical example: it rotates
108
+ a fresh `execution_token` AND `UPDATE tool_calls SET lease_token=NULL,
109
+ leased_until=NULL WHERE id IN (...) AND executed_at IS NULL` in the same locked
110
+ txn BEFORE enqueuing (`app/models/solid_loop/base.rb`, the pending-tool-calls
111
+ branch). Hosts that hand-roll their own resume must mirror BOTH steps.
112
+ Observability: the claim no-op logs a `Rails.logger.warn` (`ToolExecutionJob:
113
+ tool lease claim lost …`) with the tool_call/loop ids and the surviving
114
+ lease_token/leased_until, so the otherwise-invisible strand is diagnosable.
115
+
116
+ ### LLM heartbeat
117
+
118
+ The claim-time lease derivation above is NOT sufficient on its own for the LLM
119
+ path. `read_timeout` is an **inactivity** timeout (the silence between two
120
+ consecutive socket reads), NOT a bound on total request duration. A healthy model
121
+ can stream for tens of minutes with **every inter-chunk gap** well under
122
+ `read_timeout`, so a static `read_timeout + margin` lease would expire on a LIVE
123
+ worker mid-stream and the reaper (case 1) would reclaim a live turn — a duplicate
124
+ provider call plus a discarded result. Pre-call **MCP session initialization**
125
+ (before the first chunk) is likewise unbounded by `read_timeout`.
126
+
127
+ So `LeaseHeartbeat` (`lib/solid_loop/lease_heartbeat.rb`) renews
128
+ `lease_expires_at` for the ENTIRE claimed LLM job — MCP init + the whole stream
129
+ (incl. a silent prefill) + finalization — on a DEDICATED pooled DB connection
130
+ through the Rails executor, at ~`ttl/4` cadence. Every renew is a CONDITIONAL
131
+ `UPDATE ... WHERE id=loop_id AND status='running' AND execution_token=<ours>`, so
132
+ it can NEVER renew a lease it no longer owns; a 0-row renew sets a thread-safe
133
+ LOST-LEASE flag. The main path calls `heartbeat.check!` (raising
134
+ `SolidLoop::LostLease`) BEFORE any canonical write/checkpoint (in
135
+ `ResponseParsing`), so a worker whose lease was lost commits nothing; the job
136
+ swallows `LostLease` (recovery belongs to whoever now owns the generation). The
137
+ heartbeat stops in an `ensure` before the canonical completion, joining its thread
138
+ and returning its connection (no pool leak, no thread left running in tests).
139
+ Correctness still ultimately rests on the commit-time fence check under the row
140
+ lock; the heartbeat's job is only to keep a HEALTHY turn from being falsely
141
+ reclaimed. The tool path keeps the static per-claim lease (a tool invocation is a
142
+ single bounded client call, so its lease derivation IS sufficient — no tool
143
+ heartbeat).
144
+
145
+ ### The process-wide lease renewer
146
+
147
+ The heartbeat is a thin registration handle over a SINGLE process-wide
148
+ `SolidLoop::LeaseRenewer` (`lib/solid_loop/lease_renewer.rb`), not one thread per
149
+ turn. Each in-flight `LlmCompletionJob` registers its
150
+ `(loop_id, execution_token, lease_duration, max_duration)` on start and
151
+ deregisters when the turn ends; the renewer renews ALL registered leases from ONE
152
+ long-lived connection checked out once, so renewal is **O(1) in connections, not
153
+ O(concurrency)**. A per-turn renewer thread each checking out its own connection
154
+ per tick would livelock at full worker-pool occupancy (pool == concurrency, the
155
+ standard shape): all N connections held by busy workers, so every heartbeat blocks
156
+ in `with_connection`, treats `checkout_timeout` as a lost lease, yields its turn
157
+ uncommitted, and the reaper requeues into the same saturated pool. The single
158
+ renewer avoids this entirely. Deployment requirement: size the DB pool as
159
+ `worker_concurrency + 1` (documented in the README).
160
+
161
+ `LeaseHeartbeat` keeps the public surface (`run`/`start`/`stop`/`lost?`/`check!`)
162
+ and the ownership contract (conditional renew keyed on loop+token,
163
+ `lease_expires_at IS NOT NULL` guard, per-registration lost flag,
164
+ stop-before-commit). The renewer is created via a class-level constant mutex
165
+ (`LeaseRenewer::MUTEX`) wrapping BOTH `@instance ||= new` and the reset-to-nil
166
+ swap, so concurrent first-turns can't each construct a distinct renewer (which
167
+ would each start a thread contending for the one process-lifetime connection —
168
+ a livelock). The blocking `stop!` (thread join) runs OUTSIDE the lock so a
169
+ concurrent `instance` never waits on a join.
170
+
171
+ **Leak ceiling.** Leak safety does not depend on the job's `ensure` ever running.
172
+ A registration that never deregisters (a job that skipped its `ensure`) would
173
+ otherwise be renewed forever, so the reaper's case-1 reclaim could never fire.
174
+ Each registration carries a hard ceiling tied to the **owning agent's own legal
175
+ turn bound** — `max_duration + config.lease_leak_grace` (default `2h + 5min`).
176
+ A turn longer than `max_duration` is illegitimate by the agent's own contract, so
177
+ a legitimate turn always finishes before the ceiling (never dropped), while a
178
+ genuine leak self-expires shortly after `max_duration` → its lease lapses → the
179
+ reaper reclaims it. `max_duration` is registered alongside the other fields (the
180
+ `LlmCompletionJob` already resolves the agent; a fallback
181
+ `config.default_max_duration` covers out-of-band callers). A ceiling tied to
182
+ `lease_duration × K` instead would be wrong: a healthy turn stays registered for
183
+ its whole life, so any legitimate stream longer than K lease-widths (e.g. a small
184
+ `read_timeout` with steady chunks, 30 min vs a ~20-min K=20 ceiling) would be
185
+ dropped → its lease would lapse → the reaper would reclaim a live turn → the
186
+ replacement would hit the same ceiling → no long turn could ever complete.
187
+
188
+ The renewer intentionally runs for the whole process lifetime; its single
189
+ connection is released at process exit (no shutdown hook needed — nothing to
190
+ drain between turns). Each registration's `lost` flag is a
191
+ `Concurrent::AtomicBoolean` so the renewer-thread write and main-path read carry a
192
+ memory barrier on every Ruby engine, not only MRI+GVL.
193
+
194
+ ### Reclaim
195
+
196
+ One idempotent `SolidLoop.reap!` service on a host-installed recurrence (30–60s).
197
+ Each case is a short row-locked txn that re-checks expiry under lock (two racing
198
+ reapers ⇒ one wins):
199
+
200
+ 1. `running`, `lease_expires_at < now` → dead LLM turn: fail+hide the processing
201
+ shell, rotate the token away, loop → `queued`, re-enqueue
202
+ `LlmCompletionJob(loop_id)`. If the loop is paused/terminal/frozen, just clean
203
+ the shell.
204
+ 2. `running`, lease NULL, an unresolved tool_call with expired/absent lease →
205
+ re-enqueue `ToolExecutionJob(id, current execution_token)` (claim CAS dedupes;
206
+ healthy siblings untouched).
207
+ 3. `running`, lease NULL, zero unresolved tools → dropped-enqueue repair: loop →
208
+ `queued`, enqueue the next turn.
209
+ 4. `queued` older than a threshold → re-enqueue (repairs a lost queue row).
210
+
211
+ A backstop sweep (`reap_orphaned_shells`) over paused/terminal loops fails+hides
212
+ any orphaned `processing` shell (cases 1–4 only scan `running`). A gem-side health
213
+ check reports the last successful reap.
214
+
215
+ ### Failure reconciliation
216
+
217
+ Both jobs wrap EVERYTHING — agent `constantize`, provider resolution, MCP init,
218
+ payload build, parse, callbacks — in `rescue StandardError`; if the fence still
219
+ matches under lock: shell → failed+hidden, loop → failed (or paused by error
220
+ class), token + lease cleared. A PRE-lease error takes a no-lease,
221
+ generation-fenced failure path (the loop → failed under the token, no tool_call
222
+ write). Hard kills fall through to the reaper. A renamed/missing agent class or a
223
+ raising config hook now fails the loop VISIBLY instead of stranding it. The two
224
+ job reconcilers share a `ToolFailureReconciler` so their semantics cannot drift.
225
+
226
+ ### Steering
227
+
228
+ First act of an LLM turn after claim, in one committed row-locked txn: lock
229
+ pending steering rows `FOR UPDATE` in ID order; flip to delivered with
230
+ delivery-time `conversation_order`; commit. Payload builds from canonical history
231
+ afterward. Cancel locks the same row — no pluck/find gap. A retried turn re-reads
232
+ history (already-delivered steering included) and may drain newly-arrived
233
+ steering; both are fine WITHOUT exact-bytes retry. The loop-completion txn checks
234
+ pending steering and goes `queued` instead of `completed`.
235
+
236
+ ### Canonical writes under the row lock
237
+
238
+ The tool canonical writes are LOOP-ROW-LOCKED, not `reload`-fenced. Lock order is
239
+ **loop row → tool_call → message** (deterministic ascending id). `ToolExecution`
240
+ holds the loop lock across both-token validation AND the
241
+ `executed_at`/result/`duration_tools` write; `ResponseCreation` holds the SAME
242
+ loop lock across validation, the tool response `message.create!`, the
243
+ completion-gate transition, and the successor enqueue. A `reload` is NOT a fence;
244
+ the row lock freezes `loop.execution_token` + `tool_call.lease_token` between
245
+ validation and mutation, so a stale/revoked worker writes NOTHING. The tool BODY
246
+ runs OUTSIDE the loop lock (a long tool must never serialize pause/stop/resume
247
+ behind the loop lock). `mcp_session_id` is noncanonical telemetry folded into the
248
+ FENCED canonical write (no separate unfenced update), so a stale worker cannot
249
+ overwrite it.
250
+
251
+ ## Load-bearing — must NOT be dropped
252
+
253
+ - The single **canonical-commit transaction** with a fence check under row locks
254
+ (all durable writes happen here or nowhere).
255
+ - **CAS claim** (duplicate job delivery must no-op).
256
+ - **Transactional enqueue with verified `perform_later`** + reaper repair of lost
257
+ queue rows (the "queued forever" failure mode). `SolidLoop.enqueue!` provides
258
+ the verified-enqueue half.
259
+ - **Per-`tool_call` lease columns** (parallel is the shipped default).
260
+ - The **job-level reconciliation wrapper** (~30 lines).
261
+ - **Steering drain as a committed, row-locked txn before payload build** with
262
+ delivery-time `conversation_order`.
263
+
264
+ ## Delivery contract (README)
265
+
266
+ Exactly-once (in SolidLoop's DB): canonical assistant message + loop counters per
267
+ logical LLM turn; canonical tool response + ToolCall checkpoint per ToolCall.
268
+ At-least-once: pending job delivery (claim CAS makes duplicates harmless); LLM
269
+ provider request; tool invocation/effect (unless the handler dedupes the stable
270
+ ToolCall key atomically with its effect). Best-effort: streaming UI chunks.
271
+
272
+ ## Guarantees and their limits
273
+
274
+ 1. Wider duplicate-call window: any death between provider-send and canonical
275
+ commit re-sends the whole request (possible duplicate side effect). This window
276
+ is narrowed, not closed.
277
+ 2. Reclaim latency up to `ttl + reaper interval` (~13 min worst case at a 600s
278
+ read timeout) instead of heartbeat-scale. "Permanently stranded" is fully
279
+ closed.
280
+ 3. No exact-request-bytes retry ⇒ no provider idempotency keys offered (near-
281
+ useless anyway).
282
+ 4. No per-acquisition audit rows (logs + an optional `reclaim_count`).
283
+ 5. Mid-stream partial text hidden by default (hosts opt into showing processing
284
+ shells — a one-line host change).
285
+ 6. Migration is **additive**, not destructive: the new columns are nullable, so
286
+ old terminal loops sit at `NULL` (correct — they are not in flight). The only
287
+ cleanup is any dangling `running` loop with no `execution_token` (would trip
288
+ the CHECK) — mark it `failed`, and the reaper heals such rows anyway. A
289
+ consumer's own domain tables are untouched. "Breaking" here means we do not
290
+ write careful online-rollout code and the release notes declare incompatibility
291
+ with the prior release — not that data is destroyed.
292
+ 7. Reaper case 2's "exactly one re-enqueue" holds for the UNEXECUTED sub-case
293
+ (the CAS-claimed lease fences the losing reaper). The dropped-CONTINUATION
294
+ sub-case (a call `executed_at` set but its canonical response/successor never
295
+ committed) carries no lease to CAS, so two racing reapers may both re-enqueue
296
+ it — harmless: `ResponseCreation`'s `response_message` idempotency check plus
297
+ the completion-gate `transition_status` fence make the duplicate a no-op and
298
+ the next LLM turn still fires exactly once. (At-least-once job delivery,
299
+ claim-CAS-dedupes, per the delivery contract.)
300
+
301
+ "Two racing reapers ⇒ one acts" is LITERALLY true for cases 1, 2-unexecuted, 3,
302
+ AND 4. Cases 1 / 2-unexecuted / 3 resolve to one winner under the row lock
303
+ (expiry re-check / CAS-claimed lease / under-lock re-confirmation). Case 4
304
+ (`queued` past threshold) BUMPS `updated_at` inside the same row-locked txn
305
+ BEFORE the enqueue, so the losing reaper re-reads `updated_at < threshold` as
306
+ false and does not re-enqueue. The ONE remaining duplicate-but-harmless case is
307
+ **case-2-CONTINUATION** (above): it carries no lease/marker to CAS, so a
308
+ duplicate `ToolExecutionJob` delivery is possible but idempotent (CAS-deduped
309
+ at ResponseCreation).
310
+
311
+ ## Known limitations
312
+
313
+ - Continuation failure after a tool checkpoint surfaces late (the reaper
314
+ redelivers the checkpointed call; the tool body is not repeated).
315
+ - At-least-once external tool effects for slow in-process/custom tools reclaimed
316
+ past `default_tool_timeout` (the canonical DB result is still exactly one).
317
+ - Tool-lease derivation bounds configured HTTP MCP calls only; opaque custom
318
+ transports and cumulative setup can exceed the derived lease.
319
+ - `SolidLoop.last_reaped_at` is process-local, not fleet-wide.
320
+ - Mixed old/new workers across the lease upgrade are unsupported (drain or deploy
321
+ atomically).
322
+ - Lease timestamps are Rails UTC `:datetime`.
323
+ - Reaper scans/indexes are adequate but not scale-tuned.
324
+ - Middleware that REPLACES (rather than wraps) a canonical finalizer forfeits the
325
+ durability guarantees unless it reproduces the fences itself.
326
+
327
+ ## Configuration / tunables
328
+
329
+ ```ruby
330
+ SolidLoop.configure do |c|
331
+ c.config.lease_margin # slack (s) added on top of a client timeout when deriving a lease
332
+ c.config.default_read_timeout # fallback read_timeout when an agent's provider omits one; single source for the derived LLM lease and the HTTP client fallback
333
+ c.config.default_tool_timeout # ceiling for an in-process/custom tool's lease
334
+ c.config.lease_leak_grace # grace (s) on top of an agent's max_duration → the renewer's leak ceiling; > 0
335
+ c.config.default_max_duration # fallback max_duration (s) sizing the leak ceiling when a registration omits one
336
+ end
337
+ ```
338
+
339
+ The invariant `lease_margin > 0` guarantees `lease > read_timeout` (LLM) and
340
+ `lease > tool timeout` (tools) per claim; the heartbeat then keeps a healthy LLM
341
+ turn's lease alive for its full duration. `Adapters::Native`'s fallback read
342
+ timeout reads `default_read_timeout` (the SAME value the LLM lease derives from),
343
+ so the derived lease can never be shorter than the client timeout. `ReaperJob`
344
+ uses the `solid_loop_maintenance` queue; hosts allowlisting queues must consume
345
+ the COMPLETE set (`solid_loop_llm`, `solid_loop_tool`, `solid_loop_maintenance`).
346
+ `Reaper::Result` is a diagnostic return value, not a precise stable API;
347
+ `cleaned_shell_loops` counts LOOPS, not shell rows.
348
+
349
+ ## Test focus
350
+
351
+ The suite is a set of deterministic scenarios using a failpoint +
352
+ `SimulatedHardCrash < Exception`, on real Postgres, non-transactional where locks
353
+ must contend: kill after LLM claim; kill mid-stream (shell failed+hidden, absent
354
+ from retry payload, assert TWO HTTP calls = the documented at-least-once
355
+ boundary); redelivery after commit no-ops; renamed agent / provider
356
+ `NoMethodError` → loop failed + token cleared; pause mid-stream → late commit
357
+ fenced, resume clean; pause during long tool → late tool write fenced; tool kill
358
+ after side effect pre-`executed_at` → reinvoked once, one canonical result (stable
359
+ key); two parallel tools, one dies → sibling unaffected; cancel-vs-drain on two
360
+ connections; two reapers race one expired lease; tool-phase loop (lease NULL)
361
+ never falsely reclaimed + case-3 gap repair; steering delivered exactly once
362
+ across a retried turn; heartbeat renewal under full pool saturation; renewer
363
+ singleton creation race; the leak ceiling keeps a healthy long turn alive but
364
+ drops a leaked one.
@@ -0,0 +1,135 @@
1
+ # Decision: MCP-Only Tooling (removal of native Ruby tools)
2
+
3
+ > **Status: implemented in 0.0.3.**
4
+
5
+ **Date:** 2026-07-08
6
+ **Companion:** [mcp_transports.md](../guides/mcp_transports.md) — reference documentation for the new entities.
7
+
8
+ ## Decision
9
+
10
+ SolidLoop drops the native Ruby tool path entirely. **Every tool is an MCP tool.**
11
+ MCP becomes the single tool *interface*; transports are pluggable *implementations*
12
+ behind a small port (`SolidLoop::Mcp::Transport`):
13
+
14
+ | Transport | What it is | Ships with the gem |
15
+ |---|---|---|
16
+ | `Mcp::HttpTransport` | Streamable-HTTP MCP (today's `McpClient` Faraday path) | yes (reference adapter) |
17
+ | `Mcp::Toolset` | In-process MCP server over the host app's service layer — the new home for what used to be native Ruby tools | yes (base class + DSL) |
18
+ | `Mcp::StdioTransport` | Ephemeral child-process MCP server, spawn-per-call | yes, with a restricted contract |
19
+ | custom | Anything implementing the port | user-provided |
20
+
21
+ Users group their tools into domain toolsets ("search", "weather", "files" …) —
22
+ each toolset is a logical MCP server. Transports can be mixed per agent: one domain
23
+ external over HTTP, another in-process, another stdio.
24
+
25
+ ## Why
26
+
27
+ 1. **One execution path.** Today `ToolExecution` middleware has two resolution
28
+ branches (Ruby class match by `FUNCTION_NAME`/class name, then `mcp_tools`
29
+ lookup), `MessageBuilding` merges two schema sources, and the unknown-tool
30
+ message unions both lists. All of that collapses to one path: `loop.mcp_tools`.
31
+ 2. **Uniform observability.** Every tool call becomes a JSON-RPC envelope with a
32
+ wire log and an admin event — native calls today bypass the MCP logging.
33
+ 3. **Uniform discovery.** Schemas always come from `tools/list`; no more
34
+ hand-written `to_tool_spec` on ad-hoc classes.
35
+ 4. **Ports & adapters.** The transport port makes "local MCP" a first-class
36
+ concept instead of a special case, and lets users bring their own transport.
37
+ 5. **Timing.** At decision time, 0.0.1 and 0.0.2 were internal milestones;
38
+ 0.0.3 became the first public release. The native-tool API could therefore
39
+ be removed cleanly, with no deprecation cycle and no external users to
40
+ migrate.
41
+
42
+ ## What is removed / changed (blast radius)
43
+
44
+ Removed:
45
+ - `SolidLoop::Base#tools` (`attr_accessor :tools`, `app/models/solid_loop/base.rb`).
46
+ - The "Try Ruby Tool" branch in `ToolMiddlewares::ToolExecution#call`
47
+ (constantize + `FUNCTION_NAME`/class-name matching) and the native half of
48
+ `build_unknown_tool_message`.
49
+ - The native merge in `Middlewares::MessageBuilding`
50
+ (`env.agent.tools.map { .to_tool_spec }`); payload tools become
51
+ `env.loop.mcp_tools.ordered.map(&:to_tool_spec)` only.
52
+ - The `to_tool_spec` / `FUNCTION_NAME` conventions for user tool classes
53
+ (dummy app `CheckTaskTool`, `RaisingTool` are rewritten as a test toolset).
54
+
55
+ Changed:
56
+ - `McpClient` delegates all I/O to a `Transport` (see companion doc); the current
57
+ Faraday code moves verbatim into `Mcp::HttpTransport`.
58
+ - `McpSessionInitializer` / `McpToolExecutionService` stop reading Faraday
59
+ internals (`resp&.env&.method`, response objects) and consume the
60
+ transport-agnostic `Mcp::Result` instead. Wire logs keep working for every
61
+ transport — this preserves the observability requirement from
62
+ `docs/decisions/ruby_llm_rejected.md`.
63
+ - `Base#mcps` config accepts new entry forms (`toolset:`, `transport:`, `prefix:`)
64
+ alongside today's `url:` form. See companion doc.
65
+ - `McpSession` gains `transport_kind` (`http` / `local` / `stdio` / `custom`).
66
+ Nothing is published yet, so the column goes directly into the init
67
+ migration (`20260408000100_solid_loop_init.rb`).
68
+ - Unknown-tool feedback message lists `loop.mcp_tools.pluck(:name)` only.
69
+
70
+ Unchanged:
71
+ - `McpSession` / `McpTool` persistence, per-loop session rows, the admin UI,
72
+ the MCP Inspector, `tools:` whitelists and `required_tools` validation,
73
+ the `mcp://<server>/<tool>` metadata URL — all apply uniformly to every
74
+ transport.
75
+
76
+ ## Migration guide (native tool → toolset)
77
+
78
+ Before (0.0.2):
79
+
80
+ ```ruby
81
+ class CheckTaskTool
82
+ FUNCTION_NAME = "check_task"
83
+ def self.to_tool_spec = { type: "function", function: { name: "check_task", ... } }
84
+ def call(tool_call) = "Task verified: #{tool_call.arguments["path"]}"
85
+ end
86
+
87
+ class MyAgent < SolidLoop::Base
88
+ def tools = [CheckTaskTool]
89
+ end
90
+ ```
91
+
92
+ After:
93
+
94
+ ```ruby
95
+ class TaskTools < SolidLoop::Mcp::Toolset
96
+ server_name "tasks"
97
+
98
+ tool "check_task",
99
+ description: "Verifies the task completed successfully",
100
+ input_schema: { type: "object",
101
+ properties: { path: { type: "string" } } } do |args, ctx|
102
+ "Task verified: #{args["path"]}"
103
+ end
104
+ end
105
+
106
+ class MyAgent < SolidLoop::Base
107
+ def mcps = [ { name: :tasks, toolset: TaskTools } ]
108
+ end
109
+ ```
110
+
111
+ ## Known trade-offs
112
+
113
+ - **Ergonomics tax.** The 80% case "expose one Ruby method as a tool" gets a
114
+ toolset + explicit JSON schema instead of a bare class. The toolset DSL is the
115
+ make-or-break piece; it must stay one-screen simple. Mitigation: schema
116
+ helpers and a generator are on the table if the raw DSL feels heavy.
117
+ - **stdio does not fit the durable-async core cleanly.** A stdio server's
118
+ session state lives in a child process owned by whichever worker spawned it;
119
+ the next tool call runs in a different job, possibly on a different machine.
120
+ `StdioTransport` therefore has a restricted contract: spawn-per-call,
121
+ stateless servers only, no cross-round server memory. Stateful stdio servers
122
+ (running shells, browser sessions, REPLs) are explicitly unsupported.
123
+ - **Tool-name collisions** become likely with several servers per agent; solved
124
+ with per-entry `prefix:` + hard validation at session init (companion doc).
125
+
126
+ ## Out of scope (future work, deliberately not designed here)
127
+
128
+ - Per-tool execution policy (`auto` / `requires_approval`) for HITL gating —
129
+ orthogonal layer on top of `ToolExecution`; the toolset/port design leaves
130
+ room for it but does not define it.
131
+ - Server-initiated MCP notifications / streaming `tools/call` over local and
132
+ stdio transports; first iteration is synchronous request/response only.
133
+
134
+ Serving toolsets over HTTP and consuming MCP resources/prompts were subsequently
135
+ implemented in 0.0.4; see [mcp-server.md](mcp-server.md) for the settled design.