rails_console_ai 0.32.0 → 0.34.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +13 -0
- data/README.md +46 -5
- data/app/helpers/rails_console_ai/sessions_helper.rb +1 -1
- data/app/views/layouts/rails_console_ai/application.html.erb +5 -0
- data/app/views/rails_console_ai/sessions/index.html.erb +2 -0
- data/lib/generators/rails_console_ai/templates/initializer.rb +7 -0
- data/lib/rails_console_ai/configuration.rb +34 -6
- data/lib/rails_console_ai/conversation_engine.rb +93 -5
- data/lib/rails_console_ai/executor.rb +30 -1
- data/lib/rails_console_ai/safety_guards.rb +75 -0
- data/lib/rails_console_ai/session_logger.rb +13 -1
- data/lib/rails_console_ai/tools/registry.rb +5 -1
- data/lib/rails_console_ai/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 61fcabfed3f03e58e104b7c8b49b86ccf365db5dbae5e1dbf1f8f1b7e63d811c
|
|
4
|
+
data.tar.gz: ec844ca4e1251fb9c1f0c0172095050745ef9690a90defefbcded92eae0962f8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: f2294a51714714de69afcc908dccbcb8d5034999973c40b4533600579e8617efc1133ed918ed7925aa082a3afd5598335b5a09527d1a933b38a6ddbf8143d595
|
|
7
|
+
data.tar.gz: bce68674c9467a92eac6b7521e4f1e39f1d06c32b374f38f627513c7ab31bdda57a12a2e937654103a2322d075b594aa56b610cf2ad716f1965f1c2d6123cfad
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.34.0]
|
|
6
|
+
|
|
7
|
+
- Add an `:in_process_requests` built-in safety guard that blocks in-process HTTP dispatch against the app itself — `ActionDispatch::Integration::Session` requests (the console `app` helper) and direct Rack dispatch (`Rails.application.call`) — for all verbs including GET, since these can hang the session thread indefinitely; allowlist entries are request paths
|
|
8
|
+
- Create/update the session record with a `running` status before each turn's tool loop starts, so a turn that hangs or dies mid-loop still leaves a visible session record, and show each session's status in the admin sessions page
|
|
9
|
+
- Make session logging drop attributes the sessions table doesn't have a column for yet, so a gem newer than the table degrades to a partial row instead of losing the insert
|
|
10
|
+
- Fix the sessions page cost display after the pricing refactor
|
|
11
|
+
|
|
12
|
+
## [0.33.0]
|
|
13
|
+
|
|
14
|
+
- Support Claude Opus 5 and make it the default model for the Anthropic and Bedrock providers
|
|
15
|
+
- Add runaway-loop circuit breakers to the tool loop — break when the same error signature repeats (warn at 3 occurrences, stop at 5) or when a token budget is exceeded (nudge at 500K input tokens, force a final answer at 1M), with a tailored wrap-up prompt for each break reason
|
|
16
|
+
- Fail fast on known environment errors: a built-in hint recognizes decryption key misconfiguration (`bad decrypt` / `ActiveRecord::Encryption`) and tells the model not to retry, and apps can register their own hints via `config.error_hints`
|
|
17
|
+
|
|
5
18
|
## [0.32.0]
|
|
6
19
|
|
|
7
20
|
- Switch default models to Claude Sonnet 5 and Opus 4.8, and rework pricing to match models by family so dated snapshots and Bedrock IDs are covered
|
data/README.md
CHANGED
|
@@ -176,15 +176,17 @@ Safety guards prevent AI-generated code from causing side effects. When a guard
|
|
|
176
176
|
|
|
177
177
|
```ruby
|
|
178
178
|
RailsConsoleAi.configure do |config|
|
|
179
|
-
config.use_builtin_safety_guard :database_writes
|
|
180
|
-
config.use_builtin_safety_guard :http_mutations
|
|
181
|
-
config.use_builtin_safety_guard :mailers
|
|
179
|
+
config.use_builtin_safety_guard :database_writes # blocks INSERT/UPDATE/DELETE/DROP/etc.
|
|
180
|
+
config.use_builtin_safety_guard :http_mutations # blocks POST/PUT/PATCH/DELETE via Net::HTTP
|
|
181
|
+
config.use_builtin_safety_guard :mailers # disables ActionMailer delivery
|
|
182
|
+
config.use_builtin_safety_guard :in_process_requests # blocks in-process requests against the app itself
|
|
182
183
|
end
|
|
183
184
|
```
|
|
184
185
|
|
|
185
186
|
- **`:database_writes`** — intercepts the ActiveRecord connection adapter to block write SQL. Works on Rails 5+ with any database adapter.
|
|
186
187
|
- **`:http_mutations`** — intercepts `Net::HTTP#request` to block non-GET/HEAD/OPTIONS requests. Covers libraries built on Net::HTTP (HTTParty, RestClient, Faraday).
|
|
187
188
|
- **`:mailers`** — sets `ActionMailer::Base.perform_deliveries = false` during execution.
|
|
189
|
+
- **`:in_process_requests`** — blocks `ActionDispatch::Integration::Session` requests (the console `app` helper) and direct Rack dispatch (`Rails.application.call`). These run the app's full middleware stack inside the current process and can deadlock or hang the session thread indefinitely, so **all verbs are blocked, including GET**. Allowlist entries are request paths.
|
|
188
190
|
|
|
189
191
|
### Custom Guards
|
|
190
192
|
|
|
@@ -291,7 +293,7 @@ RailsConsoleAi.configure do |config|
|
|
|
291
293
|
end
|
|
292
294
|
```
|
|
293
295
|
|
|
294
|
-
Default model: `claude-sonnet-5`. Thinking model: `claude-opus-
|
|
296
|
+
Default model: `claude-sonnet-5`. Thinking model: `claude-opus-5`. Prompt caching is enabled automatically.
|
|
295
297
|
|
|
296
298
|
### OpenAI
|
|
297
299
|
|
|
@@ -318,7 +320,7 @@ RailsConsoleAi.configure do |config|
|
|
|
318
320
|
config.provider = :bedrock
|
|
319
321
|
config.bedrock_region = 'us-east-1'
|
|
320
322
|
# config.model = 'us.anthropic.claude-sonnet-5' # default
|
|
321
|
-
# config.thinking_model = 'us.anthropic.claude-opus-
|
|
323
|
+
# config.thinking_model = 'us.anthropic.claude-opus-5' # default
|
|
322
324
|
end
|
|
323
325
|
```
|
|
324
326
|
|
|
@@ -396,9 +398,48 @@ RailsConsoleAi.configure do |config|
|
|
|
396
398
|
config.timeout = 30 # HTTP timeout in seconds
|
|
397
399
|
config.max_tool_rounds = 200 # safety cap on tool-use loops
|
|
398
400
|
config.code_search_paths = %w[app] # directories for list_files / search_code
|
|
401
|
+
|
|
402
|
+
# Runaway-loop circuit breakers (see "Runaway sessions" below)
|
|
403
|
+
config.token_nudge_threshold = 500_000 # input tokens in one tool loop → nudge model to wrap up (nil disables)
|
|
404
|
+
config.token_stop_threshold = 1_000_000 # input tokens in one tool loop → force a final answer (nil disables)
|
|
399
405
|
end
|
|
400
406
|
```
|
|
401
407
|
|
|
408
|
+
### Runaway sessions & known-issue hints
|
|
409
|
+
|
|
410
|
+
Two mechanisms stop the LLM from burning tokens on a dead end:
|
|
411
|
+
|
|
412
|
+
**Known-issue hints** — when executed code fails (or prints a rescued error) matching a
|
|
413
|
+
known environment-level problem, the tool result includes explicit guidance telling the
|
|
414
|
+
model not to retry. The built-in hint recognizes decryption failures
|
|
415
|
+
(`OpenSSL::Cipher::CipherError` / "bad decrypt" / `ActiveRecord::Encryption` errors),
|
|
416
|
+
which indicate a missing or placeholder encryption key in the console process — an
|
|
417
|
+
environment issue no amount of retrying can fix. Repeat occurrences escalate the message.
|
|
418
|
+
Apps can add their own:
|
|
419
|
+
|
|
420
|
+
```ruby
|
|
421
|
+
RailsConsoleAi.configure do |config|
|
|
422
|
+
config.error_hints << {
|
|
423
|
+
name: :vpn_required,
|
|
424
|
+
pattern: /Errno::ECONNREFUSED.*10\.8\./,
|
|
425
|
+
hint: "This host is only reachable over the VPN, which this console does not have. " \
|
|
426
|
+
"Do not retry; report the limitation to the user."
|
|
427
|
+
}
|
|
428
|
+
end
|
|
429
|
+
```
|
|
430
|
+
|
|
431
|
+
**Circuit breakers** — inside a single tool loop, the engine tracks:
|
|
432
|
+
|
|
433
|
+
- *Identical tool calls* (same tool + same args): warns at 3, breaks at 5.
|
|
434
|
+
- *Repeated error signatures* (same normalized error from **different** code): warns at 3,
|
|
435
|
+
breaks at 5. This catches "new code, same dead end" loops that identical-call detection
|
|
436
|
+
misses. Digits are normalized so varying record IDs don't disguise the same failure.
|
|
437
|
+
- *Token budget*: at `token_nudge_threshold` cumulative input tokens the model is told to
|
|
438
|
+
wrap up; at `token_stop_threshold` the loop is stopped and a final answer is forced.
|
|
439
|
+
|
|
440
|
+
When any breaker trips, the model is asked to summarize what it established, what it
|
|
441
|
+
could not determine and why, and what a human should do next — instead of iterating.
|
|
442
|
+
|
|
402
443
|
### Code Search Paths
|
|
403
444
|
|
|
404
445
|
By default, `list_files` and `search_code` only look in `app/`. If your project has code in other directories (e.g. a frontend in `public/portal`, or shared code in `lib`), add them:
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
module RailsConsoleAi
|
|
2
2
|
module SessionsHelper
|
|
3
3
|
def estimated_cost(session)
|
|
4
|
-
pricing = Configuration
|
|
4
|
+
pricing = Configuration.pricing_for(session.model)
|
|
5
5
|
return nil unless pricing
|
|
6
6
|
|
|
7
7
|
(session.input_tokens * pricing[:input]) + (session.output_tokens * pricing[:output])
|
|
@@ -35,6 +35,11 @@
|
|
|
35
35
|
.badge-one_shot { background: #d4edda; color: #155724; }
|
|
36
36
|
.badge-interactive { background: #cce5ff; color: #004085; }
|
|
37
37
|
.badge-explain { background: #fff3cd; color: #856404; }
|
|
38
|
+
.badge-status-running { background: #fff3cd; color: #856404; }
|
|
39
|
+
.badge-status-queued { background: #e2e3e5; color: #383d41; }
|
|
40
|
+
.badge-status-ready { background: #d4edda; color: #155724; }
|
|
41
|
+
.badge-status-failed { background: #f8d7da; color: #721c24; }
|
|
42
|
+
.badge-status-aborted { background: #f8d7da; color: #721c24; }
|
|
38
43
|
.meta-card {
|
|
39
44
|
background: #fff; border-radius: 8px; padding: 20px;
|
|
40
45
|
box-shadow: 0 1px 3px rgba(0,0,0,0.1); margin-bottom: 24px;
|
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
<th>Name</th>
|
|
18
18
|
<th style="max-width: 400px;">Query</th>
|
|
19
19
|
<th>Mode</th>
|
|
20
|
+
<th>Status</th>
|
|
20
21
|
<th>Tokens</th>
|
|
21
22
|
<th>Cost</th>
|
|
22
23
|
<th>Duration</th>
|
|
@@ -31,6 +32,7 @@
|
|
|
31
32
|
<td><%= session.name.present? ? session.name : '-' %></td>
|
|
32
33
|
<td class="query-cell"><a href="<%= rails_console_ai.session_path(session) %>" title="<%= h session.query.truncate(200) %>"><%= truncate(session.query.gsub(/\s+/, ' ').strip, length: 80) %></a></td>
|
|
33
34
|
<td><span class="badge badge-<%= session.mode %>"><%= session.mode %></span></td>
|
|
35
|
+
<td><% status = session.try(:status) %><%= status.present? ? content_tag(:span, status, class: "badge badge-status-#{status}") : '-' %></td>
|
|
34
36
|
<td class="mono"><%= session.input_tokens + session.output_tokens %></td>
|
|
35
37
|
<td class="mono"><%= format_cost(session) %></td>
|
|
36
38
|
<td class="mono"><%= session.duration_ms ? "#{session.duration_ms}ms" : '-' %></td>
|
|
@@ -72,6 +72,13 @@ RailsConsoleAi.configure do |config|
|
|
|
72
72
|
# Built-in guard for mailers — disables ActionMailer delivery:
|
|
73
73
|
# config.use_builtin_safety_guard :mailers
|
|
74
74
|
#
|
|
75
|
+
# Built-in guard for in-process requests — blocks ActionDispatch::Integration::Session
|
|
76
|
+
# (the console `app` helper) and direct Rack dispatch against the running app.
|
|
77
|
+
# These run the full middleware stack inside this process and can hang the session
|
|
78
|
+
# thread indefinitely, so ALL verbs are blocked (including GET). Strongly recommended
|
|
79
|
+
# for Slack/API channels:
|
|
80
|
+
# config.use_builtin_safety_guard :in_process_requests
|
|
81
|
+
#
|
|
75
82
|
# config.safety_guard :jobs do |&execute|
|
|
76
83
|
# Sidekiq::Testing.fake! { execute.call }
|
|
77
84
|
# end
|
|
@@ -13,6 +13,7 @@ module RailsConsoleAi
|
|
|
13
13
|
# (removed on opus-4-7+, sonnet-5, and fable-5).
|
|
14
14
|
MODEL_FAMILIES = {
|
|
15
15
|
'claude-fable-5' => { input: 10.0, output: 50.0, max_tokens: 16_000, temperature: false },
|
|
16
|
+
'claude-opus-5' => { input: 5.0, output: 25.0, max_tokens: 16_000, temperature: false },
|
|
16
17
|
'claude-opus-4-8' => { input: 5.0, output: 25.0, max_tokens: 16_000, temperature: false },
|
|
17
18
|
'claude-opus-4-7' => { input: 5.0, output: 25.0, max_tokens: 16_000, temperature: false },
|
|
18
19
|
'claude-opus-4-6' => { input: 5.0, output: 25.0, max_tokens: 16_000, temperature: true },
|
|
@@ -47,9 +48,30 @@ module RailsConsoleAi
|
|
|
47
48
|
}
|
|
48
49
|
end
|
|
49
50
|
|
|
51
|
+
# Known environment-level failures the executor recognizes and explains to the
|
|
52
|
+
# LLM on the FIRST occurrence, so it doesn't burn rounds rediscovering them
|
|
53
|
+
# through trial and error. Each entry: { name:, pattern:, hint: }.
|
|
54
|
+
# The pattern is matched against the execution error AND the captured output
|
|
55
|
+
# (to catch errors rescued and printed by the generated code itself).
|
|
56
|
+
DEFAULT_ERROR_HINTS = [
|
|
57
|
+
{
|
|
58
|
+
name: :decryption_failure,
|
|
59
|
+
pattern: /OpenSSL::Cipher::CipherError|bad decrypt|ActiveRecord::Encryption::Errors/i,
|
|
60
|
+
hint: "Decryption failed — this console process's encryption key (e.g. ENV['ENCRYPTION_KEY']) " \
|
|
61
|
+
"appears to be missing, invalid, or a placeholder for the data you are reading. This is an " \
|
|
62
|
+
"environment configuration issue, not a data issue, and it affects EVERY encrypted field in " \
|
|
63
|
+
"this session. Retrying with different code (reloading records, toggling encryption flags, " \
|
|
64
|
+
"decrypting other fields or records) will fail the same way. Do NOT retry. Report this " \
|
|
65
|
+
"limitation to the user, tell them the encryption key needs to be configured for this " \
|
|
66
|
+
"environment, and answer using only what you can determine without decrypting."
|
|
67
|
+
}
|
|
68
|
+
].freeze
|
|
69
|
+
|
|
50
70
|
attr_accessor :provider, :api_key, :model, :thinking_model, :max_tokens,
|
|
51
71
|
:auto_execute, :temperature,
|
|
52
72
|
:timeout, :debug, :max_tool_rounds,
|
|
73
|
+
:error_hints,
|
|
74
|
+
:token_nudge_threshold, :token_stop_threshold,
|
|
53
75
|
:storage_adapter, :memories_enabled,
|
|
54
76
|
:session_logging, :connection_class,
|
|
55
77
|
:admin_username, :admin_password,
|
|
@@ -75,6 +97,9 @@ module RailsConsoleAi
|
|
|
75
97
|
@timeout = 30
|
|
76
98
|
@debug = false
|
|
77
99
|
@max_tool_rounds = 200
|
|
100
|
+
@error_hints = DEFAULT_ERROR_HINTS.dup
|
|
101
|
+
@token_nudge_threshold = 500_000 # input tokens in one tool loop → nudge model to wrap up (nil disables)
|
|
102
|
+
@token_stop_threshold = 1_000_000 # input tokens in one tool loop → force a final answer (nil disables)
|
|
78
103
|
@storage_adapter = nil
|
|
79
104
|
@memories_enabled = true
|
|
80
105
|
@session_logging = true
|
|
@@ -147,12 +172,13 @@ module RailsConsoleAi
|
|
|
147
172
|
end
|
|
148
173
|
|
|
149
174
|
# Register a built-in safety guard by name.
|
|
150
|
-
# Available: :database_writes, :http_mutations, :mailers
|
|
175
|
+
# Available: :database_writes, :http_mutations, :mailers, :in_process_requests
|
|
151
176
|
#
|
|
152
177
|
# Options:
|
|
153
178
|
# allow: Array of strings or regexps to allowlist for this guard.
|
|
154
|
-
# - :http_mutations
|
|
155
|
-
# - :database_writes
|
|
179
|
+
# - :http_mutations → hosts (e.g. "s3.amazonaws.com", /googleapis\.com/)
|
|
180
|
+
# - :database_writes → table names (e.g. "rails_console_ai_sessions")
|
|
181
|
+
# - :in_process_requests → request paths (e.g. "/health")
|
|
156
182
|
def use_builtin_safety_guard(name, allow: nil)
|
|
157
183
|
require 'rails_console_ai/safety_guards'
|
|
158
184
|
guard_name = name.to_sym
|
|
@@ -163,8 +189,10 @@ module RailsConsoleAi
|
|
|
163
189
|
safety_guards.add(:http_mutations, &BuiltinGuards.http_mutations)
|
|
164
190
|
when :mailers
|
|
165
191
|
safety_guards.add(:mailers, &BuiltinGuards.mailers)
|
|
192
|
+
when :in_process_requests
|
|
193
|
+
safety_guards.add(:in_process_requests, &BuiltinGuards.in_process_requests)
|
|
166
194
|
else
|
|
167
|
-
raise ConfigurationError, "Unknown built-in safety guard: #{name}. Available: database_writes, http_mutations, mailers"
|
|
195
|
+
raise ConfigurationError, "Unknown built-in safety guard: #{name}. Available: database_writes, http_mutations, mailers, in_process_requests"
|
|
168
196
|
end
|
|
169
197
|
|
|
170
198
|
if allow
|
|
@@ -222,13 +250,13 @@ module RailsConsoleAi
|
|
|
222
250
|
|
|
223
251
|
case @provider
|
|
224
252
|
when :anthropic
|
|
225
|
-
'claude-opus-
|
|
253
|
+
'claude-opus-5'
|
|
226
254
|
when :openai
|
|
227
255
|
'gpt-5.3-codex'
|
|
228
256
|
when :local
|
|
229
257
|
@local_model
|
|
230
258
|
when :bedrock
|
|
231
|
-
'us.anthropic.claude-opus-
|
|
259
|
+
'us.anthropic.claude-opus-5'
|
|
232
260
|
end
|
|
233
261
|
end
|
|
234
262
|
|
|
@@ -7,6 +7,8 @@ module RailsConsoleAi
|
|
|
7
7
|
LARGE_OUTPUT_PREVIEW_CHARS = 16_000 # chars — how much of the output the LLM sees upfront
|
|
8
8
|
LOOP_WARN_THRESHOLD = 3 # same tool+args repeated → inject warning
|
|
9
9
|
LOOP_BREAK_THRESHOLD = 5 # same tool+args repeated → break loop
|
|
10
|
+
REPEAT_ERROR_WARN_THRESHOLD = 3 # same error signature (any args) → inject warning
|
|
11
|
+
REPEAT_ERROR_BREAK_THRESHOLD = 5 # same error signature (any args) → force wrap-up
|
|
10
12
|
|
|
11
13
|
def initialize(binding_context:, channel:, slack_thread_ts: nil, slack_channel_name: nil)
|
|
12
14
|
@binding_context = binding_context
|
|
@@ -46,6 +48,7 @@ module RailsConsoleAi
|
|
|
46
48
|
if executed && @executor.last_error && !@executor.last_safety_error
|
|
47
49
|
error_msg = "Code execution failed with error: #{@executor.last_error}"
|
|
48
50
|
error_msg = error_msg[0..1000] + '...' if error_msg.length > 1000
|
|
51
|
+
error_msg += "\n\n#{@executor.last_error_hint}" if @executor.last_error_hint
|
|
49
52
|
conversation << { role: :assistant, content: @_last_result_text }
|
|
50
53
|
conversation << { role: :user, content: error_msg }
|
|
51
54
|
|
|
@@ -239,6 +242,8 @@ module RailsConsoleAi
|
|
|
239
242
|
end
|
|
240
243
|
|
|
241
244
|
def execute_direct(raw_code)
|
|
245
|
+
@interactive_query ||= "> #{raw_code}"
|
|
246
|
+
log_interactive_turn(status: 'running')
|
|
242
247
|
exec_result = @executor.execute_unsafe(raw_code)
|
|
243
248
|
|
|
244
249
|
output_parts = []
|
|
@@ -257,14 +262,31 @@ module RailsConsoleAi
|
|
|
257
262
|
end
|
|
258
263
|
@history << { role: :user, content: context_msg, output_id: output_id }
|
|
259
264
|
|
|
260
|
-
@interactive_query ||= "> #{raw_code}"
|
|
261
265
|
@last_interactive_code = raw_code
|
|
262
266
|
@last_interactive_output = @executor.last_output
|
|
263
267
|
@last_interactive_result = exec_result ? exec_result.inspect : nil
|
|
264
268
|
@last_interactive_executed = true
|
|
269
|
+
log_interactive_turn(status: 'ready')
|
|
265
270
|
end
|
|
266
271
|
|
|
272
|
+
# Wraps run_turn with session-row status tracking. The row is created/updated
|
|
273
|
+
# with status 'running' BEFORE the tool loop starts, so a turn that hangs or
|
|
274
|
+
# dies mid-loop (hung eval, OOM-killed pod, deploy) still leaves a visible
|
|
275
|
+
# session record instead of vanishing without a trace.
|
|
267
276
|
def send_and_execute
|
|
277
|
+
log_interactive_turn(status: 'running')
|
|
278
|
+
status = run_turn
|
|
279
|
+
log_interactive_turn(status: status == :error ? 'failed' : 'ready')
|
|
280
|
+
status
|
|
281
|
+
rescue Interrupt
|
|
282
|
+
log_interactive_turn(status: 'ready')
|
|
283
|
+
raise
|
|
284
|
+
rescue StandardError
|
|
285
|
+
log_interactive_turn(status: 'failed')
|
|
286
|
+
raise
|
|
287
|
+
end
|
|
288
|
+
|
|
289
|
+
def run_turn
|
|
268
290
|
begin
|
|
269
291
|
result, tool_messages, last_llm_stats = send_query(nil, conversation: @history)
|
|
270
292
|
rescue Providers::ProviderError => e
|
|
@@ -347,6 +369,7 @@ module RailsConsoleAi
|
|
|
347
369
|
elsif @executor.last_error
|
|
348
370
|
error_msg = "Code execution failed with error: #{@executor.last_error}"
|
|
349
371
|
error_msg = error_msg[0..1000] + '...' if error_msg.length > 1000
|
|
372
|
+
error_msg += "\n\n#{@executor.last_error_hint}" if @executor.last_error_hint
|
|
350
373
|
@history << { role: :user, content: error_msg }
|
|
351
374
|
:error
|
|
352
375
|
else
|
|
@@ -564,7 +587,7 @@ module RailsConsoleAi
|
|
|
564
587
|
|
|
565
588
|
# --- Session logging ---
|
|
566
589
|
|
|
567
|
-
def log_interactive_turn
|
|
590
|
+
def log_interactive_turn(status: nil)
|
|
568
591
|
require 'rails_console_ai/session_logger'
|
|
569
592
|
session_attrs = {
|
|
570
593
|
conversation: @history,
|
|
@@ -576,6 +599,7 @@ module RailsConsoleAi
|
|
|
576
599
|
executed: @last_interactive_executed,
|
|
577
600
|
console_output: @channel.respond_to?(:console_capture_string) ? @channel.console_capture_string : nil
|
|
578
601
|
}
|
|
602
|
+
session_attrs[:status] = status if status
|
|
579
603
|
|
|
580
604
|
if @interactive_session_id
|
|
581
605
|
SessionLogger.update(@interactive_session_id, session_attrs)
|
|
@@ -796,7 +820,11 @@ module RailsConsoleAi
|
|
|
796
820
|
last_tool_names = []
|
|
797
821
|
|
|
798
822
|
exhausted = false
|
|
823
|
+
wrap_up_reason = nil
|
|
799
824
|
tool_call_counts = Hash.new(0)
|
|
825
|
+
error_sig_counts = Hash.new(0)
|
|
826
|
+
warned_error_sigs = Set.new
|
|
827
|
+
token_nudge_sent = false
|
|
800
828
|
|
|
801
829
|
max_rounds.times do |round|
|
|
802
830
|
if @channel.cancelled?
|
|
@@ -918,6 +946,12 @@ module RailsConsoleAi
|
|
|
918
946
|
$stderr.puts "\e[35m[debug] tool result (#{tool_result.to_s.length} chars)\e[0m"
|
|
919
947
|
end
|
|
920
948
|
|
|
949
|
+
# Track repeated error signatures across DIFFERENT tool calls — the
|
|
950
|
+
# identical-call loop detection below misses "same dead end, new code".
|
|
951
|
+
if (sig = error_signature(tool_result))
|
|
952
|
+
error_sig_counts[sig] += 1
|
|
953
|
+
end
|
|
954
|
+
|
|
921
955
|
tool_msg = provider.format_tool_result(tc[:id], tool_result)
|
|
922
956
|
full_text = tool_result.to_s
|
|
923
957
|
output_id = @executor.store_output(full_text)
|
|
@@ -942,23 +976,68 @@ module RailsConsoleAi
|
|
|
942
976
|
if tool_call_counts[key] >= LOOP_BREAK_THRESHOLD
|
|
943
977
|
@channel.display_status(" Loop detected: #{tc[:name]} called #{tool_call_counts[key]} times with same args — stopping.")
|
|
944
978
|
exhausted = true
|
|
979
|
+
wrap_up_reason ||= :tool_loop
|
|
945
980
|
elsif tool_call_counts[key] >= LOOP_WARN_THRESHOLD
|
|
946
981
|
@channel.display_status(" Warning: #{tc[:name]} called #{tool_call_counts[key]} times with same args — consider a different approach.")
|
|
947
982
|
messages << { role: :user, content: "You are repeating the same tool call (#{tc[:name]}) with the same arguments. This is not making progress. Try a different approach or provide your answer now." }
|
|
948
983
|
end
|
|
949
984
|
end
|
|
985
|
+
|
|
986
|
+
# Circuit breaker: the same error signature recurring across different
|
|
987
|
+
# tool calls means no new information is being gained (e.g. every
|
|
988
|
+
# decryption attempt failing with "bad decrypt" regardless of the code).
|
|
989
|
+
error_sig_counts.each do |sig, count|
|
|
990
|
+
if count >= REPEAT_ERROR_BREAK_THRESHOLD
|
|
991
|
+
@channel.display_status(" Circuit breaker: same error hit #{count} times (#{sig}) — stopping.")
|
|
992
|
+
exhausted = true
|
|
993
|
+
wrap_up_reason ||= :repeated_errors
|
|
994
|
+
elsif count >= REPEAT_ERROR_WARN_THRESHOLD && !warned_error_sigs.include?(sig)
|
|
995
|
+
warned_error_sigs << sig
|
|
996
|
+
@channel.display_status(" Warning: same error hit #{count} times — nudging model to change strategy.")
|
|
997
|
+
messages << { role: :user, content: "You have now hit the same error #{count} times (#{sig}). Trying variations of the same approach is not producing new information. If this error cannot be resolved from this session, stop investigating it: summarize what you have established, state what you could not determine and why, and give the user your best answer." }
|
|
998
|
+
end
|
|
999
|
+
end
|
|
1000
|
+
|
|
1001
|
+
# Circuit breaker: token budget for a single tool loop.
|
|
1002
|
+
config = RailsConsoleAi.configuration
|
|
1003
|
+
if config.token_stop_threshold && total_input >= config.token_stop_threshold
|
|
1004
|
+
@channel.display_status(" Token budget exceeded (#{format_tokens(total_input)} input tokens this request) — forcing wrap-up.")
|
|
1005
|
+
exhausted = true
|
|
1006
|
+
wrap_up_reason ||= :token_budget
|
|
1007
|
+
elsif config.token_nudge_threshold && total_input >= config.token_nudge_threshold && !token_nudge_sent
|
|
1008
|
+
token_nudge_sent = true
|
|
1009
|
+
@channel.display_status(" High token usage (#{format_tokens(total_input)} input tokens this request) — nudging model to wrap up.")
|
|
1010
|
+
messages << { role: :user, content: "This investigation has consumed #{format_tokens(total_input)} input tokens without reaching a conclusion. Wrap up now: stop opening new lines of investigation, summarize what you have established, state what you could not determine and why, and give the user your best answer. Only make another tool call if you are confident a single call will resolve the question." }
|
|
1011
|
+
end
|
|
1012
|
+
|
|
950
1013
|
break if exhausted
|
|
951
1014
|
|
|
952
1015
|
# If the user declined execution, don't call the LLM again —
|
|
953
1016
|
# just return to the prompt so they can correct their request.
|
|
954
1017
|
break if @executor.last_cancelled?
|
|
955
1018
|
|
|
956
|
-
|
|
1019
|
+
if round == max_rounds - 1
|
|
1020
|
+
exhausted = true
|
|
1021
|
+
wrap_up_reason ||= :round_cap
|
|
1022
|
+
end
|
|
957
1023
|
end
|
|
958
1024
|
|
|
959
1025
|
if exhausted
|
|
960
|
-
|
|
961
|
-
|
|
1026
|
+
final_nudge =
|
|
1027
|
+
case wrap_up_reason
|
|
1028
|
+
when :repeated_errors
|
|
1029
|
+
"You have hit the same error repeatedly without gaining new information. Do not attempt any further tool calls. Give the user a final answer now: (1) what you established, (2) what you could not determine and the blocking error, and (3) what a human should do next."
|
|
1030
|
+
when :token_budget
|
|
1031
|
+
"This session has exceeded its token budget. Do not attempt any further tool calls. Give the user a final answer now: (1) what you established, (2) what you could not determine and why, and (3) what a human should do next."
|
|
1032
|
+
when :tool_loop
|
|
1033
|
+
"You kept repeating the same tool call with the same arguments. Do not attempt any further tool calls. Provide your best answer now based on what you've learned so far."
|
|
1034
|
+
else
|
|
1035
|
+
"You've used all available tool rounds. Please provide your best answer now based on what you've learned so far."
|
|
1036
|
+
end
|
|
1037
|
+
if wrap_up_reason.nil? || wrap_up_reason == :round_cap
|
|
1038
|
+
$stdout.puts "\e[33m Hit tool round limit (#{max_rounds}). Forcing final answer. Increase with: RailsConsoleAi.configure { |c| c.max_tool_rounds = 200 }\e[0m"
|
|
1039
|
+
end
|
|
1040
|
+
messages << { role: :user, content: final_nudge }
|
|
962
1041
|
result = provider.chat(messages, system_prompt: active_system_prompt)
|
|
963
1042
|
total_input += result.input_tokens || 0
|
|
964
1043
|
total_output += result.output_tokens || 0
|
|
@@ -1167,6 +1246,15 @@ module RailsConsoleAi
|
|
|
1167
1246
|
str.length > max ? str[0..max] + '...' : str
|
|
1168
1247
|
end
|
|
1169
1248
|
|
|
1249
|
+
# Extract a normalized error signature from a tool result so we can detect
|
|
1250
|
+
# "same failure, different code" loops. Digits are collapsed so record IDs,
|
|
1251
|
+
# counts, and offsets don't make identical failures look distinct.
|
|
1252
|
+
def error_signature(tool_result)
|
|
1253
|
+
line = tool_result.to_s[/^ERROR: [^\n]+/]
|
|
1254
|
+
return nil unless line
|
|
1255
|
+
line.gsub(/\d+/, 'N')[0, 160]
|
|
1256
|
+
end
|
|
1257
|
+
|
|
1170
1258
|
# Wraps mid-task user messages with explicit framing so the model treats them
|
|
1171
1259
|
# as a real-time interruption that supersedes the prior task, rather than as
|
|
1172
1260
|
# a reply to the most recent tool result.
|
|
@@ -44,7 +44,8 @@ module RailsConsoleAi
|
|
|
44
44
|
class Executor
|
|
45
45
|
CODE_REGEX = /```ruby\s*\n(.*?)```/m
|
|
46
46
|
|
|
47
|
-
attr_reader :binding_context, :last_error, :last_safety_error, :last_safety_exception
|
|
47
|
+
attr_reader :binding_context, :last_error, :last_safety_error, :last_safety_exception,
|
|
48
|
+
:last_error_hint
|
|
48
49
|
attr_accessor :on_prompt
|
|
49
50
|
|
|
50
51
|
def initialize(binding_context, channel: nil)
|
|
@@ -55,6 +56,7 @@ module RailsConsoleAi
|
|
|
55
56
|
@output_store = {}
|
|
56
57
|
@output_counter = 0
|
|
57
58
|
@active_skill_bypass_methods = Set.new
|
|
59
|
+
@error_hint_counts = Hash.new(0)
|
|
58
60
|
end
|
|
59
61
|
|
|
60
62
|
def extract_code(response)
|
|
@@ -98,6 +100,7 @@ module RailsConsoleAi
|
|
|
98
100
|
@last_error = nil
|
|
99
101
|
@last_safety_error = false
|
|
100
102
|
@last_safety_exception = nil
|
|
103
|
+
@last_error_hint = nil
|
|
101
104
|
captured_output = StringIO.new
|
|
102
105
|
old_stdout = $stdout
|
|
103
106
|
# Three capture strategies:
|
|
@@ -140,6 +143,8 @@ module RailsConsoleAi
|
|
|
140
143
|
display_result(result) if display
|
|
141
144
|
|
|
142
145
|
@last_output = captured_output.string
|
|
146
|
+
# Code may have rescued a known failure and printed it — still surface the hint.
|
|
147
|
+
@last_error_hint = build_error_hint(@last_output)
|
|
143
148
|
result
|
|
144
149
|
rescue Interrupt
|
|
145
150
|
restore_stdout(use_thread_local, old_stdout)
|
|
@@ -174,6 +179,7 @@ module RailsConsoleAi
|
|
|
174
179
|
return nil
|
|
175
180
|
end
|
|
176
181
|
@last_error = "#{e.class}: #{e.message}"
|
|
182
|
+
@last_error_hint = build_error_hint(@last_error, captured_output&.string)
|
|
177
183
|
backtrace = e.backtrace.first(3).map { |line| " #{line}" }.join("\n")
|
|
178
184
|
log_execution_error("Error: #{@last_error}\n#{backtrace}")
|
|
179
185
|
@last_output = captured_output&.string
|
|
@@ -346,6 +352,29 @@ module RailsConsoleAi
|
|
|
346
352
|
|
|
347
353
|
private
|
|
348
354
|
|
|
355
|
+
# Match the error/output against configured known-issue patterns
|
|
356
|
+
# (Configuration#error_hints). Returns a guidance string for the LLM, or nil.
|
|
357
|
+
# The first occurrence explains the issue; repeats escalate so the model
|
|
358
|
+
# stops probing the same dead end.
|
|
359
|
+
def build_error_hint(*texts)
|
|
360
|
+
text = texts.compact.join("\n")
|
|
361
|
+
return nil if text.strip.empty?
|
|
362
|
+
|
|
363
|
+
hint_def = Array(RailsConsoleAi.configuration.error_hints).find do |h|
|
|
364
|
+
h[:pattern] && text.match?(h[:pattern])
|
|
365
|
+
end
|
|
366
|
+
return nil unless hint_def
|
|
367
|
+
|
|
368
|
+
count = (@error_hint_counts[hint_def[:name]] += 1)
|
|
369
|
+
if count == 1
|
|
370
|
+
"KNOWN ISSUE DETECTED (#{hint_def[:name]}): #{hint_def[:hint]}"
|
|
371
|
+
else
|
|
372
|
+
"KNOWN ISSUE (#{hint_def[:name]}) — this is failure ##{count} for the same underlying problem. " \
|
|
373
|
+
"You were already told this cannot succeed from this session. Stop retrying now and report the " \
|
|
374
|
+
"limitation to the user. Reminder: #{hint_def[:hint]}"
|
|
375
|
+
end
|
|
376
|
+
end
|
|
377
|
+
|
|
349
378
|
def restore_stdout(use_thread_local, old_stdout)
|
|
350
379
|
if use_thread_local
|
|
351
380
|
Thread.current[:capture_io] = nil
|
|
@@ -364,6 +364,81 @@ module RailsConsoleAi
|
|
|
364
364
|
}
|
|
365
365
|
end
|
|
366
366
|
|
|
367
|
+
# Blocks in-process HTTP dispatch against the running app itself.
|
|
368
|
+
# An ActionDispatch::Integration::Session request (the console `app` helper,
|
|
369
|
+
# or a manually built integration session) runs the app's full middleware
|
|
370
|
+
# stack inside the current process and can deadlock or hang the session
|
|
371
|
+
# thread indefinitely — so ALL verbs are blocked, including GET.
|
|
372
|
+
module InProcessRequestBlocker
|
|
373
|
+
def process(*args, **kwargs, &block)
|
|
374
|
+
RailsConsoleAi::BuiltinGuards.check_in_process_request!(args[0], args[1])
|
|
375
|
+
super
|
|
376
|
+
end
|
|
377
|
+
end
|
|
378
|
+
|
|
379
|
+
# Backstop for the same hazard via direct Rack dispatch
|
|
380
|
+
# (e.g. Rails.application.call(env)), which bypasses Integration::Session.
|
|
381
|
+
module EngineCallBlocker
|
|
382
|
+
def call(env, *args)
|
|
383
|
+
if env.is_a?(Hash)
|
|
384
|
+
RailsConsoleAi::BuiltinGuards.check_in_process_request!(env['REQUEST_METHOD'], env['PATH_INFO'])
|
|
385
|
+
end
|
|
386
|
+
super
|
|
387
|
+
end
|
|
388
|
+
end
|
|
389
|
+
|
|
390
|
+
def self.check_in_process_request!(http_method, path)
|
|
391
|
+
return unless Thread.current[:rails_console_ai_block_in_process_requests]
|
|
392
|
+
return if Thread.current[:rails_console_ai_bypass_guards]
|
|
393
|
+
|
|
394
|
+
key = path.to_s
|
|
395
|
+
guards = RailsConsoleAi.configuration.safety_guards
|
|
396
|
+
return if !key.empty? && guards.allowed?(:in_process_requests, key)
|
|
397
|
+
|
|
398
|
+
label = [http_method.to_s.upcase, key].reject(&:empty?).join(' ')
|
|
399
|
+
raise RailsConsoleAi::SafetyError.new(
|
|
400
|
+
"In-process HTTP request blocked (#{label.empty? ? 'app dispatch' : label}). " \
|
|
401
|
+
"Dispatching a request through the app's own middleware stack from this session " \
|
|
402
|
+
"can hang the process indefinitely, even for GET. Do not retry via another route " \
|
|
403
|
+
"or Rack — call the controller's underlying service or model code directly instead.",
|
|
404
|
+
guard: :in_process_requests,
|
|
405
|
+
blocked_key: key.empty? ? nil : key
|
|
406
|
+
)
|
|
407
|
+
end
|
|
408
|
+
|
|
409
|
+
def self.in_process_requests
|
|
410
|
+
->(&block) {
|
|
411
|
+
ensure_in_process_blocker_installed!
|
|
412
|
+
prev = Thread.current[:rails_console_ai_block_in_process_requests]
|
|
413
|
+
Thread.current[:rails_console_ai_block_in_process_requests] = true
|
|
414
|
+
begin
|
|
415
|
+
block.call
|
|
416
|
+
ensure
|
|
417
|
+
Thread.current[:rails_console_ai_block_in_process_requests] = prev
|
|
418
|
+
end
|
|
419
|
+
}
|
|
420
|
+
end
|
|
421
|
+
|
|
422
|
+
def self.ensure_in_process_blocker_installed!
|
|
423
|
+
return if @in_process_blocker_installed
|
|
424
|
+
|
|
425
|
+
begin
|
|
426
|
+
require 'action_dispatch'
|
|
427
|
+
require 'action_dispatch/testing/integration'
|
|
428
|
+
rescue LoadError, NameError
|
|
429
|
+
nil # actionpack not (fully) available — the Engine backstop may still apply
|
|
430
|
+
end
|
|
431
|
+
|
|
432
|
+
if defined?(ActionDispatch::Integration::Session) &&
|
|
433
|
+
!ActionDispatch::Integration::Session.ancestors.include?(InProcessRequestBlocker)
|
|
434
|
+
ActionDispatch::Integration::Session.prepend(InProcessRequestBlocker)
|
|
435
|
+
end
|
|
436
|
+
if defined?(Rails::Engine) && !Rails::Engine.ancestors.include?(EngineCallBlocker)
|
|
437
|
+
Rails::Engine.prepend(EngineCallBlocker)
|
|
438
|
+
end
|
|
439
|
+
@in_process_blocker_installed = true
|
|
440
|
+
end
|
|
441
|
+
|
|
367
442
|
def self.ensure_http_blocker_installed!
|
|
368
443
|
return if @http_blocker_installed
|
|
369
444
|
|
|
@@ -32,7 +32,7 @@ module RailsConsoleAi
|
|
|
32
32
|
opts = attrs[:options]
|
|
33
33
|
create_attrs[:options] = opts.is_a?(String) ? opts : opts.to_json
|
|
34
34
|
end
|
|
35
|
-
record = session_class.create!(create_attrs)
|
|
35
|
+
record = session_class.create!(filter_to_columns(create_attrs))
|
|
36
36
|
record.id
|
|
37
37
|
rescue => e
|
|
38
38
|
msg = "RailsConsoleAi: session logging failed: #{e.class}: #{e.message}"
|
|
@@ -70,6 +70,7 @@ module RailsConsoleAi
|
|
|
70
70
|
updates[:result] = attrs[:result] if attrs.key?(:result)
|
|
71
71
|
updates[:error_message] = attrs[:error_message] if attrs.key?(:error_message)
|
|
72
72
|
|
|
73
|
+
updates = filter_to_columns(updates)
|
|
73
74
|
session_class.where(id: id).update_all(updates) unless updates.empty?
|
|
74
75
|
rescue => e
|
|
75
76
|
msg = "RailsConsoleAi: session update failed: #{e.class}: #{e.message}"
|
|
@@ -80,6 +81,17 @@ module RailsConsoleAi
|
|
|
80
81
|
|
|
81
82
|
private
|
|
82
83
|
|
|
84
|
+
# Drop attrs the table doesn't have a column for, so a gem that's newer
|
|
85
|
+
# than the table (e.g. status added before RailsConsoleAi.migrate! ran)
|
|
86
|
+
# degrades to a partial row instead of losing the whole insert.
|
|
87
|
+
def filter_to_columns(attrs)
|
|
88
|
+
return attrs unless session_class.respond_to?(:column_names)
|
|
89
|
+
cols = session_class.column_names.map(&:to_s)
|
|
90
|
+
attrs.select { |k, _| cols.include?(k.to_s) }
|
|
91
|
+
rescue StandardError
|
|
92
|
+
attrs
|
|
93
|
+
end
|
|
94
|
+
|
|
83
95
|
def table_exists?
|
|
84
96
|
# Only cache positive results — retry on failure so transient
|
|
85
97
|
# errors (boot timing, connection not ready) don't stick forever
|
|
@@ -698,13 +698,16 @@ module RailsConsoleAi
|
|
|
698
698
|
end
|
|
699
699
|
|
|
700
700
|
if @executor.last_error
|
|
701
|
-
|
|
701
|
+
result = "ERROR: #{@executor.last_error}"
|
|
702
|
+
result += "\n\n#{@executor.last_error_hint}" if @executor.last_error_hint
|
|
703
|
+
return result
|
|
702
704
|
end
|
|
703
705
|
|
|
704
706
|
output = @executor.last_output
|
|
705
707
|
parts = []
|
|
706
708
|
parts << "Output:\n#{output.strip}" if output && !output.strip.empty?
|
|
707
709
|
parts << "Return value: #{exec_result.inspect}"
|
|
710
|
+
parts << @executor.last_error_hint if @executor.last_error_hint
|
|
708
711
|
parts.join("\n\n")
|
|
709
712
|
end
|
|
710
713
|
|
|
@@ -808,6 +811,7 @@ module RailsConsoleAi
|
|
|
808
811
|
step_report = "Step #{i + 1} (#{step['description']}):\n"
|
|
809
812
|
if error
|
|
810
813
|
step_report += "ERROR: #{error}\n"
|
|
814
|
+
step_report += "#{@executor.last_error_hint}\n" if @executor.last_error_hint
|
|
811
815
|
end
|
|
812
816
|
if output && !output.strip.empty?
|
|
813
817
|
step_report += "Output: #{output.strip}\n"
|