rails_error_dashboard 0.8.0 → 0.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (33) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +34 -1
  3. data/app/controllers/rails_error_dashboard/application_controller.rb +5 -0
  4. data/app/controllers/rails_error_dashboard/errors_controller.rb +12 -0
  5. data/app/controllers/rails_error_dashboard/webhooks_controller.rb +35 -1
  6. data/app/jobs/rails_error_dashboard/storm_flush_job.rb +19 -0
  7. data/app/jobs/rails_error_dashboard/storm_notification_job.rb +74 -0
  8. data/app/models/rails_error_dashboard/storm_event.rb +34 -0
  9. data/app/views/layouts/rails_error_dashboard.html.erb +21 -0
  10. data/app/views/rails_error_dashboard/errors/_issue_section.html.erb +1 -0
  11. data/app/views/rails_error_dashboard/errors/storms.html.erb +91 -0
  12. data/config/routes.rb +1 -0
  13. data/db/migrate/20260306000002_add_instance_variables_to_error_logs.rb +7 -1
  14. data/db/migrate/20260306000003_create_rails_error_dashboard_swallowed_exceptions.rb +4 -0
  15. data/db/migrate/20260307000001_create_rails_error_dashboard_diagnostic_dumps.rb +4 -0
  16. data/db/migrate/20260613000001_create_storm_events.rb +28 -0
  17. data/lib/generators/rails_error_dashboard/install/templates/initializer.rb +49 -2
  18. data/lib/rails_error_dashboard/commands/create_issue.rb +1 -1
  19. data/lib/rails_error_dashboard/commands/flush_storm_counts.rb +188 -0
  20. data/lib/rails_error_dashboard/commands/link_existing_issue.rb +3 -1
  21. data/lib/rails_error_dashboard/commands/log_error.rb +70 -12
  22. data/lib/rails_error_dashboard/configuration.rb +73 -7
  23. data/lib/rails_error_dashboard/queries/storm_history.rb +39 -0
  24. data/lib/rails_error_dashboard/services/issue_tracker_client.rb +8 -5
  25. data/lib/rails_error_dashboard/services/linear_issue_client.rb +248 -0
  26. data/lib/rails_error_dashboard/services/storm_protection/circuit_breaker.rb +195 -0
  27. data/lib/rails_error_dashboard/services/storm_protection/count_buffer.rb +100 -0
  28. data/lib/rails_error_dashboard/services/storm_protection/fingerprint_buckets.rb +123 -0
  29. data/lib/rails_error_dashboard/services/storm_protection/gate.rb +258 -0
  30. data/lib/rails_error_dashboard/subscribers/issue_tracker_subscriber.rb +12 -0
  31. data/lib/rails_error_dashboard/version.rb +1 -1
  32. data/lib/rails_error_dashboard.rb +7 -0
  33. metadata +15 -3
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 002c2e5d338c585425599802271a90c5bac425dc86867da1aa10a0c7a478d1e8
4
- data.tar.gz: 077d7d7a7f957bee799cf39c96632532e3c74259e83310605c6068d57436aec3
3
+ metadata.gz: c51d8221b7ca00ce0f9242f98f3594fd86028a2c98c464fcac28aac34f6d2a3b
4
+ data.tar.gz: 597373253abc68befcf70edab809093c320740fca1a575bcc8cae7fd6ae7e3e5
5
5
  SHA512:
6
- metadata.gz: 8e6ef9f87aae8c200e997bf761184eb37c86d0637dfdd383fd6cc003f3c3b9fa30b7906fbaf6377670d932a60da5a9046202d4a48ab63264ce916c861ac20b91
7
- data.tar.gz: 67858acc12ad29cb412b0c049b776b7193c9ce8d45febfa4bc23e079a1afd7ff64a2a85bd81516c999cc57861631ed3f89ea62bc36544370811e8cff858c8678
6
+ metadata.gz: 2bcae54eb97ce19345f5ca8a92e515c7b44d9957afc985885bda168afddadc24b3640b7fc3dcdfbe5e80637564975be851d7aeb2d2be14fbdfb8b38bf769300c
7
+ data.tar.gz: f269598ef3e5c0b4a7761be4bb4ddeeef32f662dd18f6e75dca292f108804b8d73f4d95e0f30e86a550e51478ec66d43431bb722740caf7d0149f98addb9c042
data/README.md CHANGED
@@ -79,6 +79,28 @@ Error capture from controllers, jobs, and middleware. Custom-designed dashboard
79
79
 
80
80
  ### Optional Features
81
81
 
82
+ <details>
83
+ <summary><strong>Storm Protection — Circuit Breaker + Adaptive Sampling</strong></summary>
84
+
85
+ When the error rate spikes (a bad deploy throwing thousands of errors a minute), the nightmare scenario for any in-process tracker is amplifying the outage with its own database writes. Storm protection makes the gem **provably degrade itself first** — ON by default.
86
+
87
+ - **Per-fingerprint caps:** past N occurrences/minute per error, context is shed, then rows are sampled deterministically (a fresh exemplar is always kept each minute)
88
+ - **Global circuit breaker:** sustained floods flip the gem to count-only mode — zero per-event I/O, exact in-memory counts reconciled onto error records every 30s. Async mode is gated too (a SolidQueue enqueue is itself a DB write)
89
+ - **One storm notification** replaces hundreds of per-error pings; auto-issue creation is capped (default 5 per 10 min) so a storm of new errors can't open 500 GitHub/Linear issues
90
+ - **Honest accounting:** a dashboard banner during/after the storm, a Storm History page with exact counts of everything shed, and `reached_open`/peak-rate per episode. Counts are never extrapolated
91
+ - **Calm-weather economy:** after 25 full-context captures of the same error per day, context is sampled (occurrence counting unaffected)
92
+ - **Fails open:** any internal storm-protection error means full capture. Protection can never be the thing that loses an error
93
+
94
+ ```ruby
95
+ config.enable_storm_protection = true # default
96
+ config.storm_open_threshold_per_second = 50 # per process
97
+ ```
98
+
99
+ All thresholds are per process and individually configurable. Disable with one flag.
100
+
101
+ **Measured overhead** (Apple Silicon, Ruby 4.0): 2.4µs/error with protection active and calm, 2.95µs in count-only mode, 0.2µs when disabled — against a 5µs budget. The check is a digest plus an atomic increment; there is no I/O on the hot path.
102
+ </details>
103
+
82
104
  <details>
83
105
  <summary><strong>Breadcrumbs — Request Activity Trail</strong></summary>
84
106
 
@@ -233,7 +255,7 @@ Payload contract matches the `LlmCallEvent` value object — see [`docs/LLM_OBSE
233
255
  </details>
234
256
 
235
257
  <details>
236
- <summary><strong>Issue Tracking — GitHub, GitLab, Codeberg</strong></summary>
258
+ <summary><strong>Issue Tracking — GitHub, GitLab, Codeberg, Linear</strong></summary>
237
259
 
238
260
  One switch connects errors to your issue tracker. Platform becomes the source of truth — status, assignees, labels, and comments are mirrored live in the dashboard.
239
261
 
@@ -250,6 +272,17 @@ config.issue_tracker_token = ENV["RED_BOT_TOKEN"]
250
272
  # That's it — provider and repo auto-detected from git_repository_url
251
273
  ```
252
274
 
275
+ Linear works too — it's not a git forge, so set the provider and team key explicitly:
276
+
277
+ ```ruby
278
+ config.enable_issue_tracking = true
279
+ config.issue_tracker_provider = :linear
280
+ config.issue_tracker_repo = "ENG" # Linear team key (issues land as ENG-123)
281
+ config.issue_tracker_token = ENV["RED_BOT_TOKEN"] # lin_api_... personal API key
282
+ ```
283
+
284
+ Closing maps to the team's first `completed` workflow state, reopening to `unstarted`/`backlog`. Two-way sync uses Linear webhooks (`Linear-Signature` HMAC verification).
285
+
253
286
  [Complete documentation →](docs/guides/CONFIGURATION.md)
254
287
  </details>
255
288
 
@@ -77,6 +77,11 @@ module RailsErrorDashboard
77
77
  def set_common_view_variables
78
78
  @applications = Application.ordered_by_name.pluck(:name, :id) rescue []
79
79
  @default_credentials_warning = RailsErrorDashboard.configuration.default_credentials? rescue false
80
+ # Only query when a before_action (e.g. ErrorsController#load_storm_banner)
81
+ # hasn't already loaded it this request — the error renderer runs after
82
+ # those callbacks, so reuse their result instead of querying a second time.
83
+ # Tracked by a flag rather than the value, since the common case is nil.
84
+ @storm_banner_event = Queries::StormHistory.banner_event unless @storm_banner_loaded
80
85
  end
81
86
  end
82
87
  end
@@ -5,6 +5,7 @@ module RailsErrorDashboard
5
5
  before_action :authenticate_dashboard_user!
6
6
  before_action :set_application_context
7
7
  before_action :check_default_credentials
8
+ before_action :load_storm_banner
8
9
 
9
10
  FILTERABLE_PARAMS = %i[
10
11
  error_type
@@ -329,6 +330,12 @@ module RailsErrorDashboard
329
330
  @pagy, @releases = pagy(:offset, all_releases, limit: params[:per_page] || 25)
330
331
  end
331
332
 
333
+ def storms
334
+ result = Queries::StormHistory.call
335
+ @active_storm = result[:active]
336
+ @storm_events = result[:events]
337
+ end
338
+
332
339
  def user_impact
333
340
  days = days_param(default: 30)
334
341
  @days = days
@@ -729,6 +736,11 @@ module RailsErrorDashboard
729
736
  @default_credentials_warning = RailsErrorDashboard.configuration.default_credentials?
730
737
  end
731
738
 
739
+ def load_storm_banner
740
+ @storm_banner_loaded = true
741
+ @storm_banner_event = Queries::StormHistory.banner_event
742
+ end
743
+
732
744
  def authenticate_dashboard_user!
733
745
  auth_lambda = RailsErrorDashboard.configuration.authenticate_with
734
746
 
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module RailsErrorDashboard
4
- # Receives webhooks from GitHub/GitLab/Codeberg for two-way issue sync.
4
+ # Receives webhooks from GitHub/GitLab/Codeberg/Linear for two-way issue sync.
5
5
  #
6
6
  # When an issue is closed/reopened on the platform, the corresponding
7
7
  # error in the dashboard is resolved/reopened to match.
@@ -10,6 +10,7 @@ module RailsErrorDashboard
10
10
  # - GitHub: X-Hub-Signature-256 (HMAC-SHA256)
11
11
  # - GitLab: X-Gitlab-Token (shared secret)
12
12
  # - Codeberg: X-Gitea-Signature (HMAC-SHA256)
13
+ # - Linear: Linear-Signature (HMAC-SHA256)
13
14
  class WebhooksController < ActionController::Base
14
15
  skip_before_action :verify_authenticity_token
15
16
 
@@ -29,6 +30,8 @@ module RailsErrorDashboard
29
30
  handle_gitlab(payload)
30
31
  when "codeberg"
31
32
  handle_codeberg(payload)
33
+ when "linear"
34
+ handle_linear(payload)
32
35
  else
33
36
  head :not_found
34
37
  return
@@ -69,6 +72,8 @@ module RailsErrorDashboard
69
72
  verify_gitlab_token(secret)
70
73
  when "codeberg"
71
74
  verify_codeberg_signature(body, secret)
75
+ when "linear"
76
+ verify_linear_signature(body, secret)
72
77
  else
73
78
  false
74
79
  end
@@ -99,6 +104,14 @@ module RailsErrorDashboard
99
104
  ActiveSupport::SecurityUtils.secure_compare(expected, signature)
100
105
  end
101
106
 
107
+ def verify_linear_signature(body, secret)
108
+ signature = request.headers["Linear-Signature"]
109
+ return false unless signature
110
+
111
+ expected = OpenSSL::HMAC.hexdigest("SHA256", secret, body)
112
+ ActiveSupport::SecurityUtils.secure_compare(expected, signature)
113
+ end
114
+
102
115
  def parse_payload
103
116
  JSON.parse(request.body.read)
104
117
  rescue JSON::ParserError
@@ -162,6 +175,27 @@ module RailsErrorDashboard
162
175
  end
163
176
  end
164
177
 
178
+ # Linear: Issue webhook fires with action create/update/remove. State changes
179
+ # arrive as updates with the old stateId in updatedFrom. Linear has no
180
+ # open/closed binary — completed/canceled state types map to resolved.
181
+ def handle_linear(payload)
182
+ return unless payload["type"] == "Issue" && payload["action"] == "update"
183
+ return unless payload["updatedFrom"]&.key?("stateId")
184
+
185
+ issue_number = payload.dig("data", "number")
186
+ return unless issue_number
187
+
188
+ error = find_error_by_issue(issue_number, "linear")
189
+ return unless error
190
+
191
+ state_type = payload.dig("data", "state", "type")
192
+ if state_type.in?(%w[completed canceled])
193
+ resolve_error(error, "Completed on Linear by #{payload.dig("actor", "name")}")
194
+ elsif state_type.in?(%w[triage backlog unstarted started])
195
+ reopen_error(error)
196
+ end
197
+ end
198
+
165
199
  def find_error_by_issue(issue_number, provider)
166
200
  ErrorLog.find_by(
167
201
  external_issue_number: issue_number,
@@ -0,0 +1,19 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RailsErrorDashboard
4
+ # Persists storm-protection count snapshots (mirrors SwallowedExceptionFlushJob):
5
+ # the gate accumulates counts in memory with zero I/O, snapshots are handed
6
+ # to this job at most once per flush interval, and ALL DB writes happen here.
7
+ class StormFlushJob < ApplicationJob
8
+ queue_as :default
9
+
10
+ def perform(entries: [], overflow: 0, episode: nil)
11
+ entries = entries.map { |e| e.respond_to?(:stringify_keys) ? e.stringify_keys : e }
12
+ episode = episode.stringify_keys if episode.respond_to?(:stringify_keys)
13
+
14
+ Commands::FlushStormCounts.call(entries: entries, overflow: overflow, episode: episode)
15
+ rescue => e
16
+ Rails.logger.error("[RailsErrorDashboard] StormFlushJob failed: #{e.class} - #{e.message}")
17
+ end
18
+ end
19
+ end
@@ -0,0 +1,74 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RailsErrorDashboard
4
+ # Sends the SINGLE "error storm in progress" notification per storm episode.
5
+ #
6
+ # During a storm, per-error notifications are suppressed (500 Slack pings
7
+ # help nobody) — this one message replaces them. The gate guarantees at
8
+ # most one enqueue per episode; this job just delivers.
9
+ class StormNotificationJob < ApplicationJob
10
+ queue_as :default
11
+
12
+ # @param started_at [String] ISO8601 episode start
13
+ # @param state [String] breaker state at notification time ("shedding"/"open")
14
+ def perform(started_at:, state: "shedding")
15
+ config = RailsErrorDashboard.configuration
16
+ message = build_message(started_at, state, config)
17
+
18
+ if config.enable_slack_notifications && config.slack_webhook_url.present?
19
+ post_json(config.slack_webhook_url, { text: message })
20
+ end
21
+
22
+ if config.enable_discord_notifications && config.discord_webhook_url.present?
23
+ post_json(config.discord_webhook_url, { content: message })
24
+ end
25
+
26
+ if config.enable_webhook_notifications && config.webhook_urls.any?
27
+ payload = {
28
+ event: "error_storm_detected",
29
+ started_at: started_at,
30
+ state: state,
31
+ application: app_name(config)
32
+ }
33
+ config.webhook_urls.each { |url| post_json(url, payload) }
34
+ end
35
+ rescue => e
36
+ Rails.logger.error("[RailsErrorDashboard] StormNotificationJob failed: #{e.class} - #{e.message}")
37
+ end
38
+
39
+ private
40
+
41
+ def build_message(started_at, state, config)
42
+ mode = state == "open" ? "count-only mode (occurrences tallied, detail paused)" : "shedding mode (context sampling active)"
43
+ dashboard = (config.dashboard_base_url || "").chomp("/")
44
+ link = dashboard.present? ? " Dashboard: #{dashboard}/errors/storms" : ""
45
+
46
+ ":warning: Error storm detected in #{app_name(config)} at #{started_at}. " \
47
+ "Storm protection engaged — #{mode}. Per-error notifications are " \
48
+ "suppressed until the storm subsides; exact counts are preserved.#{link}"
49
+ end
50
+
51
+ def app_name(config)
52
+ config.application_name || ENV["APPLICATION_NAME"] ||
53
+ (defined?(Rails) && Rails.application.class.module_parent_name) || "Rails Application"
54
+ end
55
+
56
+ def post_json(url, payload)
57
+ if defined?(HTTParty)
58
+ HTTParty.post(url, body: payload.to_json,
59
+ headers: { "Content-Type" => "application/json" }, timeout: 10)
60
+ else
61
+ uri = URI(url)
62
+ http = Net::HTTP.new(uri.host, uri.port)
63
+ http.use_ssl = uri.scheme == "https"
64
+ http.open_timeout = 5
65
+ http.read_timeout = 10
66
+ request = Net::HTTP::Post.new(uri.path, { "Content-Type" => "application/json" })
67
+ request.body = payload.to_json
68
+ http.request(request)
69
+ end
70
+ rescue => e
71
+ Rails.logger.error("[RailsErrorDashboard] Storm notification post failed: #{e.message}")
72
+ end
73
+ end
74
+ end
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module RailsErrorDashboard
4
+ # One row per storm-protection episode (per process). Powers the dashboard
5
+ # banner and the storm history page. Counts are exact, not extrapolated.
6
+ # Inherits ErrorLogsRecord so separate-database routing applies.
7
+ class StormEvent < ErrorLogsRecord
8
+ self.table_name = "rails_error_dashboard_storm_events"
9
+
10
+ scope :active, -> { where(ended_at: nil) }
11
+ scope :recent_first, -> { order(started_at: :desc) }
12
+ scope :ended_within, ->(duration) { where.not(ended_at: nil).where(ended_at: duration.ago..) }
13
+
14
+ def active?
15
+ ended_at.nil?
16
+ end
17
+
18
+ def duration_seconds
19
+ return nil unless ended_at
20
+
21
+ (ended_at - started_at).round
22
+ end
23
+
24
+ # @return [Array<Hash>] top fingerprints by count, [] when absent/corrupt
25
+ def top_fingerprints_list
26
+ return [] if top_fingerprints.blank?
27
+
28
+ parsed = JSON.parse(top_fingerprints)
29
+ parsed.is_a?(Array) ? parsed : []
30
+ rescue JSON::ParserError
31
+ []
32
+ end
33
+ end
34
+ end
@@ -1036,6 +1036,9 @@ tr[data-red-row-href]:hover .sev-bar { opacity: 1 !important; }
1036
1036
  <% if RailsErrorDashboard.configuration.enable_diagnostic_dump %>
1037
1037
  <% diag_items << { path: diagnostic_dumps_errors_path(nav_params), icon: 'bi-clipboard-pulse', label: 'Diagnostics' } %>
1038
1038
  <% end %>
1039
+ <% if RailsErrorDashboard.configuration.enable_storm_protection %>
1040
+ <% diag_items << { path: storms_errors_path(nav_params), icon: 'bi-cloud-lightning-rain', label: 'Storms' } %>
1041
+ <% end %>
1039
1042
 
1040
1043
  <% if diag_items.any? %>
1041
1044
  <div style="margin-bottom: var(--space-2);" id="navDiagSection">
@@ -1175,6 +1178,24 @@ tr[data-red-row-href]:hover .sev-bar { opacity: 1 !important; }
1175
1178
  </div>
1176
1179
  <script<%= " nonce=\"#{red_csp_nonce}\"".html_safe if red_csp_nonce %>>try { if (sessionStorage.getItem('red_dismiss_creds_warning') === '1') { document.getElementById('security-warning').style.display = 'none'; } } catch(e) {}</script>
1177
1180
  <% end %>
1181
+ <% if defined?(@storm_banner_event) && @storm_banner_event %>
1182
+ <% storm_active = @storm_banner_event.active? %>
1183
+ <div id="storm-banner" class="alert <%= storm_active ? 'alert-danger' : 'alert-info' %>" style="display: flex; align-items: center; gap: 10px; margin-top: var(--space-2); margin-bottom: var(--space-4); border-left: 4px solid <%= storm_active ? 'var(--status-critical)' : 'var(--status-info)' %>;">
1184
+ <i class="bi <%= storm_active ? 'bi-cloud-lightning-rain-fill' : 'bi-cloud-check' %>" style="font-size: 18px; flex-shrink: 0;"></i>
1185
+ <div style="flex: 1;">
1186
+ <% if storm_active %>
1187
+ <strong>Error storm in progress</strong> (since <%= @storm_banner_event.started_at.strftime("%H:%M %Z") %>) —
1188
+ storm protection engaged. Occurrences are being counted exactly;
1189
+ <%= @storm_banner_event.reached_open ? "detail capture is paused (count-only mode)" : "context capture is sampled" %>.
1190
+ <% else %>
1191
+ <strong>Storm protection engaged recently</strong> (ended <%= @storm_banner_event.ended_at.strftime("%H:%M %Z") %>) —
1192
+ <%= number_with_delimiter(@storm_banner_event.events_counted_only.to_i + @storm_banner_event.events_overflow.to_i) %> occurrences
1193
+ were recorded as exact counts with sampled detail.
1194
+ <% end %>
1195
+ <a href="<%= storms_errors_path %>" style="margin-left: 4px;">View storm history</a>
1196
+ </div>
1197
+ </div>
1198
+ <% end %>
1178
1199
  <%= yield %>
1179
1200
  </main>
1180
1201
 
@@ -18,6 +18,7 @@
18
18
  when "github" then "bi-github"
19
19
  when "gitlab" then "bi-gitlab"
20
20
  when "codeberg" then "bi-git"
21
+ when "linear" then "bi-kanban"
21
22
  else "bi-link-45deg"
22
23
  end %>
23
24
  <% issue = @platform_issue %>
@@ -0,0 +1,91 @@
1
+ <% content_for :page_title, "Storm History" %>
2
+
3
+ <div>
4
+ <div class="d-flex justify-content-between align-items-center mb-4">
5
+ <h1 style="font-size: 20px; font-weight: 700; margin: 0;">
6
+ <i class="bi bi-cloud-lightning-rain me-2"></i>
7
+ Storm History
8
+ </h1>
9
+ </div>
10
+
11
+ <p class="text-muted" style="font-size: 13px; margin-bottom: var(--space-4);">
12
+ When the error rate spikes (a bad deploy, a dependency outage), storm protection
13
+ limits the gem's own database writes so it never amplifies the incident.
14
+ Occurrences are always counted exactly — only per-event detail is sampled.
15
+ Thresholds are configurable per process via <code>storm_*</code> options.
16
+ </p>
17
+
18
+ <% if @active_storm %>
19
+ <div class="alert alert-danger" style="border-left: 4px solid var(--status-critical);">
20
+ <i class="bi bi-cloud-lightning-rain-fill me-1"></i>
21
+ <strong>Storm in progress</strong> since <%= @active_storm.started_at.strftime("%Y-%m-%d %H:%M %Z") %> —
22
+ peak rate <%= number_with_delimiter(@active_storm.peak_rate_per_minute) %>/min.
23
+ </div>
24
+ <% end %>
25
+
26
+ <% if @storm_events.empty? %>
27
+ <div class="red-empty-state">
28
+ <i class="bi bi-cloud-sun display-1 text-muted mb-3"></i>
29
+ <div class="red-empty-state-title">No Storms Recorded</div>
30
+ <p class="text-muted">
31
+ Storm protection is standing by. If an error flood ever hits this app,
32
+ episodes will appear here with exact counts of what was shed.
33
+ </p>
34
+ </div>
35
+ <% else %>
36
+ <div class="card">
37
+ <div class="card-body" style="padding: 0;">
38
+ <table class="table" style="margin: 0;">
39
+ <thead>
40
+ <tr>
41
+ <th>Started</th>
42
+ <th>Duration</th>
43
+ <th>Peak Rate</th>
44
+ <th>Mode</th>
45
+ <th class="text-end">Counted (exact)</th>
46
+ <th class="text-end">Overflow</th>
47
+ <th>Top Errors</th>
48
+ </tr>
49
+ </thead>
50
+ <tbody>
51
+ <% @storm_events.each do |event| %>
52
+ <tr>
53
+ <td style="white-space: nowrap;">
54
+ <%= event.started_at.strftime("%Y-%m-%d %H:%M") %>
55
+ <% if event.active? %>
56
+ <span class="badge bg-danger ms-1">active</span>
57
+ <% end %>
58
+ </td>
59
+ <td>
60
+ <% if event.duration_seconds %>
61
+ <%= ActiveSupport::Duration.build(event.duration_seconds).inspect %>
62
+ <% else %>
63
+ &mdash;
64
+ <% end %>
65
+ </td>
66
+ <td><%= number_with_delimiter(event.peak_rate_per_minute) %>/min</td>
67
+ <td>
68
+ <% if event.reached_open %>
69
+ <span class="badge bg-danger">count-only</span>
70
+ <% else %>
71
+ <span class="badge bg-warning text-dark">shedding</span>
72
+ <% end %>
73
+ </td>
74
+ <td class="text-end"><%= number_with_delimiter(event.events_counted_only) %></td>
75
+ <td class="text-end"><%= number_with_delimiter(event.events_overflow) %></td>
76
+ <td style="max-width: 320px;">
77
+ <% event.top_fingerprints_list.first(3).each do |fp| %>
78
+ <div style="font-size: 12px; font-family: var(--font-mono); overflow: hidden; text-overflow: ellipsis; white-space: nowrap;" title="<%= fp["message"] %>">
79
+ <strong><%= fp["class"] %></strong>
80
+ <span class="text-muted">&times;<%= number_with_delimiter(fp["count"]) %></span>
81
+ </div>
82
+ <% end %>
83
+ </td>
84
+ </tr>
85
+ <% end %>
86
+ </tbody>
87
+ </table>
88
+ </div>
89
+ </div>
90
+ <% end %>
91
+ </div>
data/config/routes.rb CHANGED
@@ -40,6 +40,7 @@ RailsErrorDashboard::Engine.routes.draw do
40
40
  get :activestorage_health_summary
41
41
  get :llm_health_summary
42
42
  get :releases
43
+ get :storms
43
44
  get :user_impact
44
45
  get :diagnostic_dumps
45
46
  post :create_diagnostic_dump
@@ -1,7 +1,13 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  class AddInstanceVariablesToErrorLogs < ActiveRecord::Migration[7.0]
4
- def change
4
+ def up
5
+ return if column_exists?(:rails_error_dashboard_error_logs, :instance_variables)
6
+
5
7
  add_column :rails_error_dashboard_error_logs, :instance_variables, :text
6
8
  end
9
+
10
+ def down
11
+ remove_column :rails_error_dashboard_error_logs, :instance_variables if column_exists?(:rails_error_dashboard_error_logs, :instance_variables)
12
+ end
7
13
  end
@@ -2,6 +2,10 @@
2
2
 
3
3
  class CreateRailsErrorDashboardSwallowedExceptions < ActiveRecord::Migration[7.0]
4
4
  def change
5
+ # Guard against the squashed schema migration having already created this
6
+ # table — without it, every later migration is silently cancelled.
7
+ return if table_exists?(:rails_error_dashboard_swallowed_exceptions)
8
+
5
9
  create_table :rails_error_dashboard_swallowed_exceptions do |t|
6
10
  t.string :exception_class, null: false, limit: 250
7
11
  t.string :raise_location, null: false, limit: 250
@@ -2,6 +2,10 @@
2
2
 
3
3
  class CreateRailsErrorDashboardDiagnosticDumps < ActiveRecord::Migration[7.0]
4
4
  def change
5
+ # Guard against the squashed schema migration having already created this
6
+ # table — without it, every later migration is silently cancelled.
7
+ return if table_exists?(:rails_error_dashboard_diagnostic_dumps)
8
+
5
9
  create_table :rails_error_dashboard_diagnostic_dumps do |t|
6
10
  t.references :application, null: false,
7
11
  foreign_key: { to_table: :rails_error_dashboard_applications }
@@ -0,0 +1,28 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Storm protection honesty layer: one row per storm episode, powering the
4
+ # dashboard banner ("storm detected, counts recorded, detail sampled") and
5
+ # the storm history page. Small table — a few rows per incident.
6
+ class CreateStormEvents < ActiveRecord::Migration[7.0]
7
+ def change
8
+ return if table_exists?(:rails_error_dashboard_storm_events)
9
+
10
+ create_table :rails_error_dashboard_storm_events do |t|
11
+ t.datetime :started_at, null: false
12
+ t.datetime :ended_at # NULL while the storm is active
13
+ t.integer :peak_rate_per_minute, default: 0
14
+ t.boolean :reached_open, default: false # true if count-only mode engaged
15
+ t.bigint :events_total, default: 0 # count-only total = counted_only + overflow (excludes :lite/:full rows)
16
+ t.bigint :events_counted_only, default: 0 # counted in memory, no rows
17
+ t.bigint :events_overflow, default: 0 # beyond the bounded map — exact total, anonymous identity
18
+ t.integer :fingerprints_affected, default: 0
19
+ t.text :top_fingerprints # JSON: top 5 by count [{class, message, count}]
20
+ t.timestamps
21
+ end
22
+
23
+ add_index :rails_error_dashboard_storm_events, :ended_at,
24
+ name: "index_red_storm_events_on_ended_at"
25
+ add_index :rails_error_dashboard_storm_events, :started_at,
26
+ name: "index_red_storm_events_on_started_at"
27
+ end
28
+ end
@@ -519,7 +519,43 @@ RailsErrorDashboard.configure do |config|
519
519
  # Safe to enable on production OTel pipelines.
520
520
 
521
521
  # ============================================================================
522
- # ISSUE TRACKING (GitHub / GitLab / Codeberg)
522
+ # STORM PROTECTION (circuit breaker + adaptive sampling) — ON by default
523
+ # ============================================================================
524
+ #
525
+ # When the error rate spikes (bad deploy, dependency outage), storm
526
+ # protection limits the gem's own database writes so it never amplifies
527
+ # the incident. Occurrences are ALWAYS counted exactly — only per-event
528
+ # detail (context payloads, occurrence rows) is sampled under load.
529
+ #
530
+ # How it degrades, in order:
531
+ # 1. Per-fingerprint cap: past N/min, context is shed, then rows sampled
532
+ # 2. Global breaker: shedding (context off) → open (count-only mode)
533
+ # 3. Per-error notifications replaced by ONE "storm in progress" message
534
+ # 4. Counts reconciled onto error records every flush interval
535
+ #
536
+ # All thresholds are PER PROCESS (each Puma worker runs its own breaker).
537
+ #
538
+ # config.enable_storm_protection = true
539
+ # config.storm_fingerprint_full_per_minute = 30 # full-fidelity captures per fingerprint/min
540
+ # config.storm_occurrence_sample_keep_every = 10 # past the cap, keep every Nth occurrence
541
+ # config.storm_shedding_threshold_per_second = 10 # global rate entering shedding state
542
+ # config.storm_open_threshold_per_second = 50 # global rate opening the breaker (count-only)
543
+ # config.storm_cooldown_seconds = 60 # open → half-open probe delay
544
+ # config.storm_notification = true # one notification per storm episode
545
+ #
546
+ # Always-on issue cap (a storm of NEW critical errors must not open
547
+ # hundreds of GitHub/Linear issues):
548
+ # config.auto_issue_rate_limit_count = 5
549
+ # config.auto_issue_rate_limit_window_minutes = 10
550
+ #
551
+ # Calm-weather context economy: an error seen 1000x/day doesn't need 1000
552
+ # breadcrumb trails. After N full-context captures per fingerprint per day,
553
+ # context is kept every Mth time (occurrence rows are unaffected):
554
+ # config.context_sampling_threshold_per_day = 25
555
+ # config.context_sampling_keep_every = 10
556
+
557
+ # ============================================================================
558
+ # ISSUE TRACKING (GitHub / GitLab / Codeberg / Linear)
523
559
  # ============================================================================
524
560
  #
525
561
  # One switch enables everything: issue creation, auto-create on first
@@ -534,7 +570,7 @@ RailsErrorDashboard.configure do |config|
534
570
  # - Priority → labels from platform (with colors)
535
571
  # - Snooze and Mute remain (no platform equivalent)
536
572
  #
537
- # Setup:
573
+ # Setup (GitHub/GitLab/Codeberg):
538
574
  # 1. Create a RED bot account on GitHub/GitLab/Codeberg
539
575
  # 2. Generate a token and set RED_BOT_TOKEN env var
540
576
  # 3. Set git_repository_url above (already used for source code linking)
@@ -543,6 +579,17 @@ RailsErrorDashboard.configure do |config|
543
579
  # config.enable_issue_tracking = true
544
580
  # config.issue_tracker_token = ENV["RED_BOT_TOKEN"]
545
581
  #
582
+ # Setup (Linear):
583
+ # Linear is not a git forge, so it cannot be auto-detected from
584
+ # git_repository_url — set provider and team key explicitly. Issues are
585
+ # created in the team matching the key (e.g. "ENG" for ENG-123 issues).
586
+ # Generate a personal API key under Settings > Security & access.
587
+ #
588
+ # config.enable_issue_tracking = true
589
+ # config.issue_tracker_provider = :linear
590
+ # config.issue_tracker_repo = "ENG" # Linear team key
591
+ # config.issue_tracker_token = ENV["RED_BOT_TOKEN"] # lin_api_... key
592
+ #
546
593
  # Optional overrides:
547
594
  # config.issue_tracker_labels = ["bug"] # Labels added to new issues
548
595
  # config.issue_tracker_auto_create_severities = [:critical, :high] # Auto-create threshold
@@ -2,7 +2,7 @@
2
2
 
3
3
  module RailsErrorDashboard
4
4
  module Commands
5
- # Command: Create an issue on the configured issue tracker (GitHub/GitLab/Codeberg)
5
+ # Command: Create an issue on the configured issue tracker (GitHub/GitLab/Codeberg/Linear)
6
6
  #
7
7
  # Creates the issue via the provider API, then stores the issue URL, number,
8
8
  # and provider on the error record for linking.