rails_error_dashboard 0.11.9 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +4 -4
- data/app/controllers/rails_error_dashboard/errors_controller.rb +14 -4
- data/app/controllers/rails_error_dashboard/webhooks_controller.rb +79 -6
- data/app/jobs/rails_error_dashboard/add_issue_recurrence_comment_job.rb +4 -2
- data/app/jobs/rails_error_dashboard/application_job.rb +56 -14
- data/app/jobs/rails_error_dashboard/async_error_logging_job.rb +22 -4
- data/app/jobs/rails_error_dashboard/close_linked_issue_job.rb +4 -2
- data/app/jobs/rails_error_dashboard/create_issue_job.rb +1 -1
- data/app/jobs/rails_error_dashboard/reopen_linked_issue_job.rb +4 -2
- data/app/jobs/rails_error_dashboard/retention_cleanup_job.rb +25 -0
- data/app/jobs/rails_error_dashboard/storm_flush_job.rb +22 -4
- data/app/models/rails_error_dashboard/error_log.rb +47 -0
- data/app/models/rails_error_dashboard/storm_flush_batch.rb +50 -0
- data/app/views/rails_error_dashboard/errors/_request_context.html.erb +27 -1
- data/app/views/rails_error_dashboard/errors/_stats.html.erb +6 -0
- data/app/views/rails_error_dashboard/errors/overview.html.erb +13 -1
- data/config/locales/de.yml +9 -0
- data/config/locales/en.yml +35 -0
- data/config/locales/es.yml +9 -0
- data/config/locales/fr.yml +10 -1
- data/config/locales/it.yml +9 -0
- data/config/locales/ja.yml +9 -0
- data/config/locales/pl.yml +9 -0
- data/config/locales/pt-BR.yml +9 -0
- data/config/locales/ru.yml +9 -0
- data/config/locales/uk.yml +9 -0
- data/config/locales/zh-CN.yml +9 -0
- data/db/migrate/20260915000001_add_group_identity_unique_index_to_error_logs.rb +207 -0
- data/db/migrate/20260915000002_add_issue_repo_identity_to_error_logs.rb +89 -0
- data/db/migrate/20260915000003_add_context_provenance_to_error_logs.rb +42 -0
- data/db/migrate/20260915000004_create_storm_flush_batches.rb +50 -0
- data/lib/rails_error_dashboard/commands/create_issue.rb +10 -2
- data/lib/rails_error_dashboard/commands/find_or_increment_error.rb +135 -4
- data/lib/rails_error_dashboard/commands/flush_storm_counts.rb +139 -34
- data/lib/rails_error_dashboard/commands/link_existing_issue.rb +20 -2
- data/lib/rails_error_dashboard/commands/log_error.rb +206 -19
- data/lib/rails_error_dashboard/configuration.rb +4 -3
- data/lib/rails_error_dashboard/engine.rb +28 -0
- data/lib/rails_error_dashboard/integrations/tracer.rb +26 -7
- data/lib/rails_error_dashboard/queries/dashboard_stats.rb +147 -61
- data/lib/rails_error_dashboard/queries/user_impact_summary.rb +57 -7
- data/lib/rails_error_dashboard/services/error_hash_generator.rb +61 -19
- data/lib/rails_error_dashboard/services/issue_tracker_client.rb +38 -0
- data/lib/rails_error_dashboard/services/storm_protection/count_buffer.rb +22 -3
- data/lib/rails_error_dashboard/services/storm_protection/gate.rb +101 -14
- data/lib/rails_error_dashboard/version.rb +1 -1
- metadata +9 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2c23f8c75c71f71913d2f5964f25a38642373f2f720e8996377a5877e54a1e6a
|
|
4
|
+
data.tar.gz: 95e1db7a2687ed2900e3a02a2f33d1a1c21fc70c4597f9d9bf5d5855356d2f98
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: acb081a03269304c9c70b63ee996bd8d446a525bd506f401851d8249ad9bffe2ead0bf63798be64c231604176e03c614c761e4f551b9827802b7881b99f3b1b0
|
|
7
|
+
data.tar.gz: d28393ca63c1b9639daeb2409cdc3f20074f905e1cbc9e3a89ae7b881dbe2666bfbaf53052d1172803de0ca46d58ff7ae8b4bd6f4d9b24be3ffa61ae1283a20e
|
data/README.md
CHANGED
|
@@ -679,14 +679,14 @@ end
|
|
|
679
679
|
|
|
680
680
|
## Languages
|
|
681
681
|
|
|
682
|
-
RED ships in English with machine-translated previews for
|
|
682
|
+
RED ships in English, with French reviewed by a native speaker and machine-translated previews for nine further languages, covering the dashboard, its emails and its notification payloads. Native-speaking Rails developers are invited to review and improve them; once a locale has been reviewed it is marked individually as community-reviewed, as French now is. Eleven locales ship:
|
|
683
683
|
|
|
684
684
|
| Locale | Language | Status |
|
|
685
685
|
|---|---|---|
|
|
686
686
|
| `en` | English | Source language |
|
|
687
687
|
| `de` | Deutsch | Machine-translated, unreviewed |
|
|
688
688
|
| `es` | Español | Machine-translated, unreviewed |
|
|
689
|
-
| `fr` | Français |
|
|
689
|
+
| `fr` | Français | Community-reviewed by a native speaker |
|
|
690
690
|
| `pt-BR` | Português (Brasil) | Machine-translated, unreviewed |
|
|
691
691
|
| `ja` | 日本語 | Machine-translated, unreviewed |
|
|
692
692
|
| `ru` | Русский | Machine-translated, unreviewed |
|
|
@@ -701,9 +701,9 @@ config.dashboard_locale = "de" # en, de, es, fr, pt-BR, ja, ru, uk, pl, it, zh-
|
|
|
701
701
|
|
|
702
702
|
Users can also switch language per-session from the picker in the dashboard navbar, which overrides the configured default for them alone.
|
|
703
703
|
|
|
704
|
-
**
|
|
704
|
+
**French has been reviewed by a native speaker; everything but English and French is machine-translated and has not been.** That is stated plainly rather than as "beta", which would imply a review process that has not happened — RED's maintainer reads only English. Key structure, interpolation variables and plural categories *are* verified mechanically in every locale; outside French, wording, register and idiom are not verified by anyone. A wrong or missing translation falls back to **English**, never to a broken page.
|
|
705
705
|
|
|
706
|
-
**Corrections are very welcome, and a one-key PR is a perfectly good PR.** If you read one of these languages, [every locale has an open issue](https://github.com/AnjanJ/rails_error_dashboard/issues?q=is%3Aissue+is%3Aopen+label%3Atranslation%3Aneeds-review) tracking its review — comment there, or [report a bad translation](https://github.com/AnjanJ/rails_error_dashboard/issues/new?template=translation_report.yml) without touching any code. You do not need to know Ruby, and you are not expected to review a whole file.
|
|
706
|
+
**Corrections are very welcome, and a one-key PR is a perfectly good PR.** If you read one of these languages, [every unreviewed locale has an open issue](https://github.com/AnjanJ/rails_error_dashboard/issues?q=is%3Aissue+is%3Aopen+label%3Atranslation%3Aneeds-review) tracking its review — comment there, or [report a bad translation](https://github.com/AnjanJ/rails_error_dashboard/issues/new?template=translation_report.yml) without touching any code. You do not need to know Ruby, and you are not expected to review a whole file.
|
|
707
707
|
|
|
708
708
|
RED translates through its own private I18n backend, so it never reads, writes or mutates your application's `I18n` configuration — your locale and its `available_locales` are untouched.
|
|
709
709
|
|
|
@@ -769,13 +769,23 @@ module RailsErrorDashboard
|
|
|
769
769
|
@current_application_id.present? ? { application_id: @current_application_id } : {}
|
|
770
770
|
end
|
|
771
771
|
|
|
772
|
+
# Cache key for one linked issue's platform data. Includes the repository
|
|
773
|
+
# (or Linear team), because provider + number alone is not unique: issue 42
|
|
774
|
+
# exists in every repository, and Linear numbers issues per team.
|
|
775
|
+
def issue_cache_key(kind, error)
|
|
776
|
+
repo = error.respond_to?(:external_issue_repo) ? error.external_issue_repo : nil
|
|
777
|
+
[ "red", kind, error.external_issue_provider, repo.presence || "-", error.external_issue_number ].join("/")
|
|
778
|
+
end
|
|
779
|
+
|
|
772
780
|
def fetch_platform_issue(error)
|
|
773
781
|
return nil unless error.external_issue_url.present? && error.external_issue_number.present?
|
|
774
782
|
return nil unless RailsErrorDashboard.configuration.enable_issue_tracking
|
|
775
783
|
|
|
776
|
-
|
|
784
|
+
# The repository belongs in the key: two repositories sharing an issue
|
|
785
|
+
# number would otherwise read each other's cached state.
|
|
786
|
+
cache_key = issue_cache_key("issue_state", error)
|
|
777
787
|
Rails.cache.fetch(cache_key, expires_in: 60.seconds) do
|
|
778
|
-
client = Services::IssueTrackerClient.
|
|
788
|
+
client = Services::IssueTrackerClient.for_error(error)
|
|
779
789
|
return nil unless client
|
|
780
790
|
|
|
781
791
|
result = client.fetch_issue(number: error.external_issue_number)
|
|
@@ -790,9 +800,9 @@ module RailsErrorDashboard
|
|
|
790
800
|
return [] unless RailsErrorDashboard.configuration.enable_issue_tracking
|
|
791
801
|
|
|
792
802
|
# Cache for 60 seconds to avoid API hammering on page refreshes
|
|
793
|
-
cache_key = "
|
|
803
|
+
cache_key = issue_cache_key("issue_comments", error)
|
|
794
804
|
Rails.cache.fetch(cache_key, expires_in: 60.seconds) do
|
|
795
|
-
client = Services::IssueTrackerClient.
|
|
805
|
+
client = Services::IssueTrackerClient.for_error(error)
|
|
796
806
|
return [] unless client
|
|
797
807
|
|
|
798
808
|
result = client.fetch_comments(number: error.external_issue_number, per_page: 20)
|
|
@@ -127,7 +127,7 @@ module RailsErrorDashboard
|
|
|
127
127
|
issue_number = payload.dig("issue", "number")
|
|
128
128
|
return unless issue_number
|
|
129
129
|
|
|
130
|
-
error = find_error_by_issue(issue_number, "github")
|
|
130
|
+
error = find_error_by_issue(issue_number, "github", repo_from_payload("github", payload))
|
|
131
131
|
return unless error
|
|
132
132
|
|
|
133
133
|
case payload["action"]
|
|
@@ -146,7 +146,7 @@ module RailsErrorDashboard
|
|
|
146
146
|
issue_iid = payload.dig("object_attributes", "iid")
|
|
147
147
|
return unless issue_iid
|
|
148
148
|
|
|
149
|
-
error = find_error_by_issue(issue_iid, "gitlab")
|
|
149
|
+
error = find_error_by_issue(issue_iid, "gitlab", repo_from_payload("gitlab", payload))
|
|
150
150
|
return unless error
|
|
151
151
|
|
|
152
152
|
case action
|
|
@@ -164,7 +164,7 @@ module RailsErrorDashboard
|
|
|
164
164
|
issue_number = payload.dig("issue", "number")
|
|
165
165
|
return unless issue_number
|
|
166
166
|
|
|
167
|
-
error = find_error_by_issue(issue_number, "codeberg")
|
|
167
|
+
error = find_error_by_issue(issue_number, "codeberg", repo_from_payload("codeberg", payload))
|
|
168
168
|
return unless error
|
|
169
169
|
|
|
170
170
|
case payload["action"]
|
|
@@ -185,7 +185,7 @@ module RailsErrorDashboard
|
|
|
185
185
|
issue_number = payload.dig("data", "number")
|
|
186
186
|
return unless issue_number
|
|
187
187
|
|
|
188
|
-
error = find_error_by_issue(issue_number, "linear")
|
|
188
|
+
error = find_error_by_issue(issue_number, "linear", repo_from_payload("linear", payload))
|
|
189
189
|
return unless error
|
|
190
190
|
|
|
191
191
|
state_type = payload.dig("data", "state", "type")
|
|
@@ -196,11 +196,84 @@ module RailsErrorDashboard
|
|
|
196
196
|
end
|
|
197
197
|
end
|
|
198
198
|
|
|
199
|
-
|
|
200
|
-
|
|
199
|
+
# The repository (or Linear team) this payload is about.
|
|
200
|
+
#
|
|
201
|
+
# Every provider already sends it and RED already parses the rest of the
|
|
202
|
+
# payload around it; it was simply never read. Linear has no repository --
|
|
203
|
+
# issue numbers are scoped to a team, which is exactly why it needs this
|
|
204
|
+
# most -- so the team key is taken from the issue identifier ("ENG-123")
|
|
205
|
+
# or the team object.
|
|
206
|
+
def repo_from_payload(provider, payload)
|
|
207
|
+
case provider
|
|
208
|
+
when "github", "codeberg"
|
|
209
|
+
payload.dig("repository", "full_name")
|
|
210
|
+
when "gitlab"
|
|
211
|
+
payload.dig("project", "path_with_namespace") ||
|
|
212
|
+
payload.dig("object_attributes", "project", "path_with_namespace")
|
|
213
|
+
when "linear"
|
|
214
|
+
key = payload.dig("data", "team", "key")
|
|
215
|
+
key ||= payload.dig("data", "identifier").to_s[/\A([A-Za-z][A-Za-z0-9]*)-\d+\z/, 1]
|
|
216
|
+
key&.upcase
|
|
217
|
+
end
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
# Match the FULL issue identity: provider, number and repository.
|
|
221
|
+
#
|
|
222
|
+
# A validly signed webhook from one repository used to resolve an error
|
|
223
|
+
# linked to a different repository that happened to share an issue number.
|
|
224
|
+
# That is not a signature bypass -- the request is genuine -- but it acts
|
|
225
|
+
# on the wrong error, which matters for shared databases, several linked
|
|
226
|
+
# repositories, a repository rename, and Linear's team-scoped numbering.
|
|
227
|
+
#
|
|
228
|
+
# A row that DOES record a repository is never matched by a different one.
|
|
229
|
+
#
|
|
230
|
+
# Rows linked before the repository was recorded have NULL there, and
|
|
231
|
+
# cannot simply be adopted by whoever asks first -- that would leave the
|
|
232
|
+
# original defect intact for exactly the rows most likely to be affected.
|
|
233
|
+
# Their stored issue URL is corroborating evidence: it names the repository
|
|
234
|
+
# the link was made against. If it disagrees with the payload, this webhook
|
|
235
|
+
# is about a different issue that merely shares a number.
|
|
236
|
+
#
|
|
237
|
+
# Only a legacy row whose URL cannot be parsed at all (an unrecognised
|
|
238
|
+
# forge) falls back to matching on provider + number, as before -- there is
|
|
239
|
+
# no evidence either way there, and dropping the event would break a
|
|
240
|
+
# working link.
|
|
241
|
+
def find_error_by_issue(issue_number, provider, repo = nil)
|
|
242
|
+
scope = ErrorLog.where(
|
|
201
243
|
external_issue_number: issue_number,
|
|
202
244
|
external_issue_provider: provider
|
|
203
245
|
)
|
|
246
|
+
return scope.first unless ErrorLog.column_names.include?("external_issue_repo")
|
|
247
|
+
|
|
248
|
+
return scope.first if repo.blank?
|
|
249
|
+
|
|
250
|
+
exact = scope.where(external_issue_repo: repo).first
|
|
251
|
+
return exact if exact
|
|
252
|
+
|
|
253
|
+
legacy = scope.where(external_issue_repo: nil).find { |error|
|
|
254
|
+
url_repo = repo_from_issue_url(provider, error.external_issue_url)
|
|
255
|
+
url_repo.nil? || url_repo.casecmp?(repo.to_s)
|
|
256
|
+
}
|
|
257
|
+
# The identity is now established, so the row is precise from here on.
|
|
258
|
+
legacy&.update_columns(external_issue_repo: repo)
|
|
259
|
+
legacy
|
|
260
|
+
end
|
|
261
|
+
|
|
262
|
+
# The repository a stored issue URL was linked against, using the same
|
|
263
|
+
# patterns LinkExistingIssue parses links with -- one definition, so the
|
|
264
|
+
# two can never disagree about what a URL means.
|
|
265
|
+
#
|
|
266
|
+
# nil when the URL is absent or from an unrecognised forge.
|
|
267
|
+
def repo_from_issue_url(provider, url)
|
|
268
|
+
return nil if url.blank?
|
|
269
|
+
|
|
270
|
+
pattern = Commands::LinkExistingIssue::PROVIDER_PATTERNS[provider.to_s.to_sym]
|
|
271
|
+
return nil unless pattern
|
|
272
|
+
|
|
273
|
+
match = url.match(pattern)
|
|
274
|
+
return nil unless match
|
|
275
|
+
|
|
276
|
+
provider.to_s == "linear" ? match[1].upcase : match[1]
|
|
204
277
|
end
|
|
205
278
|
|
|
206
279
|
def resolve_error(error, message)
|
|
@@ -9,7 +9,7 @@ module RailsErrorDashboard
|
|
|
9
9
|
class AddIssueRecurrenceCommentJob < ApplicationJob
|
|
10
10
|
queue_as :error_notifications
|
|
11
11
|
|
|
12
|
-
retry_on StandardError, wait:
|
|
12
|
+
retry_on StandardError, wait: RailsErrorDashboard::ApplicationJob::POLYNOMIAL_BACKOFF, attempts: 2
|
|
13
13
|
discard_on ActiveRecord::RecordNotFound
|
|
14
14
|
|
|
15
15
|
THROTTLE_INTERVAL = 3600 # 1 hour
|
|
@@ -27,7 +27,9 @@ module RailsErrorDashboard
|
|
|
27
27
|
error = ErrorLog.find(error_log_id)
|
|
28
28
|
return unless error.external_issue_url.present? && error.external_issue_number.present?
|
|
29
29
|
|
|
30
|
-
|
|
30
|
+
# The repository this issue was actually opened in, not whatever the
|
|
31
|
+
# global configuration currently points at.
|
|
32
|
+
client = Services::IssueTrackerClient.for_error(error)
|
|
31
33
|
return unless client
|
|
32
34
|
|
|
33
35
|
comment = "Error occurred again (#{error.occurrence_count} total occurrences).\n\n"
|
|
@@ -5,24 +5,66 @@ module RailsErrorDashboard
|
|
|
5
5
|
# the request-scoped locale directly — see Concerns::LocalizedJob for why.
|
|
6
6
|
include Concerns::LocalizedJob
|
|
7
7
|
|
|
8
|
+
# Polynomial backoff, as a Proc rather than the :polynomially_longer
|
|
9
|
+
# symbol.
|
|
10
|
+
#
|
|
11
|
+
# That symbol arrived in Rails 7.1; on 7.0 -- which this gem still
|
|
12
|
+
# supports -- retry_on raises "Couldn't determine a delay based on
|
|
13
|
+
# :polynomially_longer" the moment a retry is actually scheduled. The
|
|
14
|
+
# symbol therefore passed every 7.1+ CI row and failed every 7.0 one.
|
|
15
|
+
#
|
|
16
|
+
# This is the same formula Rails uses: (executions ** 4) + 2, with jitter.
|
|
17
|
+
POLYNOMIAL_BACKOFF = ->(executions) {
|
|
18
|
+
((executions**4) + 2) + (Kernel.rand * (executions**4) * 0.15)
|
|
19
|
+
}
|
|
20
|
+
|
|
8
21
|
# CRITICAL: Ensure job failures don't break the app or spam error logs
|
|
9
|
-
# Retry failed jobs with polynomial backoff, but limit attempts
|
|
10
|
-
|
|
22
|
+
# Retry failed jobs with polynomial backoff, but limit attempts.
|
|
23
|
+
#
|
|
24
|
+
# This must be the ONLY StandardError handler on this class. A
|
|
25
|
+
# `rescue_from StandardError` used to sit below it and log-then-re-raise:
|
|
26
|
+
# rescue_from is resolved in reverse registration order, so it won the
|
|
27
|
+
# lookup every time and retry_on's handler was never consulted. Retries
|
|
28
|
+
# still happened (the re-raise reached the adapter) but the polynomial
|
|
29
|
+
# backoff never applied -- failures retried immediately, three times, on
|
|
30
|
+
# every job in the gem. The logging that block provided now runs from
|
|
31
|
+
# retry_on's own exhaustion block, where it fires once, after the last
|
|
32
|
+
# attempt, with the same detail.
|
|
33
|
+
retry_on StandardError, wait: RailsErrorDashboard::ApplicationJob::POLYNOMIAL_BACKOFF, attempts: 3 do |job, error|
|
|
34
|
+
Rails.logger.error("[RailsErrorDashboard] Job #{job.class.name} failed: #{error.class} - #{error.message}")
|
|
35
|
+
Rails.logger.error("Job arguments: #{job.arguments.inspect}")
|
|
36
|
+
Rails.logger.error("Attempt: #{job.executions}/3")
|
|
37
|
+
Rails.logger.error(error.backtrace&.first(10)&.join("\n")) if error.backtrace
|
|
38
|
+
Rails.logger.error("[RailsErrorDashboard] Job #{job.class.name} discarded after #{job.executions} attempts")
|
|
39
|
+
end
|
|
11
40
|
|
|
12
|
-
#
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
41
|
+
# Did this perform_later actually reach the queue?
|
|
42
|
+
#
|
|
43
|
+
# Active Job does NOT raise on every failed handoff: from Rails 7.2 an
|
|
44
|
+
# ActiveJob::EnqueueError raised by the adapter is caught inside
|
|
45
|
+
# perform_later, which then returns false and leaves enqueue_error set on
|
|
46
|
+
# the job. A caller that only rescues therefore treats a dropped job as a
|
|
47
|
+
# successful one. Callbacks that abort the enqueue behave the same way.
|
|
48
|
+
#
|
|
49
|
+
# Returns true for adapters/versions that predate successfully_enqueued?
|
|
50
|
+
# (Rails 7.0/7.1 let the error propagate instead, so the caller's rescue
|
|
51
|
+
# is what catches it there).
|
|
52
|
+
#
|
|
53
|
+
# @param job [ActiveJob::Base, false, nil] whatever perform_later returned
|
|
54
|
+
# @return [Boolean]
|
|
55
|
+
def self.enqueued?(job)
|
|
56
|
+
return false unless job
|
|
57
|
+
return job.successfully_enqueued? if job.respond_to?(:successfully_enqueued?)
|
|
19
58
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
raise exception if executions < 3
|
|
59
|
+
true
|
|
60
|
+
end
|
|
23
61
|
|
|
24
|
-
|
|
25
|
-
|
|
62
|
+
# Why a handoff failed, for the log line that reports it.
|
|
63
|
+
# @param job [ActiveJob::Base, false, nil]
|
|
64
|
+
# @return [String]
|
|
65
|
+
def self.enqueue_failure_reason(job)
|
|
66
|
+
(job.respond_to?(:enqueue_error) && job.enqueue_error&.message) ||
|
|
67
|
+
"perform_later returned #{job.inspect}"
|
|
26
68
|
end
|
|
27
69
|
end
|
|
28
70
|
end
|
|
@@ -23,11 +23,29 @@ module RailsErrorDashboard
|
|
|
23
23
|
context[:_serialized_cause_chain] = exception_data[:cause_chain]
|
|
24
24
|
end
|
|
25
25
|
|
|
26
|
-
# Log the error synchronously in the background job
|
|
27
|
-
#
|
|
28
|
-
|
|
26
|
+
# Log the error synchronously in the background job.
|
|
27
|
+
# .new(...).call bypasses the async check (we're already async);
|
|
28
|
+
# worker: true makes an unreachable error store raise instead of being
|
|
29
|
+
# swallowed, so this job fails and retries rather than acknowledging a
|
|
30
|
+
# capture it never wrote.
|
|
31
|
+
Commands::LogError.new(exception, context, worker: true).call
|
|
32
|
+
rescue *Commands::LogError::RETRYABLE_STORE_ERRORS => e
|
|
33
|
+
# The error database is unreachable. Protecting a user request and
|
|
34
|
+
# deciding whether a background job succeeded are different contracts:
|
|
35
|
+
# swallowing here told Active Job the capture was delivered when nothing
|
|
36
|
+
# was written, so the payload was gone for good. Let it fail instead and
|
|
37
|
+
# let retry_on schedule another attempt.
|
|
38
|
+
#
|
|
39
|
+
# Rails reports this job failure to Rails.error, so RED will try to
|
|
40
|
+
# capture it too; that capture fails the same way and is swallowed by
|
|
41
|
+
# LogError's own rescue. ErrorReporter's recursion guard (issue #114)
|
|
42
|
+
# plus the attempt cap bound this to a few wasted attempts, not a loop.
|
|
43
|
+
Rails.logger.error("AsyncErrorLoggingJob: error storage unavailable (#{e.class}: #{e.message}) — will retry")
|
|
44
|
+
raise
|
|
29
45
|
rescue => e
|
|
30
|
-
#
|
|
46
|
+
# A payload problem (unparseable arguments, a class that cannot be
|
|
47
|
+
# reconstructed). Retrying replays the identical payload, so it would
|
|
48
|
+
# fail identically three times and still be discarded. Log and drop.
|
|
31
49
|
Rails.logger.error("AsyncErrorLoggingJob failed: #{e.message}")
|
|
32
50
|
Rails.logger.error("Backtrace: #{e.backtrace&.first(5)&.join("\n")}")
|
|
33
51
|
end
|
|
@@ -8,14 +8,16 @@ module RailsErrorDashboard
|
|
|
8
8
|
class CloseLinkedIssueJob < ApplicationJob
|
|
9
9
|
queue_as :error_notifications
|
|
10
10
|
|
|
11
|
-
retry_on StandardError, wait:
|
|
11
|
+
retry_on StandardError, wait: RailsErrorDashboard::ApplicationJob::POLYNOMIAL_BACKOFF, attempts: 2
|
|
12
12
|
discard_on ActiveRecord::RecordNotFound
|
|
13
13
|
|
|
14
14
|
def perform(error_log_id)
|
|
15
15
|
error = ErrorLog.find(error_log_id)
|
|
16
16
|
return unless error.external_issue_url.present? && error.external_issue_number.present?
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
# The repository this issue was actually opened in, not whatever the
|
|
19
|
+
# global configuration currently points at.
|
|
20
|
+
client = Services::IssueTrackerClient.for_error(error)
|
|
19
21
|
return unless client
|
|
20
22
|
|
|
21
23
|
# Add resolution comment
|
|
@@ -12,7 +12,7 @@ module RailsErrorDashboard
|
|
|
12
12
|
class CreateIssueJob < ApplicationJob
|
|
13
13
|
queue_as :error_notifications
|
|
14
14
|
|
|
15
|
-
retry_on StandardError, wait:
|
|
15
|
+
retry_on StandardError, wait: RailsErrorDashboard::ApplicationJob::POLYNOMIAL_BACKOFF, attempts: 3
|
|
16
16
|
discard_on ActiveRecord::RecordNotFound
|
|
17
17
|
|
|
18
18
|
# Simple circuit breaker — class-level failure tracking
|
|
@@ -8,14 +8,16 @@ module RailsErrorDashboard
|
|
|
8
8
|
class ReopenLinkedIssueJob < ApplicationJob
|
|
9
9
|
queue_as :error_notifications
|
|
10
10
|
|
|
11
|
-
retry_on StandardError, wait:
|
|
11
|
+
retry_on StandardError, wait: RailsErrorDashboard::ApplicationJob::POLYNOMIAL_BACKOFF, attempts: 2
|
|
12
12
|
discard_on ActiveRecord::RecordNotFound
|
|
13
13
|
|
|
14
14
|
def perform(error_log_id)
|
|
15
15
|
error = ErrorLog.find(error_log_id)
|
|
16
16
|
return unless error.external_issue_url.present? && error.external_issue_number.present?
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
# The repository this issue was actually opened in, not whatever the
|
|
19
|
+
# global configuration currently points at.
|
|
20
|
+
client = Services::IssueTrackerClient.for_error(error)
|
|
19
21
|
return unless client
|
|
20
22
|
|
|
21
23
|
# Reopen the issue
|
|
@@ -25,6 +25,7 @@ module RailsErrorDashboard
|
|
|
25
25
|
# errors — clean them up BEFORE the early return below, which fires
|
|
26
26
|
# whenever no error logs happen to be expired.
|
|
27
27
|
cleanup_rack_attack_events(cutoff)
|
|
28
|
+
cleanup_storm_flush_batches(cutoff)
|
|
28
29
|
|
|
29
30
|
expired_scope = ErrorLog.where("occurred_at < ?", cutoff)
|
|
30
31
|
return 0 if expired_scope.none?
|
|
@@ -61,6 +62,30 @@ module RailsErrorDashboard
|
|
|
61
62
|
|
|
62
63
|
private
|
|
63
64
|
|
|
65
|
+
# Expire the storm batch ledger. Its only job is to make a replayed batch
|
|
66
|
+
# a no-op, and a replay arrives within the job's retry window (minutes),
|
|
67
|
+
# so rows well past the retention cutoff protect nothing. Own rescue, like
|
|
68
|
+
# the rack-attack cleanup: a failure here must never block error cleanup.
|
|
69
|
+
def cleanup_storm_flush_batches(cutoff)
|
|
70
|
+
return unless RailsErrorDashboard.configuration.enable_storm_protection
|
|
71
|
+
return unless StormFlushBatch.table_exists?
|
|
72
|
+
|
|
73
|
+
deleted = 0
|
|
74
|
+
StormFlushBatch.where("applied_at < ?", cutoff).in_batches(of: 1000) do |batch|
|
|
75
|
+
deleted += batch.delete_all
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
if deleted > 0
|
|
79
|
+
RailsErrorDashboard::Logger.info(
|
|
80
|
+
"[RailsErrorDashboard] Retention cleanup: deleted #{deleted} storm flush batch records"
|
|
81
|
+
)
|
|
82
|
+
end
|
|
83
|
+
rescue => e
|
|
84
|
+
RailsErrorDashboard::Logger.debug(
|
|
85
|
+
"[RailsErrorDashboard] Storm flush batch retention cleanup failed: #{e.class} - #{e.message}"
|
|
86
|
+
)
|
|
87
|
+
end
|
|
88
|
+
|
|
64
89
|
# Expire aggregated Rack Attack event rows. Isolated in its own rescue so a
|
|
65
90
|
# failure here (e.g. table not yet migrated) never blocks error cleanup.
|
|
66
91
|
def cleanup_rack_attack_events(cutoff)
|
|
@@ -5,15 +5,33 @@ module RailsErrorDashboard
|
|
|
5
5
|
# the gate accumulates counts in memory with zero I/O, snapshots are handed
|
|
6
6
|
# to this job at most once per flush interval, and ALL DB writes happen here.
|
|
7
7
|
class StormFlushJob < ApplicationJob
|
|
8
|
+
# Raised when the command reports that nothing in the batch could be
|
|
9
|
+
# written, so retry_on can schedule another attempt.
|
|
10
|
+
class FlushFailed < StandardError; end
|
|
11
|
+
|
|
8
12
|
queue_as :default
|
|
9
13
|
|
|
10
|
-
def perform(entries: [], overflow: 0, episode: nil)
|
|
14
|
+
def perform(entries: [], overflow: 0, episode: nil, batch_id: nil)
|
|
11
15
|
entries = entries.map { |e| e.respond_to?(:stringify_keys) ? e.stringify_keys : e }
|
|
12
16
|
episode = episode.stringify_keys if episode.respond_to?(:stringify_keys)
|
|
13
17
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
18
|
+
# batch_id identifies THIS batch across retries: ApplicationJob replays
|
|
19
|
+
# the identical payload, and the ledger has to recognise it.
|
|
20
|
+
result = Commands::FlushStormCounts.call(
|
|
21
|
+
entries: entries, overflow: overflow, episode: episode, batch_id: batch_id
|
|
22
|
+
)
|
|
23
|
+
|
|
24
|
+
# A batch that reconciled nothing because every write failed is not a
|
|
25
|
+
# delivered batch. Fail the job so Active Job retries it rather than
|
|
26
|
+
# dropping counts that were only ever held in one process's memory.
|
|
27
|
+
if result.is_a?(Hash) && result[:success] == false
|
|
28
|
+
raise FlushFailed, "storm flush reconciled nothing: #{result[:error]}"
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
result
|
|
32
|
+
rescue *Commands::LogError::RETRYABLE_STORE_ERRORS => e
|
|
33
|
+
Rails.logger.error("[RailsErrorDashboard] StormFlushJob: error storage unavailable (#{e.class}: #{e.message}) — will retry")
|
|
34
|
+
raise
|
|
17
35
|
end
|
|
18
36
|
end
|
|
19
37
|
end
|
|
@@ -79,6 +79,7 @@ module RailsErrorDashboard
|
|
|
79
79
|
# Set defaults and tracking
|
|
80
80
|
before_validation :set_defaults, on: :create
|
|
81
81
|
before_create :set_tracking_fields
|
|
82
|
+
before_save :sync_group_window
|
|
82
83
|
before_create :set_priority_score
|
|
83
84
|
|
|
84
85
|
# Turbo Stream broadcasting
|
|
@@ -98,6 +99,52 @@ module RailsErrorDashboard
|
|
|
98
99
|
self.first_seen_at ||= Time.current
|
|
99
100
|
self.last_seen_at ||= Time.current
|
|
100
101
|
self.occurrence_count ||= 1
|
|
102
|
+
set_group_window
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
# The immutable bucket that completes an unresolved group's database
|
|
106
|
+
# identity (see 20260915000001). Stamped once, at creation, and never
|
|
107
|
+
# rewritten -- not on increment, not on reopen: the row keeps the identity
|
|
108
|
+
# it was created with for its whole life, which is exactly what makes it
|
|
109
|
+
# usable as an index key.
|
|
110
|
+
#
|
|
111
|
+
# Two captures racing to create the same group are milliseconds apart and
|
|
112
|
+
# therefore share a bucket, so one of them loses on the unique index and
|
|
113
|
+
# takes the existing RecordNotUnique retry path. A capture that arrives
|
|
114
|
+
# after the previous group's 24 h window has rolled over lands in a
|
|
115
|
+
# different bucket and is allowed to open a new group, which is intended
|
|
116
|
+
# behaviour.
|
|
117
|
+
def set_group_window
|
|
118
|
+
return unless respond_to?(:group_window=)
|
|
119
|
+
return if group_window.present?
|
|
120
|
+
|
|
121
|
+
self.group_window = computed_group_window
|
|
122
|
+
rescue StandardError
|
|
123
|
+
# A bad occurred_at must never block a capture; a NULL bucket simply
|
|
124
|
+
# falls back to the pre-migration behaviour for this one row.
|
|
125
|
+
nil
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
# The bucket is DERIVED from occurred_at, so it must follow it. In
|
|
129
|
+
# production occurred_at is written once, at creation, and never rewritten
|
|
130
|
+
# -- so this is a no-op there and the bucket is immutable in practice.
|
|
131
|
+
# But anything that does move occurred_at (a fixture ageing a row, a
|
|
132
|
+
# backfill, a data repair) would otherwise leave the bucket pointing at the
|
|
133
|
+
# old window, and the row would sit in an index slot that no longer matches
|
|
134
|
+
# its own timestamp: a later capture computing the correct bucket would
|
|
135
|
+
# collide with a row the 24 h lookup had already excluded.
|
|
136
|
+
def sync_group_window
|
|
137
|
+
return unless respond_to?(:group_window=)
|
|
138
|
+
return unless has_attribute?(:occurred_at) && will_save_change_to_attribute?(:occurred_at)
|
|
139
|
+
|
|
140
|
+
self.group_window = computed_group_window
|
|
141
|
+
rescue StandardError
|
|
142
|
+
nil
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
def computed_group_window
|
|
146
|
+
basis = occurred_at || first_seen_at || Time.current
|
|
147
|
+
basis.utc.strftime("%Y-%m-%d")
|
|
101
148
|
end
|
|
102
149
|
|
|
103
150
|
def set_priority_score
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RailsErrorDashboard
|
|
4
|
+
# One row per storm count batch that has been reconciled onto ErrorLog rows.
|
|
5
|
+
#
|
|
6
|
+
# Storm counts are applied additively (occurrence_count + N), which is not
|
|
7
|
+
# idempotent, so a replayed batch would count twice. The digest is inserted
|
|
8
|
+
# in the same transaction as the increments: either both land or neither
|
|
9
|
+
# does, and a replay violates the unique index instead of double-counting.
|
|
10
|
+
#
|
|
11
|
+
# Inherits ErrorLogsRecord so separate-database routing applies.
|
|
12
|
+
class StormFlushBatch < ErrorLogsRecord
|
|
13
|
+
self.table_name = "rails_error_dashboard_storm_flush_batches"
|
|
14
|
+
|
|
15
|
+
# The identity of a batch: which fingerprints, how many of each, the
|
|
16
|
+
# overflow bucket, and the episode it belongs to.
|
|
17
|
+
#
|
|
18
|
+
# Sorted, so the same snapshot always digests to the same value whatever
|
|
19
|
+
# order the entries arrive in. Counts are included, so a LATER batch for
|
|
20
|
+
# the same fingerprints is a different identity and is applied normally --
|
|
21
|
+
# only a genuine replay of the same counts is suppressed.
|
|
22
|
+
#
|
|
23
|
+
# @param entries [Array<Hash>] snapshot entries
|
|
24
|
+
# @param overflow [Integer]
|
|
25
|
+
# @param episode [Hash, nil]
|
|
26
|
+
# @return [String] 64-character hex digest
|
|
27
|
+
def self.digest_for(entries:, overflow: 0, episode: nil, batch_id: nil)
|
|
28
|
+
# A batch id minted when the buffer was swapped identifies this batch
|
|
29
|
+
# exactly: a retry carries the same one, and two separate batches never
|
|
30
|
+
# share one. The content digest below is the fallback for a caller that
|
|
31
|
+
# has none (an older in-flight job, or a direct call).
|
|
32
|
+
return Digest::SHA256.hexdigest("batch:#{batch_id}") if batch_id.present?
|
|
33
|
+
|
|
34
|
+
parts = Array(entries).map { |entry|
|
|
35
|
+
entry = entry.with_indifferent_access if entry.respond_to?(:with_indifferent_access)
|
|
36
|
+
next nil unless entry.is_a?(Hash)
|
|
37
|
+
|
|
38
|
+
# gate_key identifies the fingerprint; opaque_identity is carried for
|
|
39
|
+
# grouping and is included so two different errors that somehow shared
|
|
40
|
+
# a gate_key could never collide here.
|
|
41
|
+
[ entry["gate_key"], entry["opaque_identity"], entry["count"].to_i,
|
|
42
|
+
entry["first_seen_at"], entry["last_seen_at"] ].join(":")
|
|
43
|
+
}.compact.sort
|
|
44
|
+
|
|
45
|
+
episode_part = episode.is_a?(Hash) ? episode["started_at"].to_s : ""
|
|
46
|
+
|
|
47
|
+
Digest::SHA256.hexdigest([ parts.join("|"), overflow.to_i, episode_part ].join("#"))
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
@@ -2,13 +2,39 @@
|
|
|
2
2
|
<%# The locale is part of the cache key: this fragment contains translated
|
|
3
3
|
text, and without it the first locale to render an error would be
|
|
4
4
|
served to every other one. %>
|
|
5
|
-
<% cache [error, '
|
|
5
|
+
<% cache [error, 'request_context_v6', red_locale] do %>
|
|
6
6
|
<div class="card mb-4" id="section-request-context">
|
|
7
7
|
<div class="card-header">
|
|
8
8
|
<h5 class="mb-0"><i class="bi bi-globe"></i> <%= red_t("red.errors.request_context.title") %></h5>
|
|
9
9
|
</div>
|
|
10
10
|
<div class="card-body">
|
|
11
11
|
<table class="table table-sm">
|
|
12
|
+
<%# Provenance: which occurrence supplied everything below. The row
|
|
13
|
+
is a group; this evidence is one moment of failure. %>
|
|
14
|
+
<% if error.respond_to?(:context_captured_at) %>
|
|
15
|
+
<tr>
|
|
16
|
+
<th width="200"><%= red_t("red.errors.request_context.snapshot") %></th>
|
|
17
|
+
<td>
|
|
18
|
+
<% if error.context_captured_at.present? %>
|
|
19
|
+
<%= red_t("red.errors.request_context.snapshot_captured_html",
|
|
20
|
+
time: tag.span(error.context_captured_at.utc.iso8601, class: "local-time")) %>
|
|
21
|
+
<% if error.last_seen_at.present? && error.context_captured_at < error.last_seen_at - 1.second %>
|
|
22
|
+
<span class="badge bg-warning text-dark ms-1"><%= red_t("red.errors.request_context.snapshot_stale") %></span>
|
|
23
|
+
<% end %>
|
|
24
|
+
<% else %>
|
|
25
|
+
<span class="text-muted"><%= red_t("red.errors.request_context.snapshot_unknown") %></span>
|
|
26
|
+
<% end %>
|
|
27
|
+
<% if error.respond_to?(:context_fidelity) %>
|
|
28
|
+
<% case error.context_fidelity %>
|
|
29
|
+
<% when "lite" %>
|
|
30
|
+
<div><small class="text-muted"><%= red_t("red.errors.request_context.snapshot_fidelity_lite") %></small></div>
|
|
31
|
+
<% when "minimal" %>
|
|
32
|
+
<div><small class="text-muted"><%= red_t("red.errors.request_context.snapshot_fidelity_minimal") %></small></div>
|
|
33
|
+
<% end %>
|
|
34
|
+
<% end %>
|
|
35
|
+
</td>
|
|
36
|
+
</tr>
|
|
37
|
+
<% end %>
|
|
12
38
|
<tr>
|
|
13
39
|
<th width="200"><%= red_t("red.errors.request_context.url") %></th>
|
|
14
40
|
<td>
|
|
@@ -1,3 +1,9 @@
|
|
|
1
|
+
<%# Rendered live over ActionCable by ErrorBroadcaster.broadcast_stats, which
|
|
2
|
+
names this partial by its FULL STRING PATH -- no `render "stats"` call
|
|
3
|
+
exists, so a search for one finds nothing. It is in scope for any change
|
|
4
|
+
to the stats contract.
|
|
5
|
+
|
|
6
|
+
today / this_week count EVENTS (summed occurrence_count), not groups. %>
|
|
1
7
|
<div class="row g-4">
|
|
2
8
|
<div class="col">
|
|
3
9
|
<div class="card stat-card">
|