error_radar 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +60 -0
- data/app/mailers/error_radar/digest_mailer.rb +31 -0
- data/app/mailers/error_radar/error_mailer.rb +14 -7
- data/app/views/error_radar/digest_mailer/digest.html.erb +168 -0
- data/app/views/error_radar/digest_mailer/digest.text.erb +45 -0
- data/lib/error_radar/configuration.rb +19 -0
- data/lib/error_radar/digest.rb +87 -0
- data/lib/error_radar/notifications/discord.rb +19 -10
- data/lib/error_radar/notifications/email.rb +2 -2
- data/lib/error_radar/notifications/slack.rb +25 -17
- data/lib/error_radar/notifications/webhook.rb +16 -5
- data/lib/error_radar/notifier.rb +70 -32
- data/lib/error_radar/spike_detector.rb +60 -0
- data/lib/error_radar/version.rb +1 -1
- data/lib/error_radar.rb +2 -0
- data/lib/generators/error_radar/install/templates/initializer.rb +21 -3
- data/lib/tasks/error_radar.rake +19 -0
- metadata +6 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e3337550bf775b5f8898cd083bf92b679e913a68166e03064614a16949808bbd
|
|
4
|
+
data.tar.gz: dd37cf03eaa472383509efb76aa77e9760dcbb38a06bf9eb380b73080888671f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 872ec9263dffcdb9c4df391c4259577a0a15805a54cc9e0f8116f0942bee6b05e289b072764e950028dc4fc5dfb786831e864c86d7284f2f88d1b4b71bb20c93
|
|
7
|
+
data.tar.gz: 7e23681c9968c2b2248e3f7ee93e487f2570cce77139acff21e7e1f6f5d2236c58cc09506fff434dd6cec7401cdce5cc26bd9af3321aa5cc43ad3371e12f12b9
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,66 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.9.0] - 2026-07-03
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- **Digest email**: a periodic summary email (HTML + plain text) with:
|
|
9
|
+
- Summary row: new errors, open + in progress, resolved this period, total in DB
|
|
10
|
+
- Unresolved-by-severity bar chart (inline table)
|
|
11
|
+
- Top 10 unresolved errors (linked to detail page when `app_host` is set)
|
|
12
|
+
- New errors first seen in the period
|
|
13
|
+
- Unresolved-by-category breakdown
|
|
14
|
+
- "Open Error Radar →" button
|
|
15
|
+
- **`config.digest_enabled`** (default: `false`) — gates delivery; set `true`
|
|
16
|
+
then schedule the rake task.
|
|
17
|
+
- **`config.digest_recipients`** — separate recipient list for digests; falls
|
|
18
|
+
back to `config.email_recipients` if empty.
|
|
19
|
+
- **`ErrorRadar::Digest.deliver`** — programmatic delivery:
|
|
20
|
+
`ErrorRadar::Digest.deliver(since: 24.hours.ago, period: :daily)`
|
|
21
|
+
- **Rake tasks**:
|
|
22
|
+
- `rake error_radar:digest` — last 24 hours (daily digest)
|
|
23
|
+
- `rake error_radar:digest:weekly` — last 7 days (weekly digest)
|
|
24
|
+
- Both accept `SINCE="2024-01-01 08:00"` env override for custom windows
|
|
25
|
+
- **`DigestMailer`** (`app/mailers/error_radar/digest_mailer.rb`) —
|
|
26
|
+
ActionMailer class with HTML + text templates.
|
|
27
|
+
|
|
28
|
+
### Notes
|
|
29
|
+
- Requires ActionMailer configured in the host app (same requirement as the
|
|
30
|
+
existing `new_error` notification email).
|
|
31
|
+
- Scheduling is left to the host app's cron/scheduler — no in-process scheduler
|
|
32
|
+
dependency. Heroku Scheduler, whenever, clockwork, solid-cron all work.
|
|
33
|
+
- Subject line: `[App] Daily Digest — N new · N open · N resolved`
|
|
34
|
+
|
|
35
|
+
## [0.8.0] - 2026-07-03
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
- **Rate-based spike alerts**: add `:spike` to `config.notify_on` to fire a
|
|
39
|
+
notification when a single error exceeds `config.spike_threshold` (default: 10)
|
|
40
|
+
occurrences within `config.spike_window_minutes` (default: 5). Re-alerts once
|
|
41
|
+
per window after the first trigger.
|
|
42
|
+
- **`ErrorRadar::SpikeDetector`**: detects spikes using two strategies — if
|
|
43
|
+
`track_occurrences` is enabled it queries `error_radar_occurrences` for an
|
|
44
|
+
exact cross-process count; otherwise falls back to an in-memory ring buffer
|
|
45
|
+
(per-process, resets on restart). Memory is capped at 1 000 timestamps per
|
|
46
|
+
fingerprint.
|
|
47
|
+
- **`config.spike_threshold`** (default: 10) and **`config.spike_window_minutes`**
|
|
48
|
+
(default: 5) — tune the spike trigger.
|
|
49
|
+
- **Spike throttle**: a spike alert re-fires at most once per window
|
|
50
|
+
(e.g. every 5 minutes), using a separate throttle key from `:critical`/`:all`.
|
|
51
|
+
- **Event-aware notification channels**: all channels (Slack, Discord, email,
|
|
52
|
+
webhook) now receive the notification event (`:new_error`, `:spike`,
|
|
53
|
+
`:recurring`) and format accordingly:
|
|
54
|
+
- Slack: `:warning:` emoji, "danger" button colour, hit-count in title
|
|
55
|
+
- Discord: orange embed colour (`#ff6600`) for spikes
|
|
56
|
+
- Email subject: `[App] SPIKE ErrorClass: N hits in M min`
|
|
57
|
+
- Webhook payload: `event: "spike"` + `spike: { count:, window_minutes: }`
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
- `Notifier.dispatch` now determines a single *event type* before dispatching,
|
|
61
|
+
preventing a log from matching both `:spike` and `:critical` simultaneously.
|
|
62
|
+
- All channel `deliver` methods accept an optional `event` argument (default
|
|
63
|
+
`:recurring`) — backwards-compatible if called directly.
|
|
64
|
+
|
|
5
65
|
## [0.7.0] - 2026-07-03
|
|
6
66
|
|
|
7
67
|
### Added
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ErrorRadar
|
|
4
|
+
class DigestMailer < ActionMailer::Base
|
|
5
|
+
layout false
|
|
6
|
+
|
|
7
|
+
def digest(data, period: :daily)
|
|
8
|
+
@data = data
|
|
9
|
+
@period = period
|
|
10
|
+
@app_name = ErrorRadar.config.app_name ||
|
|
11
|
+
(defined?(Rails) && Rails.application ? Rails.application.class.module_parent_name : 'App')
|
|
12
|
+
@app_host = ErrorRadar.config.app_host.to_s.chomp('/')
|
|
13
|
+
|
|
14
|
+
period_label = period == :weekly ? 'Weekly' : 'Daily'
|
|
15
|
+
subject = "[#{@app_name}] #{period_label} Digest — " \
|
|
16
|
+
"#{data[:new_this_period]} new · " \
|
|
17
|
+
"#{data[:unresolved]} open · " \
|
|
18
|
+
"#{data[:resolved_this_period]} resolved"
|
|
19
|
+
|
|
20
|
+
recipients = ErrorRadar.config.digest_recipients.any? ?
|
|
21
|
+
ErrorRadar.config.digest_recipients :
|
|
22
|
+
ErrorRadar.config.email_recipients
|
|
23
|
+
|
|
24
|
+
mail(
|
|
25
|
+
to: recipients,
|
|
26
|
+
from: ErrorRadar.config.email_from || 'noreply@localhost',
|
|
27
|
+
subject: subject
|
|
28
|
+
)
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
|
@@ -4,14 +4,21 @@ module ErrorRadar
|
|
|
4
4
|
class ErrorMailer < ActionMailer::Base
|
|
5
5
|
layout false
|
|
6
6
|
|
|
7
|
-
def new_error(error_log)
|
|
8
|
-
@error
|
|
9
|
-
@
|
|
10
|
-
@
|
|
11
|
-
@
|
|
7
|
+
def new_error(error_log, event = :recurring)
|
|
8
|
+
@error = error_log
|
|
9
|
+
@event = event
|
|
10
|
+
@spike_data = error_log.instance_variable_get(:@spike_data)
|
|
11
|
+
@url = build_url
|
|
12
|
+
@app_name = ErrorRadar::Notifier.app_name
|
|
12
13
|
|
|
13
|
-
|
|
14
|
-
|
|
14
|
+
subject = case event
|
|
15
|
+
when :spike
|
|
16
|
+
"[#{@app_name}] SPIKE #{@error.error_class}: #{@spike_data[:count]} hits in #{@spike_data[:window_minutes]} min"
|
|
17
|
+
when :new_error
|
|
18
|
+
"[#{@app_name}] New #{@error.severity}: #{@error.error_class}"
|
|
19
|
+
else
|
|
20
|
+
"[#{@app_name}] #{@error.severity.capitalize}: #{@error.error_class}"
|
|
21
|
+
end
|
|
15
22
|
|
|
16
23
|
mail(
|
|
17
24
|
to: ErrorRadar.config.email_recipients,
|
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
<!DOCTYPE html>
|
|
2
|
+
<html lang="en">
|
|
3
|
+
<head>
|
|
4
|
+
<meta charset="utf-8">
|
|
5
|
+
<meta name="viewport" content="width=device-width,initial-scale=1">
|
|
6
|
+
<%
|
|
7
|
+
sev_colors = { 'critical' => '#7b001c', 'error' => '#dc3545', 'warning' => '#fd7e14', 'info' => '#17a2b8' }
|
|
8
|
+
period_label = @period == :weekly ? 'Weekly' : 'Daily'
|
|
9
|
+
fmt = ->(t) { t&.strftime('%Y-%m-%d %H:%M') }
|
|
10
|
+
%>
|
|
11
|
+
</head>
|
|
12
|
+
<body style="margin:0;padding:24px 8px;background:#f3f4f6;font-family:-apple-system,'Segoe UI',Roboto,Arial,sans-serif;color:#1f2937">
|
|
13
|
+
<div style="max-width:640px;margin:0 auto">
|
|
14
|
+
|
|
15
|
+
<%# ── Header ── %>
|
|
16
|
+
<div style="background:#1f2937;color:#fff;padding:20px 24px;border-radius:10px 10px 0 0">
|
|
17
|
+
<h1 style="margin:0 0 4px;font-size:18px">📊 <%= @app_name %> — <%= period_label %> Error Digest</h1>
|
|
18
|
+
<div style="font-size:12px;color:#9ca3af">
|
|
19
|
+
<%= fmt.call(@data[:period_start]) %> → <%= fmt.call(@data[:period_end]) %>
|
|
20
|
+
</div>
|
|
21
|
+
</div>
|
|
22
|
+
|
|
23
|
+
<%# ── Summary cards ── %>
|
|
24
|
+
<div style="background:#fff;padding:20px 24px">
|
|
25
|
+
<table width="100%" cellpadding="0" cellspacing="0" style="margin-bottom:24px">
|
|
26
|
+
<tr>
|
|
27
|
+
<td width="25%" style="padding:4px">
|
|
28
|
+
<div style="background:#fef3c7;border-radius:8px;padding:14px 10px;text-align:center">
|
|
29
|
+
<div style="font-size:26px;font-weight:700;color:#92400e"><%= @data[:new_this_period] %></div>
|
|
30
|
+
<div style="font-size:11px;color:#78350f;margin-top:2px">New errors</div>
|
|
31
|
+
</div>
|
|
32
|
+
</td>
|
|
33
|
+
<td width="25%" style="padding:4px">
|
|
34
|
+
<div style="background:#fee2e2;border-radius:8px;padding:14px 10px;text-align:center">
|
|
35
|
+
<div style="font-size:26px;font-weight:700;color:#991b1b"><%= @data[:unresolved] %></div>
|
|
36
|
+
<div style="font-size:11px;color:#7f1d1d;margin-top:2px">Open + in progress</div>
|
|
37
|
+
</div>
|
|
38
|
+
</td>
|
|
39
|
+
<td width="25%" style="padding:4px">
|
|
40
|
+
<div style="background:#d1fae5;border-radius:8px;padding:14px 10px;text-align:center">
|
|
41
|
+
<div style="font-size:26px;font-weight:700;color:#065f46"><%= @data[:resolved_this_period] %></div>
|
|
42
|
+
<div style="font-size:11px;color:#064e3b;margin-top:2px">Resolved</div>
|
|
43
|
+
</div>
|
|
44
|
+
</td>
|
|
45
|
+
<td width="25%" style="padding:4px">
|
|
46
|
+
<div style="background:#f3f4f6;border-radius:8px;padding:14px 10px;text-align:center">
|
|
47
|
+
<div style="font-size:26px;font-weight:700;color:#374151"><%= @data[:total] %></div>
|
|
48
|
+
<div style="font-size:11px;color:#6b7280;margin-top:2px">Total in DB</div>
|
|
49
|
+
</div>
|
|
50
|
+
</td>
|
|
51
|
+
</tr>
|
|
52
|
+
</table>
|
|
53
|
+
|
|
54
|
+
<%# ── Severity breakdown ── %>
|
|
55
|
+
<% if @data[:by_severity].any? %>
|
|
56
|
+
<div style="margin-bottom:24px">
|
|
57
|
+
<h2 style="margin:0 0 10px;font-size:14px;font-weight:600;color:#374151;text-transform:uppercase;letter-spacing:.05em">Unresolved by severity</h2>
|
|
58
|
+
<table width="100%" cellpadding="0" cellspacing="0">
|
|
59
|
+
<% %w[critical error warning info].each do |sev| %>
|
|
60
|
+
<% count = @data[:by_severity][sev].to_i %>
|
|
61
|
+
<% next if count.zero? %>
|
|
62
|
+
<% pct = (@data[:unresolved] > 0 ? count.to_f / @data[:unresolved] * 100 : 0).round %>
|
|
63
|
+
<tr style="margin-bottom:6px">
|
|
64
|
+
<td width="80" style="padding:3px 0">
|
|
65
|
+
<span style="display:inline-block;padding:2px 10px;border-radius:10px;color:#fff;font-size:11px;font-weight:600;background:<%= sev_colors[sev] || '#6c757d' %>"><%= sev %></span>
|
|
66
|
+
</td>
|
|
67
|
+
<td style="padding:3px 8px">
|
|
68
|
+
<div style="background:#e5e7eb;border-radius:4px;height:10px;overflow:hidden">
|
|
69
|
+
<div style="background:<%= sev_colors[sev] || '#6c757d' %>;height:10px;width:<%= pct %>%"></div>
|
|
70
|
+
</div>
|
|
71
|
+
</td>
|
|
72
|
+
<td width="40" style="padding:3px 0;text-align:right;font-size:13px;font-weight:600"><%= count %></td>
|
|
73
|
+
</tr>
|
|
74
|
+
<% end %>
|
|
75
|
+
</table>
|
|
76
|
+
</div>
|
|
77
|
+
<% end %>
|
|
78
|
+
|
|
79
|
+
<%# ── Top unresolved ── %>
|
|
80
|
+
<% if @data[:top_unresolved].any? %>
|
|
81
|
+
<h2 style="margin:0 0 10px;font-size:14px;font-weight:600;color:#374151;text-transform:uppercase;letter-spacing:.05em">Top 10 unresolved (by occurrences)</h2>
|
|
82
|
+
<table width="100%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;margin-bottom:24px">
|
|
83
|
+
<thead>
|
|
84
|
+
<tr style="background:#f9fafb;border-bottom:2px solid #e5e7eb">
|
|
85
|
+
<th style="text-align:left;padding:7px 8px;font-size:11px;color:#6b7280;font-weight:600">Error</th>
|
|
86
|
+
<th style="text-align:left;padding:7px 8px;font-size:11px;color:#6b7280;font-weight:600;width:70px">Severity</th>
|
|
87
|
+
<th style="text-align:right;padding:7px 8px;font-size:11px;color:#6b7280;font-weight:600;width:60px">Hits</th>
|
|
88
|
+
<th style="text-align:left;padding:7px 8px;font-size:11px;color:#6b7280;font-weight:600;width:100px">Last seen</th>
|
|
89
|
+
</tr>
|
|
90
|
+
</thead>
|
|
91
|
+
<tbody>
|
|
92
|
+
<% @data[:top_unresolved].each do |e| %>
|
|
93
|
+
<% url = @app_host.present? ? "#{@app_host}/error_radar/errors/#{e.id}" : nil %>
|
|
94
|
+
<tr style="border-bottom:1px solid #f3f4f6">
|
|
95
|
+
<td style="padding:8px 8px">
|
|
96
|
+
<div style="font-size:12px;font-family:monospace;font-weight:600;color:#1f2937">
|
|
97
|
+
<% if url %><a href="<%= url %>" style="color:#2563eb;text-decoration:none"><%= e.error_class %></a><% else %><%= e.error_class %><% end %>
|
|
98
|
+
</div>
|
|
99
|
+
<div style="font-size:11px;color:#6b7280;margin-top:2px"><%= e.short_message.truncate(80) %></div>
|
|
100
|
+
</td>
|
|
101
|
+
<td style="padding:8px 8px">
|
|
102
|
+
<span style="display:inline-block;padding:2px 8px;border-radius:8px;color:#fff;font-size:10px;font-weight:600;background:<%= sev_colors[e.severity] || '#6c757d' %>"><%= e.severity %></span>
|
|
103
|
+
</td>
|
|
104
|
+
<td style="padding:8px 8px;text-align:right;font-size:13px;font-weight:700;color:#1f2937"><%= e.occurrences %></td>
|
|
105
|
+
<td style="padding:8px 8px;font-size:11px;color:#6b7280"><%= e.last_seen_at&.strftime('%m/%d %H:%M') %></td>
|
|
106
|
+
</tr>
|
|
107
|
+
<% end %>
|
|
108
|
+
</tbody>
|
|
109
|
+
</table>
|
|
110
|
+
<% end %>
|
|
111
|
+
|
|
112
|
+
<%# ── New errors this period ── %>
|
|
113
|
+
<% if @data[:recent_new].any? %>
|
|
114
|
+
<h2 style="margin:0 0 10px;font-size:14px;font-weight:600;color:#374151;text-transform:uppercase;letter-spacing:.05em">🆕 New this period (<%= @data[:new_this_period] %> total)</h2>
|
|
115
|
+
<table width="100%" cellpadding="0" cellspacing="0" style="border-collapse:collapse;margin-bottom:24px">
|
|
116
|
+
<% @data[:recent_new].each do |e| %>
|
|
117
|
+
<% url = @app_host.present? ? "#{@app_host}/error_radar/errors/#{e.id}" : nil %>
|
|
118
|
+
<tr style="border-bottom:1px solid #f3f4f6">
|
|
119
|
+
<td style="padding:7px 8px">
|
|
120
|
+
<span style="display:inline-block;padding:2px 8px;border-radius:8px;color:#fff;font-size:10px;font-weight:600;background:<%= sev_colors[e.severity] || '#6c757d' %>"><%= e.severity %></span>
|
|
121
|
+
</td>
|
|
122
|
+
<td style="padding:7px 8px">
|
|
123
|
+
<div style="font-size:12px;font-family:monospace;font-weight:600">
|
|
124
|
+
<% if url %><a href="<%= url %>" style="color:#2563eb;text-decoration:none"><%= e.error_class %></a><% else %><%= e.error_class %><% end %>
|
|
125
|
+
</div>
|
|
126
|
+
<div style="font-size:11px;color:#6b7280"><%= e.short_message.truncate(90) %></div>
|
|
127
|
+
</td>
|
|
128
|
+
<td style="padding:7px 8px;font-size:11px;color:#6b7280;white-space:nowrap"><%= fmt.call(e.first_seen_at) %></td>
|
|
129
|
+
</tr>
|
|
130
|
+
<% end %>
|
|
131
|
+
</table>
|
|
132
|
+
<% end %>
|
|
133
|
+
|
|
134
|
+
<%# ── Category breakdown ── %>
|
|
135
|
+
<% if @data[:by_category].any? %>
|
|
136
|
+
<h2 style="margin:0 0 10px;font-size:14px;font-weight:600;color:#374151;text-transform:uppercase;letter-spacing:.05em">Unresolved by category</h2>
|
|
137
|
+
<table width="100%" cellpadding="0" cellspacing="0" style="margin-bottom:20px">
|
|
138
|
+
<% @data[:by_category].sort_by { |_, v| -v }.each do |cat, count| %>
|
|
139
|
+
<tr>
|
|
140
|
+
<td width="140" style="padding:3px 0;font-size:12px;color:#374151"><%= cat.to_s.humanize %></td>
|
|
141
|
+
<td style="padding:3px 8px">
|
|
142
|
+
<div style="background:#e5e7eb;border-radius:4px;height:8px;overflow:hidden">
|
|
143
|
+
<% pct = (@data[:unresolved] > 0 ? count.to_f / @data[:unresolved] * 100 : 0).round %>
|
|
144
|
+
<div style="background:#2563eb;height:8px;width:<%= pct %>%"></div>
|
|
145
|
+
</div>
|
|
146
|
+
</td>
|
|
147
|
+
<td width="36" style="padding:3px 0;text-align:right;font-size:12px;font-weight:600;color:#374151"><%= count %></td>
|
|
148
|
+
</tr>
|
|
149
|
+
<% end %>
|
|
150
|
+
</table>
|
|
151
|
+
<% end %>
|
|
152
|
+
|
|
153
|
+
<% if @app_host.present? %>
|
|
154
|
+
<div style="text-align:center;margin-top:16px">
|
|
155
|
+
<a href="<%= @app_host %>/error_radar" style="display:inline-block;background:#2563eb;color:#fff;padding:11px 24px;border-radius:8px;text-decoration:none;font-size:14px;font-weight:600">Open Error Radar →</a>
|
|
156
|
+
</div>
|
|
157
|
+
<% end %>
|
|
158
|
+
</div>
|
|
159
|
+
|
|
160
|
+
<%# ── Footer ── %>
|
|
161
|
+
<div style="text-align:center;padding:16px 0;font-size:11px;color:#9ca3af">
|
|
162
|
+
Sent by <strong>Error Radar</strong> · <%= @app_name %> ·
|
|
163
|
+
Add <code>rake error_radar:digest</code> to your cron schedule.
|
|
164
|
+
</div>
|
|
165
|
+
|
|
166
|
+
</div>
|
|
167
|
+
</body>
|
|
168
|
+
</html>
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
<%= @period == :weekly ? 'Weekly' : 'Daily' %> Error Digest — <%= @app_name %>
|
|
2
|
+
<%= '=' * 60 %>
|
|
3
|
+
Period: <%= @data[:period_start]&.strftime('%Y-%m-%d %H:%M') %> → <%= @data[:period_end]&.strftime('%Y-%m-%d %H:%M') %>
|
|
4
|
+
|
|
5
|
+
SUMMARY
|
|
6
|
+
New errors this period : <%= @data[:new_this_period] %>
|
|
7
|
+
Resolved this period : <%= @data[:resolved_this_period] %>
|
|
8
|
+
Open + in progress : <%= @data[:unresolved] %>
|
|
9
|
+
Total in DB : <%= @data[:total] %>
|
|
10
|
+
|
|
11
|
+
UNRESOLVED BY SEVERITY
|
|
12
|
+
<% %w[critical error warning info].each do |sev| %>
|
|
13
|
+
<% count = @data[:by_severity][sev].to_i %><% next if count.zero? %>
|
|
14
|
+
<%= sev.ljust(10) %> <%= count %>
|
|
15
|
+
<% end %>
|
|
16
|
+
|
|
17
|
+
TOP 10 UNRESOLVED (by occurrences)
|
|
18
|
+
<% @data[:top_unresolved].each_with_index do |e, i| %>
|
|
19
|
+
<%= (i + 1).to_s.rjust(2) %>. [<%= e.severity.upcase %>] <%= e.error_class %> (×<%= e.occurrences %>)
|
|
20
|
+
<%= e.short_message.truncate(100) %>
|
|
21
|
+
Last seen: <%= e.last_seen_at&.strftime('%Y-%m-%d %H:%M') %>
|
|
22
|
+
<% if @app_host.present? %>
|
|
23
|
+
<%= @app_host %>/error_radar/errors/<%= e.id %>
|
|
24
|
+
<% end %>
|
|
25
|
+
<% end %>
|
|
26
|
+
|
|
27
|
+
<% if @data[:recent_new].any? %>
|
|
28
|
+
NEW ERRORS THIS PERIOD (<%= @data[:new_this_period] %> total, showing <%= [@data[:recent_new].size, 10].min %>)
|
|
29
|
+
<% @data[:recent_new].each do |e| %>
|
|
30
|
+
· [<%= e.severity.upcase %>] <%= e.error_class %> — <%= e.first_seen_at&.strftime('%Y-%m-%d %H:%M') %>
|
|
31
|
+
<%= e.short_message.truncate(100) %>
|
|
32
|
+
<% end %>
|
|
33
|
+
<% end %>
|
|
34
|
+
|
|
35
|
+
UNRESOLVED BY CATEGORY
|
|
36
|
+
<% @data[:by_category].sort_by { |_, v| -v }.each do |cat, count| %>
|
|
37
|
+
<%= cat.to_s.humanize.ljust(20) %> <%= count %>
|
|
38
|
+
<% end %>
|
|
39
|
+
|
|
40
|
+
<% if @app_host.present? %>
|
|
41
|
+
Open dashboard: <%= @app_host %>/error_radar
|
|
42
|
+
<% end %>
|
|
43
|
+
|
|
44
|
+
--
|
|
45
|
+
Sent by Error Radar · Add `rake error_radar:digest` to your cron schedule.
|
|
@@ -69,6 +69,19 @@ module ErrorRadar
|
|
|
69
69
|
@error_callbacks << block
|
|
70
70
|
end
|
|
71
71
|
|
|
72
|
+
# Digest email ────────────────────────────────────────────────────────────
|
|
73
|
+
# Set to true to enable digest delivery (rake error_radar:digest).
|
|
74
|
+
attr_accessor :digest_enabled
|
|
75
|
+
# Separate recipient list for digests. Falls back to email_recipients if empty.
|
|
76
|
+
attr_accessor :digest_recipients
|
|
77
|
+
|
|
78
|
+
# Spike detection ─────────────────────────────────────────────────────────
|
|
79
|
+
# Number of occurrences within spike_window_minutes that triggers a spike alert.
|
|
80
|
+
# Add :spike to notify_on to enable: config.notify_on = [:new_error, :spike]
|
|
81
|
+
attr_accessor :spike_threshold
|
|
82
|
+
# Rolling time window in minutes for counting occurrences.
|
|
83
|
+
attr_accessor :spike_window_minutes
|
|
84
|
+
|
|
72
85
|
# Occurrences ─────────────────────────────────────────────────────────────
|
|
73
86
|
# When true, each individual error hit is stored in error_radar_occurrences.
|
|
74
87
|
# Requires running: bin/rails generate error_radar:upgrade_v060 && bin/rails db:migrate
|
|
@@ -156,6 +169,12 @@ module ErrorRadar
|
|
|
156
169
|
@app_host = nil
|
|
157
170
|
@error_callbacks = []
|
|
158
171
|
|
|
172
|
+
@digest_enabled = false
|
|
173
|
+
@digest_recipients = []
|
|
174
|
+
|
|
175
|
+
@spike_threshold = 10
|
|
176
|
+
@spike_window_minutes = 5
|
|
177
|
+
|
|
159
178
|
@track_occurrences = false
|
|
160
179
|
@max_occurrences_per_error = 200
|
|
161
180
|
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ErrorRadar
|
|
4
|
+
# Gathers error statistics for a time window and delivers a digest email.
|
|
5
|
+
# Intended to be called from a cron job or scheduled task:
|
|
6
|
+
#
|
|
7
|
+
# # Daily (runs every morning via cron / Heroku Scheduler):
|
|
8
|
+
# rake error_radar:digest
|
|
9
|
+
#
|
|
10
|
+
# # Weekly:
|
|
11
|
+
# rake error_radar:digest:weekly
|
|
12
|
+
#
|
|
13
|
+
# # From code:
|
|
14
|
+
# ErrorRadar::Digest.deliver(since: 24.hours.ago, period: :daily)
|
|
15
|
+
module Digest
|
|
16
|
+
def self.deliver(since: nil, period: :daily)
|
|
17
|
+
return unless ErrorRadar.config.digest_enabled
|
|
18
|
+
|
|
19
|
+
since ||= period == :weekly ? 7.days.ago : 24.hours.ago
|
|
20
|
+
data = build(since: since)
|
|
21
|
+
|
|
22
|
+
require 'error_radar/mailers/digest_mailer'
|
|
23
|
+
DigestMailer.digest(data, period: period).deliver_now
|
|
24
|
+
rescue StandardError => e
|
|
25
|
+
ErrorRadar::Tracking.warn_internal("Digest.deliver failed: #{e.class}: #{e.message}")
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def self.build(since:)
|
|
29
|
+
now = Time.current
|
|
30
|
+
counts = ErrorLog.group(:status).count
|
|
31
|
+
inv = ErrorLog.statuses.invert
|
|
32
|
+
|
|
33
|
+
open_count = counts[ErrorLog.statuses['open']] || 0
|
|
34
|
+
in_prog_count = counts[ErrorLog.statuses['in_progress']] || 0
|
|
35
|
+
resolved = counts[ErrorLog.statuses['resolved']] || 0
|
|
36
|
+
ignored = counts[ErrorLog.statuses['ignored']] || 0
|
|
37
|
+
total = counts.values.sum
|
|
38
|
+
unresolved = open_count + in_prog_count
|
|
39
|
+
|
|
40
|
+
new_this_period = ErrorLog.where('first_seen_at >= ?', since).count
|
|
41
|
+
resolved_this_period = ErrorLog.where('resolved_at >= ?', since).count
|
|
42
|
+
reopened_this_period = ErrorLog.unresolved.where('resolved_at < ?', since)
|
|
43
|
+
.where('last_seen_at >= ?', since).count
|
|
44
|
+
|
|
45
|
+
top_unresolved = ErrorLog.unresolved.order(occurrences: :desc).limit(10).to_a
|
|
46
|
+
recent_new = ErrorLog.where('first_seen_at >= ?', since)
|
|
47
|
+
.order(first_seen_at: :desc).limit(10).to_a
|
|
48
|
+
|
|
49
|
+
by_severity = ErrorLog.unresolved.group(:severity).count
|
|
50
|
+
.transform_keys { |k| ErrorLog.severities.invert[k] || k.to_s }
|
|
51
|
+
by_category = ErrorLog.unresolved.group(:category).count
|
|
52
|
+
.transform_keys { |k| ErrorLog.categories.invert[k] || k.to_s }
|
|
53
|
+
|
|
54
|
+
trend = build_trend(since, now)
|
|
55
|
+
|
|
56
|
+
{
|
|
57
|
+
period_start: since,
|
|
58
|
+
period_end: now,
|
|
59
|
+
total: total,
|
|
60
|
+
open: open_count,
|
|
61
|
+
in_progress: in_prog_count,
|
|
62
|
+
unresolved: unresolved,
|
|
63
|
+
resolved_total: resolved,
|
|
64
|
+
ignored_total: ignored,
|
|
65
|
+
new_this_period: new_this_period,
|
|
66
|
+
resolved_this_period: resolved_this_period,
|
|
67
|
+
reopened_this_period: reopened_this_period,
|
|
68
|
+
top_unresolved: top_unresolved,
|
|
69
|
+
recent_new: recent_new,
|
|
70
|
+
by_severity: by_severity,
|
|
71
|
+
by_category: by_category,
|
|
72
|
+
trend: trend
|
|
73
|
+
}
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
def self.build_trend(since, now)
|
|
77
|
+
days = [((now - since) / 86_400).ceil, 30].min
|
|
78
|
+
ErrorLog.where(last_seen_at: days.days.ago..)
|
|
79
|
+
.pluck(:last_seen_at)
|
|
80
|
+
.group_by { |t| t.to_date }
|
|
81
|
+
.transform_values(&:size)
|
|
82
|
+
.sort.to_h
|
|
83
|
+
.transform_keys { |d| d.strftime('%m/%d') }
|
|
84
|
+
end
|
|
85
|
+
private_class_method :build_trend
|
|
86
|
+
end
|
|
87
|
+
end
|
|
@@ -13,10 +13,10 @@ module ErrorRadar
|
|
|
13
13
|
'warning' => 0xfd7e14,
|
|
14
14
|
'info' => 0x17a2b8
|
|
15
15
|
}.freeze
|
|
16
|
+
SPIKE_COLOR = 0xff6600
|
|
16
17
|
|
|
17
|
-
def self.deliver(log)
|
|
18
|
-
url
|
|
19
|
-
payload = build_payload(log)
|
|
18
|
+
def self.deliver(log, event = :recurring)
|
|
19
|
+
url = URI(ErrorRadar.config.discord_webhook_url)
|
|
20
20
|
|
|
21
21
|
http = Net::HTTP.new(url.host, url.port)
|
|
22
22
|
http.use_ssl = url.scheme == 'https'
|
|
@@ -25,17 +25,26 @@ module ErrorRadar
|
|
|
25
25
|
|
|
26
26
|
req = Net::HTTP::Post.new(url)
|
|
27
27
|
req['Content-Type'] = 'application/json'
|
|
28
|
-
req.body =
|
|
28
|
+
req.body = build_payload(log, event).to_json
|
|
29
29
|
|
|
30
30
|
http.request(req)
|
|
31
31
|
rescue StandardError => e
|
|
32
32
|
ErrorRadar::Tracking.warn_internal("Discord notification failed: #{e.message}")
|
|
33
33
|
end
|
|
34
34
|
|
|
35
|
-
def self.build_payload(log)
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
35
|
+
def self.build_payload(log, event)
|
|
36
|
+
spike_data = log.instance_variable_get(:@spike_data)
|
|
37
|
+
app = ErrorRadar::Notifier.app_name
|
|
38
|
+
link = ErrorRadar::Notifier.error_url(log)
|
|
39
|
+
|
|
40
|
+
prefix, color = case event
|
|
41
|
+
when :spike
|
|
42
|
+
["⚠️ Spike (#{spike_data[:count]} hits / #{spike_data[:window_minutes]} min)", SPIKE_COLOR]
|
|
43
|
+
when :new_error
|
|
44
|
+
['🆕 New error', SEV_COLORS[log.severity] || 0x6c757d]
|
|
45
|
+
else
|
|
46
|
+
['🔁 Recurring error', SEV_COLORS[log.severity] || 0x6c757d]
|
|
47
|
+
end
|
|
39
48
|
|
|
40
49
|
fields = [
|
|
41
50
|
{ name: 'Source', value: (log.source || 'unknown').truncate(100), inline: true },
|
|
@@ -43,12 +52,12 @@ module ErrorRadar
|
|
|
43
52
|
{ name: 'Severity', value: log.severity.to_s, inline: true },
|
|
44
53
|
{ name: 'Occurrences', value: log.occurrences.to_s, inline: true }
|
|
45
54
|
]
|
|
46
|
-
fields << { name: 'View', value: "[Error Radar](#{
|
|
55
|
+
fields << { name: 'View', value: "[Error Radar](#{link})", inline: false } if link
|
|
47
56
|
|
|
48
57
|
embed = {
|
|
49
58
|
title: "#{prefix}: #{log.error_class}",
|
|
50
59
|
description: "```#{log.message.to_s.truncate(300)}```",
|
|
51
|
-
color:
|
|
60
|
+
color: color,
|
|
52
61
|
fields: fields,
|
|
53
62
|
footer: { text: "[#{app}] Error Radar" },
|
|
54
63
|
timestamp: log.last_seen_at&.iso8601
|
|
@@ -5,11 +5,11 @@ module ErrorRadar
|
|
|
5
5
|
# Delivers notification emails via ActionMailer (must be configured in
|
|
6
6
|
# the host app). Uses deliver_later so it doesn't block the request cycle.
|
|
7
7
|
module Email
|
|
8
|
-
def self.deliver(log)
|
|
8
|
+
def self.deliver(log, event = :recurring)
|
|
9
9
|
return unless defined?(::ActionMailer::Base)
|
|
10
10
|
|
|
11
11
|
require 'error_radar/mailers/error_mailer'
|
|
12
|
-
ErrorRadar::ErrorMailer.new_error(log).deliver_later
|
|
12
|
+
ErrorRadar::ErrorMailer.new_error(log, event).deliver_later
|
|
13
13
|
rescue StandardError => e
|
|
14
14
|
ErrorRadar::Tracking.warn_internal("Email notification failed: #{e.message}")
|
|
15
15
|
end
|
|
@@ -14,40 +14,47 @@ module ErrorRadar
|
|
|
14
14
|
'info' => ':large_blue_circle:'
|
|
15
15
|
}.freeze
|
|
16
16
|
|
|
17
|
-
def self.deliver(log)
|
|
18
|
-
url
|
|
19
|
-
payload = build_payload(log)
|
|
17
|
+
def self.deliver(log, event = :recurring)
|
|
18
|
+
url = URI(ErrorRadar.config.slack_webhook_url)
|
|
20
19
|
|
|
21
20
|
http = Net::HTTP.new(url.host, url.port)
|
|
22
21
|
http.use_ssl = url.scheme == 'https'
|
|
23
22
|
http.open_timeout = 5
|
|
24
23
|
http.read_timeout = 5
|
|
25
24
|
|
|
26
|
-
req
|
|
27
|
-
req['Content-Type']
|
|
28
|
-
req.body
|
|
25
|
+
req = Net::HTTP::Post.new(url)
|
|
26
|
+
req['Content-Type'] = 'application/json'
|
|
27
|
+
req.body = build_payload(log, event).to_json
|
|
29
28
|
|
|
30
29
|
http.request(req)
|
|
31
30
|
rescue StandardError => e
|
|
32
31
|
ErrorRadar::Tracking.warn_internal("Slack notification failed: #{e.message}")
|
|
33
32
|
end
|
|
34
33
|
|
|
35
|
-
def self.build_payload(log)
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
34
|
+
def self.build_payload(log, event)
|
|
35
|
+
spike_data = log.instance_variable_get(:@spike_data)
|
|
36
|
+
app = ErrorRadar::Notifier.app_name
|
|
37
|
+
link = ErrorRadar::Notifier.error_url(log)
|
|
38
|
+
channel = ErrorRadar.config.slack_channel
|
|
39
|
+
|
|
40
|
+
emoji, title = case event
|
|
41
|
+
when :spike
|
|
42
|
+
[':warning:', "Spike — #{spike_data[:count]} hits in #{spike_data[:window_minutes]} min: *#{log.error_class}*"]
|
|
43
|
+
when :new_error
|
|
44
|
+
[SEV_EMOJI[log.severity] || ':white_circle:', "New error: *#{log.error_class}*"]
|
|
45
|
+
else
|
|
46
|
+
[SEV_EMOJI[log.severity] || ':white_circle:', "#{log.severity.capitalize} error: *#{log.error_class}*"]
|
|
47
|
+
end
|
|
41
48
|
|
|
42
49
|
blocks = [
|
|
43
|
-
{ type: 'section', text: { type: 'mrkdwn', text: "*[#{app}]* #{title}" } },
|
|
50
|
+
{ type: 'section', text: { type: 'mrkdwn', text: "#{emoji} *[#{app}]* #{title}" } },
|
|
44
51
|
{
|
|
45
52
|
type: 'section',
|
|
46
53
|
fields: [
|
|
47
54
|
{ type: 'mrkdwn', text: "*Source*\n#{(log.source || 'unknown').truncate(80)}" },
|
|
48
55
|
{ type: 'mrkdwn', text: "*Category*\n#{log.category}" },
|
|
49
56
|
{ type: 'mrkdwn', text: "*Severity*\n#{log.severity}" },
|
|
50
|
-
{ type: 'mrkdwn', text: "*
|
|
57
|
+
{ type: 'mrkdwn', text: "*Total occurrences*\n#{log.occurrences}" }
|
|
51
58
|
]
|
|
52
59
|
},
|
|
53
60
|
{
|
|
@@ -56,13 +63,14 @@ module ErrorRadar
|
|
|
56
63
|
}
|
|
57
64
|
]
|
|
58
65
|
|
|
59
|
-
if
|
|
66
|
+
if link
|
|
60
67
|
blocks << {
|
|
61
68
|
type: 'actions',
|
|
62
69
|
elements: [{
|
|
63
70
|
type: 'button',
|
|
64
|
-
text:
|
|
65
|
-
url:
|
|
71
|
+
text: { type: 'plain_text', text: 'View in Error Radar', emoji: true },
|
|
72
|
+
url: link,
|
|
73
|
+
style: event == :spike ? 'danger' : 'primary'
|
|
66
74
|
}]
|
|
67
75
|
}
|
|
68
76
|
end
|
|
@@ -9,7 +9,7 @@ module ErrorRadar
|
|
|
9
9
|
# Generic outbound webhook. POSTs a JSON payload to any URL.
|
|
10
10
|
# Useful for PagerDuty, OpsGenie, custom scripts, etc.
|
|
11
11
|
module Webhook
|
|
12
|
-
def self.deliver(log, url:)
|
|
12
|
+
def self.deliver(log, url:, event: :recurring)
|
|
13
13
|
uri = URI(url)
|
|
14
14
|
|
|
15
15
|
http = Net::HTTP.new(uri.host, uri.port)
|
|
@@ -20,16 +20,18 @@ module ErrorRadar
|
|
|
20
20
|
req = Net::HTTP::Post.new(uri)
|
|
21
21
|
req['Content-Type'] = 'application/json'
|
|
22
22
|
req['User-Agent'] = "ErrorRadar/#{ErrorRadar::VERSION}"
|
|
23
|
-
req.body = build_payload(log).to_json
|
|
23
|
+
req.body = build_payload(log, event).to_json
|
|
24
24
|
|
|
25
25
|
http.request(req)
|
|
26
26
|
rescue StandardError => e
|
|
27
27
|
ErrorRadar::Tracking.warn_internal("Webhook (#{url}) notification failed: #{e.message}")
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
-
def self.build_payload(log)
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
def self.build_payload(log, event)
|
|
31
|
+
spike_data = log.instance_variable_get(:@spike_data)
|
|
32
|
+
|
|
33
|
+
payload = {
|
|
34
|
+
event: event.to_s,
|
|
33
35
|
app: ErrorRadar::Notifier.app_name,
|
|
34
36
|
error_class: log.error_class,
|
|
35
37
|
source: log.source,
|
|
@@ -42,6 +44,15 @@ module ErrorRadar
|
|
|
42
44
|
last_seen_at: log.last_seen_at&.iso8601,
|
|
43
45
|
url: ErrorRadar::Notifier.error_url(log)
|
|
44
46
|
}
|
|
47
|
+
|
|
48
|
+
if spike_data
|
|
49
|
+
payload[:spike] = {
|
|
50
|
+
count: spike_data[:count],
|
|
51
|
+
window_minutes: spike_data[:window_minutes]
|
|
52
|
+
}
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
payload
|
|
45
56
|
end
|
|
46
57
|
end
|
|
47
58
|
end
|
data/lib/error_radar/notifier.rb
CHANGED
|
@@ -5,68 +5,105 @@ module ErrorRadar
|
|
|
5
5
|
# callbacks) after an ErrorLog is persisted. Called from Tracking.capture and
|
|
6
6
|
# Tracking.notify. Never raises — a broken notification must not affect the
|
|
7
7
|
# host application.
|
|
8
|
+
#
|
|
9
|
+
# Notification events:
|
|
10
|
+
# :new_error — first time a fingerprint is seen (no throttle)
|
|
11
|
+
# :spike — error rate exceeds config.spike_threshold in the window
|
|
12
|
+
# :recurring — matches :critical or :all rule, throttled to 1/hour
|
|
8
13
|
module Notifier
|
|
9
|
-
THROTTLE_MUTEX
|
|
10
|
-
#
|
|
11
|
-
|
|
14
|
+
THROTTLE_MUTEX = Mutex.new
|
|
15
|
+
# key => Time of last notification (in-memory, resets on restart)
|
|
16
|
+
# Keys: fingerprint (for :recurring) or "spike:fingerprint" (for :spike)
|
|
17
|
+
THROTTLE = {}
|
|
12
18
|
THROTTLE_INTERVAL = 3600 # seconds between repeat alerts per fingerprint
|
|
13
19
|
|
|
14
20
|
def self.dispatch(log)
|
|
15
21
|
return unless ErrorRadar.config.enabled
|
|
16
|
-
return unless should_fire?(log)
|
|
17
22
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
23
|
+
rules = Array(ErrorRadar.config.notify_on).map(&:to_sym)
|
|
24
|
+
return if rules.empty?
|
|
25
|
+
|
|
26
|
+
# Record the hit for in-memory spike tracking before we test the count.
|
|
27
|
+
if rules.include?(:spike)
|
|
28
|
+
require 'error_radar/spike_detector'
|
|
29
|
+
SpikeDetector.record_hit(log.fingerprint)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
event = determine_event(log, rules)
|
|
33
|
+
return unless event
|
|
34
|
+
|
|
35
|
+
fire_all(log, event)
|
|
24
36
|
rescue StandardError => e
|
|
25
37
|
ErrorRadar::Tracking.warn_internal("Notifier.dispatch failed: #{e.message}")
|
|
26
38
|
end
|
|
27
39
|
|
|
28
40
|
# ── Fire decision ──────────────────────────────────────────────────────
|
|
29
|
-
def self.should_fire?(log)
|
|
30
|
-
rules = Array(ErrorRadar.config.notify_on).map(&:to_sym)
|
|
31
|
-
return false if rules.empty?
|
|
32
41
|
|
|
33
|
-
|
|
34
|
-
|
|
42
|
+
def self.determine_event(log, rules)
|
|
43
|
+
# :new_error fires exactly once per fingerprint, no throttle
|
|
44
|
+
return :new_error if rules.include?(:new_error) && log.new_fingerprint?
|
|
35
45
|
|
|
36
|
-
# :
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
46
|
+
# :spike — rate-based alert when occurrences in window exceed threshold
|
|
47
|
+
if rules.include?(:spike)
|
|
48
|
+
spike_count = SpikeDetector.check(log)
|
|
49
|
+
if spike_count
|
|
50
|
+
window_secs = ErrorRadar.config.spike_window_minutes * 60
|
|
51
|
+
if throttle_ok?("spike:#{log.fingerprint}", window_secs)
|
|
52
|
+
log.instance_variable_set(:@spike_data, {
|
|
53
|
+
count: spike_count,
|
|
54
|
+
window_minutes: ErrorRadar.config.spike_window_minutes
|
|
55
|
+
})
|
|
56
|
+
return :spike
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
# :critical / :all — throttled recurring alerts
|
|
62
|
+
matches = rules.include?(:all) ||
|
|
63
|
+
(rules.include?(:critical) && log.severity_critical?)
|
|
64
|
+
return :recurring if matches && throttle_ok?(log.fingerprint, THROTTLE_INTERVAL)
|
|
65
|
+
|
|
66
|
+
nil
|
|
40
67
|
end
|
|
41
68
|
|
|
42
|
-
def self.throttle_ok?(
|
|
69
|
+
def self.throttle_ok?(key, interval = THROTTLE_INTERVAL)
|
|
43
70
|
THROTTLE_MUTEX.synchronize do
|
|
44
|
-
last = THROTTLE[
|
|
45
|
-
ok = last.nil? || (Time.current - last) >=
|
|
46
|
-
THROTTLE[
|
|
71
|
+
last = THROTTLE[key]
|
|
72
|
+
ok = last.nil? || (Time.current - last) >= interval
|
|
73
|
+
THROTTLE[key] = Time.current if ok
|
|
47
74
|
ok
|
|
48
75
|
end
|
|
49
76
|
end
|
|
50
77
|
|
|
51
78
|
# ── Channel dispatchers ────────────────────────────────────────────────
|
|
52
|
-
|
|
79
|
+
|
|
80
|
+
def self.fire_all(log, event)
|
|
81
|
+
cfg = ErrorRadar.config
|
|
82
|
+
send_slack(log, event) if cfg.slack_webhook_url.to_s.start_with?('http')
|
|
83
|
+
send_discord(log, event) if cfg.discord_webhook_url.to_s.start_with?('http')
|
|
84
|
+
send_email(log, event) if cfg.email_recipients.any?
|
|
85
|
+
cfg.webhook_urls.each { |url| send_webhook(log, url, event) }
|
|
86
|
+
cfg.error_callbacks.each { |cb| safe_call(cb, log) }
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
def self.send_slack(log, event)
|
|
53
90
|
require 'error_radar/notifications/slack'
|
|
54
|
-
Notifications::Slack.deliver(log)
|
|
91
|
+
Notifications::Slack.deliver(log, event)
|
|
55
92
|
end
|
|
56
93
|
|
|
57
|
-
def self.send_discord(log)
|
|
94
|
+
def self.send_discord(log, event)
|
|
58
95
|
require 'error_radar/notifications/discord'
|
|
59
|
-
Notifications::Discord.deliver(log)
|
|
96
|
+
Notifications::Discord.deliver(log, event)
|
|
60
97
|
end
|
|
61
98
|
|
|
62
|
-
def self.send_email(log)
|
|
99
|
+
def self.send_email(log, event)
|
|
63
100
|
require 'error_radar/notifications/email'
|
|
64
|
-
Notifications::Email.deliver(log)
|
|
101
|
+
Notifications::Email.deliver(log, event)
|
|
65
102
|
end
|
|
66
103
|
|
|
67
|
-
def self.send_webhook(log, url)
|
|
104
|
+
def self.send_webhook(log, url, event)
|
|
68
105
|
require 'error_radar/notifications/webhook'
|
|
69
|
-
Notifications::Webhook.deliver(log, url: url)
|
|
106
|
+
Notifications::Webhook.deliver(log, url: url, event: event)
|
|
70
107
|
end
|
|
71
108
|
|
|
72
109
|
def self.safe_call(cb, log)
|
|
@@ -75,7 +112,8 @@ module ErrorRadar
|
|
|
75
112
|
ErrorRadar::Tracking.warn_internal("on_error callback failed: #{e.message}")
|
|
76
113
|
end
|
|
77
114
|
|
|
78
|
-
#
|
|
115
|
+
# ── Helpers ───────────────────────────────────────────────────────────
|
|
116
|
+
|
|
79
117
|
def self.error_url(log)
|
|
80
118
|
host = ErrorRadar.config.app_host.to_s.chomp('/')
|
|
81
119
|
return nil if host.empty?
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ErrorRadar
|
|
4
|
+
# Detects sudden error-rate spikes for a given fingerprint.
|
|
5
|
+
#
|
|
6
|
+
# Strategy (in order of preference):
|
|
7
|
+
# 1. If track_occurrences is on — query error_radar_occurrences for exact
|
|
8
|
+
# count in the window. Accurate across processes and survives restarts.
|
|
9
|
+
# 2. In-memory ring buffer per fingerprint. Fast, zero extra DB queries, but
|
|
10
|
+
# resets on process restart and is per-worker (not shared across Puma workers).
|
|
11
|
+
module SpikeDetector
|
|
12
|
+
MUTEX = Mutex.new
|
|
13
|
+
# fingerprint => Array<Time> of recent hit timestamps (in-memory fallback)
|
|
14
|
+
HITS = Hash.new { |h, k| h[k] = [] }
|
|
15
|
+
# Cap per-key buffer to avoid unbounded growth on very high-volume errors
|
|
16
|
+
MAX_HITS_PER_KEY = 1000
|
|
17
|
+
|
|
18
|
+
# Record a hit in the in-memory ring buffer.
|
|
19
|
+
# Called by Notifier before spike detection so the current hit is counted.
|
|
20
|
+
def self.record_hit(fingerprint)
|
|
21
|
+
MUTEX.synchronize do
|
|
22
|
+
arr = HITS[fingerprint]
|
|
23
|
+
arr << Time.current
|
|
24
|
+
arr.shift while arr.size > MAX_HITS_PER_KEY
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
# Returns the number of recent hits if >= config threshold, false otherwise.
|
|
29
|
+
def self.check(log)
|
|
30
|
+
threshold = ErrorRadar.config.spike_threshold.to_i
|
|
31
|
+
window = ErrorRadar.config.spike_window_minutes.to_i
|
|
32
|
+
return false unless threshold > 0 && window > 0
|
|
33
|
+
|
|
34
|
+
count = recent_count(log, window)
|
|
35
|
+
count >= threshold ? count : false
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
class << self
|
|
39
|
+
private
|
|
40
|
+
|
|
41
|
+
def recent_count(log, window_minutes)
|
|
42
|
+
if ErrorRadar.config.track_occurrences && defined?(ErrorRadar::ErrorOccurrence)
|
|
43
|
+
begin
|
|
44
|
+
return ErrorRadar::ErrorOccurrence
|
|
45
|
+
.where(error_log_id: log.id)
|
|
46
|
+
.where('occurred_at >= ?', window_minutes.minutes.ago)
|
|
47
|
+
.count
|
|
48
|
+
rescue ActiveRecord::StatementInvalid
|
|
49
|
+
# Table not yet created — fall through to in-memory
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
cutoff = window_minutes.minutes.ago
|
|
54
|
+
MUTEX.synchronize do
|
|
55
|
+
(HITS[log.fingerprint] || []).count { |t| t >= cutoff }
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
end
|
|
59
|
+
end
|
|
60
|
+
end
|
data/lib/error_radar/version.rb
CHANGED
data/lib/error_radar.rb
CHANGED
|
@@ -5,6 +5,8 @@ require 'error_radar/configuration'
|
|
|
5
5
|
require 'error_radar/tracking'
|
|
6
6
|
require 'error_radar/notifier'
|
|
7
7
|
require 'error_radar/cleanup'
|
|
8
|
+
require 'error_radar/spike_detector'
|
|
9
|
+
require 'error_radar/digest'
|
|
8
10
|
require 'error_radar/middleware'
|
|
9
11
|
require 'error_radar/server_monitor'
|
|
10
12
|
require 'error_radar/engine'
|
|
@@ -15,10 +15,19 @@ ErrorRadar.configure do |config|
|
|
|
15
15
|
config.install_rails_admin = true # register the ErrorLog board (auto, if RailsAdmin present)
|
|
16
16
|
|
|
17
17
|
# --- Notifications ---
|
|
18
|
-
# When to fire:
|
|
19
|
-
#
|
|
20
|
-
#
|
|
18
|
+
# When to fire:
|
|
19
|
+
# :new_error — first occurrence of a fingerprint (default, no throttle)
|
|
20
|
+
# :critical — any critical-severity occurrence, throttled to 1/hour
|
|
21
|
+
# :all — every occurrence, throttled to 1/hour per fingerprint
|
|
22
|
+
# :spike — rate spike: N hits in M minutes (configure below)
|
|
21
23
|
config.notify_on = [:new_error]
|
|
24
|
+
# config.notify_on = [:new_error, :spike]
|
|
25
|
+
|
|
26
|
+
# --- Spike detection (requires :spike in notify_on) ---
|
|
27
|
+
# Fire an alert when a single error hits >= spike_threshold times within
|
|
28
|
+
# spike_window_minutes. Re-alerts once per window after that.
|
|
29
|
+
# config.spike_threshold = 10 # occurrences …
|
|
30
|
+
# config.spike_window_minutes = 5 # … in this many minutes
|
|
22
31
|
|
|
23
32
|
# Slack incoming webhook (https://api.slack.com/messaging/webhooks)
|
|
24
33
|
# config.slack_webhook_url = ENV['SLACK_WEBHOOK_URL']
|
|
@@ -84,6 +93,15 @@ ErrorRadar.configure do |config|
|
|
|
84
93
|
# config.github_token = ENV['GITHUB_TOKEN'] # PAT with repo scope
|
|
85
94
|
# config.github_repo = 'myorg/myapp' # "owner/repo" format
|
|
86
95
|
|
|
96
|
+
# --- Digest Email ---
|
|
97
|
+
# Sends a periodic summary of errors via email. Requires ActionMailer.
|
|
98
|
+
# Run via cron / Heroku Scheduler:
|
|
99
|
+
# rake error_radar:digest # last 24 hours (daily)
|
|
100
|
+
# rake error_radar:digest:weekly # last 7 days
|
|
101
|
+
# SINCE="2024-01-01 08:00" rake error_radar:digest # custom window
|
|
102
|
+
# config.digest_enabled = true
|
|
103
|
+
# config.digest_recipients = ['team@myapp.com'] # falls back to email_recipients
|
|
104
|
+
|
|
87
105
|
# --- Occurrence History ---
|
|
88
106
|
# Store each individual error hit so you can see context/backtrace per occurrence.
|
|
89
107
|
# Requires running: bin/rails generate error_radar:upgrade_v060 && bin/rails db:migrate
|
data/lib/tasks/error_radar.rake
CHANGED
|
@@ -1,6 +1,25 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
namespace :error_radar do
|
|
4
|
+
desc 'Send a daily digest email summarising the last 24 hours (requires config.digest_enabled = true)'
|
|
5
|
+
task digest: :environment do
|
|
6
|
+
require 'error_radar/digest'
|
|
7
|
+
since = ENV['SINCE'].presence ? Time.parse(ENV['SINCE']) : 24.hours.ago
|
|
8
|
+
ErrorRadar::Digest.deliver(since: since, period: :daily)
|
|
9
|
+
puts '[ErrorRadar] Daily digest sent.'
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
namespace :digest do
|
|
13
|
+
desc 'Send a weekly digest email summarising the last 7 days (requires config.digest_enabled = true)'
|
|
14
|
+
task weekly: :environment do
|
|
15
|
+
require 'error_radar/digest'
|
|
16
|
+
since = ENV['SINCE'].presence ? Time.parse(ENV['SINCE']) : 7.days.ago
|
|
17
|
+
ErrorRadar::Digest.deliver(since: since, period: :weekly)
|
|
18
|
+
puts '[ErrorRadar] Weekly digest sent.'
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
|
|
4
23
|
desc 'Delete old resolved/ignored ErrorLogs per config.retention_days and config.max_records'
|
|
5
24
|
task cleanup: :environment do
|
|
6
25
|
result = ErrorRadar::Cleanup.run
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: error_radar
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.9.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- chienbn9x
|
|
@@ -63,12 +63,15 @@ files:
|
|
|
63
63
|
- app/controllers/error_radar/application_controller.rb
|
|
64
64
|
- app/controllers/error_radar/dashboard_controller.rb
|
|
65
65
|
- app/controllers/error_radar/errors_controller.rb
|
|
66
|
+
- app/mailers/error_radar/digest_mailer.rb
|
|
66
67
|
- app/mailers/error_radar/error_mailer.rb
|
|
67
68
|
- app/models/error_radar/application_record.rb
|
|
68
69
|
- app/models/error_radar/error_log.rb
|
|
69
70
|
- app/models/error_radar/error_occurrence.rb
|
|
70
71
|
- app/views/error_radar/dashboard/index.html.erb
|
|
71
72
|
- app/views/error_radar/dashboard/show.html.erb
|
|
73
|
+
- app/views/error_radar/digest_mailer/digest.html.erb
|
|
74
|
+
- app/views/error_radar/digest_mailer/digest.text.erb
|
|
72
75
|
- app/views/error_radar/error_mailer/new_error.html.erb
|
|
73
76
|
- app/views/error_radar/error_mailer/new_error.text.erb
|
|
74
77
|
- app/views/error_radar/errors/index.html.erb
|
|
@@ -79,6 +82,7 @@ files:
|
|
|
79
82
|
- lib/error_radar/capture_job.rb
|
|
80
83
|
- lib/error_radar/cleanup.rb
|
|
81
84
|
- lib/error_radar/configuration.rb
|
|
85
|
+
- lib/error_radar/digest.rb
|
|
82
86
|
- lib/error_radar/engine.rb
|
|
83
87
|
- lib/error_radar/integrations/active_job.rb
|
|
84
88
|
- lib/error_radar/integrations/github.rb
|
|
@@ -92,6 +96,7 @@ files:
|
|
|
92
96
|
- lib/error_radar/notifications/webhook.rb
|
|
93
97
|
- lib/error_radar/notifier.rb
|
|
94
98
|
- lib/error_radar/server_monitor.rb
|
|
99
|
+
- lib/error_radar/spike_detector.rb
|
|
95
100
|
- lib/error_radar/tracking.rb
|
|
96
101
|
- lib/error_radar/version.rb
|
|
97
102
|
- lib/generators/error_radar/install/install_generator.rb
|