solid_queue_web 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +61 -6
- data/app/assets/stylesheets/solid_queue_web/_04_table.css +5 -1
- data/app/controllers/solid_queue_web/dashboard_controller.rb +2 -0
- data/app/controllers/solid_queue_web/failed_jobs_controller.rb +20 -5
- data/app/controllers/solid_queue_web/history_controller.rb +15 -5
- data/app/controllers/solid_queue_web/jobs_controller.rb +53 -25
- data/app/controllers/solid_queue_web/queues_controller.rb +5 -0
- data/app/helpers/solid_queue_web/application_helper.rb +21 -0
- data/app/javascript/solid_queue_web/application.js +2 -0
- data/app/javascript/solid_queue_web/filters_controller.js +34 -0
- data/app/services/solid_queue_web/slow_job_alert.rb +70 -0
- data/app/services/solid_queue_web/stale_process_alert.rb +68 -0
- data/app/views/solid_queue_web/failed_jobs/index.html.erb +12 -9
- data/app/views/solid_queue_web/history/index.html.erb +12 -9
- data/app/views/solid_queue_web/jobs/index.html.erb +38 -26
- data/app/views/solid_queue_web/queues/index.html.erb +4 -4
- data/app/views/solid_queue_web/queues/jobs/index.html.erb +2 -2
- data/app/views/solid_queue_web/recurring_tasks/index.html.erb +2 -2
- data/app/views/solid_queue_web/scheduled_jobs/update.turbo_stream.erb +1 -1
- data/app/views/solid_queue_web/search/index.html.erb +1 -1
- data/config/importmap.rb +1 -0
- data/lib/solid_queue_web/version.rb +1 -1
- data/lib/solid_queue_web.rb +14 -1
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c0d9757682265e639856df718934858d99390e8ca06ff14cb575b117db105932
|
|
4
|
+
data.tar.gz: a6c0ca7e495f76467cde1aabb0f54674c1439a5489913b4fe9d95837df3acd03
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 41ff634f54cb990c14c57870835f62e091a36130c26e0ecdca0a9fbe07f6ed70b9f128a41d22b59177f03cd21497c3bda9fc17e3044e3fcb843a6031ad9ece3d
|
|
7
|
+
data.tar.gz: 15ed48fde349d58b1f2778cd9b0cf5a8e502ccd74f610a23368665f938fe36c3d66041400eeb96b3aa196d03a5dd56c7becfbd6b743b055cba60ee01e4cb6c3b
|
data/README.md
CHANGED
|
@@ -39,21 +39,22 @@ SolidQueueWeb surfaces all of this in a browser UI available at any route you ch
|
|
|
39
39
|
|
|
40
40
|
- **Dashboard** — stat cards showing counts for ready, scheduled, running, blocked, and failed jobs, plus queues, recurring tasks, and processes; "Done (1h)" and "Done (24h)" throughput cards; a "Throughput — Last 12 Hours" bar chart (blue) and a "Queue Depth — Last 12 Hours" bar chart (purple) showing hourly snapshots of active job count; pure CSS, no charting library; auto-refreshes every 5 seconds
|
|
41
41
|
- **Queues** — all queues sorted by name with size; oldest ready job latency (color-coded, with UTC timestamp tooltip); Done (24h) and Failed (24h) throughput counts; a mini 12-bar failure rate sparkline per queue showing failure % per hour over the last 12 hours; pause/resume controls
|
|
42
|
-
- **Jobs** — filterable by status (ready, scheduled, claimed, blocked, failed), queue, and priority; search by job class name with dynamic auto-submit; time-based period filter (1 h / 24 h / 7 d); discard individual or all jobs; Turbo Frame navigation so only the table updates on filter or search; auto-refreshes every 10 seconds
|
|
42
|
+
- **Jobs** — filterable by status (ready, scheduled, claimed, blocked, failed), queue, and priority; search by job class name with dynamic auto-submit; time-based period filter (1 h / 24 h / 7 d); sortable by class, queue, priority, and enqueued-at; sort state is preserved across filter, period, and status tab changes; discard individual or all jobs; Turbo Frame navigation so only the table updates on filter or search; auto-refreshes every 10 seconds
|
|
43
43
|
- **Scheduled job management** — reschedule a scheduled job to run immediately ("Run Now") or push its `scheduled_at` forward by 1 h, 24 h, or 7 d; Turbo Stream responses update the row in place; "Run All Now" bulk action promotes every scheduled job in the current filtered view in a single operation
|
|
44
|
-
- **Failed jobs** — list of failed executions with error details; search by class name; filter by queue; time-based period filter; retry or discard individually or in bulk; bulk retry with configurable stagger (+5s / +10s / +30s / +1m) to avoid thundering herd on recovery
|
|
44
|
+
- **Failed jobs** — list of failed executions with error details; search by class name; filter by queue; time-based period filter; sortable by class, queue, and failed-at; retry or discard individually or in bulk; bulk retry with configurable stagger (+5s / +10s / +30s / +1m) to avoid thundering herd on recovery
|
|
45
45
|
- **Job detail** — full arguments, timestamps, blocked-until date, and error backtrace; action buttons based on job status; failed jobs show an editable arguments textarea so you can correct a bad payload and retry in one step without redeploying
|
|
46
46
|
- **Queue management** — pause and resume individual queues; queue-scoped job list with status filter, search, and discard
|
|
47
47
|
- **Recurring tasks** — all configured recurring tasks with cron schedule, next run time, last run time, and static/dynamic classification; "Run Now" button enqueues a task immediately without waiting for its next scheduled run
|
|
48
48
|
- **Processes** — workers, dispatchers, and supervisors with heartbeat health status; auto-refreshes every 10 seconds
|
|
49
49
|
- **Global search** — search across all job statuses at once by class name substring; results grouped by status with match count and direct links to filtered views; native datalist autocomplete pre-populated from all known job classes; auto-submits on selection
|
|
50
50
|
- **Targeted bulk actions** — checkboxes on the jobs and failed jobs lists for selecting individual rows; selection bar shows count and action buttons ("Discard Selected" for jobs, "Retry Selected" / "Discard Selected" for failed jobs); select-all checkbox in the table header
|
|
51
|
-
- **Job history** — browsable list of all finished jobs with class name, queue, duration, and finished timestamp; filterable by period (1h / 24h / 7d), queue, and class name search; Done (1h) / Done (24h) dashboard cards link directly to the filtered history view; auto-refreshes every 10 seconds
|
|
51
|
+
- **Job history** — browsable list of all finished jobs with class name, queue, duration, and finished timestamp; filterable by period (1h / 24h / 7d), queue, and class name search; sortable by class, queue, and finished-at; Done (1h) / Done (24h) dashboard cards link directly to the filtered history view; auto-refreshes every 10 seconds
|
|
52
52
|
- **Dark mode** — ☽/☀ toggle in the header; preference persists to `localStorage` and defaults to the OS `prefers-color-scheme` on first visit; zero extra dependencies — implemented via CSS custom properties and a small Stimulus controller
|
|
53
53
|
- **Dashboard quick actions** — "Retry All Failed" and "Discard All Blocked" cards appear on the dashboard only when the respective count is non-zero; one-click bulk operations with confirm dialogs, keeping the dashboard clean when everything is healthy
|
|
54
54
|
- **CSV export** — "Export CSV" button on the jobs, failed jobs, and history pages downloads all records matching the current filters; columns are tailored per view
|
|
55
55
|
- **Slow job detection** — when `slow_job_threshold` is configured, claimed jobs running longer than the threshold are flagged with an orange row, a "slow" badge, and a "Running For" duration column on the Running tab; a "Slow Jobs" warning card appears on the dashboard with a link to the Running tab
|
|
56
|
-
- **
|
|
56
|
+
- **Job wait time** — the Running tab shows a "Wait Time" column with how long each job waited in the queue from enqueue to pickup; also exported as `wait_time_seconds` in the claimed-status CSV
|
|
57
|
+
- **Webhook alerts** — set `alert_webhook_url` and `alert_failure_threshold` to receive a POST request whenever the failed job count meets or exceeds the threshold; set `alert_queue_thresholds` for per-queue depth alerts; set `alert_slow_job_count_threshold` (requires `slow_job_threshold`) for slow-job count alerts; set `alert_stale_process_threshold` for stale-worker alerts; all fire asynchronously with a configurable cooldown (default 1 h) to prevent repeated alerts
|
|
57
58
|
- **Performance analytics** — per-job-class statistics at `/jobs/performance` showing run count, average, p50, p95, p99, standard deviation, min, and max duration; sorted by p95 descending so the slowest classes surface first; high std dev surfaces inconsistent jobs worth investigating; period filter scopes to 1h / 24h / 7d or all time; each class name links to the filtered History view
|
|
58
59
|
- **Failed job trend chart** — a "Failures — Last 12 Hours" bar chart on the dashboard shows failures per hour over the last 12 hours; bars are red, making failure spikes visible before clicking into the failed jobs list
|
|
59
60
|
- **Error frequency report** — `GET /jobs/failed_jobs/errors` groups all failed jobs by error class and message prefix, shows a count per group, and surfaces a sample backtrace in an expandable row; sorted by count descending so the most common errors appear first; accessible via the "Error Summary" button on the Failed Jobs page
|
|
@@ -106,9 +107,12 @@ SolidQueueWeb.configure do |config|
|
|
|
106
107
|
config.slow_job_threshold = 5.minutes # flag claimed jobs running longer than this (default: nil = disabled)
|
|
107
108
|
config.alert_webhook_url = "https://hooks.example.com/solid-queue" # POST target — string or array (default: nil = disabled)
|
|
108
109
|
config.alert_failure_threshold = 10 # fire when failed count >= this (default: nil = disabled)
|
|
109
|
-
config.alert_queue_thresholds
|
|
110
|
-
config.
|
|
110
|
+
config.alert_queue_thresholds = { "critical" => 50, "default" => 200 } # fire when queue depth >= threshold (default: {})
|
|
111
|
+
config.alert_slow_job_count_threshold = 5 # fire when slow job count >= this (default: nil = disabled)
|
|
112
|
+
config.alert_stale_process_threshold = 1 # fire when stale process count >= this (default: nil = disabled)
|
|
113
|
+
config.alert_webhook_cooldown = 1800 # seconds between repeated alerts per alert type (default: 3600)
|
|
111
114
|
config.connects_to = { reading: :reading, writing: :writing } # read replica (default: nil)
|
|
115
|
+
config.time_zone = "America/New_York" # display timezone for all timestamps (default: nil = UTC)
|
|
112
116
|
end
|
|
113
117
|
|
|
114
118
|
SolidQueueWeb.authenticate do
|
|
@@ -181,6 +185,57 @@ The same `alert_webhook_url` endpoint(s) receive the payload, with a distinct ev
|
|
|
181
185
|
|
|
182
186
|
Cooldown is tracked independently per queue, so a persistently deep "critical" queue does not suppress alerts for "default". The shared `alert_webhook_cooldown` setting applies to each queue separately.
|
|
183
187
|
|
|
188
|
+
## Slow job alerts
|
|
189
|
+
|
|
190
|
+
Set `alert_slow_job_count_threshold` to fire a webhook when the number of currently-running slow jobs meets or exceeds a count. This requires `slow_job_threshold` to also be configured — it defines what "slow" means.
|
|
191
|
+
|
|
192
|
+
```ruby
|
|
193
|
+
SolidQueueWeb.configure do |config|
|
|
194
|
+
config.slow_job_threshold = 5.minutes # a job is "slow" if it has been claimed longer than this
|
|
195
|
+
config.alert_slow_job_count_threshold = 3 # fire when >= 3 jobs are slow
|
|
196
|
+
config.alert_webhook_url = "https://hooks.example.com/solid-queue"
|
|
197
|
+
config.alert_webhook_cooldown = 1800 # don't re-fire for 30 minutes (default: 3600)
|
|
198
|
+
end
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
The same `alert_webhook_url` endpoint(s) receive the payload with a distinct event type:
|
|
202
|
+
|
|
203
|
+
```json
|
|
204
|
+
{
|
|
205
|
+
"event": "slow_job_threshold_exceeded",
|
|
206
|
+
"slow_job_count": 5,
|
|
207
|
+
"threshold": 3,
|
|
208
|
+
"fired_at": "2026-05-28T08:00:00Z"
|
|
209
|
+
}
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
The alert fires on every dashboard page load while the condition persists, subject to the cooldown window.
|
|
213
|
+
|
|
214
|
+
## Stale process alerts
|
|
215
|
+
|
|
216
|
+
Set `alert_stale_process_threshold` to fire a webhook when the number of stale workers meets or exceeds a count. A process is considered stale when its `last_heartbeat_at` has not been updated within `SolidQueue.process_alive_threshold` (default 5 minutes). A stale worker means jobs in its queues have silently stopped processing.
|
|
217
|
+
|
|
218
|
+
```ruby
|
|
219
|
+
SolidQueueWeb.configure do |config|
|
|
220
|
+
config.alert_stale_process_threshold = 1 # fire when any process goes stale
|
|
221
|
+
config.alert_webhook_url = "https://hooks.example.com/solid-queue"
|
|
222
|
+
config.alert_webhook_cooldown = 1800 # don't re-fire for 30 minutes (default: 3600)
|
|
223
|
+
end
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
The same `alert_webhook_url` endpoint(s) receive the payload with a distinct event type:
|
|
227
|
+
|
|
228
|
+
```json
|
|
229
|
+
{
|
|
230
|
+
"event": "stale_process_detected",
|
|
231
|
+
"stale_process_count": 2,
|
|
232
|
+
"threshold": 1,
|
|
233
|
+
"fired_at": "2026-05-28T08:00:00Z"
|
|
234
|
+
}
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
The alert fires on every dashboard page load while the condition persists, subject to the cooldown window.
|
|
238
|
+
|
|
184
239
|
## Metrics endpoint
|
|
185
240
|
|
|
186
241
|
`GET /jobs/metrics.json` returns a machine-readable JSON document suitable for Prometheus scraping, uptime monitors, or external dashboards. No configuration is required — the endpoint is available as soon as the engine is mounted.
|
|
@@ -56,4 +56,8 @@ tbody tr:hover { background: var(--bg); }
|
|
|
56
56
|
|
|
57
57
|
.sqd-row--slow { background: rgba(253, 126, 20, 0.07); }
|
|
58
58
|
.sqd-row--slow:hover { background: rgba(253, 126, 20, 0.13); }
|
|
59
|
-
.sqd-slow-duration { color: var(--warning); font-weight: 600; }
|
|
59
|
+
.sqd-slow-duration { color: var(--warning); font-weight: 600; }
|
|
60
|
+
|
|
61
|
+
th a { color: inherit; text-decoration: none; }
|
|
62
|
+
th a:hover { color: var(--primary); }
|
|
63
|
+
.sqd-sort-indicator { margin-left: 0.2rem; }
|
|
@@ -4,7 +4,7 @@ module SolidQueueWeb
|
|
|
4
4
|
|
|
5
5
|
def index
|
|
6
6
|
respond_to do |format|
|
|
7
|
-
format.html { @pagy, @failed_jobs = pagy(filtered_scope.order(
|
|
7
|
+
format.html { @pagy, @failed_jobs = pagy(filtered_scope.order(sort_expression)) }
|
|
8
8
|
format.csv do
|
|
9
9
|
send_data failed_jobs_csv,
|
|
10
10
|
filename: "failed-jobs-#{Date.today}.csv",
|
|
@@ -25,7 +25,7 @@ module SolidQueueWeb
|
|
|
25
25
|
def failed_jobs_csv
|
|
26
26
|
CSV.generate(headers: true) do |csv|
|
|
27
27
|
csv << %w[id class_name queue_name error_class error_message failed_at]
|
|
28
|
-
filtered_scope.order(
|
|
28
|
+
filtered_scope.order(sort_expression).each do |execution|
|
|
29
29
|
job = execution.job
|
|
30
30
|
error = execution.error || {}
|
|
31
31
|
csv << [job.id, job.class_name, job.queue_name,
|
|
@@ -42,10 +42,25 @@ module SolidQueueWeb
|
|
|
42
42
|
notice: "#{jobs.size} #{"job".pluralize(jobs.size)} discarded."
|
|
43
43
|
end
|
|
44
44
|
|
|
45
|
+
def sortable_columns
|
|
46
|
+
%w[class_name queue_name created_at]
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
def sort_expression
|
|
50
|
+
sql_col = case @sort
|
|
51
|
+
when "class_name" then "solid_queue_jobs.class_name"
|
|
52
|
+
when "queue_name" then "solid_queue_jobs.queue_name"
|
|
53
|
+
else "solid_queue_failed_executions.created_at"
|
|
54
|
+
end
|
|
55
|
+
Arel.sql("#{sql_col} #{@direction == 'asc' ? 'ASC' : 'DESC'}")
|
|
56
|
+
end
|
|
57
|
+
|
|
45
58
|
def set_filter_params
|
|
46
|
-
@queue
|
|
47
|
-
@search
|
|
48
|
-
@period
|
|
59
|
+
@queue = params[:queue].presence
|
|
60
|
+
@search = params[:q].presence
|
|
61
|
+
@period = params[:period].presence_in(PERIOD_DURATIONS.keys)
|
|
62
|
+
@sort = params[:sort].presence_in(sortable_columns) || "created_at"
|
|
63
|
+
@direction = params[:direction] == "asc" ? "asc" : "desc"
|
|
49
64
|
end
|
|
50
65
|
|
|
51
66
|
def filtered_scope
|
|
@@ -1,9 +1,11 @@
|
|
|
1
1
|
module SolidQueueWeb
|
|
2
2
|
class HistoryController < ApplicationController
|
|
3
3
|
def index
|
|
4
|
-
@queue
|
|
5
|
-
@search
|
|
6
|
-
@period
|
|
4
|
+
@queue = params[:queue].presence
|
|
5
|
+
@search = params[:q].presence
|
|
6
|
+
@period = params[:period].presence_in(PERIOD_DURATIONS.keys)
|
|
7
|
+
@sort = params[:sort].presence_in(sortable_columns) || "finished_at"
|
|
8
|
+
@direction = params[:direction] == "asc" ? "asc" : "desc"
|
|
7
9
|
|
|
8
10
|
scope = SolidQueue::Job.where.not(finished_at: nil)
|
|
9
11
|
scope = scope.where(queue_name: @queue) if @queue.present?
|
|
@@ -11,7 +13,7 @@ module SolidQueueWeb
|
|
|
11
13
|
scope = scope.where("finished_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
|
|
12
14
|
|
|
13
15
|
respond_to do |format|
|
|
14
|
-
format.html { @pagy, @jobs = pagy(scope.order(
|
|
16
|
+
format.html { @pagy, @jobs = pagy(scope.order(sort_expression)) }
|
|
15
17
|
format.csv do
|
|
16
18
|
send_data history_csv(scope),
|
|
17
19
|
filename: "job-history-#{Date.today}.csv",
|
|
@@ -22,10 +24,18 @@ module SolidQueueWeb
|
|
|
22
24
|
|
|
23
25
|
private
|
|
24
26
|
|
|
27
|
+
def sortable_columns
|
|
28
|
+
%w[class_name queue_name finished_at]
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def sort_expression
|
|
32
|
+
Arel.sql("#{@sort} #{@direction == 'asc' ? 'ASC' : 'DESC'}")
|
|
33
|
+
end
|
|
34
|
+
|
|
25
35
|
def history_csv(scope)
|
|
26
36
|
CSV.generate(headers: true) do |csv|
|
|
27
37
|
csv << %w[id class_name queue_name duration_seconds finished_at]
|
|
28
|
-
scope.order(
|
|
38
|
+
scope.order(sort_expression).each do |job|
|
|
29
39
|
duration = job.finished_at && job.created_at ? (job.finished_at - job.created_at).round : nil
|
|
30
40
|
csv << [job.id, job.class_name, job.queue_name, duration, job.finished_at.iso8601]
|
|
31
41
|
end
|
|
@@ -1,22 +1,16 @@
|
|
|
1
1
|
module SolidQueueWeb
|
|
2
2
|
class JobsController < ApplicationController
|
|
3
|
-
|
|
4
|
-
@status = params[:status].presence_in(Job::STATUSES) || "ready"
|
|
5
|
-
@search = params[:q].presence
|
|
6
|
-
@period = params[:period].presence_in(PERIOD_DURATIONS.keys)
|
|
7
|
-
@priority = params[:priority].presence
|
|
8
|
-
|
|
9
|
-
scope = Job::EXECUTION_MODELS[@status].includes(:job)
|
|
10
|
-
scope = scope.references(:job).where("solid_queue_jobs.class_name LIKE ?", "%#{@search}%") if @search.present?
|
|
11
|
-
scope = scope.references(:job).where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
|
|
12
|
-
scope = scope.references(:job).where("solid_queue_jobs.priority = ?", @priority.to_i) if @priority.present?
|
|
13
|
-
scope = scope.order(created_at: :desc)
|
|
3
|
+
before_action :set_filters, only: [:index, :destroy]
|
|
14
4
|
|
|
15
|
-
|
|
16
|
-
|
|
5
|
+
def index
|
|
6
|
+
scope = job_scope
|
|
17
7
|
|
|
18
8
|
respond_to do |format|
|
|
19
|
-
format.html
|
|
9
|
+
format.html do
|
|
10
|
+
@priority_options = Job::EXECUTION_MODELS[@status].joins(:job)
|
|
11
|
+
.distinct.pluck("solid_queue_jobs.priority").sort
|
|
12
|
+
@pagy, @jobs = pagy(scope)
|
|
13
|
+
end
|
|
20
14
|
format.csv do
|
|
21
15
|
send_data jobs_csv(scope),
|
|
22
16
|
filename: "jobs-#{@status}-#{Date.today}.csv",
|
|
@@ -33,9 +27,6 @@ module SolidQueueWeb
|
|
|
33
27
|
end
|
|
34
28
|
|
|
35
29
|
def destroy
|
|
36
|
-
@status = params[:status]
|
|
37
|
-
@period = params[:period].presence_in(PERIOD_DURATIONS.keys)
|
|
38
|
-
@priority = params[:priority].presence
|
|
39
30
|
model = Job.execution_model_for!(@status)
|
|
40
31
|
if params[:id]
|
|
41
32
|
@execution = model.find(params[:id])
|
|
@@ -43,33 +34,70 @@ module SolidQueueWeb
|
|
|
43
34
|
@remaining_count = filtered_scope(model).count
|
|
44
35
|
respond_to do |format|
|
|
45
36
|
format.turbo_stream
|
|
46
|
-
format.html { redirect_to
|
|
37
|
+
format.html { redirect_to jobs_return_path, notice: "Job discarded." }
|
|
47
38
|
end
|
|
48
39
|
else
|
|
49
40
|
jobs = filtered_scope(model).map(&:job)
|
|
50
41
|
model.discard_all_from_jobs(jobs)
|
|
51
|
-
redirect_to
|
|
52
|
-
notice: "#{jobs.size} #{"job".pluralize(jobs.size)} discarded."
|
|
42
|
+
redirect_to jobs_return_path, notice: "#{jobs.size} #{"job".pluralize(jobs.size)} discarded."
|
|
53
43
|
end
|
|
54
44
|
rescue ArgumentError => e
|
|
55
|
-
redirect_to
|
|
45
|
+
redirect_to jobs_return_path, alert: e.message
|
|
56
46
|
rescue => e
|
|
57
|
-
redirect_to
|
|
58
|
-
alert: "Could not discard #{params[:id] ? "job" : "jobs"}: #{e.message}"
|
|
47
|
+
redirect_to jobs_return_path, alert: "Could not discard #{params[:id] ? "job" : "jobs"}: #{e.message}"
|
|
59
48
|
end
|
|
60
49
|
|
|
61
50
|
private
|
|
62
51
|
|
|
52
|
+
def set_filters
|
|
53
|
+
@status = params[:status].presence_in(Job::STATUSES) || "ready"
|
|
54
|
+
@search = params[:q].presence
|
|
55
|
+
@period = params[:period].presence_in(PERIOD_DURATIONS.keys)
|
|
56
|
+
@priority = params[:priority].presence
|
|
57
|
+
@sort = params[:sort].presence_in(sortable_columns) || "created_at"
|
|
58
|
+
@direction = params[:direction] == "asc" ? "asc" : "desc"
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def job_scope
|
|
62
|
+
scope = Job::EXECUTION_MODELS[@status].includes(:job).references(:job).order(sort_expression)
|
|
63
|
+
scope = scope.where("solid_queue_jobs.class_name LIKE ?", "%#{@search}%") if @search.present?
|
|
64
|
+
scope = scope.where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
|
|
65
|
+
scope = scope.where("solid_queue_jobs.priority = ?", @priority.to_i) if @priority.present?
|
|
66
|
+
scope
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def sortable_columns
|
|
70
|
+
%w[class_name queue_name priority created_at]
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def sort_expression
|
|
74
|
+
sql_col = case @sort
|
|
75
|
+
when "class_name" then "solid_queue_jobs.class_name"
|
|
76
|
+
when "queue_name" then "solid_queue_jobs.queue_name"
|
|
77
|
+
when "priority" then "solid_queue_jobs.priority"
|
|
78
|
+
else "#{Job::EXECUTION_MODELS[@status].quoted_table_name}.created_at"
|
|
79
|
+
end
|
|
80
|
+
Arel.sql("#{sql_col} #{@direction == 'asc' ? 'ASC' : 'DESC'}")
|
|
81
|
+
end
|
|
82
|
+
|
|
63
83
|
def jobs_csv(scope)
|
|
64
84
|
CSV.generate(headers: true) do |csv|
|
|
65
|
-
|
|
85
|
+
headers = %w[id class_name queue_name status priority enqueued_at]
|
|
86
|
+
headers << "wait_time_seconds" if @status == "claimed"
|
|
87
|
+
csv << headers
|
|
66
88
|
scope.each do |execution|
|
|
67
89
|
job = execution.job
|
|
68
|
-
|
|
90
|
+
row = [job.id, job.class_name, job.queue_name, @status, job.priority, job.created_at.iso8601]
|
|
91
|
+
row << (execution.created_at - job.created_at).to_i if @status == "claimed"
|
|
92
|
+
csv << row
|
|
69
93
|
end
|
|
70
94
|
end
|
|
71
95
|
end
|
|
72
96
|
|
|
97
|
+
def jobs_return_path
|
|
98
|
+
jobs_path(status: @status, period: @period, sort: @sort, direction: @direction)
|
|
99
|
+
end
|
|
100
|
+
|
|
73
101
|
def filtered_scope(model)
|
|
74
102
|
scope = model.includes(:job)
|
|
75
103
|
scope = scope.references(:job).where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
|
|
@@ -7,6 +7,11 @@ module SolidQueueWeb
|
|
|
7
7
|
@failed_24h = stats.failed_24h
|
|
8
8
|
@oldest_ready = stats.oldest_ready
|
|
9
9
|
@failure_sparklines = stats.failure_sparklines
|
|
10
|
+
@queue_sizes = SolidQueue::ReadyExecution
|
|
11
|
+
.joins(:job)
|
|
12
|
+
.group("solid_queue_jobs.queue_name")
|
|
13
|
+
.count
|
|
14
|
+
@paused_queue_names = SolidQueue::Pause.pluck(:queue_name).to_set
|
|
10
15
|
end
|
|
11
16
|
end
|
|
12
17
|
end
|
|
@@ -1,11 +1,32 @@
|
|
|
1
1
|
module SolidQueueWeb
|
|
2
2
|
module ApplicationHelper
|
|
3
|
+
def sort_header_th(label, col, url_proc, current_sort:, current_dir:)
|
|
4
|
+
is_active = current_sort == col
|
|
5
|
+
next_dir = (is_active && current_dir == "desc") ? "asc" : "desc"
|
|
6
|
+
indicator = is_active ? content_tag(:span, current_dir == "desc" ? "↓" : "↑", class: "sqd-sort-indicator") : nil
|
|
7
|
+
tag_opts = { scope: "col" }
|
|
8
|
+
tag_opts[:"aria-sort"] = current_dir == "asc" ? "ascending" : "descending" if is_active
|
|
9
|
+
content_tag(:th, **tag_opts) do
|
|
10
|
+
link_to(url_proc.call(sort: col, direction: next_dir)) do
|
|
11
|
+
safe_join([label, indicator].compact)
|
|
12
|
+
end
|
|
13
|
+
end
|
|
14
|
+
end
|
|
15
|
+
|
|
3
16
|
def inline_styles
|
|
4
17
|
dir = SolidQueueWeb::Engine.root.join("app/assets/stylesheets/solid_queue_web")
|
|
5
18
|
css = dir.glob("_*.css").sort.map(&:read).join("\n")
|
|
6
19
|
content_tag(:style, css.html_safe)
|
|
7
20
|
end
|
|
8
21
|
|
|
22
|
+
def format_timestamp(time, format: "%Y-%m-%d %H:%M:%S")
|
|
23
|
+
return "—" unless time
|
|
24
|
+
|
|
25
|
+
tz = SolidQueueWeb.time_zone
|
|
26
|
+
displayed = tz ? time.in_time_zone(tz) : time.utc
|
|
27
|
+
displayed.strftime(format)
|
|
28
|
+
end
|
|
29
|
+
|
|
9
30
|
def format_duration(seconds)
|
|
10
31
|
s = seconds.to_i
|
|
11
32
|
return "< 1s" if s < 1
|
|
@@ -4,9 +4,11 @@ import SearchController from "solid_queue_web/search_controller"
|
|
|
4
4
|
import RefreshController from "solid_queue_web/refresh_controller"
|
|
5
5
|
import SelectionController from "solid_queue_web/selection_controller"
|
|
6
6
|
import ThemeController from "solid_queue_web/theme_controller"
|
|
7
|
+
import FiltersController from "solid_queue_web/filters_controller"
|
|
7
8
|
|
|
8
9
|
const application = Application.start()
|
|
9
10
|
application.register("search", SearchController)
|
|
10
11
|
application.register("refresh", RefreshController)
|
|
11
12
|
application.register("selection", SelectionController)
|
|
12
13
|
application.register("theme", ThemeController)
|
|
14
|
+
application.register("filters", FiltersController)
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import { Controller } from "@hotwired/stimulus"
|
|
2
|
+
|
|
3
|
+
export default class extends Controller {
|
|
4
|
+
static values = { page: String }
|
|
5
|
+
|
|
6
|
+
connect() {
|
|
7
|
+
const url = new URL(window.location.href)
|
|
8
|
+
const params = url.searchParams
|
|
9
|
+
const page = this.pageValue
|
|
10
|
+
let changed = false
|
|
11
|
+
|
|
12
|
+
const keys = page === "jobs" ? ["status", "period"] : ["period"]
|
|
13
|
+
keys.forEach(key => {
|
|
14
|
+
if (!params.has(key)) {
|
|
15
|
+
const saved = localStorage.getItem(`sqd-${page}-${key}`)
|
|
16
|
+
if (saved) { params.set(key, saved); changed = true }
|
|
17
|
+
}
|
|
18
|
+
})
|
|
19
|
+
|
|
20
|
+
if (changed) window.location.replace(url.toString())
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
saveStatus({ params: { status } }) {
|
|
24
|
+
if (status) localStorage.setItem(`sqd-${this.pageValue}-status`, status)
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
savePeriod({ params: { period } }) {
|
|
28
|
+
if (period) {
|
|
29
|
+
localStorage.setItem(`sqd-${this.pageValue}-period`, period)
|
|
30
|
+
} else {
|
|
31
|
+
localStorage.removeItem(`sqd-${this.pageValue}-period`)
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
}
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "json"
|
|
3
|
+
require "uri"
|
|
4
|
+
|
|
5
|
+
module SolidQueueWeb
|
|
6
|
+
class SlowJobAlert
|
|
7
|
+
MUTEX = Mutex.new
|
|
8
|
+
|
|
9
|
+
class << self
|
|
10
|
+
def call
|
|
11
|
+
return unless configured?
|
|
12
|
+
|
|
13
|
+
slow_count = SolidQueue::ClaimedExecution
|
|
14
|
+
.where("created_at <= ?", SolidQueueWeb.slow_job_threshold.ago)
|
|
15
|
+
.count
|
|
16
|
+
|
|
17
|
+
return if slow_count < SolidQueueWeb.alert_slow_job_count_threshold
|
|
18
|
+
return unless should_fire?
|
|
19
|
+
|
|
20
|
+
urls = webhook_urls
|
|
21
|
+
Thread.new { urls.each { |url| post(url, slow_count) } }
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def reset!
|
|
25
|
+
MUTEX.synchronize { @last_fired_at = nil }
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
def configured?
|
|
31
|
+
SolidQueueWeb.slow_job_threshold.present? &&
|
|
32
|
+
SolidQueueWeb.alert_slow_job_count_threshold.present? &&
|
|
33
|
+
webhook_urls.any?
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def webhook_urls
|
|
37
|
+
Array(SolidQueueWeb.alert_webhook_url).flatten.compact.select(&:present?)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
def should_fire?
|
|
41
|
+
MUTEX.synchronize do
|
|
42
|
+
cooldown = SolidQueueWeb.alert_webhook_cooldown
|
|
43
|
+
return false if @last_fired_at && Time.current - @last_fired_at < cooldown
|
|
44
|
+
|
|
45
|
+
@last_fired_at = Time.current
|
|
46
|
+
true
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def post(url_string, slow_count)
|
|
51
|
+
uri = URI.parse(url_string)
|
|
52
|
+
payload = JSON.generate(
|
|
53
|
+
event: "slow_job_threshold_exceeded",
|
|
54
|
+
slow_job_count: slow_count,
|
|
55
|
+
threshold: SolidQueueWeb.alert_slow_job_count_threshold,
|
|
56
|
+
fired_at: Time.current.iso8601
|
|
57
|
+
)
|
|
58
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
59
|
+
http.use_ssl = uri.scheme == "https"
|
|
60
|
+
http.open_timeout = 5
|
|
61
|
+
http.read_timeout = 10
|
|
62
|
+
request = Net::HTTP::Post.new(uri.path.presence || "/", "Content-Type" => "application/json")
|
|
63
|
+
request.body = payload
|
|
64
|
+
http.request(request)
|
|
65
|
+
rescue => e
|
|
66
|
+
Rails.logger.error("[SolidQueueWeb] Slow job alert webhook failed: #{e.message}")
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
require "net/http"
|
|
2
|
+
require "json"
|
|
3
|
+
require "uri"
|
|
4
|
+
|
|
5
|
+
module SolidQueueWeb
|
|
6
|
+
class StaleProcessAlert
|
|
7
|
+
MUTEX = Mutex.new
|
|
8
|
+
|
|
9
|
+
class << self
|
|
10
|
+
def call
|
|
11
|
+
return unless configured?
|
|
12
|
+
|
|
13
|
+
stale_count = SolidQueue::Process
|
|
14
|
+
.where("last_heartbeat_at < ?", SolidQueue.process_alive_threshold.ago)
|
|
15
|
+
.count
|
|
16
|
+
|
|
17
|
+
return if stale_count < SolidQueueWeb.alert_stale_process_threshold
|
|
18
|
+
return unless should_fire?
|
|
19
|
+
|
|
20
|
+
urls = webhook_urls
|
|
21
|
+
Thread.new { urls.each { |url| post(url, stale_count) } }
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
def reset!
|
|
25
|
+
MUTEX.synchronize { @last_fired_at = nil }
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
private
|
|
29
|
+
|
|
30
|
+
def configured?
|
|
31
|
+
SolidQueueWeb.alert_stale_process_threshold.present? && webhook_urls.any?
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def webhook_urls
|
|
35
|
+
Array(SolidQueueWeb.alert_webhook_url).flatten.compact.select(&:present?)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def should_fire?
|
|
39
|
+
MUTEX.synchronize do
|
|
40
|
+
cooldown = SolidQueueWeb.alert_webhook_cooldown
|
|
41
|
+
return false if @last_fired_at && Time.current - @last_fired_at < cooldown
|
|
42
|
+
|
|
43
|
+
@last_fired_at = Time.current
|
|
44
|
+
true
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def post(url_string, stale_count)
|
|
49
|
+
uri = URI.parse(url_string)
|
|
50
|
+
payload = JSON.generate(
|
|
51
|
+
event: "stale_process_detected",
|
|
52
|
+
stale_process_count: stale_count,
|
|
53
|
+
threshold: SolidQueueWeb.alert_stale_process_threshold,
|
|
54
|
+
fired_at: Time.current.iso8601
|
|
55
|
+
)
|
|
56
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
|
57
|
+
http.use_ssl = uri.scheme == "https"
|
|
58
|
+
http.open_timeout = 5
|
|
59
|
+
http.read_timeout = 10
|
|
60
|
+
request = Net::HTTP::Post.new(uri.path.presence || "/", "Content-Type" => "application/json")
|
|
61
|
+
request.body = payload
|
|
62
|
+
http.request(request)
|
|
63
|
+
rescue => e
|
|
64
|
+
Rails.logger.error("[SolidQueueWeb] Stale process alert webhook failed: #{e.message}")
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
@@ -34,17 +34,19 @@
|
|
|
34
34
|
<input type="hidden" name="queue" value="<%= @queue %>">
|
|
35
35
|
<% end %>
|
|
36
36
|
<input type="hidden" name="period" value="<%= @period %>">
|
|
37
|
+
<input type="hidden" name="sort" value="<%= @sort %>">
|
|
38
|
+
<input type="hidden" name="direction" value="<%= @direction %>">
|
|
37
39
|
<input class="sqd-search__input" type="search" name="q" value="<%= @search %>"
|
|
38
40
|
placeholder="Filter by job class…" autocomplete="off" aria-label="Filter by job class"
|
|
39
41
|
data-action="input->search#filter">
|
|
40
42
|
<% if @search.present? %>
|
|
41
43
|
<%= link_to "Clear", failed_jobs_path(queue: @queue, period: @period), class: "sqd-btn sqd-btn--muted" %>
|
|
42
44
|
<% end %>
|
|
43
|
-
<div class="sqd-period-filter" role="group" aria-label="Time period">
|
|
44
|
-
<%= link_to "All", failed_jobs_path(queue: @queue, q: @search), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil }, aria_label: "All time" %>
|
|
45
|
-
<%= link_to "1h", failed_jobs_path(queue: @queue, q: @search, period: "1h"), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil }, aria_label: "Last 1 hour" %>
|
|
46
|
-
<%= link_to "24h", failed_jobs_path(queue: @queue, q: @search, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours" %>
|
|
47
|
-
<%= link_to "7d", failed_jobs_path(queue: @queue, q: @search, period: "7d"), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil }, aria_label: "Last 7 days" %>
|
|
45
|
+
<div class="sqd-period-filter" role="group" aria-label="Time period" data-controller="filters" data-filters-page-value="failed">
|
|
46
|
+
<%= link_to "All", failed_jobs_path(queue: @queue, q: @search), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil }, aria_label: "All time", data: { action: "click->filters#savePeriod", filters_period_param: "" } %>
|
|
47
|
+
<%= link_to "1h", failed_jobs_path(queue: @queue, q: @search, period: "1h"), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil }, aria_label: "Last 1 hour", data: { action: "click->filters#savePeriod", filters_period_param: "1h" } %>
|
|
48
|
+
<%= link_to "24h", failed_jobs_path(queue: @queue, q: @search, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours", data: { action: "click->filters#savePeriod", filters_period_param: "24h" } %>
|
|
49
|
+
<%= link_to "7d", failed_jobs_path(queue: @queue, q: @search, period: "7d"), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil }, aria_label: "Last 7 days", data: { action: "click->filters#savePeriod", filters_period_param: "7d" } %>
|
|
48
50
|
</div>
|
|
49
51
|
</form>
|
|
50
52
|
|
|
@@ -86,10 +88,11 @@
|
|
|
86
88
|
data-action="change->selection#selectAll"
|
|
87
89
|
aria-label="Select all failed jobs">
|
|
88
90
|
</th>
|
|
89
|
-
|
|
90
|
-
|
|
91
|
+
<% sort_url = ->(p) { failed_jobs_path(queue: @queue, q: @search, period: @period, **p) } %>
|
|
92
|
+
<%= sort_header_th("Job Class", "class_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
93
|
+
<%= sort_header_th("Queue", "queue_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
91
94
|
<th scope="col">Error</th>
|
|
92
|
-
|
|
95
|
+
<%= sort_header_th("Failed At", "created_at", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
93
96
|
<th scope="col"><span class="sqd-sr-only">Actions</span></th>
|
|
94
97
|
</tr>
|
|
95
98
|
</thead>
|
|
@@ -117,7 +120,7 @@
|
|
|
117
120
|
<span style="color:var(--muted)">—</span>
|
|
118
121
|
<% end %>
|
|
119
122
|
</td>
|
|
120
|
-
<td class="sqd-mono"><%= execution.created_at
|
|
123
|
+
<td class="sqd-mono"><%= format_timestamp(execution.created_at) %></td>
|
|
121
124
|
<td class="sqd-row-actions">
|
|
122
125
|
<%= button_to "Retry", retry_failed_job_path(execution), method: :post,
|
|
123
126
|
class: "sqd-btn sqd-btn--primary sqd-btn--sm" %>
|
|
@@ -14,17 +14,19 @@
|
|
|
14
14
|
<input type="hidden" name="queue" value="<%= @queue %>">
|
|
15
15
|
<% end %>
|
|
16
16
|
<input type="hidden" name="period" value="<%= @period %>">
|
|
17
|
+
<input type="hidden" name="sort" value="<%= @sort %>">
|
|
18
|
+
<input type="hidden" name="direction" value="<%= @direction %>">
|
|
17
19
|
<input class="sqd-search__input" type="search" name="q" value="<%= @search %>"
|
|
18
20
|
placeholder="Filter by job class…" autocomplete="off" aria-label="Filter by job class"
|
|
19
21
|
data-action="input->search#filter">
|
|
20
22
|
<% if @search.present? %>
|
|
21
23
|
<%= link_to "Clear", history_path(queue: @queue, period: @period), class: "sqd-btn sqd-btn--muted" %>
|
|
22
24
|
<% end %>
|
|
23
|
-
<div class="sqd-period-filter" role="group" aria-label="Time period">
|
|
24
|
-
<%= link_to "All", history_path(queue: @queue, q: @search), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil } %>
|
|
25
|
-
<%= link_to "1h", history_path(queue: @queue, q: @search, period: "1h"), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil } %>
|
|
26
|
-
<%= link_to "24h", history_path(queue: @queue, q: @search, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil } %>
|
|
27
|
-
<%= link_to "7d", history_path(queue: @queue, q: @search, period: "7d"), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil } %>
|
|
25
|
+
<div class="sqd-period-filter" role="group" aria-label="Time period" data-controller="filters" data-filters-page-value="history">
|
|
26
|
+
<%= link_to "All", history_path(queue: @queue, q: @search), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil }, data: { action: "click->filters#savePeriod", filters_period_param: "" } %>
|
|
27
|
+
<%= link_to "1h", history_path(queue: @queue, q: @search, period: "1h"), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil }, data: { action: "click->filters#savePeriod", filters_period_param: "1h" } %>
|
|
28
|
+
<%= link_to "24h", history_path(queue: @queue, q: @search, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, data: { action: "click->filters#savePeriod", filters_period_param: "24h" } %>
|
|
29
|
+
<%= link_to "7d", history_path(queue: @queue, q: @search, period: "7d"), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil }, data: { action: "click->filters#savePeriod", filters_period_param: "7d" } %>
|
|
28
30
|
</div>
|
|
29
31
|
</form>
|
|
30
32
|
|
|
@@ -39,11 +41,12 @@
|
|
|
39
41
|
<div class="sqd-card">
|
|
40
42
|
<table>
|
|
41
43
|
<thead>
|
|
44
|
+
<% sort_url = ->(p) { history_path(queue: @queue, q: @search, period: @period, **p) } %>
|
|
42
45
|
<tr>
|
|
43
|
-
|
|
44
|
-
|
|
46
|
+
<%= sort_header_th("Job Class", "class_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
47
|
+
<%= sort_header_th("Queue", "queue_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
45
48
|
<th scope="col">Duration</th>
|
|
46
|
-
|
|
49
|
+
<%= sort_header_th("Finished At", "finished_at", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
47
50
|
</tr>
|
|
48
51
|
</thead>
|
|
49
52
|
<tbody>
|
|
@@ -55,7 +58,7 @@
|
|
|
55
58
|
class: "sqd-mono", style: "color: inherit;" %>
|
|
56
59
|
</td>
|
|
57
60
|
<td class="sqd-mono"><%= format_duration(job.finished_at - job.created_at) %></td>
|
|
58
|
-
<td class="sqd-mono"><%= job.finished_at
|
|
61
|
+
<td class="sqd-mono"><%= format_timestamp(job.finished_at) %></td>
|
|
59
62
|
</tr>
|
|
60
63
|
<% end %>
|
|
61
64
|
</tbody>
|
|
@@ -3,17 +3,18 @@
|
|
|
3
3
|
<%= turbo_frame_tag "jobs-table", data: { turbo_action: "advance", controller: "refresh", refresh_interval_value: SolidQueueWeb.default_refresh_interval } do %>
|
|
4
4
|
<% discardable = SolidQueueWeb::Job::DISCARDABLE.include?(@status) %>
|
|
5
5
|
|
|
6
|
+
<div data-controller="filters" data-filters-page-value="jobs">
|
|
6
7
|
<div class="sqd-page-header">
|
|
7
8
|
<div class="sqd-filters">
|
|
8
|
-
<%= link_to "Ready", jobs_path(status: "ready", q: @search, period: @period, priority: @priority), class: @status == "ready" ? "active" : "" %>
|
|
9
|
-
<%= link_to "Scheduled", jobs_path(status: "scheduled", q: @search, period: @period, priority: @priority), class: @status == "scheduled" ? "active" : "" %>
|
|
10
|
-
<%= link_to "Running", jobs_path(status: "claimed", q: @search, period: @period, priority: @priority), class: @status == "claimed" ? "active" : "" %>
|
|
11
|
-
<%= link_to "Blocked", jobs_path(status: "blocked", q: @search, period: @period, priority: @priority), class: @status == "blocked" ? "active" : "" %>
|
|
12
|
-
<%= link_to "Failed", jobs_path(status: "failed", q: @search, period: @period, priority: @priority), class: @status == "failed" ? "active" : "" %>
|
|
9
|
+
<%= link_to "Ready", jobs_path(status: "ready", q: @search, period: @period, priority: @priority, sort: @sort, direction: @direction), class: @status == "ready" ? "active" : "", data: { action: "click->filters#saveStatus", filters_status_param: "ready" } %>
|
|
10
|
+
<%= link_to "Scheduled", jobs_path(status: "scheduled", q: @search, period: @period, priority: @priority, sort: @sort, direction: @direction), class: @status == "scheduled" ? "active" : "", data: { action: "click->filters#saveStatus", filters_status_param: "scheduled" } %>
|
|
11
|
+
<%= link_to "Running", jobs_path(status: "claimed", q: @search, period: @period, priority: @priority, sort: @sort, direction: @direction), class: @status == "claimed" ? "active" : "", data: { action: "click->filters#saveStatus", filters_status_param: "claimed" } %>
|
|
12
|
+
<%= link_to "Blocked", jobs_path(status: "blocked", q: @search, period: @period, priority: @priority, sort: @sort, direction: @direction), class: @status == "blocked" ? "active" : "", data: { action: "click->filters#saveStatus", filters_status_param: "blocked" } %>
|
|
13
|
+
<%= link_to "Failed", jobs_path(status: "failed", q: @search, period: @period, priority: @priority, sort: @sort, direction: @direction), class: @status == "failed" ? "active" : "", data: { action: "click->filters#saveStatus", filters_status_param: "failed" } %>
|
|
13
14
|
</div>
|
|
14
15
|
<% if @jobs.any? %>
|
|
15
16
|
<div class="sqd-actions">
|
|
16
|
-
<%= link_to "Export CSV", jobs_path(format: :csv, status: @status, q: @search, period: @period),
|
|
17
|
+
<%= link_to "Export CSV", jobs_path(format: :csv, status: @status, q: @search, period: @period, sort: @sort, direction: @direction),
|
|
17
18
|
class: "sqd-btn sqd-btn--muted", data: { turbo: false } %>
|
|
18
19
|
<% if @status == "scheduled" %>
|
|
19
20
|
<%= button_to "Run All Now", run_all_now_scheduled_jobs_path,
|
|
@@ -36,6 +37,8 @@
|
|
|
36
37
|
<form class="sqd-search" action="<%= jobs_path %>" method="get" data-controller="search">
|
|
37
38
|
<input type="hidden" name="status" value="<%= @status %>">
|
|
38
39
|
<input type="hidden" name="period" value="<%= @period %>">
|
|
40
|
+
<input type="hidden" name="sort" value="<%= @sort %>">
|
|
41
|
+
<input type="hidden" name="direction" value="<%= @direction %>">
|
|
39
42
|
<input class="sqd-search__input" type="search" name="q" value="<%= @search %>"
|
|
40
43
|
placeholder="Filter by job class…" autocomplete="off" aria-label="Filter by job class"
|
|
41
44
|
data-action="input->search#filter">
|
|
@@ -49,22 +52,25 @@
|
|
|
49
52
|
</select>
|
|
50
53
|
<% end %>
|
|
51
54
|
<% if @search.present? || @priority.present? %>
|
|
52
|
-
<%= link_to "Clear", jobs_path(status: @status, period: @period), class: "sqd-btn sqd-btn--muted" %>
|
|
55
|
+
<%= link_to "Clear", jobs_path(status: @status, period: @period, sort: @sort, direction: @direction), class: "sqd-btn sqd-btn--muted" %>
|
|
53
56
|
<% end %>
|
|
54
57
|
<div class="sqd-period-filter" role="group" aria-label="Time period">
|
|
55
|
-
<%= link_to "All", jobs_path(status: @status, q: @search, priority: @priority), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil }, aria_label: "All time" %>
|
|
56
|
-
<%= link_to "1h", jobs_path(status: @status, q: @search, priority: @priority, period: "1h"), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil }, aria_label: "Last 1 hour" %>
|
|
57
|
-
<%= link_to "24h", jobs_path(status: @status, q: @search, priority: @priority, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours" %>
|
|
58
|
-
<%= link_to "7d", jobs_path(status: @status, q: @search, priority: @priority, period: "7d"), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil }, aria_label: "Last 7 days" %>
|
|
58
|
+
<%= link_to "All", jobs_path(status: @status, q: @search, priority: @priority, sort: @sort, direction: @direction), class: @period.nil? ? "active" : "", aria: { current: @period.nil? ? "true" : nil }, aria_label: "All time", data: { action: "click->filters#savePeriod", filters_period_param: "" } %>
|
|
59
|
+
<%= link_to "1h", jobs_path(status: @status, q: @search, priority: @priority, period: "1h", sort: @sort, direction: @direction), class: @period == "1h" ? "active" : "", aria: { current: @period == "1h" ? "true" : nil }, aria_label: "Last 1 hour", data: { action: "click->filters#savePeriod", filters_period_param: "1h" } %>
|
|
60
|
+
<%= link_to "24h", jobs_path(status: @status, q: @search, priority: @priority, period: "24h", sort: @sort, direction: @direction), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours", data: { action: "click->filters#savePeriod", filters_period_param: "24h" } %>
|
|
61
|
+
<%= link_to "7d", jobs_path(status: @status, q: @search, priority: @priority, period: "7d", sort: @sort, direction: @direction), class: @period == "7d" ? "active" : "", aria: { current: @period == "7d" ? "true" : nil }, aria_label: "Last 7 days", data: { action: "click->filters#savePeriod", filters_period_param: "7d" } %>
|
|
59
62
|
</div>
|
|
60
63
|
</form>
|
|
64
|
+
</div>
|
|
61
65
|
|
|
62
66
|
<% if discardable && @jobs.any? %>
|
|
63
67
|
<div data-controller="selection">
|
|
64
68
|
<%= form_tag job_selection_path, method: :delete, id: "job-selection-form",
|
|
65
69
|
data: { turbo_confirm: "Discard selected jobs? This cannot be undone." } do %>
|
|
66
|
-
<%= hidden_field_tag :status,
|
|
67
|
-
<%= hidden_field_tag :period,
|
|
70
|
+
<%= hidden_field_tag :status, @status %>
|
|
71
|
+
<%= hidden_field_tag :period, @period %>
|
|
72
|
+
<%= hidden_field_tag :sort, @sort %>
|
|
73
|
+
<%= hidden_field_tag :direction, @direction %>
|
|
68
74
|
<% end %>
|
|
69
75
|
|
|
70
76
|
<div class="sqd-selection-bar" data-selection-target="bar" style="display: none;">
|
|
@@ -83,11 +89,12 @@
|
|
|
83
89
|
data-action="change->selection#selectAll"
|
|
84
90
|
aria-label="Select all jobs">
|
|
85
91
|
</th>
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
92
|
+
<% sort_url = ->(p) { jobs_path(status: @status, q: @search, period: @period, priority: @priority, **p) } %>
|
|
93
|
+
<%= sort_header_th("Job Class", "class_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
94
|
+
<%= sort_header_th("Queue", "queue_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
95
|
+
<%= sort_header_th("Priority", "priority", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
89
96
|
<th scope="col">Scheduled At</th>
|
|
90
|
-
|
|
97
|
+
<%= sort_header_th("Enqueued At", "created_at", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
91
98
|
<th scope="col"><span class="sqd-sr-only">Actions</span></th>
|
|
92
99
|
</tr>
|
|
93
100
|
</thead>
|
|
@@ -111,9 +118,9 @@
|
|
|
111
118
|
</td>
|
|
112
119
|
<td><%= job.priority %></td>
|
|
113
120
|
<td id="scheduled_at_<%= execution.id %>" class="sqd-mono">
|
|
114
|
-
<%= job.scheduled_at
|
|
121
|
+
<%= format_timestamp(job.scheduled_at) %>
|
|
115
122
|
</td>
|
|
116
|
-
<td class="sqd-mono"><%= job.created_at
|
|
123
|
+
<td class="sqd-mono"><%= format_timestamp(job.created_at) %></td>
|
|
117
124
|
<td class="sqd-row-actions">
|
|
118
125
|
<% if @status == "scheduled" %>
|
|
119
126
|
<%= button_to "Run Now", scheduled_job_path(execution),
|
|
@@ -130,7 +137,7 @@
|
|
|
130
137
|
<% end %>
|
|
131
138
|
<%= button_to "Discard", job_path(execution),
|
|
132
139
|
method: :delete,
|
|
133
|
-
params: { status: @status, period: @period },
|
|
140
|
+
params: { status: @status, period: @period, sort: @sort, direction: @direction },
|
|
134
141
|
class: "sqd-btn sqd-btn--danger sqd-btn--sm",
|
|
135
142
|
data: { confirm: "Discard this job?" } %>
|
|
136
143
|
</td>
|
|
@@ -149,13 +156,15 @@
|
|
|
149
156
|
<table>
|
|
150
157
|
<thead>
|
|
151
158
|
<tr>
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
159
|
+
<% sort_url = ->(p) { jobs_path(status: @status, q: @search, period: @period, priority: @priority, **p) } %>
|
|
160
|
+
<%= sort_header_th("Job Class", "class_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
161
|
+
<%= sort_header_th("Queue", "queue_name", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
162
|
+
<%= sort_header_th("Priority", "priority", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
155
163
|
<th scope="col">Scheduled At</th>
|
|
156
|
-
|
|
164
|
+
<%= sort_header_th("Enqueued At", "created_at", sort_url, current_sort: @sort, current_dir: @direction) %>
|
|
157
165
|
<% if @status == "claimed" %>
|
|
158
166
|
<th scope="col">Running For</th>
|
|
167
|
+
<th scope="col">Wait Time</th>
|
|
159
168
|
<% end %>
|
|
160
169
|
</tr>
|
|
161
170
|
</thead>
|
|
@@ -177,13 +186,16 @@
|
|
|
177
186
|
</td>
|
|
178
187
|
<td><%= job.priority %></td>
|
|
179
188
|
<td class="sqd-mono">
|
|
180
|
-
<%= job.scheduled_at
|
|
189
|
+
<%= format_timestamp(job.scheduled_at) %>
|
|
181
190
|
</td>
|
|
182
|
-
<td class="sqd-mono"><%= job.created_at
|
|
191
|
+
<td class="sqd-mono"><%= format_timestamp(job.created_at) %></td>
|
|
183
192
|
<% if @status == "claimed" %>
|
|
184
193
|
<td class="sqd-mono<%= slow ? " sqd-slow-duration" : "" %>">
|
|
185
194
|
<%= time_ago_in_words(execution.created_at) %>
|
|
186
195
|
</td>
|
|
196
|
+
<td class="sqd-mono">
|
|
197
|
+
<%= format_duration(execution.created_at - job.created_at) %>
|
|
198
|
+
</td>
|
|
187
199
|
<% end %>
|
|
188
200
|
</tr>
|
|
189
201
|
<% end %>
|
|
@@ -21,12 +21,12 @@
|
|
|
21
21
|
<% @queues.each do |queue| %>
|
|
22
22
|
<tr>
|
|
23
23
|
<td class="sqd-mono"><%= queue.name %></td>
|
|
24
|
-
<td><%= queue.
|
|
24
|
+
<td><%= @queue_sizes[queue.name] || 0 %></td>
|
|
25
25
|
<td>
|
|
26
26
|
<% if (oldest = @oldest_ready[queue.name]) %>
|
|
27
27
|
<% age = Time.current - oldest %>
|
|
28
28
|
<% latency_color = age > 86_400 ? "var(--danger)" : age > 3_600 ? "var(--warning)" : "inherit" %>
|
|
29
|
-
<abbr title="<%= oldest
|
|
29
|
+
<abbr title="<%= format_timestamp(oldest) %>">
|
|
30
30
|
<span style="color: <%= latency_color %>"><%= format_duration(age) %></span>
|
|
31
31
|
</abbr>
|
|
32
32
|
<% else %>
|
|
@@ -52,14 +52,14 @@
|
|
|
52
52
|
<% end %>
|
|
53
53
|
</td>
|
|
54
54
|
<td>
|
|
55
|
-
<% if queue.
|
|
55
|
+
<% if @paused_queue_names.include?(queue.name) %>
|
|
56
56
|
<span class="sqd-badge sqd-badge--paused">Paused</span>
|
|
57
57
|
<% else %>
|
|
58
58
|
<span class="sqd-badge sqd-badge--running">Running</span>
|
|
59
59
|
<% end %>
|
|
60
60
|
</td>
|
|
61
61
|
<td class="sqd-row-actions">
|
|
62
|
-
<% if queue.
|
|
62
|
+
<% if @paused_queue_names.include?(queue.name) %>
|
|
63
63
|
<%= button_to "Resume", queue_pause_path(queue.name), method: :delete,
|
|
64
64
|
class: "sqd-btn sqd-btn--primary sqd-btn--sm" %>
|
|
65
65
|
<% else %>
|
|
@@ -64,9 +64,9 @@
|
|
|
64
64
|
</td>
|
|
65
65
|
<td><%= job.priority %></td>
|
|
66
66
|
<td class="sqd-mono">
|
|
67
|
-
<%= job.scheduled_at
|
|
67
|
+
<%= format_timestamp(job.scheduled_at) %>
|
|
68
68
|
</td>
|
|
69
|
-
<td class="sqd-mono"><%= job.created_at
|
|
69
|
+
<td class="sqd-mono"><%= format_timestamp(job.created_at) %></td>
|
|
70
70
|
<% if discardable %>
|
|
71
71
|
<td class="sqd-row-actions">
|
|
72
72
|
<%= button_to "Discard", queue_job_path(queue_name: @queue, id: execution),
|
|
@@ -43,7 +43,7 @@
|
|
|
43
43
|
<td class="sqd-mono">
|
|
44
44
|
<%
|
|
45
45
|
next_run = begin
|
|
46
|
-
task.next_time
|
|
46
|
+
format_timestamp(task.next_time, format: "%Y-%m-%d %H:%M")
|
|
47
47
|
rescue
|
|
48
48
|
nil
|
|
49
49
|
end
|
|
@@ -52,7 +52,7 @@
|
|
|
52
52
|
</td>
|
|
53
53
|
<td class="sqd-mono">
|
|
54
54
|
<% last_run = task.last_enqueued_time %>
|
|
55
|
-
<%= last_run
|
|
55
|
+
<%= format_timestamp(last_run, format: "%Y-%m-%d %H:%M") %>
|
|
56
56
|
</td>
|
|
57
57
|
<td>
|
|
58
58
|
<% if task.static? %>
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
<% else %>
|
|
4
4
|
<%= turbo_stream.replace "scheduled_at_#{@execution.id}" do %>
|
|
5
5
|
<td id="scheduled_at_<%= @execution.id %>" class="sqd-mono">
|
|
6
|
-
<%= @execution.scheduled_at
|
|
6
|
+
<%= format_timestamp(@execution.scheduled_at) %>
|
|
7
7
|
</td>
|
|
8
8
|
<% end %>
|
|
9
9
|
<% end %>
|
|
@@ -52,7 +52,7 @@
|
|
|
52
52
|
<tr>
|
|
53
53
|
<td><%= link_to job.class_name, job_path(job), class: "sqd-table-link" %></td>
|
|
54
54
|
<td class="sqd-mono"><%= job.queue_name %></td>
|
|
55
|
-
<td class="sqd-mono"><%= job.created_at
|
|
55
|
+
<td class="sqd-mono"><%= format_timestamp(job.created_at) %></td>
|
|
56
56
|
</tr>
|
|
57
57
|
<% end %>
|
|
58
58
|
</tbody>
|
data/config/importmap.rb
CHANGED
|
@@ -3,3 +3,4 @@ pin "solid_queue_web/search_controller", to: "solid_queue_web/search_controller.
|
|
|
3
3
|
pin "solid_queue_web/refresh_controller", to: "solid_queue_web/refresh_controller.js"
|
|
4
4
|
pin "solid_queue_web/selection_controller", to: "solid_queue_web/selection_controller.js"
|
|
5
5
|
pin "solid_queue_web/theme_controller", to: "solid_queue_web/theme_controller.js"
|
|
6
|
+
pin "solid_queue_web/filters_controller", to: "solid_queue_web/filters_controller.js"
|
data/lib/solid_queue_web.rb
CHANGED
|
@@ -6,7 +6,8 @@ module SolidQueueWeb
|
|
|
6
6
|
class << self
|
|
7
7
|
attr_writer :page_size, :dashboard_refresh_interval, :default_refresh_interval, :search_results_limit,
|
|
8
8
|
:slow_job_threshold, :alert_webhook_url, :alert_failure_threshold, :alert_webhook_cooldown,
|
|
9
|
-
:alert_queue_thresholds, :
|
|
9
|
+
:alert_queue_thresholds, :alert_slow_job_count_threshold, :alert_stale_process_threshold,
|
|
10
|
+
:connects_to, :time_zone
|
|
10
11
|
|
|
11
12
|
def page_size
|
|
12
13
|
@page_size || 25
|
|
@@ -44,10 +45,22 @@ module SolidQueueWeb
|
|
|
44
45
|
@alert_queue_thresholds || {}
|
|
45
46
|
end
|
|
46
47
|
|
|
48
|
+
def alert_slow_job_count_threshold
|
|
49
|
+
@alert_slow_job_count_threshold
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def alert_stale_process_threshold
|
|
53
|
+
@alert_stale_process_threshold
|
|
54
|
+
end
|
|
55
|
+
|
|
47
56
|
def connects_to
|
|
48
57
|
@connects_to
|
|
49
58
|
end
|
|
50
59
|
|
|
60
|
+
def time_zone
|
|
61
|
+
@time_zone
|
|
62
|
+
end
|
|
63
|
+
|
|
51
64
|
def configure
|
|
52
65
|
yield self
|
|
53
66
|
end
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: solid_queue_web
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Chuck Smith
|
|
@@ -144,6 +144,7 @@ files:
|
|
|
144
144
|
- app/controllers/solid_queue_web/search_controller.rb
|
|
145
145
|
- app/helpers/solid_queue_web/application_helper.rb
|
|
146
146
|
- app/javascript/solid_queue_web/application.js
|
|
147
|
+
- app/javascript/solid_queue_web/filters_controller.js
|
|
147
148
|
- app/javascript/solid_queue_web/refresh_controller.js
|
|
148
149
|
- app/javascript/solid_queue_web/search_controller.js
|
|
149
150
|
- app/javascript/solid_queue_web/selection_controller.js
|
|
@@ -158,6 +159,8 @@ files:
|
|
|
158
159
|
- app/services/solid_queue_web/metrics_payload.rb
|
|
159
160
|
- app/services/solid_queue_web/queue_depth_alert.rb
|
|
160
161
|
- app/services/solid_queue_web/queue_stats.rb
|
|
162
|
+
- app/services/solid_queue_web/slow_job_alert.rb
|
|
163
|
+
- app/services/solid_queue_web/stale_process_alert.rb
|
|
161
164
|
- app/views/layouts/solid_queue_web/application.html.erb
|
|
162
165
|
- app/views/solid_queue_web/dashboard/index.html.erb
|
|
163
166
|
- app/views/solid_queue_web/failed_jobs/errors/index.html.erb
|