solid_queue_web 0.9.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 462f81d84dead7a68833c768775eda6cb0b55b66862d58fa1b49b504fa417644
-  data.tar.gz: cc388f882d5709e8d92a778a2c6ccaa39d04022c62c9bc7e89320e416c5148d2
+  metadata.gz: 65caaec8c90225e3645116b206b9a3f81c4ff2389b2dd688e2e3fccca8a0121d
+  data.tar.gz: 217dd8f2c0dcdaf13fe69cff9f3bf1c0857a416e88de8643248fc54a85efe917
 SHA512:
-  metadata.gz: '087a8cb99798a1d21df1347008380c58242a98247d48b80cba0865c4bb0e46e800d37a2c21eb887fc516c6c724c5e138ebbc8a517dc62e730b1d2ea08fe6b5d3'
-  data.tar.gz: 3885a114fa3f31db4b949a3b397070734b76c6e447d82bc7cd5e67628163760e03a2b70e90c1ef532143170752430de6ba1dd5a18fa0c52ff379462e90dc8d30
+  metadata.gz: ed20d5b71f733c2e7eea80a249ff42c45094507f129dd69899b5938bc9594f575cb0a5958fc3e94bbf7b28fda0c70256257ba1651738d42dfa5c0661e61a7b73
+  data.tar.gz: d7ab81704638739b1108e666d87797a0f4d21c690d428ec321ce04ae98a3af15e181d783e5a498fd9e892b8b1e8041648d9666c356c1aebc194dbc76173cfdfe

data/README.md

@@ -35,11 +35,12 @@ SolidQueueWeb surfaces all of this in a browser UI available at any route you ch
 
 - **Dashboard** — stat cards showing counts for ready, scheduled, running, blocked, and failed jobs, plus queues, recurring tasks, and processes; "Done (1h)" and "Done (24h)" throughput cards; a "Throughput — Last 12 Hours" bar chart (blue) and a "Queue Depth — Last 12 Hours" bar chart (purple) showing hourly snapshots of active job count; pure CSS, no charting library; auto-refreshes every 5 seconds
 - **Queues** — all queues sorted by name with size; oldest ready job latency (color-coded, with UTC timestamp tooltip); Done (24h) and Failed (24h) throughput counts; a mini 12-bar failure rate sparkline per queue showing failure % per hour over the last 12 hours; pause/resume controls
-- **Jobs** — filterable by status (ready, scheduled, claimed, blocked, failed) and by queue; search by job class name with dynamic auto-submit; time-based period filter (1 h / 24 h / 7 d); discard individual or all jobs; Turbo Frame navigation so only the table updates on filter or search; auto-refreshes every 10 seconds
-- **Failed jobs** — list of failed executions with error details; search by class name; filter by queue; time-based period filter; retry or discard individually or in bulk
+- **Jobs** — filterable by status (ready, scheduled, claimed, blocked, failed), queue, and priority; search by job class name with dynamic auto-submit; time-based period filter (1 h / 24 h / 7 d); discard individual or all jobs; Turbo Frame navigation so only the table updates on filter or search; auto-refreshes every 10 seconds
+- **Scheduled job management** — reschedule a scheduled job to run immediately ("Run Now") or push its `scheduled_at` forward by 1 h, 24 h, or 7 d; Turbo Stream responses update the row in place; "Run All Now" bulk action promotes every scheduled job in the current filtered view in a single operation
+- **Failed jobs** — list of failed executions with error details; search by class name; filter by queue; time-based period filter; retry or discard individually or in bulk; bulk retry with configurable stagger (+5s / +10s / +30s / +1m) to avoid thundering herd on recovery
 - **Job detail** — full arguments, timestamps, blocked-until date, and error backtrace; action buttons based on job status
 - **Queue management** — pause and resume individual queues; queue-scoped job list with status filter, search, and discard
-- **Recurring tasks** — all configured recurring tasks with cron schedule, next run time, last run time, and static/dynamic classification
+- **Recurring tasks** — all configured recurring tasks with cron schedule, next run time, last run time, and static/dynamic classification; "Run Now" button enqueues a task immediately without waiting for its next scheduled run
 - **Processes** — workers, dispatchers, and supervisors with heartbeat health status; auto-refreshes every 10 seconds
 - **Global search** — search across all job statuses at once by class name substring; results grouped by status with match count and direct links to filtered views; native datalist autocomplete pre-populated from all known job classes; auto-submits on selection
 - **Targeted bulk actions** — checkboxes on the jobs and failed jobs lists for selecting individual rows; selection bar shows count and action buttons ("Discard Selected" for jobs, "Retry Selected" / "Discard Selected" for failed jobs); select-all checkbox in the table header
@@ -48,6 +49,9 @@ SolidQueueWeb surfaces all of this in a browser UI available at any route you ch
 - **Dashboard quick actions** — "Retry All Failed" and "Discard All Blocked" cards appear on the dashboard only when the respective count is non-zero; one-click bulk operations with confirm dialogs, keeping the dashboard clean when everything is healthy
 - **CSV export** — "Export CSV" button on the jobs, failed jobs, and history pages downloads all records matching the current filters; columns are tailored per view
 - **Slow job detection** — when `slow_job_threshold` is configured, claimed jobs running longer than the threshold are flagged with an orange row, a "slow" badge, and a "Running For" duration column on the Running tab; a "Slow Jobs" warning card appears on the dashboard with a link to the Running tab
+- **Webhook alerts** — set `alert_webhook_url` and `alert_failure_threshold` to receive a POST request whenever the failed job count meets or exceeds the threshold; fires asynchronously so dashboard performance is unaffected; a configurable cooldown (default 1 h) prevents repeated alerts while the count stays elevated
+- **Performance analytics** — per-job-class statistics at `/jobs/performance` showing run count, average, p50, p95, min, and max duration; sorted by p95 descending so the slowest classes surface first; period filter scopes to 1h / 24h / 7d or all time; each class name links to the filtered History view
+- **Metrics / health endpoint** — `GET /jobs/metrics.json` returns a machine-readable JSON document with job counts, throughput, per-queue depth and pause state, and process health summary; suitable for Prometheus scraping, uptime monitors, or external dashboards; `slow_jobs` count included when `slow_job_threshold` is configured
 
 ## Screenshots
 
@@ -98,6 +102,10 @@ SolidQueueWeb.configure do |config|
   config.default_refresh_interval   = 30_000 # jobs/processes/history auto-refresh in ms (default: 10_000)
   config.search_results_limit       = 10     # max results per status in global search (default: 25)
   config.slow_job_threshold         = 5.minutes # flag claimed jobs running longer than this (default: nil = disabled)
+  config.alert_webhook_url          = "https://hooks.example.com/solid-queue" # POST target (default: nil = disabled)
+  config.alert_failure_threshold    = 10         # fire when failed count >= this (default: nil = disabled)
+  config.alert_webhook_cooldown     = 1800       # seconds between repeated alerts (default: 3600)
+  config.connects_to                = { reading: :reading, writing: :writing } # read replica (default: nil)
 end
 
 SolidQueueWeb.authenticate do
@@ -109,19 +117,103 @@ end
 
 No authentication is enforced by default. When the `authenticate` block returns falsy, HTTP Basic auth is used as a fallback.
 
+## Webhook alerts
+
+Set `alert_webhook_url` and `alert_failure_threshold` to receive a POST request whenever the failed job count meets or exceeds the threshold. This is useful for paging an on-call team or triggering a Slack notification via an incoming webhook.
+
+```ruby
+SolidQueueWeb.configure do |config|
+  config.alert_webhook_url       = "https://hooks.slack.com/services/..."
+  config.alert_failure_threshold = 10    # fire when >= 10 jobs have failed
+  config.alert_webhook_cooldown  = 1800  # don't re-fire for 30 minutes (default: 3600)
+end
+```
+
+The request body is JSON:
+
+```json
+{
+  "event": "failure_threshold_exceeded",
+  "failure_count": 14,
+  "threshold": 10,
+  "fired_at": "2026-05-21T12:34:56Z"
+}
+```
+
+The webhook fires asynchronously in a background thread so dashboard page loads are never delayed. HTTP errors are logged to `Rails.logger` and swallowed. The cooldown window prevents repeated alerts while the count stays elevated — the clock resets on each app restart.
+
+## Metrics endpoint
+
+`GET /jobs/metrics.json` returns a machine-readable JSON document suitable for Prometheus scraping, uptime monitors, or external dashboards. No configuration is required — the endpoint is available as soon as the engine is mounted.
+
+```
+GET /jobs/metrics.json
+```
+
+Example response:
+
+```json
+{
+  "generated_at": "2026-05-21T12:00:00Z",
+  "jobs": {
+    "ready": 12,
+    "scheduled": 8,
+    "claimed": 3,
+    "blocked": 5,
+    "failed": 9
+  },
+  "throughput": {
+    "completed_1h": 15,
+    "completed_24h": 87
+  },
+  "queues": [
+    { "name": "critical", "depth": 2, "paused": false },
+    { "name": "default",  "depth": 4, "paused": false },
+    { "name": "mailers",  "depth": 3, "paused": true  }
+  ],
+  "processes": {
+    "total": 4,
+    "healthy": 4,
+    "stale": 0,
+    "by_kind": { "Dispatcher": 1, "Supervisor": 1, "Worker": 2 }
+  }
+}
+```
+
+When `slow_job_threshold` is configured, a `slow_jobs` integer is also included at the top level.
+
+The endpoint respects the same authentication and `connects_to` settings as the rest of the dashboard. A process is counted as **stale** when its `last_heartbeat_at` is older than `SolidQueue.process_alive_threshold` (default: 5 minutes).
+
+## Read replica support
+
+Set `connects_to` with both `reading:` and `writing:` keys to enable automatic role switching. GET requests are routed to the reading role; POST/DELETE/PATCH requests use the writing role.
+
+```ruby
+SolidQueueWeb.configure do |config|
+  # Route dashboard reads to the replica, writes to primary:
+  config.connects_to = { reading: :reading, writing: :writing }
+end
+```
+
+The role names must match what Solid Queue's models are configured with (set via `SolidQueue.connects_to` in your app). To pin all requests to a single role instead (e.g. to bypass automatic read/write splitting middleware), pass a plain `role:` or `shard:` hash:
+
+```ruby
+config.connects_to = { role: :writing }
+```
+
+When `connects_to` is `nil` (the default), no connection switching occurs and single-database apps are unaffected.
+
 ## Roadmap
 
-Planned features, roughly ordered by priority:  
+Post-1.0 planned features:
 
 **Operations**
-- Scheduled job management — reschedule a job to run immediately, or push its `scheduled_at` forward
-- Bulk retry with delay — retry all failed jobs with a configurable stagger to avoid thundering herd
 - Admin audit log — record who retried or discarded which jobs and when (requires host-app user identity)
+- Failed job retry with modified arguments — edit the arguments JSON from the job detail page before retrying; useful for correcting bad payloads without redeploying
 
-**Infrastructure**
-- Webhook / alert config — POST to a URL when the failure count exceeds a threshold
-- Multi-database support — when Solid Queue runs on a separate database from the host app
-- Read replica support — route dashboard queries to a replica to avoid impacting the primary
+**Notifications**
+- Multiple webhook targets — support an array of `alert_webhook_url` values so alerts can fan out to Slack, PagerDuty, and custom endpoints simultaneously
+- Queue depth alert — fire a webhook when a queue's ready job count exceeds a configurable threshold (complements the existing failure-count alert)
 
 Pull requests for any of these are welcome. See [Contributing](#contributing) below.
 

data/Rakefile

@@ -3,11 +3,13 @@ require "bundler/setup"
 require "bundler/gem_tasks"
 require "rubocop/rake_task"
 require "rspec/core/rake_task"
+require "bundler/audit/task"
 
 RuboCop::RakeTask.new
 RSpec::Core::RakeTask.new(:spec)
+Bundler::Audit::Task.new
 
-task default: [:rubocop, :spec]
+task default: ["bundle:audit:update", "bundle:audit:check", :rubocop, :spec]
 
 namespace :dev do
   def dummy_env

data/app/assets/stylesheets/solid_queue_web/_07_forms.css

@@ -76,6 +76,23 @@
   color: #fff;
 }
 
+.sqd-select {
+  padding: 0.35rem 0.6rem;
+  border: 1px solid var(--border);
+  border-radius: 5px;
+  font-size: 13px;
+  background: var(--surface);
+  color: var(--text);
+  line-height: 1.5;
+  cursor: pointer;
+}
+
+.sqd-select:focus {
+  outline: 2px solid var(--primary);
+  outline-offset: -1px;
+  border-color: var(--primary);
+}
+
 .sqd-period-filter {
   display: flex;
   align-items: center;

data/app/controllers/solid_queue_web/application_controller.rb

@@ -4,12 +4,30 @@ module SolidQueueWeb
   class ApplicationController < ActionController::Base
     include Pagy::Method
 
-    PERIOD_DURATIONS = { "1h" => 1.hour, "24h" => 24.hours, "7d" => 7.days }.freeze
+    PERIOD_DURATIONS   = { "1h" => 1.hour, "24h" => 24.hours, "7d" => 7.days }.freeze
+    STAGGER_INTERVALS  = { "5s" => 5.seconds, "10s" => 10.seconds, "30s" => 30.seconds, "1m" => 1.minute }.freeze
 
     before_action :authenticate!
+    around_action :with_database_connection
 
     private
 
+    def with_database_connection
+      config = SolidQueueWeb.connects_to
+      return yield unless config
+
+      if replica_configured?(config)
+        role = request.get? ? config[:reading] : config[:writing]
+        ActiveRecord::Base.connected_to(role: role) { yield }
+      else
+        ActiveRecord::Base.connected_to(**config) { yield }
+      end
+    end
+
+    def replica_configured?(config)
+      config.key?(:reading) && config.key?(:writing)
+    end
+
     def authenticate!
       return unless (auth = SolidQueueWeb.authenticate)
 

data/app/controllers/solid_queue_web/dashboard_controller.rb

@@ -2,6 +2,7 @@ module SolidQueueWeb
   class DashboardController < ApplicationController
     def index
       @stats = DashboardStats.new
+      AlertWebhook.call(failure_count: @stats.counts[:failed])
     end
   end
 end

data/app/controllers/solid_queue_web/jobs_controller.rb

@@ -1,16 +1,20 @@
 module SolidQueueWeb
   class JobsController < ApplicationController
-    before_action :set_status, only: [:destroy, :discard_selected]
-
     def index
-      @status = params[:status].presence_in(Job::STATUSES) || "ready"
-      @search = params[:q].presence
-      @period = params[:period].presence_in(PERIOD_DURATIONS.keys)
+      @status   = params[:status].presence_in(Job::STATUSES) || "ready"
+      @search   = params[:q].presence
+      @period   = params[:period].presence_in(PERIOD_DURATIONS.keys)
+      @priority = params[:priority].presence
+
       scope = Job::EXECUTION_MODELS[@status].includes(:job)
-      scope = scope.references(:job).where("solid_queue_jobs.class_name LIKE ?", "%#{@search}%") if @search.present?
+      scope = scope.references(:job).where("solid_queue_jobs.class_name LIKE ?", "%#{@search}%")         if @search.present?
       scope = scope.references(:job).where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
+      scope = scope.references(:job).where("solid_queue_jobs.priority = ?", @priority.to_i)              if @priority.present?
       scope = scope.order(created_at: :desc)
 
+      @priority_options = Job::EXECUTION_MODELS[@status].joins(:job)
+        .distinct.pluck("solid_queue_jobs.priority").sort
+
       respond_to do |format|
         format.html { @pagy, @jobs = pagy(scope) }
         format.csv do
@@ -25,11 +29,14 @@ module SolidQueueWeb
       @job = SolidQueue::Job
         .includes(:ready_execution, :scheduled_execution, :claimed_execution, :blocked_execution, :failed_execution)
         .find(params[:id])
-      @execution_status = derive_status(@job)
+      @execution_status = Job.derive_status(@job)
     end
 
     def destroy
-      model = execution_model_for!(@status)
+      @status   = params[:status]
+      @period   = params[:period].presence_in(PERIOD_DURATIONS.keys)
+      @priority = params[:priority].presence
+      model = Job.execution_model_for!(@status)
       if params[:id]
         @execution = model.find(params[:id])
         @execution.discard
@@ -63,29 +70,11 @@ module SolidQueueWeb
       end
     end
 
-    def derive_status(job)
-      return "failed"    if job.failed_execution.present?
-      return "claimed"   if job.claimed_execution.present?
-      return "blocked"   if job.blocked_execution.present?
-      return "ready"     if job.ready_execution.present?
-      return "scheduled" if job.scheduled_execution.present?
-      "finished"
-    end
-
-    def set_status
-      @status = params[:status]
-      @period = params[:period].presence_in(PERIOD_DURATIONS.keys)
-    end
-
     def filtered_scope(model)
       scope = model.includes(:job)
       scope = scope.references(:job).where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
+      scope = scope.references(:job).where("solid_queue_jobs.priority = ?", @priority.to_i)                  if @priority.present?
       scope
     end
-
-    def execution_model_for!(status)
-      raise ArgumentError, "Cannot discard #{status} jobs from this page." unless Job::DISCARDABLE.include?(status)
-      Job::EXECUTION_MODELS[status]
-    end
   end
 end

data/app/controllers/solid_queue_web/metrics_controller.rb

@@ -0,0 +1,7 @@
+module SolidQueueWeb
+  class MetricsController < ApplicationController
+    def index
+      render json: MetricsPayload.new.to_h
+    end
+  end
+end

data/app/controllers/solid_queue_web/performance_controller.rb

@@ -0,0 +1,12 @@
+module SolidQueueWeb
+  class PerformanceController < ApplicationController
+    def index
+      @period = params[:period].presence_in(PERIOD_DURATIONS.keys)
+
+      scope = SolidQueue::Job.where.not(finished_at: nil)
+      scope = scope.where("finished_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
+
+      @rows = JobPerformanceStats.new(scope).rows
+    end
+  end
+end

data/app/controllers/solid_queue_web/recurring_tasks/runs_controller.rb

@@ -0,0 +1,18 @@
+module SolidQueueWeb
+  class RecurringTasks::RunsController < ApplicationController
+    def create
+      task = SolidQueue::RecurringTask.find_by!(key: params[:recurring_task_key])
+      result = task.enqueue(at: Time.current)
+
+      if result
+        redirect_to recurring_tasks_path, notice: "\"#{task.key}\" queued for immediate execution."
+      else
+        redirect_to recurring_tasks_path, alert: "Could not enqueue \"#{task.key}\" — it may have just run."
+      end
+    rescue ActiveRecord::RecordNotFound
+      redirect_to recurring_tasks_path, alert: "Recurring task not found."
+    rescue => e
+      redirect_to recurring_tasks_path, alert: "Could not run task: #{e.message}"
+    end
+  end
+end

data/app/controllers/solid_queue_web/retry_failed_jobs_controller.rb

@@ -5,15 +5,36 @@ module SolidQueueWeb
     def create
       executions = params[:id] ? [SolidQueue::FailedExecution.find(params[:id])] : filtered_scope.to_a
       jobs = executions.map(&:job)
-      SolidQueue::FailedExecution.retry_all(jobs)
+
+      if params[:stagger].present? && executions.size > 1
+        interval = STAGGER_INTERVALS[params[:stagger]]
+        raise ArgumentError, "Invalid stagger interval." unless interval
+        executions.each_with_index do |execution, i|
+          execution.job.update!(scheduled_at: i.zero? ? nil : Time.current + (i * interval))
+          execution.retry
+        end
+      else
+        SolidQueue::FailedExecution.retry_all(jobs)
+      end
       redirect_to failed_jobs_path(queue: @queue, q: @search, period: @period),
-        notice: "#{jobs.size} #{"job".pluralize(jobs.size)} queued for retry."
+        notice: retry_notice(jobs.size)
+    rescue ArgumentError => e
+      redirect_to failed_jobs_path, alert: e.message
     rescue => e
       redirect_to failed_jobs_path, alert: "Could not retry job: #{e.message}"
     end
 
     private
 
+    def retry_notice(count)
+      label = "#{count} #{"job".pluralize(count)}"
+      if params[:stagger].present? && count > 1
+        "#{label} queued for retry, staggered #{params[:stagger]} apart."
+      else
+        "#{label} queued for retry."
+      end
+    end
+
     def set_filter_params
       @queue  = params[:queue].presence
       @search = params[:q].presence

data/app/controllers/solid_queue_web/scheduled_jobs_controller.rb

@@ -0,0 +1,54 @@
+module SolidQueueWeb
+  class ScheduledJobsController < ApplicationController
+    def create
+      @period = params[:period].presence_in(PERIOD_DURATIONS.keys)
+      job_ids = scheduled_scope.pluck("solid_queue_jobs.id")
+
+      SolidQueue::ScheduledExecution.where(job_id: job_ids).update_all(scheduled_at: 1.second.ago)
+      SolidQueue::Job.where(id: job_ids).update_all(scheduled_at: 1.second.ago)
+
+      redirect_to jobs_path(status: "scheduled", period: @period),
+        notice: "#{job_ids.size} #{"job".pluralize(job_ids.size)} scheduled to run immediately."
+    rescue => e
+      redirect_to jobs_path(status: "scheduled", period: @period),
+        alert: "Could not run jobs: #{e.message}"
+    end
+
+    def update
+      @execution = SolidQueue::ScheduledExecution.find(params[:id])
+      @period    = params[:period].presence_in(PERIOD_DURATIONS.keys)
+      @run_now   = params[:offset] == "now"
+      new_time   = resolve_new_time(@execution, params[:offset])
+
+      @execution.update!(scheduled_at: new_time)
+      @execution.job.update!(scheduled_at: new_time)
+
+      respond_to do |format|
+        format.turbo_stream
+        format.html do
+          notice = @run_now ? "Job scheduled to run immediately." : "Job rescheduled by +#{params[:offset]}."
+          redirect_to jobs_path(status: "scheduled", period: @period), notice: notice
+        end
+      end
+    rescue ArgumentError => e
+      redirect_to jobs_path(status: "scheduled"), alert: e.message
+    rescue => e
+      redirect_to jobs_path(status: "scheduled"), alert: "Could not reschedule job: #{e.message}"
+    end
+
+    private
+
+    def scheduled_scope
+      scope = SolidQueue::ScheduledExecution.joins(:job)
+      scope = scope.where("solid_queue_jobs.created_at >= ?", PERIOD_DURATIONS[@period].ago) if @period.present?
+      scope
+    end
+
+    def resolve_new_time(execution, offset)
+      return 1.second.ago if offset == "now"
+      raise ArgumentError, "Invalid offset." unless PERIOD_DURATIONS.key?(offset)
+
+      execution.scheduled_at + PERIOD_DURATIONS[offset]
+    end
+  end
+end

data/app/models/solid_queue_web/job.rb

@@ -1,6 +1,6 @@
 module SolidQueueWeb
   class Job
-    STATUSES = %w[ready scheduled claimed blocked failed].freeze
+    STATUSES    = %w[ready scheduled claimed blocked failed].freeze
     DISCARDABLE = %w[ready scheduled blocked].freeze
     EXECUTION_MODELS = {
       "ready"     => SolidQueue::ReadyExecution,
@@ -9,5 +9,21 @@ module SolidQueueWeb
       "blocked"   => SolidQueue::BlockedExecution,
       "failed"    => SolidQueue::FailedExecution
     }.freeze
+
+    def self.execution_model_for!(status)
+      raise ArgumentError, "Cannot discard #{status} jobs from this page." unless DISCARDABLE.include?(status)
+
+      EXECUTION_MODELS[status]
+    end
+
+    def self.derive_status(job)
+      return "failed"    if job.failed_execution.present?
+      return "claimed"   if job.claimed_execution.present?
+      return "blocked"   if job.blocked_execution.present?
+      return "ready"     if job.ready_execution.present?
+      return "scheduled" if job.scheduled_execution.present?
+
+      "finished"
+    end
   end
 end

data/app/services/solid_queue_web/alert_webhook.rb

@@ -0,0 +1,58 @@
+require "net/http"
+require "json"
+require "uri"
+
+module SolidQueueWeb
+  class AlertWebhook
+    MUTEX = Mutex.new
+
+    class << self
+      def call(failure_count:)
+        return unless configured?
+        return if failure_count < SolidQueueWeb.alert_failure_threshold
+        return unless should_fire?
+
+        Thread.new { post(SolidQueueWeb.alert_webhook_url, failure_count) }
+      end
+
+      def reset!
+        MUTEX.synchronize { @last_fired_at = nil }
+      end
+
+      private
+
+      def configured?
+        SolidQueueWeb.alert_webhook_url.present? && SolidQueueWeb.alert_failure_threshold.present?
+      end
+
+      def should_fire?
+        MUTEX.synchronize do
+          cooldown = SolidQueueWeb.alert_webhook_cooldown
+          return false if @last_fired_at && Time.current - @last_fired_at < cooldown
+
+          @last_fired_at = Time.current
+          true
+        end
+      end
+
+      def post(url_string, failure_count)
+        uri = URI.parse(url_string)
+        payload = JSON.generate(
+          event:         "failure_threshold_exceeded",
+          failure_count: failure_count,
+          threshold:     SolidQueueWeb.alert_failure_threshold,
+          fired_at:      Time.current.iso8601
+        )
+        http = Net::HTTP.new(uri.host, uri.port)
+        http.use_ssl = uri.scheme == "https"
+        http.open_timeout = 5
+        http.read_timeout = 10
+        request = Net::HTTP::Post.new(uri.path.presence || "/", "Content-Type" => "application/json")
+        request.body = payload
+        http.request(request)
+      rescue => e
+        Rails.logger.error("[SolidQueueWeb] Alert webhook failed: #{e.message}")
+      end
+    end
+  end
+end

data/app/services/solid_queue_web/job_performance_stats.rb

@@ -0,0 +1,38 @@
+module SolidQueueWeb
+  class JobPerformanceStats
+    Row = Struct.new(:class_name, :count, :avg, :p50, :p95, :min, :max, keyword_init: true)
+
+    def initialize(scope)
+      @scope = scope
+    end
+
+    def rows
+      grouped = @scope.pluck(:class_name, :created_at, :finished_at)
+        .group_by(&:first)
+
+      grouped.map do |class_name, records|
+        durations = records.map { |_, created, finished| (finished - created).to_f }.sort
+        Row.new(
+          class_name: class_name,
+          count:      durations.size,
+          avg:        mean(durations),
+          p50:        percentile(durations, 50),
+          p95:        percentile(durations, 95),
+          min:        durations.first,
+          max:        durations.last
+        )
+      end.sort_by { |r| -r.p95 }
+    end
+
+    private
+
+    def mean(sorted)
+      sorted.sum / sorted.size
+    end
+
+    def percentile(sorted, pct)
+      idx = [(pct / 100.0 * sorted.size).ceil - 1, 0].max
+      sorted[idx]
+    end
+  end
+end

data/app/services/solid_queue_web/metrics_payload.rb

@@ -0,0 +1,66 @@
+module SolidQueueWeb
+  class MetricsPayload
+    def initialize
+      @now = Time.current
+    end
+
+    def to_h
+      payload = {
+        generated_at: @now.iso8601,
+        jobs:         job_counts,
+        throughput:   throughput,
+        queues:       queue_list,
+        processes:    process_summary
+      }
+      slow = slow_jobs_count
+      payload[:slow_jobs] = slow unless slow.nil?
+      payload
+    end
+
+    private
+
+    def job_counts
+      {
+        ready:     SolidQueue::ReadyExecution.count,
+        scheduled: SolidQueue::ScheduledExecution.count,
+        claimed:   SolidQueue::ClaimedExecution.count,
+        blocked:   SolidQueue::BlockedExecution.count,
+        failed:    SolidQueue::FailedExecution.count
+      }
+    end
+
+    def throughput
+      finished_times = SolidQueue::Job.where(finished_at: 24.hours.ago..@now).pluck(:finished_at)
+      {
+        completed_1h:  finished_times.count { |t| t >= 1.hour.ago },
+        completed_24h: finished_times.size
+      }
+    end
+
+    def queue_list
+      depths = SolidQueue::ReadyExecution
+        .joins(:job)
+        .group("solid_queue_jobs.queue_name")
+        .count
+      SolidQueue::Queue.all.sort_by(&:name).map do |q|
+        { name: q.name, depth: depths[q.name] || 0, paused: q.paused? }
+      end
+    end
+
+    def process_summary
+      processes = SolidQueue::Process.all.to_a
+      threshold = SolidQueue.process_alive_threshold.ago
+      {
+        total:   processes.size,
+        healthy: processes.count { |p| p.last_heartbeat_at >= threshold },
+        stale:   processes.count { |p| p.last_heartbeat_at < threshold },
+        by_kind: processes.group_by(&:kind).transform_values(&:size)
+      }
+    end
+
+    def slow_jobs_count
+      threshold = SolidQueueWeb.slow_job_threshold
+      SolidQueue::ClaimedExecution.where("created_at <= ?", threshold.ago).count if threshold
+    end
+  end
+end

data/app/views/layouts/solid_queue_web/application.html.erb

@@ -21,6 +21,7 @@
           <li><%= link_to "Queues", queues_path, class: current_page?(queues_path) ? "active" : "", aria: { current: current_page?(queues_path) ? "page" : nil } %></li>
           <li><%= link_to "Jobs", jobs_path, class: current_page?(jobs_path) ? "active" : "", aria: { current: current_page?(jobs_path) ? "page" : nil } %></li>
           <li><%= link_to "History", history_path, class: current_page?(history_path) ? "active" : "", aria: { current: current_page?(history_path) ? "page" : nil } %></li>
+          <li><%= link_to "Performance", performance_path, class: current_page?(performance_path) ? "active" : "", aria: { current: current_page?(performance_path) ? "page" : nil } %></li>
           <li><%= link_to "Failed", failed_jobs_path, class: current_page?(failed_jobs_path) ? "active" : "", aria: { current: current_page?(failed_jobs_path) ? "page" : nil } %></li>
           <li><%= link_to "Recurring", recurring_tasks_path, class: current_page?(recurring_tasks_path) ? "active" : "", aria: { current: current_page?(recurring_tasks_path) ? "page" : nil } %></li>
           <li><%= link_to "Processes", processes_path, class: current_page?(processes_path) ? "active" : "", aria: { current: current_page?(processes_path) ? "page" : nil } %></li>

data/app/views/solid_queue_web/failed_jobs/index.html.erb

@@ -9,6 +9,16 @@
             params: { queue: @queue, q: @search, period: @period },
             class: "sqd-btn sqd-btn--primary",
             data: { confirm: "Retry all #{@failed_jobs.size} failed jobs?" } %>
+      <% if @failed_jobs.size > 1 %>
+        <% %w[5s 10s 30s 1m].each do |interval| %>
+          <%= button_to "+#{interval}", retry_all_failed_jobs_path,
+                method: :post,
+                params: { stagger: interval, queue: @queue, q: @search, period: @period },
+                class: "sqd-btn sqd-btn--muted sqd-btn--sm",
+                title: "Retry all, staggered #{interval} apart",
+                data: { confirm: "Retry #{@failed_jobs.size} failed jobs staggered #{interval} apart?" } %>
+        <% end %>
+      <% end %>
       <%= button_to "Discard All", discard_all_failed_jobs_path,
             method: :post,
             params: { queue: @queue, q: @search, period: @period },

data/app/views/solid_queue_web/jobs/index.html.erb

@@ -5,22 +5,29 @@
 
 <div class="sqd-page-header">
   <div class="sqd-filters">
-    <%= link_to "Ready",     jobs_path(status: "ready",     q: @search, period: @period), class: @status == "ready"     ? "active" : "" %>
-    <%= link_to "Scheduled", jobs_path(status: "scheduled", q: @search, period: @period), class: @status == "scheduled" ? "active" : "" %>
-    <%= link_to "Running",   jobs_path(status: "claimed",   q: @search, period: @period), class: @status == "claimed"   ? "active" : "" %>
-    <%= link_to "Blocked",   jobs_path(status: "blocked",   q: @search, period: @period), class: @status == "blocked"   ? "active" : "" %>
-    <%= link_to "Failed",    jobs_path(status: "failed",    q: @search, period: @period), class: @status == "failed"    ? "active" : "" %>
+    <%= link_to "Ready",     jobs_path(status: "ready",     q: @search, period: @period, priority: @priority), class: @status == "ready"     ? "active" : "" %>
+    <%= link_to "Scheduled", jobs_path(status: "scheduled", q: @search, period: @period, priority: @priority), class: @status == "scheduled" ? "active" : "" %>
+    <%= link_to "Running",   jobs_path(status: "claimed",   q: @search, period: @period, priority: @priority), class: @status == "claimed"   ? "active" : "" %>
+    <%= link_to "Blocked",   jobs_path(status: "blocked",   q: @search, period: @period, priority: @priority), class: @status == "blocked"   ? "active" : "" %>
+    <%= link_to "Failed",    jobs_path(status: "failed",    q: @search, period: @period, priority: @priority), class: @status == "failed"    ? "active" : "" %>
   </div>
   <% if @jobs.any? %>
     <div class="sqd-actions">
       <%= link_to "Export CSV", jobs_path(format: :csv, status: @status, q: @search, period: @period),
             class: "sqd-btn sqd-btn--muted", data: { turbo: false } %>
+      <% if @status == "scheduled" %>
+        <%= button_to "Run All Now", run_all_now_scheduled_jobs_path,
+              method: :post,
+              params: { period: @period },
+              class: "sqd-btn sqd-btn--primary",
+              data: { confirm: "Run all #{@pagy.count} scheduled jobs immediately?" } %>
+      <% end %>
       <% if discardable %>
         <%= button_to "Discard All", discard_all_jobs_path,
               method: :post,
               params: { status: @status, period: @period },
               class: "sqd-btn sqd-btn--danger",
-              data: { confirm: "Discard all #{@jobs.size} #{@status} jobs? This cannot be undone." } %>
+              data: { confirm: "Discard all #{@pagy.count} #{@status} jobs? This cannot be undone." } %>
       <% end %>
     </div>
   <% end %>
@@ -32,14 +39,23 @@
   <input class="sqd-search__input" type="search" name="q" value="<%= @search %>"
          placeholder="Filter by job class…" autocomplete="off" aria-label="Filter by job class"
          data-action="input->search#filter">
-  <% if @search.present? %>
+  <% if @priority_options.size > 1 %>
+    <select name="priority" class="sqd-select" aria-label="Filter by priority"
+            onchange="this.form.submit()">
+      <option value="" <%= @priority.nil? ? "selected" : "" %>>All priorities</option>
+      <% @priority_options.each do |p| %>
+        <option value="<%= p %>" <%= @priority.to_s == p.to_s ? "selected" : "" %>>Priority <%= p %></option>
+      <% end %>
+    </select>
+  <% end %>
+  <% if @search.present? || @priority.present? %>
     <%= link_to "Clear", jobs_path(status: @status, period: @period), class: "sqd-btn sqd-btn--muted" %>
   <% end %>
   <div class="sqd-period-filter" role="group" aria-label="Time period">
-    <%= link_to "All", jobs_path(status: @status, q: @search),                       class: @period.nil?       ? "active" : "", aria: { current: @period.nil?       ? "true" : nil }, aria_label: "All time" %>
-    <%= link_to "1h",  jobs_path(status: @status, q: @search, period: "1h"),  class: @period == "1h"  ? "active" : "", aria: { current: @period == "1h"  ? "true" : nil }, aria_label: "Last 1 hour" %>
-    <%= link_to "24h", jobs_path(status: @status, q: @search, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours" %>
-    <%= link_to "7d",  jobs_path(status: @status, q: @search, period: "7d"),  class: @period == "7d"  ? "active" : "", aria: { current: @period == "7d"  ? "true" : nil }, aria_label: "Last 7 days" %>
+    <%= link_to "All", jobs_path(status: @status, q: @search, priority: @priority),                       class: @period.nil?       ? "active" : "", aria: { current: @period.nil?       ? "true" : nil }, aria_label: "All time" %>
+    <%= link_to "1h",  jobs_path(status: @status, q: @search, priority: @priority, period: "1h"),  class: @period == "1h"  ? "active" : "", aria: { current: @period == "1h"  ? "true" : nil }, aria_label: "Last 1 hour" %>
+    <%= link_to "24h", jobs_path(status: @status, q: @search, priority: @priority, period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil }, aria_label: "Last 24 hours" %>
+    <%= link_to "7d",  jobs_path(status: @status, q: @search, priority: @priority, period: "7d"),  class: @period == "7d"  ? "active" : "", aria: { current: @period == "7d"  ? "true" : nil }, aria_label: "Last 7 days" %>
   </div>
 </form>
 
@@ -94,11 +110,24 @@
                       class: "sqd-mono", style: "color: inherit;" %>
               </td>
               <td><%= job.priority %></td>
-              <td class="sqd-mono">
+              <td id="scheduled_at_<%= execution.id %>" class="sqd-mono">
                 <%= job.scheduled_at ? job.scheduled_at.strftime("%Y-%m-%d %H:%M:%S") : "—" %>
               </td>
               <td class="sqd-mono"><%= job.created_at.strftime("%Y-%m-%d %H:%M:%S") %></td>
               <td class="sqd-row-actions">
+                <% if @status == "scheduled" %>
+                  <%= button_to "Run Now", scheduled_job_path(execution),
+                        method: :patch,
+                        params: { offset: "now", period: @period },
+                        class: "sqd-btn sqd-btn--primary sqd-btn--sm",
+                        data: { confirm: "Run this job immediately?" } %>
+                  <% %w[1h 24h 7d].each do |offset| %>
+                    <%= button_to "+#{offset}", scheduled_job_path(execution),
+                          method: :patch,
+                          params: { offset: offset, period: @period },
+                          class: "sqd-btn sqd-btn--muted sqd-btn--sm" %>
+                  <% end %>
+                <% end %>
                 <%= button_to "Discard", job_path(execution),
                       method: :delete,
                       params: { status: @status, period: @period },

data/app/views/solid_queue_web/performance/index.html.erb

@@ -0,0 +1,50 @@
+<div class="sqd-page-header">
+  <h1 class="sqd-page-title">Performance</h1>
+</div>
+
+<form class="sqd-search" action="<%= performance_path %>" method="get">
+  <div class="sqd-period-filter" role="group" aria-label="Time period">
+    <%= link_to "All",  performance_path,                      class: @period.nil?       ? "active" : "", aria: { current: @period.nil?       ? "true" : nil } %>
+    <%= link_to "1h",   performance_path(period: "1h"),  class: @period == "1h"  ? "active" : "", aria: { current: @period == "1h"  ? "true" : nil } %>
+    <%= link_to "24h",  performance_path(period: "24h"), class: @period == "24h" ? "active" : "", aria: { current: @period == "24h" ? "true" : nil } %>
+    <%= link_to "7d",   performance_path(period: "7d"),  class: @period == "7d"  ? "active" : "", aria: { current: @period == "7d"  ? "true" : nil } %>
+  </div>
+</form>
+
+<% if @rows.any? %>
+  <div class="sqd-card" style="margin-top: 1rem;">
+    <table>
+      <thead>
+        <tr>
+          <th scope="col">Job Class</th>
+          <th scope="col" style="text-align: right;">Runs</th>
+          <th scope="col" style="text-align: right;">Avg</th>
+          <th scope="col" style="text-align: right;">p50</th>
+          <th scope="col" style="text-align: right;">p95</th>
+          <th scope="col" style="text-align: right;">Min</th>
+          <th scope="col" style="text-align: right;">Max</th>
+        </tr>
+      </thead>
+      <tbody>
+        <% @rows.each do |row| %>
+          <tr>
+            <td>
+              <%= link_to row.class_name, history_path(q: row.class_name, period: @period),
+                    class: "sqd-table-link" %>
+            </td>
+            <td class="sqd-mono" style="text-align: right;"><%= row.count %></td>
+            <td class="sqd-mono" style="text-align: right;"><%= format_duration(row.avg) %></td>
+            <td class="sqd-mono" style="text-align: right;"><%= format_duration(row.p50) %></td>
+            <td class="sqd-mono" style="text-align: right;"><%= format_duration(row.p95) %></td>
+            <td class="sqd-mono" style="text-align: right;"><%= format_duration(row.min) %></td>
+            <td class="sqd-mono" style="text-align: right;"><%= format_duration(row.max) %></td>
+          </tr>
+        <% end %>
+      </tbody>
+    </table>
+  </div>
+<% else %>
+  <div class="sqd-card" style="margin-top: 1rem;">
+    <div class="sqd-empty">No finished jobs found<%= " in the last #{@period}" if @period %>.</div>
+  </div>
+<% end %>

data/app/views/solid_queue_web/recurring_tasks/index.html.erb

@@ -14,6 +14,7 @@
           <th scope="col">Next Run</th>
           <th scope="col">Last Run</th>
           <th scope="col">Type</th>
+          <th scope="col"><span class="sqd-sr-only">Actions</span></th>
         </tr>
       </thead>
       <tbody>
@@ -60,6 +61,12 @@
                 <span class="sqd-badge sqd-badge--dynamic">Dynamic</span>
               <% end %>
             </td>
+            <td class="sqd-row-actions">
+              <%= button_to "Run Now", recurring_task_run_path(task.key),
+                    method: :post,
+                    class: "sqd-btn sqd-btn--primary sqd-btn--sm",
+                    data: { confirm: "Run \"#{task.key}\" immediately?" } %>
+            </td>
           </tr>
         <% end %>
       </tbody>

data/app/views/solid_queue_web/scheduled_jobs/update.turbo_stream.erb

@@ -0,0 +1,9 @@
+<% if @run_now %>
+  <%= turbo_stream.remove "execution_#{@execution.id}" %>
+<% else %>
+  <%= turbo_stream.replace "scheduled_at_#{@execution.id}" do %>
+    <td id="scheduled_at_<%= @execution.id %>" class="sqd-mono">
+      <%= @execution.scheduled_at.strftime("%Y-%m-%d %H:%M:%S") %>
+    </td>
+  <% end %>
+<% end %>

data/config/routes.rb

@@ -2,10 +2,14 @@ SolidQueueWeb::Engine.routes.draw do
   root to: "dashboard#index"
   resource :blocked_jobs, only: [:destroy]
 
-  get "search", to: "search#index", as: :search
-  get "history", to: "history#index", as: :history
+  get "metrics", to: "metrics#index", as: :metrics, defaults: { format: :json }
+  get "search",      to: "search#index",      as: :search
+  get "history",     to: "history#index",     as: :history
+  get "performance", to: "performance#index", as: :performance
 
-  resources :recurring_tasks, only: [:index]
+  resources :recurring_tasks, only: [:index], param: :key do
+    resource :run, only: [:create], controller: "recurring_tasks/runs"
+  end
   resources :processes, only: [:index]
   resources :queues, only: [:index], param: :name do
     resource :pause, only: [:create, :destroy], controller: "queues/pauses"
@@ -18,6 +22,12 @@ SolidQueueWeb::Engine.routes.draw do
 
   # Singular selection resources must be defined before the member routes of their
   # parent resources, otherwise DELETE /list/selection matches /list/:id first.
+  resources :scheduled_jobs, only: [:update] do
+    collection do
+      post :run_all_now, action: :create
+    end
+  end
+
   resource :job_selection, path: "list/selection", only: [:destroy], controller: "jobs/selections"
   resources :jobs, path: "list", only: [:index, :show, :destroy] do
     collection do

data/lib/solid_queue_web/version.rb

@@ -1,3 +1,3 @@
 module SolidQueueWeb
-  VERSION = "0.9.0"
+  VERSION = "1.0.0"
 end

data/lib/solid_queue_web.rb

@@ -5,7 +5,8 @@ require "solid_queue_web/engine"
 module SolidQueueWeb
   class << self
     attr_writer :page_size, :dashboard_refresh_interval, :default_refresh_interval, :search_results_limit,
-                :slow_job_threshold
+                :slow_job_threshold, :alert_webhook_url, :alert_failure_threshold, :alert_webhook_cooldown,
+                :connects_to
 
     def page_size
       @page_size || 25
@@ -27,6 +28,22 @@ module SolidQueueWeb
       @slow_job_threshold
     end
 
+    def alert_webhook_url
+      @alert_webhook_url
+    end
+
+    def alert_failure_threshold
+      @alert_failure_threshold
+    end
+
+    def alert_webhook_cooldown
+      @alert_webhook_cooldown || 3600
+    end
+
+    def connects_to
+      @connects_to
+    end
+
     def configure
       yield self
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue_web
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Chuck Smith
@@ -126,12 +126,16 @@ files:
 - app/controllers/solid_queue_web/history_controller.rb
 - app/controllers/solid_queue_web/jobs/selections_controller.rb
 - app/controllers/solid_queue_web/jobs_controller.rb
+- app/controllers/solid_queue_web/metrics_controller.rb
+- app/controllers/solid_queue_web/performance_controller.rb
 - app/controllers/solid_queue_web/processes_controller.rb
 - app/controllers/solid_queue_web/queues/jobs_controller.rb
 - app/controllers/solid_queue_web/queues/pauses_controller.rb
 - app/controllers/solid_queue_web/queues_controller.rb
+- app/controllers/solid_queue_web/recurring_tasks/runs_controller.rb
 - app/controllers/solid_queue_web/recurring_tasks_controller.rb
 - app/controllers/solid_queue_web/retry_failed_jobs_controller.rb
+- app/controllers/solid_queue_web/scheduled_jobs_controller.rb
 - app/controllers/solid_queue_web/search_controller.rb
 - app/helpers/solid_queue_web/application_helper.rb
 - app/javascript/solid_queue_web/application.js
@@ -142,7 +146,10 @@ files:
 - app/jobs/solid_queue_web/application_job.rb
 - app/models/solid_queue_web/application_record.rb
 - app/models/solid_queue_web/job.rb
+- app/services/solid_queue_web/alert_webhook.rb
 - app/services/solid_queue_web/dashboard_stats.rb
+- app/services/solid_queue_web/job_performance_stats.rb
+- app/services/solid_queue_web/metrics_payload.rb
 - app/services/solid_queue_web/queue_stats.rb
 - app/views/layouts/solid_queue_web/application.html.erb
 - app/views/solid_queue_web/dashboard/index.html.erb
@@ -151,11 +158,13 @@ files:
 - app/views/solid_queue_web/jobs/destroy.turbo_stream.erb
 - app/views/solid_queue_web/jobs/index.html.erb
 - app/views/solid_queue_web/jobs/show.html.erb
+- app/views/solid_queue_web/performance/index.html.erb
 - app/views/solid_queue_web/processes/index.html.erb
 - app/views/solid_queue_web/queues/index.html.erb
 - app/views/solid_queue_web/queues/jobs/destroy.turbo_stream.erb
 - app/views/solid_queue_web/queues/jobs/index.html.erb
 - app/views/solid_queue_web/recurring_tasks/index.html.erb
+- app/views/solid_queue_web/scheduled_jobs/update.turbo_stream.erb
 - app/views/solid_queue_web/search/index.html.erb
 - config/importmap.rb
 - config/routes.rb