queue_pulse 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ce1008bbb8305bb0a7c884df8ca88bcd44484c7f177ec1a400eee970c8387721
+  data.tar.gz: f7e29e0722bd4278169e254651b5266dbda28b77248cd928579b7318a546cfca
+SHA512:
+  metadata.gz: 54437eb5f66dba1c5ea6db4d3208c1b18288c71656690a6f8212bb2a9d67a99d393b4e87da6e668425f5309b42326be3b90d40ddb477c64c5e2eb7d0f1ef1970
+  data.tar.gz: d75c324f3bc757080309e7b053365ebb79ff1fa77b4c94f2bf2a549d43d5743464f9dd54f3c80f294538dc55cb9a434a9a64e5f84c4aa61f9ea6823cd2a1914a

data/CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+All notable changes to this project are documented here.
+This project adheres to [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-06-09
+Initial open-source release (Phase 1 — the free core).
+
+### Added
+- Read-only monitoring of Solid Queue with **no migration and no extra service**.
+- Five checks: job failures (with burst-collapsing), queue latency, queue depth,
+  stuck (long-running) jobs, and worker-liveness (dead-worker) detection.
+- Notifiers: Slack, generic Webhook, Email (ActionMailer), plus a Test notifier.
+- Alert deduplication via a failed-execution high-water mark and per-condition cooldowns,
+  backed by `Rails.cache` (with an in-process fallback).
+- Execution via `QueuePulse.check!`, `QueuePulse::CheckJob` (for Solid Queue recurring tasks),
+  a `queue_pulse:check` rake task, and an opt-in in-process poller.
+- Full exception isolation: a failing check or notifier never propagates to the host app.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 QueuePulse
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,105 @@
+# QueuePulse
+
+**Know the moment your Solid Queue jobs break — without running Datadog.**
+
+[Solid Queue](https://github.com/rails/solid_queue) is the default Active Job backend in Rails 8.
+Its dashboard, [Mission Control Jobs](https://github.com/rails/mission_control-jobs), is great for
+*looking* at jobs — but it has **no alerting** and **no historical metrics**. So today you either
+find out your jobs are failing when a customer emails you, or you bolt on a heavyweight APM
+(Datadog/New Relic) that's overkill for a small app.
+
+QueuePulse fills that gap. It reads Solid Queue's existing tables (read-only, **no migration, no extra
+service**) and pings you the moment something goes wrong:
+
+- 🔴 **Job failures** — alerted the instant a job lands in `failed_executions`
+- 🟠 **Queue latency** — oldest job has been waiting too long (work is backing up)
+- 🟠 **Queue depth** — a queue is piling up beyond your threshold
+- 🟠 **Stuck jobs** — a job has been running far longer than it should
+- 🔴 **Dead workers** — no worker/dispatcher heartbeat = nothing is processing jobs
+
+Delivered to **Slack, email, or any webhook**.
+
+> Status: early. The free gem (this repo) is the open-source core. A hosted dashboard with
+> historical metrics and AI failure summaries is on the roadmap —
+> **[join the waitlist »](https://queue-pulse.vercel.app)**.
+
+## Install
+
+```ruby
+# Gemfile
+gem "queue_pulse"
+```
+
+```bash
+bundle install
+```
+
+## Configure
+
+```ruby
+# config/initializers/queue_pulse.rb
+QueuePulse.configure do |config|
+  config.add_notifier QueuePulse::Notifiers::Slack.new(webhook_url: ENV["SLACK_WEBHOOK_URL"])
+
+  # Optional — every setting has a sensible default:
+  config.queue_latency_threshold = 300   # seconds a job may wait before alerting
+  config.queue_depth_threshold   = 1_000 # ready jobs per queue before alerting
+  config.stuck_job_threshold     = 600   # seconds a running job may take before alerting
+  config.alert_cooldown          = 900   # seconds before re-alerting the same condition
+  config.environment_label       = Rails.env
+end
+```
+
+### Run the checks
+
+Schedule it with Solid Queue's own recurring tasks (recommended):
+
+```yaml
+# config/recurring.yml
+queue_pulse_check:
+  class: QueuePulse::CheckJob
+  schedule: every minute
+```
+
+Or run on demand:
+
+```bash
+bin/rails queue_pulse:check
+```
+
+Or run an in-process poller (opt-in):
+
+```ruby
+QueuePulse.start_poller!  # checks every config.poll_interval seconds
+```
+
+## Notifiers
+
+```ruby
+QueuePulse::Notifiers::Slack.new(webhook_url: "https://hooks.slack.com/...")
+QueuePulse::Notifiers::Webhook.new(url: "https://example.com/hook", headers: { "Authorization" => "Bearer ..." })
+QueuePulse::Notifiers::Email.new(to: "ops@example.com")
+```
+
+Add your own by subclassing `QueuePulse::Notifiers::Base` and implementing `#deliver(alerts)`.
+
+## Design principles
+
+- **No migration, no extra service.** Reads Solid Queue tables directly; dedupe state lives in `Rails.cache`.
+- **Safe by default.** Read-only queries, every check and notifier is exception-isolated — QueuePulse can never take down your app. Raw job arguments are never sent unless you opt in.
+- **Quiet.** Cooldowns and burst-collapsing mean you get signal, not spam.
+
+## Requirements
+
+Ruby ≥ 3.1, Rails ≥ 7.1, solid_queue ≥ 1.0.
+
+## Development
+
+```bash
+bundle install
+rake test
+```
+
+## License
+
+MIT — see [LICENSE.txt](LICENSE.txt).

data/lib/queue_pulse/alert.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  # An immutable description of something worth notifying about.
+  #
+  # fingerprint is the dedupe/cooldown key: alerts sharing a fingerprint are
+  # considered "the same ongoing condition" and are rate-limited together.
+  class Alert
+    SEVERITIES = %i[info warning critical].freeze
+
+    attr_reader :type, :severity, :title, :message, :context, :fingerprint
+
+    def initialize(type:, severity:, title:, message:, fingerprint:, context: {})
+      @type = type.to_sym
+      @severity = severity.to_sym
+      unless SEVERITIES.include?(@severity)
+        raise ArgumentError, "severity must be one of #{SEVERITIES.inspect}"
+      end
+
+      @title = title
+      @message = message
+      @context = context.freeze
+      @fingerprint = fingerprint
+      freeze
+    end
+
+    def to_h
+      {
+        type: type,
+        severity: severity,
+        title: title,
+        message: message,
+        context: context,
+        fingerprint: fingerprint
+      }
+    end
+
+    SEVERITY_EMOJI = { info: "🔵", warning: "🟠", critical: "🔴" }.freeze
+
+    def emoji
+      SEVERITY_EMOJI.fetch(severity, "⚪️")
+    end
+  end
+end

data/lib/queue_pulse/check_job.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  # Schedule this via Solid Queue recurring tasks to run checks periodically:
+  #
+  #   # config/recurring.yml
+  #   queue_pulse_check:
+  #     class: QueuePulse::CheckJob
+  #     schedule: every minute
+  class CheckJob < ActiveJob::Base
+    queue_as :queue_pulse
+
+    def perform
+      QueuePulse.check!
+    end
+  end
+end

data/lib/queue_pulse/checks/base.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Checks are pure-ish: given (config, source, state) they return Alerts.
+    # The only side effect they own is updating dedupe/cooldown state at the
+    # moment they decide to emit (keeps cooldown logic next to the data that
+    # produces it). Delivery is the Monitor's job.
+    class Base
+      def initialize(config, source, state)
+        @config = config
+        @source = source
+        @state = state
+      end
+
+      # @return [Array<QueuePulse::Alert>]
+      def call
+        raise NotImplementedError
+      end
+
+      private
+
+      attr_reader :config, :source, :state
+
+      # Emit only if this fingerprint is outside its cooldown window, marking it
+      # so it won't re-alert until the window passes (Requirements US-3.3/3.4).
+      def with_cooldown(fingerprint)
+        return nil if state.alerted_recently?(fingerprint)
+
+        alert = yield
+        state.mark_alerted(fingerprint, config.alert_cooldown) if alert
+        alert
+      end
+
+      def env_suffix
+        config.environment_label ? " [#{config.environment_label}]" : ""
+      end
+    end
+  end
+end

data/lib/queue_pulse/checks/failures.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Detects newly failed jobs since the last pass using a high-water mark on
+    # solid_queue_failed_executions.id. On first ever run it establishes the
+    # baseline silently (so installing the gem doesn't replay history).
+    # Above failure_burst_threshold it collapses into a single summary alert.
+    # (Requirements US-2.)
+    class Failures < Base
+      def call
+        hwm = state.failed_high_water_mark
+
+        if hwm.nil?
+          baseline = source.max_failed_execution_id || 0
+          state.set_failed_high_water_mark(baseline)
+          return []
+        end
+
+        burst = config.failure_burst_threshold
+        recent = source.failed_executions_since(hwm, limit: burst + 1)
+        return [] if recent.empty?
+
+        new_hwm = recent.map(&:id).max
+
+        if recent.size > burst
+          total = source.failed_count_since(hwm)
+          state.set_failed_high_water_mark(source.max_failed_execution_id || new_hwm)
+          [burst_alert(total)]
+        else
+          state.set_failed_high_water_mark(new_hwm)
+          recent.map { |f| failure_alert(f) }
+        end
+      end
+
+      private
+
+      def failure_alert(failure)
+        Alert.new(
+          type: :job_failed,
+          severity: :critical,
+          title: "Job failed: #{failure.job_class}#{env_suffix}",
+          message: "#{failure.job_class} on queue \"#{failure.queue_name}\" failed: #{failure.error_message}",
+          fingerprint: "job_failed:#{failure.id}",
+          context: {
+            job_class: failure.job_class,
+            queue_name: failure.queue_name,
+            error: failure.error_message,
+            failed_at: failure.failed_at
+          }
+        )
+      end
+
+      def burst_alert(total)
+        Alert.new(
+          type: :job_failure_burst,
+          severity: :critical,
+          title: "#{total} jobs failed#{env_suffix}",
+          message: "#{total} background jobs failed since the last check. " \
+                   "Open Solid Queue / Mission Control to inspect.",
+          fingerprint: "job_failure_burst",
+          context: { count: total }
+        )
+      end
+    end
+  end
+end

data/lib/queue_pulse/checks/queue_depth.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Alerts when a queue's ready depth exceeds queue_depth_threshold.
+    # Cooldown is per-queue. (Requirements US-3.2, US-3.3.)
+    class QueueDepth < Base
+      def call
+        threshold = config.queue_depth_threshold
+
+        source.queue_stats.filter_map do |stat|
+          next if stat.depth < threshold
+
+          with_cooldown("queue_depth:#{stat.queue_name}") do
+            depth_alert(stat)
+          end
+        end
+      end
+
+      private
+
+      def depth_alert(stat)
+        Alert.new(
+          type: :queue_depth,
+          severity: :warning,
+          title: "Queue \"#{stat.queue_name}\" is backing up#{env_suffix}",
+          message: "\"#{stat.queue_name}\" has #{stat.depth} jobs waiting " \
+                   "(threshold #{config.queue_depth_threshold}).",
+          fingerprint: "queue_depth:#{stat.queue_name}",
+          context: { queue_name: stat.queue_name, depth: stat.depth }
+        )
+      end
+    end
+  end
+end

data/lib/queue_pulse/checks/queue_latency.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Alerts when the oldest ready job in a queue has been waiting longer than
+    # queue_latency_threshold — i.e. work is backing up before users feel it.
+    # Cooldown is per-queue. (Requirements US-3.1, US-3.3, US-3.4.)
+    class QueueLatency < Base
+      def call
+        threshold = config.queue_latency_threshold
+        now = source.now
+
+        source.queue_stats.filter_map do |stat|
+          next unless stat.oldest_enqueued_at
+
+          latency = (now - stat.oldest_enqueued_at).to_i
+          next if latency < threshold
+
+          with_cooldown("queue_latency:#{stat.queue_name}") do
+            latency_alert(stat, latency)
+          end
+        end
+      end
+
+      private
+
+      def latency_alert(stat, latency)
+        Alert.new(
+          type: :queue_latency,
+          severity: :warning,
+          title: "Queue \"#{stat.queue_name}\" is delayed#{env_suffix}",
+          message: "Oldest job in \"#{stat.queue_name}\" has waited #{latency}s " \
+                   "(threshold #{config.queue_latency_threshold}s), #{stat.depth} jobs waiting.",
+          fingerprint: "queue_latency:#{stat.queue_name}",
+          context: { queue_name: stat.queue_name, latency_seconds: latency, depth: stat.depth }
+        )
+      end
+    end
+  end
+end

data/lib/queue_pulse/checks/stuck_jobs.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Alerts on jobs that have been claimed (running) longer than
+    # stuck_job_threshold. Cooldown is per job_id so each stuck job alerts once
+    # per window. (Requirements US-4.)
+    class StuckJobs < Base
+      def call
+        threshold = config.stuck_job_threshold
+        now = source.now
+
+        source.stuck_claimed_executions(older_than: threshold).filter_map do |claimed|
+          running = (now - claimed.claimed_at).to_i
+
+          with_cooldown("stuck_job:#{claimed.job_id}") do
+            stuck_alert(claimed, running)
+          end
+        end
+      end
+
+      private
+
+      def stuck_alert(claimed, running)
+        Alert.new(
+          type: :stuck_job,
+          severity: :warning,
+          title: "Job stuck: #{claimed.job_class}#{env_suffix}",
+          message: "#{claimed.job_class} on \"#{claimed.queue_name}\" has been running #{running}s " \
+                   "(threshold #{config.stuck_job_threshold}s). It may be hung.",
+          fingerprint: "stuck_job:#{claimed.job_id}",
+          context: {
+            job_id: claimed.job_id,
+            job_class: claimed.job_class,
+            queue_name: claimed.queue_name,
+            running_seconds: running
+          }
+        )
+      end
+    end
+  end
+end

data/lib/queue_pulse/checks/worker_liveness.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  module Checks
+    # Alerts when no worker/dispatcher process has a fresh heartbeat — meaning
+    # nothing is processing jobs. Critical, because the queue silently stops.
+    # (Requirements US-5.) Only fires when at least one process was ever
+    # registered, to avoid false alarms before the first boot.
+    class WorkerLiveness < Base
+      def call
+        return [] if source.total_process_count.to_i.zero?
+
+        live = source.live_process_count(heartbeat_within: config.worker_heartbeat_threshold)
+        return [] if live.positive?
+
+        alert = with_cooldown("workers_down") { workers_down_alert }
+        Array(alert)
+      end
+
+      private
+
+      def workers_down_alert
+        Alert.new(
+          type: :workers_down,
+          severity: :critical,
+          title: "No live Solid Queue workers#{env_suffix}",
+          message: "No worker/dispatcher has sent a heartbeat in the last " \
+                   "#{config.worker_heartbeat_threshold}s. Background jobs are NOT being processed.",
+          fingerprint: "workers_down",
+          context: { heartbeat_threshold: config.worker_heartbeat_threshold }
+        )
+      end
+    end
+  end
+end

data/lib/queue_pulse/configuration.rb

@@ -0,0 +1,87 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  # Holds all tunable settings. Sensible defaults mean a user only has to add a
+  # notifier to get value. See docs/requirements.md (US-1, US-3).
+  class Configuration
+    # All checks shipped in Phase 1, in evaluation order.
+    ALL_CHECKS = %i[failures queue_latency queue_depth stuck_jobs worker_liveness].freeze
+
+    attr_accessor :enabled,
+                  :poll_interval,
+                  :queue_latency_threshold,
+                  :queue_depth_threshold,
+                  :stuck_job_threshold,
+                  :worker_heartbeat_threshold,
+                  :alert_cooldown,
+                  :failure_burst_threshold,
+                  :check_timeout,
+                  :notifiers,
+                  :enabled_checks,
+                  :environment_label,
+                  :include_arguments,
+                  :logger
+
+    attr_writer :cache
+
+    def initialize
+      @enabled = true
+      @poll_interval = 30                 # seconds (Railtie poller)
+      @queue_latency_threshold = 300      # seconds a job may wait before alerting
+      @queue_depth_threshold = 1_000      # ready jobs per queue before alerting
+      @stuck_job_threshold = 600          # seconds a claimed job may run before alerting
+      @worker_heartbeat_threshold = 120   # seconds before a process is "dead"
+      @alert_cooldown = 900               # seconds to suppress duplicate state alerts
+      @failure_burst_threshold = 25       # collapse into a summary above this many
+      @check_timeout = 25                 # seconds; abort a slow pass
+      @notifiers = []
+      @enabled_checks = ALL_CHECKS.dup
+      @environment_label = default_environment
+      @include_arguments = false          # never send raw job arguments by default
+      @logger = default_logger
+      @cache = nil
+    end
+
+    def add_notifier(notifier)
+      @notifiers << notifier
+      notifier
+    end
+
+    def check_enabled?(name)
+      @enabled_checks.include?(name)
+    end
+
+    # Cache store for dedupe/cooldown state. Prefers Rails.cache, falls back to
+    # an in-process MemoryStore so the gem works even without a configured cache.
+    def cache
+      @cache ||= rails_cache || MemoryStore.new
+    end
+
+    private
+
+    def rails_cache
+      return nil unless defined?(Rails)
+
+      store = Rails.respond_to?(:cache) ? Rails.cache : nil
+      # A NullStore would silently drop our dedupe state → prefer MemoryStore.
+      return nil if store.nil? || store.class.name.to_s.include?("NullStore")
+
+      store
+    end
+
+    def default_environment
+      return ENV["RAILS_ENV"] if ENV["RAILS_ENV"]
+      return Rails.env.to_s if defined?(Rails) && Rails.respond_to?(:env)
+
+      ENV["RACK_ENV"] || "development"
+    end
+
+    def default_logger
+      if defined?(Rails) && Rails.respond_to?(:logger) && Rails.logger
+        Rails.logger
+      else
+        Logger.new($stdout).tap { |l| l.level = Logger::INFO }
+      end
+    end
+  end
+end

data/lib/queue_pulse/memory_store.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module QueuePulse
+  # Minimal in-process cache used as a fallback when Rails.cache is unavailable
+  # (e.g. plain Ruby tests, or apps with no cache store configured).
+  #
+  # Implements the small subset of the ActiveSupport::Cache interface QueuePulse
+  # relies on: #read, #write (with :expires_in), #delete. Thread-safe.
+  class MemoryStore
+    Entry = Struct.new(:value, :expires_at)
+
+    def initialize
+      @data = {}
+      @mutex = Mutex.new
+    end
+
+    def read(key)
+      @mutex.synchronize do
+        entry = @data[key]
+        return nil unless entry
+        if entry.expires_at && entry.expires_at <= monotonic_now
+          @data.delete(key)
+          return nil
+        end
+        entry.value
+      end
+    end
+
+    def write(key, value, expires_in: nil)
+      @mutex.synchronize do
+        expires_at = expires_in ? monotonic_now + expires_in : nil
+        @data[key] = Entry.new(value, expires_at)
+      end
+      true
+    end
+
+    def delete(key)
+      @mutex.synchronize { @data.delete(key) }
+      true
+    end
+
+    private
+
+    def monotonic_now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+  end
+end