pgbus 0.0.1

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 mhenrixon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,417 @@
+# Pgbus
+
+PostgreSQL-native job processing and event bus for Rails, built on [PGMQ](https://github.com/tembo-io/pgmq).
+
+**Why Pgbus?** If you already run PostgreSQL, you don't need Redis for background jobs. Pgbus gives you ActiveJob integration, AMQP-style topic routing, dead letter queues, worker memory management, and a live dashboard -- all backed by your existing database.
+
+[![Ruby](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml/badge.svg)](https://github.com/mhenrixon/pgbus/actions/workflows/main.yml)
+
+## Table of contents
+
+- [Features](#features)
+- [Requirements](#requirements)
+- [Installation](#installation)
+- [Quick start](#quick-start)
+- [Concurrency controls](#concurrency-controls)
+- [Batches](#batches)
+- [Configuration reference](#configuration-reference)
+- [Architecture](#architecture)
+- [CLI](#cli)
+- [Dashboard](#dashboard)
+- [Database tables](#database-tables)
+- [Switching from another backend](#switching-from-another-backend)
+- [Development](#development)
+- [License](#license)
+
+## Features
+
+- **ActiveJob adapter** -- drop-in replacement, zero config migration from other backends
+- **Event bus** -- publish/subscribe with AMQP-style topic routing (`orders.#`, `payments.*`)
+- **Dead letter queues** -- automatic DLQ routing after configurable retries
+- **Worker recycling** -- memory, job count, and lifetime limits prevent runaway processes
+- **LISTEN/NOTIFY** -- instant wake-up, polling as fallback only
+- **Idempotent events** -- deduplication via `(event_id, handler_class)` unique index
+- **Live dashboard** -- Turbo Frames auto-refresh, no ActionCable required
+- **Supervisor/worker model** -- forked processes with heartbeat monitoring
+
+## Requirements
+
+- Ruby >= 3.3
+- Rails >= 7.1
+- PostgreSQL with the [PGMQ extension](https://github.com/tembo-io/pgmq)
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "pgbus"
+```
+
+Then install the PGMQ extension in your database:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS pgmq;
+```
+
+## Quick start
+
+### 1. Configure (optional)
+
+Pgbus works with zero config in Rails -- it uses your existing `ActiveRecord` connection. For custom setups, create `config/pgbus.yml`:
+
+```yaml
+production:
+  queue_prefix: myapp
+  default_queue: default
+  pool_size: 10
+  max_retries: 5
+  workers:
+    - queues: [default, mailers]
+      threads: 10
+    - queues: [critical]
+      threads: 5
+  event_consumers:
+    - queues: [orders, payments]
+      threads: 5
+  max_jobs_per_worker: 10000
+  max_memory_mb: 512
+  max_worker_lifetime: 3600
+```
+
+Or configure in an initializer:
+
+```ruby
+# config/initializers/pgbus.rb
+Pgbus.configure do |config|
+  config.queue_prefix = "myapp"
+  config.max_retries = 5
+  config.max_jobs_per_worker = 10_000
+  config.max_memory_mb = 512
+  config.max_worker_lifetime = 3600
+
+  config.workers = [
+    { queues: %w[default mailers], threads: 10 },
+    { queues: %w[critical], threads: 5 }
+  ]
+end
+```
+
+### 2. Use as ActiveJob backend
+
+```ruby
+# config/application.rb
+config.active_job.queue_adapter = :pgbus
+```
+
+That's it. Your existing jobs work unchanged:
+
+```ruby
+class OrderConfirmationJob < ApplicationJob
+  queue_as :mailers
+
+  def perform(order)
+    OrderMailer.confirmation(order).deliver_now
+  end
+end
+
+# Enqueue
+OrderConfirmationJob.perform_later(order)
+
+# Schedule
+OrderConfirmationJob.set(wait: 5.minutes).perform_later(order)
+```
+
+### 3. Event bus (optional)
+
+Publish events with AMQP-style topic routing:
+
+```ruby
+# Publish an event
+Pgbus::EventBus::Publisher.publish(
+  "orders.created",
+  { order_id: order.id, total: order.total }
+)
+
+# Publish with delay
+Pgbus::EventBus::Publisher.publish_later(
+  "invoices.due",
+  { invoice_id: invoice.id },
+  delay: 30.days
+)
+```
+
+Subscribe with handlers:
+
+```ruby
+# app/handlers/order_created_handler.rb
+class OrderCreatedHandler < Pgbus::EventBus::Handler
+  idempotent!  # Deduplicate by (event_id, handler_class)
+
+  def handle(event)
+    order_id = event.payload["order_id"]
+    Analytics.track_order(order_id)
+    InventoryService.reserve(order_id)
+  end
+end
+
+# Register in an initializer
+Pgbus::EventBus::Registry.instance.subscribe(
+  "orders.created",
+  OrderCreatedHandler
+)
+
+# Wildcard patterns
+Pgbus::EventBus::Registry.instance.subscribe(
+  "orders.#",           # matches orders.created, orders.updated, orders.shipped.confirmed
+  OrderAuditHandler
+)
+```
+
+### 4. Start workers
+
+```bash
+bundle exec pgbus start
+```
+
+This boots a supervisor that manages:
+- **Workers** -- process ActiveJob queues
+- **Dispatcher** -- runs maintenance tasks (idempotency cleanup, stale process reaping)
+- **Consumers** -- process event bus messages
+
+### 5. Mount the dashboard
+
+```ruby
+# config/routes.rb
+mount Pgbus::Engine => "/pgbus"
+```
+
+The dashboard shows queues, jobs, processes, failures, dead letter messages, and event subscribers. It auto-refreshes via Turbo Frames with no WebSocket dependency.
+
+Protect it in production:
+
+```ruby
+Pgbus.configure do |config|
+  config.web_auth = ->(request) {
+    request.env["warden"].user&.admin?
+  }
+end
+```
+
+## Concurrency controls
+
+Limit how many jobs with the same key can run concurrently:
+
+```ruby
+class ProcessOrderJob < ApplicationJob
+  include Pgbus::Concurrency
+
+  limits_concurrency to: 1,
+                     key: ->(order_id) { "ProcessOrder-#{order_id}" },
+                     duration: 15.minutes,
+                     on_conflict: :block
+
+  def perform(order_id)
+    # Only one job per order_id runs at a time
+  end
+end
+```
+
+### Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `to:` | (required) | Maximum concurrent jobs for the same key |
+| `key:` | Job class name | Proc receiving job arguments, returns a string key |
+| `duration:` | `15.minutes` | Safety expiry for the semaphore (crashed worker recovery) |
+| `on_conflict:` | `:block` | What to do when the limit is reached |
+
+### Conflict strategies
+
+| Strategy | Behavior |
+|----------|----------|
+| `:block` | Hold the job in a blocked queue. It is automatically released when a slot opens or the semaphore expires. |
+| `:discard` | Silently drop the job. |
+| `:raise` | Raise `Pgbus::ConcurrencyLimitExceeded` so the caller can handle it. |
+
+### How it works
+
+1. **Enqueue**: The adapter checks a semaphore table for the concurrency key. If under the limit, it increments the counter and sends the job to PGMQ. If at the limit, it applies the `on_conflict` strategy.
+2. **Complete**: After a job succeeds or is dead-lettered, the executor decrements the semaphore and releases the next blocked job (if any).
+3. **Safety net**: The dispatcher periodically cleans up expired semaphores and orphaned blocked executions to recover from crashed workers.
+
+## Batches
+
+Coordinate groups of jobs with callbacks when all complete:
+
+```ruby
+batch = Pgbus::Batch.new(
+  on_finish: BatchFinishedJob,
+  on_success: BatchSucceededJob,
+  on_discard: BatchFailedJob,
+  description: "Import users",
+  properties: { initiated_by: current_user.id }
+)
+
+batch.enqueue do
+  users.each { |user| ImportUserJob.perform_later(user.id) }
+end
+```
+
+### Callbacks
+
+| Callback | Fired when |
+|----------|------------|
+| `on_finish` | All jobs completed (success or discard) |
+| `on_success` | All jobs completed successfully (zero discarded) |
+| `on_discard` | At least one job was dead-lettered |
+
+Callback jobs receive the batch `properties` hash as their argument:
+
+```ruby
+class BatchFinishedJob < ApplicationJob
+  def perform(properties)
+    user = User.find(properties["initiated_by"])
+    ImportMailer.complete(user).deliver_later
+  end
+end
+```
+
+### How it works
+
+1. `Batch.new(...)` creates a tracking row in `pgbus_batches` with `status: "pending"`
+2. `batch.enqueue { ... }` tags each enqueued job with the `pgbus_batch_id` in its payload
+3. After each job completes or is dead-lettered, the executor atomically updates the batch counters
+4. When `completed_jobs + discarded_jobs == total_jobs`, the batch status flips to `"finished"` and callback jobs are enqueued
+5. The dispatcher cleans up finished batches older than 7 days
+
+## Configuration reference
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `database_url` | `nil` | PostgreSQL connection URL (auto-detected in Rails) |
+| `queue_prefix` | `"pgbus"` | Prefix for all PGMQ queue names |
+| `default_queue` | `"default"` | Default queue for jobs without explicit queue |
+| `pool_size` | `5` | Connection pool size |
+| `workers` | `[{queues: ["default"], threads: 5}]` | Worker process definitions |
+| `event_consumers` | `nil` | Event consumer process definitions (same format as workers) |
+| `polling_interval` | `0.1` | Seconds between polls (LISTEN/NOTIFY is primary) |
+| `visibility_timeout` | `30` | Seconds before unacked message becomes visible again |
+| `max_retries` | `5` | Failed reads before routing to dead letter queue |
+| `max_jobs_per_worker` | `nil` | Recycle worker after N jobs (nil = unlimited) |
+| `max_memory_mb` | `nil` | Recycle worker when memory exceeds N MB |
+| `max_worker_lifetime` | `nil` | Recycle worker after N seconds |
+| `listen_notify` | `true` | Use PGMQ's LISTEN/NOTIFY for instant wake-up |
+| `dispatch_interval` | `1.0` | Seconds between dispatcher maintenance ticks |
+| `idempotency_ttl` | `604800` | Seconds to keep processed event records (7 days, cleaned hourly) |
+| `web_auth` | `nil` | Lambda for dashboard authentication |
+| `web_refresh_interval` | `5000` | Dashboard auto-refresh interval in milliseconds |
+| `web_live_updates` | `true` | Enable Turbo Frames auto-refresh on dashboard |
+
+## Architecture
+
+```text
+Supervisor (fork manager)
+  ├── Worker 1        (queues: [default, mailers], threads: 10)
+  ├── Worker 2        (queues: [critical], threads: 5)
+  ├── Dispatcher      (maintenance: idempotency cleanup, stale process reaping)
+  └── Consumer        (event bus topics)
+
+PostgreSQL + PGMQ
+  ├── pgbus_default          (job queue)
+  ├── pgbus_default_dlq      (dead letter queue)
+  ├── pgbus_critical         (job queue)
+  ├── pgbus_critical_dlq     (dead letter queue)
+  └── pgbus_mailers          (job queue)
+```
+
+### How it works
+
+1. **Enqueue**: ActiveJob serializes the job to JSON, Pgbus sends it to the appropriate PGMQ queue
+2. **Read**: Workers poll queues (or wake instantly via LISTEN/NOTIFY) and claim messages with a visibility timeout
+3. **Execute**: The job is deserialized and executed within the Rails executor
+4. **Archive/Retry**: On success, the message is archived. On failure, the visibility timeout expires and the message becomes available again. PGMQ's `read_ct` tracks delivery attempts
+5. **Dead letter**: When `read_ct` exceeds `max_retries`, the message is moved to the `_dlq` queue for manual inspection
+
+### Worker recycling
+
+Unlike solid_queue, Pgbus workers recycle themselves to prevent memory bloat:
+
+```ruby
+Pgbus.configure do |config|
+  config.max_jobs_per_worker = 10_000  # Restart after 10k jobs
+  config.max_memory_mb = 512           # Restart if memory exceeds 512MB
+  config.max_worker_lifetime = 3600    # Restart after 1 hour
+end
+```
+
+When a limit is hit, the worker drains its thread pool, exits, and the supervisor forks a fresh process.
+
+## CLI
+
+```bash
+pgbus start     # Start supervisor with workers + dispatcher
+pgbus status    # Show running processes
+pgbus queues    # List queues with depth/metrics
+pgbus version   # Print version
+pgbus help      # Show help
+```
+
+## Dashboard
+
+The dashboard is a mountable Rails engine at `/pgbus` with:
+
+- **Overview** -- queue depths, enqueued count, active processes, failure count
+- **Queues** -- per-queue metrics, purge actions
+- **Jobs** -- enqueued and failed jobs, retry/discard actions
+- **Dead letter** -- DLQ messages with retry/discard, bulk actions
+- **Processes** -- active workers/dispatcher/consumers with heartbeat status
+- **Events** -- registered subscribers and processed events
+
+All tables use Turbo Frames for periodic auto-refresh without page reloads.
+
+## Database tables
+
+Pgbus uses these tables (created via PGMQ and migrations):
+
+| Table | Purpose |
+|-------|---------|
+| `q_pgbus_*` | PGMQ job queues (managed by PGMQ) |
+| `a_pgbus_*` | PGMQ archive tables (managed by PGMQ) |
+| `pgbus_processes` | Heartbeat tracking for workers/dispatcher/consumers |
+| `pgbus_failed_events` | Failed event dispatch records |
+| `pgbus_processed_events` | Idempotency deduplication (event_id, handler_class) |
+| `pgbus_semaphores` | Concurrency control counting semaphores |
+| `pgbus_blocked_executions` | Jobs waiting for a concurrency semaphore slot |
+| `pgbus_batches` | Batch tracking with job counters and callback config |
+
+## Switching from another backend
+
+Already using a different job processor? These guides walk you through the migration:
+
+- **[Switch from Sidekiq](docs/switch_from_sidekiq.md)** -- remove Redis, convert native workers, replace middleware with callbacks
+- **[Switch from SolidQueue](docs/switch_from_solid_queue.md)** -- similar architecture, swap config format, gain LISTEN/NOTIFY + worker recycling
+- **[Switch from GoodJob](docs/switch_from_good_job.md)** -- both PostgreSQL-native, swap advisory locks for PGMQ visibility timeouts
+
+See [docs/README.md](docs/README.md) for a full feature comparison table.
+
+## Development
+
+```bash
+bundle install
+bundle exec rake          # Run tests + rubocop
+bundle exec rspec         # Run tests only
+bundle exec rubocop       # Run linter only
+```
+
+System tests use Playwright via Capybara:
+
+```bash
+bun install
+bunx --bun playwright install chromium
+bundle exec rspec spec/system/
+```
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.pattern = "spec/pgbus/**/*_spec.rb"
+end
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/app/controllers/pgbus/api/stats_controller.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Pgbus
+  module Api
+    class StatsController < ApplicationController
+      def show
+        render json: data_source.summary_stats
+      end
+    end
+  end
+end

data/app/controllers/pgbus/application_controller.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class ApplicationController < ActionController::Base
+    include Web::Authentication
+
+    protect_from_forgery with: :exception
+
+    layout "pgbus/application"
+
+    helper Pgbus::ApplicationHelper
+
+    private
+
+    def data_source
+      @data_source ||= Pgbus.configuration.web_data_source || Web::DataSource.new
+    end
+
+    def page_param
+      [params[:page].to_i, 1].max
+    end
+
+    def per_page
+      Pgbus.configuration.web_per_page
+    end
+
+    def turbo_frame_request?
+      request.headers["Turbo-Frame"].present? || params[:frame].present?
+    end
+
+    def render_frame(partial)
+      render partial: partial, layout: false
+    end
+  end
+end

data/app/controllers/pgbus/dashboard_controller.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class DashboardController < ApplicationController
+    def show
+      case params[:frame]
+      when "stats"
+        @stats = data_source.summary_stats
+        render_frame("pgbus/dashboard/stats_cards")
+      when "queues"
+        @queues = data_source.queues_with_metrics
+        render_frame("pgbus/dashboard/queues_table")
+      when "processes"
+        @processes = data_source.processes
+        render_frame("pgbus/dashboard/processes_table")
+      when "failures"
+        @recent_failures = data_source.failed_events(page: 1, per_page: 5)
+        render_frame("pgbus/dashboard/recent_failures")
+      else
+        @stats = data_source.summary_stats
+        @queues = data_source.queues_with_metrics
+        @processes = data_source.processes
+        @recent_failures = data_source.failed_events(page: 1, per_page: 5)
+      end
+    end
+  end
+end

data/app/controllers/pgbus/dead_letter_controller.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class DeadLetterController < ApplicationController
+    def index
+      @messages = data_source.dlq_messages(page: page_param, per_page: per_page)
+      render_frame("pgbus/dead_letter/messages_table") if params[:frame] == "list"
+    end
+
+    def show
+      @message = data_source.dlq_message_detail(params[:id].to_i)
+    end
+
+    def retry
+      queue_name = params[:queue_name].to_s
+      unless queue_name.end_with?(Pgbus.configuration.dead_letter_queue_suffix)
+        return redirect_to dead_letter_index_path, alert: "Invalid DLQ queue."
+      end
+
+      if data_source.retry_dlq_message(queue_name, params[:id])
+        redirect_to dead_letter_index_path, notice: "Message re-enqueued to original queue."
+      else
+        redirect_to dead_letter_index_path, alert: "Could not retry message."
+      end
+    end
+
+    def discard
+      queue_name = params[:queue_name].to_s
+      unless queue_name.end_with?(Pgbus.configuration.dead_letter_queue_suffix)
+        return redirect_to dead_letter_index_path, alert: "Invalid DLQ queue."
+      end
+
+      if data_source.discard_dlq_message(queue_name, params[:id])
+        redirect_to dead_letter_index_path, notice: "Message discarded."
+      else
+        redirect_to dead_letter_index_path, alert: "Could not discard message."
+      end
+    end
+
+    def retry_all
+      count = data_source.retry_all_dlq
+      redirect_to dead_letter_index_path, notice: "Re-enqueued #{count} DLQ messages."
+    end
+
+    def discard_all
+      count = data_source.discard_all_dlq
+      redirect_to dead_letter_index_path, notice: "Discarded #{count} DLQ messages."
+    end
+  end
+end

data/app/controllers/pgbus/events_controller.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class EventsController < ApplicationController
+    def index
+      @events = data_source.processed_events(page: page_param, per_page: per_page)
+      @subscribers = data_source.registered_subscribers
+    end
+
+    def show
+      @event = data_source.processed_event(params[:id])
+    end
+
+    def replay
+      event = data_source.processed_event(params[:id])
+      if event && data_source.replay_event(event)
+        redirect_to events_path, notice: "Event replayed."
+      else
+        redirect_to events_path, alert: "Event not found or could not be replayed."
+      end
+    end
+  end
+end

data/app/controllers/pgbus/jobs_controller.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class JobsController < ApplicationController
+    def index
+      if params[:frame] == "failed"
+        @failed = data_source.failed_events(page: page_param, per_page: per_page)
+        render_frame("pgbus/jobs/failed_table")
+      elsif params[:frame] == "enqueued"
+        @jobs = params[:status] == "failed" ? [] : data_source.jobs(queue_name: params[:queue], page: page_param, per_page: per_page)
+        render_frame("pgbus/jobs/enqueued_table")
+      else
+        @jobs = params[:status] == "failed" ? [] : data_source.jobs(queue_name: params[:queue], page: page_param, per_page: per_page)
+        @failed = data_source.failed_events(page: page_param, per_page: per_page)
+      end
+    end
+
+    def show
+      @job = data_source.failed_event(params[:id])
+    end
+
+    def retry
+      if data_source.retry_failed_event(params[:id])
+        redirect_to jobs_path, notice: "Job re-enqueued."
+      else
+        redirect_to jobs_path, alert: "Could not retry job."
+      end
+    end
+
+    def discard
+      if data_source.discard_failed_event(params[:id])
+        redirect_to jobs_path, notice: "Job discarded."
+      else
+        redirect_to jobs_path, alert: "Could not discard job."
+      end
+    end
+
+    def retry_all
+      count = data_source.retry_all_failed
+      redirect_to jobs_path, notice: "Re-enqueued #{count} jobs."
+    end
+
+    def discard_all
+      count = data_source.discard_all_failed
+      redirect_to jobs_path, notice: "Discarded #{count} jobs."
+    end
+  end
+end

data/app/controllers/pgbus/processes_controller.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class ProcessesController < ApplicationController
+    def index
+      @processes = data_source.processes
+      render_frame("pgbus/processes/processes_table") if params[:frame] == "list"
+    end
+  end
+end

data/app/controllers/pgbus/queues_controller.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Pgbus
+  class QueuesController < ApplicationController
+    def index
+      @queues = data_source.queues_with_metrics
+    end
+
+    def show
+      @queue = data_source.queue_detail(params[:name])
+      redirect_to queues_path, alert: "Queue not found." and return unless @queue
+
+      @messages = data_source.jobs(queue_name: params[:name], page: page_param, per_page: per_page)
+    end
+
+    def purge
+      data_source.purge_queue(params[:name])
+      redirect_to queue_path(name: params[:name]), notice: "Queue purged."
+    end
+  end
+end