pgbus 0.0.1

data/docs/switch_from_sidekiq.md

@@ -0,0 +1,226 @@
+# Switch from Sidekiq to Pgbus
+
+## Overview
+
+Sidekiq uses Redis as its message broker. Pgbus uses PostgreSQL via PGMQ. Switching eliminates Redis from your infrastructure and gives you dead letter queues, worker recycling, and an event bus -- all backed by your existing database.
+
+**Effort estimate:** Low if you use ActiveJob exclusively. Medium-high if you use native Sidekiq workers or Pro/Enterprise features.
+
+## Step 1: Update dependencies
+
+```ruby
+# Gemfile
+
+# Remove
+gem "sidekiq"
+gem "sidekiq-cron"          # if used
+gem "sidekiq-unique-jobs"   # if used
+
+# Add
+gem "pgbus"
+```
+
+```bash
+bundle install
+rails generate pgbus:install
+rails db:migrate
+```
+
+## Step 2: Switch the adapter
+
+```ruby
+# config/application.rb (or config/environments/production.rb)
+
+# Before
+config.active_job.queue_adapter = :sidekiq
+
+# After
+config.active_job.queue_adapter = :pgbus
+```
+
+## Step 3: Convert native Sidekiq workers
+
+If all your jobs inherit from `ApplicationJob` (ActiveJob), skip this step -- they work unchanged.
+
+If you have native Sidekiq workers using `include Sidekiq::Job`, convert them:
+
+```ruby
+# Before: Native Sidekiq worker
+class HardWorker
+  include Sidekiq::Job
+  sidekiq_options queue: :critical, retry: 5
+
+  def perform(user_id, action)
+    user = User.find(user_id)
+    # ...
+  end
+end
+
+# Enqueue
+HardWorker.perform_async(user.id, "activate")
+HardWorker.perform_at(5.minutes.from_now, user.id, "activate")
+```
+
+```ruby
+# After: ActiveJob
+class HardWorker < ApplicationJob
+  queue_as :critical
+  retry_on StandardError, wait: :polynomially_longer, attempts: 5
+
+  def perform(user_id, action)
+    user = User.find(user_id)
+    # ...
+  end
+end
+
+# Enqueue
+HardWorker.perform_later(user.id, "activate")
+HardWorker.set(wait: 5.minutes).perform_later(user.id, "activate")
+```
+
+### API mapping
+
+| Sidekiq | ActiveJob / Pgbus |
+|---------|-------------------|
+| `perform_async(args)` | `perform_later(args)` |
+| `perform_at(time, args)` | `.set(wait_until: time).perform_later(args)` |
+| `perform_in(duration, args)` | `.set(wait: duration).perform_later(args)` |
+| `sidekiq_options queue: :name` | `queue_as :name` |
+| `sidekiq_options retry: N` | `retry_on StandardError, attempts: N` |
+| `sidekiq_retries_exhausted` | `discard_on` + `after_discard` callback |
+
+## Step 4: Replace middleware with ActiveJob callbacks
+
+Sidekiq middleware wraps job push (client) and execution (server) in a Rack-style chain. ActiveJob provides equivalent hooks.
+
+```ruby
+# Before: Sidekiq server middleware
+class LoggingMiddleware
+  include Sidekiq::ServerMiddleware
+  def call(job_instance, job_payload, queue)
+    Rails.logger.info("Starting #{job_payload['class']}")
+    yield
+    Rails.logger.info("Finished #{job_payload['class']}")
+  end
+end
+```
+
+```ruby
+# After: ActiveJob callback (add to ApplicationJob or per-job)
+class ApplicationJob < ActiveJob::Base
+  around_perform do |job, block|
+    Rails.logger.info("Starting #{job.class.name}")
+    block.call
+    Rails.logger.info("Finished #{job.class.name}")
+  end
+end
+```
+
+### Callback mapping
+
+| Sidekiq middleware | ActiveJob callback |
+|--------------------|-------------------|
+| Server middleware (before yield) | `before_perform` |
+| Server middleware (around yield) | `around_perform` |
+| Server middleware (after yield) | `after_perform` |
+| Client middleware (before yield) | `before_enqueue` |
+| Client middleware (around yield) | `around_enqueue` |
+| Client middleware (after yield) | `after_enqueue` |
+
+## Step 5: Configure workers
+
+```yaml
+# Before: config/sidekiq.yml
+:concurrency: 10
+:queues:
+  - [critical, 3]
+  - [default, 2]
+  - [low, 1]
+```
+
+```yaml
+# After: config/pgbus.yml
+production:
+  workers:
+    - queues: [critical]
+      threads: 5
+    - queues: [default, low]
+      threads: 10
+  max_jobs_per_worker: 10000
+  max_memory_mb: 512
+  max_worker_lifetime: 3600
+```
+
+## Step 6: Replace the dashboard
+
+```ruby
+# Before: config/routes.rb
+require "sidekiq/web"
+mount Sidekiq::Web => "/sidekiq"
+
+# After:
+mount Pgbus::Engine => "/pgbus"
+```
+
+If you used `Sidekiq::Web`'s authentication:
+
+```ruby
+Pgbus.configure do |config|
+  config.web_auth = ->(request) {
+    request.env["warden"].user&.admin?
+  }
+end
+```
+
+## Step 7: Update process management
+
+```bash
+# Before
+bundle exec sidekiq
+
+# After
+bundle exec pgbus start
+```
+
+Update your `Procfile`, systemd units, or container entrypoints accordingly.
+
+## Step 8: Remove Redis
+
+Once all Sidekiq jobs have drained and you've verified Pgbus is processing correctly, remove Redis from your infrastructure:
+
+- Remove `REDIS_URL` / Sidekiq Redis config from environment
+- Remove `Sidekiq.configure_server` / `Sidekiq.configure_client` blocks
+- Remove Redis from `docker-compose.yml`, Terraform, etc.
+
+## What you gain
+
+- **No Redis dependency** -- one fewer service to operate
+- **Dead letter queues** -- failed jobs route to `_dlq` queues after `max_retries`, visible in the dashboard
+- **Worker recycling** -- memory, job count, and lifetime limits prevent the memory bloat that plagues long-running Sidekiq processes
+- **Event bus** -- AMQP-style topic routing for pub/sub, built into the same infrastructure
+- **LISTEN/NOTIFY** -- instant job wake-up without polling overhead
+
+## What you lose (for now)
+
+| Sidekiq feature | Status in Pgbus |
+|-----------------|-----------------|
+| Batches (Pro) | `Pgbus::Batch` with on_finish/on_success/on_discard callbacks |
+| Rate limiting (Enterprise) | Planned |
+| Concurrency controls (Enterprise) | `Pgbus::Concurrency` with `limits_concurrency` DSL |
+| Unique jobs (Enterprise / `sidekiq-unique-jobs`) | Partial -- event bus has idempotency; job-level dedup planned |
+| Cron / recurring jobs (`sidekiq-cron`) | Planned |
+| Real-time metrics (Sidekiq Web) | Pgbus dashboard covers queue depth, failures, processes |
+
+## Gotchas
+
+1. **Argument serialization**: Sidekiq passes raw JSON; ActiveJob uses GlobalID for ActiveRecord objects. If you pass raw IDs (`user.id`), both work the same. If you pass AR objects (`user`), ActiveJob serializes via GlobalID automatically.
+
+2. **`Sidekiq::Testing.fake!`**: Replace with `ActiveJob::TestHelper`:
+   ```ruby
+   include ActiveJob::TestHelper
+   assert_enqueued_jobs 1 { MyJob.perform_later(args) }
+   ```
+
+3. **`Sidekiq.redis { |conn| ... }`**: If you use Sidekiq's Redis connection for custom caching or distributed locks, you'll need a separate solution (e.g., PostgreSQL advisory locks, `with_advisory_lock` gem).
+
+4. **Job priorities**: Sidekiq uses queue weight ordering. Pgbus processes queues in the order listed in the worker config. Put higher-priority queues first.

data/docs/switch_from_solid_queue.md

@@ -0,0 +1,247 @@
+# Switch from SolidQueue to Pgbus
+
+## Overview
+
+SolidQueue and Pgbus are both PostgreSQL-backed job processors with similar architectures: supervisor/worker process model, `FOR UPDATE SKIP LOCKED` for contention-free polling, and forked processes. The migration is straightforward since both are ActiveJob adapters.
+
+**Key differences:** Pgbus adds LISTEN/NOTIFY for instant wake-up (SolidQueue only polls), dead letter queues, worker recycling, and an event bus. Pgbus uses PGMQ under the hood instead of custom tables.
+
+**Effort estimate:** Low. Both are pure ActiveJob adapters, so your jobs work unchanged.
+
+## Step 1: Update dependencies
+
+```ruby
+# Gemfile
+
+# Remove
+gem "solid_queue"
+gem "mission_control-jobs"  # if used
+
+# Add
+gem "pgbus"
+```
+
+```bash
+bundle install
+rails generate pgbus:install
+rails db:migrate
+```
+
+## Step 2: Switch the adapter
+
+```ruby
+# config/application.rb
+
+# Before
+config.active_job.queue_adapter = :solid_queue
+
+# After
+config.active_job.queue_adapter = :pgbus
+```
+
+If you set the adapter per-environment or use `config.solid_queue.connects_to` for a separate queue database, update those too.
+
+## Step 3: Convert worker configuration
+
+```yaml
+# Before: config/queue.yml (SolidQueue)
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  workers:
+    - queues: "critical"
+      threads: 5
+      processes: 2
+      polling_interval: 0.1
+    - queues: "default,low"
+      threads: 3
+      processes: 3
+      polling_interval: 1
+```
+
+```yaml
+# After: config/pgbus.yml
+production:
+  workers:
+    - queues: [critical]
+      threads: 5
+    - queues: [critical]
+      threads: 5
+    - queues: [default, low]
+      threads: 3
+    - queues: [default, low]
+      threads: 3
+    - queues: [default, low]
+      threads: 3
+  max_retries: 5
+  max_jobs_per_worker: 10000
+  max_memory_mb: 512
+```
+
+> **Note:** SolidQueue's `processes: N` forks N identical workers. In Pgbus, list the same worker config N times, or let the supervisor handle it via configuration (one entry per process).
+
+### Configuration mapping
+
+| SolidQueue | Pgbus | Notes |
+|------------|-------|-------|
+| `polling_interval` | `polling_interval` | Pgbus defaults to 0.1s; LISTEN/NOTIFY makes this a fallback only |
+| `threads` | `threads` | Same concept |
+| `processes` | Repeat worker entry | One entry per forked process |
+| `dispatchers[].batch_size` | N/A | Pgbus dispatcher does maintenance, not dispatch |
+| `queues: "a,b"` (string) | `queues: [a, b]` (array) | Different format |
+| `queues: "*"` (wildcard) | List queues explicitly | PGMQ queues are explicit |
+
+## Step 4: Remove concurrency controls
+
+SolidQueue's `limits_concurrency` is a SolidQueue-specific mixin. Remove it from your jobs:
+
+```ruby
+# Before: SolidQueue concurrency control
+class ProcessOrderJob < ApplicationJob
+  limits_concurrency to: 1,
+                     key: ->(order) { order.account_id },
+                     duration: 15.minutes,
+                     on_conflict: :block
+
+  def perform(order)
+    # ...
+  end
+end
+```
+
+```ruby
+# After: Remove the SolidQueue mixin
+class ProcessOrderJob < ApplicationJob
+  def perform(order)
+    # ...
+  end
+end
+```
+
+> Pgbus supports concurrency controls via `Pgbus::Concurrency`:
+> ```ruby
+> class ProcessOrderJob < ApplicationJob
+>   include Pgbus::Concurrency
+>   limits_concurrency to: 1,
+>                      key: ->(order) { order.account_id },
+>                      duration: 15.minutes,
+>                      on_conflict: :block
+>   def perform(order)
+>     # ...
+>   end
+> end
+> ```
+
+## Step 5: Migrate recurring tasks
+
+If you use SolidQueue's `config/recurring.yml`:
+
+```yaml
+# Before: config/recurring.yml (SolidQueue)
+production:
+  daily_cleanup:
+    class: CleanupJob
+    schedule: "every day at 2am"
+    queue: maintenance
+  hourly_sync:
+    class: SyncJob
+    schedule: "0 * * * *"
+    args: [42, "sync"]
+```
+
+Pgbus does not yet have built-in recurring task support. Options:
+
+1. **Use the `whenever` gem** for cron-based scheduling:
+   ```ruby
+   # config/schedule.rb (whenever gem)
+   every 1.day, at: "2:00 am" do
+     runner "CleanupJob.perform_later"
+   end
+   every :hour do
+     runner "SyncJob.perform_later(42, 'sync')"
+   end
+   ```
+
+2. **Use system cron** directly:
+   ```cron
+   0 2 * * * cd /app && bin/rails runner "CleanupJob.perform_later"
+   0 * * * * cd /app && bin/rails runner "SyncJob.perform_later(42, 'sync')"
+   ```
+
+3. Wait for Pgbus recurring task support (planned).
+
+## Step 6: Replace the dashboard
+
+```ruby
+# Before: config/routes.rb
+mount MissionControl::Jobs::Engine, at: "/jobs"
+
+# After:
+mount Pgbus::Engine => "/pgbus"
+```
+
+## Step 7: Update process management
+
+```bash
+# Before
+bundle exec rake solid_queue:start
+
+# After
+bundle exec pgbus start
+```
+
+## Step 8: Clean up SolidQueue tables
+
+After verifying Pgbus processes jobs correctly and SolidQueue's tables are drained:
+
+```ruby
+class RemoveSolidQueue < ActiveRecord::Migration[7.1]
+  def up
+    drop_table :solid_queue_blocked_executions, if_exists: true
+    drop_table :solid_queue_claimed_executions, if_exists: true
+    drop_table :solid_queue_failed_executions, if_exists: true
+    drop_table :solid_queue_pauses, if_exists: true
+    drop_table :solid_queue_processes, if_exists: true
+    drop_table :solid_queue_ready_executions, if_exists: true
+    drop_table :solid_queue_recurring_executions, if_exists: true
+    drop_table :solid_queue_recurring_tasks, if_exists: true
+    drop_table :solid_queue_scheduled_executions, if_exists: true
+    drop_table :solid_queue_semaphores, if_exists: true
+    drop_table :solid_queue_jobs, if_exists: true
+  end
+end
+```
+
+## What you gain
+
+- **LISTEN/NOTIFY** -- SolidQueue only polls (100-150ms latency baseline). Pgbus wakes workers instantly via PostgreSQL LISTEN/NOTIFY, with polling as fallback only.
+- **Dead letter queues** -- SolidQueue marks jobs as failed but keeps them in the same table. Pgbus routes failures to dedicated `_dlq` queues after `max_retries` for clear separation.
+- **Worker recycling** -- SolidQueue workers run indefinitely. Pgbus recycles workers by job count, memory, or lifetime to prevent memory bloat.
+- **Event bus** -- AMQP-style pub/sub with topic routing, not available in SolidQueue.
+- **PGMQ under the hood** -- battle-tested message queue extension with visibility timeouts and atomic operations.
+
+## What you lose (for now)
+
+| SolidQueue feature | Status in Pgbus |
+|--------------------|-----------------|
+| `limits_concurrency` | `Pgbus::Concurrency` with `limits_concurrency` DSL |
+| `config/recurring.yml` | Planned |
+| Queue pausing (`SolidQueue::Queue.pause`) | Planned |
+| Separate queue database | Not planned (PGMQ lives in your primary DB) |
+| Numeric job priorities | PGMQ reads in FIFO order per queue; use separate queues for priority |
+
+## Gotchas
+
+1. **PgBouncer**: If you run PgBouncer in transaction mode, LISTEN/NOTIFY won't work. Use session mode or direct connections for Pgbus worker processes. This also applies to PGMQ's `FOR UPDATE SKIP LOCKED`.
+
+2. **Separate queue database**: SolidQueue supports `connects_to` for a dedicated queue DB. Pgbus requires PGMQ in the same database it connects to. If you need isolation, use a separate PostgreSQL database with PGMQ installed and configure `Pgbus.configuration.database_url`.
+
+3. **Queue naming**: SolidQueue uses bare queue names (`default`). Pgbus prefixes all queues (`pgbus_default`). Your `queue_as :default` declarations work unchanged -- the prefix is applied automatically.
+
+4. **`ActionMailer::MailDeliveryJob`**: This can bypass the application-level adapter setting in some Rails versions. If mailer jobs don't appear in Pgbus, add to `ApplicationMailer`:
+   ```ruby
+   class ApplicationMailer < ActionMailer::Base
+     self.deliver_later_queue_name = :mailers
+   end
+   ```

data/exe/pgbus

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "pgbus"
+require "pgbus/cli"
+
+Pgbus::CLI.start(ARGV)

data/lib/generators/pgbus/install_generator.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Pgbus
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Install Pgbus: create migration, config file, and binstub"
+
+      def create_migration
+        migration_template "migration.rb.erb", "db/migrate/install_pgbus.rb"
+      end
+
+      def create_config_file
+        template "pgbus.yml.erb", "config/pgbus.yml"
+      end
+
+      def create_binstub
+        template "pgbus_binstub.erb", "bin/pgbus"
+        chmod "bin/pgbus", 0o755
+      end
+
+      def configure_active_job
+        application_config = "config.active_job.queue_adapter = :pgbus"
+
+        return unless File.exist?(File.join(destination_root, "config", "application.rb"))
+
+        inject_into_file "config/application.rb",
+                         "\n    #{application_config}\n",
+                         after: "class Application < Rails::Application\n"
+      end
+
+      def display_post_install
+        say ""
+        say "Pgbus installed successfully!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run: rails db:migrate"
+        say "  2. Edit config/pgbus.yml to configure workers"
+        say "  3. Start processing: bin/pgbus start"
+        say ""
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/generators/pgbus/templates/migration.rb.erb

@@ -0,0 +1,114 @@
+class InstallPgbus < ActiveRecord::Migration<%= migration_version %>
+  def up
+    # Install PGMQ extension
+    enable_extension "pgmq"
+
+    # Idempotent event processing deduplication
+    create_table :pgbus_processed_events do |t|
+      t.string :event_id, null: false
+      t.string :handler_class, null: false
+      t.datetime :processed_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+    end
+
+    add_index :pgbus_processed_events, [:event_id, :handler_class],
+      unique: true, name: "idx_pgbus_processed_events_unique"
+    add_index :pgbus_processed_events, :processed_at,
+      name: "idx_pgbus_processed_events_cleanup"
+
+    # Process registry (heartbeats, monitoring)
+    create_table :pgbus_processes do |t|
+      t.string :kind, null: false
+      t.string :hostname
+      t.integer :pid
+      t.jsonb :metadata, default: {}
+      t.datetime :last_heartbeat_at
+      t.timestamps
+    end
+
+    add_index :pgbus_processes, :last_heartbeat_at,
+      name: "idx_pgbus_processes_heartbeat"
+
+    # Failed events log (for dashboard visibility)
+    create_table :pgbus_failed_events do |t|
+      t.string :queue_name, null: false
+      t.bigint :msg_id
+      t.jsonb :payload
+      t.jsonb :headers
+      t.string :error_class
+      t.text :error_message
+      t.text :backtrace
+      t.integer :retry_count, default: 0
+      t.datetime :failed_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+    end
+
+    add_index :pgbus_failed_events, :queue_name, name: "idx_pgbus_failed_events_queue"
+    add_index :pgbus_failed_events, :failed_at, name: "idx_pgbus_failed_events_time"
+
+    # Concurrency semaphores (counting locks for job concurrency limits)
+    create_table :pgbus_semaphores do |t|
+      t.string :key, null: false
+      t.integer :value, null: false, default: 0
+      t.integer :max_value, null: false, default: 1
+      t.datetime :expires_at, null: false
+      t.datetime :created_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+    end
+
+    add_index :pgbus_semaphores, :key,
+      unique: true, name: "idx_pgbus_semaphores_key"
+    add_index :pgbus_semaphores, :expires_at,
+      name: "idx_pgbus_semaphores_expired"
+
+    # Blocked executions (jobs waiting for a concurrency semaphore)
+    create_table :pgbus_blocked_executions do |t|
+      t.string :concurrency_key, null: false
+      t.string :queue_name, null: false
+      t.jsonb :payload, null: false
+      t.integer :priority, null: false, default: 0
+      t.datetime :expires_at, null: false
+      t.datetime :created_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+    end
+
+    add_index :pgbus_blocked_executions, [:concurrency_key, :priority, :created_at],
+      name: "idx_pgbus_blocked_release_order"
+
+    # Batch tracking (coordinated job groups with callbacks)
+    create_table :pgbus_batches do |t|
+      t.string :batch_id, null: false
+      t.string :description
+      t.string :on_finish_class
+      t.string :on_success_class
+      t.string :on_discard_class
+      t.jsonb :properties, default: {}
+      t.integer :total_jobs, null: false, default: 0
+      t.integer :completed_jobs, null: false, default: 0
+      t.integer :failed_jobs, null: false, default: 0
+      t.integer :discarded_jobs, null: false, default: 0
+      t.string :status, null: false, default: "pending"
+      t.datetime :created_at, null: false, default: -> { "CURRENT_TIMESTAMP" }
+      t.datetime :finished_at
+    end
+
+    add_index :pgbus_batches, :batch_id,
+      unique: true, name: "idx_pgbus_batches_batch_id"
+    add_index :pgbus_batches, :status,
+      name: "idx_pgbus_batches_status"
+
+    # Create default queues via PGMQ
+    execute "SELECT pgmq.create('pgbus_default')"
+    execute "SELECT pgmq.create('pgbus_default_dlq')"
+  end
+
+  def down
+    execute "SELECT pgmq.drop_queue('pgbus_default_dlq')"
+    execute "SELECT pgmq.drop_queue('pgbus_default')"
+
+    drop_table :pgbus_batches
+    drop_table :pgbus_blocked_executions
+    drop_table :pgbus_semaphores
+    drop_table :pgbus_failed_events
+    drop_table :pgbus_processes
+    drop_table :pgbus_processed_events
+
+    disable_extension "pgmq"
+  end
+end