pgbus 0.0.1 → 0.1.2

data/docs/switch_from_good_job.md

@@ -1,279 +0,0 @@
-# Switch from GoodJob to Pgbus
-
-## Overview
-
-GoodJob and Pgbus are both PostgreSQL-native job processors with LISTEN/NOTIFY support. The biggest architectural difference: GoodJob uses advisory locks and a `good_jobs` table, while Pgbus uses PGMQ (a dedicated message queue extension) with visibility timeouts. Both are pure ActiveJob adapters.
-
-**Effort estimate:** Low if you use standard ActiveJob. Medium if you rely on GoodJob's concurrency controls, batches, or cron.
-
-## Step 1: Update dependencies
-
-```ruby
-# Gemfile
-
-# Remove
-gem "good_job"
-
-# Add
-gem "pgbus"
-```
-
-```bash
-bundle install
-rails generate pgbus:install
-rails db:migrate
-```
-
-## Step 2: Switch the adapter
-
-```ruby
-# config/application.rb
-
-# Before
-config.active_job.queue_adapter = :good_job
-
-# After
-config.active_job.queue_adapter = :pgbus
-```
-
-Remove GoodJob configuration:
-
-```ruby
-# Before: Remove this block
-config.good_job = {
-  execution_mode: :async,
-  queues: "critical:5;default:3;*:1",
-  max_threads: 5,
-  poll_interval: 10,
-  enable_cron: true,
-  enable_listen_notify: true,
-  # ...
-}
-```
-
-## Step 3: Convert worker configuration
-
-```ruby
-# Before: GoodJob config (in application.rb or environment files)
-config.good_job = {
-  execution_mode: :external,  # or :async
-  queues: "critical:5;default:3;low:1",
-  poll_interval: 10,
-  max_threads: 5,
-  shutdown_timeout: 30,
-}
-```
-
-```yaml
-# After: config/pgbus.yml
-production:
-  workers:
-    - queues: [critical]
-      threads: 5
-    - queues: [default]
-      threads: 3
-    - queues: [low]
-      threads: 1
-  max_retries: 5
-  max_jobs_per_worker: 10000
-  max_memory_mb: 512
-  max_worker_lifetime: 3600
-```
-
-### Configuration mapping
-
-| GoodJob | Pgbus | Notes |
-|---------|-------|-------|
-| `execution_mode: :external` | `bundle exec pgbus start` | Separate process (recommended for production) |
-| `execution_mode: :async` | N/A | Pgbus always runs as separate processes |
-| `queues: "critical:5;default:3"` | `workers:` array | One entry per worker process |
-| `max_threads` | `threads` per worker | Per-worker, not global |
-| `poll_interval` | `polling_interval` | Pgbus defaults to 0.1s; LISTEN/NOTIFY is primary |
-| `enable_listen_notify` | `listen_notify` | Both support LISTEN/NOTIFY |
-| `shutdown_timeout` | Handled by supervisor | Graceful shutdown on SIGTERM |
-| `on_thread_error` | Configure via `Pgbus.logger` | Error reporting via logging |
-| `preserve_job_records` | Always archived | PGMQ archives completed messages |
-
-## Step 4: Remove concurrency controls
-
-GoodJob's concurrency extensions are GoodJob-specific. Remove them:
-
-```ruby
-# Before
-class ProcessOrderJob < ApplicationJob
-  include GoodJob::ActiveJobExtensions::Concurrency
-
-  good_job_control_concurrency_with(
-    total_limit: 1,
-    enqueue_limit: 2,
-    perform_limit: 1,
-    enqueue_throttle: [10, 1.minute],
-    perform_throttle: [100, 1.hour],
-    key: -> { "ProcessOrder-#{arguments.first.id}" }
-  )
-
-  def perform(order)
-    # ...
-  end
-end
-```
-
-```ruby
-# After
-class ProcessOrderJob < ApplicationJob
-  def perform(order)
-    # ...
-  end
-end
-```
-
-> Pgbus supports concurrency controls via `Pgbus::Concurrency`:
-> ```ruby
-> class ProcessOrderJob < ApplicationJob
->   include Pgbus::Concurrency
->   limits_concurrency to: 1,
->                      key: -> { "ProcessOrder-#{arguments.first.id}" },
->                      duration: 15.minutes,
->                      on_conflict: :block
->   def perform(order)
->     # ...
->   end
-> end
-> ```
-
-## Step 5: Migrate batches
-
-If you use `GoodJob::Batch`:
-
-```ruby
-# Before: GoodJob batches
-GoodJob::Batch.enqueue(
-  on_finish: BatchCallbackJob,
-  on_success: SuccessNotifyJob,
-  description: "Import users"
-) do
-  users.each { |u| ImportUserJob.perform_later(u) }
-end
-```
-
-Pgbus does not yet have batch support. Workaround using a coordinator job:
-
-```ruby
-# After: Coordinator pattern
-class ImportUsersJob < ApplicationJob
-  def perform(user_ids)
-    user_ids.each { |id| ImportUserJob.perform_later(id) }
-    # Track completion via a counter in the database or Redis
-  end
-end
-```
-
-## Step 6: Migrate cron / recurring jobs
-
-```ruby
-# Before: GoodJob cron
-config.good_job.enable_cron = true
-config.good_job.cron = {
-  daily_cleanup: {
-    cron: "0 2 * * *",
-    class: "CleanupJob",
-    set: { priority: -10, queue: "maintenance" },
-  },
-  hourly_sync: {
-    cron: "0 * * * *",
-    class: "SyncJob",
-    args: [42],
-  }
-}
-```
-
-Pgbus does not yet have built-in recurring task support. Options:
-
-1. **Use the `whenever` gem**:
-   ```ruby
-   # config/schedule.rb
-   every 1.day, at: "2:00 am" do
-     runner "CleanupJob.perform_later"
-   end
-   every :hour do
-     runner "SyncJob.perform_later(42)"
-   end
-   ```
-
-2. **Use system cron** directly:
-   ```cron
-   0 2 * * * cd /app && bin/rails runner "CleanupJob.perform_later"
-   0 * * * * cd /app && bin/rails runner "SyncJob.perform_later(42)"
-   ```
-
-3. Wait for Pgbus recurring task support (planned).
-
-## Step 7: Replace the dashboard
-
-```ruby
-# Before: config/routes.rb
-mount GoodJob::Engine => "good_job"
-
-# After:
-mount Pgbus::Engine => "/pgbus"
-```
-
-## Step 8: Update process management
-
-```bash
-# Before (external mode)
-bundle exec good_job start
-
-# After
-bundle exec pgbus start
-```
-
-If you ran GoodJob in `async` mode (in-process), note that Pgbus always runs as separate forked processes managed by a supervisor. Update your deployment accordingly -- you need `bundle exec pgbus start` as a separate process.
-
-## Step 9: Clean up GoodJob tables
-
-After verifying Pgbus is processing correctly and GoodJob's tables are drained:
-
-```ruby
-class RemoveGoodJob < ActiveRecord::Migration[7.1]
-  def up
-    drop_table :good_jobs, if_exists: true
-    drop_table :good_job_batches, if_exists: true
-    drop_table :good_job_executions, if_exists: true
-    drop_table :good_job_processes, if_exists: true
-    drop_table :good_job_settings, if_exists: true
-  end
-end
-```
-
-## What you gain
-
-- **Dead letter queues** -- GoodJob retries in-place and marks jobs as discarded. Pgbus routes failures to dedicated `_dlq` queues for clear operational visibility.
-- **Worker recycling** -- GoodJob workers run indefinitely. Pgbus recycles by job count, memory, or lifetime to prevent memory bloat.
-- **Event bus** -- AMQP-style pub/sub with topic routing and idempotent handlers.
-- **PGMQ** -- purpose-built message queue extension with atomic read/archive/delete, visibility timeouts, and `SKIP LOCKED` under the hood.
-- **Supervisor/fork model** -- isolated worker processes. A memory leak or crash in one worker doesn't affect others.
-
-## What you lose (for now)
-
-| GoodJob feature | Status in Pgbus |
-|-----------------|-----------------|
-| Concurrency controls (`good_job_control_concurrency_with`) | `Pgbus::Concurrency` with `limits_concurrency` DSL |
-| Throttling (`enqueue_throttle`, `perform_throttle`) | Planned |
-| Batches (`GoodJob::Batch`) | `Pgbus::Batch` with on_finish/on_success/on_discard callbacks |
-| Cron / recurring jobs | Planned |
-| Async execution mode (in-process) | Not planned (forked processes only) |
-| Capsules (isolated thread pools) | Workers are isolated by design (forked processes) |
-| Advisory locks | Replaced by PGMQ visibility timeouts |
-
-## Gotchas
-
-1. **PgBouncer**: Both GoodJob and Pgbus use LISTEN/NOTIFY, which requires session-mode PgBouncer or direct connections. If you already had GoodJob working with PgBouncer, the same configuration applies.
-
-2. **No async mode**: GoodJob can run in-process (`:async` mode) alongside your Rails app server. Pgbus requires a separate supervisor process. Make sure your deployment runs `bundle exec pgbus start` alongside your web server.
-
-3. **Advisory locks vs. visibility timeouts**: GoodJob uses PostgreSQL advisory locks to claim jobs. Pgbus uses PGMQ's visibility timeout -- a claimed message becomes invisible for `visibility_timeout` seconds. If a worker crashes without archiving the message, it automatically becomes available again after the timeout expires. This is more resilient than advisory locks, which release on disconnect.
-
-4. **Job record preservation**: GoodJob has `preserve_job_records` for keeping completed job records. PGMQ automatically archives completed messages to `a_pgbus_*` tables, which serve a similar purpose for debugging and auditing.
-
-5. **Queue naming**: GoodJob uses bare queue names. Pgbus prefixes all queues (`pgbus_default`). Your `queue_as` declarations work unchanged -- the prefix is applied automatically.

data/docs/switch_from_sidekiq.md

@@ -1,226 +0,0 @@
-# Switch from Sidekiq to Pgbus
-
-## Overview
-
-Sidekiq uses Redis as its message broker. Pgbus uses PostgreSQL via PGMQ. Switching eliminates Redis from your infrastructure and gives you dead letter queues, worker recycling, and an event bus -- all backed by your existing database.
-
-**Effort estimate:** Low if you use ActiveJob exclusively. Medium-high if you use native Sidekiq workers or Pro/Enterprise features.
-
-## Step 1: Update dependencies
-
-```ruby
-# Gemfile
-
-# Remove
-gem "sidekiq"
-gem "sidekiq-cron"          # if used
-gem "sidekiq-unique-jobs"   # if used
-
-# Add
-gem "pgbus"
-```
-
-```bash
-bundle install
-rails generate pgbus:install
-rails db:migrate
-```
-
-## Step 2: Switch the adapter
-
-```ruby
-# config/application.rb (or config/environments/production.rb)
-
-# Before
-config.active_job.queue_adapter = :sidekiq
-
-# After
-config.active_job.queue_adapter = :pgbus
-```
-
-## Step 3: Convert native Sidekiq workers
-
-If all your jobs inherit from `ApplicationJob` (ActiveJob), skip this step -- they work unchanged.
-
-If you have native Sidekiq workers using `include Sidekiq::Job`, convert them:
-
-```ruby
-# Before: Native Sidekiq worker
-class HardWorker
-  include Sidekiq::Job
-  sidekiq_options queue: :critical, retry: 5
-
-  def perform(user_id, action)
-    user = User.find(user_id)
-    # ...
-  end
-end
-
-# Enqueue
-HardWorker.perform_async(user.id, "activate")
-HardWorker.perform_at(5.minutes.from_now, user.id, "activate")
-```
-
-```ruby
-# After: ActiveJob
-class HardWorker < ApplicationJob
-  queue_as :critical
-  retry_on StandardError, wait: :polynomially_longer, attempts: 5
-
-  def perform(user_id, action)
-    user = User.find(user_id)
-    # ...
-  end
-end
-
-# Enqueue
-HardWorker.perform_later(user.id, "activate")
-HardWorker.set(wait: 5.minutes).perform_later(user.id, "activate")
-```
-
-### API mapping
-
-| Sidekiq | ActiveJob / Pgbus |
-|---------|-------------------|
-| `perform_async(args)` | `perform_later(args)` |
-| `perform_at(time, args)` | `.set(wait_until: time).perform_later(args)` |
-| `perform_in(duration, args)` | `.set(wait: duration).perform_later(args)` |
-| `sidekiq_options queue: :name` | `queue_as :name` |
-| `sidekiq_options retry: N` | `retry_on StandardError, attempts: N` |
-| `sidekiq_retries_exhausted` | `discard_on` + `after_discard` callback |
-
-## Step 4: Replace middleware with ActiveJob callbacks
-
-Sidekiq middleware wraps job push (client) and execution (server) in a Rack-style chain. ActiveJob provides equivalent hooks.
-
-```ruby
-# Before: Sidekiq server middleware
-class LoggingMiddleware
-  include Sidekiq::ServerMiddleware
-  def call(job_instance, job_payload, queue)
-    Rails.logger.info("Starting #{job_payload['class']}")
-    yield
-    Rails.logger.info("Finished #{job_payload['class']}")
-  end
-end
-```
-
-```ruby
-# After: ActiveJob callback (add to ApplicationJob or per-job)
-class ApplicationJob < ActiveJob::Base
-  around_perform do |job, block|
-    Rails.logger.info("Starting #{job.class.name}")
-    block.call
-    Rails.logger.info("Finished #{job.class.name}")
-  end
-end
-```
-
-### Callback mapping
-
-| Sidekiq middleware | ActiveJob callback |
-|--------------------|-------------------|
-| Server middleware (before yield) | `before_perform` |
-| Server middleware (around yield) | `around_perform` |
-| Server middleware (after yield) | `after_perform` |
-| Client middleware (before yield) | `before_enqueue` |
-| Client middleware (around yield) | `around_enqueue` |
-| Client middleware (after yield) | `after_enqueue` |
-
-## Step 5: Configure workers
-
-```yaml
-# Before: config/sidekiq.yml
-:concurrency: 10
-:queues:
-  - [critical, 3]
-  - [default, 2]
-  - [low, 1]
-```
-
-```yaml
-# After: config/pgbus.yml
-production:
-  workers:
-    - queues: [critical]
-      threads: 5
-    - queues: [default, low]
-      threads: 10
-  max_jobs_per_worker: 10000
-  max_memory_mb: 512
-  max_worker_lifetime: 3600
-```
-
-## Step 6: Replace the dashboard
-
-```ruby
-# Before: config/routes.rb
-require "sidekiq/web"
-mount Sidekiq::Web => "/sidekiq"
-
-# After:
-mount Pgbus::Engine => "/pgbus"
-```
-
-If you used `Sidekiq::Web`'s authentication:
-
-```ruby
-Pgbus.configure do |config|
-  config.web_auth = ->(request) {
-    request.env["warden"].user&.admin?
-  }
-end
-```
-
-## Step 7: Update process management
-
-```bash
-# Before
-bundle exec sidekiq
-
-# After
-bundle exec pgbus start
-```
-
-Update your `Procfile`, systemd units, or container entrypoints accordingly.
-
-## Step 8: Remove Redis
-
-Once all Sidekiq jobs have drained and you've verified Pgbus is processing correctly, remove Redis from your infrastructure:
-
-- Remove `REDIS_URL` / Sidekiq Redis config from environment
-- Remove `Sidekiq.configure_server` / `Sidekiq.configure_client` blocks
-- Remove Redis from `docker-compose.yml`, Terraform, etc.
-
-## What you gain
-
-- **No Redis dependency** -- one fewer service to operate
-- **Dead letter queues** -- failed jobs route to `_dlq` queues after `max_retries`, visible in the dashboard
-- **Worker recycling** -- memory, job count, and lifetime limits prevent the memory bloat that plagues long-running Sidekiq processes
-- **Event bus** -- AMQP-style topic routing for pub/sub, built into the same infrastructure
-- **LISTEN/NOTIFY** -- instant job wake-up without polling overhead
-
-## What you lose (for now)
-
-| Sidekiq feature | Status in Pgbus |
-|-----------------|-----------------|
-| Batches (Pro) | `Pgbus::Batch` with on_finish/on_success/on_discard callbacks |
-| Rate limiting (Enterprise) | Planned |
-| Concurrency controls (Enterprise) | `Pgbus::Concurrency` with `limits_concurrency` DSL |
-| Unique jobs (Enterprise / `sidekiq-unique-jobs`) | Partial -- event bus has idempotency; job-level dedup planned |
-| Cron / recurring jobs (`sidekiq-cron`) | Planned |
-| Real-time metrics (Sidekiq Web) | Pgbus dashboard covers queue depth, failures, processes |
-
-## Gotchas
-
-1. **Argument serialization**: Sidekiq passes raw JSON; ActiveJob uses GlobalID for ActiveRecord objects. If you pass raw IDs (`user.id`), both work the same. If you pass AR objects (`user`), ActiveJob serializes via GlobalID automatically.
-
-2. **`Sidekiq::Testing.fake!`**: Replace with `ActiveJob::TestHelper`:
-   ```ruby
-   include ActiveJob::TestHelper
-   assert_enqueued_jobs 1 { MyJob.perform_later(args) }
-   ```
-
-3. **`Sidekiq.redis { |conn| ... }`**: If you use Sidekiq's Redis connection for custom caching or distributed locks, you'll need a separate solution (e.g., PostgreSQL advisory locks, `with_advisory_lock` gem).
-
-4. **Job priorities**: Sidekiq uses queue weight ordering. Pgbus processes queues in the order listed in the worker config. Put higher-priority queues first.