riverqueue 0.11.0 → 0.12.0

data/docs/migrations.md

@@ -0,0 +1,121 @@
+# Schema migrations
+
+River Ruby includes a migration API and the `river` command. Neither needs
+Go installed. The gems bundle byte-for-byte copies of the PostgreSQL and SQLite
+SQL from Go's River drivers, pinned to release v0.48.0 through migration 008;
+only the upstream schema template placeholders are substituted when executing
+them.
+
+## Command line
+
+Install `riverqueue-sequel` or `riverqueue-activerecord` and the application's
+database gem (`pg` or `sqlite3`). The command auto-detects the River driver gem
+available in your bundle, preferring Sequel when both are available. This choice
+does not need to match your application's driver: both run the same migrations.
+Create the database first, then run:
+
+```sh
+bundle exec river migrate-status --database-url postgres://localhost/my_app
+bundle exec river migrate-up --database-url postgres://localhost/my_app
+
+# SQLite, using Sequel:
+bundle exec river migrate-up --database-url sqlite://storage/river.sqlite3
+```
+
+`DATABASE_URL` supplies the URL when `--database-url` is omitted. ActiveRecord
+uses `sqlite3:` URLs instead of Sequel's `sqlite:` URLs. To target an existing
+PostgreSQL schema, pass `--schema jobs`; otherwise the connection's current
+schema is used. Schema names must be simple SQL identifiers. SQLite always
+uses its main schema.
+
+Up applies all pending versions. `--target N` stops at version N; `--steps N`
+limits the number applied. `--dry-run` lists the plan without executing SQL.
+Down defaults to one version and requires explicit confirmation:
+
+```sh
+bundle exec river migrate-down --database-url postgres://localhost/my_app --dry-run
+bundle exec river migrate-down --database-url postgres://localhost/my_app --yes
+```
+
+Down migrations can delete jobs and other data. Back up the database and stop
+workers first. `--target 0 --yes` removes the complete selected migration line.
+
+## Ruby API
+
+```ruby
+driver = River::Driver::Sequel.new(DB)
+migrator = River::Migrator.new(driver)
+
+migrator.status # Version/name records with an applied boolean.
+migrator.migrate # Up to the newest bundled version.
+migrator.migrate(dry_run: true, target: 8)
+migrator.migrate(direction: :down) # One version; no interactive confirmation.
+```
+
+The API accepts either driver and optionally `schema:` for PostgreSQL. `migrate`
+returns the migration records applied (or planned in dry-run mode), with their
+version, name, and original up/down SQL. Each version's SQL and history update
+commit in their own transaction. A failing version rolls back while earlier
+versions remain committed, so rerunning resumes safely. Do not wrap the migrator
+in an application transaction: some PostgreSQL changes require a real commit
+before the next version can run.
+
+An up migration targeting an already-applied version does nothing, even when
+newer migrations are pending.
+
+## Go compatibility and Pro
+
+The migrator reads and writes Go's `river_migration` history, including the
+legacy pre-version-5 format. Either language can continue from the other's
+applied versions. Unknown newer versions and gaps in history cause an error
+rather than guessing what SQL to run. It does not baseline an existing schema
+that has no River migration history.
+
+Pro SQL is distributed only inside `riverqueue-pro`, not the public core gem.
+SQLite main migration 008 rebuilds `river_job` with `AUTOINCREMENT` so deleted
+job IDs are not reused. Its upgrade and downgrade refuse to run while Pro's
+workflow/sequence schema is installed; coordinate that upgrade with Pro before
+running it on an existing Pro database. PostgreSQL migration 008 is a no-op.
+
+Migrate main first, then Pro:
+
+```sh
+bundle exec river migrate-up --database-url postgres://localhost/my_app
+bundle exec river migrate-up --database-url postgres://localhost/my_app --line pro
+```
+
+```ruby
+require "riverqueue-pro"
+
+River::Migrator.new(driver).migrate
+River::Pro::Migrator.new(driver).migrate
+```
+
+Remove other migration lines before downgrading main. Run only one migration
+process at a time across languages. Ruby migrators take a PostgreSQL advisory
+lock per schema; SQLite takes a write lock per version and checks for concurrent
+history changes. These are not shared locks with Go's migration runner.
+
+## Updating the bundled SQL
+
+`migration/manifest.json` records the upstream commit and each file's SHA-256.
+Synchronize or verify the public migrations against a local Go checkout:
+
+```sh
+ruby scripts/sync_migrations.rb ../river
+make verify
+make verify RIVER_PATH=/path/to/river
+```
+
+`make verify` defaults to `../river` and checks SQL contents, filenames, the
+license, and the manifest's checksums and source revision. CI fetches the public
+River repository at that recorded revision and runs the same check; it does not
+require a sibling checkout or access to the private Pro repository.
+
+Pro migrations have their own manifest and sync script in the private
+`riverqueue-ruby-pro` repository. Run that repository's `make verify` against
+the Go Pro checkout; the public sync script handles only the main migrations.
+
+Review the upstream changes and run the complete test matrix before publishing.
+Do not edit the copied SQL independently. Upstream changes are bundled in gem
+releases; users do not need the Go repositories or network access at runtime.

data/docs/testing.md

@@ -0,0 +1,173 @@
+# Testing River jobs
+
+Testing helpers ship in `riverqueue`. They use your real PostgreSQL or SQLite
+database through either driver; there is no fake queue, global testing mode,
+implicit migration, or automatic cleanup. Require them explicitly:
+
+```ruby
+require "riverqueue/testing"
+```
+
+Use an isolated test database/schema and a stopped client with registered workers.
+Do not run background consumers or concurrent tests against the same job rows.
+Your test suite owns migrations and cleanup. Transactional tests work when the
+client uses the same database connection as the test transaction: synchronous
+execution stays on the calling thread. Sharing a database URL is not sufficient.
+
+## RSpec
+
+Add `rspec-expectations` (or `rspec`) to your test bundle, then:
+
+```ruby
+require "riverqueue/testing/rspec"
+
+RSpec.configure do |config|
+  config.include River::Testing::RSpec
+end
+
+expect { enqueue_order(42) }.to insert_job(
+  client,
+  args: {"order_id" => 42},
+  kind: :fulfill_order,
+  queue: :orders
+)
+
+expect { enqueue_orders }.to insert_jobs(client, count: 3, kind: :fulfill_order)
+expect { ignore_duplicate_order }.not_to insert_job(client)
+```
+
+Attributes support composable RSpec expectations, including nested hash/array
+matchers and time tolerances. Literal hashes still require all their keys to
+match; use `a_hash_including` for a subset. JSON object keys are strings.
+Identifier filters (`kind:`, `queue:`, and `state:`) accept symbols or strings;
+literal JSON values and nested matchers are not coerced.
+
+```ruby
+expect { enqueue_order(42) }.to insert_job(
+  client,
+  args: a_hash_including("order_id" => 42),
+  scheduled_at: be_within(1).of(expected_time)
+)
+
+expect { enqueue_orders }.to insert_jobs(client, kind: :fulfill_order).at_least(2)
+expect { enqueue_orders }.to insert_jobs(client).at_most(5)
+expect { enqueue_orders }.to insert_jobs(client).exactly(3)
+```
+
+`insert_job` and `insert_jobs` default to exactly one matching new row. `count:`
+on `insert_jobs` remains shorthand for an exact count. Count chains accept
+nonnegative integers; the last chain determines the count requirement. Other,
+nonmatching jobs do not affect the count. Negation means **zero** matching rows,
+not merely failure of the positive count requirement—even with `at_most` or
+`at_least`. Use a positive `.exactly(0)` to assert a zero count explicitly.
+
+Use `have_job` to inspect existing persisted jobs rather than insertions inside
+a block. It defaults to at least one match and supports the same count chains:
+
+```ruby
+expect(client).to have_job(kind: :fulfill_order, queue: :orders)
+expect(client).to have_job(args: a_hash_including("order_id" => 42)).exactly(1)
+expect(client).not_to have_job(state: :discarded)
+
+expect { enqueue_order_and_notification }.to(
+  insert_job(client, kind: :fulfill_order)
+    .and(insert_job(client, kind: :notify_order))
+)
+```
+
+`have_job` includes every job state unless filtered with `state:`; it does not
+mean only waiting jobs. Insertion matchers support block expectations and
+`have_job` supports value expectations. Compound insertion expectations execute
+the application block once. Matchers only read persisted rows; they do not run
+workers or clean up data. `River::Testing.jobs(client)` also returns all rows
+through paginated reads for custom assertions.
+
+These richer comparisons are RSpec-only. Framework-neutral and Minitest
+insertion assertions use the same identifier normalization, but otherwise retain
+exact attribute equality and exact counts.
+
+## Minitest and other frameworks
+
+Add `minitest` to your test bundle, then explicitly include its integration:
+
+```ruby
+require "minitest/autorun"
+require "riverqueue/testing/minitest"
+
+class OrderTest < Minitest::Test
+  include River::Testing::Minitest
+
+  def test_enqueue
+    row = assert_job_inserted client, args: {"order_id" => 42}, kind: :fulfill_order do
+      enqueue_order(42)
+    end
+    assert_equal "orders", row.queue
+
+    assert_jobs_inserted(client, count: 3) { enqueue_orders }
+    assert_no_jobs_inserted(client) { ignore_duplicate_order }
+  end
+end
+```
+
+`client`, `enqueue_order`, and `enqueue_orders` above are application/test fixture
+methods. The integration uses Minitest's assertion counts and failure reporting;
+it does not enable autorun itself. Other frameworks can include
+`River::Testing::Assertions` (failures raise `River::Testing::AssertionError`) or
+use `River::Testing.inserted_jobs(client, **attributes) { ... }` directly.
+
+Insertion checks compare row IDs before and after the block. A uniqueness conflict
+returning an existing row does not count as an insertion. Rows deleted or rolled
+back before the block ends cannot be observed. Exceptions from your block propagate.
+
+## Execute one real attempt
+
+```ruby
+result = River::Testing.perform_job(client, row.id)
+
+expect(result).to have_attributes(error: nil, outcome: :completed)
+expect(result.job).to have_attributes(attempt: 1, state: "completed")
+```
+
+This claims the specified job atomically, then uses the real runtime, including
+plugins, timeout, retries, error handling, snoozes, cancellation, resumable steps,
+metadata/output persistence, and finalization. It does not start consumer threads,
+maintenance services, or periodic producers. The client cannot be started or used
+for another synchronous attempt until execution finishes.
+
+`ExecutionResult` contains `id`, `error` (the original exception, or `nil`), `job`
+(the row after execution, or `nil` if deleted), and `outcome`: `:completed`,
+`:retried`, `:discarded`, `:cancelled`, `:snoozed`, `:interrupted`, or `:deleted`.
+Worker errors are returned, not re-raised; assert the outcome so a failed worker
+cannot accidentally pass your test. Database/finalization failures can still raise.
+
+Minitest and framework-neutral assertion modules also provide
+`assert_job_completed(result)`, `assert_job_cancelled(result)`, and
+`assert_job_discarded(result)`; each returns the result on success.
+
+Only available, scheduled, or retryable jobs are eligible. Future jobs require an
+explicit override; this does not change their original `scheduled_at`:
+
+```ruby
+result = River::Testing.perform_job(client, row.id, allow_scheduled: true)
+```
+
+Direct execution bypasses queue pause, worker capacity, and Pro concurrency claim
+controls. It is a worker test helper, not a scheduler simulator. Pro plugins may
+have additional effects (such as a batch worker claiming additional jobs).
+Use regular threaded integration tests for these behaviors and maintenance-driven
+workflow progression. With Rails, continue to use `ActiveJob::TestHelper` for
+adapter-independent tests; use these helpers with a real River adapter/client to
+test persistence and execution.
+
+## Drain a queue
+
+```ruby
+results = River::Testing.drain(client, max_jobs: 20, queue: :orders)
+expect(results.map(&:outcome)).to all(eq(:completed))
+```
+
+Draining runs due jobs sequentially in priority, scheduled-time, then ID order,
+including jobs inserted by workers. It does not sleep for future jobs or run
+maintenance. Due scheduled/retryable jobs can be claimed directly. The default
+limit is 100 attempts; if runnable work remains at the limit, it raises
+`River::Testing::DrainLimitError`. Attempts already performed remain persisted.

data/docs/workers.md

@@ -0,0 +1,126 @@
+# Dedicated worker processes
+
+Run River consumers separately from web processes with `river worker`. The
+command owns one foreground process: it boots your application, starts a client,
+handles stop signals, and waits for workers. It does not daemonize, fork
+children, run migrations, or supervise replacement processes.
+
+## Plain Ruby
+
+Install `riverqueue`, your driver gem, and the database gem that driver needs.
+Apply [migrations](./migrations.md) before starting consumers. Create a Ruby
+configuration file whose **last expression returns an unstarted client**:
+
+```ruby
+# config/river.rb
+require_relative "../app" # Boots your application and defines SortWorker.
+require "riverqueue-sequel"
+
+River::Client.new(
+  River::Driver::Sequel.new(
+    Sequel.connect(ENV.fetch("DATABASE_URL"), max_connections: 20)
+  ),
+  config: River::Config.new(
+    queues: {ruby: 10},
+    workers: River::Workers.new.add(SortWorker)
+  )
+)
+```
+
+Run from your application's root directory:
+
+```sh
+bundle exec river worker --config config/river.rb
+```
+
+Do not call `.start` in the configuration file or install your own signal
+handlers. Configure at least one queue and register its workers. Producers must
+insert into the same database/schema and queue names. Ten workers means up to
+ten concurrent job threads in this process, not ten OS processes.
+
+The configuration file is evaluated as Ruby, not parsed as data; use only trusted
+application code. It can return an ActiveRecord-backed client or a
+`River::Pro::Client` instead. Pro must be installed and required by the application;
+the command does not load it automatically. Pool sizing is application-specific:
+allow connections for queue producers, maintenance, and application database work
+as well as worker threads. The example's pool size is illustrative.
+
+## Rails
+
+Install and configure [riverqueue-rails](../rails/riverqueue-rails/README.md), then
+run from the Rails application root:
+
+```sh
+RAILS_ENV=production bundle exec river worker --rails
+```
+
+This boots `config/environment.rb` and builds a consumer from
+`Rails.application.config.river`, including the configured connection class,
+Active Job worker, and Rails execution wrapping. Configure native workers through
+the same integration when needed. Keep web processes insertion-only; do not start
+a consumer in a web initializer.
+
+The generated `bin/jobs start` also invokes the shared runner. Use the `river`
+command for the CLI options shown here. Do not combine `--rails` with `--config`.
+
+## Stopping
+
+```sh
+bundle exec river worker --config config/river.rb --stop-timeout 60
+```
+
+The graceful stop timeout defaults to 30 seconds for plain Ruby. Rails uses
+`config.river.stop_timeout`; `--stop-timeout` overrides it. The value must
+be finite and nonnegative; zero skips the grace period. It is separate from the
+per-job execution timeout.
+
+| Signal / event        | Behavior                                                                                 |
+|-----------------------|------------------------------------------------------------------------------------------|
+| `SIGTSTP`             | Calls `client.stop(wait: false)`: stops accepting work but keeps the process alive.         |
+| `SIGTERM` / `SIGINT`   | Requests a stop, waits for active attempts, then exits.                                    |
+| Another stop signal   | Skips the remaining grace period and interrupts active attempts.                          |
+| Grace period expires  | Interrupts active attempts and allows five seconds for finalization.                      |
+| Finalization expires  | Forces process exit with status 1, bypassing `at_exit` handlers.                           |
+
+After interruption, a further stop signal can force exit before the finalization
+window ends. In-flight fetches or maintenance operations may finish after stop
+is requested. `SIGTSTP` does not pause queues in the database, does not start the
+stop deadline, and does not exit automatically when work drains. Send
+`SIGTERM` afterward to complete stop; there is no signal to resume fetching.
+
+Attempts successfully finalized as interrupted become available again without
+consuming the attempt. If the process dies before finalization, running jobs are
+left for normal stuck-job recovery. Workers must tolerate retries; stop cannot
+roll back external side effects. Keep an external supervisor's kill deadline as
+a backstop for uninterruptible native calls or blocked application code.
+
+For `river worker`, a clean drain exits with status 0. Boot failures, detected
+runtime-thread failures, and interrupted stops return status 1; forced
+termination also exits with status 1. Lifecycle messages go to standard output;
+worker logging uses the client's configured logger.
+
+## Deployment
+
+Configure your deployment platform's worker command as one of the commands above.
+Run multiple independent instances for multiple processes; queue concurrency is
+per process. A shell wrapper should use `exec` so signals reach the worker:
+
+```sh
+exec bundle exec river worker --config config/river.rb --stop-timeout 30
+```
+
+Give the supervisor more than the graceful timeout plus the five-second
+finalization window before it forcibly kills the process, with additional margin
+for scheduling and cleanup. Migrate once as a deployment step, not on each worker
+boot. Configure restart policy in the supervisor, accounting for status 1 after
+an interrupted stop.
+
+For a rolling replacement, start new instances, confirm startup and database
+connectivity, then send `SIGTERM` to old instances and allow them to drain. An
+optional earlier `SIGTSTP` stops old instances from accepting more work. The
+`River worker: ready pid=...` log means the initial database check and client start
+have completed; it is not a health endpoint or a guarantee of ongoing queue health.
+
+River does not provide a multiprocess supervisor, rolling-restart coordinator,
+PID-file management, or a readiness probe server. Those remain deployment-platform
+responsibilities.

data/exe/river

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "riverqueue"
+require_relative "../lib/cli"
+
+exit River::CLI.run(ARGV)

data/lib/cli.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+require "optparse"
+require_relative "migration_cli"
+require_relative "worker_runner"
+
+module River
+  # Shared command dispatch for migrations and dedicated worker processes.
+  module CLI
+    # Runs a River command and returns its process exit status.
+    def self.run(argv, err: $stderr, out: $stdout)
+      return worker(argv.drop(1), out: out) if argv.first == "worker"
+
+      if argv.empty? || %w[-h --help].include?(argv.first)
+        out.puts("Usage: river worker|migrate-up|migrate-down|migrate-status [options]")
+        return 0
+      end
+
+      MigrationCLI.run(argv, err: err, out: out)
+    rescue StandardError, LoadError, SyntaxError => error
+      err.puts("Worker failed: #{error.class}: #{error.message}")
+      1
+    end
+
+    def self.worker(argv, out:)
+      options = {} #: Hash[Symbol, untyped]
+      parser = OptionParser.new do |args|
+        args.banner = "Usage: river worker --config FILE | --rails [options]"
+        args.on("--config FILE", "Ruby file returning an unstarted River client") { |value| options[:config] = value }
+        args.on("--rails", "Boot config/environment.rb in the current directory") { options[:rails] = true }
+        args.on("--stop-timeout SECONDS", Float) { |value| options[:stop_timeout] = value }
+        args.on("-h", "--help") do
+          out.puts(args)
+          return 0
+        end
+      end
+
+      args = argv.dup
+      parser.parse!(args)
+      raise ArgumentError, parser.banner unless args.empty? && (!!options[:config] ^ !!options[:rails])
+
+      if options[:rails]
+        require File.expand_path("config/environment.rb")
+        require "riverqueue-rails"
+        Object.const_get("River::Rails::Runner").start(out: out, stop_timeout: options[:stop_timeout])
+      else
+        path = File.expand_path(options[:config])
+        client = TOPLEVEL_BINDING.eval(File.read(path), path)
+        raise ArgumentError, "configuration must return a River::Client" unless client.is_a?(River::Client)
+
+        WorkerRunner.new(client, out: out, stop_timeout: options.fetch(:stop_timeout, 30)).run
+      end
+    end
+  end
+end