async-background 0.7.1 → 0.7.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a8f8593628f08094cc53b6c69eff34e3a4be9e8c12de5bd72f1447857aab2682
-  data.tar.gz: e9e441eb2b6f775cd52d2704c8a2b24d9ac771a24beb07d33f78d81285d41f1c
+  metadata.gz: '095ad7028f5492c5f116a86adbbacaabe39167586405f696910e6ff799900831'
+  data.tar.gz: 2eb448ca15863b2ae6177a8ab2f3c4d348b9ecc9453f5603ec50363879c9f50d
 SHA512:
-  metadata.gz: 194c82e280d24821e756849cc9f3f74db8cbadc424d4984fb3a58d10d21864cbe13b1adfa84aa3bf1bd6b9996d59e1d423d5c865c2d1ec33846679ed2ab936f0
-  data.tar.gz: 518bc2a692d0ade6ea02ff18ebca2a3078e02365ffd77fe9850d8bccc64b7243ea871efeec2913f33fe7ecdaeb8fa6a56067e57f36d5852adf59302cc656416e
+  metadata.gz: 9533784a209703347f448b7b5a497c92415a339f4a0c3d2f808f569068f5db64a833d2ae7098b3c64d4dbab497d68da22ca3cd2381d746d552fe1bd98f2f555b
+  data.tar.gz: 4091871c888e1b80832cd764c6c44bfb6e6ca1335f21456855e303d0469ca0127ab1e463d4e23318aabc5e171010d6b3b55764c88f7fa179e650b3d934bf8b7d

data/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 0.7.2
+
+- Harden queue execution, retries, shutdown, and metrics.
+- Add schema v1, optional dashboard indexes, and a faster enqueue path.
+
 ## 0.7.1
 
 ### Features

data/README.md

@@ -13,7 +13,7 @@ A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https:/
 - Ruby >= 3.3
 - `async ~> 2.0`, `fugit ~> 1.0`
 - `sqlite3 ~> 2.0` (optional, for the job queue)
-- `async-utilization ~> 0.3` (optional, for metrics)
+- `async-utilization >= 0.3, < 0.5` (optional, for metrics)
 
 ## Install
 
@@ -21,7 +21,7 @@ A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https:/
 # Gemfile
 gem "async-background"
 gem "sqlite3", "~> 2.0"            # optional
-gem "async-utilization", "~> 0.3"  # optional
+gem "async-utilization", ">= 0.3", "< 0.5"  # optional
 ```
 
 ## ➡️ [Get Started](docs/GET_STARTED.md)
@@ -92,9 +92,8 @@ Without this, you will get database crashes in multi-process mode. See [Get Star
 **Don't share SQLite connections across `fork()`.** The gem opens connections lazily after fork, but if you create a `Queue::Store` manually for schema setup, close it before forking:
 
 ```ruby
-store = Async::Background::Queue::Store.new(path: db_path)
-store.ensure_database!
-store.close  # ← before fork
+Async::Background::Queue.migrate!(path: db_path) # ← once, before fork
+# Every process opens its own Store lazily after fork.
 ```
 
 **Two clocks, on purpose.** Interval jobs use `CLOCK_MONOTONIC` (immune to NTP drift). Cron jobs use wall-clock time, because "every day at 3am" needs to mean 3am.
@@ -127,19 +126,61 @@ The dynamic queue runs alongside it:
 
 Jobs are persisted in SQLite, so a missed wake-up is never a lost job — workers also poll every 5 seconds as a safety net.
 
+### Schema migration during deploy
+
+Run queue migrations once in the release/pre-deploy step, before starting new web or worker
+processes. This serializes the schema upgrade with `BEGIN IMMEDIATE`, records the version in
+SQLite, and avoids a first producer doing DDL under live queue traffic:
+
+```ruby
+Async::Background::Queue.migrate!(path: ENV.fetch("QUEUE_DB_PATH"))
+```
+
+A fresh database still self-initializes on first use for local development, but explicit
+migration is the production path. For an existing queue, finish or stop 0.7.1 producers/workers,
+run the migration once, then start 0.7.2 processes.
+
+### Future dashboard indexes
+
+The queue does **not** install dashboard indexes by default. They slow every enqueue even though
+pending rows never enter terminal or in-flight read-model indexes. When the 1.0 dashboard module
+is enabled, its installer will run this once in the same release step:
+
+```ruby
+Async::Background::Queue.prepare_dashboard!(path: ENV.fetch("QUEUE_DB_PATH"))
+```
+
+It adds three compact indexes: one each for cursor-sorted done and failed jobs, plus one for
+the bounded in-flight list. It does not change queue behavior or rerun the core migration.
+
 ## Metrics
 
-With `async-utilization` installed, per-worker stats land in shared memory at `/tmp/async-background.shm` with lock-free updates.
+Metrics are an optional integration with `async-utilization` (`>= 0.3`, `< 0.5`). The
+background worker remains fully functional when that gem is absent. With it installed, each
+worker publishes counters to a shared-memory segment.
 
 ```ruby
+runner.metrics.enabled?
 runner.metrics.values
 # => { total_runs: 142, total_successes: 140, total_failures: 2,
 #      total_timeouts: 0, total_skips: 5, active_jobs: 1, ... }
 
 Async::Background::Metrics.read_all(total_workers: 2)
+# => [{ worker: 1, ... }, { worker: 2, ... }]
 ```
 
-Without the gem, metrics are silently disabled — zero overhead.
+`Metrics.read_all` returns `[]` until the optional gem is installed and a worker has created
+the file, so an observer can render an unavailable state without rescuing `LoadError`. Its
+snapshot is lock-free best effort: cumulative fields (`total_runs`, `total_successes`,
+`total_failures`, `total_timeouts`, `total_skips`) are counters; `active_jobs`,
+`last_run_at`, and `last_duration_ms` are gauges. Fields can describe adjacent moments in time
+rather than one globally atomic instant.
+
+By default the file is `/tmp/async-background.shm`. Set `ASYNC_BACKGROUND_METRICS_PATH`
+or pass `metrics_shm_path:` to `Runner.new` when another observer runs in a separate process
+or container; both sides must see the same mounted file.
+
+
 
 ## License
 

data/async-background.gemspec

@@ -36,8 +36,9 @@ Gem::Specification.new do |spec|
 
   # Optional: add to your own Gemfile if you need these features
   #   gem 'sqlite3',          '~> 2.0'  # dynamic job queue
-  #   gem 'async-utilization', '~> 0.3' # shared-memory worker metrics
+  #   gem 'async-utilization', '>= 0.3', '< 0.5' # shared-memory worker metrics
 
   spec.add_development_dependency 'rake',  '~> 13.0'
   spec.add_development_dependency 'rspec', '~> 3.12'
+  spec.add_development_dependency 'async-utilization', '>= 0.3', '< 0.5'
 end

data/lib/async/background/job.rb

@@ -4,6 +4,7 @@ module Async
   module Background
     module Job
       DEFAULT_TIMEOUT = 120
+      EMPTY_OPTIONS   = {}.freeze
       BACKOFFS        = %i[fixed linear exponential].freeze
       DEFAULT_JITTER_FOR = { fixed: 0.0, linear: 0.0, exponential: 0.5 }.freeze
 
@@ -55,15 +56,16 @@ module Async
       module ClassMethods
         def perform_now(*args) = new.perform(*args)
 
-        def perform_async(*args, options: {})     = Async::Background::Queue.enqueue(self, *args, options: options)
-        def perform_in(delay, *args, options: {}) = Async::Background::Queue.enqueue_in(delay, self, *args, options: options)
-        def perform_at(time, *args, options: {})  = Async::Background::Queue.enqueue_at(time, self, *args, options: options)
+        def perform_async(*args, options: EMPTY_OPTIONS)     = Async::Background::Queue.enqueue(self, *args, options: options)
+        def perform_in(delay, *args, options: EMPTY_OPTIONS) = Async::Background::Queue.enqueue_in(delay, self, *args, options: options)
+        def perform_at(time, *args, options: EMPTY_OPTIONS)  = Async::Background::Queue.enqueue_at(time, self, *args, options: options)
 
         def options(**values)
           @options = Options.new(**values).to_h.compact
         end
 
         def resolve_options = @options || {}
+        def queue_options = @options || EMPTY_OPTIONS
       end
 
       def perform(*)

data/lib/async/background/metrics.rb

@@ -1,143 +1,214 @@
 # frozen_string_literal: true
 
+require 'tmpdir'
+
 module Async
   module Background
     class Metrics
       SCHEMA_FIELDS = {
-        total_runs:       :u64,
-        total_successes:  :u64,
-        total_failures:   :u64,
-        total_timeouts:   :u64,
-        total_skips:      :u64,
-        active_jobs:      :u32,
-        last_run_at:      :u64,
+        total_runs: :u64,
+        total_successes: :u64,
+        total_failures: :u64,
+        total_timeouts: :u64,
+        total_skips: :u64,
+        active_jobs: :u32,
+        last_run_at: :u64,
         last_duration_ms: :u32
       }.freeze
 
-      attr_reader :registry
+      EMPTY_HANDLES = {}.freeze
 
-      def initialize(worker_index:, total_workers:, shm_path: self.class.default_shm_path)
-        require 'async/utilization'
+      attr_reader :registry, :shm_path, :unavailable_reason
 
+      def initialize(worker_index:, total_workers:, shm_path: self.class.default_shm_path)
+        @enabled = false
         @registry = nil
-        @enabled  = false
-        @registry = ::Async::Utilization::Registry.new
-        @enabled  = true
-        ensure_shm!(total_workers, shm_path)
-        attach_observer!(worker_index, total_workers, shm_path)
-      rescue LoadError, ArgumentError
-        @registry = nil
-        @enabled  = false
+        @metric_handles = EMPTY_HANDLES
+        @shm_path = shm_path
+        @unavailable_reason = nil
+
+        validate_worker!(worker_index, total_workers)
+        initialize_registry!(worker_index, total_workers, shm_path)
+      rescue LoadError => error
+        mark_unavailable!(error)
       end
 
-      def enabled?
-        @enabled
+      def enabled? = @enabled
+
+      def job_started(_entry)
+        return unless enabled?
+
+        increment(:total_runs)
+        increment(:active_jobs)
+        set(:last_run_at, Process.clock_gettime(Process::CLOCK_REALTIME).to_i)
       end
 
-      def job_started(entry)
-        return unless @enabled
+      def job_succeeded(_entry, duration)
+        return unless enabled?
 
-        @registry.increment(:total_runs)
-        @registry.increment(:active_jobs)
-        @registry.set(:last_run_at, Process.clock_gettime(Process::CLOCK_REALTIME).to_i)
+        increment(:total_successes)
+        set(:last_duration_ms, duration_to_milliseconds(duration))
       end
 
       def job_finished(entry, duration)
-        return unless @enabled
+        job_succeeded(entry, duration)
+        job_stopped(entry)
+      end
 
-        @registry.decrement(:active_jobs)
-        @registry.increment(:total_successes)
-        @registry.set(:last_duration_ms, (duration * 1000).to_i)
+      def job_failed(_entry, _error)
+        increment(:total_failures) if enabled?
       end
 
-      def job_failed(entry, error)
-        return unless @enabled
+      def job_timed_out(_entry)
+        increment(:total_timeouts) if enabled?
+      end
 
-        @registry.decrement(:active_jobs)
-        @registry.increment(:total_failures)
+      def job_stopped(_entry)
+        decrement(:active_jobs) if enabled?
       end
 
-      def job_timed_out(entry)
-        return unless @enabled
+      def job_skipped(_entry)
+        increment(:total_skips) if enabled?
+      end
 
-        @registry.decrement(:active_jobs)
-        @registry.increment(:total_timeouts)
+      def values
+        enabled? ? registry.values : {}
       end
 
-      def job_skipped(entry)
-        return unless @enabled
+      class << self
+        def available?
+          load_utilization!
+          true
+        rescue LoadError
+          false
+        end
 
-        @registry.increment(:total_skips)
-      end
+        def load_utilization!
+          require 'async/utilization'
+        end
 
-      def values
-        return {} unless @enabled
+        def schema
+          load_utilization!
+          ::Async::Utilization::Schema.build(SCHEMA_FIELDS)
+        end
 
-        @registry.values
-      end
+        def read_all(total_workers:, path: default_shm_path)
+          validate_total_workers!(total_workers)
+          return [] unless available? && File.file?(path)
 
-      def self.schema
-        require 'async/utilization'
-        ::Async::Utilization::Schema.build(SCHEMA_FIELDS)
-      end
+          layout = schema
+          segment = segment_size
+          required_size = segment * total_workers
+
+          File.open(path, 'rb') do |file|
+            return [] if file.size < required_size
 
-      # Read metrics for all workers from the shm file.
-      # No server needed — just reads the mmap'd file.
-      #
-      #   Async::Background::Metrics.read_all(total_workers: 2)
-      #   # => [
-      #   #   { worker: 1, total_runs: 142, active_jobs: 1, ... },
-      #   #   { worker: 2, total_runs: 98,  active_jobs: 0, ... }
-      #   # ]
-      #
-      def self.read_all(total_workers:, path: default_shm_path)
-        require 'async/utilization'
+            buffer = IO::Buffer.map(file, required_size, 0, IO::Buffer::READONLY)
+            decode_workers(buffer, layout, segment, total_workers)
+          end
+        rescue Errno::ENOENT
+          []
+        end
 
-        s = schema
-        segment = segment_size
-        file_size = segment * total_workers
+        def default_shm_path
+          ENV.fetch('ASYNC_BACKGROUND_METRICS_PATH') { File.join(Dir.tmpdir, 'async-background.shm') }
+        end
 
-        buffer = File.open(path, "rb") do |f|
-          IO::Buffer.map(f, file_size, 0)
+        def segment_size
+          SCHEMA_FIELDS.sum { |_, type| IO::Buffer.size_of(type) }
         end
 
-        (1..total_workers).map do |i|
-          base = (i - 1) * segment
-          row = { worker: i }
-          s.fields.each do |field|
-            row[field.name] = buffer.get_value(field.type, base + field.offset)
-          end
-          row
+        private
+
+        def validate_total_workers!(total_workers)
+          return if total_workers.is_a?(Integer) && total_workers.positive?
+
+          raise ArgumentError, 'total_workers must be a positive Integer'
+        end
+
+        def decode_workers(buffer, schema, segment, total_workers)
+          (1..total_workers).map do |worker|
+            decode_worker(buffer, schema, segment, worker)
+          end.freeze
+        end
+
+        def decode_worker(buffer, schema, segment, worker)
+          offset = (worker - 1) * segment
+          schema.fields.each_with_object(worker: worker) do |field, values|
+            values[field.name] = buffer.get_value(field.type, offset + field.offset)
+          end.freeze
         end
       end
 
-      def self.default_shm_path
-        File.join(Dir.tmpdir, "async-background.shm")
+      private
+
+      def initialize_registry!(worker_index, total_workers, path)
+        self.class.load_utilization!
+        ensure_shm!(total_workers, path)
+
+        @registry = ::Async::Utilization::Registry.new
+        unless @registry.respond_to?(:metric)
+          raise LoadError, 'async-utilization >= 0.3 is required for metrics'
+        end
+
+        attach_observer!(worker_index, path)
+        @metric_handles = SCHEMA_FIELDS.keys.to_h { |name| [name, @registry.metric(name)] }.freeze
+        @enabled = true
       end
 
-      def self.segment_size
-        SCHEMA_FIELDS.sum { |_, type| IO::Buffer.size_of(type) }
+      def mark_unavailable!(error)
+        @registry = nil
+        @metric_handles = EMPTY_HANDLES
+        @unavailable_reason = error.message
       end
 
-      private
+      def increment(name)
+        metric(name).increment
+      end
+
+      def decrement(name)
+        metric(name).decrement
+      end
+
+      def set(name, value)
+        metric(name).set(value)
+      end
+
+      def metric(name)
+        @metric_handles.fetch(name)
+      end
+
+      def duration_to_milliseconds(duration)
+        (duration * 1000).to_i
+      end
+
+      def validate_worker!(worker_index, total_workers)
+        self.class.send(:validate_total_workers!, total_workers)
+        return if worker_index.is_a?(Integer) && worker_index.between?(1, total_workers)
+
+        raise ArgumentError, 'worker_index must be an Integer between 1 and total_workers'
+      end
 
       def ensure_shm!(total_workers, path)
-        required = self.class.segment_size * total_workers
+        required_size = self.class.segment_size * total_workers
 
-        File.open(path, File::CREAT | File::RDWR, 0644) do |f|
-          f.flock(File::LOCK_EX)
-          f.truncate(required) if f.size < required
-          f.flock(File::LOCK_UN)
+        File.open(path, File::CREAT | File::RDWR, 0o644) do |file|
+          file.flock(File::LOCK_EX)
+          file.truncate(required_size) if file.size < required_size
+        ensure
+          file.flock(File::LOCK_UN) rescue nil
         end
       end
 
-      def attach_observer!(worker_index, total_workers, path)
+      def attach_observer!(worker_index, path)
         segment = self.class.segment_size
-        offset  = (worker_index - 1) * segment
         observer = ::Async::Utilization::Observer.open(
-          self.class.schema, path, segment, offset
+          self.class.schema,
+          path,
+          segment,
+          (worker_index - 1) * segment
         )
-        @registry.observer = observer
+        registry.observer = observer
       end
     end
   end

data/lib/async/background/queue/client.rb

@@ -1,12 +1,11 @@
 # frozen_string_literal: true
 
 require_relative '../clock'
+require_relative 'store'
 
 module Async
   module Background
     module Queue
-      EMPTY_OPTIONS = {}.freeze
-
       class Client
         include Clock
 
@@ -15,17 +14,17 @@ module Async
           @notifier = notifier
         end
 
-        def push(class_name, args = [], run_at = nil, options: {})
+        def push(class_name, args = EMPTY_ARGS, run_at = nil, options: EMPTY_OPTIONS)
           id = @store.enqueue(class_name, args, run_at, options: options)
           @notifier&.notify_all
           id
         end
 
-        def push_in(delay, class_name, args = [], options: {})
+        def push_in(delay, class_name, args = EMPTY_ARGS, options: EMPTY_OPTIONS)
           push(class_name, args, realtime_now + delay.to_f, options: options)
         end
 
-        def push_at(time, class_name, args = [], options: {})
+        def push_at(time, class_name, args = EMPTY_ARGS, options: EMPTY_OPTIONS)
           run_at = time.respond_to?(:to_f) ? time.to_f : time
           push(class_name, args, run_at, options: options)
         end
@@ -34,19 +33,27 @@ module Async
       class << self
         attr_accessor :default_client
 
-        def enqueue(job_class, *args, options: {})
+        def migrate!(path: Store.default_path, options: {})
+          Store.migrate!(path: path, options: options)
+        end
+
+        def prepare_dashboard!(path: Store.default_path, options: {})
+          Store.prepare_dashboard!(path: path, options: options)
+        end
+
+        def enqueue(job_class, *args, options: EMPTY_OPTIONS)
           ensure_configured!
-          default_client.push(resolve_class_name(job_class), args, nil, options: build_options(job_class, options))
+          default_client.push(resolve_class_name(job_class), normalized_args(args), nil, options: build_options(job_class, options))
         end
 
-        def enqueue_in(delay, job_class, *args, options: {})
+        def enqueue_in(delay, job_class, *args, options: EMPTY_OPTIONS)
           ensure_configured!
-          default_client.push_in(delay, resolve_class_name(job_class), args, options: build_options(job_class, options))
+          default_client.push_in(delay, resolve_class_name(job_class), normalized_args(args), options: build_options(job_class, options))
         end
 
-        def enqueue_at(time, job_class, *args, options: {})
+        def enqueue_at(time, job_class, *args, options: EMPTY_OPTIONS)
           ensure_configured!
-          default_client.push_at(time, resolve_class_name(job_class), args, options: build_options(job_class, options))
+          default_client.push_at(time, resolve_class_name(job_class), normalized_args(args), options: build_options(job_class, options))
         end
 
         private
@@ -55,8 +62,11 @@ module Async
         private_constant :RETRY_KEYS
 
         def build_options(job_class, call_site)
-          call_site ||= {}
-          merged = resolve_options(job_class).merge(call_site.compact)
+          call_site ||= EMPTY_OPTIONS
+          class_options = resolve_options(job_class)
+          return EMPTY_OPTIONS if class_options.empty? && call_site.empty?
+
+          merged = class_options.merge(call_site.compact)
           apply_retry_overrides!(merged, call_site)
 
           merged.empty? ? EMPTY_OPTIONS : Job::Options.new(**merged).to_h.compact
@@ -72,6 +82,8 @@ module Async
           raise "Async::Background::Queue not configured" unless default_client
         end
 
+        def normalized_args(args) = args.empty? ? EMPTY_ARGS : args
+
         def resolve_class_name(job_class)
           return job_class if job_class.is_a?(String)
           return job_class.name if job_class.respond_to?(:perform_now)
@@ -80,9 +92,15 @@ module Async
         end
 
         def resolve_options(job_class)
-          return {} unless job_class.respond_to?(:resolve_options)
+          return EMPTY_OPTIONS unless job_class.respond_to?(:resolve_options)
+
+          options = if job_class.respond_to?(:queue_options)
+            job_class.queue_options
+          else
+            job_class.resolve_options
+          end
 
-          job_class.resolve_options.dup
+          options.empty? ? EMPTY_OPTIONS : options
         end
       end
     end

data/lib/async/background/queue/options.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module Async
+  module Background
+    module Queue
+      EMPTY_ARGS = [].freeze
+      EMPTY_OPTIONS = {}.freeze
+
+      SYNCHRONOUS_LEVELS = {normal: 'NORMAL', full: 'FULL', extra: 'EXTRA'}.freeze
+      WAL_AUTOCHECKPOINT_RANGE = 100..10_000
+      DEFAULT_STORE_OPTIONS = {mmap: true, synchronous: :normal, wal_autocheckpoint: 1_000}.freeze
+      DEFAULTS = DEFAULT_STORE_OPTIONS
+      MMAP_SIZE = 268_435_456
+
+      StoreOptions = Data.define(:mmap, :synchronous, :wal_autocheckpoint) do
+        def self.build(value = {})
+          return value if value.is_a?(self)
+
+          new(**DEFAULT_STORE_OPTIONS, **value)
+        end
+
+        def initialize(mmap:, synchronous:, wal_autocheckpoint:)
+          validate_mmap!(mmap)
+          validate_synchronous!(synchronous)
+          validate_wal_autocheckpoint!(wal_autocheckpoint)
+
+          super
+        end
+
+        def synchronous_pragma = SYNCHRONOUS_LEVELS.fetch(synchronous)
+        def mmap_size = mmap ? MMAP_SIZE : 0
+
+        def pragma_sql
+          <<~SQL
+            PRAGMA journal_mode       = WAL;
+            PRAGMA synchronous        = #{synchronous_pragma};
+            PRAGMA mmap_size          = #{mmap_size};
+            PRAGMA cache_size         = -16000;
+            PRAGMA temp_store         = MEMORY;
+            PRAGMA journal_size_limit = 67108864;
+            PRAGMA wal_autocheckpoint = #{wal_autocheckpoint};
+          SQL
+        end
+
+        private
+
+        def validate_mmap!(value)
+          return if value == true || value == false
+
+          raise ArgumentError, "mmap must be true or false, got #{value.inspect}"
+        end
+
+        def validate_synchronous!(value)
+          return if SYNCHRONOUS_LEVELS.key?(value)
+
+          raise ArgumentError,
+                "synchronous must be one of #{SYNCHRONOUS_LEVELS.keys.inspect}, got #{value.inspect}"
+        end
+
+        def validate_wal_autocheckpoint!(value)
+          return if value.is_a?(Integer) && WAL_AUTOCHECKPOINT_RANGE.cover?(value)
+
+          raise ArgumentError,
+                "wal_autocheckpoint must be an Integer in #{WAL_AUTOCHECKPOINT_RANGE}, " \
+                "got #{value.inspect}"
+        end
+      end
+    end
+  end
+end