async-background 0.7.2 → 1.0.1

data/README.md

@@ -1,36 +1,67 @@
 # Async::Background
 
-A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https://github.com/socketry/async) ecosystem. Built for [Falcon](https://github.com/socketry/falcon), works with any Async app.
+A lightweight cron, interval, and job-queue scheduler for Ruby's
+[Async](https://github.com/socketry/async) ecosystem. Built for
+[Falcon](https://github.com/socketry/falcon), works with any Async app.
+
+- **Cron & interval scheduling** on a single event loop with a min-heap.
+- **Dynamic job queue** backed by SQLite, with delayed jobs
+  (`perform_in` / `perform_at`).
+- **Cross-process wake-ups** over Unix domain sockets — web workers can
+  enqueue and instantly wake background workers.
+- **Multi-process safe** — deterministic worker sharding, no duplicate
+  execution.
+- **Per-job timeouts**, skip-on-overlap, startup jitter, optional metrics.
 
-- **Cron & interval scheduling** on a single event loop with a min-heap
-- **Dynamic job queue** backed by SQLite, with delayed jobs (`perform_in` / `perform_at`)
-- **Cross-process wake-ups** over Unix domain sockets — web workers can enqueue and instantly wake background workers
-- **Multi-process safe** — deterministic worker sharding, no duplicate execution
-- **Per-job timeouts**, skip-on-overlap, startup jitter, optional metrics
+---
+
+## Why Async? Why fibers?
+
+The whole gem is built around the assumption that Falcon's reactor schedules
+many fibers on top of one OS thread per process — so the dashboard's SSE
+stream, the cron scheduler, and the queue worker all share that one thread
+cooperatively. A blocked fiber yields; a blocked thread doesn't.
+
+![Threads vs fibers under different Ruby web servers](docs/fibers-vs-threads.svg)
+
+That's also why the dashboard (since 1.0.1) runs its SSE loop entirely inside
+the request fiber: zero extra threads, zero `ConditionVariable`, ~4 KB per
+open tab.
+
+---
 
 ## Requirements
 
-- Ruby >= 3.3
-- `async ~> 2.0`, `fugit ~> 1.0`
-- `sqlite3 ~> 2.0` (optional, for the job queue)
-- `async-utilization >= 0.3, < 0.5` (optional, for metrics)
+| Dependency           | Version          | Required?            |
+| -------------------- | ---------------- | -------------------- |
+| Ruby                 | `>= 3.3`         | yes                  |
+| `async`              | `~> 2.0`         | yes                  |
+| `fugit`              | `~> 1.0`         | yes                  |
+| `sqlite3`            | `~> 2.0`         | for the queue & dashboard |
+| `async-utilization`  | `>= 0.3, < 0.5`  | for metrics          |
+
+---
 
 ## Install
 
 ```ruby
 # Gemfile
 gem "async-background"
-gem "sqlite3", "~> 2.0"            # optional
-gem "async-utilization", ">= 0.3", "< 0.5"  # optional
+
+gem "sqlite3",           "~> 2.0"          # if you use the queue or dashboard
+gem "async-utilization", ">= 0.3", "< 0.5" # if you want worker metrics
 ```
 
-## ➡️ [Get Started](docs/GET_STARTED.md)
+---
+
+## ➡️ [Get started](docs/GET_STARTED.md)
 
-Full setup walkthrough: schedule config, Falcon integration, Docker, queue, delayed jobs.
+A four-step walkthrough: schedule config, Falcon integration, Docker, queue,
+delayed jobs.
 
 ---
 
-## Quick Look
+## Quick look
 
 ```ruby
 class SendEmailJob
@@ -59,108 +90,71 @@ daily_report:
   timeout: 120
 ```
 
-| Key | Description |
-|---|---|
-| `class` | Job class — must include `Async::Background::Job` |
-| `every` / `cron` | One of: interval in seconds, or cron expression |
-| `timeout` | Max execution time in seconds (default: 30) |
-| `worker` | Pin to a specific worker. Default: `crc32(name) % total_workers` |
+| Key              | Description                                                     |
+| ---------------- | --------------------------------------------------------------- |
+| `class`          | Job class — must include `Async::Background::Job`.              |
+| `every` / `cron` | Interval in seconds, or a cron expression. Exactly one.         |
+| `timeout`        | Max execution time in seconds. Default: 30.                     |
+| `worker`         | Pin to a specific worker. Default: `crc32(name) % total_workers`. |
 
 ---
 
 ## Gotchas
 
-### Docker: SQLite requires a named volume
-
-The SQLite database **must not** live on Docker's `overlay2` filesystem. The `overlay2` driver breaks coherence between `write()` and `mmap()`, which corrupts SQLite WAL under concurrent access.
-
-```yaml
-# docker-compose.yml
-services:
-  app:
-    volumes:
-      - queue-data:/app/tmp/queue   # ← named volume, NOT overlay2
-
-volumes:
-  queue-data:
-```
-
-Without this, you will get database crashes in multi-process mode. See [Get Started → Step 3](docs/GET_STARTED.md#step-3-docker) for details. If you can't use a named volume, pass `queue_mmap: false` to disable mmap entirely.
-
-### Other gotchas
-
-**Don't share SQLite connections across `fork()`.** The gem opens connections lazily after fork, but if you create a `Queue::Store` manually for schema setup, close it before forking:
+**Docker + SQLite — use a named volume.**
+SQLite's database must not live on Docker's default `overlay2` filesystem:
+`overlay2` breaks coherence between `write()` and `mmap()`, which corrupts
+the WAL under concurrent access. Mount the queue directory as a named volume,
+or pass `mmap: false` to `Store.new`. See
+[Get Started → Docker](docs/GET_STARTED.md#step-3--docker-setup).
 
-```ruby
-Async::Background::Queue.migrate!(path: db_path) # ← once, before fork
-# Every process opens its own Store lazily after fork.
-```
+**Don't share SQLite connections across `fork()`.**
+The gem opens connections lazily after fork. If you build a `Store` manually
+for schema setup, close it before forking.
 
-**Two clocks, on purpose.** Interval jobs use `CLOCK_MONOTONIC` (immune to NTP drift). Cron jobs use wall-clock time, because "every day at 3am" needs to mean 3am.
+**Two clocks, on purpose.**
+Interval jobs use `CLOCK_MONOTONIC` so NTP drift can't fire them twice. Cron
+jobs use wall-clock time, because "every day at 3am" needs to mean 3am.
 
 ---
 
 ## How it works
 
+A single Async task sleeps until the next entry is due, then dispatches it
+under a semaphore that caps concurrency. Overlapping ticks are skipped and
+rescheduled.
+
 ```
-schedule.yml ─► build_heap ─► MinHeap<Entry> ─► scheduler loop ─► Semaphore ─► run_job
+schedule.yml → build_heap → MinHeap<Entry> → scheduler loop → Semaphore → run_job
 ```
 
-A single Async task sleeps until the next entry is due, then dispatches it under a semaphore that caps concurrency. Overlapping ticks are skipped and rescheduled.
-
 The dynamic queue runs alongside it:
 
 ```
-   Producer (web/console)              Consumer (background worker)
-          │                                       │
-          ▼                                       ▼
-    Queue::Client                          Queue::Store#fetch
-   push / push_in / push_at                (run_at <= now)
-          │                                       ▲
-          ▼                                       │
-    Queue::Store ──── SQLite (jobs) ──── SocketWaker
-          │                                       ▲
-          └───────► SocketNotifier ───────────────┘
-                    (UNIX socket wake-up, ~80µs)
-```
-
-Jobs are persisted in SQLite, so a missed wake-up is never a lost job — workers also poll every 5 seconds as a safety net.
-
-### Schema migration during deploy
-
-Run queue migrations once in the release/pre-deploy step, before starting new web or worker
-processes. This serializes the schema upgrade with `BEGIN IMMEDIATE`, records the version in
-SQLite, and avoids a first producer doing DDL under live queue traffic:
-
-```ruby
-Async::Background::Queue.migrate!(path: ENV.fetch("QUEUE_DB_PATH"))
+   Producer (web / console)                Consumer (background worker)
+          │                                          │
+          ▼                                          ▼
+    Queue::Client                            Queue::Store#fetch
+   push / push_in / push_at                  (run_at <= now)
+          │                                          ▲
+          ▼                                          │
+    Queue::Store ──── SQLite (jobs) ──── SocketWaker │
+          │                                          ▲
+          └─────────► SocketNotifier ────────────────┘
+                      (UNIX socket wake-up, ~80µs)
 ```
 
-A fresh database still self-initializes on first use for local development, but explicit
-migration is the production path. For an existing queue, finish or stop 0.7.1 producers/workers,
-run the migration once, then start 0.7.2 processes.
+Jobs are persisted in SQLite, so a missed wake-up is never a lost job —
+workers also poll every 5 seconds as a safety net.
 
-### Future dashboard indexes
-
-The queue does **not** install dashboard indexes by default. They slow every enqueue even though
-pending rows never enter terminal or in-flight read-model indexes. When the 1.0 dashboard module
-is enabled, its installer will run this once in the same release step:
-
-```ruby
-Async::Background::Queue.prepare_dashboard!(path: ENV.fetch("QUEUE_DB_PATH"))
-```
-
-It adds three compact indexes: one each for cursor-sorted done and failed jobs, plus one for
-the bounded in-flight list. It does not change queue behavior or rerun the core migration.
+---
 
 ## Metrics
 
-Metrics are an optional integration with `async-utilization` (`>= 0.3`, `< 0.5`). The
-background worker remains fully functional when that gem is absent. With it installed, each
-worker publishes counters to a shared-memory segment.
+Metrics are an optional integration with `async-utilization`. With the gem
+installed, each worker publishes counters to a shared-memory segment:
 
 ```ruby
-runner.metrics.enabled?
 runner.metrics.values
 # => { total_runs: 142, total_successes: 140, total_failures: 2,
 #      total_timeouts: 0, total_skips: 5, active_jobs: 1, ... }
@@ -169,19 +163,13 @@ Async::Background::Metrics.read_all(total_workers: 2)
 # => [{ worker: 1, ... }, { worker: 2, ... }]
 ```
 
-`Metrics.read_all` returns `[]` until the optional gem is installed and a worker has created
-the file, so an observer can render an unavailable state without rescuing `LoadError`. Its
-snapshot is lock-free best effort: cumulative fields (`total_runs`, `total_successes`,
-`total_failures`, `total_timeouts`, `total_skips`) are counters; `active_jobs`,
-`last_run_at`, and `last_duration_ms` are gauges. Fields can describe adjacent moments in time
-rather than one globally atomic instant.
-
-By default the file is `/tmp/async-background.shm`. Set `ASYNC_BACKGROUND_METRICS_PATH`
-or pass `metrics_shm_path:` to `Runner.new` when another observer runs in a separate process
-or container; both sides must see the same mounted file.
-
+Without the gem, `runner.metrics.enabled?` is `false` and `read_all` returns
+`[]` — no `LoadError` to rescue. Configuration and the cross-container
+shared-memory path are covered in
+[Get Started → Optional metrics](docs/GET_STARTED.md#appendix-optional-metrics).
 
+---
 
 ## License
 
-MIT
+MIT.

data/async-background.gemspec

@@ -33,12 +33,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'async',   '~> 2.0'
   spec.add_dependency 'console', '~> 1.0'
   spec.add_dependency 'fugit',   '~> 1.0'
+  spec.add_dependency 'base64',  '~> 0.2'
 
   # Optional: add to your own Gemfile if you need these features
-  #   gem 'sqlite3',          '~> 2.0'  # dynamic job queue
+  #   gem 'sqlite3', '~> 2.0'
   #   gem 'async-utilization', '>= 0.3', '< 0.5' # shared-memory worker metrics
 
   spec.add_development_dependency 'rake',  '~> 13.0'
   spec.add_development_dependency 'rspec', '~> 3.12'
+  spec.add_development_dependency 'rack',  '~> 3.0'
   spec.add_development_dependency 'async-utilization', '>= 0.3', '< 0.5'
 end

data/lib/async/background/metrics.rb

@@ -191,10 +191,12 @@ module Async
 
       def ensure_shm!(total_workers, path)
         required_size = self.class.segment_size * total_workers
+        page_size = IO::Buffer::PAGE_SIZE
+        mapped_size = ((required_size + page_size - 1) / page_size) * page_size
 
         File.open(path, File::CREAT | File::RDWR, 0o644) do |file|
           file.flock(File::LOCK_EX)
-          file.truncate(required_size) if file.size < required_size
+          file.truncate(mapped_size) if file.size < mapped_size
         ensure
           file.flock(File::LOCK_UN) rescue nil
         end

data/lib/async/background/queue/schema.rb

@@ -13,7 +13,12 @@ module Async
         VERSION = 1
         MIGRATION_BUSY_TIMEOUT_MS = 30_000
         CORE_INDEXES = %w[idx_jobs_pending].freeze
-        DASHBOARD_INDEXES = %w[idx_jobs_done_finished_at idx_jobs_failed_finished_at idx_jobs_running].freeze
+        DASHBOARD_INDEXES = %w[
+          idx_jobs_done_finished_at
+          idx_jobs_failed_finished_at
+          idx_jobs_executing_started_at
+          idx_jobs_claimed_locked_at
+        ].freeze
         REQUIRED_INDEXES = CORE_INDEXES
 
         module_function

data/lib/async/background/queue/sql.rb

@@ -192,13 +192,24 @@ module Async
           WHERE status = 'failed'
         SQL
 
-        CREATE_RUNNING_INDEX = <<~SQL.freeze
-          CREATE INDEX IF NOT EXISTS idx_jobs_running
+        CREATE_EXECUTING_INDEX = <<~SQL.freeze
+          CREATE INDEX IF NOT EXISTS idx_jobs_executing_started_at
+          ON jobs(started_at)
+          WHERE status = 'running' AND started_at IS NOT NULL
+        SQL
+
+        CREATE_CLAIMED_INDEX = <<~SQL.freeze
+          CREATE INDEX IF NOT EXISTS idx_jobs_claimed_locked_at
           ON jobs(locked_at)
-          WHERE status = 'running'
+          WHERE status = 'running' AND started_at IS NULL
         SQL
 
-        CREATE_DASHBOARD_INDEXES = [CREATE_DONE_INDEX, CREATE_FAILED_INDEX, CREATE_RUNNING_INDEX].freeze
+        CREATE_DASHBOARD_INDEXES = [
+          CREATE_DONE_INDEX,
+          CREATE_FAILED_INDEX,
+          CREATE_EXECUTING_INDEX,
+          CREATE_CLAIMED_INDEX
+        ].freeze
       end
     end
   end

data/lib/async/background/runner/schedule.rb

@@ -13,6 +13,8 @@ module Async
         private
 
         def build_heap(config_path)
+          return MinHeap.new if config_path.nil?
+
           schedule = load_schedule(config_path)
           build_entries(schedule, monotonic_now)
         end

data/lib/async/background/runner.rb

@@ -29,8 +29,11 @@ module Async
                   :metrics,
                   :queue_store
 
+      # `config_path: nil` explicitly disables recurring jobs. This keeps the
+      # dynamic SQLite queue usable on its own; a supplied path remains strict
+      # so a typo cannot silently disable scheduled work.
       def initialize(
-        config_path:,
+        config_path: nil,
         job_count: 2,
         worker_index:,
         total_workers:,
@@ -53,8 +56,9 @@ module Async
 
         @drain_barrier = ::Async::Barrier.new
         @semaphore = ::Async::Semaphore.new(job_count, parent: @drain_barrier)
-        @heap = build_heap(config_path)
+        @heap = config_path.nil? ? MinHeap.new : build_heap(config_path)
         setup_queue(queue_socket_dir, queue_db_path, queue_mmap)
+        validate_work_source!(config_path)
       end
 
       def run
@@ -82,6 +86,11 @@ module Async
       private
 
       def scheduler_loop(task)
+        # Queue-only workers have no heap entry to sleep on. Keep the runner
+        # alive until #stop / SIGTERM wakes this condition; the queue listener
+        # continues independently in its own Async task.
+        return shutdown.wait if heap.empty? && @listen_queue
+
         loop do
           entry = heap.peek
           break unless entry
@@ -93,6 +102,12 @@ module Async
         end
       end
 
+      def validate_work_source!(config_path)
+        return unless config_path.nil? && !@listen_queue
+
+        raise ConfigError, 'Runner requires config_path or queue_socket_dir'
+      end
+
       def wait_for_next_entry(task, entry)
         wait = [entry.next_run_at - monotonic_now, MIN_SLEEP_TIME].max
         wait_with_shutdown(task, wait)

data/lib/async/background/version.rb

@@ -2,6 +2,6 @@
 
 module Async
   module Background
-    VERSION = '0.7.2'
+    VERSION = '1.0.1'
   end
 end

data/lib/async/background/web/app.rb

@@ -0,0 +1,159 @@
+# frozen_string_literal: true
+
+module Async
+  module Background
+    module Web
+      class App
+        def initialize(config)
+          @config = config.validate!
+          @logger = @config.logger
+          @auth = Auth.new(@config.auth, logger: @logger)
+          @snapshot = Snapshot.new(path: @config.queue_path, counts_cache_ttl: @config.counts_cache_ttl).open!
+          @metrics_reader = build_metrics_reader
+          @serializer = Serializer.new(@config)
+          @event_hub = build_event_hub
+          @router = Router.new
+        end
+
+        def call(env)
+          head = env['REQUEST_METHOD'] == 'HEAD'
+          response = handle(env, head: head)
+          return response unless head
+
+          status, headers, _body = response
+          [status, headers, []]
+        end
+
+        def close
+          @event_hub&.close
+          @snapshot.close
+          self
+        end
+
+        private
+
+        def handle(env, head:)
+          return Response.unauthorized unless @auth.authorized?(env)
+
+          route = @router.match(env)
+          return Response.not_found unless route
+
+          if head && route == :stream
+            return @config.transport == :sse ? [200, Response.sse_headers, []] : Response.not_found
+          end
+
+          dispatch(route, env)
+        rescue RequestError => error
+          Response.bad_request(error.message)
+        rescue UnavailableError, ClosedError
+          Response.unavailable
+        rescue StandardError => error
+          # Do not turn internal class names, paths or database errors into an
+          # unauthenticated information disclosure channel — but do surface
+          # them to the operator via the configured logger.
+          log_internal_error(env, error)
+          Response.internal_error
+        end
+
+        def build_metrics_reader
+          return unless @config.metrics_enabled?
+
+          MetricsReader.new(path: @config.metrics_path, total_workers: @config.total_workers)
+        end
+
+        def build_event_hub
+          return unless @config.transport == :sse
+
+          EventHub.new(@snapshot, @serializer, metrics_reader: @metrics_reader)
+        end
+
+        def dispatch(route, env)
+          case route
+          when :index then Response.html(Assets.render_index(@config))
+          when :javascript then Response.javascript(Assets::JS)
+          when :stylesheet then Response.stylesheet(Assets::CSS)
+          when :overview then overview_response
+          when :executing then in_flight_response(:executing, env)
+          when :claimed then in_flight_response(:claimed, env)
+          when :done then terminal_response(:done, env)
+          when :failed then terminal_response(:failed, env)
+          when :pending then pending_response(env)
+          when :metrics then metrics_response
+          when :config then config_response
+          when :stream then stream_response
+          else Response.not_found
+          end
+        end
+
+        def overview_response
+          Response.json(@serializer.overview(@snapshot.overview, metrics_payload))
+        end
+
+        def in_flight_response(kind, env)
+          request = Request.new(env, @config)
+          rows = kind == :executing ? @snapshot.executing(limit: request.limit) : @snapshot.claimed(limit: request.limit)
+          payload = kind == :executing ? @serializer.executing(rows) : @serializer.claimed(rows)
+          Response.json({items: payload})
+        end
+
+        def terminal_response(kind, env)
+          request = Request.new(env, @config)
+          cursor = request.finished_cursor
+          rows = kind == :done ? @snapshot.recent_done(limit: request.limit, cursor: cursor) :
+            @snapshot.recent_failed(limit: request.limit, cursor: cursor)
+          payload = kind == :done ? @serializer.done(rows) : @serializer.failed(rows)
+          Response.json(payload)
+        end
+
+        def pending_response(env)
+          request = Request.new(env, @config)
+          rows = @snapshot.pending(limit: request.limit, cursor: request.pending_cursor)
+          Response.json(@serializer.pending(rows))
+        end
+
+        def metrics_response
+          Response.json(metrics_payload || {available: false, workers: [], totals: MetricsReader::EMPTY_TOTALS})
+        end
+
+        def metrics_payload
+          @metrics_reader&.aggregated
+        end
+
+        def config_response
+          Response.json(
+            {
+              title: @config.title,
+              poll_interval_ms: @config.poll_interval_ms,
+              transport: @config.transport.to_s,
+              expose_args: @config.expose_args,
+              list_limit: @config.list_limit,
+              mount_path: @config.mount_path
+            }
+          )
+        end
+
+        def stream_response
+          return Response.not_found unless @config.transport == :sse
+
+          Response.sse(
+            Stream.new(
+              @event_hub,
+              heartbeat_seconds: @config.stream_heartbeat_seconds,
+              retry_ms: @config.stream_retry_ms,
+              poll_seconds: @config.stream_poll_seconds,
+              logger: @logger
+            )
+          )
+        end
+
+        def log_internal_error(env, error)
+          @logger&.error(
+            "[async-background-web] internal error on " \
+            "#{env['REQUEST_METHOD']} #{env['PATH_INFO']}: " \
+            "#{error.class}: #{error.message}"
+          )
+        end
+      end
+    end
+  end
+end