async-background 0.7.2 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '095ad7028f5492c5f116a86adbbacaabe39167586405f696910e6ff799900831'
-  data.tar.gz: 2eb448ca15863b2ae6177a8ab2f3c4d348b9ecc9453f5603ec50363879c9f50d
+  metadata.gz: 3d7244ccfb7156c423229ed59f36df9354d293711bab53e6a87d943743bc7f44
+  data.tar.gz: 84752c52033b89c5b957d39976f5d025a56215b378138eda8813c05635c5d58e
 SHA512:
-  metadata.gz: 9533784a209703347f448b7b5a497c92415a339f4a0c3d2f808f569068f5db64a833d2ae7098b3c64d4dbab497d68da22ca3cd2381d746d552fe1bd98f2f555b
-  data.tar.gz: 4091871c888e1b80832cd764c6c44bfb6e6ca1335f21456855e303d0469ca0127ab1e463d4e23318aabc5e171010d6b3b55764c88f7fa179e650b3d934bf8b7d
+  metadata.gz: 6e90dd0f6be9ff7361aba7d84e7a7cdfdd712b351938c50dc8b84358d10a20f65514880a3eac3ab2ad8c0adf5164c06f9668ead03df5161fc4482b4e27aa03ec
+  data.tar.gz: 1867b17975c63966aeaf437414795b3b5d9083c9871a9f6c44dc6b7280651bbf46a8c929b4751349f969ffbdf72c32aeb8c6965634d149e74a0a352e4ef958df

data/CHANGELOG.md

@@ -1,5 +1,112 @@
 # Changelog
 
+## Unreleased
+
+### Dashboard SSE hardening
+
+- Make SSE the default dashboard transport. `:polling` remains an explicit compatibility fallback.
+- Replace one `PRAGMA data_version` loop **per browser connection** with one shared `EventHub` watcher per Rack app process while at least one dashboard tab is connected. It fans complete overview snapshots to every SSE body through latest-value mailboxes, so slow tabs do not accumulate unbounded event queues.
+- Use complete snapshots on connect and after a change rather than a replay log: reconnecting EventSource clients cannot miss the current queue state.
+- Coalesce client-side list refreshes after a burst of changes, cancel stale list fetches on tab switches, and retain previous pages when `Load more` is used.
+- Add SSE retry control, 25-second heartbeat frames, `no-cache, no-transform`, and `X-Accel-Buffering: no`. Remove the hop-by-hop `Connection` response header.
+- Fingerprint asset URLs and cache immutable assets by digest, so a dashboard deploy cannot leave a browser on incompatible HTML/JS/CSS.
+- Add coverage for event fan-out, latest-value coalescing, clean stream shutdown, forced overview reads, and the SSE configuration constraints.
+
+## 1.1.0
+
+Server-Sent Events transport for the dashboard. Replaces HTTP polling as the recommended transport.
+
+### Added
+
+- **SSE transport for the dashboard.** Set `c.transport = :sse` in `Async::Background::Web.configure` and the dashboard now uses a single long-lived `text/event-stream` connection per browser tab instead of polling `/api/overview` every 2 seconds. The browser opens `EventSource(mount_path + '/api/stream')` once; the server pushes an `overview` event whenever `PRAGMA data_version` changes, and a `:keepalive` comment frame every 30 seconds. Result: 1 HTTP connection per dashboard tab regardless of how long it stays open, instead of 30 req/min per tab.
+
+  - New module `Async::Background::Web::Stream` implements the event loop as a Rack streaming body (responds to `#each`, yields SSE frames). Holds no state across requests.
+  - New route `GET /api/stream` returns `200 text/event-stream` when `transport == :sse`, `404` otherwise. Subject to the same auth gate as every other endpoint.
+  - New `Response.sse(body)` helper sets the correct headers including `x-accel-buffering: no` (disables nginx buffering for the streaming response).
+  - JS client (`assets.rb`) detects `state.config.transport === 'sse'` at boot and chooses `EventSource` over `setInterval(tick, ...)`. Both transports share the same `applyOverview()` and `refreshActiveList()` handlers, so the UI behaves identically.
+
+- **`Configuration#transport`** with default `:polling` (backward compatible) and accepted values `:polling | :sse`. Validation rejects anything else with `ConfigurationError`. The chosen transport is exposed at `/api/config` so the client knows which path to take.
+
+### Migration
+
+Existing deployments keep working unchanged. To opt into SSE:
+
+```ruby
+Async::Background::Web.configure do |c|
+  c.queue_path = ...
+  c.auth = ->(env) { ... }
+  c.transport = :sse
+end
+```
+
+### Server compatibility note
+
+SSE holds the request thread/fiber open for the lifetime of the dashboard tab. **Recommended for Falcon**, which handles long-lived connections natively via fibers. **Puma works** but each open dashboard tab holds one worker thread for its lifetime — fine for an admin dashboard with a handful of operators, problematic if many concurrent operators would starve the worker pool. **Unicorn does not work** for SSE since its blocking worker model can't hold long-lived connections without timeouts; stay on `:polling` there.
+
+### Backend-side polling
+
+The server still polls `PRAGMA data_version` every 500ms inside the snapshot connection to detect changes. This is a connection-local PRAGMA call, microseconds, never hits a rate limiter. Client-facing transport is push.
+
+### Tests
+
+- New `spec/async/background/web/stream_spec.rb` — covers overview event on data_version change, heartbeat after idle, graceful exit on `EPIPE`/`IOError`, error frame on `ClosedError`/`UnavailableError`.
+- Extended `spec/async/background/web/app_spec.rb` — `/api/stream` returns 404 on polling default, 200 text/event-stream on `:sse`, 401 without auth.
+- Extended `spec/async/background/web/configuration_spec.rb` — accepts `:sse`, rejects unknown transports.
+
+## 1.0.0
+
+First stable release. The queue execution contract from 0.7.2 (claim-token CAS, lifecycle columns, barrier-based shutdown drain, per-status partial indexes, versioned migrations) is now considered the public API.
+
+### Features
+
+- **Web dashboard.** Rack-mountable read-only UI under `require 'async/background/web'`. Vanilla HTML/CSS/JS, no JS framework, no npm.
+  - Endpoints: `GET /`, `GET /assets/app.css`, `GET /assets/app.js`, `GET /api/overview`, `GET /api/executing`, `GET /api/claimed`, `GET /api/pending`, `GET /api/done`, `GET /api/failed`, `GET /api/metrics`, `GET /api/config`.
+  - Default transport is JSON polling (`poll_interval_ms`, default 2000). SSE adapter for Falcon is intentionally deferred to a later release; the dashboard already coalesces work via a shared overview cache, so adding SSE later is a backward-compatible change.
+  - Read path runs through `Async::Background::Web::Snapshot`, which opens SQLite with `file:?mode=ro`, wraps a `Mutex` around a single shared connection, and uses one read transaction per endpoint and caches each overview as one consistent snapshot.
+  - Distinguishes `Executing` (`status='running' AND started_at IS NOT NULL`) from `Claimed` (`status='running' AND started_at IS NULL`).
+  - Overview snapshot cache for `counts_cache_ttl` seconds (default 3.0) so a busy queue does not turn the dashboard into a hot reader.
+  - Cursor pagination for `done`/`failed`/`pending` using `(finished_at, id)` / `(run_at, id)` tuples. Stable on ties.
+  - Args hidden by default (`expose_args: false`); when enabled, content runs through `redact_args`. All user content rendered through `textContent`, never `innerHTML`.
+  - Auth hook is **mandatory**. `Configuration#validate!` rejects an unconfigured `auth`. There is no permissive default.
+
+  - Add the optional Rack dashboard for the SQLite queue.
+  - Make sqlite3 an explicit runtime dependency for queue/dashboard installs.
+
+### Configuration
+
+```ruby
+require 'async/background/web'
+
+Async::Background::Queue::Store.prepare_dashboard!(path: '/var/lib/app/queue.db')
+
+Async::Background::Web.configure do |c|
+  c.queue_path        = '/var/lib/app/queue.db'
+  c.auth              = ->(env) { env['warden'].user&.admin? }
+  c.expose_args       = false
+  c.metrics_path      = '/run/app/async-background.shm'
+  c.total_workers     = 4
+  c.counts_cache_ttl  = 3.0
+  c.poll_interval_ms  = 2000
+  c.list_limit        = 50
+  c.mount_path        = '/admin/background'
+  c.title             = 'My App background jobs'
+end
+
+run Async::Background::Web.app
+```
+
+### Dependencies
+
+- `rack` is an optional dependency. Required only when `require 'async/background/web'` is loaded. Core gem and worker processes do not require it.
+
+### Breaking changes from 0.7.x
+
+None beyond what 0.7.2 already shipped. The 1.0 line locks the existing contract:
+
+- `Queue::Store#fetch` returns `claim_token` in the result hash.
+- All terminal `Queue::Store` methods (`complete`, `fail`, `retry_or_fail`) require the `claim_token:` kwarg and return CAS success boolean / `:retried` / `:failed` / `nil`.
+- Schema is versioned via `PRAGMA user_version`. Use `Queue::Store.migrate!(path:)` to upgrade. Use `Queue::Store.prepare_dashboard!(path:)` from the dashboard process to lazily create dashboard-only indexes (per-status partial indexes for `done` / `failed`, plus separate `executing` and `claimed` indexes).
+
 ## 0.7.2
 
 - Harden queue execution, retries, shutdown, and metrics.
@@ -102,7 +209,7 @@
   - Proper job distribution validation across worker pool
 - **Test fixtures** — dedicated `ci/fixtures/jobs.rb` and `ci/fixtures/schedule.yml` for scenario testing
 
-### Bug Fixes  
+### Bug Fixes
 - **SQLite busy timeout** — added `PRAGMA busy_timeout = 5000` to `Queue::Store` to prevent `SQLITE_BUSY` errors under concurrent multi-process database access
 - **Enhanced Queue::Notifier error handling** — restructured IO error handling with clearer categorization:
   - `WRITE_DROPPED` for write failures (`IO::WaitWritable`, `Errno::EAGAIN`, `IOError`, `Errno::EPIPE`) — all non-fatal as job is already in store

data/README.md

@@ -12,7 +12,7 @@ A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https:/
 
 - Ruby >= 3.3
 - `async ~> 2.0`, `fugit ~> 1.0`
-- `sqlite3 ~> 2.0` (optional, for the job queue)
+- `sqlite3 ~> 2.0` (optional, storage)
 - `async-utilization >= 0.3, < 0.5` (optional, for metrics)
 
 ## Install
@@ -20,7 +20,7 @@ A lightweight cron, interval, and job-queue scheduler for Ruby's [Async](https:/
 ```ruby
 # Gemfile
 gem "async-background"
-gem "sqlite3", "~> 2.0"            # optional
+gem "sqlite3", "~> 2.0"  # optional
 gem "async-utilization", ">= 0.3", "< 0.5"  # optional
 ```
 
@@ -126,6 +126,12 @@ The dynamic queue runs alongside it:
 
 Jobs are persisted in SQLite, so a missed wake-up is never a lost job — workers also poll every 5 seconds as a safety net.
 
+### Queue-only workers
+
+Recurring schedules are optional. A worker that serves only dynamic jobs starts with
+`config_path: nil` and a `queue_socket_dir`; it does not need a placeholder schedule file.
+A supplied schedule path stays strict and raises when the file is missing or empty.
+
 ### Schema migration during deploy
 
 Run queue migrations once in the release/pre-deploy step, before starting new web or worker
@@ -140,18 +146,18 @@ A fresh database still self-initializes on first use for local development, but
 migration is the production path. For an existing queue, finish or stop 0.7.1 producers/workers,
 run the migration once, then start 0.7.2 processes.
 
-### Future dashboard indexes
+### Dashboard indexes
 
 The queue does **not** install dashboard indexes by default. They slow every enqueue even though
-pending rows never enter terminal or in-flight read-model indexes. When the 1.0 dashboard module
-is enabled, its installer will run this once in the same release step:
+pending rows never enter terminal or in-flight read-model indexes. Enable them once before
+mounting the dashboard:
 
 ```ruby
 Async::Background::Queue.prepare_dashboard!(path: ENV.fetch("QUEUE_DB_PATH"))
 ```
 
-It adds three compact indexes: one each for cursor-sorted done and failed jobs, plus one for
-the bounded in-flight list. It does not change queue behavior or rerun the core migration.
+It adds four compact indexes: cursor-sorted `done` / `failed` history plus separate
+`executing` / `claimed` in-flight lists. It does not change queue behavior or rerun the core migration.
 
 ## Metrics
 

data/async-background.gemspec

@@ -33,12 +33,14 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'async',   '~> 2.0'
   spec.add_dependency 'console', '~> 1.0'
   spec.add_dependency 'fugit',   '~> 1.0'
+  spec.add_dependency 'base64',  '~> 0.2'
 
   # Optional: add to your own Gemfile if you need these features
-  #   gem 'sqlite3',          '~> 2.0'  # dynamic job queue
+  #   gem 'sqlite3', '~> 2.0'
   #   gem 'async-utilization', '>= 0.3', '< 0.5' # shared-memory worker metrics
 
   spec.add_development_dependency 'rake',  '~> 13.0'
   spec.add_development_dependency 'rspec', '~> 3.12'
+  spec.add_development_dependency 'rack',  '~> 3.0'
   spec.add_development_dependency 'async-utilization', '>= 0.3', '< 0.5'
 end

data/lib/async/background/metrics.rb

@@ -191,10 +191,12 @@ module Async
 
       def ensure_shm!(total_workers, path)
         required_size = self.class.segment_size * total_workers
+        page_size = IO::Buffer::PAGE_SIZE
+        mapped_size = ((required_size + page_size - 1) / page_size) * page_size
 
         File.open(path, File::CREAT | File::RDWR, 0o644) do |file|
           file.flock(File::LOCK_EX)
-          file.truncate(required_size) if file.size < required_size
+          file.truncate(mapped_size) if file.size < mapped_size
         ensure
           file.flock(File::LOCK_UN) rescue nil
         end

data/lib/async/background/queue/schema.rb

@@ -13,7 +13,12 @@ module Async
         VERSION = 1
         MIGRATION_BUSY_TIMEOUT_MS = 30_000
         CORE_INDEXES = %w[idx_jobs_pending].freeze
-        DASHBOARD_INDEXES = %w[idx_jobs_done_finished_at idx_jobs_failed_finished_at idx_jobs_running].freeze
+        DASHBOARD_INDEXES = %w[
+          idx_jobs_done_finished_at
+          idx_jobs_failed_finished_at
+          idx_jobs_executing_started_at
+          idx_jobs_claimed_locked_at
+        ].freeze
         REQUIRED_INDEXES = CORE_INDEXES
 
         module_function

data/lib/async/background/queue/sql.rb

@@ -192,13 +192,24 @@ module Async
           WHERE status = 'failed'
         SQL
 
-        CREATE_RUNNING_INDEX = <<~SQL.freeze
-          CREATE INDEX IF NOT EXISTS idx_jobs_running
+        CREATE_EXECUTING_INDEX = <<~SQL.freeze
+          CREATE INDEX IF NOT EXISTS idx_jobs_executing_started_at
+          ON jobs(started_at)
+          WHERE status = 'running' AND started_at IS NOT NULL
+        SQL
+
+        CREATE_CLAIMED_INDEX = <<~SQL.freeze
+          CREATE INDEX IF NOT EXISTS idx_jobs_claimed_locked_at
           ON jobs(locked_at)
-          WHERE status = 'running'
+          WHERE status = 'running' AND started_at IS NULL
         SQL
 
-        CREATE_DASHBOARD_INDEXES = [CREATE_DONE_INDEX, CREATE_FAILED_INDEX, CREATE_RUNNING_INDEX].freeze
+        CREATE_DASHBOARD_INDEXES = [
+          CREATE_DONE_INDEX,
+          CREATE_FAILED_INDEX,
+          CREATE_EXECUTING_INDEX,
+          CREATE_CLAIMED_INDEX
+        ].freeze
       end
     end
   end

data/lib/async/background/runner/schedule.rb

@@ -13,6 +13,8 @@ module Async
         private
 
         def build_heap(config_path)
+          return MinHeap.new if config_path.nil?
+
           schedule = load_schedule(config_path)
           build_entries(schedule, monotonic_now)
         end

data/lib/async/background/runner.rb

@@ -29,8 +29,11 @@ module Async
                   :metrics,
                   :queue_store
 
+      # `config_path: nil` explicitly disables recurring jobs. This keeps the
+      # dynamic SQLite queue usable on its own; a supplied path remains strict
+      # so a typo cannot silently disable scheduled work.
       def initialize(
-        config_path:,
+        config_path: nil,
         job_count: 2,
         worker_index:,
         total_workers:,
@@ -53,8 +56,9 @@ module Async
 
         @drain_barrier = ::Async::Barrier.new
         @semaphore = ::Async::Semaphore.new(job_count, parent: @drain_barrier)
-        @heap = build_heap(config_path)
+        @heap = config_path.nil? ? MinHeap.new : build_heap(config_path)
         setup_queue(queue_socket_dir, queue_db_path, queue_mmap)
+        validate_work_source!(config_path)
       end
 
       def run
@@ -82,6 +86,11 @@ module Async
       private
 
       def scheduler_loop(task)
+        # Queue-only workers have no heap entry to sleep on. Keep the runner
+        # alive until #stop / SIGTERM wakes this condition; the queue listener
+        # continues independently in its own Async task.
+        return shutdown.wait if heap.empty? && @listen_queue
+
         loop do
           entry = heap.peek
           break unless entry
@@ -93,6 +102,12 @@ module Async
         end
       end
 
+      def validate_work_source!(config_path)
+        return unless config_path.nil? && !@listen_queue
+
+        raise ConfigError, 'Runner requires config_path or queue_socket_dir'
+      end
+
       def wait_for_next_entry(task, entry)
         wait = [entry.next_run_at - monotonic_now, MIN_SLEEP_TIME].max
         wait_with_shutdown(task, wait)

data/lib/async/background/version.rb

@@ -2,6 +2,6 @@
 
 module Async
   module Background
-    VERSION = '0.7.2'
+    VERSION = '1.0.0'
   end
 end

data/lib/async/background/web/app.rb

@@ -0,0 +1,138 @@
+# frozen_string_literal: true
+
+module Async
+  module Background
+    module Web
+      class App
+        def initialize(config)
+          @config = config.validate!
+          @auth = Auth.new(@config.auth)
+          @snapshot = Snapshot.new(path: @config.queue_path, counts_cache_ttl: @config.counts_cache_ttl).open!
+          @metrics_reader = build_metrics_reader
+          @serializer = Serializer.new(@config)
+          @event_hub = build_event_hub
+          @router = Router.new
+        end
+
+        def call(env)
+          return Response.unauthorized unless @auth.authorized?(env)
+
+          route = @router.match(env)
+          return Response.not_found unless route
+
+          dispatch(route, env)
+        rescue RequestError => error
+          Response.bad_request(error.message)
+        rescue UnavailableError, ClosedError
+          Response.unavailable
+        rescue StandardError
+          # Do not turn internal class names, paths or database errors into an
+          # unauthenticated information disclosure channel.
+          Response.internal_error
+        end
+
+        def close
+          @event_hub&.close
+          @snapshot.close
+          self
+        end
+
+        private
+
+        def build_metrics_reader
+          return unless @config.metrics_enabled?
+
+          MetricsReader.new(path: @config.metrics_path, total_workers: @config.total_workers)
+        end
+
+        def build_event_hub
+          return unless @config.transport == :sse
+
+          EventHub.new(
+            @snapshot,
+            @serializer,
+            metrics_reader: @metrics_reader,
+            poll_seconds: @config.stream_poll_seconds
+          )
+        end
+
+        def dispatch(route, env)
+          case route
+          when :index then Response.html(Assets.render_index(@config))
+          when :javascript then Response.javascript(Assets::JS)
+          when :stylesheet then Response.stylesheet(Assets::CSS)
+          when :overview then overview_response
+          when :executing then in_flight_response(:executing, env)
+          when :claimed then in_flight_response(:claimed, env)
+          when :done then terminal_response(:done, env)
+          when :failed then terminal_response(:failed, env)
+          when :pending then pending_response(env)
+          when :metrics then metrics_response
+          when :config then config_response
+          when :stream then stream_response
+          else Response.not_found
+          end
+        end
+
+        def overview_response
+          Response.json(@serializer.overview(@snapshot.overview, metrics_payload))
+        end
+
+        def in_flight_response(kind, env)
+          request = Request.new(env, @config)
+          rows = kind == :executing ? @snapshot.executing(limit: request.limit) : @snapshot.claimed(limit: request.limit)
+          payload = kind == :executing ? @serializer.executing(rows) : @serializer.claimed(rows)
+          Response.json({items: payload})
+        end
+
+        def terminal_response(kind, env)
+          request = Request.new(env, @config)
+          cursor = request.finished_cursor
+          rows = kind == :done ? @snapshot.recent_done(limit: request.limit, cursor: cursor) :
+            @snapshot.recent_failed(limit: request.limit, cursor: cursor)
+          payload = kind == :done ? @serializer.done(rows) : @serializer.failed(rows)
+          Response.json(payload)
+        end
+
+        def pending_response(env)
+          request = Request.new(env, @config)
+          rows = @snapshot.pending(limit: request.limit, cursor: request.pending_cursor)
+          Response.json(@serializer.pending(rows))
+        end
+
+        def metrics_response
+          Response.json(metrics_payload || {available: false, workers: [], totals: MetricsReader::EMPTY_TOTALS})
+        end
+
+        def metrics_payload
+          @metrics_reader&.aggregated
+        end
+
+        def config_response
+          Response.json(
+            {
+              title: @config.title,
+              poll_interval_ms: @config.poll_interval_ms,
+              transport: @config.transport.to_s,
+              expose_args: @config.expose_args,
+              list_limit: @config.list_limit,
+              mount_path: @config.mount_path
+            }
+          )
+        end
+
+        def stream_response
+          return Response.not_found unless @config.transport == :sse
+
+          Response.sse(
+            Stream.new(
+              @event_hub,
+              heartbeat_seconds: @config.stream_heartbeat_seconds,
+              retry_ms: @config.stream_retry_ms
+            )
+          )
+        end
+      end
+    end
+  end
+end