workhorse 1.5.2 → 2.0.0.rc1

data/README.md

@@ -35,8 +35,8 @@ What it does not do:
 
 * Ruby `>= 3.0` (may work with earlier versions but is untested)
 * Rails `>= 7.0`
-* A database and table handler that properly supports row-level locking (such as
-  MySQL with InnoDB, PostgreSQL, or Oracle).
+* One of the supported databases (see [Database support](#database-support)):
+  MySQL / MariaDB with InnoDB, or Oracle. **PostgreSQL is not supported.**
 * If you are planning on using the daemons handler:
   * An operating system and file system that supports file locking.
   * MRI Ruby (aka "CRuby") as jRuby does not support `fork`. See the
@@ -60,22 +60,96 @@ What it does not do:
 
    This generates:
 
-   * A database migration for creating a table named `jobs`
+   * Two database migrations, creating the tables `jobs` and
+     `workhorse_schedules`
    * The initializer `config/initializers/workhorse.rb` for global configuration
      * This can be skipped using the `--skip-initializer` flag
    * The daemon worker script `bin/workhorse.rb`
 
    Please customize the initializer and worker script to your liking.
 
-### Oracle
+### Database support
 
-When using Oracle databases, make sure your schema has access to the package
-`DBMS_LOCK`:
+Workhorse serialises job pickup using a database-level lock, which is
+necessarily written against a specific database's dialect. Two families are
+implemented:
+
+| Database          | Supported | Lock used            | Covered by CI |
+|-------------------|-----------|----------------------|---------------|
+| MySQL / MariaDB   | Yes       | `GET_LOCK`           | Yes, against both the `mysql2` and the `trilogy` adapter |
+| Oracle 12c+       | Yes       | `DBMS_LOCK`          | Yes, against `activerecord-oracle_enhanced-adapter` |
+| PostgreSQL        | **No**    | —                    | — |
+| Everything else   | **No**    | —                    | — |
+
+There is no PostgreSQL implementation: workers emit `GET_LOCK` on every poll,
+which PostgreSQL does not provide, so a worker fails on its first poll.
+Supporting it would mean an advisory-lock dialect of its own
+(`pg_advisory_lock`) and is not currently planned. Note that InnoDB is required
+on MySQL / MariaDB, as MyISAM supports neither transactions nor row-level
+locking.
+
+Oracle 12c is the minimum, as job selection limits its rows with
+`FETCH FIRST … ROWS ONLY`. When using Oracle, make sure your schema has access
+to the package `DBMS_LOCK`:
 
 ```
 GRANT execute ON DBMS_LOCK TO <schema-name>;
 ```
 
+## Upgrading from 1.x
+
+Nothing breaks without migrating: [scheduling](#scheduling) is inert
+until a schedule is declared, [notifications](#notifications) are off until a
+notifier is selected, and the new job columns are only written when used. To
+take the new features up, add this migration:
+
+```ruby
+class UpgradeWorkhorseToV2 < ActiveRecord::Migration[7.1]
+  def change
+    add_column :jobs, :expires_at, :datetime, null: true
+    add_column :jobs, :max_lateness, :integer, null: true
+
+    create_table :workhorse_schedules do |t|
+      t.string :key, null: false
+      t.string :cron, null: false
+      t.string :timezone, null: true
+      t.boolean :enabled, null: false, default: true
+      t.datetime :next_at, null: false
+      t.datetime :last_enqueued_at, null: true
+      t.datetime :last_occurrence, null: true
+      t.integer :last_job_id, null: true
+      t.timestamps null: false
+    end
+
+    # The index names are given explicitly because the ones Rails would
+    # derive are longer than some tools accept.
+    add_index :workhorse_schedules, :key,
+              unique: true, length: 191, name: 'idx_wh_schedules_key'
+    add_index :workhorse_schedules, %i[enabled next_at],
+              name: 'idx_wh_schedules_due'
+
+    add_index :jobs, %i[state perform_at],
+              length: { state: 191 }, name: 'idx_jobs_state_perform_at'
+    add_index :jobs, %i[state priority created_at],
+              length: { state: 191 }, name: 'idx_jobs_state_prio_created'
+    add_index :jobs, %i[state expires_at],
+              length: { state: 191 }, name: 'idx_jobs_state_expires_at'
+
+    # Now redundant, as `state` leads both indexes above
+    remove_index :jobs, :state
+  end
+end
+```
+
+On Oracle, drop every `length:` option above — it indexes the whole column and
+rejects a prefix length. The generated migrations do this for you; this one is
+written out by hand.
+
+Two behaviour changes worth knowing about, both described in the changelog: a
+forked daemon worker no longer runs `at_exit` handlers registered by the
+process that started it, and `Workhorse::Jobs::CleanupSucceededJobs` now also
+deletes jobs in the new `expired` state.
+
 ## Queuing jobs
 
 ### Basic jobs
@@ -122,86 +196,155 @@ If you do not want to pass any parameters to the operation, just omit the third
 Workhorse.enqueue_op Operations::Jobs::CleanUpDatabase, queue: :maintenance, priority: 2
 ```
 
-### Scheduling
+## Scheduling
 
-Workhorse has no out-of-the-box functionality to support scheduling of regular
-jobs, such as maintenance or backup jobs. There are two primary ways of
-achieving regular execution:
+Workhorse runs jobs on a schedule itself, without an external scheduler
+process. Schedules are declared in code and their state is kept in the
+database:
 
-1. Rescheduling by the same job after successful execution and setting
-   `perform_at`
+```ruby
+# config/initializers/workhorse.rb
+Workhorse.schedules do
+  schedule 'cleanup_jobs',
+           job:  'Workhorse::Jobs::CleanupSucceededJobs',
+           cron: '10 0 * * *'
+
+  schedule 'morning_digest',
+           job:          'Jobs::MorningDigest',
+           cron:         '0 8 * * 1-5',
+           timezone:     'Europe/Zurich',
+           queue:        :reports,
+           priority:     -10,
+           catch_up:     :skip,
+           grace:        15.minutes,
+           max_lateness: 60.seconds
+end
+```
 
-   This is simple to set up and requires no additional dependencies. However,
-   the time taken to execute a job and the time delay caused by the polling
-   interval cannot easily be factored into the calculation of the interval,
-   leading to a slight shift in effective execution date. (This can be mitigated
-   by scheduling the job before knowing whether the current run will succeed.
-   Proceed down this path at your own peril!)
+Each schedule owns a row in `workhorse_schedules` holding the next occurrence
+that has not been materialised yet. Workers reconcile those rows against the
+declarations above on startup, and materialise the occurrences that have come
+due during their regular poll. There is no scheduler process to keep alive and
+no single point of failure: any worker will do.
 
-   *Example:* A job that takes 5 seconds to run and reschedules itself every
-   10 minutes. If started at 12:00 sharp, after one hour it will execute at
-   13:00:30 at the earliest due to cumulative execution time.
+### Why the occurrence is a row
 
-   In its most basic form, the `perform` method of a job would look as follows:
+An in-memory scheduler computes the next occurrence from *now*, so an
+occurrence whose time passes while it is not running never happens and leaves
+no trace — a deployment, a restart or a crash at the wrong minute silently
+skips a nightly job. Because the next occurrence is persisted here, a worker
+coming back at any later point still sees that it is due, and the schedule
+decides what to do about it.
 
-   ```ruby
-   class MyJob
-     def perform
-       # Do all the work
+### Catch-up
 
-       # Perform again after 10 minutes (600 seconds)
-       Workhorse.enqueue MyJob.new, perform_at: Time.now + 600
-     end
-   end
-   ```
+What should happen to an occurrence whose time has passed depends on the job,
+so it is stated per schedule:
 
-2. Using an external scheduler
+| `catch_up`  | Behaviour                                                | Suits                                   |
+|-------------|----------------------------------------------------------|-----------------------------------------|
+| `:run_once` | Collapse all missed occurrences into one (**default**)    | Cleanup, maintenance, idempotent work   |
+| `:run`      | Materialise each, up to `max_catch_up` (default 10)       | Per-period reports that must all exist  |
+| `:skip`     | Drop those older than `grace`                             | "Send the 08:00 digest"                 |
 
-   A more elaborate setup requires an external scheduler, but which can still be
-   called from Ruby. One such scheduler is
-   [rufus-scheduler](https://github.com/jmettraux/rufus-scheduler). A small
-   example of an adapted `bin/workhorse.rb` to accommodate for the additional
-   cog in the mechanism is given below:
+`:run_once` is the default deliberately: after a long outage it is the safe
+behaviour. A schedule running every minute that was down for a day would
+otherwise enqueue 1440 jobs at once, which `max_catch_up` also guards against.
 
-   ```ruby
-   #!/usr/bin/env ruby
+`:skip` requires `grace`, as without one it has no way to tell an occurrence
+that is a moment late from one that is a day late.
 
-   require './config/environment'
+### Lateness and deadlines
 
-   Workhorse::Daemon::ShellHandler.run do |daemon|
-     # Start scheduler process
-     daemon.worker 'Scheduler' do
-       scheduler = Rufus::Scheduler.new
+A materialised job's `perform_at` is the occurrence's own time, not the moment
+it was enqueued. The difference between it and `started_at` is therefore the
+lateness of that occurrence, available on every job as
+`Workhorse::DbJob#lateness`, and derivable in SQL from the `started_at` and
+`perform_at` columns.
 
-       scheduler.cron '0/10 * * * *' do
-         Workhorse.enqueue Workhorse::Jobs::CleanupSucceededJobs.new
-       end
+Two options act on it, both settable on a schedule:
 
-       Signal.trap 'TERM' do
-         scheduler.shutdown
-       end
+* **`max_lateness`** — seconds the job may start late before
+  `Workhorse.on_job_late` is called. The job still runs; it was just late.
+* **`expires_after`** — seconds after the occurrence at which the job is no
+  longer worth running. It is then set to state `expired` and
+  `Workhorse.on_job_expired` is called instead of it being performed. For
+  "send the 08:00 reminder", running it at 11:40 is often worse than not
+  running it at all.
 
-       scheduler.join
-     end
+Hand-enqueued jobs take `max_lateness:` under the same name, and the deadline
+as an absolute time rather than an offset:
 
-     # Start 5 worker processes with 3 threads each
-     5.times do
-       daemon.worker do
-         Workhorse::Worker.start_and_wait(pool_size: 3, polling_interval: 10, logger: Rails.logger)
-       end
-     end
-   end
-   ```
+```ruby
+Workhorse.enqueue(job, expires_at: 1.hour.from_now, max_lateness: 60)
+```
+
+```ruby
+Workhorse.setup do |config|
+  config.on_job_expired = proc do |db_job|
+    ExceptionNotifier.notify_exception(
+      StandardError.new("Job #{db_job.id} (#{db_job.description}) expired")
+    )
+  end
+
+  config.on_job_late = proc do |db_job, lateness|
+    ExceptionNotifier.notify_exception(
+      StandardError.new("Job #{db_job.id} started #{lateness.round}s late")
+    )
+  end
+end
+```
+
+Both callbacks are best-effort: anything they raise goes to
+`Workhorse.on_exception` and never affects the worker or the job. An expiry is
+logged at `warn` whether or not a callback is configured.
+
+Expired jobs stay in the table like any other finished job.
+`Workhorse::Jobs::CleanupSucceededJobs` removes them along with succeeded ones;
+pass `states: [Workhorse::DbJob::STATE_SUCCEEDED]` to keep them.
+
+### Detecting schedules that stopped
+
+Neither callback can fire for a job that was never created, so if no worker is
+polling or the global lock is stuck, occurrences simply stop being
+materialised and nothing says so. `Workhorse::Jobs::DetectLateSchedulesJob`
+covers that case by reporting schedules whose next occurrence lies well in the
+past:
 
-   This allows starting and stopping the daemon with the usual interface.
-   Note that the scheduler is handled like a Workhorse worker, the consequence
-   of which is that only one 'worker' should be started by the ShellHandler.
-   Otherwise there would be multiple jobs scheduled at the same time.
+```ruby
+Workhorse.schedules do
+  schedule 'detect_late_schedules',
+           job:  'Workhorse::Jobs::DetectLateSchedulesJob',
+           cron: '*/30 * * * *'
+end
+```
+
+It is itself performed by a worker, so it reports a stall only while at least
+one worker is still running — use it alongside external monitoring rather than
+instead of it.
+
+### Timezones
+
+Without `timezone`, a cron expression is read in the process's local time.
+Given one, occurrences are computed in that zone, including across daylight
+saving changes. A `30 2 * * *` schedule in `Europe/Zurich` has no occurrence
+on the day the clocks go forward, because 02:30 does not exist that day — and
+exactly one on the day they go back, although 02:30 happens twice.
+
+### Disabling a schedule
+
+Setting `enabled` to `false` on the row stops its occurrences from being
+materialised, without a deployment:
+
+```ruby
+Workhorse::Schedule.find_by(key: 'morning_digest').update!(enabled: false)
+```
 
-   Please refer to the documentation for
-   [rufus-scheduler](https://github.com/jmettraux/rufus-scheduler) (or the
-   scheduler of your choice) for further options concerning the timing of the
-   jobs.
+Removing a schedule from the declarations deletes its row: the next worker to
+start after a day has passed without any worker declaring it removes it. The delay matters during a rolling deployment, where
+the old and the new version run at once: were rows removed immediately, each
+version would delete the other's schedules and reset the occurrences they were
+waiting for.
 
 ## Configuring and starting workers
 
@@ -278,6 +421,100 @@ polling interval.
 This setting is recommended for all setups and may eventually be enabled by
 default.
 
+### Notifications
+
+Instant repolling only helps *after* a job has been performed. A worker that is
+idle and waiting for new work still sleeps out its entire polling interval, so
+a job enqueued just after a poll waits almost a full interval before it starts.
+
+Shortening the polling interval is the obvious remedy and a poor one: every
+poll acquires a global database lock, so more frequent polling across several
+workers increases contention, and a worker that fails to acquire the lock skips
+its poll and waits another whole interval. Polling costs the same whether or
+not anything is happening.
+
+*Notifications* turn the question around. Enqueuing a job announces it, and a
+waiting worker polls straight away instead of sleeping out its interval:
+
+```ruby
+# config/initializers/workhorse.rb
+Workhorse.setup do |config|
+  config.notifier = :file
+end
+```
+
+Polling remains the floor. A notification that is never delivered — a worker
+that was restarting, an enqueue from a host that cannot reach the others —
+costs latency and nothing else, as the regular poll still finds the job. For
+the same reason, raise `polling_interval` only as far as you are willing to
+wait when a notification *is* missed.
+
+Note what this does and does not do to database load. A notification only
+brings the next poll forward; it never replaces one, so the saving comes from
+raising `polling_interval`, not from notifications by themselves. An
+announcement also wakes *every* worker with a free thread, and all but the one
+that wins the job take the global lock for nothing. So a workload that enqueues
+in bursts while many workers sit idle can take the lock more often than plain
+polling would, rather than less. A worker whose threads are all busy does not
+react, so the effect is bounded by how much spare capacity there is.
+
+Two notifiers ship with workhorse:
+
+#### `:file`
+
+Touches a single file, which waiting workers stat once per 0.1 seconds on a
+tick the poller performs anyway. It costs no database work at all and around a
+microsecond of CPU per check, and a job starts within roughly 100 milliseconds.
+
+It requires that the processes enqueueing jobs and the workers share a
+filesystem, which in practice means the same host. Where they do not, the
+touch never reaches those workers and they fall back to polling.
+
+The file defaults to `tmp/pids/workhorse.wake` below the Rails root and can be
+moved with `config.notification_path`. Every process involved must agree on it.
+
+#### `:redis`
+
+Publishes on a Redis pub/sub channel, for deployments whose workers do not
+share a filesystem with the application. Redis is a soft dependency: it is not
+declared as a dependency of this gem and workhorse never requires it — supply
+a client through `config.notification_redis`.
+
+```ruby
+Workhorse.setup do |config|
+  config.notifier = :redis
+  config.notification_redis = -> { Redis.new(url: ENV['REDIS_URL']) }
+  config.notification_channel = 'workhorse:jobs' # optional
+end
+```
+
+`notification_redis` is given something callable above because a subscribed
+Redis connection cannot be used for anything else: `subscribe` occupies it
+until it returns. The notifier calls it once for publishing and once for the
+subscriber, so the two get separate connections.
+
+Passing a client rather than a callable also works, but the subscriber then
+falls back to `dup` — in redis-rb a shallow copy that may share the
+connection, in which case publishing can block behind the subscription.
+
+The channel can be changed at any point before a worker starts; the
+subscriber thread captures the one it was started with.
+
+#### Writing your own
+
+Subclass `Workhorse::Notifiers::Base` and assign an instance to
+`config.notifier`. A notifier announces jobs with `notify` and exposes a
+`token` that changes whenever a notification has arrived; workers compare it
+against the last value they saw, so several workers in one process stay
+independent of one another. `notify(queue: nil)` must never raise — a job must
+still be enqueued when it cannot be announced. `start` and `stop` are optional
+hooks, called as a worker starts and shuts down, for anything that needs a
+thread or a connection of its own.
+
+Note that jobs with a future `perform_at` are not announced, as a woken worker
+would find nothing to do; they are picked up by the regular poll once they are
+due.
+
 ## Transactions
 
 By default, each job is run in an individual database transaction. An exception
@@ -412,6 +649,7 @@ DbJob.locked
 DbJob.started
 DbJob.succeeded
 DbJob.failed
+DbJob.expired
 ```
 ### Resetting jobs
 
@@ -419,8 +657,8 @@ Jobs in a state other than `waiting` are either being processed or else already
 in a final state such as `succeeded` and won't be performed again. Workhorse
 provides an API method for resetting jobs in the following cases:
 
-* A job has succeeded or failed (states `succeeded` and `failed`) and needs to
-  re-run. In these cases, perform a non-forced reset:
+* A job has succeeded, failed or expired (states `succeeded`, `failed` and
+  `expired`) and needs to re-run. In these cases, perform a non-forced reset:
 
   ```ruby
   db_job.reset!
@@ -461,8 +699,9 @@ configuration or else using `self.queue_adapter` in a job class inheriting from
 Per default, jobs remain in the database, no matter in which state. This can
 eventually lead to a very large jobs database. You are advised to clean your
 jobs database on a regular interval. Workhorse provides the job
-`Workhorse::Jobs::CleanupSucceededJobs` for this purpose that cleans up all
-succeeded jobs. You can run this using your scheduler in a specific interval.
+`Workhorse::Jobs::CleanupSucceededJobs` for this purpose, which cleans up
+succeeded and expired jobs — pass `states:` to narrow that. Schedule it with
+`Workhorse.schedules`, see [Scheduling](#scheduling).
 
 ## Memory handling
 
@@ -587,6 +826,11 @@ In the event that this still happens, Workhorse takes the following steps:
 - Retries acquiring the lock on the next poll.
 - Calls the `on_exception` callback (if configured) after a configurable number of consecutive failures to obtain the lock.
 
+Failures on a poll that a notification or an instant repoll brought forward
+are expected — several workers race for the lock and all but one lose — so
+they are logged at `debug` only and do not count towards
+`max_global_lock_fails`.
+
 The maximum number of consecutive failures can be configured using
 `config.max_global_lock_fails`, which defaults to 10.
 

data/Rakefile

@@ -16,6 +16,7 @@ task :gemspec do
     spec.add_dependency 'activesupport', '>= 7.0.0'
     spec.add_dependency 'activerecord', '>= 7.0.0'
     spec.add_dependency 'concurrent-ruby'
+    spec.add_dependency 'fugit'
   end
 
   File.write('workhorse.gemspec', gemspec.to_ruby.strip)

data/VERSION

@@ -1 +1 @@
-1.5.2
+2.0.0.rc1

data/bin/rubocop

@@ -1 +1,5 @@
-bundle exec rubocop "$@"
+#!/bin/bash
+
+BASE_PATH=`(cd $(dirname $0)/.. && pwd -P)`
+
+BUNDLE_GEMFILE=$BASE_PATH/Gemfile $BASE_PATH/bin/ruby -e "require 'rubygems';require 'bundler/setup'; load Gem.activate_bin_path('rubocop', 'rubocop', '>= 0.a')" -- $@

data/lib/generators/workhorse/install_generator.rb

@@ -7,12 +7,21 @@ module Workhorse
 
     source_root File.expand_path('templates', __dir__)
 
+    # Returns the version for the next generated migration. Counts up rather
+    # than returning the current time, as several migrations are generated
+    # within the same second and would otherwise collide.
     def self.next_migration_number(_dir)
-      Time.now.utc.strftime('%Y%m%d%H%M%S')
+      @next_migration_number = [
+        Time.now.utc.strftime('%Y%m%d%H%M%S').to_i,
+        (@next_migration_number || 0) + 1
+      ].max
+
+      return @next_migration_number.to_s
     end
 
     def install_migration
       migration_template 'create_table_jobs.rb', 'db/migrate/create_table_jobs.rb'
+      migration_template 'create_table_workhorse_schedules.rb', 'db/migrate/create_table_workhorse_schedules.rb'
     end
 
     def install_daemon_script

data/lib/generators/workhorse/templates/config/initializers/workhorse.rb

@@ -23,4 +23,59 @@ Workhorse.setup do |config|
   #   # Do something with exception, i.e.
   #   # ExceptionNotifier.notify_exception(exception)
   # end
+
+  # Seconds the daemon's `stop` waits for a worker to finish what it is doing
+  # before killing it. Set to nil to wait indefinitely.
+  #
+  # config.shutdown_timeout = 300
+
+  # Enable this to let an enqueued job start without waiting for the next
+  # poll. Use :file where the workers share a filesystem with the application
+  # and :redis where they do not. Polling stays the floor either way, so raise
+  # the polling interval only as far as you are willing to wait when a
+  # notification is missed.
+  #
+  # config.notifier = :file
+  # config.notification_path = Rails.root.join('tmp', 'pids', 'workhorse.wake')
+  #
+  # config.notifier = :redis
+  # config.notification_redis = -> { Redis.new(url: ENV['REDIS_URL']) }
+  # config.notification_channel = 'workhorse:jobs'
+
+  # Enable and configure these to be told about a job that passed its
+  # `expires_at` before any worker got to it, and about one that started later
+  # than its `max_lateness` allows. Neither can affect the worker or the job.
+  #
+  # config.on_job_expired = proc do |db_job|
+  #   # Do something with the job, i.e.
+  #   # ExceptionNotifier.notify_exception(
+  #   #   StandardError.new("Job #{db_job.id} (#{db_job.description}) expired")
+  #   # )
+  # end
+  #
+  # config.on_job_late = proc do |db_job, lateness|
+  #   # Do something with the job, i.e.
+  #   # ExceptionNotifier.notify_exception(
+  #   #   StandardError.new("Job #{db_job.id} started #{lateness.round}s late")
+  #   # )
+  # end
 end
+
+# Jobs that run on a schedule. Each of these owns a row in the
+# `workhorse_schedules` table holding the next occurrence that has not been
+# materialized yet, so an occurrence whose time passes while nothing is
+# running is not lost. See the README for the catch-up policies.
+#
+# Workhorse.schedules do
+#   schedule 'cleanup_jobs',
+#            job:  'Workhorse::Jobs::CleanupSucceededJobs',
+#            cron: '10 0 * * *'
+#
+#   schedule 'detect_stale_jobs',
+#            job:  'Workhorse::Jobs::DetectStaleJobsJob',
+#            cron: '30 * * * *'
+#
+#   schedule 'detect_late_schedules',
+#            job:  'Workhorse::Jobs::DetectLateSchedulesJob',
+#            cron: '*/30 * * * *'
+# end

data/lib/generators/workhorse/templates/create_table_jobs.rb

@@ -17,17 +17,30 @@ class CreateTableJobs < ActiveRecord::Migration[7.1]
       t.integer :priority, null: false
       t.datetime :perform_at, null: true
 
+      # Deadline; the job is then set to state 'expired' rather than performed.
+      t.datetime :expires_at, null: true
+
+      # Seconds the job may start later than its perform_at before
+      # Workhorse.on_job_late is called.
+      t.integer :max_lateness, null: true
+
       t.string :description, null: true
 
       t.timestamps null: false
     end
 
+    # The index names are given explicitly because the ones Rails would derive
+    # exceed the 30 characters Oracle allows before 12.2.
     if oracle?
       add_index :jobs, :queue
-      add_index :jobs, :state
+      add_index :jobs, %i[state perform_at], name: 'idx_jobs_state_perform_at'
+      add_index :jobs, %i[state priority created_at], name: 'idx_jobs_state_prio_created'
+      add_index :jobs, %i[state expires_at], name: 'idx_jobs_state_expires_at'
     else
       add_index :jobs, :queue, length: 191
-      add_index :jobs, :state, length: 191
+      add_index :jobs, %i[state perform_at], length: { state: 191 }, name: 'idx_jobs_state_perform_at'
+      add_index :jobs, %i[state priority created_at], length: { state: 191 }, name: 'idx_jobs_state_prio_created'
+      add_index :jobs, %i[state expires_at], length: { state: 191 }, name: 'idx_jobs_state_expires_at'
     end
     add_index :jobs, :perform_at
   end

data/lib/generators/workhorse/templates/create_table_workhorse_schedules.rb

@@ -0,0 +1,42 @@
+class CreateTableWorkhorseSchedules < ActiveRecord::Migration[7.1]
+  def change
+    # Schedules are addressed by key; the job class and its options live in
+    # the Workhorse.schedules definition rather than here.
+    create_table :workhorse_schedules, force: true do |t|
+      t.string :key, null: false
+
+      # The cron expression and timezone are kept here so that a change to
+      # either is recognised on reconciliation.
+      t.string :cron, null: false
+      t.string :timezone, null: true
+
+      # Lets a schedule be switched off without a deployment.
+      t.boolean :enabled, null: false, default: true
+
+      # The next occurrence that has not been materialised yet.
+      t.datetime :next_at, null: false
+
+      t.datetime :last_enqueued_at, null: true
+      t.datetime :last_occurrence, null: true
+      t.integer :last_job_id, null: true
+
+      t.timestamps null: false
+    end
+
+    # The index names are given explicitly because the ones Rails would derive
+    # exceed the 30 characters Oracle allows before 12.2.
+    if oracle?
+      add_index :workhorse_schedules, :key, unique: true, name: 'idx_wh_schedules_key'
+    else
+      add_index :workhorse_schedules, :key, unique: true, length: 191, name: 'idx_wh_schedules_key'
+    end
+
+    add_index :workhorse_schedules, %i[enabled next_at], name: 'idx_wh_schedules_due'
+  end
+
+  private
+
+  def oracle?
+    ActiveRecord::Base.connection.adapter_name == 'OracleEnhanced'
+  end
+end

data/lib/workhorse/daemon/shell_handler.rb

@@ -146,7 +146,10 @@ module Workhorse
 
     def self.acquire_lock(lockfile_path, flags)
       if Workhorse.lock_shell_commands
-        lockfile = File.open(lockfile_path, 'a') # rubocop:disable Style/FileOpen
+        # Not the block form: the lockfile is returned to the caller, which
+        # holds the flock for as long as the command runs.
+        # rubocop:disable-next Style/FileOpen
+        lockfile = File.open(lockfile_path, 'a')
         result = lockfile.flock(flags)
 
         if result == false