solid_queue 0.7.0 → 0.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 23836c755cda6dd9bec594148ad92ea19740da4125e9115d17c44bd2543d752c
-  data.tar.gz: 40d91182c13f155a86879f04abcf4232a50f3502f4038b5e9e07a3c15da8def2
+  metadata.gz: '053856cc5d0d234f55fa42184371360bca7278a37130bc4230656c48ac0f8e74'
+  data.tar.gz: f31c7c8e48f67874365e788cb301c41e5a3be8dc77f28bf5e848f01645076928
 SHA512:
-  metadata.gz: f8ac683f1a0e635762d0d93b9b74f712c5fc5465381ea51230448dc2f176de67aafb0ee7031ae9e9cfee73edcee7bd70352e78aefd49ae25ea71a7485af95aff
-  data.tar.gz: b02eac304a30fb8c7e9e2627446a31bb56d22740e863b1d2749838429416f886da78d19d2000b533e3672576387ee075b8efb3c330f6177b5cd76a27955cd433
+  metadata.gz: cdd486cd413c30d10deea24ba32f8f9e06000263a619f0fb175f08695a53f83a55eae56a19aa626e97af1e37a882c16be40441a158569ca06cb3140681414c8e
+  data.tar.gz: 1545db515ce86ad1f5e327533be2cfeba19a2b5bb162139dfa40584482a9d1b576775c97d79d4aa6baa5aff36fb3a7579088e9e9ef4852e03f79544fc5a0091e

data/README.md

@@ -6,43 +6,52 @@ Besides regular job enqueuing and processing, Solid Queue supports delayed jobs,
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
-## Installation and usage
-Add this line to your application's Gemfile:
+## Installation
 
-```ruby
-gem "solid_queue"
-```
+Solid Queue is configured by default in new Rails 8 applications. But if you're running an earlier version, you can add it manually following these steps:
 
-And then execute:
-```bash
-$ bundle
-```
+1. `bundle add solid_queue`
+2. `bin/rails solid_queue:install`
 
-Or install it yourself as:
-```bash
-$ gem install solid_queue
-```
+This will configure Solid Queue as the production Active Job backend, create `config/solid_queue.yml`, and create the `db/queue_schema.rb`.
 
-Now, you need to install the necessary migrations and configure the Active Job's adapter. You can do both at once using the provided generator:
+You will then have to add the configuration for the queue database in `config/database.yml`. If you're using sqlite, it'll look like this:
 
-```bash
-$ bin/rails generate solid_queue:install
+```yaml
+production:
+  primary:
+    <<: *default
+    database: storage/production.sqlite3
+  queue:
+    <<: *default
+    database: storage/production_queue.sqlite3
+    migrations_paths: db/queue_migrate
 ```
 
-This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
+...or if you're using MySQL/PostgreSQL/Trilogy:
 
-Alternatively, you can skip setting the Active Job's adapter with:
-```bash
-$ bin/rails generate solid_queue:install --skip_adapter
+```yaml
+production:
+  primary: &primary_production
+    <<: *default
+    database: app_production
+    username: app
+    password: <%= ENV["APP_DATABASE_PASSWORD"] %>
+  queue:
+    <<: *primary_production
+    database: app_production_queue
+    migrations_paths: db/queue_migrate
 ```
 
-And set Solid Queue as your Active Job's queue backend manually, in your environment config:
-```ruby
-# config/environments/production.rb
-config.active_job.queue_adapter = :solid_queue
-```
+Then run `db:prepare` in production to ensure the database is created and the schema is loaded.
+
+Now you're ready to start processing jobs by running `bin/jobs` on the server that's doing the work. This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
+
+For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
-Or you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+## Incremental adoption
+
+If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the `config.active_job.queue_adapter` set to your old backend, and then set the `queue_adapter` directly in the jobs you're moving:
 
 ```ruby
 # app/jobs/my_job.rb
@@ -52,24 +61,9 @@ class MyJob < ApplicationJob
   # ...
 end
 ```
+## High performance requirements
 
-Finally, you need to run the migrations:
-
-```bash
-$ bin/rails db:migrate
-```
-
-After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them. You can use the provided binstub:`
-```bash
-$ bin/jobs
-```
-
-This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
-
-For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
-
-## Requirements
-Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
+Solid Queue was designed for the highest throughput when used with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue. You can also use it with SQLite on smaller applications.
 
 ## Configuration
 
@@ -135,8 +129,8 @@ Here's an overview of the different options:
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 - `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
 
-
 ### Queue order and priorities
+
 As mentioned above, if you specify a list of queues for a worker, these will be polled in the order given, such as for the list `real_time,background`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`.
 
 Active Job also supports positive integer priorities when enqueuing jobs. In Solid Queue, the smaller the value, the higher the priority. The default is `0`.
@@ -159,66 +153,53 @@ When receiving a `QUIT` signal, if workers still have jobs in-flight, these will
 If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats, which will release any claimed jobs back to their queues. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
 
 
-### Dedicated database configuration
+### Database configuration
 
-Solid Queue can be configured to run on a different database than the main application.
+You can configure the database used by Solid Queue via the `config.solid_queue.connects_to` option in the `config/application.rb` or `config/environments/production.rb` config files. By default, a single database is used for both writing and reading called `queue` to match the database configuration you set up during the install.
 
-Configure the `connects_to` option in `config/application.rb` or your environment config, with the custom database configuration that will be used in the abstract `SolidQueue::Record` Active Record model.
+All the options available to Active Record for multiple databases can be used here.
 
-```ruby
-  # Use a separate DB for Solid Queue
-  config.solid_queue.connects_to = { database: { writing: :solid_queue_primary, reading: :solid_queue_replica } }
-  ```
+## Lifecycle hooks
 
-Add the dedicated database configuration to `config/database.yml`, differentiating between the main app's database and the dedicated `solid_queue` database. Make sure to include the `migrations_paths` for the solid queue database. This is where migration files for Solid Queue tables will reside.
+In Solid queue, you can hook into two different points in the supervisor's life:
+- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
+- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
 
-```yml
-default: &default
-  adapter: sqlite3
-  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
-  timeout: 5000
+And into two different points in a worker's life:
+- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
+- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
 
-solid_queue: &solid_queue
-  <<: *default
-  migrations_paths: db/solid_queue_migrate
+You can use the following methods with a block to do this:
+```ruby
+SolidQueue.on_start
+SolidQueue.on_stop
 
-development:
-  primary:
-    <<: *default
-    # ...
-  solid_queue_primary:
-    <<: *solid_queue
-    # ...
-  solid_queue_replica:
-    <<: *solid_queue
-    # ...
+SolidQueue.on_worker_start
+SolidQueue.on_worker_stop
 ```
 
-Install migrations and specify the dedicated database name with the `--database` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
-
-```bash
-$ bin/rails g solid_queue:install --database solid_queue
+For example:
+```ruby
+SolidQueue.on_start { start_metrics_server }
+SolidQueue.on_stop { stop_metrics_server }
 ```
 
-Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`) without a `--database` option, the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
-
-Finally, run the migrations:
-
-```bash
-$ bin/rails db:migrate
-```
+These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
 
 ### Other configuration settings
+
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
 
 There are several settings that control how Solid Queue works that you can set as well:
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
 - `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
-- `on_thread_error`: custom lambda/Proc to call when there's an error within a thread that takes the exception raised as argument. Defaults to
+- `on_thread_error`: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to
 
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
+  **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
+
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
@@ -231,11 +212,13 @@ There are several settings that control how Solid Queue works that you can set a
 - `enqueue_after_transaction_commit`: whether the job queuing is deferred to after the current Active Record transaction is committed. The default is `false`. [Read more](https://github.com/rails/rails/pull/51426).
 
 ## Errors when enqueuing
+
 Solid Queue will raise a `SolidQueue::Job::EnqueueError` for any Active Record errors that happen when enqueuing a job. The reason for not raising `ActiveJob::EnqueueError` is that this one gets handled by Active Job, causing `perform_later` to return `false` and set `job.enqueue_error`, yielding the job to a block that you need to pass to `perform_later`. This works very well for your own jobs, but makes failure very hard to handle for jobs enqueued by Rails or other gems, such as `Turbo::Streams::BroadcastJob` or `ActiveStorage::AnalyzeJob`, because you don't control the call to `perform_later` in that cases.
 
 In the case of recurring tasks, if such error is raised when enqueuing the job corresponding to the task, it'll be handled and logged but it won't bubble up.
 
 ## Concurrency controls
+
 Solid Queue extends Active Job with concurrency controls, that allows you to limit how many jobs of a certain type or with certain arguments can run at the same time. When limited in this way, jobs will be blocked from running, and they'll stay blocked until another job finishes and unblocks them, or after the set expiry time (concurrency limit's _duration_) elapses. Jobs are never discarded or lost, only blocked.
 
 ```ruby
@@ -283,6 +266,8 @@ In this case, if we have a `Box::MovePostingsByContactToDesignatedBoxJob` job en
 
 Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked. In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
 
+Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
+
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
 
 ## Failed jobs and retries
@@ -299,35 +284,15 @@ failed_execution.discard # This will delete the job from the system
 However, we recommend taking a look at [mission_control-jobs](https://github.com/rails/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs.
 
 ## Puma plugin
+
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add
 ```ruby
 plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
-
-## Jobs and transactional integrity
-:warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.
-
-By default, Solid Queue runs in the same DB as your app, and job enqueuing is _not_ deferred until any ongoing transaction is committed, which means that by default, you'll be taking advantage of this transactional integrity.
-
-If you prefer not to rely on this, or avoid relying on it unintentionally, you should make sure that:
-- You set [`config.active_job.enqueue_after_transaction_commit`](https://edgeguides.rubyonrails.org/configuring.html#config-active-job-enqueue-after-transaction-commit) to `always`, if you're using Rails 7.2+.
-- Or, your jobs relying on specific records are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
-- Or, you configure a different database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
-
-  ```ruby
-  class ApplicationRecord < ActiveRecord::Base
-    self.abstract_class = true
-
-    connects_to database: { writing: :primary, reading: :replica }
-  ```
-
-  ```ruby
-  config.solid_queue.connects_to = { database: { writing: :primary, reading: :replica } }
-  ```
-
 ## Recurring tasks
+
 Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by dispatcher processes and as such, they can be defined in the dispatcher's configuration like this:
 ```yml
   dispatchers:

data/UPGRADING.md

@@ -1,6 +1,9 @@
+# Upgrading to version 0.8.x
+*IMPORTANT*: This version collapsed all migrations into a single `db/queue_schema.rb`, that will use a separate `queue` database. If you're upgrading from a version < 0.6.0, you need to upgrade to 0.6.0 first, ensure all migrations are up-to-date, and then upgrade further.
+
 # Upgrading to version 0.7.x
 
-This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor. It includes also a minor migration.
+This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor.
 
 To install both the binstub `bin/jobs` and the migration, you can just run
 ```

data/lib/generators/solid_queue/install/USAGE

@@ -5,6 +5,6 @@ Example:
     bin/rails generate solid_queue:install
 
     This will perform the following:
-        Installs solid_queue migrations
+        Adds solid_queue db schema
         Replaces Active Job's adapter in environment configuration
         Installs bin/jobs binstub to start the supervisor

data/lib/generators/solid_queue/install/install_generator.rb

@@ -3,33 +3,17 @@
 class SolidQueue::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
-  class_option :skip_adapter, type: :boolean, default: nil, desc: "Skip setting Solid Queue as the Active Job's adapter"
-  class_option :database, type: :string, default: nil, desc: "The database to use for migrations, if different from the primary one."
-
-  def add_solid_queue
-    unless options[:skip_adapter]
-      if (env_config = Pathname(destination_root).join("config/environments/production.rb")).exist?
-        say "Setting solid_queue as Active Job's queue adapter"
-        gsub_file env_config, /(# )?config\.active_job\.queue_adapter\s+=.*/, "config.active_job.queue_adapter = :solid_queue"
-      end
-    end
-
-    if File.exist?("config/solid_queue.yml")
-      say "Skipping sample configuration as config/solid_queue.yml exists"
-    else
-      say "Copying sample configuration"
-      copy_file "config.yml", "config/solid_queue.yml"
-    end
-
-    say "Copying binstub"
-    copy_file "jobs", "bin/jobs"
+  def copy_files
+    template "config/solid_queue.yml"
+    template "db/queue_schema.rb"
+    template "bin/jobs"
     chmod "bin/jobs", 0755 & ~File.umask, verbose: false
   end
 
-  def create_migrations
-    say "Installing database migrations"
-    arguments = [ "FROM=solid_queue" ]
-    arguments << "DATABASE=#{options[:database]}" if options[:database].present?
-    rails_command "railties:install:migrations #{arguments.join(" ")}", inline: true
+  def configure_active_job_adapter
+    gsub_file Pathname(destination_root).join("config/environments/production.rb"),
+      /(# )?config\.active_job\.queue_adapter\s+=.*/,
+      "config.active_job.queue_adapter = :solid_queue\n" +
+      "  config.solid_queue.connects_to = { database: { writing: :queue } }\n"
   end
 end

data/lib/generators/solid_queue/install/templates/config/solid_queue.yml

@@ -0,0 +1,18 @@
+default: &default
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  workers:
+    - queues: "*"
+      threads: 3
+      processes: <%%= ENV.fetch("JOB_CONCURRENCY", 1) %>
+      polling_interval: 0.1
+
+development:
+  <<: *default
+
+test:
+  <<: *default
+
+production:
+  <<: *default

data/lib/generators/solid_queue/install/templates/db/queue_schema.rb

@@ -0,0 +1,129 @@
+ActiveRecord::Schema[7.1].define(version: 2024_09_04_193154) do
+  create_table "solid_queue_blocked_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.string "queue_name", null: false
+    t.integer "priority", default: 0, null: false
+    t.string "concurrency_key", null: false
+    t.datetime "expires_at", null: false
+    t.datetime "created_at", null: false
+    t.index [ "concurrency_key", "priority", "job_id" ], name: "index_solid_queue_blocked_executions_for_release"
+    t.index [ "expires_at", "concurrency_key" ], name: "index_solid_queue_blocked_executions_for_maintenance"
+    t.index [ "job_id" ], name: "index_solid_queue_blocked_executions_on_job_id", unique: true
+  end
+
+  create_table "solid_queue_claimed_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.bigint "process_id"
+    t.datetime "created_at", null: false
+    t.index [ "job_id" ], name: "index_solid_queue_claimed_executions_on_job_id", unique: true
+    t.index [ "process_id", "job_id" ], name: "index_solid_queue_claimed_executions_on_process_id_and_job_id"
+  end
+
+  create_table "solid_queue_failed_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.text "error"
+    t.datetime "created_at", null: false
+    t.index [ "job_id" ], name: "index_solid_queue_failed_executions_on_job_id", unique: true
+  end
+
+  create_table "solid_queue_jobs", force: :cascade do |t|
+    t.string "queue_name", null: false
+    t.string "class_name", null: false
+    t.text "arguments"
+    t.integer "priority", default: 0, null: false
+    t.string "active_job_id"
+    t.datetime "scheduled_at"
+    t.datetime "finished_at"
+    t.string "concurrency_key"
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
+    t.index [ "active_job_id" ], name: "index_solid_queue_jobs_on_active_job_id"
+    t.index [ "class_name" ], name: "index_solid_queue_jobs_on_class_name"
+    t.index [ "finished_at" ], name: "index_solid_queue_jobs_on_finished_at"
+    t.index [ "queue_name", "finished_at" ], name: "index_solid_queue_jobs_for_filtering"
+    t.index [ "scheduled_at", "finished_at" ], name: "index_solid_queue_jobs_for_alerting"
+  end
+
+  create_table "solid_queue_pauses", force: :cascade do |t|
+    t.string "queue_name", null: false
+    t.datetime "created_at", null: false
+    t.index [ "queue_name" ], name: "index_solid_queue_pauses_on_queue_name", unique: true
+  end
+
+  create_table "solid_queue_processes", force: :cascade do |t|
+    t.string "kind", null: false
+    t.datetime "last_heartbeat_at", null: false
+    t.bigint "supervisor_id"
+    t.integer "pid", null: false
+    t.string "hostname"
+    t.text "metadata"
+    t.datetime "created_at", null: false
+    t.string "name", null: false
+    t.index [ "last_heartbeat_at" ], name: "index_solid_queue_processes_on_last_heartbeat_at"
+    t.index [ "name", "supervisor_id" ], name: "index_solid_queue_processes_on_name_and_supervisor_id", unique: true
+    t.index [ "supervisor_id" ], name: "index_solid_queue_processes_on_supervisor_id"
+  end
+
+  create_table "solid_queue_ready_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.string "queue_name", null: false
+    t.integer "priority", default: 0, null: false
+    t.datetime "created_at", null: false
+    t.index [ "job_id" ], name: "index_solid_queue_ready_executions_on_job_id", unique: true
+    t.index [ "priority", "job_id" ], name: "index_solid_queue_poll_all"
+    t.index [ "queue_name", "priority", "job_id" ], name: "index_solid_queue_poll_by_queue"
+  end
+
+  create_table "solid_queue_recurring_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.string "task_key", null: false
+    t.datetime "run_at", null: false
+    t.datetime "created_at", null: false
+    t.index [ "job_id" ], name: "index_solid_queue_recurring_executions_on_job_id", unique: true
+    t.index [ "task_key", "run_at" ], name: "index_solid_queue_recurring_executions_on_task_key_and_run_at", unique: true
+  end
+
+  create_table "solid_queue_recurring_tasks", force: :cascade do |t|
+    t.string "key", null: false
+    t.string "schedule", null: false
+    t.string "command", limit: 2048
+    t.string "class_name"
+    t.text "arguments"
+    t.string "queue_name"
+    t.integer "priority", default: 0
+    t.boolean "static", default: true, null: false
+    t.text "description"
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
+    t.index [ "key" ], name: "index_solid_queue_recurring_tasks_on_key", unique: true
+    t.index [ "static" ], name: "index_solid_queue_recurring_tasks_on_static"
+  end
+
+  create_table "solid_queue_scheduled_executions", force: :cascade do |t|
+    t.integer "job_id", null: false
+    t.string "queue_name", null: false
+    t.integer "priority", default: 0, null: false
+    t.datetime "scheduled_at", null: false
+    t.datetime "created_at", null: false
+    t.index [ "job_id" ], name: "index_solid_queue_scheduled_executions_on_job_id", unique: true
+    t.index [ "scheduled_at", "priority", "job_id" ], name: "index_solid_queue_dispatch_all"
+  end
+
+  create_table "solid_queue_semaphores", force: :cascade do |t|
+    t.string "key", null: false
+    t.integer "value", default: 1, null: false
+    t.datetime "expires_at", null: false
+    t.datetime "created_at", null: false
+    t.datetime "updated_at", null: false
+    t.index [ "expires_at" ], name: "index_solid_queue_semaphores_on_expires_at"
+    t.index [ "key", "value" ], name: "index_solid_queue_semaphores_on_key_and_value"
+    t.index [ "key" ], name: "index_solid_queue_semaphores_on_key", unique: true
+  end
+
+  add_foreign_key "solid_queue_blocked_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+  add_foreign_key "solid_queue_claimed_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+  add_foreign_key "solid_queue_failed_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+  add_foreign_key "solid_queue_ready_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+  add_foreign_key "solid_queue_recurring_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+  add_foreign_key "solid_queue_scheduled_executions", "solid_queue_jobs", column: "job_id", on_delete: :cascade
+end

data/lib/solid_queue/lifecycle_hooks.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module LifecycleHooks
+    extend ActiveSupport::Concern
+
+    included do
+      mattr_reader :lifecycle_hooks, default: { start: [], stop: [] }
+    end
+
+    class_methods do
+      def on_start(&block)
+        self.lifecycle_hooks[:start] << block
+      end
+
+      def on_stop(&block)
+        self.lifecycle_hooks[:stop] << block
+      end
+
+      def clear_hooks
+        self.lifecycle_hooks[:start] = []
+        self.lifecycle_hooks[:stop] = []
+      end
+    end
+
+    private
+      def run_start_hooks
+        run_hooks_for :start
+      end
+
+      def run_stop_hooks
+        run_hooks_for :stop
+      end
+
+      def run_hooks_for(event)
+        self.class.lifecycle_hooks.fetch(event, []).each do |block|
+          block.call
+        rescue Exception => exception
+          handle_thread_error(exception)
+        end
+      end
+  end
+end

data/lib/solid_queue/processes/runnable.rb

@@ -7,11 +7,7 @@ module SolidQueue::Processes
     attr_writer :mode
 
     def start
-      @stopped = false
-
-      SolidQueue.instrument(:start_process, process: self) do
-        run_callbacks(:boot) { boot }
-      end
+      boot
 
       if running_async?
         @thread = create_thread { run }
@@ -25,10 +21,6 @@ module SolidQueue::Processes
       @thread&.join
     end
 
-    def alive?
-      !running_async? || @thread.alive?
-    end
-
     private
       DEFAULT_MODE = :async
 
@@ -37,9 +29,15 @@ module SolidQueue::Processes
       end
 
       def boot
-        if running_as_fork?
-          register_signal_handlers
-          set_procline
+        SolidQueue.instrument(:start_process, process: self) do
+          run_callbacks(:boot) do
+            @stopped = false
+
+            if running_as_fork?
+              register_signal_handlers
+              set_procline
+            end
+          end
         end
       end
 

data/lib/solid_queue/supervisor.rb

@@ -2,6 +2,7 @@
 
 module SolidQueue
   class Supervisor < Processes::Base
+    include LifecycleHooks
     include Maintenance, Signals, Pidfiled
 
     class << self
@@ -27,6 +28,7 @@ module SolidQueue
 
     def start
       boot
+      run_start_hooks
 
       start_processes
       launch_maintenance_task
@@ -36,6 +38,7 @@ module SolidQueue
 
     def stop
       @stopped = true
+      run_stop_hooks
     end
 
     private

data/lib/solid_queue/tasks.rb

@@ -1,4 +1,9 @@
 namespace :solid_queue do
+  desc "Install Solid Queue"
+  task :install do
+    Rails::Command.invoke :generate, [ "solid_queue:install" ]
+  end
+
   desc "start solid_queue supervisor to dispatch and process jobs"
   task start: :environment do
     SolidQueue::Supervisor.start

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.7.0"
+  VERSION = "0.8.1"
 end

data/lib/solid_queue/worker.rb

@@ -2,6 +2,11 @@
 
 module SolidQueue
   class Worker < Processes::Poller
+    include LifecycleHooks
+
+    after_boot :run_start_hooks
+    before_shutdown :run_stop_hooks
+
     attr_accessor :queues, :pool
 
     def initialize(**options)

data/lib/solid_queue.rb

@@ -43,6 +43,16 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
+  delegate :on_start, :on_stop, to: Supervisor
+
+  def on_worker_start(...)
+    Worker.on_start(...)
+  end
+
+  def on_worker_stop(...)
+    Worker.on_stop(...)
+  end
+
   def supervisor?
     supervisor
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.1
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-02 00:00:00.000000000 Z
+date: 2024-09-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -203,7 +203,6 @@ files:
 - README.md
 - Rakefile
 - UPGRADING.md
-- app/jobs/solid_queue/recurring_job.rb
 - app/models/solid_queue/blocked_execution.rb
 - app/models/solid_queue/claimed_execution.rb
 - app/models/solid_queue/execution.rb
@@ -231,19 +230,13 @@ files:
 - app/models/solid_queue/scheduled_execution.rb
 - app/models/solid_queue/semaphore.rb
 - config/routes.rb
-- db/migrate/20231211200639_create_solid_queue_tables.rb
-- db/migrate/20240110143450_add_missing_index_to_blocked_executions.rb
-- db/migrate/20240218110712_create_recurring_executions.rb
-- db/migrate/20240719134516_create_recurring_tasks.rb
-- db/migrate/20240811173327_add_name_to_processes.rb
-- db/migrate/20240813160053_make_name_not_null.rb
-- db/migrate/20240819165045_change_solid_queue_recurring_tasks_static_to_not_null.rb
 - lib/active_job/concurrency_controls.rb
 - lib/active_job/queue_adapters/solid_queue_adapter.rb
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
-- lib/generators/solid_queue/install/templates/config.yml
-- lib/generators/solid_queue/install/templates/jobs
+- lib/generators/solid_queue/install/templates/bin/jobs
+- lib/generators/solid_queue/install/templates/config/solid_queue.yml
+- lib/generators/solid_queue/install/templates/db/queue_schema.rb
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
 - lib/solid_queue/app_executor.rb
@@ -253,6 +246,7 @@ files:
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
 - lib/solid_queue/dispatcher/recurring_schedule.rb
 - lib/solid_queue/engine.rb
+- lib/solid_queue/lifecycle_hooks.rb
 - lib/solid_queue/log_subscriber.rb
 - lib/solid_queue/pool.rb
 - lib/solid_queue/processes/base.rb
@@ -282,9 +276,11 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
-  configuration and new migrations. Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
+  Upgrading to Solid Queue 0.8.0 from < 0.6.0? You need to upgrade to 0.6.0 first. Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
   for upgrade instructions.
+
+  Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
+  configuration and new migrations.
 rdoc_options: []
 require_paths:
 - lib

data/app/jobs/solid_queue/recurring_job.rb

@@ -1,9 +0,0 @@
-# frozen_string_literal: true
-
-class SolidQueue::RecurringJob < ActiveJob::Base
-  def perform(command)
-    SolidQueue.instrument(:run_command, command: command) do
-      eval(command, TOPLEVEL_BINDING, __FILE__, __LINE__)
-    end
-  end
-end

data/db/migrate/20231211200639_create_solid_queue_tables.rb

@@ -1,100 +0,0 @@
-class CreateSolidQueueTables < ActiveRecord::Migration[7.0]
-  def change
-    create_table :solid_queue_jobs do |t|
-      t.string :queue_name, null: false
-      t.string :class_name, null: false, index: true
-      t.text :arguments
-      t.integer :priority, default: 0, null: false
-      t.string :active_job_id, index: true
-      t.datetime :scheduled_at
-      t.datetime :finished_at, index: true
-      t.string :concurrency_key
-
-      t.timestamps
-
-      t.index [ :queue_name, :finished_at ], name: "index_solid_queue_jobs_for_filtering"
-      t.index [ :scheduled_at, :finished_at ], name: "index_solid_queue_jobs_for_alerting"
-    end
-
-    create_table :solid_queue_scheduled_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.string :queue_name, null: false
-      t.integer :priority, default: 0, null: false
-      t.datetime :scheduled_at, null: false
-
-      t.datetime :created_at, null: false
-
-      t.index [ :scheduled_at, :priority, :job_id ], name: "index_solid_queue_dispatch_all"
-    end
-
-    create_table :solid_queue_ready_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.string :queue_name, null: false
-      t.integer :priority, default: 0, null: false
-
-      t.datetime :created_at, null: false
-
-      t.index [ :priority, :job_id ], name: "index_solid_queue_poll_all"
-      t.index [ :queue_name, :priority, :job_id ], name: "index_solid_queue_poll_by_queue"
-    end
-
-    create_table :solid_queue_claimed_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.bigint :process_id
-      t.datetime :created_at, null: false
-
-      t.index [ :process_id, :job_id ]
-    end
-
-    create_table :solid_queue_blocked_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.string :queue_name, null: false
-      t.integer :priority, default: 0, null: false
-      t.string :concurrency_key, null: false
-      t.datetime :expires_at, null: false
-
-      t.datetime :created_at, null: false
-
-      t.index [ :expires_at, :concurrency_key ], name: "index_solid_queue_blocked_executions_for_maintenance"
-    end
-
-    create_table :solid_queue_failed_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.text :error
-      t.datetime :created_at, null: false
-    end
-
-    create_table :solid_queue_pauses do |t|
-      t.string :queue_name, null: false, index: { unique: true }
-      t.datetime :created_at, null: false
-    end
-
-    create_table :solid_queue_processes do |t|
-      t.string :kind, null: false
-      t.datetime :last_heartbeat_at, null: false, index: true
-      t.bigint :supervisor_id, index: true
-
-      t.integer :pid, null: false
-      t.string :hostname
-      t.text :metadata
-
-      t.datetime :created_at, null: false
-    end
-
-    create_table :solid_queue_semaphores do |t|
-      t.string :key, null: false, index: { unique: true }
-      t.integer :value, default: 1, null: false
-      t.datetime :expires_at, null: false, index: true
-
-      t.timestamps
-
-      t.index [ :key, :value ], name: "index_solid_queue_semaphores_on_key_and_value"
-    end
-
-    add_foreign_key :solid_queue_blocked_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-    add_foreign_key :solid_queue_claimed_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-    add_foreign_key :solid_queue_failed_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-    add_foreign_key :solid_queue_ready_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-    add_foreign_key :solid_queue_scheduled_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-  end
-end

data/db/migrate/20240110143450_add_missing_index_to_blocked_executions.rb

@@ -1,5 +0,0 @@
-class AddMissingIndexToBlockedExecutions < ActiveRecord::Migration[7.1]
-  def change
-    add_index :solid_queue_blocked_executions, [ :concurrency_key, :priority, :job_id ], name: "index_solid_queue_blocked_executions_for_release"
-  end
-end

data/db/migrate/20240218110712_create_recurring_executions.rb

@@ -1,14 +0,0 @@
-class CreateRecurringExecutions < ActiveRecord::Migration[7.1]
-  def change
-    create_table :solid_queue_recurring_executions do |t|
-      t.references :job, index: { unique: true }, null: false
-      t.string :task_key, null: false
-      t.datetime :run_at, null: false
-      t.datetime :created_at, null: false
-
-      t.index [ :task_key, :run_at ], unique: true
-    end
-
-    add_foreign_key :solid_queue_recurring_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
-  end
-end

data/db/migrate/20240719134516_create_recurring_tasks.rb

@@ -1,20 +0,0 @@
-class CreateRecurringTasks < ActiveRecord::Migration[7.1]
-  def change
-    create_table :solid_queue_recurring_tasks do |t|
-      t.string :key, null: false, index: { unique: true }
-      t.string :schedule, null: false
-      t.string :command, limit: 2048
-      t.string :class_name
-      t.text :arguments
-
-      t.string :queue_name
-      t.integer :priority, default: 0
-
-      t.boolean :static, default: true, index: true
-
-      t.text :description
-
-      t.timestamps
-    end
-  end
-end

data/db/migrate/20240811173327_add_name_to_processes.rb

@@ -1,5 +0,0 @@
-class AddNameToProcesses < ActiveRecord::Migration[7.1]
-  def change
-    add_column :solid_queue_processes, :name, :string
-  end
-end

data/db/migrate/20240813160053_make_name_not_null.rb

@@ -1,16 +0,0 @@
-class MakeNameNotNull < ActiveRecord::Migration[7.1]
-  def up
-    SolidQueue::Process.where(name: nil).find_each do |process|
-      process.name ||= [ process.kind.downcase, SecureRandom.hex(10) ].join("-")
-      process.save!
-    end
-
-    change_column :solid_queue_processes, :name, :string, null: false
-    add_index :solid_queue_processes, [ :name, :supervisor_id ], unique: true
-  end
-
-  def down
-    remove_index :solid_queue_processes, [ :name, :supervisor_id ]
-    change_column :solid_queue_processes, :name, :string, null: true
-  end
-end

data/db/migrate/20240819165045_change_solid_queue_recurring_tasks_static_to_not_null.rb

@@ -1,5 +0,0 @@
-class ChangeSolidQueueRecurringTasksStaticToNotNull < ActiveRecord::Migration[7.1]
-  def change
-    change_column_null :solid_queue_recurring_tasks, :static, false, true
-  end
-end

data/lib/generators/solid_queue/install/templates/config.yml

@@ -1,18 +0,0 @@
-# default: &default
-#   dispatchers:
-#     - polling_interval: 1
-#       batch_size: 500
-#   workers:
-#     - queues: "*"
-#       threads: 3
-#       processes: 1
-#       polling_interval: 0.1
-#
-# development:
-#  <<: *default
-#
-# test:
-#  <<: *default
-#
-# production:
-#  <<: *default