solid_queue 0.9.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b0a582a0c2b6b0025baed737e35b41dbe2dc687b0fbd7a221909b6350321a717
-  data.tar.gz: 85053c1ab6f8b7d5b66855631a003e789306499eba71d100ae91a32e39757b97
+  metadata.gz: 553dc884dd38c24c16196da9cceb6e7b40d02bd3f5c005bc980065232b4b1682
+  data.tar.gz: dba6309954b326305526417c05c1a5878a0195d759e8cd6fb6c4453ee61cd6c0
 SHA512:
-  metadata.gz: 630b5c980be4b1e152544e46eb38922706b620765ffb1b9c18ff81a9c00a221be2d0ed2f723ecd7d39391a01119785b3be42db408f7e48a6b0afd67ab62088eb
-  data.tar.gz: 4451b96428dfbaa8213d9fff225dfeaa5826aa10d6ab68151effbba01df866c8c19110e4c6f147a25b70fa34827c6a6c03381b1fb7681934b036791ba4f16faa
+  metadata.gz: dc6ad2d8abd513cd9f357a1190c2173c978b46dd800d1b3f4040364cd70c44b5271da7108901c67a2811586d746607eac4f999c8886a56a1942a228f0f3d1d11
+  data.tar.gz: 999a90c18c09350609670b29bdd3f5e2596815f0a5edb194b225634b9169eca9efaf3e8efda23cfe346e6947e9477b983bbcae1a8a1adf3c6fc2ae9affd4e2ff

data/README.md

@@ -15,7 +15,7 @@ Solid Queue is configured by default in new Rails 8 applications. But if you're
 
 This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
 
-Once you've done that, you will then have to add the configuration for the queue database in `config/database.yml`. If you're using sqlite, it'll look like this:
+Once you've done that, you will then have to add the configuration for the queue database in `config/database.yml`. If you're using SQLite, it'll look like this:
 
 ```yaml
 production:
@@ -51,9 +51,12 @@ Now you're ready to start processing jobs by running `bin/jobs` on the server th
 
 For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
+**Note**: future changes to the schema will come in the form of regular migrations.
+
+
 ### Single database configuration
 
-It's also possibile to use one single database for both production data:
+Running Solid Queue in a separate database is recommended, but it's also possible to use one single database for both the app and the queue. Just follow these steps:
 
 1. Copy the contents of `db/queue_schema.rb` into a normal migration and delete `db/queue_schema.rb`
 2. Remove `config.solid_queue.connects_to` from `production.rb`
@@ -220,6 +223,7 @@ There are several settings that control how Solid Queue works that you can set a
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
+
   **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
 
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
@@ -231,7 +235,6 @@ There are several settings that control how Solid Queue works that you can set a
 - `preserve_finished_jobs`: whether to keep finished jobs in the `solid_queue_jobs` table—defaults to `true`.
 - `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true—defaults to 1 day. **Note:** Right now, there's no automatic cleanup of finished jobs. You'd need to do this by periodically invoking `SolidQueue::Job.clear_finished_in_batches`, but this will happen automatically in the near future.
 - `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
-- `enqueue_after_transaction_commit`: whether the job queuing is deferred to after the current Active Record transaction is committed. The default is `false`. [Read more](https://github.com/rails/rails/pull/51426).
 
 ## Errors when enqueuing
 
@@ -256,7 +259,7 @@ class MyJob < ApplicationJob
 
 When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as `to`) that yield the same `key` will be performed concurrently, and this guarantee will last for `duration` for each job enqueued. Note that there's no guarantee about _the order of execution_, only about jobs being performed at the same time (overlapping).
 
-The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as _ready_. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccesfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from _blocked_ to _ready_. Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the `duration` as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the `duration` is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting.
+The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as _ready_. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccessfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from _blocked_ to _ready_. Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the `duration` as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the `duration` is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting.
 
 
 For example:
@@ -308,6 +311,33 @@ failed_execution.discard # This will delete the job from the system
 
 However, we recommend taking a look at [mission_control-jobs](https://github.com/rails/mission_control-jobs), a dashboard where, among other things, you can examine and retry/discard failed jobs.
 
+### Error reporting on jobs
+
+Some error tracking services that integrate with Rails, such as Sentry or Rollbar, hook into [Active Job](https://guides.rubyonrails.org/active_job_basics.html#exceptions) and automatically report not handled errors that happen during job execution. However, if your error tracking system doesn't, or if you need some custom reporting, you can hook into Active Job yourself. A possible way of doing this would be:
+
+```ruby
+# application_job.rb
+class ApplicationJob < ActiveJob::Base
+  rescue_from(Exception) do |exception|
+    Rails.error.report(exception)
+    raise exception
+  end
+end
+```
+
+Note that, you will have to duplicate the above logic on `ActionMailer::MailDeliveryJob` too. That is because `ActionMailer` doesn't inherit from `ApplicationJob` but instead uses `ActionMailer::MailDeliveryJob` which inherits from `ActiveJob::Base`.
+
+```ruby
+# application_mailer.rb
+
+class ApplicationMailer < ActionMailer::Base
+  ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception|
+    Rails.error.report(exception)
+    raise exception
+  end
+end
+```
+
 ## Puma plugin
 
 We provide a Puma plugin if you want to run the Solid Queue's supervisor together with Puma and have Puma monitor and manage it. You just need to add
@@ -316,6 +346,29 @@ plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
+
+## Jobs and transactional integrity
+:warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed and viceversa, and ensuring that your job won't be enqueued until the transaction within which you're enqueing it is committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.
+
+Because this can be quite tricky and many people shouldn't need to worry about it, by default Solid Queue is configured in a different database as the main app, **job enqueuing is deferred until any ongoing transaction is committed** thanks to Active Job's built-in capability to do this. This means that even if you run Solid Queue in the same DB as your app, you won't be taking advantage of this transactional integrity.
+
+If you prefer to change this, you can set [`config.active_job.enqueue_after_transaction_commit`](https://edgeguides.rubyonrails.org/configuring.html#config-active-job-enqueue-after-transaction-commit) to `never`. You can also set this on a per-job basis.
+
+If you set that to `never` but still want to make sure you're not inadvertently on transactional integrity, you can make sure that:
+- Your jobs relying on specific data are always enqueued on [`after_commit` callbacks](https://guides.rubyonrails.org/active_record_callbacks.html#after-commit-and-after-rollback) or otherwise from a place where you're certain that whatever data the job will use has been committed to the database before the job is enqueued.
+- Or, you configure a different database for Solid Queue, even if it's the same as your app, ensuring that a different connection on the thread handling requests or running jobs for your app will be used to enqueue jobs. For example:
+
+  ```ruby
+  class ApplicationRecord < ActiveRecord::Base
+    self.abstract_class = true
+
+    connects_to database: { writing: :primary, reading: :replica }
+  ```
+
+  ```ruby
+  config.solid_queue.connects_to = { database: { writing: :primary, reading: :replica } }
+  ```
+
 ## Recurring tasks
 
 Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by the scheduler process and are defined in their own configuration file. By default, the file is located in `config/recurring.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_RECURRING_SCHEDULE` or by using the `--recurring_schedule_file` option with `bin/jobs`, like this:
@@ -327,13 +380,14 @@ bin/jobs --recurring_schedule_file=config/schedule.yml
 The configuration itself looks like this:
 
 ```yml
-a_periodic_job:
-  class: MyJob
-  args: [ 42, { status: "custom_status" } ]
-  schedule: every second
-a_cleanup_task:
-  command: "DeletedStuff.clear_all"
-  schedule: every day at 9am
+production:
+  a_periodic_job:
+    class: MyJob
+    args: [ 42, { status: "custom_status" } ]
+    schedule: every second
+  a_cleanup_task:
+    command: "DeletedStuff.clear_all"
+    schedule: every day at 9am
 ```
 
 Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a `class`, which will be the job class to enqueue, or a `command`, which will be eval'ed in the context of a job (`SolidQueue::RecurringJob`) that will be enqueued according to its schedule, in the `solid_queue_recurring` queue.
@@ -355,6 +409,8 @@ Tasks are enqueued at their corresponding times by the scheduler, and each task
 
 It's possible to run multiple schedulers with the same `recurring_tasks` configuration, for example, if you have multiple servers for redundancy, and you run the `scheduler` in more than one of them. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
 
+**Note**: a single recurring schedule is supported, so you can have multiple schedulers using the same schedule, but not multiple schedulers using different configurations.
+
 Finally, it's possible to configure jobs that aren't handled by Solid Queue. That is, you can have a job like this in your app:
 ```ruby
 class MyResqueJob < ApplicationJob

data/UPGRADING.md

@@ -1,26 +1,23 @@
+# Upgrading to version 1.x
+The value returned for `enqueue_after_transaction_commit?` has changed to `true`, and it's no longer configurable. If you want to change this, you need to use Active Job's configuration options.
+
 # Upgrading to version 0.9.x
 This version has two breaking changes regarding configuration:
 - The default configuration file has changed from `config/solid_queue.yml` to `config/queue.yml`.
 - Recurring tasks are now defined in `config/recurring.yml` (by default). Before, they would be defined as part of the _dispatcher_ configuration. Now they've been upgraded to their own configuration file, and a dedicated process (the _scheduler_) to manage them. Check the _Recurring tasks_ section in the `README` to learn how to configure them in detail. They still follow the same format as before when they lived under `dispatchers > recurring_tasks`.
 
 # Upgrading to version 0.8.x
-*IMPORTANT*: This version collapsed all migrations into a single `db/queue_schema.rb`, that will use a separate `queue` database. If you're upgrading from a version < 0.6.0, you need to upgrade to 0.6.0 first, ensure all migrations are up-to-date, and then upgrade further.
+*IMPORTANT*: This version collapsed all migrations into a single `db/queue_schema.rb`, that will use a separate `queue` database on install. If you're upgrading from a version < 0.6.0, you need to upgrade to 0.6.0 first, ensure all migrations are up-to-date, and then upgrade further. You don't have to switch to a separate `queue` database or use the new `db/queue_schema.rb` file, these are for people starting on a version >= 0.8.x. You can continue using your existing database (be it separate or the same as your app) as long as you run all migrations defined up to version 0.6.0.
 
 # Upgrading to version 0.7.x
 
 This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor.
 
-To install both the binstub `bin/jobs` and the migration, you can just run
+To install the binstub `bin/jobs`, you can just run:
 ```
 bin/rails generate solid_queue:install
 ```
 
-Or, if you're using a different database for Solid Queue:
-
-```bash
-$ bin/rails generate solid_queue:install --database <the_name_of_your_solid_queue_db>
-```
-
 
 # Upgrading to version 0.6.x
 

data/app/models/solid_queue/process.rb

@@ -20,7 +20,11 @@ class SolidQueue::Process < SolidQueue::Record
   end
 
   def heartbeat
-    touch(:last_heartbeat_at)
+    # Clear any previous changes before locking, for example, in case a previous heartbeat
+    # failed because of a DB issue (with SQLite depending on configuration, a BusyException
+    # is not rare) and we still have the unpersisted value
+    restore_attributes
+    with_lock { touch(:last_heartbeat_at) }
   end
 
   def deregister(pruned: false)

data/lib/active_job/queue_adapters/solid_queue_adapter.rb

@@ -9,7 +9,7 @@ module ActiveJob
     #   Rails.application.config.active_job.queue_adapter = :solid_queue
     class SolidQueueAdapter
       def enqueue_after_transaction_commit?
-        SolidQueue.enqueue_after_transaction_commit
+        true
       end
 
       def enqueue(active_job) # :nodoc:

data/lib/generators/solid_queue/install/templates/config/recurring.yml

@@ -1,9 +1,10 @@
-# periodic_cleanup:
-#   class: CleanSoftDeletedRecordsJob
-#   queue: background
-#   args: [ 1000, { batch_size: 500 } ]
-#   schedule: every hour
-# periodic_command:
-#   command: "SoftDeletedRecord.due.delete_all"
-#   priority: 2
-#   schedule: at 5am every day
+# production:
+#   periodic_cleanup:
+#     class: CleanSoftDeletedRecordsJob
+#     queue: background
+#     args: [ 1000, { batch_size: 500 } ]
+#     schedule: every hour
+#   periodic_command:
+#     command: "SoftDeletedRecord.due.delete_all"
+#     priority: 2
+#     schedule: at 5am every day

data/lib/generators/solid_queue/install/templates/db/queue_schema.rb

@@ -1,4 +1,4 @@
-ActiveRecord::Schema[7.1].define(version: 2024_09_04_193154) do
+ActiveRecord::Schema[7.1].define(version: 1) do
   create_table "solid_queue_blocked_executions", force: :cascade do |t|
     t.bigint "job_id", null: false
     t.string "queue_name", null: false

data/lib/solid_queue/cli.rb

@@ -5,16 +5,15 @@ require "thor"
 module SolidQueue
   class Cli < Thor
     class_option :config_file, type: :string, aliases: "-c",
-      default: Configuration::DEFAULT_CONFIG_FILE_PATH,
-      desc: "Path to config file",
+      desc: "Path to config file (default: #{Configuration::DEFAULT_CONFIG_FILE_PATH}).",
       banner: "SOLID_QUEUE_CONFIG"
 
     class_option :recurring_schedule_file, type: :string,
-      default: Configuration::DEFAULT_RECURRING_SCHEDULE_FILE_PATH,
-      desc: "Path to recurring schedule definition",
+      desc: "Path to recurring schedule definition (default: #{Configuration::DEFAULT_RECURRING_SCHEDULE_FILE_PATH}).",
       banner: "SOLID_QUEUE_RECURRING_SCHEDULE"
 
-    class_option :skip_recurring, type: :boolean, default: false
+    class_option :skip_recurring, type: :boolean, default: false,
+      desc: "Whether to skip recurring tasks scheduling"
 
     def self.exit_on_failure?
       true

data/lib/solid_queue/configuration.rb

@@ -111,7 +111,7 @@ module SolidQueue
       end
 
       def recurring_tasks_config
-        @recurring_tasks ||= config_from options[:recurring_schedule_file]
+        @recurring_tasks_config ||= config_from options[:recurring_schedule_file]
       end
 
 

data/lib/solid_queue/processes/registrable.rb

@@ -53,7 +53,7 @@ module SolidQueue::Processes
       end
 
       def heartbeat
-        process.with_lock { process.heartbeat }
+        process.heartbeat
       rescue ActiveRecord::RecordNotFound
         self.process = nil
         wake_up

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.9.0"
+  VERSION = "1.0.0"
 end

data/lib/solid_queue.rb

@@ -32,8 +32,6 @@ module SolidQueue
 
   mattr_accessor :shutdown_timeout, default: 5.seconds
 
-  mattr_accessor :enqueue_after_transaction_commit, default: false
-
   mattr_accessor :silence_polling, default: true
 
   mattr_accessor :supervisor_pidfile

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-11 00:00:00.000000000 Z
+date: 2024-09-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord