solid_queue 0.3.4 → 0.4.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: b4a85fb424e127543352846ee1b0b4aa958d4c77a7a9e1c26a8082464ae7bebc
4
- data.tar.gz: f15b4db5c77c14af4b989fc8abe2670fd49006853ca9bfce0080e8b15df733c3
3
+ metadata.gz: cb47bf3ee9dffa1300c093abb260394d088f100b3c16a026d755f706d9f7a852
4
+ data.tar.gz: 10cc1b6f866d148d0c1cefce7587614aaca3e938f3f04370c15318e5b8d3509f
5
5
  SHA512:
6
- metadata.gz: ba913d760e6f1ac8bac01f68b3ffeb61811952d13a7862731c517dea0ec3e8938b8fb5194113e356fe8a35a02f4d95b306f327772e21f8f6098349bd1235dc53
7
- data.tar.gz: 7418c2db5be34b59e123395ee3edf519d703b6e0488a564a85c24dc22344d67e894ad8bb378f88a3cd6e0b9085602be1850dab60c168816e92b081364b720594
6
+ metadata.gz: 71054017fcd26421f25140db9c929518ee011210fd6fea51e77352abf2a128289e965843dc1cb1fe97b62fa003d571e1de3631fcab329914b069ef9e0621806b
7
+ data.tar.gz: d08497fc98f9498aeaa3449182fc13cb90f71c859c67d55e0e5c14803bb1d5445e7717e1e3aca578fcb22a09c47126d465306e0220f54d6af4a8f8027fef0072
data/README.md CHANGED
@@ -66,7 +66,7 @@ $ bundle exec rake solid_queue:start
66
66
 
67
67
  This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
68
68
 
69
- For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. If you'd like to designate some machines to be only dispatchers or only workers, use `bundle exec rake solid_queue:dispatch` or `bundle exec rake solid_queue:work`, respectively.
69
+ For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
70
70
 
71
71
  ## Requirements
72
72
  Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
@@ -75,10 +75,12 @@ Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as t
75
75
 
76
76
  ### Workers and dispatchers
77
77
 
78
- We have three types of processes in Solid Queue:
78
+ We have three types of actors in Solid Queue:
79
79
  - _Workers_ are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table.
80
80
  - _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
81
- - The _supervisor_ forks workers and dispatchers according to the configuration, controls their heartbeats, and sends them signals to stop and start them when needed.
81
+ - The _supervisor_ runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.
82
+
83
+ By default, Solid Queue runs in `fork` mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher. There's also an `async` mode where each worker and dispatcher will be run as a thread of the supervisor process. This can be used with [the provided Puma plugin](#puma-plugin)
82
84
 
83
85
  By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
84
86
 
@@ -98,7 +100,18 @@ production:
98
100
  processes: 3
99
101
  ```
100
102
 
101
- Everything is optional. If no configuration is provided, Solid Queue will run with one dispatcher and one worker with default settings.
103
+ Everything is optional. If no configuration at all is provided, Solid Queue will run with one dispatcher and one worker with default settings. If you want to run only dispatchers or workers, you just need to include that section alone in the configuration. For example, with the following configuration:
104
+
105
+ ```yml
106
+ production:
107
+ dispatchers:
108
+ - polling_interval: 1
109
+ batch_size: 500
110
+ concurrency_maintenance_interval: 300
111
+ ```
112
+ the supervisor will run 1 dispatcher and no workers.
113
+
114
+ Here's an overview of the different options:
102
115
 
103
116
  - `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `1` second for dispatchers and `0.1` seconds for workers.
104
117
  - `batch_size`: the dispatcher will dispatch jobs in batches of this size. The default is 500.
@@ -118,7 +131,7 @@ Everything is optional. If no configuration is provided, Solid Queue will run wi
118
131
 
119
132
  Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
120
133
  - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
121
- - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
134
+ - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. **Note**: this option will be ignored if [running in `async` mode](#running-as-a-fork-or-asynchronously).
122
135
  - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
123
136
  - `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
124
137
 
@@ -232,7 +245,8 @@ class MyJob < ApplicationJob
232
245
  # ...
233
246
  ```
234
247
  - `key` is the only required parameter, and it can be a symbol, a string or a proc that receives the job arguments as parameters and will be used to identify the jobs that need to be limited together. If the proc returns an Active Record record, the key will be built from its class name and `id`.
235
- - `to` is `1` by default, and `duration` is set to `SolidQueue.default_concurrency_control_period` by default, which itself defaults to `3 minutes`, but that you can configure as well.
248
+ - `to` is `1` by default.
249
+ - `duration` is set to `SolidQueue.default_concurrency_control_period` by default, which itself defaults to `3 minutes`, but that you can configure as well.
236
250
  - `group` is used to control the concurrency of different job classes together. It defaults to the job class name.
237
251
 
238
252
  When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as `to`) that yield the same `key` will be performed concurrently, and this guarantee will last for `duration` for each job enqueued. Note that there's no guarantee about _the order of execution_, only about jobs being performed at the same time (overlapping).
@@ -291,6 +305,18 @@ plugin :solid_queue
291
305
  ```
292
306
  to your `puma.rb` configuration.
293
307
 
308
+ ### Running as a fork or asynchronously
309
+
310
+ By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.
311
+
312
+ Alternatively, workers and dispatchers can be run within the same Puma process(s). To do so just configure the plugin as:
313
+
314
+ ```ruby
315
+ plugin :solid_queue
316
+ solid_queue_mode :async
317
+ ```
318
+
319
+ Note that in this case, the `processes` configuration option will be ignored.
294
320
 
295
321
  ## Jobs and transactional integrity
296
322
  :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.
@@ -15,9 +15,14 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
15
15
  def claiming(job_ids, process_id, &block)
16
16
  job_data = Array(job_ids).collect { |job_id| { job_id: job_id, process_id: process_id } }
17
17
 
18
- insert_all!(job_data)
19
- where(job_id: job_ids, process_id: process_id).load.tap do |claimed|
20
- block.call(claimed)
18
+ SolidQueue.instrument(:claim, process_id: process_id, job_ids: job_ids) do |payload|
19
+ insert_all!(job_data)
20
+ where(job_id: job_ids, process_id: process_id).load.tap do |claimed|
21
+ block.call(claimed)
22
+
23
+ payload[:size] = claimed.size
24
+ payload[:claimed_job_ids] = claimed.map(&:job_id)
25
+ end
21
26
  end
22
27
  end
23
28
 
@@ -0,0 +1,20 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SolidQueue
4
+ class Process
5
+ module Executor
6
+ extend ActiveSupport::Concern
7
+
8
+ included do
9
+ has_many :claimed_executions
10
+
11
+ after_destroy -> { claimed_executions.release_all }, if: :claims_executions?
12
+ end
13
+
14
+ private
15
+ def claims_executions?
16
+ kind == "Worker"
17
+ end
18
+ end
19
+ end
20
+ end
@@ -1,19 +1,23 @@
1
1
  # frozen_string_literal: true
2
2
 
3
- module SolidQueue::Process::Prunable
4
- extend ActiveSupport::Concern
3
+ module SolidQueue
4
+ class Process
5
+ module Prunable
6
+ extend ActiveSupport::Concern
5
7
 
6
- included do
7
- scope :prunable, -> { where(last_heartbeat_at: ..SolidQueue.process_alive_threshold.ago) }
8
- end
8
+ included do
9
+ scope :prunable, -> { where(last_heartbeat_at: ..SolidQueue.process_alive_threshold.ago) }
10
+ end
9
11
 
10
- class_methods do
11
- def prune
12
- SolidQueue.instrument :prune_processes, size: 0 do |payload|
13
- prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
14
- payload[:size] += batch.size
12
+ class_methods do
13
+ def prune
14
+ SolidQueue.instrument :prune_processes, size: 0 do |payload|
15
+ prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
16
+ payload[:size] += batch.size
15
17
 
16
- batch.each { |process| process.deregister(pruned: true) }
18
+ batch.each { |process| process.deregister(pruned: true) }
19
+ end
20
+ end
17
21
  end
18
22
  end
19
23
  end
@@ -1,19 +1,18 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  class SolidQueue::Process < SolidQueue::Record
4
- include Prunable
4
+ include Executor, Prunable
5
5
 
6
- belongs_to :supervisor, class_name: "SolidQueue::Process", optional: true, inverse_of: :forks
7
- has_many :forks, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id, dependent: :destroy
8
- has_many :claimed_executions
6
+ belongs_to :supervisor, class_name: "SolidQueue::Process", optional: true, inverse_of: :supervisees
7
+ has_many :supervisees, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id, dependent: :destroy
9
8
 
10
9
  store :metadata, coder: JSON
11
10
 
12
- after_destroy -> { claimed_executions.release_all }
13
-
14
11
  def self.register(**attributes)
15
- SolidQueue.instrument :register_process, **attributes do
16
- create!(attributes.merge(last_heartbeat_at: Time.current))
12
+ SolidQueue.instrument :register_process, **attributes do |payload|
13
+ create!(attributes.merge(last_heartbeat_at: Time.current)).tap do |process|
14
+ payload[:process_id] = process.id
15
+ end
17
16
  end
18
17
  rescue Exception => error
19
18
  SolidQueue.instrument :register_process, **attributes.merge(error: error)
@@ -25,7 +24,9 @@ class SolidQueue::Process < SolidQueue::Record
25
24
  end
26
25
 
27
26
  def deregister(pruned: false)
28
- SolidQueue.instrument :deregister_process, process: self, pruned: pruned, claimed_size: claimed_executions.size do |payload|
27
+ SolidQueue.instrument :deregister_process, process: self, pruned: pruned do |payload|
28
+ payload[:claimed_size] = claimed_executions.size if claims_executions?
29
+
29
30
  destroy!
30
31
  rescue Exception => error
31
32
  payload[:error] = error
@@ -1,28 +1,50 @@
1
1
  require "puma/plugin"
2
2
 
3
+ module Puma
4
+ class DSL
5
+ def solid_queue_mode(mode = :fork)
6
+ @options[:solid_queue_mode] = mode.to_sym
7
+ end
8
+ end
9
+ end
10
+
3
11
  Puma::Plugin.create do
4
- attr_reader :puma_pid, :solid_queue_pid, :log_writer
12
+ attr_reader :puma_pid, :solid_queue_pid, :log_writer, :solid_queue_supervisor
5
13
 
6
14
  def start(launcher)
7
15
  @log_writer = launcher.log_writer
8
16
  @puma_pid = $$
9
17
 
10
- in_background do
11
- monitor_solid_queue
18
+ if launcher.options[:solid_queue_mode] == :async
19
+ start_async(launcher)
20
+ else
21
+ start_forked(launcher)
12
22
  end
23
+ end
13
24
 
14
- launcher.events.on_booted do
15
- @solid_queue_pid = fork do
16
- Thread.new { monitor_puma }
17
- SolidQueue::Supervisor.start(mode: :all)
25
+ private
26
+ def start_forked(launcher)
27
+ in_background do
28
+ monitor_solid_queue
18
29
  end
30
+
31
+ launcher.events.on_booted do
32
+ @solid_queue_pid = fork do
33
+ Thread.new { monitor_puma }
34
+ SolidQueue::Supervisor.start(mode: :fork)
35
+ end
36
+ end
37
+
38
+ launcher.events.on_stopped { stop_solid_queue }
39
+ launcher.events.on_restart { stop_solid_queue }
19
40
  end
20
41
 
21
- launcher.events.on_stopped { stop_solid_queue }
22
- launcher.events.on_restart { stop_solid_queue }
23
- end
42
+ def start_async(launcher)
43
+ launcher.events.on_booted { @solid_queue_supervisor = SolidQueue::Supervisor.start(mode: :async) }
44
+ launcher.events.on_stopped { solid_queue_supervisor.stop }
45
+ launcher.events.on_restart { solid_queue_supervisor.stop; solid_queue_supervisor.start }
46
+ end
24
47
 
25
- private
26
48
  def stop_solid_queue
27
49
  Process.waitpid(solid_queue_pid, Process::WNOHANG)
28
50
  log "Stopping Solid Queue..."
@@ -17,38 +17,30 @@ module SolidQueue
17
17
  recurring_tasks: []
18
18
  }
19
19
 
20
- def initialize(mode: :work, load_from: nil)
21
- @mode = mode
20
+ def initialize(mode: :fork, load_from: nil)
21
+ @mode = mode.to_s.inquiry
22
22
  @raw_config = config_from(load_from)
23
23
  end
24
24
 
25
25
  def processes
26
- case mode
27
- when :dispatch then dispatchers
28
- when :work then workers
29
- when :all then dispatchers + workers
30
- else raise "Invalid mode #{mode}"
31
- end
26
+ dispatchers + workers
32
27
  end
33
28
 
34
29
  def workers
35
- if mode.in? %i[ work all]
36
- workers_options.flat_map do |worker_options|
37
- processes = worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
38
- processes.times.map { Worker.new(**worker_options.with_defaults(WORKER_DEFAULTS)) }
30
+ workers_options.flat_map do |worker_options|
31
+ processes = if mode.fork?
32
+ worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
33
+ else
34
+ WORKER_DEFAULTS[:processes]
39
35
  end
40
- else
41
- []
36
+ processes.times.map { Worker.new(**worker_options.with_defaults(WORKER_DEFAULTS)) }
42
37
  end
43
38
  end
44
39
 
45
40
  def dispatchers
46
- if mode.in? %i[ dispatch all]
47
- dispatchers_options.map do |dispatcher_options|
48
- recurring_tasks = parse_recurring_tasks dispatcher_options[:recurring_tasks]
49
-
50
- Dispatcher.new **dispatcher_options.merge(recurring_tasks: recurring_tasks).with_defaults(DISPATCHER_DEFAULTS)
51
- end
41
+ dispatchers_options.map do |dispatcher_options|
42
+ recurring_tasks = parse_recurring_tasks dispatcher_options[:recurring_tasks]
43
+ Dispatcher.new **dispatcher_options.merge(recurring_tasks: recurring_tasks).with_defaults(DISPATCHER_DEFAULTS)
52
44
  end
53
45
  end
54
46
 
@@ -68,15 +60,19 @@ module SolidQueue
68
60
  end
69
61
 
70
62
  def workers_options
71
- @workers_options ||= (raw_config[:workers] || [ WORKER_DEFAULTS ])
63
+ @workers_options ||= options_from_raw_config(:workers, WORKER_DEFAULTS)
72
64
  .map { |options| options.dup.symbolize_keys }
73
65
  end
74
66
 
75
67
  def dispatchers_options
76
- @dispatchers_options ||= (raw_config[:dispatchers] || [ DISPATCHER_DEFAULTS ])
68
+ @dispatchers_options ||= options_from_raw_config(:dispatchers, DISPATCHER_DEFAULTS)
77
69
  .map { |options| options.dup.symbolize_keys }
78
70
  end
79
71
 
72
+ def options_from_raw_config(key, defaults)
73
+ raw_config.empty? ? [ defaults ] : Array(raw_config[key])
74
+ end
75
+
80
76
  def parse_recurring_tasks(tasks)
81
77
  Array(tasks).map do |id, options|
82
78
  Dispatcher::RecurringTask.from_configuration(id, **options)
@@ -25,7 +25,7 @@ module SolidQueue
25
25
  end
26
26
 
27
27
  def stop
28
- @concurrency_maintenance_task.shutdown
28
+ @concurrency_maintenance_task&.shutdown
29
29
  end
30
30
 
31
31
  private
@@ -1,9 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SolidQueue
4
- class Dispatcher < Processes::Base
5
- include Processes::Poller
6
-
4
+ class Dispatcher < Processes::Poller
7
5
  attr_accessor :batch_size, :concurrency_maintenance, :recurring_schedule
8
6
 
9
7
  after_boot :start_concurrency_maintenance, :load_recurring_schedule
@@ -13,10 +11,11 @@ module SolidQueue
13
11
  options = options.dup.with_defaults(SolidQueue::Configuration::DISPATCHER_DEFAULTS)
14
12
 
15
13
  @batch_size = options[:batch_size]
16
- @polling_interval = options[:polling_interval]
17
14
 
18
15
  @concurrency_maintenance = ConcurrencyMaintenance.new(options[:concurrency_maintenance_interval], options[:batch_size]) if options[:concurrency_maintenance]
19
16
  @recurring_schedule = RecurringSchedule.new(options[:recurring_tasks])
17
+
18
+ super(**options)
20
19
  end
21
20
 
22
21
  def metadata
@@ -7,6 +7,10 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
7
7
  debug formatted_event(event, action: "Dispatch scheduled jobs", **event.payload.slice(:batch_size, :size))
8
8
  end
9
9
 
10
+ def claim(event)
11
+ debug formatted_event(event, action: "Claim jobs", **event.payload.slice(:process_id, :job_ids, :claimed_job_ids, :size))
12
+ end
13
+
10
14
  def release_many_claimed(event)
11
15
  debug formatted_event(event, action: "Release claimed jobs", **event.payload.slice(:size))
12
16
  end
@@ -40,19 +44,16 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
40
44
  end
41
45
 
42
46
  def enqueue_recurring_task(event)
43
- attributes = event.payload.slice(:task, :active_job_id, :enqueue_error, :at)
47
+ attributes = event.payload.slice(:task, :active_job_id, :enqueue_error)
48
+ attributes[:at] = event.payload[:at]&.iso8601
44
49
 
45
- if event.payload[:other_adapter]
46
- action = attributes[:active_job_id].present? ? "Enqueued recurring task outside Solid Queue" : "Error enqueuing recurring task"
47
- info formatted_event(event, action: action, **attributes)
50
+ if attributes[:active_job_id].nil? && event.payload[:skipped].nil?
51
+ error formatted_event(event, action: "Error enqueuing recurring task", **attributes)
52
+ elsif event.payload[:other_adapter]
53
+ debug formatted_event(event, action: "Enqueued recurring task outside Solid Queue", **attributes)
48
54
  else
49
- action = case
50
- when event.payload[:skipped].present? then "Skipped recurring task – already dispatched"
51
- when attributes[:active_job_id].nil? then "Error enqueuing recurring task"
52
- else "Enqueued recurring task"
53
- end
54
-
55
- info formatted_event(event, action: action, **attributes)
55
+ action = event.payload[:skipped].present? ? "Skipped recurring task – already dispatched" : "Enqueued recurring task"
56
+ debug formatted_event(event, action: action, **attributes)
56
57
  end
57
58
  end
58
59
 
@@ -61,7 +62,8 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
61
62
 
62
63
  attributes = {
63
64
  pid: process.pid,
64
- hostname: process.hostname
65
+ hostname: process.hostname,
66
+ process_id: process.process_id
65
67
  }.merge(process.metadata)
66
68
 
67
69
  info formatted_event(event, action: "Started #{process.kind}", **attributes)
@@ -72,15 +74,16 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
72
74
 
73
75
  attributes = {
74
76
  pid: process.pid,
75
- hostname: process.hostname
77
+ hostname: process.hostname,
78
+ process_id: process.process_id
76
79
  }.merge(process.metadata)
77
80
 
78
- info formatted_event(event, action: "Shut down #{process.kind}", **attributes)
81
+ info formatted_event(event, action: "Shutdown #{process.kind}", **attributes)
79
82
  end
80
83
 
81
84
  def register_process(event)
82
85
  process_kind = event.payload[:kind]
83
- attributes = event.payload.slice(:pid, :hostname)
86
+ attributes = event.payload.slice(:pid, :hostname, :process_id)
84
87
 
85
88
  if error = event.payload[:error]
86
89
  warn formatted_event(event, action: "Error registering #{process_kind}", **attributes.merge(error: formatted_error(error)))
@@ -96,9 +99,9 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
96
99
  process_id: process.id,
97
100
  pid: process.pid,
98
101
  hostname: process.hostname,
99
- last_heartbeat_at: process.last_heartbeat_at,
100
- claimed_size: process.claimed_executions.size,
101
- pruned: event.payload
102
+ last_heartbeat_at: process.last_heartbeat_at.iso8601,
103
+ claimed_size: event.payload[:claimed_size],
104
+ pruned: event.payload[:pruned]
102
105
  }
103
106
 
104
107
  if error = event.payload[:error]
@@ -117,7 +120,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
117
120
  end
118
121
 
119
122
  def graceful_termination(event)
120
- attributes = event.payload.slice(:supervisor_pid, :supervised_pids)
123
+ attributes = event.payload.slice(:process_id, :supervisor_pid, :supervised_processes)
121
124
 
122
125
  if event.payload[:shutdown_timeout_exceeded]
123
126
  warn formatted_event(event, action: "Supervisor wasn't terminated gracefully - shutdown timeout exceeded", **attributes)
@@ -127,7 +130,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
127
130
  end
128
131
 
129
132
  def immediate_termination(event)
130
- info formatted_event(event, action: "Supervisor terminated immediately", **event.payload.slice(:supervisor_pid, :supervised_pids))
133
+ info formatted_event(event, action: "Supervisor terminated immediately", **event.payload.slice(:process_id, :supervisor_pid, :supervised_processes))
131
134
  end
132
135
 
133
136
  def unhandled_signal_error(event)
@@ -8,12 +8,5 @@ module SolidQueue::Processes
8
8
  extend ActiveModel::Callbacks
9
9
  define_model_callbacks :boot, :shutdown
10
10
  end
11
-
12
- private
13
- def boot
14
- end
15
-
16
- def shutdown
17
- end
18
11
  end
19
12
  end
@@ -1,13 +1,13 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SolidQueue::Processes
4
- module Poller
5
- extend ActiveSupport::Concern
6
-
4
+ class Poller < Base
7
5
  include Runnable
8
6
 
9
- included do
10
- attr_accessor :polling_interval
7
+ attr_accessor :polling_interval
8
+
9
+ def initialize(polling_interval:, **options)
10
+ @polling_interval = polling_interval
11
11
  end
12
12
 
13
13
  def metadata
@@ -16,11 +16,7 @@ module SolidQueue::Processes
16
16
 
17
17
  private
18
18
  def run
19
- if mode.async?
20
- @thread = Thread.new { start_loop }
21
- else
22
- start_loop
23
- end
19
+ start_loop
24
20
  end
25
21
 
26
22
  def start_loop
@@ -43,6 +39,9 @@ module SolidQueue::Processes
43
39
  raise NotImplementedError
44
40
  end
45
41
 
42
+ def shutdown
43
+ end
44
+
46
45
  def with_polling_volume
47
46
  if SolidQueue.silence_polling? && ActiveRecord::Base.logger
48
47
  ActiveRecord::Base.logger.silence { yield }
@@ -5,7 +5,7 @@ module SolidQueue::Processes
5
5
  # Sets the procline ($0)
6
6
  # solid-queue-supervisor(0.1.0): <string>
7
7
  def procline(string)
8
- $0 = "solid-queue-#{self.class.name.demodulize.downcase}(#{SolidQueue::VERSION}): #{string}"
8
+ $0 = "solid-queue-#{self.class.name.demodulize.underscore.dasherize}(#{SolidQueue::VERSION}): #{string}"
9
9
  end
10
10
  end
11
11
  end
@@ -11,12 +11,16 @@ module SolidQueue::Processes
11
11
  after_shutdown :deregister
12
12
  end
13
13
 
14
+ def process_id
15
+ process&.id
16
+ end
17
+
14
18
  private
15
19
  attr_accessor :process
16
20
 
17
21
  def register
18
22
  @process = SolidQueue::Process.register \
19
- kind: self.class.name.demodulize,
23
+ kind: kind,
20
24
  pid: pid,
21
25
  hostname: hostname,
22
26
  supervisor: try(:supervisor),
@@ -36,6 +40,10 @@ module SolidQueue::Processes
36
40
  wrap_in_app_executor { heartbeat }
37
41
  end
38
42
 
43
+ @heartbeat_task.add_observer do |_, _, error|
44
+ handle_thread_error(error) if error
45
+ end
46
+
39
47
  @heartbeat_task.execute
40
48
  end
41
49
 
@@ -7,20 +7,32 @@ module SolidQueue::Processes
7
7
  attr_writer :mode
8
8
 
9
9
  def start
10
- @stopping = false
10
+ @stopped = false
11
11
 
12
12
  SolidQueue.instrument(:start_process, process: self) do
13
13
  run_callbacks(:boot) { boot }
14
14
  end
15
15
 
16
- run
16
+ if running_async?
17
+ @thread = create_thread { run }
18
+ else
19
+ run
20
+ end
17
21
  end
18
22
 
19
23
  def stop
20
- @stopping = true
24
+ @stopped = true
21
25
  @thread&.join
22
26
  end
23
27
 
28
+ def name
29
+ @name ||= [ kind.downcase, SecureRandom.hex(6) ].join("-")
30
+ end
31
+
32
+ def alive?
33
+ !running_async? || @thread.alive?
34
+ end
35
+
24
36
  private
25
37
  DEFAULT_MODE = :async
26
38
 
@@ -29,22 +41,22 @@ module SolidQueue::Processes
29
41
  end
30
42
 
31
43
  def boot
32
- if supervised?
44
+ if running_as_fork?
33
45
  register_signal_handlers
34
46
  set_procline
35
47
  end
36
48
  end
37
49
 
38
50
  def shutting_down?
39
- stopping? || supervisor_went_away? || finished?
51
+ stopped? || (running_as_fork? && supervisor_went_away?) || finished?
40
52
  end
41
53
 
42
54
  def run
43
55
  raise NotImplementedError
44
56
  end
45
57
 
46
- def stopping?
47
- @stopping
58
+ def stopped?
59
+ @stopped
48
60
  end
49
61
 
50
62
  def finished?
@@ -61,5 +73,24 @@ module SolidQueue::Processes
61
73
  def running_inline?
62
74
  mode.inline?
63
75
  end
76
+
77
+ def running_async?
78
+ mode.async?
79
+ end
80
+
81
+ def running_as_fork?
82
+ mode.fork?
83
+ end
84
+
85
+
86
+ def create_thread(&block)
87
+ Thread.new do
88
+ Thread.current.name = name
89
+ block.call
90
+ rescue Exception => exception
91
+ handle_thread_error(exception)
92
+ raise
93
+ end
94
+ end
64
95
  end
65
96
  end