que 0.11.3 → 2.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.github/workflows/tests.yml +51 -0
- data/.gitignore +2 -0
- data/.ruby-version +1 -0
- data/CHANGELOG.md +502 -97
- data/Dockerfile +20 -0
- data/LICENSE.txt +1 -1
- data/README.md +205 -59
- data/auto/dev +21 -0
- data/auto/pre-push-hook +30 -0
- data/auto/psql +9 -0
- data/auto/test +5 -0
- data/auto/test-postgres-14 +17 -0
- data/bin/que +8 -81
- data/docker-compose.yml +47 -0
- data/docs/README.md +881 -0
- data/lib/que/active_job/extensions.rb +114 -0
- data/lib/que/active_record/connection.rb +51 -0
- data/lib/que/active_record/model.rb +48 -0
- data/lib/que/command_line_interface.rb +259 -0
- data/lib/que/connection.rb +198 -0
- data/lib/que/connection_pool.rb +78 -0
- data/lib/que/job.rb +210 -103
- data/lib/que/job_buffer.rb +255 -0
- data/lib/que/job_methods.rb +176 -0
- data/lib/que/listener.rb +176 -0
- data/lib/que/locker.rb +507 -0
- data/lib/que/metajob.rb +47 -0
- data/lib/que/migrations/4/down.sql +48 -0
- data/lib/que/migrations/4/up.sql +267 -0
- data/lib/que/migrations/5/down.sql +73 -0
- data/lib/que/migrations/5/up.sql +76 -0
- data/lib/que/migrations/6/down.sql +8 -0
- data/lib/que/migrations/6/up.sql +8 -0
- data/lib/que/migrations/7/down.sql +5 -0
- data/lib/que/migrations/7/up.sql +13 -0
- data/lib/que/migrations.rb +37 -18
- data/lib/que/poller.rb +274 -0
- data/lib/que/rails/railtie.rb +12 -0
- data/lib/que/result_queue.rb +35 -0
- data/lib/que/sequel/model.rb +52 -0
- data/lib/que/utils/assertions.rb +62 -0
- data/lib/que/utils/constantization.rb +19 -0
- data/lib/que/utils/error_notification.rb +68 -0
- data/lib/que/utils/freeze.rb +20 -0
- data/lib/que/utils/introspection.rb +50 -0
- data/lib/que/utils/json_serialization.rb +21 -0
- data/lib/que/utils/logging.rb +79 -0
- data/lib/que/utils/middleware.rb +46 -0
- data/lib/que/utils/queue_management.rb +18 -0
- data/lib/que/utils/ruby2_keywords.rb +19 -0
- data/lib/que/utils/transactions.rb +34 -0
- data/lib/que/version.rb +5 -1
- data/lib/que/worker.rb +145 -149
- data/lib/que.rb +103 -159
- data/que.gemspec +17 -4
- data/scripts/docker-entrypoint +14 -0
- data/scripts/test +6 -0
- metadata +59 -95
- data/.rspec +0 -2
- data/.travis.yml +0 -17
- data/Gemfile +0 -24
- data/docs/advanced_setup.md +0 -106
- data/docs/customizing_que.md +0 -200
- data/docs/error_handling.md +0 -47
- data/docs/inspecting_the_queue.md +0 -114
- data/docs/logging.md +0 -50
- data/docs/managing_workers.md +0 -80
- data/docs/migrating.md +0 -30
- data/docs/multiple_queues.md +0 -27
- data/docs/shutting_down_safely.md +0 -7
- data/docs/using_plain_connections.md +0 -41
- data/docs/using_sequel.md +0 -31
- data/docs/writing_reliable_jobs.md +0 -117
- data/lib/generators/que/install_generator.rb +0 -24
- data/lib/generators/que/templates/add_que.rb +0 -13
- data/lib/que/adapters/active_record.rb +0 -54
- data/lib/que/adapters/base.rb +0 -127
- data/lib/que/adapters/connection_pool.rb +0 -16
- data/lib/que/adapters/pg.rb +0 -21
- data/lib/que/adapters/pond.rb +0 -16
- data/lib/que/adapters/sequel.rb +0 -20
- data/lib/que/railtie.rb +0 -16
- data/lib/que/rake_tasks.rb +0 -59
- data/lib/que/sql.rb +0 -152
- data/spec/adapters/active_record_spec.rb +0 -152
- data/spec/adapters/connection_pool_spec.rb +0 -22
- data/spec/adapters/pg_spec.rb +0 -41
- data/spec/adapters/pond_spec.rb +0 -22
- data/spec/adapters/sequel_spec.rb +0 -57
- data/spec/gemfiles/Gemfile1 +0 -18
- data/spec/gemfiles/Gemfile2 +0 -18
- data/spec/spec_helper.rb +0 -118
- data/spec/support/helpers.rb +0 -19
- data/spec/support/jobs.rb +0 -35
- data/spec/support/shared_examples/adapter.rb +0 -37
- data/spec/support/shared_examples/multi_threaded_adapter.rb +0 -46
- data/spec/travis.rb +0 -23
- data/spec/unit/connection_spec.rb +0 -14
- data/spec/unit/customization_spec.rb +0 -251
- data/spec/unit/enqueue_spec.rb +0 -245
- data/spec/unit/helper_spec.rb +0 -12
- data/spec/unit/logging_spec.rb +0 -101
- data/spec/unit/migrations_spec.rb +0 -84
- data/spec/unit/pool_spec.rb +0 -365
- data/spec/unit/run_spec.rb +0 -14
- data/spec/unit/states_spec.rb +0 -50
- data/spec/unit/stats_spec.rb +0 -46
- data/spec/unit/transaction_spec.rb +0 -36
- data/spec/unit/work_spec.rb +0 -407
- data/spec/unit/worker_spec.rb +0 -167
- data/tasks/benchmark.rb +0 -3
- data/tasks/rspec.rb +0 -14
- data/tasks/safe_shutdown.rb +0 -67
data/docs/advanced_setup.md
DELETED
@@ -1,106 +0,0 @@
|
|
1
|
-
## Advanced Setup
|
2
|
-
|
3
|
-
### Using ActiveRecord Without Rails
|
4
|
-
|
5
|
-
If you're using both Rails and ActiveRecord, the README describes how to get started with Que (which is pretty straightforward, since Que includes a Railtie that handles a lot of setup for you). Otherwise, you'll need to do some manual setup.
|
6
|
-
|
7
|
-
If you're using ActiveRecord outside of Rails, you'll need to tell Que to piggyback on its connection pool after you've connected to the database:
|
8
|
-
|
9
|
-
```ruby
|
10
|
-
ActiveRecord::Base.establish_connection(ENV['DATABASE_URL'])
|
11
|
-
|
12
|
-
require 'que'
|
13
|
-
Que.connection = ActiveRecord
|
14
|
-
```
|
15
|
-
|
16
|
-
Then you can queue jobs just as you would in Rails:
|
17
|
-
|
18
|
-
```ruby
|
19
|
-
ActiveRecord::Base.transaction do
|
20
|
-
@user = User.create(params[:user])
|
21
|
-
SendRegistrationEmail.enqueue :user_id => @user.id
|
22
|
-
end
|
23
|
-
```
|
24
|
-
|
25
|
-
There are other docs to read if you're using [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [plain Postgres connections](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md) (with no ORM at all) instead of ActiveRecord.
|
26
|
-
|
27
|
-
### Forking Servers
|
28
|
-
|
29
|
-
If you want to run a worker pool in your web process and you're using a forking webserver like Phusion Passenger (in smart spawning mode), Unicorn or Puma in some configurations, you'll want to set `Que.mode = :off` in your application configuration and only start up the worker pool in the child processes after the DB connection has been reestablished. So, for Puma:
|
30
|
-
|
31
|
-
```ruby
|
32
|
-
# config/puma.rb
|
33
|
-
on_worker_boot do
|
34
|
-
ActiveRecord::Base.establish_connection
|
35
|
-
|
36
|
-
Que.mode = :async
|
37
|
-
end
|
38
|
-
```
|
39
|
-
|
40
|
-
And for Unicorn:
|
41
|
-
|
42
|
-
```ruby
|
43
|
-
# config/unicorn.rb
|
44
|
-
after_fork do |server, worker|
|
45
|
-
ActiveRecord::Base.establish_connection
|
46
|
-
|
47
|
-
Que.mode = :async
|
48
|
-
end
|
49
|
-
```
|
50
|
-
|
51
|
-
And for Phusion Passenger:
|
52
|
-
|
53
|
-
```ruby
|
54
|
-
# config.ru
|
55
|
-
if defined?(PhusionPassenger)
|
56
|
-
PhusionPassenger.on_event(:starting_worker_process) do |forked|
|
57
|
-
if forked
|
58
|
-
Que.mode = :async
|
59
|
-
end
|
60
|
-
end
|
61
|
-
end
|
62
|
-
```
|
63
|
-
|
64
|
-
If there's other setup you want to do for workers, such as setting up the
|
65
|
-
configuration, you'll need to do that manually as well.
|
66
|
-
|
67
|
-
### Managing the Jobs Table
|
68
|
-
|
69
|
-
After you've connected Que to the database, you can manage the jobs table:
|
70
|
-
|
71
|
-
```ruby
|
72
|
-
# Create/update the jobs table to the latest schema version:
|
73
|
-
Que.migrate!
|
74
|
-
```
|
75
|
-
|
76
|
-
You'll want to migrate to a specific version if you're using migration files, to ensure that they work the same way even when you upgrade Que in the future:
|
77
|
-
|
78
|
-
```ruby
|
79
|
-
# Update the schema to version #3.
|
80
|
-
Que.migrate! :version => 3
|
81
|
-
|
82
|
-
# To reverse the migration, drop the jobs table entirely:
|
83
|
-
Que.migrate! :version => 0
|
84
|
-
```
|
85
|
-
|
86
|
-
There's also a helper method to clear all jobs from the jobs table:
|
87
|
-
|
88
|
-
```ruby
|
89
|
-
Que.clear!
|
90
|
-
```
|
91
|
-
|
92
|
-
### Other Setup
|
93
|
-
|
94
|
-
You'll need to set Que's mode manually:
|
95
|
-
|
96
|
-
```ruby
|
97
|
-
# Start the worker pool:
|
98
|
-
Que.mode = :async
|
99
|
-
|
100
|
-
# Or, when testing:
|
101
|
-
Que.mode = :sync
|
102
|
-
```
|
103
|
-
|
104
|
-
Be sure to read the docs on [managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md) for more information on using the worker pool.
|
105
|
-
|
106
|
-
You'll also want to set up [logging](https://github.com/chanks/que/blob/master/docs/logging.md) and an [error handler](https://github.com/chanks/que/blob/master/docs/error_handling.md) to track errors raised by jobs.
|
data/docs/customizing_que.md
DELETED
@@ -1,200 +0,0 @@
|
|
1
|
-
## Customizing Que
|
2
|
-
|
3
|
-
One of Que's goals to be easily extensible and hackable (and if anyone has any suggestions on ways to accomplish that, please [open an issue](https://github.com/chanks/que/issues)). This document is meant to demonstrate some of the ways Que can be used to accomplish different tasks that it's not already designed for.
|
4
|
-
|
5
|
-
Some of these features may be moved into core Que at some point, depending on how commonly useful they are.
|
6
|
-
|
7
|
-
### Recurring Jobs
|
8
|
-
|
9
|
-
Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
|
10
|
-
|
11
|
-
```ruby
|
12
|
-
class CronJob < Que::Job
|
13
|
-
# Default repetition interval in seconds. Can be overridden in
|
14
|
-
# subclasses. Can use 1.minute if using Rails.
|
15
|
-
INTERVAL = 60
|
16
|
-
|
17
|
-
attr_reader :start_at, :end_at, :run_again_at, :time_range
|
18
|
-
|
19
|
-
def _run
|
20
|
-
args = attrs[:args].first
|
21
|
-
@start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
|
22
|
-
@run_again_at = @end_at + self.class::INTERVAL
|
23
|
-
@time_range = @start_at...@end_at
|
24
|
-
|
25
|
-
super
|
26
|
-
|
27
|
-
args['start_at'] = @end_at.to_f
|
28
|
-
args['end_at'] = @run_again_at.to_f
|
29
|
-
self.class.enqueue(args, run_at: @run_again_at)
|
30
|
-
end
|
31
|
-
end
|
32
|
-
|
33
|
-
class MyCronJob < CronJob
|
34
|
-
INTERVAL = 3600
|
35
|
-
|
36
|
-
def run(args)
|
37
|
-
User.where(created_at: time_range).each { ... }
|
38
|
-
end
|
39
|
-
end
|
40
|
-
|
41
|
-
# To enqueue:
|
42
|
-
tf = Time.now
|
43
|
-
t0 = Time.now - 3600
|
44
|
-
MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
|
45
|
-
```
|
46
|
-
|
47
|
-
Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use job arguments to track start and end times. This lets us correct for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
|
48
|
-
|
49
|
-
This also compensates for clock drift. `Time.now` on one of your application servers may not match `Time.now` on another application server may not match `now()` on your database server. The best way to stay reliable is have a single authoritative source on what the current time is, and your best source for authoritative information is always your database (this is why Que uses Postgres' `now()` function when locking jobs, by the way).
|
50
|
-
|
51
|
-
Note also the use of the triple-dot range, which results in a query like `SELECT "users".* FROM "users" WHERE ("users"."created_at" >= '2014-01-08 10:00:00.000000' AND "users"."created_at" < '2014-01-08 11:00:00.000000')` instead of a BETWEEN condition. This ensures that a user created at 11:00 am exactly isn't processed twice, by the jobs starting at both 10 am and 11 am.
|
52
|
-
|
53
|
-
Finally, by passing both the start and end times for the period to be processed, and only using the interval to calculate the period for the following job, we make it easy to change the interval at which the job runs, without the risk of missing or double-processing any users.
|
54
|
-
|
55
|
-
### DelayedJob-style Jobs
|
56
|
-
|
57
|
-
DelayedJob offers a simple API for delaying methods to objects:
|
58
|
-
|
59
|
-
```ruby
|
60
|
-
@user.delay.activate!(@device)
|
61
|
-
```
|
62
|
-
|
63
|
-
The API is pleasant, but implementing it requires storing marshalled Ruby objects in the database, which is both inefficient and prone to bugs - for example, if you deploy an update that changes the name of an instance variable (a contained, internal change that might seem completely innocuous), the marshalled objects in the database will retain the old instance variable name and will behave unexpectedly when unmarshalled into the new Ruby code.
|
64
|
-
|
65
|
-
This is the danger of mixing the ephemeral state of a Ruby object in memory with the more permanent state of a database row. The advantage of Que's API is that, since your arguments are forced through a JSON serialization/deserialization process, it becomes your responsibility when designing a job class to establish an API for yourself (what the arguments to the job are and what they mean) that you will have to stick to in the future.
|
66
|
-
|
67
|
-
That said, if you want to queue jobs in the DelayedJob style, that can be done relatively easily:
|
68
|
-
|
69
|
-
```ruby
|
70
|
-
class Delayed < Que::Job
|
71
|
-
def run(receiver, method, args)
|
72
|
-
Marshal.load(receiver).send method, *Marshal.load(args)
|
73
|
-
end
|
74
|
-
end
|
75
|
-
|
76
|
-
class DelayedAction
|
77
|
-
def initialize(receiver)
|
78
|
-
@receiver = receiver
|
79
|
-
end
|
80
|
-
|
81
|
-
def method_missing(method, *args)
|
82
|
-
Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
|
83
|
-
end
|
84
|
-
end
|
85
|
-
|
86
|
-
class Object
|
87
|
-
def delay
|
88
|
-
DelayedAction.new(self)
|
89
|
-
end
|
90
|
-
end
|
91
|
-
```
|
92
|
-
|
93
|
-
You can replace Marshal with YAML if you like.
|
94
|
-
|
95
|
-
### QueueClassic-style Jobs
|
96
|
-
|
97
|
-
You may find it a hassle to keep an individual class file for each type of job. QueueClassic has a simpler design, wherein you simply give it a class method to call, like:
|
98
|
-
|
99
|
-
```ruby
|
100
|
-
QC.enqueue("Kernel.puts", "hello world")
|
101
|
-
```
|
102
|
-
|
103
|
-
You can mimic this style with Que by using a simple job class:
|
104
|
-
|
105
|
-
```ruby
|
106
|
-
class Command < Que::Job
|
107
|
-
def run(method, *args)
|
108
|
-
receiver, message = method.split('.')
|
109
|
-
Object.const_get(receiver).send(message, *args)
|
110
|
-
end
|
111
|
-
end
|
112
|
-
|
113
|
-
# Then:
|
114
|
-
|
115
|
-
Command.enqueue "Kernel.puts", "hello world"
|
116
|
-
```
|
117
|
-
|
118
|
-
### Retaining Finished Jobs
|
119
|
-
|
120
|
-
Que deletes jobs from the queue as they are worked, in order to keep the `que_jobs` table and index small and efficient. If you have a need to hold onto finished jobs, the recommended way to do this is to add a second table to hold them, and then insert them there as they are deleted from the queue. You can use Ruby's inheritance mechanics to do this cleanly:
|
121
|
-
|
122
|
-
```ruby
|
123
|
-
Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
|
124
|
-
# Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
|
125
|
-
|
126
|
-
class MyJobClass < Que::Job
|
127
|
-
def destroy
|
128
|
-
Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
|
129
|
-
super
|
130
|
-
end
|
131
|
-
end
|
132
|
-
```
|
133
|
-
|
134
|
-
Then just have your job classes inherit from MyJobClass instead of Que::Job. If you need to query the jobs table and you want to include both finished and unfinished jobs, you might use:
|
135
|
-
|
136
|
-
```ruby
|
137
|
-
Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
|
138
|
-
Que.execute "SELECT * FROM all_jobs"
|
139
|
-
```
|
140
|
-
|
141
|
-
Alternately, if you want a more foolproof solution and you're not scared of PostgreSQL, you can use a trigger:
|
142
|
-
|
143
|
-
```sql
|
144
|
-
CREATE FUNCTION please_save_my_job()
|
145
|
-
RETURNS trigger
|
146
|
-
LANGUAGE plpgsql
|
147
|
-
AS $$
|
148
|
-
BEGIN
|
149
|
-
INSERT INTO finished_jobs SELECT (OLD).*;
|
150
|
-
RETURN OLD;
|
151
|
-
END;
|
152
|
-
$$;
|
153
|
-
|
154
|
-
CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();
|
155
|
-
```
|
156
|
-
|
157
|
-
### Not Retrying Certain Failed Jobs
|
158
|
-
|
159
|
-
By default, when jobs fail, Que reschedules them to be retried later. If instead you'd like certain jobs to not be retried, and instead move them elsewhere to be examined later, you can accomplish that easily. First, we need a place for the failed jobs to be stored:
|
160
|
-
|
161
|
-
```sql
|
162
|
-
CREATE TABLE failed_jobs AS SELECT * FROM que_jobs LIMIT 0
|
163
|
-
```
|
164
|
-
|
165
|
-
Then, create a module that you can use in the jobs you don't want to retry:
|
166
|
-
|
167
|
-
```ruby
|
168
|
-
module SkipRetries
|
169
|
-
def run(*args)
|
170
|
-
super
|
171
|
-
rescue
|
172
|
-
sql = <<-SQL
|
173
|
-
WITH failed AS (
|
174
|
-
DELETE
|
175
|
-
FROM que_jobs
|
176
|
-
WHERE queue = $1::text
|
177
|
-
AND priority = $2::smallint
|
178
|
-
AND run_at = $3::timestamptz
|
179
|
-
AND job_id = $4::bigint
|
180
|
-
RETURNING *
|
181
|
-
)
|
182
|
-
INSERT INTO failed_jobs
|
183
|
-
SELECT * FROM failed;
|
184
|
-
SQL
|
185
|
-
|
186
|
-
Que.execute sql, @attrs.values_at(:queue, :priority, :run_at, :job_id)
|
187
|
-
|
188
|
-
raise # Reraises caught error.
|
189
|
-
end
|
190
|
-
end
|
191
|
-
|
192
|
-
class RunOnceJob < Que::Job
|
193
|
-
prepend SkipRetries
|
194
|
-
|
195
|
-
def run(*args)
|
196
|
-
# Do something - if this job runs an error it'll be moved to the
|
197
|
-
# failed_jobs table and not retried.
|
198
|
-
end
|
199
|
-
end
|
200
|
-
```
|
data/docs/error_handling.md
DELETED
@@ -1,47 +0,0 @@
|
|
1
|
-
## Error Handling
|
2
|
-
|
3
|
-
If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
|
4
|
-
|
5
|
-
If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's. Alternately, you can define your own retry logic by setting an interval to delay each time, or a callable that accepts the number of failures and returns an interval:
|
6
|
-
|
7
|
-
```ruby
|
8
|
-
class MyJob < Que::Job
|
9
|
-
# Just retry a failed job every 5 seconds:
|
10
|
-
@retry_interval = 5
|
11
|
-
|
12
|
-
# Always retry this job immediately (not recommended, or transient
|
13
|
-
# errors will spam your error reporting):
|
14
|
-
@retry_interval = 0
|
15
|
-
|
16
|
-
# Increase the delay by 30 seconds every time this job fails:
|
17
|
-
@retry_interval = proc { |count| count * 30 }
|
18
|
-
end
|
19
|
-
```
|
20
|
-
|
21
|
-
Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
|
22
|
-
|
23
|
-
If you're using an error notification system (highly recommended, of course), you can hook Que into it by setting a callable as the error handler:
|
24
|
-
|
25
|
-
```ruby
|
26
|
-
Que.error_handler = proc do |error, job|
|
27
|
-
# Do whatever you want with the error object or job row here.
|
28
|
-
|
29
|
-
# Note that the job passed is not the actual job object, but the hash
|
30
|
-
# representing the job row in the database, which looks like:
|
31
|
-
|
32
|
-
# {
|
33
|
-
# "queue" => "my_queue",
|
34
|
-
# "priority" => 100,
|
35
|
-
# "run_at" => 2015-03-06 11:07:08 -0500,
|
36
|
-
# "job_id" => 65,
|
37
|
-
# "job_class" => "MyJob",
|
38
|
-
# "args" => ['argument', 78],
|
39
|
-
# "error_count" => 0
|
40
|
-
# }
|
41
|
-
|
42
|
-
# This is done because the job may not have been able to be deserialized
|
43
|
-
# properly, if the name of the job class was changed or the job is being
|
44
|
-
# retrieved and worked by the wrong app. The job argument may also be
|
45
|
-
# nil, if there was a connection failure or something similar.
|
46
|
-
end
|
47
|
-
```
|
@@ -1,114 +0,0 @@
|
|
1
|
-
## Inspecting the Queue
|
2
|
-
|
3
|
-
In order to remain simple and compatible with any ORM (or no ORM at all), Que is really just a very thin wrapper around some raw SQL. There are two methods available that query the jobs table and Postgres' system catalogs to retrieve information on the current state of the queue:
|
4
|
-
|
5
|
-
### Job Stats
|
6
|
-
|
7
|
-
You can call `Que.job_stats` to return some aggregate data on the types of jobs currently in the queue. Example output:
|
8
|
-
|
9
|
-
```ruby
|
10
|
-
[
|
11
|
-
{
|
12
|
-
"job_class"=>"ChargeCreditCard",
|
13
|
-
"count"=>"10",
|
14
|
-
"count_working"=>"4",
|
15
|
-
"count_errored"=>"2",
|
16
|
-
"highest_error_count"=>"5",
|
17
|
-
"oldest_run_at"=>"2014-01-04 21:24:55.817129+00"
|
18
|
-
},
|
19
|
-
{
|
20
|
-
"job_class"=>"SendRegistrationEmail",
|
21
|
-
"count"=>"8",
|
22
|
-
"count_working"=>"0",
|
23
|
-
"count_errored"=>"0",
|
24
|
-
"highest_error_count"=>"0",
|
25
|
-
"oldest_run_at"=>"2014-01-04 22:24:55.81532+00"
|
26
|
-
}
|
27
|
-
]
|
28
|
-
```
|
29
|
-
|
30
|
-
This tells you that, for instance, there are ten ChargeCreditCard jobs in the queue, four of which are currently being worked, and two of which have experienced errors. One of them has started to process but experienced an error five times. The oldest_run_at is helpful for determining how long jobs have been sitting around, if you have backlog.
|
31
|
-
|
32
|
-
### Worker States
|
33
|
-
|
34
|
-
You can call `Que.worker_states` to return some information on every worker touching the queue (not just those in the current process). Example output:
|
35
|
-
|
36
|
-
```ruby
|
37
|
-
[
|
38
|
-
{
|
39
|
-
"priority"=>"2",
|
40
|
-
"run_at"=>"2014-01-04 22:35:55.772324+00",
|
41
|
-
"job_id"=>"4592",
|
42
|
-
"job_class"=>"ChargeCreditCard",
|
43
|
-
"args"=>"[345,56]",
|
44
|
-
"error_count"=>"0",
|
45
|
-
"last_error"=>nil,
|
46
|
-
"pg_backend_pid"=>"1175",
|
47
|
-
"pg_state"=>"idle",
|
48
|
-
"pg_state_changed_at"=>"2014-01-04 22:35:55.777785+00",
|
49
|
-
"pg_last_query"=>"SELECT * FROM users",
|
50
|
-
"pg_last_query_started_at"=>"2014-01-04 22:35:55.777519+00",
|
51
|
-
"pg_transaction_started_at"=>nil,
|
52
|
-
"pg_waiting_on_lock"=>"f"
|
53
|
-
}
|
54
|
-
]
|
55
|
-
```
|
56
|
-
|
57
|
-
In this case, there is only one worker currently working the queue. The first seven fields are the attributes of the job it is currently running. The next seven fields are information about that worker's Postgres connection, and are taken from `pg_stat_activity` - see [Postgres' documentation](http://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-ACTIVITY-VIEW) for more information on interpreting these fields.
|
58
|
-
|
59
|
-
* `pg_backend_pid` - The pid of the Postgres process serving this worker. This is useful if you wanted to kill that worker's connection, for example, by running "SELECT pg_terminate_backend(1175)". This would free up the job to be attempted by another worker.
|
60
|
-
* `pg_state` - The state of the Postgres backend. It may be "active" if the worker is currently running a query or "idle"/"idle in transaction" if it is not. It may also be in one of a few other less common states.
|
61
|
-
* `pg_state_changed_at` - The timestamp for when the backend's state was last changed. If the backend is idle, this would reflect the time that the last query finished.
|
62
|
-
* `pg_last_query` - The text of the current or most recent query that the worker sent to the database.
|
63
|
-
* `pg_last_query_started_at` - The timestamp for when the last query began to run.
|
64
|
-
* `pg_transaction_started_at` - The timestamp for when the worker's current transaction (if any) began.
|
65
|
-
* `pg_waiting_on_lock` - Whether or not the worker is waiting for a lock in Postgres to be released.
|
66
|
-
|
67
|
-
### Custom Queries
|
68
|
-
|
69
|
-
If you want to query the jobs table yourself to see what's been queued or to check the state of various jobs, you can always use Que to execute whatever SQL you want:
|
70
|
-
|
71
|
-
```ruby
|
72
|
-
Que.execute("select count(*) from que_jobs") #=> [{"count"=>"492"}]
|
73
|
-
```
|
74
|
-
|
75
|
-
If you want to use ActiveRecord's features when querying, you can define your own model around Que's job table:
|
76
|
-
|
77
|
-
```ruby
|
78
|
-
class QueJob < ActiveRecord::Base
|
79
|
-
end
|
80
|
-
|
81
|
-
# Or:
|
82
|
-
|
83
|
-
class MyJob < ActiveRecord::Base
|
84
|
-
self.table_name = :que_jobs
|
85
|
-
end
|
86
|
-
```
|
87
|
-
|
88
|
-
Then you can query just as you would with any other model. Since the jobs table has a composite primary key, however, you probably won't be able to update or destroy jobs this way, though.
|
89
|
-
|
90
|
-
If you're using Sequel, you can use the same technique:
|
91
|
-
|
92
|
-
```ruby
|
93
|
-
class QueJob < Sequel::Model
|
94
|
-
end
|
95
|
-
|
96
|
-
# Or:
|
97
|
-
|
98
|
-
class MyJob < Sequel::Model(:que_jobs)
|
99
|
-
end
|
100
|
-
```
|
101
|
-
|
102
|
-
And note that Sequel *does* support composite primary keys:
|
103
|
-
|
104
|
-
```ruby
|
105
|
-
job = QueJob.where(:job_class => "ChargeCreditCard").first
|
106
|
-
job.priority = 1
|
107
|
-
job.save
|
108
|
-
```
|
109
|
-
|
110
|
-
Or, you can just use Sequel's dataset methods:
|
111
|
-
|
112
|
-
```ruby
|
113
|
-
DB[:que_jobs].where{priority > 3}.all
|
114
|
-
```
|
data/docs/logging.md
DELETED
@@ -1,50 +0,0 @@
|
|
1
|
-
## Logging
|
2
|
-
|
3
|
-
By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running via the `que` executable). So, your logs will look something like:
|
4
|
-
|
5
|
-
```
|
6
|
-
I, [2014-01-12T05:07:31.094201 #4687] INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
|
7
|
-
```
|
8
|
-
|
9
|
-
Of course you can have it log wherever you like:
|
10
|
-
|
11
|
-
```ruby
|
12
|
-
Que.logger = Logger.new(...)
|
13
|
-
```
|
14
|
-
|
15
|
-
You can use Que's logger in your jobs anywhere you like:
|
16
|
-
|
17
|
-
```ruby
|
18
|
-
class MyJob
|
19
|
-
def run
|
20
|
-
Que.log :my_output => "my string"
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
#=> I, [2014-01-12T05:13:11.006776 #4914] INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
|
25
|
-
```
|
26
|
-
|
27
|
-
Que will always add a 'lib' key, so you can easily filter its output from that of other sources, and the object_id of the thread that emitted the log, so you can follow the actions of a particular worker if you wish. You can also pass a :level key to set the level of the output:
|
28
|
-
|
29
|
-
```ruby
|
30
|
-
Que.log :level => :debug, :my_output => 'my string'
|
31
|
-
#=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
|
32
|
-
```
|
33
|
-
|
34
|
-
If you don't like JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
|
35
|
-
|
36
|
-
```ruby
|
37
|
-
Que.log_formatter = proc do |data|
|
38
|
-
"Thread number #{data[:thread]} experienced a #{data[:event]}"
|
39
|
-
end
|
40
|
-
```
|
41
|
-
|
42
|
-
If the log formatter returns nil or false, a nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
|
43
|
-
|
44
|
-
```ruby
|
45
|
-
Que.log_formatter = proc do |data|
|
46
|
-
if [:job_worked, :job_unavailable].include?(data[:event])
|
47
|
-
JSON.dump(data)
|
48
|
-
end
|
49
|
-
end
|
50
|
-
```
|
data/docs/managing_workers.md
DELETED
@@ -1,80 +0,0 @@
|
|
1
|
-
## Managing Workers
|
2
|
-
|
3
|
-
Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
|
4
|
-
|
5
|
-
When the worker pool is active (when you set `Que.mode = :async`), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
|
6
|
-
|
7
|
-
Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
|
8
|
-
|
9
|
-
JRuby and Rubinius, on the other hand, have no global interpreter lock, and so can make use of multiple CPU cores - you could potentially set the number of workers very high for them. You should experiment to find the best setting for your use case.
|
10
|
-
|
11
|
-
You can change the number of workers in the pool whenever you like by setting the `worker_count` option:
|
12
|
-
|
13
|
-
```ruby
|
14
|
-
Que.worker_count = 8
|
15
|
-
```
|
16
|
-
|
17
|
-
### Working Jobs Via Executable
|
18
|
-
|
19
|
-
If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
|
20
|
-
|
21
|
-
```shell
|
22
|
-
# Run a pool of 4 workers:
|
23
|
-
que
|
24
|
-
|
25
|
-
# Or configure the number of workers:
|
26
|
-
que --worker-count 8
|
27
|
-
```
|
28
|
-
|
29
|
-
See `que -h` for a list of command-line options.
|
30
|
-
|
31
|
-
### Thread-Unsafe Application Code
|
32
|
-
|
33
|
-
If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to turn the worker pool off by default:
|
34
|
-
|
35
|
-
```ruby
|
36
|
-
Que.mode = :off
|
37
|
-
```
|
38
|
-
|
39
|
-
This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate process, like so:
|
40
|
-
|
41
|
-
```shell
|
42
|
-
que --worker-count 1
|
43
|
-
```
|
44
|
-
|
45
|
-
### The Wake Interval
|
46
|
-
|
47
|
-
If a worker checks the job queue and finds no jobs ready for it to work, it will fall asleep. In order to make sure that newly-available jobs don't go unworked, a worker is awoken every so often to check for available work. By default, this happens every five seconds, but you can make it happen more or less often by setting a custom wake_interval:
|
48
|
-
|
49
|
-
```ruby
|
50
|
-
Que.wake_interval = 2 # In Rails, 2.seconds also works fine.
|
51
|
-
```
|
52
|
-
|
53
|
-
You can also choose to never let workers wake up on their own:
|
54
|
-
|
55
|
-
```ruby
|
56
|
-
# Never wake up any workers:
|
57
|
-
Que.wake_interval = nil
|
58
|
-
```
|
59
|
-
|
60
|
-
If you do this, though, you'll need to wake workers manually.
|
61
|
-
|
62
|
-
### Manually Waking Workers
|
63
|
-
|
64
|
-
Regardless of the `wake_interval` setting, you can always wake workers manually:
|
65
|
-
|
66
|
-
```ruby
|
67
|
-
# Wake up a single worker to check the queue for work:
|
68
|
-
Que.wake!
|
69
|
-
|
70
|
-
# Wake up all workers in this process to check for work:
|
71
|
-
Que.wake_all!
|
72
|
-
```
|
73
|
-
|
74
|
-
`Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.
|
75
|
-
|
76
|
-
### Connection Pool Size
|
77
|
-
|
78
|
-
For the job locking system to work properly, each worker thread needs to reserve a database connection from the connection pool for the period of time between when it locks a job and when it releases that lock (which won't happen until the job has been finished and deleted from the queue).
|
79
|
-
|
80
|
-
So, for example, if you're running 6 workers via the executable, you'll want to make sure that whatever connection pool Que is using (usually ActiveRecord's) has a maximum size of at least 6. If you're running those workers in a web process, you'll want the size to be at least 6 plus however many connections you expect your application to need for serving web requests (which may only be one if you're using Rails in single-threaded mode, or many more if you're running a threaded web server like Puma).
|
data/docs/migrating.md
DELETED
@@ -1,30 +0,0 @@
|
|
1
|
-
## Migrating
|
2
|
-
|
3
|
-
Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
|
4
|
-
|
5
|
-
```ruby
|
6
|
-
class UpdateQue < ActiveRecord::Migration
|
7
|
-
def self.up
|
8
|
-
Que.migrate! :version => 3
|
9
|
-
end
|
10
|
-
|
11
|
-
def self.down
|
12
|
-
Que.migrate! :version => 2
|
13
|
-
end
|
14
|
-
end
|
15
|
-
```
|
16
|
-
|
17
|
-
This will make sure that your database schema stays consistent with your codebase. If you're looking for something quicker and dirtier, you can always manually migrate in a console session:
|
18
|
-
|
19
|
-
```ruby
|
20
|
-
# Change schema to version 3.
|
21
|
-
Que.migrate! :version => 3
|
22
|
-
|
23
|
-
# Update to whatever the latest schema version is.
|
24
|
-
Que.migrate!
|
25
|
-
|
26
|
-
# Check your current schema version.
|
27
|
-
Que.db_version #=> 3
|
28
|
-
```
|
29
|
-
|
30
|
-
Note that you can remove Que from your database completely by migrating to version 0.
|
data/docs/multiple_queues.md
DELETED
@@ -1,27 +0,0 @@
|
|
1
|
-
## Multiple Queues
|
2
|
-
|
3
|
-
Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
|
4
|
-
|
5
|
-
```shell
|
6
|
-
que --queue-name credit_cards
|
7
|
-
```
|
8
|
-
|
9
|
-
Then you can set jobs to be enqueued in that queue specifically:
|
10
|
-
|
11
|
-
```ruby
|
12
|
-
ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
|
13
|
-
|
14
|
-
# Or:
|
15
|
-
|
16
|
-
class ProcessCreditCard < Que::Job
|
17
|
-
# Set a default queue for this job class; this can be overridden by
|
18
|
-
# passing the :queue parameter to enqueue like above.
|
19
|
-
@queue = 'credit_cards'
|
20
|
-
end
|
21
|
-
```
|
22
|
-
|
23
|
-
In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
|
24
|
-
|
25
|
-
```ruby
|
26
|
-
Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'
|
27
|
-
```
|
@@ -1,7 +0,0 @@
|
|
1
|
-
## Shutting Down Safely
|
2
|
-
|
3
|
-
To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
|
4
|
-
|
5
|
-
To prevent this, Que will block a Ruby process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
|
6
|
-
|
7
|
-
So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.
|