que 0.11.3 → 2.2.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (114) hide show
  1. checksums.yaml +5 -5
  2. data/.github/workflows/tests.yml +51 -0
  3. data/.gitignore +2 -0
  4. data/.ruby-version +1 -0
  5. data/CHANGELOG.md +502 -97
  6. data/Dockerfile +20 -0
  7. data/LICENSE.txt +1 -1
  8. data/README.md +205 -59
  9. data/auto/dev +21 -0
  10. data/auto/pre-push-hook +30 -0
  11. data/auto/psql +9 -0
  12. data/auto/test +5 -0
  13. data/auto/test-postgres-14 +17 -0
  14. data/bin/que +8 -81
  15. data/docker-compose.yml +47 -0
  16. data/docs/README.md +881 -0
  17. data/lib/que/active_job/extensions.rb +114 -0
  18. data/lib/que/active_record/connection.rb +51 -0
  19. data/lib/que/active_record/model.rb +48 -0
  20. data/lib/que/command_line_interface.rb +259 -0
  21. data/lib/que/connection.rb +198 -0
  22. data/lib/que/connection_pool.rb +78 -0
  23. data/lib/que/job.rb +210 -103
  24. data/lib/que/job_buffer.rb +255 -0
  25. data/lib/que/job_methods.rb +176 -0
  26. data/lib/que/listener.rb +176 -0
  27. data/lib/que/locker.rb +507 -0
  28. data/lib/que/metajob.rb +47 -0
  29. data/lib/que/migrations/4/down.sql +48 -0
  30. data/lib/que/migrations/4/up.sql +267 -0
  31. data/lib/que/migrations/5/down.sql +73 -0
  32. data/lib/que/migrations/5/up.sql +76 -0
  33. data/lib/que/migrations/6/down.sql +8 -0
  34. data/lib/que/migrations/6/up.sql +8 -0
  35. data/lib/que/migrations/7/down.sql +5 -0
  36. data/lib/que/migrations/7/up.sql +13 -0
  37. data/lib/que/migrations.rb +37 -18
  38. data/lib/que/poller.rb +274 -0
  39. data/lib/que/rails/railtie.rb +12 -0
  40. data/lib/que/result_queue.rb +35 -0
  41. data/lib/que/sequel/model.rb +52 -0
  42. data/lib/que/utils/assertions.rb +62 -0
  43. data/lib/que/utils/constantization.rb +19 -0
  44. data/lib/que/utils/error_notification.rb +68 -0
  45. data/lib/que/utils/freeze.rb +20 -0
  46. data/lib/que/utils/introspection.rb +50 -0
  47. data/lib/que/utils/json_serialization.rb +21 -0
  48. data/lib/que/utils/logging.rb +79 -0
  49. data/lib/que/utils/middleware.rb +46 -0
  50. data/lib/que/utils/queue_management.rb +18 -0
  51. data/lib/que/utils/ruby2_keywords.rb +19 -0
  52. data/lib/que/utils/transactions.rb +34 -0
  53. data/lib/que/version.rb +5 -1
  54. data/lib/que/worker.rb +145 -149
  55. data/lib/que.rb +103 -159
  56. data/que.gemspec +17 -4
  57. data/scripts/docker-entrypoint +14 -0
  58. data/scripts/test +6 -0
  59. metadata +59 -95
  60. data/.rspec +0 -2
  61. data/.travis.yml +0 -17
  62. data/Gemfile +0 -24
  63. data/docs/advanced_setup.md +0 -106
  64. data/docs/customizing_que.md +0 -200
  65. data/docs/error_handling.md +0 -47
  66. data/docs/inspecting_the_queue.md +0 -114
  67. data/docs/logging.md +0 -50
  68. data/docs/managing_workers.md +0 -80
  69. data/docs/migrating.md +0 -30
  70. data/docs/multiple_queues.md +0 -27
  71. data/docs/shutting_down_safely.md +0 -7
  72. data/docs/using_plain_connections.md +0 -41
  73. data/docs/using_sequel.md +0 -31
  74. data/docs/writing_reliable_jobs.md +0 -117
  75. data/lib/generators/que/install_generator.rb +0 -24
  76. data/lib/generators/que/templates/add_que.rb +0 -13
  77. data/lib/que/adapters/active_record.rb +0 -54
  78. data/lib/que/adapters/base.rb +0 -127
  79. data/lib/que/adapters/connection_pool.rb +0 -16
  80. data/lib/que/adapters/pg.rb +0 -21
  81. data/lib/que/adapters/pond.rb +0 -16
  82. data/lib/que/adapters/sequel.rb +0 -20
  83. data/lib/que/railtie.rb +0 -16
  84. data/lib/que/rake_tasks.rb +0 -59
  85. data/lib/que/sql.rb +0 -152
  86. data/spec/adapters/active_record_spec.rb +0 -152
  87. data/spec/adapters/connection_pool_spec.rb +0 -22
  88. data/spec/adapters/pg_spec.rb +0 -41
  89. data/spec/adapters/pond_spec.rb +0 -22
  90. data/spec/adapters/sequel_spec.rb +0 -57
  91. data/spec/gemfiles/Gemfile1 +0 -18
  92. data/spec/gemfiles/Gemfile2 +0 -18
  93. data/spec/spec_helper.rb +0 -118
  94. data/spec/support/helpers.rb +0 -19
  95. data/spec/support/jobs.rb +0 -35
  96. data/spec/support/shared_examples/adapter.rb +0 -37
  97. data/spec/support/shared_examples/multi_threaded_adapter.rb +0 -46
  98. data/spec/travis.rb +0 -23
  99. data/spec/unit/connection_spec.rb +0 -14
  100. data/spec/unit/customization_spec.rb +0 -251
  101. data/spec/unit/enqueue_spec.rb +0 -245
  102. data/spec/unit/helper_spec.rb +0 -12
  103. data/spec/unit/logging_spec.rb +0 -101
  104. data/spec/unit/migrations_spec.rb +0 -84
  105. data/spec/unit/pool_spec.rb +0 -365
  106. data/spec/unit/run_spec.rb +0 -14
  107. data/spec/unit/states_spec.rb +0 -50
  108. data/spec/unit/stats_spec.rb +0 -46
  109. data/spec/unit/transaction_spec.rb +0 -36
  110. data/spec/unit/work_spec.rb +0 -407
  111. data/spec/unit/worker_spec.rb +0 -167
  112. data/tasks/benchmark.rb +0 -3
  113. data/tasks/rspec.rb +0 -14
  114. data/tasks/safe_shutdown.rb +0 -67
@@ -1,106 +0,0 @@
1
- ## Advanced Setup
2
-
3
- ### Using ActiveRecord Without Rails
4
-
5
- If you're using both Rails and ActiveRecord, the README describes how to get started with Que (which is pretty straightforward, since Que includes a Railtie that handles a lot of setup for you). Otherwise, you'll need to do some manual setup.
6
-
7
- If you're using ActiveRecord outside of Rails, you'll need to tell Que to piggyback on its connection pool after you've connected to the database:
8
-
9
- ```ruby
10
- ActiveRecord::Base.establish_connection(ENV['DATABASE_URL'])
11
-
12
- require 'que'
13
- Que.connection = ActiveRecord
14
- ```
15
-
16
- Then you can queue jobs just as you would in Rails:
17
-
18
- ```ruby
19
- ActiveRecord::Base.transaction do
20
- @user = User.create(params[:user])
21
- SendRegistrationEmail.enqueue :user_id => @user.id
22
- end
23
- ```
24
-
25
- There are other docs to read if you're using [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [plain Postgres connections](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md) (with no ORM at all) instead of ActiveRecord.
26
-
27
- ### Forking Servers
28
-
29
- If you want to run a worker pool in your web process and you're using a forking webserver like Phusion Passenger (in smart spawning mode), Unicorn or Puma in some configurations, you'll want to set `Que.mode = :off` in your application configuration and only start up the worker pool in the child processes after the DB connection has been reestablished. So, for Puma:
30
-
31
- ```ruby
32
- # config/puma.rb
33
- on_worker_boot do
34
- ActiveRecord::Base.establish_connection
35
-
36
- Que.mode = :async
37
- end
38
- ```
39
-
40
- And for Unicorn:
41
-
42
- ```ruby
43
- # config/unicorn.rb
44
- after_fork do |server, worker|
45
- ActiveRecord::Base.establish_connection
46
-
47
- Que.mode = :async
48
- end
49
- ```
50
-
51
- And for Phusion Passenger:
52
-
53
- ```ruby
54
- # config.ru
55
- if defined?(PhusionPassenger)
56
- PhusionPassenger.on_event(:starting_worker_process) do |forked|
57
- if forked
58
- Que.mode = :async
59
- end
60
- end
61
- end
62
- ```
63
-
64
- If there's other setup you want to do for workers, such as setting up the
65
- configuration, you'll need to do that manually as well.
66
-
67
- ### Managing the Jobs Table
68
-
69
- After you've connected Que to the database, you can manage the jobs table:
70
-
71
- ```ruby
72
- # Create/update the jobs table to the latest schema version:
73
- Que.migrate!
74
- ```
75
-
76
- You'll want to migrate to a specific version if you're using migration files, to ensure that they work the same way even when you upgrade Que in the future:
77
-
78
- ```ruby
79
- # Update the schema to version #3.
80
- Que.migrate! :version => 3
81
-
82
- # To reverse the migration, drop the jobs table entirely:
83
- Que.migrate! :version => 0
84
- ```
85
-
86
- There's also a helper method to clear all jobs from the jobs table:
87
-
88
- ```ruby
89
- Que.clear!
90
- ```
91
-
92
- ### Other Setup
93
-
94
- You'll need to set Que's mode manually:
95
-
96
- ```ruby
97
- # Start the worker pool:
98
- Que.mode = :async
99
-
100
- # Or, when testing:
101
- Que.mode = :sync
102
- ```
103
-
104
- Be sure to read the docs on [managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md) for more information on using the worker pool.
105
-
106
- You'll also want to set up [logging](https://github.com/chanks/que/blob/master/docs/logging.md) and an [error handler](https://github.com/chanks/que/blob/master/docs/error_handling.md) to track errors raised by jobs.
@@ -1,200 +0,0 @@
1
- ## Customizing Que
2
-
3
- One of Que's goals to be easily extensible and hackable (and if anyone has any suggestions on ways to accomplish that, please [open an issue](https://github.com/chanks/que/issues)). This document is meant to demonstrate some of the ways Que can be used to accomplish different tasks that it's not already designed for.
4
-
5
- Some of these features may be moved into core Que at some point, depending on how commonly useful they are.
6
-
7
- ### Recurring Jobs
8
-
9
- Que's support for scheduling jobs makes it easy to implement reliable recurring jobs. For example, suppose you want to run a job every hour that processes the users created in that time:
10
-
11
- ```ruby
12
- class CronJob < Que::Job
13
- # Default repetition interval in seconds. Can be overridden in
14
- # subclasses. Can use 1.minute if using Rails.
15
- INTERVAL = 60
16
-
17
- attr_reader :start_at, :end_at, :run_again_at, :time_range
18
-
19
- def _run
20
- args = attrs[:args].first
21
- @start_at, @end_at = Time.at(args.delete('start_at')), Time.at(args.delete('end_at'))
22
- @run_again_at = @end_at + self.class::INTERVAL
23
- @time_range = @start_at...@end_at
24
-
25
- super
26
-
27
- args['start_at'] = @end_at.to_f
28
- args['end_at'] = @run_again_at.to_f
29
- self.class.enqueue(args, run_at: @run_again_at)
30
- end
31
- end
32
-
33
- class MyCronJob < CronJob
34
- INTERVAL = 3600
35
-
36
- def run(args)
37
- User.where(created_at: time_range).each { ... }
38
- end
39
- end
40
-
41
- # To enqueue:
42
- tf = Time.now
43
- t0 = Time.now - 3600
44
- MyCronJob.enqueue :start_at => t0.to_f, :end_at => tf.to_f
45
- ```
46
-
47
- Note that instead of using Time.now in our database query, and requeueing the job at 1.hour.from_now, we use job arguments to track start and end times. This lets us correct for delays in running the job. Suppose that there's a backlog of priority jobs, or that the worker briefly goes down, and this job, which was supposed to run at 11:00 a.m. isn't run until 11:05 a.m. A lazier implementation would look for users created after 1.hour.ago, and miss those that signed up between 10:00 a.m. and 10:05 a.m.
48
-
49
- This also compensates for clock drift. `Time.now` on one of your application servers may not match `Time.now` on another application server may not match `now()` on your database server. The best way to stay reliable is have a single authoritative source on what the current time is, and your best source for authoritative information is always your database (this is why Que uses Postgres' `now()` function when locking jobs, by the way).
50
-
51
- Note also the use of the triple-dot range, which results in a query like `SELECT "users".* FROM "users" WHERE ("users"."created_at" >= '2014-01-08 10:00:00.000000' AND "users"."created_at" < '2014-01-08 11:00:00.000000')` instead of a BETWEEN condition. This ensures that a user created at 11:00 am exactly isn't processed twice, by the jobs starting at both 10 am and 11 am.
52
-
53
- Finally, by passing both the start and end times for the period to be processed, and only using the interval to calculate the period for the following job, we make it easy to change the interval at which the job runs, without the risk of missing or double-processing any users.
54
-
55
- ### DelayedJob-style Jobs
56
-
57
- DelayedJob offers a simple API for delaying methods to objects:
58
-
59
- ```ruby
60
- @user.delay.activate!(@device)
61
- ```
62
-
63
- The API is pleasant, but implementing it requires storing marshalled Ruby objects in the database, which is both inefficient and prone to bugs - for example, if you deploy an update that changes the name of an instance variable (a contained, internal change that might seem completely innocuous), the marshalled objects in the database will retain the old instance variable name and will behave unexpectedly when unmarshalled into the new Ruby code.
64
-
65
- This is the danger of mixing the ephemeral state of a Ruby object in memory with the more permanent state of a database row. The advantage of Que's API is that, since your arguments are forced through a JSON serialization/deserialization process, it becomes your responsibility when designing a job class to establish an API for yourself (what the arguments to the job are and what they mean) that you will have to stick to in the future.
66
-
67
- That said, if you want to queue jobs in the DelayedJob style, that can be done relatively easily:
68
-
69
- ```ruby
70
- class Delayed < Que::Job
71
- def run(receiver, method, args)
72
- Marshal.load(receiver).send method, *Marshal.load(args)
73
- end
74
- end
75
-
76
- class DelayedAction
77
- def initialize(receiver)
78
- @receiver = receiver
79
- end
80
-
81
- def method_missing(method, *args)
82
- Delayed.enqueue Marshal.dump(@receiver), method, Marshal.dump(args)
83
- end
84
- end
85
-
86
- class Object
87
- def delay
88
- DelayedAction.new(self)
89
- end
90
- end
91
- ```
92
-
93
- You can replace Marshal with YAML if you like.
94
-
95
- ### QueueClassic-style Jobs
96
-
97
- You may find it a hassle to keep an individual class file for each type of job. QueueClassic has a simpler design, wherein you simply give it a class method to call, like:
98
-
99
- ```ruby
100
- QC.enqueue("Kernel.puts", "hello world")
101
- ```
102
-
103
- You can mimic this style with Que by using a simple job class:
104
-
105
- ```ruby
106
- class Command < Que::Job
107
- def run(method, *args)
108
- receiver, message = method.split('.')
109
- Object.const_get(receiver).send(message, *args)
110
- end
111
- end
112
-
113
- # Then:
114
-
115
- Command.enqueue "Kernel.puts", "hello world"
116
- ```
117
-
118
- ### Retaining Finished Jobs
119
-
120
- Que deletes jobs from the queue as they are worked, in order to keep the `que_jobs` table and index small and efficient. If you have a need to hold onto finished jobs, the recommended way to do this is to add a second table to hold them, and then insert them there as they are deleted from the queue. You can use Ruby's inheritance mechanics to do this cleanly:
121
-
122
- ```ruby
123
- Que.execute "CREATE TABLE finished_jobs AS SELECT * FROM que_jobs LIMIT 0"
124
- # Or, better, use a proper CREATE TABLE with not-null constraints, and add whatever indexes you like.
125
-
126
- class MyJobClass < Que::Job
127
- def destroy
128
- Que.execute "INSERT INTO finished_jobs SELECT * FROM que_jobs WHERE queue = $1::text AND priority = $2::integer AND run_at = $3::timestamptz AND job_id = $4::bigint", @attrs.values_at(:queue, :priority, :run_at, :job_id)
129
- super
130
- end
131
- end
132
- ```
133
-
134
- Then just have your job classes inherit from MyJobClass instead of Que::Job. If you need to query the jobs table and you want to include both finished and unfinished jobs, you might use:
135
-
136
- ```ruby
137
- Que.execute "CREATE VIEW all_jobs AS SELECT * FROM que_jobs UNION ALL SELECT * FROM finished_jobs"
138
- Que.execute "SELECT * FROM all_jobs"
139
- ```
140
-
141
- Alternately, if you want a more foolproof solution and you're not scared of PostgreSQL, you can use a trigger:
142
-
143
- ```sql
144
- CREATE FUNCTION please_save_my_job()
145
- RETURNS trigger
146
- LANGUAGE plpgsql
147
- AS $$
148
- BEGIN
149
- INSERT INTO finished_jobs SELECT (OLD).*;
150
- RETURN OLD;
151
- END;
152
- $$;
153
-
154
- CREATE TRIGGER keep_all_my_old_jobs BEFORE DELETE ON que_jobs FOR EACH ROW EXECUTE PROCEDURE please_save_my_job();
155
- ```
156
-
157
- ### Not Retrying Certain Failed Jobs
158
-
159
- By default, when jobs fail, Que reschedules them to be retried later. If instead you'd like certain jobs to not be retried, and instead move them elsewhere to be examined later, you can accomplish that easily. First, we need a place for the failed jobs to be stored:
160
-
161
- ```sql
162
- CREATE TABLE failed_jobs AS SELECT * FROM que_jobs LIMIT 0
163
- ```
164
-
165
- Then, create a module that you can use in the jobs you don't want to retry:
166
-
167
- ```ruby
168
- module SkipRetries
169
- def run(*args)
170
- super
171
- rescue
172
- sql = <<-SQL
173
- WITH failed AS (
174
- DELETE
175
- FROM que_jobs
176
- WHERE queue = $1::text
177
- AND priority = $2::smallint
178
- AND run_at = $3::timestamptz
179
- AND job_id = $4::bigint
180
- RETURNING *
181
- )
182
- INSERT INTO failed_jobs
183
- SELECT * FROM failed;
184
- SQL
185
-
186
- Que.execute sql, @attrs.values_at(:queue, :priority, :run_at, :job_id)
187
-
188
- raise # Reraises caught error.
189
- end
190
- end
191
-
192
- class RunOnceJob < Que::Job
193
- prepend SkipRetries
194
-
195
- def run(*args)
196
- # Do something - if this job runs an error it'll be moved to the
197
- # failed_jobs table and not retried.
198
- end
199
- end
200
- ```
@@ -1,47 +0,0 @@
1
- ## Error Handling
2
-
3
- If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
4
-
5
- If a given job fails repeatedly, Que will retry it at exponentially-increasing intervals equal to (failure_count^4 + 3) seconds. This means that a job will be retried 4 seconds after its first failure, 19 seconds after its second, 84 seconds after its third, 259 seconds after its fourth, and so on until it succeeds. This pattern is very similar to DelayedJob's. Alternately, you can define your own retry logic by setting an interval to delay each time, or a callable that accepts the number of failures and returns an interval:
6
-
7
- ```ruby
8
- class MyJob < Que::Job
9
- # Just retry a failed job every 5 seconds:
10
- @retry_interval = 5
11
-
12
- # Always retry this job immediately (not recommended, or transient
13
- # errors will spam your error reporting):
14
- @retry_interval = 0
15
-
16
- # Increase the delay by 30 seconds every time this job fails:
17
- @retry_interval = proc { |count| count * 30 }
18
- end
19
- ```
20
-
21
- Unlike DelayedJob, however, there is currently no maximum number of failures after which jobs will be deleted. Que's assumption is that if a job is erroring perpetually (and not just transiently), you will want to take action to get the job working properly rather than simply losing it silently.
22
-
23
- If you're using an error notification system (highly recommended, of course), you can hook Que into it by setting a callable as the error handler:
24
-
25
- ```ruby
26
- Que.error_handler = proc do |error, job|
27
- # Do whatever you want with the error object or job row here.
28
-
29
- # Note that the job passed is not the actual job object, but the hash
30
- # representing the job row in the database, which looks like:
31
-
32
- # {
33
- # "queue" => "my_queue",
34
- # "priority" => 100,
35
- # "run_at" => 2015-03-06 11:07:08 -0500,
36
- # "job_id" => 65,
37
- # "job_class" => "MyJob",
38
- # "args" => ['argument', 78],
39
- # "error_count" => 0
40
- # }
41
-
42
- # This is done because the job may not have been able to be deserialized
43
- # properly, if the name of the job class was changed or the job is being
44
- # retrieved and worked by the wrong app. The job argument may also be
45
- # nil, if there was a connection failure or something similar.
46
- end
47
- ```
@@ -1,114 +0,0 @@
1
- ## Inspecting the Queue
2
-
3
- In order to remain simple and compatible with any ORM (or no ORM at all), Que is really just a very thin wrapper around some raw SQL. There are two methods available that query the jobs table and Postgres' system catalogs to retrieve information on the current state of the queue:
4
-
5
- ### Job Stats
6
-
7
- You can call `Que.job_stats` to return some aggregate data on the types of jobs currently in the queue. Example output:
8
-
9
- ```ruby
10
- [
11
- {
12
- "job_class"=>"ChargeCreditCard",
13
- "count"=>"10",
14
- "count_working"=>"4",
15
- "count_errored"=>"2",
16
- "highest_error_count"=>"5",
17
- "oldest_run_at"=>"2014-01-04 21:24:55.817129+00"
18
- },
19
- {
20
- "job_class"=>"SendRegistrationEmail",
21
- "count"=>"8",
22
- "count_working"=>"0",
23
- "count_errored"=>"0",
24
- "highest_error_count"=>"0",
25
- "oldest_run_at"=>"2014-01-04 22:24:55.81532+00"
26
- }
27
- ]
28
- ```
29
-
30
- This tells you that, for instance, there are ten ChargeCreditCard jobs in the queue, four of which are currently being worked, and two of which have experienced errors. One of them has started to process but experienced an error five times. The oldest_run_at is helpful for determining how long jobs have been sitting around, if you have backlog.
31
-
32
- ### Worker States
33
-
34
- You can call `Que.worker_states` to return some information on every worker touching the queue (not just those in the current process). Example output:
35
-
36
- ```ruby
37
- [
38
- {
39
- "priority"=>"2",
40
- "run_at"=>"2014-01-04 22:35:55.772324+00",
41
- "job_id"=>"4592",
42
- "job_class"=>"ChargeCreditCard",
43
- "args"=>"[345,56]",
44
- "error_count"=>"0",
45
- "last_error"=>nil,
46
- "pg_backend_pid"=>"1175",
47
- "pg_state"=>"idle",
48
- "pg_state_changed_at"=>"2014-01-04 22:35:55.777785+00",
49
- "pg_last_query"=>"SELECT * FROM users",
50
- "pg_last_query_started_at"=>"2014-01-04 22:35:55.777519+00",
51
- "pg_transaction_started_at"=>nil,
52
- "pg_waiting_on_lock"=>"f"
53
- }
54
- ]
55
- ```
56
-
57
- In this case, there is only one worker currently working the queue. The first seven fields are the attributes of the job it is currently running. The next seven fields are information about that worker's Postgres connection, and are taken from `pg_stat_activity` - see [Postgres' documentation](http://www.postgresql.org/docs/current/static/monitoring-stats.html#PG-STAT-ACTIVITY-VIEW) for more information on interpreting these fields.
58
-
59
- * `pg_backend_pid` - The pid of the Postgres process serving this worker. This is useful if you wanted to kill that worker's connection, for example, by running "SELECT pg_terminate_backend(1175)". This would free up the job to be attempted by another worker.
60
- * `pg_state` - The state of the Postgres backend. It may be "active" if the worker is currently running a query or "idle"/"idle in transaction" if it is not. It may also be in one of a few other less common states.
61
- * `pg_state_changed_at` - The timestamp for when the backend's state was last changed. If the backend is idle, this would reflect the time that the last query finished.
62
- * `pg_last_query` - The text of the current or most recent query that the worker sent to the database.
63
- * `pg_last_query_started_at` - The timestamp for when the last query began to run.
64
- * `pg_transaction_started_at` - The timestamp for when the worker's current transaction (if any) began.
65
- * `pg_waiting_on_lock` - Whether or not the worker is waiting for a lock in Postgres to be released.
66
-
67
- ### Custom Queries
68
-
69
- If you want to query the jobs table yourself to see what's been queued or to check the state of various jobs, you can always use Que to execute whatever SQL you want:
70
-
71
- ```ruby
72
- Que.execute("select count(*) from que_jobs") #=> [{"count"=>"492"}]
73
- ```
74
-
75
- If you want to use ActiveRecord's features when querying, you can define your own model around Que's job table:
76
-
77
- ```ruby
78
- class QueJob < ActiveRecord::Base
79
- end
80
-
81
- # Or:
82
-
83
- class MyJob < ActiveRecord::Base
84
- self.table_name = :que_jobs
85
- end
86
- ```
87
-
88
- Then you can query just as you would with any other model. Since the jobs table has a composite primary key, however, you probably won't be able to update or destroy jobs this way, though.
89
-
90
- If you're using Sequel, you can use the same technique:
91
-
92
- ```ruby
93
- class QueJob < Sequel::Model
94
- end
95
-
96
- # Or:
97
-
98
- class MyJob < Sequel::Model(:que_jobs)
99
- end
100
- ```
101
-
102
- And note that Sequel *does* support composite primary keys:
103
-
104
- ```ruby
105
- job = QueJob.where(:job_class => "ChargeCreditCard").first
106
- job.priority = 1
107
- job.save
108
- ```
109
-
110
- Or, you can just use Sequel's dataset methods:
111
-
112
- ```ruby
113
- DB[:que_jobs].where{priority > 3}.all
114
- ```
data/docs/logging.md DELETED
@@ -1,50 +0,0 @@
1
- ## Logging
2
-
3
- By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running via the `que` executable). So, your logs will look something like:
4
-
5
- ```
6
- I, [2014-01-12T05:07:31.094201 #4687] INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
7
- ```
8
-
9
- Of course you can have it log wherever you like:
10
-
11
- ```ruby
12
- Que.logger = Logger.new(...)
13
- ```
14
-
15
- You can use Que's logger in your jobs anywhere you like:
16
-
17
- ```ruby
18
- class MyJob
19
- def run
20
- Que.log :my_output => "my string"
21
- end
22
- end
23
-
24
- #=> I, [2014-01-12T05:13:11.006776 #4914] INFO -- : {"lib":"que","thread":24960,"my_output":"my string"}
25
- ```
26
-
27
- Que will always add a 'lib' key, so you can easily filter its output from that of other sources, and the object_id of the thread that emitted the log, so you can follow the actions of a particular worker if you wish. You can also pass a :level key to set the level of the output:
28
-
29
- ```ruby
30
- Que.log :level => :debug, :my_output => 'my string'
31
- #=> D, [2014-01-12T05:16:15.221941 #5088] DEBUG -- : {"lib":"que","thread":24960,"my_output":"my string"}
32
- ```
33
-
34
- If you don't like JSON, you can also customize the format of the logging output by passing a callable object (such as a proc) to Que.log_formatter=. The proc should take a hash (the keys are symbols) and return a string. The keys and values are just as you would expect from the JSON output:
35
-
36
- ```ruby
37
- Que.log_formatter = proc do |data|
38
- "Thread number #{data[:thread]} experienced a #{data[:event]}"
39
- end
40
- ```
41
-
42
- If the log formatter returns nil or false, a nothing will be logged at all. You could use this to narrow down what you want to emit, for example:
43
-
44
- ```ruby
45
- Que.log_formatter = proc do |data|
46
- if [:job_worked, :job_unavailable].include?(data[:event])
47
- JSON.dump(data)
48
- end
49
- end
50
- ```
@@ -1,80 +0,0 @@
1
- ## Managing Workers
2
-
3
- Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
4
-
5
- When the worker pool is active (when you set `Que.mode = :async`), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
6
-
7
- Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
8
-
9
- JRuby and Rubinius, on the other hand, have no global interpreter lock, and so can make use of multiple CPU cores - you could potentially set the number of workers very high for them. You should experiment to find the best setting for your use case.
10
-
11
- You can change the number of workers in the pool whenever you like by setting the `worker_count` option:
12
-
13
- ```ruby
14
- Que.worker_count = 8
15
- ```
16
-
17
- ### Working Jobs Via Executable
18
-
19
- If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
20
-
21
- ```shell
22
- # Run a pool of 4 workers:
23
- que
24
-
25
- # Or configure the number of workers:
26
- que --worker-count 8
27
- ```
28
-
29
- See `que -h` for a list of command-line options.
30
-
31
- ### Thread-Unsafe Application Code
32
-
33
- If your application code is not thread-safe, you won't want any workers to be processing jobs while anything else is happening in the Ruby process. So, you'll want to turn the worker pool off by default:
34
-
35
- ```ruby
36
- Que.mode = :off
37
- ```
38
-
39
- This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate process, like so:
40
-
41
- ```shell
42
- que --worker-count 1
43
- ```
44
-
45
- ### The Wake Interval
46
-
47
- If a worker checks the job queue and finds no jobs ready for it to work, it will fall asleep. In order to make sure that newly-available jobs don't go unworked, a worker is awoken every so often to check for available work. By default, this happens every five seconds, but you can make it happen more or less often by setting a custom wake_interval:
48
-
49
- ```ruby
50
- Que.wake_interval = 2 # In Rails, 2.seconds also works fine.
51
- ```
52
-
53
- You can also choose to never let workers wake up on their own:
54
-
55
- ```ruby
56
- # Never wake up any workers:
57
- Que.wake_interval = nil
58
- ```
59
-
60
- If you do this, though, you'll need to wake workers manually.
61
-
62
- ### Manually Waking Workers
63
-
64
- Regardless of the `wake_interval` setting, you can always wake workers manually:
65
-
66
- ```ruby
67
- # Wake up a single worker to check the queue for work:
68
- Que.wake!
69
-
70
- # Wake up all workers in this process to check for work:
71
- Que.wake_all!
72
- ```
73
-
74
- `Que.wake_all!` is helpful if there are no jobs available and all your workers go to sleep, and then you queue a large number of jobs. Typically, it will take a little while for the entire pool of workers get going again - a new one will wake up every `wake_interval` seconds, but it will take up to `wake_interval * worker_count` seconds for all of them to get going. `Que.wake_all!` can get them all moving immediately.
75
-
76
- ### Connection Pool Size
77
-
78
- For the job locking system to work properly, each worker thread needs to reserve a database connection from the connection pool for the period of time between when it locks a job and when it releases that lock (which won't happen until the job has been finished and deleted from the queue).
79
-
80
- So, for example, if you're running 6 workers via the executable, you'll want to make sure that whatever connection pool Que is using (usually ActiveRecord's) has a maximum size of at least 6. If you're running those workers in a web process, you'll want the size to be at least 6 plus however many connections you expect your application to need for serving web requests (which may only be one if you're using Rails in single-threaded mode, or many more if you're running a threaded web server like Puma).
data/docs/migrating.md DELETED
@@ -1,30 +0,0 @@
1
- ## Migrating
2
-
3
- Some new releases of Que may require updates to the database schema. It's recommended that you integrate these updates alongside your other database migrations. For example, when Que released version 0.6.0, the schema version was updated from 2 to 3. If you're running ActiveRecord, you could make a migration to perform this upgrade like so:
4
-
5
- ```ruby
6
- class UpdateQue < ActiveRecord::Migration
7
- def self.up
8
- Que.migrate! :version => 3
9
- end
10
-
11
- def self.down
12
- Que.migrate! :version => 2
13
- end
14
- end
15
- ```
16
-
17
- This will make sure that your database schema stays consistent with your codebase. If you're looking for something quicker and dirtier, you can always manually migrate in a console session:
18
-
19
- ```ruby
20
- # Change schema to version 3.
21
- Que.migrate! :version => 3
22
-
23
- # Update to whatever the latest schema version is.
24
- Que.migrate!
25
-
26
- # Check your current schema version.
27
- Que.db_version #=> 3
28
- ```
29
-
30
- Note that you can remove Que from your database completely by migrating to version 0.
@@ -1,27 +0,0 @@
1
- ## Multiple Queues
2
-
3
- Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
4
-
5
- ```shell
6
- que --queue-name credit_cards
7
- ```
8
-
9
- Then you can set jobs to be enqueued in that queue specifically:
10
-
11
- ```ruby
12
- ProcessCreditCard.enqueue current_user.id, :queue => 'credit_cards'
13
-
14
- # Or:
15
-
16
- class ProcessCreditCard < Que::Job
17
- # Set a default queue for this job class; this can be overridden by
18
- # passing the :queue parameter to enqueue like above.
19
- @queue = 'credit_cards'
20
- end
21
- ```
22
-
23
- In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
24
-
25
- ```ruby
26
- Que.enqueue current_user.id, :job_class => 'ProcessCreditCard', :queue => 'credit_cards'
27
- ```
@@ -1,7 +0,0 @@
1
- ## Shutting Down Safely
2
-
3
- To ensure safe operation, Que needs to be very careful in how it shuts down. When a Ruby process ends normally, it calls Thread#kill on any threads that are still running - unfortunately, if a thread is in the middle of a transaction when this happens, there is a risk that it will be prematurely commited, resulting in data corruption. See [here](http://blog.headius.com/2008/02/ruby-threadraise-threadkill-timeoutrb.html) and [here](http://coderrr.wordpress.com/2011/05/03/beware-of-threadkill-or-your-activerecord-transactions-are-in-danger-of-being-partially-committed/) for more detail on this.
4
-
5
- To prevent this, Que will block a Ruby process from exiting until all jobs it is working have completed normally. Unfortunately, if you have long-running jobs, this may take a very long time (and if something goes wrong with a job's logic, it may never happen). The solution in this case is SIGKILL - luckily, Ruby processes that are killed via SIGKILL will end without using Thread#kill on its running threads. This is safer than exiting normally - when PostgreSQL loses the connection it will simply roll back the open transaction, if any, and unlock the job so it can be retried later by another worker. Be sure to read [Writing Reliable Jobs](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md) for information on how to design your jobs to fail safely.
6
-
7
- So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.