que 0.11.2 → 0.11.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +2 -0
- data/CHANGELOG.md +10 -0
- data/README.md +3 -2
- data/bin/que +7 -5
- data/docs/advanced_setup.md +5 -0
- data/docs/logging.md +1 -1
- data/docs/managing_workers.md +8 -12
- data/docs/multiple_queues.md +1 -1
- data/docs/using_plain_connections.md +2 -17
- data/lib/generators/que/install_generator.rb +2 -0
- data/lib/generators/que/templates/add_que.rb +2 -0
- data/lib/que.rb +2 -0
- data/lib/que/adapters/active_record.rb +2 -0
- data/lib/que/adapters/base.rb +2 -0
- data/lib/que/adapters/connection_pool.rb +2 -0
- data/lib/que/adapters/pg.rb +2 -0
- data/lib/que/adapters/pond.rb +2 -0
- data/lib/que/adapters/sequel.rb +2 -0
- data/lib/que/job.rb +2 -0
- data/lib/que/migrations.rb +2 -0
- data/lib/que/railtie.rb +2 -0
- data/lib/que/rake_tasks.rb +4 -0
- data/lib/que/sql.rb +2 -0
- data/lib/que/version.rb +3 -1
- data/lib/que/worker.rb +2 -0
- data/spec/adapters/active_record_spec.rb +2 -0
- data/spec/adapters/connection_pool_spec.rb +2 -0
- data/spec/adapters/pg_spec.rb +2 -0
- data/spec/adapters/pond_spec.rb +2 -0
- data/spec/adapters/sequel_spec.rb +2 -0
- data/spec/spec_helper.rb +2 -0
- data/spec/support/helpers.rb +2 -0
- data/spec/support/jobs.rb +2 -0
- data/spec/support/shared_examples/adapter.rb +2 -0
- data/spec/support/shared_examples/multi_threaded_adapter.rb +2 -0
- data/spec/travis.rb +2 -0
- data/spec/unit/connection_spec.rb +2 -0
- data/spec/unit/customization_spec.rb +2 -0
- data/spec/unit/enqueue_spec.rb +2 -0
- data/spec/unit/helper_spec.rb +2 -0
- data/spec/unit/logging_spec.rb +2 -0
- data/spec/unit/migrations_spec.rb +2 -0
- data/spec/unit/pool_spec.rb +2 -0
- data/spec/unit/run_spec.rb +2 -0
- data/spec/unit/states_spec.rb +2 -0
- data/spec/unit/stats_spec.rb +2 -0
- data/spec/unit/transaction_spec.rb +2 -0
- data/spec/unit/work_spec.rb +2 -0
- data/spec/unit/worker_spec.rb +2 -0
- data/tasks/benchmark.rb +2 -0
- data/tasks/rspec.rb +2 -0
- data/tasks/safe_shutdown.rb +2 -0
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 36663fa76366e576b7da98780e9c71519148fc10
|
4
|
+
data.tar.gz: dd37b18e13b152b2f2f876439b491ec3820465a4
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f6d752b03080ae3d1a04b3afdf21b886a9d615011d38d713015c47aadd1f389bb231ed9d553b4c7beba382bbdfbeda97510d7212debc527f6f6daedef074b646
|
7
|
+
data.tar.gz: 7e087a89f476a2fefe2426e1d9d6143045748cda09fb61254ee0645d670583a74f4bef137bf13cc04f232e45ccfc6d91c7e0337e6b4ab004aed88aadd0a48702
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,13 @@
|
|
1
|
+
### 0.11.3 (2016-02-26)
|
2
|
+
|
3
|
+
* Fixed bug with displaying the current version of the que executable. (#122) (hardbap)
|
4
|
+
|
5
|
+
* Output to STDOUT when running via the executable or rake task is no longer buffered. This prevented logging in some cases. (#129) (hmarr)
|
6
|
+
|
7
|
+
* Officially added support for Ruby 2.2 and 2.3.
|
8
|
+
|
9
|
+
* String literals are now frozen on Ruby 2.3.
|
10
|
+
|
1
11
|
### 0.11.2 (2015-09-09)
|
2
12
|
|
3
13
|
* Fix Job class constantizing when ActiveSupport isn't loaded. (#121) (godfat)
|
data/README.md
CHANGED
@@ -100,8 +100,8 @@ ChargeCreditCard.enqueue current_user.id, :credit_card_id => card.id, :run_at =>
|
|
100
100
|
|
101
101
|
To determine what happens when a job is queued, you can set Que's mode. There are a few options for the mode:
|
102
102
|
|
103
|
-
* `Que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's
|
104
|
-
* `Que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread.
|
103
|
+
* `Que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's an executable included that will do this, `que`). This is the default when running `bin/rails console`.
|
104
|
+
* `Que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
|
105
105
|
* `Que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.enqueue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
|
106
106
|
|
107
107
|
**If you're using ActiveRecord to dump your database's schema, [set your schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly.** (You can use schema_format as :ruby if you want but keep in mind this is highly advised against, as some parts of Que will not work.)
|
@@ -112,6 +112,7 @@ To determine what happens when a job is queued, you can set Que's mode. There ar
|
|
112
112
|
* [que-web](https://github.com/statianzo/que-web) is a Sinatra-based UI for inspecting your job queue.
|
113
113
|
* [que-testing](https://github.com/statianzo/que-testing) allows making assertions on enqueued jobs.
|
114
114
|
* [que-go](https://github.com/bgentry/que-go) is a port of Que for the Go programming language. It uses the same table structure, so that you can use the same job queue from Ruby and Go applications.
|
115
|
+
* [wisper-que](https://github.com/joevandyk/wisper-que) adds support for Que to [wisper](https://github.com/krisleech/wisper).
|
115
116
|
|
116
117
|
If you have a project that uses or relates to Que, feel free to submit a PR adding it to the list!
|
117
118
|
|
data/bin/que
CHANGED
@@ -4,6 +4,8 @@ require 'optparse'
|
|
4
4
|
require 'ostruct'
|
5
5
|
require 'logger'
|
6
6
|
|
7
|
+
$stdout.sync = true
|
8
|
+
|
7
9
|
options = OpenStruct.new
|
8
10
|
|
9
11
|
OptionParser.new do |opts|
|
@@ -26,7 +28,7 @@ OptionParser.new do |opts|
|
|
26
28
|
end
|
27
29
|
|
28
30
|
opts.on('-v', '--version', "Show Que version") do
|
29
|
-
$stdout.puts "Que version #{Version}"
|
31
|
+
$stdout.puts "Que version #{Que::Version}"
|
30
32
|
exit 0
|
31
33
|
end
|
32
34
|
|
@@ -57,7 +59,7 @@ end
|
|
57
59
|
Que.logger ||= Logger.new(STDOUT)
|
58
60
|
|
59
61
|
begin
|
60
|
-
if log_level = options.log_level
|
62
|
+
if log_level = (options.log_level || ENV['QUE_LOG_LEVEL'])
|
61
63
|
Que.logger.level = Logger.const_get(log_level.upcase)
|
62
64
|
end
|
63
65
|
rescue NameError
|
@@ -65,9 +67,9 @@ rescue NameError
|
|
65
67
|
exit 1
|
66
68
|
end
|
67
69
|
|
68
|
-
Que.queue_name = options.queue_name
|
69
|
-
Que.worker_count = options.worker_count || Que.worker_count || 4
|
70
|
-
Que.wake_interval = options.wake_interval || Que.wake_interval || 0.1
|
70
|
+
Que.queue_name = options.queue_name || ENV['QUE_QUEUE'] || Que.queue_name || nil
|
71
|
+
Que.worker_count = (options.worker_count || ENV['QUE_WORKER_COUNT'] || Que.worker_count || 4).to_i
|
72
|
+
Que.wake_interval = (options.wake_interval || ENV['QUE_WAKE_INTERVAL'] || Que.wake_interval || 0.1).to_f
|
71
73
|
Que.mode = :async
|
72
74
|
|
73
75
|
stop = false
|
data/docs/advanced_setup.md
CHANGED
@@ -1,5 +1,7 @@
|
|
1
1
|
## Advanced Setup
|
2
2
|
|
3
|
+
### Using ActiveRecord Without Rails
|
4
|
+
|
3
5
|
If you're using both Rails and ActiveRecord, the README describes how to get started with Que (which is pretty straightforward, since Que includes a Railtie that handles a lot of setup for you). Otherwise, you'll need to do some manual setup.
|
4
6
|
|
5
7
|
If you're using ActiveRecord outside of Rails, you'll need to tell Que to piggyback on its connection pool after you've connected to the database:
|
@@ -59,6 +61,9 @@ if defined?(PhusionPassenger)
|
|
59
61
|
end
|
60
62
|
```
|
61
63
|
|
64
|
+
If there's other setup you want to do for workers, such as setting up the
|
65
|
+
configuration, you'll need to do that manually as well.
|
66
|
+
|
62
67
|
### Managing the Jobs Table
|
63
68
|
|
64
69
|
After you've connected Que to the database, you can manage the jobs table:
|
data/docs/logging.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
## Logging
|
2
2
|
|
3
|
-
By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running
|
3
|
+
By default, Que logs important information in JSON to either Rails' logger (when running in a Rails web process) or STDOUT (when running via the `que` executable). So, your logs will look something like:
|
4
4
|
|
5
5
|
```
|
6
6
|
I, [2014-01-12T05:07:31.094201 #4687] INFO -- : {"lib":"que","thread":104928,"event":"job_worked","elapsed":0.01045,"job":{"priority":"1","run_at":"2014-01-12 05:07:31.081877+00","job_id":"4","job_class":"MyJob","args":[],"error_count":"0"}}
|
data/docs/managing_workers.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
Que provides a pool of workers to process jobs in a multithreaded fashion - this allows you to save memory by working many jobs simultaneously in the same process.
|
4
4
|
|
5
|
-
When the worker pool is active (
|
5
|
+
When the worker pool is active (when you set `Que.mode = :async`), the default number of workers is 4. This is fine for most use cases, but the ideal number for your app will depend on your interpreter and what types of jobs you're running.
|
6
6
|
|
7
7
|
Ruby MRI has a global interpreter lock (GIL), which prevents it from using more than one CPU core at a time. Having multiple workers running makes sense if your jobs tend to spend a lot of time in I/O (waiting on complex database queries, sending emails, making HTTP requests, etc.), as most jobs do. However, if your jobs are doing a lot of work in Ruby, they'll be spending a lot of time blocking each other, and having too many workers running will just slow everything down.
|
8
8
|
|
@@ -14,23 +14,19 @@ You can change the number of workers in the pool whenever you like by setting th
|
|
14
14
|
Que.worker_count = 8
|
15
15
|
```
|
16
16
|
|
17
|
-
### Working Jobs Via
|
17
|
+
### Working Jobs Via Executable
|
18
18
|
|
19
19
|
If you don't want to burden your web processes with too much work and want to run workers in a background process instead, similar to how most other queues work, you can:
|
20
20
|
|
21
21
|
```shell
|
22
22
|
# Run a pool of 4 workers:
|
23
|
-
|
23
|
+
que
|
24
24
|
|
25
25
|
# Or configure the number of workers:
|
26
|
-
|
26
|
+
que --worker-count 8
|
27
27
|
```
|
28
28
|
|
29
|
-
|
30
|
-
|
31
|
-
```shell
|
32
|
-
QUE_QUEUE=other_queue QUE_WORKER_COUNT=2 QUE_WAKE_INTERVAL=0.5 rake que:work
|
33
|
-
```
|
29
|
+
See `que -h` for a list of command-line options.
|
34
30
|
|
35
31
|
### Thread-Unsafe Application Code
|
36
32
|
|
@@ -40,10 +36,10 @@ If your application code is not thread-safe, you won't want any workers to be pr
|
|
40
36
|
Que.mode = :off
|
41
37
|
```
|
42
38
|
|
43
|
-
This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate
|
39
|
+
This will prevent Que from trying to process jobs in the background of your web processes. In order to actually work jobs, you'll want to run a single worker at a time, and to do so via a separate process, like so:
|
44
40
|
|
45
41
|
```shell
|
46
|
-
|
42
|
+
que --worker-count 1
|
47
43
|
```
|
48
44
|
|
49
45
|
### The Wake Interval
|
@@ -81,4 +77,4 @@ Que.wake_all!
|
|
81
77
|
|
82
78
|
For the job locking system to work properly, each worker thread needs to reserve a database connection from the connection pool for the period of time between when it locks a job and when it releases that lock (which won't happen until the job has been finished and deleted from the queue).
|
83
79
|
|
84
|
-
So, for example, if you're running 6 workers
|
80
|
+
So, for example, if you're running 6 workers via the executable, you'll want to make sure that whatever connection pool Que is using (usually ActiveRecord's) has a maximum size of at least 6. If you're running those workers in a web process, you'll want the size to be at least 6 plus however many connections you expect your application to need for serving web requests (which may only be one if you're using Rails in single-threaded mode, or many more if you're running a threaded web server like Puma).
|
data/docs/multiple_queues.md
CHANGED
@@ -3,7 +3,7 @@
|
|
3
3
|
Que supports the use of multiple queues in a single job table. This feature is intended to support the case where multiple applications (with distinct codebases) are sharing the same database. For instance, you might have a separate Ruby application that handles only processing credit cards. In that case, you can run that application's workers against a specific queue:
|
4
4
|
|
5
5
|
```shell
|
6
|
-
|
6
|
+
que --queue-name credit_cards
|
7
7
|
```
|
8
8
|
|
9
9
|
Then you can set jobs to be enqueued in that queue specifically:
|
@@ -1,21 +1,6 @@
|
|
1
1
|
## Using Plain Postgres Connections
|
2
2
|
|
3
|
-
If you're not using an ORM like ActiveRecord or Sequel, you can
|
4
|
-
|
5
|
-
```ruby
|
6
|
-
require 'uri'
|
7
|
-
require 'pg'
|
8
|
-
|
9
|
-
uri = URI.parse(ENV['DATABASE_URL'])
|
10
|
-
|
11
|
-
Que.connection = PG::Connection.open :host => uri.host,
|
12
|
-
:user => uri.user,
|
13
|
-
:password => uri.password,
|
14
|
-
:port => uri.port || 5432,
|
15
|
-
:dbname => uri.path[1..-1]
|
16
|
-
```
|
17
|
-
|
18
|
-
If you want to be able to use multithreading to run multiple jobs simultaneously in the same process, though, you'll need the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
|
3
|
+
If you're not using an ORM like ActiveRecord or Sequel, you can use one of two gems to manage a connection pool for your PG connections. The first is the ConnectionPool gem (be sure to add `gem 'connection_pool'` to your Gemfile):
|
19
4
|
|
20
5
|
```ruby
|
21
6
|
require 'uri'
|
@@ -53,4 +38,4 @@ Que.connection = Pond.new :maximum_size => 10 do
|
|
53
38
|
end
|
54
39
|
```
|
55
40
|
|
56
|
-
Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.
|
41
|
+
Please be aware that if you're using ActiveRecord or Sequel to manage your data, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data. In general, your app should probably be using a connection pool, and Que should probably hook into whatever connection pool you're already using.
|
data/lib/que.rb
CHANGED
data/lib/que/adapters/base.rb
CHANGED
data/lib/que/adapters/pg.rb
CHANGED
data/lib/que/adapters/pond.rb
CHANGED
data/lib/que/adapters/sequel.rb
CHANGED
data/lib/que/job.rb
CHANGED
data/lib/que/migrations.rb
CHANGED
data/lib/que/railtie.rb
CHANGED
data/lib/que/rake_tasks.rb
CHANGED
@@ -1,6 +1,10 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
namespace :que do
|
2
4
|
desc "Process Que's jobs using a worker pool"
|
3
5
|
task :work => :environment do
|
6
|
+
$stdout.sync = true
|
7
|
+
|
4
8
|
$stdout.puts "The que:work rake task has been deprecated and will be removed in Que 1.0. Please transition to the que command line interface instead."
|
5
9
|
|
6
10
|
if defined?(::Rails) && Rails.respond_to?(:application)
|
data/lib/que/sql.rb
CHANGED
data/lib/que/version.rb
CHANGED
data/lib/que/worker.rb
CHANGED
data/spec/adapters/pg_spec.rb
CHANGED
data/spec/adapters/pond_spec.rb
CHANGED
data/spec/spec_helper.rb
CHANGED
data/spec/support/helpers.rb
CHANGED
data/spec/support/jobs.rb
CHANGED
data/spec/travis.rb
CHANGED
data/spec/unit/enqueue_spec.rb
CHANGED
data/spec/unit/helper_spec.rb
CHANGED
data/spec/unit/logging_spec.rb
CHANGED
data/spec/unit/pool_spec.rb
CHANGED
data/spec/unit/run_spec.rb
CHANGED
data/spec/unit/states_spec.rb
CHANGED
data/spec/unit/stats_spec.rb
CHANGED
data/spec/unit/work_spec.rb
CHANGED
data/spec/unit/worker_spec.rb
CHANGED
data/tasks/benchmark.rb
CHANGED
data/tasks/rspec.rb
CHANGED
data/tasks/safe_shutdown.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: que
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.11.
|
4
|
+
version: 0.11.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Chris Hanks
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2016-02-26 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -125,7 +125,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
125
125
|
version: '0'
|
126
126
|
requirements: []
|
127
127
|
rubyforge_project:
|
128
|
-
rubygems_version: 2.
|
128
|
+
rubygems_version: 2.5.1
|
129
129
|
signing_key:
|
130
130
|
specification_version: 4
|
131
131
|
summary: A PostgreSQL-based Job Queue
|