que 1.0.0.beta2 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.github/workflows/tests.yml +43 -0
- data/CHANGELOG.md +242 -6
- data/README.md +55 -8
- data/bin/command_line_interface.rb +50 -26
- data/docs/README.md +785 -32
- data/lib/que/active_record/connection.rb +1 -1
- data/lib/que/active_record/model.rb +1 -1
- data/lib/que/connection.rb +11 -1
- data/lib/que/job_buffer.rb +27 -22
- data/lib/que/job_methods.rb +4 -0
- data/lib/que/locker.rb +27 -26
- data/lib/que/rails/railtie.rb +0 -2
- data/lib/que/sequel/model.rb +14 -16
- data/lib/que/version.rb +1 -1
- data/lib/que/worker.rb +37 -15
- data/que.gemspec +2 -2
- metadata +14 -30
- data/CHANGELOG.1.0.beta.md +0 -127
- data/docs/active_job.md +0 -6
- data/docs/advanced_setup.md +0 -49
- data/docs/command_line_interface.md +0 -45
- data/docs/error_handling.md +0 -94
- data/docs/inspecting_the_queue.md +0 -64
- data/docs/job_helper_methods.md +0 -27
- data/docs/logging.md +0 -31
- data/docs/managing_workers.md +0 -25
- data/docs/middleware.md +0 -36
- data/docs/migrating.md +0 -27
- data/docs/multiple_queues.md +0 -31
- data/docs/shutting_down_safely.md +0 -7
- data/docs/using_plain_connections.md +0 -65
- data/docs/using_sequel.md +0 -33
- data/docs/writing_reliable_jobs.md +0 -108
data/docs/using_sequel.md
DELETED
|
@@ -1,33 +0,0 @@
|
|
|
1
|
-
## Using Sequel
|
|
2
|
-
|
|
3
|
-
If you're using Sequel, with or without Rails, you'll need to give Que a specific database instance to use:
|
|
4
|
-
|
|
5
|
-
```ruby
|
|
6
|
-
DB = Sequel.connect(ENV['DATABASE_URL'])
|
|
7
|
-
Que.connection = DB
|
|
8
|
-
```
|
|
9
|
-
|
|
10
|
-
Then you can safely use the same database object to transactionally protect your jobs:
|
|
11
|
-
|
|
12
|
-
```ruby
|
|
13
|
-
class MyJob < Que::Job
|
|
14
|
-
def run(user_id:)
|
|
15
|
-
# Do stuff.
|
|
16
|
-
|
|
17
|
-
DB.transaction do
|
|
18
|
-
# Make changes to the database.
|
|
19
|
-
|
|
20
|
-
# Destroying this job will be protected by the same transaction.
|
|
21
|
-
destroy
|
|
22
|
-
end
|
|
23
|
-
end
|
|
24
|
-
end
|
|
25
|
-
|
|
26
|
-
# Or, in your controller action:
|
|
27
|
-
DB.transaction do
|
|
28
|
-
@user = User.create(params[:user])
|
|
29
|
-
MyJob.enqueue user_id: @user.id
|
|
30
|
-
end
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
Sequel automatically wraps model persistance actions (create, update, destroy) in transactions, so you can simply call #enqueue methods from your models' callbacks, if you wish.
|
|
@@ -1,108 +0,0 @@
|
|
|
1
|
-
## Writing Reliable Jobs
|
|
2
|
-
|
|
3
|
-
Que does everything it can to ensure that jobs are worked exactly once, but if something bad happens when a job is halfway completed, there's no way around it - the job will need be repeated over again from the beginning, probably by a different worker. When you're writing jobs, you need to be prepared for this to happen.
|
|
4
|
-
|
|
5
|
-
The safest type of job is one that reads in data, either from the database or from external APIs, then does some number crunching and writes the results to the database. These jobs are easy to make safe - simply write the results to the database inside a transaction, and also destroy the job inside that transaction, like so:
|
|
6
|
-
|
|
7
|
-
```ruby
|
|
8
|
-
class UpdateWidgetPrice < Que::Job
|
|
9
|
-
def run(widget_id)
|
|
10
|
-
widget = Widget[widget_id]
|
|
11
|
-
price = ExternalService.get_widget_price(widget_id)
|
|
12
|
-
|
|
13
|
-
ActiveRecord::Base.transaction do
|
|
14
|
-
# Make changes to the database.
|
|
15
|
-
widget.update price: price
|
|
16
|
-
|
|
17
|
-
# Mark the job as destroyed, so it doesn't run again.
|
|
18
|
-
destroy
|
|
19
|
-
end
|
|
20
|
-
end
|
|
21
|
-
end
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
Here, you're taking advantage of the guarantees of an [ACID](https://en.wikipedia.org/wiki/ACID) database. The job is destroyed along with the other changes, so either the write will succeed and the job will be run only once, or it will fail and the database will be left untouched. But even if it fails, the job can simply be retried, and there are no lingering effects from the first attempt, so no big deal.
|
|
25
|
-
|
|
26
|
-
The more difficult type of job is one that makes changes that can't be controlled transactionally. For example, writing to an external service:
|
|
27
|
-
|
|
28
|
-
```ruby
|
|
29
|
-
class ChargeCreditCard < Que::Job
|
|
30
|
-
def run(user_id, credit_card_id)
|
|
31
|
-
CreditCardService.charge(credit_card_id, amount: "$10.00")
|
|
32
|
-
|
|
33
|
-
ActiveRecord::Base.transaction do
|
|
34
|
-
User.where(id: user_id).update_all charged_at: Time.now
|
|
35
|
-
destroy
|
|
36
|
-
end
|
|
37
|
-
end
|
|
38
|
-
end
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
What if the process abruptly dies after we tell the provider to charge the credit card, but before we finish the transaction? Que will retry the job, but there's no way to tell where (or even if) it failed the first time. The credit card will be charged a second time, and then you've got an angry customer. The ideal solution in this case is to make the job [idempotent](https://en.wikipedia.org/wiki/Idempotence), meaning that it will have the same effect no matter how many times it is run:
|
|
42
|
-
|
|
43
|
-
```ruby
|
|
44
|
-
class ChargeCreditCard < Que::Job
|
|
45
|
-
def run(user_id, credit_card_id)
|
|
46
|
-
unless CreditCardService.check_for_previous_charge(credit_card_id)
|
|
47
|
-
CreditCardService.charge(credit_card_id, amount: "$10.00")
|
|
48
|
-
end
|
|
49
|
-
|
|
50
|
-
ActiveRecord::Base.transaction do
|
|
51
|
-
User.where(id: user_id).update_all charged_at: Time.now
|
|
52
|
-
destroy
|
|
53
|
-
end
|
|
54
|
-
end
|
|
55
|
-
end
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
This makes the job slightly more complex, but reliable (or, at least, as reliable as your credit card service).
|
|
59
|
-
|
|
60
|
-
Finally, there are some jobs where you won't want to write to the database at all:
|
|
61
|
-
|
|
62
|
-
```ruby
|
|
63
|
-
class SendVerificationEmail < Que::Job
|
|
64
|
-
def run(email_address)
|
|
65
|
-
Mailer.verification_email(email_address).deliver
|
|
66
|
-
end
|
|
67
|
-
end
|
|
68
|
-
```
|
|
69
|
-
|
|
70
|
-
In this case, we don't have any no way to prevent the occasional double-sending of an email. But, for ease of use, you can leave out the transaction and the `destroy` call entirely - Que will recognize that the job wasn't destroyed and will clean it up for you.
|
|
71
|
-
|
|
72
|
-
### Timeouts
|
|
73
|
-
|
|
74
|
-
Long-running jobs aren't necessarily a problem for the database, since the overhead of an individual job is very small (just an advisory lock held in memory). But jobs that hang indefinitely can tie up a worker and [block the Ruby process from exiting gracefully](https://github.com/chanks/que/blob/master/docs/shutting_down_safely.md), which is a pain.
|
|
75
|
-
|
|
76
|
-
If there's part of your job that is prone to hang (due to an API call or other HTTP request that never returns, for example), you can (and should) timeout those parts of your job. For example, consider a job that needs to make an HTTP request and then write to the database:
|
|
77
|
-
|
|
78
|
-
```ruby
|
|
79
|
-
class ScrapeStuff < Que::Job
|
|
80
|
-
def run(url_to_scrape)
|
|
81
|
-
result = YourHTTPLibrary.get(url_to_scrape)
|
|
82
|
-
|
|
83
|
-
ActiveRecord::Base.transaction do
|
|
84
|
-
# Insert result...
|
|
85
|
-
|
|
86
|
-
destroy
|
|
87
|
-
end
|
|
88
|
-
end
|
|
89
|
-
end
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
That request could take a very long time, or never return at all. Let's use the timeout feature that almost all HTTP libraries offer some version of:
|
|
93
|
-
|
|
94
|
-
```ruby
|
|
95
|
-
class ScrapeStuff < Que::Job
|
|
96
|
-
def run(url_to_scrape)
|
|
97
|
-
result = YourHTTPLibrary.get(url_to_scrape, timeout: 5)
|
|
98
|
-
|
|
99
|
-
ActiveRecord::Base.transaction do
|
|
100
|
-
# Insert result...
|
|
101
|
-
|
|
102
|
-
destroy
|
|
103
|
-
end
|
|
104
|
-
end
|
|
105
|
-
end
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
Now, if the request takes more than five seconds, an error will be raised (probably - check your library's documentation) and Que will just retry the job later.
|