cloudtasker 0.6.0 → 0.9.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (41) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +3 -0
  3. data/.rubocop.yml +5 -0
  4. data/.travis.yml +3 -3
  5. data/CHANGELOG.md +38 -0
  6. data/README.md +142 -26
  7. data/_config.yml +1 -0
  8. data/app/controllers/cloudtasker/worker_controller.rb +21 -5
  9. data/cloudtasker.gemspec +2 -2
  10. data/docs/BATCH_JOBS.md +29 -4
  11. data/docs/CRON_JOBS.md +18 -14
  12. data/exe/cloudtasker +13 -1
  13. data/gemfiles/google_cloud_tasks_1.0.gemfile.lock +26 -9
  14. data/gemfiles/google_cloud_tasks_1.1.gemfile.lock +26 -9
  15. data/gemfiles/google_cloud_tasks_1.2.gemfile.lock +27 -10
  16. data/gemfiles/google_cloud_tasks_1.3.gemfile.lock +26 -9
  17. data/gemfiles/rails_5.2.gemfile.lock +26 -9
  18. data/gemfiles/rails_6.0.gemfile.lock +27 -10
  19. data/lib/cloudtasker.rb +0 -1
  20. data/lib/cloudtasker/backend/google_cloud_task.rb +65 -12
  21. data/lib/cloudtasker/backend/memory_task.rb +5 -3
  22. data/lib/cloudtasker/backend/redis_task.rb +24 -13
  23. data/lib/cloudtasker/batch/batch_progress.rb +11 -2
  24. data/lib/cloudtasker/batch/job.rb +18 -4
  25. data/lib/cloudtasker/cli.rb +6 -5
  26. data/lib/cloudtasker/cloud_task.rb +4 -2
  27. data/lib/cloudtasker/config.rb +30 -9
  28. data/lib/cloudtasker/cron/job.rb +2 -2
  29. data/lib/cloudtasker/cron/schedule.rb +26 -14
  30. data/lib/cloudtasker/local_server.rb +44 -22
  31. data/lib/cloudtasker/redis_client.rb +10 -7
  32. data/lib/cloudtasker/unique_job/job.rb +2 -2
  33. data/lib/cloudtasker/version.rb +1 -1
  34. data/lib/cloudtasker/worker.rb +46 -10
  35. data/lib/cloudtasker/worker_handler.rb +7 -5
  36. data/lib/cloudtasker/worker_logger.rb +1 -1
  37. data/lib/cloudtasker/worker_wrapper.rb +52 -0
  38. data/lib/tasks/setup_queue.rake +12 -2
  39. metadata +6 -6
  40. data/Gemfile.lock +0 -280
  41. data/lib/cloudtasker/railtie.rb +0 -10
data/docs/BATCH_JOBS.md CHANGED
@@ -2,14 +2,14 @@
2
2
 
3
3
  **Note**: this extension requires redis
4
4
 
5
- The Cloudtasker batch job extension allows to add sub-jobs to regular jobs. This adds the ability to enqueue a list of jobs and track their overall progression as a groupd of jobs (the batch). This extension allows jobs to define callbacks in their worker to track completion of the batch and take action based on that.
5
+ The Cloudtasker batch job extension allows to add sub-jobs to regular jobs. This adds the ability to enqueue a list of jobs and track their overall progression as a group of jobs (a "batch"). This extension allows jobs to define callbacks in their worker to track completion of the batch and take actions based on that.
6
6
 
7
7
  ## Configuration
8
8
 
9
9
  You can enable batch jobs by adding the following to your cloudtasker initializer:
10
10
  ```ruby
11
11
  # The batch job extension is optional and must be explicitly required
12
- require 'cloudtasker/batch_job'
12
+ require 'cloudtasker/batch'
13
13
 
14
14
  Cloudtasker.configure do |config|
15
15
  # Specify your redis url.
@@ -53,14 +53,39 @@ The following callbacks are available on your workers to track the progress of t
53
53
 
54
54
  | Callback | Argument | Description |
55
55
  |------|-------------|-----------|
56
- | `on_batch_node_complete` | `The child job` | Invoked when any descendant (e.g. sub-sub job) successfully completess |
56
+ | `on_batch_node_complete` | `The child job` | Invoked when any descendant (e.g. sub-sub job) successfully completes |
57
57
  | `on_child_complete` | `The child job` | Invoked when a direct descendant successfully completes |
58
58
  | `on_child_error` | `The child job` | Invoked when a child fails |
59
59
  | `on_child_dead` | `The child job` | Invoked when a child has exhausted all of its retries |s
60
60
  | `on_batch_complete` | none | Invoked when all chidren have finished or died |
61
61
 
62
+ ## Queue management
63
+
64
+ Jobs added to a batch inherit the queue of the parent. It is possible to specify a different queue when adding a job to a batch using `add_to_queue` batch method.
65
+
66
+ E.g.
67
+
68
+ ```ruby
69
+ def perform
70
+ batch.add_to_queue(:critical, SubWorker, arg1, arg2, arg3)
71
+ end
72
+ ```
73
+
62
74
  ## Batch completion
63
75
 
64
76
  Batches complete when all children have successfully completed or died (all retries exhausted).
65
77
 
66
- Jobs that fail in a batch will be retried based on the `max_retries` setting configured globally or on the worker itself. The batch will be considered `pending` while workers retry. Therefore it may be a good idea to reduce the number of retries on your workers using `cloudtasker_options max_retries: 5` to ensure your batches don't hang for too long.
78
+ Jobs that fail in a batch will be retried based on the `max_retries` setting configured globally or on the worker itself. The batch will be considered `pending` while workers retry. Therefore it may be a good idea to reduce the number of retries on your workers using `cloudtasker_options max_retries: 5` to ensure your batches don't hang for too long.
79
+
80
+ ## Batch progress tracking
81
+
82
+ You can access progression statistics in callback using `batch.progress`. See the [BatchProgress](../lib/cloudtasker/batch/batch_progress.rb) class for more details.
83
+
84
+ E.g.
85
+ ```ruby
86
+ def on_batch_node_complete(_child_job)
87
+ logger.info("Total: #{batch.progress.total}")
88
+ logger.info("Completed: #{batch.progress.completed}")
89
+ logger.info("Progress: #{batch.progress.percent.to_i}%")
90
+ end
91
+ ```
data/docs/CRON_JOBS.md CHANGED
@@ -9,7 +9,7 @@ The Cloudtasker cron job extension allows you to register workers to run at fixe
9
9
  You can schedule cron jobs by adding the following to your cloudtasker initializer:
10
10
  ```ruby
11
11
  # The cron job extension is optional and must be explicitly required
12
- require 'cloudtasker/cron_job'
12
+ require 'cloudtasker/cron'
13
13
 
14
14
  Cloudtasker.configure do |config|
15
15
  # Specify your redis url.
@@ -18,18 +18,22 @@ Cloudtasker.configure do |config|
18
18
  end
19
19
 
20
20
  # Specify all your cron jobs below. This will synchronize your list of cron jobs (cron jobs previously created and not listed below will be removed).
21
- Cloudtasker::Cron::Schedule.load_from_hash!(
22
- # Run job every minute
23
- some_schedule_name: {
24
- worker: 'SomeCronWorker',
25
- cron: '* * * * *'
26
- },
27
- # Run job every hour on the fifteenth minute
28
- other_cron_schedule: {
29
- worker: 'OtherCronWorker',
30
- cron: '15 * * * *'
31
- }
32
- )
21
+ unless Rails.env.test?
22
+ Cloudtasker::Cron::Schedule.load_from_hash!(
23
+ # Run job every minute
24
+ some_schedule_name: {
25
+ worker: 'SomeCronWorker',
26
+ cron: '* * * * *'
27
+ },
28
+ # Run job every hour on the fifteenth minute
29
+ other_cron_schedule: {
30
+ worker: 'OtherCronWorker',
31
+ cron: '15 * * * *',
32
+ queue: 'critical'
33
+ args: ['foo', 'bar']
34
+ }
35
+ )
36
+ end
33
37
  ```
34
38
 
35
39
  ## Using a configuration file
@@ -56,7 +60,7 @@ Then register the jobs inside your Cloudtasker initializer this way:
56
60
  # ... Cloudtasker configuration ...
57
61
 
58
62
  schedule_file = 'config/cloudtasker_cron.yml'
59
- if File.exist?(schedule_file)
63
+ if File.exist?(schedule_file) && !Rails.env.test?
60
64
  Cloudtasker::Cron::Schedule.load_from_hash!(YAML.load_file(schedule_file))
61
65
  end
62
66
  ```
data/exe/cloudtasker CHANGED
@@ -3,9 +3,21 @@
3
3
 
4
4
  require 'bundler/setup'
5
5
  require 'cloudtasker/cli'
6
+ require 'optparse'
7
+
8
+ options = {}
9
+ OptionParser.new do |opts|
10
+ opts.banner = 'Usage: cloudtasker [options]'
11
+
12
+ opts.on('-q QUEUE', '--queue=QUEUE', 'Queue to process and number of threads. ' \
13
+ "Examples: '-q critical' | '-q critical,2' | '-q critical,3 -q defaults,2'") do |o|
14
+ options[:queues] ||= []
15
+ options[:queues] << o.split(',')
16
+ end
17
+ end.parse!
6
18
 
7
19
  begin
8
- Cloudtasker::CLI.run
20
+ Cloudtasker::CLI.run(options)
9
21
  rescue StandardError => e
10
22
  raise e if $DEBUG
11
23
 
data/gemfiles/google_cloud_tasks_1.0.gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: ..
3
3
  specs:
4
- cloudtasker (0.2.0)
4
+ cloudtasker (0.9.0)
5
5
  activesupport
6
6
  fugit
7
7
  google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
82
82
  erubi (1.9.0)
83
83
  et-orbi (1.2.2)
84
84
  tzinfo
85
- faraday (0.17.0)
85
+ faraday (0.17.1)
86
86
  multipart-post (>= 1.2, < 3)
87
+ faraday-http-cache (2.0.0)
88
+ faraday (~> 0.8)
87
89
  fugit (1.3.3)
88
90
  et-orbi (~> 1.1, >= 1.1.8)
89
91
  raabro (~> 1.1)
92
+ github_changelog_generator (1.15.0)
93
+ activesupport
94
+ faraday-http-cache
95
+ multi_json
96
+ octokit (~> 4.6)
97
+ rainbow (>= 2.2.1)
98
+ rake (>= 10.0)
99
+ retriable (~> 3.0)
90
100
  globalid (0.4.2)
91
101
  activesupport (>= 4.2.0)
92
102
  google-cloud-tasks (1.0.0)
@@ -99,7 +109,7 @@ GEM
99
109
  googleauth (~> 0.9)
100
110
  grpc (~> 1.24)
101
111
  rly (~> 0.2.3)
102
- google-protobuf (3.10.1)
112
+ google-protobuf (3.11.1)
103
113
  googleapis-common-protos (1.3.9)
104
114
  google-protobuf (~> 3.0)
105
115
  googleapis-common-protos-types (~> 1.0)
@@ -124,14 +134,14 @@ GEM
124
134
  concurrent-ruby (~> 1.0)
125
135
  jaro_winkler (1.5.4)
126
136
  jwt (2.2.1)
127
- loofah (2.3.1)
137
+ loofah (2.4.0)
128
138
  crass (~> 1.0.2)
129
139
  nokogiri (>= 1.5.9)
130
140
  mail (2.7.1)
131
141
  mini_mime (>= 0.1.1)
132
142
  marcel (0.3.3)
133
143
  mimemagic (~> 0.3.2)
134
- memoist (0.16.1)
144
+ memoist (0.16.2)
135
145
  method_source (0.9.2)
136
146
  mimemagic (0.3.3)
137
147
  mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
140
150
  multi_json (1.14.1)
141
151
  multipart-post (2.1.1)
142
152
  nio4r (2.5.2)
143
- nokogiri (1.10.5)
153
+ nokogiri (1.10.7)
144
154
  mini_portile2 (~> 2.4.0)
155
+ octokit (4.14.0)
156
+ sawyer (~> 0.8.0, >= 0.5.3)
145
157
  os (1.0.1)
146
- parallel (1.19.0)
158
+ parallel (1.19.1)
147
159
  parser (2.6.5.0)
148
160
  ast (~> 2.4.0)
149
161
  public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
180
192
  rainbow (3.0.0)
181
193
  rake (10.5.0)
182
194
  redis (4.1.3)
195
+ retriable (3.1.2)
183
196
  rly (0.2.3)
184
197
  rspec (3.9.0)
185
198
  rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
209
222
  rainbow (>= 2.2.2, < 4.0)
210
223
  ruby-progressbar (~> 1.7)
211
224
  unicode-display_width (>= 1.4.0, < 1.7)
212
- rubocop-rspec (1.36.0)
225
+ rubocop-rspec (1.37.0)
213
226
  rubocop (>= 0.68.1)
214
227
  ruby-progressbar (1.10.1)
215
228
  safe_yaml (1.0.5)
229
+ sawyer (0.8.2)
230
+ addressable (>= 2.3.5)
231
+ faraday (> 0.8, < 2.0)
216
232
  signet (0.12.0)
217
233
  addressable (~> 2.3)
218
234
  faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
239
255
  websocket-driver (0.7.1)
240
256
  websocket-extensions (>= 0.1.0)
241
257
  websocket-extensions (0.1.4)
242
- zeitwerk (2.2.1)
258
+ zeitwerk (2.2.2)
243
259
 
244
260
  PLATFORMS
245
261
  ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
248
264
  appraisal
249
265
  bundler (~> 2.0)
250
266
  cloudtasker!
267
+ github_changelog_generator
251
268
  google-cloud-tasks (= 1.0)
252
269
  rails
253
270
  rake (~> 10.0)
data/gemfiles/google_cloud_tasks_1.1.gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: ..
3
3
  specs:
4
- cloudtasker (0.2.0)
4
+ cloudtasker (0.9.0)
5
5
  activesupport
6
6
  fugit
7
7
  google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
82
82
  erubi (1.9.0)
83
83
  et-orbi (1.2.2)
84
84
  tzinfo
85
- faraday (0.17.0)
85
+ faraday (0.17.1)
86
86
  multipart-post (>= 1.2, < 3)
87
+ faraday-http-cache (2.0.0)
88
+ faraday (~> 0.8)
87
89
  fugit (1.3.3)
88
90
  et-orbi (~> 1.1, >= 1.1.8)
89
91
  raabro (~> 1.1)
92
+ github_changelog_generator (1.15.0)
93
+ activesupport
94
+ faraday-http-cache
95
+ multi_json
96
+ octokit (~> 4.6)
97
+ rainbow (>= 2.2.1)
98
+ rake (>= 10.0)
99
+ retriable (~> 3.0)
90
100
  globalid (0.4.2)
91
101
  activesupport (>= 4.2.0)
92
102
  google-cloud-tasks (1.1.0)
@@ -99,7 +109,7 @@ GEM
99
109
  googleauth (~> 0.9)
100
110
  grpc (~> 1.24)
101
111
  rly (~> 0.2.3)
102
- google-protobuf (3.10.1)
112
+ google-protobuf (3.11.1)
103
113
  googleapis-common-protos (1.3.9)
104
114
  google-protobuf (~> 3.0)
105
115
  googleapis-common-protos-types (~> 1.0)
@@ -124,14 +134,14 @@ GEM
124
134
  concurrent-ruby (~> 1.0)
125
135
  jaro_winkler (1.5.4)
126
136
  jwt (2.2.1)
127
- loofah (2.3.1)
137
+ loofah (2.4.0)
128
138
  crass (~> 1.0.2)
129
139
  nokogiri (>= 1.5.9)
130
140
  mail (2.7.1)
131
141
  mini_mime (>= 0.1.1)
132
142
  marcel (0.3.3)
133
143
  mimemagic (~> 0.3.2)
134
- memoist (0.16.1)
144
+ memoist (0.16.2)
135
145
  method_source (0.9.2)
136
146
  mimemagic (0.3.3)
137
147
  mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
140
150
  multi_json (1.14.1)
141
151
  multipart-post (2.1.1)
142
152
  nio4r (2.5.2)
143
- nokogiri (1.10.5)
153
+ nokogiri (1.10.7)
144
154
  mini_portile2 (~> 2.4.0)
155
+ octokit (4.14.0)
156
+ sawyer (~> 0.8.0, >= 0.5.3)
145
157
  os (1.0.1)
146
- parallel (1.19.0)
158
+ parallel (1.19.1)
147
159
  parser (2.6.5.0)
148
160
  ast (~> 2.4.0)
149
161
  public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
180
192
  rainbow (3.0.0)
181
193
  rake (10.5.0)
182
194
  redis (4.1.3)
195
+ retriable (3.1.2)
183
196
  rly (0.2.3)
184
197
  rspec (3.9.0)
185
198
  rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
209
222
  rainbow (>= 2.2.2, < 4.0)
210
223
  ruby-progressbar (~> 1.7)
211
224
  unicode-display_width (>= 1.4.0, < 1.7)
212
- rubocop-rspec (1.36.0)
225
+ rubocop-rspec (1.37.0)
213
226
  rubocop (>= 0.68.1)
214
227
  ruby-progressbar (1.10.1)
215
228
  safe_yaml (1.0.5)
229
+ sawyer (0.8.2)
230
+ addressable (>= 2.3.5)
231
+ faraday (> 0.8, < 2.0)
216
232
  signet (0.12.0)
217
233
  addressable (~> 2.3)
218
234
  faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
239
255
  websocket-driver (0.7.1)
240
256
  websocket-extensions (>= 0.1.0)
241
257
  websocket-extensions (0.1.4)
242
- zeitwerk (2.2.1)
258
+ zeitwerk (2.2.2)
243
259
 
244
260
  PLATFORMS
245
261
  ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
248
264
  appraisal
249
265
  bundler (~> 2.0)
250
266
  cloudtasker!
267
+ github_changelog_generator
251
268
  google-cloud-tasks (= 1.1)
252
269
  rails
253
270
  rake (~> 10.0)
data/gemfiles/google_cloud_tasks_1.2.gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: ..
3
3
  specs:
4
- cloudtasker (0.2.0)
4
+ cloudtasker (0.9.0)
5
5
  activesupport
6
6
  fugit
7
7
  google-cloud-tasks
@@ -82,11 +82,21 @@ GEM
82
82
  erubi (1.9.0)
83
83
  et-orbi (1.2.2)
84
84
  tzinfo
85
- faraday (0.17.0)
85
+ faraday (0.17.1)
86
86
  multipart-post (>= 1.2, < 3)
87
+ faraday-http-cache (2.0.0)
88
+ faraday (~> 0.8)
87
89
  fugit (1.3.3)
88
90
  et-orbi (~> 1.1, >= 1.1.8)
89
91
  raabro (~> 1.1)
92
+ github_changelog_generator (1.15.0)
93
+ activesupport
94
+ faraday-http-cache
95
+ multi_json
96
+ octokit (~> 4.6)
97
+ rainbow (>= 2.2.1)
98
+ rake (>= 10.0)
99
+ retriable (~> 3.0)
90
100
  globalid (0.4.2)
91
101
  activesupport (>= 4.2.0)
92
102
  google-cloud-tasks (1.2.0)
@@ -99,7 +109,7 @@ GEM
99
109
  googleauth (~> 0.9)
100
110
  grpc (~> 1.24)
101
111
  rly (~> 0.2.3)
102
- google-protobuf (3.10.1-universal-darwin)
112
+ google-protobuf (3.11.1)
103
113
  googleapis-common-protos (1.3.9)
104
114
  google-protobuf (~> 3.0)
105
115
  googleapis-common-protos-types (~> 1.0)
@@ -113,7 +123,7 @@ GEM
113
123
  multi_json (~> 1.11)
114
124
  os (>= 0.9, < 2.0)
115
125
  signet (~> 0.12)
116
- grpc (1.25.0-universal-darwin)
126
+ grpc (1.25.0)
117
127
  google-protobuf (~> 3.8)
118
128
  googleapis-common-protos-types (~> 1.0)
119
129
  grpc-google-iam-v1 (0.6.9)
@@ -124,14 +134,14 @@ GEM
124
134
  concurrent-ruby (~> 1.0)
125
135
  jaro_winkler (1.5.4)
126
136
  jwt (2.2.1)
127
- loofah (2.3.1)
137
+ loofah (2.4.0)
128
138
  crass (~> 1.0.2)
129
139
  nokogiri (>= 1.5.9)
130
140
  mail (2.7.1)
131
141
  mini_mime (>= 0.1.1)
132
142
  marcel (0.3.3)
133
143
  mimemagic (~> 0.3.2)
134
- memoist (0.16.1)
144
+ memoist (0.16.2)
135
145
  method_source (0.9.2)
136
146
  mimemagic (0.3.3)
137
147
  mini_mime (1.0.2)
@@ -140,10 +150,12 @@ GEM
140
150
  multi_json (1.14.1)
141
151
  multipart-post (2.1.1)
142
152
  nio4r (2.5.2)
143
- nokogiri (1.10.5)
153
+ nokogiri (1.10.7)
144
154
  mini_portile2 (~> 2.4.0)
155
+ octokit (4.14.0)
156
+ sawyer (~> 0.8.0, >= 0.5.3)
145
157
  os (1.0.1)
146
- parallel (1.19.0)
158
+ parallel (1.19.1)
147
159
  parser (2.6.5.0)
148
160
  ast (~> 2.4.0)
149
161
  public_suffix (4.0.1)
@@ -180,6 +192,7 @@ GEM
180
192
  rainbow (3.0.0)
181
193
  rake (10.5.0)
182
194
  redis (4.1.3)
195
+ retriable (3.1.2)
183
196
  rly (0.2.3)
184
197
  rspec (3.9.0)
185
198
  rspec-core (~> 3.9.0)
@@ -209,10 +222,13 @@ GEM
209
222
  rainbow (>= 2.2.2, < 4.0)
210
223
  ruby-progressbar (~> 1.7)
211
224
  unicode-display_width (>= 1.4.0, < 1.7)
212
- rubocop-rspec (1.36.0)
225
+ rubocop-rspec (1.37.0)
213
226
  rubocop (>= 0.68.1)
214
227
  ruby-progressbar (1.10.1)
215
228
  safe_yaml (1.0.5)
229
+ sawyer (0.8.2)
230
+ addressable (>= 2.3.5)
231
+ faraday (> 0.8, < 2.0)
216
232
  signet (0.12.0)
217
233
  addressable (~> 2.3)
218
234
  faraday (~> 0.9)
@@ -239,7 +255,7 @@ GEM
239
255
  websocket-driver (0.7.1)
240
256
  websocket-extensions (>= 0.1.0)
241
257
  websocket-extensions (0.1.4)
242
- zeitwerk (2.2.1)
258
+ zeitwerk (2.2.2)
243
259
 
244
260
  PLATFORMS
245
261
  ruby
@@ -248,6 +264,7 @@ DEPENDENCIES
248
264
  appraisal
249
265
  bundler (~> 2.0)
250
266
  cloudtasker!
267
+ github_changelog_generator
251
268
  google-cloud-tasks (= 1.2)
252
269
  rails
253
270
  rake (~> 10.0)