cloudtasker 0.5.0 → 0.8.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.gitignore +3 -0
- data/.rubocop.yml +5 -0
- data/.travis.yml +3 -3
- data/CHANGELOG.md +45 -0
- data/README.md +163 -26
- data/Rakefile +6 -0
- data/_config.yml +1 -0
- data/app/controllers/cloudtasker/worker_controller.rb +6 -6
- data/cloudtasker.gemspec +3 -2
- data/docs/BATCH_JOBS.md +29 -4
- data/docs/CRON_JOBS.md +18 -14
- data/exe/cloudtasker +13 -1
- data/gemfiles/google_cloud_tasks_1.0.gemfile.lock +26 -9
- data/gemfiles/google_cloud_tasks_1.1.gemfile.lock +26 -9
- data/gemfiles/google_cloud_tasks_1.2.gemfile.lock +27 -10
- data/gemfiles/google_cloud_tasks_1.3.gemfile.lock +26 -9
- data/gemfiles/rails_5.2.gemfile.lock +25 -8
- data/gemfiles/rails_6.0.gemfile.lock +27 -10
- data/lib/cloudtasker.rb +0 -1
- data/lib/cloudtasker/backend/google_cloud_task.rb +41 -8
- data/lib/cloudtasker/backend/memory_task.rb +5 -3
- data/lib/cloudtasker/backend/redis_task.rb +24 -13
- data/lib/cloudtasker/batch/batch_progress.rb +11 -2
- data/lib/cloudtasker/batch/job.rb +18 -4
- data/lib/cloudtasker/cli.rb +6 -5
- data/lib/cloudtasker/cloud_task.rb +4 -2
- data/lib/cloudtasker/config.rb +21 -9
- data/lib/cloudtasker/cron/job.rb +2 -2
- data/lib/cloudtasker/cron/schedule.rb +26 -14
- data/lib/cloudtasker/local_server.rb +44 -22
- data/lib/cloudtasker/redis_client.rb +5 -6
- data/lib/cloudtasker/unique_job/job.rb +2 -2
- data/lib/cloudtasker/version.rb +1 -1
- data/lib/cloudtasker/worker.rb +46 -10
- data/lib/cloudtasker/worker_handler.rb +5 -3
- data/lib/cloudtasker/worker_logger.rb +1 -1
- data/lib/cloudtasker/worker_wrapper.rb +52 -0
- data/lib/tasks/setup_queue.rake +12 -2
- metadata +21 -6
- data/Gemfile.lock +0 -263
- data/lib/cloudtasker/railtie.rb +0 -10
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c4c3ce66208b1b45f5ec361f098ac3ceb1db562f2b4af45b3d42ef6ce54b6aac
|
4
|
+
data.tar.gz: 52ed1b155fea5ce6c307498f058b4b57d115b97e73ac7d0e1ab685c10c62672d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: a3a4276e4494fc8912480a06cfe81f283340b516ec45cc9a504ffe02c733b42c3dfe3e14e9d132aacf93ff5d4f25534459d85467426497bc46383fbfc4c7fa08
|
7
|
+
data.tar.gz: ce32054e9e9f6fe496bcd3c36b6e15d012720834f71c237da1cbe82119a6a09d23c4b1463c11044261e890f3e1262fc94ed6dd42e86f72bfb340f79fed6cbbba
|
data/.gitignore
CHANGED
data/.rubocop.yml
CHANGED
data/.travis.yml
CHANGED
@@ -2,11 +2,11 @@
|
|
2
2
|
language: ruby
|
3
3
|
cache: bundler
|
4
4
|
rvm:
|
5
|
-
- 2.3
|
6
|
-
- 2.4
|
7
5
|
- 2.5.5
|
6
|
+
services:
|
7
|
+
- redis-server
|
8
8
|
before_install: gem install bundler -v 2.0.2
|
9
|
-
before_script: rubocop
|
9
|
+
before_script: bundle exec rubocop
|
10
10
|
gemfile:
|
11
11
|
- gemfiles/google_cloud_tasks_1.0.gemfile
|
12
12
|
- gemfiles/google_cloud_tasks_1.1.gemfile
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,45 @@
|
|
1
|
+
# Changelog
|
2
|
+
|
3
|
+
## [v0.8.1](https://github.com/keypup-io/cloudtasker/tree/v0.8.1) (2019-12-03)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.8.0...v0.8.1)
|
6
|
+
|
7
|
+
**Fixed bugs:**
|
8
|
+
- Local dev server: ensure job queue name is kept when taks is retried
|
9
|
+
- Rails/Controller: bypass Rails munge logic to preserve nil values inside job arguments.
|
10
|
+
|
11
|
+
## [v0.8.0](https://github.com/keypup-io/cloudtasker/tree/v0.8.0) (2019-11-27)
|
12
|
+
|
13
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.7.0...v0.8.0)
|
14
|
+
|
15
|
+
## [v0.7.0](https://github.com/keypup-io/cloudtasker/tree/v0.7.0) (2019-11-25)
|
16
|
+
|
17
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.6.0...v0.7.0)
|
18
|
+
|
19
|
+
## [v0.6.0](https://github.com/keypup-io/cloudtasker/tree/v0.6.0) (2019-11-25)
|
20
|
+
|
21
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.5.0...v0.6.0)
|
22
|
+
|
23
|
+
## [v0.5.0](https://github.com/keypup-io/cloudtasker/tree/v0.5.0) (2019-11-25)
|
24
|
+
|
25
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.4.0...v0.5.0)
|
26
|
+
|
27
|
+
## [v0.4.0](https://github.com/keypup-io/cloudtasker/tree/v0.4.0) (2019-11-25)
|
28
|
+
|
29
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.3.0...v0.4.0)
|
30
|
+
|
31
|
+
## [v0.3.0](https://github.com/keypup-io/cloudtasker/tree/v0.3.0) (2019-11-25)
|
32
|
+
|
33
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.2.0...v0.3.0)
|
34
|
+
|
35
|
+
## [v0.2.0](https://github.com/keypup-io/cloudtasker/tree/v0.2.0) (2019-11-18)
|
36
|
+
|
37
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.1.0...v0.2.0)
|
38
|
+
|
39
|
+
## [v0.1.0](https://github.com/keypup-io/cloudtasker/tree/v0.1.0) (2019-11-17)
|
40
|
+
|
41
|
+
[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/c137feb1ceaaaa4e2fecac0d1f0b4c73151ae002...v0.1.0)
|
42
|
+
|
43
|
+
|
44
|
+
|
45
|
+
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
|
data/README.md
CHANGED
@@ -1,14 +1,40 @@
|
|
1
|
+
[![Build Status](https://travis-ci.org/keypup-io/cloudtasker.svg?branch=master)](https://travis-ci.org/keypup-io/cloudtasker) [![Gem Version](https://badge.fury.io/rb/cloudtasker.svg)](https://badge.fury.io/rb/cloudtasker)
|
2
|
+
|
1
3
|
# Cloudtasker
|
2
4
|
|
3
5
|
Background jobs for Ruby using Google Cloud Tasks.
|
4
6
|
|
5
7
|
Cloudtasker provides an easy to manage interface to Google Cloud Tasks for background job processing. Workers can be defined programmatically using the Cloudtasker DSL and enqueued for processing using a simple to use API.
|
6
8
|
|
7
|
-
Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually
|
9
|
+
Cloudtasker is particularly suited for serverless applications only responding to HTTP requests and where running a dedicated job processing is not an option (e.g. deploy via [Cloud Run](https://cloud.google.com/run)). All jobs enqueued in Cloud Tasks via Cloudtasker eventually get processed by your application via HTTP requests.
|
8
10
|
|
9
11
|
Cloudtasker also provides optional modules for running [cron jobs](docs/CRON_JOBS.md), [batch jobs](docs/BATCH_JOBS.md) and [unique jobs](docs/UNIQUE_JOBS.md).
|
10
12
|
|
11
|
-
A local processing server is also available in development. This local server processes jobs in lieu of Cloud Tasks and
|
13
|
+
A local processing server is also available in development. This local server processes jobs in lieu of Cloud Tasks and allows you to work offline.
|
14
|
+
|
15
|
+
## Summary
|
16
|
+
|
17
|
+
1. [Installation](#installation)
|
18
|
+
2. [Get started with Rails](#get-started-with-rails)
|
19
|
+
3. [Configuring Cloudtasker](#configuring-cloudtasker)
|
20
|
+
1. [Cloud Tasks authentication & permissions](#cloud-tasks-authentication--permissions)
|
21
|
+
2. [Cloudtasker initializer](#cloudtasker-initializer)
|
22
|
+
4. [Enqueuing jobs](#enqueuing-jobs)
|
23
|
+
5. [Managing worker queues](#managing-worker-queues)
|
24
|
+
1. [Creating queues](#creating-queues)
|
25
|
+
2. [Assigning queues to workers](#assigning-queues-to-workers)
|
26
|
+
6. [Extensions](#extensions)
|
27
|
+
7. [Working locally](#working-locally)
|
28
|
+
1. [Option 1: Cloudtasker local server](#option-1-cloudtasker-local-server)
|
29
|
+
2. [Option 2: Using ngrok](#option-2-using-ngrok)
|
30
|
+
8. [Logging](#logging)
|
31
|
+
1. [Configuring a logger](#configuring-a-logger)
|
32
|
+
2. [Logging context](#logging-context)
|
33
|
+
9. [Error Handling](#error-handling)
|
34
|
+
1. [HTTP Error codes](#http-error-codes)
|
35
|
+
2. [Error callbacks](#error-callbacks)
|
36
|
+
3. [Max retries](#max-retries)
|
37
|
+
10. [Best practices building workers](#best-practices-building-workers)
|
12
38
|
|
13
39
|
## Installation
|
14
40
|
|
@@ -77,13 +103,13 @@ Launch Rails and the local Cloudtasker processing server (or add `cloudtasker` t
|
|
77
103
|
> cloudtasker
|
78
104
|
```
|
79
105
|
|
80
|
-
Open a Rails console and enqueue
|
106
|
+
Open a Rails console and enqueue some jobs
|
81
107
|
```ruby
|
82
108
|
# Process job as soon as possible
|
83
109
|
DummyWorker.perform_async('foo')
|
84
110
|
|
85
111
|
# Process job in 60 seconds
|
86
|
-
DummyWorker.perform_in(
|
112
|
+
DummyWorker.perform_in(60, 'foo')
|
87
113
|
```
|
88
114
|
|
89
115
|
Your Rails logs should display the following:
|
@@ -136,13 +162,31 @@ Cloudtasker.configure do |config|
|
|
136
162
|
# config.secret = 'some-long-token'
|
137
163
|
|
138
164
|
#
|
139
|
-
# Specify the details of your Google Cloud Task
|
165
|
+
# Specify the details of your Google Cloud Task location.
|
140
166
|
#
|
141
167
|
# This not required in development using the Cloudtasker local server.
|
142
168
|
#
|
143
169
|
config.gcp_location_id = 'us-central1' # defaults to 'us-east1'
|
144
170
|
config.gcp_project_id = 'my-gcp-project'
|
145
|
-
|
171
|
+
|
172
|
+
#
|
173
|
+
# Specify the namespace for your Cloud Task queues.
|
174
|
+
#
|
175
|
+
# The gem assumes that a least a default queue named 'my-app-default'
|
176
|
+
# exists in Cloud Tasks. You can create this default queue using the
|
177
|
+
# gcloud SDK or via the `rake cloudtasker:setup_queue` task if you use Rails.
|
178
|
+
#
|
179
|
+
# Workers can be scheduled on different queues. The name of the queue
|
180
|
+
# in Cloud Tasks is always assumed to be prefixed with the prefix below.
|
181
|
+
#
|
182
|
+
# E.g.
|
183
|
+
# Setting `cloudtasker_options queue: 'critical'` on a worker means that
|
184
|
+
# the worker will be pushed to 'my-app-critical' in Cloud Tasks.
|
185
|
+
#
|
186
|
+
# Specific queues can be created in Cloud Tasks using the gcloud SDK or
|
187
|
+
# via the `rake cloudtasker:setup_queue name=<queue_name>` task.
|
188
|
+
#
|
189
|
+
config.gcp_queue_prefix = 'my-app'
|
146
190
|
|
147
191
|
#
|
148
192
|
# Specify the publicly accessible host for your application
|
@@ -194,7 +238,7 @@ Cloudtasker.configure do |config|
|
|
194
238
|
end
|
195
239
|
```
|
196
240
|
|
197
|
-
If
|
241
|
+
If the default queue `<gcp_queue_prefix>-default` does not exist in Cloud Tasks you should [create it using the gcloud sdk](https://cloud.google.com/tasks/docs/creating-queues).
|
198
242
|
|
199
243
|
Alternatively with Rails you can simply run the following rake task if you have queue admin permissions (`cloudtasks.queues.get` and `cloudtasks.queues.create`).
|
200
244
|
```bash
|
@@ -214,10 +258,15 @@ MyWorker.perform_in(5 * 60, arg1, arg2)
|
|
214
258
|
# or with Rails
|
215
259
|
MyWorker.perform_in(5.minutes, arg1, arg2)
|
216
260
|
|
217
|
-
# Worker will be processed on specific date
|
261
|
+
# Worker will be processed on a specific date
|
218
262
|
MyWorker.perform_at(Time.parse('2025-01-01 00:50:00Z'), arg1, arg2)
|
219
263
|
# also with Rails
|
220
264
|
MyWorker.perform_at(3.days.from_now, arg1, arg2)
|
265
|
+
|
266
|
+
# With all options, including which queue to run the worker on.
|
267
|
+
MyWorker.schedule(args: [arg1, arg2], time_at: Time.parse('2025-01-01 00:50:00Z'), queue: 'critical')
|
268
|
+
# or
|
269
|
+
MyWorker.schedule(args: [arg1, arg2], time_in: 5 * 60, queue: 'critical')
|
221
270
|
```
|
222
271
|
|
223
272
|
Cloudtasker also provides a helper for re-enqueuing jobs. Re-enqueued jobs keep the same worker id. Some middlewares may rely on this to track the fact that that a job didn't actually complete (e.g. Cloustasker batch). This is optional and you can always fallback to using exception management (raise an error) to retry/re-enqueue jobs.
|
@@ -241,6 +290,52 @@ class FetchResourceWorker
|
|
241
290
|
end
|
242
291
|
```
|
243
292
|
|
293
|
+
## Managing worker queues
|
294
|
+
|
295
|
+
Cloudtasker allows you to manage several queues and distribute workers across them based on job priority. By default jobs are pushed to the `default` queue, which is `<gcp_queue_prefix>-default` in Cloud Tasks.
|
296
|
+
|
297
|
+
### Creating queues
|
298
|
+
|
299
|
+
More queues can be created using the gcloud sdk or the `cloudtasker:setup_queue` rake task.
|
300
|
+
|
301
|
+
E.g. Create a `critical` queue with a concurrency of 5 via the gcloud SDK
|
302
|
+
```bash
|
303
|
+
gcloud tasks queues create <gcp_queue_prefix>-critical --max-concurrent-dispatches=5
|
304
|
+
```
|
305
|
+
|
306
|
+
E.g. Create a `real-time` queue with a concurrency of 15 via the rake task (Rails only)
|
307
|
+
```bash
|
308
|
+
rake cloudtasker:setup_queue name=real-time concurrency=15
|
309
|
+
```
|
310
|
+
|
311
|
+
When running the Cloudtasker local processing server, you can specify the concurrency for each queue using:
|
312
|
+
```bash
|
313
|
+
cloudtasker -q critical,5 -q important,4 -q default,3
|
314
|
+
```
|
315
|
+
|
316
|
+
### Assigning queues to workers
|
317
|
+
|
318
|
+
Queues can be assigned to workers via the `cloudtasker_options` directive on the worker class:
|
319
|
+
|
320
|
+
```ruby
|
321
|
+
# app/workers/critical_worker.rb
|
322
|
+
|
323
|
+
class CriticalWorker
|
324
|
+
include Cloudtasker::Worker
|
325
|
+
|
326
|
+
cloudtasker_options queue: :critical
|
327
|
+
|
328
|
+
def perform(some_arg)
|
329
|
+
logger.info("This is a critical job run with arg=#{some_arg}.")
|
330
|
+
end
|
331
|
+
end
|
332
|
+
```
|
333
|
+
|
334
|
+
Queues can also be assigned at runtime when scheduling a job:
|
335
|
+
```ruby
|
336
|
+
CriticalWorker.schedule(args: [1], queue: :important)
|
337
|
+
```
|
338
|
+
|
244
339
|
## Extensions
|
245
340
|
Cloudtasker comes with three optional features:
|
246
341
|
- Cron Jobs [[docs](docs/CRON_JOBS.md)]: Run jobs at fixed intervals.
|
@@ -256,7 +351,7 @@ When working locally on your application it is usually not possible to have a pu
|
|
256
351
|
### Option 1: Cloudtasker local server
|
257
352
|
The Cloudtasker local server is a ruby daemon that looks for jobs pushed to Redis and sends them to your application via HTTP POST requests. The server mimics the way Google Cloud Tasks works, but locally!
|
258
353
|
|
259
|
-
You can configure your
|
354
|
+
You can configure your application to use the Cloudtasker local server using the following initializer:
|
260
355
|
```ruby
|
261
356
|
# config/initializers/cloudtasker.rb
|
262
357
|
|
@@ -272,25 +367,28 @@ end
|
|
272
367
|
|
273
368
|
The Cloudtasker server can then be started using:
|
274
369
|
```bash
|
275
|
-
cloudtasker
|
276
|
-
# or
|
277
370
|
bundle exec cloudtasker
|
278
371
|
```
|
279
372
|
|
280
373
|
You can as well define a Procfile to manage the cloudtasker process via foreman. Then use `foreman start` to launch both your Rails server and the Cloudtasker local server.
|
281
374
|
```yaml
|
282
375
|
# Procfile
|
283
|
-
web: rails s
|
284
|
-
worker: cloudtasker
|
376
|
+
web: bundle exec rails s
|
377
|
+
worker: bundle exec cloudtasker
|
378
|
+
```
|
379
|
+
|
380
|
+
Note that the local development server runs with `5` concurrent threads by default. You can tune the number of threads per queue by running `cloudtasker` the following options:
|
381
|
+
```bash
|
382
|
+
bundle exec cloudtasker -q critical,5 -q important,4 -q default,3
|
285
383
|
```
|
286
384
|
|
287
385
|
### Option 2: Using ngrok
|
288
386
|
|
289
387
|
Want to test your application end to end with Google Cloud Task? Then [ngrok](https://ngrok.io) is the way to go.
|
290
388
|
|
291
|
-
First start your ngrok tunnel
|
389
|
+
First start your ngrok tunnel:
|
292
390
|
```bash
|
293
|
-
ngrok
|
391
|
+
ngrok http 3000
|
294
392
|
```
|
295
393
|
|
296
394
|
Take note of your ngrok domain and configure Cloudtasker to use Google Cloud Task in development via ngrok.
|
@@ -299,9 +397,9 @@ Take note of your ngrok domain and configure Cloudtasker to use Google Cloud Tas
|
|
299
397
|
|
300
398
|
Cloudtasker.configure do |config|
|
301
399
|
# Specify your Google Cloud Task queue configuration
|
302
|
-
|
303
|
-
|
304
|
-
|
400
|
+
config.gcp_location_id = 'us-central1'
|
401
|
+
config.gcp_project_id = 'my-gcp-project'
|
402
|
+
config.gcp_queue_prefix = 'my-app'
|
305
403
|
|
306
404
|
# Use your ngrok domain as the processor host
|
307
405
|
config.processor_host = 'https://your-tunnel-id.ngrok.io'
|
@@ -313,7 +411,7 @@ end
|
|
313
411
|
|
314
412
|
Finally start Rails to accept jobs from Google Cloud Tasks
|
315
413
|
```bash
|
316
|
-
rails s
|
414
|
+
bundle exec rails s
|
317
415
|
```
|
318
416
|
|
319
417
|
## Logging
|
@@ -322,7 +420,7 @@ There are several options available to configure logging and logging context.
|
|
322
420
|
### Configuring a logger
|
323
421
|
Cloudtasker uses `Rails.logger` if Rails is available and falls back on a plain ruby logger `Logger.new(STDOUT)` if not.
|
324
422
|
|
325
|
-
It is also possible to configure your own logger. For example you can setup Cloudtasker with [semantic_logger](http://rocketjob.github.io/semantic_logger) by doing the following your initializer:
|
423
|
+
It is also possible to configure your own logger. For example you can setup Cloudtasker with [semantic_logger](http://rocketjob.github.io/semantic_logger) by doing the following in your initializer:
|
326
424
|
```ruby
|
327
425
|
# config/initializers/cloudtasker.rb
|
328
426
|
|
@@ -368,7 +466,7 @@ Cloudtasker::WorkerLogger.log_context_processor = lambda { |worker|
|
|
368
466
|
}
|
369
467
|
```
|
370
468
|
|
371
|
-
You could also decide to log all available context
|
469
|
+
You could also decide to log all available context - including arguments passed to `perform` - for specific workers only:
|
372
470
|
```ruby
|
373
471
|
# app/workers/full_context_worker.rb
|
374
472
|
|
@@ -383,7 +481,7 @@ class FullContextWorker
|
|
383
481
|
end
|
384
482
|
```
|
385
483
|
|
386
|
-
See the [Cloudtasker::Worker class](
|
484
|
+
See the [Cloudtasker::Worker class](lib/cloudtasker/worker.rb) for more information on attributes available to be logged in your `log_context_processor` proc.
|
387
485
|
|
388
486
|
## Error Handling
|
389
487
|
|
@@ -394,7 +492,7 @@ Jobs failing will automatically return an HTTP error to Cloud Task and trigger a
|
|
394
492
|
Jobs failing will automatically return the following HTTP error code to Cloud Tasks, based on the actual reason:
|
395
493
|
|
396
494
|
| Code | Description |
|
397
|
-
|
495
|
+
|------|-------------|
|
398
496
|
| 205 | The job is dead and has been removed from the queue |
|
399
497
|
| 404 | The job has specified an incorrect worker class. |
|
400
498
|
| 422 | An error happened during the execution of the worker (`perform` method) |
|
@@ -515,7 +613,7 @@ MyWorker.new.perform({ 'foo' => 'bar', 'baz' => { 'key' => 'value' } })
|
|
515
613
|
```
|
516
614
|
|
517
615
|
### Be careful with default arguments
|
518
|
-
Default arguments passed to the `perform` method are not actually considered as job arguments. Default arguments will therefore be ignored in contextual logging and by extensions relying on arguments such as the
|
616
|
+
Default arguments passed to the `perform` method are not actually considered as job arguments. Default arguments will therefore be ignored in contextual logging and by extensions relying on arguments such as the [unique job](docs/UNIQUE_JOBS.md) extension.
|
519
617
|
|
520
618
|
Consider the following worker:
|
521
619
|
```ruby
|
@@ -566,6 +664,45 @@ Rails.cache.write(payload_id, data)
|
|
566
664
|
BigPayloadWorker.perform_async(payload_id)
|
567
665
|
```
|
568
666
|
|
667
|
+
### Sizing the concurrency of your queues
|
668
|
+
|
669
|
+
When defining the max concurrency of your queues (`max_concurrent_dispatches` in Cloud Tasks) you must keep in mind the maximum number of threads that your application provides. Otherwise your application threads may eventually get exhausted and your users will experience outages if all your web threads are busy running jobs.
|
670
|
+
|
671
|
+
#### With server based applications
|
672
|
+
|
673
|
+
Let's consider an application deployed in production with 3 instances, each having `RAILS_MAX_THREADS` set to `20`. This gives us a total of `60` threads available.
|
674
|
+
|
675
|
+
Now let's say that we distribute jobs across two queues: `default` and `critical`. We can set the concurrency of each queue depending on the profile of the application:
|
676
|
+
|
677
|
+
E.g. 1: The application serves requests from web users and runs backgrounds jobs in a balanced way
|
678
|
+
```
|
679
|
+
concurrency for default queue: 20
|
680
|
+
concurrency for critical queue: 10
|
681
|
+
|
682
|
+
Total threads consumed by jobs at most: 30
|
683
|
+
Total threads always available to web users at worst: 30
|
684
|
+
```
|
685
|
+
|
686
|
+
E.g. 2: The application is a micro-service API heavily focused on running jobs (e.g. data processing)
|
687
|
+
```
|
688
|
+
concurrency for default queue: 35
|
689
|
+
concurrency for critical queue: 15
|
690
|
+
|
691
|
+
Total threads consumed by jobs at most: 50
|
692
|
+
Total threads always available to API clients at worst: 10
|
693
|
+
```
|
694
|
+
|
695
|
+
Also always ensure that your total number of threads does not exceed the available number of database connections (if you use any).
|
696
|
+
|
697
|
+
#### With serverless applications
|
698
|
+
|
699
|
+
In a serverless context your application will be scaled up/down based on traffic. When we say 'traffic' this includes requests from Cloud Tasks to run jobs.
|
700
|
+
|
701
|
+
Because your application is auto-scaled - and assuming you haven't set a maximum - your job processing capacity if theoretically unlimited. The main limiting factor in a serverless context becomes external constraints such as the number of database connections available.
|
702
|
+
|
703
|
+
To size the concurrency of your queues you should therefore take the most limiting factor - which is often the database connection pool size of relational databases - and use the calculations of the previous section with this limiting factor as the capping parameter instead of threads.
|
704
|
+
|
705
|
+
|
569
706
|
## Development
|
570
707
|
|
571
708
|
After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
@@ -574,7 +711,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
574
711
|
|
575
712
|
## Contributing
|
576
713
|
|
577
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
714
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/keypup-io/cloudtasker. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
578
715
|
|
579
716
|
## License
|
580
717
|
|
@@ -582,7 +719,7 @@ The gem is available as open source under the terms of the [MIT License](https:/
|
|
582
719
|
|
583
720
|
## Code of Conduct
|
584
721
|
|
585
|
-
Everyone interacting in the Cloudtasker project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/
|
722
|
+
Everyone interacting in the Cloudtasker project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/keypup-io/cloudtasker/blob/master/CODE_OF_CONDUCT.md).
|
586
723
|
|
587
724
|
## Author
|
588
725
|
|
data/Rakefile
CHANGED
@@ -2,7 +2,13 @@
|
|
2
2
|
|
3
3
|
require 'bundler/gem_tasks'
|
4
4
|
require 'rspec/core/rake_task'
|
5
|
+
require 'github_changelog_generator/task'
|
5
6
|
|
6
7
|
RSpec::Core::RakeTask.new(:spec)
|
7
8
|
|
8
9
|
task default: :spec
|
10
|
+
|
11
|
+
GitHubChangelogGenerator::RakeTask.new :changelog do |config|
|
12
|
+
config.user = 'keypup-io'
|
13
|
+
config.project = 'cloudtasker'
|
14
|
+
end
|