batches_task_processor 0.3.0 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +7 -6
- data/lib/batches_task_processor/processor.rb +1 -1
- data/lib/batches_task_processor/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 6b83bdfebcd837e7ee632a01502d5796ad29ad096e63486cd4d9513dd5882908
|
|
4
|
+
data.tar.gz: a8e2fbfafe6c76ca0ddab0810b24797dbc5ce3b6eceb84f15c2b587078a8d558
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a2aca1fb9e37d1c3c7e7af5e51cc593bcbbf1d83f37654bb81f9ef0cc45740a5d33467f3ead66a20f93fedf25f5ac1b81568a49019673ee0f47bf61bf3fd2629
|
|
7
|
+
data.tar.gz: 43d671b910781c69ac56bee5c363213c7590c162e5f17cf48882e31ea5dcc20ae7311186b30c5d9b9fdbc037a3e342097543b46d084112477c794057ccc89612
|
data/README.md
CHANGED
|
@@ -12,14 +12,14 @@ And then execute: `bundle install && bundle exec rake db:migrate`
|
|
|
12
12
|
|
|
13
13
|
## Usage
|
|
14
14
|
- Register a new task:
|
|
15
|
-
The following will process
|
|
15
|
+
The following will process 200k items with 10 jobs parallelly each one in charge of 20k items (recommended `preload_job_items` for performance reasons):
|
|
16
16
|
```ruby
|
|
17
17
|
task = BatchesTaskProcessor::Model.create!(
|
|
18
18
|
key: 'my_process',
|
|
19
|
-
data: Article.all.limit(
|
|
19
|
+
data: Article.all.limit(200000).pluck(:id),
|
|
20
20
|
qty_jobs: 10,
|
|
21
21
|
preload_job_items: 'Article.where(id: items)',
|
|
22
|
-
process_item: 'puts "my article: #{item.id}"'
|
|
22
|
+
process_item: 'puts "my article ID: #{item.id}"'
|
|
23
23
|
)
|
|
24
24
|
task.start!
|
|
25
25
|
```
|
|
@@ -27,10 +27,11 @@ And then execute: `bundle install && bundle exec rake db:migrate`
|
|
|
27
27
|
|
|
28
28
|
## Task api
|
|
29
29
|
- `task.start!` starts the task (initializes the jobs)
|
|
30
|
-
- `task.cancel` cancels the task and stops processing the items
|
|
30
|
+
- `task.cancel` cancels the task at any time and stops processing the items
|
|
31
31
|
- `task.export` exports the items that were processed in a csv file
|
|
32
|
-
- `task.
|
|
33
|
-
|
|
32
|
+
- `task.status` prints the current status of the task
|
|
33
|
+
- `task.items` returns the items that were processed so far
|
|
34
|
+
Each item includes the following attributes: `# { key: 'value from items', result: "value returned from the process_item callback", error_details: "error message from the process_message callback if failed" }`
|
|
34
35
|
|
|
35
36
|
## TODO
|
|
36
37
|
- update tests
|
|
@@ -70,7 +70,7 @@ module BatchesTaskProcessor
|
|
|
70
70
|
end
|
|
71
71
|
|
|
72
72
|
def start_process_item(item, job, key, index)
|
|
73
|
-
log "Processing #{job
|
|
73
|
+
log "Processing key: #{key}, job: #{job}, counter: #{index}/#{task_model.qty_items_job}"
|
|
74
74
|
result = process_item(item)
|
|
75
75
|
task_model.items.create!(key: key, result: result.to_s[0..255])
|
|
76
76
|
rescue => e
|