wor-batchifier 0.0.1 → 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
data/.rubocop.yml CHANGED
@@ -1,25 +1,25 @@
1
- # This is the configuration used to check the rubocop source code.
2
-
3
- AllCops:
4
- Exclude:
5
- - spec/spec_helper.rb
6
- - db/**/*
7
- - docs/**/*
8
- - config/**/*
9
- - bin/**/*
10
- - spec/**/**
11
-
12
- Documentation:
13
- Enabled: false
14
-
15
- LineLength:
16
- Max: 99
17
-
18
- HasAndBelongsToMany:
19
- Enabled: false
20
-
21
- Style/FrozenStringLiteralComment:
22
- Enabled: false
23
-
24
- Metrics/BlockLength:
25
- ExcludedMethods: ['register', 'describe', 'show', 'shared_examples', 'context', 'form', 'has_many', 'inputs']
1
+ # This is the configuration used to check the rubocop source code.
2
+
3
+ AllCops:
4
+ Exclude:
5
+ - spec/spec_helper.rb
6
+ - db/**/*
7
+ - docs/**/*
8
+ - config/**/*
9
+ - bin/**/*
10
+ - spec/**/**
11
+
12
+ Documentation:
13
+ Enabled: false
14
+
15
+ LineLength:
16
+ Max: 99
17
+
18
+ HasAndBelongsToMany:
19
+ Enabled: false
20
+
21
+ Style/FrozenStringLiteralComment:
22
+ Enabled: false
23
+
24
+ Metrics/BlockLength:
25
+ ExcludedMethods: ['register', 'describe', 'show', 'shared_examples', 'context', 'form', 'has_many', 'inputs']
data/Gemfile CHANGED
@@ -1,4 +1,4 @@
1
- source 'https://rubygems.org'
2
-
3
- # Specify your gem's dependencies in wor-batchifier.gemspec
4
- gemspec
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in wor-batchifier.gemspec
4
+ gemspec
data/README.md CHANGED
@@ -1,134 +1,134 @@
1
- # Wolox on Rails - Batchifier
2
- [![Gem Version](https://badge.fury.io/rb/wor-batchifier.svg)](https://badge.fury.io/rb/wor-batchifier)
3
- [![Dependency Status](https://gemnasium.com/badges/github.com/Wolox/wor-batchifier.svg)](https://gemnasium.com/github.com/Wolox/wor-batchifier)
4
- [![Build Status](https://travis-ci.org/Wolox/wor-batchifier.svg)](https://travis-ci.org/Wolox/wor-batchifier)
5
- [![Code Climate](https://codeclimate.com/github/Wolox/wor-batchifier/badges/gpa.svg)](https://codeclimate.com/github/Wolox/wor-authentication)
6
- [![Test Coverage](https://codeclimate.com/github/Wolox/wor-batchifier/badges/coverage.svg)](https://codeclimate.com/github/Wolox/wor-batchifier/coverage)
7
-
8
- Gem that allows you to easily divide requests that send a lot of information into several requests with smaller chunks of data, taking care of the response from each one and providing it joint together based on different strategies for parsing said response.
9
-
10
- ## Installation
11
-
12
- Add this line to your application's Gemfile:
13
-
14
- ```ruby
15
- gem 'wor-batchifier'
16
- ```
17
-
18
- And then execute:
19
-
20
- $ bundle
21
-
22
- Or install it yourself as:
23
-
24
- $ gem install wor-batchifier
25
-
26
- ## Usage
27
-
28
- ### Basic use:
29
-
30
- The first step is to define a parent controller from which all other controllers will have to extend to have access to the batchifier's methods. So, let's do that in our `ApplicationController.rb`:
31
-
32
- ```ruby
33
- class ApplicationController < ActionController::Base
34
- include Wor::Batchifier
35
- end
36
- ```
37
-
38
- You can also include the batchifier just in the controllers you intend to use it in.
39
-
40
- The final step, is to find any request you wish to perform with smaller chunks of data and utilize the batchifier's methods to divide it into smaller tasks.
41
-
42
- For example, let's pretend we have an endpoint called bulk_request that communicates with a third API and sends a lot of information to be utilized.
43
-
44
- ```ruby
45
- def bulk_request
46
- ThirdAPI.bulk_request(params[:information])
47
- end
48
- ```
49
-
50
- Now we will partition that request into chunks using the batchifier:
51
-
52
- ```ruby
53
- def bulk_request
54
- execute_in_batches(params[:information], batch_size: 100, strategy: :add) do |chunks|
55
- ThirdAPI.bulk_request(chunks)
56
- end
57
- end
58
- ```
59
-
60
- The batchifier will take three parameters, the first one being the information that needs to be partitioned, then the batch_size we wish to utilize and finally the symbol of the strategy that will be implemented on the response of each batch.
61
-
62
- ### Available strategies
63
-
64
- - Add: For each request, it joins together each response no matter the result.
65
- - Maintain-Unique: It will only add the results that are not present already in the response.
66
- - No-Response: It will not provide any response whatsoever.
67
-
68
- ### Adding new strategies
69
-
70
- Should you desire to add new strategies, it's as simple as creating a new class and defining a method called `merge_strategy` which will hold the logic that will be implemented to parse the response. Let's look at an example:
71
-
72
- ```ruby
73
- module Wor
74
- module Batchifier
75
- class MaintainUnique < Strategy
76
- def merge_strategy(response,rec)
77
- return response.merge(rec) { |_, v1, _| v1 }
78
- end
79
- end
80
- end
81
- end
82
- ```
83
-
84
- The `merge_strategy` will receive two parameters, the first being "response" which is the total response which will be returned from `execute_in_batches`, and "rec" is the recursive response from each batch that will be added to response in each iteration. If you want to merge or do something else entirely, you have the option to do so.
85
-
86
- All strategies have a `base_case` which by default is `{}` but if you wish to override it, you can define your own in your strategy by simply adding a method called `base_case` which should return the value you desire for your own personal case.
87
-
88
- ```ruby
89
- def base_case
90
- # Your base value for your batches.
91
- end
92
- ```
93
-
94
- The new class that will hold the method `merge_strategy` should inherit the class `Strategy`. If the strategy doesn't define the method, an exception will be raised when trying to utilize it
95
- warning that it does not respect the contract set by the `Strategy` Interface.
96
-
97
- ## Contributing
98
-
99
- 1. Fork it
100
- 2. Create your feature branch (`git checkout -b my-new-feature`)
101
- 3. Commit your changes (`git commit -am 'Add some feature'`)
102
- 4. Run rubocop lint (`rubocop -R --format simple`)
103
- 5. Run rspec tests (`bundle exec rspec`)
104
- 6. Push your branch (`git push origin my-new-feature`)
105
- 7. Create a new Pull Request
106
-
107
- ## About ##
108
-
109
- This project is maintained by [Diego Raffo](https://github.com/enanodr) along with [Pedro Jara](https://github.com/redwarewolf) and it was written by [Wolox](http://www.wolox.com.ar).
110
- ![Wolox](https://raw.githubusercontent.com/Wolox/press-kit/master/logos/logo_banner.png)
111
-
112
- ## License
113
-
114
- **wor-batchifier** is available under the MIT [license](https://raw.githubusercontent.com/Wolox/wor-batchifier/master/LICENSE.md).
115
-
116
- Copyright (c) 2017 Wolox
117
-
118
- Permission is hereby granted, free of charge, to any person obtaining a copy
119
- of this software and associated documentation files (the "Software"), to deal
120
- in the Software without restriction, including without limitation the rights
121
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
122
- copies of the Software, and to permit persons to whom the Software is
123
- furnished to do so, subject to the following conditions:
124
-
125
- The above copyright notice and this permission notice shall be included in
126
- all copies or substantial portions of the Software.
127
-
128
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
129
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
130
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
131
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
132
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
133
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
134
- THE SOFTWARE.
1
+ # Wolox on Rails - Batchifier
2
+ [![Gem Version](https://badge.fury.io/rb/wor-batchifier.svg)](https://badge.fury.io/rb/wor-batchifier)
3
+ [![Dependency Status](https://gemnasium.com/badges/github.com/Wolox/wor-batchifier.svg)](https://gemnasium.com/github.com/Wolox/wor-batchifier)
4
+ [![Build Status](https://travis-ci.org/Wolox/wor-batchifier.svg)](https://travis-ci.org/Wolox/wor-batchifier)
5
+ [![Code Climate](https://codeclimate.com/github/Wolox/wor-batchifier/badges/gpa.svg)](https://codeclimate.com/github/Wolox/wor-authentication)
6
+ [![Test Coverage](https://codeclimate.com/github/Wolox/wor-batchifier/badges/coverage.svg)](https://codeclimate.com/github/Wolox/wor-batchifier/coverage)
7
+
8
+ Gem that allows you to easily divide requests that send a lot of information into several requests with smaller chunks of data, taking care of the response from each one and providing it joint together based on different strategies for parsing said response.
9
+
10
+ ## Installation
11
+
12
+ Add this line to your application's Gemfile:
13
+
14
+ ```ruby
15
+ gem 'wor-batchifier'
16
+ ```
17
+
18
+ And then execute:
19
+
20
+ $ bundle
21
+
22
+ Or install it yourself as:
23
+
24
+ $ gem install wor-batchifier
25
+
26
+ ## Usage
27
+
28
+ ### Basic use:
29
+
30
+ The first step is to define a parent controller from which all other controllers will have to extend to have access to the batchifier's methods. So, let's do that in our `ApplicationController.rb`:
31
+
32
+ ```ruby
33
+ class ApplicationController < ActionController::Base
34
+ include Wor::Batchifier
35
+ end
36
+ ```
37
+
38
+ You can also include the batchifier just in the controllers you intend to use it in.
39
+
40
+ The final step, is to find any request you wish to perform with smaller chunks of data and utilize the batchifier's methods to divide it into smaller tasks.
41
+
42
+ For example, let's pretend we have an endpoint called bulk_request that communicates with a third API and sends a lot of information to be utilized.
43
+
44
+ ```ruby
45
+ def bulk_request
46
+ ThirdAPI.bulk_request(params[:information])
47
+ end
48
+ ```
49
+
50
+ Now we will partition that request into chunks using the batchifier:
51
+
52
+ ```ruby
53
+ def bulk_request
54
+ execute_in_batches(params[:information], batch_size: 100, strategy: :add) do |chunks|
55
+ ThirdAPI.bulk_request(chunks)
56
+ end
57
+ end
58
+ ```
59
+
60
+ The batchifier will take three parameters, the first one being the information that needs to be partitioned, then the batch_size we wish to utilize and finally the symbol of the strategy that will be implemented on the response of each batch.
61
+
62
+ ### Available strategies
63
+
64
+ - Add: For each request, it joins together each response no matter the result.
65
+ - Maintain-Unique: It will only add the results that are not present already in the response.
66
+ - No-Response: It will not provide any response whatsoever.
67
+
68
+ ### Adding new strategies
69
+
70
+ Should you desire to add new strategies, it's as simple as creating a new class and defining a method called `merge_strategy` which will hold the logic that will be implemented to parse the response. Let's look at an example:
71
+
72
+ ```ruby
73
+ module Wor
74
+ module Batchifier
75
+ class MaintainUnique < Strategy
76
+ def merge_strategy(response,rec)
77
+ return response.merge(rec) { |_, v1, _| v1 }
78
+ end
79
+ end
80
+ end
81
+ end
82
+ ```
83
+
84
+ The `merge_strategy` will receive two parameters, the first being "response" which is the total response which will be returned from `execute_in_batches`, and "rec" is the recursive response from each batch that will be added to response in each iteration. If you want to merge or do something else entirely, you have the option to do so.
85
+
86
+ All strategies have a `base_case` which by default is `{}` but if you wish to override it, you can define your own in your strategy by simply adding a method called `base_case` which should return the value you desire for your own personal case.
87
+
88
+ ```ruby
89
+ def base_case
90
+ # Your base value for your batches.
91
+ end
92
+ ```
93
+
94
+ The new class that will hold the method `merge_strategy` should inherit the class `Strategy`. If the strategy doesn't define the method, an exception will be raised when trying to utilize it
95
+ warning that it does not respect the contract set by the `Strategy` Interface.
96
+
97
+ ## Contributing
98
+
99
+ 1. Fork it
100
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
101
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
102
+ 4. Run rubocop lint (`rubocop -R --format simple`)
103
+ 5. Run rspec tests (`bundle exec rspec`)
104
+ 6. Push your branch (`git push origin my-new-feature`)
105
+ 7. Create a new Pull Request
106
+
107
+ ## About ##
108
+
109
+ This project is maintained by [Diego Raffo](https://github.com/enanodr) along with [Pedro Jara](https://github.com/redwarewolf) and it was written by [Wolox](http://www.wolox.com.ar).
110
+ ![Wolox](https://raw.githubusercontent.com/Wolox/press-kit/master/logos/logo_banner.png)
111
+
112
+ ## License
113
+
114
+ **wor-batchifier** is available under the MIT [license](https://raw.githubusercontent.com/Wolox/wor-batchifier/master/LICENSE.md).
115
+
116
+ Copyright (c) 2017 Wolox
117
+
118
+ Permission is hereby granted, free of charge, to any person obtaining a copy
119
+ of this software and associated documentation files (the "Software"), to deal
120
+ in the Software without restriction, including without limitation the rights
121
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
122
+ copies of the Software, and to permit persons to whom the Software is
123
+ furnished to do so, subject to the following conditions:
124
+
125
+ The above copyright notice and this permission notice shall be included in
126
+ all copies or substantial portions of the Software.
127
+
128
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
129
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
130
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
131
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
132
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
133
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
134
+ THE SOFTWARE.
data/Rakefile CHANGED
@@ -1,6 +1,6 @@
1
- require 'bundler/gem_tasks'
2
- require 'rspec/core/rake_task'
3
-
4
- RSpec::Core::RakeTask.new(:spec)
5
-
6
- task default: :spec
1
+ require 'bundler/gem_tasks'
2
+ require 'rspec/core/rake_task'
3
+
4
+ RSpec::Core::RakeTask.new(:spec)
5
+
6
+ task default: :spec
@@ -1,37 +1,36 @@
1
- require_relative 'batchifier/exceptions'
2
- require_relative 'batchifier/interface'
3
- require_relative 'batchifier/strategy'
4
- require_relative 'batchifier/add'
5
- require_relative 'batchifier/no_response'
6
- require_relative 'batchifier/array_merge'
7
- require_relative 'batchifier/maintain_unique'
8
-
9
- module Wor
10
- module Batchifier
11
-
12
- # This module exports the function execute_in_batches, that needs a collections and
13
- # => optionaly a batch_size and a merge strategy. It will slice the collection and
14
- # => apply the chozen strategy to all chunks and merge the results. It expects the responses
15
- # => to be Hash. It can ignore them if the given strategy is no_response
16
- def execute_in_batches(collection, batch_size: 100, strategy: :add)
17
- strategy_class = classify_strategy(strategy)
18
- collection.lazy.each_slice(batch_size).inject(strategy_class.new.base_case) do |rec, chunk|
19
- response = yield(chunk)
20
- merge(response, rec, strategy)
21
- end
22
- end
23
-
24
- private
25
-
26
- def merge(response, rec, strategy)
27
- return Wor::Batchifier.classify_strategy(strategy).new.merge_strategy(response,rec)
28
- end
29
-
30
- def classify_strategy(strategy)
31
- strategy_class_name = strategy.to_s.split('_').collect(&:capitalize).join
32
- Kernel.const_get(strategy_class_name)
33
- rescue NameError => e
34
- raise Wor::Batchifier::Exceptions::StrategyNotFound
35
- end
36
- end
37
- end
1
+ require_relative 'batchifier/exceptions'
2
+ require_relative 'batchifier/strategy'
3
+ require_relative 'batchifier/add'
4
+ require_relative 'batchifier/no_response'
5
+ require_relative 'batchifier/array_merge'
6
+ require_relative 'batchifier/maintain_unique'
7
+
8
+ module Wor
9
+ module Batchifier
10
+
11
+ # This module exports the function execute_in_batches, that needs a collections and
12
+ # => optionaly a batch_size and a merge strategy. It will slice the collection and
13
+ # => apply the chozen strategy to all chunks and merge the results. It expects the responses
14
+ # => to be Hash. It can ignore them if the given strategy is no_response
15
+ def execute_in_batches(collection, batch_size: 100, strategy: :add)
16
+ strategy_class = classify_strategy(strategy)
17
+ collection.lazy.each_slice(batch_size).inject(strategy_class.new.base_case) do |rec, chunk|
18
+ response = yield(chunk)
19
+ merge(response, rec, strategy)
20
+ end
21
+ end
22
+
23
+ private
24
+
25
+ def merge(response, rec, strategy)
26
+ return classify_strategy(strategy).new.merge_strategy(response,rec)
27
+ end
28
+
29
+ def classify_strategy(strategy)
30
+ strategy_class_name = strategy.to_s.split('_').collect(&:capitalize).join
31
+ Kernel.const_get("Wor::Batchifier::#{strategy_class_name}")
32
+ rescue NameError => e
33
+ raise Wor::Batchifier::Exceptions::StrategyNotFound
34
+ end
35
+ end
36
+ end
@@ -1,9 +1,9 @@
1
- module Wor
2
- module Batchifier
3
- class Add < Strategy
4
- def merge_strategy(response,rec)
5
- return response.merge(rec) { |_, v1, v2| v1 + v2 }
6
- end
7
- end
8
- end
9
- end
1
+ module Wor
2
+ module Batchifier
3
+ class Add < Strategy
4
+ def merge_strategy(response,rec)
5
+ return response.merge(rec) { |_, v1, v2| v1 + v2 }
6
+ end
7
+ end
8
+ end
9
+ end
@@ -1,10 +1,10 @@
1
- module Wor
2
- module Batchifier
3
- class ArrayMerge < Strategy
4
- def merge_strategy(response,rec)
5
- rec = [] if rec.empty?
6
- return rec + response
7
- end
8
- end
9
- end
10
- end
1
+ module Wor
2
+ module Batchifier
3
+ class ArrayMerge < Strategy
4
+ def merge_strategy(response,rec)
5
+ rec = [] if rec.empty?
6
+ return rec + response
7
+ end
8
+ end
9
+ end
10
+ end