kraps 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: d1f08b6fa0f725c63e3750f4b3bf04479622b40160a6364f708d91d37c0b1948
+  data.tar.gz: '0681d837d852846cc6c115fe9dae3075a0e5b3bb8b8eae2d90d8a23ec26581e3'
+SHA512:
+  metadata.gz: 354ab3129ef1713c8229af54945251069c98d681e2db5c716d93b5925576b601751c23c1d502cb72ddaea5df5fc91e6eceb4590a619de730ae65f0762662da21
+  data.tar.gz: 0647fc85f445bc634f70e2e10feab325c3df6aec3a30d2af4b1a792e82b9adf1a31e8bb339465e10944bce927933956e8d88e72b37fb7d84027ec569441781d6

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.rubocop.yml

@@ -0,0 +1,79 @@
+AllCops:
+  NewCops: enable
+
+Naming/FileName:
+  Exclude:
+    - lib/map-reduce-ruby.rb
+
+Style/StringConcatenation:
+  Exclude:
+    - spec/**/*.rb
+
+Lint/UnreachableLoop:
+  Exclude:
+    - spec/**/*.rb
+
+Metrics/BlockLength:
+  Enabled: false
+
+Gemspec/RequiredRubyVersion:
+  Enabled: false
+
+Style/MutableConstant:
+  Enabled: false
+
+Metrics/MethodLength:
+  Enabled: false
+
+Style/Documentation:
+  Enabled: false
+
+Style/NumericPredicate:
+  Enabled: false
+
+Metrics/AbcSize:
+  Enabled: false
+
+Metrics/CyclomaticComplexity:
+  Enabled: false
+
+Metrics/PerceivedComplexity:
+  Enabled: false
+
+Style/StringLiterals:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Style/StringLiteralsInInterpolation:
+  Enabled: true
+  EnforcedStyle: double_quotes
+
+Layout/LineLength:
+  Max: 250
+
+Style/FrozenStringLiteralComment:
+  EnforcedStyle: never
+
+Style/ObjectThen:
+  Enabled: false
+
+Gemspec/RequireMFA:
+  Enabled: false
+
+Lint/EmptyBlock:
+  Enabled: false
+
+Metrics/ModuleLength:
+  Enabled: false
+
+Metrics/ParameterLists:
+  Enabled: false
+
+Metrics/ClassLength:
+  Enabled: false
+
+Lint/EmptyClass:
+  Enabled: false
+
+Style/WordArray:
+  Enabled: false

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,84 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment for our community include:
+
+* Demonstrating empathy and kindness toward other people
+* Being respectful of differing opinions, viewpoints, and experiences
+* Giving and gracefully accepting constructive feedback
+* Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+* Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+* The use of sexualized language or imagery, and sexual attention or
+  advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or email
+  address, without their explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Enforcement Responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at benjamin.vetter@wlw.de. All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement Guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary Ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent Ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior,  harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0,
+available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
+
+Community Impact Guidelines were inspired by [Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see the FAQ at
+https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/translations.

data/Gemfile

@@ -0,0 +1,8 @@
+source "https://rubygems.org"
+
+# Specify your gem's dependencies in kraps.gemspec
+gemspec
+
+gem "rake", "~> 13.0"
+
+gem "rspec", "~> 3.0"

data/Gemfile.lock

@@ -0,0 +1,113 @@
+PATH
+  remote: .
+  specs:
+    kraps (0.1.0)
+      attachie
+      distributed_job
+      map-reduce-ruby (>= 2.1.1)
+      redis
+      ruby-progressbar
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activesupport (6.1.7)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 1.6, < 2)
+      minitest (>= 5.1)
+      tzinfo (~> 2.0)
+      zeitwerk (~> 2.3)
+    ast (2.4.2)
+    attachie (1.2.0)
+      activesupport
+      aws-sdk-s3
+      connection_pool
+      mime-types
+    aws-eventstream (1.2.0)
+    aws-partitions (1.649.0)
+    aws-sdk-core (3.164.0)
+      aws-eventstream (~> 1, >= 1.0.2)
+      aws-partitions (~> 1, >= 1.525.0)
+      aws-sigv4 (~> 1.1)
+      jmespath (~> 1, >= 1.6.1)
+    aws-sdk-kms (1.58.0)
+      aws-sdk-core (~> 3, >= 3.127.0)
+      aws-sigv4 (~> 1.1)
+    aws-sdk-s3 (1.116.0)
+      aws-sdk-core (~> 3, >= 3.127.0)
+      aws-sdk-kms (~> 1)
+      aws-sigv4 (~> 1.4)
+    aws-sigv4 (1.5.2)
+      aws-eventstream (~> 1, >= 1.0.2)
+    concurrent-ruby (1.1.10)
+    connection_pool (2.3.0)
+    diff-lcs (1.5.0)
+    distributed_job (3.1.0)
+      redis (>= 4.1.0)
+    i18n (1.12.0)
+      concurrent-ruby (~> 1.0)
+    jmespath (1.6.1)
+    json (2.6.2)
+    lazy_priority_queue (0.1.1)
+    map-reduce-ruby (2.1.1)
+      json
+      lazy_priority_queue
+    mime-types (3.4.1)
+      mime-types-data (~> 3.2015)
+    mime-types-data (3.2022.0105)
+    minitest (5.16.3)
+    parallel (1.22.1)
+    parser (3.1.2.1)
+      ast (~> 2.4.1)
+    rainbow (3.1.1)
+    rake (13.0.6)
+    redis (5.0.5)
+      redis-client (>= 0.9.0)
+    redis-client (0.10.0)
+      connection_pool
+    regexp_parser (2.5.0)
+    rexml (3.2.5)
+    rspec (3.11.0)
+      rspec-core (~> 3.11.0)
+      rspec-expectations (~> 3.11.0)
+      rspec-mocks (~> 3.11.0)
+    rspec-core (3.11.0)
+      rspec-support (~> 3.11.0)
+    rspec-expectations (3.11.1)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.11.0)
+    rspec-mocks (3.11.1)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.11.0)
+    rspec-support (3.11.1)
+    rubocop (1.36.0)
+      json (~> 2.3)
+      parallel (~> 1.10)
+      parser (>= 3.1.2.1)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 1.8, < 3.0)
+      rexml (>= 3.2.5, < 4.0)
+      rubocop-ast (>= 1.20.1, < 2.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 1.4.0, < 3.0)
+    rubocop-ast (1.21.0)
+      parser (>= 3.1.1.0)
+    ruby-progressbar (1.11.0)
+    tzinfo (2.0.5)
+      concurrent-ruby (~> 1.0)
+    unicode-display_width (2.3.0)
+    zeitwerk (2.6.1)
+
+PLATFORMS
+  ruby
+  x86_64-linux
+
+DEPENDENCIES
+  bundler
+  kraps!
+  rake (~> 13.0)
+  rspec (~> 3.0)
+  rubocop
+
+BUNDLED WITH
+   2.3.24

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2022 Benjamin Vetter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,333 @@
+# Kraps
+
+**Easily process big data in ruby**
+
+Kraps allows to process and perform calculations on very large datasets in
+parallel using a map/reduce framework and runs on a background job framework
+you already have. You just need some space on your filesystem, S3 as a storage
+layer with temporary lifecycle policy enabled, the already mentioned background
+job framework (like sidekiq, shoryuken, etc) and redis to keep track of the
+progress. Most things you most likely already have in place anyways.
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+    $ bundle add kraps
+
+If bundler is not being used to manage dependencies, install the gem by executing:
+
+    $ gem install kraps
+
+## Usage
+
+The first thing you need to do is to tell Kraps about your desired
+configuration in an initializer for example:
+
+```ruby
+Kraps.configure(
+  driver: Kraps::Drivers::S3Driver.new(s3_client: Aws::S3::Client.new("..."), bucket: "some-bucket", prefix: "temp/kraps/"),
+  redis: Redis.new,
+  namespace: "my-application", # An optional namespace to be used for redis keys, default: nil
+  job_ttl: 24.hours, # Job information in redis will automatically be removed after this amount of time, default: 24 hours
+  show_progress: true # Whether or not to show the progress in the terminal when executing jobs, default: true
+  enqueuer: ->(worker, json) { worker.perform_async(json) } # Allows to customize the enqueueing of worker jobs
+)
+```
+
+Afterwards, create a job class, which tells Kraps what your job should do.
+Therefore, you create some class with a `call` method, and optionally some
+arguments. Let's create a simple job, which reads search log files to analyze
+how often search queries have been searched:
+
+```ruby
+class SearchLogCounter
+  def call(start_date:, end_date:)
+    job = Kraps::Job.new(worker: MyKrapsWorker)
+
+    job = job.parallelize(partitions: 128) do |collector|
+      (Date.parse(start_date)..Date.parse(end_date)).each do |date|
+        collector.call(date.to_s)
+      end
+    end
+
+    job = job.map do |date, _, collector|
+      # fetch log file for the date from e.g. s3
+
+      File.open(logfile).each_line do |line|
+        data = JSON.parse(line)
+
+        collector.call(data["q"], 1)
+      end
+    end
+
+    job = job.reduce do |_, count1, count2|
+      count1 + count2
+    end
+
+    job = job.each_partition do |partition, pairs|
+      tempfile = Tempfile.new
+
+      pairs.each do |q, count|
+        tempfile.puts(JSON.generate(q: q, count: count))
+      end
+
+      # store tempfile on e.g. s3
+    ensure
+      tempfile.close(true)
+    end
+
+    job
+  end
+end
+```
+
+Please note that this represents a specification of your job. It should be as
+free as possible from side effects, because your background jobs must also be
+able to take this specification to be told what to do as Kraps will run the job
+with maximum concurrency.
+
+Next thing you need to do: create the background worker which runs arbitrary
+Kraps job steps. Assuming you have sidekiq in place:
+
+```ruby
+class MyKrapsWorker
+  include Sidekiq::Worker
+
+  def perform(json)
+    Kraps::Worker.new(json, memory_limit: 128.megabytes, chunk_limit: 64, concurrency: 8).call(retries: 3)
+  end
+end
+```
+
+The `json` argument is automatically enqueued by Kraps and contains everything
+it needs to know about the job and step to execute. The `memory_limit` tells
+Kraps how much memory it is allowed to allocate for temporary chunks, etc. This
+value depends on the memory size of your container/server and how much worker
+threads your background queue spawns. Let's say your container/server has 2
+gigabytes of memory and your background framework spawns 5 threads.
+Theoretically, you might be able to give 300-400 megabytes to Kraps then. The
+`chunk_limit` ensures that only the specified amount of chunks are processed in
+a single run. A run basically means: it takes up to `chunk_limit` chunks,
+reduces them and pushes the result as a new chunk to the list of chunks to
+process. Thus, if your number of file descriptors is unlimited, you want to set
+it to a higher number to avoid the overhead of multiple runs. `concurrency`
+tells Kraps how much threads to use to concurrently upload/download files from
+the storage layer. Finally, `retries` specifies how often Kraps should retry
+the job step in case of errors. Kraps will sleep for 5 seconds between those
+retries. Please note that it's not yet possible to use the retry mechanism of
+your background job framework with Kraps.
+
+Now, executing your job is super easy:
+
+```ruby
+Kraps::Runner.new(SearchLogCounter).call(start_date: '2018-01-01', end_date: '2022-01-01')
+```
+
+This will execute all steps of your job, where the parts of a step are executed
+in parallel, depending on the number of background job workers you have.
+
+The runner by default also shows the progress of the execution:
+
+```
+SearchLogCounter: job 1/1, step 1/4, token 2407e38eb58233ae3cecaec86fa6a6ec, Time: 00:00:05, 356/356 (100%) => parallelize
+SearchLogCounter: job 1/1, step 2/4, token 7f11a04c754389359f67c1e7627468c6, Time: 00:08:00, 128/128 (100%) => map
+SearchLogCounter: job 1/1, step 3/4, token b602198bfeab20ff205a00af36e43402, Time: 00:03:00, 128/128 (100%) => reduce
+SearchLogCounter: job 1/1, step 4/4, token d18acbb22bbd30faff7265c179d4ec5a, Time: 00:02:00, 128/128 (100%) => each_partition
+```
+
+How many "parts" a step has mostly boils down to the number of partitions you
+specify in the job respectively steps. More concretely, As your data consists
+of `(key, value)` pairs, the number of partitions specifies how your data gets
+split. Kraps assigns every `key` to a partition, either using a custom
+`partitioner` or the default built in hash partitioner. The hash partitioner
+simply calculates a hash of your key modulo the number of partitions and the
+resulting partition number is the partition where the respective key is
+assigned to. A partitioner is a callable which gets the key as argument and
+returns a partition number. The built in hash partitioner looks similar to this
+one:
+
+```ruby
+partitioner = proc { |key| Digest::SHA1.hexdigest(key.inspect)[0..4].to_i(16) % 128 } # 128 partitions
+```
+
+Please note, it's important that the partitioner and the specified number of
+partitions stays in sync. When you use a custom partitioner, please make sure
+that the partitioner operates on the same number of partitions you specify.
+
+## Datatypes
+
+Be aware that Kraps converts everything you pass to it to JSON sooner or later,
+i.e. symbols will be converted to strings, etc. Therefore, it is recommended to
+only use the json compatible datatypes right from the start. However, the keys
+that you pass to Kraps additionally must be properly sortable, such that it is
+recommended to only use strings, numbers and arrays or a combination of those
+for the keys. For more information, please check out
+https://github.com/mrkamel/map-reduce-ruby/#limitations-for-keys
+
+## Storage
+
+Kraps stores temporary results of steps in a storage layer. Currently, only S3
+is supported besides a in memory driver used for testing purposes. Please be
+aware that Kraps does not clean up any files from the storage layer, as it
+would be a safe thing to do in case of errors anyways. Instead, Kraps relies on
+lifecycle features of modern object storage systems. Therefore, it is recommend
+to e.g. configure a lifecycle policy to delete any files after e.g. 7 days
+either for a whole bucket or for a certain prefix like e.g. `temp/` and tell
+Kraps about the prefix to use (e.g. `temp/kraps/`).
+
+```ruby
+Kraps::Drivers::S3Driver.new(s3_client: Aws::S3::Client.new("..."), bucket: "some-bucket", prefix: "temp/kraps/"),
+```
+
+If you set up the lifecycle policy for the whole bucket instead and Kraps is
+the only user of the bucket, then no prefix needs to be specified.
+
+## API
+
+Your jobs can use the following list of methods. Please note that you don't
+always need to specify all the parameters listed here. Especially `partitions`,
+`partitioner` and `worker` are used from the previous step unless changed in
+the next one.
+
+* `parallelize`: Used to seed the job with initial data
+
+```ruby
+job.parallelize(partitions: 128, partitioner: partitioner, worker: MyKrapsWorker) do |collector|
+  ["item1", "item2", "item3"].each do |item|
+    collector.call(item)
+  end
+end
+```
+
+The block must use the collector to feed Kraps with individual items. The
+items are used as keys and the values are set to `nil`.
+
+* `map`: Maps the key value pairs to other key value pairs
+
+```ruby
+job.map(partitions: 128, partitioner: partitioner, worker: MyKrapsWorker) do |key, value, collector|
+  collector.call("changed #{key}", "changed #{value}")
+end
+```
+
+The block gets each key-value pair passed and the `collector` block can be
+called as often as neccessary. This is also the reason why `map` can not simply
+return the new key-value pair, but the `collector` must be used instead.
+
+* `reduce`: Reduces the values of pairs having the same key
+
+```ruby
+job.reduce(worker: MyKrapsWorker) do |key, value1, value2|
+  value1 + value2
+end
+```
+
+When the same key exists multiple times in the data, kraps feeds the values
+into your reduce block and expects to get one value returned. This happens
+until every key exists only once.
+
+The `key` itself is also passed to the block for the case that you need to
+customize the reduce calculation according to the value of the key. However,
+most of the time, this is not neccessary and the key can simply be ignored.
+
+* `repartition`: Used to change the partitioning
+
+```ruby
+job.repartition(partitions: 128, partitioner: partitioner, worker: MyKrapsWorker)
+```
+
+Repartitions all data into the specified number of partitions and using the
+specified partitioner.
+
+* `each_partition`: Passes the partition number and all data of each partition
+  as a lazy enumerable
+
+```ruby
+job.each_partition do |partition, pairs|
+  pairs.each do |key, value|
+    # ...
+  end
+end
+```
+
+## More Complex Jobs
+
+Please note that a job class can return multiple jobs and jobs can build up on
+each other. Let's assume that we additionally want to calculate a total number
+of searches made:
+
+```ruby
+class SearchLogCounter
+  def call(start_date:, end_date:)
+    count_job = Kraps::Job.new(worker: SomeBackgroundWorker)
+
+    count_job = count_job.parallelize(partitions: 128) do |collector|
+      (Date.parse(start_date)..Date.parse(end_date)).each do |date|
+        collector.call(date.to_s)
+      end
+    end
+
+    count_job = count_job.map do |date, _, collector|
+      # ...
+
+      collector.call(data["q"], 1)
+
+      # ...
+    end
+
+    count_job = count_job.reduce do |_, count1, count2|
+      count1 + count2
+    end
+
+    sum_job = count_job.map do |q, count, collector|
+      collector.call('sum', count)
+    end
+
+    sum_job = sum_job.reduce do |_, count1, count2|
+      count1 + count2
+    end
+
+    # ...
+
+    [count_job, sum_job]
+  end
+end
+```
+
+When you execute the job, Kraps will execute the jobs one after another and as
+the jobs build up on each other, Kraps will execute the steps shared by both
+jobs only once.
+
+## Dependencies
+
+Kraps is built on top of
+[map-reduce-ruby](https://github.com/mrkamel/map-reduce-ruby) for the
+map/reduce framework,
+[distributed_job](https://github.com/mrkamel/distributed_job)
+to keep track of the job/step status,
+[attachie](https://github.com/mrkamel/attachie) to interact with the storage
+layer (s3),
+[ruby-progressbar](https://github.com/jfelchner/ruby-progressbar) to
+report the progress in the terminal.
+
+It is highly recommended to check out `map-reduce-ruby` to dig into internals
+and performance details.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at
+https://github.com/mrkamel/kraps. This project is intended to be a safe,
+welcoming space for collaboration, and contributors are expected to adhere to
+the [code of conduct](https://github.com/mrkamel/kraps/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the
+[MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Kraps project's codebases, issue trackers, chat
+rooms and mailing lists is expected to follow the
+[code of conduct](https://github.com/mrkamel/kraps/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/docker-compose.yml

@@ -0,0 +1,6 @@
+version: '2' 
+services:
+  elasticsearch:
+    image: redis
+    ports:
+      - 6379:6379

data/lib/kraps/actions.rb

@@ -0,0 +1,10 @@
+module Kraps
+  module Actions
+    ALL = [
+      PARALLELIZE = "parallelize",
+      MAP = "map",
+      REDUCE = "reduce",
+      EACH_PARTITION = "each_partition"
+    ]
+  end
+end