kraps 0.7.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 19635ced3e745d44313ed3bc416ef73eb555134c591725f4dab7b38208e21393
-  data.tar.gz: bb7f679c7e2cd053744d1c7d857629de1912a305880d0538bdab418de3861ba1
+  metadata.gz: 8b24f67ff2122dc82d372eb421c379ac42f415674999958f42f71a2cdbee1a33
+  data.tar.gz: d8297482eb38a30cb8ff6a7761a544833e3795a8f3a045343f6a10c370d886ca
 SHA512:
-  metadata.gz: 91273ba54ea33c6d5cb1b4f335ad8039c35601953fdf1e6b9b2ac3117ceb25d81e9be569e5c8b70deda22b53a4c72f04d60ffb4b251badf5c5a64d13d399f36c
-  data.tar.gz: 4e563257fcba0c9f457b363da4f43d000ee239a5179e2f965071bb3df27e362cdf7bc9950a1624520cb862acfb0fde89e96b0a4987d79a93524230c4b84619cd
+  metadata.gz: 5543d1a8af8fa12007d38f00d9aa515eb1edf254d8bbc2aa8c133c0101dbe7ccaef5f0330929b6b84456a81eec7e5744091deace2d16ed2826f14cd56432db8f
+  data.tar.gz: 14992d608157562da3af98207681a66cf0a9a9566861fe79c949ccdec9db6cb344bfb070147fc820eac70b076c15db373896f80b5bd8d5f8714f9ce75d7eb7c8

data/.rubocop.yml

@@ -16,6 +16,12 @@ Lint/UnreachableLoop:
 Metrics/BlockLength:
   Enabled: false
 
+Style/HashEachMethods:
+  Enabled: false
+
+Style/ZeroLengthPredicate:
+  Enabled: false
+
 Gemspec/RequiredRubyVersion:
   Enabled: false
 

data/CHANGELOG.md

@@ -1,5 +1,41 @@
 # CHANGELOG
 
+## v0.9.0
+
+* Argments are no longer passed to the `call` method, but to the
+  initializer instead
+
+Before:
+
+```ruby
+class MyJob
+  def call(arg1, arg2)
+    # ...
+  end
+end
+```
+
+After:
+
+```ruby
+class MyJob
+  def initializer(arg1, arg2)
+    @arg1 = arg1
+    @arg2 = arg2
+  end
+
+  def call
+    # ...
+  end
+end
+```
+
+## v0.8.0
+
+* Use number of partitions of previous step for `jobs` option by default
+* Changed `combine` to receive a `collector`
+* Added mandatory `concurrency` argument to `load`
+
 ## v0.7.0
 
 * Added a `jobs` option to the actions to limit the concurrency

data/Gemfile

@@ -3,6 +3,7 @@ source "https://rubygems.org"
 # Specify your gem's dependencies in kraps.gemspec
 gemspec
 
-gem "rake", "~> 13.0"
-
-gem "rspec", "~> 3.0"
+gem "bundler"
+gem "rake"
+gem "rspec"
+gem "rubocop"

data/README.md

@@ -38,16 +38,21 @@ Kraps.configure(
 
 Afterwards, create a job class, which tells Kraps what your job should do.
 Therefore, you create some class with a `call` method, and optionally some
-arguments. Let's create a simple job, which reads search log files to analyze
-how often search queries have been searched:
+arguments passed to its initializer. Let's create a simple job, which reads
+search log files to analyze how often search queries have been searched:
 
 ```ruby
 class SearchLogCounter
-  def call(start_date:, end_date:)
+  def initialize(start_date:, end_date:)
+    @start_date = start_date
+    @end_date = end_date
+  end
+
+  def call
     job = Kraps::Job.new(worker: MyKrapsWorker)
 
     job = job.parallelize(partitions: 128) do |collector|
-      (Date.parse(start_date)..Date.parse(end_date)).each do |date|
+      (Date.parse(@start_date)..Date.parse(@end_date)).each do |date|
         collector.call(date.to_s)
       end
     end
@@ -214,6 +219,10 @@ job.parallelize(partitions: 128, partitioner: partitioner, worker: MyKrapsWorker
 end
 ```
 
+Please note, that `parallelize` itself is not parallelized but rather
+parallelizes the data you feed into Kraps within `parallelize` by splitting it
+into the number of `partitions` specified.
+
 The block must use the collector to feed Kraps with individual items. The
 items are used as keys and the values are set to `nil`.
 
@@ -232,10 +241,11 @@ return the new key-value pair, but the `collector` must be used instead.
 The `jobs` argument can be useful when you need to access an external data
 source, like a relational database and you want to limit the number of workers
 accessing the store concurrently to avoid overloading it. If you don't specify
-it, it will be identical to the number of partitions. It is recommended to only
-use it for steps where you need to throttle the concurrency, because it will of
-course slow down the processing. The `jobs` argument only applies to the
-current step. The following steps don't inherit the argument, but reset it.
+it, it will be identical to the number of partitions of the previous step. It
+is recommended to only use it for steps where you need to throttle the
+concurrency, because it will of course slow down the processing. The `jobs`
+argument only applies to the current step. The following steps don't inherit
+the argument, but reset it.
 
 * `map_partitions`: Maps the key value pairs to other key value pairs, but the
   block receives all data of each partition as an enumerable and sorted by key.
@@ -273,8 +283,8 @@ most of the time, this is not neccessary and the key can simply be ignored.
   passed job result are completely omitted.
 
 ```ruby
-  job.combine(other_job, worker: MyKrapsWorker, jobs: 8) do |key, value1, value2|
-    (value1 || {}).merge(value2 || {})
+  job.combine(other_job, worker: MyKrapsWorker, jobs: 8) do |key, value1, value2, collector|
+    collector.call(key, (value1 || {}).merge(value2 || {}))
   end
 ```
 
@@ -316,10 +326,12 @@ It creates a folder for every partition and stores one or more chunks in there.
 * `load`: Loads the previously dumped data
 
 ```ruby
-job.load(prefix: "path/to/dump", partitions: 32, partitioner: Kraps::HashPartitioner.new, worker: MyKrapsWorker)
+job.load(prefix: "path/to/dump", partitions: 32, concurrency: 8, partitioner: Kraps::HashPartitioner.new, worker: MyKrapsWorker)
 ```
 
-The number of partitions and the partitioner must be specified.
+The number of partitions, the partitioner and concurrency must be specified.
+The concurrency specifies the number of threads used for downloading chunks in
+parallel.
 
 Please note that every API method accepts a `before` callable:
 
@@ -342,11 +354,16 @@ of searches made:
 
 ```ruby
 class SearchLogCounter
-  def call(start_date:, end_date:)
+  def initialize(start_date:, end_date:)
+    @start_date = start_date
+    @end_date = end_date
+  end
+
+  def call
     count_job = Kraps::Job.new(worker: SomeBackgroundWorker)
 
     count_job = count_job.parallelize(partitions: 128) do |collector|
-      (Date.parse(start_date)..Date.parse(end_date)).each do |date|
+      (Date.parse(@start_date)..Date.parse(@end_date)).each do |date|
         collector.call(date.to_s)
       end
     end

data/lib/kraps/downloader.rb

@@ -0,0 +1,19 @@
+module Kraps
+  class Downloader
+    def self.download_all(prefix:, concurrency:)
+      temp_paths = TempPaths.new
+
+      files = Kraps.driver.list(prefix: prefix).sort
+
+      temp_paths_index = files.each_with_object({}) do |file, hash|
+        hash[file] = temp_paths.add
+      end
+
+      Parallelizer.each(files, concurrency) do |file|
+        Kraps.driver.download(file, temp_paths_index[file].path)
+      end
+
+      temp_paths
+    end
+  end
+end

data/lib/kraps/job.rb

@@ -30,12 +30,14 @@ module Kraps
     def map(partitions: nil, partitioner: nil, jobs: nil, worker: @worker, before: nil, &block)
       fresh.tap do |job|
         job.instance_eval do
+          jobs = [jobs, @partitions].compact.min
+
           @partitions = partitions if partitions
           @partitioner = partitioner if partitioner
 
           @steps << Step.new(
             action: Actions::MAP,
-            jobs: [jobs, @partitions].compact.min,
+            jobs: jobs,
             partitions: @partitions,
             partitioner: @partitioner,
             worker: worker,
@@ -49,12 +51,14 @@ module Kraps
     def map_partitions(partitions: nil, partitioner: nil, jobs: nil, worker: @worker, before: nil, &block)
       fresh.tap do |job|
         job.instance_eval do
+          jobs = [jobs, @partitions].compact.min
+
           @partitions = partitions if partitions
           @partitioner = partitioner if partitioner
 
           @steps << Step.new(
             action: Actions::MAP_PARTITIONS,
-            jobs: [jobs, @partitions].compact.min,
+            jobs: jobs,
             partitions: @partitions,
             partitioner: @partitioner,
             worker: worker,
@@ -135,7 +139,7 @@ module Kraps
       end
     end
 
-    def load(prefix:, partitions:, partitioner:, worker: @worker)
+    def load(prefix:, partitions:, partitioner:, concurrency:, worker: @worker)
       job = parallelize(partitions: partitions, partitioner: proc { |key, _| key }, worker: worker) do |collector|
         (0...partitions).each do |partition|
           collector.call(partition)
@@ -143,20 +147,19 @@ module Kraps
       end
 
       job.map_partitions(partitioner: partitioner, worker: worker) do |partition, _, collector|
-        tempfile = Tempfile.new
-
-        path = File.join(prefix, partition.to_s, "chunk.json")
-        next unless Kraps.driver.exists?(path)
+        temp_paths = Downloader.download_all(prefix: File.join(prefix, partition.to_s, "/"), concurrency: concurrency)
 
-        Kraps.driver.download(path, tempfile.path)
+        temp_paths.each do |temp_path|
+          File.open(temp_path.path) do |stream|
+            stream.each_line do |line|
+              key, value = JSON.parse(line)
 
-        tempfile.each_line do |line|
-          key, value = JSON.parse(line)
-
-          collector.call(key, value)
+              collector.call(key, value)
+            end
+          end
         end
       ensure
-        tempfile&.close(true)
+        temp_paths&.delete
       end
     end
 

data/lib/kraps/runner.rb

@@ -5,7 +5,7 @@ module Kraps
     end
 
     def call(*args, **kwargs)
-      JobResolver.new.call(@klass.new.call(*args, **kwargs)).tap do |jobs|
+      JobResolver.new.call(@klass.new(*args, **kwargs).call).tap do |jobs|
         jobs.each_with_index do |job, job_index|
           job.steps.each_with_index.inject(nil) do |frame, (_, step_index)|
             StepRunner.new(
@@ -100,7 +100,7 @@ module Kraps
 
       def push_and_wait(enum:, job_count: nil)
         redis_queue = RedisQueue.new(redis: Kraps.redis, token: SecureRandom.hex, namespace: Kraps.namespace, ttl: Kraps.job_ttl)
-        progress_bar = build_progress_bar("#{@klass}: job #{@job_index + 1}/#{@jobs.size}, step #{@step_index + 1}/#{@job.steps.size}, token #{redis_queue.token}, %a, %c/%C (%p%) => #{@step.action}")
+        progress_bar = build_progress_bar("#{@klass}: job #{@job_index + 1}/#{@jobs.size}, step #{@step_index + 1}/#{@job.steps.size}, #{@step.jobs || "?"} jobs, token #{redis_queue.token}, %a, %c/%C (%p%) => #{@step.action}")
 
         total = 0
 

data/lib/kraps/version.rb

@@ -1,3 +1,3 @@
 module Kraps
-  VERSION = "0.7.0"
+  VERSION = "0.9.0"
 end

data/lib/kraps/worker.rb

@@ -157,7 +157,7 @@ module Kraps
       implementation = Object.new
       implementation.define_singleton_method(:map) do |&block|
         combine_method.call(enum1, enum2) do |key, value1, value2|
-          block.call(key, current_step.block.call(key, value1, value2))
+          current_step.block.call(key, value1, value2, block)
         end
       end
 
@@ -270,23 +270,11 @@ module Kraps
     end
 
     def download_all(token:, partition:)
-      temp_paths = TempPaths.new
-
-      files = Kraps.driver.list(prefix: Kraps.driver.with_prefix("#{token}/#{partition}/")).sort
-
-      temp_paths_index = files.each_with_object({}) do |file, hash|
-        hash[file] = temp_paths.add
-      end
-
-      Parallelizer.each(files, @concurrency) do |file|
-        Kraps.driver.download(file, temp_paths_index[file].path)
-      end
-
-      temp_paths
+      Downloader.download_all(prefix: Kraps.driver.with_prefix("#{token}/#{partition}/"), concurrency: @concurrency)
     end
 
     def jobs
-      @jobs ||= JobResolver.new.call(@args["klass"].constantize.new.call(*@args["args"], **@args["kwargs"].transform_keys(&:to_sym)))
+      @jobs ||= JobResolver.new.call(@args["klass"].constantize.new(*@args["args"], **@args["kwargs"].transform_keys(&:to_sym)).call)
     end
 
     def job

data/lib/kraps.rb

@@ -19,6 +19,7 @@ require_relative "kraps/runner"
 require_relative "kraps/step"
 require_relative "kraps/frame"
 require_relative "kraps/worker"
+require_relative "kraps/downloader"
 
 module Kraps
   class Error < StandardError; end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kraps
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Benjamin Vetter
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2022-12-02 00:00:00.000000000 Z
+date: 2024-03-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: attachie
@@ -66,48 +66,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rubocop
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Kraps allows to process and perform calculations on very large datasets
   in parallel
 email:
@@ -121,13 +79,13 @@ files:
 - CHANGELOG.md
 - CODE_OF_CONDUCT.md
 - Gemfile
-- Gemfile.lock
 - LICENSE.txt
 - README.md
 - Rakefile
 - docker-compose.yml
 - lib/kraps.rb
 - lib/kraps/actions.rb
+- lib/kraps/downloader.rb
 - lib/kraps/drivers.rb
 - lib/kraps/frame.rb
 - lib/kraps/hash_partitioner.rb

data/Gemfile.lock

@@ -1,108 +0,0 @@
-PATH
-  remote: .
-  specs:
-    kraps (0.7.0)
-      attachie
-      map-reduce-ruby (>= 3.0.0)
-      redis
-      ruby-progressbar
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    activesupport (7.0.4)
-      concurrent-ruby (~> 1.0, >= 1.0.2)
-      i18n (>= 1.6, < 2)
-      minitest (>= 5.1)
-      tzinfo (~> 2.0)
-    ast (2.4.2)
-    attachie (1.2.0)
-      activesupport
-      aws-sdk-s3
-      connection_pool
-      mime-types
-    aws-eventstream (1.2.0)
-    aws-partitions (1.657.0)
-    aws-sdk-core (3.166.0)
-      aws-eventstream (~> 1, >= 1.0.2)
-      aws-partitions (~> 1, >= 1.651.0)
-      aws-sigv4 (~> 1.5)
-      jmespath (~> 1, >= 1.6.1)
-    aws-sdk-kms (1.59.0)
-      aws-sdk-core (~> 3, >= 3.165.0)
-      aws-sigv4 (~> 1.1)
-    aws-sdk-s3 (1.117.1)
-      aws-sdk-core (~> 3, >= 3.165.0)
-      aws-sdk-kms (~> 1)
-      aws-sigv4 (~> 1.4)
-    aws-sigv4 (1.5.2)
-      aws-eventstream (~> 1, >= 1.0.2)
-    concurrent-ruby (1.1.10)
-    connection_pool (2.3.0)
-    diff-lcs (1.5.0)
-    i18n (1.12.0)
-      concurrent-ruby (~> 1.0)
-    jmespath (1.6.1)
-    json (2.6.2)
-    lazy_priority_queue (0.1.1)
-    map-reduce-ruby (3.0.0)
-      json
-      lazy_priority_queue
-    mime-types (3.4.1)
-      mime-types-data (~> 3.2015)
-    mime-types-data (3.2022.0105)
-    minitest (5.16.3)
-    parallel (1.22.1)
-    parser (3.1.2.1)
-      ast (~> 2.4.1)
-    rainbow (3.1.1)
-    rake (13.0.6)
-    redis (5.0.5)
-      redis-client (>= 0.9.0)
-    redis-client (0.11.2)
-      connection_pool
-    regexp_parser (2.6.0)
-    rexml (3.2.5)
-    rspec (3.12.0)
-      rspec-core (~> 3.12.0)
-      rspec-expectations (~> 3.12.0)
-      rspec-mocks (~> 3.12.0)
-    rspec-core (3.12.0)
-      rspec-support (~> 3.12.0)
-    rspec-expectations (3.12.0)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
-    rspec-mocks (3.12.0)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
-    rspec-support (3.12.0)
-    rubocop (1.38.0)
-      json (~> 2.3)
-      parallel (~> 1.10)
-      parser (>= 3.1.2.1)
-      rainbow (>= 2.2.2, < 4.0)
-      regexp_parser (>= 1.8, < 3.0)
-      rexml (>= 3.2.5, < 4.0)
-      rubocop-ast (>= 1.23.0, < 2.0)
-      ruby-progressbar (~> 1.7)
-      unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.23.0)
-      parser (>= 3.1.1.0)
-    ruby-progressbar (1.11.0)
-    tzinfo (2.0.5)
-      concurrent-ruby (~> 1.0)
-    unicode-display_width (2.3.0)
-
-PLATFORMS
-  ruby
-  x86_64-linux
-
-DEPENDENCIES
-  bundler
-  kraps!
-  rake (~> 13.0)
-  rspec (~> 3.0)
-  rubocop
-
-BUNDLED WITH
-   2.3.24