lowkiq 1.0.2 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bd62b77923e5cedfcfefb44de439a6ca698931df329ef1790c25158173f18f68
-  data.tar.gz: 83a4bc80cf936a734fa15c7f2569644fa5a52eb4b40b5c09553c8b7ce030174b
+  metadata.gz: 1737d4cbb8058600c3c3b162c6d48b8749d5f1750524538f73300fbcd0090269
+  data.tar.gz: 6014eb81fa0550a93efa7e04d2785bc4616f8f004563c906fd9f3ee930ba3183
 SHA512:
-  metadata.gz: 4684bf9142a115de9c844f0fd3d59810a27204b94b580db64947edac1440da8a24ac4efc55843d2905f50f815da665a088cd4d1604ae4a5c5d4fa3d08e6e7223
-  data.tar.gz: 26f8179bbd911011d830b25ad7fb42113b7a5bf98ea0c84c716b2ad412a462f48a0620e4189dc454f3a91b6237b140f25599f5d259dbeec902e32910399efe05
+  metadata.gz: 16bb89de9f1f25a745602e70ae9e172fb85cbbad9b9ed50061fd6c3c74dd0122c204c7b6c973bbb02e2c891bc45449882da479a3b59b5108b65073b9d000e8a2
+  data.tar.gz: 0d9c5316a08272e539c34a1cda2e688348b93f3c255ec9c2faef8f3db7363e1e8e87402a4cda880cce95249e2120686f86d05bdba5bb117c3cefb079885bc52f

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# 1.1.0
+
+* Timestamps are float rather than int. #23
+* Due to problems with autoloading, you now need to manually assign a list of workers. #22
+
+```ruby
+Lowkiq.workers = [ ATestWorker, ATest2Worker ]
+```

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    lowkiq (1.0.0)
+    lowkiq (1.1.0)
       connection_pool (~> 2.2, >= 2.2.2)
       rack (>= 1.5.0)
       redis (>= 4.0.1, < 5)
@@ -9,13 +9,22 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    connection_pool (2.2.2)
+    byebug (11.1.3)
+    coderay (1.1.3)
+    connection_pool (2.2.3)
     diff-lcs (1.3)
+    method_source (1.0.0)
+    pry (0.13.1)
+      coderay (~> 1.1)
+      method_source (~> 1.0)
+    pry-byebug (3.9.0)
+      byebug (~> 11.0)
+      pry (~> 0.13.0)
     rack (2.2.2)
     rack-test (1.1.0)
       rack (>= 1.0, < 3)
     rake (12.3.3)
-    redis (4.1.3)
+    redis (4.2.5)
     rspec (3.9.0)
       rspec-core (~> 3.9.0)
       rspec-expectations (~> 3.9.0)
@@ -36,10 +45,11 @@ PLATFORMS
 DEPENDENCIES
   bundler (~> 2.1.0)
   lowkiq!
+  pry-byebug (~> 3.9.0)
   rack-test (~> 1.1)
   rake (~> 12.3.0)
   rspec (~> 3.0)
   rspec-mocks (~> 3.8)
 
 BUNDLED WITH
-   2.1.2
+   2.1.4

data/README.md

@@ -16,6 +16,7 @@ Ordered background jobs processing
 * [Api](#api)
 * [Ring app](#ring-app)
 * [Configuration](#configuration)
+* [Performance](#performance)
 * [Execution](#execution)
 * [Shutdown](#shutdown)
 * [Debug](#debug)
@@ -31,9 +32,9 @@ Ordered background jobs processing
 ## Rationale
 
 We've faced some problems using Sidekiq while processing messages from a side system.
-For instance, the message is a data of an order in particular time.
-The side system will send a new data of an order on an every change.
-Orders are frequently updated and a queue containts some closely located messages of the same order.
+For instance, the message is the data of an order at a particular time.
+The side system will send new data of an order on every change.
+Orders are frequently updated and a queue contains some closely located messages of the same order.
 
 Sidekiq doesn't guarantee a strict message order, because a queue is processed by multiple threads.
 For example, we've received 2 messages: M1 and M2.
@@ -42,8 +43,8 @@ so M2 can be processed before M1.
 
 Parallel processing of such kind of messages can result in:
 
-+ dead locks
-+ overwriting new data with old one
++ deadlocks
++ overwriting new data with an old one
 
 Lowkiq has been created to eliminate such problems by avoiding parallel task processing within one entity.
 
@@ -51,17 +52,17 @@ Lowkiq has been created to eliminate such problems by avoiding parallel task pro
 
 Lowkiq's queues are reliable i.e.,
 Lowkiq saves information about a job being processed
-and returns incompleted jobs back to the queue on startup.
+and returns uncompleted jobs to the queue on startup.
 
 Jobs in queues are ordered by preassigned execution time, so they are not FIFO queues.
 
-Every job has it's own identifier. Lowkiq guarantees that jobs with equal id are processed by the same thread.
+Every job has its identifier. Lowkiq guarantees that jobs with equal IDs are processed by the same thread.
 
 Every queue is divided into a permanent set of shards.
-A job is placed into particular shard based on an id of the job.
+A job is placed into a particular shard based on an id of the job.
 So jobs with the same id are always placed into the same shard.
 All jobs of the shard are always processed with the same thread.
-This guarantees the sequently processing of jobs with the same ids and excludes the possibility of locks.
+This guarantees the sequential processing of jobs with the same ids and excludes the possibility of locks.
 
 Besides the id, every job has a payload.
 Payloads are accumulated for jobs with the same id.
@@ -70,8 +71,8 @@ It's useful when you need to process only the last message and drop all previous
 
 A worker corresponds to a queue and contains a job processing logic.
 
-Fixed amount of threads is used to process all job of all queues.
-Adding or removing queues or it's shards won't affect the amount of threads.
+The fixed number of threads is used to process all jobs of all queues.
+Adding or removing queues or their shards won't affect the number of threads.
 
 ## Sidekiq comparison
 
@@ -82,45 +83,46 @@ But if you use plugins like
 [sidekiq-merger](https://github.com/dtaniwaki/sidekiq-merger)
 or implement your own lock system, you should look at Lowkiq.
 
-For example, sidekiq-grouping accumulates a batch of jobs than enqueues it and accumulates a next batch.
-With this approach queue can contains two batches with a data of the same order.
+For example, sidekiq-grouping accumulates a batch of jobs then enqueues it and accumulates the next batch.
+With this approach, a queue can contain two batches with data of the same order.
 These batches are parallel processed with different threads, so we come back to the initial problem.
 
-Lowkiq was designed to avoid any types of locking.
+Lowkiq was designed to avoid any type of locking.
 
 Furthermore, Lowkiq's queues are reliable. Only Sidekiq Pro or plugins can add such functionality.
 
-This [benchmark](examples/benchmark) shows overhead on redis usage.
-This is the results for 5 threads, 100,000 blank jobs:
+This [benchmark](examples/benchmark) shows overhead on Redis usage.
+These are the results for 5 threads, 100,000 blank jobs:
 
-+ lowkiq: 214 sec or 2.14 ms per job
-+ sidekiq: 29 sec or 0.29 ms per job
++ lowkiq: 155 sec or 1.55 ms per job
++ lowkiq +hiredis: 80 sec or 0.80 ms per job
++ sidekiq: 15 sec or 0.15 ms per job
 
 This difference is related to different queues structure.
 Sidekiq uses one list for all workers and fetches the job entirely for O(1).
-Lowkiq uses several data structures, including sorted sets for storing ids of jobs.
+Lowkiq uses several data structures, including sorted sets for keeping ids of jobs.
 So fetching only an id of a job takes O(log(N)).
 
 ## Queue
 
 Please, look at [the presentation](https://docs.google.com/presentation/d/e/2PACX-1vRdwA2Ck22r26KV1DbY__XcYpj2FdlnR-2G05w1YULErnJLB_JL1itYbBC6_JbLSPOHwJ0nwvnIHH2A/pub?start=false&loop=false&delayms=3000).
 
-Every job has following attributes:
+Every job has the following attributes:
 
 + `id` is a job identifier with string type.
-+ `payloads` is a sorted set of payloads ordered by it's score. Payload is an object. Score is a real number.
-+ `perform_in` is planned execution time. It's unix timestamp with real number type.
++ `payloads` is a sorted set of payloads ordered by its score. A payload is an object. A score is a real number.
++ `perform_in` is planned execution time. It's a Unix timestamp with a real number type.
 + `retry_count` is amount of retries. It's a real number.
 
-For example, `id` can be an identifier of replicated entity.
-`payloads` is a sorted set ordered by score of payload and resulted by grouping a payload of job by it's `id`.
-`payload` can be a ruby object, because it is serialized by `Marshal.dump`.
-`score` can be `payload`'s creation date (unix timestamp) or it's incremental version number.
-By default `score` and `perform_in` are current unix timestamp.
+For example, `id` can be an identifier of a replicated entity.
+`payloads` is a sorted set ordered by a score of payload and resulted by grouping a payload of the job by its `id`.
+`payload` can be a ruby object because it is serialized by `Marshal.dump`.
+`score` can be `payload`'s creation date (Unix timestamp) or it's an incremental version number.
+By default, `score` and `perform_in` are current Unix timestamp.
 `retry_count` for new unprocessed job equals to `-1`,
 for one-time failed is `0`, so the planned retries are counted, not the performed ones.
 
-A job execution can be unsuccessful. In this case, its `retry_count` is incremented, new `perform_in` is calculated with determined formula and it moves back to a queue.
+Job execution can be unsuccessful. In this case, its `retry_count` is incremented, the new `perform_in` is calculated with determined formula, and it moves back to a queue.
 
 In case of `retry_count` is getting `>=` `max_retry_count` an element of `payloads` with less (oldest) score is moved to a morgue,
 rest elements are moved back to the queue, wherein `retry_count` and `perform_in` are reset to `-1` and `now()` respectively.
@@ -144,14 +146,14 @@ If `max_retry_count = 1`, retries stop.
 
 They are applied when:
 
-+ a job had been in a queue and a new one with the same id was added
-+ a job was failed, but a new one with the same id had been added
-+ a job from morgue was moved back to queue, but queue had had a job with the same id
++ a job has been in a queue and a new one with the same id is added
++ a job is failed, but a new one with the same id has been added
++ a job from a morgue is moved back to a queue, but the queue has had a job with the same id
 
 Algorithm:
 
-+ payloads is merged, minimal score is chosen for equal payloads
-+ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the the job from the queue
++ payloads are merged, the minimal score is chosen for equal payloads
++ if a new job and queued job is merged, `perform_in` and `retry_count` is taken from the job from the queue
 + if a failed job and queued job is merged, `perform_in` and `retry_count` is taken from the failed one
 + if morgue job and queued job is merged, `perform_in = now()`, `retry_count = -1`
 
@@ -170,8 +172,8 @@ Example:
 { id: "1", payloads: #{"v1": 1, "v2": 3, "v3": 4}, retry_count: 0, perform_in: 1536323288 }
 ```
 
-Morgue is a part of the queue. Jobs in morgue are not processed.
-A job in morgue has following attributes:
+A morgue is a part of a queue. Jobs in a morgue are not processed.
+A job in a morgue has the following attributes:
 
 + id is the job identifier
 + payloads
@@ -214,6 +216,12 @@ module ATestWorker
 end
 ```
 
+And then you have to add it to Lowkiq in your initializer file due to problems with autoloading:
+
+```ruby
+Lowkiq.workers = [ ATestWorker ]
+```
+
 Default values:
 
 ```ruby
@@ -232,10 +240,10 @@ end
 ATestWorker.perform_async [
   { id: 0 },
   { id: 1, payload: { attr: 'v1' } },
-  { id: 2, payload: { attr: 'v1' }, score: Time.now.to_i, perform_in: Time.now.to_i },
+  { id: 2, payload: { attr: 'v1' }, score: Time.now.to_f, perform_in: Time.now.to_f },
 ]
 # payload by default equals to ""
-# score and perform_in by default equals to Time.now.to_i
+# score and perform_in by default equals to Time.now.to_f
 ```
 
 It is possible to redefine `perform_async` and calculate `id`, `score` и `perform_in` in a worker code:
@@ -268,10 +276,11 @@ ATestWorker.perform_async 1000.times.map { |id| { payload: {id: id} } }
 
 ## Configuration
 
-Default options and values are:
+Options and their default values are:
 
++ `Lowkiq.workers = []`- list of workers to use. Since 1.1.0.
 + `Lowkiq.poll_interval = 1` - delay in seconds between queue polling for new jobs.
-   Used only if the queue was empty at previous cycle or error was occured.
+   Used only if a queue was empty in a previous cycle or an error occurred.
 + `Lowkiq.threads_per_node = 5` - threads per node.
 + `Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL') }` - redis connection options
 + `Lowkiq.client_pool_size = 5` - redis pool size for queueing jobs
@@ -281,6 +290,8 @@ Default options and values are:
 + `Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }` is a scheduler
 + `Lowkiq.build_splitter = ->() { Lowkiq.build_default_splitter }` is a splitter
 + `Lowkiq.last_words = ->(ex) {}` is an exception handler of descendants of `StandardError` caused the process stop
++ `Lowkiq.dump_payload = Marshal.method :dump`
++ `Lowkiq.load_payload = Marshal.method :load`
 
 ```ruby
 $logger = Logger.new(STDOUT)
@@ -301,13 +312,29 @@ Lowkiq.server_middlewares << -> (worker, batch, &block) do
 end
 ```
 
+## Performance
+
+Use [hiredis](https://github.com/redis/hiredis-rb) for better performance.
+
+```ruby
+# Gemfile
+
+gem "hiredis"
+```
+
+```ruby
+# config
+
+Lowkiq.redis = ->() { Redis.new url: ENV.fetch('REDIS_URL'), driver: :hiredis }
+```
+
 ## Execution
 
 `lowkiq -r ./path_to_app`
 
 `path_to_app.rb` must load app. [Example](examples/dummy/lib/app.rb).
 
-Lazy loading of workers modules is unacceptable.
+The lazy loading of worker modules is unacceptable.
 For preliminarily loading modules use
 `require`
 or [`require_dependency`](https://api.rubyonrails.org/classes/ActiveSupport/Dependencies/Loadable.html#method-i-require_dependency)
@@ -315,15 +342,15 @@ for Ruby on Rails.
 
 ## Shutdown
 
-Send TERM or INT signal to process (Ctrl-C).
-Process will wait for executed jobs to finish.
+Send TERM or INT signal to the process (Ctrl-C).
+The process will wait for executed jobs to finish.
 
-Note that if queue is empty, process sleeps `poll_interval` seconds,
+Note that if a queue is empty, the process sleeps `poll_interval` seconds,
 therefore, the process will not stop until the `poll_interval` seconds have passed.
 
 ## Debug
 
-To get trace of all threads of app:
+To get trace of all threads of an app:
 
 ```
 kill -TTIN <pid>
@@ -337,11 +364,22 @@ docker-compose run --rm --service-port app bash
 bundle
 rspec
 cd examples/dummy ; bundle exec ../../exe/lowkiq -r ./lib/app.rb
+
+# open localhost:8080
+```
+
+```
+docker-compose run --rm --service-port frontend bash
+npm run dumb
+# open localhost:8081
+
+# npm run build
+# npm run web-api
 ```
 
 ## Exceptions
 
-`StandardError` thrown by worker are handled with middleware. Such exceptions doesn't lead to process stop.
+`StandardError` thrown by a worker are handled with middleware. Such exceptions don't lead to process stops.
 
 All other exceptions cause the process to stop.
 Lowkiq will wait for job execution by other threads.
@@ -364,15 +402,18 @@ end
 ```ruby
 # config/initializers/lowkiq.rb
 
-# loading all lowkiq workers
-Dir["#{Rails.root}/app/lowkiq_workers/**/*.rb"].each { |file| require_dependency file }
-
 # configuration:
 # Lowkiq.redis = -> { Redis.new url: ENV.fetch('LOWKIQ_REDIS_URL') }
 # Lowkiq.threads_per_node = ENV.fetch('LOWKIQ_THREADS_PER_NODE').to_i
 # Lowkiq.client_pool_size = ENV.fetch('LOWKIQ_CLIENT_POOL_SIZE').to_i
 # ...
 
+# since 1.1.0
+Lowkiq.workers = [
+  ATestWorker,
+  OtherCoolWorker
+]
+
 Lowkiq.server_middlewares << -> (worker, batch, &block) do
   logger = Rails.logger
   tag = "#{worker}-#{Thread.current.object_id}"
@@ -460,10 +501,10 @@ worker C: 0
 worker D: 0, 1
 ```
 
-Lowkiq uses fixed amount of threads for job processing, therefore it is necessary to distribute shards between threads.
+Lowkiq uses a fixed number of threads for job processing, therefore it is necessary to distribute shards between threads.
 Splitter does it.
 
-To define a set of shards, which is being processed by thread, lets move them to one list:
+To define a set of shards, which is being processed by a thread, let's move them to one list:
 
 ```
 A0, A1, A2, B0, B1, B2, B3, C0, D0, D1
@@ -480,7 +521,7 @@ t1: A1, B1, C0
 t2: A2, B2, D0
 ```
 
-Besides Default Lowkiq has ByNode splitter. It allows to divide the load by several processes (nodes).
+Besides Default Lowkiq has the ByNode splitter. It allows dividing the load by several processes (nodes).
 
 ```
 Lowkiq.build_splitter = -> () do
@@ -491,7 +532,7 @@ Lowkiq.build_splitter = -> () do
 end
 ```
 
-So, instead of single process you need to execute multiple ones and to set environment variables up:
+So, instead of a single process, you need to execute multiple ones and to set environment variables up:
 
 ```
 # process 0
@@ -503,18 +544,18 @@ LOWKIQ_NUMBER_OF_NODES=2 LOWKIQ_NODE_NUMBER=1 bundle exec lowkiq -r ./lib/app.rb
 
 Summary amount of threads are equal product of `ENV.fetch('LOWKIQ_NUMBER_OF_NODES')` and `Lowkiq.threads_per_node`.
 
-You can also write your own splitter if your app needs extra distribution of shards between threads or nodes.
+You can also write your own splitter if your app needs an extra distribution of shards between threads or nodes.
 
 ## Scheduler
 
-Every thread processes a set of shards. Scheduler select shard for processing.
-Every thread has it's own instance of scheduler.
+Every thread processes a set of shards. The scheduler selects shard for processing.
+Every thread has its own instance of the scheduler.
 
 Lowkiq has 2 schedulers for your choice.
-`Seq` sequentally looks over shards.
+`Seq` sequentially looks over shards.
 `Lag`  chooses shard with the oldest job minimizing the lag. It's used by default.
 
-Scheduler can be set up through settings:
+The scheduler can be set up through settings:
 
 ```
 Lowkiq.build_scheduler = ->() { Lowkiq.build_seq_scheduler }
@@ -527,13 +568,13 @@ Lowkiq.build_scheduler = ->() { Lowkiq.build_lag_scheduler }
 ### `SomeWorker.shards_count`
 
 Sum of `shards_count` of all workers shouldn't be less than `Lowkiq.threads_per_node`
-otherwise threads will stay idle.
+otherwise, threads will stay idle.
 
 Sum of `shards_count` of all workers can be equal to `Lowkiq.threads_per_node`.
-In this case thread processes a single shard. This makes sense only with uniform queue load.
+In this case, a thread processes a single shard. This makes sense only with a uniform queue load.
 
 Sum of `shards_count` of all workers can be more than `Lowkiq.threads_per_node`.
-In this case `shards_count` can be counted as a priority.
+In this case, `shards_count` can be counted as a priority.
 The larger it is, the more often the tasks of this queue will be processed.
 
 There is no reason to set `shards_count` of one worker more than `Lowkiq.threads_per_node`,
@@ -541,8 +582,8 @@ because every thread will handle more than one shard from this queue, so it incr
 
 ### `SomeWorker.max_retry_count`
 
-From `retry_in` and `max_retry_count`, you can calculate approximate time that payload of job will be in a queue.
-After `max_retry_count` is reached the payload with a minimal score will be moved to a morgue.
+From `retry_in` and `max_retry_count`, you can calculate the approximate time that a payload of a job will be in a queue.
+After `max_retry_count` is reached a payload with a minimal score will be moved to a morgue.
 
 For default `retry_in` we receive the following table.
 
@@ -570,9 +611,9 @@ end
 
 ## Changing of worker's shards amount
 
-Try to count amount of shards right away and don't change it in future.
+Try to count the number of shards right away and don't change it in the future.
 
-If you can disable adding of new jobs, wait for queues to get empty and deploy the new version of code with changed amount of shards.
+If you can disable adding of new jobs, wait for queues to get empty, and deploy the new version of code with a changed amount of shards.
 
 If you can't do it, follow the next steps:
 
@@ -590,7 +631,7 @@ module ATestWorker
 end
 ```
 
-Set the number of shards and new queue name:
+Set the number of shards and the new queue name:
 
 ```ruby
 module ATestWorker
@@ -605,7 +646,7 @@ module ATestWorker
 end
 ```
 
-Add a worker moving jobs from the old queue to a new one:
+Add a worker moving jobs from the old queue to the new one:
 
 ```ruby
 module ATestMigrationWorker

data/README.ru.md

@@ -247,10 +247,10 @@ end
 ATestWorker.perform_async [
   { id: 0 },
   { id: 1, payload: { attr: 'v1' } },
-  { id: 2, payload: { attr: 'v1' }, score: Time.now.to_i, perform_in: Time.now.to_i },
+  { id: 2, payload: { attr: 'v1' }, score: Time.now.to_f, perform_in: Time.now.to_f },
 ]
 # payload по умолчанию равен ""
-# score и perform_in по умолчанию равны Time.now.to_i
+# score и perform_in по умолчанию равны Time.now.to_f
 ```
 
 Вы можете переопределить `perform_async` и вычислять `id`, `score` и `perform_in` в воркере:

data/docker-compose.yml

@@ -20,7 +20,7 @@ services:
     image: redis:5-alpine
 
   frontend:
-    image: node:12.16
+    image: node:12.16.3
     ports:
       - "8081:8081"
     working_dir: /usr/src/app

data/lib/lowkiq.rb

@@ -4,11 +4,12 @@ require "zlib"
 require "json"
 require "ostruct"
 require "optparse"
+require "digest"
 
 require "lowkiq/version"
 require "lowkiq/utils"
+require "lowkiq/script"
 
-require "lowkiq/extend_tracker"
 require "lowkiq/option_parser"
 
 require "lowkiq/splitters/default"
@@ -19,7 +20,6 @@ require "lowkiq/schedulers/seq"
 
 require "lowkiq/server"
 
-require "lowkiq/queue/marshal"
 require "lowkiq/queue/keys"
 require "lowkiq/queue/fetch"
 require "lowkiq/queue/queue"
@@ -40,7 +40,9 @@ module Lowkiq
                   :redis, :client_pool_size, :pool_timeout,
                   :server_middlewares, :on_server_init,
                   :build_scheduler, :build_splitter,
-                  :last_words
+                  :last_words,
+                  :dump_payload, :load_payload,
+                  :workers
 
     def server_redis_pool
       @server_redis_pool ||= ConnectionPool.new(size: threads_per_node, timeout: pool_timeout, &redis)
@@ -61,10 +63,6 @@ module Lowkiq
       end
     end
 
-    def workers
-      Worker.extended_modules
-    end
-
     def shard_handlers
       self.workers.flat_map do |w|
         ShardHandler.build_many w, self.server_wrapper
@@ -108,4 +106,7 @@ module Lowkiq
   self.build_scheduler = ->() { Lowkiq.build_lag_scheduler }
   self.build_splitter = ->() { Lowkiq.build_default_splitter }
   self.last_words = ->(ex) {}
+  self.dump_payload = ::Marshal.method :dump
+  self.load_payload = ::Marshal.method :load
+  self.workers = []
 end

data/lib/lowkiq/queue/fetch.rb

@@ -21,7 +21,7 @@ module Lowkiq
             id: x[0],
             perform_in: x[1][0],
             retry_count: x[1][1],
-            payloads: x[1][2].map { |(payload, score)| [Marshal.load_payload(payload), score] },
+            payloads: x[1][2].map { |(payload, score)| [Lowkiq.load_payload.call(payload), score] },
             error: x[1][3],
           }.compact
         end.compact
@@ -41,7 +41,7 @@ module Lowkiq
           {
             id: x[0],
             updated_at: x[1][0],
-            payloads: x[1][1].map { |(payload, score)| [Marshal.load_payload(payload), score] },
+            payloads: x[1][1].map { |(payload, score)| [Lowkiq.load_payload.call(payload), score] },
             error: x[1][2],
           }.compact
         end.compact

data/lib/lowkiq/queue/keys.rb

@@ -39,14 +39,26 @@ module Lowkiq
         [@prefix, :errors].join(':')
       end
 
-      def processing_key(shard)
-        [@prefix, :processing, shard].join(':')
-      end
-
       def processing_length_by_shard_hash
         [@prefix, :processing_length_by_shard].join(':')
       end
 
+      def processing_ids_with_perform_in_hash(shard)
+        [@prefix, :processing, :ids_with_perform_in, shard].join(':')
+      end
+
+      def processing_ids_with_retry_count_hash(shard)
+        [@prefix, :processing, :ids_with_retry_count, shard].join(':')
+      end
+
+      def processing_payloads_zset(id)
+        [@prefix, :processing, :payloads, id].join(':')
+      end
+
+      def processing_errors_hash(shard)
+        [@prefix, :processing, :errors, shard].join(':')
+      end
+
       def morgue_all_ids_lex_zset
         [@prefix, :morgue, :all_ids_lex].join(':')
       end

data/lib/lowkiq/queue/queue.rb

@@ -29,7 +29,7 @@ module Lowkiq
               redis.zadd @keys.all_ids_scored_by_retry_count_zset, retry_count, id, nx: true
 
               redis.zadd @keys.ids_scored_by_perform_in_zset(shard), perform_in, id, nx: true
-              redis.zadd @keys.payloads_zset(id), score, Marshal.dump_payload(payload), nx: true
+              redis.zadd @keys.payloads_zset(id), score, Lowkiq.dump_payload.call(payload), nx: true
             end
           end
         end
@@ -37,44 +37,56 @@ module Lowkiq
 
       def pop(shard, limit:)
         @pool.with do |redis|
-          data = nil
-          tx = redis.watch @keys.ids_scored_by_perform_in_zset(shard) do
-            ids = redis.zrangebyscore @keys.ids_scored_by_perform_in_zset(shard),
-                                      0, @timestamp.call,
-                                      limit: [0, limit]
-
-            if ids.empty?
-              redis.unwatch
-              return []
-            end
+          ids = redis.zrangebyscore @keys.ids_scored_by_perform_in_zset(shard),
+                                    0, @timestamp.call,
+                                    limit: [0, limit]
+          return [] if ids.empty?
 
-            data = @fetch.fetch(redis, :pipelined, ids)
+          res = redis.multi do |redis|
+            redis.hset @keys.processing_length_by_shard_hash, shard, ids.length
 
-            redis.multi do
-              _delete redis, ids
-              redis.set  @keys.processing_key(shard), Marshal.dump_data(data)
-              redis.hset @keys.processing_length_by_shard_hash, shard, data.length
+            ids.each do |id|
+              redis.zrem @keys.all_ids_lex_zset, id
+              redis.zrem @keys.ids_scored_by_perform_in_zset(shard), id
+
+              Script.zremhset redis,
+                              @keys.all_ids_scored_by_perform_in_zset,
+                              @keys.processing_ids_with_perform_in_hash(shard),
+                              id
+              Script.zremhset redis,
+                              @keys.all_ids_scored_by_retry_count_zset,
+                              @keys.processing_ids_with_retry_count_hash(shard),
+                              id
+              redis.rename @keys.payloads_zset(id),
+                           @keys.processing_payloads_zset(id)
+              Script.hmove redis,
+                           @keys.errors_hash,
+                           @keys.processing_errors_hash(shard),
+                           id
             end
-          end until tx
+            processing_data_pipeline(redis, shard, ids)
+          end
 
-          data
+          res.shift 1 + ids.length * 6
+          processing_data_build res, ids
         end
       end
 
       def push_back(batch)
         @pool.with do |redis|
-          batch.each do |job|
-            id          = job.fetch(:id)
-            perform_in  = job.fetch(:perform_in, @timestamp.call)
-            retry_count = job.fetch(:retry_count, -1)
-            payloads    = job.fetch(:payloads).map do |(payload, score)|
-              [score, Marshal.dump_payload(payload)]
-            end
-            error       = job.fetch(:error, nil)
+          timestamp = @timestamp.call
+          redis.multi do |redis|
+            batch.each do |job|
+              id          = job.fetch(:id)
+              perform_in  = job.fetch(:perform_in, timestamp)
+              retry_count = job.fetch(:retry_count, -1)
+              payloads    = job.fetch(:payloads).map do |(payload, score)|
+                [score, Lowkiq.dump_payload.call(payload)]
+              end
+              error       = job.fetch(:error, nil)
 
-            shard = id_to_shard id
+              shard = id_to_shard id
 
-            redis.multi do
               redis.zadd @keys.all_ids_lex_zset, 0, id
               redis.zadd @keys.all_ids_scored_by_perform_in_zset, perform_in, id
               redis.zadd @keys.all_ids_scored_by_retry_count_zset, retry_count, id
@@ -88,40 +100,52 @@ module Lowkiq
         end
       end
 
-      def ack(shard, result = nil)
+      def ack(shard, data, result = nil)
+        ids = data.map { |job| job[:id] }
+        length = ids.length
+
         @pool.with do |redis|
-          length = redis.hget(@keys.processing_length_by_shard_hash, shard).to_i
           redis.multi do
-            redis.del  @keys.processing_key(shard)
+            redis.del @keys.processing_ids_with_perform_in_hash(shard)
+            redis.del @keys.processing_ids_with_retry_count_hash(shard)
+            redis.del @keys.processing_errors_hash(shard)
+            ids.each do |id|
+              redis.del @keys.processing_payloads_zset(id)
+            end
             redis.hdel @keys.processing_length_by_shard_hash, shard
-
             redis.incrby @keys.processed_key, length if result == :success
-            redis.incrby @keys.failed_key, length    if result == :fail
+            redis.incrby @keys.failed_key,    length if result == :fail
           end
         end
       end
 
       def processing_data(shard)
-        data = @pool.with do |redis|
-          redis.get @keys.processing_key(shard)
-        end
-        return [] if data.nil?
+        @pool.with do |redis|
+          ids = redis.hkeys @keys.processing_ids_with_perform_in_hash(shard)
+          return [] if ids.empty?
 
-        Marshal.load_data data
+          res = redis.multi do |redis|
+            processing_data_pipeline redis, shard, ids
+          end
+
+          processing_data_build res, ids
+        end
       end
 
       def push_to_morgue(batch)
         @pool.with do |redis|
-          batch.each do |job|
-            id       = job.fetch(:id)
-            payloads = job.fetch(:payloads).map do |(payload, score)|
-              [score, Marshal.dump_payload(payload)]
-            end
-            error    = job.fetch(:error, nil)
+          timestamp = @timestamp.call
+          redis.multi do
+            batch.each do |job|
+              id       = job.fetch(:id)
+              payloads = job.fetch(:payloads).map do |(payload, score)|
+                [score, Lowkiq.dump_payload.call(payload)]
+              end
+              error    = job.fetch(:error, nil)
+
 
-            redis.multi do
               redis.zadd @keys.morgue_all_ids_lex_zset, 0, id
-              redis.zadd @keys.morgue_all_ids_scored_by_updated_at_zset, @timestamp.call, id
+              redis.zadd @keys.morgue_all_ids_scored_by_updated_at_zset, timestamp, id
               redis.zadd @keys.morgue_payloads_zset(id), payloads, nx: true
 
               redis.hset @keys.morgue_errors_hash, id, error unless error.nil?
@@ -146,7 +170,15 @@ module Lowkiq
       def delete(ids)
         @pool.with do |redis|
           redis.multi do
-            _delete redis, ids
+            ids.each do |id|
+              shard = id_to_shard id
+              redis.zrem @keys.all_ids_lex_zset, id
+              redis.zrem @keys.all_ids_scored_by_perform_in_zset, id
+              redis.zrem @keys.all_ids_scored_by_retry_count_zset, id
+              redis.zrem @keys.ids_scored_by_perform_in_zset(shard), id
+              redis.del  @keys.payloads_zset(id)
+              redis.hdel @keys.errors_hash, id
+            end
           end
         end
       end
@@ -161,17 +193,33 @@ module Lowkiq
         Zlib.crc32(id.to_s) % @shards_count
       end
 
-      def _delete(redis, ids)
+      def processing_data_pipeline(redis, shard, ids)
+        redis.hgetall @keys.processing_ids_with_perform_in_hash(shard)
+        redis.hgetall @keys.processing_ids_with_retry_count_hash(shard)
+        redis.hgetall @keys.processing_errors_hash(shard)
+
         ids.each do |id|
-          shard = id_to_shard id
-          redis.zrem @keys.all_ids_lex_zset, id
-          redis.zrem @keys.all_ids_scored_by_perform_in_zset, id
-          redis.zrem @keys.all_ids_scored_by_retry_count_zset, id
-          redis.zrem @keys.ids_scored_by_perform_in_zset(shard), id
-          redis.del  @keys.payloads_zset(id)
-          redis.hdel @keys.errors_hash, id
+          redis.zrange @keys.processing_payloads_zset(id), 0, -1, with_scores: true
         end
       end
+
+      def processing_data_build(arr, ids)
+        ids_with_perform_in = arr.shift
+        ids_with_retry_count = arr.shift
+        errors = arr.shift
+        payloads = arr
+
+        ids.zip(payloads).map do |(id, payloads)|
+          next if payloads.empty?
+          {
+            id: id,
+            perform_in: ids_with_perform_in[id].to_f,
+            retry_count: ids_with_retry_count[id].to_f,
+            payloads: payloads.map { |(payload, score)| [Lowkiq.load_payload.call(payload), score] },
+            error: errors[id]
+          }.compact
+        end.compact
+      end
     end
   end
 end

data/lib/lowkiq/queue/queue_metrics.rb

@@ -64,11 +64,9 @@ module Lowkiq
 
       def coerce_lag(res)
         _id, score = res.first
-
-        return 0 if score.nil?
-        return 1 if score == 0 # на случай Actions#perform_all_jobs_now
-        lag = @timestamp.call - score.to_i
-        return 0 if lag < 0
+        return 0.0 if score.nil?
+        lag = @timestamp.call - score
+        return 0.0 if lag < 0.0
         lag
       end
 

data/lib/lowkiq/queue/shard_metrics.rb

@@ -41,10 +41,9 @@ module Lowkiq
 
       def coerce_lag(res)
         _id, score = res.first
-
-        return 0 if score.nil?
-        lag = @timestamp.call - score.to_i
-        return 0 if lag < 0
+        return 0.0 if score.nil?
+        lag = @timestamp.call - score
+        return 0.0 if lag < 0.0
         lag
       end
     end

data/lib/lowkiq/schedulers/lag.rb

@@ -12,7 +12,7 @@ module Lowkiq
           metrics = @metrics.call identifiers
           shard_handler, _lag =
                          shard_handlers.zip(metrics.map(&:lag))
-                           .select { |(_, lag)| lag > 0 }
+                           .select { |(_, lag)| lag > 0.0 }
                            .max_by { |(_, lag)| lag }
 
           if shard_handler

data/lib/lowkiq/script.rb

@@ -0,0 +1,42 @@
+module Lowkiq
+  module Script
+    module_function
+
+    ALL = {
+      hmove: <<-LUA,
+        local source = KEYS[1]
+        local destination = KEYS[2]
+        local key = ARGV[1]
+        local value = redis.call('hget', source, key)
+        if value then
+          redis.call('hdel', source, key)
+          redis.call('hset', destination, key, value)
+        end
+      LUA
+      zremhset: <<-LUA
+        local source = KEYS[1]
+        local destination = KEYS[2]
+        local member = ARGV[1]
+        local score = redis.call('zscore', source, member)
+        if score then
+          redis.call('zrem', source, member)
+          redis.call('hset', destination, member, score)
+        end
+      LUA
+    }.transform_values { |v| { sha: Digest::SHA1.hexdigest(v), source: v } }.freeze
+
+    def load!(redis)
+      ALL.each do |_, item|
+        redis.script(:load, item[:source])
+      end
+    end
+
+    def hmove(redis, source, destination, key)
+      redis.evalsha ALL[:hmove][:sha], keys: [source, destination], argv: [key]
+    end
+
+    def zremhset(redis, source, destination, member)
+      redis.evalsha ALL[:zremhset][:sha], keys: [source, destination], argv: [member]
+    end
+  end
+end

data/lib/lowkiq/server.rb

@@ -17,6 +17,10 @@ module Lowkiq
     end
 
     def start
+      Lowkiq.server_redis_pool.with do |redis|
+        Script.load! redis
+      end
+
       @shard_handlers_by_thread.each do |handlers|
         handlers.each(&:restore)
       end

data/lib/lowkiq/shard_handler.rb

@@ -31,7 +31,7 @@ module Lowkiq
           @worker.perform batch
         end
 
-        @queue.ack @shard_index, :success
+        @queue.ack @shard_index, data, :success
         true
       rescue => ex
         fail! data, ex
@@ -39,7 +39,7 @@ module Lowkiq
 
         @queue.push_back back
         @queue.push_to_morgue morgue
-        @queue.ack @shard_index, :fail
+        @queue.ack @shard_index, data, :fail
         false
       end
     end
@@ -48,7 +48,7 @@ module Lowkiq
       data = @queue.processing_data @shard_index
       return if data.nil?
       @queue.push_back data
-      @queue.ack @shard_index
+      @queue.ack @shard_index, data
     end
 
     private

data/lib/lowkiq/utils.rb

@@ -29,7 +29,7 @@ module Lowkiq
 
     module Timestamp
       def self.now
-        Time.now.to_i
+        Time.now.to_f
       end
     end
   end

data/lib/lowkiq/version.rb

@@ -1,3 +1,3 @@
 module Lowkiq
-  VERSION = "1.0.2"
+  VERSION = "1.1.0"
 end

data/lib/lowkiq/web.rb

@@ -28,6 +28,7 @@ module Lowkiq
 
     APP = Rack::Builder.new do
       map "/api" do
+        use Rack::ContentType, "application/json"
         run Api
       end
 
@@ -35,6 +36,7 @@ module Lowkiq
         run Rack::File.new ASSETS, { 'Cache-Control' => 'public, max-age=86400' }
       end
 
+      use Rack::ContentType, "text/html"
       run HTML
     end
 

data/lib/lowkiq/web/api.rb

@@ -18,7 +18,7 @@ module Lowkiq
           total = {
             length:        metrics.map(&:length).reduce(&:+).to_i,
             morgue_length: metrics.map(&:morgue_length).reduce(&:+).to_i,
-            lag:           metrics.map(&:lag).max.to_i,
+            lag:           metrics.map(&:lag).max.to_f,
           }
           {
             total: total,

data/lib/lowkiq/worker.rb

@@ -1,7 +1,5 @@
 module Lowkiq
   module Worker
-    extend ExtendTracker
-
     attr_accessor :shards_count,
                   :batch_size,
                   :max_retry_count,

data/lowkiq.gemspec

@@ -34,4 +34,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "rspec", "~> 3.0"
   spec.add_development_dependency "rspec-mocks", "~> 3.8"
   spec.add_development_dependency "rack-test", "~> 1.1"
+  spec.add_development_dependency "pry-byebug", "~> 3.9.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: lowkiq
 version: !ruby/object:Gem::Version
-  version: 1.0.2
+  version: 1.1.0
 platform: ruby
 authors:
 - Mikhail Kuzmin
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-03-17 00:00:00.000000000 Z
+date: 2021-01-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
@@ -134,6 +134,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.1'
+- !ruby/object:Gem::Dependency
+  name: pry-byebug
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.9.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 3.9.0
 description: Lowkiq
 email:
 - Mihail.Kuzmin@bia-tech.ru
@@ -144,6 +158,7 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - ".rspec"
+- CHANGELOG.md
 - Gemfile
 - Gemfile.lock
 - LICENSE.md
@@ -157,12 +172,10 @@ files:
 - docker-compose.yml
 - exe/lowkiq
 - lib/lowkiq.rb
-- lib/lowkiq/extend_tracker.rb
 - lib/lowkiq/option_parser.rb
 - lib/lowkiq/queue/actions.rb
 - lib/lowkiq/queue/fetch.rb
 - lib/lowkiq/queue/keys.rb
-- lib/lowkiq/queue/marshal.rb
 - lib/lowkiq/queue/queries.rb
 - lib/lowkiq/queue/queue.rb
 - lib/lowkiq/queue/queue_metrics.rb
@@ -170,6 +183,7 @@ files:
 - lib/lowkiq/redis_info.rb
 - lib/lowkiq/schedulers/lag.rb
 - lib/lowkiq/schedulers/seq.rb
+- lib/lowkiq/script.rb
 - lib/lowkiq/server.rb
 - lib/lowkiq/shard_handler.rb
 - lib/lowkiq/splitters/by_node.rb
@@ -201,7 +215,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: Lowkiq

data/lib/lowkiq/extend_tracker.rb

@@ -1,13 +0,0 @@
-module Lowkiq
-  module ExtendTracker
-    def extended(mod)
-      @extended_modules ||= []
-      @extended_modules << mod
-      @extended_modules.sort_by!(&:name).uniq!
-    end
-
-    def extended_modules
-      @extended_modules
-    end
-  end
-end

data/lib/lowkiq/queue/marshal.rb

@@ -1,23 +0,0 @@
-module Lowkiq
-  module Queue
-    module Marshal
-      class << self
-        def dump_payload(data)
-          ::Marshal.dump data
-        end
-
-        def load_payload(str)
-          ::Marshal.load str
-        end
-
-        def dump_data(data)
-          ::Marshal.dump data
-        end
-
-        def load_data(str)
-          ::Marshal.load str
-        end
-      end
-    end
-  end
-end