jiggler 0.1.0.rc4 → 0.1.0.rc5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4e6545c58aa0c8c04bbeb879c82e1a954eb0e13bb92971a148aaaaeaf17d1b0c
-  data.tar.gz: fa12547490937572301bdee5024988c233d363f574cbf92467e3bfa9fe16b8ff
+  metadata.gz: 215183a38b3a7a32e692478a7aa6126fb0f78f36891708ab95e657c85e0a0c6f
+  data.tar.gz: bfc6e357e0917dd2175aa5bb2454a3243685f383199f69c4c58354502a460bfd
 SHA512:
-  metadata.gz: 464c1b31e6d0b200fdaad5c6a1da857a0a9ecabc7e239202d483178a15293d96f27b866d86287f9c91576294f794dd799393eaf11694858a0d664b3b96138dfb
-  data.tar.gz: 5e407cea521fc909160dbc637e9716f09026cc98c98e0397c129cc62dd891c20d6b582947c3944dadd2ff6165d561fd34feffc4988af7202bcc8410c87beb07c
+  metadata.gz: 006d0b3cc279afcf64f8abc2f75e46b4ff62a2668ee34a2fdb61bd8f4cd455b92593073697e27583d840a46b676cf90523dd470fe8dd4467c93e7f922d7a56c9
+  data.tar.gz: 49c1bdb550fd4721980130f7f63ffbc0b628d3a373123b52ac248de99b747cd67d2b7addf8bb7175b33b862f0e51a6b06031f62c70bf3dd7e76b2d53f13675c1

data/README.md

@@ -7,7 +7,7 @@ Jiggler is a [Sidekiq](https://github.com/mperham/sidekiq)-inspired background j
 
 *Jiggler is based on Sidekiq implementation, and re-uses most of its concepts and ideas.*
 
-**NOTE**: Altrough some performance results may look interesting, it's absolutly not recommended to switch to it from well-tested stable solutions. Jiggler has a meager set of features and a very basic monitoring. It's a small indie gem made purely for fun and to gain some hand-on experience with async and fibers. It isn't tested with production projects and might have not-yet-discovered issues. \
+**NOTE**: Jiggler is a small gem made purely for fun and to gain some hand-on experience with async and fibers. It isn't tested with production projects and might have not-yet-discovered issues. Use at your own risk. \
 However, it's good to play around and/or to try it in the name of science.
 
 ### Installation
@@ -17,6 +17,11 @@ Install the gem:
 gem install jiggler
 ```
 
+Use `--pre` for release candidates:
+```
+gem install jiggler --pre
+```
+
 Start Jiggler server as a separate process with bin command:
 ```
 jiggler -r <FILE_PATH>
@@ -28,204 +33,7 @@ Run `jiggler --help` to see the list of command line arguments.
 
 ### Performance
 
-The tests were run on local (Ubuntu 22.04, Intel(R) Core(TM) i7 6700HQ 2.60GHz). \
-On the other configurations the results may differ significantly, f.e. with Apple M1 Max chip it treats some IO operations as blocking and shows a poor performance ಠ_ಥ.
-
-Ruby `3.2.0` \
-Redis `7.0.7` \
-Poller interval `5s` \
-Monitoring interval `10s` \
-Logging level `WARN`
-
-#### Noop task measures
-
-```ruby
-def perform
-  # just an empty job doing nothing
-end
-```
-
-The parent process enqueues the jobs, starts the monitoring, and then forks the child job-processor-process. Thus, `RSS` value is affected by the number of jobs uploaded in the parent process. See `bin/jigglerload` to see the load test structure and measuring. \
-1_000_000 jobs were enqueued in 100k batches x10.
-
-| Job Processor    | Concurrency | Jobs      | Time      | Start RSS  | Finish RSS    |
-|------------------|-------------|-----------|-----------|------------|---------------|
-| Sidekiq 7.0.2    | 5           | 100_000   | 20.01 sec | 132_080 kb | 103_168 kb (GC) |
-| Jiggler 0.1.0    | 5           | 100_000   | 14.25 sec | 87_532 kb  | 91_464 kb |
-| -                |             |           |           |            |           |
-| Sidekiq 7.0.2    | 10          | 100_000   | 20.49 sec | 132_164 kb | 125_768 kb (GC) |
-| Jiggler 0.1.0    | 10          | 100_000   | 13.25 sec | 87_688 kb  | 91_625 kb |
-| -                |             |           |           |            |           |
-| Sidekiq 7.0.2    | 5           | 1_000_000 | 186.90 sec | 159_712 kb | 186_224 kb |
-| Jiggler 0.1.0    | 5           | 1_000_000 | 123.13 sec | 113_212 kb | 116_336 kb |
-| -                |             |           |            |            |            |
-| Sidekiq 7.0.2    | 10          | 1_000_000 | 186.94 sec | 159_000 kb | 192_780 kb |
-| Jiggler 0.1.0    | 10          | 1_000_000 | 115.56 sec | 113_656 kb | 116_896 kb |
-
-
-#### IO tests
-
-The idea of the next tests is to simulate jobs with different kinds of IO tasks. \
-Ruby 3 has introduced fiber scheduler interface, which allows to implement hooks related to IO/blocking operations.\
-The context switching won't work well in case IO is performed by C-extentions which are not aware of Ruby scheduler. 
-
-##### NET/HTTP requests
-
-Spin-up a local sinatra app to exclude network issues while testing HTTP requests (it uses `falcon` web server).
-
-```ruby
-require "sinatra"
-
-class MyApp < Sinatra::Base
-  get "/hello" do
-    sleep(0.2)
-    "Hello World!"
-  end
-end
-```
-
-Then, the code which is going to be performed within the workers should make a `net/http` request to the local endpoint.
-
-```ruby
-# a single job takes ~0.21s to perform
-def perform
-  uri = URI("http://127.0.0.1:9292/hello")
-  res = Net::HTTP.get_response(uri)
-  puts "Request Error!!!" unless res.is_a?(Net::HTTPSuccess)
-end
-```
-
-It's not recommended to run sidekiq with high concurrency values, setting it for the sake of test. \
-The time difference for these samples is small-ish, however the memory consumption is less with the fibers. \
-Since fibers have relatively small memory foot-print and context switching is also relatively cheap, it's possible to set concurrency to higher values within Jiggler without too much trade-offs.
-
-| Job Processor    | Concurrency | Jobs  | Time to complete  | Start RSS | Finish RSS | %CPU |
-|------------------|-------------|-------|-------------------|-----------|------------|------|
-| Sidekiq 7.0.2    | 5           | 1_000 | 43.74 sec         | 30_444 kb | 45_124 kb  | 5.9 |
-| Jiggler 0.1.0    | 5           | 1_000 | 43.65 sec         | 33_476 kb | 34_144 kb  | 2.9 |
-| -                |             |       |                   |           |            |      |
-| Sidekiq 7.0.2    | 10          | 1_000 | 23.05 sec         | 30_604 kb | 50_292 kb  | 10.93 |
-| Jiggler 0.1.0    | 10          | 1_000 | 22.86 sec         | 32_416 kb | 34_128 kb  | 5.69 |
-| -                |             |       |                   |           |            |      |
-| Sidekiq 7.0.2    | 15          | 1_000 | 16.17 sec         | 30_636 kb | 55_144 kb  | 16.47 |
-| Jiggler 0.1.0    | 15          | 1_000 | 15.87 sec         | 33_328 kb | 34_548 kb  | 8.25 |
-
-**NOTE**: Jiggler has more dependencies, so with small load `start RSS` takes more space.
-
-##### PostgreSQL connection/queries
-
-`pg` gem supports Ruby's `Fiber.scheduler` starting from 1.3.0 version. Make sure yours DB-adapter supports it.
-
-```ruby
-### global namespace
-require "pg"
-
-$pg_pool = ConnectionPool.new(size: CONCURRENCY) do
-  PG.connect({ dbname: "test", password: "test", user: "test" })
-end
-
-### worker context
-# a single job takes ~0.102s to perform
-def perform
-  $pg_pool.with do |conn|
-    conn.exec("SELECT pg_sleep(0.1)")
-  end
-end
-```
-
-| Job Processor    | Concurrency | Jobs  | Time      | Start RSS | Finish RSS | %CPU |
-|------------------|-------------|-------|-----------|-----------|------------|------|
-| Sidekiq 7.0.2    | 5           | 1_000 | 23.44 sec | 31_436 kb | 48_856 kb  | 7.56 |
-| Jiggler 0.1.0    | 5           | 1_000 | 23.20 sec | 35_312 kb | 38_592 kb  | 2.91 |
-| -                |             |       |           |           |            |      |
-| Sidekiq 7.0.2    | 10          | 1_000 | 13.15 sec | 31_272 kb | 52_808 kb  | 13.76 |
-| Jiggler 0.1.0    | 10          | 1_000 | 12.65 sec | 35_296 kb | 38_784 kb  | 6.11 |
-| -                |             |       |           |           |            |      |
-| Sidekiq 7.0.2    | 15          | 1_000 | 9.63 sec  | 31_016 kb | 59_868 kb  | 20.32 |
-| Jiggler 0.1.0    | 15          | 1_000 | 9.17 sec  | 35_188 kb | 38_948 kb  | 9.26 |
-
-##### File IO
-
-```ruby
-def perform(file_name, id)
-  File.open(file_name, "a") { |f| f.write("#{id}\n") }
-end
-```
-
-| Job Processor    | Concurrency | Jobs   | Time      | Start RSS    | Finish RSS | %CPU  |
-|------------------|-------------|--------|-----------|--------------|------------|-------|
-| Sidekiq 7.0.2    | 5           | 30_000 | 11.94 sec | 61_944 kb    | 71_948 kb  | 94.34 |
-| Jiggler 0.1.0    | 5           | 30_000 | 7.87 sec  | 50_140 kb    | 51_272 kb  | 61.7  |
-| -                |             |        |           |              |            |       |
-| Sidekiq 7.0.2    | 10          | 30_000 | 11.6 sec  | 62_020 kb    | 78_952 kb  | 94.44 |
-| Jiggler 0.1.0    | 10          | 30_000 | 7.17 sec  | 50_060 kb    | 51_464 kb  | 69.25 |
-| -                |             |        |           |              |            |       |
-| Sidekiq 7.0.2    | 15          | 30_000 | 11.24 sec | 62_016 kb    | 83_808 kb  | 94.16 |
-| Jiggler 0.1.0    | 15          | 30_000 | 7.02 sec  | 49_988 kb    | 51_428 kb  | 70.3  |
-
-
-Jiggler is effective only for tasks with a lot of IO. You must test the concurrency setting with your jobs to find out what configuration works best for your payload.
-
-#### Simulate CPU-only job
-
-With CPU-heavy jobs Jiggler has poor performance. Just to make sure it's generally able to work with CPU-only payloads:
-
-```ruby
-def fib(n)
-  if n <= 1
-    1
-  else
-    (fib(n-1) + fib(n-2))
-  end
-end
-
-# a single job takes ~0.035s to perform
-def perform(_idx)
-  fib(25)
-end
-```
-
-| Job Processor    | Concurrency | Jobs | Time     | Start RSS  | Finish RSS |
-|------------------|-------------|------|----------|------------|------------|
-| Sidekiq 7.0.2    | 5           | 100  | 5.81 sec | 27_792 kb  | 42_464 kb |
-| Jiggler 0.1.0    | 5           | 100  | 5.29 sec | 31_304 kb  | 32_320 kb |
-| -                |             |      |          |            |           |
-| Sidekiq 7.0.2    | 10          | 100  | 5.63 sec | 28_044 kb  | 47_640 kb |
-| Jiggler 0.1.0    | 10          | 100  | 5.43 sec | 32_316 kb  | 32_548 kb |
-
-#### IO Event selector
-
-`IO_EVENT_SELECTOR` is an env variable which allows to specify the event selector used by the Ruby scheduler. \
-On default it uses `Epoll` (`IO_EVENT_SELECTOR=EPoll`). \
-Another available option is `URing` (`IO_EVENT_SELECTOR=URing`). Underneath it uses `io_uring` library. It is a Linux kernel library that provides a high-performance interface for asynchronous I/O operations. It was introduced in Linux kernel version 5.1 and aims to address some of the limitations and scalability issues of the existing AIO (Asynchronous I/O) interface.
-In the future it might bring a lot of performance boost into Ruby fibers world (once `async` project fully adopts it), but at the moment in the most cases its performance is similar to `EPoll`, yet it could give some boost with File IO.
-
-#### Socketry stack
-
-The gem allows to use libs from `socketry` stack (https://github.com/socketry) within workers. \
-F.e. when making HTTP requests using `async/http/internet` to the Sinatra app described above:
-
-```ruby
-### global namespace
-require "async/http/internet"
-$internet = Async::HTTP::Internet.new
-
-### worker context
-def perform
-  uri = "https://127.0.0.1/hello"
-  res = $internet.get(uri)
-  res.finish
-  puts "Request Error!!!" unless res.status == 200
-end
-```
-
-| Job Processor    | Concurrency | Jobs  | Time      | Start RSS | Finish RSS | %CPU |
-|------------------|-------------|-------|-----------|-----------|------------|------|
-| Jiggler 0.1.0    | 5           | 1_000 | 43.23 sec | 34_340 kb | 38_488 kb  | 1.51 |
-| -                |             |       |           |           |            |      |
-| Jiggler 0.1.0    | 10          | 1_000 | 22.67 sec | 34_552 kb | 38_600 kb  | 2.75 |
-| -                |             |       |           |           |            |      |
-| Jiggler 0.1.0    | 15          | 1_000 | 15.88 sec | 34_332 kb | 38_544 kb  | 4.06 |
+[Jiggler 0.1.0rc4 performance results](/docs/perf_results_0.1.0rc4.md)
 
 ### Getting Started
 
@@ -255,9 +63,34 @@ Jiggler.configure do |config|
   config[:redis_url]   = ENV["REDIS_URL"] # On default fetches the value from ENV["REDIS_URL"]
   config[:queues]      = ["shippers"]     # An array of queue names the server is going to listen to. On default uses ["default"]
   config[:config_file] = "./jiggler.yml"  # .yml file with Jiggler settings
+  config[:mode]        = :at_most_once    # at_most_once and at_least_once modes supported. Defaults to :at_least_once
+end
+```
+
+`at_least_once` mode grants reliability for the regular enqueued jobs which are going to be executed by workers. The scheduled jobs (the ones planned to be executed at a specific time, or which failed and going to be retried) still support only `at_most_once` strategy. The support for them is going to be added in the upcoming versions.
+
+On default all queues have the same priority (equals to 0). Higher number means higher prio. \
+It's possible to specify custom priorities as follows:
+
+```ruby
+Jiggler.configure do |config|
+  config[:queues] = [["shippers", 0], ["shipments", 1], ["delivery", 2]]
 end
 ```
 
+#### IO Event selector
+
+`IO_EVENT_SELECTOR` is an env variable which allows to specify the event selector used by the Ruby scheduler. \
+On default it uses `Epoll` (`IO_EVENT_SELECTOR=EPoll`). \
+Another available option is `URing` (`IO_EVENT_SELECTOR=URing`). Underneath it uses `io_uring` library. It is a Linux kernel library that provides a high-performance interface for asynchronous I/O operations. It was introduced in Linux kernel version 5.1 and aims to address some of the limitations and scalability issues of the existing AIO (Asynchronous I/O) interface.
+In the future it might bring a lot of performance boost into Ruby fibers world (once `async` project fully adopts it), but at the moment in the most cases its performance is similar to `EPoll`, yet it could give some boost with File IO.
+
+#### Socketry stack
+
+The gem allows to use libs from `socketry` stack (https://github.com/socketry) within workers.
+
+#### Core concepts
+
 Internally Jiggler server consists of 3 parts: `Manager`, `Poller`, `Monitor`. \
 `Manager` is responsible for workers. \
 `Poller` fetches data for retries and scheduled jobs. \
@@ -377,7 +210,7 @@ Jiggler.configure_client do |config|
   config[:client_redis_pool] = my_async_redis_pool
 end
 
-# or use build-in async pool with
+# or use built-in async pool with
 require "async/pool"
 
 Jiggler.configure_client do |config|
@@ -416,3 +249,7 @@ docker-compose run --rm web -- bundle exec rspec
 ```
 
 To run the load tests modify the `docker-compose.yml` to point to `bin/jigglerload`
+
+### Contributing
+
+Fork & Pull Request.

data/bin/jiggler

@@ -9,7 +9,7 @@ begin
   cli.parse_and_init
   
   cli.config.logger.info("Jiggler is starting in #{Jiggler.config[:environment].upcase} ✯⸜(*❛‿❛)⸝✯")
-  cli.config.logger.info("Jiggler version=#{Jiggler::VERSION} pid=#{Process.pid} concurrency=#{cli.config[:concurrency]} queues=#{cli.config[:queues].join(',')}")
+  cli.config.logger.info("Jiggler version=#{Jiggler::VERSION} pid=#{Process.pid} concurrency=#{cli.config[:concurrency]} queues=#{cli.config.sorted_queues.join(',')}")
 
   cli.start
 rescue => e

data/lib/jiggler/at_least_once/acknowledger.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Jiggler
+  module AtLeastOnce
+    class Acknowledger < BaseAcknowledger
+      def initialize(config)
+        super
+        @runners = []
+        @queue = Queue.new
+      end
+
+      def ack(job)
+        @queue.push(job)
+      end
+  
+      def start
+        @config[:concurrency].times do
+          @runners << safe_async('Acknowledger') do
+            while (job = @queue.pop) != nil
+              begin
+                job.ack
+              rescue StandardError => err
+                log_error(err, context: '\'Could not acknowledge a job\'', job: job)
+              end
+            end
+            logger.debug('Acknowledger exits')
+          rescue Async::Stop
+            logger.debug('Acknowledger received stop signal')
+          end
+        end
+      end
+      
+      def wait
+        @runners.each(&:wait)
+      end
+  
+      def terminate
+        logger.debug('Suspending the acknowledger')
+        @queue.close
+      end
+    end
+  end
+end

data/lib/jiggler/at_least_once/fetcher.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+require 'fc'
+
+module Jiggler
+  module AtLeastOnce
+    class Fetcher < BaseFetcher
+      TIMEOUT = 2.0 # 2 seconds of waiting for brpoplpush
+      RESERVE_QUEUE_SUFFIX = 'in_progress'
+
+      attr_reader :producers
+
+      def initialize(config, collection)
+        super
+        @tasks_queue = FastContainers::PriorityQueue.new(:min)
+        @condition = Async::Notification.new
+        @consumers_queue = Queue.new
+      end
+
+      CurrentJob = Struct.new(:queue, :args, :reserve_queue, :config, keyword_init: true) do
+        def ack
+          config.with_sync_redis do |conn|
+            conn.call('LREM', reserve_queue, 1, args)
+          end
+        end
+      end
+
+      def start
+        config.sorted_queues_data.each do |queue, data|
+          config[:fetchers_concurrency].times do
+            safe_async("'Fetcher for #{queue}'") do
+              list = data[:list]
+              rlist = in_process_queue(list)
+              loop do
+                if @consumers_queue.num_waiting.zero? && !@done
+                  @condition.wait # supposed to block here until consumers notify
+                end
+                break if @done
+
+                args = config.with_sync_redis do |conn|
+                  conn.blocking_call(false, 'BRPOPLPUSH', list, rlist, TIMEOUT)
+                end
+                # no requeue logic rn as we expect monitor to handle
+                # in-process-tasks list for this process
+                break if @done
+
+                next if args.nil? 
+
+                @tasks_queue.push(job(list, args, rlist), data[:priority]) 
+                @consumers_queue.push('') # to unblock any waiting consumer
+              end
+              logger.debug("Fetcher for #{queue} stopped")
+            rescue Async::Stop
+              logger.debug("Fetcher for #{queue} received stop signal")
+            end
+          end
+        end
+      end
+
+      def fetch
+        @condition.signal if signal?
+        return :done if @consumers_queue.pop.nil?
+
+        @tasks_queue.pop
+      end
+
+      def suspend
+        logger.debug("Suspending the fetcher")
+        @done = true
+        @condition.signal
+        @consumers_queue.close # unblocks awaiting consumers
+      end
+
+      private
+
+      def queue_signaling_limit
+        @queue_signaling_limit ||= [config[:concurrency] / 2, 1].max
+      end
+
+      def signal?
+        @consumers_queue.size < config[:concurrency]
+      end
+
+      def in_process_queue(queue)
+        "#{queue}:#{RESERVE_QUEUE_SUFFIX}:#{collection.uuid}"
+      end
+
+      def job(queue, args, reserve_queue)
+        CurrentJob.new(queue:, args:, reserve_queue:, config:)
+      end
+    end
+  end
+end

data/lib/jiggler/at_most_once/acknowledger.rb

@@ -0,0 +1,21 @@
+module Jiggler
+  module AtMostOnce
+    class Acknowledger < BaseAcknowledger
+      def ack(job)
+        # noop
+      end
+  
+      def start
+        # noop
+      end
+      
+      def wait
+        # noop
+      end
+  
+      def terminate
+        # noop
+      end
+    end
+  end
+end

data/lib/jiggler/at_most_once/fetcher.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Jiggler
+  module AtMostOnce
+    class Fetcher < BaseFetcher
+      TIMEOUT = 2.0 # 2 seconds of waiting for brpop
+
+      CurrentJob = Struct.new(:queue, :args)
+
+      def start
+        # noop, we just block directly during the fetch
+      end
+
+      def fetch
+        return :done if @done
+
+        q, args = config.with_sync_redis do |conn|
+          conn.blocking_call(false, 'BRPOP', *config.sorted_lists, TIMEOUT)
+        end
+
+        if @done
+          requeue(q, args) unless q.nil?
+          return :done
+        end
+
+        job(q, args) unless q.nil?
+      end
+
+      def suspend
+        @done = true
+      end
+
+      private
+
+      def requeue(queue, args)
+        config.with_sync_redis do |conn|
+          conn.call('RPUSH', queue, args)
+        end
+      end
+
+      def job(queue, args)
+        CurrentJob.new(queue, args)
+      end
+    end
+  end
+end

data/lib/jiggler/base_acknowledger.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module Jiggler
+  class BaseAcknowledger
+    include Support::Helper
+
+    def initialize(config)
+      @config = config
+    end
+  end
+end

data/lib/jiggler/base_fetcher.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module Jiggler
+  class BaseFetcher
+    include Support::Helper
+
+    attr_reader :config, :collection
+
+    def initialize(config, collection)
+      @config = config
+      @collection = collection
+    end
+  end
+end

data/lib/jiggler/cleaner.rb

@@ -108,15 +108,24 @@ module Jiggler
     def prn_dead_set(conn)
       conn.call('DEL', config.dead_set)
     end
-
+    
+    # it deletes in_progress queues as well
     def prn_all_queues(conn)
-      queues = conn.call('SCAN', '0', 'MATCH', config.queue_scan_key).last
-      conn.call('DEL', *queues) unless queues.empty?
+      cursor = '0'
+      loop do
+        cursor, queues = conn.call('SCAN', cursor, 'MATCH', config.queue_scan_key)
+        conn.call('DEL', *queues) unless queues.empty?
+        break if cursor == '0'
+      end
     end
 
     def prn_all_processes(conn)
-      processes = conn.call('SCAN', '0', 'MATCH', config.process_scan_key).last
-      conn.call('DEL', *processes) unless processes.empty?
+      cursor = '0'
+      loop do
+        cursor, processes = conn.call('SCAN', cursor, 'MATCH', config.process_scan_key)
+        conn.call('DEL', *processes) unless processes.empty?
+        break if cursor == '0'
+      end
     end
 
     def prn_failures_counter(conn)

data/lib/jiggler/cli.rb

@@ -3,10 +3,11 @@
 require 'singleton'
 require 'optparse'
 require 'yaml'
-require 'set'
+require 'erb'
 require 'async'
 require 'async/io/trap'
 require 'async/pool'
+require 'debug'
 
 module Jiggler
   class CLI
@@ -45,7 +46,7 @@ module Jiggler
     def start
       return unless ping_redis
       @cond = Async::Condition.new
-      Async do
+      Async do |task|
         setup_signal_handlers
         patch_scheduler
         @launcher = Launcher.new(config)

data/lib/jiggler/config.rb

@@ -25,11 +25,15 @@ module Jiggler
       stats_interval: 10,
       poller_enabled: true,
       poll_interval: 5,
+      # used in scheduled/requeuer
+      in_process_interval: 120,
       dead_timeout: 180 * 24 * 60 * 60, # 6 months in seconds
+      mode: :at_least_once,
       # client settings
       client_concurrency: 10,
       client_redis_pool: nil,
-      client_async: false,
+      fetchers_concurrency: 1,
+      client_async: false
     }
 
     def initialize(options = {})
@@ -80,11 +84,42 @@ module Jiggler
       @queue_scan_key ||= "#{queue_prefix}*"
     end
 
-    def prefixed_queues
-      @prefixed_queues ||= @options[:queues].map do |name| 
-        "#{QUEUE_PREFIX}#{name}" 
+    def at_least_once?
+      @options[:mode] == :at_least_once
+    end
+
+    def queues_data
+      @queues_data ||= begin
+        queues = {}
+
+        @options[:queues].each do |queue|
+          name, priority = queue
+          # by default all queues have the same priority
+          priority ||= 0
+
+          queues[name] = {
+            priority: priority,
+            # list is a redis list key for a queue
+            list: "#{QUEUE_PREFIX}#{name}",
+          }
+        end
+        
+        queues
       end
     end
+    
+    # sort in descending order (higher priority first)
+    def sorted_queues_data
+      @sorted_queues_data ||= queues_data.sort_by { |_, v| -v[:priority] }
+    end
+    
+    def sorted_lists
+      @sorted_lists ||= sorted_queues_data.map { |_, v| v[:list] }
+    end
+
+    def sorted_queues
+      @sorted_queues ||= sorted_queues_data.map { |k, _| k }
+    end
 
     def with_async_redis
       Async do
@@ -109,6 +144,15 @@ module Jiggler
           :redis_url
         )
 
+        if at_least_once?
+          # for acknowledgers
+          opts[:concurrency] *= 2
+          # for queue fetchers
+          opts[:concurrency] += @options[:fetchers_concurrency] * sorted_queues.count
+          # extra poller task to cleanup leftover in-process queues
+          opts[:concurrency] += 1
+        end
+
         opts[:concurrency] += 2 # monitor + safety margin
         opts[:concurrency] += 1 if @options[:poller_enabled]
         opts[:async] = true