resque-fair-share 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 620b34d14c123d949f93b7df791906a6523a33cc593a6b4aac952ef2a5023d2a
+  data.tar.gz: 47c6b05a2167fd496a99ef0f1ccf30c45a9339f866ac4e838f8d9c2ff22986f7
+SHA512:
+  metadata.gz: df5fe8d9683f859b351e9265609a4fbb86c11a3a43ceb79cce4eb6c293cda9c851a7abb8f7cc1486d2388b2c29dded84b5702cd3d5fd07c17f2cfbd9958fe7a4
+  data.tar.gz: 4d3a14190919a666aadadd7a5c0c4bb9a2189dcbf57086ac6cf1a7b812222ad3c1d2785e92ef7469addfbeee29760c1406f39c1d49b22a2fe68de5881bfcc3f0

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+# Changelog
+
+## [Unreleased]
+
+- Initial project scaffolding

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Duarte Reis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,222 @@
+# resque-fair-share
+
+**Requires Resque >= 2.0**
+
+Partition-level queue fairness for Resque.
+
+A Resque plugin that implements weighted fair-share scheduling at the worker level. Instead of FIFO processing, workers rotate through partitions (accounts, tenants, teams, etc.) proportionally so that high-volume partitions don't starve smaller ones (inspired by Weighted Fair Queuing).
+
+## Problem
+
+A single large account enqueues thousands of jobs and occupies the queue for hours while smaller accounts wait. Standard FIFO processing means whoever enqueues first gets processed first, regardless of how many jobs they already have in flight.
+
+## How it works
+
+- **On enqueue**: routes the job into a per-partition sub-queue and registers the partition in a priority ZSET (scored by in-flight count). Also increments a pending counter in Redis.
+- **On dequeue (worker-side)**: a Lua script runs atomically that:
+  1. Queries the ZSET for the lowest-scored eligible partition via `ZRANGEBYSCORE ... LIMIT 0 1` (O(log k))
+  2. LPOPs from that partition's sub-queue (O(1))
+  3. Removes the item from the main Resque queue for consistency
+- **On perform**: transitions the job from pending to in-flight, and back down on completion or failure
+- **Hard cap**: configurable `max_in_flight` (default: 50 concurrent jobs per partition across all workers). Partitions at capacity are skipped entirely during dequeue.
+- **Observability**: optional StatsD gauges of per-partition queue depth
+
+The queue always drains at full speed. No jobs are artificially delayed or moved to holding queues.
+
+Unlike a scan-window approach, the sub-queue strategy always sees all active partitions regardless of how many jobs one tenant has enqueued. A single tenant flooding the queue cannot starve others.
+
+## Why not resque-restriction?
+
+`resque-restriction` pauses excess jobs into a holding queue and releases them on a timer. Fair-share instead interleaves partitions dynamically. High-volume partitions still process, just not at the expense of everyone else.
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'resque-fair-share'
+```
+
+## Configuration
+
+```ruby
+Resque::FairShare.configure do |config|
+  config.dequeue_strategy = :sub_queues  # :sub_queues (default) or :legacy_scan
+  config.max_in_flight = 50              # hard cap per partition across all workers
+  config.partition_key = :account_id     # default key to look up in job args
+  config.statsd_client = MyApp.statsd    # optional, must respond to #gauge
+  config.scan_size = 100                 # legacy_scan only: queue items to examine
+end
+```
+
+## Usage
+
+### Basic (hash argument with a known key)
+
+The simplest case. Your job receives a hash containing the partition identifier:
+
+```ruby
+class ProcessRecordsJob
+  extend Resque::Plugins::FairShare
+
+  @queue = :default
+  fair_share_on :account_id
+
+  def self.perform(params)
+    account_id = params['account_id']
+    # ...
+  end
+end
+
+Resque.enqueue(ProcessRecordsJob, account_id: 42, record_ids: [1, 2, 3])
+```
+
+### Positional arguments (block form)
+
+When the partition value isn't in a hash, use a block to extract it:
+
+```ruby
+class SyncTenantJob
+  extend Resque::Plugins::FairShare
+
+  @queue = :sync
+  fair_share_on { |args| args[0] }
+
+  def self.perform(tenant_id, options)
+    # ...
+  end
+end
+
+Resque.enqueue(SyncTenantJob, 'tenant_123', full: true)
+```
+
+**Note**: with the default `sub_queues` strategy, the block form works fully for both enqueue and dequeue (partition routing is resolved in Ruby at enqueue time). With `legacy_scan`, the Lua script can only do hash-key lookup, so ensure the partition value is also present as a key in a hash argument.
+
+### Falling back to global config
+
+If you don't call `fair_share_on`, the plugin uses `config.partition_key` (default: `:account_id`) and searches all hash arguments for that key.
+
+```ruby
+class SimpleJob
+  extend Resque::Plugins::FairShare
+
+  @queue = :default
+
+  def self.perform(params)
+    # ...
+  end
+end
+
+Resque::FairShare.configure { |c| c.partition_key = :org_id }
+Resque.enqueue(SimpleJob, org_id: 7, data: 'hello')
+```
+
+## Redis keys
+
+The plugin maintains these keys in Redis (under the `resque:` namespace):
+
+| Key | Type | Purpose |
+|-----|------|---------|
+| `fair_share:{queue}:{value}:pending` | STRING | Jobs enqueued but not yet picked up |
+| `fair_share:{queue}:{value}:in_flight` | STRING | Jobs currently being processed |
+| `fair_share:{queue}:sub:{value}` | LIST | Per-partition sub-queue (sub_queues strategy) |
+| `fair_share:{queue}:partitions` | ZSET | Active partitions scored by in_flight count (sub_queues strategy) |
+
+## StatsD metrics
+
+When `config.statsd_client` is set, the plugin emits gauges after each counter mutation:
+
+- `resque.fair_share.{queue}.{partition_value}.pending`
+- `resque.fair_share.{queue}.{partition_value}.in_flight`
+
+Any object responding to `#gauge(name, value)` works (Datadog's `dogstatsd-ruby`, `statsd-instrument`, etc.).
+
+## Performance
+
+The sub-queue dequeue runs atomically inside Redis. Partition selection is O(log k) via a ZSET scored by in-flight count. The LPOP from the chosen sub-queue is O(1). This scales to high-cardinality partition keys (100k+ unique tenants) without degradation.
+
+### Dequeue overhead vs plain Resque
+
+| Scenario | Plain avg | Fair-share avg | Overhead |
+|----------|-----------|----------------|----------|
+| 1k jobs / 1 tenant (FIFO-equivalent) | 0.23ms | 0.25ms | ~6% |
+| 1k jobs / 10 tenants | 0.24ms | 0.25ms | ~6% |
+| 1k jobs / 100 tenants | 0.25ms | 0.26ms | ~4% |
+| 10k jobs / 10 tenants | 0.23ms | 0.26ms | ~12% |
+| 10k jobs / 100 tenants | 0.20ms | 0.25ms | ~24% |
+| 10k jobs / 1000 tenants | 0.24ms | 0.30ms | ~25% |
+| 100k jobs / 100 tenants | 0.22ms | 0.35ms | ~58% |
+| 100k jobs / 1000 tenants | 0.23ms | 0.70ms | ~199% |
+
+The overhead grows with queue depth because of the LREM on the main queue (kept for resque-web compatibility). In absolute terms, even the worst case (100k items) stays under 1ms per dequeue.
+
+### Skewed and saturated scenarios
+
+| Scenario | Plain avg | Fair-share avg | Overhead |
+|----------|-----------|----------------|----------|
+| 10k jobs / 100 tenants (90% from 1 tenant) | 0.24ms | 0.24ms | ~0% |
+| 100k jobs / 1000 tenants (99% from 1 tenant) | 0.21ms | 0.22ms | ~1% |
+| 10k jobs / 100 tenants (80% saturated) | 0.17ms | 0.28ms | ~61% |
+
+When one tenant dominates, the ZSET picks the correct (underserved) partition in O(log k) with no scan. The plain Resque LPOP picks the dominant tenant's job every time (fast but unfair).
+
+### Enqueue overhead
+
+| Scenario | Plain avg | Fair-share avg | Overhead |
+|----------|-----------|----------------|----------|
+| 500 jobs / 1 tenant | 0.22ms | 0.62ms | ~186% |
+| 500 jobs / 10 tenants | 0.20ms | 0.65ms | ~222% |
+| 500 jobs / 100 tenants | 0.25ms | 0.59ms | ~133% |
+| 1000 jobs / 500 tenants | 0.21ms | 0.62ms | ~195% |
+
+The fair-share enqueue does 3 Redis operations per job (RPUSH to main queue, RPUSH to sub-queue, ZADD NX to ZSET) plus an INCR for the pending counter. The constant ~0.4ms additional cost is independent of queue depth or partition cardinality.
+
+### Redis memory overhead
+
+| Scenario | Plain | Fair-share | Ratio |
+|----------|-------|------------|-------|
+| 1k jobs / 10 tenants | 47 KB | 96 KB | 2.0x |
+| 1k jobs / 100 tenants | 48 KB | 115 KB | 2.4x |
+| 10k jobs / 100 tenants | 481 KB | 978 KB | 2.0x |
+| 10k jobs / 1000 tenants | 490 KB | 1269 KB | 2.6x |
+
+Memory overhead is ~2x from dual-writing job payloads to both the main queue and sub-queues. The ZSET and counter keys add negligible overhead. At 10k jobs with 1000 tenants, the total fair-share footprint is ~1.2 MB.
+
+### Running benchmarks
+
+```bash
+bundle exec rspec --tag benchmark
+```
+
+## Migrating from legacy_scan
+
+If you have existing jobs in the queue that were enqueued before the sub-queue strategy was enabled, run the migration to backfill sub-queues:
+
+```ruby
+Resque::FairShare::Migration.perform('default')
+```
+
+This reads the main queue in batches, extracts partition values, and populates the sub-queues. It's safe to run while workers are active (new jobs are dual-written by the hooks). You can also roll back to the legacy strategy at any time:
+
+```ruby
+Resque::FairShare.configure { |c| c.dequeue_strategy = :legacy_scan }
+```
+
+## Development
+
+```bash
+docker compose up -d   # starts Redis on port 6380
+bundle install
+bundle exec rspec      # runs unit + integration tests
+bundle exec rubocop    # linting
+```
+
+Or point to an existing Redis:
+
+```bash
+REDIS_URL=redis://localhost:$PORT/15 bundle exec rspec
+```
+
+## License
+
+MIT

data/lib/resque/fair_share/configuration.rb

@@ -0,0 +1,32 @@
+module Resque
+  module FairShare
+    class Configuration
+      # @return [Symbol] Dequeue strategy (:sub_queues or :legacy_scan).
+      attr_accessor :dequeue_strategy
+
+      # @return [Integer] Number of queue items the Lua script examines per dequeue (legacy_scan only).
+      attr_accessor :scan_size
+
+      # @return [Integer] Hard cap on concurrent jobs per partition across all workers.
+      attr_accessor :max_in_flight
+
+      # @return [Symbol] Job argument key used to identify partitions (e.g. :account_id, :tenant_id).
+      attr_accessor :partition_key
+
+      # @return [Integer, nil] Zero-based index into the job args array for positional partition extraction.
+      attr_accessor :partition_index
+
+      # @return [#gauge, nil] Optional StatsD-compatible client for emitting per-partition depth gauges.
+      attr_accessor :statsd_client
+
+      def initialize
+        @dequeue_strategy = :sub_queues
+        @scan_size = 100
+        @max_in_flight = 50
+        @partition_key = :account_id
+        @partition_index = nil
+        @statsd_client = nil
+      end
+    end
+  end
+end

data/lib/resque/fair_share/dequeue.rb

@@ -0,0 +1,54 @@
+module Resque
+  module FairShare
+    class Dequeue
+      LEGACY_SCRIPT = File.read(File.expand_path('../../../lua/fair_dequeue.lua', __dir__))
+      SUB_QUEUE_SCRIPT = File.read(File.expand_path('../../../lua/fair_dequeue_v2.lua', __dir__))
+
+      def self.perform(queue)
+        config = Resque::FairShare.configuration
+
+        if config.dequeue_strategy == :sub_queues
+          perform_sub_queues(queue, config)
+        else
+          perform_legacy(queue, config)
+        end
+      end
+
+      def self.perform_sub_queues(queue, config)
+        namespace = Resque.redis.namespace
+        partitions_key = "#{namespace}:fair_share:#{queue}:partitions"
+        queue_key = "#{namespace}:queue:#{queue}"
+        sub_prefix = "#{namespace}:fair_share:#{queue}:sub:"
+
+        result = Resque.redis.redis.eval(
+          SUB_QUEUE_SCRIPT,
+          keys: [partitions_key, queue_key],
+          argv: [sub_prefix, config.max_in_flight]
+        )
+
+        return unless result
+
+        JSON.parse(result)
+      end
+
+      def self.perform_legacy(queue, config)
+        namespace = Resque.redis.namespace
+        queue_key = "#{namespace}:queue:#{queue}"
+        prefix = "#{namespace}:fair_share:#{queue}"
+        partition_index = config.partition_index ? (config.partition_index + 1).to_s : '0'
+
+        result = Resque.redis.redis.eval(
+          LEGACY_SCRIPT,
+          keys: [queue_key],
+          argv: [config.scan_size, config.partition_key.to_s, prefix, config.max_in_flight, partition_index]
+        )
+
+        return unless result
+
+        JSON.parse(result)
+      end
+
+      private_class_method :perform_sub_queues, :perform_legacy
+    end
+  end
+end

data/lib/resque/fair_share/hooks.rb

@@ -0,0 +1,8 @@
+require_relative '../plugins/fair_share'
+
+module Resque
+  module FairShare
+    # @deprecated Use {Resque::Plugins::FairShare} directly.
+    Hooks = Resque::Plugins::FairShare
+  end
+end

data/lib/resque/fair_share/migration.rb

@@ -0,0 +1,56 @@
+module Resque
+  module FairShare
+    # Backfills per-partition sub-queues from the existing main Resque queue.
+    # Safe to run while workers are active (new jobs are dual-written by hooks).
+    module Migration
+      BATCH_SIZE = 500
+
+      def self.perform(queue, partition_key: nil)
+        key = partition_key || Resque::FairShare.configuration.partition_key
+        namespace = Resque.redis.namespace
+        queue_key = "#{namespace}:queue:#{queue}"
+        raw_redis = Resque.redis.redis
+
+        offset = 0
+        migrated = 0
+
+        loop do
+          items = raw_redis.lrange(queue_key, offset, offset + BATCH_SIZE - 1)
+          break if items.empty?
+
+          migrated += migrate_batch(items, queue, key, namespace, raw_redis)
+          offset += BATCH_SIZE
+        end
+
+        migrated
+      end
+
+      def self.migrate_batch(items, queue, key, namespace, raw_redis)
+        count = 0
+        items.each do |item|
+          partition_value = extract_value(JSON.parse(item)['args'], key)
+          next unless partition_value
+
+          sub_key = "#{namespace}:fair_share:#{queue}:sub:#{partition_value}"
+          partitions_key = "#{namespace}:fair_share:#{queue}:partitions"
+
+          raw_redis.rpush(sub_key, item)
+          raw_redis.zadd(partitions_key, 0, partition_value.to_s, nx: true)
+          count += 1
+        end
+        count
+      end
+
+      def self.extract_value(args, key)
+        return nil unless args.is_a?(Array)
+
+        hash = args.find { |a| a.is_a?(Hash) }
+        return nil unless hash
+
+        hash[key.to_s] || hash[key.to_sym]
+      end
+
+      private_class_method :migrate_batch, :extract_value
+    end
+  end
+end

data/lib/resque/fair_share/reserve.rb

@@ -0,0 +1,28 @@
+module Resque
+  module FairShare
+    # Overrides Resque::Job.reserve to use fair-share dequeuing instead of LPOP.
+    # Prepended onto Resque::Job's singleton class on require.
+    module Reserve
+      def reserve(queue)
+        if fair_share_active?(queue)
+          payload = Resque::FairShare::Dequeue.perform(queue)
+          return new(queue, payload) if payload
+
+          nil
+        else
+          super
+        end
+      end
+
+      private
+
+      def fair_share_active?(queue)
+        namespace = Resque.redis.namespace
+        partitions_key = "#{namespace}:fair_share:#{queue}:partitions"
+        Resque.redis.redis.exists?(partitions_key)
+      end
+    end
+  end
+end
+
+Resque::Job.singleton_class.prepend(Resque::FairShare::Reserve)

data/lib/resque/fair_share/version.rb

@@ -0,0 +1,5 @@
+module Resque
+  module FairShare
+    VERSION = '0.1.0'.freeze
+  end
+end

data/lib/resque/fair_share.rb

@@ -0,0 +1,26 @@
+require 'resque'
+require_relative 'fair_share/version'
+require_relative 'fair_share/configuration'
+require_relative 'plugins/fair_share'
+require_relative 'fair_share/hooks'
+require_relative 'fair_share/dequeue'
+require_relative 'fair_share/reserve'
+require_relative 'fair_share/migration'
+
+module Resque
+  module FairShare
+    class << self
+      def configuration
+        @configuration ||= Configuration.new
+      end
+
+      def configure
+        yield(configuration)
+      end
+
+      def reset_configuration!
+        @configuration = Configuration.new
+      end
+    end
+  end
+end

data/lib/resque/plugins/fair_share.rb

@@ -0,0 +1,141 @@
+module Resque
+  module Plugins
+    # Resque job hooks for fair-share scheduling.
+    # Extend this module in your job class to track per-partition pending and
+    # in-flight counters in Redis.
+    #
+    #   class MyJob
+    #     extend Resque::Plugins::FairShare
+    #     @queue = :default
+    #     fair_share_on :account_id
+    #     def self.perform(params); end
+    #   end
+    module FairShare
+      ENQUEUE_SCRIPT = File.read(File.expand_path('../../../lua/fair_enqueue.lua', __dir__))
+
+      def fair_share_on(key = nil, &block)
+        if block
+          @_fair_share_resolver = block
+        elsif key
+          @_fair_share_key = key
+        end
+      end
+
+      def after_enqueue_fair_share(*args)
+        partition_value = extract_partition_value(args)
+
+        redis.incr(redis_key(partition_value, 'pending'))
+        emit_gauge(partition_value, 'pending')
+
+        enqueue_to_sub_queue(partition_value, args)
+      end
+
+      def before_perform_fair_share(*args)
+        partition_value = extract_partition_value(args)
+
+        in_flight_key = redis_key(partition_value, 'in_flight')
+        in_flight = redis.get(in_flight_key).to_i
+        unless in_flight < Resque::FairShare.configuration.max_in_flight
+          re_enqueue(partition_value, args)
+
+          raise Resque::Job::DontPerform,
+                "Max in-flight limit reached for partition #{partition_value}. " \
+                "Current: #{in_flight}, max: #{Resque::FairShare.configuration.max_in_flight}"
+        end
+
+        redis.incr(in_flight_key)
+        update_partition_score(partition_value)
+        emit_gauge(partition_value, 'in_flight')
+        redis.decr(redis_key(partition_value, 'pending'))
+        emit_gauge(partition_value, 'pending')
+      end
+
+      def after_perform_fair_share(*args)
+        partition_value = extract_partition_value(args)
+
+        redis.decr(redis_key(partition_value, 'in_flight'))
+        update_partition_score(partition_value)
+        emit_gauge(partition_value, 'in_flight')
+      end
+
+      def on_failure_fair_share(_exception, *args)
+        partition_value = extract_partition_value(args)
+
+        redis.decr(redis_key(partition_value, 'in_flight'))
+        update_partition_score(partition_value)
+        emit_gauge(partition_value, 'in_flight')
+      end
+
+      private
+
+      def fair_share_partition_key
+        @_fair_share_key || Resque::FairShare.configuration.partition_key
+      end
+
+      def redis_key(partition_value, counter_type)
+        "fair_share:#{@queue}:#{partition_value}:#{counter_type}"
+      end
+
+      def redis
+        Resque.redis
+      end
+
+      def extract_partition_value(args)
+        if @_fair_share_resolver
+          value = @_fair_share_resolver.call(args)
+          raise ArgumentError, 'fair_share_on block returned nil' unless value
+
+          return value
+        end
+
+        key = fair_share_partition_key
+        hash = args.find { |a| a.is_a?(Hash) }
+        raise ArgumentError, 'No hash argument found in job arguments' unless hash
+
+        value = hash[key.to_s] || hash[key]
+        raise ArgumentError, "Partition key #{key} not found in job arguments" unless value
+
+        value
+      end
+
+      def emit_gauge(partition_value, counter_type)
+        client = Resque::FairShare.configuration.statsd_client
+        return unless client
+
+        value = redis.get(redis_key(partition_value, counter_type)).to_i
+        client.gauge("resque.fair_share.#{@queue}.#{partition_value}.#{counter_type}", value)
+      end
+
+      def enqueue_to_sub_queue(partition_value, args)
+        return unless Resque::FairShare.configuration.dequeue_strategy == :sub_queues
+
+        namespace = Resque.redis.namespace
+        sub_key = "#{namespace}:fair_share:#{@queue}:sub:#{partition_value}"
+        partitions_key = "#{namespace}:fair_share:#{@queue}:partitions"
+        job_json = Resque.encode(class: to_s, args: args)
+
+        Resque.redis.redis.eval(
+          ENQUEUE_SCRIPT,
+          keys: [sub_key, partitions_key],
+          argv: [job_json, partition_value.to_s]
+        )
+      end
+
+      def update_partition_score(partition_value)
+        return unless Resque::FairShare.configuration.dequeue_strategy == :sub_queues
+
+        namespace = Resque.redis.namespace
+        partitions_key = "#{namespace}:fair_share:#{@queue}:partitions"
+        in_flight = redis.get(redis_key(partition_value, 'in_flight')).to_i
+
+        # Update ZSET score to reflect current in_flight count (only if member exists)
+        Resque.redis.redis.zadd(partitions_key, in_flight, partition_value.to_s, xx: true)
+      end
+
+      def re_enqueue(partition_value, args)
+        Resque.push(@queue.to_s, class: to_s, args: args)
+        enqueue_to_sub_queue(partition_value, args)
+      end
+    end
+  end
+end

data/lib/resque-fair-share.rb

@@ -0,0 +1 @@
+require 'resque/fair_share'

data/lua/fair_dequeue.lua

@@ -0,0 +1,182 @@
+-- ============================================================================
+-- fair_dequeue.lua
+-- ============================================================================
+--
+-- PURPOSE:
+-- Instead of Resque's default behavior (LPOP = always take the oldest job),
+-- this script picks the "fairest" job from the front of the queue.
+--
+-- WHAT "FAIR" MEANS HERE:
+-- We want to give every partition (tenant/account) a roughly equal share of
+-- worker time. If tenant A has 10 jobs running and tenant B has 0 jobs running,
+-- we should pick tenant B's job next, even if tenant A's jobs were enqueued first.
+--
+-- THE ALGORITHM:
+-- 1. Look at the first N jobs in the queue (the "scan window")
+-- 2. For each job, check how many jobs that partition already has running (in_flight)
+-- 3. Pick the job whose partition has the LOWEST in_flight count
+-- 4. Remove that job from the queue and return it
+--
+-- This means:
+-- - Partitions with fewer running jobs get priority (they're underserved)
+-- - Partitions at their max_in_flight limit are skipped entirely
+-- - On a tie, the job closest to the front of the queue wins (natural FIFO)
+-- - If all partitions in the scan window are at max, return nil (nothing to do)
+--
+-- WHY A LUA SCRIPT:
+-- This needs to be atomic. Between reading the queue and removing the chosen job,
+-- no other worker should be able to grab the same job. Redis executes Lua scripts
+-- as a single atomic operation, so no race conditions.
+--
+-- ============================================================================
+-- INPUTS
+-- ============================================================================
+--
+-- KEYS[1] = The Redis key for the queue list
+--           e.g. "resque:queue:default"
+--           This is a Redis LIST where each element is a JSON-encoded job payload
+--           like: {"class":"ProcessRecords","args":[{"tenant_id":42,"data":"..."}]}
+--
+-- ARGV[1] = scan_size (number, as string)
+--           How many items to examine from the front of the queue.
+--           Larger = fairer decisions but more CPU per dequeue.
+--           Smaller = faster but might miss a better candidate further back.
+--           Typical value: 100
+--
+-- ARGV[2] = partition_key (string)
+--           The JSON field name to group jobs by, e.g. "tenant_id"
+--           We look for this key inside the first hash argument of the job.
+--
+-- ARGV[3] = key_prefix (string)
+--           The prefix for Redis counter keys, e.g. "resque:fair_share:default"
+--           We append ":{partition_value}:in_flight" to look up running counts.
+--
+-- ARGV[4] = max_in_flight (number, as string)
+--           The hard cap on concurrent jobs per partition.
+--           0 means no limit (every partition is always eligible).
+--
+-- ARGV[5] = partition_index (number, as string)
+--           1-based index into the job args array for positional extraction.
+--           0 means disabled (only hash-key lookup is used).
+--           When set, if no hash arg contains partition_key, the script uses
+--           args[partition_index] as the partition value directly.
+--
+-- ============================================================================
+-- OUTPUT
+-- ============================================================================
+--
+-- Returns the raw JSON string of the selected job (to be parsed by Ruby),
+-- or nil if no eligible job was found.
+--
+-- ============================================================================
+
+-- Parse inputs. ARGV values are always strings in Redis Lua.
+local scan_size = tonumber(ARGV[1])
+local partition_key = ARGV[2]
+local prefix = ARGV[3]
+local max_in_flight = tonumber(ARGV[4])
+local partition_index = tonumber(ARGV[5])
+
+-- ============================================================================
+-- STEP 1: Read the scan window
+-- ============================================================================
+-- LRANGE returns elements from index 0 to scan_size-1 (inclusive on both ends).
+-- If the list has fewer elements, it just returns what's available.
+-- This does NOT remove anything from the list; it's a read-only peek.
+local items = redis.call("LRANGE", KEYS[1], 0, scan_size - 1)
+
+-- Nothing in the queue, nothing to do.
+if #items == 0 then
+    return nil
+end
+
+-- ============================================================================
+-- STEP 2: Score each job and find the best candidate
+-- ============================================================================
+-- We scan through the items and track the "best" job seen so far.
+-- "Best" = belongs to the partition with the lowest in_flight count.
+
+local best_item = nil   -- The raw JSON string we'll return and remove
+local best_score = nil  -- The in_flight count of the best partition so far
+
+for i, item in ipairs(items) do
+    -- Each `item` is a JSON string like:
+    -- {"class":"ProcessRecords","args":[{"tenant_id":42,"data":"hello"}]}
+
+    -- Decode it into a Lua table so we can inspect the args
+    local payload = cjson.decode(item)
+
+    -- Resque stores job arguments as an array under the "args" key.
+    -- payload["args"] is a Lua table (1-indexed array).
+    local job_args = payload["args"]
+
+    -- Extract the partition value. First try hash-key lookup, then positional.
+    local partition_value = nil
+    for _, arg in ipairs(job_args) do
+        if type(arg) == "table" and arg[partition_key] ~= nil then
+            partition_value = arg[partition_key]
+            break
+        end
+    end
+
+    if partition_value == nil and partition_index > 0 then
+        partition_value = job_args[partition_index]
+    end
+
+    -- If this job doesn't have a partition value, skip it.
+    if partition_value ~= nil then
+
+        -- ================================================================
+        -- STEP 2a: Look up how many jobs this partition has running
+        -- ================================================================
+        -- Build the key: e.g. "resque:fair_share:default:42:in_flight"
+        local in_flight_key = prefix .. ":" .. tostring(partition_value) .. ":in_flight"
+
+        -- GET returns a string or nil. Convert to number, default to 0.
+        -- A partition with no key means it has nothing running yet.
+        local in_flight = tonumber(redis.call("GET", in_flight_key)) or 0
+
+        -- ================================================================
+        -- STEP 2b: Check if this partition is eligible
+        -- ================================================================
+        -- Skip partitions that have hit their max. They're "full" and
+        -- should not take on more work right now.
+        if max_in_flight == 0 or in_flight < max_in_flight then
+
+            -- ============================================================
+            -- STEP 2c: Is this better than our current best?
+            -- ============================================================
+            -- Lower in_flight = more underserved = should get priority.
+            -- On equal scores, the first one we saw wins (closer to queue front).
+            if best_score == nil or in_flight < best_score then
+                best_score = in_flight
+                best_item = item
+            end
+
+            -- Optimization: 0 in-flight is the minimum possible score.
+            -- We can't find anything better, so stop scanning early.
+            if best_score == 0 then
+                break
+            end
+        end
+    end
+end
+
+-- ============================================================================
+-- STEP 3: Remove the winner from the queue and return it
+-- ============================================================================
+if best_item ~= nil then
+    -- LREM removes elements from a list by value.
+    -- Arguments: key, count, value
+    --   count=1 means "remove the first occurrence scanning from head to tail"
+    -- We use the exact JSON string from LRANGE, so it matches perfectly.
+    -- This is atomic with the rest of the script; no other worker can grab it.
+    redis.call("LREM", KEYS[1], 1, best_item)
+    return best_item
+end
+
+-- No eligible job found. Either:
+-- - Every partition in the scan window is at max_in_flight
+-- - Every job in the scan window is malformed (no partition key)
+-- The worker will get nil and try again on the next poll cycle.
+return nil

data/lua/fair_dequeue_v2.lua

@@ -0,0 +1,50 @@
+-- fair_dequeue_v2.lua
+--
+-- Per-partition sub-queue dequeue with O(log k) partition selection via ZSET.
+-- The partitions ZSET is scored by in_flight count, so the member with the
+-- lowest score is always the most underserved partition.
+--
+-- KEYS[1] = partitions ZSET (e.g. "resque:fair_share:default:partitions")
+-- KEYS[2] = main queue LIST (e.g. "resque:queue:default")
+--
+-- ARGV[1] = sub-queue key prefix (e.g. "resque:fair_share:default:sub:")
+-- ARGV[2] = max_in_flight (number as string, 0 = no limit)
+
+local sub_prefix = ARGV[1]
+local max_in_flight = tonumber(ARGV[2])
+
+-- Step 1: Find eligible partitions starting from the lowest score.
+-- ZRANGEBYSCORE returns members scored between min and max, ordered by score.
+-- We ask for a small batch and check eligibility.
+local max_score = "inf"
+if max_in_flight > 0 then
+    max_score = tostring(max_in_flight - 1)
+end
+
+local candidates = redis.call("ZRANGEBYSCORE", KEYS[1], 0, max_score, "LIMIT", 0, 1)
+
+if #candidates == 0 then
+    return nil
+end
+
+local best_partition = candidates[1]
+
+-- Step 2: LPOP from the best partition's sub-queue
+local sub_key = sub_prefix .. best_partition
+local item = redis.call("LPOP", sub_key)
+
+if item == nil then
+    -- Stale ZSET entry (sub-queue already drained)
+    redis.call("ZREM", KEYS[1], best_partition)
+    return nil
+end
+
+-- Step 3: Remove from main queue for consistency
+redis.call("LREM", KEYS[2], 1, item)
+
+-- Step 4: Clean up ZSET if sub-queue is now empty
+if redis.call("LLEN", sub_key) == 0 then
+    redis.call("ZREM", KEYS[1], best_partition)
+end
+
+return item

data/lua/fair_enqueue.lua

@@ -0,0 +1,17 @@
+-- fair_enqueue.lua
+--
+-- Atomically routes a job into its partition sub-queue and registers the
+-- partition in the priority ZSET (scored by in_flight count).
+--
+-- KEYS[1] = sub-queue key (e.g. "resque:fair_share:default:sub:42")
+-- KEYS[2] = partitions ZSET (e.g. "resque:fair_share:default:partitions")
+--
+-- ARGV[1] = job JSON payload
+-- ARGV[2] = partition value (string)
+
+redis.call("RPUSH", KEYS[1], ARGV[1])
+
+-- NX: only add if not already a member (preserves current score for active partitions)
+redis.call("ZADD", KEYS[2], "NX", 0, ARGV[2])
+
+return 1

metadata

@@ -0,0 +1,92 @@
+--- !ruby/object:Gem::Specification
+name: resque-fair-share
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Duarte Reis
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2026-07-21 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+description: A Resque plugin that implements weighted fair-share scheduling at the
+  worker level. Instead of FIFO processing, workers rotate through accounts proportionally
+  so that high-volume accounts don't starve smaller ones.
+email:
+- duarte.reis@zendesk.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/resque-fair-share.rb
+- lib/resque/fair_share.rb
+- lib/resque/fair_share/configuration.rb
+- lib/resque/fair_share/dequeue.rb
+- lib/resque/fair_share/hooks.rb
+- lib/resque/fair_share/migration.rb
+- lib/resque/fair_share/reserve.rb
+- lib/resque/fair_share/version.rb
+- lib/resque/plugins/fair_share.rb
+- lua/fair_dequeue.lua
+- lua/fair_dequeue_v2.lua
+- lua/fair_enqueue.lua
+homepage: https://github.com/duarteareiasdosreis/resque-fair-share
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/duarteareiasdosreis/resque-fair-share
+  source_code_uri: https://github.com/duarteareiasdosreis/resque-fair-share
+  changelog_uri: https://github.com/duarteareiasdosreis/resque-fair-share/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.1.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: Account-level queue fairness for Resque
+test_files: []