sidekiq-queue-throttled 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f04395444525e11488352d423c291bbefa3d15cd0415cea21a6a248fe9ec11f1
+  data.tar.gz: 97e5ee1c7501c66e598fa6638a30a2cb1ba39801dbef610456aa590e2300685c
+SHA512:
+  metadata.gz: c8e536136b78764fe9fa29ac5666839fdac9c81f53d6edf23a0725abb9459966f6a5e0b35c3b47099d665a569e28fc222a6bbc5e49f4b4597c8bce0cd7d51db8
+  data.tar.gz: 8dfae89423e246c267efa76eda12237e10885917cb519aea643f6e2868d100dd1db518a6f9d5fe5b27bcb01f41c9bb9b4ca80277ae289a97426a4e6e6556ad28

data/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+- Initial release
+- Queue-level concurrency limits
+- Job-level throttling with concurrency and rate limits
+- Redis-based coordination
+- Thread-safe implementation
+- Comprehensive configuration options
+- Middleware integration with Sidekiq
+- DSL for job throttling configuration
+- Error handling and logging
+- Production-ready features
+
+### Features
+- Support for queue limits in sidekiq.yml configuration
+- Programmatic queue limit configuration
+- Concurrency-based job throttling with custom key suffixes
+- Rate-based job throttling with time windows
+- Automatic job rescheduling when limits are reached
+- Redis key management with TTL
+- Concurrent access safety with read-write locks
+- Comprehensive validation of configuration options
+
+## [1.0.0] - 2024-01-01
+
+### Added
+- Initial release of sidekiq-queue-throttled gem
+- Queue-level concurrency limiting
+- Job-level throttling capabilities
+- Redis-based coordination system
+- Thread-safe implementation
+- Comprehensive configuration system
+- Middleware integration
+- DSL for job configuration
+- Error handling and logging
+- Production-ready features
+
+### Technical Details
+- Uses Concurrent::ReentrantReadWriteLock for thread safety
+- Redis-based counters with TTL for memory management
+- Automatic job rescheduling with configurable delays
+- Support for both concurrency and rate limiting
+- Flexible key suffix resolution (Proc, Symbol, String)
+- Comprehensive validation of configuration options
+- Integration with Sidekiq's middleware chain 

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Sidekiq Queue Throttled
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

data/README.md

@@ -0,0 +1,303 @@
+# Sidekiq Queue Throttled
+
+A production-ready Sidekiq gem that combines queue-level concurrency limits with job-level throttling capabilities. This gem provides the best of both `sidekiq-limit_fetch` and `sidekiq-throttled` in a single, well-tested package.
+
+## Features
+
+- **Queue-level limits**: Set maximum concurrent jobs per queue
+- **Job-level throttling**: Limit jobs by concurrency or rate
+- **Redis-based**: Scalable across multiple Sidekiq processes
+- **Production-ready**: Comprehensive error handling and logging
+- **Thread-safe**: Uses concurrent primitives for safety
+- **Configurable**: Flexible configuration options
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sidekiq-queue-throttled'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+## Configuration
+
+### Queue Limits
+
+Configure queue limits in your `sidekiq.yml`:
+
+```yaml
+:concurrency: 25
+:queues:
+  - [default, 1]
+  - [high, 2]
+  - [low, 1]
+
+:limits:
+  default: 100
+  high: 50
+  low: 200
+```
+
+Or configure programmatically:
+
+```ruby
+Sidekiq::QueueThrottled.configure do |config|
+  config.set_queue_limit(:default, 100)
+  config.set_queue_limit(:high, 50)
+  config.set_queue_limit(:low, 200)
+end
+```
+
+### Job Throttling
+
+Include the `Sidekiq::QueueThrottled::Job` module in your job classes:
+
+```ruby
+class MyJob
+  include Sidekiq::Job
+  include Sidekiq::QueueThrottled::Job
+
+  sidekiq_options queue: :my_queue
+
+  # Concurrency-based throttling
+  sidekiq_throttle(
+    concurrency: { 
+      limit: 10, 
+      key_suffix: -> (user_id) { user_id } 
+    }
+  )
+
+  def perform(user_id)
+    # Your job logic here
+  end
+end
+```
+
+## Usage Examples
+
+### Basic Queue Limiting
+
+```ruby
+# sidekiq.yml
+:limits:
+  email_queue: 10
+  processing_queue: 5
+
+# This ensures email_queue never has more than 10 concurrent jobs
+# and processing_queue never has more than 5 concurrent jobs
+```
+
+### Concurrency-based Job Throttling
+
+```ruby
+class UserNotificationJob
+  include Sidekiq::Job
+  include Sidekiq::QueueThrottled::Job
+
+  sidekiq_options queue: :notifications
+
+  # Allow maximum 3 concurrent jobs per user
+  sidekiq_throttle(
+    concurrency: { 
+      limit: 3, 
+      key_suffix: -> (user_id) { user_id } 
+    }
+  )
+
+  def perform(user_id, message)
+    # Send notification to user
+  end
+end
+```
+
+### Rate-based Job Throttling
+
+```ruby
+class APICallJob
+  include Sidekiq::Job
+  include Sidekiq::QueueThrottled::Job
+
+  sidekiq_options queue: :api_calls
+
+  # Allow maximum 100 jobs per hour per API key
+  sidekiq_throttle(
+    rate: { 
+      limit: 100, 
+      period: 3600, # 1 hour in seconds
+      key_suffix: -> (api_key) { api_key } 
+    }
+  )
+
+  def perform(api_key, endpoint)
+    # Make API call
+  end
+end
+```
+
+### Complex Throttling with Multiple Parameters
+
+```ruby
+class DataProcessingJob
+  include Sidekiq::Job
+  include Sidekiq::QueueThrottled::Job
+
+  sidekiq_options queue: :processing
+
+  # Allow maximum 5 concurrent jobs per organization per data_type
+  sidekiq_throttle(
+    concurrency: { 
+      limit: 5, 
+      key_suffix: -> (org_id, data_type) { "#{org_id}:#{data_type}" } 
+    }
+  )
+
+  def perform(org_id, data_type, data)
+    # Process data
+  end
+end
+```
+
+## Configuration Options
+
+### Global Configuration
+
+```ruby
+Sidekiq::QueueThrottled.configure do |config|
+  # Redis key prefix for all keys
+  config.redis_key_prefix = "sidekiq:queue_throttled"
+  
+  # TTL for throttle counters (in seconds)
+  config.throttle_ttl = 3600 # 1 hour
+  
+  # TTL for queue locks (in seconds)
+  config.lock_ttl = 300 # 5 minutes
+  
+  # Delay before rescheduling blocked jobs (in seconds)
+  config.retry_delay = 5
+end
+```
+
+### Custom Logger
+
+```ruby
+Sidekiq::QueueThrottled.logger = Rails.logger
+```
+
+### Custom Redis Connection
+
+```ruby
+Sidekiq::QueueThrottled.redis = Redis.new(url: ENV['REDIS_URL'])
+```
+
+## API Reference
+
+### Queue Limiter
+
+```ruby
+limiter = Sidekiq::QueueThrottled::QueueLimiter.new(queue_name, limit)
+
+# Acquire a lock (returns lock_id or false)
+lock_id = limiter.acquire_lock
+
+# Release a lock
+limiter.release_lock(lock_id)
+
+# Get current count
+current_count = limiter.current_count
+
+# Get available slots
+available = limiter.available_slots
+
+# Reset counters
+limiter.reset!
+```
+
+### Job Throttler
+
+```ruby
+throttler = Sidekiq::QueueThrottled::JobThrottler.new(job_class, throttle_config)
+
+# Check if job can be processed
+can_process = throttler.can_process?(args)
+
+# Acquire a slot
+acquired = throttler.acquire_slot(args)
+
+# Release a slot
+throttler.release_slot(args)
+```
+
+## Monitoring and Debugging
+
+### Redis Keys
+
+The gem uses the following Redis key patterns:
+
+- Queue counters: `sidekiq:queue_throttled:queue:{queue_name}:counter`
+- Queue locks: `sidekiq:queue_throttled:queue:{queue_name}:lock`
+- Concurrency counters: `sidekiq:queue_throttled:concurrency:{job_class}:{key_suffix}`
+- Rate counters: `sidekiq:queue_throttled:rate:{job_class}:{key_suffix}:{window}`
+
+### Logging
+
+The gem logs important events:
+
+```
+INFO: Queue limit reached for email_queue, rescheduling job
+INFO: Job throttling limit reached for UserNotificationJob, rescheduling job
+ERROR: Failed to release lock for queue email_queue: Redis connection error
+```
+
+## Testing
+
+```ruby
+# In your test setup
+require 'sidekiq/queue_throttled'
+
+RSpec.configure do |config|
+  config.before(:each) do
+    # Reset all counters
+    Sidekiq::QueueThrottled.redis.flushdb
+  end
+end
+
+# Test queue limits
+RSpec.describe "Queue Limiting" do
+  it "respects queue limits" do
+    limiter = Sidekiq::QueueThrottled::QueueLimiter.new("test_queue", 2)
+    
+    expect(limiter.acquire_lock).to be_truthy
+    expect(limiter.acquire_lock).to be_truthy
+    expect(limiter.acquire_lock).to be_falsey # Limit reached
+  end
+end
+```
+
+## Performance Considerations
+
+- **Redis Operations**: The gem uses Redis for coordination, so ensure your Redis instance can handle the load
+- **Memory Usage**: Counters are stored in Redis with TTL, so memory usage is bounded
+- **Network Latency**: Consider Redis network latency when setting TTL values
+- **Concurrent Access**: The gem uses thread-safe primitives for concurrent access
+
+## Contributing
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -am 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Create a new Pull Request
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md) for a list of changes and version history. 

data/lib/sidekiq/queue_throttled/configuration.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module QueueThrottled
+    class Configuration
+      attr_accessor :queue_limits, :redis_key_prefix, :throttle_ttl, :lock_ttl, :retry_delay
+
+      def initialize
+        @queue_limits = {}
+        @redis_key_prefix = 'sidekiq:queue_throttled'
+        @throttle_ttl = 3600 # 1 hour
+        @lock_ttl = 300 # 5 minutes
+        @retry_delay = 5 # 5 seconds
+      end
+
+      def queue_limit(queue_name)
+        @queue_limits[queue_name.to_s] || @queue_limits[queue_name.to_sym]
+      end
+
+      def set_queue_limit(queue_name, limit)
+        @queue_limits[queue_name.to_s] = limit.to_i
+      end
+
+      def load_from_sidekiq_config!(sidekiq_config = nil)
+        limits = sidekiq_config&.dig(:limits) || sidekiq_config&.dig('limits')
+        return unless limits
+
+        limits.each do |queue_name, limit|
+          set_queue_limit(queue_name, limit)
+        end
+      end
+
+      def load_from_yaml!(yaml_content)
+        require 'yaml'
+        config = YAML.safe_load(yaml_content)
+        limits = config['limits'] || config[:limits]
+        return unless limits
+
+        limits.each do |queue_name, limit|
+          set_queue_limit(queue_name, limit)
+        end
+      end
+
+      def validate!
+        @queue_limits.each do |queue_name, limit|
+          unless limit.is_a?(Integer) && limit.positive?
+            raise ArgumentError, "Queue limit for '#{queue_name}' must be a positive integer, got: #{limit}"
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/queue_throttled/job.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module QueueThrottled
+    module Job
+      def self.included(base)
+        base.extend(ClassMethods)
+        base.class_eval do
+          # Simple class attribute implementation
+          class << self
+            attr_accessor :sidekiq_throttle_config
+          end
+
+          def self.sidekiq_throttle_config
+            @sidekiq_throttle_config
+          end
+
+          def self.sidekiq_throttle_config=(value)
+            @sidekiq_throttle_config = value
+          end
+        end
+      end
+
+      module ClassMethods
+        def sidekiq_throttle(options = {})
+          validate_throttle_options!(options)
+          self.sidekiq_throttle_config = options
+        end
+
+        private
+
+        def validate_throttle_options!(options)
+          return if options.empty?
+
+          if options[:concurrency] && options[:rate]
+            raise ArgumentError, 'Cannot specify both concurrency and rate limits'
+          end
+
+          if options[:concurrency]
+            validate_concurrency_options!(options[:concurrency])
+          end
+
+          if options[:rate]
+            validate_rate_options!(options[:rate])
+          end
+        end
+
+        def validate_concurrency_options!(concurrency)
+          unless concurrency.is_a?(Hash)
+            raise ArgumentError, 'Concurrency must be a hash'
+          end
+
+          unless concurrency[:limit].is_a?(Integer) && concurrency[:limit].positive?
+            raise ArgumentError, 'Concurrency limit must be a positive integer'
+          end
+
+          unless concurrency[:key_suffix]
+            raise ArgumentError, 'Concurrency key_suffix is required'
+          end
+        end
+
+        def validate_rate_options!(rate)
+          unless rate.is_a?(Hash)
+            raise ArgumentError, 'Rate must be a hash'
+          end
+
+          unless rate[:limit].is_a?(Integer) && rate[:limit].positive?
+            raise ArgumentError, 'Rate limit must be a positive integer'
+          end
+
+          unless rate[:period].nil? || (rate[:period].is_a?(Integer) && rate[:period].positive?)
+            raise ArgumentError, 'Rate period must be a positive integer'
+          end
+
+          unless rate[:key_suffix]
+            raise ArgumentError, 'Rate key_suffix is required'
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/queue_throttled/job_throttler.rb

@@ -0,0 +1,155 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module QueueThrottled
+    class JobThrottler
+      attr_reader :job_class, :throttle_config, :redis
+
+      def initialize(job_class, throttle_config, redis = nil)
+        @job_class = job_class
+        @throttle_config = throttle_config
+        @redis = redis || Sidekiq::QueueThrottled.redis
+        @mutex = Concurrent::ReentrantReadWriteLock.new
+      end
+
+      def can_process?(args)
+        return true unless @throttle_config
+
+        @mutex.with_read_lock do
+          check_concurrency_limit(args) && check_rate_limit(args)
+        end
+      end
+
+      def acquire_slot(args)
+        return true unless @throttle_config
+
+        @mutex.with_write_lock do
+          return false unless can_process?(args)
+
+          if @throttle_config[:concurrency]
+            acquire_concurrency_slot(args)
+          elsif @throttle_config[:rate]
+            acquire_rate_slot(args)
+          else
+            true
+          end
+        end
+      end
+
+      def release_slot(args)
+        return true unless @throttle_config
+
+        @mutex.with_write_lock do
+          if @throttle_config[:concurrency]
+            release_concurrency_slot(args)
+          end
+          # Rate limiting slots are not released - they expire naturally
+        end
+
+        true
+      rescue StandardError => e
+        Sidekiq::QueueThrottled.logger.error "Failed to release slot for job #{@job_class}: #{e.message}"
+        false
+      end
+
+      private
+
+      def check_concurrency_limit(args)
+        return true unless @throttle_config[:concurrency]
+
+        config = @throttle_config[:concurrency]
+        limit = config[:limit]
+        key_suffix = resolve_key_suffix(config[:key_suffix], args)
+        current_count = get_concurrency_count(key_suffix)
+
+        current_count < limit
+      end
+
+      def check_rate_limit(args)
+        return true unless @throttle_config[:rate]
+
+        config = @throttle_config[:rate]
+        limit = config[:limit]
+        period = config[:period] || 60
+        key_suffix = resolve_key_suffix(config[:key_suffix], args)
+
+        current_count = get_rate_count(key_suffix, period)
+        current_count < limit
+      end
+
+      def acquire_concurrency_slot(args)
+        config = @throttle_config[:concurrency]
+        key_suffix = resolve_key_suffix(config[:key_suffix], args)
+        key = concurrency_key(key_suffix)
+
+        @redis.multi do |multi|
+          multi.incr(key)
+          multi.expire(key, Sidekiq::QueueThrottled.configuration.throttle_ttl)
+        end
+
+        true
+      end
+
+      def release_concurrency_slot(args)
+        config = @throttle_config[:concurrency]
+        key_suffix = resolve_key_suffix(config[:key_suffix], args)
+        key = concurrency_key(key_suffix)
+
+        @redis.multi do |multi|
+          multi.decr(key)
+          multi.expire(key, Sidekiq::QueueThrottled.configuration.throttle_ttl)
+        end
+
+        true
+      end
+
+      def acquire_rate_slot(args)
+        config = @throttle_config[:rate]
+        period = config[:period] || 60
+        key_suffix = resolve_key_suffix(config[:key_suffix], args)
+        key = rate_key(key_suffix, period)
+
+        @redis.multi do |multi|
+          multi.incr(key)
+          multi.expire(key, period)
+        end
+
+        true
+      end
+
+      def get_concurrency_count(key_suffix)
+        key = concurrency_key(key_suffix)
+        count = @redis.get(key)
+        count ? count.to_i : 0
+      end
+
+      def get_rate_count(key_suffix, period)
+        key = rate_key(key_suffix, period)
+        count = @redis.get(key)
+        count ? count.to_i : 0
+      end
+
+      def concurrency_key(key_suffix)
+        "#{Sidekiq::QueueThrottled.configuration.redis_key_prefix}:concurrency:#{@job_class}:#{key_suffix}"
+      end
+
+      def rate_key(key_suffix, period)
+        window = Time.now.to_i / period
+        "#{Sidekiq::QueueThrottled.configuration.redis_key_prefix}:rate:#{@job_class}:#{key_suffix}:#{window}"
+      end
+
+      def resolve_key_suffix(key_suffix, args)
+        case key_suffix
+        when Proc
+          key_suffix.call(*args)
+        when Symbol
+          args.first.send(key_suffix) if args.first.respond_to?(key_suffix)
+        when String
+          key_suffix
+        else
+          'default'
+        end.to_s
+      end
+    end
+  end
+end