fiber_job 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8a6c46d9a9278acfb132e182abe3f176e96e1b948aeaee30a722793b4b47b222
+  data.tar.gz: 70d45d9ee91164f8a5016bfc5cbae46dd4250de0a45d6658f442f4c4c78a72e2
+SHA512:
+  metadata.gz: f2a203f5d2124903dff417ca2562862a3326e496037e8d56e542629cbabbbfe1a129c9888bc31b047d522617db2afb18de22ec112d2c0ec9b37525f48046aedf
+  data.tar.gz: fc605f346003929db83d530b44024cc2f402c503041d1bb8ef890281c5762c581b943f628b289f4bcd5437a5391607c73db0a80c76391ff1de13d7cc5609220f

data/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-07-20
+
+### Added
+- Initial release of FiberJob
+- Hybrid Redis + Async::Queue architecture for optimal performance
+- Fiber-based job processing with async/await patterns
+- Job scheduling with delayed and scheduled execution
+- Cron job support with standard cron expressions
+- Built-in retry logic with exponential backoff
+- Priority queue support
+- Per-queue concurrency control with semaphores
+- Comprehensive failure tracking and monitoring
+- Production-ready logging system
+- Full YARD documentation coverage
+
+### Features
+- **Core**: Job enqueueing, processing, and lifecycle management
+- **Scheduling**: Immediate, delayed, and cron-based job execution
+- **Concurrency**: Advanced fiber pools with semaphore control
+- **Persistence**: Redis-backed job storage with atomic operations
+- **Monitoring**: Queue statistics and failed job inspection
+- **Configuration**: Flexible per-queue and global settings

data/README.md

@@ -0,0 +1,290 @@
+# FiberJob
+
+A high-performance, Redis-based background job processing library for Ruby built on modern fiber-based concurrency. FiberJob combines the persistence of Redis with the speed of async fibers to deliver exceptional performance and reliability.
+
+## Architecture Highlights
+
+FiberJob is a experimental gem that uses a architecture that sets it apart from traditional job queues:
+
+### **Hybrid Redis + Async::Queue Design**
+- **Redis for persistence**: Durable job storage with atomic operations and scheduling
+- **Async::Queue for speed**: Lightning-fast in-memory job processing with fiber-based concurrency
+- **Best of both worlds**: Reliability of Redis + performance of in-memory queues
+
+### **Advanced Fiber Management**
+- **Separation of concerns**: Independent polling fibers fetch from Redis while processing fibers execute jobs
+- **Per-queue fiber pools**: Isolated concurrency control with `Async::Semaphore` for optimal resource utilization
+- **Non-blocking operations**: All I/O operations use async/await patterns for maximum throughput
+
+### **Production-Optimized Performance**
+- **Minimal Redis contention**: Single polling fiber per queue reduces Redis load
+- **Fast job execution**: Jobs flow through in-memory `Async::Queue` for sub-millisecond processing
+- **Scalable concurrency**: Configurable fiber pools scale efficiently without thread overhead
+
+## Features
+
+- **Fiber-Based Job Processing**: Execute jobs using modern Ruby async/fiber concurrency
+- **Hybrid Queue Architecture**: Redis persistence + in-memory async queues for optimal performance
+- **Job Scheduling**: Schedule jobs for future execution with precise timing
+- **Cron Jobs**: Define recurring jobs with cron expressions
+- **Retry Logic**: Built-in exponential backoff with configurable retry policies
+- **Priority Queues**: Support for high-priority job execution
+- **Advanced Concurrency**: Per-queue fiber pools with semaphore-controlled execution
+- **Failure Tracking**: Store and inspect failed jobs for debugging
+- **Redis Integration**: Optimized Redis usage with connection pooling and atomic operations
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'fiber_job'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+Or install it yourself as:
+
+```bash
+$ gem install fiber_job
+```
+
+## Quick Start
+
+### 1. Configuration
+
+```ruby
+require 'fiber_job'
+
+FiberJob.configure do |config|
+  config.redis_url = 'redis://localhost:6379/0'
+  config.queues = [:default, :high, :low]
+  config.queue_concurrency = {
+    default: 5,
+    high: 10,
+    low: 2
+  }
+  config.log_level = :info
+end
+```
+
+### 2. Define a Job
+
+```ruby
+class EmailJob < FiberJob::Job
+  def perform(user_id, message)
+    user = User.find(user_id)
+    UserMailer.notification(user, message).deliver_now
+    FiberJob.logger.info "Email sent to user #{user_id}"
+  end
+end
+```
+
+### 3. Enqueue Jobs
+
+```ruby
+# Immediate execution
+EmailJob.perform_async(123, "Welcome to our platform!")
+
+# Delayed execution
+EmailJob.perform_in(1.hour, 456, "Don't forget to complete your profile")
+
+# Scheduled execution
+EmailJob.perform_at(Date.tomorrow.beginning_of_day, 789, "Daily newsletter")
+```
+
+### 4. Start Workers
+
+```ruby
+# Start a worker for all configured queues
+worker = FiberJob::Worker.new
+worker.start
+
+# Start a worker for specific queues
+worker = FiberJob::Worker.new(queues: [:high, :default])
+worker.start
+```
+
+## How the Hybrid Architecture Works
+
+FiberJob's unique architecture combines Redis persistence with fiber-based in-memory processing:
+
+```
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   Redis Queue   │───▶│  Polling Fiber  │───▶│ Async::Queue    │
+│   (Persistent)  │    │  (Single/Queue) │    │ (In-Memory)     │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+                                                        │
+                                                        ▼
+                                              ┌─────────────────┐
+                                              │ Processing Pool │
+                                              │ (Fiber Pool +   │
+                                              │  Semaphore)     │
+                                              └─────────────────┘
+```
+
+### Fiber Pool Architecture
+
+```ruby
+# Each queue gets its own fiber ecosystem
+Sync do |task|
+  @queues.each do |queue_name|
+    # 1. Single polling fiber per queue (minimal Redis load)
+    task.async { poll_redis_queue(queue_name) }
+
+    # 2. Multiple processing fibers per queue (parallel execution)
+    concurrency.times do
+      task.async { process_job_queue(queue_name) }
+    end
+  end
+end
+
+# Jobs flow: Redis → Polling Fiber → Async::Queue → Processing Fiber Pool
+def poll_redis_queue(queue_name)
+  while @running
+    job_data = Queue.pop(queue_name, timeout: 1.0)  # Redis operation
+    @job_queues[queue_name].enqueue(job_data) if job_data  # Fast async queue
+  end
+end
+
+def process_job_queue(queue_name)
+  while @running
+    job_data = @job_queues[queue_name].dequeue  # Instant fiber operation
+    @managers[queue_name].execute { execute_job(job_data) }  # Semaphore-controlled
+  end
+end
+```
+
+### Performance Benefits
+
+- **Sub-millisecond job pickup**: Jobs are instantly available in `Async::Queue`
+- **Reduced Redis load**: One polling fiber per queue instead of multiple workers competing
+- **Optimal concurrency**: Semaphore ensures exact concurrency limits without oversubscription
+- **Zero blocking**: All operations use async/await patterns for maximum throughput
+
+## Job Configuration
+
+### Custom Job Settings
+
+```ruby
+class ComplexJob < FiberJob::Job
+  def initialize
+    super
+    @queue = :high_priority
+    @max_retries = 5
+    @timeout = 600  # 10 minutes
+  end
+
+  def perform(data)
+    # Complex processing logic
+  end
+
+  def retry_delay(attempt)
+    # Custom retry strategy: linear backoff
+    attempt * 60  # 1min, 2min, 3min...
+  end
+end
+```
+
+### Priority Retry
+
+```ruby
+class CriticalJob < FiberJob::Job
+  def priority_retry?
+    true  # Failed jobs go to front of queue on retry
+  end
+end
+```
+
+## Cron Jobs
+
+Define recurring jobs with cron expressions:
+
+```ruby
+class DailyReportJob < FiberJob::CronJob
+  cron '0 9 * * *'  # Every day at 9 AM
+
+  def execute_cron_job
+    # Generate and send daily reports
+    Report.generate_daily
+    FiberJob.logger.info "Daily report generated"
+  end
+end
+
+class HourlyCleanupJob < FiberJob::CronJob
+  cron '0 * * * *'  # Every hour
+
+  def execute_cron_job
+    # Cleanup old data
+    TempData.cleanup_old_records
+  end
+end
+
+# Register cron jobs (automatically done on startup)
+FiberJob.register_cron_jobs
+```
+
+## Queue Management
+
+### Queue Statistics
+
+```ruby
+stats = FiberJob::Queue.stats(:default)
+puts "Queue size: #{stats[:size]}"
+puts "Scheduled jobs: #{stats[:scheduled]}"
+puts "Processing: #{stats[:processing]}"
+```
+
+### Failed Jobs
+
+```ruby
+# View failed jobs
+failed_jobs = FiberJob::Queue.failed_jobs
+failed_jobs.each do |job|
+  puts "Failed: #{job['class']} - #{job['error']}"
+end
+
+# Note: clear_failed_jobs method was removed as it was unused
+```
+
+## Configuration Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `redis_url` | `redis://localhost:6379` | Redis connection URL |
+| `concurrency` | `2` | Global default concurrency |
+| `queues` | `[:default]` | List of queues to process |
+| `queue_concurrency` | `{default: 2}` | Per-queue concurrency settings |
+| `log_level` | `:info` | Logging level (debug, info, warn, error, fatal) |
+
+## Environment Variables
+
+- `REDIS_URL`: Redis connection URL
+- `FIBER_JOB_LOG_LEVEL`: Logging level
+
+## Requirements
+
+- Ruby 3.1+
+- Redis 5.0+
+
+## License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## Roadmap
+
+- [ ] Middleware support for job lifecycle hooks
+- [ ] Dead letter queue for permanently failed jobs
+- [ ] Metrics and monitoring integration
+- [ ] ActiveJob adapter
+- [ ] Web UI for job monitoring and management
+
+## Support
+
+- Create an issue on GitHub for bug reports or feature requests
+- Check existing issues for solutions to common problems
+- Review the documentation for detailed API information

data/bin/fiber_job

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative '../lib/fiber_job'
+
+# Simple CLI for FiberJob
+command = ARGV[0]
+
+case command
+when 'worker'
+  puts "Starting FiberJob worker..."
+  FiberJob::ProcessManager.start_worker
+when 'version'
+  puts "FiberJob version #{FiberJob::VERSION}"
+when nil, 'help'
+  puts <<~HELP
+    FiberJob - High-performance fiber-based job processing
+
+    Usage:
+      fiber_job worker                Start a worker process
+      fiber_job version               Show version
+      fiber_job help                  Show this help
+
+    Examples:
+      fiber_job worker                # Start worker with default configuration
+      REDIS_URL=redis://localhost:6379/1 fiber_job worker  # Custom Redis URL
+  HELP
+else
+  puts "Unknown command: #{command}"
+  puts "Run 'fiber_job help' for usage information"
+  exit 1
+end

data/lib/fiber_job/client.rb

@@ -0,0 +1,103 @@
+# frozen_string_literal: true
+
+module FiberJob
+  # Client handles job enqueueing and scheduling operations.
+  # Provides the core interface for adding jobs to queues with immediate,
+  # delayed, or scheduled execution.
+  #
+  # This class is used internally by {FiberJob::Job} class methods and can also
+  # be used directly for advanced job management scenarios.
+  #
+  # @example Direct usage
+  #   FiberJob::Client.enqueue(MyJob, arg1, arg2)
+  #   FiberJob::Client.enqueue_in(3600, MyJob, arg1, arg2)  # 1 hour delay
+  #   FiberJob::Client.enqueue_at(Time.parse("2024-01-01"), MyJob, arg1)
+  #
+  # @author FiberJob Team
+  # @since 1.0.0
+  # @see FiberJob::Job
+  # @see FiberJob::Queue
+  class Client
+    # Enqueues a job for immediate execution.
+    # The job will be added to the appropriate queue and processed
+    # by the next available worker.
+    #
+    # @param job_class [Class] The job class to execute (must inherit from FiberJob::Job)
+    # @param args [Array] Arguments to pass to the job's perform method
+    # @return [String] Unique job identifier
+    #
+    # @example Enqueue a job immediately
+    #   FiberJob::Client.enqueue(EmailJob, user.id, "Welcome!")
+    #
+    # @raise [ArgumentError] If job_class is not a valid job class
+    def self.enqueue(job_class, *args)
+      payload = {
+        'class' => job_class.name,
+        'args' => args,
+        'enqueued_at' => Time.now.to_f
+      }
+
+      queue_name = job_class.queue
+      Queue.push(queue_name, payload)
+
+      FiberJob.logger.info "Enqueued #{job_class.name} with args: #{args.inspect}"
+    end
+
+    # Enqueues a job for execution after a specified delay.
+    # The job will be scheduled for future execution and moved to the
+    # regular queue when the delay period expires.
+    #
+    # @param delay_seconds [Numeric] Number of seconds to delay execution
+    # @param job_class [Class] The job class to execute (must inherit from FiberJob::Job)
+    # @param args [Array] Arguments to pass to the job's perform method
+    # @return [String] Unique job identifier
+    #
+    # @example Enqueue with delay
+    #   FiberJob::Client.enqueue_in(300, EmailJob, user.id, "5 minute reminder")
+    #   FiberJob::Client.enqueue_in(1.hour, ReportJob, date: Date.today)
+    #
+    # @raise [ArgumentError] If delay_seconds is negative or job_class is invalid
+    def self.enqueue_in(delay_seconds, job_class, *args)
+      scheduled_at = Time.now.to_f + delay_seconds
+      payload = {
+        'class' => job_class.name,
+        'args' => args,
+        'enqueued_at' => Time.now.to_f
+      }
+
+      queue_name = job_class.queue
+      Queue.schedule(queue_name, payload, scheduled_at)
+
+      FiberJob.logger.info "Scheduled #{job_class.name} to run in #{delay_seconds}s"
+    end
+
+    # Enqueues a job for execution at a specific time.
+    # The job will be scheduled and executed when the specified time is reached.
+    #
+    # @param timestamp [Time, Integer] Specific time or Unix timestamp for execution
+    # @param job_class [Class] The job class to execute (must inherit from FiberJob::Job)
+    # @param args [Array] Arguments to pass to the job's perform method
+    # @return [String] Unique job identifier
+    #
+    # @example Enqueue at specific time
+    #   tomorrow_9am = Time.parse("2024-01-02 09:00:00")
+    #   FiberJob::Client.enqueue_at(tomorrow_9am, DailyReportJob, date: Date.today)
+    #
+    #   # Using Unix timestamp
+    #   FiberJob::Client.enqueue_at(1672531200, MaintenanceJob, type: "cleanup")
+    #
+    # @raise [ArgumentError] If timestamp is in the past or job_class is invalid
+    def self.enqueue_at(timestamp, job_class, *args)
+      payload = {
+        'class' => job_class.name,
+        'args' => args,
+        'enqueued_at' => Time.now.to_f
+      }
+
+      queue_name = job_class.queue
+      Queue.schedule(queue_name, payload, timestamp.to_f)
+
+      FiberJob.logger.info "Scheduled #{job_class.name} to run at #{Time.at(timestamp)}"
+    end
+  end
+end

data/lib/fiber_job/concurrency.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require 'async'
+require 'async/semaphore'
+
+module FiberJob
+  # The ConcurrencyManager class is a lightweight wrapper around Async::Semaphore that
+  # controls how many jobs can execute simultaneously within each queue
+  class ConcurrencyManager
+    def initialize(max_concurrency: 5)
+      @semaphore = Async::Semaphore.new(max_concurrency)
+    end
+
+    def execute(&block)
+      @semaphore.async(&block)
+    end
+  end
+end

data/lib/fiber_job/config.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+require 'logger'
+
+module FiberJob
+  # Configuration class for FiberJob library settings.
+  # Manages Redis connection, worker concurrency, queue configuration,
+  # and logging settings. Supports both global and per-queue configuration.
+  #
+  # @example Basic configuration
+  #   FiberJob.configure do |config|
+  #     config.redis_url = 'redis://localhost:6379/0'
+  #     config.concurrency = 5
+  #     config.queues = [:default, :high, :low]
+  #     config.log_level = :debug
+  #   end
+  #
+  # @example Per-queue concurrency
+  #   FiberJob.configure do |config|
+  #     config.queue_concurrency = {
+  #       default: 5,
+  #       high: 10,
+  #       low: 2
+  #     }
+  #   end
+  #
+  class Config
+    # @!attribute [rw] redis_url
+    #   @return [String] Redis connection URL
+    # @!attribute [rw] concurrency
+    #   @return [Integer] Global default concurrency level
+    # @!attribute [rw] queues
+    #   @return [Array<Symbol>] List of queue names to process
+    # @!attribute [rw] queue_concurrency
+    #   @return [Hash] Per-queue concurrency settings
+    # @!attribute [rw] logger
+    #   @return [Logger] Logger instance for application logging
+    # @!attribute [rw] log_level
+    #   @return [Symbol] Logging level (:debug, :info, :warn, :error)
+    attr_accessor :redis_url, :concurrency, :queues, :queue_concurrency, :logger, :log_level
+
+    # Initializes configuration with sensible defaults.
+    # Values can be overridden through environment variables or configuration blocks.
+    #
+    # @return [void]
+    #
+    # Environment variables:
+    # - REDIS_URL: Redis connection URL (default: redis://localhost:6379)
+    # - FIBER_JOB_LOG_LEVEL: Logging level (default: info)
+    def initialize
+      @redis_url = ENV['REDIS_URL'] || 'redis://localhost:6379'
+      @concurrency = 2  # Global default fallback
+      @queues = [:default]
+      @queue_concurrency = { default: 2 }  # Per-queue concurrency
+      @log_level = ENV['FIBER_JOB_LOG_LEVEL']&.to_sym || :info
+      @logger = ::Logger.new($stdout)
+      @logger.level = ::Logger.const_get(@log_level.to_s.upcase)
+    end
+
+    # Returns the concurrency setting for a specific queue.
+    # Falls back to the global concurrency setting if no queue-specific
+    # setting is configured.
+    #
+    # @param queue_name [String, Symbol] Name of the queue
+    # @return [Integer] Concurrency level for the specified queue
+    #
+    # @example Get queue concurrency
+    #   config.concurrency_for_queue(:high)  # => 10
+    #   config.concurrency_for_queue(:unknown)  # => 2 (global default)
+    def concurrency_for_queue(queue_name)
+      @queue_concurrency[queue_name.to_sym] || @concurrency
+    end
+  end
+end

data/lib/fiber_job/cron.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module FiberJob
+  class Cron
+    def self.redis
+      @redis ||= Redis.new(url: FiberJob.config.redis_url)
+    end
+
+    def self.register(cron_job_class)
+      job_name = cron_job_class.name
+      cron_expression = cron_job_class.cron_expression
+
+      redis.hset('cron:jobs', job_name, JSON.dump({ 'class' => job_name,
+                                                    'cron' => cron_expression,
+                                                    'queue' => cron_job_class.new.queue,
+                                                    'registered_at' => Time.now.to_f }))
+
+      unless redis.exists?("cron:next_run:#{job_name}")
+        next_time = cron_job_class.next_run_time
+        schedule_job(cron_job_class, next_time)
+      end
+
+      FiberJob.logger.info "Registered cron job: #{job_name} (#{cron_expression})"
+    end
+
+    def self.schedule_job(cron_job_class, run_time)
+      job_name = cron_job_class.name
+
+      # Set next run time
+      redis.set("cron:next_run:#{job_name}", run_time.to_f)
+
+      # Add to sorted set for efficient scanning
+      redis.zadd('cron:schedule', run_time.to_f, job_name)
+
+      FiberJob.logger.debug "Scheduled #{job_name} for #{run_time}"
+    end
+
+    def self.due_jobs(current_time = Time.now)
+      job_names = redis.zrangebyscore('cron:schedule', 0, current_time.to_f)
+
+      job_names.map do |job_name|
+        job_data = JSON.parse(redis.hget('cron:jobs', job_name))
+        next unless job_data
+
+        redis.zrem('cron:schedule', job_name)
+
+        job_data
+      end.compact
+    end
+
+    def self.registered_jobs
+      jobs = redis.hgetall('cron:jobs')
+      jobs.transform_values { |data| JSON.parse(data) }
+    end
+
+    def self.clear_all
+      redis.del('cron:jobs', 'cron:schedule')
+      redis.keys('cron:next_run:*').each { |key| redis.del(key) }
+    end
+  end
+end

data/lib/fiber_job/cron_job.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative 'cron_parser'
+
+module FiberJob
+  class CronJob < Job
+    def self.cron(cron_expression)
+      @cron_expression = cron_expression
+    end
+
+    def self.cron_expression
+      @cron_expression || raise("No cron expression defined for #{name}")
+    end
+
+    def self.next_run_time(from_time = Time.now)
+      CronParser.next_run(@cron_expression, from_time)
+    end
+
+    def self.register
+      Cron.register(self)
+    end
+
+    # Override perform to add automatic rescheduling
+    def perform(*args)
+      execute_cron_job(*args)
+      schedule_next_run
+    end
+
+    # Subclasses implement this instead of perform
+    def execute_cron_job(*args)
+      raise NotImplementedError, 'Subclasses must implement execute_cron_job'
+    end
+
+    private
+
+    def schedule_next_run
+      next_time = self.class.next_run_time
+      Cron.schedule_job(self.class, next_time)
+    end
+  end
+end