queekiq 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 4310b94018510b1e6353861e3d6b282c2141f214b56fba92628f61291ac38f6b
+  data.tar.gz: 52b6ddc84908e73f43e1253f20292ca59284b622503040096ff35463fd630547
+SHA512:
+  metadata.gz: 41275ff885c8b44f5d8d9c5583227355a7dc858846c4d5438c3543d2a4877c6810a35ba3eb3befaf1d2e3b4e9beb31f38e7cb638d08d323faa00bf36c1cacecc
+  data.tar.gz: 4c471bd854b4a3ed23a4e460377c6936ddcc86c2c7377e9a0ce03e3d7e7de18b71deb6dcc6f923341a8840e595167f4cdbe5d65f8a010fd3395f1dc72ea4ab07

data/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.1.0]
+
+- Initial extraction: `Queekiq::Queue` (Redis Streams + consumer group, reclaim,
+  deferral), `Queekiq::Worker` (consumer loop), `Queekiq::Configuration`.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Your Name
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,153 @@
+# Queekiq
+
+A small, framework-agnostic work queue on **Redis Streams**.
+
+- Consumer groups, so any number of worker processes share one queue.
+- At-least-once delivery: messages stay pending until acked, and messages
+  abandoned by a crashed worker are reclaimed by another.
+- **Deferral**: a handler can say "not yet" and the message comes back after a
+  short delay, which is enough to build per-key ordering without locks.
+- A ready-made worker loop with graceful shutdown.
+
+No dependency on Rails (it uses `Rails.logger` automatically if present).
+Requires Ruby >= 3.1, Redis >= 7.0 and the `redis` gem (4.8 up to 6.x).
+
+## Installation
+
+```ruby
+gem "queekiq", github: "your-user/queekiq"   # or path: "../queekiq" while developing
+```
+
+## Quick start
+
+```ruby
+# config/initializers/queekiq.rb (optional; defaults to ENV["REDIS_URL"])
+Queekiq.configure do |c|
+  c.redis_url = ENV.fetch("REDIS_URL", "redis://localhost:6379/0")
+end
+
+EMAILS = Queekiq::Queue.new("emails")
+
+# Producer (web process, console, ...)
+EMAILS.enqueue(user.id)          # payloads are strings; use JSON for structured data
+
+# Consumer (bin/email_worker)
+worker = Queekiq::Worker.new(queue: EMAILS) do |message|
+  Mailer.deliver(message.payload)
+  :sent                          # any return value is the "outcome"
+end
+worker.start                     # traps INT/TERM, runs until stopped
+```
+
+Run as many worker processes as you like; each message goes to exactly one of
+them. Consumer names default to `hostname-pid`, which is unique per process,
+including under Docker.
+
+## How it works
+
+| Piece | Redis structure | Default key |
+|---|---|---|
+| Messages | stream | `queekiq:{name}:stream` |
+| Workers | consumer group | `queekiq-<name>` |
+| Deferred payloads | sorted set (score = due time in ms) | `queekiq:{name}:deferred` |
+
+Each worker iteration:
+
+1. **Promote** deferred payloads that are due back onto the stream (atomically,
+   in a Lua script, so none are lost or promoted twice).
+2. **Reclaim** pending messages idle for longer than `reclaim_after_ms`
+   (default 30 s), i.e. ones whose worker died.
+3. **Read** new messages, blocking up to `block_ms` (default 1 s).
+
+Each message goes through your handler and is then acked.
+
+### Delivery semantics
+
+Delivery is **at-least-once**. A message can be handled more than once (a worker
+dies after the handler finished but before the ack, or a handler is slower than
+`reclaim_after_ms`), so handlers must be idempotent. Set `reclaim_after_ms`
+comfortably above your slowest handler.
+
+### Outcomes and deferral
+
+The handler receives a `Queekiq::Message` (`#id`, `#payload`) and returns an
+outcome. If the outcome is in `defer_on` (default `[:deferred]`) the payload is
+scheduled to come back after the queue's `defer_delay_ms` (default 200 ms). The
+original delivery is acked either way.
+
+That is the building block for ordering. For example, process jobs for the same
+account strictly in submission order, with the ordering rule living in your
+database:
+
+```ruby
+Queekiq::Worker.new(queue: queue) do |message|
+  job = Job.find(message.payload)
+  next :already_done unless job.pending?
+  next :deferred if Job.earlier_pending_for_same_account?(job)   # come back later
+
+  job.run!
+  :done
+end
+```
+
+### Errors
+
+If the handler raises, the worker logs it, calls `on_error` (if given, e.g. to
+report to an error tracker) and leaves the message **unacked**, so it is
+redelivered after `reclaim_after_ms`. A message that always raises will
+therefore be retried forever; handlers that should give up should rescue their
+own errors, record the failure and return an outcome instead.
+
+```ruby
+Queekiq::Worker.new(queue: queue, on_error: ->(error, message) { Sentry.capture_exception(error) }) { ... }
+```
+
+## Configuration
+
+Global (`Queekiq.configure`): `redis_url`, `redis` (an existing client, e.g. a
+shared one), `logger` (defaults to `Rails.logger`, else stdout).
+
+`Queekiq::Queue.new(name, **options)`:
+
+| Option | Default | |
+|---|---|---|
+| `stream`, `group`, `deferred_key` | derived from `name` | Override to adopt existing keys. |
+| `field` | `"payload"` | Stream field holding the payload. |
+| `reclaim_after_ms` | `30_000` | Idle time before a pending message is reclaimed. |
+| `defer_delay_ms` | `200` | Default delay for `#defer`. |
+| `delete_on_ack` | `false` | `XDEL` entries once acked so the stream doesn't grow without bound. |
+| `redis` | `Queekiq.redis` | A client dedicated to this queue. |
+
+`Queekiq::Worker.new(queue:, handler: nil, **options, &block)`:
+`consumer`, `logger`, `defer_on` (`[:deferred]`), `block_ms` (`1_000`, also the
+worst-case time to stop), `on_error`.
+
+Use one Redis connection per worker thread: a blocking read occupies its
+connection.
+
+On Redis Cluster, `stream` and `deferred_key` must share a hash tag (the
+defaults do).
+
+## Low-level API
+
+`Queekiq::Queue` can be used without `Worker`: `enqueue`, `defer`,
+`promote_due!`, `ensure_group!`, `read`, `reclaim`, `ack`, plus `size`,
+`deferred_size` and `clear!` (for tests).
+
+## Development
+
+```bash
+docker compose up -d redis   # or any Redis >= 7 on REDIS_URL
+bundle install
+bundle exec rspec
+```
+
+## Not included (yet)
+
+Delivery-count limits / dead-letter stream, stream trimming beyond
+`delete_on_ack`, delayed enqueue of new work (only deferral of existing
+payloads), metrics.
+
+## License
+
+MIT.

data/lib/queekiq/configuration.rb

@@ -0,0 +1,38 @@
+require "logger"
+require "redis"
+
+module Queekiq
+  # Global defaults shared by every Queue and Worker that doesn't get its own
+  # `redis:` / `logger:` argument.
+  class Configuration
+    DEFAULT_REDIS_URL = "redis://localhost:6379/0".freeze
+
+    attr_accessor :redis_url
+    attr_writer :redis, :logger
+
+    def initialize
+      @redis_url = ENV.fetch("REDIS_URL", DEFAULT_REDIS_URL)
+    end
+
+    # A shared client, built lazily from `redis_url` unless one was assigned.
+    def redis
+      @redis ||= Redis.new(url: redis_url)
+    end
+
+    # Falls back to Rails.logger when running inside Rails, otherwise to a
+    # logger on $stdout.
+    def logger
+      @logger || default_logger
+    end
+
+    private
+
+    def default_logger
+      if defined?(::Rails) && ::Rails.respond_to?(:logger) && ::Rails.logger
+        ::Rails.logger
+      else
+        @default_logger ||= Logger.new($stdout)
+      end
+    end
+  end
+end

data/lib/queekiq/message.rb

@@ -0,0 +1,5 @@
+module Queekiq
+  # One delivery from a queue. `id` is the Redis stream entry id (needed to
+  # ack); `payload` is the string that was passed to Queue#enqueue.
+  Message = Struct.new(:id, :payload, keyword_init: true)
+end

data/lib/queekiq/queue.rb

@@ -0,0 +1,147 @@
+module Queekiq
+  # A durable work queue: a Redis Stream read through a consumer group, plus a
+  # sorted set of "deferred" payloads that are pushed back onto the stream once
+  # their delay has elapsed.
+  #
+  #   queue = Queekiq::Queue.new("emails")
+  #   queue.enqueue("user-42")
+  #   queue.ensure_group!
+  #   queue.read(consumer: "worker-1").each { |m| ...; queue.ack(m) }
+  #
+  # Delivery is at-least-once: a message stays pending until it is acked, and
+  # #reclaim hands pending messages that have been idle too long to another
+  # consumer. Handlers must therefore be idempotent.
+  #
+  # Requires Redis >= 7.0.
+  class Queue
+    DEFAULT_FIELD = "payload".freeze
+    DEFAULT_RECLAIM_AFTER_MS = 30_000
+    DEFAULT_DEFER_DELAY_MS = 200
+
+    # Moves due members of the deferred set onto the stream in one atomic step,
+    # so a crash can't lose a payload between "removed" and "re-enqueued", and
+    # two workers can't both promote the same one.
+    #   KEYS: deferred set, stream   ARGV: now_ms, limit, field name
+    PROMOTE_SCRIPT = <<~LUA.freeze
+      local due = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, ARGV[2])
+      for _, member in ipairs(due) do
+        redis.call('ZREM', KEYS[1], member)
+        redis.call('XADD', KEYS[2], '*', ARGV[3], member)
+      end
+      return #due
+    LUA
+
+    attr_reader :name, :stream, :group, :deferred_key, :field, :reclaim_after_ms, :defer_delay_ms
+
+    # name             - used to derive the default Redis keys.
+    # stream/group/deferred_key
+    #                  - override the derived names, e.g. to adopt keys an
+    #                    application already uses. (On Redis Cluster the stream
+    #                    and deferred key must share a hash tag.)
+    # field            - the stream field holding the payload.
+    # reclaim_after_ms - idle time after which a pending message may be reclaimed.
+    # defer_delay_ms   - default delay for #defer.
+    # delete_on_ack    - also XDEL entries once acked, so the stream doesn't
+    #                    grow forever. Only safe with a single consumer group
+    #                    (which is what Queekiq uses).
+    # redis            - a client for this queue; defaults to Queekiq.redis.
+    def initialize(name, stream: nil, group: nil, deferred_key: nil, field: DEFAULT_FIELD,
+                   reclaim_after_ms: DEFAULT_RECLAIM_AFTER_MS, defer_delay_ms: DEFAULT_DEFER_DELAY_MS,
+                   delete_on_ack: false, redis: nil)
+      raise ArgumentError, "queue name can't be blank" if name.to_s.strip.empty?
+
+      @name = name.to_s
+      @stream = stream || "queekiq:{#{@name}}:stream"
+      @group = group || "queekiq-#{@name}"
+      @deferred_key = deferred_key || "queekiq:{#{@name}}:deferred"
+      @field = field
+      @reclaim_after_ms = reclaim_after_ms
+      @defer_delay_ms = defer_delay_ms
+      @delete_on_ack = delete_on_ack
+      @redis = redis
+    end
+
+    def redis
+      @redis || Queekiq.redis
+    end
+
+    # Appends a message to the stream. `payload` is stored as a string (use
+    # JSON yourself for structured data). Returns the stream entry id.
+    def enqueue(payload)
+      redis.xadd(stream, { field => payload.to_s })
+    end
+
+    # Schedules `payload` to be re-enqueued after `delay_ms`. The deferred set
+    # holds one entry per distinct payload: deferring the same payload twice
+    # just moves its due time.
+    def defer(payload, delay_ms: defer_delay_ms)
+      redis.zadd(deferred_key, now_ms + delay_ms, payload.to_s)
+    end
+
+    # Re-enqueues up to `limit` deferred payloads whose delay has elapsed.
+    # Returns how many were promoted.
+    def promote_due!(limit: 50)
+      redis.eval(PROMOTE_SCRIPT, keys: [ deferred_key, stream ], argv: [ now_ms, limit, field ])
+    end
+
+    # Creates the consumer group (and the stream if needed). Safe to call
+    # repeatedly. The group starts at the beginning of the stream, so messages
+    # enqueued before the group existed are still delivered.
+    def ensure_group!
+      redis.xgroup(:create, stream, group, "0", mkstream: true)
+    rescue Redis::CommandError => e
+      raise unless e.message.include?("BUSYGROUP")
+    end
+
+    # Blocks up to `block_ms` for new messages for `consumer`.
+    # Returns an array of Message (empty when nothing arrived). Messages stay
+    # pending until #ack.
+    def read(consumer:, count: 1, block_ms: 5_000)
+      result = redis.xreadgroup(group, consumer, stream, ">", count: count, block: block_ms)
+      build_messages(result&.dig(stream) || [])
+    end
+
+    # Takes over pending messages that have been idle for reclaim_after_ms
+    # (their consumer probably died) and returns them as Messages.
+    def reclaim(consumer:, count: 10)
+      result = redis.xautoclaim(stream, group, consumer, reclaim_after_ms, "0-0", count: count)
+      entries = result.is_a?(Hash) ? result["entries"] : result[1]
+      build_messages(entries || [])
+    end
+
+    # Marks a message as done. Accepts a Message or a bare stream entry id.
+    def ack(message_or_id)
+      id = message_or_id.respond_to?(:id) ? message_or_id.id : message_or_id
+      redis.xack(stream, group, id)
+      redis.xdel(stream, id) if @delete_on_ack
+    end
+
+    # Number of entries currently in the stream.
+    def size
+      redis.xlen(stream)
+    end
+
+    # Number of payloads waiting in the deferred set.
+    def deferred_size
+      redis.zcard(deferred_key)
+    end
+
+    # Deletes the stream (and its group) and the deferred set. Meant for tests.
+    def clear!
+      redis.del(stream, deferred_key)
+    end
+
+    private
+
+    def now_ms
+      (Time.now.to_f * 1000).to_i
+    end
+
+    def build_messages(entries)
+      entries.map do |id, fields|
+        payload = fields.fetch(field) { raise Error, "stream entry #{id} has no #{field.inspect} field" }
+        Message.new(id: id, payload: payload)
+      end
+    end
+  end
+end

data/lib/queekiq/version.rb

@@ -0,0 +1,3 @@
+module Queekiq
+  VERSION = "0.1.0".freeze
+end

data/lib/queekiq/worker.rb

@@ -0,0 +1,96 @@
+require "socket"
+
+module Queekiq
+  # The consumer loop. Each iteration it promotes due deferrals, picks up
+  # messages other consumers abandoned, then blocks briefly for new ones; every
+  # message goes through the handler and is acked afterwards.
+  #
+  #   worker = Queekiq::Worker.new(queue: queue) do |message|
+  #     process(message.payload)   # return an outcome, e.g. :done or :deferred
+  #   end
+  #   worker.start                 # traps INT/TERM and runs until stopped
+  #
+  # The handler receives a Queekiq::Message and may return anything. If the
+  # outcome is one of `defer_on` (default: :deferred) the payload is put on the
+  # queue's deferred set to be retried later. After the handler returns, the
+  # message is acked whatever the outcome.
+  #
+  # If the handler raises, the error is logged, `on_error` is called, and the
+  # message is left unacked: it will be redelivered once the queue's
+  # reclaim_after_ms has passed. A handler that must not be retried forever
+  # should rescue its own errors and return an outcome instead.
+  class Worker
+    attr_reader :queue, :consumer
+
+    def self.default_consumer
+      "#{Socket.gethostname}-#{Process.pid}"
+    end
+
+    # handler  - anything responding to #call(message); alternatively pass a block.
+    # consumer - unique per worker process (default: hostname + pid).
+    # defer_on - outcomes that trigger Queue#defer.
+    # block_ms - how long one read blocks; also the worst-case stop latency.
+    # on_error - optional #call(error, message), e.g. to report to an error tracker.
+    def initialize(queue:, handler: nil, consumer: self.class.default_consumer, logger: nil,
+                   defer_on: [ :deferred ], block_ms: 1_000, on_error: nil, &block)
+      @handler = handler || block
+      raise ArgumentError, "pass a handler: or a block" unless @handler
+
+      @queue = queue
+      @consumer = consumer
+      @logger = logger
+      @defer_on = Array(defer_on)
+      @block_ms = block_ms
+      @on_error = on_error
+      @running = true
+    end
+
+    def logger
+      @logger || Queekiq.logger
+    end
+
+    # Ask the loop to finish its current iteration and return.
+    def stop
+      @running = false
+    end
+
+    # Convenience for worker scripts: unbuffered stdout, graceful stop on
+    # INT/TERM, then #run.
+    def start(signals: %w[INT TERM])
+      $stdout.sync = true
+      signals.each { |signal| trap(signal) { stop } }
+      run
+    end
+
+    def run
+      queue.ensure_group!
+      logger.info("[queekiq] #{consumer} started on #{queue.name}")
+
+      while @running
+        queue.promote_due!
+        handle(queue.reclaim(consumer: consumer))
+        handle(queue.read(consumer: consumer, block_ms: @block_ms))
+      end
+
+      logger.info("[queekiq] #{consumer} stopped")
+    end
+
+    # Processes messages in the order given. Public so it can be unit tested
+    # without a running loop.
+    def handle(messages)
+      messages.each { |message| process(message) }
+    end
+
+    private
+
+    def process(message)
+      outcome = @handler.call(message)
+      queue.defer(message.payload) if @defer_on.include?(outcome)
+      logger.info("[queekiq] #{consumer} payload=#{message.payload} outcome=#{outcome.inspect}")
+      queue.ack(message)
+    rescue StandardError => e
+      logger.error("[queekiq] #{consumer} payload=#{message.payload} failed: #{e.class}: #{e.message}")
+      @on_error&.call(e, message)
+    end
+  end
+end

data/lib/queekiq.rb

@@ -0,0 +1,33 @@
+require "queekiq/version"
+require "queekiq/configuration"
+require "queekiq/message"
+require "queekiq/queue"
+require "queekiq/worker"
+
+module Queekiq
+  class Error < StandardError; end
+
+  class << self
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    # Queekiq.configure { |c| c.redis_url = "redis://redis:6379/1" }
+    def configure
+      yield configuration
+    end
+
+    def redis
+      configuration.redis
+    end
+
+    def logger
+      configuration.logger
+    end
+
+    # Drops all configuration, including the memoized Redis client.
+    def reset!
+      @configuration = nil
+    end
+  end
+end

metadata

@@ -0,0 +1,75 @@
+--- !ruby/object:Gem::Specification
+name: queekiq
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Marco Aviles
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.8'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
+description: 'Queekiq is a small, framework-agnostic job queue built on Redis Streams:
+  consumer groups for parallel workers, reclaiming of abandoned messages, delayed
+  re-enqueueing (deferral) and a graceful consumer loop.'
+email:
+- gdmarav374@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/queekiq.rb
+- lib/queekiq/configuration.rb
+- lib/queekiq/message.rb
+- lib/queekiq/queue.rb
+- lib/queekiq/version.rb
+- lib/queekiq/worker.rb
+homepage: https://github.com/mhero/queekiq
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/mhero/queekiq
+  changelog_uri: https://github.com/mhero/queekiq/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.22
+specification_version: 4
+summary: Durable Redis Streams work queue with consumer groups, deferral and a worker
+  loop.
+test_files: []