queekiq 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4310b94018510b1e6353861e3d6b282c2141f214b56fba92628f61291ac38f6b
-  data.tar.gz: 52b6ddc84908e73f43e1253f20292ca59284b622503040096ff35463fd630547
+  metadata.gz: 92b27e1bc09a36c133ab56236cb3f0086c92e92b6cabee6ab5d4e742e156dbe2
+  data.tar.gz: 94945eaf96bc1f8a5c373505f61356bf52ea816349f2bfcb98c21e055242b7df
 SHA512:
-  metadata.gz: 41275ff885c8b44f5d8d9c5583227355a7dc858846c4d5438c3543d2a4877c6810a35ba3eb3befaf1d2e3b4e9beb31f38e7cb638d08d323faa00bf36c1cacecc
-  data.tar.gz: 4c471bd854b4a3ed23a4e460377c6936ddcc86c2c7377e9a0ce03e3d7e7de18b71deb6dcc6f923341a8840e595167f4cdbe5d65f8a010fd3395f1dc72ea4ab07
+  metadata.gz: 00b5b5204f50da5694d9b262ee28898a9127d20bfbc7cab1ed706d7004913305f5793b79b489bc43ebc09f3c49677a9b53f318d497cc480bd5ee307527603a1b
+  data.tar.gz: 302a3eb8b14294cf0bf8df6b813277734350c5f432fedcd52bb340ebcb074dca49ccd51758ad4f60afb3cfe0c55222179faaba2ba9421c9cbdae8db93233432e

data/CHANGELOG.md

@@ -2,6 +2,64 @@
 
 ## [Unreleased]
 
+## [0.2.0]
+
+### Added
+
+- **Message envelope.** Every message carries `attempt`, `defers` and `enqueued_at`
+  (readable on `Queekiq::Message`), which survive deferrals and retries.
+- **Serializers.** `serializer:` on `Queue` (`Serializers::Raw`, the default, or
+  `Serializers::JSON`, or your own `dump`/`load` object). Payloads are decoded
+  lazily, so an undecodable payload fails inside the handler and is retried and
+  dead-lettered instead of crashing the worker.
+- **Retries with backoff.** A handler that raises is retried after `retry_backoff`
+  (exponential with jitter). New `Queekiq::Backoff`.
+- **Dead-letter stream.** After `max_attempts` (default 5) failed attempts, or
+  `max_defers` deferrals (default: unlimited), a message moves to the queue's
+  dead stream with the reason and error. `Queue#dead_messages`, `#requeue_dead`,
+  `#delete_dead`, `#dead_size`. Messages whose workers keep dying are dead-lettered
+  too, without running the handler again.
+- **Deferral backoff.** `defer_backoff` grows the delay each time a message is
+  deferred (default 200 ms up to 5 s).
+- **Heartbeat.** While a handler runs, the worker keeps the message from being
+  reclaimed (`Queue#touch`, `heartbeat_interval_ms`), so `reclaim_after_ms` only
+  has to cover a worker dying, not your slowest job. A worker that lost its
+  message to a reclaim can't steal it back.
+- **Resilient worker loop.** Lost Redis connections are retried with backoff
+  (`reconnect_backoff`); a stream or group that disappeared (Redis restarted
+  empty) is recreated.
+- **Safer shutdown.** `stop` finishes the message in hand and, if that takes longer
+  than `shutdown_timeout` (default 25 s), abandons it so another worker reclaims it.
+- **Observability.** `Queue#stats` (backlog, pending, oldest pending idle time,
+  deferred, dead, per-consumer pending) and instrumentation events
+  (`process`, `defer`, `retry`, `dead_letter`, `reclaim`, `connection_error`,
+  all `.queekiq`) via `Queekiq.subscribe`, or ActiveSupport::Notifications through
+  `config.instrumenter`. Log lines are now structured `key=value`.
+- GitHub Actions CI: lint, specs on Ruby 3.1-3.4 / redis gem 4.8-6.x / Redis 7.0 and
+  7.4, and a build-and-install check; Dependabot for actions and gems.
+
+### Changed
+
+- `Queue#defer(message)` now takes the delivered message and acks it in the same
+  atomic step; a bare payload still works for scheduling something new. The
+  worker no longer calls `ack` after a deferral.
+- Failed messages are retried with backoff instead of waiting out
+  `reclaim_after_ms`, and are dead-lettered after `max_attempts`. Pass
+  `max_attempts: nil` to retry forever.
+- Deferral delays now grow (200 ms, 400 ms, ... up to 5 s). Use
+  `defer_backoff: Queekiq::Backoff.fixed(200)` for the old fixed delay.
+- `Queue.new(defer_delay_ms:)` is replaced by `defer_backoff:`.
+- `Message` is a class, not a Struct.
+- `clear!` also deletes the dead stream.
+
+### Upgrading from 0.1
+
+- Deferred entries are now stored as `attempt|defers|enqueued_at|payload`. A 0.1
+  worker would promote one of those with the whole string as its payload, so
+  stop all 0.1 workers before starting 0.2 ones. 0.2 reads everything 0.1 wrote.
+- Specs that stub `Queue#defer` and expect the payload need to expect the
+  message: `have_received(:defer).with(having_attributes(payload: "..."))`.
+
 ## [0.1.0]
 
 - Initial extraction: `Queekiq::Queue` (Redis Streams + consumer group, reclaim,

data/README.md

@@ -1,16 +1,20 @@
 # Queekiq
 
-A small, framework-agnostic work queue on **Redis Streams**.
+A small, framework-agnostic work queue on **Redis Streams**. No dependencies
+beyond the `redis` gem.
 
-- Consumer groups, so any number of worker processes share one queue.
-- At-least-once delivery: messages stay pending until acked, and messages
+- **Parallel workers** through consumer groups: every message goes to exactly one worker.
+- **At-least-once delivery**: messages stay pending until acked, and messages
   abandoned by a crashed worker are reclaimed by another.
-- **Deferral**: a handler can say "not yet" and the message comes back after a
-  short delay, which is enough to build per-key ordering without locks.
-- A ready-made worker loop with graceful shutdown.
+- **Retries with backoff and a dead-letter stream** for messages that keep failing.
+- **Deferral**: a handler can say "not yet" and the message comes back shortly,
+  which is enough to build per-key ordering without locks.
+- **A worker loop that survives**: heartbeats for long jobs, graceful shutdown,
+  reconnects when Redis goes away.
+- **Observable**: queue stats, instrumentation events, structured logs.
 
-No dependency on Rails (it uses `Rails.logger` automatically if present).
-Requires Ruby >= 3.1, Redis >= 7.0 and the `redis` gem (4.8 up to 6.x).
+Works with or without Rails (it uses `Rails.logger` automatically if present).
+Requires Ruby >= 3.1, Redis >= 7.0 and the `redis` gem (4.8 up to 6.x; redis 6.x itself needs Ruby >= 3.2).
 
 ## Installation
 
@@ -26,22 +30,21 @@ Queekiq.configure do |c|
   c.redis_url = ENV.fetch("REDIS_URL", "redis://localhost:6379/0")
 end
 
-EMAILS = Queekiq::Queue.new("emails")
+EMAILS = Queekiq::Queue.new("emails", serializer: Queekiq::Serializers::JSON)
 
 # Producer (web process, console, ...)
-EMAILS.enqueue(user.id)          # payloads are strings; use JSON for structured data
+EMAILS.enqueue("user_id" => user.id, "template" => "welcome")
 
 # Consumer (bin/email_worker)
 worker = Queekiq::Worker.new(queue: EMAILS) do |message|
-  Mailer.deliver(message.payload)
+  Mailer.deliver(**message.payload.transform_keys(&:to_sym))
   :sent                          # any return value is the "outcome"
 end
 worker.start                     # traps INT/TERM, runs until stopped
 ```
 
-Run as many worker processes as you like; each message goes to exactly one of
-them. Consumer names default to `hostname-pid`, which is unique per process,
-including under Docker.
+Run as many worker processes as you like. Consumer names default to
+`hostname-pid`, which is unique per process, including under Docker.
 
 ## How it works
 
@@ -49,104 +52,219 @@ including under Docker.
 |---|---|---|
 | Messages | stream | `queekiq:{name}:stream` |
 | Workers | consumer group | `queekiq-<name>` |
-| Deferred payloads | sorted set (score = due time in ms) | `queekiq:{name}:deferred` |
+| Deferred and retrying messages | sorted set (score = due time in ms) | `queekiq:{name}:deferred` |
+| Messages that gave up | stream | `queekiq:{name}:dead` |
 
 Each worker iteration:
 
-1. **Promote** deferred payloads that are due back onto the stream (atomically,
+1. **Promote** deferred messages that are due back onto the stream (atomically,
    in a Lua script, so none are lost or promoted twice).
 2. **Reclaim** pending messages idle for longer than `reclaim_after_ms`
    (default 30 s), i.e. ones whose worker died.
 3. **Read** new messages, blocking up to `block_ms` (default 1 s).
 
-Each message goes through your handler and is then acked.
+Each message goes through your handler and is then settled:
+
+| Handler | What happens |
+|---|---|
+| returns normally | the message is acked |
+| returns an outcome in `defer_on` (default `:deferred`) | it comes back after a delay (`defer_backoff`); dead-lettered after `max_defers` |
+| raises | it is retried after a delay (`retry_backoff`); dead-lettered after `max_attempts` |
+| the worker dies mid-handler | another worker reclaims it after `reclaim_after_ms` |
 
 ### Delivery semantics
 
 Delivery is **at-least-once**. A message can be handled more than once (a worker
-dies after the handler finished but before the ack, or a handler is slower than
-`reclaim_after_ms`), so handlers must be idempotent. Set `reclaim_after_ms`
-comfortably above your slowest handler.
+dies after the handler finished but before the ack), so handlers must be
+idempotent. Ordering is never guaranteed by the queue itself; see Deferral.
+
+## Messages and serializers
+
+The handler receives a `Queekiq::Message`:
+
+| | |
+|---|---|
+| `payload` | what you enqueued, decoded by the serializer (lazily) |
+| `raw` | the payload as stored, a String |
+| `id` | the stream entry id |
+| `attempt` | 1 on the first delivery; +1 per retry or per delivery that never finished |
+| `defers` | how many times it has been deferred |
+| `enqueued_at` | when it was first enqueued (epoch ms); `latency_ms` is the age |
 
-### Outcomes and deferral
+Serializers are anything with `dump(object) -> String` and `load(String) -> object`:
+`Serializers::Raw` (default, plain strings, like Redis itself), `Serializers::JSON`
+(hash keys come back as strings), or your own (MessagePack, Marshal, ...).
+Because decoding is lazy, a payload that can't be decoded raises inside your
+handler, and goes through retries and the dead stream like any other failure.
 
-The handler receives a `Queekiq::Message` (`#id`, `#payload`) and returns an
-outcome. If the outcome is in `defer_on` (default `[:deferred]`) the payload is
-scheduled to come back after the queue's `defer_delay_ms` (default 200 ms). The
-original delivery is acked either way.
+## Deferral and ordering
+
+A handler returning `:deferred` means "not yet, try again shortly". The message
+keeps its history, and each deferral waits a little longer (200 ms, 400 ms, ...
+up to 5 s by default).
 
 That is the building block for ordering. For example, process jobs for the same
-account strictly in submission order, with the ordering rule living in your
-database:
+account strictly in submission order, with the rule living in your database:
 
 ```ruby
 Queekiq::Worker.new(queue: queue) do |message|
   job = Job.find(message.payload)
   next :already_done unless job.pending?
-  next :deferred if Job.earlier_pending_for_same_account?(job)   # come back later
+  next :deferred if Job.earlier_pending_for_same_account?(job)
 
   job.run!
   :done
 end
 ```
 
-### Errors
+The ordering key can be anything your database can compare: a sequence number,
+a ULID, a timestamp, or a state machine ("a refund can only apply to a captured
+payment"). Use `max_defers` if a message that waits forever should end up in the
+dead stream instead.
+
+## Failures, retries and dead letters
 
-If the handler raises, the worker logs it, calls `on_error` (if given, e.g. to
-report to an error tracker) and leaves the message **unacked**, so it is
-redelivered after `reclaim_after_ms`. A message that always raises will
-therefore be retried forever; handlers that should give up should rescue their
-own errors, record the failure and return an outcome instead.
+A raising handler is logged, passed to `on_error` (e.g. to report to an error
+tracker), and retried after `retry_backoff` (default 1 s, doubling, up to 5 min,
+with jitter). After `max_attempts` (default 5) it is moved to the dead stream
+with the reason and the error. A message whose workers keep *dying* is
+dead-lettered the same way, without running the handler again.
 
 ```ruby
-Queekiq::Worker.new(queue: queue, on_error: ->(error, message) { Sentry.capture_exception(error) }) { ... }
+queue.dead_size                              # => 1
+dead = queue.dead_messages.first             # DeadMessage
+dead.raw; dead.reason; dead.error_class; dead.error_message; dead.attempt
+queue.requeue_dead(dead.id)                  # back on the queue, attempt reset
+queue.delete_dead(dead.id)
 ```
 
+Set `max_attempts: nil` to retry forever. A handler that should never be
+retried can rescue its own errors and return an outcome instead.
+
+## Long jobs
+
+While a handler runs, the worker touches its message every `reclaim_after_ms / 3`
+(`heartbeat_interval_ms`; `false` disables it). A slow job is therefore never
+reclaimed and run twice just for being slow, while a worker that really died
+stops heartbeating and is reclaimed after `reclaim_after_ms`. A worker that lost
+its message (it was reclaimed anyway) can't take it back.
+
+## Shutdown
+
+`worker.stop` (called for you on INT/TERM by `start`) lets the message in hand
+finish. If that takes longer than `shutdown_timeout` (default 25 s), the worker
+gives up on it and exits; the message stays pending and another worker reclaims
+it. Set `shutdown_timeout:` below your platform's kill grace period (Docker's is
+10 s by default, Kubernetes' 30 s).
+
+## When Redis goes away
+
+Connection errors don't crash the worker: it waits (`reconnect_backoff`, 0.5 s
+doubling up to 30 s) and tries again. A stream or consumer group that vanished
+(Redis restarted without persistence) is recreated. Messages being handled stay
+pending and are reclaimed afterwards.
+
+## Observability
+
+```ruby
+queue.stats
+# => { backlog: 12, pending: 2, oldest_pending_idle_ms: 340, deferred: 3, dead: 0,
+#      consumers: [{ name: "web1-123", pending: 1, idle_ms: 5 }, ...] }
+```
+
+`oldest_pending_idle_ms` is the one to alert on: a large value means workers are
+stuck or dead. `backlog` is nil if Redis can't tell (after entries were deleted).
+
+Workers emit events with these names, each with `queue:` and `consumer:` and a
+`duration_ms:`:
+
+| Event | Extra payload |
+|---|---|
+| `process.queekiq` | `message`, `outcome`, `latency_ms`, `exception` if the handler raised |
+| `defer.queekiq` | `message`, `delay_ms` |
+| `retry.queekiq` | `message`, `delay_ms`, `error` |
+| `dead_letter.queekiq` | `message`, `reason`, `error` |
+| `reclaim.queekiq` | `count` |
+| `connection_error.queekiq` | `error`, `retry_in_ms`, `failures` |
+
+```ruby
+Queekiq.subscribe("process.queekiq") do |_name, payload|
+  StatsD.timing("queekiq.process", payload[:duration_ms], tags: [ "queue:#{payload[:queue]}" ])
+  StatsD.timing("queekiq.latency", payload[:latency_ms]) if payload[:latency_ms]
+end
+
+# Or in Rails, use ActiveSupport::Notifications instead:
+Queekiq.configure { |c| c.instrumenter = ActiveSupport::Notifications }
+```
+
+Logs are one `key=value` line per event, e.g.
+`[queekiq] event=retry queue=emails consumer=web1-123 id=1-0 payload=42 attempt=1 delay_ms=1043`.
+Payloads are truncated to 100 characters, but don't put secrets in them.
+
 ## Configuration
 
-Global (`Queekiq.configure`): `redis_url`, `redis` (an existing client, e.g. a
-shared one), `logger` (defaults to `Rails.logger`, else stdout).
+Global (`Queekiq.configure`): `redis_url`, `redis` (an existing client),
+`logger` (defaults to `Rails.logger`, else stdout), `instrumenter`.
 
 `Queekiq::Queue.new(name, **options)`:
 
 | Option | Default | |
 |---|---|---|
-| `stream`, `group`, `deferred_key` | derived from `name` | Override to adopt existing keys. |
+| `stream`, `group`, `deferred_key`, `dead_key` | derived from `name` | Override to adopt existing keys. |
 | `field` | `"payload"` | Stream field holding the payload. |
+| `serializer` | `Serializers::Raw` | How payloads are stored. |
 | `reclaim_after_ms` | `30_000` | Idle time before a pending message is reclaimed. |
-| `defer_delay_ms` | `200` | Default delay for `#defer`. |
+| `max_attempts` | `5` | Failed attempts before dead-lettering; `nil` = forever. |
+| `max_defers` | `nil` | Deferrals before dead-lettering; `nil` = forever. |
+| `retry_backoff` | `Backoff.new(base_ms: 1_000, max_ms: 300_000)` | Delay after a failure. |
+| `defer_backoff` | `Backoff.new(base_ms: 200, max_ms: 5_000)` | Delay after a deferral. |
 | `delete_on_ack` | `false` | `XDEL` entries once acked so the stream doesn't grow without bound. |
 | `redis` | `Queekiq.redis` | A client dedicated to this queue. |
 
 `Queekiq::Worker.new(queue:, handler: nil, **options, &block)`:
-`consumer`, `logger`, `defer_on` (`[:deferred]`), `block_ms` (`1_000`, also the
-worst-case time to stop), `on_error`.
 
-Use one Redis connection per worker thread: a blocking read occupies its
-connection.
+| Option | Default | |
+|---|---|---|
+| `consumer` | `hostname-pid` | Unique per worker process. |
+| `defer_on` | `[:deferred]` | Outcomes that defer the message. |
+| `block_ms` | `1_000` | How long one read blocks. |
+| `on_error` | `nil` | `#call(error, message)`. |
+| `heartbeat_interval_ms` | `reclaim_after_ms / 3` | `false` disables. |
+| `shutdown_timeout` | `25` (seconds) | `nil` waits forever. |
+| `reconnect_backoff` | `Backoff.new(base_ms: 500, max_ms: 30_000)` | Pause while Redis is down. |
+| `logger` | `Queekiq.logger` | |
+
+`Queekiq::Backoff.new(base_ms:, max_ms:, factor: 2, jitter: 0.2)` is exponential
+with jitter; `Backoff.fixed(ms)` is constant; anything with `delay_ms(attempt)` works.
 
-On Redis Cluster, `stream` and `deferred_key` must share a hash tag (the
-defaults do).
+Use one Redis connection per worker thread: a blocking read occupies its
+connection. On Redis Cluster, the stream, deferred and dead keys must share a
+hash tag (the defaults do).
 
 ## Low-level API
 
-`Queekiq::Queue` can be used without `Worker`: `enqueue`, `defer`,
-`promote_due!`, `ensure_group!`, `read`, `reclaim`, `ack`, plus `size`,
-`deferred_size` and `clear!` (for tests).
+`Queekiq::Queue` can be used without `Worker`: `enqueue`, `defer`, `retry_later`,
+`dead_letter`, `promote_due!`, `ensure_group!`, `read`, `reclaim`, `touch`, `ack`,
+plus `stats`, `size`, `deferred_size`, `dead_size` and `clear!` (for tests).
 
 ## Development
 
 ```bash
 docker compose up -d redis   # or any Redis >= 7 on REDIS_URL
 bundle install
-bundle exec rspec
+bundle exec rake             # specs
+bundle exec rubocop
 ```
 
+CI (GitHub Actions) runs the lint, the specs on Ruby 3.1-3.4, the redis gem 4.8,
+5.x and 6.x and Redis 7.0 and 7.4, and builds and installs the gem.
+To run the specs against another redis gem locally: `REDIS_GEM_VERSION="~> 5.0" bundle update redis`.
+
 ## Not included (yet)
 
-Delivery-count limits / dead-letter stream, stream trimming beyond
-`delete_on_ack`, delayed enqueue of new work (only deferral of existing
-payloads), metrics.
+Per-key rate limits or leases, sharded streams for strict per-key ordering,
+scheduling brand-new work for later (only deferral and retries), replay and
+multiple consumer groups, a web UI, and an in-memory backend for tests.
 
 ## License
 

data/lib/queekiq/backoff.rb

@@ -0,0 +1,31 @@
+module Queekiq
+  # Exponential delay with jitter and a cap: base, base*factor, base*factor^2,
+  # ... never more than max_ms, each randomly shifted by up to +/- `jitter`
+  # (a fraction, 0.2 = 20%).
+  #
+  # Anything responding to #delay_ms(attempt) can be used where a backoff is
+  # expected; Backoff.fixed(ms) gives a constant delay.
+  class Backoff
+    attr_reader :base_ms, :max_ms, :factor, :jitter
+
+    def self.fixed(ms)
+      new(base_ms: ms, max_ms: ms, factor: 1, jitter: 0)
+    end
+
+    def initialize(base_ms: 200, max_ms: 30_000, factor: 2, jitter: 0.2, random: Random)
+      @base_ms = base_ms
+      @max_ms = max_ms
+      @factor = factor.to_f
+      @jitter = jitter
+      @random = random
+    end
+
+    # attempt is 1-based: delay_ms(1) is the delay before the first retry.
+    def delay_ms(attempt)
+      steps = [ attempt.to_i - 1, 0 ].max
+      delay = [ base_ms * (factor**steps), max_ms ].min
+      delay *= 1 + (((@random.rand * 2) - 1) * jitter) if jitter.positive?
+      [ delay.round, 0 ].max
+    end
+  end
+end

data/lib/queekiq/configuration.rb

@@ -8,7 +8,7 @@ module Queekiq
     DEFAULT_REDIS_URL = "redis://localhost:6379/0".freeze
 
     attr_accessor :redis_url
-    attr_writer :redis, :logger
+    attr_writer :redis, :logger, :instrumenter
 
     def initialize
       @redis_url = ENV.fetch("REDIS_URL", DEFAULT_REDIS_URL)
@@ -25,6 +25,12 @@ module Queekiq
       @logger || default_logger
     end
 
+    # Receives the events workers emit (see Instrumenter). Defaults to a
+    # built-in Instrumenter; anything with the same #instrument call shape works.
+    def instrumenter
+      @instrumenter ||= Instrumenter.new
+    end
+
     private
 
     def default_logger

data/lib/queekiq/instrumenter.rb

@@ -0,0 +1,54 @@
+module Queekiq
+  # A tiny in-process event bus, so metrics and tracing can hook in without
+  # Queekiq depending on any monitoring library.
+  #
+  #   Queekiq.subscribe("process.queekiq") do |name, payload|
+  #     StatsD.timing("queekiq.process", payload[:duration_ms])
+  #   end
+  #
+  # Workers call `instrument(name, payload) { |payload| ... }`. The block's
+  # duration is added to the payload as :duration_ms and, if it raises, the error
+  # as :exception. Subscribers that raise are ignored.
+  #
+  # The call shape matches ActiveSupport::Notifications, so in Rails you can use
+  # it instead: `Queekiq.configure { |c| c.instrumenter = ActiveSupport::Notifications }`
+  # (then subscribe through ActiveSupport).
+  class Instrumenter
+    def initialize
+      @subscribers = []
+    end
+
+    # pattern: nil (every event), a String (exact event name) or a Regexp.
+    def subscribe(pattern = nil, &block)
+      raise ArgumentError, "a block is required" unless block
+
+      @subscribers << [ pattern, block ]
+      block
+    end
+
+    def instrument(name, payload = {})
+      started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      begin
+        block_given? ? yield(payload) : nil
+      rescue StandardError => e
+        payload[:exception] = [ e.class.name, e.message ]
+        raise
+      ensure
+        payload[:duration_ms] = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - started) * 1000).round(2)
+        publish(name, payload)
+      end
+    end
+
+    private
+
+    def publish(name, payload)
+      @subscribers.each do |pattern, block|
+        next unless pattern.nil? || pattern === name # rubocop:disable Style/CaseEquality
+
+        block.call(name, payload)
+      rescue StandardError
+        nil # a broken subscriber must never break message processing
+      end
+    end
+  end
+end

data/lib/queekiq/message.rb

@@ -1,5 +1,50 @@
 module Queekiq
-  # One delivery from a queue. `id` is the Redis stream entry id (needed to
-  # ack); `payload` is the string that was passed to Queue#enqueue.
-  Message = Struct.new(:id, :payload, keyword_init: true)
+  # One delivery from a queue.
+  #
+  # id          - the Redis stream entry id (needed to ack); nil for a message
+  #               that was never delivered.
+  # payload     - the enqueued object, decoded with the queue's serializer. It is
+  #               decoded on first use, so a payload that can't be decoded fails
+  #               inside the handler (and is retried / dead-lettered) instead of
+  #               crashing the worker.
+  # raw         - the payload exactly as stored in Redis (a String).
+  # attempt     - 1 for the first delivery; goes up each time the message is
+  #               redelivered after a failure or after its worker died.
+  # defers      - how many times the message has been deferred so far.
+  # enqueued_at - when it was first enqueued, in epoch milliseconds (nil for
+  #               entries written by Queekiq 0.1).
+  class Message
+    NOT_LOADED = Object.new.freeze
+    private_constant :NOT_LOADED
+
+    attr_reader :id, :raw, :attempt, :defers, :enqueued_at
+
+    # Pass `payload:` (it is serialized to get `raw`) or `raw:` (decoded lazily).
+    def initialize(id:, payload: NOT_LOADED, raw: nil, serializer: Serializers::Raw,
+                   attempt: 1, defers: 0, enqueued_at: nil)
+      raise ArgumentError, "pass payload: or raw:" if payload.equal?(NOT_LOADED) && raw.nil?
+
+      @id = id
+      @serializer = serializer
+      @payload = payload
+      @raw = raw.nil? ? serializer.dump(payload) : raw
+      @attempt = attempt
+      @defers = defers
+      @enqueued_at = enqueued_at
+    end
+
+    def payload
+      @payload = @serializer.load(@raw) if @payload.equal?(NOT_LOADED)
+      @payload
+    end
+
+    # Milliseconds since the message was first enqueued, or nil if unknown.
+    def latency_ms(now_ms = (Time.now.to_f * 1000).to_i)
+      now_ms - enqueued_at if enqueued_at
+    end
+
+    def inspect
+      "#<#{self.class} id=#{id.inspect} raw=#{raw[0, 60].inspect} attempt=#{attempt} defers=#{defers}>"
+    end
+  end
 end