sidekiq-buffered 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: cf1f0ec18cf0aa67f5a920af04b9ad90395b0e92498a294e3662fa4f705a34c4
+  data.tar.gz: 149f62313b6613c8cb924b1d1c4b82a1383d07ce1aa3ad974648ef385322a355
+SHA512:
+  metadata.gz: c7ac58054551cc0ddc95adeb9b5c0fedf5858f4d2c372cafd1441ec22dc749709698157ca42df36840af48467e3ddfce3663a86e28ea9c2b30d2986f36772def
+  data.tar.gz: bc09e4814a9787cc9b8427710a3215086fa930e989e593d51a87463db253420ca61aea1cecbaed1c41b806cd9ccb6fdff4e01b0abff7c772966282ec010a938c

data/.rspec

@@ -0,0 +1,2 @@
+--require spec_helper
+--format documentation

data/.rubocop.yml

@@ -0,0 +1,64 @@
+plugins:
+  - rubocop-performance
+  - rubocop-rspec
+
+AllCops:
+  TargetRubyVersion: 3.2
+  NewCops: enable
+  SuggestExtensions: false
+  Exclude:
+    - tmp/**/*
+    - vendor/**/*
+    - pkg/**/*
+
+Naming/FileName:
+  Exclude:
+    - lib/sidekiq-buffered.rb
+
+Style/Documentation:
+  Enabled: false
+
+Style/FrozenStringLiteralComment:
+  Enabled: false
+
+Style/StringLiterals:
+  EnforcedStyle: double_quotes
+
+Style/TrailingCommaInArrayLiteral:
+  EnforcedStyleForMultiline: consistent_comma
+
+Style/TrailingCommaInHashLiteral:
+  EnforcedStyleForMultiline: consistent_comma
+
+Style/TrailingCommaInArguments:
+  EnforcedStyleForMultiline: consistent_comma
+
+Layout/FirstArrayElementIndentation:
+  EnforcedStyle: consistent
+
+Gemspec/DevelopmentDependencies:
+  Enabled: false
+
+Metrics/AbcSize:
+  Max: 25
+
+Metrics/ClassLength:
+  Enabled: false
+
+Metrics/MethodLength:
+  Max: 30
+
+RSpec/MultipleExpectations:
+  Enabled: false
+
+RSpec/ExampleLength:
+  Max: 20
+
+RSpec/NestedGroups:
+  Enabled: false
+
+RSpec/DescribeClass:
+  Enabled: false
+
+Metrics/ParameterLists:
+  CountKeywordArgs: false

data/.tool-versions

@@ -0,0 +1 @@
+ruby 4.0.6

data/CHANGELOG.md

@@ -0,0 +1,32 @@
+# Changelog
+
+All notable changes to this project are documented in this file. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project adheres to
+[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-09-20
+
+First release.
+
+### Added
+
+- `Sidekiq::Buffered` mixin with a `buffered` declaration that takes `max_size`, `max_wait`,
+  an optional `group_by` key, and an optional per-job `lease_timeout`.
+- Buffering through the native `perform_async`, `perform_bulk`, and `perform_in` APIs, which
+  keep their argument shapes and still return input JIDs.
+- Batches dispatched as `Sidekiq::Buffered::Processor` jobs that name the original job class
+  as `wrapped` and `display_class`, so Sidekiq logs, metrics, and Web attribute them correctly.
+- A poller that flushes buffers whose `max_wait` has elapsed and a heartbeat that keeps
+  running batches leased, both started inside Sidekiq server processes.
+- Redis lease-based claiming with recovery of batches whose worker died, retry of failed
+  batches under the inputs' retry options, and manual replay of exhausted batches.
+- Global settings for `poll_interval`, `lease_timeout`, `recovery_interval`,
+  `recovery_max_interval`, and `failed_retention`.
+- Read-only batch context through `buffered_batch` inside `perform` and
+  `Sidekiq::Buffered.current_batch` in server middleware.
+- Batch inspection, statistics, lifecycle events, and buffer controls.
+- Active Job support and scheduled inputs.
+- Sidekiq testing support: fake-mode `drain` and `perform_one` execute buffered batches, and
+  `perform_inline` runs an explicit batch.
+
+[0.1.0]: https://github.com/ksarunas/sidekiq-buffered/releases/tag/v0.1.0

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+gemspec
+
+gem "activejob", ENV.fetch("ACTIVEJOB_VERSION") if ENV["ACTIVEJOB_VERSION"]
+gem "sidekiq", ENV.fetch("SIDEKIQ_VERSION") if ENV["SIDEKIQ_VERSION"]

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sarunas
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,432 @@
+# sidekiq-buffered
+
+Buffer individually enqueued Sidekiq job inputs and invoke `perform` once with an array of
+those inputs.
+
+## Installation
+
+Add the gem to your application and require it where Sidekiq is configured:
+
+```ruby
+require "sidekiq-buffered"
+```
+
+Requiring the gem prepends a client middleware and a server middleware to Sidekiq's chains
+and, inside a Sidekiq server process, starts two background threads: a poller that flushes
+buffers whose wait has elapsed, and a heartbeat that keeps running batches leased. Nothing
+else changes for jobs that do not declare `buffered`. The gem uses only Sidekiq's public
+extension points and does not reopen Sidekiq or Active Job classes; see
+[Design notes](#design-notes) for the list.
+
+## Quick start
+
+```ruby
+class JukeboxJob
+  include Sidekiq::Job
+  include Sidekiq::Buffered
+
+  buffered(
+    max_size: 50,
+    max_wait: 2,
+    group_by: :diner,
+  )
+
+  def perform(items)
+    diner = items.first.fetch("diner")
+    tracks = items.map { |item| item.fetch("track") }
+    play_on_jukebox(diner, tracks)
+  end
+end
+
+JukeboxJob.perform_async(
+  diner: "mel's",
+  track: { title: "Rebel Rebel", artist: "David Bowie" },
+)
+
+JukeboxJob.perform_bulk([
+  [{ diner: "mel's", track: { title: "Blitzkrieg Bop" } }],
+  [{ diner: "peachy's", track: { title: "Teenage Kicks" } }],
+])
+```
+
+Each enqueue takes one Hash. `perform` receives an array of those hashes, string-keyed, from a
+single group. `perform_async` and `perform_bulk` keep their native argument shapes and still
+return input JIDs. `batch_size:` remains Sidekiq's enqueue chunking; `max_size:` is the
+execution batch size.
+
+Inside `perform`, `buffered_batch` exposes a read-only context with `id`, `job_class`,
+`group`, `input_jids`, and `attempts`. Use the batch or input IDs for idempotency keys and
+correlation. Server middleware can read the same context as `Sidekiq::Buffered.current_batch`;
+it is cleared after execution, including on exceptions.
+
+Calling `Sidekiq::Client.push` or `push_bulk` directly with a buffered class still buffers the
+inputs, but Sidekiq returns `nil` for pushes its client middleware diverts, and each input is
+admitted in its own Redis call. Use the job class methods or `Sidekiq::Buffered::Client` to
+keep input JIDs and atomic chunk admission.
+
+## How a batch moves
+
+1. **Admission.** Enqueueing writes each input into a Redis buffer. A buffer holds inputs of
+   one job class, one grouping value, and one set of execution options and middleware
+   metadata. All immediate inputs in a Sidekiq enqueue chunk are admitted in one Redis script
+   call.
+2. **Materialization.** When a buffer reaches `max_size`, the script immediately cuts a batch
+   from it. Otherwise the poller cuts a batch once `max_wait` has elapsed since the buffer's
+   first input.
+3. **Dispatch.** Each batch is queued as a `Sidekiq::Buffered::Processor` job whose only
+   argument is the batch ID, on the inputs' queue and with their retry options. The payload
+   names the original job class as its `wrapped` and `display_class`, so Sidekiq's logs,
+   metrics, and Web attribute the batch to that class.
+4. **Claim and execution.** The server middleware claims the batch under a Redis lease before
+   the rest of the server chain runs, then the processor calls `perform(items)`.
+5. **Acknowledgement.** After the chain succeeds, the middleware acknowledges the batch, which
+   deletes it. A batch that fails stays in Redis for retry, recovery, or manual replay.
+
+Delivery is at-least-once: a crash after `perform` succeeded but before acknowledgement runs
+the batch again, and a failure after `perform` in downstream middleware may repeat application
+side effects. Keep application work idempotent on the batch or input IDs.
+
+## Configuration
+
+### Per job
+
+```ruby
+buffered max_size: 50, max_wait: 2, group_by: :diner, lease_timeout: 120
+```
+
+| Option          | Required | Meaning                                                                              |
+|-----------------|----------|--------------------------------------------------------------------------------------|
+| `max_size`      | yes      | Inputs per batch. A full buffer is cut into a batch immediately.                     |
+| `max_wait`      | yes      | Seconds a partial buffer waits, from its first input, before the poller flushes it.  |
+| `group_by`      | no       | Argument key whose value separates buffers. Every input must carry it.               |
+| `lease_timeout` | no       | Overrides the global lease timeout for this job's batches (see below).               |
+
+The declaration is inherited by subclasses; redeclaring `buffered` overrides it for that
+subclass only. Each concrete job class has its own buffers.
+
+Inputs with different Sidekiq execution options (`queue`, `retry`, `retry_for`,
+`retry_queue`, `dead`, `backtrace`) form separate buffers, and a batch runs with the options
+its inputs were enqueued with. Metadata attached by other client middleware, such as
+persisted `CurrentAttributes`, trace fields, and tags, also separates buffers and is retained
+for execution and replay, so a batch never adopts the last producer's tenant context.
+Request-specific trace IDs therefore reduce batch sizes. Server middleware still wraps the
+processor once per batch. The `buffered_context` and `buffered_active_job_context` payload
+fields are reserved for the gem.
+
+### Global
+
+```ruby
+Sidekiq::Buffered.lease_timeout = 60
+Sidekiq::Buffered.failed_retention = 7 * 24 * 60 * 60
+```
+
+| Setting                 | Default | Meaning                                                                                                 |
+|-------------------------|---------|---------------------------------------------------------------------------------------------------------|
+| `poll_interval`         | `0.25`  | Target seconds between poller ticks across the whole cluster. Positive number.                         |
+| `lease_timeout`         | `60`    | Seconds until a crashed worker's batch becomes claimable again. Positive Integer.                      |
+| `recovery_interval`     | `30`    | Seconds after dispatch before recovery first checks whether a processor is still queued. Positive Integer. |
+| `recovery_max_interval` | `300`   | Upper bound for the recovery check backoff. Positive Integer.                                           |
+| `failed_retention`      | `nil`   | Seconds to keep exhausted batches before deleting them, or `nil` to keep them until replayed.          |
+
+`poll_interval` is a cluster target: each process stretches its own idle interval by the
+number of Sidekiq processes, with random jitter, the way Sidekiq's scheduler does, and shortens
+it to the next known buffer deadline or recovery check.
+
+Because running batches are kept alive by the heartbeat, `lease_timeout` bounds recovery delay
+after a worker dies, not how long `perform` may run. A job-level override is fixed when its
+batch materializes and also extends that batch's retry protection window.
+
+Cleanup under `failed_retention` permanently removes the gem's stored payloads and their
+outstanding counts. It is separate from Sidekiq's dead-job retention. Retained failures keep
+counting in `Sidekiq::Buffered.stats`, so inspect and resolve them rather than letting them
+accumulate.
+
+The gem imposes no cap on buffered inputs. A backlog lives in Redis like any Sidekiq queue
+backlog; watch `outstanding_inputs` and `stored_bytes` in `Sidekiq::Buffered.stats` and alert
+on them the way you alert on queue depth.
+
+## Failures and recovery
+
+**Only one worker runs a batch at a time.** A Redis lease makes the claim exclusive. Each
+Sidekiq process runs a heartbeat thread that extends the leases of batches it is executing at
+a third of the lease timeout. If the heartbeat cannot reach Redis for a whole lease timeout,
+the lease expires and another worker can claim the batch. The original worker logs
+`lease lost`, emits a `lease_lost` event, and its acknowledgement is refused.
+
+**Failures retry through Sidekiq.** A processor retry is a Sidekiq retry of the same job, with
+the same JID. On failure the gem records the error and keeps the batch out of recovery for
+Sidekiq's maximum default retry delay plus the lease timeout, so recovery never races a retry
+that Sidekiq is about to run. The retrying processor itself can reclaim immediately; the
+window is not a minimum execution time. A failure callback from an earlier execution JID
+cannot overwrite a newer owner's state. Failures in downstream middleware, before or after
+`perform`, retain the batch for retry. Middleware that does not yield leaves the batch
+available for recovery rather than discarding it.
+
+The buffered job class's own `sidekiq_retry_in` and `sidekiq_retries_exhausted` callbacks run
+for the batch's processor payload, whose arguments contain the batch ID. A `sidekiq_retry_in`
+callback that returns `:discard` acknowledges the batch; `:kill` sends it straight to
+exhaustion. Sidekiq resolves these callbacks through the payload's `wrapped` class, so
+`buffered` prepends `Sidekiq::Buffered::RetryCallbacks` to the job class's singleton; the
+callbacks Sidekiq sees record the batch state first and then run the declared ones.
+
+**Exhausted batches wait for you.** After Sidekiq's retries are exhausted, the batch stays in
+Redis marked `failed` and is not revived until it is replayed. `Batch#retry!` replays it
+immediately and removes the processor's Dead entry. Sidekiq Web's individual and bulk Dead-set
+retry actions also work: they re-enqueue the exhausted processor under its original JID, the
+gem recognizes the redelivery on claim and replays the batch, and the batch stays `failed` in
+inspection until a worker picks it up. A stale Dead entry for a batch that already completed
+or was replayed elsewhere is a no-op. `failed_retention` deletes exhausted batches
+automatically when set.
+
+**Shutdown releases the lease.** When Sidekiq shuts down while a batch is running, it
+interrupts `perform` and requeues the processor. The gem releases the lease at that point so
+the requeued processor reclaims the batch as soon as a worker picks it up.
+
+**Crashes are recovered by the poller.** A batch whose processor disappeared, for example
+because the queue was flushed or the worker died before claiming, is re-dispatched once its
+recovery deadline passes. Before re-enqueueing, recovery checks whether the dispatched
+processor is still waiting in its queue. Repeated checks back off from `recovery_interval` to
+`recovery_max_interval`. Recovered batch IDs are logged at info level. Buffer deadlines,
+lease expiry, and recovery decisions all use Redis time, so producers and workers with skewed
+clocks agree; Sidekiq's own scheduler controls when scheduled inputs are admitted.
+
+## Inspecting batches
+
+```ruby
+batch = Sidekiq::Buffered::Batch.find(batch_id)
+batch.items
+batch.input_jids
+batch.state      # "queued", "running", "retrying", or "failed"
+batch.attempts   # Successful claims since creation or manual replay
+batch.last_error # { "class" => "RuntimeError", "message" => "...", "at" => 1789750000.0 }
+batch.failed_at  # Unix timestamp when retries were exhausted, or nil
+batch.retry!
+batch.discard!   # Permanently remove an exhausted batch; state becomes "discarded"
+
+Sidekiq::Buffered::Batch.failed(job_class: JukeboxJob, group: "mel's", limit: 20, offset: 20)
+```
+
+`Batch.find` returns a snapshot; call `reload` or fetch again to refresh it. A successful
+delivery removes the batch, so this API is not a history of completed work. Manual replay
+clears the last error and attempt count. Only exhausted batches can be discarded; discarding
+removes the batch's inputs from the `stats` counts and deletes the processor's Dead entry.
+Error details retain the exception class, at most 2,048 UTF-8 bytes of its message, and the
+failure time; they do not include backtraces. Exception messages can contain application
+data, so treat inspection access like access to Sidekiq job payloads.
+
+Batch states are the latest recorded lifecycle state: `queued` includes batches waiting for
+dispatch, `running` means a worker claimed the batch, `retrying` means a failure is awaiting
+retry, and `failed` means retries were exhausted. A crashed worker's batch stays `running`
+until recovery. States are not live scans of Sidekiq's queues or proof that a worker is alive.
+
+### Statistics
+
+```ruby
+Sidekiq::Buffered.stats # Optional job_class: and group: filters
+# => {
+#      outstanding_inputs: 187,
+#      stored_bytes: 42000,
+#      pending_inputs: 23,
+#      oldest_pending_age: 1.4,
+#      oldest_input_age: 62.1,
+#      batches: { queued: 2, running: 1, retrying: 3, failed: 1 }
+#    }
+```
+
+`outstanding_inputs` and `stored_bytes` include partial buffers and unacknowledged batches,
+failures included. Inputs shared by separate middleware contexts are counted once.
+`pending_inputs` counts inputs still in partial buffers, excluding materialized batches.
+`oldest_pending_age` is the age in seconds of the oldest such input; `oldest_input_age` also
+covers queued, running, retrying, and failed batches. Ages are `nil` when no timestamp is
+available, which can be the case for inputs buffered before timestamp tracking existed.
+
+Inspection is read-only and pages through Redis indexes in groups of 100 with pipelined
+reads. Its cost grows with the backlog, so use it for periodic operational checks rather than
+on every enqueue. Results are best-effort snapshots while work changes concurrently.
+
+### Events
+
+```ruby
+Sidekiq::Buffered.on_event = ->(event, details) { Metrics.increment("buffered.#{event}") }
+```
+
+Events are `appended`, `claimed`, `claim_skipped`, `acknowledged`, `acknowledgement_lost`,
+`lease_lost`, `released`, `retrying`, `failed`, `recovered`, and `discarded`. Details contain
+identifiers and counts, not input payloads. Handlers run synchronously, must be thread-safe,
+and should return quickly; handler exceptions are logged without changing delivery. Assign
+`nil` to disable the handler.
+
+## Buffer controls
+
+```ruby
+JukeboxJob.flush(group: "mel's")               # Dispatch partial buffers now; returns batch IDs
+Sidekiq::Buffered.flush(job_class: JukeboxJob) # All groups for this class
+```
+
+Flush and failed-list pagination are best-effort under concurrent changes. Grouping controls
+batch membership only; it does not serialize batches from the same group or guarantee their
+completion order.
+
+## Active Job
+
+Active Job support is opt-in. Add `activejob` (or Rails) to your application and select this
+gem's adapter on each buffered job:
+
+```ruby
+require "active_job/queue_adapters/sidekiq_buffered_adapter"
+
+class JukeboxJob < ActiveJob::Base
+  include Sidekiq::Buffered
+
+  self.queue_adapter = :sidekiq_buffered
+  buffered max_size: 50, max_wait: 2, group_by: :diner
+
+  def perform(items)
+    play_on_jukebox(items.first.fetch(:diner), items.map { |item| item.fetch(:track) })
+  end
+end
+
+JukeboxJob.perform_later(diner: "mel's", track: "Rebel Rebel")
+JukeboxJob.set(wait: 5.minutes).perform_later(diner: "mel's", track: "Later")
+```
+
+Use `:sidekiq_buffered` only for jobs that declare `buffered`; other jobs keep their existing
+adapter. Including `Sidekiq::Buffered` under the ordinary `:sidekiq` adapter does not enable
+buffering. Active Job is an optional dependency and is not loaded for native Sidekiq jobs.
+
+Each enqueue accepts one Hash. Active Job serializes inputs, preserving symbol keys,
+GlobalIDs, and its other supported argument types. A batch executes through
+`ActiveJob::Base.execute` as **one Active Job** with an array of deserialized input hashes,
+so perform callbacks, instrumentation, `rescue_from`, `retry_on`, and `discard_on` apply once
+to the whole batch. Enqueue callbacks apply to each input. `perform_now` remains ordinary,
+unbuffered Active Job execution and expects the array that `perform` takes.
+
+The batch's Active Job `job_id` is the batch ID and its `provider_job_id` is the processor
+JID. Input jobs keep their own job IDs and receive individual Sidekiq provider JIDs. Inputs
+with different grouping values, queues, locales, or timezones do not share a batch; the batch
+inherits its first input's other Active Job metadata. Batches execute on the inputs' queue and
+Active Job retries use their selected queue. Numeric Active Job priority does not control
+Sidekiq execution order.
+
+`retry_on` schedules the complete batch through a Sidekiq `ActiveJobRetry` wrapper without
+rebuffering it. Once that enqueue succeeds the original stored batch is acknowledged, and
+subsequent retries and terminal failures live in Sidekiq, outside buffered batch inspection
+and retention. The gem observes Active Job's `enqueue_retry.active_job` instrumentation; a
+retry that was announced but not accepted by Sidekiq, for example because an enqueue callback
+aborted or a client middleware rejected it, raises `Sidekiq::Buffered::RetryEnqueueError` and
+keeps the source batch for Sidekiq retry. `discard_on` also acknowledges the batch. Unhandled
+exceptions keep the original batch in the gem's Sidekiq retry and replay lifecycle. Each such
+processor attempt starts a fresh Active Job execution; Active Job's execution counter carries
+forward only across Active Job retries. Delivery remains at-least-once.
+
+On Active Job 7.2 and later the adapter inherits `ActiveJob::QueueAdapters::AbstractAdapter`
+and answers `enqueue_after_transaction_commit?` with `true`, like Sidekiq's own adapter, so
+jobs enqueued inside a transaction are buffered after it commits when Rails asks. Scheduled
+Active Job inputs follow the delay-first behavior described next.
+
+## Scheduled inputs
+
+```ruby
+JukeboxJob.perform_in(5 * 60, diner: "mel's", track: { title: "Later" })
+JukeboxJob.perform_at(Time.now + 300, diner: "mel's", track: { title: "Later" })
+JukeboxJob.set(at: Time.now.to_f + 300).perform_bulk([
+  [{ diner: "mel's", track: { title: "One" } }],
+  [{ diner: "mel's", track: { title: "Two" } }],
+])
+```
+
+Scheduling is **delay first, buffer second**. Each scheduled input is stored as a
+`Sidekiq::Buffered::ScheduledInput` admission job in Sidekiq's scheduled set. When it is due,
+Sidekiq queues it and a worker admits the original argument hash to the buffer, keeping the
+original input JID. A past time follows the immediate buffering path. Bulk scheduling supports
+per-input timestamps through `Sidekiq::Client.push_bulk`'s `at` array; inputs with identical
+timestamps have no guaranteed admission order. Scheduled inputs are handed to Sidekiq's
+schedule one by one rather than in bulk chunks.
+
+A new buffer's `max_wait` starts at admission, not at the scheduling call. If the group already
+has buffered inputs, the scheduled input joins that buffer without resetting its deadline. A
+full batch can run immediately after admission. Execution can be later than the requested time
+because of Sidekiq's scheduler polling, queue load, and the batch wait.
+
+Scheduling options such as queue and retry count are preserved for the admission job, and batch
+execution preserves them through the processor. Workers must consume the admission job's queue.
+Admission jobs do not invoke the original job class's retry callbacks. Until admitted,
+scheduled inputs do not appear in `Sidekiq::Buffered.stats`. If admission fails, Sidekiq
+applies the payload's retry settings; inspect waiting admissions in Sidekiq's Scheduled,
+Queues, Retry, or Dead views. Exhausted admissions never entered a batch and follow Sidekiq's
+dead-job settings, including an explicit `retry: false` or `dead: false`; the gem's
+`failed_retention` does not apply to them. A failure after buffer admission can admit an input
+again, so application processing should tolerate duplicates.
+
+## Testing your jobs
+
+In Sidekiq fake mode, inputs stay in the original job's `.jobs` array and nothing is written
+to Redis. `MyBufferedJob.drain` groups those inputs and executes full and partial batches
+deterministically; `perform_one` executes one full or partial batch. A failed drain leaves
+unattempted inputs queued, following Sidekiq's testing semantics. Inline mode executes each
+enqueue call immediately, splitting bulk inputs by group, context, and `max_size`, and does not
+wait for `max_wait`.
+
+`perform_inline(hash)` executes a one-item batch and `perform_inline([hash, ...])` executes an
+explicit batch. Leases, timing, and retries need integration tests with Sidekiq testing
+disabled and a dedicated Redis instance.
+
+## Design notes
+
+**Sidekiq extension points.** The gem uses only public hooks:
+
+- `Sidekiq::Buffered::ClientMiddleware` is prepended to the client middleware chain so it runs
+  after the rest of the chain. It validates and normalizes buffered inputs, captures the
+  metadata other client middleware attach, and diverts immediate inputs into Redis by halting
+  the push. Scheduled inputs are rewritten into `ScheduledInput` admission jobs.
+- `buffered` sets the job's `client_class` Sidekiq option to `Sidekiq::Buffered::Client`, a
+  `Sidekiq::Client` subclass that admits each `perform_bulk` chunk atomically and returns input
+  JIDs. Leave `client_class` unset on buffered jobs, or point it at a subclass of this client.
+- `buffered` also overrides `set`, `perform_inline`, `perform_sync`, and `perform_bulk` on the
+  job class so synchronous execution accepts one input hash and bypasses buffering, and
+  prepends `Sidekiq::Buffered::RetryCallbacks` to the class's singleton so Sidekiq's `wrapped`
+  callback lookup reaches the batch's retry bookkeeping.
+- `Sidekiq::Buffered::ServerMiddleware` is prepended to the server middleware chain. It claims
+  the batch, registers it with the heartbeat, and releases the lease on `Sidekiq::Shutdown`.
+  Keep it first if you customize middleware order.
+- `Sidekiq::Buffered::Processor` and `Sidekiq::Buffered::ScheduledInput` are ordinary Sidekiq
+  jobs, and Active Job support is an ordinary queue adapter.
+
+**Poller ticks.** A tick is one Redis script that returns the Redis time and everything due.
+Due buffers are then flushed in a single pipeline, up to 500 per tick, and a full page
+triggers an immediate follow-up tick. Consecutive poller errors back off exponentially to 30
+seconds; a successful tick restores the normal interval.
+
+**Recovery and queue scans.** Recovery needs the exact processor payload Sidekiq queued in
+order to check the queue with Redis `LPOS`. The gem's own dispatch captures it through the
+client middleware; processors that Sidekiq itself re-enqueues from its retry or Dead sets are
+not recorded, so an extreme backlog after such a re-push can dispatch a duplicate processor.
+Duplicates are harmless because only one claim succeeds. `LPOS` cost grows with queue length,
+and the recovery backoff limits repeated scans during a sustained backlog.
+
+**Identifiers and compatibility.** Batch IDs are opaque strings combining a random UUID
+namespace with a sequence, so independent Redis datasets never share small integer IDs.
+Low-level `Sidekiq::Client` payloads use Sidekiq's string-keyed contract; ordinary job
+argument hashes can use symbol keys. Redis Cluster is not supported; use a standalone Redis
+deployment suitable for the selected Sidekiq version.
+
+## Development
+
+Run `bundle install`, then:
+
+```bash
+REDIS_URL=redis://localhost:6379/15 bundle exec rake
+```
+
+That runs RSpec and RuboCop. The specs use a real Redis instance and clear the selected test
+database before and after each example, and stop the poller and heartbeat threads after every
+example. Use a dedicated test Redis (`REDIS_URL`) and optionally `SIDEKIQ_BUFFERED_TEST_DB`
+(default 15); never point the suite at application data. The lifecycle specs run the actual
+Sidekiq processor, server middleware, retry handler, and scheduled retry enqueuer, advancing
+the clock rather than sleeping through retry backoff.
+
+CI covers Ruby 3.2 with Sidekiq 7.0 and Active Job 7.0, Ruby 3.3 with Sidekiq 7.3 and Active
+Job 7.2, Ruby 3.4 with Sidekiq 8.0 and Active Job 8.0, and Ruby 4.0 with Sidekiq 8.1 and
+Active Job 8.1. Set `SIDEKIQ_VERSION` and `ACTIVEJOB_VERSION` when resolving a bundle to
+select another combination. The Active Job bulk-enqueue test is skipped on Active Job 7.0,
+which has no `perform_all_later` API.

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/active_job/queue_adapters/sidekiq_buffered_adapter.rb

@@ -0,0 +1,61 @@
+require "active_job"
+require "sidekiq-buffered"
+require "sidekiq/buffered/active_job_execution"
+require "sidekiq/buffered/active_job_retry"
+
+module ActiveJob
+  module QueueAdapters
+    # Active Job 7.2 introduced AbstractAdapter and asks adapters whether jobs
+    # enqueued inside a transaction should wait for its commit. Buffering after
+    # commit is the safe answer, matching Sidekiq's own adapter.
+    SIDEKIQ_BUFFERED_ADAPTER_BASE = defined?(AbstractAdapter) ? AbstractAdapter : Object
+
+    class SidekiqBufferedAdapter < SIDEKIQ_BUFFERED_ADAPTER_BASE
+      def enqueue(job)
+        return enqueue_retry(job) if job.executions.positive?
+
+        enqueue_input(job)
+      end
+
+      def enqueue_at(job, timestamp)
+        return enqueue_retry(job, timestamp) if job.executions.positive?
+
+        enqueue_input(job, timestamp)
+      end
+
+      private
+
+      def enqueue_retry(job, timestamp = nil)
+        wrapper = Sidekiq::Buffered::ActiveJobRetry.set(wrapped: job.class, queue: job.queue_name)
+        jid = if timestamp
+                wrapper.perform_at(timestamp, job.serialize)
+              else
+                wrapper.perform_async(job.serialize)
+              end
+        raise ActiveJob::EnqueueError, "Sidekiq client middleware rejected the buffered batch retry" if jid.nil?
+
+        Sidekiq::Buffered::ActiveJobExecution.record_handoff(job.job_id)
+        job.provider_job_id = jid
+      end
+
+      def enqueue_input(job, timestamp = nil)
+        unless Sidekiq::Buffered.registered?(job.class)
+          raise ArgumentError, "#{job.class} must declare buffered options"
+        end
+        unless job.arguments.size == 1 && job.arguments.first.is_a?(Hash)
+          raise ArgumentError, "#{job.class} expects a single Hash argument"
+        end
+
+        data = job.serialize
+        payload = {
+          "class" => job.class.name,
+          "queue" => job.queue_name,
+          "args" => data.fetch("arguments"),
+          "active_job" => data,
+        }
+        payload["at"] = timestamp if timestamp
+        job.provider_job_id = Sidekiq::Buffered::Client.new.push(payload)
+      end
+    end
+  end
+end