rails-hyperdrive-martian-sidekiq 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +55 -0
- data/LICENSE.txt +21 -0
- data/README.md +30 -0
- data/hyperdrive.yml +18 -0
- data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-fair-tenant.md +54 -0
- data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-transactions.md +50 -0
- data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq.md.erb +27 -0
- data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/skills/martian-sidekiq/SKILL.md.erb +610 -0
- data/lib/rails-hyperdrive-martian-sidekiq/version.rb +3 -0
- data/lib/rails-hyperdrive-martian-sidekiq.rb +4 -0
- metadata +61 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: e8cf2c9fdd45e821390b7dfa7febc832f8744beee536ccc3c2b02432687c9329
|
|
4
|
+
data.tar.gz: acc1a38e5ad757cdcafa48d675a3efeacf251ef637433db33eb64189b9488832
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: b058ae7780539d58d8576077c1bab2a877ba8ebe65a69d138756a2ead16cd57d8ab25c26cd13e3bd25768846295ac0be63ab93eca7d1870a09ec14dac434bc43
|
|
7
|
+
data.tar.gz: 69f00169e4e42233b5737b016357b37f1e37f170488b6e68a0263235e341764acff4a6fa71f0cad4cbac84b31c15067310179578061bb7e33359cf03b1aa6124
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-09-01
|
|
11
|
+
|
|
12
|
+
First release.
|
|
13
|
+
|
|
14
|
+
### Added
|
|
15
|
+
|
|
16
|
+
- **`martian-sidekiq` skill** — a model-invoked playbook for Sidekiq in Rails,
|
|
17
|
+
loaded lazily when the work touches jobs. Covers idempotency strategies and
|
|
18
|
+
a pre-merge checklist, argument rules, retry counts and custom backoff,
|
|
19
|
+
transactional enqueues, uniqueness, queue layout, concurrency vs throughput,
|
|
20
|
+
multi-tenant fairness, bulk enqueueing, rate limiting, testing, observability,
|
|
21
|
+
a red-flag catalogue, and escalation ladders for duplicate, stuck, and
|
|
22
|
+
backing-up jobs.
|
|
23
|
+
- **`martian-sidekiq` guideline** — the non-negotiable rules, loaded eagerly:
|
|
24
|
+
no `Sidekiq::Testing.inline!`, small primitive arguments, no enqueue inside
|
|
25
|
+
an open transaction, idempotent `perform`, concurrency as the risk factor,
|
|
26
|
+
smearing for rate-limited work.
|
|
27
|
+
- **`martian-sidekiq-transactions` guideline** — service-level enqueues with
|
|
28
|
+
[after_commit_everywhere](https://github.com/Envek/after_commit_everywhere),
|
|
29
|
+
including the two footguns its README calls out (never register the callbacks
|
|
30
|
+
on an AR model; `after_rollback` raises outside a transaction).
|
|
31
|
+
- **`martian-sidekiq-fair-tenant` guideline** — multi-tenant fairness with
|
|
32
|
+
[sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant): rule
|
|
33
|
+
ordering (the last matching rule wins), tenant resolution from job arguments,
|
|
34
|
+
and the silent failure when a throttled queue is missing from the Sidekiq
|
|
35
|
+
config.
|
|
36
|
+
- **`hyperdrive.yml` manifest** gating every artifact on `sidekiq >= 7.0, < 9.0`.
|
|
37
|
+
The two gem-specific guidelines carry an additional `all:` gate on their gem,
|
|
38
|
+
so an app without it never carries them in context.
|
|
39
|
+
|
|
40
|
+
### Conditional content
|
|
41
|
+
|
|
42
|
+
The skill and the base guideline are ERB templates rendered against the app's
|
|
43
|
+
resolved bundle, so the installed text matches the stack:
|
|
44
|
+
|
|
45
|
+
- **Sidekiq Pro** — `batch.jobs` pipelining for bulk enqueues.
|
|
46
|
+
- **Sidekiq Enterprise** — `Sidekiq::Limiter` for rate-limited work, and its
|
|
47
|
+
built-in `unique_for:` in place of a third-party locking gem.
|
|
48
|
+
- **rspec-sidekiq** — `have_enqueued_sidekiq_job` in the testing guidance.
|
|
49
|
+
- **after_commit_everywhere**, **sidekiq-fair_tenant**, **sidekiq-unique-jobs** —
|
|
50
|
+
where the gem is bundled the skill gives orientation and defers to the
|
|
51
|
+
guideline; where it is not, the skill proposes the gem with a `bundle add`
|
|
52
|
+
line and a worked example, and never adds a dependency silently.
|
|
53
|
+
|
|
54
|
+
[Unreleased]: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq/compare/v0.1.0...HEAD
|
|
55
|
+
[0.1.0]: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq/releases/tag/v0.1.0
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 izhanov
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# rails-hyperdrive-martian-sidekiq
|
|
2
|
+
|
|
3
|
+
Companion gem for [rails-hyperdrive](https://github.com/rails-hyperdrive/rails-hyperdrive) — Sidekiq guidance for AI coding agents working in Rails projects.
|
|
4
|
+
|
|
5
|
+
Ships four artifacts, all gated on `sidekiq >= 7.0, < 9.0` being in the app's bundle:
|
|
6
|
+
|
|
7
|
+
- **`martian-sidekiq` skill** — a lazily-loaded playbook for writing idempotent, well-retried, transactionally-safe Sidekiq jobs: idempotency strategies, retry/backoff design, uniqueness, queue layout, multi-tenant fairness, testing, escalation ladders.
|
|
8
|
+
- **`martian-sidekiq` guideline** — the non-negotiable rules (no `Sidekiq::Testing.inline!`, small primitive arguments, no enqueue inside a transaction, …), loaded eagerly.
|
|
9
|
+
- **`martian-sidekiq-transactions` guideline** — how to enqueue from a service with [after_commit_everywhere](https://github.com/Envek/after_commit_everywhere).
|
|
10
|
+
- **`martian-sidekiq-fair-tenant` guideline** — keeping one tenant from starving the queue with [sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant).
|
|
11
|
+
|
|
12
|
+
The last two are additionally gated on their gem being in the bundle, so apps without it never carry them in context.
|
|
13
|
+
|
|
14
|
+
Both are ERB templates: sections for Sidekiq Pro/Enterprise, [sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant), and rspec-sidekiq render against the app's bundle — apps that have the gem get usage guidance, apps that don't get a suggestion where it would help.
|
|
15
|
+
|
|
16
|
+
## Install
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
# Gemfile
|
|
20
|
+
group :development do
|
|
21
|
+
gem "rails-hyperdrive"
|
|
22
|
+
gem "rails-hyperdrive-martian-sidekiq"
|
|
23
|
+
end
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Then run `bin/rails hyperdrive:init`. Artifacts install to `.claude/skills/martian-sidekiq/SKILL.md` and `.claude/hyperdrive/guidelines/`.
|
|
27
|
+
|
|
28
|
+
## License
|
|
29
|
+
|
|
30
|
+
MIT — see [LICENSE.txt](LICENSE.txt).
|
data/hyperdrive.yml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Default gate for every artifact this companion ships: install only when the
|
|
2
|
+
# app's bundle has a compatible Sidekiq.
|
|
3
|
+
gems:
|
|
4
|
+
- sidekiq: ">= 7.0, < 9.0"
|
|
5
|
+
|
|
6
|
+
guidelines:
|
|
7
|
+
# A per-entry gate REPLACES the default one above, so Sidekiq must be
|
|
8
|
+
# re-stated in each: these are only useful when the app has both gems.
|
|
9
|
+
martian-sidekiq-transactions.md:
|
|
10
|
+
gems:
|
|
11
|
+
all:
|
|
12
|
+
- sidekiq: ">= 7.0, < 9.0"
|
|
13
|
+
- after_commit_everywhere
|
|
14
|
+
martian-sidekiq-fair-tenant.md:
|
|
15
|
+
gems:
|
|
16
|
+
all:
|
|
17
|
+
- sidekiq: ">= 7.0, < 9.0"
|
|
18
|
+
- sidekiq-fair_tenant
|
data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-fair-tenant.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: martian-sidekiq-fair-tenant
|
|
3
|
+
description: Keeping one tenant from starving the queue with sidekiq-fair_tenant.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
### Fair scheduling between tenants
|
|
7
|
+
|
|
8
|
+
This project has
|
|
9
|
+
[sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant): once a
|
|
10
|
+
tenant enqueues more than a threshold within a sliding window, their overflow
|
|
11
|
+
is re-routed to lower-weight throttled queues, so one noisy tenant cannot
|
|
12
|
+
starve the rest.
|
|
13
|
+
|
|
14
|
+
Declare the rules on jobs a single tenant can trigger in bulk:
|
|
15
|
+
|
|
16
|
+
```ruby
|
|
17
|
+
class SomeJob
|
|
18
|
+
include Sidekiq::Job
|
|
19
|
+
sidekiq_options queue: "default",
|
|
20
|
+
fair_tenant_queues: [
|
|
21
|
+
{ queue: "throttled_2x", threshold: 100, per: 1.hour },
|
|
22
|
+
{ queue: "throttled_4x", threshold: 10, per: 1.minute }
|
|
23
|
+
]
|
|
24
|
+
|
|
25
|
+
# Or enqueue with SomeJob.set(fair_tenant: "...").perform_async(...)
|
|
26
|
+
def self.fair_tenant(*perform_args)
|
|
27
|
+
perform_args.first # e.g. the account id
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Rules for using it:
|
|
33
|
+
|
|
34
|
+
- **Every throttled queue MUST be in the Sidekiq config with a lower weight
|
|
35
|
+
than the queue it throttles** (`[default, 4]`, `[throttled_2x, 2]`,
|
|
36
|
+
`[throttled_4x, 1]`). Re-routing into a queue no worker consumes is a
|
|
37
|
+
silent black hole — the jobs are enqueued, never drained, and nothing
|
|
38
|
+
errors. Adding a rule and its queue config is one change, never two.
|
|
39
|
+
- **When several rules match, the LAST one wins.** Order them from the
|
|
40
|
+
loosest window to the tightest, so the most aggressive throttling is last
|
|
41
|
+
and takes effect during a real burst.
|
|
42
|
+
- **Derive the tenant from the job's arguments**, via `self.fair_tenant` or an
|
|
43
|
+
explicit `.set(fair_tenant:)`. Deriving it from ambient state (`Current.*`,
|
|
44
|
+
a thread local) is unreliable: the value that matters is the one available
|
|
45
|
+
wherever the job is enqueued from, including re-enqueues outside a request.
|
|
46
|
+
- **Don't hand-roll fairness.** A queue per tenant, a `sleep` in `perform`, or
|
|
47
|
+
an "is this tenant over quota?" check inside the job all enforce fairness
|
|
48
|
+
after the queue order is already wrong — and the in-job variants burn a
|
|
49
|
+
worker slot doing it.
|
|
50
|
+
|
|
51
|
+
Fair scheduling reorders work; it does not reduce it. Throttled tenants still
|
|
52
|
+
get their jobs run, just later — so this is not a substitute for idempotency,
|
|
53
|
+
retry limits, or an actual rate limit against an external API. See the
|
|
54
|
+
`martian-sidekiq` skill for those.
|
data/lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-transactions.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: martian-sidekiq-transactions
|
|
3
|
+
description: Enqueuing jobs around DB transactions with after_commit_everywhere.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
### Enqueuing jobs around transactions
|
|
7
|
+
|
|
8
|
+
**Never enqueue inside an open transaction.** Sidekiq lives in Redis and does
|
|
9
|
+
not participate in the DB transaction: the worker can pick the job up before
|
|
10
|
+
the COMMIT (and find nothing) or after a ROLLBACK (and act on a record that
|
|
11
|
+
never existed).
|
|
12
|
+
|
|
13
|
+
This project has
|
|
14
|
+
[after_commit_everywhere](https://github.com/Envek/after_commit_everywhere),
|
|
15
|
+
so the enqueue belongs in an `after_commit` block in the service that owns the
|
|
16
|
+
use case — not in a model callback:
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
class ChargeInvoice
|
|
20
|
+
include AfterCommitEverywhere
|
|
21
|
+
|
|
22
|
+
def call(user:, amount:)
|
|
23
|
+
invoice = Invoice.create!(user:, amount:)
|
|
24
|
+
after_commit { ChargeInvoiceJob.perform_async(invoice.id) }
|
|
25
|
+
invoice
|
|
26
|
+
end
|
|
27
|
+
end
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Rules for using it:
|
|
31
|
+
|
|
32
|
+
- **Outside a transaction the block runs immediately.** That is the point:
|
|
33
|
+
the service stays correct whether or not a caller wrapped it in a
|
|
34
|
+
transaction. Don't guard it with your own `current_transaction` check.
|
|
35
|
+
- **Never call `after_commit` / `after_rollback` on an ActiveRecord model
|
|
36
|
+
instance or class.** That registers the callback on every instance,
|
|
37
|
+
including future ones. Include the module in the service object, or call
|
|
38
|
+
`AfterCommitEverywhere.after_commit { ... }` directly.
|
|
39
|
+
- **`after_rollback` raises when called outside a transaction** (unlike
|
|
40
|
+
`after_commit`). If the code path can run either way, guard it with
|
|
41
|
+
`in_transaction?`.
|
|
42
|
+
- **Prefer this over a model `after_commit` callback** when the enqueue
|
|
43
|
+
belongs to a use case rather than to the record's lifecycle. A model
|
|
44
|
+
callback fires for every flow that touches the model, including fixtures,
|
|
45
|
+
imports, and admin edits.
|
|
46
|
+
|
|
47
|
+
Enqueuing after the commit does not make the job safe on its own — it still
|
|
48
|
+
runs at-least-once, so `perform` needs its early-exit guards. See the
|
|
49
|
+
`martian-sidekiq` skill for the full transactional-enqueue options and
|
|
50
|
+
idempotency strategies.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: martian-sidekiq
|
|
3
|
+
description: Non-negotiable Sidekiq rules — testing, arguments, transactions, concurrency.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
### Sidekiq & Background Jobs
|
|
7
|
+
|
|
8
|
+
- **Never use `Sidekiq::Testing.inline!` in tests.** It runs ALL enqueued jobs,
|
|
9
|
+
including ones from factories and callbacks. Instead: `Sidekiq::Job.clear_all`,
|
|
10
|
+
run the code, then `SpecificWorker.drain`.
|
|
11
|
+
- **Keep job arguments small (< 1 KB) and primitive.** Pass IDs, refetch in
|
|
12
|
+
`perform`; never AR objects, symbols, or Time objects.
|
|
13
|
+
- **Never enqueue inside an open DB transaction.** The worker can run before
|
|
14
|
+
the COMMIT (or after a ROLLBACK). Enqueue after the transaction or from
|
|
15
|
+
`after_commit`.
|
|
16
|
+
- **Every job must be idempotent.** Retries are at-least-once; start `perform`
|
|
17
|
+
with early-exit guards.
|
|
18
|
+
- **Concurrency is the risk factor, not job count.** Millions of well-behaved
|
|
19
|
+
jobs at low concurrency are fine; 100 misbehaving jobs at high concurrency
|
|
20
|
+
can take the site down.
|
|
21
|
+
- **For rate-limited work, smear the schedule** (`perform_in` / `push_bulk`
|
|
22
|
+
with `at:`)<% if gem?("sidekiq-ent") %> and combine it with `Sidekiq::Limiter` for fine-grained
|
|
23
|
+
concurrency control — neither alone is sufficient<% else %> and back off on 429s via `sidekiq_retry_in` — neither
|
|
24
|
+
alone is sufficient<% end %>.
|
|
25
|
+
|
|
26
|
+
The `martian-sidekiq` skill has the full playbook (idempotency strategies,
|
|
27
|
+
retry design, uniqueness, queue layout, escalation ladders).
|
|
@@ -0,0 +1,610 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: martian-sidekiq
|
|
3
|
+
description: "Writing and reviewing Sidekiq jobs in Rails projects. Use when (1) creating new worker/job classes, (2) reviewing job code for retry, idempotency, or transactional bugs, (3) debugging duplicated/lost/stuck jobs, (4) tuning queue layout, concurrency, or backpressure, (5) writing or fixing tests that enqueue, drain, or assert on jobs. MUST use when working with `perform_async`, `perform_in`, `perform_bulk`, `sidekiq_options`, `retry`, `sidekiq-cron`, `sidekiq-unique-jobs`, `sidekiq-fair_tenant`, `Sidekiq::Testing`, or transactional enqueues. Covers: Sidekiq, ActiveJob-on-Sidekiq, idempotency, retries, uniqueness, batching, rate limiting, multi-tenant fairness / noisy neighbors, transactional outbox, queue layout, testing, observability."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Sidekiq for Rails
|
|
7
|
+
|
|
8
|
+
A skill for writing idempotent, well-retried, transactionally-safe Sidekiq jobs in Rails projects.
|
|
9
|
+
|
|
10
|
+
## Thinking Prompts
|
|
11
|
+
|
|
12
|
+
Before writing a job, ask yourself:
|
|
13
|
+
|
|
14
|
+
- "If this job runs twice with the same arguments, what breaks?" -- if anything breaks, the job is not idempotent; fix that first
|
|
15
|
+
- "Could the caller's DB transaction roll back AFTER the job is enqueued?" -- if yes, you have a transactional outbox bug, see [Enqueue inside a transaction](#enqueue-inside-a-transaction)
|
|
16
|
+
- "What state does this job mutate that another worker could mutate concurrently?" -- if any, you need uniqueness or row-level locking
|
|
17
|
+
- "If this raises on retry attempt 25, what does the user see and what does on-call see?" -- design the failure mode, don't discover it
|
|
18
|
+
- "Is this job doing real work, or just orchestrating other jobs?" -- orchestration belongs in a Batch / Workflow, not a long `perform`
|
|
19
|
+
|
|
20
|
+
When reviewing existing jobs, scan for the three failure modes:
|
|
21
|
+
|
|
22
|
+
- "Are arguments small, JSON-serializable, and stable?" -- AR objects, symbols, ActiveSupport::TimeWithZone in args are the #1 source of `JSON::GeneratorError` and silent argument drift
|
|
23
|
+
- "Is `sidekiq_options retry:` set explicitly?" -- a missing `retry:` directive is fine (defaults to 25), but jobs that should NEVER retry (idempotency-violating, user-visible side effects) need `retry: false` or `retry: 0`
|
|
24
|
+
- "Does `perform` start with an early-exit guard for the no-op case?" -- if the record was deleted, soft-deleted, or already processed, return early; don't fight to make a no-op idempotent later
|
|
25
|
+
|
|
26
|
+
Then annotate jobs with `[ok]`/`[warning]`/`[error]`:
|
|
27
|
+
|
|
28
|
+
- `[ok]` -- safe to retry, transactional, single responsibility
|
|
29
|
+
- `[warning]` -- works today but coupled to caller state or queue config
|
|
30
|
+
- `[error]` -- not idempotent, or enqueued inside a transaction that can roll back
|
|
31
|
+
|
|
32
|
+
If a job's `perform` body has >3 conditional branches that ALSO trigger external side effects (HTTP, email, payments), extract each branch to its own job. The top-level job becomes a dispatcher; each child job is independently retryable.
|
|
33
|
+
|
|
34
|
+
### Job kind selection
|
|
35
|
+
|
|
36
|
+
| What you're doing | Job kind | Why |
|
|
37
|
+
|-------------------|----------|-----|
|
|
38
|
+
| Send one email / one webhook | `ActionMailer#deliver_later` / a single Sidekiq worker | One side effect = one job |
|
|
39
|
+
| Process N records, each independent | `perform_bulk` enqueuing N child jobs | Each child retries independently; failure of one does not block others |
|
|
40
|
+
| Fan-out + collect (N children, then aggregate) | `Sidekiq::Batch` (Pro) or per-record idempotency keys with a final "all done?" check | Naïve fan-out without coordination drops the aggregate step on partial failure |
|
|
41
|
+
| Long-running orchestration with pauses | Workflow gem (Acidic Job, Sidekiq Workflows, or external) | A single `perform` that sleeps or polls is a footgun |
|
|
42
|
+
| Recurring background work | `sidekiq-cron` or `sidekiq-scheduler`, never `loop { sleep }` | Server restarts kill in-process loops silently |
|
|
43
|
+
| One-off scheduled work (e.g. send reminder in 7d) | `perform_in(7.days, ...)` | Survives deploys; cancel via the job id if the source record changes |
|
|
44
|
+
|
|
45
|
+
### When a side effect lives in the wrong layer
|
|
46
|
+
|
|
47
|
+
| Situation | Action |
|
|
48
|
+
|-----------|--------|
|
|
49
|
+
| `after_commit` callback enqueues a job from a model | Acceptable for low-coupling notifications; risky when the model is created in many flows -- prefer enqueuing from the service that owns the use case<% if !gem?("after_commit_everywhere") %> (`after_commit_everywhere` makes that possible without a model callback)<% end %> |
|
|
50
|
+
| Controller enqueues 5 jobs in a row | Move to a single coordinator job + `perform_bulk` so the request returns fast and retries are unified |
|
|
51
|
+
| Caller awaits the result of a job | Wrong shape -- either do the work synchronously or return a "processing" state and let the client poll |
|
|
52
|
+
| Job calls another job's `.new.perform` inline | Loses retry isolation; either inline the logic OR enqueue with `perform_async` |
|
|
53
|
+
|
|
54
|
+
## Quick Reference: Idempotency
|
|
55
|
+
|
|
56
|
+
### Make every job idempotent
|
|
57
|
+
|
|
58
|
+
A job MUST be safe to run >=2 times with the same arguments. Sidekiq retries by default, and at-least-once delivery is non-negotiable. Idempotency is not a nice-to-have; it is the entry criterion.
|
|
59
|
+
|
|
60
|
+
| Strategy | When to use | How |
|
|
61
|
+
|----------|-------------|-----|
|
|
62
|
+
| Early-exit guard | Side effect is "set X to value Y" | `return if record.already_in_state?(...)` at the top of `perform` |
|
|
63
|
+
| Unique constraint | Side effect is "insert row Z" | Add a DB unique index on the natural key; rescue `ActiveRecord::RecordNotUnique` and treat as success |
|
|
64
|
+
| Idempotency key | Side effect is an external HTTP call (Stripe, Sendgrid, etc.) | Pass a deterministic key (job args hash, source record id + version) so the remote API dedupes |
|
|
65
|
+
| Optimistic locking | Side effect is "update X based on its current value" | Use `lock_version` and rescue `StaleObjectError` -- the retry will reload and re-check |
|
|
66
|
+
| Application-level dedupe | Side effect is "enqueue a notification" | Store the enqueue intent (`UNIQUE (user_id, notification_kind, day)`) before enqueuing |
|
|
67
|
+
|
|
68
|
+
### Pre-merge checklist: is this job idempotent?
|
|
69
|
+
|
|
70
|
+
Before merging a new job, verify ALL of these:
|
|
71
|
+
- [ ] Running `perform(*same_args)` twice produces the same end state
|
|
72
|
+
- [ ] No external HTTP side effect runs without an idempotency key OR a pre-check
|
|
73
|
+
- [ ] No "find or create" pattern uses `first_or_create` without a unique index backing it
|
|
74
|
+
- [ ] No model uses `update` on a derived counter (`count += 1`) -- use atomic `update_counters` or a SQL `UPDATE ... SET col = col + 1`
|
|
75
|
+
- [ ] If the job sends an email, the recipient is the SAME record the job was enqueued for (not a "current user" looked up at run time)
|
|
76
|
+
- [ ] Arguments contain IDs and primitives only -- never AR instances, Symbols, or `Time.current`
|
|
77
|
+
|
|
78
|
+
### Arguments: keep them small and serializable
|
|
79
|
+
|
|
80
|
+
Aim for well under 1 KB per job: arguments live in Redis for the job's whole
|
|
81
|
+
lifetime (queue + retries + scheduled set), so large arguments consume Redis
|
|
82
|
+
memory across millions of jobs and can cause incidents. Pass IDs and look the
|
|
83
|
+
data up in `perform`; never serialize objects as arguments.
|
|
84
|
+
|
|
85
|
+
| Pattern | Why it breaks | Fix |
|
|
86
|
+
|---------|---------------|-----|
|
|
87
|
+
| `perform_async(user)` (an AR object) | Sidekiq JSON-serializes the object's full attributes -- snapshot drifts from DB by the time the worker runs; also wastes Redis | Pass `user.id`, refetch in `perform` |
|
|
88
|
+
| `perform_async(:some_symbol)` | Symbols round-trip as strings -- `case kind` branches silently fall through | Pass strings explicitly, or freeze a constants module the worker maps |
|
|
89
|
+
| `perform_async(Time.current)` | Time serializes to a String with the local zone; UTC vs zoned drift bites months later | Pass an integer epoch or an ISO8601 String the worker re-parses |
|
|
90
|
+
| `perform_async(some_hash_with_symbol_keys)` | Keys come back as strings; `args[:user_id]` returns nil | Use string keys, or convert with `args.with_indifferent_access` at the top of `perform` |
|
|
91
|
+
| `perform_async(large_payload)` (>1 MB) | Bloats Redis, slows fetch, may exceed Redis value limit | Persist the payload (DB / S3) and pass the id |
|
|
92
|
+
|
|
93
|
+
### Refetch pattern
|
|
94
|
+
|
|
95
|
+
```ruby
|
|
96
|
+
class ChargeInvoiceJob
|
|
97
|
+
include Sidekiq::Job
|
|
98
|
+
sidekiq_options queue: :payments, retry: 10
|
|
99
|
+
|
|
100
|
+
def perform(invoice_id)
|
|
101
|
+
invoice = Invoice.find_by(id: invoice_id)
|
|
102
|
+
return unless invoice # deleted between enqueue and run
|
|
103
|
+
return if invoice.charged? || invoice.void? # idempotency guard
|
|
104
|
+
return unless invoice.ready_to_charge? # state-machine precondition
|
|
105
|
+
|
|
106
|
+
invoice.charge!(idempotency_key: "invoice-#{invoice.id}-v#{invoice.lock_version}")
|
|
107
|
+
end
|
|
108
|
+
end
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Three guards in a row at the top of `perform` is the canonical Sidekiq shape. If you find yourself writing a fourth, the precondition belongs in a state-machine method on the model.
|
|
112
|
+
|
|
113
|
+
## Quick Reference: Retries
|
|
114
|
+
|
|
115
|
+
### Default retry behavior
|
|
116
|
+
|
|
117
|
+
- `sidekiq_options retry: 25` (the default) -> ~21 days of exponential backoff with jitter; after that the job lands in the Dead Set for 6 months.
|
|
118
|
+
- `sidekiq_options retry: false` (or `retry: 0`) -> failures go straight to the Dead Set (or are discarded with `dead: false`).
|
|
119
|
+
- `sidekiq_options retry: N` -> N retries, then Dead Set.
|
|
120
|
+
|
|
121
|
+
Pick a retry count that matches the worst tolerable latency. A payment confirmation that becomes useless after 10 minutes should not retry for 21 days.
|
|
122
|
+
|
|
123
|
+
### Custom backoff
|
|
124
|
+
|
|
125
|
+
```ruby
|
|
126
|
+
class ExternalApiJob
|
|
127
|
+
include Sidekiq::Job
|
|
128
|
+
sidekiq_options retry: 8
|
|
129
|
+
|
|
130
|
+
sidekiq_retry_in do |count, exception|
|
|
131
|
+
case exception
|
|
132
|
+
when RateLimited then count * 60 # linear backoff for 429s
|
|
133
|
+
when Net::OpenTimeout then 10 * (count ** 2) # quadratic for transient network
|
|
134
|
+
else :kill # any other error -> Dead Set immediately
|
|
135
|
+
end
|
|
136
|
+
end
|
|
137
|
+
|
|
138
|
+
def perform(...)
|
|
139
|
+
# ...
|
|
140
|
+
end
|
|
141
|
+
end
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
### Don't retry on the caller's behalf
|
|
145
|
+
|
|
146
|
+
| Pattern | Why | Fix |
|
|
147
|
+
|---------|-----|-----|
|
|
148
|
+
| `begin; do_thing; rescue; retry; end` inside `perform` | Blocks the worker thread; one bad call holds a slot for minutes | Re-raise; let Sidekiq retry the job with backoff |
|
|
149
|
+
| `sleep 5; retry` for a 429 | Burns a thread on a sleep; doesn't survive worker restart | Raise a `RateLimited` exception; configure `sidekiq_retry_in` to back off |
|
|
150
|
+
| `loop { break if done }` polling for an external state | Same thread cost; deploys orphan the loop | Re-enqueue the job with `perform_in(30.seconds, ...)` |
|
|
151
|
+
| Retrying inside a transaction | The transaction is held open across retries; locks pile up | Move the retry boundary outside the transaction (Sidekiq's retry, not yours) |
|
|
152
|
+
|
|
153
|
+
### Stop retrying when retry is pointless
|
|
154
|
+
|
|
155
|
+
Some exceptions should NOT retry:
|
|
156
|
+
|
|
157
|
+
- `ActiveRecord::RecordNotFound` for the primary arg -- the record was deleted; retrying 24 more times changes nothing
|
|
158
|
+
- `ArgumentError` / `TypeError` / `NoMethodError` -- a code bug, the next attempt will fail the same way
|
|
159
|
+
- Authorization failures (`Pundit::NotAuthorizedError`) -- state hasn't changed; retry won't fix it
|
|
160
|
+
- `JSON::ParserError` on a payload arg -- the payload is malformed; retrying re-parses the same bytes
|
|
161
|
+
|
|
162
|
+
```ruby
|
|
163
|
+
def perform(record_id)
|
|
164
|
+
record = Record.find(record_id)
|
|
165
|
+
# ...
|
|
166
|
+
rescue ActiveRecord::RecordNotFound
|
|
167
|
+
# already deleted -- treat as success, don't raise
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
# OR, if a class of error must never retry:
|
|
171
|
+
sidekiq_options retry: 5
|
|
172
|
+
sidekiq_retry_in do |count, exception|
|
|
173
|
+
:kill if exception.is_a?(Pundit::NotAuthorizedError)
|
|
174
|
+
end
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Rate-limited external work: smear the schedule, then throttle
|
|
178
|
+
|
|
179
|
+
For work bounded by an external rate limit (an API allowing N calls/minute),
|
|
180
|
+
one mechanism is not enough — combine two:
|
|
181
|
+
|
|
182
|
+
1. **Smearing** sets the coarse schedule: spread the enqueues over time with
|
|
183
|
+
`perform_in(i * interval, ...)` or `push_bulk` with `at:` timestamps instead
|
|
184
|
+
of dumping 10k jobs into the queue at once.
|
|
185
|
+
<%- if gem?("sidekiq-ent") -%>
|
|
186
|
+
2. **`Sidekiq::Limiter`** (Enterprise) handles fine-grained concurrency: wrap
|
|
187
|
+
the API call in a `rate` or `concurrent` limiter so bursts and retries
|
|
188
|
+
still respect the limit. Smearing alone drifts under retries; the limiter
|
|
189
|
+
alone turns the queue into a thundering herd of rescheduled jobs.
|
|
190
|
+
<%- else -%>
|
|
191
|
+
2. **Backoff on 429** handles the fine grain: raise a dedicated `RateLimited`
|
|
192
|
+
error when the API says no, and give it a linear `sidekiq_retry_in` backoff
|
|
193
|
+
(see Custom backoff above). Smearing alone drifts under retries; backoff
|
|
194
|
+
alone turns the queue into a thundering herd.
|
|
195
|
+
<%- end -%>
|
|
196
|
+
|
|
197
|
+
### Bulk enqueueing
|
|
198
|
+
|
|
199
|
+
`perform_async` in a loop pays one Redis round-trip per job. For N > ~100,
|
|
200
|
+
use `perform_bulk` / `Sidekiq::Client.push_bulk` — one round-trip per 1,000
|
|
201
|
+
jobs, and each child job still retries independently.
|
|
202
|
+
<%- if gem?("sidekiq-pro") -%>
|
|
203
|
+
|
|
204
|
+
With Sidekiq Pro, enqueue inside a `batch.jobs` block: it collects all Redis
|
|
205
|
+
commands and executes them via pipelining, dramatically faster than individual
|
|
206
|
+
`perform_async` calls — and you get completion callbacks for free.
|
|
207
|
+
<%- end -%>
|
|
208
|
+
|
|
209
|
+
### `sidekiq_retries_exhausted`
|
|
210
|
+
|
|
211
|
+
Use it to record a structured failure when retries are gone -- do NOT use it to swallow the failure silently.
|
|
212
|
+
|
|
213
|
+
```ruby
|
|
214
|
+
sidekiq_retries_exhausted do |msg, exception|
|
|
215
|
+
Bugsnag.notify(exception) { |r| r.add_metadata(:sidekiq, msg) }
|
|
216
|
+
# OPTIONAL: mark the source record as needing manual intervention
|
|
217
|
+
Invoice.find_by(id: msg["args"].first)&.mark_failed!(reason: exception.message)
|
|
218
|
+
end
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
## Critical Rules
|
|
222
|
+
|
|
223
|
+
### Enqueue inside a transaction
|
|
224
|
+
|
|
225
|
+
If your code looks like this, you have a bug:
|
|
226
|
+
|
|
227
|
+
```ruby
|
|
228
|
+
# WRONG -- transactional outbox violation
|
|
229
|
+
ActiveRecord::Base.transaction do
|
|
230
|
+
invoice = Invoice.create!(user:, amount:)
|
|
231
|
+
ChargeInvoiceJob.perform_async(invoice.id) # may run BEFORE the COMMIT
|
|
232
|
+
end
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
Sidekiq runs in Redis; your DB transaction has not yet committed. The worker picks up the job within milliseconds and finds `Invoice.find_by(id: ...)` returns `nil`. On retry the record exists; on bad luck the retry happens after the outer transaction rolls back, charging a phantom invoice.
|
|
236
|
+
|
|
237
|
+
Pick one of four fixes, in increasing order of machinery:
|
|
238
|
+
|
|
239
|
+
1. **Move the enqueue out of the transaction (simplest):**
|
|
240
|
+
```ruby
|
|
241
|
+
invoice = ActiveRecord::Base.transaction { Invoice.create!(user:, amount:) }
|
|
242
|
+
ChargeInvoiceJob.perform_async(invoice.id)
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
2. **Defer the enqueue to `after_commit`:**
|
|
246
|
+
```ruby
|
|
247
|
+
class Invoice < ApplicationRecord
|
|
248
|
+
after_commit on: :create do
|
|
249
|
+
ChargeInvoiceJob.perform_async(id)
|
|
250
|
+
end
|
|
251
|
+
end
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
3. **Defer the enqueue to `after_commit` from the service that owns the use
|
|
255
|
+
case** — the shape you want when the enqueue is part of a use case rather
|
|
256
|
+
than a property of the model, and the one that survives a caller wrapping
|
|
257
|
+
your service in an outer transaction:
|
|
258
|
+
<%- if gem?("after_commit_everywhere") -%>
|
|
259
|
+
|
|
260
|
+
This project has
|
|
261
|
+
[after_commit_everywhere](https://github.com/Envek/after_commit_everywhere):
|
|
262
|
+
`include AfterCommitEverywhere` in the service and wrap the enqueue in
|
|
263
|
+
`after_commit { ... }`. Outside a transaction the block runs immediately,
|
|
264
|
+
so the service stays correct whether or not a caller wrapped it.
|
|
265
|
+
|
|
266
|
+
The worked example and the two footguns (never register the callbacks on
|
|
267
|
+
an AR model; `after_rollback` raises outside a transaction) are in the
|
|
268
|
+
`martian-sidekiq-transactions` guideline, already in your context — follow
|
|
269
|
+
it rather than re-deriving the shape here.
|
|
270
|
+
<%- else -%>
|
|
271
|
+
|
|
272
|
+
Rails only offers `after_commit` as a model callback, so the plain version
|
|
273
|
+
of this means pushing a use-case concern down into the model. When that
|
|
274
|
+
tension shows up, propose
|
|
275
|
+
[after_commit_everywhere](https://github.com/Envek/after_commit_everywhere)
|
|
276
|
+
to the user — it makes `after_commit` / `after_rollback` callable from
|
|
277
|
+
service objects, and running outside a transaction simply executes the
|
|
278
|
+
block immediately, so the service stays correct whether or not a caller
|
|
279
|
+
wrapped it. Propose the dependency; don't add it silently. If the user
|
|
280
|
+
agrees:
|
|
281
|
+
|
|
282
|
+
```sh
|
|
283
|
+
bundle add after_commit_everywhere
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
then move the enqueue into the service:
|
|
287
|
+
|
|
288
|
+
```ruby
|
|
289
|
+
class ChargeInvoice
|
|
290
|
+
include AfterCommitEverywhere
|
|
291
|
+
|
|
292
|
+
def call(user:, amount:)
|
|
293
|
+
invoice = Invoice.create!(user:, amount:)
|
|
294
|
+
after_commit { ChargeInvoiceJob.perform_async(invoice.id) }
|
|
295
|
+
invoice
|
|
296
|
+
end
|
|
297
|
+
end
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
Don't call the methods on an AR model directly — that registers the
|
|
301
|
+
callback on every instance, including future ones.
|
|
302
|
+
<%- end -%>
|
|
303
|
+
|
|
304
|
+
4. **Use a transactional outbox** (e.g. [sidekiq-grouping](https://github.com/gzigzigzeo/sidekiq-grouping), `acidic_job`, or a hand-rolled `outbox_events` table polled by a worker). Required when you need exactly-once-ish semantics across the DB and Redis. Most apps don't need this; the first three suffice.
|
|
305
|
+
|
|
306
|
+
ActiveJob-on-Sidekiq has the same trap. ActiveJob 7.2+ adds `enqueue_after_transaction_commit`, which can be set per-queue-adapter and defers every enqueue automatically -- enable it globally if your team prefers the guarantee implicit; the explicit fixes above keep the lifecycle visible at the call site.
|
|
307
|
+
|
|
308
|
+
### Uniqueness: prevent duplicate enqueues, not just duplicate work
|
|
309
|
+
|
|
310
|
+
Idempotency in `perform` makes a job safe to RUN twice. Uniqueness makes it impossible to ENQUEUE the same work twice. The two are independent and complementary.
|
|
311
|
+
|
|
312
|
+
Use a uniqueness lock when:
|
|
313
|
+
|
|
314
|
+
- The job is expensive (re-running it wastes minutes, not milliseconds)
|
|
315
|
+
- The user can trigger enqueues faster than the worker can drain them (a UI button, a webhook flood)
|
|
316
|
+
- The work has a natural "subject" (user, order, report) and concurrent runs on the same subject corrupt state
|
|
317
|
+
|
|
318
|
+
Reach for the DB first — it is the option with the fewest moving parts:
|
|
319
|
+
|
|
320
|
+
- A DB unique index on the source intent, preferred whenever the lock must outlive Redis (e.g. an `outbox_events(unique_key)` index). A lock that vanishes with a Redis flush was never a guarantee.
|
|
321
|
+
- An application-level `INSERT ... ON CONFLICT DO NOTHING` on a `job_intents` table before enqueueing.
|
|
322
|
+
|
|
323
|
+
A unique DB index plus an early-exit guard in `perform` beats a Redis-based lock for most cases: it survives Redis loss, it is visible in schema, and it needs no extra dependency.
|
|
324
|
+
<%- if gem?("sidekiq-unique-jobs") -%>
|
|
325
|
+
|
|
326
|
+
When the lock genuinely belongs in Redis, this project already has
|
|
327
|
+
[`sidekiq-unique-jobs`](https://github.com/mhenrixon/sidekiq-unique-jobs).
|
|
328
|
+
Use `lock: :until_executed` for "no duplicate while pending", and
|
|
329
|
+
`lock: :until_and_while_executing` for "no duplicate while pending OR
|
|
330
|
+
running". Always pair a lock with `lock_timeout` / `lock_ttl` — an
|
|
331
|
+
orphaned lock (worker killed mid-job) otherwise blocks that subject
|
|
332
|
+
indefinitely.
|
|
333
|
+
<%- if gem?("sidekiq-ent") -%>
|
|
334
|
+
|
|
335
|
+
Note that Sidekiq Enterprise (also in this bundle) ships uniqueness built in
|
|
336
|
+
via `sidekiq_options unique_for: 10.minutes`. For new jobs prefer it over the
|
|
337
|
+
gem — one less locking layer to reason about.
|
|
338
|
+
<%- end -%>
|
|
339
|
+
<%- elsif gem?("sidekiq-ent") -%>
|
|
340
|
+
|
|
341
|
+
This project has Sidekiq Enterprise, which ships uniqueness built in — use
|
|
342
|
+
`sidekiq_options unique_for: 10.minutes` rather than adding a third-party
|
|
343
|
+
locking gem on top of it.
|
|
344
|
+
<%- else -%>
|
|
345
|
+
|
|
346
|
+
When neither DB option fits and the lock genuinely belongs in Redis, propose
|
|
347
|
+
[`sidekiq-unique-jobs`](https://github.com/mhenrixon/sidekiq-unique-jobs) to
|
|
348
|
+
the user — it is the de-facto standard for uniqueness locks on OSS Sidekiq
|
|
349
|
+
(uniqueness itself is an Enterprise feature). Propose the dependency; don't
|
|
350
|
+
add it silently. If the user agrees:
|
|
351
|
+
|
|
352
|
+
```sh
|
|
353
|
+
bundle add sidekiq-unique-jobs
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
The stable 8.x line supports Sidekiq 7 and 8, so this resolves without a
|
|
357
|
+
version pin. Its README documents Sidekiq >= 8 — that is the unreleased 9.0
|
|
358
|
+
line; don't pin to a prerelease on its account.
|
|
359
|
+
|
|
360
|
+
then declare the lock on the job, always with a TTL so a worker killed
|
|
361
|
+
mid-job cannot block its subject forever:
|
|
362
|
+
|
|
363
|
+
```ruby
|
|
364
|
+
sidekiq_options lock: :until_executed, lock_ttl: 30.minutes
|
|
365
|
+
```
|
|
366
|
+
<%- end -%>
|
|
367
|
+
|
|
368
|
+
### Queue layout: latency-based, not feature-based
|
|
369
|
+
|
|
370
|
+
Group queues by SLA, not by domain. Three queues -- `critical`, `default`, `low` -- handles 95% of apps; add a fourth (e.g. `payments`, `mailers`) only when one queue's latency profile is incompatible with the rest.
|
|
371
|
+
|
|
372
|
+
| Queue | Typical work | Concurrency strategy |
|
|
373
|
+
|-------|--------------|----------------------|
|
|
374
|
+
| `critical` | Auth, billing confirmations, security notifications | Always drained first; size with headroom |
|
|
375
|
+
| `default` | Most async work | Bulk of capacity |
|
|
376
|
+
| `low` | Reports, exports, analytics, cache warming | Throttled or moved to a separate process |
|
|
377
|
+
| (named) | Strictly different SLA (e.g. webhooks with strict ordering) | Isolated workers |
|
|
378
|
+
|
|
379
|
+
Bad signs: queues named `users_queue`, `posts_queue`, `model_xyz_queue` -- they fragment capacity without a latency justification.
|
|
380
|
+
|
|
381
|
+
### Fair scheduling between tenants (noisy neighbors)
|
|
382
|
+
|
|
383
|
+
In a multi-tenant app, one tenant bulk-triggering thousands of jobs starves
|
|
384
|
+
everyone else on the same queue: capacity is fine, ordering is not. The fix is
|
|
385
|
+
fairness, not more workers — once a tenant exceeds a threshold, route their
|
|
386
|
+
overflow to lower-weight "throttled" queues so other tenants' jobs keep
|
|
387
|
+
flowing. Don't hand-roll this with one-queue-per-tenant or in-job sleeps.
|
|
388
|
+
<%- if gem?("sidekiq-fair_tenant") -%>
|
|
389
|
+
|
|
390
|
+
This project uses [sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant):
|
|
391
|
+
declare `fair_tenant_queues:` rules on the jobs one tenant can trigger in
|
|
392
|
+
bulk, and give each rule a throttled queue that exists in the Sidekiq config
|
|
393
|
+
with a lower weight.
|
|
394
|
+
|
|
395
|
+
The rule syntax, tenant resolution, ordering semantics (the LAST matching rule
|
|
396
|
+
wins), and the silent-black-hole failure when a throttled queue is missing
|
|
397
|
+
from the config are all in the `martian-sidekiq-fair-tenant` guideline,
|
|
398
|
+
already in your context — follow it rather than re-deriving the shape here.
|
|
399
|
+
<%- else -%>
|
|
400
|
+
|
|
401
|
+
This project does not use
|
|
402
|
+
[sidekiq-fair_tenant](https://github.com/Envek/sidekiq-fair_tenant). It is the
|
|
403
|
+
ready-made answer here and needs no Sidekiq Pro/Enterprise: enqueues are
|
|
404
|
+
counted per tenant in a sliding Redis window, and a tenant over its threshold
|
|
405
|
+
gets re-routed to lower-weight throttled queues automatically.
|
|
406
|
+
|
|
407
|
+
**Propose it to the user** whenever a multi-tenant app hits noisy-neighbor
|
|
408
|
+
starvation, or when you catch someone building the hand-rolled version
|
|
409
|
+
(a queue per tenant, an in-job `sleep`, a "is this tenant over quota?" check
|
|
410
|
+
inside `perform`). Propose the dependency — never add it silently. If the user
|
|
411
|
+
agrees:
|
|
412
|
+
|
|
413
|
+
```sh
|
|
414
|
+
bundle add sidekiq-fair_tenant
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
then declare re-routing rules on the heavy jobs and tell the gem who the
|
|
418
|
+
tenant is:
|
|
419
|
+
|
|
420
|
+
```ruby
|
|
421
|
+
class SomeJob
|
|
422
|
+
include Sidekiq::Job
|
|
423
|
+
sidekiq_options queue: "default",
|
|
424
|
+
fair_tenant_queues: [
|
|
425
|
+
{ queue: "throttled_2x", threshold: 100, per: 1.hour },
|
|
426
|
+
{ queue: "throttled_4x", threshold: 10, per: 1.minute }
|
|
427
|
+
]
|
|
428
|
+
|
|
429
|
+
# Or enqueue with SomeJob.set(fair_tenant: "...").perform_async(...)
|
|
430
|
+
def self.fair_tenant(*perform_args)
|
|
431
|
+
perform_args.first # e.g. the account id
|
|
432
|
+
end
|
|
433
|
+
end
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
and list the throttled queues with LOWER weights in the Sidekiq config —
|
|
437
|
+
without this the re-routed jobs land in a queue nobody drains:
|
|
438
|
+
|
|
439
|
+
```yaml
|
|
440
|
+
:queues:
|
|
441
|
+
- [default, 4]
|
|
442
|
+
- [throttled_2x, 2]
|
|
443
|
+
- [throttled_4x, 1]
|
|
444
|
+
```
|
|
445
|
+
<%- end -%>
|
|
446
|
+
|
|
447
|
+
### Concurrency vs throughput
|
|
448
|
+
|
|
449
|
+
Concurrency is the key risk factor, not total job count: 10 million
|
|
450
|
+
well-behaved jobs draining at low concurrency is fine, while 100 misbehaving
|
|
451
|
+
jobs at high concurrency can take the site down (DB pool exhaustion, external
|
|
452
|
+
API hammering, lock contention). Size concurrency for the worst job on the
|
|
453
|
+
queue, not the average one.
|
|
454
|
+
|
|
455
|
+
A worker process has `concurrency: N` threads sharing one Ruby VM. Each thread holds:
|
|
456
|
+
|
|
457
|
+
- A DB connection from the pool (`pool: N` in `database.yml`, or `pool >= N + a few`)
|
|
458
|
+
- A Redis connection
|
|
459
|
+
- Whatever the job grabs (HTTP client, S3 client, etc.)
|
|
460
|
+
|
|
461
|
+
Common mistakes:
|
|
462
|
+
|
|
463
|
+
- `concurrency: 25` with `pool: 5` -- 20 threads stall on connection checkout, surfaces as random `ConnectionTimeoutError`
|
|
464
|
+
- `concurrency: 25` for a CPU-bound job that pegs one core -- the other 24 threads compete for the GVL and you lose ~70% throughput vs `concurrency: 5` x 5 processes
|
|
465
|
+
- Mixing IO-bound and CPU-bound jobs on the same process at high concurrency -- the CPU-bound ones starve the IO-bound ones
|
|
466
|
+
|
|
467
|
+
Match concurrency to the workload, not the marketing number. For IO-bound jobs, 5-10 is usually enough; the bottleneck is downstream (DB, external API), not Ruby.
|
|
468
|
+
|
|
469
|
+
### Observability is part of the job, not optional
|
|
470
|
+
|
|
471
|
+
Every job should emit, at minimum:
|
|
472
|
+
|
|
473
|
+
- `start` / `success` / `failure` events with the job class, jid, args summary (never full args -- args can contain PII)
|
|
474
|
+
- Latency from `enqueued_at` -> `started_at` (queue latency) and `started_at` -> `finished_at` (work latency)
|
|
475
|
+
- The retry count if > 0
|
|
476
|
+
|
|
477
|
+
Use [`sidekiq` middleware](https://github.com/sidekiq/sidekiq/wiki/Middleware) for cross-cutting instrumentation; don't sprinkle `Rails.logger.info` inside each `perform`.
|
|
478
|
+
|
|
479
|
+
For Rails 7.1+, prefer the `ActiveSupport::Notifications` instrumentation built into Sidekiq's ActiveJob adapter -- subscribe in one place.
|
|
480
|
+
|
|
481
|
+
## Testing Jobs
|
|
482
|
+
|
|
483
|
+
Default to `Sidekiq::Testing.fake!` (the out-of-the-box mode): enqueued jobs
|
|
484
|
+
accumulate in in-memory arrays; assert on them and drain explicitly.
|
|
485
|
+
|
|
486
|
+
**Never use `Sidekiq::Testing.inline!` in tests.** It runs ALL enqueued jobs —
|
|
487
|
+
including ones enqueued from factories and model callbacks, potentially
|
|
488
|
+
hundreds of unrelated jobs per example. Drain only the workers the test needs:
|
|
489
|
+
|
|
490
|
+
```ruby
|
|
491
|
+
# Bad
|
|
492
|
+
Sidekiq::Testing.inline! { do_something }
|
|
493
|
+
|
|
494
|
+
# Good
|
|
495
|
+
Sidekiq::Job.clear_all # discard jobs enqueued by factories / setup
|
|
496
|
+
do_something
|
|
497
|
+
SomeSpecificWorker.drain # run only the jobs this test is about
|
|
498
|
+
```
|
|
499
|
+
|
|
500
|
+
Split coverage by layer:
|
|
501
|
+
|
|
502
|
+
- **Job logic** — call `described_class.new.perform(args)` directly; no
|
|
503
|
+
Sidekiq machinery involved, no drain needed.
|
|
504
|
+
- **"The caller enqueues the job"** — assert on the enqueue in the caller's
|
|
505
|
+
test<% if gem?("rspec-sidekiq") %> with `have_enqueued_sidekiq_job` (rspec-sidekiq)<% else %> (`expect { call }.to change(SomeWorker.jobs, :size).by(1)`)<% end %>;
|
|
506
|
+
don't execute the job there.
|
|
507
|
+
- **Scheduling** — assert on `SomeWorker.jobs.last["at"]`, don't sleep.
|
|
508
|
+
|
|
509
|
+
## Red Flags
|
|
510
|
+
|
|
511
|
+
### Idempotency & state
|
|
512
|
+
|
|
513
|
+
| Pattern | Why | Fix |
|
|
514
|
+
|---------|-----|-----|
|
|
515
|
+
| `perform` body with no early-exit guard | Retry repeats the side effect | Add guards: `return if already_done?`, `return unless record` |
|
|
516
|
+
| `User.find(args[0]).update!(counter: user.counter + 1)` | Lost update under retry / concurrency | `User.where(id: ...).update_all('counter = counter + 1')` or `update_counters` |
|
|
517
|
+
| `record.update_columns(state: "done")` to skip a callback that also enqueues a job | Hides the side effect; downstream consumers stop firing | Fix the callback, not the bypass |
|
|
518
|
+
| `find_or_create_by` without a unique index | Race window between `find` and `create` -- both retries succeed at `create`, one raises | Add the unique index AND rescue `RecordNotUnique` |
|
|
519
|
+
| `Stripe::Charge.create(amount:, customer:)` with no `idempotency_key` | Retried job double-charges | Always pass `idempotency_key: "<deterministic-string>"` to external APIs that support it |
|
|
520
|
+
|
|
521
|
+
### Transactions & enqueue ordering
|
|
522
|
+
|
|
523
|
+
| Pattern | Why | Fix |
|
|
524
|
+
|---------|-----|-----|
|
|
525
|
+
| `Job.perform_async(...)` inside an open transaction | Worker may run before COMMIT | Move outside, or use `after_commit`, or enable `enqueue_after_transaction_commit` |
|
|
526
|
+
| `perform_async(...)` inside a `with_lock` block | Same as above plus holds a row lock longer | Capture the work to do, exit the block, then enqueue |
|
|
527
|
+
| Multiple `perform_async` calls in a controller action | Increases latency, fragments retries | Enqueue a single coordinator job that does the fan-out |
|
|
528
|
+
|
|
529
|
+
### Arguments
|
|
530
|
+
|
|
531
|
+
| Pattern | Why | Fix |
|
|
532
|
+
|---------|-----|-----|
|
|
533
|
+
| `perform_async(active_record_instance)` | Sidekiq dumps the attributes hash; PII leaks to Redis; data goes stale | Pass `id`, refetch in `perform` |
|
|
534
|
+
| `perform_async(:some_symbol)` | Round-trips as String; `case x when :sym` falls through | Pass strings, or use a constants module |
|
|
535
|
+
| `perform_async(Time.now)` | Zone drift, format drift | Pass epoch integer or ISO8601 String |
|
|
536
|
+
| `perform_async({user_id: 1})` then `args[:user_id]` | Key comes back as `"user_id"`; result is `nil` | Use string keys, or `args.with_indifferent_access` |
|
|
537
|
+
| `perform_async(big_array_of_records)` | Redis bloat; >1 MB job is a smell | Persist + pass id, or `perform_bulk` with per-element jobs |
|
|
538
|
+
|
|
539
|
+
### Retries
|
|
540
|
+
|
|
541
|
+
| Pattern | Why | Fix |
|
|
542
|
+
|---------|-----|-----|
|
|
543
|
+
| `sidekiq_options retry: false` on a job that DOES fail transiently | One blip lands the job in the Dead Set forever | Use a small `retry: N` with custom backoff |
|
|
544
|
+
| Custom `rescue => e; retry end` inside `perform` | Bypasses Sidekiq's backoff and observability | Re-raise; let Sidekiq handle it |
|
|
545
|
+
| 25-retry default on a time-sensitive job | Retries for 21 days; the work has been stale for 20 of them | Set `retry: 3` and a custom backoff matching the SLA |
|
|
546
|
+
| Retrying authorization / "record gone" errors | Wastes retries; never recovers | Rescue and return; or configure `sidekiq_retry_in` to `:kill` |
|
|
547
|
+
|
|
548
|
+
### Concurrency & queue layout
|
|
549
|
+
|
|
550
|
+
| Pattern | Why | Fix |
|
|
551
|
+
|---------|-----|-----|
|
|
552
|
+
| One queue per model (`posts_queue`, `users_queue`, ...) | Fragments capacity; no SLA differentiation | Collapse to `critical` / `default` / `low` |
|
|
553
|
+
| One queue per tenant / account | Unbounded queue count; weights become unmanageable | Keep shared queues + threshold-based re-routing<% if !gem?("sidekiq-fair_tenant") %> (propose `sidekiq-fair_tenant`)<% end %> |
|
|
554
|
+
| A quota check inside `perform` that re-enqueues or sleeps when the tenant is over | Burns a worker slot to enforce fairness; the queue order is already wrong by then | Enforce fairness at enqueue time by re-routing to a throttled queue |
|
|
555
|
+
| `concurrency: 25` everywhere | DB pool exhaustion; GVL contention for CPU work | Match concurrency to workload kind |
|
|
556
|
+
| Long-running jobs on the same queue as short jobs | Head-of-line blocking | Move long jobs to a dedicated queue with `-q long,1` weight |
|
|
557
|
+
| `loop { ...; sleep n }` inside a job | Deploys orphan the loop; thread held forever | `sidekiq-cron` or `perform_in` |
|
|
558
|
+
|
|
559
|
+
### Job kind
|
|
560
|
+
|
|
561
|
+
| Pattern | Why | Fix |
|
|
562
|
+
|---------|-----|-----|
|
|
563
|
+
| One job loops over 10k records calling external APIs | One failure restarts the whole 10k loop | `perform_bulk` enqueues N child jobs |
|
|
564
|
+
| Job calls `OtherJob.new.perform(...)` inline | Loses retry isolation, observability, and queue routing | `OtherJob.perform_async(...)` |
|
|
565
|
+
| `Job.perform_in(0, ...)` to "defer slightly" | The work runs immediately; this just wastes a Redis round-trip | Either inline the work or `perform_async` |
|
|
566
|
+
| `Sidekiq::Worker` mixed in (old) instead of `Sidekiq::Job` (7.x+) | Old name, identical semantics | Rename to `Sidekiq::Job` |
|
|
567
|
+
|
|
568
|
+
### Testing
|
|
569
|
+
|
|
570
|
+
| Pattern | Why | Fix |
|
|
571
|
+
|---------|-----|-----|
|
|
572
|
+
| `Sidekiq::Testing.inline!` | Runs every enqueued job, including factory/callback strays | `clear_all` + drain the specific worker (see Testing Jobs) |
|
|
573
|
+
| Draining all queues to test one flow | Hides which jobs the flow actually enqueues | Drain named workers; assert the rest stayed enqueued |
|
|
574
|
+
| Sleeping to wait for a scheduled job | Wastes wall-clock, still flaky | Assert on the scheduled set / `jobs.last["at"]` |
|
|
575
|
+
|
|
576
|
+
## Escalation Ladders
|
|
577
|
+
|
|
578
|
+
### "This job is double-running / I see duplicate side effects"
|
|
579
|
+
|
|
580
|
+
1. Check if the job is **idempotent** -- does `perform(*same_args)` twice produce the same end state? If not, fix that first; retries are real and rare events will always produce duplicates without idempotency.
|
|
581
|
+
2. Check **uniqueness** -- is the same `(class, args)` enqueued more than once? Inspect with `Sidekiq::Queue.new("default").select { |j| j.klass == "..." }`. If yes, add a uniqueness lock or a `job_intents` unique index.
|
|
582
|
+
3. Check **transactional ordering** -- is the enqueue inside a transaction that may roll back? Move it out or use `after_commit`.
|
|
583
|
+
4. Check for **multiple producers** -- a callback AND a service AND a controller all enqueuing the same job is a common cause.
|
|
584
|
+
|
|
585
|
+
### "This job is silently not running"
|
|
586
|
+
|
|
587
|
+
1. Inspect the queue: `Sidekiq::Queue.new("queue_name").size`.
|
|
588
|
+
2. Inspect the retry set: `Sidekiq::RetrySet.new.select { |j| j.klass == "..." }` -- often the job is failing and you don't see logs.
|
|
589
|
+
3. Inspect the dead set: `Sidekiq::DeadSet.new.select { |j| ... }` -- 25 silent retries land here.
|
|
590
|
+
4. Inspect the scheduled set: `Sidekiq::ScheduledSet.new` -- `perform_in` lives here.
|
|
591
|
+
5. Confirm the worker process is consuming the right queues: `sidekiq -q critical -q default -q low` order matters (left-to-right priority).
|
|
592
|
+
|
|
593
|
+
### "This job is too slow / queue is backing up"
|
|
594
|
+
|
|
595
|
+
1. **Profile one example**: time `perform` for a representative arg set. If `perform` itself is slow, the queue backup is downstream of the worker.
|
|
596
|
+
2. **Check N+1**: most "slow Sidekiq" is "slow ActiveRecord". Add `bullet` or `prosopite` in dev; eager-load.
|
|
597
|
+
3. **Check external API latency**: a 2s API call * 10k jobs = 5.5 hours single-threaded. Either parallelize (`perform_bulk` + concurrency) or batch the API calls.
|
|
598
|
+
4. **Check concurrency vs. pool**: `concurrency: 25` with `pool: 5` stalls 20 threads on DB checkout.
|
|
599
|
+
5. **Check queue weights**: `sidekiq -q critical,4 -q default,2 -q low,1` -- without weights, all queues are equal and `low` starves `critical`.
|
|
600
|
+
6. **Check for a noisy tenant**: if most queued jobs belong to one account, capacity is fine — fairness isn't. Re-route that tenant's overflow to throttled lower-weight queues<% if !gem?("sidekiq-fair_tenant") %>, and propose `sidekiq-fair_tenant` rather than hand-rolling the throttling<% end %> (see Fair scheduling between tenants).
|
|
601
|
+
7. **Last resort: split processes**. Move `low` to its own dyno/pod with separate concurrency tuning so `critical` is never starved.
|
|
602
|
+
|
|
603
|
+
### "How do I cancel a scheduled job?"
|
|
604
|
+
|
|
605
|
+
Sidekiq does not give jids a real cancel API. Patterns:
|
|
606
|
+
|
|
607
|
+
1. **Source-of-truth check inside `perform`**: store the intent on a record (`reminder_scheduled_for`), and `return` from `perform` if `reminder_scheduled_for != args[1]` (i.e. it was rescheduled or cancelled). This is the canonical pattern.
|
|
608
|
+
2. **Manual scheduled-set removal**: `Sidekiq::ScheduledSet.new.find { |j| j.jid == jid }&.delete` -- works but the jid is the only handle; keep it on the source record.
|
|
609
|
+
|
|
610
|
+
Never delete the job from the queue while it might already be running.
|
metadata
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: rails-hyperdrive-martian-sidekiq
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- izhanov
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies: []
|
|
12
|
+
description: |
|
|
13
|
+
Companion gem for rails-hyperdrive. Ships the `martian-sidekiq` skill — a procedural,
|
|
14
|
+
model-invoked guide for writing idempotent, well-retried Sidekiq jobs in Rails
|
|
15
|
+
projects — plus an always-on guideline with the non-negotiable Sidekiq rules.
|
|
16
|
+
Installed by `bin/rails hyperdrive:init`.
|
|
17
|
+
email:
|
|
18
|
+
- aibek.izhanov@evilmartians.com
|
|
19
|
+
executables: []
|
|
20
|
+
extensions: []
|
|
21
|
+
extra_rdoc_files: []
|
|
22
|
+
files:
|
|
23
|
+
- CHANGELOG.md
|
|
24
|
+
- LICENSE.txt
|
|
25
|
+
- README.md
|
|
26
|
+
- hyperdrive.yml
|
|
27
|
+
- lib/rails-hyperdrive-martian-sidekiq.rb
|
|
28
|
+
- lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-fair-tenant.md
|
|
29
|
+
- lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq-transactions.md
|
|
30
|
+
- lib/rails-hyperdrive-martian-sidekiq/hyperdrive/guidelines/martian-sidekiq.md.erb
|
|
31
|
+
- lib/rails-hyperdrive-martian-sidekiq/hyperdrive/skills/martian-sidekiq/SKILL.md.erb
|
|
32
|
+
- lib/rails-hyperdrive-martian-sidekiq/version.rb
|
|
33
|
+
homepage: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq
|
|
34
|
+
licenses:
|
|
35
|
+
- MIT
|
|
36
|
+
metadata:
|
|
37
|
+
homepage_uri: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq
|
|
38
|
+
source_code_uri: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq
|
|
39
|
+
changelog_uri: https://github.com/rails-hyperdrive/rails-hyperdrive-martian-sidekiq/blob/main/CHANGELOG.md
|
|
40
|
+
allowed_push_host: https://rubygems.org
|
|
41
|
+
rubygems_mfa_required: 'true'
|
|
42
|
+
hyperdrive_targets: sidekiq
|
|
43
|
+
hyperdrive_artifacts: skill,guideline
|
|
44
|
+
rdoc_options: []
|
|
45
|
+
require_paths:
|
|
46
|
+
- lib
|
|
47
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
48
|
+
requirements:
|
|
49
|
+
- - ">="
|
|
50
|
+
- !ruby/object:Gem::Version
|
|
51
|
+
version: 3.2.0
|
|
52
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
53
|
+
requirements:
|
|
54
|
+
- - ">="
|
|
55
|
+
- !ruby/object:Gem::Version
|
|
56
|
+
version: '0'
|
|
57
|
+
requirements: []
|
|
58
|
+
rubygems_version: 4.0.6
|
|
59
|
+
specification_version: 4
|
|
60
|
+
summary: 'Rails Hyperdrive companion gem: Sidekiq skill for AI coding agents.'
|
|
61
|
+
test_files: []
|