wurk 1.0.1 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +3 -0
- data/app/controllers/wurk/api_controller.rb +22 -1
- data/lib/generators/wurk/install/templates/wurk.rb +4 -3
- data/lib/wurk/batch.rb +9 -0
- data/lib/wurk/capsule.rb +18 -4
- data/lib/wurk/cli.rb +12 -0
- data/lib/wurk/client.rb +9 -1
- data/lib/wurk/compat.rb +19 -0
- data/lib/wurk/configuration.rb +18 -1
- data/lib/wurk/fetcher/reliable.rb +1 -1
- data/lib/wurk/launcher.rb +29 -4
- data/lib/wurk/limiter/leaky.rb +1 -1
- data/lib/wurk/limiter.rb +10 -0
- data/lib/wurk/railtie.rb +19 -1
- data/lib/wurk/unique.rb +21 -1
- data/lib/wurk/version.rb +1 -1
- data/lib/wurk/worker.rb +64 -3
- data/lib/wurk.rb +36 -2
- data/vendor/assets/dashboard/assets/geist-sans-latin-400-normal-BOaIZNA2.woff +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-400-normal-gapTbOY8.woff2 +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-500-normal-CN2lyvyL.woff +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-500-normal-uokXdC-Q.woff2 +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-600-normal-CA1yjETN.woff +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-600-normal-DFOURf8L.woff2 +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-700-normal-BmN9tIp5.woff2 +0 -0
- data/vendor/assets/dashboard/assets/geist-sans-latin-700-normal-CjScfYeH.woff +0 -0
- data/vendor/assets/dashboard/assets/index-BCGWUExk.js +143 -0
- data/vendor/assets/dashboard/assets/index-MNiNmpDL.css +1 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-cyrillic-500-normal-DJqRU3vO.woff +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-cyrillic-500-normal-DmUKJPL_.woff2 +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-greek-500-normal-D7SFKleX.woff +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-greek-500-normal-JpySY46c.woff2 +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-latin-500-normal-BWZEU5yA.woff2 +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-latin-500-normal-CJOVTJB7.woff +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-latin-ext-500-normal-Cut-4mMH.woff2 +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-latin-ext-500-normal-ckzbgY84.woff +0 -0
- data/vendor/assets/dashboard/assets/jetbrains-mono-vietnamese-500-normal-DNRqzVM1.woff +0 -0
- data/vendor/assets/dashboard/index.html +2 -2
- data/vendor/assets/dashboard/wurk-manifest.json +2 -2
- metadata +61 -4
- data/vendor/assets/dashboard/assets/index-9CFRWpfG.js +0 -77
- data/vendor/assets/dashboard/assets/index-CW8AFQIv.css +0 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 4f6df1c551958881248c0b6a97709e2a4999ec57f976cac37371d7c3e1c5d4e7
|
|
4
|
+
data.tar.gz: 450a316b29bada67e8deac2f1411ec1b084f57a2af48e752800b5aed01c9a0af
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 077cad32a71630367889448188e4bcaa59ed16b5853f37caf01efa5f907c939a4937d04fccb8df21eb256d9b685a50c2dce01cd041e5c5870573ee4c77eb7ef5
|
|
7
|
+
data.tar.gz: ec16e8f19b58332974d0e38cdb5ecb665b508698465435de29014ce2ff395ff1b374a827b098bcdd22abf1db9dc36d4692c3900e6b8091c1c22b0ebb1be878d1
|
data/README.md
CHANGED
|
@@ -56,7 +56,10 @@ Plus Wurk extras: a worker topology DSL, a Kubernetes liveness/readiness listene
|
|
|
56
56
|
## Documentation
|
|
57
57
|
|
|
58
58
|
- **[Website](https://developerz-ai.github.io/wurk/)** · **[Wiki / full docs](https://github.com/developerz-ai/wurk/wiki)** — the pitch, install, and the complete guide.
|
|
59
|
+
- **[API reference (YARD)](https://developerz-ai.github.io/wurk/api/)** — generated docs for the public classes (`Wurk::Worker`, `Wurk::Client`, `Wurk::Configuration`, `Wurk::Batch`, `Wurk::Limiter`, `Wurk::Unique`, and the `Sidekiq::*` aliases). Machine-readable map for AI agents: **[llms.txt](https://developerz-ai.github.io/wurk/llms.txt)**.
|
|
59
60
|
- **[Getting started & architecture](https://github.com/developerz-ai/wurk/blob/main/docs/idea/01-overview.md)** — how the swarm, manager, fetcher, and processor fit together.
|
|
61
|
+
- **[Starting the worker](https://github.com/developerz-ai/wurk/blob/main/docs/running.md)** — Rails auto-start, the `wurk`/`wurkswarm` runners, and running standalone without Rails.
|
|
62
|
+
- **[Active Job adapter](https://github.com/developerz-ai/wurk/blob/main/docs/active-job.md)** — run `ActiveJob`/`deliver_later` on Wurk with `queue_adapter = :wurk`.
|
|
60
63
|
- **[Migrating from Sidekiq](#migrating-from-sidekiq)** — the one-line swap and what to expect.
|
|
61
64
|
- **API reference (parity specs):** [Sidekiq OSS](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-free.md) · [Pro](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-pro.md) · [Enterprise](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-ent.md) — the authoritative surface Wurk matches exactly.
|
|
62
65
|
- **[Securing the dashboard](https://github.com/developerz-ai/wurk/blob/main/docs/dashboard.md)** · **[Metrics history](https://github.com/developerz-ai/wurk/blob/main/docs/metrics-history.md)**
|
|
@@ -44,7 +44,7 @@ module Wurk
|
|
|
44
44
|
def meta
|
|
45
45
|
config = ::Wurk::Web.config
|
|
46
46
|
render json: {
|
|
47
|
-
read_only: config.read_only
|
|
47
|
+
read_only: config.read_only? || !mutations_authorized?(config),
|
|
48
48
|
read_only_message: config.read_only_message,
|
|
49
49
|
custom_tabs: config.custom_tabs
|
|
50
50
|
}
|
|
@@ -294,6 +294,27 @@ module Wurk
|
|
|
294
294
|
|
|
295
295
|
private
|
|
296
296
|
|
|
297
|
+
# Engine-relative path probed as a representative mutation. Must be a real
|
|
298
|
+
# mutating route (POST /api/retries — bulk retry/delete/kill) so that a
|
|
299
|
+
# path-sensitive hook resolves it the same way the Authorization middleware
|
|
300
|
+
# will resolve the actual mutation. Probing `request.path_info` (which here
|
|
301
|
+
# is the GET /api/meta path) would let such a hook allow the probe while
|
|
302
|
+
# still 403ing real mutations, reviving the "button shows, then 403s" gap.
|
|
303
|
+
MUTATION_PROBE_PATH = '/api/retries'
|
|
304
|
+
|
|
305
|
+
# Per-request read-only signal for the SPA. When a registered authorization
|
|
306
|
+
# hook would reject a *mutating* request for this user (e.g. a viewer role
|
|
307
|
+
# that may GET but not retry/kill), report `read_only` so the SPA hides the
|
|
308
|
+
# destructive actions — the Authorization middleware already 403s the
|
|
309
|
+
# mutation itself, this just stops the buttons from showing. With no hook
|
|
310
|
+
# registered, `authorized?` is always true, so this is a no-op and the flag
|
|
311
|
+
# keeps reflecting the global read-only mode. Probes POST on a canonical
|
|
312
|
+
# mutating path (SAFE_METHODS are GET/HEAD/OPTIONS) so a path-sensitive hook
|
|
313
|
+
# answers for a real mutation, not the GET /api/meta request carrying it.
|
|
314
|
+
def mutations_authorized?(config)
|
|
315
|
+
config.authorized?(request.env, 'POST', MUTATION_PROBE_PATH)
|
|
316
|
+
end
|
|
317
|
+
|
|
297
318
|
# Resolves a single entry by "<score>|<jid>" key and applies a whitelisted
|
|
298
319
|
# action. 400 on an unknown action, 404 when the key matches nothing (e.g.
|
|
299
320
|
# the entry was already retried/deleted from another tab).
|
|
@@ -16,9 +16,10 @@ Wurk.configure_server do |config|
|
|
|
16
16
|
# Seconds to let in-flight jobs finish on shutdown (Sidekiq-compatible key):
|
|
17
17
|
# config[:timeout] = 25
|
|
18
18
|
|
|
19
|
-
#
|
|
20
|
-
#
|
|
21
|
-
# queues.
|
|
19
|
+
# By default Wurk forks one worker process per CPU core (set WURK_COUNT, or
|
|
20
|
+
# SIDEKIQ_COUNT, to override). For full control declare a topology — `flat`
|
|
21
|
+
# spawns N identical forks; use `slot`s for dedicated queues. Total in-flight
|
|
22
|
+
# jobs = forks × concurrency. See docs/idea/03-process-model.md.
|
|
22
23
|
# config.topology = Wurk::Topology.flat(count: 2, queues: %w[critical default low], concurrency: 10)
|
|
23
24
|
end
|
|
24
25
|
|
data/lib/wurk/batch.rb
CHANGED
|
@@ -9,6 +9,15 @@ module Wurk
|
|
|
9
9
|
# Sidekiq Pro Batches. Group jobs, attach success/complete/death callbacks,
|
|
10
10
|
# track progress. Spec: docs/target/sidekiq-pro.md §2.
|
|
11
11
|
#
|
|
12
|
+
# @example Define a batch with a success callback
|
|
13
|
+
# batch = Sidekiq::Batch.new
|
|
14
|
+
# batch.description = "Nightly import"
|
|
15
|
+
# batch.on(:success, ImportCallback, "user_id" => user.id)
|
|
16
|
+
# batch.jobs do
|
|
17
|
+
# rows.each { |r| ImportRowJob.perform_async(r.id) }
|
|
18
|
+
# end
|
|
19
|
+
# batch.bid # => the batch id, persisted in Redis
|
|
20
|
+
#
|
|
12
21
|
# Lifecycle:
|
|
13
22
|
# 1. `Batch.new` allocates a fresh BID; the batch is `mutable?` until the
|
|
14
23
|
# first `#jobs` block flushes — that flush HSETs the core hash, ZADDs
|
data/lib/wurk/capsule.rb
CHANGED
|
@@ -38,7 +38,7 @@ module Wurk
|
|
|
38
38
|
# %w[high default low] → mode :strict, weights all 0
|
|
39
39
|
# %w[high,3 default,2 low,1] → mode :weighted, weights {q=>w}
|
|
40
40
|
# %w[a,1 b,1 c,1] → mode :random, weights all 1
|
|
41
|
-
#
|
|
41
|
+
# `@queues` is expanded by weight so a uniform shuffle gives weighted
|
|
42
42
|
# fairness (e.g. ["high","high","high","default","default","low"]).
|
|
43
43
|
def queues=(val)
|
|
44
44
|
parsed = Array(val).map { |entry| parse_queue_entry(entry) }
|
|
@@ -153,13 +153,27 @@ module Wurk
|
|
|
153
153
|
fetcher
|
|
154
154
|
end
|
|
155
155
|
|
|
156
|
+
# Accepts both the CLI/string form (`"name"` / `"name,weight"`) and the
|
|
157
|
+
# nested-array form Sidekiq's YAML produces for weighted queues
|
|
158
|
+
# (`:queues: - [critical, 2]` parses to `["critical", 2]`). Without the Array
|
|
159
|
+
# branch a real sidekiq.yml crashes at boot with `Integer(): " 2]"` (#241).
|
|
156
160
|
def parse_queue_entry(entry)
|
|
161
|
+
if entry.is_a?(::Array)
|
|
162
|
+
raise ArgumentError, "queue entry must be `[name]` or `[name, weight]`: `#{entry}`" if entry.size > 2
|
|
163
|
+
|
|
164
|
+
return parse_pair_queue_entry(entry[0], entry[1], entry)
|
|
165
|
+
end
|
|
166
|
+
|
|
157
167
|
qname, weight_str = entry.to_s.split(',', 2)
|
|
158
|
-
qname
|
|
168
|
+
parse_pair_queue_entry(qname, weight_str, entry)
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def parse_pair_queue_entry(name, weight, entry)
|
|
172
|
+
qname = name.to_s.strip
|
|
159
173
|
raise ArgumentError, "queue name cannot be empty: `#{entry}`" if qname.empty?
|
|
160
|
-
return [qname, 0] if
|
|
174
|
+
return [qname, 0] if weight.nil?
|
|
161
175
|
|
|
162
|
-
weight = Integer(
|
|
176
|
+
weight = Integer(weight)
|
|
163
177
|
raise ArgumentError, "queue weight must be > 0: `#{entry}`" if weight <= 0
|
|
164
178
|
|
|
165
179
|
[qname, weight]
|
data/lib/wurk/cli.rb
CHANGED
|
@@ -398,10 +398,22 @@ module Wurk
|
|
|
398
398
|
|
|
399
399
|
def boot_rails_application(require_path)
|
|
400
400
|
require 'rails'
|
|
401
|
+
define_active_job_adapter
|
|
401
402
|
require ::File.expand_path("#{require_path}/config/environment.rb")
|
|
402
403
|
@config[:tag] ||= default_tag(::Rails.root) if defined?(::Rails) && ::Rails.respond_to?(:root)
|
|
403
404
|
end
|
|
404
405
|
|
|
406
|
+
# Standalone mode never loads the engine, so the ActiveJob `:wurk` /
|
|
407
|
+
# `:sidekiq` adapters it normally defines are absent. Define them BEFORE the
|
|
408
|
+
# app boots — an app with `config.active_job.queue_adapter = :wurk` resolves
|
|
409
|
+
# the adapter during environment boot, which otherwise raises `uninitialized
|
|
410
|
+
# constant WurkAdapter` (#253). `require` no-ops for a mounted-engine app
|
|
411
|
+
# that already loaded it, and the adapter file itself rescues a missing
|
|
412
|
+
# ActiveJob gem, so a non-ActiveJob app is unaffected.
|
|
413
|
+
def define_active_job_adapter
|
|
414
|
+
require_relative '../active_job/queue_adapters/wurk_adapter'
|
|
415
|
+
end
|
|
416
|
+
|
|
405
417
|
def initialize_logger
|
|
406
418
|
return unless @config[:verbose] || ENV['DEBUG_INVOCATION'] == '1'
|
|
407
419
|
|
data/lib/wurk/client.rb
CHANGED
|
@@ -7,7 +7,14 @@ require_relative 'lua'
|
|
|
7
7
|
module Wurk
|
|
8
8
|
# Enqueue interface. Pipelined LPUSH / ZADD writes against the canonical
|
|
9
9
|
# Sidekiq Redis schema — never change keys, JSON shape, or score format here:
|
|
10
|
-
# wire-compat is sacred.
|
|
10
|
+
# wire-compat is sacred. Most apps enqueue through the {Wurk::Worker} DSL
|
|
11
|
+
# (`MyJob.perform_async`), which routes here; reach for Client directly only to
|
|
12
|
+
# push a raw job hash or to drive the bulk/scheduled path explicitly.
|
|
13
|
+
#
|
|
14
|
+
# @example Push a raw job hash
|
|
15
|
+
# Wurk::Client.new.push("class" => "MyJob", "args" => [1, 2], "queue" => "default")
|
|
16
|
+
# @example Bulk enqueue in one round-trip
|
|
17
|
+
# Wurk::Client.new.push_bulk("class" => "MyJob", "args" => [[1], [2], [3]])
|
|
11
18
|
#
|
|
12
19
|
# Spec: docs/target/sidekiq-free.md §7.
|
|
13
20
|
class Client
|
|
@@ -36,6 +43,7 @@ module Wurk
|
|
|
36
43
|
copy
|
|
37
44
|
end
|
|
38
45
|
|
|
46
|
+
# @param item [Hash] job payload; must carry `class` and `args`, may carry `at`, `queue`, `jid`, etc.
|
|
39
47
|
# @return [String, nil] jid; nil when client middleware halts the push.
|
|
40
48
|
def push(item)
|
|
41
49
|
normed = normalize_item(item)
|
data/lib/wurk/compat.rb
CHANGED
|
@@ -5,6 +5,25 @@
|
|
|
5
5
|
#
|
|
6
6
|
# Spec: docs/target/sidekiq-{free,pro,ent}.md.
|
|
7
7
|
|
|
8
|
+
# The `Sidekiq::*` compatibility namespace. Every public Wurk class is exposed
|
|
9
|
+
# here under its Sidekiq name so an existing Sidekiq/Pro/Enterprise app runs on
|
|
10
|
+
# Wurk after a one-line gem swap. `Sidekiq::Job` / `Sidekiq::Worker` are
|
|
11
|
+
# {Wurk::Job} / {Wurk::Worker}; `Sidekiq.configure_server` is
|
|
12
|
+
# {Wurk.configure_server}; `Sidekiq::Batch`, `Sidekiq::Limiter`,
|
|
13
|
+
# `Sidekiq::Client`, `Sidekiq::Pro::Web`, `Sidekiq::Enterprise` all resolve to
|
|
14
|
+
# their Wurk implementations. This alias surface is the drop-in contract and is
|
|
15
|
+
# never broken.
|
|
16
|
+
#
|
|
17
|
+
# @example The same code targets Sidekiq or Wurk unchanged
|
|
18
|
+
# class MyJob
|
|
19
|
+
# include Sidekiq::Job
|
|
20
|
+
# def perform(id); end
|
|
21
|
+
# end
|
|
22
|
+
# Sidekiq.configure_server { |c| c.concurrency = 10 }
|
|
23
|
+
#
|
|
24
|
+
# @note `Sidekiq.pro?` and `Sidekiq.ent?` return `false` — Wurk ships the Pro/Ent
|
|
25
|
+
# features for free but advertises as OSS. Don't gate behavior on them.
|
|
26
|
+
# @see https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md Migration guide
|
|
8
27
|
module Sidekiq
|
|
9
28
|
# Version stamps mirror Sidekiq's OSS release Wurk targets for compat.
|
|
10
29
|
# Third-party gems version-gate on these; raise the MAJOR only when the
|
data/lib/wurk/configuration.rb
CHANGED
|
@@ -112,8 +112,14 @@ module Wurk
|
|
|
112
112
|
|
|
113
113
|
# --- Default capsule shortcuts ---------------------------------------
|
|
114
114
|
|
|
115
|
+
# @return [Integer] threads per worker process for the default capsule
|
|
116
|
+
# (default 5). The *process* count is separate — set via `WURK_COUNT`
|
|
117
|
+
# (defaults to the CPU count). With a single capsule, total in-flight
|
|
118
|
+
# jobs = `WURK_COUNT × concurrency`; with multiple capsules see
|
|
119
|
+
# {#total_concurrency} for the cluster aggregate.
|
|
115
120
|
def concurrency = default_capsule.concurrency
|
|
116
121
|
|
|
122
|
+
# @param val [Integer] threads per worker process
|
|
117
123
|
def concurrency=(val)
|
|
118
124
|
default_capsule.concurrency = val
|
|
119
125
|
end
|
|
@@ -251,7 +257,18 @@ module Wurk
|
|
|
251
257
|
|
|
252
258
|
# Yields a Wurk::Cron::Manager so the host app can register periodic
|
|
253
259
|
# jobs at boot. Manager state is shared per-process so multiple
|
|
254
|
-
# `config.periodic` blocks accumulate (matches Sidekiq Ent §2.1).
|
|
260
|
+
# `config.periodic` blocks accumulate (matches Sidekiq Ent §2.1). This is
|
|
261
|
+
# the native replacement for the `sidekiq-cron` gem.
|
|
262
|
+
#
|
|
263
|
+
# @example Register cron jobs at boot
|
|
264
|
+
# Wurk.configure_server do |config|
|
|
265
|
+
# config.periodic do |mgr|
|
|
266
|
+
# mgr.register("*/5 * * * *", ReportJob)
|
|
267
|
+
# mgr.register("0 0 * * *", NightlyJob, tz: "UTC")
|
|
268
|
+
# end
|
|
269
|
+
# end
|
|
270
|
+
# @yieldparam mgr [Wurk::Cron::Manager] call `mgr.register(cron, JobClass, **opts)`
|
|
271
|
+
# @return [Wurk::Cron::Manager]
|
|
255
272
|
#
|
|
256
273
|
# Spec: docs/target/sidekiq-ent.md §2.
|
|
257
274
|
def periodic
|
|
@@ -96,7 +96,7 @@ module Wurk
|
|
|
96
96
|
|
|
97
97
|
# Prefixed queue keys (`queue:<name>`) in fetch order. Strict mode
|
|
98
98
|
# preserves declaration order. Random/weighted shuffle each call —
|
|
99
|
-
#
|
|
99
|
+
# `@queues` is pre-expanded by weight in Capsule#queues=, so uniform
|
|
100
100
|
# shuffle yields weighted fairness; .uniq trims duplicates. Paused
|
|
101
101
|
# queues are filtered after shuffle so the membership test runs on
|
|
102
102
|
# the smallest possible set.
|
data/lib/wurk/launcher.rb
CHANGED
|
@@ -48,7 +48,12 @@ module Wurk
|
|
|
48
48
|
def initialize(config, embedded: false)
|
|
49
49
|
@config = config
|
|
50
50
|
@embedded = embedded
|
|
51
|
+
# Two separate flags, deliberately. @done = "quieted" (stop fetching, stay
|
|
52
|
+
# alive, report quiet=true). @stopped = "shutting down" (terminate the
|
|
53
|
+
# heartbeat loop). Quiet must NOT stop the heartbeat — otherwise a quieted
|
|
54
|
+
# process never publishes quiet=true and expires out of the live set (#236).
|
|
51
55
|
@done = false
|
|
56
|
+
@stopped = false
|
|
52
57
|
@managers = config.capsules.values.map { |cap| Manager.new(cap) }
|
|
53
58
|
@poller = build_poller
|
|
54
59
|
@cron_poller = build_cron_poller
|
|
@@ -127,6 +132,7 @@ module Wurk
|
|
|
127
132
|
# CAS-release the cluster lock now (planned shutdown) so a follower can
|
|
128
133
|
# take over immediately instead of waiting out the TTL.
|
|
129
134
|
@leader&.stop
|
|
135
|
+
stop_heartbeat
|
|
130
136
|
clear_heartbeat
|
|
131
137
|
fire_event(:exit, reverse: true)
|
|
132
138
|
end
|
|
@@ -217,11 +223,30 @@ module Wurk
|
|
|
217
223
|
@health_server&.stop
|
|
218
224
|
end
|
|
219
225
|
|
|
220
|
-
#
|
|
221
|
-
#
|
|
222
|
-
#
|
|
226
|
+
# Terminate the heartbeat loop and wait for it to exit before clear_heartbeat
|
|
227
|
+
# removes us from the `processes` SET — otherwise a final in-flight beat could
|
|
228
|
+
# SADD us back right after the SREM. Wakes the thread out of its BEAT_PAUSE
|
|
229
|
+
# sleep so shutdown isn't delayed up to a full interval.
|
|
230
|
+
def stop_heartbeat
|
|
231
|
+
@stopped = true
|
|
232
|
+
thread = @heartbeat_thread
|
|
233
|
+
return unless thread
|
|
234
|
+
|
|
235
|
+
begin
|
|
236
|
+
thread.wakeup
|
|
237
|
+
rescue ThreadError
|
|
238
|
+
nil
|
|
239
|
+
end
|
|
240
|
+
thread.join(BEAT_PAUSE)
|
|
241
|
+
end
|
|
242
|
+
|
|
243
|
+
# Heartbeat thread loop. `safe_thread` already wraps exceptions. Loops on
|
|
244
|
+
# `@stopped` — NOT `@done` — so a *quieted* process keeps beating and publishes
|
|
245
|
+
# `quiet=true` instead of vanishing from the live set (#236). Only `#stop`
|
|
246
|
+
# flips `@stopped`; its `Thread#wakeup` breaks the sleep so the loop re-checks
|
|
247
|
+
# `@stopped` and exits without waiting out the interval.
|
|
223
248
|
def start_heartbeat
|
|
224
|
-
until @
|
|
249
|
+
until @stopped
|
|
225
250
|
heartbeat
|
|
226
251
|
sleep BEAT_PAUSE
|
|
227
252
|
end
|
data/lib/wurk/limiter/leaky.rb
CHANGED
|
@@ -5,7 +5,7 @@ require_relative 'base'
|
|
|
5
5
|
module Wurk
|
|
6
6
|
module Limiter
|
|
7
7
|
# Leaky bucket: drain rate = bucket_size / drain ops/sec. Stored as a
|
|
8
|
-
# HASH of {level, last} — Lua compares level vs bucket_size after
|
|
8
|
+
# HASH of `{level, last}` — Lua compares level vs bucket_size after
|
|
9
9
|
# leaking elapsed * drain_per_sec.
|
|
10
10
|
class Leaky < Base
|
|
11
11
|
WAIT_SLEEP = 0.05
|
data/lib/wurk/limiter.rb
CHANGED
|
@@ -11,6 +11,16 @@ module Wurk
|
|
|
11
11
|
# clock skew across hosts doesn't matter inside one Redis. Spec:
|
|
12
12
|
# docs/target/sidekiq-ent.md §1.
|
|
13
13
|
#
|
|
14
|
+
# @example Throttle to 50 concurrent uses, waiting up to the default timeout
|
|
15
|
+
# STRIPE = Sidekiq::Limiter.concurrent("stripe", 50)
|
|
16
|
+
#
|
|
17
|
+
# class ChargeJob
|
|
18
|
+
# include Sidekiq::Job
|
|
19
|
+
# def perform(id)
|
|
20
|
+
# STRIPE.within_limit { Stripe::Charge.create(...) }
|
|
21
|
+
# end
|
|
22
|
+
# end
|
|
23
|
+
#
|
|
14
24
|
# Layout (one file per type under `lib/wurk/limiter/`):
|
|
15
25
|
# * `Limiter::Base` owns the metadata write (lmtr:{name}) + the global
|
|
16
26
|
# `lmtr-list` registration so the Web UI can list every limiter, and
|
data/lib/wurk/railtie.rb
CHANGED
|
@@ -36,7 +36,25 @@ module Wurk
|
|
|
36
36
|
# and the swarm boot. Console mode is detected reliably here — the console
|
|
37
37
|
# command file defines ::Rails::Console before initializers run.
|
|
38
38
|
def self.skip_boot?
|
|
39
|
-
ENV['WURK_DISABLED'] == '1' ||
|
|
39
|
+
ENV['WURK_DISABLED'] == '1' ||
|
|
40
|
+
building? ||
|
|
41
|
+
defined?(::Rails::Console) ||
|
|
42
|
+
::Rails.env.test?
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
# A build/precompile step must never fork the swarm (#247). The default
|
|
46
|
+
# Rails Dockerfile runs `SECRET_KEY_BASE_DUMMY=1 ./bin/rails
|
|
47
|
+
# assets:precompile`; that loads `:environment` → fires after_initialize,
|
|
48
|
+
# but there's no Redis during `docker build`, so a fork would hang/fail the
|
|
49
|
+
# build. Same for other env-loading rake tasks (db:prepare, db:migrate).
|
|
50
|
+
# The real server path is unaffected: `rails server` / `puma` boot through
|
|
51
|
+
# Rails::Command, not Rake, and don't set the dummy secret.
|
|
52
|
+
def self.building?
|
|
53
|
+
return true if ENV.key?('SECRET_KEY_BASE_DUMMY')
|
|
54
|
+
|
|
55
|
+
defined?(::Rake) && ::Rake.application.top_level_tasks.any?
|
|
56
|
+
rescue StandardError
|
|
57
|
+
false
|
|
40
58
|
end
|
|
41
59
|
end
|
|
42
60
|
end
|
data/lib/wurk/unique.rb
CHANGED
|
@@ -25,6 +25,22 @@ module Wurk
|
|
|
25
25
|
# holding the owning JID. Scheduled jobs extend the TTL by the delay so
|
|
26
26
|
# the lock covers the entire wait+execution window (§3.4).
|
|
27
27
|
#
|
|
28
|
+
# This is the native replacement for the `sidekiq-unique-jobs` gem; see the
|
|
29
|
+
# migration guide for the `lock:` → `unique_until:` translation table.
|
|
30
|
+
#
|
|
31
|
+
# @example Enable globally, then declare per worker
|
|
32
|
+
# Sidekiq::Enterprise.unique! # activate the middleware once, at boot
|
|
33
|
+
#
|
|
34
|
+
# class ChargeJob
|
|
35
|
+
# include Sidekiq::Job
|
|
36
|
+
# sidekiq_options unique_for: 10.minutes, unique_until: :success
|
|
37
|
+
#
|
|
38
|
+
# # optional: customize the dedup key
|
|
39
|
+
# def self.sidekiq_unique_context(job)
|
|
40
|
+
# job["args"].first # dedup on the first arg only
|
|
41
|
+
# end
|
|
42
|
+
# end
|
|
43
|
+
#
|
|
28
44
|
# Spec: docs/target/sidekiq-ent.md §3.
|
|
29
45
|
module Unique
|
|
30
46
|
KEY_PREFIX = 'unique:'
|
|
@@ -131,7 +147,11 @@ module Wurk
|
|
|
131
147
|
# (skip). Returns nil when uniqueness should be skipped.
|
|
132
148
|
def self.coerce_ttl(value)
|
|
133
149
|
return nil if value.nil? || value == false
|
|
134
|
-
|
|
150
|
+
# `.to_i`, not `value`: ActiveSupport::Duration overrides `is_a?(Integer)`
|
|
151
|
+
# to return true (it delegates to its underlying value), so `1.hour` passes
|
|
152
|
+
# this guard — returning the raw Duration handed redis-client a non-Integer
|
|
153
|
+
# EX arg and raised TypeError at enqueue (#253).
|
|
154
|
+
return value.to_i if value.is_a?(Integer) && value.positive?
|
|
135
155
|
return value.to_i if value.is_a?(Numeric)
|
|
136
156
|
return value.to_i if duration_like?(value)
|
|
137
157
|
|
data/lib/wurk/version.rb
CHANGED
data/lib/wurk/worker.rb
CHANGED
|
@@ -3,9 +3,27 @@
|
|
|
3
3
|
require_relative 'worker/setter'
|
|
4
4
|
|
|
5
5
|
module Wurk
|
|
6
|
-
# The user-facing DSL
|
|
7
|
-
#
|
|
8
|
-
# `
|
|
6
|
+
# The user-facing job DSL. `include Wurk::Worker` — or its modern alias
|
|
7
|
+
# `Sidekiq::Job` / `Sidekiq::Worker` — onto a class to make it a background
|
|
8
|
+
# job. The class gains `sidekiq_options`, the `perform_*` enqueue methods,
|
|
9
|
+
# `set`, and the retry-hook DSL; each instance gains `jid`, `logger`,
|
|
10
|
+
# `interrupted?`, and the batch helpers.
|
|
11
|
+
#
|
|
12
|
+
# @example A minimal job
|
|
13
|
+
# class HardJob
|
|
14
|
+
# include Sidekiq::Job
|
|
15
|
+
# sidekiq_options queue: "critical", retry: 5
|
|
16
|
+
#
|
|
17
|
+
# def perform(user_id, opts = {})
|
|
18
|
+
# # ... your work ...
|
|
19
|
+
# end
|
|
20
|
+
# end
|
|
21
|
+
#
|
|
22
|
+
# HardJob.perform_async(42, "fast" => true) # enqueue now
|
|
23
|
+
# HardJob.perform_in(5.minutes, 42) # enqueue later
|
|
24
|
+
#
|
|
25
|
+
# @see Wurk::Worker::ClassMethods the enqueue + options DSL added to the class
|
|
26
|
+
# @see https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md Migration guide
|
|
9
27
|
#
|
|
10
28
|
# Spec: docs/target/sidekiq-free.md §6 (Sidekiq::Job).
|
|
11
29
|
module Worker
|
|
@@ -65,7 +83,17 @@ module Wurk
|
|
|
65
83
|
batch.valid?
|
|
66
84
|
end
|
|
67
85
|
|
|
86
|
+
# Class-level DSL mixed into every job class by {Wurk::Worker}. These are
|
|
87
|
+
# the public enqueue and configuration entry points.
|
|
68
88
|
module ClassMethods # rubocop:disable Metrics/ModuleLength
|
|
89
|
+
# Set per-class job options (merged over any inherited options).
|
|
90
|
+
#
|
|
91
|
+
# @example
|
|
92
|
+
# sidekiq_options queue: "mailers", retry: 3, unique_for: 10.minutes
|
|
93
|
+
# @param opts [Hash] any of `queue:`, `retry:`, `dead:`, `backtrace:`,
|
|
94
|
+
# `expires_in:`, `tags:`, `pool:`, `unique_for:`, … (see the migration
|
|
95
|
+
# guide's sidekiq_options table for the full set)
|
|
96
|
+
# @return [Hash] the merged, string-keyed options hash
|
|
69
97
|
def sidekiq_options(opts = {})
|
|
70
98
|
merged = get_sidekiq_options.merge(opts.transform_keys(&:to_s))
|
|
71
99
|
@sidekiq_options_hash = merged
|
|
@@ -92,24 +120,57 @@ module Wurk
|
|
|
92
120
|
self.sidekiq_retries_exhausted_block = block
|
|
93
121
|
end
|
|
94
122
|
|
|
123
|
+
# Enqueue the job to run as soon as a worker is free. Arguments are
|
|
124
|
+
# forwarded to `#perform` and must be JSON-serializable
|
|
125
|
+
# (string/number/bool/nil/array/hash).
|
|
126
|
+
#
|
|
127
|
+
# @example
|
|
128
|
+
# EmailJob.perform_async(user.id, "welcome")
|
|
129
|
+
# @return [String, nil] the job id (jid), or nil if a client middleware
|
|
130
|
+
# halted the push
|
|
95
131
|
def perform_async(*)
|
|
96
132
|
Wurk::Worker::Setter.new(self, {}).perform_async(*)
|
|
97
133
|
end
|
|
98
134
|
|
|
135
|
+
# Run `#perform` synchronously in the current thread (no Redis). Useful in
|
|
136
|
+
# tests and for inline execution.
|
|
137
|
+
#
|
|
138
|
+
# @return [Object] the return value of `#perform`
|
|
99
139
|
def perform_inline(*)
|
|
100
140
|
new.perform(*)
|
|
101
141
|
end
|
|
102
142
|
alias perform_sync perform_inline
|
|
103
143
|
|
|
144
|
+
# Schedule the job for later. `perform_at` is an alias taking an absolute
|
|
145
|
+
# time; `perform_in` takes a relative interval.
|
|
146
|
+
#
|
|
147
|
+
# @example
|
|
148
|
+
# ReminderJob.perform_in(1.hour, lead.id)
|
|
149
|
+
# ReminderJob.perform_at(Time.now + 3600, lead.id)
|
|
150
|
+
# @param interval [Numeric, Time] seconds-from-now, or an absolute Time
|
|
151
|
+
# @return [String, nil] the job id (jid)
|
|
104
152
|
def perform_in(interval, *)
|
|
105
153
|
Wurk::Worker::Setter.new(self, {}).perform_in(interval, *)
|
|
106
154
|
end
|
|
107
155
|
alias perform_at perform_in
|
|
108
156
|
|
|
157
|
+
# Enqueue many jobs in one round-trip via the Lua bulk path.
|
|
158
|
+
#
|
|
159
|
+
# @example
|
|
160
|
+
# ImportJob.perform_bulk([[1], [2], [3]])
|
|
161
|
+
# @param items [Array<Array>] one args array per job
|
|
162
|
+
# @return [Array<String>] the job ids, in order
|
|
109
163
|
def perform_bulk(items, **)
|
|
110
164
|
Wurk::Worker::Setter.new(self, {}).perform_bulk(items, **)
|
|
111
165
|
end
|
|
112
166
|
|
|
167
|
+
# Return a per-call option carrier so a single enqueue can override
|
|
168
|
+
# class-level options (queue, scheduling, pool, …).
|
|
169
|
+
#
|
|
170
|
+
# @example
|
|
171
|
+
# ReportJob.set(queue: "low").perform_async(account.id)
|
|
172
|
+
# @param opts [Hash] per-call overrides
|
|
173
|
+
# @return [Wurk::Worker::Setter]
|
|
113
174
|
def set(opts)
|
|
114
175
|
Wurk::Worker::Setter.new(self, opts)
|
|
115
176
|
end
|
data/lib/wurk.rb
CHANGED
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
# Standalone entry point. Loading "wurk" must work without Rails.
|
|
4
|
-
# The engine and railtie live under "wurk/rails"
|
|
5
|
-
# when
|
|
4
|
+
# The engine and railtie live under "wurk/rails"; they're auto-loaded at the
|
|
5
|
+
# end of this file when Rails is present (drop-in parity, #246), and stay
|
|
6
|
+
# unloaded otherwise so a no-Rails boot pulls in zero Rails code.
|
|
6
7
|
|
|
7
8
|
require_relative 'wurk/version'
|
|
8
9
|
require_relative 'wurk/keys'
|
|
@@ -71,6 +72,18 @@ require_relative 'wurk/deploy'
|
|
|
71
72
|
|
|
72
73
|
require 'json'
|
|
73
74
|
|
|
75
|
+
# Wurk — a 100% API-compatible, free, faster drop-in for Sidekiq + Sidekiq Pro
|
|
76
|
+
# + Sidekiq Enterprise. Same Redis key schema, same job JSON, same Ruby DSL;
|
|
77
|
+
# real parallelism via a fork-based swarm. Every public class is also exposed
|
|
78
|
+
# under its `Sidekiq::*` name (see {Sidekiq}).
|
|
79
|
+
#
|
|
80
|
+
# Start here:
|
|
81
|
+
# * {Wurk::Worker} / `Sidekiq::Job` — define and enqueue jobs.
|
|
82
|
+
# * {Wurk::Configuration} — `Wurk.configure_server { |config| ... }`; note
|
|
83
|
+
# `WURK_COUNT` (forked processes) × `concurrency` (threads) = total in-flight.
|
|
84
|
+
# * {Wurk::Batch}, {Wurk::Limiter}, {Wurk::Unique} — Pro/Enterprise features, free.
|
|
85
|
+
#
|
|
86
|
+
# @see https://github.com/developerz-ai/wurk/blob/main/docs/migrate-from-sidekiq.md Migration guide
|
|
74
87
|
module Wurk
|
|
75
88
|
class Error < StandardError; end
|
|
76
89
|
|
|
@@ -280,3 +293,24 @@ require_relative 'wurk/api/fast'
|
|
|
280
293
|
# fully defined first. compat.rb only redefines names, it does not gate
|
|
281
294
|
# behavior, so trailing the load order is safe.
|
|
282
295
|
require_relative 'wurk/compat'
|
|
296
|
+
|
|
297
|
+
# Drop-in parity (#246): stock Sidekiq auto-loads its Rails integration when
|
|
298
|
+
# Rails is present (`lib/sidekiq.rb`: `require "sidekiq/rails" if
|
|
299
|
+
# defined?(Rails::Engine)`). Without the equivalent here, a host that follows
|
|
300
|
+
# the README/migration guide with a plain `gem "wurk"` (no `require:`) boots
|
|
301
|
+
# with the engine/railtie unloaded, so the `:wurk` / `:sidekiq` ActiveJob
|
|
302
|
+
# adapter constant is never defined and `config.active_job.queue_adapter =
|
|
303
|
+
# :wurk` raises NameError during the railtie phase. Loading the engine here
|
|
304
|
+
# (idempotent with an explicit `require "wurk/rails"`) closes that gap while
|
|
305
|
+
# keeping standalone, no-Rails boot fully Rails-free.
|
|
306
|
+
#
|
|
307
|
+
# Also gate on ActionDispatch::Routing::RouteSet: the engine's class body calls
|
|
308
|
+
# `isolate_namespace Wurk`, which under railties >= 8.1 eager-reads
|
|
309
|
+
# `ActionDispatch::Routing::RouteSet` to set the engine's @route_set_class.
|
|
310
|
+
# Ecosystem test helpers (sidekiq-cron, …) load `rails/engine/railties` for the
|
|
311
|
+
# railtie API alone — Rails::Engine is defined but ActionDispatch isn't — so a
|
|
312
|
+
# bare `defined?(Rails::Engine)` check would crash with `uninitialized constant
|
|
313
|
+
# Rails::Engine::Configuration::ActionDispatch` mid-`require "sidekiq"`. In a
|
|
314
|
+
# real Rails host both are loaded by `rails/all` before Bundler.require, so the
|
|
315
|
+
# stricter gate is invisible there.
|
|
316
|
+
require_relative 'wurk/rails' if defined?(Rails::Engine) && defined?(::ActionDispatch::Routing::RouteSet)
|