sidekiq-unique-jobs 7.0.0.beta26 → 7.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of sidekiq-unique-jobs might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/CHANGELOG.md +89 -11
- data/README.md +154 -36
- data/lib/sidekiq-unique-jobs.rb +0 -2
- data/lib/sidekiq_unique_jobs.rb +1 -0
- data/lib/sidekiq_unique_jobs/changelog.rb +11 -4
- data/lib/sidekiq_unique_jobs/config.rb +2 -2
- data/lib/sidekiq_unique_jobs/constants.rb +4 -0
- data/lib/sidekiq_unique_jobs/digests.rb +1 -1
- data/lib/sidekiq_unique_jobs/job.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock/validator.rb +2 -1
- data/lib/sidekiq_unique_jobs/lock/while_executing.rb +1 -1
- data/lib/sidekiq_unique_jobs/lock_args.rb +4 -4
- data/lib/sidekiq_unique_jobs/lock_config.rb +2 -0
- data/lib/sidekiq_unique_jobs/locksmith.rb +1 -1
- data/lib/sidekiq_unique_jobs/lua/delete_by_digest.lua +1 -1
- data/lib/sidekiq_unique_jobs/lua/lock.lua +1 -2
- data/lib/sidekiq_unique_jobs/lua/reap_orphans.lua +8 -7
- data/lib/sidekiq_unique_jobs/lua/shared/_find_digest_in_process_set.lua +9 -2
- data/lib/sidekiq_unique_jobs/middleware.rb +0 -57
- data/lib/sidekiq_unique_jobs/on_conflict/replace.rb +9 -8
- data/lib/sidekiq_unique_jobs/on_conflict/reschedule.rb +1 -1
- data/lib/sidekiq_unique_jobs/orphans/lua_reaper.rb +1 -1
- data/lib/sidekiq_unique_jobs/orphans/reaper.rb +10 -0
- data/lib/sidekiq_unique_jobs/orphans/ruby_reaper.rb +9 -2
- data/lib/sidekiq_unique_jobs/redis/entity.rb +9 -3
- data/lib/sidekiq_unique_jobs/redis/sorted_set.rb +27 -0
- data/lib/sidekiq_unique_jobs/server.rb +48 -0
- data/lib/sidekiq_unique_jobs/sidekiq_unique_jobs.rb +1 -1
- data/lib/sidekiq_unique_jobs/sidekiq_worker_methods.rb +1 -1
- data/lib/sidekiq_unique_jobs/testing.rb +2 -1
- data/lib/sidekiq_unique_jobs/version.rb +1 -1
- data/lib/sidekiq_unique_jobs/web.rb +26 -9
- data/lib/sidekiq_unique_jobs/web/helpers.rb +24 -3
- data/lib/sidekiq_unique_jobs/web/views/changelogs.erb +54 -0
- data/lib/sidekiq_unique_jobs/web/views/locks.erb +1 -1
- metadata +25 -9
- data/lib/sidekiq_unique_jobs/profiler.rb +0 -55
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 791ef31e7978f706fa00198280417ebb4f230d8bf5225f5e914cc8092b347b1e
|
4
|
+
data.tar.gz: b085a1ac1974d74a73248cd931eff8a387f5017e06e66a863844616e6ac29184
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 71cb338220b1d3bb77b9834f5c4fe42c48532b95bf794bbad286382f1a12297b8411276b1e802c921203a345fd25151feb22b63dd284d111eea2018f29ec4ea1
|
7
|
+
data.tar.gz: 7cca4aa29025359b33cd914b402c446a4e9b03cb98f3b0839732d2bae03b2503f5d3425d57cd2efbd4705a438c334b36f8dd2b60a279cf8e505b188c4a3003bc
|
data/CHANGELOG.md
CHANGED
@@ -2,7 +2,84 @@
|
|
2
2
|
|
3
3
|
## [Unreleased](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/HEAD)
|
4
4
|
|
5
|
-
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/
|
5
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0...HEAD)
|
6
|
+
|
7
|
+
|
8
|
+
|
9
|
+
## [v7.0.0](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0) (2021-01-20)
|
10
|
+
|
11
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.25...v7.0.0)
|
12
|
+
|
13
|
+
**Implemented enhancements:**
|
14
|
+
|
15
|
+
- Give user full control over adding middleware [\#566](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/566) ([mhenrixon](https://github.com/mhenrixon))
|
16
|
+
|
17
|
+
**Fixed bugs:**
|
18
|
+
|
19
|
+
- Race condition in ruby reaper [\#559](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/559)
|
20
|
+
- Fix until and while executed and improve documentation [\#397](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/397)
|
21
|
+
- Fix 532 [\#549](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/549) ([mhenrixon](https://github.com/mhenrixon))
|
22
|
+
|
23
|
+
**Closed issues:**
|
24
|
+
|
25
|
+
- Rejected jobs are still displayed as 'Queued' with gem 'sidekiq-status' on /sidekiq/statuses [\#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564)
|
26
|
+
- Is it possible to have a :until\_executed lock with an expiration time? [\#524](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/524)
|
27
|
+
|
28
|
+
## [v7.0.0.beta29](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta29) (2021-01-16)
|
29
|
+
|
30
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta28...v7.0.0.beta29)
|
31
|
+
|
32
|
+
**Implemented enhancements:**
|
33
|
+
|
34
|
+
- Fix coverage reporting and add coverage [\#565](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/565) ([mhenrixon](https://github.com/mhenrixon))
|
35
|
+
|
36
|
+
**Fixed bugs:**
|
37
|
+
|
38
|
+
- Ruby Reaper active check incorrect [\#557](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/557)
|
39
|
+
- Routes with authentication should work with web [\#562](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/562) ([mhenrixon](https://github.com/mhenrixon))
|
40
|
+
|
41
|
+
**Closed issues:**
|
42
|
+
|
43
|
+
- Can't add the lock tab o nthe website when there is authentication through devise [\#561](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/561)
|
44
|
+
|
45
|
+
## [v7.0.0.beta28](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta28) (2021-01-07)
|
46
|
+
|
47
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta27...v7.0.0.beta28)
|
48
|
+
|
49
|
+
**Fixed bugs:**
|
50
|
+
|
51
|
+
- lock\_args does not work when you define the lock\_args argument and default lock\_args function at the same time. [\#548](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/548)
|
52
|
+
- incorrect `:while\_executing` behavior [\#432](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/432)
|
53
|
+
- Fix race condition to avoid reaping active jobs [\#563](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/563) ([mhenrixon](https://github.com/mhenrixon))
|
54
|
+
- Fix active check for the worker hash [\#558](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/558) ([mhenrixon](https://github.com/mhenrixon))
|
55
|
+
- Prefer lock\_prefix not unique\_prefix [\#554](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/554) ([mhenrixon](https://github.com/mhenrixon))
|
56
|
+
- Fix issue 432 [\#552](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/552) ([mhenrixon](https://github.com/mhenrixon))
|
57
|
+
|
58
|
+
## [v7.0.0.beta27](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta27) (2020-11-03)
|
59
|
+
|
60
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v7.0.0.beta26...v7.0.0.beta27)
|
61
|
+
|
62
|
+
**Implemented enhancements:**
|
63
|
+
|
64
|
+
- Adds coverage for regression purposes [\#550](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/550) ([mhenrixon](https://github.com/mhenrixon))
|
65
|
+
|
66
|
+
**Fixed bugs:**
|
67
|
+
|
68
|
+
- Rename lock\_args to lock\_args\_method [\#551](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/551) ([mhenrixon](https://github.com/mhenrixon))
|
69
|
+
|
70
|
+
**Closed issues:**
|
71
|
+
|
72
|
+
- Documentation incorrect for `delete\_by\_digest` [\#547](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/547)
|
73
|
+
- Locked jobs after kill -9 with while\_executing lock [\#546](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/546)
|
74
|
+
|
75
|
+
## [v7.0.0.beta26](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta26) (2020-10-28)
|
76
|
+
|
77
|
+
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.25...v7.0.0.beta26)
|
78
|
+
|
79
|
+
**Implemented enhancements:**
|
80
|
+
|
81
|
+
- How to disable Reaper [\#543](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/543)
|
82
|
+
- Allow disabling of reaper [\#544](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/544) ([mhenrixon](https://github.com/mhenrixon))
|
6
83
|
|
7
84
|
**Merged pull requests:**
|
8
85
|
|
@@ -88,6 +165,7 @@
|
|
88
165
|
**Fixed bugs:**
|
89
166
|
|
90
167
|
- Prevent indefinitely looping entries [\#516](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/516) ([mhenrixon](https://github.com/mhenrixon))
|
168
|
+
- Don't bypass lock\_args if arguments are empty [\#511](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/511) ([nhippenmeyer](https://github.com/nhippenmeyer))
|
91
169
|
|
92
170
|
**Closed issues:**
|
93
171
|
|
@@ -169,7 +247,6 @@
|
|
169
247
|
- Brpoplpush::RedisScript::LuaError: WRONGTYPE Operation against a key holding the wrong kind of value [\#491](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/491)
|
170
248
|
- Lua script bug [\#489](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/489)
|
171
249
|
- Reaper will delete locks for running jobs [\#488](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/488)
|
172
|
-
- Fix access to hash members [\#496](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/496) ([mhenrixon](https://github.com/mhenrixon))
|
173
250
|
- Fix cursor assignment [\#494](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/494) ([mhenrixon](https://github.com/mhenrixon))
|
174
251
|
- Prevent reaping of active jobs [\#493](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/493) ([mhenrixon](https://github.com/mhenrixon))
|
175
252
|
|
@@ -187,6 +264,7 @@
|
|
187
264
|
|
188
265
|
**Fixed bugs:**
|
189
266
|
|
267
|
+
- Fix access to hash members [\#496](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/496) ([mhenrixon](https://github.com/mhenrixon))
|
190
268
|
- Use thread-safe digest creation mechanism [\#484](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/484) ([mhenrixon](https://github.com/mhenrixon))
|
191
269
|
|
192
270
|
## [v7.0.0.beta13](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v7.0.0.beta13) (2020-03-26)
|
@@ -422,6 +500,7 @@
|
|
422
500
|
|
423
501
|
**Fixed bugs:**
|
424
502
|
|
503
|
+
- Include redis directory [\#430](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/430) ([mhenrixon](https://github.com/mhenrixon))
|
425
504
|
- Allow Sidekiq::Context to be used for logging [\#429](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/429) ([mhenrixon](https://github.com/mhenrixon))
|
426
505
|
- Fix sidekiq develop [\#426](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/426) ([mhenrixon](https://github.com/mhenrixon))
|
427
506
|
- Reap as many orphans as advertised [\#403](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/403) ([mhenrixon](https://github.com/mhenrixon))
|
@@ -448,7 +527,6 @@
|
|
448
527
|
|
449
528
|
- Duplicate job was pushed \( v6.0.13 \) [\#414](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/414)
|
450
529
|
- Constant SidekiqUniqueJobs::Web::Digests not found [\#396](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/396)
|
451
|
-
- Include redis directory [\#430](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/430) ([mhenrixon](https://github.com/mhenrixon))
|
452
530
|
- Fix v6 [\#428](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/428) ([mhenrixon](https://github.com/mhenrixon))
|
453
531
|
|
454
532
|
**Closed issues:**
|
@@ -569,7 +647,7 @@
|
|
569
647
|
|
570
648
|
- Version 5: Job ID Hash Entries Not Removed if Unique Key Expires [\#346](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/346)
|
571
649
|
- Move the lpush last [\#354](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/354) ([mhenrixon](https://github.com/mhenrixon))
|
572
|
-
-
|
650
|
+
- Fix the broken expiration [\#316](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/316) ([mhenrixon](https://github.com/mhenrixon))
|
573
651
|
|
574
652
|
**Closed issues:**
|
575
653
|
|
@@ -591,6 +669,7 @@
|
|
591
669
|
**Fixed bugs:**
|
592
670
|
|
593
671
|
- Always Remove Job ID from uniquejobs Hash [\#347](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/347) ([chadrschroeder](https://github.com/chadrschroeder))
|
672
|
+
- Convert expiration time to integer [\#330](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/330) ([dareddov](https://github.com/dareddov))
|
594
673
|
|
595
674
|
**Closed issues:**
|
596
675
|
|
@@ -631,10 +710,6 @@
|
|
631
710
|
|
632
711
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.3...v6.0.4)
|
633
712
|
|
634
|
-
**Fixed bugs:**
|
635
|
-
|
636
|
-
- Fix the broken expiration [\#316](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/316) ([mhenrixon](https://github.com/mhenrixon))
|
637
|
-
|
638
713
|
**Closed issues:**
|
639
714
|
|
640
715
|
- Question about until\_timeout with 6.0.0 [\#303](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/303)
|
@@ -682,6 +757,7 @@
|
|
682
757
|
|
683
758
|
- Dead jobs [\#308](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/308) ([mhenrixon](https://github.com/mhenrixon))
|
684
759
|
- Fix require path for sidekiq\_unique\_jobs/web [\#305](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/305) ([soundasleep](https://github.com/soundasleep))
|
760
|
+
- Update README.md [\#300](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/300) ([pirj](https://github.com/pirj))
|
685
761
|
|
686
762
|
## [v6.0.0](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.0) (2018-07-27)
|
687
763
|
|
@@ -700,7 +776,6 @@
|
|
700
776
|
**Merged pull requests:**
|
701
777
|
|
702
778
|
- Add more details about testing uniqueness [\#301](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/301) ([mhenrixon](https://github.com/mhenrixon))
|
703
|
-
- Update README.md [\#300](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/300) ([pirj](https://github.com/pirj))
|
704
779
|
|
705
780
|
## [v6.0.0.rc7](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.0.rc7) (2018-07-23)
|
706
781
|
|
@@ -745,7 +820,6 @@
|
|
745
820
|
**Fixed bugs:**
|
746
821
|
|
747
822
|
- bundle exec jobs console does not work [\#253](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/253)
|
748
|
-
- Rename command line binary [\#287](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/287) ([mhenrixon](https://github.com/mhenrixon))
|
749
823
|
|
750
824
|
## [v6.0.0.rc4](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.0.rc4) (2018-06-30)
|
751
825
|
|
@@ -756,6 +830,10 @@
|
|
756
830
|
- Prepare for v6 [\#286](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/286) ([mhenrixon](https://github.com/mhenrixon))
|
757
831
|
- Only unlock not delete [\#285](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/285) ([mhenrixon](https://github.com/mhenrixon))
|
758
832
|
|
833
|
+
**Fixed bugs:**
|
834
|
+
|
835
|
+
- Rename command line binary [\#287](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/287) ([mhenrixon](https://github.com/mhenrixon))
|
836
|
+
|
759
837
|
## [v6.0.0.rc3](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.0.rc3) (2018-06-29)
|
760
838
|
|
761
839
|
[Full Changelog](https://github.com/mhenrixon/sidekiq-unique-jobs/compare/v6.0.0.rc2...v6.0.0.rc3)
|
@@ -819,7 +897,6 @@
|
|
819
897
|
**Implemented enhancements:**
|
820
898
|
|
821
899
|
- Code smells [\#275](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/275) ([mhenrixon](https://github.com/mhenrixon))
|
822
|
-
- Reject while scheduling [\#273](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/273) ([mhenrixon](https://github.com/mhenrixon))
|
823
900
|
- Improve testing [\#272](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/272) ([mhenrixon](https://github.com/mhenrixon))
|
824
901
|
|
825
902
|
## [v6.0.0.beta](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.0.beta) (2018-06-17)
|
@@ -828,6 +905,7 @@
|
|
828
905
|
|
829
906
|
**Implemented enhancements:**
|
830
907
|
|
908
|
+
- Reject while scheduling [\#273](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/273) ([mhenrixon](https://github.com/mhenrixon))
|
831
909
|
- Until and while executing [\#271](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/271) ([mhenrixon](https://github.com/mhenrixon))
|
832
910
|
- Solidify master [\#270](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/270) ([mhenrixon](https://github.com/mhenrixon))
|
833
911
|
- Minor adjustments [\#268](https://github.com/mhenrixon/sidekiq-unique-jobs/pull/268) ([mhenrixon](https://github.com/mhenrixon))
|
data/README.md
CHANGED
@@ -3,9 +3,12 @@
|
|
3
3
|
<!-- MarkdownTOC -->
|
4
4
|
|
5
5
|
- [Introduction](#introduction)
|
6
|
-
- [
|
7
|
-
- [Installation](#installation)
|
6
|
+
- [Usage](#usage)
|
7
|
+
- [Installation](#installation)
|
8
|
+
- [Add the middleware](#add-the-middleware)
|
9
|
+
- [Your first worker](#your-first-worker)
|
8
10
|
- [Support Me](#support-me)
|
11
|
+
- [Requirements](#requirements)
|
9
12
|
- [General Information](#general-information)
|
10
13
|
- [Global Configuration](#global-configuration)
|
11
14
|
- [debug_lua](#debug_lua)
|
@@ -18,9 +21,11 @@
|
|
18
21
|
- [reaper_count](#reaper_count)
|
19
22
|
- [reaper_interval](#reaper_interval)
|
20
23
|
- [reaper_timeout](#reaper_timeout)
|
21
|
-
- [
|
24
|
+
- [lock_prefix](#lock_prefix)
|
22
25
|
- [lock_info](#lock_info)
|
23
26
|
- [Worker Configuration](#worker-configuration)
|
27
|
+
- [lock_info](#lock_info-1)
|
28
|
+
- [lock_prefix](#lock_prefix-1)
|
24
29
|
- [lock_ttl](#lock_ttl-1)
|
25
30
|
- [lock_timeout](#lock_timeout-1)
|
26
31
|
- [unique_across_queues](#unique_across_queues)
|
@@ -39,13 +44,15 @@
|
|
39
44
|
- [replace](#replace)
|
40
45
|
- [Reschedule](#reschedule)
|
41
46
|
- [Custom Strategies](#custom-strategies)
|
42
|
-
- [Usage](#usage)
|
47
|
+
- [Usage](#usage-1)
|
43
48
|
- [Finer Control over Uniqueness](#finer-control-over-uniqueness)
|
44
49
|
- [After Unlock Callback](#after-unlock-callback)
|
45
50
|
- [Logging](#logging)
|
46
51
|
- [Cleanup Dead Locks](#cleanup-dead-locks)
|
47
52
|
- [Other Sidekiq gems](#other-sidekiq-gems)
|
53
|
+
- [apartment-sidekiq](#apartment-sidekiq)
|
48
54
|
- [sidekiq-global_id](#sidekiq-global_id)
|
55
|
+
- [sidekiq-status](#sidekiq-status)
|
49
56
|
- [Debugging](#debugging)
|
50
57
|
- [Sidekiq Web](#sidekiq-web)
|
51
58
|
- [Show Locks](#show-locks)
|
@@ -61,30 +68,19 @@
|
|
61
68
|
|
62
69
|
## Introduction
|
63
70
|
|
64
|
-
The
|
71
|
+
This gem adds unique constraints to the sidekiq queues. The uniqueness is achieved by acquiring locks for a hash of a queue name, a worker class, and job's arguments. By default, only one lock for a given hash can be acquired. What happens when a lock can't be acquired is governed by a chosen `on_conflict`strategy.
|
65
72
|
|
66
73
|
This is the documentation for the master branch. You can find the documentation for each release by navigating to its tag.
|
67
74
|
|
68
75
|
Here are links to some of the old versions
|
69
76
|
|
70
|
-
- [v6.0.
|
77
|
+
- [v6.0.25](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v6.0.25)
|
71
78
|
- [v5.0.10](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v5.0.10)
|
72
79
|
- [v4.0.18](https://github.com/mhenrixon/sidekiq-unique-jobs/tree/v4.0.18)
|
73
80
|
|
74
|
-
##
|
75
|
-
|
76
|
-
- Sidekiq `>= 4.0` (`>= 5.2` recommended)
|
77
|
-
- Ruby:
|
78
|
-
- MRI `>= 2.3` (`>= 2.5` recommended)
|
79
|
-
- JRuby `>= 9.0` (`>= 9.2` recommended)
|
80
|
-
- Truffleruby
|
81
|
-
- Redis Server `>= 3.0.2` (`>= 3.2` recommended)
|
82
|
-
- [ActiveJob officially not supported][48]
|
83
|
-
- [redis-namespace officially not supported][49]
|
84
|
-
|
85
|
-
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
81
|
+
## Usage
|
86
82
|
|
87
|
-
|
83
|
+
### Installation
|
88
84
|
|
89
85
|
Add this line to your application's Gemfile:
|
90
86
|
|
@@ -98,16 +94,83 @@ And then execute:
|
|
98
94
|
bundle
|
99
95
|
```
|
100
96
|
|
101
|
-
|
97
|
+
### Add the middleware
|
102
98
|
|
103
|
-
|
104
|
-
|
99
|
+
Before v7, the middleware was configured automatically. Since some people reported issues with other gems (see [Other Sidekiq Gems](#other-sidekiq-gems)) it was decided to give full control over to the user.
|
100
|
+
|
101
|
+
*NOTE* if you want to use the reaper you also need to configure the server middleware.
|
102
|
+
|
103
|
+
[A full and hopefully working example](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12)
|
104
|
+
|
105
|
+
```ruby
|
106
|
+
Sidekiq.configure_server do |config|
|
107
|
+
config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
|
108
|
+
|
109
|
+
config.client_middleware do |chain|
|
110
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
111
|
+
end
|
112
|
+
|
113
|
+
config.server_middleware do |chain|
|
114
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
115
|
+
end
|
116
|
+
|
117
|
+
SidekiqUniqueJobs::Server.configure(config)
|
118
|
+
end
|
119
|
+
|
120
|
+
Sidekiq.configure_client do |config|
|
121
|
+
config.redis = { url: ENV["REDIS_URL"], driver: :hiredis }
|
122
|
+
|
123
|
+
config.client_middleware do |chain|
|
124
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
125
|
+
end
|
126
|
+
end
|
105
127
|
```
|
106
128
|
|
129
|
+
### Your first worker
|
130
|
+
|
131
|
+
```ruby
|
132
|
+
# frozen_string_literal: true
|
133
|
+
|
134
|
+
class UntilExecutedWorker
|
135
|
+
include Sidekiq::Worker
|
136
|
+
|
137
|
+
sidekiq_options queue: :special,
|
138
|
+
retry: false,
|
139
|
+
lock: :until_executed,
|
140
|
+
lock_info: true,
|
141
|
+
lock_timeout: 0,
|
142
|
+
lock_prefix: "special",
|
143
|
+
lock_ttl: 0,
|
144
|
+
lock_limit: 5
|
145
|
+
|
146
|
+
def perform
|
147
|
+
logger.info("cowboy")
|
148
|
+
sleep(1) # hardcore processing
|
149
|
+
logger.info("beebop")
|
150
|
+
end
|
151
|
+
end
|
152
|
+
|
153
|
+
```
|
154
|
+
|
155
|
+
You can read more about the worker configuration in [Worker Configuration](#worker-configuration) below.
|
156
|
+
|
107
157
|
## Support Me
|
108
158
|
|
109
159
|
Want to show me some ❤️ for the hard work I do on this gem? You can use the following PayPal link: [https://paypal.me/mhenrixon1](https://paypal.me/mhenrixon1). Any amount is welcome and let me tell you it feels good to be appreciated. Even a dollar makes me super excited about all of this.
|
110
160
|
|
161
|
+
## Requirements
|
162
|
+
|
163
|
+
- Sidekiq `>= 5.0` (`>= 5.2` recommended)
|
164
|
+
- Ruby:
|
165
|
+
- MRI `>= 2.5` (`>= 2.6` recommended)
|
166
|
+
- JRuby `>= 9.0` (`>= 9.2` recommended)
|
167
|
+
- Truffleruby
|
168
|
+
- Redis Server `>= 3.2` (`>= 5.0` recommended)
|
169
|
+
- [ActiveJob officially not supported][48]
|
170
|
+
- [redis-namespace officially not supported][49]
|
171
|
+
|
172
|
+
See [Sidekiq requirements][24] for detailed requirements of Sidekiq itself (be sure to check the right sidekiq version).
|
173
|
+
|
111
174
|
## General Information
|
112
175
|
|
113
176
|
See [Interaction w/ Sidekiq](https://github.com/mhenrixon/sidekiq-unique-jobs/wiki/How-this-gem-interacts-with-Sidekiq) on how the gem interacts with Sidekiq.
|
@@ -235,10 +298,10 @@ SidekiqUniqueJobs.config.reaper_timeout #=> 10
|
|
235
298
|
|
236
299
|
The number of seconds to wait for the reaper to finish before raising a TimeoutError. This is done to ensure that the next time we reap isn't getting stuck due to the previous process already running.
|
237
300
|
|
238
|
-
###
|
301
|
+
### lock_prefix
|
239
302
|
|
240
303
|
```ruby
|
241
|
-
SidekiqUniqueJobs.config.
|
304
|
+
SidekiqUniqueJobs.config.lock_prefix #=> "uniquejobs"
|
242
305
|
```
|
243
306
|
|
244
307
|
Use if you want a different key prefix for the keys in redis.
|
@@ -253,11 +316,27 @@ Using lock info will create an additional key for the lock with a json object co
|
|
253
316
|
|
254
317
|
## Worker Configuration
|
255
318
|
|
319
|
+
### lock_info
|
320
|
+
|
321
|
+
Lock info gathers information about a specific lock. It collects things like which `lock_args` where used to compute the `lock_digest` that is used for maintaining uniqueness.
|
322
|
+
|
323
|
+
```ruby
|
324
|
+
sidekiq_options lock_info: false # this is the default, set to true to turn on
|
325
|
+
```
|
326
|
+
|
327
|
+
### lock_prefix
|
328
|
+
|
329
|
+
Use if you want a different key prefix for the keys in redis.
|
330
|
+
|
331
|
+
```ruby
|
332
|
+
sidekiq_options lock_prefix: "uniquejobs" # this is the default value
|
333
|
+
```
|
334
|
+
|
256
335
|
### lock_ttl
|
257
336
|
|
258
337
|
Lock TTL decides how long to wait after the job has been successfully processed before making it possible to reuse that lock.
|
259
338
|
|
260
|
-
|
339
|
+
Starting from `v7` the expiration will take place when the job is pushed to the queue.
|
261
340
|
|
262
341
|
```ruby
|
263
342
|
sidekiq_options lock_ttl: nil # default - don't expire keys
|
@@ -278,6 +357,8 @@ sidekiq_options lock_timeout: nil # lock indefinitely, this process won't contin
|
|
278
357
|
|
279
358
|
This configuration option is slightly misleading. It doesn't disregard the queue on other jobs. Just on itself, this means that a worker that might schedule jobs into multiple queues will be able to have uniqueness enforced on all queues it is pushed to.
|
280
359
|
|
360
|
+
This is mainly intended for `Worker.set(queue: :another).perform_async`.
|
361
|
+
|
281
362
|
```ruby
|
282
363
|
class Worker
|
283
364
|
include Sidekiq::Worker
|
@@ -428,7 +509,7 @@ The last one is log which can be be used with the lock `UntilExecuted` and `Unti
|
|
428
509
|
It is possible for locks to have different conflict strategy for the client and server. This is useful for `:until_and_while_executing`.
|
429
510
|
|
430
511
|
```ruby
|
431
|
-
sidekiq_options lock: :until_and_while_executing,
|
512
|
+
sidekiq_options lock: :until_and_while_executing,
|
432
513
|
on_conflict: { client: :log, server: :reject }
|
433
514
|
```
|
434
515
|
|
@@ -536,7 +617,7 @@ The method or the proc can return a modified version of args without the transie
|
|
536
617
|
class UniqueJobWithFilterMethod
|
537
618
|
include Sidekiq::Worker
|
538
619
|
sidekiq_options lock: :until_and_while_executing,
|
539
|
-
|
620
|
+
lock_args_method: :lock_args # this is default and will be used if such a method is defined
|
540
621
|
|
541
622
|
def self.lock_args(args)
|
542
623
|
[ args[0], args[2][:type] ]
|
@@ -549,7 +630,7 @@ end
|
|
549
630
|
class UniqueJobWithFilterProc
|
550
631
|
include Sidekiq::Worker
|
551
632
|
sidekiq_options lock: :until_executed,
|
552
|
-
|
633
|
+
lock_args_method: ->(args) { [ args.first ] }
|
553
634
|
|
554
635
|
...
|
555
636
|
|
@@ -561,7 +642,7 @@ It is possible to ensure different types of unique args based on context. I can'
|
|
561
642
|
```ruby
|
562
643
|
class UniqueJobWithFilterMethod
|
563
644
|
include Sidekiq::Worker
|
564
|
-
sidekiq_options lock: :until_and_while_executing,
|
645
|
+
sidekiq_options lock: :until_and_while_executing, lock_args_method: :lock_args
|
565
646
|
|
566
647
|
def self.lock_args(args)
|
567
648
|
if Sidekiq::ProcessSet.new.size > 1
|
@@ -622,8 +703,8 @@ For sidekiq versions before 5.1 a `sidekiq_retries_exhausted` block is required
|
|
622
703
|
```ruby
|
623
704
|
class MyWorker
|
624
705
|
sidekiq_retries_exhausted do |msg, _ex|
|
625
|
-
digest = msg['
|
626
|
-
SidekiqUniqueJobs::Digests.delete_by_digest(digest) if digest
|
706
|
+
digest = msg['lock_digest']
|
707
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
627
708
|
end
|
628
709
|
end
|
629
710
|
```
|
@@ -633,18 +714,38 @@ Starting in v5.1, Sidekiq can also fire a global callback when a job dies:
|
|
633
714
|
```ruby
|
634
715
|
Sidekiq.configure_server do |config|
|
635
716
|
config.death_handlers << ->(job, _ex) do
|
636
|
-
digest = job['
|
637
|
-
SidekiqUniqueJobs::Digests.delete_by_digest(digest) if digest
|
717
|
+
digest = job['lock_digest']
|
718
|
+
SidekiqUniqueJobs::Digests.new.delete_by_digest(digest) if digest
|
638
719
|
end
|
639
720
|
end
|
640
721
|
```
|
641
722
|
|
642
723
|
### Other Sidekiq gems
|
643
724
|
|
725
|
+
#### apartment-sidekiq
|
726
|
+
|
727
|
+
It was reported in [#536](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/536) that the order of the Sidekiq middleware needs to be as follows.
|
728
|
+
|
729
|
+
```ruby
|
730
|
+
Sidekiq.client_middleware do |chain|
|
731
|
+
chain.add Apartment::Sidekiq::Middleware::Client
|
732
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
733
|
+
end
|
734
|
+
|
735
|
+
Sidekiq.server_middleware do |chain|
|
736
|
+
chain.add Apartment::Sidekiq::Middleware::Server
|
737
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
738
|
+
end
|
739
|
+
```
|
740
|
+
|
741
|
+
The reason being that this gem needs to be configured AFTER the apartment gem or the apartment will not be able to be considered for uniqueness
|
742
|
+
|
644
743
|
#### sidekiq-global_id
|
645
744
|
|
646
745
|
It was reported in [#235](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/235) that the order of the Sidekiq middleware needs to be as follows.
|
647
746
|
|
747
|
+
For a working setup check the following [file](https://github.com/mhenrixon/sidekiq-unique-jobs/blob/master/myapp/config/sidekiq.rb#L12).
|
748
|
+
|
648
749
|
```ruby
|
649
750
|
Sidekiq.client_middleware do |chain|
|
650
751
|
chain.add Sidekiq::GlobalId::ClientMiddleware
|
@@ -652,12 +753,30 @@ Sidekiq.client_middleware do |chain|
|
|
652
753
|
end
|
653
754
|
|
654
755
|
Sidekiq.server_middleware do |chain|
|
655
|
-
chain.add SidekiqUniqueJobs::Middleware::Server
|
656
756
|
chain.add Sidekiq::GlobalId::ServerMiddleware
|
757
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
657
758
|
end
|
658
759
|
```
|
659
760
|
|
660
|
-
|
761
|
+
The reason for this is that the global id needs to be set before the unique jobs middleware runs. Otherwise that won't be available for uniqueness.
|
762
|
+
|
763
|
+
#### sidekiq-status
|
764
|
+
|
765
|
+
It was reported in [#564](https://github.com/mhenrixon/sidekiq-unique-jobs/issues/564) that the order of the middleware needs to be as follows.
|
766
|
+
|
767
|
+
```ruby
|
768
|
+
Sidekiq.client_middleware do |chain|
|
769
|
+
chain.add Sidekiq::Status::ClientMiddleware, expiration: 10.minutes
|
770
|
+
chain.add SidekiqUniqueJobs::Middleware::Client
|
771
|
+
end
|
772
|
+
|
773
|
+
Sidekiq.server_middleware do |chain|
|
774
|
+
chain.add SidekiqUniqueJobs::Middleware::Server
|
775
|
+
chain.add Sidekiq::Status::ServerMiddleware, expiration: 10.minutes
|
776
|
+
end
|
777
|
+
```
|
778
|
+
|
779
|
+
The reason for this is that if a job is duplicated it shouldn't end up with the status middleware at all. Status is just a monitor so to prevent clashes, leftovers and ensure cleanup. The status middleware should run after uniqueness on client and before on server. This will lead to less surprises.
|
661
780
|
|
662
781
|
## Debugging
|
663
782
|
|
@@ -728,7 +847,6 @@ If you are not using RSpec (a lot of people prefer minitest or test unit) you ca
|
|
728
847
|
assert SidekiqUniqueJobs.validate_worker!(BadWorker.get_sidekiq_options)
|
729
848
|
```
|
730
849
|
|
731
|
-
|
732
850
|
### Uniqueness
|
733
851
|
|
734
852
|
This has been probably the most confusing part of this gem. People get really confused with how unreliable the unique jobs have been. I there for decided to do what Mike is doing for sidekiq enterprise. Read the section about unique jobs: [Enterprise unique jobs][]
|