sidekiq 4.2.4 → 6.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (134) hide show
  1. checksums.yaml +5 -5
  2. data/Changes.md +445 -0
  3. data/LICENSE +1 -1
  4. data/README.md +21 -34
  5. data/bin/sidekiq +26 -2
  6. data/bin/sidekiqload +28 -38
  7. data/bin/sidekiqmon +8 -0
  8. data/lib/generators/sidekiq/templates/worker_spec.rb.erb +1 -1
  9. data/lib/generators/sidekiq/templates/worker_test.rb.erb +2 -2
  10. data/lib/generators/sidekiq/worker_generator.rb +21 -13
  11. data/lib/sidekiq/api.rb +347 -213
  12. data/lib/sidekiq/cli.rb +221 -212
  13. data/lib/sidekiq/client.rb +75 -52
  14. data/lib/sidekiq/delay.rb +41 -0
  15. data/lib/sidekiq/exception_handler.rb +12 -16
  16. data/lib/sidekiq/extensions/action_mailer.rb +13 -22
  17. data/lib/sidekiq/extensions/active_record.rb +13 -10
  18. data/lib/sidekiq/extensions/class_methods.rb +14 -11
  19. data/lib/sidekiq/extensions/generic_proxy.rb +10 -4
  20. data/lib/sidekiq/fetch.rb +38 -31
  21. data/lib/sidekiq/job_logger.rb +63 -0
  22. data/lib/sidekiq/job_retry.rb +263 -0
  23. data/lib/sidekiq/launcher.rb +169 -70
  24. data/lib/sidekiq/logger.rb +166 -0
  25. data/lib/sidekiq/manager.rb +17 -20
  26. data/lib/sidekiq/middleware/chain.rb +15 -5
  27. data/lib/sidekiq/middleware/i18n.rb +5 -7
  28. data/lib/sidekiq/monitor.rb +133 -0
  29. data/lib/sidekiq/paginator.rb +18 -14
  30. data/lib/sidekiq/processor.rb +161 -70
  31. data/lib/sidekiq/rails.rb +30 -73
  32. data/lib/sidekiq/redis_connection.rb +67 -20
  33. data/lib/sidekiq/scheduled.rb +61 -35
  34. data/lib/sidekiq/sd_notify.rb +149 -0
  35. data/lib/sidekiq/systemd.rb +24 -0
  36. data/lib/sidekiq/testing/inline.rb +2 -1
  37. data/lib/sidekiq/testing.rb +54 -26
  38. data/lib/sidekiq/util.rb +48 -15
  39. data/lib/sidekiq/version.rb +2 -1
  40. data/lib/sidekiq/web/action.rb +15 -15
  41. data/lib/sidekiq/web/application.rb +112 -89
  42. data/lib/sidekiq/web/csrf_protection.rb +180 -0
  43. data/lib/sidekiq/web/helpers.rb +153 -73
  44. data/lib/sidekiq/web/router.rb +27 -19
  45. data/lib/sidekiq/web.rb +64 -109
  46. data/lib/sidekiq/worker.rb +164 -41
  47. data/lib/sidekiq.rb +86 -60
  48. data/sidekiq.gemspec +24 -22
  49. data/web/assets/images/apple-touch-icon.png +0 -0
  50. data/web/assets/javascripts/application.js +25 -27
  51. data/web/assets/javascripts/dashboard.js +34 -38
  52. data/web/assets/stylesheets/application-dark.css +160 -0
  53. data/web/assets/stylesheets/application-rtl.css +246 -0
  54. data/web/assets/stylesheets/application.css +402 -12
  55. data/web/assets/stylesheets/bootstrap-rtl.min.css +9 -0
  56. data/web/assets/stylesheets/bootstrap.css +2 -2
  57. data/web/locales/ar.yml +81 -0
  58. data/web/locales/de.yml +14 -2
  59. data/web/locales/en.yml +4 -0
  60. data/web/locales/es.yml +4 -3
  61. data/web/locales/fa.yml +80 -0
  62. data/web/locales/fr.yml +3 -3
  63. data/web/locales/he.yml +79 -0
  64. data/web/locales/ja.yml +9 -4
  65. data/web/locales/lt.yml +83 -0
  66. data/web/locales/pl.yml +4 -4
  67. data/web/locales/ru.yml +4 -0
  68. data/web/locales/ur.yml +80 -0
  69. data/web/locales/vi.yml +83 -0
  70. data/web/views/_footer.erb +5 -2
  71. data/web/views/_job_info.erb +3 -2
  72. data/web/views/_nav.erb +4 -18
  73. data/web/views/_paging.erb +1 -1
  74. data/web/views/busy.erb +57 -19
  75. data/web/views/dashboard.erb +3 -3
  76. data/web/views/dead.erb +2 -2
  77. data/web/views/layout.erb +13 -2
  78. data/web/views/morgue.erb +19 -12
  79. data/web/views/queue.erb +22 -12
  80. data/web/views/queues.erb +13 -3
  81. data/web/views/retries.erb +22 -13
  82. data/web/views/retry.erb +3 -3
  83. data/web/views/scheduled.erb +7 -4
  84. metadata +42 -194
  85. data/.github/contributing.md +0 -32
  86. data/.github/issue_template.md +0 -4
  87. data/.gitignore +0 -12
  88. data/.travis.yml +0 -12
  89. data/3.0-Upgrade.md +0 -70
  90. data/4.0-Upgrade.md +0 -53
  91. data/COMM-LICENSE +0 -95
  92. data/Ent-Changes.md +0 -146
  93. data/Gemfile +0 -29
  94. data/Pro-2.0-Upgrade.md +0 -138
  95. data/Pro-3.0-Upgrade.md +0 -44
  96. data/Pro-Changes.md +0 -585
  97. data/Rakefile +0 -9
  98. data/bin/sidekiqctl +0 -99
  99. data/code_of_conduct.md +0 -50
  100. data/lib/sidekiq/core_ext.rb +0 -106
  101. data/lib/sidekiq/logging.rb +0 -106
  102. data/lib/sidekiq/middleware/server/active_record.rb +0 -13
  103. data/lib/sidekiq/middleware/server/logging.rb +0 -40
  104. data/lib/sidekiq/middleware/server/retry_jobs.rb +0 -205
  105. data/test/config.yml +0 -9
  106. data/test/env_based_config.yml +0 -11
  107. data/test/fake_env.rb +0 -1
  108. data/test/fixtures/en.yml +0 -2
  109. data/test/helper.rb +0 -75
  110. data/test/test_actors.rb +0 -138
  111. data/test/test_api.rb +0 -528
  112. data/test/test_cli.rb +0 -418
  113. data/test/test_client.rb +0 -266
  114. data/test/test_exception_handler.rb +0 -56
  115. data/test/test_extensions.rb +0 -127
  116. data/test/test_fetch.rb +0 -50
  117. data/test/test_launcher.rb +0 -95
  118. data/test/test_logging.rb +0 -35
  119. data/test/test_manager.rb +0 -50
  120. data/test/test_middleware.rb +0 -158
  121. data/test/test_processor.rb +0 -235
  122. data/test/test_rails.rb +0 -22
  123. data/test/test_redis_connection.rb +0 -132
  124. data/test/test_retry.rb +0 -326
  125. data/test/test_retry_exhausted.rb +0 -149
  126. data/test/test_scheduled.rb +0 -115
  127. data/test/test_scheduling.rb +0 -58
  128. data/test/test_sidekiq.rb +0 -107
  129. data/test/test_testing.rb +0 -143
  130. data/test/test_testing_fake.rb +0 -357
  131. data/test/test_testing_inline.rb +0 -94
  132. data/test/test_util.rb +0 -13
  133. data/test/test_web.rb +0 -726
  134. data/test/test_web_helpers.rb +0 -54
data/Ent-Changes.md DELETED
@@ -1,146 +0,0 @@
1
- Sidekiq Enterprise Changelog
2
- =======================
3
-
4
- Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
5
-
6
- 1.3.2
7
- -------------
8
-
9
- - Upgrade encryption to use OpenSSL's more secure GCM mode. [#3060]
10
-
11
- 1.3.1
12
- -------------
13
-
14
- - Fix multi-process memory monitoring on CentOS 6.x [#3063]
15
- - Polish the new encryption feature a bit.
16
-
17
- 1.3.0
18
- -------------
19
-
20
- - **BETA** [New encryption feature](https://github.com/mperham/sidekiq/wiki/Ent-Encryption)
21
- which automatically encrypts the last argument of a Worker, aka the secret bag.
22
-
23
- 1.2.4
24
- -------------
25
-
26
- - Fix issue causing some minutely jobs to execute every other minute.
27
- - Log a warning if slow periodic processing causes us to miss a clock tick.
28
-
29
- 1.2.3
30
- -------------
31
-
32
- - Periodic jobs could stop executing until process restart if Redis goes down [#3047]
33
-
34
- 1.2.2
35
- -------------
36
-
37
- - Add API to check if a unique lock is present. See [#2932] for details.
38
- - Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
39
- - Add option for tuning which Bundler groups get preloaded with `sidekiqswarm` [#3025]
40
- ```
41
- SIDEKIQ_PRELOAD=default,production bin/sidekiqswarm ...
42
- # Use an empty value for maximum application compatibility
43
- SIDEKIQ_PRELOAD= bin/sidekiqswarm ...
44
- ```
45
-
46
- 1.2.1
47
- -------------
48
-
49
- - Multi-Process mode can now monitor the RSS memory of children and
50
- restart any that grow too large. To limit children to 1GB each:
51
- ```
52
- MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
53
- ```
54
-
55
- 1.2.0
56
- -------------
57
-
58
- - **NEW FEATURE** Multi-process mode! Sidekiq Enterprise can now fork multiple worker
59
- processes, enabling significant memory savings. See the [wiki
60
- documentation](https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process) for details.
61
-
62
-
63
- 0.7.10
64
- -------------
65
-
66
- - More precise gemspec dependency versioning
67
-
68
- 1.1.0
69
- -------------
70
-
71
- - **NEW FEATURE** Historical queue metrics, [documented in the wiki](https://github.com/mperham/sidekiq/wiki/Ent-Historical-Metrics) [#2719]
72
-
73
- 0.7.9, 1.0.2
74
- -------------
75
-
76
- - Window limiters can now accept arbitrary window sizes [#2686]
77
- - Fix race condition in window limiters leading to non-stop OverLimit [#2704]
78
- - Fix invalid overage counts when nesting concurrent limiters
79
-
80
- 1.0.1
81
- ----------
82
-
83
- - Fix crash in periodic subsystem when a follower shuts down, thanks
84
- to @justinko for reporting.
85
-
86
- 1.0.0
87
- ----------
88
-
89
- - Enterprise 1.x targets Sidekiq 4.x.
90
- - Rewrite several features to remove Celluloid dependency. No
91
- functional changes.
92
-
93
- 0.7.8
94
- ----------
95
-
96
- - Fix `unique_for: false` [#2658]
97
-
98
-
99
- 0.7.7
100
- ----------
101
-
102
- - Enterprise 0.x targets Sidekiq 3.x.
103
- - Fix racy shutdown event which could lead to disappearing periodic
104
- jobs, requires Sidekiq >= 3.5.3.
105
- - Add new :leader event which is fired when a process gains leadership.
106
-
107
- 0.7.6
108
- ----------
109
-
110
- - Redesign how overrated jobs are rescheduled to avoid creating new
111
- jobs. [#2619]
112
-
113
- 0.7.5
114
- ----------
115
-
116
- - Fix dynamic creation of concurrent limiters [#2617]
117
-
118
- 0.7.4
119
- ----------
120
- - Add additional check to prevent duplicate periodic job creation
121
- - Allow user-specified TTLs for rate limiters [#2607]
122
- - Paginate rate limiter index page [#2606]
123
-
124
- 0.7.3
125
- ----------
126
-
127
- - Rework `Sidekiq::Limiter` redis handling to match global redis handling.
128
- - Allow user to customize rate limit backoff logic and handle custom
129
- rate limit errors.
130
- - Fix scalability issue with Limiter index page.
131
-
132
- 0.7.2
133
- ----------
134
-
135
- - Fix typo which prevented limiters with '0' in their names.
136
-
137
- 0.7.1
138
- ----------
139
-
140
- - Fix issue where unique scheduled jobs can't be enqueued upon schedule
141
- due to the existing unique lock. [#2499]
142
-
143
- 0.7.0
144
- ----------
145
-
146
- Initial release.
data/Gemfile DELETED
@@ -1,29 +0,0 @@
1
- source 'https://rubygems.org'
2
- gemspec
3
-
4
- gem 'rails', '5.0.0'
5
- gem "hiredis"
6
- gem 'simplecov'
7
- gem 'minitest'
8
- gem 'minitest-utils'
9
- gem 'toxiproxy'
10
-
11
- platforms :rbx do
12
- gem 'rubysl', '~> 2.0' # if using anything in the ruby standard library
13
- gem 'psych' # if using yaml
14
- gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
15
- end
16
-
17
- platforms :ruby do
18
- gem 'sqlite3'
19
- end
20
-
21
- platforms :mri do
22
- gem 'pry-byebug'
23
- gem 'ruby-prof'
24
- end
25
-
26
- #platforms :jruby do
27
- #gem 'jruby-openssl'
28
- #gem 'activerecord-jdbcsqlite3-adapter'
29
- #end
data/Pro-2.0-Upgrade.md DELETED
@@ -1,138 +0,0 @@
1
- # Upgrading to Sidekiq Pro 2.0
2
-
3
- Sidekiq Pro 2.0 allows nested batches for more complex job workflows
4
- and provides a new reliable scheduler which uses Lua to guarantee
5
- atomicity and much higher performance.
6
-
7
- It also removes deprecated APIs, changes the batch data format and
8
- how features are activated. Read carefully to ensure your upgrade goes
9
- smoothly.
10
-
11
- Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
12
- recommended; Redis 2.4 or 2.6 will work but some functionality will not be
13
- available.
14
-
15
- **Note that you CANNOT go back to Pro 1.x once you've created batches
16
- with 2.x. The new batches will not process correctly with 1.x.**
17
-
18
- **If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
19
- latest 1.x version and run it for a week before upgrading to 2.0.**
20
-
21
- ## Nested Batches
22
-
23
- Batches can now be nested within the `jobs` method.
24
- This feature enables Sidekiq Pro to handle workflow processing of any size
25
- and complexity!
26
-
27
- ```ruby
28
- a = Sidekiq::Batch.new
29
- a.on(:success, SomeCallback)
30
- a.jobs do
31
- SomeWork.perform_async
32
-
33
- b = Sidekiq::Batch.new
34
- b.on(:success, MyCallback)
35
- b.jobs do
36
- OtherWork.perform_async
37
- end
38
- end
39
- ```
40
-
41
- Parent batch callbacks are not processed until all child batch callbacks have
42
- run successfully. In the example above, `MyCallback` will always fire
43
- before `SomeCallback` because `b` is considered a child of `a`.
44
-
45
- Of course you can dynamically add child batches while a batch job is executing.
46
-
47
- ```ruby
48
- def perform(*args)
49
- do_something(args)
50
-
51
- if more_work?
52
- # Sidekiq::Worker#batch returns the Batch this job is part of.
53
- batch.jobs do
54
- b = Sidekiq::Batch.new
55
- b.on(:success, MyCallback)
56
- b.jobs do
57
- OtherWork.perform_async
58
- end
59
- end
60
- end
61
- end
62
- ```
63
-
64
- More context: [#1485]
65
-
66
- ## Batch Data
67
-
68
- The batch data model was overhauled. Batch data should take
69
- significantly less space in Redis now. A simple benchmark shows 25%
70
- savings but real world savings should be even greater.
71
-
72
- * Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
73
- "vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
74
- strings, e.g. "4a3fc67d30370edf".
75
- * In 1.x, batch data was not removed until it naturally expired in Redis.
76
- In 2.x, all data for a batch is removed from Redis once the batch has
77
- run any success callbacks.
78
- * Because of the former point, batch expiry is no longer a concern.
79
- Batch expiry is hardcoded to 30 days and is no longer user-tunable.
80
- * Failed batch jobs no longer automatically store any associated
81
- backtrace in Redis.
82
-
83
- **There's no data migration required. Sidekiq Pro 2.0 transparently handles
84
- both old and new format.**
85
-
86
- More context: [#2130]
87
-
88
- ## Reliability
89
-
90
- 2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
91
- scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
92
- lots of jobs.
93
-
94
- **Two caveats**:
95
- - Client-side middleware is not executed
96
- for each job when enqueued with the reliable scheduler. No Sidekiq or
97
- Sidekiq Pro functionality is affected by this change but some 3rd party
98
- plugins might be.
99
- - The Lua script used inside the reliable scheduler is not safe for use
100
- with Redis Cluster or other multi-master Redis solutions.
101
- It is safe to use with Redis Sentinel or a typical master/slave replication setup.
102
-
103
- **You no longer require anything to use the Reliability features.**
104
-
105
- * Activate reliable fetch and/or the new reliable scheduler:
106
- ```ruby
107
- Sidekiq.configure_server do |config|
108
- config.reliable_fetch!
109
- config.reliable_scheduler!
110
- end
111
- ```
112
- * Activate reliable push:
113
- ```ruby
114
- Sidekiq::Client.reliable_push!
115
- ```
116
-
117
- More context: [#2130]
118
-
119
- ## Other Changes
120
-
121
- * You must require `sidekiq/pro/notifications` if you want to use the
122
- existing notification schemes. I don't recommend using them as the
123
- newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
124
- * Several classes have been renamed. Generally these classes are ones
125
- you should not need to require/use in your own code, e.g. the Batch
126
- middleware.
127
- * You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
128
- Pro will set it to the jid of the callback job. [#2178]
129
- * There's now an official API to iterate all known Batches [#2191]
130
- ```ruby
131
- Sidekiq::BatchSet.new.each {|status| p status.bid }
132
- ```
133
- * The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
134
-
135
- ## Thanks
136
-
137
- Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
138
- feedback during the release process. Thank you guys!
data/Pro-3.0-Upgrade.md DELETED
@@ -1,44 +0,0 @@
1
- # Welcome to Sidekiq Pro 3.0!
2
-
3
- Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
4
-
5
- ## What's New
6
-
7
- * **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
8
- ago and contains **many** useful features which Sidekiq couldn't
9
- leverage until now. **Redis 3.0.3 or greater is recommended** for large
10
- scale use.
11
-
12
- * Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
13
- you will need to pull it in yourself.
14
-
15
- * Pausing and unpausing queues is now instantaneous, no more polling!
16
-
17
- * Reliable fetch has been re-implemented due to the fetch changes in
18
- Sidekiq 4.0.
19
-
20
- * Support for platforms without persistent hostnames. Since the reliable\_fetch
21
- algorithm requires a persistent hostname, an alternative reliability
22
- algorithm is now available for platforms like Heroku and Docker:
23
- ```ruby
24
- Sidekiq.configure_server do |config|
25
- config.timed_fetch!
26
- end
27
- ```
28
- The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
29
-
30
- * The old 'sidekiq/notifications' features have been removed.
31
-
32
- ## Upgrade
33
-
34
- First, make sure you are using Redis 2.8 or greater. Next:
35
-
36
- * Upgrade to the latest Sidekiq Pro 2.x.
37
- ```ruby
38
- gem 'sidekiq-pro', '< 3'
39
- ```
40
- * Fix any deprecation warnings you see.
41
- * Upgrade to 3.x.
42
- ```ruby
43
- gem 'sidekiq-pro', '< 4'
44
- ```