sidekiq 5.2.4 → 7.2.4

Sign up to get free protection for your applications and to get access to all the features.
Files changed (153) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +672 -8
  3. data/LICENSE.txt +9 -0
  4. data/README.md +48 -51
  5. data/bin/multi_queue_bench +271 -0
  6. data/bin/sidekiq +22 -3
  7. data/bin/sidekiqload +213 -115
  8. data/bin/sidekiqmon +11 -0
  9. data/lib/generators/sidekiq/job_generator.rb +57 -0
  10. data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
  11. data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
  12. data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
  13. data/lib/sidekiq/api.rb +623 -352
  14. data/lib/sidekiq/capsule.rb +127 -0
  15. data/lib/sidekiq/cli.rb +214 -229
  16. data/lib/sidekiq/client.rb +127 -102
  17. data/lib/sidekiq/component.rb +68 -0
  18. data/lib/sidekiq/config.rb +287 -0
  19. data/lib/sidekiq/deploy.rb +62 -0
  20. data/lib/sidekiq/embedded.rb +61 -0
  21. data/lib/sidekiq/fetch.rb +49 -42
  22. data/lib/sidekiq/job.rb +374 -0
  23. data/lib/sidekiq/job_logger.rb +33 -7
  24. data/lib/sidekiq/job_retry.rb +157 -108
  25. data/lib/sidekiq/job_util.rb +107 -0
  26. data/lib/sidekiq/launcher.rb +206 -106
  27. data/lib/sidekiq/logger.rb +131 -0
  28. data/lib/sidekiq/manager.rb +43 -46
  29. data/lib/sidekiq/metrics/query.rb +156 -0
  30. data/lib/sidekiq/metrics/shared.rb +95 -0
  31. data/lib/sidekiq/metrics/tracking.rb +140 -0
  32. data/lib/sidekiq/middleware/chain.rb +113 -56
  33. data/lib/sidekiq/middleware/current_attributes.rb +95 -0
  34. data/lib/sidekiq/middleware/i18n.rb +7 -7
  35. data/lib/sidekiq/middleware/modules.rb +21 -0
  36. data/lib/sidekiq/monitor.rb +146 -0
  37. data/lib/sidekiq/paginator.rb +28 -16
  38. data/lib/sidekiq/processor.rb +126 -117
  39. data/lib/sidekiq/rails.rb +52 -38
  40. data/lib/sidekiq/redis_client_adapter.rb +111 -0
  41. data/lib/sidekiq/redis_connection.rb +41 -112
  42. data/lib/sidekiq/ring_buffer.rb +29 -0
  43. data/lib/sidekiq/scheduled.rb +112 -50
  44. data/lib/sidekiq/sd_notify.rb +149 -0
  45. data/lib/sidekiq/systemd.rb +24 -0
  46. data/lib/sidekiq/testing/inline.rb +6 -5
  47. data/lib/sidekiq/testing.rb +91 -90
  48. data/lib/sidekiq/transaction_aware_client.rb +51 -0
  49. data/lib/sidekiq/version.rb +3 -1
  50. data/lib/sidekiq/web/action.rb +20 -11
  51. data/lib/sidekiq/web/application.rb +202 -80
  52. data/lib/sidekiq/web/csrf_protection.rb +183 -0
  53. data/lib/sidekiq/web/helpers.rb +165 -114
  54. data/lib/sidekiq/web/router.rb +23 -19
  55. data/lib/sidekiq/web.rb +68 -107
  56. data/lib/sidekiq/worker_compatibility_alias.rb +13 -0
  57. data/lib/sidekiq.rb +92 -182
  58. data/sidekiq.gemspec +25 -16
  59. data/web/assets/images/apple-touch-icon.png +0 -0
  60. data/web/assets/javascripts/application.js +152 -61
  61. data/web/assets/javascripts/base-charts.js +106 -0
  62. data/web/assets/javascripts/chart.min.js +13 -0
  63. data/web/assets/javascripts/chartjs-plugin-annotation.min.js +7 -0
  64. data/web/assets/javascripts/dashboard-charts.js +182 -0
  65. data/web/assets/javascripts/dashboard.js +35 -293
  66. data/web/assets/javascripts/metrics.js +298 -0
  67. data/web/assets/stylesheets/application-dark.css +147 -0
  68. data/web/assets/stylesheets/application-rtl.css +10 -93
  69. data/web/assets/stylesheets/application.css +124 -522
  70. data/web/assets/stylesheets/bootstrap.css +1 -1
  71. data/web/locales/ar.yml +71 -65
  72. data/web/locales/cs.yml +62 -62
  73. data/web/locales/da.yml +60 -53
  74. data/web/locales/de.yml +65 -53
  75. data/web/locales/el.yml +43 -24
  76. data/web/locales/en.yml +86 -66
  77. data/web/locales/es.yml +70 -54
  78. data/web/locales/fa.yml +65 -65
  79. data/web/locales/fr.yml +83 -62
  80. data/web/locales/gd.yml +99 -0
  81. data/web/locales/he.yml +65 -64
  82. data/web/locales/hi.yml +59 -59
  83. data/web/locales/it.yml +53 -53
  84. data/web/locales/ja.yml +75 -64
  85. data/web/locales/ko.yml +52 -52
  86. data/web/locales/lt.yml +83 -0
  87. data/web/locales/nb.yml +61 -61
  88. data/web/locales/nl.yml +52 -52
  89. data/web/locales/pl.yml +45 -45
  90. data/web/locales/pt-br.yml +83 -55
  91. data/web/locales/pt.yml +51 -51
  92. data/web/locales/ru.yml +68 -63
  93. data/web/locales/sv.yml +53 -53
  94. data/web/locales/ta.yml +60 -60
  95. data/web/locales/uk.yml +62 -61
  96. data/web/locales/ur.yml +64 -64
  97. data/web/locales/vi.yml +83 -0
  98. data/web/locales/zh-cn.yml +43 -16
  99. data/web/locales/zh-tw.yml +42 -8
  100. data/web/views/_footer.erb +18 -3
  101. data/web/views/_job_info.erb +21 -4
  102. data/web/views/_metrics_period_select.erb +12 -0
  103. data/web/views/_nav.erb +1 -1
  104. data/web/views/_paging.erb +2 -0
  105. data/web/views/_poll_link.erb +3 -6
  106. data/web/views/_summary.erb +7 -7
  107. data/web/views/busy.erb +79 -29
  108. data/web/views/dashboard.erb +48 -18
  109. data/web/views/dead.erb +3 -3
  110. data/web/views/filtering.erb +7 -0
  111. data/web/views/layout.erb +3 -1
  112. data/web/views/metrics.erb +91 -0
  113. data/web/views/metrics_for_job.erb +59 -0
  114. data/web/views/morgue.erb +14 -15
  115. data/web/views/queue.erb +33 -24
  116. data/web/views/queues.erb +19 -5
  117. data/web/views/retries.erb +16 -17
  118. data/web/views/retry.erb +3 -3
  119. data/web/views/scheduled.erb +17 -15
  120. metadata +71 -72
  121. data/.github/contributing.md +0 -32
  122. data/.github/issue_template.md +0 -11
  123. data/.gitignore +0 -15
  124. data/.travis.yml +0 -17
  125. data/3.0-Upgrade.md +0 -70
  126. data/4.0-Upgrade.md +0 -53
  127. data/5.0-Upgrade.md +0 -56
  128. data/Appraisals +0 -9
  129. data/COMM-LICENSE +0 -95
  130. data/Ent-Changes.md +0 -225
  131. data/Gemfile +0 -29
  132. data/LICENSE +0 -9
  133. data/Pro-2.0-Upgrade.md +0 -138
  134. data/Pro-3.0-Upgrade.md +0 -44
  135. data/Pro-4.0-Upgrade.md +0 -35
  136. data/Pro-Changes.md +0 -752
  137. data/Rakefile +0 -9
  138. data/bin/sidekiqctl +0 -237
  139. data/code_of_conduct.md +0 -50
  140. data/gemfiles/rails_4.gemfile +0 -31
  141. data/gemfiles/rails_5.gemfile +0 -31
  142. data/lib/generators/sidekiq/worker_generator.rb +0 -49
  143. data/lib/sidekiq/core_ext.rb +0 -1
  144. data/lib/sidekiq/delay.rb +0 -42
  145. data/lib/sidekiq/exception_handler.rb +0 -29
  146. data/lib/sidekiq/extensions/action_mailer.rb +0 -57
  147. data/lib/sidekiq/extensions/active_record.rb +0 -40
  148. data/lib/sidekiq/extensions/class_methods.rb +0 -40
  149. data/lib/sidekiq/extensions/generic_proxy.rb +0 -31
  150. data/lib/sidekiq/logging.rb +0 -122
  151. data/lib/sidekiq/middleware/server/active_record.rb +0 -23
  152. data/lib/sidekiq/util.rb +0 -66
  153. data/lib/sidekiq/worker.rb +0 -215
data/Pro-2.0-Upgrade.md DELETED
@@ -1,138 +0,0 @@
1
- # Upgrading to Sidekiq Pro 2.0
2
-
3
- Sidekiq Pro 2.0 allows nested batches for more complex job workflows
4
- and provides a new reliable scheduler which uses Lua to guarantee
5
- atomicity and much higher performance.
6
-
7
- It also removes deprecated APIs, changes the batch data format and
8
- how features are activated. Read carefully to ensure your upgrade goes
9
- smoothly.
10
-
11
- Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
12
- recommended; Redis 2.4 or 2.6 will work but some functionality will not be
13
- available.
14
-
15
- **Note that you CANNOT go back to Pro 1.x once you've created batches
16
- with 2.x. The new batches will not process correctly with 1.x.**
17
-
18
- **If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
19
- latest 1.x version and run it for a week before upgrading to 2.0.**
20
-
21
- ## Nested Batches
22
-
23
- Batches can now be nested within the `jobs` method.
24
- This feature enables Sidekiq Pro to handle workflow processing of any size
25
- and complexity!
26
-
27
- ```ruby
28
- a = Sidekiq::Batch.new
29
- a.on(:success, SomeCallback)
30
- a.jobs do
31
- SomeWork.perform_async
32
-
33
- b = Sidekiq::Batch.new
34
- b.on(:success, MyCallback)
35
- b.jobs do
36
- OtherWork.perform_async
37
- end
38
- end
39
- ```
40
-
41
- Parent batch callbacks are not processed until all child batch callbacks have
42
- run successfully. In the example above, `MyCallback` will always fire
43
- before `SomeCallback` because `b` is considered a child of `a`.
44
-
45
- Of course you can dynamically add child batches while a batch job is executing.
46
-
47
- ```ruby
48
- def perform(*args)
49
- do_something(args)
50
-
51
- if more_work?
52
- # Sidekiq::Worker#batch returns the Batch this job is part of.
53
- batch.jobs do
54
- b = Sidekiq::Batch.new
55
- b.on(:success, MyCallback)
56
- b.jobs do
57
- OtherWork.perform_async
58
- end
59
- end
60
- end
61
- end
62
- ```
63
-
64
- More context: [#1485]
65
-
66
- ## Batch Data
67
-
68
- The batch data model was overhauled. Batch data should take
69
- significantly less space in Redis now. A simple benchmark shows 25%
70
- savings but real world savings should be even greater.
71
-
72
- * Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
73
- "vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
74
- strings, e.g. "4a3fc67d30370edf".
75
- * In 1.x, batch data was not removed until it naturally expired in Redis.
76
- In 2.x, all data for a batch is removed from Redis once the batch has
77
- run any success callbacks.
78
- * Because of the former point, batch expiry is no longer a concern.
79
- Batch expiry is hardcoded to 30 days and is no longer user-tunable.
80
- * Failed batch jobs no longer automatically store any associated
81
- backtrace in Redis.
82
-
83
- **There's no data migration required. Sidekiq Pro 2.0 transparently handles
84
- both old and new format.**
85
-
86
- More context: [#2130]
87
-
88
- ## Reliability
89
-
90
- 2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
91
- scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
92
- lots of jobs.
93
-
94
- **Two caveats**:
95
- - Client-side middleware is not executed
96
- for each job when enqueued with the reliable scheduler. No Sidekiq or
97
- Sidekiq Pro functionality is affected by this change but some 3rd party
98
- plugins might be.
99
- - The Lua script used inside the reliable scheduler is not safe for use
100
- with Redis Cluster or other multi-master Redis solutions.
101
- It is safe to use with Redis Sentinel or a typical master/slave replication setup.
102
-
103
- **You no longer require anything to use the Reliability features.**
104
-
105
- * Activate reliable fetch and/or the new reliable scheduler:
106
- ```ruby
107
- Sidekiq.configure_server do |config|
108
- config.reliable_fetch!
109
- config.reliable_scheduler!
110
- end
111
- ```
112
- * Activate reliable push:
113
- ```ruby
114
- Sidekiq::Client.reliable_push!
115
- ```
116
-
117
- More context: [#2130]
118
-
119
- ## Other Changes
120
-
121
- * You must require `sidekiq/pro/notifications` if you want to use the
122
- existing notification schemes. I don't recommend using them as the
123
- newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
124
- * Several classes have been renamed. Generally these classes are ones
125
- you should not need to require/use in your own code, e.g. the Batch
126
- middleware.
127
- * You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
128
- Pro will set it to the jid of the callback job. [#2178]
129
- * There's now an official API to iterate all known Batches [#2191]
130
- ```ruby
131
- Sidekiq::BatchSet.new.each {|status| p status.bid }
132
- ```
133
- * The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
134
-
135
- ## Thanks
136
-
137
- Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
138
- feedback during the release process. Thank you guys!
data/Pro-3.0-Upgrade.md DELETED
@@ -1,44 +0,0 @@
1
- # Welcome to Sidekiq Pro 3.0!
2
-
3
- Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
4
-
5
- ## What's New
6
-
7
- * **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
8
- ago and contains **many** useful features which Sidekiq couldn't
9
- leverage until now. **Redis 3.0.3 or greater is recommended** for large
10
- scale use.
11
-
12
- * Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
13
- you will need to pull it in yourself.
14
-
15
- * Pausing and unpausing queues is now instantaneous, no more polling!
16
-
17
- * Reliable fetch has been re-implemented due to the fetch changes in
18
- Sidekiq 4.0.
19
-
20
- * Support for platforms without persistent hostnames. Since the reliable\_fetch
21
- algorithm requires a persistent hostname, an alternative reliability
22
- algorithm is now available for platforms like Heroku and Docker:
23
- ```ruby
24
- Sidekiq.configure_server do |config|
25
- config.timed_fetch!
26
- end
27
- ```
28
- The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
29
-
30
- * The old 'sidekiq/notifications' features have been removed.
31
-
32
- ## Upgrade
33
-
34
- First, make sure you are using Redis 2.8 or greater. Next:
35
-
36
- * Upgrade to the latest Sidekiq Pro 2.x.
37
- ```ruby
38
- gem 'sidekiq-pro', '< 3'
39
- ```
40
- * Fix any deprecation warnings you see.
41
- * Upgrade to 3.x.
42
- ```ruby
43
- gem 'sidekiq-pro', '< 4'
44
- ```
data/Pro-4.0-Upgrade.md DELETED
@@ -1,35 +0,0 @@
1
- # Welcome to Sidekiq Pro 4.0!
2
-
3
- Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
4
-
5
- ## What's New
6
-
7
- * Batches now "die" if any of their jobs die. You can enumerate the set
8
- of dead batches and their associated dead jobs. The success callback
9
- for a dead batch will never fire unless these jobs are fixed.
10
- ```ruby
11
- Sidekiq::Batch::DeadSet.new.each do |status|
12
- status.dead? # => true
13
- status.dead_jobs # => [...]
14
- end
15
- ```
16
- This API allows you to enumerate the batches which need help.
17
- If you fix the issue and the dead jobs succeed, the batch will succeed.
18
- * The older `reliable_fetch` and `timed_fetch` algorithms have been
19
- removed. Only super\_fetch is available in 4.0.
20
- * The statsd middleware has been tweaked to remove support for legacy,
21
- pre-3.6.0 configuration and add relevant tags.
22
- * Requires Sidekiq 5.0.5+.
23
-
24
- ## Upgrade
25
-
26
- * Upgrade to the latest Sidekiq Pro 3.x.
27
- ```ruby
28
- gem 'sidekiq-pro', '< 4'
29
- ```
30
- * Fix any deprecation warnings you see.
31
- * Upgrade to 4.x.
32
- ```ruby
33
- gem 'sidekiq-pro', '< 5'
34
- ```
35
-