sr-sidekiq 4.1.6

Sign up to get free protection for your applications and to get access to all the features.
Files changed (186) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +12 -0
  3. data/3.0-Upgrade.md +70 -0
  4. data/4.0-Upgrade.md +50 -0
  5. data/COMM-LICENSE (sidekiq) +95 -0
  6. data/Changes.md +1241 -0
  7. data/Ent-Changes.md +112 -0
  8. data/Gemfile +29 -0
  9. data/LICENSE (sidekiq) +9 -0
  10. data/LICENSE (sr-sidekiq) +5 -0
  11. data/Pro-2.0-Upgrade.md +138 -0
  12. data/Pro-3.0-Upgrade.md +44 -0
  13. data/Pro-Changes.md +539 -0
  14. data/README.md +8 -0
  15. data/Rakefile +9 -0
  16. data/bin/sidekiq +18 -0
  17. data/bin/sidekiqctl +99 -0
  18. data/bin/sidekiqload +167 -0
  19. data/code_of_conduct.md +50 -0
  20. data/lib/generators/sidekiq/templates/worker.rb.erb +9 -0
  21. data/lib/generators/sidekiq/templates/worker_spec.rb.erb +6 -0
  22. data/lib/generators/sidekiq/templates/worker_test.rb.erb +8 -0
  23. data/lib/generators/sidekiq/worker_generator.rb +49 -0
  24. data/lib/sidekiq.rb +237 -0
  25. data/lib/sidekiq/api.rb +844 -0
  26. data/lib/sidekiq/cli.rb +389 -0
  27. data/lib/sidekiq/client.rb +260 -0
  28. data/lib/sidekiq/core_ext.rb +106 -0
  29. data/lib/sidekiq/exception_handler.rb +31 -0
  30. data/lib/sidekiq/extensions/action_mailer.rb +57 -0
  31. data/lib/sidekiq/extensions/active_record.rb +40 -0
  32. data/lib/sidekiq/extensions/class_methods.rb +40 -0
  33. data/lib/sidekiq/extensions/generic_proxy.rb +25 -0
  34. data/lib/sidekiq/fetch.rb +81 -0
  35. data/lib/sidekiq/launcher.rb +160 -0
  36. data/lib/sidekiq/logging.rb +106 -0
  37. data/lib/sidekiq/manager.rb +137 -0
  38. data/lib/sidekiq/middleware/chain.rb +150 -0
  39. data/lib/sidekiq/middleware/i18n.rb +42 -0
  40. data/lib/sidekiq/middleware/server/active_record.rb +13 -0
  41. data/lib/sidekiq/middleware/server/logging.rb +40 -0
  42. data/lib/sidekiq/middleware/server/retry_jobs.rb +205 -0
  43. data/lib/sidekiq/paginator.rb +43 -0
  44. data/lib/sidekiq/processor.rb +186 -0
  45. data/lib/sidekiq/rails.rb +39 -0
  46. data/lib/sidekiq/redis_connection.rb +97 -0
  47. data/lib/sidekiq/scheduled.rb +146 -0
  48. data/lib/sidekiq/testing.rb +316 -0
  49. data/lib/sidekiq/testing/inline.rb +29 -0
  50. data/lib/sidekiq/util.rb +62 -0
  51. data/lib/sidekiq/version.rb +4 -0
  52. data/lib/sidekiq/web.rb +278 -0
  53. data/lib/sidekiq/web_helpers.rb +255 -0
  54. data/lib/sidekiq/worker.rb +121 -0
  55. data/sidekiq.gemspec +26 -0
  56. data/sr-sidekiq-4.1.3.gem +0 -0
  57. data/sr-sidekiq-4.1.4.gem +0 -0
  58. data/sr-sidekiq-4.1.5.gem +0 -0
  59. data/test/config.yml +9 -0
  60. data/test/env_based_config.yml +11 -0
  61. data/test/fake_env.rb +1 -0
  62. data/test/fixtures/en.yml +2 -0
  63. data/test/helper.rb +75 -0
  64. data/test/test_actors.rb +138 -0
  65. data/test/test_api.rb +528 -0
  66. data/test/test_cli.rb +406 -0
  67. data/test/test_client.rb +262 -0
  68. data/test/test_exception_handler.rb +56 -0
  69. data/test/test_extensions.rb +127 -0
  70. data/test/test_fetch.rb +50 -0
  71. data/test/test_launcher.rb +85 -0
  72. data/test/test_logging.rb +35 -0
  73. data/test/test_manager.rb +50 -0
  74. data/test/test_middleware.rb +158 -0
  75. data/test/test_processor.rb +201 -0
  76. data/test/test_rails.rb +22 -0
  77. data/test/test_redis_connection.rb +127 -0
  78. data/test/test_retry.rb +326 -0
  79. data/test/test_retry_exhausted.rb +149 -0
  80. data/test/test_scheduled.rb +115 -0
  81. data/test/test_scheduling.rb +50 -0
  82. data/test/test_sidekiq.rb +107 -0
  83. data/test/test_testing.rb +143 -0
  84. data/test/test_testing_fake.rb +357 -0
  85. data/test/test_testing_inline.rb +94 -0
  86. data/test/test_util.rb +13 -0
  87. data/test/test_web.rb +614 -0
  88. data/test/test_web_helpers.rb +54 -0
  89. data/web/assets/images/bootstrap/glyphicons-halflings-white.png +0 -0
  90. data/web/assets/images/bootstrap/glyphicons-halflings.png +0 -0
  91. data/web/assets/images/favicon.ico +0 -0
  92. data/web/assets/images/logo.png +0 -0
  93. data/web/assets/images/status-sd8051fd480.png +0 -0
  94. data/web/assets/images/status/active.png +0 -0
  95. data/web/assets/images/status/idle.png +0 -0
  96. data/web/assets/javascripts/application.js +88 -0
  97. data/web/assets/javascripts/dashboard.js +300 -0
  98. data/web/assets/javascripts/locales/README.md +27 -0
  99. data/web/assets/javascripts/locales/jquery.timeago.ar.js +96 -0
  100. data/web/assets/javascripts/locales/jquery.timeago.bg.js +18 -0
  101. data/web/assets/javascripts/locales/jquery.timeago.bs.js +49 -0
  102. data/web/assets/javascripts/locales/jquery.timeago.ca.js +18 -0
  103. data/web/assets/javascripts/locales/jquery.timeago.cs.js +18 -0
  104. data/web/assets/javascripts/locales/jquery.timeago.cy.js +20 -0
  105. data/web/assets/javascripts/locales/jquery.timeago.da.js +18 -0
  106. data/web/assets/javascripts/locales/jquery.timeago.de.js +18 -0
  107. data/web/assets/javascripts/locales/jquery.timeago.el.js +18 -0
  108. data/web/assets/javascripts/locales/jquery.timeago.en-short.js +20 -0
  109. data/web/assets/javascripts/locales/jquery.timeago.en.js +20 -0
  110. data/web/assets/javascripts/locales/jquery.timeago.es.js +18 -0
  111. data/web/assets/javascripts/locales/jquery.timeago.et.js +18 -0
  112. data/web/assets/javascripts/locales/jquery.timeago.fa.js +22 -0
  113. data/web/assets/javascripts/locales/jquery.timeago.fi.js +28 -0
  114. data/web/assets/javascripts/locales/jquery.timeago.fr-short.js +16 -0
  115. data/web/assets/javascripts/locales/jquery.timeago.fr.js +17 -0
  116. data/web/assets/javascripts/locales/jquery.timeago.he.js +18 -0
  117. data/web/assets/javascripts/locales/jquery.timeago.hr.js +49 -0
  118. data/web/assets/javascripts/locales/jquery.timeago.hu.js +18 -0
  119. data/web/assets/javascripts/locales/jquery.timeago.hy.js +18 -0
  120. data/web/assets/javascripts/locales/jquery.timeago.id.js +18 -0
  121. data/web/assets/javascripts/locales/jquery.timeago.it.js +16 -0
  122. data/web/assets/javascripts/locales/jquery.timeago.ja.js +19 -0
  123. data/web/assets/javascripts/locales/jquery.timeago.ko.js +17 -0
  124. data/web/assets/javascripts/locales/jquery.timeago.lt.js +20 -0
  125. data/web/assets/javascripts/locales/jquery.timeago.mk.js +20 -0
  126. data/web/assets/javascripts/locales/jquery.timeago.nl.js +20 -0
  127. data/web/assets/javascripts/locales/jquery.timeago.no.js +18 -0
  128. data/web/assets/javascripts/locales/jquery.timeago.pl.js +31 -0
  129. data/web/assets/javascripts/locales/jquery.timeago.pt-br.js +16 -0
  130. data/web/assets/javascripts/locales/jquery.timeago.pt.js +16 -0
  131. data/web/assets/javascripts/locales/jquery.timeago.ro.js +18 -0
  132. data/web/assets/javascripts/locales/jquery.timeago.rs.js +49 -0
  133. data/web/assets/javascripts/locales/jquery.timeago.ru.js +34 -0
  134. data/web/assets/javascripts/locales/jquery.timeago.sk.js +18 -0
  135. data/web/assets/javascripts/locales/jquery.timeago.sl.js +44 -0
  136. data/web/assets/javascripts/locales/jquery.timeago.sv.js +18 -0
  137. data/web/assets/javascripts/locales/jquery.timeago.th.js +20 -0
  138. data/web/assets/javascripts/locales/jquery.timeago.tr.js +16 -0
  139. data/web/assets/javascripts/locales/jquery.timeago.uk.js +34 -0
  140. data/web/assets/javascripts/locales/jquery.timeago.uz.js +19 -0
  141. data/web/assets/javascripts/locales/jquery.timeago.zh-cn.js +20 -0
  142. data/web/assets/javascripts/locales/jquery.timeago.zh-tw.js +20 -0
  143. data/web/assets/stylesheets/application.css +754 -0
  144. data/web/assets/stylesheets/bootstrap.css +9 -0
  145. data/web/locales/cs.yml +78 -0
  146. data/web/locales/da.yml +68 -0
  147. data/web/locales/de.yml +69 -0
  148. data/web/locales/el.yml +68 -0
  149. data/web/locales/en.yml +79 -0
  150. data/web/locales/es.yml +69 -0
  151. data/web/locales/fr.yml +78 -0
  152. data/web/locales/hi.yml +75 -0
  153. data/web/locales/it.yml +69 -0
  154. data/web/locales/ja.yml +78 -0
  155. data/web/locales/ko.yml +68 -0
  156. data/web/locales/nb.yml +77 -0
  157. data/web/locales/nl.yml +68 -0
  158. data/web/locales/pl.yml +59 -0
  159. data/web/locales/pt-br.yml +68 -0
  160. data/web/locales/pt.yml +67 -0
  161. data/web/locales/ru.yml +78 -0
  162. data/web/locales/sv.yml +68 -0
  163. data/web/locales/ta.yml +75 -0
  164. data/web/locales/uk.yml +76 -0
  165. data/web/locales/zh-cn.yml +68 -0
  166. data/web/locales/zh-tw.yml +68 -0
  167. data/web/views/_footer.erb +17 -0
  168. data/web/views/_job_info.erb +88 -0
  169. data/web/views/_nav.erb +66 -0
  170. data/web/views/_paging.erb +23 -0
  171. data/web/views/_poll_js.erb +5 -0
  172. data/web/views/_poll_link.erb +7 -0
  173. data/web/views/_status.erb +4 -0
  174. data/web/views/_summary.erb +40 -0
  175. data/web/views/busy.erb +94 -0
  176. data/web/views/dashboard.erb +75 -0
  177. data/web/views/dead.erb +34 -0
  178. data/web/views/layout.erb +32 -0
  179. data/web/views/morgue.erb +71 -0
  180. data/web/views/queue.erb +45 -0
  181. data/web/views/queues.erb +28 -0
  182. data/web/views/retries.erb +74 -0
  183. data/web/views/retry.erb +34 -0
  184. data/web/views/scheduled.erb +54 -0
  185. data/web/views/scheduled_job_info.erb +8 -0
  186. metadata +408 -0
data/Ent-Changes.md ADDED
@@ -0,0 +1,112 @@
1
+ Sidekiq Enterprise Changelog
2
+ =======================
3
+
4
+ Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
5
+
6
+ HEAD
7
+ -------------
8
+
9
+ - Add API to check if a unique lock is present. See [#2932] for details.
10
+ - Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
11
+
12
+ 1.2.1
13
+ -------------
14
+
15
+ - Multi-Process mode can now monitor the RSS memory of children and
16
+ restart any that grow too large. To limit children to 1GB each:
17
+ ```
18
+ MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
19
+ ```
20
+
21
+ 1.2.0
22
+ -------------
23
+
24
+ - **NEW FEATURE** Multi-process mode! Sidekiq Enterprise can now fork multiple worker
25
+ processes, enabling significant memory savings. See the [wiki
26
+ documentation](https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process) for details.
27
+
28
+
29
+ 0.7.10
30
+ -------------
31
+
32
+ - More precise gemspec dependency versioning
33
+
34
+ 1.1.0
35
+ -------------
36
+
37
+ - **NEW FEATURE** Historical queue metrics, [documented in the wiki](https://github.com/mperham/sidekiq/wiki/Ent-Historical-Metrics) [#2719]
38
+
39
+ 0.7.9, 1.0.2
40
+ -------------
41
+
42
+ - Window limiters can now accept arbitrary window sizes [#2686]
43
+ - Fix race condition in window limiters leading to non-stop OverLimit [#2704]
44
+ - Fix invalid overage counts when nesting concurrent limiters
45
+
46
+ 1.0.1
47
+ ----------
48
+
49
+ - Fix crash in periodic subsystem when a follower shuts down, thanks
50
+ to @justinko for reporting.
51
+
52
+ 1.0.0
53
+ ----------
54
+
55
+ - Enterprise 1.x targets Sidekiq 4.x.
56
+ - Rewrite several features to remove Celluloid dependency. No
57
+ functional changes.
58
+
59
+ 0.7.8
60
+ ----------
61
+
62
+ - Fix `unique_for: false` [#2658]
63
+
64
+
65
+ 0.7.7
66
+ ----------
67
+
68
+ - Enterprise 0.x targets Sidekiq 3.x.
69
+ - Fix racy shutdown event which could lead to disappearing periodic
70
+ jobs, requires Sidekiq >= 3.5.3.
71
+ - Add new :leader event which is fired when a process gains leadership.
72
+
73
+ 0.7.6
74
+ ----------
75
+
76
+ - Redesign how overrated jobs are rescheduled to avoid creating new
77
+ jobs. [#2619]
78
+
79
+ 0.7.5
80
+ ----------
81
+
82
+ - Fix dynamic creation of concurrent limiters [#2617]
83
+
84
+ 0.7.4
85
+ ----------
86
+ - Add additional check to prevent duplicate periodic job creation
87
+ - Allow user-specified TTLs for rate limiters [#2607]
88
+ - Paginate rate limiter index page [#2606]
89
+
90
+ 0.7.3
91
+ ----------
92
+
93
+ - Rework `Sidekiq::Limiter` redis handling to match global redis handling.
94
+ - Allow user to customize rate limit backoff logic and handle custom
95
+ rate limit errors.
96
+ - Fix scalability issue with Limiter index page.
97
+
98
+ 0.7.2
99
+ ----------
100
+
101
+ - Fix typo which prevented limiters with '0' in their names.
102
+
103
+ 0.7.1
104
+ ----------
105
+
106
+ - Fix issue where unique scheduled jobs can't be enqueued upon schedule
107
+ due to the existing unique lock. [#2499]
108
+
109
+ 0.7.0
110
+ ----------
111
+
112
+ Initial release.
data/Gemfile ADDED
@@ -0,0 +1,29 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+
4
+ gem "hiredis"
5
+ gem 'rails', '~> 4.2'
6
+ gem 'simplecov'
7
+ gem 'minitest'
8
+ gem 'minitest-utils'
9
+ gem 'toxiproxy'
10
+
11
+ platforms :rbx do
12
+ gem 'rubysl', '~> 2.0' # if using anything in the ruby standard library
13
+ gem 'psych' # if using yaml
14
+ gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
15
+ end
16
+
17
+ platforms :ruby do
18
+ gem 'sqlite3'
19
+ end
20
+
21
+ platforms :mri do
22
+ gem 'pry-byebug'
23
+ gem 'ruby-prof'
24
+ end
25
+
26
+ platforms :jruby do
27
+ gem 'jruby-openssl'
28
+ gem 'activerecord-jdbcsqlite3-adapter'
29
+ end
@@ -0,0 +1,9 @@
1
+ Copyright (c) Contributed Systems LLC
2
+
3
+ Sidekiq is an Open Source project licensed under the terms of
4
+ the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
+ for license text.
6
+
7
+ Sidekiq Pro has a commercial-friendly license allowing private forks
8
+ and modifications of Sidekiq. Please see http://sidekiq.org/pro/ for
9
+ more detail. You can find the commercial license terms in COMM-LICENSE.
@@ -0,0 +1,5 @@
1
+ Copyright (c) seanhuber.com
2
+
3
+ SR-Sidekiq is an Open Source project licensed under the terms of
4
+ the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
+ for license text.
@@ -0,0 +1,138 @@
1
+ # Upgrading to Sidekiq Pro 2.0
2
+
3
+ Sidekiq Pro 2.0 allows nested batches for more complex job workflows
4
+ and provides a new reliable scheduler which uses Lua to guarantee
5
+ atomicity and much higher performance.
6
+
7
+ It also removes deprecated APIs, changes the batch data format and
8
+ how features are activated. Read carefully to ensure your upgrade goes
9
+ smoothly.
10
+
11
+ Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
12
+ recommended; Redis 2.4 or 2.6 will work but some functionality will not be
13
+ available.
14
+
15
+ **Note that you CANNOT go back to Pro 1.x once you've created batches
16
+ with 2.x. The new batches will not process correctly with 1.x.**
17
+
18
+ **If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
19
+ latest 1.x version and run it for a week before upgrading to 2.0.**
20
+
21
+ ## Nested Batches
22
+
23
+ Batches can now be nested within the `jobs` method.
24
+ This feature enables Sidekiq Pro to handle workflow processing of any size
25
+ and complexity!
26
+
27
+ ```ruby
28
+ a = Sidekiq::Batch.new
29
+ a.on(:success, SomeCallback)
30
+ a.jobs do
31
+ SomeWork.perform_async
32
+
33
+ b = Sidekiq::Batch.new
34
+ b.on(:success, MyCallback)
35
+ b.jobs do
36
+ OtherWork.perform_async
37
+ end
38
+ end
39
+ ```
40
+
41
+ Parent batch callbacks are not processed until all child batch callbacks have
42
+ run successfully. In the example above, `MyCallback` will always fire
43
+ before `SomeCallback` because `b` is considered a child of `a`.
44
+
45
+ Of course you can dynamically add child batches while a batch job is executing.
46
+
47
+ ```ruby
48
+ def perform(*args)
49
+ do_something(args)
50
+
51
+ if more_work?
52
+ # Sidekiq::Worker#batch returns the Batch this job is part of.
53
+ batch.jobs do
54
+ b = Sidekiq::Batch.new
55
+ b.on(:success, MyCallback)
56
+ b.jobs do
57
+ OtherWork.perform_async
58
+ end
59
+ end
60
+ end
61
+ end
62
+ ```
63
+
64
+ More context: [#1485]
65
+
66
+ ## Batch Data
67
+
68
+ The batch data model was overhauled. Batch data should take
69
+ significantly less space in Redis now. A simple benchmark shows 25%
70
+ savings but real world savings should be even greater.
71
+
72
+ * Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
73
+ "vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
74
+ strings, e.g. "4a3fc67d30370edf".
75
+ * In 1.x, batch data was not removed until it naturally expired in Redis.
76
+ In 2.x, all data for a batch is removed from Redis once the batch has
77
+ run any success callbacks.
78
+ * Because of the former point, batch expiry is no longer a concern.
79
+ Batch expiry is hardcoded to 30 days and is no longer user-tunable.
80
+ * Failed batch jobs no longer automatically store any associated
81
+ backtrace in Redis.
82
+
83
+ **There's no data migration required. Sidekiq Pro 2.0 transparently handles
84
+ both old and new format.**
85
+
86
+ More context: [#2130]
87
+
88
+ ## Reliability
89
+
90
+ 2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
91
+ scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
92
+ lots of jobs.
93
+
94
+ **Two caveats**:
95
+ - Client-side middleware is not executed
96
+ for each job when enqueued with the reliable scheduler. No Sidekiq or
97
+ Sidekiq Pro functionality is affected by this change but some 3rd party
98
+ plugins might be.
99
+ - The Lua script used inside the reliable scheduler is not safe for use
100
+ with Redis Cluster or other multi-master Redis solutions.
101
+ It is safe to use with Redis Sentinel or a typical master/slave replication setup.
102
+
103
+ **You no longer require anything to use the Reliability features.**
104
+
105
+ * Activate reliable fetch and/or the new reliable scheduler:
106
+ ```ruby
107
+ Sidekiq.configure_server do |config|
108
+ config.reliable_fetch!
109
+ config.reliable_scheduler!
110
+ end
111
+ ```
112
+ * Activate reliable push:
113
+ ```ruby
114
+ Sidekiq::Client.reliable_push!
115
+ ```
116
+
117
+ More context: [#2130]
118
+
119
+ ## Other Changes
120
+
121
+ * You must require `sidekiq/pro/notifications` if you want to use the
122
+ existing notification schemes. I don't recommend using them as the
123
+ newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
124
+ * Several classes have been renamed. Generally these classes are ones
125
+ you should not need to require/use in your own code, e.g. the Batch
126
+ middleware.
127
+ * You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
128
+ Pro will set it to the jid of the callback job. [#2178]
129
+ * There's now an official API to iterate all known Batches [#2191]
130
+ ```ruby
131
+ Sidekiq::BatchSet.new.each {|status| p status.bid }
132
+ ```
133
+ * The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
134
+
135
+ ## Thanks
136
+
137
+ Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
138
+ feedback during the release process. Thank you guys!
@@ -0,0 +1,44 @@
1
+ # Welcome to Sidekiq Pro 3.0!
2
+
3
+ Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
4
+
5
+ ## What's New
6
+
7
+ * **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
8
+ ago and contains **many** useful features which Sidekiq couldn't
9
+ leverage until now. **Redis 3.0.3 or greater is recommended** for large
10
+ scale use.
11
+
12
+ * Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
13
+ you will need to pull it in yourself.
14
+
15
+ * Pausing and unpausing queues is now instantaneous, no more polling!
16
+
17
+ * Reliable fetch has been re-implemented due to the fetch changes in
18
+ Sidekiq 4.0.
19
+
20
+ * Support for platforms without persistent hostnames. Since the reliable\_fetch
21
+ algorithm requires a persistent hostname, an alternative reliability
22
+ algorithm is now available for platforms like Heroku and Docker:
23
+ ```ruby
24
+ Sidekiq.configure_server do |config|
25
+ config.timed_fetch!
26
+ end
27
+ ```
28
+ The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
29
+
30
+ * The old 'sidekiq/notifications' features have been removed.
31
+
32
+ ## Upgrade
33
+
34
+ First, make sure you are using Redis 2.8 or greater. Next:
35
+
36
+ * Upgrade to the latest Sidekiq Pro 2.x.
37
+ ```ruby
38
+ gem 'sidekiq-pro', '< 3'
39
+ ```
40
+ * Fix any deprecation warnings you see.
41
+ * Upgrade to 3.x.
42
+ ```ruby
43
+ gem 'sidekiq-pro', '< 4'
44
+ ```
data/Pro-Changes.md ADDED
@@ -0,0 +1,539 @@
1
+ Sidekiq Pro Changelog
2
+ =======================
3
+
4
+ Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
5
+
6
+ 3.2.1
7
+ ---------
8
+
9
+ - timed\_fetch now works with namespaces. [ryansch]
10
+
11
+
12
+ 3.2.0
13
+ ---------
14
+
15
+ - Fixed detection of missing batches, `NoSuchBatch` should be raised
16
+ properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
17
+ longer in Redis.
18
+ - Remove support for Pro 1.x format batches. This version will no
19
+ longer seamlessly process batches created with Sidekiq Pro 1.x.
20
+ As always, upgrade one major version at a time to ensure a smooth
21
+ transition.
22
+ - Fix edge case where a parent batch could expire before a child batch
23
+ was finished processing, leading to missing batches [#2889]
24
+
25
+ 2.1.5
26
+ ---------
27
+
28
+ - Fix edge case where a parent batch could expire before a child batch
29
+ was finished processing, leading to missing batches [#2889]
30
+
31
+ 3.1.0
32
+ ---------
33
+
34
+ - New container-friendly fetch algorithm: `timed_fetch`. See the
35
+ [wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
36
+ for trade offs between the two reliability options. You should
37
+ use this if you are on Heroku, Docker, Amazon ECS or EBS or
38
+ another container-based system.
39
+
40
+
41
+ 3.0.6
42
+ ---------
43
+
44
+ - Fix race condition on reliable fetch shutdown
45
+
46
+ 3.0.5
47
+ ---------
48
+
49
+ - Statsd metrics now account for ActiveJob class names
50
+ - Allow reliable fetch internals to be overridden [jonhyman]
51
+
52
+ 3.0.4
53
+ ---------
54
+
55
+ - Queue pausing no longer requires reliable fetch. [#2786]
56
+
57
+ 3.0.3, 2.1.4
58
+ ------------
59
+
60
+ - Convert Lua-based `Sidekiq::Queue#delete_by_class` to Ruby-based, to
61
+ avoid O(N^2) performance and possible Redis failure. [#2806]
62
+
63
+ 3.0.2
64
+ -----------
65
+
66
+ - Make job registration with batch part of the atomic push so batch
67
+ metadata can't get out of sync with the job data. [#2714]
68
+
69
+ 3.0.1
70
+ -----------
71
+
72
+ - Remove a number of Redis version checks since we can assume 2.8+ now.
73
+ - Fix expiring jobs client middleware not loaded on server
74
+
75
+ 3.0.0
76
+ -----------
77
+
78
+ - See the [Pro 3.0 release notes](Pro-3.0-Upgrade.md).
79
+
80
+ 2.1.3
81
+ -----------
82
+
83
+ - Don't enable strict priority if using weighted queueing like `-q a,1 -q b,1`
84
+ - Safer JSON mangling in Lua [#2639]
85
+
86
+ 2.1.2
87
+ -----------
88
+
89
+ - Lock Sidekiq Pro 2.x to Sidekiq 3.x.
90
+
91
+ 2.1.1
92
+ -----------
93
+
94
+ - Make ShardSet lazier so Redis can first be initialized at startup. [#2603]
95
+
96
+
97
+ 2.1.0
98
+ -----------
99
+
100
+ - Explicit support for sharding batches. You list your Redis shards and
101
+ Sidekiq Pro will randomly spread batches across the shards. The BID
102
+ will indicate which shard contains the batch data. Jobs within a
103
+ batch may be spread across all shards too. [#2548, jonhyman]
104
+ - Officially deprecate Sidekiq::Notifications code. Notifications have
105
+ been undocumented for months now. [#2575]
106
+
107
+
108
+ 2.0.8
109
+ -----------
110
+
111
+ - Fix reliable scheduler mangling large numeric arguments. Lua's CJSON
112
+ library cannot accurately encode numbers larger than 14 digits! [#2478]
113
+
114
+ 2.0.7
115
+ -----------
116
+
117
+ - Optimize delete of enormous batches (100,000s of jobs) [#2458]
118
+
119
+ 2.0.6, 1.9.3
120
+ --------------
121
+
122
+ - CSRF protection in Sidekiq 3.4.2 broke job filtering in the Web UI [#2442]
123
+ - Sidekiq Pro 1.x is now limited to Sidekiq < 3.5.0.
124
+
125
+ 2.0.5
126
+ -----------
127
+
128
+ - Atomic scheduler now sets `enqueued_at` [#2414]
129
+ - Batches now account for jobs which are stopped by client middleware [#2406]
130
+ - Ignore redundant calls to `Sidekiq::Client.reliable_push!` [#2408]
131
+
132
+ 2.0.4
133
+ -----------
134
+
135
+ - Reliable push now supports sharding [#2409]
136
+ - Reliable push now only catches Redis exceptions [#2307]
137
+
138
+ 2.0.3
139
+ -----------
140
+
141
+ - Display Batch callback data on the Batch details page. [#2347]
142
+ - Fix incompatibility with Pro Web and Rack middleware. [#2344] Thank
143
+ you to Jason Clark for the tip on how to fix it.
144
+
145
+ 2.0.2
146
+ -----------
147
+
148
+ - Multiple Web UIs can now run in the same process. [#2267] If you have
149
+ multiple Redis shards, you can mount UIs for all in the same process:
150
+ ```ruby
151
+ POOL1 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6379/0") }
152
+ POOL2 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6378/0") }
153
+
154
+ mount Sidekiq::Pro::Web => '/sidekiq' # default
155
+ mount Sidekiq::Pro::Web.with(redis_pool: POOL1), at: '/sidekiq1', as: 'sidekiq1' # shard1
156
+ mount Sidekiq::Pro::Web.with(redis_pool: POOL2), at: '/sidekiq2', as: 'sidekiq2' # shard2
157
+ ```
158
+ - **SECURITY** Fix batch XSS in error data. Thanks to moneybird.com for
159
+ reporting the issue.
160
+
161
+ 2.0.1
162
+ -----------
163
+
164
+ - Add `batch.callback_queue` so batch callbacks can use a higher
165
+ priority queue than jobs. [#2200]
166
+ - Gracefully recover if someone runs `SCRIPT FLUSH` on Redis. [#2240]
167
+ - Ignore errors when attempting `bulk_requeue`, allowing clean shutdown
168
+
169
+ 2.0.0
170
+ -----------
171
+
172
+ - See [the Upgrade Notes](Pro-2.0-Upgrade.md) for detailed notes.
173
+
174
+ 1.9.2
175
+ -----------
176
+
177
+ - As of 1/1/2015, Sidekiq Pro is hosted on a new dedicated server.
178
+ Happy new year and let's hope for 100% uptime!
179
+ - Fix bug in reliable\_fetch where jobs could be duplicated if a Sidekiq
180
+ process crashed and you were using weighted queues. [#2120]
181
+
182
+ 1.9.1
183
+ -----------
184
+
185
+ - **SECURITY** Fix XSS in batch description, thanks to intercom.io for reporting the
186
+ issue. If you don't use batch descriptions, you don't need the fix.
187
+
188
+ 1.9.0
189
+ -----------
190
+
191
+ - Add new expiring jobs feature [#1982]
192
+ - Show batch expiration on Batch details page [#1981]
193
+ - Add '$' batch success token to the pubsub support. [#1953]
194
+
195
+
196
+ 1.8.0
197
+ -----------
198
+
199
+ - Fix race condition where Batches can complete
200
+ before they have been fully defined or only half-defined. Requires
201
+ Sidekiq 3.2.3. [#1919]
202
+
203
+
204
+ 1.7.6
205
+ -----------
206
+
207
+ - Quick release to verify #1919
208
+
209
+
210
+ 1.7.5
211
+ -----------
212
+
213
+ - Fix job filtering within the Dead tab.
214
+ - Add APIs and wiki documentation for invalidating jobs within a batch.
215
+
216
+
217
+ 1.7.4
218
+ -----------
219
+
220
+ - Awesome ANSI art startup banner!
221
+
222
+
223
+ 1.7.3
224
+ -----------
225
+
226
+ - Batch callbacks should use the same queue as the associated jobs.
227
+
228
+ 1.7.2
229
+ -----------
230
+
231
+ - **DEPRECATION** Use `Batch#on(:complete)` instead of `Batch#notify`.
232
+ The specific Campfire, HipChat, email and other notification schemes
233
+ will be removed in 2.0.0.
234
+ - Remove batch from UI when successful. [#1745]
235
+ - Convert batch callbacks to be asynchronous jobs for error handling [#1744]
236
+
237
+ 1.7.1
238
+ -----------
239
+
240
+ - Fix for paused queues being processed for a few seconds when starting
241
+ a new Sidekiq process.
242
+ - Add a 5 sec delay when starting reliable fetch on Heroku to minimize
243
+ any duplicate job processing with another process shutting down.
244
+
245
+ 1.7.0
246
+ -----------
247
+
248
+ - Add ability to pause reliable queues via API.
249
+ ```ruby
250
+ q = Sidekiq::Queue.new("critical")
251
+ q.pause!
252
+ q.paused? # => true
253
+ q.unpause!
254
+ ```
255
+
256
+ Sidekiq polls Redis every 10 seconds for paused queues so pausing will take
257
+ a few seconds to take effect.
258
+
259
+ 1.6.0
260
+ -----------
261
+
262
+ - Compatible with Sidekiq 3.
263
+
264
+ 1.5.1
265
+ -----------
266
+
267
+ - Due to a breaking API change in Sidekiq 3.0, this version is limited
268
+ to Sidekiq 2.x.
269
+
270
+ 1.5.0
271
+ -----------
272
+
273
+ - Fix issue on Heroku where reliable fetch could orphan jobs [#1573]
274
+
275
+
276
+ 1.4.3
277
+ -----------
278
+
279
+ - Reverse sorting of Batches in Web UI [#1098]
280
+ - Refactoring for Sidekiq 3.0, Pro now requires Sidekiq 2.17.5
281
+
282
+ 1.4.2
283
+ -----------
284
+
285
+ - Tolerate expired Batches in the web UI.
286
+ - Fix 100% CPU usage when using weighted queues and reliable fetch.
287
+
288
+ 1.4.1
289
+ -----------
290
+
291
+ - Add batch progress bar to batch detail page. [#1398]
292
+ - Fix race condition in initializing Lua scripts
293
+
294
+
295
+ 1.4.0
296
+ -----------
297
+
298
+ - Default batch expiration has been extended to 3 days, from 1 day previously.
299
+ - Batches now sort in the Web UI according to expiry time, not creation time.
300
+ - Add user-configurable batch expiry. If your batches might take longer
301
+ than 72 hours to process, you can extend the expiration date.
302
+
303
+ ```ruby
304
+ b = Sidekiq::Batch.new
305
+ b.expires_in 5.days
306
+ ...
307
+ ```
308
+
309
+ 1.3.2
310
+ -----------
311
+
312
+ - Lazy load Lua scripts so a Redis connection is not required on bootup.
313
+
314
+ 1.3.1
315
+ -----------
316
+
317
+ - Fix a gemspec packaging issue which broke the Batch UI.
318
+
319
+ 1.3.0
320
+ -----------
321
+
322
+ Thanks to @jonhyman for his contributions to this Sidekiq Pro release.
323
+
324
+ This release includes new functionality based on the SCAN command newly
325
+ added to Redis 2.8. Pro still works with Redis 2.4 but some
326
+ functionality will be unavailable.
327
+
328
+ - Job Filtering in the Web UI!
329
+ You can now filter retries and scheduled jobs in the Web UI so you
330
+ only see the jobs relevant to your needs. Queues cannot be filtered;
331
+ Redis does not provide the same SCAN operation on the LIST type.
332
+ **Redis 2.8**
333
+ ![Filtering](https://f.cloud.github.com/assets/2911/1619465/f47529f2-5657-11e3-8cd1-33899eb72aad.png)
334
+ - SCAN support in the Sidekiq::SortedSet API. Here's an example that
335
+ finds all jobs which contain the substring "Warehouse::OrderShip"
336
+ and deletes all matching retries. If the set is large, this API
337
+ will be **MUCH** faster than standard iteration using each.
338
+ **Redis 2.8**
339
+ ```ruby
340
+ Sidekiq::RetrySet.new.scan("Warehouse::OrderShip") do |job|
341
+ job.delete
342
+ end
343
+ ```
344
+
345
+ - Sidekiq::Batch#jobs now returns the set of JIDs added to the batch.
346
+ - Sidekiq::Batch#jids returns the complete set of JIDs associated with the batch.
347
+ - Sidekiq::Batch#remove\_jobs(jid, jid, ...) removes JIDs from the set, allowing early termination of jobs if they become irrelevant according to application logic.
348
+ - Sidekiq::Batch#include?(jid) allows jobs to check if they are still
349
+ relevant to a Batch and exit early if not.
350
+ - Sidekiq::SortedSet#find\_job(jid) now uses server-side Lua if possible **Redis 2.6** [jonhyman]
351
+ - The statsd integration now sets global job counts:
352
+ ```ruby
353
+ jobs.count
354
+ jobs.success
355
+ jobs.failure
356
+ ```
357
+
358
+ - Change shutdown logic to push leftover jobs in the private queue back
359
+ into the public queue when shutting down with Reliable Fetch. This
360
+ allows the safe decommission of a Sidekiq Pro process when autoscaling. [jonhyman]
361
+ - Add support for weighted random fetching with Reliable Fetch [jonhyman]
362
+ - Pro now requires Sidekiq 2.17.0
363
+
364
+ 1.2.5
365
+ -----------
366
+
367
+ - Convert Batch UI to use Sidekiq 2.16's support for extension localization.
368
+ - Update reliable\_push to work with Sidekiq::Client refactoring in 2.16
369
+ - Pro now requires Sidekiq 2.16.0
370
+
371
+ 1.2.4
372
+ -----------
373
+
374
+ - Convert Batch UI to Bootstrap 3
375
+ - Pro now requires Sidekiq 2.15.0
376
+ - Add Sidekiq::Batch::Status#delete [#1205]
377
+
378
+ 1.2.3
379
+ -----------
380
+
381
+ - Pro now requires Sidekiq 2.14.0
382
+ - Fix bad exception handling in batch callbacks [#1134]
383
+ - Convert Batch UI to ERB
384
+
385
+ 1.2.2
386
+ -----------
387
+
388
+ - Problem with reliable fetch which could lead to lost jobs when Sidekiq
389
+ is shut down normally. Thanks to MikaelAmborn for the report. [#1109]
390
+
391
+ 1.2.1
392
+ -----------
393
+
394
+ - Forgot to push paging code necessary for `delete_job` performance.
395
+
396
+ 1.2.0
397
+ -----------
398
+
399
+ - **LEAK** Fix batch key which didn't expire in Redis. Keys match
400
+ /b-[a-f0-9]{16}-pending/, e.g. "b-4f55163ddba10aa0-pending" [#1057]
401
+ - **Reliable fetch now supports multiple queues**, using the algorithm spec'd
402
+ by @jackrg [#1102]
403
+ - Fix issue with reliable\_push where it didn't return the JID for a pushed
404
+ job when sending previously cached jobs to Redis.
405
+ - Add fast Sidekiq::Queue#delete\_job(jid) API which leverages Lua so job lookup is
406
+ 100% server-side. Benchmark vs Sidekiq's Job#delete API. **Redis 2.6**
407
+
408
+ ```
409
+ Sidekiq Pro API
410
+ 0.030000 0.020000 0.050000 ( 1.640659)
411
+ Sidekiq API
412
+ 17.250000 2.220000 19.470000 ( 22.193300)
413
+ ```
414
+
415
+ - Add fast Sidekiq::Queue#delete\_by\_class(klass) API to remove all
416
+ jobs of a given type. Uses server-side Lua for performance. **Redis 2.6**
417
+
418
+ 1.1.0
419
+ -----------
420
+
421
+ - New `sidekiq/pro/reliable_push` which makes Sidekiq::Client resiliant
422
+ to Redis network failures. [#793]
423
+ - Move `sidekiq/reliable_fetch` to `sidekiq/pro/reliable_fetch`
424
+
425
+
426
+ 1.0.0
427
+ -----------
428
+
429
+ - Sidekiq Pro changelog moved to mperham/sidekiq for public visibility.
430
+ - Add new Rack endpoint for easy polling of batch status via JavaScript. See `sidekiq/rack/batch_status`
431
+
432
+ 0.9.3
433
+ -----------
434
+
435
+ - Fix bad /batches path in Web UI
436
+ - Fix Sinatra conflict with sidekiq-failures
437
+
438
+ 0.9.2
439
+ -----------
440
+
441
+ - Fix issue with lifecycle notifications not firing.
442
+
443
+ 0.9.1
444
+ -----------
445
+
446
+ - Update due to Sidekiq API changes.
447
+
448
+ 0.9.0
449
+ -----------
450
+
451
+ - Rearchitect Sidekiq's Fetch code to support different fetch
452
+ strategies. Add a ReliableFetch strategy which works with Redis'
453
+ RPOPLPUSH to ensure we don't lose messages, even when the Sidekiq
454
+ process crashes unexpectedly. [mperham/sidekiq#607]
455
+
456
+ 0.8.2
457
+ -----------
458
+
459
+ - Reimplement existing notifications using batch on_complete events.
460
+
461
+ 0.8.1
462
+ -----------
463
+
464
+ - Rejigger batch callback notifications.
465
+
466
+
467
+ 0.8.0
468
+ -----------
469
+
470
+ - Add new Batch 'callback' notification support, for in-process
471
+ notification.
472
+ - Symbolize option keys passed to Pony [mperham/sidekiq#603]
473
+ - Batch no longer requires the Web UI since Web UI usage is optional.
474
+ You must require is manually in your Web process:
475
+
476
+ ```ruby
477
+ require 'sidekiq/web'
478
+ require 'sidekiq/batch/web'
479
+ mount Sidekiq::Web => '/sidekiq'
480
+ ```
481
+
482
+
483
+ 0.7.1
484
+ -----------
485
+
486
+ - Worker instances can access the associated jid and bid via simple
487
+ accessors.
488
+ - Batches can now be modified while being processed so, e.g. a batch
489
+ job can add additional jobs to its own batch.
490
+
491
+ ```ruby
492
+ def perform(...)
493
+ batch = Sidekiq::Batch.new(bid) # instantiate batch associated with this job
494
+ batch.jobs do
495
+ SomeWorker.perform_async # add another job
496
+ end
497
+ end
498
+ ```
499
+
500
+ - Save error backtraces in batch's failure info for display in Web UI.
501
+ - Clean up email notification a bit.
502
+
503
+
504
+ 0.7.0
505
+ -----------
506
+
507
+ - Add optional batch description
508
+ - Mutable batches. Batches can now be modified to add additional jobs
509
+ at runtime. Example would be a batch job which needs to create more
510
+ jobs based on the data it is processing.
511
+
512
+ ```ruby
513
+ batch = Sidekiq::Batch.new(bid)
514
+ batch.jobs do
515
+ # define more jobs here
516
+ end
517
+ ```
518
+ - Fix issues with symbols vs strings in option hashes
519
+
520
+
521
+ 0.6.1
522
+ -----------
523
+
524
+ - Webhook notification support
525
+
526
+
527
+ 0.6
528
+ -----------
529
+
530
+ - Redis pubsub
531
+ - Email polish
532
+
533
+
534
+ 0.5
535
+ -----------
536
+
537
+ - Batches
538
+ - Notifications
539
+ - Statsd middleware