sidekiq 6.0.7 → 6.5.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (107) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +209 -2
  3. data/LICENSE +3 -3
  4. data/README.md +11 -10
  5. data/bin/sidekiq +8 -3
  6. data/bin/sidekiqload +70 -66
  7. data/bin/sidekiqmon +1 -1
  8. data/lib/generators/sidekiq/job_generator.rb +57 -0
  9. data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
  10. data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
  11. data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
  12. data/lib/sidekiq/api.rb +180 -123
  13. data/lib/sidekiq/cli.rb +80 -45
  14. data/lib/sidekiq/client.rb +52 -71
  15. data/lib/sidekiq/{util.rb → component.rb} +11 -14
  16. data/lib/sidekiq/delay.rb +2 -0
  17. data/lib/sidekiq/extensions/action_mailer.rb +3 -2
  18. data/lib/sidekiq/extensions/active_record.rb +4 -3
  19. data/lib/sidekiq/extensions/class_methods.rb +5 -4
  20. data/lib/sidekiq/extensions/generic_proxy.rb +4 -2
  21. data/lib/sidekiq/fetch.rb +41 -30
  22. data/lib/sidekiq/job.rb +13 -0
  23. data/lib/sidekiq/job_logger.rb +16 -28
  24. data/lib/sidekiq/job_retry.rb +36 -36
  25. data/lib/sidekiq/job_util.rb +71 -0
  26. data/lib/sidekiq/launcher.rb +123 -63
  27. data/lib/sidekiq/logger.rb +11 -20
  28. data/lib/sidekiq/manager.rb +35 -34
  29. data/lib/sidekiq/middleware/chain.rb +28 -17
  30. data/lib/sidekiq/middleware/current_attributes.rb +61 -0
  31. data/lib/sidekiq/middleware/i18n.rb +6 -4
  32. data/lib/sidekiq/middleware/modules.rb +19 -0
  33. data/lib/sidekiq/monitor.rb +1 -1
  34. data/lib/sidekiq/paginator.rb +8 -8
  35. data/lib/sidekiq/processor.rb +41 -41
  36. data/lib/sidekiq/rails.rb +38 -22
  37. data/lib/sidekiq/redis_client_adapter.rb +154 -0
  38. data/lib/sidekiq/redis_connection.rb +87 -53
  39. data/lib/sidekiq/ring_buffer.rb +29 -0
  40. data/lib/sidekiq/scheduled.rb +60 -24
  41. data/lib/sidekiq/sd_notify.rb +1 -1
  42. data/lib/sidekiq/testing/inline.rb +4 -4
  43. data/lib/sidekiq/testing.rb +39 -40
  44. data/lib/sidekiq/transaction_aware_client.rb +45 -0
  45. data/lib/sidekiq/version.rb +1 -1
  46. data/lib/sidekiq/web/action.rb +2 -2
  47. data/lib/sidekiq/web/application.rb +21 -12
  48. data/lib/sidekiq/web/csrf_protection.rb +180 -0
  49. data/lib/sidekiq/web/helpers.rb +40 -34
  50. data/lib/sidekiq/web/router.rb +5 -2
  51. data/lib/sidekiq/web.rb +36 -72
  52. data/lib/sidekiq/worker.rb +136 -16
  53. data/lib/sidekiq.rb +107 -30
  54. data/sidekiq.gemspec +11 -4
  55. data/web/assets/images/apple-touch-icon.png +0 -0
  56. data/web/assets/javascripts/application.js +113 -65
  57. data/web/assets/javascripts/dashboard.js +51 -51
  58. data/web/assets/stylesheets/application-dark.css +64 -43
  59. data/web/assets/stylesheets/application-rtl.css +0 -4
  60. data/web/assets/stylesheets/application.css +42 -239
  61. data/web/locales/ar.yml +8 -2
  62. data/web/locales/en.yml +4 -1
  63. data/web/locales/es.yml +18 -2
  64. data/web/locales/fr.yml +8 -1
  65. data/web/locales/ja.yml +3 -0
  66. data/web/locales/lt.yml +1 -1
  67. data/web/locales/pl.yml +4 -4
  68. data/web/locales/pt-br.yml +27 -9
  69. data/web/locales/ru.yml +4 -0
  70. data/web/views/_footer.erb +1 -1
  71. data/web/views/_job_info.erb +1 -1
  72. data/web/views/_poll_link.erb +2 -5
  73. data/web/views/_summary.erb +7 -7
  74. data/web/views/busy.erb +51 -20
  75. data/web/views/dashboard.erb +22 -14
  76. data/web/views/dead.erb +1 -1
  77. data/web/views/layout.erb +2 -1
  78. data/web/views/morgue.erb +6 -6
  79. data/web/views/queue.erb +11 -11
  80. data/web/views/queues.erb +4 -4
  81. data/web/views/retries.erb +7 -7
  82. data/web/views/retry.erb +1 -1
  83. data/web/views/scheduled.erb +1 -1
  84. metadata +29 -51
  85. data/.circleci/config.yml +0 -60
  86. data/.github/contributing.md +0 -32
  87. data/.github/issue_template.md +0 -11
  88. data/.gitignore +0 -13
  89. data/.standard.yml +0 -20
  90. data/3.0-Upgrade.md +0 -70
  91. data/4.0-Upgrade.md +0 -53
  92. data/5.0-Upgrade.md +0 -56
  93. data/6.0-Upgrade.md +0 -72
  94. data/COMM-LICENSE +0 -97
  95. data/Ent-2.0-Upgrade.md +0 -37
  96. data/Ent-Changes.md +0 -256
  97. data/Gemfile +0 -24
  98. data/Gemfile.lock +0 -208
  99. data/Pro-2.0-Upgrade.md +0 -138
  100. data/Pro-3.0-Upgrade.md +0 -44
  101. data/Pro-4.0-Upgrade.md +0 -35
  102. data/Pro-5.0-Upgrade.md +0 -25
  103. data/Pro-Changes.md +0 -782
  104. data/Rakefile +0 -10
  105. data/code_of_conduct.md +0 -50
  106. data/lib/generators/sidekiq/worker_generator.rb +0 -57
  107. data/lib/sidekiq/exception_handler.rb +0 -27
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3dd5d7a0dd8a7d877484df4ae3b9988f15e9cc4bc0bb4da23224be0049fa733e
4
- data.tar.gz: e2b7ddb4c045817837e996356851648bf32108b37fa1abcfb72795a185b4a238
3
+ metadata.gz: dedc78179d612e64435e57f0f6644c2645c06ffdc24151c9ad51df4035b51efa
4
+ data.tar.gz: 5987eba09190793cfccbff56066b54b5308a1fe74be7551cb5b9d6d03d256c3a
5
5
  SHA512:
6
- metadata.gz: 5ce3f911b835b38ce212309bcdf81b4789e24a12898a0f2071b9adb9cbcdfa23a33bb7216c2f7c6dcab26d484f79a1271594d3218f63ed041f0027deab5a26f3
7
- data.tar.gz: 685a6890a43c990b2fd9709c49f9c82c96a487cdcf6fc0607ab15b9dbb47d52c9f797ec5094d3f92ec35c5737e661f966a6b81006ee4b7d99a7f31285180b3db
6
+ metadata.gz: 9a26b5eb6e2248870f07ede054cac8330b52ad548d27b40d3950b6d985ef5ef8359e4dfc062757dc4b479036f0b2716bacfc821a20bd63413ea7c3f772bc6ead
7
+ data.tar.gz: 46a35f7f27693364b1b6077e602d3f69e476508654157f48fcd457a70bbcdb563f8515812ef37abae207604596e1d42f78132ac54471109f913e87c11a707cdb
data/Changes.md CHANGED
@@ -1,6 +1,200 @@
1
1
  # Sidekiq Changes
2
2
 
3
- [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
4
+
5
+ 6.5.0
6
+ ---------
7
+
8
+ - Substantial refactoring of Sidekiq server internals, part of a larger effort
9
+ to reduce Sidekiq's internal usage of global methods and data, see [docs/component.md](docs/component.md),
10
+ [docs/global_to_local.md](docs/global_to_local.md) and [docs/middleware.md](docs/middleware.md).
11
+ - **Add beta support for the `redis-client` gem**. This will become the default Redis driver in Sidekiq 7.0. [#5298]
12
+ Read more: https://github.com/mperham/sidekiq/wiki/Using-redis-client
13
+ - **Add beta support for DB transaction-aware client** [#5291]
14
+ Add this line to your initializer and any jobs created during a transaction
15
+ will only be pushed to Redis **after the transaction commits**. You will need to add the
16
+ `after_commit_everywhere` gem to your Gemfile.
17
+ ```ruby
18
+ Sidekiq.transactional_push!
19
+ ```
20
+ This feature does not have a lot of production usage yet; please try it out and let us
21
+ know if you have any issues. It will be fully supported in Sidekiq 7.0 or removed if it
22
+ proves problematic.
23
+ - Fix regression with middleware arguments [#5312]
24
+
25
+ 6.4.2
26
+ ---------
27
+
28
+ - Strict argument checking now runs after client-side middleware [#5246]
29
+ - Fix page events with live polling [#5184]
30
+ - Many under-the-hood changes to remove all usage of the term "worker"
31
+ from the Sidekiq codebase and APIs. This mostly involved RDoc and local
32
+ variable names but a few constants and public APIs were changed. The old
33
+ APIs will be removed in Sidekiq 7.0.
34
+ ```
35
+ Sidekiq::DEFAULT_WORKER_OPTIONS -> Sidekiq.default_job_options
36
+ Sidekiq.default_worker_options -> Sidekiq.default_job_options
37
+ Sidekiq::Queues["default"].jobs_by_worker(HardJob) -> Sidekiq::Queues["default"].jobs_by_class(HardJob)
38
+ ```
39
+
40
+ 6.4.1
41
+ ---------
42
+
43
+ - Fix pipeline/multi deprecations in redis-rb 4.6
44
+ - Fix sidekiq.yml YAML load errors on Ruby 3.1 [#5141]
45
+ - Sharding support for `perform_bulk` [#5129]
46
+ - Refactor job logger for SPEEEEEEED
47
+
48
+ 6.4.0
49
+ ---------
50
+
51
+ - **SECURITY**: Validate input to avoid possible DoS in Web UI.
52
+ - Add **strict argument checking** [#5071]
53
+ Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
54
+ Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
55
+ This warning will switch to an exception in Sidekiq 7.0.
56
+ - Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
57
+ - Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
58
+ ```ruby
59
+ SomeJob.perform_async(args...)
60
+ SomeJob.perform_sync(args...)
61
+ SomeJob.perform_inline(args...)
62
+ ```
63
+ You can also dynamically redirect a job to run synchronously:
64
+ ```ruby
65
+ SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
66
+ ```
67
+ - Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
68
+ ```
69
+ bin/rails generate sidekiq:job ProcessOrderJob
70
+ ```
71
+ - Fix job retries losing CurrentAttributes [#5090]
72
+ - Tweak shutdown to give long-running threads time to cleanup [#5095]
73
+
74
+ 6.3.1
75
+ ---------
76
+
77
+ - Fix keyword arguments error with CurrentAttributes on Ruby 3.0 [#5048]
78
+
79
+ 6.3.0
80
+ ---------
81
+
82
+ - **BREAK**: The Web UI has been refactored to remove jQuery. Any UI extensions
83
+ which use jQuery will break.
84
+ - **FEATURE**: Sidekiq.logger has been enhanced so any `Rails.logger`
85
+ output in jobs now shows up in the Sidekiq console. Remove any logger
86
+ hacks in your initializer and see if it Just Works™ now. [#5021]
87
+ - **FEATURE**: Add `Sidekiq::Job` alias for `Sidekiq::Worker`, to better
88
+ reflect industry standard terminology. You can now do this:
89
+ ```ruby
90
+ class MyJob
91
+ include Sidekiq::Job
92
+ sidekiq_options ...
93
+ def perform(args)
94
+ end
95
+ end
96
+ ```
97
+ - **FEATURE**: Support for serializing ActiveSupport::CurrentAttributes into each job. [#4982]
98
+ ```ruby
99
+ # config/initializers/sidekiq.rb
100
+ require "sidekiq/middleware/current_attributes"
101
+ Sidekiq::CurrentAttributes.persist(Myapp::Current) # Your AS::CurrentAttributes singleton
102
+ ```
103
+ - **FEATURE**: Add `Sidekiq::Worker.perform_bulk` for enqueuing jobs in bulk,
104
+ similar to `Sidekiq::Client.push_bulk` [#5042]
105
+ ```ruby
106
+ MyJob.perform_bulk([[1], [2], [3]])
107
+ ```
108
+ - Implement `queue_as`, `wait` and `wait_until` for ActiveJob compatibility [#5003]
109
+ - Scheduler now uses Lua to reduce Redis load and network roundtrips [#5044]
110
+ - Retry Redis operation if we get an `UNBLOCKED` Redis error [#4985]
111
+ - Run existing signal traps, if any, before running Sidekiq's trap [#4991]
112
+ - Fix fetch bug when using weighted queues which caused Sidekiq to stop
113
+ processing queues randomly [#5031]
114
+
115
+ 6.2.2
116
+ ---------
117
+
118
+ - Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
119
+ - Minimize scheduler load on Redis at scale [#4882]
120
+ - Improve logging of delay jobs [#4904, BuonOno]
121
+ - Minor CSS improvements for buttons and tables, design PRs always welcome!
122
+ - Tweak Web UI `Cache-Control` header [#4966]
123
+ - Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
124
+
125
+ 6.2.1
126
+ ---------
127
+
128
+ - Update RTT warning logic to handle transient RTT spikes [#4851]
129
+ - Fix very low priority CVE on unescaped queue name [#4852]
130
+ - Add note about sessions and Rails apps in API mode
131
+
132
+ 6.2.0
133
+ ---------
134
+
135
+ - Store Redis RTT and log if poor [#4824]
136
+ - Add process/thread stats to Busy page [#4806]
137
+ - Improve Web UI on mobile devices [#4840]
138
+ - **Refactor Web UI session usage** [#4804]
139
+ Numerous people have hit "Forbidden" errors and struggled with Sidekiq's
140
+ Web UI session requirement. If you have code in your initializer for
141
+ Web sessions, it's quite possible it will need to be removed. Here's
142
+ an overview:
143
+ ```
144
+ Sidekiq::Web needs a valid Rack session for CSRF protection. If this is a Rails app,
145
+ make sure you mount Sidekiq::Web *inside* your routes in `config/routes.rb` so
146
+ Sidekiq can reuse the Rails session:
147
+
148
+ Rails.application.routes.draw do
149
+ mount Sidekiq::Web => "/sidekiq"
150
+ ....
151
+ end
152
+
153
+ If this is a bare Rack app, use a session middleware before Sidekiq::Web:
154
+
155
+ # first, use IRB to create a shared secret key for sessions and commit it
156
+ require 'securerandom'; File.open(".session.key", "w") {|f| f.write(SecureRandom.hex(32)) }
157
+
158
+ # now, update your Rack app to include the secret with a session cookie middleware
159
+ use Rack::Session::Cookie, secret: File.read(".session.key"), same_site: true, max_age: 86400
160
+ run Sidekiq::Web
161
+
162
+ If this is a Rails app in API mode, you need to enable sessions.
163
+
164
+ https://guides.rubyonrails.org/api_app.html#using-session-middlewares
165
+ ```
166
+
167
+ 6.1.3
168
+ ---------
169
+
170
+ - Warn if Redis is configured to evict data under memory pressure [#4752]
171
+ - Add process RSS on the Busy page [#4717]
172
+
173
+ 6.1.2
174
+ ---------
175
+
176
+ - Improve readability in dark mode Web UI [#4674]
177
+ - Fix Web UI crash with corrupt session [#4672]
178
+ - Allow middleware to yield arguments [#4673, @eugeneius]
179
+ - Migrate CI from CircleCI to GitHub Actions [#4677]
180
+
181
+ 6.1.1
182
+ ---------
183
+
184
+ - Jobs are now sorted by age in the Busy Workers table. [#4641]
185
+ - Fix "check all" JS logic in Web UI [#4619]
186
+
187
+ 6.1.0
188
+ ---------
189
+
190
+ - Web UI - Dark Mode fixes [#4543, natematykiewicz]
191
+ - Ensure `Rack::ContentLength` is loaded as middleware for correct Web UI responses [#4541]
192
+ - Avoid exception dumping SSL store in Redis connection logging [#4532]
193
+ - Better error messages in Sidekiq::Client [#4549]
194
+ - Remove rack-protection, reimplement CSRF protection [#4588]
195
+ - Require redis-rb 4.2 [#4591]
196
+ - Update to jquery 1.12.4 [#4593]
197
+ - Refactor internal fetch logic and API [#4602]
4
198
 
5
199
  6.0.7
6
200
  ---------
@@ -124,7 +318,7 @@ assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs_for(FooMailer).size
124
318
 
125
319
  This release has major breaking changes. Read and test carefully in production.
126
320
 
127
- - With Rails 6.0.1+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
321
+ - With Rails 6.0.2+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
128
322
  features/internals like the retry subsystem. [#4213, pirj]
129
323
  ```ruby
130
324
  class MyJob < ActiveJob::Base
@@ -160,6 +354,19 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
160
354
  - Integrate the StandardRB code formatter to ensure consistent code
161
355
  styling. [#4114, gearnode]
162
356
 
357
+ 5.2.10
358
+ ---------
359
+
360
+ - Backport fix for CVE-2022-23837.
361
+ - Migrate to `exists?` for redis-rb.
362
+ - Lock redis-rb to <4.6 to avoid deprecations.
363
+
364
+ 5.2.9
365
+ ---------
366
+
367
+ - Release Rack lock due to a cascade of CVEs. [#4566]
368
+ Pro-tip: don't lock Rack.
369
+
163
370
  5.2.8
164
371
  ---------
165
372
 
data/LICENSE CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
4
4
  the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
5
  for license text.
6
6
 
7
- Sidekiq Pro has a commercial-friendly license allowing private forks
8
- and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
9
- more detail. You can find the commercial license terms in COMM-LICENSE.
7
+ Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
8
+ You can find the commercial license in COMM-LICENSE.txt.
9
+ Please see https://sidekiq.org for purchasing options.
data/README.md CHANGED
@@ -2,11 +2,7 @@ Sidekiq
2
2
  ==============
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
5
- [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
6
- [![Test Coverage](https://codeclimate.com/github/mperham/sidekiq/badges/coverage.svg)](https://codeclimate.com/github/mperham/sidekiq/coverage)
7
- [![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
8
- [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
9
-
5
+ ![Build](https://github.com/mperham/sidekiq/workflows/CI/badge.svg)
10
6
 
11
7
  Simple, efficient background processing for Ruby.
12
8
 
@@ -40,17 +36,17 @@ Sidekiq 6.0 supports Rails 5.0+ but does not require it.
40
36
  Installation
41
37
  -----------------
42
38
 
43
- gem install sidekiq
39
+ bundle add sidekiq
44
40
 
45
41
 
46
42
  Getting Started
47
43
  -----------------
48
44
 
49
45
  See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
50
- You can watch [this Youtube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
46
+ You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
51
47
  Sidekiq and see its features in action. Here's the Web UI:
52
48
 
53
- ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
49
+ ![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
54
50
 
55
51
 
56
52
  Want to Upgrade?
@@ -84,14 +80,19 @@ Useful resources:
84
80
  Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
85
81
  See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
86
82
 
83
+ Contributing
84
+ -----------------
85
+
86
+ Please see [the contributing guidelines](https://github.com/mperham/sidekiq/blob/main/.github/contributing.md).
87
+
87
88
 
88
89
  License
89
90
  -----------------
90
91
 
91
- Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for licensing details.
92
+ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/main/LICENSE) for licensing details.
92
93
 
93
94
 
94
95
  Author
95
96
  -----------------
96
97
 
97
- Mike Perham, [@mperham@mastodon.xyz](https://mastodon.xyz/@mperham) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
98
+ Mike Perham, [@getajobmike](https://twitter.com/getajobmike) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
data/bin/sidekiq CHANGED
@@ -4,7 +4,7 @@
4
4
  # RUBYOPT=-w bundle exec sidekiq
5
5
  $TESTING = false
6
6
 
7
- require_relative '../lib/sidekiq/cli'
7
+ require_relative "../lib/sidekiq/cli"
8
8
 
9
9
  def integrate_with_systemd
10
10
  return unless ENV["NOTIFY_SOCKET"]
@@ -31,7 +31,12 @@ begin
31
31
  cli.run
32
32
  rescue => e
33
33
  raise e if $DEBUG
34
- STDERR.puts e.message
35
- STDERR.puts e.backtrace.join("\n")
34
+ if Sidekiq.error_handlers.length == 0
35
+ warn e.message
36
+ warn e.backtrace.join("\n")
37
+ else
38
+ cli.handle_exception e
39
+ end
40
+
36
41
  exit 1
37
42
  end
data/bin/sidekiqload CHANGED
@@ -4,19 +4,22 @@
4
4
  # RUBYOPT=-w bundle exec sidekiq
5
5
  $TESTING = false
6
6
 
7
- #require 'ruby-prof'
8
- require 'bundler/setup'
7
+ # require "ruby-prof"
8
+ require "bundler/setup"
9
9
  Bundler.require(:default, :load_test)
10
10
 
11
- require_relative '../lib/sidekiq/cli'
12
- require_relative '../lib/sidekiq/launcher'
11
+ require_relative "../lib/sidekiq/cli"
12
+ require_relative "../lib/sidekiq/launcher"
13
13
 
14
- include Sidekiq::Util
14
+ if ENV["SIDEKIQ_REDIS_CLIENT"]
15
+ Sidekiq::RedisConnection.adapter = :redis_client
16
+ end
15
17
 
16
18
  Sidekiq.configure_server do |config|
17
19
  config.options[:concurrency] = 10
18
- config.redis = { db: 13, port: 6380, driver: :hiredis }
19
- config.options[:queues] << 'default'
20
+ config.redis = {db: 13, port: 6380}
21
+ # config.redis = { db: 13, port: 6380, driver: :hiredis}
22
+ config.options[:queues] << "default"
20
23
  config.logger.level = Logger::ERROR
21
24
  config.average_scheduled_poll_interval = 2
22
25
  config.reliable! if defined?(Sidekiq::Pro)
@@ -29,53 +32,50 @@ class LoadWorker
29
32
  1
30
33
  end
31
34
 
32
- def perform(idx, ts=nil)
33
- puts(Time.now.to_f - ts) if ts != nil
34
- #raise idx.to_s if idx % 100 == 1
35
+ def perform(idx, ts = nil)
36
+ puts(Time.now.to_f - ts) if !ts.nil?
37
+ # raise idx.to_s if idx % 100 == 1
35
38
  end
36
39
  end
37
40
 
38
41
  # brew tap shopify/shopify
39
42
  # brew install toxiproxy
40
- # gem install toxiproxy
41
43
  # run `toxiproxy-server` in a separate terminal window.
42
- require 'toxiproxy'
44
+ require "toxiproxy"
43
45
  # simulate a non-localhost network for realer-world conditions.
44
46
  # adding 1ms of network latency has an ENORMOUS impact on benchmarks
45
47
  Toxiproxy.populate([{
46
- "name": "redis",
47
- "listen": "127.0.0.1:6380",
48
- "upstream": "127.0.0.1:6379"
48
+ name: "redis",
49
+ listen: "127.0.0.1:6380",
50
+ upstream: "127.0.0.1:6379"
49
51
  }])
50
52
 
51
53
  self_read, self_write = IO.pipe
52
- %w(INT TERM TSTP TTIN).each do |sig|
53
- begin
54
- trap sig do
55
- self_write.puts(sig)
56
- end
57
- rescue ArgumentError
58
- puts "Signal #{sig} not supported"
54
+ %w[INT TERM TSTP TTIN].each do |sig|
55
+ trap sig do
56
+ self_write.puts(sig)
59
57
  end
58
+ rescue ArgumentError
59
+ puts "Signal #{sig} not supported"
60
60
  end
61
61
 
62
- Sidekiq.redis {|c| c.flushdb}
62
+ Sidekiq.redis { |c| c.flushdb }
63
63
  def handle_signal(launcher, sig)
64
64
  Sidekiq.logger.debug "Got #{sig} signal"
65
65
  case sig
66
- when 'INT'
66
+ when "INT"
67
67
  # Handle Ctrl-C in JRuby like MRI
68
68
  # http://jira.codehaus.org/browse/JRUBY-4637
69
69
  raise Interrupt
70
- when 'TERM'
70
+ when "TERM"
71
71
  # Heroku sends TERM and then waits 30 seconds for process to exit.
72
72
  raise Interrupt
73
- when 'TSTP'
73
+ when "TSTP"
74
74
  Sidekiq.logger.info "Received TSTP, no longer accepting new work"
75
75
  launcher.quiet
76
- when 'TTIN'
76
+ when "TTIN"
77
77
  Thread.list.each do |thread|
78
- Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread['label']}"
78
+ Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread["label"]}"
79
79
  if thread.backtrace
80
80
  Sidekiq.logger.warn thread.backtrace.join("\n")
81
81
  else
@@ -89,52 +89,56 @@ def Process.rss
89
89
  `ps -o rss= -p #{Process.pid}`.chomp.to_i
90
90
  end
91
91
 
92
- iter = 10
92
+ iter = 50
93
93
  count = 10_000
94
94
 
95
95
  iter.times do
96
- arr = Array.new(count) do
97
- []
98
- end
99
- count.times do |idx|
100
- arr[idx][0] = idx
101
- end
102
- Sidekiq::Client.push_bulk('class' => LoadWorker, 'args' => arr)
96
+ arr = Array.new(count) { |idx| [idx] }
97
+ Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr)
103
98
  end
104
- Sidekiq.logger.error "Created #{count*iter} jobs"
99
+ Sidekiq.logger.error "Created #{count * iter} jobs"
105
100
 
106
101
  start = Time.now
107
102
 
108
103
  Monitoring = Thread.new do
109
- watchdog("monitor thread") do
110
- while true
104
+ while true
105
+ sleep 0.2
106
+ qsize = Sidekiq.redis do |conn|
107
+ conn.llen "queue:default"
108
+ end
109
+ total = qsize
110
+ # Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
111
+ if total == 0
112
+ Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
113
+ Sidekiq.logger.error("Now here's the latency for three jobs")
114
+
115
+ LoadWorker.perform_async(1, Time.now.to_f)
116
+ LoadWorker.perform_async(2, Time.now.to_f)
117
+ LoadWorker.perform_async(3, Time.now.to_f)
118
+
111
119
  sleep 0.2
112
- qsize = Sidekiq.redis do |conn|
113
- conn.llen "queue:default"
114
- end
115
- total = qsize
116
- #Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
117
- if total == 0
118
- Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
119
- Sidekiq.logger.error("Now here's the latency for three jobs")
120
-
121
- LoadWorker.perform_async(1, Time.now.to_f)
122
- LoadWorker.perform_async(2, Time.now.to_f)
123
- LoadWorker.perform_async(3, Time.now.to_f)
124
-
125
- sleep 0.2
126
- exit(0)
127
- end
120
+ exit(0)
128
121
  end
129
122
  end
130
123
  end
131
124
 
125
+ def with_latency(latency, &block)
126
+ Sidekiq.logger.error "Simulating #{latency}ms of latency between Sidekiq and redis"
127
+ if latency > 0
128
+ Toxiproxy[:redis].downstream(:latency, latency: latency).apply(&block)
129
+ else
130
+ yield
131
+ end
132
+ end
133
+
132
134
  begin
133
- #RubyProf::exclude_threads = [ Monitoring ]
134
- #RubyProf.start
135
- fire_event(:startup)
136
- Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
137
- Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
135
+ # RubyProf::exclude_threads = [ Monitoring ]
136
+ # RubyProf.start
137
+ events = Sidekiq.options[:lifecycle_events][:startup]
138
+ events.each(&:call)
139
+ events.clear
140
+
141
+ with_latency(Integer(ENV.fetch("LATENCY", "1"))) do
138
142
  launcher = Sidekiq::Launcher.new(Sidekiq.options)
139
143
  launcher.run
140
144
 
@@ -144,14 +148,14 @@ begin
144
148
  end
145
149
  end
146
150
  rescue SystemExit => e
147
- #Sidekiq.logger.error("Profiling...")
148
- #result = RubyProf.stop
149
- #printer = RubyProf::GraphHtmlPrinter.new(result)
150
- #printer.print(File.new("output.html", "w"), :min_percent => 1)
151
+ # Sidekiq.logger.error("Profiling...")
152
+ # result = RubyProf.stop
153
+ # printer = RubyProf::GraphHtmlPrinter.new(result)
154
+ # printer.print(File.new("output.html", "w"), :min_percent => 1)
151
155
  # normal
152
156
  rescue => e
153
157
  raise e if $DEBUG
154
- STDERR.puts e.message
155
- STDERR.puts e.backtrace.join("\n")
158
+ warn e.message
159
+ warn e.backtrace.join("\n")
156
160
  exit 1
157
161
  end
data/bin/sidekiqmon CHANGED
@@ -1,6 +1,6 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- require 'sidekiq/monitor'
3
+ require "sidekiq/monitor"
4
4
 
5
5
  section = "all"
6
6
  section = ARGV[0] if ARGV.size == 1
@@ -0,0 +1,57 @@
1
+ require "rails/generators/named_base"
2
+
3
+ module Sidekiq
4
+ module Generators # :nodoc:
5
+ class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
6
+ desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
7
+
8
+ check_class_collision suffix: "Job"
9
+
10
+ def self.default_generator_root
11
+ File.dirname(__FILE__)
12
+ end
13
+
14
+ def create_job_file
15
+ template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
16
+ end
17
+
18
+ def create_test_file
19
+ return unless test_framework
20
+
21
+ if test_framework == :rspec
22
+ create_job_spec
23
+ else
24
+ create_job_test
25
+ end
26
+ end
27
+
28
+ private
29
+
30
+ def create_job_spec
31
+ template_file = File.join(
32
+ "spec/sidekiq",
33
+ class_path,
34
+ "#{file_name}_job_spec.rb"
35
+ )
36
+ template "job_spec.rb.erb", template_file
37
+ end
38
+
39
+ def create_job_test
40
+ template_file = File.join(
41
+ "test/sidekiq",
42
+ class_path,
43
+ "#{file_name}_job_test.rb"
44
+ )
45
+ template "job_test.rb.erb", template_file
46
+ end
47
+
48
+ def file_name
49
+ @_file_name ||= super.sub(/_?job\z/i, "")
50
+ end
51
+
52
+ def test_framework
53
+ ::Rails.application.config.generators.options[:rails][:test_framework]
54
+ end
55
+ end
56
+ end
57
+ end
@@ -1,6 +1,6 @@
1
1
  <% module_namespacing do -%>
2
- class <%= class_name %>Worker
3
- include Sidekiq::Worker
2
+ class <%= class_name %>Job
3
+ include Sidekiq::Job
4
4
 
5
5
  def perform(*args)
6
6
  # Do something
@@ -1,6 +1,6 @@
1
1
  require 'rails_helper'
2
2
  <% module_namespacing do -%>
3
- RSpec.describe <%= class_name %>Worker, type: :worker do
3
+ RSpec.describe <%= class_name %>Job, type: :job do
4
4
  pending "add some examples to (or delete) #{__FILE__}"
5
5
  end
6
6
  <% end -%>
@@ -1,6 +1,6 @@
1
1
  require 'test_helper'
2
2
  <% module_namespacing do -%>
3
- class <%= class_name %>WorkerTest < Minitest::Test
3
+ class <%= class_name %>JobTest < Minitest::Test
4
4
  def test_example
5
5
  skip "add some examples to (or delete) #{__FILE__}"
6
6
  end