sidekiq 6.0.7 → 6.4.2

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (101) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +189 -2
  3. data/LICENSE +3 -3
  4. data/README.md +11 -10
  5. data/bin/sidekiq +8 -3
  6. data/bin/sidekiqload +57 -65
  7. data/bin/sidekiqmon +1 -1
  8. data/lib/generators/sidekiq/job_generator.rb +57 -0
  9. data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
  10. data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
  11. data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
  12. data/lib/sidekiq/api.rb +164 -116
  13. data/lib/sidekiq/cli.rb +49 -15
  14. data/lib/sidekiq/client.rb +51 -70
  15. data/lib/sidekiq/delay.rb +2 -0
  16. data/lib/sidekiq/extensions/action_mailer.rb +3 -2
  17. data/lib/sidekiq/extensions/active_record.rb +4 -3
  18. data/lib/sidekiq/extensions/class_methods.rb +5 -4
  19. data/lib/sidekiq/extensions/generic_proxy.rb +4 -2
  20. data/lib/sidekiq/fetch.rb +32 -23
  21. data/lib/sidekiq/job.rb +13 -0
  22. data/lib/sidekiq/job_logger.rb +16 -28
  23. data/lib/sidekiq/job_retry.rb +32 -33
  24. data/lib/sidekiq/job_util.rb +67 -0
  25. data/lib/sidekiq/launcher.rb +113 -54
  26. data/lib/sidekiq/logger.rb +11 -20
  27. data/lib/sidekiq/manager.rb +16 -18
  28. data/lib/sidekiq/middleware/chain.rb +10 -8
  29. data/lib/sidekiq/middleware/current_attributes.rb +57 -0
  30. data/lib/sidekiq/middleware/i18n.rb +4 -4
  31. data/lib/sidekiq/monitor.rb +1 -1
  32. data/lib/sidekiq/paginator.rb +8 -8
  33. data/lib/sidekiq/processor.rb +31 -31
  34. data/lib/sidekiq/rails.rb +36 -20
  35. data/lib/sidekiq/redis_connection.rb +16 -15
  36. data/lib/sidekiq/scheduled.rb +51 -16
  37. data/lib/sidekiq/sd_notify.rb +1 -1
  38. data/lib/sidekiq/testing/inline.rb +4 -4
  39. data/lib/sidekiq/testing.rb +38 -39
  40. data/lib/sidekiq/util.rb +41 -0
  41. data/lib/sidekiq/version.rb +1 -1
  42. data/lib/sidekiq/web/action.rb +2 -2
  43. data/lib/sidekiq/web/application.rb +21 -12
  44. data/lib/sidekiq/web/csrf_protection.rb +180 -0
  45. data/lib/sidekiq/web/helpers.rb +39 -33
  46. data/lib/sidekiq/web/router.rb +5 -2
  47. data/lib/sidekiq/web.rb +36 -72
  48. data/lib/sidekiq/worker.rb +135 -16
  49. data/lib/sidekiq.rb +33 -17
  50. data/sidekiq.gemspec +11 -4
  51. data/web/assets/images/apple-touch-icon.png +0 -0
  52. data/web/assets/javascripts/application.js +113 -65
  53. data/web/assets/javascripts/dashboard.js +51 -51
  54. data/web/assets/stylesheets/application-dark.css +64 -43
  55. data/web/assets/stylesheets/application-rtl.css +0 -4
  56. data/web/assets/stylesheets/application.css +42 -239
  57. data/web/locales/ar.yml +8 -2
  58. data/web/locales/en.yml +4 -1
  59. data/web/locales/es.yml +18 -2
  60. data/web/locales/fr.yml +8 -1
  61. data/web/locales/ja.yml +3 -0
  62. data/web/locales/lt.yml +1 -1
  63. data/web/locales/pl.yml +4 -4
  64. data/web/locales/ru.yml +4 -0
  65. data/web/views/_footer.erb +1 -1
  66. data/web/views/_job_info.erb +1 -1
  67. data/web/views/_poll_link.erb +2 -5
  68. data/web/views/_summary.erb +7 -7
  69. data/web/views/busy.erb +51 -20
  70. data/web/views/dashboard.erb +22 -14
  71. data/web/views/dead.erb +1 -1
  72. data/web/views/layout.erb +2 -1
  73. data/web/views/morgue.erb +6 -6
  74. data/web/views/queue.erb +11 -11
  75. data/web/views/queues.erb +4 -4
  76. data/web/views/retries.erb +7 -7
  77. data/web/views/retry.erb +1 -1
  78. data/web/views/scheduled.erb +1 -1
  79. metadata +24 -49
  80. data/.circleci/config.yml +0 -60
  81. data/.github/contributing.md +0 -32
  82. data/.github/issue_template.md +0 -11
  83. data/.gitignore +0 -13
  84. data/.standard.yml +0 -20
  85. data/3.0-Upgrade.md +0 -70
  86. data/4.0-Upgrade.md +0 -53
  87. data/5.0-Upgrade.md +0 -56
  88. data/6.0-Upgrade.md +0 -72
  89. data/COMM-LICENSE +0 -97
  90. data/Ent-2.0-Upgrade.md +0 -37
  91. data/Ent-Changes.md +0 -256
  92. data/Gemfile +0 -24
  93. data/Gemfile.lock +0 -208
  94. data/Pro-2.0-Upgrade.md +0 -138
  95. data/Pro-3.0-Upgrade.md +0 -44
  96. data/Pro-4.0-Upgrade.md +0 -35
  97. data/Pro-5.0-Upgrade.md +0 -25
  98. data/Pro-Changes.md +0 -782
  99. data/Rakefile +0 -10
  100. data/code_of_conduct.md +0 -50
  101. data/lib/generators/sidekiq/worker_generator.rb +0 -57
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3dd5d7a0dd8a7d877484df4ae3b9988f15e9cc4bc0bb4da23224be0049fa733e
4
- data.tar.gz: e2b7ddb4c045817837e996356851648bf32108b37fa1abcfb72795a185b4a238
3
+ metadata.gz: 70d899b76bdd764a2ec7e5f23b6d056494b58897daaf49db672daad3e0f60237
4
+ data.tar.gz: daae0430ecaf8eb7f172c70e628581c20a17e1a7cb1a58170b9082fc3dde1ea2
5
5
  SHA512:
6
- metadata.gz: 5ce3f911b835b38ce212309bcdf81b4789e24a12898a0f2071b9adb9cbcdfa23a33bb7216c2f7c6dcab26d484f79a1271594d3218f63ed041f0027deab5a26f3
7
- data.tar.gz: 685a6890a43c990b2fd9709c49f9c82c96a487cdcf6fc0607ab15b9dbb47d52c9f797ec5094d3f92ec35c5737e661f966a6b81006ee4b7d99a7f31285180b3db
6
+ metadata.gz: 930a0feb7ff47473eb995f8ba695468498afecb311c25296c93042ecfa00cd2bf48e19efd5ba775b4895c065705a344043e0c0e5d7b4115c2dceb1ec7cf341f9
7
+ data.tar.gz: 14ab645609e7c9f3fa954432097cd46d6e9e74e5bddc9ed1541697d13f3399add407ffb5af23c89c82651793c97e2c54b8fd4f845ca20287ce6bf033fd9caed2
data/Changes.md CHANGED
@@ -1,6 +1,180 @@
1
1
  # Sidekiq Changes
2
2
 
3
- [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
4
+
5
+ 6.4.2
6
+ ---------
7
+
8
+ - Strict argument checking now runs after client-side middleware [#5246]
9
+ - Fix page events with live polling [#5184]
10
+ - Many under-the-hood changes to remove all usage of the term "worker"
11
+ from the Sidekiq codebase and APIs. This mostly involved RDoc and local
12
+ variable names but a few constants and public APIs were changed. The old
13
+ APIs will be removed in Sidekiq 7.0.
14
+ ```
15
+ Sidekiq::DEFAULT_WORKER_OPTIONS -> Sidekiq.default_job_options
16
+ Sidekiq.default_worker_options -> Sidekiq.default_job_options
17
+ Sidekiq::Queues["default"].jobs_by_worker(HardJob) -> Sidekiq::Queues["default"].jobs_by_class(HardJob)
18
+ ```
19
+
20
+ 6.4.1
21
+ ---------
22
+
23
+ - Fix pipeline/multi deprecations in redis-rb 4.6
24
+ - Fix sidekiq.yml YAML load errors on Ruby 3.1 [#5141]
25
+ - Sharding support for `perform_bulk` [#5129]
26
+ - Refactor job logger for SPEEEEEEED
27
+
28
+ 6.4.0
29
+ ---------
30
+
31
+ - **SECURITY**: Validate input to avoid possible DoS in Web UI.
32
+ - Add **strict argument checking** [#5071]
33
+ Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
34
+ Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
35
+ This warning will switch to an exception in Sidekiq 7.0.
36
+ - Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
37
+ - Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
38
+ ```ruby
39
+ SomeJob.perform_async(args...)
40
+ SomeJob.perform_sync(args...)
41
+ SomeJob.perform_inline(args...)
42
+ ```
43
+ You can also dynamically redirect a job to run synchronously:
44
+ ```ruby
45
+ SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
46
+ ```
47
+ - Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
48
+ ```
49
+ bin/rails generate sidekiq:job ProcessOrderJob
50
+ ```
51
+ - Fix job retries losing CurrentAttributes [#5090]
52
+ - Tweak shutdown to give long-running threads time to cleanup [#5095]
53
+
54
+ 6.3.1
55
+ ---------
56
+
57
+ - Fix keyword arguments error with CurrentAttributes on Ruby 3.0 [#5048]
58
+
59
+ 6.3.0
60
+ ---------
61
+
62
+ - **BREAK**: The Web UI has been refactored to remove jQuery. Any UI extensions
63
+ which use jQuery will break.
64
+ - **FEATURE**: Sidekiq.logger has been enhanced so any `Rails.logger`
65
+ output in jobs now shows up in the Sidekiq console. Remove any logger
66
+ hacks in your initializer and see if it Just Works™ now. [#5021]
67
+ - **FEATURE**: Add `Sidekiq::Job` alias for `Sidekiq::Worker`, to better
68
+ reflect industry standard terminology. You can now do this:
69
+ ```ruby
70
+ class MyJob
71
+ include Sidekiq::Job
72
+ sidekiq_options ...
73
+ def perform(args)
74
+ end
75
+ end
76
+ ```
77
+ - **FEATURE**: Support for serializing ActiveSupport::CurrentAttributes into each job. [#4982]
78
+ ```ruby
79
+ # config/initializers/sidekiq.rb
80
+ require "sidekiq/middleware/current_attributes"
81
+ Sidekiq::CurrentAttributes.persist(Myapp::Current) # Your AS::CurrentAttributes singleton
82
+ ```
83
+ - **FEATURE**: Add `Sidekiq::Worker.perform_bulk` for enqueuing jobs in bulk,
84
+ similar to `Sidekiq::Client.push_bulk` [#5042]
85
+ ```ruby
86
+ MyJob.perform_bulk([[1], [2], [3]])
87
+ ```
88
+ - Implement `queue_as`, `wait` and `wait_until` for ActiveJob compatibility [#5003]
89
+ - Scheduler now uses Lua to reduce Redis load and network roundtrips [#5044]
90
+ - Retry Redis operation if we get an `UNBLOCKED` Redis error [#4985]
91
+ - Run existing signal traps, if any, before running Sidekiq's trap [#4991]
92
+ - Fix fetch bug when using weighted queues which caused Sidekiq to stop
93
+ processing queues randomly [#5031]
94
+
95
+ 6.2.2
96
+ ---------
97
+
98
+ - Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
99
+ - Minimize scheduler load on Redis at scale [#4882]
100
+ - Improve logging of delay jobs [#4904, BuonOno]
101
+ - Minor CSS improvements for buttons and tables, design PRs always welcome!
102
+ - Tweak Web UI `Cache-Control` header [#4966]
103
+ - Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
104
+
105
+ 6.2.1
106
+ ---------
107
+
108
+ - Update RTT warning logic to handle transient RTT spikes [#4851]
109
+ - Fix very low priority CVE on unescaped queue name [#4852]
110
+ - Add note about sessions and Rails apps in API mode
111
+
112
+ 6.2.0
113
+ ---------
114
+
115
+ - Store Redis RTT and log if poor [#4824]
116
+ - Add process/thread stats to Busy page [#4806]
117
+ - Improve Web UI on mobile devices [#4840]
118
+ - **Refactor Web UI session usage** [#4804]
119
+ Numerous people have hit "Forbidden" errors and struggled with Sidekiq's
120
+ Web UI session requirement. If you have code in your initializer for
121
+ Web sessions, it's quite possible it will need to be removed. Here's
122
+ an overview:
123
+ ```
124
+ Sidekiq::Web needs a valid Rack session for CSRF protection. If this is a Rails app,
125
+ make sure you mount Sidekiq::Web *inside* your routes in `config/routes.rb` so
126
+ Sidekiq can reuse the Rails session:
127
+
128
+ Rails.application.routes.draw do
129
+ mount Sidekiq::Web => "/sidekiq"
130
+ ....
131
+ end
132
+
133
+ If this is a bare Rack app, use a session middleware before Sidekiq::Web:
134
+
135
+ # first, use IRB to create a shared secret key for sessions and commit it
136
+ require 'securerandom'; File.open(".session.key", "w") {|f| f.write(SecureRandom.hex(32)) }
137
+
138
+ # now, update your Rack app to include the secret with a session cookie middleware
139
+ use Rack::Session::Cookie, secret: File.read(".session.key"), same_site: true, max_age: 86400
140
+ run Sidekiq::Web
141
+
142
+ If this is a Rails app in API mode, you need to enable sessions.
143
+
144
+ https://guides.rubyonrails.org/api_app.html#using-session-middlewares
145
+ ```
146
+
147
+ 6.1.3
148
+ ---------
149
+
150
+ - Warn if Redis is configured to evict data under memory pressure [#4752]
151
+ - Add process RSS on the Busy page [#4717]
152
+
153
+ 6.1.2
154
+ ---------
155
+
156
+ - Improve readability in dark mode Web UI [#4674]
157
+ - Fix Web UI crash with corrupt session [#4672]
158
+ - Allow middleware to yield arguments [#4673, @eugeneius]
159
+ - Migrate CI from CircleCI to GitHub Actions [#4677]
160
+
161
+ 6.1.1
162
+ ---------
163
+
164
+ - Jobs are now sorted by age in the Busy Workers table. [#4641]
165
+ - Fix "check all" JS logic in Web UI [#4619]
166
+
167
+ 6.1.0
168
+ ---------
169
+
170
+ - Web UI - Dark Mode fixes [#4543, natematykiewicz]
171
+ - Ensure `Rack::ContentLength` is loaded as middleware for correct Web UI responses [#4541]
172
+ - Avoid exception dumping SSL store in Redis connection logging [#4532]
173
+ - Better error messages in Sidekiq::Client [#4549]
174
+ - Remove rack-protection, reimplement CSRF protection [#4588]
175
+ - Require redis-rb 4.2 [#4591]
176
+ - Update to jquery 1.12.4 [#4593]
177
+ - Refactor internal fetch logic and API [#4602]
4
178
 
5
179
  6.0.7
6
180
  ---------
@@ -124,7 +298,7 @@ assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs_for(FooMailer).size
124
298
 
125
299
  This release has major breaking changes. Read and test carefully in production.
126
300
 
127
- - With Rails 6.0.1+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
301
+ - With Rails 6.0.2+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
128
302
  features/internals like the retry subsystem. [#4213, pirj]
129
303
  ```ruby
130
304
  class MyJob < ActiveJob::Base
@@ -160,6 +334,19 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
160
334
  - Integrate the StandardRB code formatter to ensure consistent code
161
335
  styling. [#4114, gearnode]
162
336
 
337
+ 5.2.10
338
+ ---------
339
+
340
+ - Backport fix for CVE-2022-23837.
341
+ - Migrate to `exists?` for redis-rb.
342
+ - Lock redis-rb to <4.6 to avoid deprecations.
343
+
344
+ 5.2.9
345
+ ---------
346
+
347
+ - Release Rack lock due to a cascade of CVEs. [#4566]
348
+ Pro-tip: don't lock Rack.
349
+
163
350
  5.2.8
164
351
  ---------
165
352
 
data/LICENSE CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
4
4
  the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
5
  for license text.
6
6
 
7
- Sidekiq Pro has a commercial-friendly license allowing private forks
8
- and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
9
- more detail. You can find the commercial license terms in COMM-LICENSE.
7
+ Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
8
+ You can find the commercial license in COMM-LICENSE.txt.
9
+ Please see https://sidekiq.org for purchasing options.
data/README.md CHANGED
@@ -2,11 +2,7 @@ Sidekiq
2
2
  ==============
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
5
- [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
6
- [![Test Coverage](https://codeclimate.com/github/mperham/sidekiq/badges/coverage.svg)](https://codeclimate.com/github/mperham/sidekiq/coverage)
7
- [![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
8
- [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
9
-
5
+ ![Build](https://github.com/mperham/sidekiq/workflows/CI/badge.svg)
10
6
 
11
7
  Simple, efficient background processing for Ruby.
12
8
 
@@ -40,17 +36,17 @@ Sidekiq 6.0 supports Rails 5.0+ but does not require it.
40
36
  Installation
41
37
  -----------------
42
38
 
43
- gem install sidekiq
39
+ bundle add sidekiq
44
40
 
45
41
 
46
42
  Getting Started
47
43
  -----------------
48
44
 
49
45
  See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
50
- You can watch [this Youtube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
46
+ You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
51
47
  Sidekiq and see its features in action. Here's the Web UI:
52
48
 
53
- ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
49
+ ![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
54
50
 
55
51
 
56
52
  Want to Upgrade?
@@ -84,14 +80,19 @@ Useful resources:
84
80
  Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
85
81
  See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
86
82
 
83
+ Contributing
84
+ -----------------
85
+
86
+ Please see [the contributing guidelines](https://github.com/mperham/sidekiq/blob/main/.github/contributing.md).
87
+
87
88
 
88
89
  License
89
90
  -----------------
90
91
 
91
- Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for licensing details.
92
+ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/main/LICENSE) for licensing details.
92
93
 
93
94
 
94
95
  Author
95
96
  -----------------
96
97
 
97
- Mike Perham, [@mperham@mastodon.xyz](https://mastodon.xyz/@mperham) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
98
+ Mike Perham, [@getajobmike](https://twitter.com/getajobmike) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
data/bin/sidekiq CHANGED
@@ -4,7 +4,7 @@
4
4
  # RUBYOPT=-w bundle exec sidekiq
5
5
  $TESTING = false
6
6
 
7
- require_relative '../lib/sidekiq/cli'
7
+ require_relative "../lib/sidekiq/cli"
8
8
 
9
9
  def integrate_with_systemd
10
10
  return unless ENV["NOTIFY_SOCKET"]
@@ -31,7 +31,12 @@ begin
31
31
  cli.run
32
32
  rescue => e
33
33
  raise e if $DEBUG
34
- STDERR.puts e.message
35
- STDERR.puts e.backtrace.join("\n")
34
+ if Sidekiq.error_handlers.length == 0
35
+ warn e.message
36
+ warn e.backtrace.join("\n")
37
+ else
38
+ cli.handle_exception e
39
+ end
40
+
36
41
  exit 1
37
42
  end
data/bin/sidekiqload CHANGED
@@ -4,19 +4,18 @@
4
4
  # RUBYOPT=-w bundle exec sidekiq
5
5
  $TESTING = false
6
6
 
7
- #require 'ruby-prof'
8
- require 'bundler/setup'
7
+ # require "ruby-prof"
8
+ require "bundler/setup"
9
9
  Bundler.require(:default, :load_test)
10
10
 
11
- require_relative '../lib/sidekiq/cli'
12
- require_relative '../lib/sidekiq/launcher'
13
-
14
- include Sidekiq::Util
11
+ require_relative "../lib/sidekiq/cli"
12
+ require_relative "../lib/sidekiq/launcher"
15
13
 
16
14
  Sidekiq.configure_server do |config|
17
15
  config.options[:concurrency] = 10
18
- config.redis = { db: 13, port: 6380, driver: :hiredis }
19
- config.options[:queues] << 'default'
16
+ config.redis = {db: 13, port: 6380}
17
+ # config.redis = { db: 13, port: 6380, driver: :hiredis}
18
+ config.options[:queues] << "default"
20
19
  config.logger.level = Logger::ERROR
21
20
  config.average_scheduled_poll_interval = 2
22
21
  config.reliable! if defined?(Sidekiq::Pro)
@@ -29,53 +28,50 @@ class LoadWorker
29
28
  1
30
29
  end
31
30
 
32
- def perform(idx, ts=nil)
33
- puts(Time.now.to_f - ts) if ts != nil
34
- #raise idx.to_s if idx % 100 == 1
31
+ def perform(idx, ts = nil)
32
+ puts(Time.now.to_f - ts) if !ts.nil?
33
+ # raise idx.to_s if idx % 100 == 1
35
34
  end
36
35
  end
37
36
 
38
37
  # brew tap shopify/shopify
39
38
  # brew install toxiproxy
40
- # gem install toxiproxy
41
39
  # run `toxiproxy-server` in a separate terminal window.
42
- require 'toxiproxy'
40
+ require "toxiproxy"
43
41
  # simulate a non-localhost network for realer-world conditions.
44
42
  # adding 1ms of network latency has an ENORMOUS impact on benchmarks
45
43
  Toxiproxy.populate([{
46
- "name": "redis",
47
- "listen": "127.0.0.1:6380",
48
- "upstream": "127.0.0.1:6379"
44
+ name: "redis",
45
+ listen: "127.0.0.1:6380",
46
+ upstream: "127.0.0.1:6379"
49
47
  }])
50
48
 
51
49
  self_read, self_write = IO.pipe
52
- %w(INT TERM TSTP TTIN).each do |sig|
53
- begin
54
- trap sig do
55
- self_write.puts(sig)
56
- end
57
- rescue ArgumentError
58
- puts "Signal #{sig} not supported"
50
+ %w[INT TERM TSTP TTIN].each do |sig|
51
+ trap sig do
52
+ self_write.puts(sig)
59
53
  end
54
+ rescue ArgumentError
55
+ puts "Signal #{sig} not supported"
60
56
  end
61
57
 
62
- Sidekiq.redis {|c| c.flushdb}
58
+ Sidekiq.redis { |c| c.flushdb }
63
59
  def handle_signal(launcher, sig)
64
60
  Sidekiq.logger.debug "Got #{sig} signal"
65
61
  case sig
66
- when 'INT'
62
+ when "INT"
67
63
  # Handle Ctrl-C in JRuby like MRI
68
64
  # http://jira.codehaus.org/browse/JRUBY-4637
69
65
  raise Interrupt
70
- when 'TERM'
66
+ when "TERM"
71
67
  # Heroku sends TERM and then waits 30 seconds for process to exit.
72
68
  raise Interrupt
73
- when 'TSTP'
69
+ when "TSTP"
74
70
  Sidekiq.logger.info "Received TSTP, no longer accepting new work"
75
71
  launcher.quiet
76
- when 'TTIN'
72
+ when "TTIN"
77
73
  Thread.list.each do |thread|
78
- Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread['label']}"
74
+ Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread["label"]}"
79
75
  if thread.backtrace
80
76
  Sidekiq.logger.warn thread.backtrace.join("\n")
81
77
  else
@@ -89,50 +85,46 @@ def Process.rss
89
85
  `ps -o rss= -p #{Process.pid}`.chomp.to_i
90
86
  end
91
87
 
92
- iter = 10
88
+ iter = 50
93
89
  count = 10_000
94
90
 
95
91
  iter.times do
96
- arr = Array.new(count) do
97
- []
98
- end
99
- count.times do |idx|
100
- arr[idx][0] = idx
101
- end
102
- Sidekiq::Client.push_bulk('class' => LoadWorker, 'args' => arr)
92
+ arr = Array.new(count) { |idx| [idx] }
93
+ Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr)
103
94
  end
104
- Sidekiq.logger.error "Created #{count*iter} jobs"
95
+ Sidekiq.logger.error "Created #{count * iter} jobs"
105
96
 
106
97
  start = Time.now
107
98
 
108
99
  Monitoring = Thread.new do
109
- watchdog("monitor thread") do
110
- while true
100
+ while true
101
+ sleep 0.2
102
+ qsize = Sidekiq.redis do |conn|
103
+ conn.llen "queue:default"
104
+ end
105
+ total = qsize
106
+ # Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
107
+ if total == 0
108
+ Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
109
+ Sidekiq.logger.error("Now here's the latency for three jobs")
110
+
111
+ LoadWorker.perform_async(1, Time.now.to_f)
112
+ LoadWorker.perform_async(2, Time.now.to_f)
113
+ LoadWorker.perform_async(3, Time.now.to_f)
114
+
111
115
  sleep 0.2
112
- qsize = Sidekiq.redis do |conn|
113
- conn.llen "queue:default"
114
- end
115
- total = qsize
116
- #Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
117
- if total == 0
118
- Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
119
- Sidekiq.logger.error("Now here's the latency for three jobs")
120
-
121
- LoadWorker.perform_async(1, Time.now.to_f)
122
- LoadWorker.perform_async(2, Time.now.to_f)
123
- LoadWorker.perform_async(3, Time.now.to_f)
124
-
125
- sleep 0.2
126
- exit(0)
127
- end
116
+ exit(0)
128
117
  end
129
118
  end
130
119
  end
131
120
 
132
121
  begin
133
- #RubyProf::exclude_threads = [ Monitoring ]
134
- #RubyProf.start
135
- fire_event(:startup)
122
+ # RubyProf::exclude_threads = [ Monitoring ]
123
+ # RubyProf.start
124
+ events = Sidekiq.options[:lifecycle_events][:startup]
125
+ events.each(&:call)
126
+ events.clear
127
+
136
128
  Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
137
129
  Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
138
130
  launcher = Sidekiq::Launcher.new(Sidekiq.options)
@@ -144,14 +136,14 @@ begin
144
136
  end
145
137
  end
146
138
  rescue SystemExit => e
147
- #Sidekiq.logger.error("Profiling...")
148
- #result = RubyProf.stop
149
- #printer = RubyProf::GraphHtmlPrinter.new(result)
150
- #printer.print(File.new("output.html", "w"), :min_percent => 1)
139
+ # Sidekiq.logger.error("Profiling...")
140
+ # result = RubyProf.stop
141
+ # printer = RubyProf::GraphHtmlPrinter.new(result)
142
+ # printer.print(File.new("output.html", "w"), :min_percent => 1)
151
143
  # normal
152
144
  rescue => e
153
145
  raise e if $DEBUG
154
- STDERR.puts e.message
155
- STDERR.puts e.backtrace.join("\n")
146
+ warn e.message
147
+ warn e.backtrace.join("\n")
156
148
  exit 1
157
149
  end
data/bin/sidekiqmon CHANGED
@@ -1,6 +1,6 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- require 'sidekiq/monitor'
3
+ require "sidekiq/monitor"
4
4
 
5
5
  section = "all"
6
6
  section = ARGV[0] if ARGV.size == 1
@@ -0,0 +1,57 @@
1
+ require "rails/generators/named_base"
2
+
3
+ module Sidekiq
4
+ module Generators # :nodoc:
5
+ class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
6
+ desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
7
+
8
+ check_class_collision suffix: "Job"
9
+
10
+ def self.default_generator_root
11
+ File.dirname(__FILE__)
12
+ end
13
+
14
+ def create_job_file
15
+ template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
16
+ end
17
+
18
+ def create_test_file
19
+ return unless test_framework
20
+
21
+ if test_framework == :rspec
22
+ create_job_spec
23
+ else
24
+ create_job_test
25
+ end
26
+ end
27
+
28
+ private
29
+
30
+ def create_job_spec
31
+ template_file = File.join(
32
+ "spec/sidekiq",
33
+ class_path,
34
+ "#{file_name}_job_spec.rb"
35
+ )
36
+ template "job_spec.rb.erb", template_file
37
+ end
38
+
39
+ def create_job_test
40
+ template_file = File.join(
41
+ "test/sidekiq",
42
+ class_path,
43
+ "#{file_name}_job_test.rb"
44
+ )
45
+ template "job_test.rb.erb", template_file
46
+ end
47
+
48
+ def file_name
49
+ @_file_name ||= super.sub(/_?job\z/i, "")
50
+ end
51
+
52
+ def test_framework
53
+ ::Rails.application.config.generators.options[:rails][:test_framework]
54
+ end
55
+ end
56
+ end
57
+ end
@@ -1,6 +1,6 @@
1
1
  <% module_namespacing do -%>
2
- class <%= class_name %>Worker
3
- include Sidekiq::Worker
2
+ class <%= class_name %>Job
3
+ include Sidekiq::Job
4
4
 
5
5
  def perform(*args)
6
6
  # Do something
@@ -1,6 +1,6 @@
1
1
  require 'rails_helper'
2
2
  <% module_namespacing do -%>
3
- RSpec.describe <%= class_name %>Worker, type: :worker do
3
+ RSpec.describe <%= class_name %>Job, type: :job do
4
4
  pending "add some examples to (or delete) #{__FILE__}"
5
5
  end
6
6
  <% end -%>
@@ -1,6 +1,6 @@
1
1
  require 'test_helper'
2
2
  <% module_namespacing do -%>
3
- class <%= class_name %>WorkerTest < Minitest::Test
3
+ class <%= class_name %>JobTest < Minitest::Test
4
4
  def test_example
5
5
  skip "add some examples to (or delete) #{__FILE__}"
6
6
  end