sidekiq 4.0.0 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 154fd4d987e5f7c32fad42a20176d27e958bd49f
-  data.tar.gz: a33449473a2210e1f79e3f814c50f6fe982ff101
+  metadata.gz: f2a9cd7270e53981aaf7b3f2639596ff042f5ae3
+  data.tar.gz: 7b6e4fbfee794de34d9b6b15772cbf085e05a65b
 SHA512:
-  metadata.gz: 264d2c5a210a978ca8ce9f2a78962cc4f4d62966444788b7d942644881baec212133123f3434ffafdcc21d5bd9c0130eef007195a945f86771a33da5b2d1fbea
-  data.tar.gz: 6e651a3d38dd4518a3db01abd5febdb9a4c739d16873f2ce3ffa4998bd5f5a2285303437cc6f489cb43d032590bf741006d50170470c333ed31cfa62c1168178
+  metadata.gz: dfaecc27ce4071e36b5ab994c27f59d85b4cf7f16ca05a5b786c79125c311f58fbb38b94f7e1c71e4dbf90e6645cb358102e8940a94881981f7bbc2578fd18ff
+  data.tar.gz: a98e9b87e5f4abe2580ea5fcb6f20ef1d28623eae9be4ca758d5f5b52c65dbef7a6b5df74f673449177f36494b9075573315c7ef6cd6b757ba6a59e825737833

data/.github/issue_template.md

@@ -0,0 +1,9 @@
+Ruby version:
+Sidekiq / Pro / Enterprise version(s):
+
+Please include your initializer and any error message with the full backtrace.
+
+Are you using an old version?
+Have you checked the changelog to see if your issue has been fixed in a later version?
+
+https://github.com/mperham/sidekiq/blob/master/Changes.md

data/.gitignore

@@ -10,3 +10,4 @@ vendor/
 .bundle/
 .sass-cache/
 tmp/
+pkg/*.gem

data/.travis.yml

@@ -3,14 +3,16 @@ sudo: false
 cache: bundler
 services:
   - redis-server
-rvm:
-  - 2.1
-  - 2.0.0
-  - 2.2
-  - jruby
-  - jruby-head
-  - rbx-2
+before_install:
+  - gem install bundler
+  - gem update bundler
 matrix:
-  allow_failures:
-    - rvm: rbx-2
-    - rvm: jruby-head
+  include:
+    - rvm: 2.2.4
+      env: "PERCY_ENABLE=0"
+    - rvm: 2.3.0
+      env: "PERCY_ENABLE=0"
+    - rvm: 2.4.0
+      env: "PERCY_ENABLE=1"
+    - rvm: jruby-9.1.6.0
+      env: "PERCY_ENABLE=0"

data/4.0-Upgrade.md

@@ -17,7 +17,7 @@ gem 'redis-namespace'
 * **Redis 2.8.0 or greater is required.**  Redis 2.8 was released two years
   ago and contains **many** useful features which Sidekiq couldn't
   leverage until now.  **Redis 3.0.3 or greater is recommended** for large
-  scale use.
+  scale use [#2431](https://github.com/mperham/sidekiq/issues/2431).
 
 * Jobs are now fetched from Redis in parallel, making Sidekiq more
   resilient to high network latency.  This means that Sidekiq requires
@@ -25,6 +25,9 @@ gem 'redis-namespace'
   `concurrency + 2` connections in your pool or Sidekiq will exit.
   When in doubt, let Sidekiq size the connection pool for you.
 
+* Worker data is no longer updated in real-time but rather upon every
+  heartbeat.  Don't expect the `Sidekiq::Workers` API to be millisecond-precise.
+
 * There's a new testing API based off the `Sidekiq::Queues` namespace. All
   assertions made against the Worker class still work as expected.
 ```ruby

data/5.0-Upgrade.md

@@ -0,0 +1,56 @@
+# Welcome to Sidekiq 5.0!
+
+Sidekiq 5.0 contains a reworked job dispatch and execution core to integrate
+better with the new Rails 5.0 Executor.  It also drops support for older
+versions of Ruby and Rails and adds support for RTL languages in the Web UI.
+
+## What's New
+
+* Integrate job logging and retry logic directly in with the job
+  execution logic in Sidekiq::Processor.  Previously this logic was
+  defined as middleware.  In Rails 5.0, ActiveSupport::Executor handles ActiveRecord
+  connection management, job callbacks, development mode class loading,
+  etc.  Because of its extensive responsibilities, the Executor can't be
+  integrated as Sidekiq middleware; the logging/retry logic had to be pulled out
+  too.  Sidekiq 4.2 had a hack to make it work but this redesign provides
+  a cleaner integration. [#3235]
+* The Delayed Extensions `delay`, `delay_in` and `delay_until` APIs are
+  no longer available by default.  The extensions allow you to marshal
+  job arguments as YAML, leading to cases where job payloads could be many
+  100s of KB or larger if not careful, leading to Redis networking
+  timeouts or other problems.  As noted in the Best Practices wiki page,
+  Sidekiq is designed for jobs with small, simple arguments.
+
+  Add this line to your initializer to re-enable them and get the old behavior:
+  ```ruby
+  Sidekiq::Extensions.enable_delay!
+  ```
+  The old `Sidekiq.remove_delay!` API has been removed as it is now the default. [#3299]
+* Sidekiq's quiet signal is now `TSTP` (think of it as **T**hread
+  **ST**o**P**) instead of USR1 as USR1 is not available on JRuby.
+  USR1 will continue to be supported in Sidekiq 5.x for backwards
+  compatibility and will be removed in Sidekiq 6.x. [#3302]
+* The Web UI is now bi-directional - it can render either LTR
+  (left-to-right) or RTL languages. With this change, **Farsi, Arabic,
+  Hebrew and Urdu** are officially supported. [#3381]
+* Jobs which can't be parsed due to invalid JSON are now pushed
+  immediately to the Dead set since they require manual intervention and
+  will never execute successfully as is.  The Web UI has been updated to
+  more gracefully display these jobs. [#3296]
+* **Rails 3.2** is no longer supported.
+* **Ruby 2.0 and Ruby 2.1** are no longer supported. Ruby 2.2.2+ is required.
+
+## Upgrade
+
+As always, please upgrade Sidekiq **one major version at a time**.
+If you are already running Sidekiq 4.x, then:
+
+* Upgrade to the latest Sidekiq 4.x.
+```ruby
+gem 'sidekiq', '< 5'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 5.x.
+```ruby
+gem 'sidekiq', '< 6'
+```

data/COMM-LICENSE

@@ -28,7 +28,7 @@ In order to use the Software under this Agreement, you must receive a “Source
 
 3.1 You shall not (and shall not allow any third party to): (a) decompile, disassemble, or otherwise reverse engineer the Software or attempt to reconstruct or discover any source code, underlying ideas, algorithms, file formats or programming interfaces of the Software by any means whatsoever (except and only to the extent that applicable law prohibits or restricts reverse engineering restrictions); (b) distribute, sell, sublicense, rent, lease or use the Software for time sharing, hosting, service provider or like purposes, except as expressly permitted under this Agreement; (c) redistribute the Software or Modifications other than by including the Software or a portion thereof within your own product, which must have substantially different functionality than the Software or Modifications and must not allow any third party to use the Software or Modifications, or any portions thereof, for software development or application development purposes; (d) redistribute the Software as part of a product, "appliance" or "virtual server"; (e) redistribute the Software on any server which is not directly under your control; (f) remove any product identification, proprietary, copyright or other notices contained in the Software; (g) modify any part of the Software, create a derivative work of any part of the Software (except as permitted in Section 4), or incorporate the Software, except to the extent expressly authorized in writing by Contributed Systems; (h) publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software; (i) utilize any equipment, device, software, or other means designed to circumvent or remove any form of Source URL or copy protection used by Contributed Systems in connection with the Software, or use the Software together with any authorization code, Source URL, serial number, or other copy protection device not supplied by Contributed Systems; (j) use the Software to develop a product which is competitive with any Contributed Systems product offerings; or (k) use unauthorized Source URLS or keycode(s) or distribute or publish Source URLs or keycode(s), except as may be expressly permitted by Contributed Systems in writing. If your unique Source URL is ever published, Contributed Systems reserves the right to terminate your access without notice.
 
-3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE FOR A PRODUCT THAT IS INTENDED FOR SOFTWARE OR APPLICATION DEVELOPMENT PURPOSES.
+3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE AS PART OF A PRODUCT OR SERVICE THAT PROVIDES SIMILAR FUNCTIONALITY TO THE SOFTWARE ITSELF.
 
 The Open Source version of the Software (“LGPL Version”) is licensed
 under the terms of the GNU Lesser General Public License versions 3.0

data/Changes.md

@@ -1,15 +1,224 @@
 # Sidekiq Changes
 
-4.0.0.pre1
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
+
+5.0.0
 -----------
 
-- Sidekiq's internals have been completely overhauled for performance
-  and to remove dependencies.  This has resulted in major speedups, as
-  [detailed on my blog](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
-- See the [4.0 upgrade notes](4.0-Upgrade.md) for more detail.
+- **BREAKING CHANGE** Job dispatch was refactored for safer integration with
+  Rails 5.  The **Logging** and **RetryJobs** server middleware were removed and
+  functionality integrated directly into Sidekiq::Processor.  These aren't
+  commonly used public APIs so this shouldn't impact most users.
+```
+Sidekiq::Middleware::Server::RetryJobs -> Sidekiq::JobRetry
+Sidekiq::Middleware::Server::Logging -> Sidekiq::JobLogging
+```
+- Quieting Sidekiq is now done via the TSTP signal, the USR1 signal is deprecated.
+- The `delay` extension APIs are no longer available by default, you
+  must opt into them.
+- The Web UI is now BiDi and can render RTL languages like Arabic, Farsi and Hebrew.
+- Rails 3.2 and Ruby 2.0 and 2.1 are no longer supported.
+- The `SomeWorker.set(options)` API was re-written to avoid thread-local state. [#2152]
+- Sidekiq Enterprise's encrypted jobs now display "[encrypted data]" in the Web UI instead
+  of random hex bytes.
+- Please see the [5.0 Upgrade notes](5.0-Upgrade.md) for more detail.
+
+4.2.10
+-----------
+
+- Scheduled jobs can now be moved directly to the Dead queue via API [#3390]
+- Fix edge case leading to job duplication when using Sidekiq Pro's
+  reliability feature [#3388]
+- Fix error class name display on retry page [#3348]
+- More robust latency calculation [#3340]
+
+4.2.9
+-----------
+
+- Rollback [#3303] which broke Heroku Redis users [#3311]
+- Add support for TSTP signal, for Sidekiq 5.0 forward compatibility. [#3302]
+
+4.2.8
+-----------
+
+- Fix rare edge case with Redis driver that can create duplicate jobs [#3303]
+- Fix Rails 5 loading issue [#3275]
+- Restore missing tooltips to timestamps in Web UI [#3310]
+- Work on **Sidekiq 5.0** is now active! [#3301]
+
+4.2.7
+-----------
+
+- Add new integration testing to verify code loading and job execution
+  in development and production modes with Rails 4 and 5 [#3241]
+- Fix delayed extensions in development mode [#3227, DarthSim]
+- Use Worker's `retry` default if job payload does not have a retry
+  attribute [#3234, mlarraz]
+
+4.2.6
+-----------
+
+- Run Rails Executor when in production [#3221, eugeneius]
+
+4.2.5
+-----------
+
+- Re-enable eager loading of all code when running non-development Rails 5. [#3203]
+- Better root URL handling for zany web servers [#3207]
+
+4.2.4
+-----------
+
+- Log errors coming from the Rails 5 reloader. [#3212, eugeneius]
+- Clone job data so middleware changes don't appear in Busy tab
+
+4.2.3
+-----------
+
+- Disable use of Rails 5's Reloader API in non-development modes, it has proven
+  to be unstable under load [#3154]
+- Allow disabling of Sidekiq::Web's cookie session to handle the
+  case where the app provides a session already [#3180, inkstak]
+```ruby
+Sidekiq::Web.set :sessions, false
+```
+- Fix Web UI sharding support broken in 4.2.2. [#3169]
+- Fix timestamps not updating during UI polling [#3193, shaneog]
+- Relax rack-protection version to >= 1.5.0
+- Provide consistent interface to exception handlers, changing the structure of the context hash. [#3161]
+
+4.2.2
+-----------
+
+- Fix ever-increasing cookie size with nginx [#3146, cconstantine]
+- Fix so Web UI works without trailing slash [#3158, timdorr]
+
+4.2.1
+-----------
+
+- Ensure browser does not cache JSON/AJAX responses. [#3136]
+- Support old Sinatra syntax for setting config [#3139]
+
+4.2.0
+-----------
+
+- Enable development-mode code reloading.  **With Rails 5.0+, you don't need
+  to restart Sidekiq to pick up your Sidekiq::Worker changes anymore!** [#2457]
+- **Remove Sinatra dependency**.  Sidekiq's Web UI now uses Rack directly.
+  Thank you to Sidekiq's newest committer, **badosu**, for writing the code
+  and doing a lot of testing to ensure compatibility with many different
+  3rd party plugins.  If your Web UI works with 4.1.4 but fails with
+  4.2.0, please open an issue. [#3075]
+- Allow tuning of concurrency with the `RAILS_MAX_THREADS` env var. [#2985]
+  This is the same var used by Puma so you can tune all of your systems
+  the same way:
+```sh
+web: RAILS_MAX_THREADS=5 bundle exec puma ...
+worker: RAILS_MAX_THREADS=10 bundle exec sidekiq ...
+```
+Using `-c` or `config/sidekiq.yml` overrides this setting.  I recommend
+adjusting your `config/database.yml` to use it too so connections are
+auto-scaled:
+```yaml
+  pool: <%= ENV['RAILS_MAX_THREADS'] || 5 %>
+```
+
+4.1.4
+-----------
+
+- Unlock Sinatra so a Rails 5.0 compatible version may be used [#3048]
+- Fix race condition on startup with JRuby [#3043]
+
+
+4.1.3
+-----------
+
+- Please note the Redis 3.3.0 gem has a [memory leak](https://github.com/redis/redis-rb/issues/612),
+  Redis 3.2.2 is recommended until that issue is fixed.
+- Sinatra 1.4.x is now a required dependency, avoiding cryptic errors
+  and old bugs due to people not upgrading Sinatra for years. [#3042]
+- Fixed race condition in heartbeat which could rarely lead to lingering
+  processes on the Busy tab. [#2982]
+```ruby
+# To clean up lingering processes, modify this as necessary to connect to your Redis.
+# After 60 seconds, lingering processes should disappear from the Busy page.
+
+require 'redis'
+r = Redis.new(url: "redis://localhost:6379/0")
+# uncomment if you need a namespace
+#require 'redis-namespace'
+#r = Redis::Namespace.new("foo", r)
+r.smembers("processes").each do |pro|
+  r.expire(pro, 60)
+  r.expire("#{pro}:workers", 60)
+end
+```
+
+
+4.1.2
+-----------
+
+- Fix Redis data leak with worker data when a busy Sidekiq process
+  crashes.  You can find and expire leaked data in Redis with this
+script:
+```bash
+$ redis-cli keys  "*:workers" | while read LINE ; do TTL=`redis-cli expire "$LINE" 60`; echo "$LINE"; done;
+```
+  Please note that `keys` can be dangerous to run on a large, busy Redis.  Caveat runner.
+- Freeze all string literals with Ruby 2.3. [#2741]
+- Client middleware can now stop bulk job push. [#2887]
+
+4.1.1
+-----------
+
+- Much better behavior when Redis disappears and comes back. [#2866]
+- Update FR locale [dbachet]
+- Don't fill logfile in case of Redis downtime [#2860]
+- Allow definition of a global retries_exhausted handler. [#2807]
+```ruby
+Sidekiq.configure_server do |config|
+  config.default_retries_exhausted = -> (job, ex) do
+    Sidekiq.logger.info "#{job['class']} job is now dead"
+  end
+end
+```
+
+4.1.0
+-----------
+
+- Tag quiet processes in the Web UI [#2757, jcarlson]
+- Pass last exception to sidekiq\_retries\_exhausted block [#2787, Nowaker]
+```ruby
+class MyWorker
+  include Sidekiq::Worker
+  sidekiq_retries_exhausted do |job, exception|
+  end
+end
+```
+- Add native support for ActiveJob's `set(options)` method allowing
+you to override worker options dynamically.  This should make it
+even easier to switch between ActiveJob and Sidekiq's native APIs [#2780]
+```ruby
+class MyWorker
+  include Sidekiq::Worker
+  sidekiq_options queue: 'default', retry: true
+
+  def perform(*args)
+    # do something
+  end
+end
+
+MyWorker.set(queue: 'high', retry: false).perform_async(1)
+```
+
+4.0.2
+-----------
+
+- Better Japanese translations
+- Remove `json` gem dependency from gemspec. [#2743]
 - There's a new testing API based off the `Sidekiq::Queues` namespace. All
   assertions made against the Worker class still work as expected.
-  [#2659, brandonhilkert]
+  [#2676, brandonhilkert]
 ```ruby
 assert_equal 0, Sidekiq::Queues["default"].size
 HardWorker.perform_async("log")
@@ -18,6 +227,26 @@ assert_equal "log", Sidekiq::Queues["default"].first['args'][0]
 Sidekiq::Queues.clear_all
 ```
 
+4.0.1
+-----------
+
+- Yank new queue-based testing API [#2663]
+- Fix invalid constant reference in heartbeat
+
+4.0.0
+-----------
+
+- Sidekiq's internals have been completely overhauled for performance
+  and to remove dependencies.  This has resulted in major speedups, as
+  [detailed on my blog](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
+- See the [4.0 upgrade notes](4.0-Upgrade.md) for more detail.
+
+3.5.4
+-----------
+
+- Ensure exception message is a string [#2707]
+- Revert racy Process.kill usage in sidekiqctl
+
 3.5.3
 -----------
 
@@ -45,7 +274,7 @@ Sidekiq::Queues.clear_all
 -----------
 
 - **FIX MEMORY LEAK** Under rare conditions, threads may leak [#2598, gazay]
-- Add Ukranian locale [#2561, elrakita]
+- Add Ukrainian locale [#2561, elrakita]
 - Disconnect and retry Redis operations if we see a READONLY error [#2550]
 - Add server middleware testing harness; see [wiki](https://github.com/mperham/sidekiq/wiki/Testing#testing-server-middleware) [#2534, ryansch]
 

data/Ent-Changes.md

@@ -1,9 +1,117 @@
-Sidekiq Enterprise Changelog
-=======================
+# Sidekiq Enterprise Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
-1.0.0.pre1
+1.5.2
+-------------
+
+- Fix encrypted arguments double-encrypted by retry or rate limiting [#3368]
+- Fix leak in concurrent rate limiter, run this in Rails console to clean up existing data [#3323]
+```ruby
+expiry = 1.month.to_i; Sidekiq::Limiter.redis { |c| c.scan_each(match: "lmtr-cfree-*") { |key| c.expire(key, expiry) } }
+```
+
+1.5.1
+-------------
+
+- Fix issue with census startup when not using Bundler configuration for
+  source credentials.
+
+1.5.0
+-------------
+
+- Add new web authorization API [#3251]
+- Update all sidekiqswarm env vars to use SIDEKIQ\_ prefix [#3218]
+- Add census reporting, the leader will ping contribsys nightly with aggregate usage metrics
+
+1.4.0
+-------------
+
+- No functional changes, require latest Sidekiq and Sidekiq Pro versions
+
+1.3.2
+-------------
+
+- Upgrade encryption to use OpenSSL's more secure GCM mode. [#3060]
+
+1.3.1
+-------------
+
+- Fix multi-process memory monitoring on CentOS 6.x [#3063]
+- Polish the new encryption feature a bit.
+
+1.3.0
+-------------
+
+- **BETA** [New encryption feature](https://github.com/mperham/sidekiq/wiki/Ent-Encryption)
+  which automatically encrypts the last argument of a Worker, aka the secret bag.
+
+1.2.4
+-------------
+
+- Fix issue causing some minutely jobs to execute every other minute.
+- Log a warning if slow periodic processing causes us to miss a clock tick.
+
+1.2.3
+-------------
+
+- Periodic jobs could stop executing until process restart if Redis goes down [#3047]
+
+1.2.2
+-------------
+
+- Add API to check if a unique lock is present. See [#2932] for details.
+- Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
+- Add option for tuning which Bundler groups get preloaded with `sidekiqswarm` [#3025]
+```
+SIDEKIQ_PRELOAD=default,production bin/sidekiqswarm ...
+# Use an empty value for maximum application compatibility
+SIDEKIQ_PRELOAD= bin/sidekiqswarm ...
+```
+
+1.2.1
+-------------
+
+- Multi-Process mode can now monitor the RSS memory of children and
+  restart any that grow too large.  To limit children to 1GB each:
+```
+MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
+```
+
+1.2.0
+-------------
+
+- **NEW FEATURE** Multi-process mode!  Sidekiq Enterprise can now fork multiple worker
+  processes, enabling significant memory savings.  See the [wiki
+documentation](https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process) for details.
+
+
+0.7.10
+-------------
+
+- More precise gemspec dependency versioning
+
+1.1.0
+-------------
+
+- **NEW FEATURE** Historical queue metrics, [documented in the wiki](https://github.com/mperham/sidekiq/wiki/Ent-Historical-Metrics) [#2719]
+
+0.7.9, 1.0.2
+-------------
+
+- Window limiters can now accept arbitrary window sizes [#2686]
+- Fix race condition in window limiters leading to non-stop OverLimit [#2704]
+- Fix invalid overage counts when nesting concurrent limiters
+
+1.0.1
+----------
+
+- Fix crash in periodic subsystem when a follower shuts down, thanks
+  to @justinko for reporting.
+
+1.0.0
 ----------
 
 - Enterprise 1.x targets Sidekiq 4.x.