sidekiq 3.5.4 → 5.2.7

data/Ent-Changes.md

@@ -1,8 +1,207 @@
-Sidekiq Enterprise Changelog
-=======================
+# Sidekiq Enterprise Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+1.8.1
+-------------
+
+- Fix excessive lock reclaims with concurrent limiter [#4105]
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+1.8.0
+-------------
+
+- Require Sidekiq Pro 4.0 and Sidekiq 5.2.
+- Refactor historical metrics API to use revamped Statsd support in Sidekiq Pro
+- Add a gauge to historical metrics for `default` queue latency [#4079]
+
+1.7.2
+-------------
+
+- Add PT and JA translations
+- Fix elapsed time calculations to use monotonic clock [#4000, sj26]
+- Fix edge case where flapping leadership would cause old periodic
+  jobs to be fired once [#3974]
+- Add support for sidekiqswarm memory monitoring on FreeBSD [#3884]
+
+1.7.1
+-------------
+
+- Fix Lua error in concurrent rate limiter under heavy contention
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+1.7.0
+-------------
+
+- **NEW FEATURE** [Rolling restarts](https://github.com/mperham/sidekiq/wiki/Ent-Rolling-Restarts) - great for long running jobs!
+- Adjust middleware so unique jobs that don't push aren't registered in a Batch [#3662]
+- Add new unlimited rate limiter, useful for testing [#3743]
+```ruby
+limiter = Sidekiq::Limiter.unlimited(...any args...)
+```
+
+1.6.1
+-------------
+
+- Fix crash in rate limiter middleware when used with custom exceptions [#3604]
+
+1.6.0
+-------------
+
+- Show process "leader" tag on Busy page, requires Sidekiq 5.0.2 [#2867]
+- Capture custom metrics with the `save_history` API. [#2815]
+- Implement new `unique_until: 'start'` policy option. [#3471]
+
+1.5.4
+-------------
+
+- Fix broken Cron page in Web UI [#3458]
+
+1.5.3
+-------------
+
+- Remove dependency on the algorithms gem [#3446]
+- Allow user to specify max memory in megabytes with SIDEKIQ\_MAXMEM\_MB [#3451]
+- Implement logic to detect app startup failure, sidekiqswarm will exit
+  rather than try to restart the app forever [#3450]
+- Another fix for doubly-encrypted arguments [#3368]
+
+1.5.2
+-------------
+
+- Fix encrypted arguments double-encrypted by retry or rate limiting [#3368]
+- Fix leak in concurrent rate limiter, run this in Rails console to clean up existing data [#3323]
+```ruby
+expiry = 1.month.to_i; Sidekiq::Limiter.redis { |c| c.scan_each(match: "lmtr-cfree-*") { |key| c.expire(key, expiry) } }
+```
+
+1.5.1
+-------------
+
+- Fix issue with census startup when not using Bundler configuration for
+  source credentials.
+
+1.5.0
+-------------
+
+- Add new web authorization API [#3251]
+- Update all sidekiqswarm env vars to use SIDEKIQ\_ prefix [#3218]
+- Add census reporting, the leader will ping contribsys nightly with aggregate usage metrics
+
+1.4.0
+-------------
+
+- No functional changes, require latest Sidekiq and Sidekiq Pro versions
+
+1.3.2
+-------------
+
+- Upgrade encryption to use OpenSSL's more secure GCM mode. [#3060]
+
+1.3.1
+-------------
+
+- Fix multi-process memory monitoring on CentOS 6.x [#3063]
+- Polish the new encryption feature a bit.
+
+1.3.0
+-------------
+
+- **BETA** [New encryption feature](https://github.com/mperham/sidekiq/wiki/Ent-Encryption)
+  which automatically encrypts the last argument of a Worker, aka the secret bag.
+
+1.2.4
+-------------
+
+- Fix issue causing some minutely jobs to execute every other minute.
+- Log a warning if slow periodic processing causes us to miss a clock tick.
+
+1.2.3
+-------------
+
+- Periodic jobs could stop executing until process restart if Redis goes down [#3047]
+
+1.2.2
+-------------
+
+- Add API to check if a unique lock is present. See [#2932] for details.
+- Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
+- Add option for tuning which Bundler groups get preloaded with `sidekiqswarm` [#3025]
+```
+SIDEKIQ_PRELOAD=default,production bin/sidekiqswarm ...
+# Use an empty value for maximum application compatibility
+SIDEKIQ_PRELOAD= bin/sidekiqswarm ...
+```
+
+1.2.1
+-------------
+
+- Multi-Process mode can now monitor the RSS memory of children and
+  restart any that grow too large.  To limit children to 1GB each:
+```
+MAXMEM_KB=1048576 COUNT=2 bundle exec sidekiqswarm ...
+```
+
+1.2.0
+-------------
+
+- **NEW FEATURE** Multi-process mode!  Sidekiq Enterprise can now fork multiple worker
+  processes, enabling significant memory savings.  See the [wiki
+documentation](https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process) for details.
+
+
+0.7.10
+-------------
+
+- More precise gemspec dependency versioning
+
+1.1.0
+-------------
+
+- **NEW FEATURE** Historical queue metrics, [documented in the wiki](https://github.com/mperham/sidekiq/wiki/Ent-Historical-Metrics) [#2719]
+
+0.7.9, 1.0.2
+-------------
+
+- Window limiters can now accept arbitrary window sizes [#2686]
+- Fix race condition in window limiters leading to non-stop OverLimit [#2704]
+- Fix invalid overage counts when nesting concurrent limiters
+
+1.0.1
+----------
+
+- Fix crash in periodic subsystem when a follower shuts down, thanks
+  to @justinko for reporting.
+
+1.0.0
+----------
+
+- Enterprise 1.x targets Sidekiq 4.x.
+- Rewrite several features to remove Celluloid dependency.  No
+  functional changes.
+
+0.7.8
+----------
+
+- Fix `unique_for: false` [#2658]
+
+
+0.7.7
+----------
+
+- Enterprise 0.x targets Sidekiq 3.x.
+- Fix racy shutdown event which could lead to disappearing periodic
+  jobs, requires Sidekiq >= 3.5.3.
+- Add new :leader event which is fired when a process gains leadership.
+
+0.7.6
+----------
+
+- Redesign how overrated jobs are rescheduled to avoid creating new
+  jobs. [#2619]
+
 0.7.5
 ----------
 

data/Gemfile

@@ -1,27 +1,23 @@
 source 'https://rubygems.org'
-gemspec
 
-gem 'rails', '~> 4.2'
-gem 'simplecov'
-gem 'minitest'
-gem 'toxiproxy'
+gemspec
 
-platforms :rbx do
-  gem 'rubysl', '~> 2.0'         # if using anything in the ruby standard library
-  gem 'psych'                    # if using yaml
-  gem 'minitest'                 # if using minitest
-  gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
-end
+gem 'rake'
+gem 'redis-namespace'
+gem 'rails', '~> 5.2'
+gem 'sqlite3', '~> 1.3.6', platforms: :ruby
+gem 'activerecord-jdbcsqlite3-adapter', platforms: :jruby
 
-platforms :ruby do
-  gem 'sqlite3'
+group :test do
+  gem 'minitest'
+  gem 'simplecov'
 end
 
-platforms :mri do
-  gem 'pry-byebug'
+group :development, :test do
+  gem 'pry-byebug', platforms: :mri
 end
 
-platforms :jruby do
-  gem 'jruby-openssl'
-  gem 'activerecord-jdbcsqlite3-adapter'
+group :load_test do
+  gem 'hiredis'
+  gem 'toxiproxy'
 end

data/LICENSE

@@ -5,5 +5,5 @@ the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
 for license text.
 
 Sidekiq Pro has a commercial-friendly license allowing private forks
-and modifications of Sidekiq.  Please see http://sidekiq.org/pro/ for
+and modifications of Sidekiq.  Please see https://sidekiq.org/products/pro.html for
 more detail.  You can find the commercial license terms in COMM-LICENSE.

data/Pro-3.0-Upgrade.md

@@ -0,0 +1,44 @@
+# Welcome to Sidekiq Pro 3.0!
+
+Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
+
+## What's New
+
+* **Redis 2.8.0 or greater is required.**  Redis 2.8 was released two years
+  ago and contains **many** useful features which Sidekiq couldn't
+  leverage until now.  **Redis 3.0.3 or greater is recommended** for large
+  scale use.
+
+* Sidekiq Pro no longer uses Celluloid.  If your application code uses Celluloid,
+  you will need to pull it in yourself.
+
+* Pausing and unpausing queues is now instantaneous, no more polling!
+
+* Reliable fetch has been re-implemented due to the fetch changes in
+  Sidekiq 4.0.
+
+* Support for platforms without persistent hostnames.  Since the reliable\_fetch
+  algorithm requires a persistent hostname, an alternative reliability
+algorithm is now available for platforms like Heroku and Docker:
+```ruby
+Sidekiq.configure_server do |config|
+  config.timed_fetch!
+end
+```
+  The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
+
+* The old 'sidekiq/notifications' features have been removed.
+
+## Upgrade
+
+First, make sure you are using Redis 2.8 or greater. Next:
+
+* Upgrade to the latest Sidekiq Pro 2.x.
+```ruby
+gem 'sidekiq-pro', '< 3'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 3.x.
+```ruby
+gem 'sidekiq-pro', '< 4'
+```

data/Pro-4.0-Upgrade.md

@@ -0,0 +1,35 @@
+# Welcome to Sidekiq Pro 4.0!
+
+Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
+
+## What's New
+
+* Batches now "die" if any of their jobs die.  You can enumerate the set
+  of dead batches and their associated dead jobs.  The success callback
+  for a dead batch will never fire unless these jobs are fixed.
+```ruby
+Sidekiq::Batch::DeadSet.new.each do |status|
+  status.dead? # => true
+  status.dead_jobs # => [...]
+end
+```
+This API allows you to enumerate the batches which need help.
+If you fix the issue and the dead jobs succeed, the batch will succeed.
+* The older `reliable_fetch` and `timed_fetch` algorithms have been
+  removed.  Only super\_fetch is available in 4.0.
+* The statsd middleware has been tweaked to remove support for legacy,
+  pre-3.6.0 configuration and add relevant tags.
+* Requires Sidekiq 5.0.5+.
+
+## Upgrade
+
+* Upgrade to the latest Sidekiq Pro 3.x.
+```ruby
+gem 'sidekiq-pro', '< 4'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 4.x.
+```ruby
+gem 'sidekiq-pro', '< 5'
+```
+

data/Pro-Changes.md

@@ -1,8 +1,313 @@
-Sidekiq Pro Changelog
-=======================
+# Sidekiq Pro Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+HEAD
+---------
+
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+4.0.5
+---------
+
+- Increase super\_fetch retriever thread count from 1 to 2 to make it
+  less sensitive to Redis latency.
+- Better handling of invalid job JSON by reliable scheduler [#4053]
+- Added ZH, PT, JA and RU translations.
+
+4.0.4
+---------
+
+- Update Sidekiq::Client patches to work with new Module#prepend
+  mechanism in Sidekiq 5.2.0. [#3930]
+
+4.0.3
+---------
+
+- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
+- Implement batch death callback.  This is fired the first time a job within a batch dies. [#3841]
+```ruby
+batch = Sidekiq::Batch.new
+batch.on(:death, ...)
+```
+
+4.0.2
+---------
+
+- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
+  call and resulting latency [#3790]
+- Fix possible bad statsd metric call on super\_fetch startup
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+4.0.1
+---------
+
+- Fix incompatibility with the statsd-ruby gem [#3740]
+- Add tags to Statsd metrics when using Datadog [#3744]
+
+4.0.0
+---------
+
+- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
+
+
+3.7.1
+---------
+
+- Deprecate timed\_fetch.  Switch to super\_fetch:
+```ruby
+config.super_fetch!
+```
+
+
+3.7.0
+---------
+
+- Refactor batch job success/failure to gracefully handle several edge
+  cases with regard to Sidekiq::Shutdown.  This should greatly reduce
+  the chances of seeing the long-standing "negative pending count" problem. [#3710]
+
+
+3.6.1
+---------
+
+- Add support for Datadog::Statsd, it is the recommended Statsd client.  [#3699]
+```ruby
+Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
+```
+- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
+
+
+3.6.0
+---------
+
+This release overhauls the Statsd metrics support and adds more
+metrics for tracking Pro feature usage.  In your initializer:
+```ruby
+Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
+```
+Sidekiq Pro will emit more metrics to Statsd:
+```
+jobs.expired - when a job is expired
+jobs.recovered.push - when a job is recovered by reliable_push after network outage
+jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
+batch.created - when a batch is created
+batch.complete - when a batch is completed
+batch.success - when a batch is successful
+```
+Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
+Everything should be backwards compatible with one deprecation notice.
+
+
+3.5.4
+---------
+
+- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
+- Fix case where TimedFetch might not recover some pending jobs
+- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
+- Adjust scan API to check 100 elements at a time, to minimize network round trips
+  when scanning large sets.
+
+3.5.3
+---------
+
+- Restore error check for super\_fetch's job ack [#3601]
+- Trim error messages saved in Batch's failure hash, preventing huge
+  messages from bloating Redis. [#3570]
+
+3.5.2
+---------
+
+- Fix `Status#completed?` when run against a Batch that had succeeded
+  and was deleted. [#3519]
+
+3.5.1
+---------
+
+- Work with Sidekiq 5.0.2+
+- Improve performance of super\_fetch with weighted queues [#3489]
+
+3.5.0
+---------
+
+- Add queue pause/unpause endpoints for scripting via curl [#3445]
+- Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
+- Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
+- Batch page displays Pending JIDs if less than 10 [#3130]
+- Batch page has a Search button to find associated Retries [#3130]
+- Make Batch UI progress bar more friendly to the colorblind [#3387]
+
+3.4.5
+---------
+
+- Fix potential job loss with reliable scheduler when lots of jobs are scheduled
+  at precisely the same time. Thanks to raivil for his hard work in
+  reproducing the bug. [#3371]
+
+3.4.4
+---------
+
+- Optimize super\_fetch shutdown to restart jobs quicker [#3249]
+
+3.4.3
+---------
+
+- Limit reliable scheduler to enqueue up to 100 jobs per call, minimizing Redis latency [#3332]
+- Fix bug in super\_fetch logic for queues with `_` in the name [#3339]
+
+3.4.2
+---------
+
+- Add `Batch::Status#invalidated?` API which returns true if any/all
+  JIDs were invalidated within the batch. [#3326]
+
+3.4.1
+---------
+
+- Allow super\_fetch's orphan job check to happen as often as every hour [#3273]
+- Officially deprecate reliable\_fetch algorithm, I now recommend you use `super_fetch` instead:
+```ruby
+Sidekiq.configure_server do |config|
+  config.super_fetch!
+end
+```
+Also note that Sidekiq's `-i/--index` option is no longer used/relevant with super\_fetch.
+- Don't display "Delete/Retry All" buttons when filtering in Web UI [#3243]
+- Reimplement Sidekiq::JobSet#find\_job with ZSCAN [#3197]
+
+3.4.0
+---------
+
+- Introducing the newest reliable fetching algorithm: `super_fetch`!  This
+  algorithm will replace reliable\_fetch in Pro 4.0.  super\_fetch is
+  bullet-proof across all environments, no longer requiring stable
+  hostnames or an index to be set per-process. [#3077]
+```ruby
+Sidekiq.configure_server do |config|
+  config.super_fetch!
+end
+```
+  Thank you to @jonhyman for code review and the Sidekiq Pro customers that
+  beta tested super\_fetch.
+
+3.3.3
+---------
+
+- Update Web UI extension to work with Sidekiq 4.2.0's new Web UI. [#3075]
+
+3.3.2
+---------
+
+- Minimize batch memory usage after success [#3083]
+- Extract batch's 24 hr linger expiry to a LINGER constant so it can be tuned. [#3011]
+
+3.3.1
+---------
+
+- If environment is unset, treat it as development so reliable\_fetch works as before 3.2.2.
+
+3.3.0
+---------
+
+- Don't delete batches immediately upon success but set a 24 hr expiry, this allows
+  Sidekiq::Batch::Status#poll to work, even after batch success. [#3011]
+- New `Sidekiq::PendingSet#destroy(jid)` API to remove poison pill jobs [#3015]
+
+3.2.2
+---------
+
+- A default value for -i is only set in development now, staging or
+  other environments must set an index if you wish to use reliable\_fetch. [#2971]
+- Fix nil dereference when checking for jobs over timeout in timed\_fetch
+
+
+3.2.1
+---------
+
+- timed\_fetch now works with namespaces.  [ryansch]
+
+
+3.2.0
+---------
+
+- Fixed detection of missing batches, `NoSuchBatch` should be raised
+  properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
+  longer in Redis.
+- Remove support for Pro 1.x format batches.  This version will no
+  longer seamlessly process batches created with Sidekiq Pro 1.x.
+  As always, upgrade one major version at a time to ensure a smooth
+  transition.
+- Fix edge case where a parent batch could expire before a child batch
+  was finished processing, leading to missing batches [#2889]
+
+2.1.5
+---------
+
+- Fix edge case where a parent batch could expire before a child batch
+  was finished processing, leading to missing batches [#2889]
+
+3.1.0
+---------
+
+- New container-friendly fetch algorithm: `timed_fetch`.  See the
+  [wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
+  for trade offs between the two reliability options.  You should
+  use this if you are on Heroku, Docker, Amazon ECS or EBS or
+  another container-based system.
+
+
+3.0.6
+---------
+
+- Fix race condition on reliable fetch shutdown
+
+3.0.5
+---------
+
+- Statsd metrics now account for ActiveJob class names
+- Allow reliable fetch internals to be overridden [jonhyman]
+
+3.0.4
+---------
+
+- Queue pausing no longer requires reliable fetch. [#2786]
+
+3.0.3, 2.1.4
+------------
+
+- Convert Lua-based `Sidekiq::Queue#delete_by_class` to Ruby-based, to
+  avoid O(N^2) performance and possible Redis failure. [#2806]
+
+3.0.2
+-----------
+
+- Make job registration with batch part of the atomic push so batch
+  metadata can't get out of sync with the job data. [#2714]
+
+3.0.1
+-----------
+
+- Remove a number of Redis version checks since we can assume 2.8+ now.
+- Fix expiring jobs client middleware not loaded on server
+
+3.0.0
+-----------
+
+- See the [Pro 3.0 release notes](Pro-3.0-Upgrade.md).
+
+2.1.3
+-----------
+
+- Don't enable strict priority if using weighted queueing like `-q a,1 -q b,1`
+- Safer JSON mangling in Lua [#2639]
+
+2.1.2
+-----------
+
+- Lock Sidekiq Pro 2.x to Sidekiq 3.x.
+
 2.1.1
 -----------