resque-pool 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 88e5c211d7bdebea532f4d75cc7cb8772315ec83
-  data.tar.gz: d137b2d808b9fae0bedf4b88436a844fe44e4bb5
+  metadata.gz: e5a0aca3c74178395af21e6f7ddde9add5784556
+  data.tar.gz: 9e7b1cc2747b3b82760b2aa637411fd841712ca8
 SHA512:
-  metadata.gz: 6c818659ddf5e996358875d48ed2bb7b7aa9e4a987fd4f8752e2277694aa422319bceb89d01d1f8952148d1e2239ddd21fff0510eeea5bcc4b661610b8cd2e57
-  data.tar.gz: 0c401f1af699ccb20833336bbcc12d05e22c9a629cdee1ac47573710b7e81d7bf186d3301c7d499cde21fa77502f00a9999f79935487fb89f42e893de50654b8
+  metadata.gz: d1f568bbd72acb8b1e7d1312222847b5a67d1b77fe95a56d0a8ae935064d20ba4e29a423dc8c6e5b724ae722745a3c57dc4e7ec81efd6d372ae810f028a9b8bf
+  data.tar.gz: f4b7118fc2cad3e95de6335bb5131525fc9a3af9ece1e817a423011437fc0558f17ee98b29ad0adf2fff6e516908ee7a3dae05d2d36bd198f4b064307e0f119a

data/.gitignore

@@ -0,0 +1,9 @@
+pkg/*
+*.gem
+.bundle
+*.log
+*.pid
+examples/tmp/*
+tmp/*
+vendor
+tags

data/.travis.yml

@@ -0,0 +1,13 @@
+language: ruby
+cache: bundler
+sudo: false
+services: redis
+rvm:
+  - 2.3.8
+  - 2.4.5
+  - 2.5.3
+  - 2.6.1
+
+before_install:
+  - gem update --system
+  - gem install bundler

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at nick@ekenosen.net. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/CONTRIBUTING.md

@@ -0,0 +1,23 @@
+Contributing
+============
+
+We gladly accept help and bugfixes for resque-pool. We follow the guidelines of
+[semantic versioning](http://semver.org/) for all new gem releases.
+
+Pull request guidelines
+-----------------------
+
+1. Fork it and clone your new repo
+2. Create a branch (`git checkout -b my_awesome_feature`)
+3. Commit your changes (`git add my/awesome/file.rb; git commit -m "Added my awesome feature"`)
+4. Push your changes to your fork (`git push origin my_awesome_feature`)
+5. Open a [Pull Request](/nevans/resque-pool/pulls)
+
+Pull requests should have automated test coverage.
+
+New release policy
+------------------
+
+New versions won't be released until a release candidate has been running in a
+maintainer's production environment for at least a week.  Exceptions only for
+security-related patches.

data/Changelog.md

@@ -1,3 +1,12 @@
+## 0.7.0 _unreleased_
+[full changelog](https://github.com/nevans/resque-pool/compare/v0.6.0...master).
+
+One big new feature: `--hot-swap` for [zero-downtime code
+deploys](https://github.com/nevans/resque-pool#zero-downtime-code-deploys).
+Thanks to [joshuaflanagan](https://github.com/joshuaflanagan),
+[brasic](https://github.com/brasic), and
+[ShippingEasy](https://github.com/ShippingEasy)!
+
 ## 0.6.0
 [full changelog](https://github.com/nevans/resque-pool/compare/v0.5.0...v0.6.0).
 
@@ -8,7 +17,7 @@ thanks to [joshuaflanagan](https://github.com/joshuaflanagan)!
 Lots of cleanup in this release.  Thanks to the contributers:
 
  * [joshuaflanagan](https://github.com/joshuaflanagan) Custom config loader
- * [grosser](https://github/grosser)
+ * [grosser](https://github.com/grosser)
    * ship less files in the gem
    * remove trollop dependency
    * remove -n -t -r -n -i commandline options since they were added unintentionally

data/Gemfile

@@ -0,0 +1,3 @@
+source "https://rubygems.org"
+
+gemspec

data/Gemfile.lock

@@ -0,0 +1,102 @@
+PATH
+  remote: .
+  specs:
+    resque-pool (0.7.0)
+      rake (>= 10.0, < 13.0)
+      resque (>= 1.22, < 3)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    aruba (0.14.8)
+      childprocess (>= 0.6.3, < 1.1.0)
+      contracts (~> 0.9)
+      cucumber (>= 1.3.19)
+      ffi (~> 1.9)
+      rspec-expectations (>= 2.99)
+      thor (~> 0.19)
+    backports (3.12.0)
+    builder (3.2.3)
+    childprocess (1.0.1)
+      rake (< 13.0)
+    contracts (0.16.0)
+    cucumber (3.1.2)
+      builder (>= 2.1.2)
+      cucumber-core (~> 3.2.0)
+      cucumber-expressions (~> 6.0.1)
+      cucumber-wire (~> 0.0.1)
+      diff-lcs (~> 1.3)
+      gherkin (~> 5.1.0)
+      multi_json (>= 1.7.5, < 2.0)
+      multi_test (>= 0.1.2)
+    cucumber-core (3.2.1)
+      backports (>= 3.8.0)
+      cucumber-tag_expressions (~> 1.1.0)
+      gherkin (~> 5.0)
+    cucumber-expressions (6.0.1)
+    cucumber-tag_expressions (1.1.1)
+    cucumber-wire (0.0.1)
+    diff-lcs (1.3)
+    ffi (1.10.0)
+    gherkin (5.1.0)
+    hpricot (0.8.6)
+    mono_logger (1.1.0)
+    multi_json (1.13.1)
+    multi_test (0.1.2)
+    mustache (1.1.0)
+    mustermann (1.0.3)
+    rack (2.0.6)
+    rack-protection (2.0.5)
+      rack
+    rake (12.3.2)
+    rdiscount (2.2.0.1)
+    redis (4.1.0)
+    redis-namespace (1.6.0)
+      redis (>= 3.0.4)
+    resque (2.0.0)
+      mono_logger (~> 1.0)
+      multi_json (~> 1.0)
+      redis-namespace (~> 1.6)
+      sinatra (>= 0.9.2)
+      vegas (~> 0.1.2)
+    ronn (0.7.3)
+      hpricot (>= 0.8.2)
+      mustache (>= 0.7.0)
+      rdiscount (>= 1.5.8)
+    rspec (3.8.0)
+      rspec-core (~> 3.8.0)
+      rspec-expectations (~> 3.8.0)
+      rspec-mocks (~> 3.8.0)
+    rspec-core (3.8.0)
+      rspec-support (~> 3.8.0)
+    rspec-expectations (3.8.2)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-mocks (3.8.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-support (3.8.0)
+    sinatra (2.0.5)
+      mustermann (~> 1.0)
+      rack (~> 2.0)
+      rack-protection (= 2.0.5)
+      tilt (~> 2.0)
+    thor (0.20.3)
+    tilt (2.0.9)
+    vegas (0.1.11)
+      rack (>= 1.0.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  aruba (~> 0.14.0)
+  bundler (~> 2.0)
+  cucumber (~> 3.0)
+  mustache
+  resque-pool!
+  ronn
+  rspec (~> 3.8)
+
+BUNDLED WITH
+   2.0.1

data/LICENSE.txt

@@ -1,4 +1,6 @@
-Copyright (C) 2010 by Nicholas Evans <nick@ekenosen.net>, et al.
+The MIT License (MIT)
+
+Copyright (C) 2019 by Nicholas Evans, and other contributors
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -17,4 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
-

data/README.md

@@ -4,23 +4,23 @@ Resque Pool
 [![Build Status](https://travis-ci.org/nevans/resque-pool.png)](https://travis-ci.org/nevans/resque-pool)
 [![Dependency Status](https://gemnasium.com/nevans/resque-pool.png)](https://gemnasium.com/nevans/resque-pool)
 
-Resque pool is a simple library for managing a pool of
-[resque](https://github.com/defunkt/resque) workers.  Given a a config file, it
-manages your workers for you, starting up the appropriate number of workers for
-each worker type.
+Resque pool is a daemon for managing a pool of
+[resque](https://github.com/defunkt/resque) workers.  With a simple config file,
+it manages your workers for you, starting up the appropriate number of workers
+for each worker type.
 
 Benefits
 ---------
 
 * Less config - With a simple YAML file, you can start up a pool daemon, and it
-  will monitor your workers for you.  An example init.d script, monit config,
-  and chef cookbook are provided.
-* Less memory - If you are using Ruby Enterprise Edition, or any ruby with
-  copy-on-write safe garbage collection, this should save you a *lot* of memory
-  when you are managing many workers.
+  will monitor your workers for you.
+* Less memory - If you are using ruby 2.0+ (with copy-on-write safe garbage
+  collection), this should save you a *lot* of memory when you are managing many
+  workers.
 * Faster startup - when you start many workers at once, they would normally
-  compete for CPU as they load their environments.  Resque-pool can load the
-  environment once and fork all of the workers almost instantly.
+  compete for CPU as they load their environments.  Resque-pool can load your
+  application once, then rapidly fork the workers after setup.  If a worker
+  crashes or is killed, a new worker will start up and take its place right away.
 
 Upgrading?
 -----------
@@ -80,7 +80,7 @@ end
 For normal work with fresh resque and resque-scheduler gems add next lines in lib/rake/resque.rake
 
 ```ruby
-task 'resque:pool:setup' do
+task "resque:pool:setup" do
   Resque::Pool.after_prefork do |job|
     Resque.redis.client.reconnect
   end
@@ -164,7 +164,7 @@ For example, if you wanted to vary the number of worker processes based on a
 value stored in Redis, you could do something like:
 
 ```ruby
-task resque:pool:setup do
+task "resque:pool:setup" do
   Resque::Pool.config_loader = lambda do |env|
     worker_count = Redis.current.get("pool_workers_#{env}").to_i
     {"queueA,queueB" => worker_count }
@@ -172,6 +172,17 @@ task resque:pool:setup do
 end
 ```
 
+In order to query ActiveRecord, be sure establish the database connection:
+
+```ruby
+task "resque:pool:setup" do                                           
+  Resque::Pool.config_loader = lambda do |env|
+    ActiveRecord::Base.establish_connection
+    JobQueue.pluck(:name, :workers).to_h
+  end
+end
+```
+
 The configuration loader's `#call` method will be invoked about once a second.
 This allows the configuration to constantly change, allowing you to scale the
 number of workers up or down based on different conditions.
@@ -180,14 +191,54 @@ returns. It can optionally implement a `#reset!` method, which will be invoked
 when the HUP signal is received, allowing the loader to flush its cache, or
 perform any other re-initialization.
 
+You can reduce the frequency that your configuration loader is called by
+wrapping it with `Resque::Pool::ConfigLoaders::Throttled` and specifying a time
+(in seconds) to cache the previous value:
+
+```ruby
+task "resque:pool:setup" do
+  redis_loader = lambda do |env|
+    worker_count = Redis.current.get("pool_workers_#{env}").to_i
+    { "queueA,queueB" => worker_count }
+  end
+
+  # calls through to redis_loader at most once per 10 seconds
+  Resque::Pool.config_loader = Resque::Pool::ConfigLoaders::Throttled.new(
+    redis_loader, period: 10
+  )
+end
+```
+
+See [the spec](spec/config_loaders/throttled_spec.rb) for more details.
+
+Zero-downtime code deploys
+--------------------------
+
+In a production environment you will likely want to manage the daemon using a
+process supervisor like `runit` or `god` or an init system like `systemd` or
+`upstart`.  Example configurations for some of these are included in the
+`examples` directory.  With these systems, `reload` typically sends a `HUP`
+signal, which will reload the configuration but not application code.  The
+simplest way to make workers pick up new code after a deploy is to stop and
+start the daemon.  This will result in a period where new jobs are not being
+processed.
+
+To avoid this downtime, leave the old pool running and start a new pool with
+`resque-pool --hot-swap`.
+
+The `--hot-swap` flag will turn off pidfiles (so multiple pools can run at
+once), enable a lock file (so your init system knows when the pool is running),
+and shut down other pools _after_ the workers have started for this pool.
+These behaviors can also be configured separately (see `resque-pool --help`).
+The `upstart` configs in the `examples` directory demonstrate how to supervise a
+daemonized pool with hot swap.
+
+Please be aware that this approach uses more memory than a simple restart, since
+two copies of the application code are loaded at once. _TODO: [#139](https://github.com/nevans/resque-pool/issues/139)_
+
 Other Features
 --------------
 
-An example chef cookbook is provided (and should Just Work at Engine Yard as
-is; just provide a `/data/#{app_name}/shared/config/resque-pool.yml` on your
-utility instances).  Even if you don't use chef you can use the example init.d
-and monitrc erb templates in `examples/chef_cookbook/templates/default`.
-
 You can also start a pool manager via `rake resque:pool` or from a plain old
 ruby script by calling `Resque::Pool.run`.
 
@@ -197,6 +248,10 @@ their current job) if the manager process disappears before them.
 You can specify an alternate config file by setting the `RESQUE_POOL_CONFIG` or
 with the `--config` command line option.
 
+See the `examples` directory for example `chef` cookbook and
+`god` config.  In the `chef` cookbook, you can also find example `init.d` and
+`muninrc` templates (all very out of date, pull requests welcome).
+
 TODO
 -----
 
@@ -205,8 +260,4 @@ See [the TODO list](https://github.com/nevans/resque-pool/issues) at github issu
 Contributors
 -------------
 
-* John Schult (config file can be split by environment)
-* Stephen Celis (increased gemspec sanity)
-* Vincent Agnello, Robert Kamunyori, Paul Kauders; for pairing with me at
-  B'more on Rails Open Source Hack Nights. :)
-
+See [list of contributors on github](https://github.com/nevans/resque-pool/graphs/contributors) or [in the changelog](https://github.com/nevans/resque-pool/blob/master/Changelog.md)

data/Rakefile

@@ -1,20 +1,17 @@
-require 'bundler/setup'
 require 'bundler/gem_tasks'
 
 # for loading the example config file in config/resque-pool.yml
 require 'resque/pool/tasks'
 
 require 'rspec/core/rake_task'
-RSpec::Core::RakeTask.new(:spec) do |t|
-  t.rspec_opts = ["-c", "-f progress"]
-end
+RSpec::Core::RakeTask.new(:spec)
 
 require 'cucumber/rake/task'
 Cucumber::Rake::Task.new(:features) do |c|
   c.profile = "rake"
 end
 
-task :default => [:spec, :features]
+task :default => %i[spec features]
 
 rule(/\.[1-9]$/ => [proc { |tn| "#{tn}.ronn" }]) do |t|
   name = Resque::Pool.name.sub('::','-').upcase
@@ -22,7 +19,7 @@ rule(/\.[1-9]$/ => [proc { |tn| "#{tn}.ronn" }]) do |t|
 
   manual = '--manual "%s"' % name
   organization = '--organization "%s"' % version
-  sh "ronn #{manual} #{organization} <#{t.source} >#{t.name}"
+  sh "bundle exec ronn #{manual} #{organization} <#{t.source} >#{t.name}"
 end
 
 file 'man/resque-pool.1'