daemonic 0.0.2 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c90af55fdad70a6c4a99fc782c9fa34fb7ed6939
-  data.tar.gz: 08b9b435ce0d8e4c5b87978b66f804a55b388dab
+  metadata.gz: b1446f7014c74a118db956ede363c6ba0dc31dc6
+  data.tar.gz: e519190127e484ab4ca0722928eaac7feadc9e6f
 SHA512:
-  metadata.gz: d6a23fc8d43c33e1346dce80900130ea7c703d80b05e6028cbb8cb7d0c8c44315f3aad50d1a20f17bcc849412b77376db282582e6b9a2fb16c4d38cfdf1fc444
-  data.tar.gz: e38c14642dc4d6c5b618610b62bfd30e3b315571dff5cf2cbf36ff7e040be1b3735522c85c6e8f841c20ca0986485ddc4a90ebd6a043d69592d93a6654120f74
+  metadata.gz: 584c0e7c4fb1cb899743762b553ddcf4bb76169029f41c4db8b89ab456bcb23f18fa8d7688e2d6ea26b45b9ca18de9caf9c14cd1b371365ad4144f7551795ddc
+  data.tar.gz: 76cfeaa9cdb7067afd482b96fe3de73a8e72d9baeb5ba5f775878c09c89fb962d466199b69d294b82b5512c54e6fa6fc5f9d50c89bf5e3800c0a086d0779d126

data/.gitignore

@@ -15,4 +15,8 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
-log/*.log
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log

data/.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm:
+  - 2.0.0
+  - 2.1.2
+  - rbx-2
+  - ruby-head
+  - jruby
+cache: bundler

data/{LICENSE.txt → MIT-LICENSE.txt}

@@ -1,4 +1,4 @@
-Copyright (c) 2013 TODO: Write your name
+Copyright (c) 2014 TODO: Write your name
 
 MIT License
 

data/README.md

@@ -1,127 +1,157 @@
 # Daemonic
 
-Daemonic is a tool that let's you turn your long running processes into daemons.
+Daemonic makes multi-threaded daemons easy. All you need to do is provide the
+code that actually does the thing you need to do and you're done.
 
-At a glance:
+Because Daemonic is designed to perform multi-threaded work, you need to
+organize your code to fit a "provider-consumer" pattern.
 
-* Designed to be external to and independent of your app.
-* Possibility to spawn multiple instances of your app.
-* Smart restart behavior, comparable to Unicorn.
-* Automatically restarts crashed instances of your app.
+Your worker needs to be an object that responds to the `produce` and `consume`
+methods. Behind the scenes, Daemonic creates a thread pool and delegate work to
+each of the threads.
 
-### What can Daemonic do for me?
+## Example
 
-Say you have written your own daemon. It processes messages of a queue for
-example. You can start it by running `ruby work.rb`. Now the only thing you
-need to do is turn it into a proper daemon. This means daemonizing your
-process, managing restarts, adding concurrency, managing a pid file, etc.
+Let's build a daemon that downloads and parses RSS feeds parses them. You can
+find this example in the examples directory of the project.
 
-This is a lot of work, quite error prone and frankly, very tedious. That's
-where Daemonic comes in. All you need to do, to start your message processor is:
+``` ruby
+#!/usr/bin/env ruby
+
+require "nokogiri"
+require "open-uri"
+require "daemonic"
+
+class FeedWorker
+
+  # some feeds as an example
+  URLS = %w(
+    https://blog.yourkarma.com/rss
+    http://gigaom.com/feed
+  ) * 5
+
+  # The produce method determines what to work on next.
+  # Put the work onto the queue that is passed in.
+  def produce(queue)
+    URLS.each do |url|
+      queue << url
+    end
+  end
+
+  # The consume method does the actual hard work of downloading the feed and parsing it.
+  def consume(message)
+    puts "Downloading #{message}"
+    items = Nokogiri::XML(open(message)).css("item")
+    puts "Processing #{items.size} articles from #{message}"
+  end
 
-```
-$ daemonic --command "ruby work.rb" --pid work.pid --workers 5 --daemonize
+end
+
+feed_worker = FeedWorker.new
+
+Daemonic.run(feed_worker)
 ```
 
-And voila: you now have 5 instances of your worker, each in it's own process.
-This works just like Unicorn, but with any kind of process, not just Rack apps.
-You can restart all workers by sending the `USR2` signal, just like in Unicorn
-too.
+Make the file executable:
 
 ```
-$ kill -USR2 `cat work.pid`
+$ chmod u+x rss
 ```
 
-This will restart each process individually, meaning you never lose capacity.
+Then start the daemon:
+
+```
+$ ./rss start --concurrency 10 --daemonize --pid tmp/worker.pid
+```
 
-There are some caveats though. For one, daemonic checks if the process is
-running, but it is very naive. It doesn't really know if the app is ready, just
-if it is running or not.
+And you can stop it:
 
-Secondly, if your app needs a shared resource, spawning multiple workers will
-not work. This is the case with web servers that need to bind to a port.
+```
+$ ./rss stop --pid tmp/worker.pid
+```
 
-Also, your own worker needs to respond to the `TERM` and `HUP` signals.
-Daemonic expects the app to shut down after sending the `TERM` signal. It's up
-to you to finish up what you are doing.
+Stopping might take a while, because it gently shuts down all the threads,
+letting them finish their work first.
 
 ## Installation
 
-You don't need to add daemonic to your Gemfile. You can simply install it and
-use it straight away.
+Add this line to your application's Gemfile:
 
+``` ruby
+gem 'daemonic'
 ```
-gem install daemonic
-```
-
-## Usage
 
-Daemonic accepts command line options and can also read from a configuration
-file. The configuration file contains the same options as you would pass to the
-command line. Here is an example:
+And then execute:
 
 ```
---command "ruby work.rb"
---workers 5
---pid tmp/worker.pid
---name my-worker
---log log/development.log
---daemonize
+$ bundle
 ```
 
-Then you can run Daemonic with the `config` option:
+Or install it yourself as:
 
 ```
-$ daemonic --config my-worker.conf
+$ gem install daemonic
 ```
 
-There are a bunch more options. See `daemonic --help` for the full list.
+## Usage
 
-### Signals
+When you get down to it, you only need to do these things to create a
+multi-threaded daemon:
 
-These are the signals daemonic responds to:
+* Create an executable.
+* Require daemonic.
+* Require your own worker.
+* End the executable with `Daemonic.run(my_worker)`.
 
-* `TERM`/`INT` shuts down all workers and then the master
-* `HUP` will be forwarded to each worker and the master will reload the config file
-* `USR2` will restart each worker, but not the master
-* `TTIN` increase the number of workers by one
-* `TTOU` decrease the number of workers by one
+You can get help, by running the script you created:
 
-### Creating a worker
+```
+$ ./my_worker
+```
 
-Here's an example of a basic daemonic worker:
+And get help for each command, by passing `--help` to the command:
 
-``` ruby
-exiting = false
+```
+$ ./my_worker start --help
+```
 
-trap("TERM") { exiting = true }
-trap("INT") { exiting = true }
-trap("HUP") {
-  # reload settings
-}
+## How does it work?
 
-puts "Booting worker number #{ENV["DAEMON_WORKER_NUMBER"]}"
+When starting a
+[SizedQueue](http://ruby-doc.org/stdlib-2.0.0/libdoc/thread/rdoc/SizedQueue.html
+) will be created. A bunch of threads (specified by the `--concurrency` option)
+will be spawned to listen to that queue. Each message they pick up will be sent
+to the worker object you provided to be consumed.
 
-SomePollingService.poll do
-  # do your work here
-  break if exiting
-end
+A separate thread will be started that calls your `produce` method continuously.
+Because the SizedQueue has a limit, it will block when the queue is full, until
+some of the consumers are done.
 
-puts "Shutting down worker number #{ENV["DAEMON_WORKER_NUMBER"]}"
-```
+This approach works great for queueing systems. The produce method finds the
+next item, the consume method does the actual work.
+
+## Gotchas
+
+Because Daemonic is multi-threaded, your code needs to be thread-safe.
 
-Most importantly:
+Also, MRI has a Global Interpreter Lock (GIL). This means that under MRI you
+cannot do proper multithreading. This might be a problem for you if most of
+the work you are trying to do is CPU bound. If most work is IO bound (like
+downloading stuff from the internet), this shouldn't be a problem. When one
+consumer is busy doing IO, the other consumers can run. Therefore Daemonic works
+great when your daemon is doing mostly IO.
 
-* Trap INT and TERM signals to cleanly stop your process
+Daemonic ignores all errors. This means that Daemonic will keep on running, but
+you need to make sure you still get notified of those errors.
+
+When using the restart command, you need to provide all the options you provide
+as if starting the application. Restarting only makes sense for daemonized
+processes.
 
 ## Contributing
 
-1. Fork it
+1. Fork it ( https://github.com/yourkarma/daemonic/fork )
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
-
-## TODO
-
-* Ability to create init script like behavior.
+5. Create a new Pull Request

data/Rakefile

@@ -1,9 +1,16 @@
 require "bundler/gem_tasks"
 
-require 'rake/testtask'
+begin
+  require 'cucumber'
+  require 'cucumber/rake/task'
 
-Rake::TestTask.new do |t|
-  t.pattern = "test/*_test.rb"
+  Cucumber::Rake::Task.new(:features)
+
+rescue LoadError
+  task :features do
+    puts "Cucumber not installed."
+    exit 1
+  end
 end
 
-task :default => :test
+task :default => :features

data/daemonic.gemspec

@@ -8,16 +8,19 @@ Gem::Specification.new do |spec|
   spec.version       = Daemonic::VERSION
   spec.authors       = ["iain"]
   spec.email         = ["iain@iain.nl"]
-  spec.description   = %q{Manages daemonizing your workers with basic monitoring and restart behavior.}
-  spec.summary       = %q{Manages daemonizing your workers with basic monitoring and restart behavior.}
+  spec.summary       = %q{Daemonic makes multi-threaded daemons easy.}
+  spec.description   = %q{Daemonic makes multi-threaded daemons easy.}
   spec.homepage      = "https://github.com/yourkarma/daemonic"
   spec.license       = "MIT"
 
-  spec.files         = `git ls-files`.split($/)
+  spec.files         = `git ls-files -z`.split("\x0")
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
-  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "bundler", "~> 1.6"
+  spec.add_development_dependency "rspec", ">= 3.0.0"
   spec.add_development_dependency "rake"
+  spec.add_development_dependency "cucumber"
+  spec.add_development_dependency "aruba"
 end

data/examples/init-d.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+
+# This is an example of an init.d script to wrap around your worker.
+
+set -e # exit the script if any statement returns a non-true return value
+set -u # check whether all variables are initialised
+
+# The variables for this script
+PATH=$PATH:/usr/local/bin
+TIMEOUT=15
+APP_ROOT=/some/path
+SCRIPT=./bin/my_worker
+PID=worker.pid
+USER=worker
+LOG=worker.log
+
+# make sure the path exists
+cd $APP_ROOT || exit 1
+
+case $1 in
+start)
+  su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT start --pid $PID --log $LOG --daemonize"
+  ;;
+restart)
+  su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT restart --pid $PID --log $LOG --stop-timeout $TIMEOUT"
+  ;;
+stop)
+  su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT stop --pid $PID --stop-timeout $TIMEOUT"
+  ;;
+status)
+  su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT status --pid $PID"
+  ;;
+*)
+  echo >&2 "Usage: $0 <start|stop|restart|status>"
+  exit 1
+  ;;
+esac

data/examples/rss

@@ -0,0 +1,41 @@
+#!/usr/bin/env ruby
+
+# To run this file, clone the repo, go to the examples directory and run:
+#
+#     ./rss start -c 10
+
+# This is only for the example to load the gem file inside the project
+$LOAD_PATH << File.expand_path("../../lib", __FILE__)
+
+require "nokogiri"
+require "open-uri"
+require "daemonic"
+
+class FeedWorker
+
+  # some feeds as an example
+  URLS = %w(
+    https://blog.yourkarma.com/rss
+    http://gigaom.com/feed
+  ) * 5
+
+  # The produce method determines what to work on next.
+  # Put the work onto the queue that is passed in.
+  def produce(queue)
+    URLS.each do |url|
+      queue << url
+    end
+  end
+
+  # The consume method does the actual hard work of downloading the feed and parsing it.
+  def consume(message)
+    puts "Downloading #{message}"
+    items = Nokogiri::XML(open(message)).css("item")
+    puts "Processing #{items.size} articles from #{message}"
+  end
+
+end
+
+feed_worker = FeedWorker.new
+
+Daemonic.run(feed_worker)

data/features/support/env.rb

@@ -0,0 +1 @@
+require "aruba/cucumber"

data/features/worker.feature

@@ -0,0 +1,43 @@
+Feature: Worker
+
+  Scenario: Starting and stopping
+
+    Given a file named "worker" with mode "744" and with:
+      """
+      #!/usr/bin/env ruby
+      $LOAD_PATH << File.expand_path("../../../lib", __FILE__)
+      require "daemonic"
+
+      class MyWorker
+
+        def produce(queue)
+          sleep 0.1
+          queue << "tick"
+        end
+
+        def consume(message)
+          puts message
+          sleep 0.1
+        end
+
+      end
+
+      worker = MyWorker.new
+
+      Daemonic.run(worker)
+      """
+
+    When I run `./worker start --daemonize --pid tmp/worker.pid`
+    Then the exit status should be 0
+
+    When I run `./worker status --pid tmp/worker.pid`
+    Then the exit status should be 0
+
+    When I run `./worker restart --pid tmp/worker.pid`
+    Then the exit status should be 0
+
+    When I run `./worker stop --pid tmp/worker.pid`
+    Then the exit status should be 0
+
+    When I run `./worker status --pid tmp/worker.pid`
+    Then the exit status should be 2