daemonic 0.0.2 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +5 -1
- data/.travis.yml +8 -0
- data/{LICENSE.txt → MIT-LICENSE.txt} +1 -1
- data/README.md +108 -78
- data/Rakefile +11 -4
- data/daemonic.gemspec +7 -4
- data/examples/init-d.sh +37 -0
- data/examples/rss +41 -0
- data/features/support/env.rb +1 -0
- data/features/worker.feature +43 -0
- data/lib/daemonic.rb +40 -8
- data/lib/daemonic/cli.rb +147 -40
- data/lib/daemonic/daemon.rb +126 -0
- data/lib/daemonic/pool.rb +43 -82
- data/lib/daemonic/producer.rb +49 -0
- data/lib/daemonic/version.rb +1 -1
- metadata +68 -31
- data/bin/daemonic +0 -6
- data/lib/daemonic/configuration.rb +0 -110
- data/lib/daemonic/logging.rb +0 -14
- data/lib/daemonic/master.rb +0 -119
- data/lib/daemonic/pidfile.rb +0 -64
- data/lib/daemonic/worker.rb +0 -93
- data/test/config +0 -6
- data/test/crappy_daemon.rb +0 -1
- data/test/integration_test.rb +0 -155
- data/test/test_daemon.rb +0 -10
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: b1446f7014c74a118db956ede363c6ba0dc31dc6
|
4
|
+
data.tar.gz: e519190127e484ab4ca0722928eaac7feadc9e6f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 584c0e7c4fb1cb899743762b553ddcf4bb76169029f41c4db8b89ab456bcb23f18fa8d7688e2d6ea26b45b9ca18de9caf9c14cd1b371365ad4144f7551795ddc
|
7
|
+
data.tar.gz: 76cfeaa9cdb7067afd482b96fe3de73a8e72d9baeb5ba5f775878c09c89fb962d466199b69d294b82b5512c54e6fa6fc5f9d50c89bf5e3800c0a086d0779d126
|
data/.gitignore
CHANGED
data/.travis.yml
ADDED
data/README.md
CHANGED
@@ -1,127 +1,157 @@
|
|
1
1
|
# Daemonic
|
2
2
|
|
3
|
-
Daemonic
|
3
|
+
Daemonic makes multi-threaded daemons easy. All you need to do is provide the
|
4
|
+
code that actually does the thing you need to do and you're done.
|
4
5
|
|
5
|
-
|
6
|
+
Because Daemonic is designed to perform multi-threaded work, you need to
|
7
|
+
organize your code to fit a "provider-consumer" pattern.
|
6
8
|
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
* Automatically restarts crashed instances of your app.
|
9
|
+
Your worker needs to be an object that responds to the `produce` and `consume`
|
10
|
+
methods. Behind the scenes, Daemonic creates a thread pool and delegate work to
|
11
|
+
each of the threads.
|
11
12
|
|
12
|
-
|
13
|
+
## Example
|
13
14
|
|
14
|
-
|
15
|
-
|
16
|
-
need to do is turn it into a proper daemon. This means daemonizing your
|
17
|
-
process, managing restarts, adding concurrency, managing a pid file, etc.
|
15
|
+
Let's build a daemon that downloads and parses RSS feeds parses them. You can
|
16
|
+
find this example in the examples directory of the project.
|
18
17
|
|
19
|
-
|
20
|
-
|
18
|
+
``` ruby
|
19
|
+
#!/usr/bin/env ruby
|
20
|
+
|
21
|
+
require "nokogiri"
|
22
|
+
require "open-uri"
|
23
|
+
require "daemonic"
|
24
|
+
|
25
|
+
class FeedWorker
|
26
|
+
|
27
|
+
# some feeds as an example
|
28
|
+
URLS = %w(
|
29
|
+
https://blog.yourkarma.com/rss
|
30
|
+
http://gigaom.com/feed
|
31
|
+
) * 5
|
32
|
+
|
33
|
+
# The produce method determines what to work on next.
|
34
|
+
# Put the work onto the queue that is passed in.
|
35
|
+
def produce(queue)
|
36
|
+
URLS.each do |url|
|
37
|
+
queue << url
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
# The consume method does the actual hard work of downloading the feed and parsing it.
|
42
|
+
def consume(message)
|
43
|
+
puts "Downloading #{message}"
|
44
|
+
items = Nokogiri::XML(open(message)).css("item")
|
45
|
+
puts "Processing #{items.size} articles from #{message}"
|
46
|
+
end
|
21
47
|
|
22
|
-
|
23
|
-
|
48
|
+
end
|
49
|
+
|
50
|
+
feed_worker = FeedWorker.new
|
51
|
+
|
52
|
+
Daemonic.run(feed_worker)
|
24
53
|
```
|
25
54
|
|
26
|
-
|
27
|
-
This works just like Unicorn, but with any kind of process, not just Rack apps.
|
28
|
-
You can restart all workers by sending the `USR2` signal, just like in Unicorn
|
29
|
-
too.
|
55
|
+
Make the file executable:
|
30
56
|
|
31
57
|
```
|
32
|
-
$
|
58
|
+
$ chmod u+x rss
|
33
59
|
```
|
34
60
|
|
35
|
-
|
61
|
+
Then start the daemon:
|
62
|
+
|
63
|
+
```
|
64
|
+
$ ./rss start --concurrency 10 --daemonize --pid tmp/worker.pid
|
65
|
+
```
|
36
66
|
|
37
|
-
|
38
|
-
running, but it is very naive. It doesn't really know if the app is ready, just
|
39
|
-
if it is running or not.
|
67
|
+
And you can stop it:
|
40
68
|
|
41
|
-
|
42
|
-
|
69
|
+
```
|
70
|
+
$ ./rss stop --pid tmp/worker.pid
|
71
|
+
```
|
43
72
|
|
44
|
-
|
45
|
-
|
46
|
-
to you to finish up what you are doing.
|
73
|
+
Stopping might take a while, because it gently shuts down all the threads,
|
74
|
+
letting them finish their work first.
|
47
75
|
|
48
76
|
## Installation
|
49
77
|
|
50
|
-
|
51
|
-
use it straight away.
|
78
|
+
Add this line to your application's Gemfile:
|
52
79
|
|
80
|
+
``` ruby
|
81
|
+
gem 'daemonic'
|
53
82
|
```
|
54
|
-
gem install daemonic
|
55
|
-
```
|
56
|
-
|
57
|
-
## Usage
|
58
83
|
|
59
|
-
|
60
|
-
file. The configuration file contains the same options as you would pass to the
|
61
|
-
command line. Here is an example:
|
84
|
+
And then execute:
|
62
85
|
|
63
86
|
```
|
64
|
-
|
65
|
-
--workers 5
|
66
|
-
--pid tmp/worker.pid
|
67
|
-
--name my-worker
|
68
|
-
--log log/development.log
|
69
|
-
--daemonize
|
87
|
+
$ bundle
|
70
88
|
```
|
71
89
|
|
72
|
-
|
90
|
+
Or install it yourself as:
|
73
91
|
|
74
92
|
```
|
75
|
-
$
|
93
|
+
$ gem install daemonic
|
76
94
|
```
|
77
95
|
|
78
|
-
|
96
|
+
## Usage
|
79
97
|
|
80
|
-
|
98
|
+
When you get down to it, you only need to do these things to create a
|
99
|
+
multi-threaded daemon:
|
81
100
|
|
82
|
-
|
101
|
+
* Create an executable.
|
102
|
+
* Require daemonic.
|
103
|
+
* Require your own worker.
|
104
|
+
* End the executable with `Daemonic.run(my_worker)`.
|
83
105
|
|
84
|
-
|
85
|
-
* `HUP` will be forwarded to each worker and the master will reload the config file
|
86
|
-
* `USR2` will restart each worker, but not the master
|
87
|
-
* `TTIN` increase the number of workers by one
|
88
|
-
* `TTOU` decrease the number of workers by one
|
106
|
+
You can get help, by running the script you created:
|
89
107
|
|
90
|
-
|
108
|
+
```
|
109
|
+
$ ./my_worker
|
110
|
+
```
|
91
111
|
|
92
|
-
|
112
|
+
And get help for each command, by passing `--help` to the command:
|
93
113
|
|
94
|
-
```
|
95
|
-
|
114
|
+
```
|
115
|
+
$ ./my_worker start --help
|
116
|
+
```
|
96
117
|
|
97
|
-
|
98
|
-
trap("INT") { exiting = true }
|
99
|
-
trap("HUP") {
|
100
|
-
# reload settings
|
101
|
-
}
|
118
|
+
## How does it work?
|
102
119
|
|
103
|
-
|
120
|
+
When starting a
|
121
|
+
[SizedQueue](http://ruby-doc.org/stdlib-2.0.0/libdoc/thread/rdoc/SizedQueue.html
|
122
|
+
) will be created. A bunch of threads (specified by the `--concurrency` option)
|
123
|
+
will be spawned to listen to that queue. Each message they pick up will be sent
|
124
|
+
to the worker object you provided to be consumed.
|
104
125
|
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
end
|
126
|
+
A separate thread will be started that calls your `produce` method continuously.
|
127
|
+
Because the SizedQueue has a limit, it will block when the queue is full, until
|
128
|
+
some of the consumers are done.
|
109
129
|
|
110
|
-
|
111
|
-
|
130
|
+
This approach works great for queueing systems. The produce method finds the
|
131
|
+
next item, the consume method does the actual work.
|
132
|
+
|
133
|
+
## Gotchas
|
134
|
+
|
135
|
+
Because Daemonic is multi-threaded, your code needs to be thread-safe.
|
112
136
|
|
113
|
-
|
137
|
+
Also, MRI has a Global Interpreter Lock (GIL). This means that under MRI you
|
138
|
+
cannot do proper multithreading. This might be a problem for you if most of
|
139
|
+
the work you are trying to do is CPU bound. If most work is IO bound (like
|
140
|
+
downloading stuff from the internet), this shouldn't be a problem. When one
|
141
|
+
consumer is busy doing IO, the other consumers can run. Therefore Daemonic works
|
142
|
+
great when your daemon is doing mostly IO.
|
114
143
|
|
115
|
-
|
144
|
+
Daemonic ignores all errors. This means that Daemonic will keep on running, but
|
145
|
+
you need to make sure you still get notified of those errors.
|
146
|
+
|
147
|
+
When using the restart command, you need to provide all the options you provide
|
148
|
+
as if starting the application. Restarting only makes sense for daemonized
|
149
|
+
processes.
|
116
150
|
|
117
151
|
## Contributing
|
118
152
|
|
119
|
-
1. Fork it
|
153
|
+
1. Fork it ( https://github.com/yourkarma/daemonic/fork )
|
120
154
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
121
155
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
122
156
|
4. Push to the branch (`git push origin my-new-feature`)
|
123
|
-
5. Create new Pull Request
|
124
|
-
|
125
|
-
## TODO
|
126
|
-
|
127
|
-
* Ability to create init script like behavior.
|
157
|
+
5. Create a new Pull Request
|
data/Rakefile
CHANGED
@@ -1,9 +1,16 @@
|
|
1
1
|
require "bundler/gem_tasks"
|
2
2
|
|
3
|
-
|
3
|
+
begin
|
4
|
+
require 'cucumber'
|
5
|
+
require 'cucumber/rake/task'
|
4
6
|
|
5
|
-
Rake::
|
6
|
-
|
7
|
+
Cucumber::Rake::Task.new(:features)
|
8
|
+
|
9
|
+
rescue LoadError
|
10
|
+
task :features do
|
11
|
+
puts "Cucumber not installed."
|
12
|
+
exit 1
|
13
|
+
end
|
7
14
|
end
|
8
15
|
|
9
|
-
task :default => :
|
16
|
+
task :default => :features
|
data/daemonic.gemspec
CHANGED
@@ -8,16 +8,19 @@ Gem::Specification.new do |spec|
|
|
8
8
|
spec.version = Daemonic::VERSION
|
9
9
|
spec.authors = ["iain"]
|
10
10
|
spec.email = ["iain@iain.nl"]
|
11
|
-
spec.
|
12
|
-
spec.
|
11
|
+
spec.summary = %q{Daemonic makes multi-threaded daemons easy.}
|
12
|
+
spec.description = %q{Daemonic makes multi-threaded daemons easy.}
|
13
13
|
spec.homepage = "https://github.com/yourkarma/daemonic"
|
14
14
|
spec.license = "MIT"
|
15
15
|
|
16
|
-
spec.files = `git ls-files`.split(
|
16
|
+
spec.files = `git ls-files -z`.split("\x0")
|
17
17
|
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
18
18
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
19
19
|
spec.require_paths = ["lib"]
|
20
20
|
|
21
|
-
spec.add_development_dependency "bundler", "~> 1.
|
21
|
+
spec.add_development_dependency "bundler", "~> 1.6"
|
22
|
+
spec.add_development_dependency "rspec", ">= 3.0.0"
|
22
23
|
spec.add_development_dependency "rake"
|
24
|
+
spec.add_development_dependency "cucumber"
|
25
|
+
spec.add_development_dependency "aruba"
|
23
26
|
end
|
data/examples/init-d.sh
ADDED
@@ -0,0 +1,37 @@
|
|
1
|
+
#!/bin/sh
|
2
|
+
|
3
|
+
# This is an example of an init.d script to wrap around your worker.
|
4
|
+
|
5
|
+
set -e # exit the script if any statement returns a non-true return value
|
6
|
+
set -u # check whether all variables are initialised
|
7
|
+
|
8
|
+
# The variables for this script
|
9
|
+
PATH=$PATH:/usr/local/bin
|
10
|
+
TIMEOUT=15
|
11
|
+
APP_ROOT=/some/path
|
12
|
+
SCRIPT=./bin/my_worker
|
13
|
+
PID=worker.pid
|
14
|
+
USER=worker
|
15
|
+
LOG=worker.log
|
16
|
+
|
17
|
+
# make sure the path exists
|
18
|
+
cd $APP_ROOT || exit 1
|
19
|
+
|
20
|
+
case $1 in
|
21
|
+
start)
|
22
|
+
su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT start --pid $PID --log $LOG --daemonize"
|
23
|
+
;;
|
24
|
+
restart)
|
25
|
+
su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT restart --pid $PID --log $LOG --stop-timeout $TIMEOUT"
|
26
|
+
;;
|
27
|
+
stop)
|
28
|
+
su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT stop --pid $PID --stop-timeout $TIMEOUT"
|
29
|
+
;;
|
30
|
+
status)
|
31
|
+
su --login $USER --shell /bin/sh --command "cd $APP_ROOT; $SCRIPT status --pid $PID"
|
32
|
+
;;
|
33
|
+
*)
|
34
|
+
echo >&2 "Usage: $0 <start|stop|restart|status>"
|
35
|
+
exit 1
|
36
|
+
;;
|
37
|
+
esac
|
data/examples/rss
ADDED
@@ -0,0 +1,41 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
|
3
|
+
# To run this file, clone the repo, go to the examples directory and run:
|
4
|
+
#
|
5
|
+
# ./rss start -c 10
|
6
|
+
|
7
|
+
# This is only for the example to load the gem file inside the project
|
8
|
+
$LOAD_PATH << File.expand_path("../../lib", __FILE__)
|
9
|
+
|
10
|
+
require "nokogiri"
|
11
|
+
require "open-uri"
|
12
|
+
require "daemonic"
|
13
|
+
|
14
|
+
class FeedWorker
|
15
|
+
|
16
|
+
# some feeds as an example
|
17
|
+
URLS = %w(
|
18
|
+
https://blog.yourkarma.com/rss
|
19
|
+
http://gigaom.com/feed
|
20
|
+
) * 5
|
21
|
+
|
22
|
+
# The produce method determines what to work on next.
|
23
|
+
# Put the work onto the queue that is passed in.
|
24
|
+
def produce(queue)
|
25
|
+
URLS.each do |url|
|
26
|
+
queue << url
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
# The consume method does the actual hard work of downloading the feed and parsing it.
|
31
|
+
def consume(message)
|
32
|
+
puts "Downloading #{message}"
|
33
|
+
items = Nokogiri::XML(open(message)).css("item")
|
34
|
+
puts "Processing #{items.size} articles from #{message}"
|
35
|
+
end
|
36
|
+
|
37
|
+
end
|
38
|
+
|
39
|
+
feed_worker = FeedWorker.new
|
40
|
+
|
41
|
+
Daemonic.run(feed_worker)
|
@@ -0,0 +1 @@
|
|
1
|
+
require "aruba/cucumber"
|
@@ -0,0 +1,43 @@
|
|
1
|
+
Feature: Worker
|
2
|
+
|
3
|
+
Scenario: Starting and stopping
|
4
|
+
|
5
|
+
Given a file named "worker" with mode "744" and with:
|
6
|
+
"""
|
7
|
+
#!/usr/bin/env ruby
|
8
|
+
$LOAD_PATH << File.expand_path("../../../lib", __FILE__)
|
9
|
+
require "daemonic"
|
10
|
+
|
11
|
+
class MyWorker
|
12
|
+
|
13
|
+
def produce(queue)
|
14
|
+
sleep 0.1
|
15
|
+
queue << "tick"
|
16
|
+
end
|
17
|
+
|
18
|
+
def consume(message)
|
19
|
+
puts message
|
20
|
+
sleep 0.1
|
21
|
+
end
|
22
|
+
|
23
|
+
end
|
24
|
+
|
25
|
+
worker = MyWorker.new
|
26
|
+
|
27
|
+
Daemonic.run(worker)
|
28
|
+
"""
|
29
|
+
|
30
|
+
When I run `./worker start --daemonize --pid tmp/worker.pid`
|
31
|
+
Then the exit status should be 0
|
32
|
+
|
33
|
+
When I run `./worker status --pid tmp/worker.pid`
|
34
|
+
Then the exit status should be 0
|
35
|
+
|
36
|
+
When I run `./worker restart --pid tmp/worker.pid`
|
37
|
+
Then the exit status should be 0
|
38
|
+
|
39
|
+
When I run `./worker stop --pid tmp/worker.pid`
|
40
|
+
Then the exit status should be 0
|
41
|
+
|
42
|
+
When I run `./worker status --pid tmp/worker.pid`
|
43
|
+
Then the exit status should be 2
|