dripdrop 0.0.2 → 0.0.3
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +19 -1
- data/Rakefile +0 -20
- data/VERSION +1 -1
- data/dripdrop.gemspec +3 -5
- data/lib/dripdrop/agent.rb +6 -2
- data/lib/dripdrop/collector.rb +15 -1
- data/lib/dripdrop/message.rb +15 -0
- data/lib/dripdrop/mlogger.rb +1 -0
- data/lib/dripdrop/publisher.rb +3 -0
- metadata +2 -5
- data/bin/drip-collector +0 -7
- data/lib/dripdrop/webserver.rb +0 -10
data/README.md
CHANGED
@@ -3,9 +3,25 @@
|
|
3
3
|
0MQ Based App Event Monitoring / processing.
|
4
4
|
A work in progress.
|
5
5
|
|
6
|
+
# Why use dripdrop?
|
7
|
+
|
8
|
+
You want to asynchronously process / monitor arbitrary messages from your app.
|
9
|
+
dripdrop does this well for a few reasons.
|
10
|
+
|
11
|
+
* It's fast. dripdrop doesn't slow down your app. 0MQ + Bert are fast. Sending a message never blocks.
|
12
|
+
* It's flexible. By leveraging 0MQ pub/sub sockets you can have many different processors (collectors in dripdrop) that don't impact or even care about each other
|
13
|
+
* It's easy. Check out the agent and collector examples below. You can be processing stuff in no time.
|
14
|
+
|
6
15
|
## An example with a WebSocket UI:
|
7
16
|
|
8
|
-
|
17
|
+
### You'll need to have the zmq dev libs on your machine. On OSX this means
|
18
|
+
|
19
|
+
1. Download and build zeromq from [zeromq.org](http://www.zeromq.org/area:download)
|
20
|
+
1. The agent just uses the plain zmq gem, which runs fine on ruby 1.8.7+, *EDIT!* If you use my fork of ffi-rzmq everything should work with 1.8.7! This isn't fully tested yet though *END EDIT* this is so you can use it in say your rails app. Everything else needs ruby 1.9.2 or jruby and uses Chuck Remes [ffi-rzmq](http://github.com/chuckremes/ffi-rzmq), and [zmqmachine](http://github.com/chuckremes/zmqmachine) gems which you must build yourself. I recommend using rvm to enable the use of multiple rubies on one machine.
|
21
|
+
1. zmq_forwarder comes with zmq, use this to aggregate agent messages using the example config shown below
|
22
|
+
|
23
|
+
### To run a simple example, feeding data to a websockets UI
|
24
|
+
|
9
25
|
#### Aggregate agents with zmq_forwarder (comes with zmq)
|
10
26
|
$ zmq_forwarder examples/forwarder.cfg
|
11
27
|
|
@@ -21,6 +37,8 @@ To run a simple example, feeding data to a websockets UI
|
|
21
37
|
|
22
38
|
## Example Topology
|
23
39
|
|
40
|
+
You can add as many listeners as you want, or reconfigure things any way you want. Heres how I plan on using it.
|
41
|
+
|
24
42
|
![topology](http://github.com/andrewvc/dripdrop/raw/master/doc_img/topology.png "Topology")
|
25
43
|
|
26
44
|
## Sending Messages
|
data/Rakefile
CHANGED
@@ -18,26 +18,6 @@ rescue LoadError
|
|
18
18
|
puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
|
19
19
|
end
|
20
20
|
|
21
|
-
require 'rake/testtask'
|
22
|
-
Rake::TestTask.new(:test) do |test|
|
23
|
-
test.libs << 'lib' << 'test'
|
24
|
-
test.pattern = 'test/**/test_*.rb'
|
25
|
-
test.verbose = true
|
26
|
-
end
|
27
|
-
|
28
|
-
begin
|
29
|
-
require 'rcov/rcovtask'
|
30
|
-
Rcov::RcovTask.new do |test|
|
31
|
-
test.libs << 'test'
|
32
|
-
test.pattern = 'test/**/test_*.rb'
|
33
|
-
test.verbose = true
|
34
|
-
end
|
35
|
-
rescue LoadError
|
36
|
-
task :rcov do
|
37
|
-
abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
|
38
|
-
end
|
39
|
-
end
|
40
|
-
|
41
21
|
task :test => :check_dependencies
|
42
22
|
|
43
23
|
task :default => :test
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.0.
|
1
|
+
0.0.3
|
data/dripdrop.gemspec
CHANGED
@@ -5,14 +5,14 @@
|
|
5
5
|
|
6
6
|
Gem::Specification.new do |s|
|
7
7
|
s.name = %q{dripdrop}
|
8
|
-
s.version = "0.0.
|
8
|
+
s.version = "0.0.3"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
11
|
s.authors = ["Andrew Cholakian"]
|
12
12
|
s.date = %q{2010-08-14}
|
13
13
|
s.description = %q{0MQ App stats}
|
14
14
|
s.email = %q{andrew@andrewvc.com}
|
15
|
-
s.executables = ["drip-
|
15
|
+
s.executables = ["drip-mlogger", "drip-publisher"]
|
16
16
|
s.extra_rdoc_files = [
|
17
17
|
"LICENSE",
|
18
18
|
"README.md"
|
@@ -24,7 +24,6 @@ Gem::Specification.new do |s|
|
|
24
24
|
"README.md",
|
25
25
|
"Rakefile",
|
26
26
|
"VERSION",
|
27
|
-
"bin/drip-collector",
|
28
27
|
"bin/drip-mlogger",
|
29
28
|
"bin/drip-publisher",
|
30
29
|
"doc_img/topology.png",
|
@@ -41,8 +40,7 @@ Gem::Specification.new do |s|
|
|
41
40
|
"lib/dripdrop/collector.rb",
|
42
41
|
"lib/dripdrop/message.rb",
|
43
42
|
"lib/dripdrop/mlogger.rb",
|
44
|
-
"lib/dripdrop/publisher.rb"
|
45
|
-
"lib/dripdrop/webserver.rb"
|
43
|
+
"lib/dripdrop/publisher.rb"
|
46
44
|
]
|
47
45
|
s.homepage = %q{http://github.com/andrewvc/dripdrop}
|
48
46
|
s.rdoc_options = ["--charset=UTF-8"]
|
data/lib/dripdrop/agent.rb
CHANGED
@@ -3,8 +3,11 @@ require 'zmq'
|
|
3
3
|
require 'bert'
|
4
4
|
|
5
5
|
class DripDrop
|
6
|
+
#The Agent class is a simple ZMQ Pub client. It uses DripDrop::Message messages
|
6
7
|
class Agent
|
7
8
|
attr_reader :address, :context, :socket
|
9
|
+
|
10
|
+
#address should be a string like tcp://127.0.0.1
|
8
11
|
def initialize(address)
|
9
12
|
@address = address
|
10
13
|
@context = ZMQ::Context.new(1)
|
@@ -12,8 +15,9 @@ class DripDrop
|
|
12
15
|
@socket.connect(@address)
|
13
16
|
end
|
14
17
|
|
15
|
-
|
16
|
-
|
18
|
+
#Sends a DripDrop::Message to the socket
|
19
|
+
def send_message(name,body,head={})
|
20
|
+
puts @socket.send(DripDrop::Message.new(name,:body => body, :head => head).encoded, 0)
|
17
21
|
end
|
18
22
|
end
|
19
23
|
end
|
data/lib/dripdrop/collector.rb
CHANGED
@@ -4,9 +4,12 @@ require 'uri'
|
|
4
4
|
require 'dripdrop/message'
|
5
5
|
|
6
6
|
class DripDrop
|
7
|
+
#Publishes the ZMQ messages. This is not evented as zmqmachine seems
|
8
|
+
#to max out the CPU on ZMQ::PUB sockets
|
7
9
|
class CollectorPub
|
8
10
|
attr_reader :context, :socket, :address
|
9
11
|
|
12
|
+
#Takes either as string or URI as an address
|
10
13
|
def initialize(address)
|
11
14
|
@address = address
|
12
15
|
@context = ZMQ::Context.new(1)
|
@@ -14,14 +17,17 @@ class DripDrop
|
|
14
17
|
@socket.bind(@address.to_s)
|
15
18
|
end
|
16
19
|
|
20
|
+
#Sends an already encoded DripDrop::Message
|
17
21
|
def send_message(message)
|
18
22
|
@socket.send(message)
|
19
23
|
end
|
20
24
|
end
|
21
25
|
|
26
|
+
#Listens on a zmqmachine sub_socket.
|
22
27
|
class CollectorSub
|
23
28
|
attr_reader :context, :socket, :address, :collector
|
24
29
|
|
30
|
+
#Takes a zmqmachine reactor in sub mode's context, self, address, and a CollectorPublisher
|
25
31
|
def initialize(context, collector, address, publisher=nil)
|
26
32
|
@context = context
|
27
33
|
@collector = collector
|
@@ -40,9 +46,13 @@ class DripDrop
|
|
40
46
|
end
|
41
47
|
end
|
42
48
|
end
|
43
|
-
|
49
|
+
|
50
|
+
#Collector is meant to be subclassed. It's used to provide basic pub/sub functionality.
|
51
|
+
#Subclasses should provide +on_recv+, which gets called on receipt of a message.
|
52
|
+
#If pub_addr is specified +publish+ can be called, which sends a message to the pub socket.
|
44
53
|
class Collector
|
45
54
|
attr_reader :sub_reactor, :sub_addr, :pub_addr
|
55
|
+
|
46
56
|
def initialize(sub_addr='tcp://127.0.0.1:2900',pub_addr=nil)
|
47
57
|
sub_addr_uri = URI.parse(sub_addr)
|
48
58
|
host, port = sub_addr_uri.host, sub_addr_uri.port.to_i
|
@@ -56,6 +66,7 @@ class DripDrop
|
|
56
66
|
end
|
57
67
|
end
|
58
68
|
|
69
|
+
#Run the collector. Returns the reactor, so if this is the only thing in your script, be sure to call +join+ on the reactor.
|
59
70
|
def run
|
60
71
|
puts "Run"
|
61
72
|
@sub_reactor.run do |context|
|
@@ -64,10 +75,13 @@ class DripDrop
|
|
64
75
|
@sub_reactor
|
65
76
|
end
|
66
77
|
|
78
|
+
#If pub_addr was specified when the collector was initialized, messages can be broadcast using this.
|
67
79
|
def publish(message)
|
68
80
|
@publisher.send_string(message.encoded)
|
69
81
|
end
|
70
82
|
|
83
|
+
#Intended to be overriden by a subclass.
|
84
|
+
#Receives an encoded DripDrop::Message
|
71
85
|
def on_recv(message); end
|
72
86
|
end
|
73
87
|
end
|
data/lib/dripdrop/message.rb
CHANGED
@@ -2,9 +2,20 @@ require 'rubygems'
|
|
2
2
|
require 'bert'
|
3
3
|
|
4
4
|
class DripDrop
|
5
|
+
#DripDrop::Message messages are exchanged between all tiers in the architecture
|
6
|
+
#A Message is composed of a name, head, and body. The name exists primarily for the
|
7
|
+
#purpose of native ZMQ filtering, since ZMQ can filter based on a message prefix.
|
8
|
+
#
|
9
|
+
#The name is any string consisting of non-null chars.
|
10
|
+
#The rest of the payload is a BERT encoded head and body, both of which are hashes.
|
11
|
+
#The head and body don't have rigid definitions yet, use as you please.
|
5
12
|
class Message
|
6
13
|
attr_accessor :name, :head, :body
|
7
14
|
|
15
|
+
#Create a new message.
|
16
|
+
#example:
|
17
|
+
# Message.new('mymessage', :head => {:timestamp => Time.now},
|
18
|
+
# :body => {:mykey => :myval, :other_key => ['complex']})
|
8
19
|
def initialize(name,extra={})
|
9
20
|
raise "No null chars allowed in message names!" if name.include?("\0")
|
10
21
|
|
@@ -15,14 +26,18 @@ class DripDrop
|
|
15
26
|
@body = extra[:body]
|
16
27
|
end
|
17
28
|
|
29
|
+
#The encoded message, ready to be sent across the wire via ZMQ
|
18
30
|
def encoded
|
19
31
|
"#{@name}\0#{BERT.encode({:head => @head, :body => @body})}"
|
20
32
|
end
|
21
33
|
|
34
|
+
#Convert the Message to a hash like:
|
35
|
+
#{:name => @name, :head => @head, :body => @body}
|
22
36
|
def to_hash
|
23
37
|
{:name => @name, :head => @head, :body => @body}
|
24
38
|
end
|
25
39
|
|
40
|
+
#Parses an encoded message
|
26
41
|
def self.parse(msg)
|
27
42
|
name, encoded_body = msg.split("\0",2)
|
28
43
|
decoded = BERT.decode(encoded_body)
|
data/lib/dripdrop/mlogger.rb
CHANGED
data/lib/dripdrop/publisher.rb
CHANGED
@@ -24,8 +24,11 @@ class DripDrop
|
|
24
24
|
end
|
25
25
|
end
|
26
26
|
|
27
|
+
|
28
|
+
#WebSocket server that rebroadcasts all DripDrop::Messages it subscribes to as JSON
|
27
29
|
class Publisher
|
28
30
|
attr_reader :sub_address, :sub_collector, :ws_address
|
31
|
+
|
29
32
|
def initialize(sub_address='tcp://127.0.0.1:2901',ws_address='ws://127.0.0.1:2902')
|
30
33
|
@sub_address = URI.parse(sub_address)
|
31
34
|
@ws_address = URI.parse(ws_address)
|
metadata
CHANGED
@@ -5,8 +5,8 @@ version: !ruby/object:Gem::Version
|
|
5
5
|
segments:
|
6
6
|
- 0
|
7
7
|
- 0
|
8
|
-
-
|
9
|
-
version: 0.0.
|
8
|
+
- 3
|
9
|
+
version: 0.0.3
|
10
10
|
platform: ruby
|
11
11
|
authors:
|
12
12
|
- Andrew Cholakian
|
@@ -46,7 +46,6 @@ dependencies:
|
|
46
46
|
description: 0MQ App stats
|
47
47
|
email: andrew@andrewvc.com
|
48
48
|
executables:
|
49
|
-
- drip-collector
|
50
49
|
- drip-mlogger
|
51
50
|
- drip-publisher
|
52
51
|
extensions: []
|
@@ -61,7 +60,6 @@ files:
|
|
61
60
|
- README.md
|
62
61
|
- Rakefile
|
63
62
|
- VERSION
|
64
|
-
- bin/drip-collector
|
65
63
|
- bin/drip-mlogger
|
66
64
|
- bin/drip-publisher
|
67
65
|
- doc_img/topology.png
|
@@ -79,7 +77,6 @@ files:
|
|
79
77
|
- lib/dripdrop/message.rb
|
80
78
|
- lib/dripdrop/mlogger.rb
|
81
79
|
- lib/dripdrop/publisher.rb
|
82
|
-
- lib/dripdrop/webserver.rb
|
83
80
|
has_rdoc: true
|
84
81
|
homepage: http://github.com/andrewvc/dripdrop
|
85
82
|
licenses: []
|
data/bin/drip-collector
DELETED