camayoc 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/HISTORY.md +9 -0
- data/README.md +77 -18
- data/lib/camayoc.rb +9 -5
- data/lib/camayoc/handlers/filter.rb +15 -20
- data/lib/camayoc/handlers/io.rb +7 -7
- data/lib/camayoc/handlers/logger.rb +21 -26
- data/lib/camayoc/handlers/memory.rb +39 -13
- data/lib/camayoc/handlers/redis.rb +21 -13
- data/lib/camayoc/handlers/statsd.rb +22 -10
- data/lib/camayoc/stat_event.rb +10 -3
- data/lib/camayoc/stats.rb +58 -20
- data/lib/camayoc/timer.rb +41 -0
- data/lib/camayoc/version.rb +1 -1
- data/test/handlers/filter_test.rb +39 -35
- data/test/handlers/io_test.rb +25 -11
- data/test/handlers/logger_test.rb +14 -9
- data/test/handlers/memory_test.rb +63 -12
- data/test/handlers/redis_test.rb +10 -5
- data/test/handlers/statsd_test.rb +29 -5
- data/test/registration_test.rb +12 -0
- data/test/stat_event_test.rb +2 -2
- data/test/stats_test.rb +44 -49
- data/test/test_helper.rb +39 -1
- metadata +9 -5
data/HISTORY.md
ADDED
@@ -0,0 +1,9 @@
|
|
1
|
+
## 0.2.0
|
2
|
+
|
3
|
+
* Added the Camayoc::Stats#event method to do general event logging
|
4
|
+
* Refactored handler event propagation; handlers now implement an event(stat_event) method instead of count and timing
|
5
|
+
* Logging and IO handler formatter Procs now take a single StatEvent as an argument
|
6
|
+
|
7
|
+
## 0.1.0
|
8
|
+
|
9
|
+
* Initial release
|
data/README.md
CHANGED
@@ -1,9 +1,9 @@
|
|
1
1
|
Overview
|
2
2
|
========
|
3
|
-
Camayoc is a flexible way to keep track of application stats
|
4
|
-
various logging frameworks (especially Log4r) it makes it easy to
|
5
|
-
information to multiple destinations via the use of Handlers. Right
|
6
|
-
box, it supports sending stats to the following:
|
3
|
+
Camayoc is a flexible way to keep track of application stats and events.
|
4
|
+
Inspired by various logging frameworks (especially Log4r) it makes it easy to
|
5
|
+
send stats information to multiple destinations via the use of Handlers. Right
|
6
|
+
out of the box, it supports sending stats to the following:
|
7
7
|
|
8
8
|
* A [statsd daemon](https://github.com/etsy/statsd/) for ridiculously easy [Graphite](http://graphite.wikidot.com/) stats
|
9
9
|
* An in-memory hash for in-process stats
|
@@ -12,13 +12,25 @@ box, it supports sending stats to the following:
|
|
12
12
|
|
13
13
|
Philosophy
|
14
14
|
----------
|
15
|
-
|
16
|
-
hard (believe me, we know).
|
15
|
+
Keeping track of what goes on in your application is critical. But even
|
16
|
+
critical things get ignored if they're hard (believe me, we know). Camayoc's aim
|
17
|
+
is to make logging events and capturing stats easy:
|
17
18
|
|
18
|
-
* Collecting
|
19
|
-
* All
|
20
|
-
* Organizing stats should be easy
|
21
|
-
* There should be essentially no performance cost to collecting
|
19
|
+
* Collecting this data should take just one line of code
|
20
|
+
* All data collection should be fire-and-forget with no error handling required
|
21
|
+
* Organizing stats and events should be easy
|
22
|
+
* There should be essentially no performance cost to collecting this data
|
23
|
+
|
24
|
+
Events vs Stats
|
25
|
+
---------------
|
26
|
+
In general, events are things that happen in your application that you care
|
27
|
+
about. There are two ways to keep track of events:
|
28
|
+
|
29
|
+
* Keep summary stats about event occurances around (counts, etc)
|
30
|
+
* Keep a detailed record of events that occur in a some sort of log or collection
|
31
|
+
|
32
|
+
Through its use of flexible handlers, Camayoc makes it possilbe to do one or
|
33
|
+
both of these through a single simple interface.
|
22
34
|
|
23
35
|
Examples
|
24
36
|
==========
|
@@ -96,6 +108,24 @@ There are other options as well like :if and :unless that take Procs that can
|
|
96
108
|
be executed to determine if a metric should be sent to the specified handler.
|
97
109
|
See Camayoc::Handlers::Filter for more.
|
98
110
|
|
111
|
+
Event Logging
|
112
|
+
-------------
|
113
|
+
Sometimes you may want to keep a detailed log of events that occur instead of
|
114
|
+
summarizing then via counts. Below is an example of logging event data in a
|
115
|
+
space-delimited format to an IO stream:
|
116
|
+
|
117
|
+
event_log = Camayoc["events"]
|
118
|
+
fmt = Proc.new do |event|
|
119
|
+
([event.ns_stat, Time.now.to_i] + Array(event.value)).join(" ")
|
120
|
+
end
|
121
|
+
event_log.add(Camayoc::Handlers::IO.new(STDOUT,:formatter=>fmt))
|
122
|
+
|
123
|
+
event_log.event("foo",["bar","baz"])
|
124
|
+
|
125
|
+
This will produce the following on stdout:
|
126
|
+
|
127
|
+
events:foo 1321564751 bar baz
|
128
|
+
|
99
129
|
Available Handlers
|
100
130
|
==================
|
101
131
|
Statsd
|
@@ -105,6 +135,12 @@ Class: Camayoc::Handlers::Statsd
|
|
105
135
|
This handler sends data to the statd daemon for use in graphite. If you can get
|
106
136
|
graphite and statsd going, then you'll get pretty graphs.
|
107
137
|
|
138
|
+
This handler does not support logging details about events since this isn't
|
139
|
+
really what statsd and graphite are for. Any calls to the event method will be
|
140
|
+
treated as counts in statsd. If the event value can be converted to an Integer
|
141
|
+
using Ruby's Integer method, then it will be sent as the count. Otherwise, a
|
142
|
+
value of 1 will be sent.
|
143
|
+
|
108
144
|
Memory
|
109
145
|
------
|
110
146
|
Class: Camayoc::Handlers::Memory
|
@@ -112,6 +148,10 @@ Class: Camayoc::Handlers::Memory
|
|
112
148
|
Stores counts and timing data in an in-memory hash. This might be handy for
|
113
149
|
storing some temporary in-process stats.
|
114
150
|
|
151
|
+
If you use this handler for event logging, data will be stored in an in-memory
|
152
|
+
array with a configurable max size. If that size is exceeded, older events will
|
153
|
+
be evicted to make room for new events.
|
154
|
+
|
115
155
|
Logger
|
116
156
|
------
|
117
157
|
Class: Camayoc::Handlers::Logger
|
@@ -120,6 +160,9 @@ Writes out stat values and a timestamp to a logger instance for later analysis.
|
|
120
160
|
The format and method called on the logger can be controlled by constructor
|
121
161
|
arguments. See the class for details.
|
122
162
|
|
163
|
+
This handler is best for event logging, especially when combined with a custom
|
164
|
+
formatter.
|
165
|
+
|
123
166
|
IO
|
124
167
|
--
|
125
168
|
Class: Camayoc::Handlers::IO
|
@@ -127,6 +170,8 @@ Class: Camayoc::Handlers::IO
|
|
127
170
|
Writes out stats values and a timestamp to some stream of your choice. See
|
128
171
|
class for configuration details.
|
129
172
|
|
173
|
+
Another good event logging handler.
|
174
|
+
|
130
175
|
Redis (Experimental)
|
131
176
|
--------------------
|
132
177
|
Class: Camayoc::Handlers::Redis (require "camayoc/handlers/redis" first)
|
@@ -137,23 +182,37 @@ time-based stats system. It does make it easy to collect some simple counts and
|
|
137
182
|
some timing data. You can easily retrieve the stored data from redis by using
|
138
183
|
the #get method.
|
139
184
|
|
185
|
+
This handler does not currently support event logging.
|
186
|
+
|
140
187
|
|
141
188
|
Implementing a Handler
|
142
189
|
======================
|
143
|
-
|
190
|
+
|
191
|
+
v0.2+ Incompatibility Notice
|
192
|
+
----------------------------
|
193
|
+
As of Version 0.2.0, handlers must implement an event method. In 0.1.0,
|
194
|
+
handlers had to implement count and timing methods. Now handlers are encouraged
|
195
|
+
to dispatch events as described below.
|
196
|
+
|
197
|
+
Implementation Example
|
198
|
+
----------------------
|
199
|
+
Let's say you want to implement your own handler, perhaps to MongoDB. Handlers
|
144
200
|
just need to respond to a simple interface. See Camayoc::Handlers::StatEvent
|
145
|
-
for info on the argument to the
|
201
|
+
for info on the argument to the event method.
|
146
202
|
|
147
203
|
class SomeHandler
|
148
|
-
def
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
# Do something
|
204
|
+
def event(evt)
|
205
|
+
case evt.type
|
206
|
+
when :count then handle_count(evt) # do something with a count stat
|
207
|
+
when :timing then handle_timing(evt) # do something with a timing stat
|
208
|
+
when :generic then handle_other(evt) # do something with a generic event
|
154
209
|
end
|
155
210
|
end
|
156
211
|
|
212
|
+
If you were writing a MongoDB handler, the above might increment a value in
|
213
|
+
a collection on :count and :timing and store raw data to a collection for
|
214
|
+
:generic events.
|
215
|
+
|
157
216
|
If you write a handler and would like it included in Camayoc, please fork
|
158
217
|
and send a pull request and we'll get it integrated in.
|
159
218
|
|
data/lib/camayoc.rb
CHANGED
@@ -10,7 +10,7 @@ require 'camayoc/handlers/memory'
|
|
10
10
|
require 'camayoc/handlers/statsd'
|
11
11
|
require 'camayoc/stats'
|
12
12
|
|
13
|
-
module Camayoc
|
13
|
+
module Camayoc
|
14
14
|
|
15
15
|
DELIMITER = ":"
|
16
16
|
|
@@ -19,7 +19,7 @@ module Camayoc
|
|
19
19
|
|
20
20
|
class << self
|
21
21
|
def [](name)
|
22
|
-
@lock.synchronize { @registry[name] ||= _new(name) }
|
22
|
+
@lock.synchronize { @registry[name] ||= _new(name) }
|
23
23
|
end
|
24
24
|
|
25
25
|
def new(name)
|
@@ -33,11 +33,15 @@ module Camayoc
|
|
33
33
|
def thread_safe=(value)
|
34
34
|
@thread_safe = value
|
35
35
|
end
|
36
|
-
|
36
|
+
|
37
37
|
def thread_safe?
|
38
38
|
@thread_safe == true
|
39
39
|
end
|
40
40
|
|
41
|
+
def join(*names)
|
42
|
+
names.flatten.join(DELIMITER)
|
43
|
+
end
|
44
|
+
|
41
45
|
private
|
42
46
|
def _new(name)
|
43
47
|
stats = Stats.new(name,ancestor(name))
|
@@ -58,10 +62,10 @@ module Camayoc
|
|
58
62
|
ancestor = nil
|
59
63
|
path = name.split(DELIMITER)
|
60
64
|
while path.pop && !ancestor
|
61
|
-
ancestor = @registry[
|
65
|
+
ancestor = @registry[join(path)]
|
62
66
|
end
|
63
67
|
ancestor
|
64
68
|
end
|
65
69
|
end
|
66
70
|
|
67
|
-
end
|
71
|
+
end
|
@@ -1,43 +1,38 @@
|
|
1
1
|
module Camayoc
|
2
2
|
module Handlers
|
3
3
|
class Filter
|
4
|
-
|
4
|
+
|
5
|
+
# Constructor - by default, returns true
|
6
|
+
# * +dest+ :: Handler
|
7
|
+
# * +options[]+
|
8
|
+
# * + :with+ :: Regexp matching against event namespace
|
9
|
+
# * + :if+ :: Proc taking type and event returning true
|
10
|
+
# * + :unless+ :: Converse of +if+
|
5
11
|
def initialize(dest,options={},&block)
|
6
12
|
@dest = dest
|
7
13
|
if block_given?
|
8
14
|
@filter = block
|
9
15
|
elsif options[:with]
|
10
16
|
pattern = options[:with]
|
11
|
-
@filter = Proc.new {|
|
17
|
+
@filter = Proc.new {|event| event.ns_stat =~ pattern }
|
12
18
|
elsif options[:if]
|
13
19
|
@filter = options[:if]
|
14
20
|
elsif options[:unless]
|
15
21
|
proc = options[:unless]
|
16
|
-
@filter = Proc.new do |
|
17
|
-
!proc.call(
|
22
|
+
@filter = Proc.new do |event|
|
23
|
+
!proc.call(event)
|
18
24
|
end
|
19
25
|
else
|
20
|
-
@filter = Proc.new {
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
def count(event)
|
25
|
-
if allowed?(:count,event)
|
26
|
-
@dest.count(event)
|
26
|
+
@filter = Proc.new { true }
|
27
27
|
end
|
28
28
|
end
|
29
29
|
|
30
|
-
def
|
31
|
-
if
|
32
|
-
@dest.
|
30
|
+
def event(ev)
|
31
|
+
if @filter.call(ev)
|
32
|
+
@dest.event(ev)
|
33
33
|
end
|
34
34
|
end
|
35
35
|
|
36
|
-
private
|
37
|
-
def allowed?(type,event)
|
38
|
-
@filter.call(type,event)
|
39
|
-
end
|
40
|
-
|
41
36
|
end
|
42
37
|
end
|
43
|
-
end
|
38
|
+
end
|
data/lib/camayoc/handlers/io.rb
CHANGED
@@ -1,25 +1,25 @@
|
|
1
1
|
module Camayoc
|
2
2
|
module Handlers
|
3
3
|
|
4
|
-
# Write to a raw IO stream with optional thread-safe locking before writing
|
5
|
-
# to the stream. By default, each stat message is written on a single line
|
4
|
+
# Write to a raw IO stream with optional thread-safe locking before writing
|
5
|
+
# to the stream. By default, each stat message is written on a single line
|
6
6
|
# using puts. See the options in Camayoc::Handlers:Logger for options.
|
7
7
|
class IO < Logger
|
8
8
|
|
9
9
|
include ThreadSafety
|
10
10
|
|
11
|
-
def initialize(io=$stdout,options={})
|
12
|
-
super(io,{:method=>:puts}.merge(options))
|
11
|
+
def initialize(io=$stdout,options={},&block)
|
12
|
+
super(io,{:method=>:puts}.merge(options),&block)
|
13
13
|
self.thread_safe = Camayoc.thread_safe?
|
14
14
|
end
|
15
15
|
|
16
16
|
protected
|
17
|
-
def write(
|
17
|
+
def write(event)
|
18
18
|
synchronize do
|
19
|
-
super(
|
19
|
+
super(event)
|
20
20
|
end
|
21
21
|
end
|
22
22
|
|
23
23
|
end
|
24
24
|
end
|
25
|
-
end
|
25
|
+
end
|
@@ -1,44 +1,39 @@
|
|
1
1
|
module Camayoc
|
2
2
|
module Handlers
|
3
|
-
|
4
|
-
# Write stats to a logger. Specify the method to call on the logger instance
|
5
|
-
# with :method (usually something like :info). If not :method is specified
|
6
|
-
# :debug will be called on the logger. You can control the format of the
|
3
|
+
|
4
|
+
# Write stats to a logger. Specify the method to call on the logger instance
|
5
|
+
# with :method (usually something like :info). If not :method is specified
|
6
|
+
# :debug will be called on the logger. You can control the format of the
|
7
7
|
# message passed to the logger method using the :formatter Proc.
|
8
8
|
class Logger
|
9
9
|
|
10
10
|
attr_accessor :logger, :method, :formatter
|
11
11
|
|
12
|
-
def initialize(logger, options={})
|
12
|
+
def initialize(logger, options={}, &block)
|
13
13
|
self.logger = logger
|
14
14
|
self.method = options[:method]
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
15
|
+
if block_given?
|
16
|
+
self.formatter = block
|
17
|
+
else
|
18
|
+
self.formatter = (options[:formatter] || default_formatter)
|
19
|
+
end
|
20
20
|
end
|
21
21
|
|
22
|
-
def
|
23
|
-
|
22
|
+
def event(ev)
|
23
|
+
msg = formatter.call(ev)
|
24
|
+
if @method
|
25
|
+
@logger.send(@method,msg)
|
26
|
+
else
|
27
|
+
@logger.debug(msg)
|
28
|
+
end
|
24
29
|
end
|
25
30
|
|
26
31
|
def default_formatter
|
27
|
-
Proc.new do |
|
28
|
-
"#{type} #{event.ns_stat} #{event.value} #{Time.now.utc.to_i}"
|
29
|
-
end
|
30
|
-
end
|
31
|
-
|
32
|
-
protected
|
33
|
-
def write(type,event)
|
34
|
-
msg = formatter.call(type,event)
|
35
|
-
if @method
|
36
|
-
@logger.send(@method,msg)
|
37
|
-
else
|
38
|
-
@logger.debug(msg)
|
39
|
-
end
|
32
|
+
Proc.new do |event|
|
33
|
+
"#{event.type} #{event.ns_stat} #{event.value} #{Time.now.utc.to_i}"
|
40
34
|
end
|
35
|
+
end
|
41
36
|
|
42
37
|
end
|
43
38
|
end
|
44
|
-
end
|
39
|
+
end
|
@@ -1,25 +1,25 @@
|
|
1
1
|
module Camayoc
|
2
2
|
module Handlers
|
3
3
|
class Memory
|
4
|
-
|
4
|
+
|
5
5
|
include ThreadSafety
|
6
6
|
|
7
|
+
DEFAULT_MAX_EVENTS = 1024
|
8
|
+
|
9
|
+
attr_reader :max_events
|
10
|
+
|
7
11
|
def initialize(options={})
|
8
12
|
@data = {}
|
13
|
+
@max_events = options[:max_events].to_i rescue 0
|
14
|
+
@max_events = DEFAULT_MAX_EVENTS if @max_events <= 0
|
9
15
|
self.thread_safe = Camayoc.thread_safe?
|
10
16
|
end
|
11
17
|
|
12
|
-
def
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
|
-
def timing(event)
|
21
|
-
synchronize do
|
22
|
-
(@data[event.ns_stat] ||= Timing.new) << event.value
|
18
|
+
def event(ev)
|
19
|
+
case ev.type
|
20
|
+
when :count then count(ev)
|
21
|
+
when :timing then timing(ev)
|
22
|
+
when :generic then generic(ev)
|
23
23
|
end
|
24
24
|
end
|
25
25
|
|
@@ -30,6 +30,32 @@ module Camayoc
|
|
30
30
|
end
|
31
31
|
alias_method :[], :get
|
32
32
|
|
33
|
+
private
|
34
|
+
def count(ev)
|
35
|
+
stat = ev.ns_stat
|
36
|
+
synchronize do
|
37
|
+
@data[stat] ||= 0
|
38
|
+
@data[stat] += ev.value
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def timing(ev)
|
43
|
+
synchronize do
|
44
|
+
(@data[ev.ns_stat] ||= Timing.new) << ev.value
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def generic(ev)
|
49
|
+
stat = ev.ns_stat
|
50
|
+
synchronize do
|
51
|
+
@data[stat] ||= []
|
52
|
+
@data[stat].unshift ev.value
|
53
|
+
if @data[stat].length > @max_events
|
54
|
+
@data[stat] = @data[stat][0,@max_events]
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
33
59
|
end
|
34
60
|
end
|
35
|
-
end
|
61
|
+
end
|
@@ -5,29 +5,25 @@ module Camayoc
|
|
5
5
|
|
6
6
|
# This class is experimental.
|
7
7
|
class Redis
|
8
|
-
|
8
|
+
|
9
9
|
def initialize(options={})
|
10
10
|
@namespace = options.delete(:namespace)
|
11
11
|
@redis = (options[:redis] || ::Redis.new(options))
|
12
12
|
end
|
13
13
|
|
14
|
-
def
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
ms = event.value
|
21
|
-
@redis.set(key(stat),"timing")
|
22
|
-
@redis.incrby(key("#{stat}:_count"),1)
|
23
|
-
@redis.incrby(key("#{stat}:_total"),ms)
|
14
|
+
def event(ev)
|
15
|
+
case ev.type
|
16
|
+
when :count then count(ev)
|
17
|
+
when :timing then timing(ev)
|
18
|
+
# generic not implemented!
|
19
|
+
end
|
24
20
|
end
|
25
21
|
|
26
22
|
def get(stat)
|
27
23
|
value = @redis.get(key(stat))
|
28
24
|
if value == "timing"
|
29
25
|
Timing.new(
|
30
|
-
@redis.get(key("#{stat}:_total")).to_i,
|
26
|
+
@redis.get(key("#{stat}:_total")).to_i,
|
31
27
|
@redis.get(key("#{stat}:_count")).to_i,
|
32
28
|
nil,nil # Don't support min and max right now in redis
|
33
29
|
)
|
@@ -40,10 +36,22 @@ module Camayoc
|
|
40
36
|
alias_method :[], :get
|
41
37
|
|
42
38
|
private
|
39
|
+
def count(event)
|
40
|
+
@redis.incrby(key(event.ns_stat),event.value)
|
41
|
+
end
|
42
|
+
|
43
|
+
def timing(event)
|
44
|
+
stat = event.ns_stat
|
45
|
+
ms = event.value
|
46
|
+
@redis.set(key(stat),"timing")
|
47
|
+
@redis.incrby(key("#{stat}:_count"),1)
|
48
|
+
@redis.incrby(key("#{stat}:_total"),ms)
|
49
|
+
end
|
50
|
+
|
43
51
|
def key(stat)
|
44
52
|
@namespace ? "#{@namespace}:#{stat}" : stat
|
45
53
|
end
|
46
54
|
|
47
55
|
end
|
48
56
|
end
|
49
|
-
end
|
57
|
+
end
|