statsd 0.0.4 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -1,10 +1,80 @@
1
1
  StatsD
2
2
  ======
3
3
 
4
- A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to [graphite][graphite].
4
+ A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to [graphite][graphite] or [mongodb][mongodb].
5
5
 
6
- We ([Etsy][etsy]) [blogged][blog post] about how it works and why we created it.
7
6
 
7
+ ### Installation
8
+
9
+ gem install statsd
10
+
11
+ ### Configuration
12
+
13
+ Create config.yml to your liking. There are 2 flush protocols: graphite and mongo. The former simple sends to carbon every flush interval. The latter flushes to MongoDB capped collections for 10s and 1min intervals.
14
+
15
+ Example config.yml
16
+ ---
17
+ bind: 127.0.0.1
18
+ port: 8125
19
+
20
+ # Flush interval should be your finest retention in seconds
21
+ flush_interval: 10
22
+
23
+ # Graphite
24
+ graphite_host: localhost
25
+ graphite_port: 2003
26
+
27
+ # Mongo
28
+ mongo_host: localhost
29
+ mongo_database: statsdb
30
+
31
+ # If you change these, you need to delete the capped collections yourself!
32
+ # Average mongo record size is 152 bytes
33
+ # 10s or 1min data is transient so we'll use MongoDB's capped collections. These collections are fixed in size.
34
+ # 5min and 1d data is interesting to preserve long-term. These collections are not capped.
35
+ retentions:
36
+ - name: stats_per_10s
37
+ seconds: 10
38
+ capped: true
39
+ cap_bytes: 268_435_456 # 2**28
40
+ - name: stats_per_1min
41
+ seconds: 60
42
+ capped: true
43
+ cap_bytes: 1_073_741_824 # 2**30
44
+ - name: stats_per_5min
45
+ seconds: 600
46
+ cap_bytes: 0
47
+ capped: false
48
+ - name: stats_per_day
49
+ seconds: 86400
50
+ cap_bytes: 0
51
+ capped: false
52
+
53
+
54
+ ### Server
55
+ Run the server:
56
+
57
+ Flush to Graphite (default):
58
+ statsd -c config.yml
59
+
60
+ Flush and aggregate to MongoDB:
61
+ statsd -c config.yml -m
62
+
63
+ ### Client
64
+ In your client code:
65
+
66
+ require 'rubygems'
67
+ require 'statsd'
68
+ STATSD = Statsd::Client.new('localhost',8125)
69
+
70
+ STATSD.increment('some_counter') # basic incrementing
71
+ STATSD.increment('system.nested_counter', 0.1) # incrementing with sampling (10%)
72
+
73
+ STATSD.decrement(:some_other_counter) # basic decrememting using a symbol
74
+ STATSD.decrement('system.nested_counter', 0.1) # decrementing with sampling (10%)
75
+
76
+ STATSD.timing('some_job_time', 20) # reporting job that took 20ms
77
+ STATSD.timing('some_job_time', 20, 0.05) # reporting job that took 20ms with sampling (5% sampling)
8
78
 
9
79
  Concepts
10
80
  --------
@@ -47,10 +117,15 @@ Guts
47
117
  * [UDP][udp]
48
118
  Client libraries use UDP to send information to the StatsD daemon.
49
119
 
50
- * [NodeJS][node]
120
+ * [EventMachine][eventmachine]
51
121
  * [Graphite][graphite]
122
+ * [MongoDB][mongodb]
123
+
124
+
125
+ Graphite
126
+ --------
52
127
 
53
- Graphite uses "schemas" to define the different round robin datasets it houses (analogous to RRAs in rrdtool). Here's what Etsy is using for the stats databases:
128
+ Graphite uses "schemas" to define the different round robin datasets it houses (analogous to RRAs in rrdtool):
54
129
 
55
130
  [stats]
56
131
  priority = 110
@@ -65,73 +140,17 @@ That translates to:
65
140
 
66
141
  This has been a good tradeoff so far between size-of-file (round robin databases are fixed size) and data we care about. Each "stats" database is about 3.2 megs with these retentions.
67
142
 
68
- Ruby
69
- ----
70
- A ruby version of statsd.
71
-
72
- ### Installation
73
-
74
- gem install statsd
75
-
76
- ### Configuration
77
-
78
- Edit the example config.yml and em-server.rb to your liking. There are 2 flush protocols: graphite and mongo (experimental).
79
-
80
- ### Server
81
- Run the server:
82
-
83
- ruby em-server.rb
84
-
85
- ### Client
86
- In your client code:
87
-
88
- require 'rubygems'
89
- require 'statsd'
90
- STATSD = Statsd::Client.new('localhost',8125)
91
-
92
- STATSD.increment('some_counter') # basic incrementing
93
- STATSD.increment('system.nested_counter', 0.1) # incrementing with sampling (10%)
94
-
95
- STATSD.decrement(:some_other_counter) # basic decrememting using a symbol
96
- STATSD.decrement('system.nested_counter', 0.1) # decrementing with sampling (10%)
97
-
98
- STATSD.timing('some_job_time', 20) # reporting job that took 20ms
99
- STATSD.timing('some_job_time', 20, 0.05) # reporting job that took 20ms with sampling (5% sampling)
100
-
101
143
 
102
144
  Inspiration
103
145
  -----------
104
146
 
105
147
  StatsD was inspired (heavily) by the project (of the same name) at Flickr. Here's a post where Cal Henderson described it in depth:
106
148
  [Counting and timing](http://code.flickr.com/blog/2008/10/27/counting-timing/). Cal re-released the code recently: [Perl StatsD](https://github.com/iamcal/Flickr-StatsD)
107
-
108
-
109
- Contribute
110
- ---------------------
111
-
112
- You're interested in contributing to StatsD? *AWESOME*. Here are the basic steps:
113
-
114
- fork StatsD from here: http://github.com/etsy/statsd
115
-
116
- 1. Clone your fork
117
- 2. Hack away
118
- 3. If you are adding new functionality, document it in the README
119
- 4. If necessary, rebase your commits into logical chunks, without errors
120
- 5. Push the branch up to GitHub
121
- 6. Send a pull request to the etsy/statsd project.
122
-
123
- We'll do our best to get your changes in!
149
+ Inspired by [Etsy's][etsy] [blog post][blog post].
124
150
 
125
151
  [graphite]: http://graphite.wikidot.com
126
152
  [etsy]: http://www.etsy.com
127
153
  [blog post]: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
128
- [node]: http://nodejs.org
129
154
  [udp]: http://enwp.org/udp
130
155
  [eventmachine]: http://rubyeventmachine.com/
131
156
  [mongodb]: http://www.mongodb.org/
132
-
133
- Contributors
134
- -----------------
135
-
136
- In lieu of a list of contributors, check out the commit history for the project:
137
- http://github.com/etsy/statsd/commits/master
data/bin/statsd ADDED
@@ -0,0 +1,56 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+ require 'yaml'
5
+ require 'optparse'
6
+
7
+ begin
8
+ ORIGINAL_ARGV = ARGV.dup
9
+
10
+ options = {:graphite => true, :mongo => false}
11
+
12
+ parser = OptionParser.new do |opts|
13
+ opts.banner = "Usage: statsd [options]"
14
+
15
+ opts.separator ""
16
+ opts.separator "Options:"
17
+
18
+ opts.on("-cCONFIG", "--config-file CONFIG", "Configuration file") do |x|
19
+ options[:config] = x
20
+ end
21
+
22
+ opts.on("-m", "--mongo", "Flush and aggregate stats to MongoDB") do
23
+ options[:mongo] = true
24
+ options[:graphite] = false
25
+ end
26
+
27
+ opts.on("-g", "--graphite", "Flush stats to Graphite") do
28
+ options[:graphite] = true
29
+ end
30
+
31
+ opts.on("-h", "--help", "Show this message") do
32
+ puts opts
33
+ exit
34
+ end
35
+
36
+ end
37
+
38
+ parser.parse!
39
+
40
+ # dispatch
41
+ if !options[:config]
42
+ puts parser.help
43
+ else
44
+ require 'statsd'
45
+ require 'statsd/server'
46
+ Statsd::Server::Daemon.new.run(options)
47
+ end
48
+ rescue Exception => e
49
+ if e.instance_of?(SystemExit)
50
+ raise
51
+ else
52
+ puts 'Uncaught exception'
53
+ puts e.message
54
+ puts e.backtrace.join("\n")
55
+ end
56
+ end
data/config.yml CHANGED
@@ -1,14 +1,17 @@
1
1
  ---
2
+ bind: 127.0.0.1
3
+ port: 8125
4
+
2
5
  # Flush interval should be your finest retention in seconds
3
6
  flush_interval: 10
4
7
 
5
8
  # Graphite
6
9
  graphite_host: localhost
7
- graphite_port: 8125
10
+ graphite_port: 2003
8
11
 
9
12
  # Mongo
10
- mongo_host: statsd.example.com
11
- mongo_database: statsdb
13
+ mongo_host: localhost
14
+ mongo_database: statsdb
12
15
 
13
16
  # If you change these, you need to delete the capped collections yourself!
14
17
  # Average mongo record size is 152 bytes
data/lib/statsd/echos.rb CHANGED
@@ -16,6 +16,6 @@ module EchoServer
16
16
  end
17
17
 
18
18
  EventMachine::run {
19
- EventMachine::start_server "127.0.0.1", 8125, EchoServer
20
- puts 'running dummy graphite echo server on 8125'
19
+ EventMachine::start_server "127.0.0.1", 2003, EchoServer
20
+ puts 'running dummy graphite echo server on 2003'
21
21
  }
@@ -27,7 +27,7 @@ module Statsd
27
27
  # end
28
28
 
29
29
  def flush_stats
30
- print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to Graphite"
30
+ print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to Graphite."
31
31
  stat_string = ''
32
32
  time = ::Benchmark.realtime do
33
33
  ts = Time.now.to_i
data/lib/statsd/mongo.rb CHANGED
@@ -26,8 +26,6 @@ module Statsd
26
26
  value /= flush_interval
27
27
  doc = {:stat => key, :value => value, :ts => ts_bucket, :type => "counter" }
28
28
  docs.push(doc)
29
- message = "stats.#{key} #{value} #{ts}\n"
30
- stat_string += message
31
29
  counters[key] = 0
32
30
 
33
31
  num_stats += 1
@@ -75,7 +73,6 @@ module Statsd
75
73
  num_stats += 1
76
74
  end
77
75
  end
78
- stat_string += "statsd.numStats #{num_stats} #{ts}\n"
79
76
  coll.insert(docs)
80
77
 
81
78
  aggregate(ts_bucket)
data/lib/statsd/server.rb CHANGED
@@ -1,14 +1,18 @@
1
1
  require 'eventmachine'
2
+ require 'yaml'
3
+ require 'erb'
2
4
  module Statsd
3
5
  module Server #< EM::Connection
4
- Version = '0.0.4'
6
+ Version = '0.5.0'
5
7
 
6
8
  FLUSH_INTERVAL = 10
7
9
  COUNTERS = {}
8
10
  TIMERS = {}
11
+
9
12
  def post_init
10
13
  puts "statsd server started!"
11
14
  end
15
+
12
16
  def self.get_and_clear_stats!
13
17
  counters = COUNTERS.dup
14
18
  timers = TIMERS.dup
@@ -16,6 +20,7 @@ module Statsd
16
20
  TIMERS.clear
17
21
  [counters,timers]
18
22
  end
23
+
19
24
  def receive_data(msg)
20
25
  msg.split("\n").each do |row|
21
26
  # puts row
@@ -37,5 +42,66 @@ module Statsd
37
42
  end
38
43
  end
39
44
  end
45
+
46
+ class Daemon
47
+ def run(options)
48
+ config = YAML::load(ERB.new(IO.read(options[:config])).result)
49
+
50
+ if options[:mongo]
51
+ require 'statsd/mongo'
52
+ # Setup retention store
53
+ db = ::Mongo::Connection.new(config['mongo_host']).db(config['mongo_database'])
54
+ config['retentions'].each do |retention|
55
+ collection_name = retention['name']
56
+ unless db.collection_names.include?(collection_name)
57
+ db.create_collection(collection_name, :capped => retention['capped'], :size => retention['cap_bytes'])
58
+ end
59
+ db.collection(collection_name).ensure_index([['ts', ::Mongo::ASCENDING]])
60
+ end
61
+ Statsd::Mongo.hostname = config['mongo_host']
62
+ Statsd::Mongo.database = config['mongo_database']
63
+ Statsd::Mongo.retentions = config['retentions']
64
+ Statsd::Mongo.flush_interval = config['flush_interval']
65
+ end
66
+
67
+ if options[:graphite]
68
+ require 'statsd/graphite'
69
+ end
70
+
71
+ # Start the server
72
+ EventMachine::run do
73
+ EventMachine::open_datagram_socket(config['bind'], config['port'], Statsd::Server)
74
+
75
+ # Periodically Flush
76
+ EventMachine::add_periodic_timer(config['flush_interval']) do
77
+ counters,timers = Statsd::Server.get_and_clear_stats!
78
+
79
+ # Flush Adapters
80
+ if options[:mongo]
81
+ EM.defer { Statsd::Mongo.flush_stats(counters,timers) }
82
+ end
83
+
84
+ if options[:graphite]
85
+ EventMachine.connect config['graphite_host'], config['graphite_port'], Statsd::Graphite do |conn|
86
+ conn.counters = counters
87
+ conn.timers = timers
88
+ conn.flush_interval = config['flush_interval']
89
+ conn.flush_stats
90
+ end
91
+ end
92
+
93
+ end
94
+
95
+ end
96
+
97
+ end
98
+ end
40
99
  end
41
- end
100
+ end
101
+
102
+
103
+
104
+ require 'statsd/graphite'
105
+
106
+
107
+
@@ -0,0 +1,3 @@
1
+ require './graphite'
2
+ counters = timers = []
3
+ #Statsd::Graphite.flush_stats(counters,timers)
File without changes
data/statsd.gemspec CHANGED
@@ -10,12 +10,13 @@ Gem::Specification.new do |s|
10
10
  s.email = ['quasor@me.com']
11
11
  s.homepage = "http://github.com/quasor/statsd"
12
12
  s.summary = "Ruby version of statsd."
13
- s.description = "Ruby version of statsd."
13
+ s.description = "A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite or mongo."
14
14
 
15
15
  s.required_rubygems_version = ">= 1.3.6"
16
16
 
17
- s.add_dependency "eventmachine", "~> 0.12.10"
18
- s.add_dependency "mongo", "~> 1.2.0"
17
+ s.add_dependency "eventmachine", "~> 0.12.10"
18
+ s.add_dependency "mongo", "~> 1.2.4"
19
+ s.add_dependency "erubis", "~> 2.6.6"
19
20
 
20
21
  s.files = `git ls-files`.split("\n")
21
22
  s.executables = `git ls-files`.split("\n").map{|f| f =~ /^bin\/(.*)/ ? $1 : nil}.compact
metadata CHANGED
@@ -2,7 +2,7 @@
2
2
  name: statsd
3
3
  version: !ruby/object:Gem::Version
4
4
  prerelease:
5
- version: 0.0.4
5
+ version: 0.5.0
6
6
  platform: ruby
7
7
  authors:
8
8
  - Andrew Coldham
@@ -33,52 +33,42 @@ dependencies:
33
33
  requirements:
34
34
  - - ~>
35
35
  - !ruby/object:Gem::Version
36
- version: 1.2.0
36
+ version: 1.2.4
37
37
  type: :runtime
38
38
  version_requirements: *id002
39
- description: Ruby version of statsd.
39
+ - !ruby/object:Gem::Dependency
40
+ name: erubis
41
+ prerelease: false
42
+ requirement: &id003 !ruby/object:Gem::Requirement
43
+ none: false
44
+ requirements:
45
+ - - ~>
46
+ - !ruby/object:Gem::Version
47
+ version: 2.6.6
48
+ type: :runtime
49
+ version_requirements: *id003
50
+ description: A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite or mongo.
40
51
  email:
41
52
  - quasor@me.com
42
- executables: []
43
-
53
+ executables:
54
+ - statsd
44
55
  extensions: []
45
56
 
46
57
  extra_rdoc_files: []
47
58
 
48
59
  files:
49
- - LICENSE
50
60
  - README.md
51
- - config.js
61
+ - bin/statsd
52
62
  - config.yml
53
- - em-server.rb
54
- - exampleConfig.js
55
63
  - lib/statsd.rb
56
64
  - lib/statsd/echos.rb
57
65
  - lib/statsd/graphite.rb
58
66
  - lib/statsd/mongo.rb
59
67
  - lib/statsd/server.rb
68
+ - lib/statsd/test.rb
60
69
  - netcat-example.sh
61
- - php-example.php
62
- - python_example.py
63
- - stats.js
70
+ - stats.rb
64
71
  - statsd.gemspec
65
- - webapp/Gemfile
66
- - webapp/Gemfile.lock
67
- - webapp/README.md
68
- - webapp/app.rb
69
- - webapp/bin/rackup
70
- - webapp/bin/statsd-web
71
- - webapp/config.yml
72
- - webapp/public/jquery-1.4.4.js
73
- - webapp/public/jquery.flot.js
74
- - webapp/public/jquery.flot.selection.js
75
- - webapp/vendor/cache/SystemTimer-1.2.2.gem
76
- - webapp/vendor/cache/rack-1.2.1.gem
77
- - webapp/vendor/cache/redis-2.1.1.gem
78
- - webapp/vendor/cache/sinatra-1.1.3.gem
79
- - webapp/vendor/cache/tilt-1.2.2.gem
80
- - webapp/vendor/cache/vegas-0.1.8.gem
81
- - webapp/views/chart.erb
82
72
  has_rdoc: true
83
73
  homepage: http://github.com/quasor/statsd
84
74
  licenses: []
data/LICENSE DELETED
@@ -1,22 +0,0 @@
1
- Copyright (c) 2010 Etsy
2
-
3
- Permission is hereby granted, free of charge, to any person
4
- obtaining a copy of this software and associated documentation
5
- files (the "Software"), to deal in the Software without
6
- restriction, including without limitation the rights to use,
7
- copy, modify, merge, publish, distribute, sublicense, and/or sell
8
- copies of the Software, and to permit persons to whom the
9
- Software is furnished to do so, subject to the following
10
- conditions:
11
-
12
- The above copyright notice and this permission notice shall be
13
- included in all copies or substantial portions of the Software.
14
-
15
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
17
- OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
- NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
19
- HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
20
- WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21
- FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22
- OTHER DEALINGS IN THE SOFTWARE.
data/config.js DELETED
@@ -1,39 +0,0 @@
1
- var fs = require('fs')
2
- , sys = require('sys')
3
-
4
- var Configurator = function (file) {
5
-
6
- var self = this;
7
- var config = {};
8
- var oldConfig = {};
9
-
10
- this.updateConfig = function () {
11
- sys.log('reading config file: ' + file);
12
-
13
- fs.readFile(file, function (err, data) {
14
- if (err) { throw err; }
15
- old_config = self.config;
16
-
17
- self.config = process.compile('config = ' + data, file);
18
- self.emit('configChanged', self.config);
19
- });
20
- };
21
-
22
- this.updateConfig();
23
-
24
- fs.watchFile(file, function (curr, prev) {
25
- if (curr.ino != prev.ino) { self.updateConfig(); }
26
- });
27
- };
28
-
29
- sys.inherits(Configurator, process.EventEmitter);
30
-
31
- exports.Configurator = Configurator;
32
-
33
- exports.configFile = function(file, callbackFunc) {
34
- var config = new Configurator(file);
35
- config.on('configChanged', function() {
36
- callbackFunc(config.config, config.oldConfig);
37
- });
38
- };
39
-
data/exampleConfig.js DELETED
@@ -1,8 +0,0 @@
1
- {
2
- debug:true
3
- , dumpMessages:true
4
- , graphitePort: 2003
5
- , graphiteHost: "graphite.host.com"
6
- , port: 8125
7
- }
8
-
data/php-example.php DELETED
@@ -1,96 +0,0 @@
1
- <?php
2
-
3
- /**
4
- * Sends statistics to the stats daemon over UDP
5
- *
6
- **/
7
-
8
- class StatsD {
9
-
10
- /**
11
- * Log timing information
12
- *
13
- * @param string $stats The metric to in log timing info for.
14
- * @param float $time The ellapsed time (ms) to log
15
- * @param float|1 $sampleRate the rate (0-1) for sampling.
16
- **/
17
- public static function timing($stat, $time, $sampleRate=1) {
18
- StatsD::send(array($stat => "$time|ms"), $sampleRate);
19
- }
20
-
21
- /**
22
- * Increments one or more stats counters
23
- *
24
- * @param string|array $stats The metric(s) to increment.
25
- * @param float|1 $sampleRate the rate (0-1) for sampling.
26
- * @return boolean
27
- **/
28
- public static function increment($stats, $sampleRate=1) {
29
- StatsD::updateStats($stats, 1, $sampleRate);
30
- }
31
-
32
- /**
33
- * Decrements one or more stats counters.
34
- *
35
- * @param string|array $stats The metric(s) to decrement.
36
- * @param float|1 $sampleRate the rate (0-1) for sampling.
37
- * @return boolean
38
- **/
39
- public static function decrement($stats, $sampleRate=1) {
40
- StatsD::updateStats($stats, -1, $sampleRate);
41
- }
42
-
43
- /**
44
- * Updates one or more stats counters by arbitrary amounts.
45
- *
46
- * @param string|array $stats The metric(s) to update. Should be either a string or array of metrics.
47
- * @param int|1 $delta The amount to increment/decrement each metric by.
48
- * @param float|1 $sampleRate the rate (0-1) for sampling.
49
- * @return boolean
50
- **/
51
- public static function updateStats($stats, $delta=1, $sampleRate=1) {
52
- if (!is_array($stats)) { $stats = array($stats); }
53
- $data = array();
54
- foreach($stats as $stat) {
55
- $data[$stat] = "$delta|c";
56
- }
57
-
58
- StatsD::send($data, $sampleRate);
59
- }
60
-
61
- /*
62
- * Squirt the metrics over UDP
63
- **/
64
- public static function send($data, $sampleRate=1) {
65
- $config = Config::getInstance();
66
- if (! $config->isEnabled("statsd")) { return; }
67
-
68
- // sampling
69
- $sampledData = array();
70
-
71
- if ($sampleRate < 1) {
72
- foreach ($data as $stat => $value) {
73
- if ((mt_rand() / mt_getrandmax()) <= $sampleRate) {
74
- $sampledData[$stat] = "$value|@$sampleRate";
75
- }
76
- }
77
- } else {
78
- $sampledData = $data;
79
- }
80
-
81
- if (empty($sampledData)) { return; }
82
-
83
- // Wrap this in a try/catch - failures in any of this should be silently ignored
84
- try {
85
- $host = $config->getConfig("statsd.host");
86
- $port = $config->getConfig("statsd.port");
87
- $fp = fsockopen("udp://$host", $port, $errno, $errstr);
88
- if (! $fp) { return; }
89
- foreach ($sampledData as $stat => $value) {
90
- fwrite($fp, "$stat:$value");
91
- }
92
- fclose($fp);
93
- } catch (Exception $e) {
94
- }
95
- }
96
- }