statsd 0.0.4 → 0.5.0

data/README.md

@@ -1,10 +1,80 @@
 StatsD
 ======
 
-A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to [graphite][graphite].
+A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to [graphite][graphite] or [mongodb][mongodb].
 
-We ([Etsy][etsy]) [blogged][blog post] about how it works and why we created it.
 
+### Installation
+
+    gem install statsd
+
+### Configuration
+
+Create config.yml to your liking. There are 2 flush protocols: graphite and mongo. The former simple sends to carbon every flush interval. The latter flushes to MongoDB capped collections for 10s and 1min intervals.
+
+Example config.yml
+    ---
+    bind: 127.0.0.1
+    port: 8125
+
+    # Flush interval should be your finest retention in seconds
+    flush_interval: 10        
+
+    # Graphite
+    graphite_host: localhost
+    graphite_port: 2003
+
+    # Mongo
+    mongo_host: localhost
+    mongo_database: statsdb
+
+    # If you change these, you need to delete the capped collections yourself!
+    # Average mongo record size is 152 bytes
+    # 10s or 1min data is transient so we'll use MongoDB's capped collections. These collections are fixed in size.
+    # 5min and 1d data is interesting to preserve long-term. These collections are not capped.
+    retentions: 
+        - name: stats_per_10s
+          seconds: 10
+          capped: true
+          cap_bytes: 268_435_456 # 2**28
+        - name: stats_per_1min
+          seconds: 60
+          capped: true
+          cap_bytes: 1_073_741_824 # 2**30
+        - name: stats_per_5min
+          seconds: 600
+          cap_bytes: 0 
+          capped: false
+        - name: stats_per_day
+          seconds: 86400
+          cap_bytes: 0 
+          capped: false
+
+
+### Server
+Run the server:
+
+Flush to Graphite (default):
+    statsd -c config.yml 
+
+Flush and aggregate to MongoDB:
+    statsd -c config.yml -m
+
+### Client    
+In your client code:
+
+    require 'rubygems'
+    require 'statsd'
+    STATSD = Statsd::Client.new('localhost',8125)
+
+    STATSD.increment('some_counter') # basic incrementing
+    STATSD.increment('system.nested_counter', 0.1) # incrementing with sampling (10%)
+
+    STATSD.decrement(:some_other_counter) # basic decrememting using a symbol
+    STATSD.decrement('system.nested_counter', 0.1) # decrementing with sampling (10%)
+
+    STATSD.timing('some_job_time', 20) # reporting job that took 20ms
+    STATSD.timing('some_job_time', 20, 0.05) # reporting job that took 20ms with sampling (5% sampling)
 
 Concepts
 --------
@@ -47,10 +117,15 @@ Guts
 * [UDP][udp]
   Client libraries use UDP to send information to the StatsD daemon.
 
-* [NodeJS][node]
+* [EventMachine][eventmachine]
 * [Graphite][graphite]
+* [MongoDB][mongodb]
+
+
+Graphite
+--------
 
-Graphite uses "schemas" to define the different round robin datasets it houses (analogous to RRAs in rrdtool). Here's what Etsy is using for the stats databases:
+Graphite uses "schemas" to define the different round robin datasets it houses (analogous to RRAs in rrdtool):
 
     [stats]
     priority = 110 
@@ -65,73 +140,17 @@ That translates to:
 
 This has been a good tradeoff so far between size-of-file (round robin databases are fixed size) and data we care about. Each "stats" database is about 3.2 megs with these retentions.
 
-Ruby
-----
-A ruby version of statsd. 
-
-### Installation
-
-    gem install statsd
-
-### Configuration
-
-Edit the example config.yml and em-server.rb to your liking. There are 2 flush protocols: graphite and mongo (experimental).
-
-### Server
-Run the server:
-
-    ruby em-server.rb
-
-### Client    
-In your client code:
-
-    require 'rubygems'
-    require 'statsd'
-    STATSD = Statsd::Client.new('localhost',8125)
-
-    STATSD.increment('some_counter') # basic incrementing
-    STATSD.increment('system.nested_counter', 0.1) # incrementing with sampling (10%)
-
-    STATSD.decrement(:some_other_counter) # basic decrememting using a symbol
-    STATSD.decrement('system.nested_counter', 0.1) # decrementing with sampling (10%)
-
-    STATSD.timing('some_job_time', 20) # reporting job that took 20ms
-    STATSD.timing('some_job_time', 20, 0.05) # reporting job that took 20ms with sampling (5% sampling)
-
 
 Inspiration
 -----------
 
 StatsD was inspired (heavily) by the project (of the same name) at Flickr. Here's a post where Cal Henderson described it in depth:
 [Counting and timing](http://code.flickr.com/blog/2008/10/27/counting-timing/). Cal re-released the code recently: [Perl StatsD](https://github.com/iamcal/Flickr-StatsD)
-
-
-Contribute
----------------------
-
-You're interested in contributing to StatsD? *AWESOME*. Here are the basic steps:
-
-fork StatsD from here: http://github.com/etsy/statsd
-
-1. Clone your fork
-2. Hack away
-3. If you are adding new functionality, document it in the README
-4. If necessary, rebase your commits into logical chunks, without errors
-5. Push the branch up to GitHub
-6. Send a pull request to the etsy/statsd project.
-
-We'll do our best to get your changes in!
+Inspired by [Etsy's][etsy] [blog post][blog post].
 
 [graphite]: http://graphite.wikidot.com
 [etsy]: http://www.etsy.com
 [blog post]: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
-[node]: http://nodejs.org
 [udp]: http://enwp.org/udp
 [eventmachine]: http://rubyeventmachine.com/
 [mongodb]: http://www.mongodb.org/
-
-Contributors
------------------
-
-In lieu of a list of contributors, check out the commit history for the project: 
-http://github.com/etsy/statsd/commits/master

data/bin/statsd

@@ -0,0 +1,56 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
+require 'yaml'
+require 'optparse'
+
+begin
+  ORIGINAL_ARGV = ARGV.dup
+  
+  options = {:graphite => true, :mongo => false}
+  
+  parser = OptionParser.new do |opts|
+    opts.banner = "Usage: statsd [options]"
+
+    opts.separator ""
+    opts.separator "Options:"
+
+    opts.on("-cCONFIG", "--config-file CONFIG", "Configuration file") do |x|
+     options[:config] = x
+    end
+
+    opts.on("-m", "--mongo", "Flush and aggregate stats to MongoDB") do 
+     options[:mongo] = true
+     options[:graphite] = false
+    end
+
+    opts.on("-g", "--graphite", "Flush stats to Graphite") do 
+     options[:graphite] = true
+    end
+
+    opts.on("-h", "--help", "Show this message") do
+      puts opts
+      exit
+    end
+
+  end
+
+  parser.parse!
+
+  # dispatch
+  if !options[:config] 
+    puts parser.help
+  else
+    require 'statsd'
+    require 'statsd/server'
+    Statsd::Server::Daemon.new.run(options)
+  end
+rescue Exception => e
+  if e.instance_of?(SystemExit)
+    raise
+  else
+    puts 'Uncaught exception'
+    puts e.message
+    puts e.backtrace.join("\n")
+  end
+end

data/config.yml

@@ -1,14 +1,17 @@
 ---
+bind: 127.0.0.1
+port: 8125
+
 # Flush interval should be your finest retention in seconds
 flush_interval: 10        
 
 # Graphite
 graphite_host: localhost
-graphite_port: 8125
+graphite_port: 2003
 
 # Mongo
-mongo_host: statsd.example.com
-mongo_database:     statsdb
+mongo_host: localhost
+mongo_database: statsdb
 
 # If you change these, you need to delete the capped collections yourself!
 # Average mongo record size is 152 bytes

data/lib/statsd/echos.rb

@@ -16,6 +16,6 @@ module EchoServer
 end
 
 EventMachine::run {
-  EventMachine::start_server "127.0.0.1", 8125, EchoServer
-  puts 'running dummy graphite echo server on 8125'
+  EventMachine::start_server "127.0.0.1", 2003, EchoServer
+  puts 'running dummy graphite echo server on 2003'
 }

data/lib/statsd/graphite.rb

@@ -27,7 +27,7 @@ module Statsd
     # end
           
     def flush_stats
-      print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to Graphite"
+      print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to Graphite."
       stat_string = ''
       time = ::Benchmark.realtime do
         ts = Time.now.to_i

data/lib/statsd/mongo.rb

@@ -26,8 +26,6 @@ module Statsd
           value /= flush_interval
           doc = {:stat => key, :value => value, :ts => ts_bucket, :type => "counter" }
           docs.push(doc)
-          message = "stats.#{key} #{value} #{ts}\n"
-          stat_string += message
           counters[key] = 0
 
           num_stats += 1
@@ -75,7 +73,6 @@ module Statsd
             num_stats += 1
           end
         end
-        stat_string += "statsd.numStats #{num_stats} #{ts}\n"
         coll.insert(docs)
         
        aggregate(ts_bucket)

data/lib/statsd/server.rb

@@ -1,14 +1,18 @@
 require 'eventmachine'
+require 'yaml'
+require 'erb'
 module Statsd
   module Server #< EM::Connection  
-    Version = '0.0.4'  
+    Version = '0.5.0'  
     
     FLUSH_INTERVAL = 10
     COUNTERS = {}
     TIMERS = {}
+
     def post_init
       puts "statsd server started!"
     end
+
     def self.get_and_clear_stats!
       counters = COUNTERS.dup
       timers = TIMERS.dup
@@ -16,6 +20,7 @@ module Statsd
       TIMERS.clear
       [counters,timers]
     end
+
     def receive_data(msg)    
       msg.split("\n").each do |row|
         # puts row
@@ -37,5 +42,66 @@ module Statsd
         end
       end
     end    
+
+    class Daemon
+      def run(options)
+        config = YAML::load(ERB.new(IO.read(options[:config])).result)
+
+        if options[:mongo]
+          require 'statsd/mongo'
+          # Setup retention store
+          db = ::Mongo::Connection.new(config['mongo_host']).db(config['mongo_database'])
+          config['retentions'].each do |retention|
+            collection_name = retention['name']
+            unless db.collection_names.include?(collection_name)
+              db.create_collection(collection_name, :capped => retention['capped'], :size => retention['cap_bytes']) 
+            end
+            db.collection(collection_name).ensure_index([['ts', ::Mongo::ASCENDING]])
+          end        
+          Statsd::Mongo.hostname = config['mongo_host']
+          Statsd::Mongo.database = config['mongo_database']
+          Statsd::Mongo.retentions = config['retentions']
+          Statsd::Mongo.flush_interval = config['flush_interval']
+        end
+      
+        if options[:graphite]
+          require 'statsd/graphite' 
+        end
+      
+        # Start the server
+        EventMachine::run do
+          EventMachine::open_datagram_socket(config['bind'], config['port'], Statsd::Server)  
+
+          # Periodically Flush
+          EventMachine::add_periodic_timer(config['flush_interval']) do
+            counters,timers = Statsd::Server.get_and_clear_stats!
+
+             # Flush Adapters
+            if options[:mongo]
+              EM.defer { Statsd::Mongo.flush_stats(counters,timers) } 
+            end
+
+            if options[:graphite]
+              EventMachine.connect config['graphite_host'], config['graphite_port'], Statsd::Graphite do |conn|
+                conn.counters = counters
+                conn.timers = timers
+                conn.flush_interval = config['flush_interval']
+                conn.flush_stats
+              end     
+            end
+          
+          end
+
+        end
+      
+      end
+    end
   end 
-end
+end
+
+
+
+require 'statsd/graphite'
+
+
+

data/lib/statsd/test.rb

@@ -0,0 +1,3 @@
+require './graphite'
+counters = timers = []
+#Statsd::Graphite.flush_stats(counters,timers)

data/statsd.gemspec

@@ -10,12 +10,13 @@ Gem::Specification.new do |s|
   s.email       = ['quasor@me.com']
   s.homepage    = "http://github.com/quasor/statsd"
   s.summary     = "Ruby version of statsd."
-  s.description = "Ruby version of statsd."
+  s.description = "A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite or mongo."
   
   s.required_rubygems_version = ">= 1.3.6"
 
-  s.add_dependency "eventmachine",           "~> 0.12.10"
-  s.add_dependency "mongo",           "~> 1.2.0"
+  s.add_dependency "eventmachine",  "~> 0.12.10"
+  s.add_dependency "mongo",         "~> 1.2.4"
+  s.add_dependency "erubis",        "~> 2.6.6"
 
   s.files        = `git ls-files`.split("\n")
   s.executables  = `git ls-files`.split("\n").map{|f| f =~ /^bin\/(.*)/ ? $1 : nil}.compact

metadata

@@ -2,7 +2,7 @@
 name: statsd
 version: !ruby/object:Gem::Version 
   prerelease: 
-  version: 0.0.4
+  version: 0.5.0
 platform: ruby
 authors: 
 - Andrew Coldham
@@ -33,52 +33,42 @@ dependencies:
     requirements: 
     - - ~>
       - !ruby/object:Gem::Version 
-        version: 1.2.0
+        version: 1.2.4
   type: :runtime
   version_requirements: *id002
-description: Ruby version of statsd.
+- !ruby/object:Gem::Dependency 
+  name: erubis
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        version: 2.6.6
+  type: :runtime
+  version_requirements: *id003
+description: A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite or mongo.
 email: 
 - quasor@me.com
-executables: []
-
+executables: 
+- statsd
 extensions: []
 
 extra_rdoc_files: []
 
 files: 
-- LICENSE
 - README.md
-- config.js
+- bin/statsd
 - config.yml
-- em-server.rb
-- exampleConfig.js
 - lib/statsd.rb
 - lib/statsd/echos.rb
 - lib/statsd/graphite.rb
 - lib/statsd/mongo.rb
 - lib/statsd/server.rb
+- lib/statsd/test.rb
 - netcat-example.sh
-- php-example.php
-- python_example.py
-- stats.js
+- stats.rb
 - statsd.gemspec
-- webapp/Gemfile
-- webapp/Gemfile.lock
-- webapp/README.md
-- webapp/app.rb
-- webapp/bin/rackup
-- webapp/bin/statsd-web
-- webapp/config.yml
-- webapp/public/jquery-1.4.4.js
-- webapp/public/jquery.flot.js
-- webapp/public/jquery.flot.selection.js
-- webapp/vendor/cache/SystemTimer-1.2.2.gem
-- webapp/vendor/cache/rack-1.2.1.gem
-- webapp/vendor/cache/redis-2.1.1.gem
-- webapp/vendor/cache/sinatra-1.1.3.gem
-- webapp/vendor/cache/tilt-1.2.2.gem
-- webapp/vendor/cache/vegas-0.1.8.gem
-- webapp/views/chart.erb
 has_rdoc: true
 homepage: http://github.com/quasor/statsd
 licenses: []

data/LICENSE

@@ -1,22 +0,0 @@
- Copyright (c) 2010 Etsy
-
- Permission is hereby granted, free of charge, to any person
- obtaining a copy of this software and associated documentation
- files (the "Software"), to deal in the Software without
- restriction, including without limitation the rights to use,
- copy, modify, merge, publish, distribute, sublicense, and/or sell
- copies of the Software, and to permit persons to whom the
- Software is furnished to do so, subject to the following
- conditions:
-
- The above copyright notice and this permission notice shall be
- included in all copies or substantial portions of the Software.
-
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
- OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
- HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
- WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
- FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
- OTHER DEALINGS IN THE SOFTWARE.

data/config.js

@@ -1,39 +0,0 @@
-var fs  = require('fs')
-  , sys = require('sys')
-
-var Configurator = function (file) {
-
-  var self = this;
-  var config = {};
-  var oldConfig = {};
-
-  this.updateConfig = function () {
-    sys.log('reading config file: ' + file);
-
-    fs.readFile(file, function (err, data) {
-      if (err) { throw err; }
-      old_config = self.config;
-
-      self.config = process.compile('config = ' + data, file);
-      self.emit('configChanged', self.config);
-    });
-  };
-
-  this.updateConfig();
-
-  fs.watchFile(file, function (curr, prev) {
-    if (curr.ino != prev.ino) { self.updateConfig(); }
-  });
-};
-
-sys.inherits(Configurator, process.EventEmitter);
-
-exports.Configurator = Configurator;
-
-exports.configFile = function(file, callbackFunc) {
-  var config = new Configurator(file);
-  config.on('configChanged', function() {
-    callbackFunc(config.config, config.oldConfig);
-  });
-};
-

data/exampleConfig.js

@@ -1,8 +0,0 @@
-{
-  debug:true
-, dumpMessages:true
-, graphitePort: 2003
-, graphiteHost: "graphite.host.com"
-, port: 8125
-}
-

data/php-example.php

@@ -1,96 +0,0 @@
-<?php
-
-/**
- * Sends statistics to the stats daemon over UDP
- *
- **/
-
-class StatsD {
-
-    /**
-     * Log timing information
-     *
-     * @param string $stats The metric to in log timing info for.
-     * @param float $time The ellapsed time (ms) to log
-     * @param float|1 $sampleRate the rate (0-1) for sampling.
-     **/
-    public static function timing($stat, $time, $sampleRate=1) {
-        StatsD::send(array($stat => "$time|ms"), $sampleRate);
-    }
-
-    /**
-     * Increments one or more stats counters
-     *
-     * @param string|array $stats The metric(s) to increment.
-     * @param float|1 $sampleRate the rate (0-1) for sampling.
-     * @return boolean
-     **/
-    public static function increment($stats, $sampleRate=1) {
-        StatsD::updateStats($stats, 1, $sampleRate);
-    }
-
-    /**
-     * Decrements one or more stats counters.
-     *
-     * @param string|array $stats The metric(s) to decrement.
-     * @param float|1 $sampleRate the rate (0-1) for sampling.
-     * @return boolean
-     **/
-    public static function decrement($stats, $sampleRate=1) {
-        StatsD::updateStats($stats, -1, $sampleRate);
-    }
-
-    /**
-     * Updates one or more stats counters by arbitrary amounts.
-     *
-     * @param string|array $stats The metric(s) to update. Should be either a string or array of metrics.
-     * @param int|1 $delta The amount to increment/decrement each metric by.
-     * @param float|1 $sampleRate the rate (0-1) for sampling.
-     * @return boolean
-     **/
-    public static function updateStats($stats, $delta=1, $sampleRate=1) {
-        if (!is_array($stats)) { $stats = array($stats); }
-        $data = array();
-        foreach($stats as $stat) {
-            $data[$stat] = "$delta|c";
-        }
-
-        StatsD::send($data, $sampleRate);
-    }
-
-    /*
-     * Squirt the metrics over UDP
-     **/
-    public static function send($data, $sampleRate=1) {
-        $config = Config::getInstance();
-        if (! $config->isEnabled("statsd")) { return; }
-
-        // sampling
-        $sampledData = array();
-
-        if ($sampleRate < 1) {
-            foreach ($data as $stat => $value) {
-                if ((mt_rand() / mt_getrandmax()) <= $sampleRate) {
-                    $sampledData[$stat] = "$value|@$sampleRate";
-                }
-            }
-        } else {
-            $sampledData = $data;
-        }
-
-        if (empty($sampledData)) { return; }
-
-        // Wrap this in a try/catch - failures in any of this should be silently ignored
-        try {
-            $host = $config->getConfig("statsd.host");
-            $port = $config->getConfig("statsd.port");
-            $fp = fsockopen("udp://$host", $port, $errno, $errstr);
-            if (! $fp) { return; }
-            foreach ($sampledData as $stat => $value) {
-                fwrite($fp, "$stat:$value");
-            }
-            fclose($fp);
-        } catch (Exception $e) {
-        }
-    }
-}