statsd 0.0.4

data/LICENSE

@@ -0,0 +1,22 @@
+ Copyright (c) 2010 Etsy
+
+ Permission is hereby granted, free of charge, to any person
+ obtaining a copy of this software and associated documentation
+ files (the "Software"), to deal in the Software without
+ restriction, including without limitation the rights to use,
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the
+ Software is furnished to do so, subject to the following
+ conditions:
+
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,137 @@
+StatsD
+======
+
+A network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to [graphite][graphite].
+
+We ([Etsy][etsy]) [blogged][blog post] about how it works and why we created it.
+
+
+Concepts
+--------
+
+* *buckets*
+  Each stat is in it's own "bucket". They are not predefined anywhere. Buckets can be named anything that will translate to Graphite (periods make folders, etc)
+
+* *values*
+  Each stat will have a value. How it is interpreted depends on modifiers
+  
+* *flush*
+  After the flush interval timeout (default 10 seconds), stats are munged and sent over to Graphite.
+
+Counting
+--------
+
+    gorets:1|c
+
+This is a simple counter. Add 1 to the "gorets" bucket. It stays in memory until the flush interval.
+
+
+Timing
+------
+
+    glork:320|ms
+
+The glork took 320ms to complete this time. StatsD figures out 90th percentile, average (mean), lower and upper bounds for the flush interval.
+
+Sampling
+--------
+
+    gorets:1|c|@0.1
+
+Tells StatsD that this counter is being sent sampled ever 1/10th of the time.
+
+
+Guts
+----
+
+* [UDP][udp]
+  Client libraries use UDP to send information to the StatsD daemon.
+
+* [NodeJS][node]
+* [Graphite][graphite]
+
+Graphite uses "schemas" to define the different round robin datasets it houses (analogous to RRAs in rrdtool). Here's what Etsy is using for the stats databases:
+
+    [stats]
+    priority = 110 
+    pattern = ^stats\..*
+    retentions = 10:2160,60:10080,600:262974
+
+That translates to:
+
+* 6 hours of 10 second data (what we consider "near-realtime")
+* 1 week of 1 minute data
+* 5 years of 10 minute data
+
+This has been a good tradeoff so far between size-of-file (round robin databases are fixed size) and data we care about. Each "stats" database is about 3.2 megs with these retentions.
+
+Ruby
+----
+A ruby version of statsd. 
+
+### Installation
+
+    gem install statsd
+
+### Configuration
+
+Edit the example config.yml and em-server.rb to your liking. There are 2 flush protocols: graphite and mongo (experimental).
+
+### Server
+Run the server:
+
+    ruby em-server.rb
+
+### Client    
+In your client code:
+
+    require 'rubygems'
+    require 'statsd'
+    STATSD = Statsd::Client.new('localhost',8125)
+
+    STATSD.increment('some_counter') # basic incrementing
+    STATSD.increment('system.nested_counter', 0.1) # incrementing with sampling (10%)
+
+    STATSD.decrement(:some_other_counter) # basic decrememting using a symbol
+    STATSD.decrement('system.nested_counter', 0.1) # decrementing with sampling (10%)
+
+    STATSD.timing('some_job_time', 20) # reporting job that took 20ms
+    STATSD.timing('some_job_time', 20, 0.05) # reporting job that took 20ms with sampling (5% sampling)
+
+
+Inspiration
+-----------
+
+StatsD was inspired (heavily) by the project (of the same name) at Flickr. Here's a post where Cal Henderson described it in depth:
+[Counting and timing](http://code.flickr.com/blog/2008/10/27/counting-timing/). Cal re-released the code recently: [Perl StatsD](https://github.com/iamcal/Flickr-StatsD)
+
+
+Contribute
+---------------------
+
+You're interested in contributing to StatsD? *AWESOME*. Here are the basic steps:
+
+fork StatsD from here: http://github.com/etsy/statsd
+
+1. Clone your fork
+2. Hack away
+3. If you are adding new functionality, document it in the README
+4. If necessary, rebase your commits into logical chunks, without errors
+5. Push the branch up to GitHub
+6. Send a pull request to the etsy/statsd project.
+
+We'll do our best to get your changes in!
+
+[graphite]: http://graphite.wikidot.com
+[etsy]: http://www.etsy.com
+[blog post]: http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
+[node]: http://nodejs.org
+[udp]: http://enwp.org/udp
+[eventmachine]: http://rubyeventmachine.com/
+[mongodb]: http://www.mongodb.org/
+
+Contributors
+-----------------
+
+In lieu of a list of contributors, check out the commit history for the project: 
+http://github.com/etsy/statsd/commits/master

data/config.js

@@ -0,0 +1,39 @@
+var fs  = require('fs')
+  , sys = require('sys')
+
+var Configurator = function (file) {
+
+  var self = this;
+  var config = {};
+  var oldConfig = {};
+
+  this.updateConfig = function () {
+    sys.log('reading config file: ' + file);
+
+    fs.readFile(file, function (err, data) {
+      if (err) { throw err; }
+      old_config = self.config;
+
+      self.config = process.compile('config = ' + data, file);
+      self.emit('configChanged', self.config);
+    });
+  };
+
+  this.updateConfig();
+
+  fs.watchFile(file, function (curr, prev) {
+    if (curr.ino != prev.ino) { self.updateConfig(); }
+  });
+};
+
+sys.inherits(Configurator, process.EventEmitter);
+
+exports.Configurator = Configurator;
+
+exports.configFile = function(file, callbackFunc) {
+  var config = new Configurator(file);
+  config.on('configChanged', function() {
+    callbackFunc(config.config, config.oldConfig);
+  });
+};
+

data/config.yml

@@ -0,0 +1,34 @@
+---
+# Flush interval should be your finest retention in seconds
+flush_interval: 10        
+
+# Graphite
+graphite_host: localhost
+graphite_port: 8125
+
+# Mongo
+mongo_host: statsd.example.com
+mongo_database:     statsdb
+
+# If you change these, you need to delete the capped collections yourself!
+# Average mongo record size is 152 bytes
+# 10s or 1min data is transient so we'll use MongoDB's capped collections. These collections are fixed in size.
+# 5min and 1d data is interesting to preserve long-term. These collections are not capped.
+retentions: 
+    - name: stats_per_10s
+      seconds: 10
+      capped: true
+      cap_bytes: 268_435_456 # 2**28
+    - name: stats_per_1min
+      seconds: 60
+      capped: true
+      cap_bytes: 1_073_741_824 # 2**30
+    - name: stats_per_5min
+      seconds: 600
+      cap_bytes: 0 
+      capped: false
+    - name: stats_per_day
+      seconds: 86400
+      cap_bytes: 0 
+      capped: false
+

data/em-server.rb

@@ -0,0 +1,51 @@
+require 'eventmachine'
+require 'statsd'
+require 'statsd/server'
+require 'statsd/mongo'
+require 'statsd/graphite'
+
+require 'yaml'
+require 'erb'
+ROOT = File.expand_path(File.dirname(__FILE__))
+APP_CONFIG = YAML::load(ERB.new(IO.read(File.join(ROOT,'config.yml'))).result)
+
+# Setup retention store
+db = Mongo::Connection.new(APP_CONFIG['mongo_host']).db(APP_CONFIG['mongo_database'])
+APP_CONFIG['retentions'].each do |retention|
+  collection_name = retention['name']
+  unless db.collection_names.include?(collection_name)
+    db.create_collection(collection_name, :capped => retention['capped'], :size => retention['cap_bytes']) 
+  end
+  db.collection(collection_name).ensure_index([['ts', Mongo::ASCENDING]])
+end
+
+# Start the server
+Statsd::Mongo.hostname = APP_CONFIG['mongo_host']
+Statsd::Mongo.database = APP_CONFIG['mongo_database']
+Statsd::Mongo.retentions = APP_CONFIG['retentions']
+Statsd::Mongo.flush_interval = APP_CONFIG['flush_interval']
+EventMachine::run do
+  EventMachine::open_datagram_socket('127.0.0.1', 8125, Statsd::Server)  
+  EventMachine::add_periodic_timer(APP_CONFIG['flush_interval']) do
+     counters,timers = Statsd::Server.get_and_clear_stats!
+     
+     #
+     # Flush Adapters
+     #
+     # Mongo
+     # EM.defer do 
+     #   Statsd::Mongo.flush_stats(counters,timers)
+     # end
+     #
+     
+     # Graphite
+     EventMachine.connect APP_CONFIG['graphite_host'], APP_CONFIG['graphite_port'], Statsd::Graphite do |conn|
+       conn.counters = counters
+       conn.timers = timers
+       conn.flush_interval = 10
+       conn.flush_stats
+     end     
+  end
+  
+  
+end

data/exampleConfig.js

@@ -0,0 +1,8 @@
+{
+  debug:true
+, dumpMessages:true
+, graphitePort: 2003
+, graphiteHost: "graphite.host.com"
+, port: 8125
+}
+

data/lib/statsd/echos.rb

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+#
+
+require 'rubygems'
+require 'eventmachine'
+
+module EchoServer
+  def post_init
+    puts "-- someone connected to the server!"
+  end
+
+  def receive_data data
+    puts data
+    send_data ">>> you sent: #{data}"
+  end
+end
+
+EventMachine::run {
+  EventMachine::start_server "127.0.0.1", 8125, EchoServer
+  puts 'running dummy graphite echo server on 8125'
+}

data/lib/statsd/graphite.rb

@@ -0,0 +1,91 @@
+require 'benchmark'
+require 'eventmachine'
+module Statsd
+  class Graphite < EM::Connection
+    attr_accessor :counters, :timers, :flush_interval
+    
+    def initialize(*args)
+      puts args
+      super
+      # stuff here...
+    end
+    
+    def post_init
+      # puts counters.size
+      # send_data 'Hello'
+      # puts 'hello'
+      # close_connection_after_writing
+    end
+
+    def receive_data(data)
+      p data      
+    end
+
+    # def unbind
+    #   p ' connection totally closed'
+    #   EventMachine::stop_event_loop
+    # end
+          
+    def flush_stats
+      print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to Graphite"
+      stat_string = ''
+      time = ::Benchmark.realtime do
+        ts = Time.now.to_i
+        num_stats = 0        
+        
+        # store counters
+        counters.each_pair do |key,value|
+          value /= flush_interval
+          message = "stats.#{key} #{value} #{ts}\n"
+          stat_string += message
+          counters[key] = 0
+
+          num_stats += 1
+        end
+    
+        # store timers
+        timers.each_pair do |key, values|
+          if (values.length > 0) 
+            pct_threshold = 90
+            values.sort!
+            count = values.count
+            min = values.first
+            max = values.last
+
+            mean = min
+            max_at_threshold = max
+
+            if (count > 1)
+              # strip off the top 100-threshold
+              threshold_index = (((100 - pct_threshold) / 100.0) * count).round
+              values = values[0..-threshold_index]
+              max_at_threshold = values.last
+
+              # average the remaining timings
+              sum = values.inject( 0 ) { |s,x| s+x }
+              mean = sum / values.count
+            end
+
+            message = ""
+            message += "stats.timers.#{key}.mean #{mean} #{ts}\n"
+            message += "stats.timers.#{key}.upper #{max} #{ts}\n"
+            message += "stats.timers.#{key}.upper_#{pct_threshold} #{max_at_threshold} #{ts}\n"
+            message += "stats.timers.#{key}.lower #{min} #{ts}\n"
+            message += "stats.timers.#{key}.count #{count} #{ts}\n"
+            stat_string += message
+
+            timers[key] = []
+                  
+            num_stats += 1
+          end
+        end
+        stat_string += "statsd.numStats #{num_stats} #{ts}\n"
+        
+      end 
+      # send to graphite
+      send_data stat_string
+      puts "complete. (#{time.round(3)}s)"
+      close_connection_after_writing
+    end
+  end
+end

data/lib/statsd/mongo.rb

@@ -0,0 +1,146 @@
+require 'benchmark'
+require 'mongo'
+module Statsd
+  class Mongo
+    class << self
+      attr_accessor :database, :hostname, :retentions, :flush_interval
+    end
+
+    def self.flush_stats(counters, timers)
+      raise 'Invalid retention config' if retentions.empty?
+      print "#{Time.now} Flushing #{counters.count} counters and #{timers.count} timers to MongoDB"
+      stat_string = ''
+      time = ::Benchmark.realtime do
+        docs = []
+        ts = Time.now.to_i
+        num_stats = 0        
+        retention = retentions.first # always write at the fineset granularity        
+        ts_bucket = ts / retention['seconds'].to_i * retention['seconds'].to_i
+        
+        # connect to store
+        db = ::Mongo::Connection.new(hostname).db(database)
+        coll = db.collection(retention['name'])
+
+        # store counters
+        counters.each_pair do |key,value|
+          value /= flush_interval
+          doc = {:stat => key, :value => value, :ts => ts_bucket, :type => "counter" }
+          docs.push(doc)
+          message = "stats.#{key} #{value} #{ts}\n"
+          stat_string += message
+          counters[key] = 0
+
+          num_stats += 1
+        end
+    
+        # store timers
+        timers.each_pair do |key, values|
+          if (values.length > 0) 
+            pct_threshold = 90
+            values.sort!
+            count = values.count
+            min = values.first
+            max = values.last
+
+            mean = min
+            max_at_threshold = max
+
+            if (count > 1)
+              # strip off the top 100-threshold
+              threshold_index = (((100 - pct_threshold) / 100.0) * count).round
+              values = values[0..-threshold_index]
+              max_at_threshold = values.last
+
+              # average the remaining timings
+              sum = values.inject( 0 ) { |s,x| s+x }
+              mean = sum / values.count
+            end
+
+            timers[key] = []
+          
+            # Flush Values to Store
+            doc = { :stat => key, 
+              :values => {
+                :mean => mean,
+                :max => max,
+                :min => min,
+                "upper_#{pct_threshold}".to_sym => max_at_threshold,
+                :count => count
+              },
+              :type => "timer",
+              :ts => ts_bucket
+            }
+            docs.push(doc)
+          
+            num_stats += 1
+          end
+        end
+        stat_string += "statsd.numStats #{num_stats} #{ts}\n"
+        coll.insert(docs)
+        
+       aggregate(ts_bucket)
+      end 
+      puts "complete. (#{time.round(3)}s)"
+    end
+
+    # For each coarse granularity of retention
+    #   Look up the previous bucket 
+    #     If there is no data, aggregate the finest Fill it if empty 
+    # TODO consider doing this inside Mongo with M/R
+    def self.aggregate(current_bucket)
+      db = ::Mongo::Connection.new(hostname).db(database)
+      retentions.sort_by! {|r| r['seconds']}
+      docs = []
+      fine_stats_collection = db.collection(retentions.first['name']) # Use the finest granularity for now
+      retentions[1..-1].each_with_index do |retention,index|
+        # fine_stats_collection = db.collection(retentions[index]['name'])
+        coarse_stats_collection = db.collection(retention['name'])
+        step = retention['seconds']
+        current_coarse_bucket = current_bucket / step * step - step
+        previous_coarse_bucket = current_coarse_bucket - step
+        # Look up previous bucket
+        if coarse_stats_collection.find({:ts => previous_coarse_bucket}).count == 0
+          # Aggregate
+          print '.'
+          stats_to_aggregate = fine_stats_collection.find(
+            {:ts => {"$gte" => previous_coarse_bucket, "$lt" => current_coarse_bucket}})
+          rows = stats_to_aggregate.to_a
+          count = stats_to_aggregate.count
+          rows.group_by {|r| r["stat"] }.each_pair do |name,stats|
+            case stats.first['type']
+            when 'timer' 
+              mean = stats.collect {|stat| stat['values']['mean'] }.inject( 0 ) { |s,x| s+x } / stats.count
+              max  = stats.collect {|stat| stat['values']['max'] }.max
+              min  = stats.collect {|stat| stat['values']['max'] }.min
+              upper_key = stats.first['values'].keys.find{|k| k =~ /upper_/}
+              max_at_threshold = stats.collect {|stat| stat['values'][upper_key] }.max
+              total_stats = stats.collect {|stat| stat['values']['count'] }.inject( 0 ) { |s,x| s+x }            
+              doc = { :stat => name, 
+                :values => {
+                  :mean => mean,
+                  :max => max,
+                  :min => min,
+                  upper_key.to_sym => max_at_threshold,
+                  :count => total_stats
+                },
+                :type => "timer",
+                :ts => previous_coarse_bucket
+              }
+            when 'counter'  
+              doc = {:stat => name, 
+                :value => stats.collect {|stat| stat['value'] }.inject( 0 ) { |s,x| s+x }, 
+                :ts => previous_coarse_bucket, 
+                :type => "counter"
+              }
+            else
+              raise "unknown type #{stats.first['type']}"
+            end
+            docs.push(doc)
+          end
+          coarse_stats_collection.insert(docs)          
+        end
+      end
+      
+    end
+  end
+end

data/lib/statsd/server.rb

@@ -0,0 +1,41 @@
+require 'eventmachine'
+module Statsd
+  module Server #< EM::Connection  
+    Version = '0.0.4'  
+    
+    FLUSH_INTERVAL = 10
+    COUNTERS = {}
+    TIMERS = {}
+    def post_init
+      puts "statsd server started!"
+    end
+    def self.get_and_clear_stats!
+      counters = COUNTERS.dup
+      timers = TIMERS.dup
+      COUNTERS.clear
+      TIMERS.clear
+      [counters,timers]
+    end
+    def receive_data(msg)    
+      msg.split("\n").each do |row|
+        # puts row
+        bits = row.split(':')
+        key = bits.shift.gsub(/\s+/, '_').gsub(/\//, '-').gsub(/[^a-zA-Z_\-0-9\.]/, '')
+        bits.each do |record|
+          sample_rate = 1
+          fields = record.split("|")    
+          if (fields[1].strip == "ms") 
+            TIMERS[key] ||= []
+            TIMERS[key].push(fields[0].to_i)
+          else
+            if (fields[2] && fields[2].match(/^@([\d\.]+)/)) 
+              sample_rate = fields[2].match(/^@([\d\.]+)/)[1]
+            end
+            COUNTERS[key] ||= 0
+            COUNTERS[key] += (fields[0].to_i || 1) * (1.0 / sample_rate.to_f)
+          end
+        end
+      end
+    end    
+  end 
+end

data/lib/statsd.rb

@@ -0,0 +1,75 @@
+# encoding: utf-8
+module Statsd
+
+  #
+  # Statsd::Client by Ben VandenBos 
+  # http://github.com/bvandenbos/statsd-client
+  # 
+  class Client
+
+    Version = '0.0.4'      
+    attr_accessor :host, :port
+  
+    def initialize(host='localhost', port=8125)
+      @host = host
+      @port = port
+    end
+
+    # +stat+ to log timing for
+    # +time+ is the time to log in ms
+    def timing(stat, time, sample_rate = 1)
+      send_stats "#{stat}:#{time}|ms", sample_rate
+    end
+  
+    # +stats+ can be a string or an array of strings
+    def increment(stats, sample_rate = 1)
+      update_counter stats, 1, sample_rate
+    end
+  
+    # +stats+ can be a string or an array of strings
+    def decrement(stats, sample_rate = 1)
+      update_counter stats, -1, sample_rate
+    end
+  
+    # +stats+ can be a string or array of strings
+    def update_counter(stats, delta = 1, sample_rate = 1)
+      stats = Array(stats)
+      send_stats(stats.map { |s| "#{s}:#{delta}|c" }, sample_rate)
+    end
+  
+    private
+  
+    def send_stats(data, sample_rate = 1)
+      data = Array(data)
+      sampled_data = []
+    
+      # Apply sample rate if less than one
+      if sample_rate < 1
+        data.each do |d|
+          if rand <= sample_rate
+            sampled_data << "#{d}@#{sample_rate}"
+          end
+        end
+        data = sampled_data
+      end
+    
+      return if data.empty?
+    
+      raise "host and port must be set" unless host && port
+    
+      begin
+        sock = UDPSocket.new
+        data.each do |d|
+          sock.send(d, 0, host, port)
+        end
+      rescue # silent but deadly
+      ensure
+        sock.close
+      end
+      true
+    end
+    
+  
+  end
+end 
+

data/netcat-example.sh

@@ -0,0 +1,5 @@
+nc -w 1 -u 127.0.0.1 8125 << EOF
+globs:1|c
+gorets:1|c|@0.1
+glork:320|ms
+EOF