librato-rack 0.1.0

data/LICENSE

@@ -0,0 +1,24 @@
+Copyright (c) 2012. Librato, Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of the <organization> nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL LIBRATO, INC. BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
+(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,176 @@
+librato-rack
+=======
+
+[![Build Status](https://secure.travis-ci.org/librato/librato-rack.png?branch=master)](http://travis-ci.org/librato/librato-rack)
+
+*Note: librato-rack is currently in beta and is currently recommended primarily for early-adopter use*
+
+`librato-rack` provides rack middleware which will report key statistics for your rack applications to [Librato Metrics](https://metrics.librato.com/). It will also allow you to easily track your own custom metrics. Metrics are delivered asynchronously behind the scenes so they won't affect performance of your requests.
+
+Currently Ruby 1.9.2+ is required.
+
+## Quick Start
+
+Install `librato-rack` as middleware in your application:
+
+    use Librato::Rack
+
+Configuring and relaunching your application will start the reporting of performance and request metrics. You can also track custom metrics by adding simple one-liners to your code:
+
+    # keep counts of key events
+    Librato.increment 'user.signup'
+
+    # benchmark sections of code to verify production performance
+    Librato.timing 'my.complicated.work' do
+      # do work
+    end
+
+    # track averages across requests
+    Librato.measure 'user.social_graph.nodes', user.social_graph.size
+
+## Installation & Configuration
+
+Install the gem:
+
+    $ gem install librato-rack
+
+Or add to your Gemfile if using bundler:
+
+    gem "librato-rack"
+
+In your rackup file or equivalent, require and add the middleware:
+
+    require 'librato-rack'
+    use Librato::Rack
+
+If you don't have a Metrics account already, [sign up](https://metrics.librato.com/). In order to send measurements to Metrics you need to provide your account credentials to `librato-rack`. You can provide these one of two ways:
+
+##### Use environment variables
+
+By default you can use `LIBRATO_USER` and `LIBRATO_TOKEN` to pass your account data to the middleware. While these are the only required variables, there are a few more optional environment variables you may find useful.
+
+* `LIBRATO_SOURCE` - the default source to use for submitted metrics. If this is not set, hostname of the executing machine will be the default source
+* `LIBRATO_PREFIX` - a prefix which will be appended to all metric names
+* `LIBRATO_LOG_LEVEL` - see logging section for more
+
+##### Use a configuration object
+
+If you want to do more complex configuration, use your own environment variables, or control your configuration in code, you can use a configuration object:
+
+    config = Librato::Rack::Configuration.new
+    config.user = 'myuser@mysite.com'
+    config.token = 'mytoken'
+    # …more configuration
+
+    use Librato::Rack, config
+
+See the configuration class for all available options.
+
+##### Running on Heroku
+
+If you are using the Librato Metrics Heroku addon, your `LIBRATO_USER` and `LIBRATO_TOKEN` environment variables will already be set in your Heroku environment. If you are running without the addon you will need to provide them yourself.
+
+You must also specify a custom source for your app to track properly. If an explicit source is not set, `librato-rack` will not start. You can set the source in your environment:
+
+    heroku config:add LIBRATO_SOURCE=myappname
+
+NOTE: if Heroku idles your application no measurements will be sent until it receives another request and is restarted. If you see intermittent gaps in your measurements during periods of low traffic this is the most likely cause.
+
+## Custom Measurements
+
+Tracking anything that interests you is easy with Metrics. There are four primary helpers available:
+
+#### increment
+
+Use for tracking a running total of something _across_ requests, examples:
+
+    # increment the 'sales_completed' metric by one
+    Librato.increment 'sales.completed'
+
+    # increment by five
+    Librato.increment 'items.purchased', :by => 5
+
+    # increment with a custom source
+    Librato.increment 'user.purchases', :source => user.id
+
+Other things you might track this way: user signups, requests of a certain type or to a certain route, total jobs queued or processed, emails sent or received
+
+###### Sporadic Increment Reporting
+
+Note that `increment` is primarily used for tracking the rate of occurrence of some event. Given this increment metrics are _continuous by default_: after being called on a metric once they will report on every interval, reporting zeros for any interval when increment was not called on the metric.
+
+Especially with custom sources you may want the opposite behavior - reporting a measurement only during intervals where `increment` was called on the metric:
+
+    # report a value for 'user.uploaded_file' only during non-zero intervals
+    Librato.increment 'user.uploaded_file', :source => user.id, :sporadic => true
+
+#### measure
+
+Use when you want to track an average value _per_-request. Examples:
+
+    Librato.measure 'user.social_graph.nodes', 212
+
+    # report from a custom source
+    Librato.measure 'jobs.queued', 3, :source => 'worker.12'
+
+#### timing
+
+Like `Librato.measure` this is per-request, but specialized for timing information:
+
+    Librato.timing 'twitter.lookup.time', 21.2
+
+The block form auto-submits the time it took for its contents to execute as the measurement value:
+
+    Librato.timing 'twitter.lookup.time' do
+      @twitter = Twitter.lookup(user)
+    end
+
+#### group
+
+There is also a grouping helper, to make managing nested metrics easier. So this:
+
+    Librato.measure 'memcached.gets', 20
+    Librato.measure 'memcached.sets', 2
+    Librato.measure 'memcached.hits', 18
+
+Can also be written as:
+
+    Librato.group 'memcached' do |g|
+      g.measure 'gets', 20
+      g.measure 'sets', 2
+      g.measure 'hits', 18
+    end
+
+Symbols can be used interchangably with strings for metric names.
+
+## Cross-Process Aggregation
+
+`librato-rack` submits measurements back to the Librato platform on a _per-process_ basis. By default these measurements are then combined into a single measurement per source (default is your hostname) before persisting the data.
+
+For example if you have 4 hosts with 8 unicorn instances each (i.e. 32 processes total), on the Metrics site you'll find 4 data streams (1 per host) instead of 32.
+Current pricing applies after aggregation, so in this case you will be charged for 4 streams instead of 32.
+
+If you want to report per-process instead, you can set `source_pids` to `true` in
+your config, which will append the process id to the source name used by each thread.
+
+## Troubleshooting
+
+Note that it may take 2-3 minutes for the first results to show up in your Metrics account after you have started your servers with `librato-rack` enabled and the first request has been received.
+
+For more information about startup and submissions to the Metrics service you can set your `log_level` to `debug`. If you are having an issue with a specific metric, using `trace` will add the exact measurements being sent to your logs along with other details about `librato-rack` execution. Neither of these modes are recommended long-term in production as they will add significant volume to your log file and may slow operation somewhat.
+
+Submission times are total time but submission I/O is non-blocking - your process will continue to handle requests during submissions.
+
+If you are debugging setup locally you can set `flush_interval` to something shorter (say 10s) to force submission more frequently. Don't change your `flush_interval` in production as it will not result in measurements showing up more quickly, but may affect performance.
+
+## Contribution
+
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project and submit a pull request from a feature or bugfix branch.
+* Please include tests. This is important so we don't break your changes unintentionally in a future version.
+* Please don't modify the gemspec, Rakefile, version, or changelog. If you do change these files, please isolate a separate commit so we can cherry-pick around it.
+
+## Copyright
+
+Copyright (c) 2013 [Librato Inc.](http://librato.com) See LICENSE for details.

data/Rakefile

@@ -0,0 +1,25 @@
+#!/usr/bin/env rake
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+
+# console
+desc "Open an console session preloaded with this library"
+task :console do
+  sh "pry -rubygems -r ./lib/librato-rack.rb"
+end
+
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+task :default => :test

data/lib/librato-rack.rb

@@ -0,0 +1 @@
+require 'librato/rack'

data/lib/librato/collector.rb

@@ -0,0 +1,48 @@
+module Librato
+  # collects and stores measurement values over time so they can be
+  # reported periodically to the Metrics service
+  #
+  class Collector
+    extend Forwardable
+
+    def_delegators :counters, :increment
+    def_delegators :aggregate, :measure, :timing
+
+    # access to internal aggregator object
+    def aggregate
+      @aggregator_cache ||= Aggregator.new(:prefix => @prefix)
+    end
+
+    # access to internal counters object
+    def counters
+      @counter_cache ||= CounterCache.new
+    end
+
+    # remove any accumulated but unsent metrics
+    def delete_all
+      aggregate.delete_all
+      counters.delete_all
+    end
+    alias :clear :delete_all
+
+    def group(prefix)
+      group = Group.new(self, prefix)
+      yield group
+    end
+
+    # update prefix
+    def prefix=(new_prefix)
+      @prefix = new_prefix
+      aggregate.prefix = @prefix
+    end
+
+    def prefix
+      @prefix
+    end
+
+  end
+end
+
+require 'librato/collector/aggregator'
+require 'librato/collector/counter_cache'
+require 'librato/collector/group'

data/lib/librato/collector/aggregator.rb

@@ -0,0 +1,97 @@
+module Librato
+  class Collector
+    # maintains storage of timing and measurement type measurements
+    #
+    class Aggregator
+      extend Forwardable
+
+      def_delegators :@cache, :empty?, :prefix, :prefix=
+
+      def initialize(options={})
+        @cache = Librato::Metrics::Aggregator.new(:prefix => options[:prefix])
+        @lock = Mutex.new
+      end
+
+      def [](key)
+        fetch(key)
+      end
+
+      def fetch(key, options={})
+        return nil if @cache.empty?
+        gauges = nil
+        source = options[:source]
+        @lock.synchronize { gauges = @cache.queued[:gauges] }
+        gauges.each do |metric|
+          if metric[:name] == key.to_s
+            return metric if !source && !metric[:source]
+            return metric if source.to_s == metric[:source]
+          end
+        end
+        nil
+      end
+
+      def delete_all
+        @lock.synchronize { @cache.clear }
+      end
+
+      # transfer all measurements to queue and reset internal status
+      def flush_to(queue)
+        queued = nil
+        @lock.synchronize do
+          return if @cache.empty?
+          queued = @cache.queued
+          @cache.clear
+        end
+        queue.merge!(queued) if queued
+      end
+
+      # @example Simple measurement
+      #   measure 'sources_returned', sources.length
+      #
+      # @example Simple timing in milliseconds
+      #   timing 'twitter.lookup', 2.31
+      #
+      # @example Block-based timing
+      #   timing 'db.query' do
+      #     do_my_query
+      #   end
+      #
+      # @example Custom source
+      #    measure 'user.all_orders', user.order_count, :source => user.id
+      #
+      def measure(*args, &block)
+        options = {}
+        event = args[0].to_s
+        returned = nil
+
+        # handle block or specified argument
+        if block_given?
+          start = Time.now
+          returned = yield
+          value = ((Time.now - start) * 1000.0).to_i
+        elsif args[1]
+          value = args[1]
+        else
+          raise "no value provided"
+        end
+
+        # detect options hash if present
+        if args.length > 1 and args[-1].respond_to?(:each)
+          options = args[-1]
+        end
+        source = options[:source]
+
+        @lock.synchronize do
+          if source
+            @cache.add event => {:source => source, :value => value}
+          else
+            @cache.add event => value
+          end
+        end
+        returned
+      end
+      alias :timing :measure
+
+    end
+  end
+end

data/lib/librato/collector/counter_cache.rb

@@ -0,0 +1,113 @@
+module Librato
+  class Collector
+    # maintains storage of a set of incrementable, counter-like
+    # measurements
+    #
+    class CounterCache
+      SEPARATOR = '%%'
+
+      extend Forwardable
+
+      def_delegators :@cache, :empty?
+
+      def initialize
+        @cache = {}
+        @lock = Mutex.new
+        @sporadics = Set.new
+      end
+
+      # Retrieve the current value for a given metric. This is a short
+      # form for convenience which only retrieves metrics with no custom
+      # source specified. For more options see #fetch.
+      #
+      # @param [String|Symbol] key metric name
+      # @return [Integer|Float] current value
+      def [](key)
+        fetch(key)
+      end
+
+      # removes all tracked metrics. note this removes all measurement
+      # data AND metric names any continuously tracked metrics will not
+      # report until they get another measurement
+      def delete_all
+        @lock.synchronize { @cache.clear }
+      end
+
+
+      def fetch(key, options={})
+        if options[:source]
+          key = "#{key}#{SEPARATOR}#{options[:source]}"
+        end
+        @lock.synchronize do
+          @cache[key.to_s]
+        end
+      end
+
+      # transfer all measurements to queue and reset internal status
+      def flush_to(queue)
+        counts = nil
+        @lock.synchronize do
+          # work off of a duplicate data set so we block for
+          # as little time as possible
+          counts = @cache.dup
+          reset_cache
+        end
+        counts.each do |metric, value|
+          metric, source = metric.split(SEPARATOR)
+          if source
+            queue.add metric => {:value => value, :source => source}
+          else
+            queue.add metric => value
+          end
+        end
+      end
+
+      # Increment a given metric
+      #
+      # @example Increment metric 'foo' by 1
+      #   increment :foo
+      #
+      # @example Increment metric 'bar' by 2
+      #   increment :bar, :by => 2
+      #
+      # @example Increment metric 'foo' by 1 with a custom source
+      #   increment :foo, :source => user.id
+      #
+      def increment(counter, options={})
+        if options.is_a?(Fixnum)
+          # suppport legacy style
+          options = {:by => options}
+        end
+        by = options[:by] || 1
+        if options[:source]
+          metric = "#{counter}#{SEPARATOR}#{options[:source]}"
+        else
+          metric = counter.to_s
+        end
+        if options[:sporadic]
+          make_sporadic(metric)
+        end
+        @lock.synchronize do
+          @cache[metric] ||= 0
+          @cache[metric] += by
+        end
+      end
+
+      private
+
+      def make_sporadic(metric)
+        @sporadics << metric
+      end
+
+      def reset_cache
+        # remove any source/metric pairs that aren't continuous
+        @sporadics.each { |metric| @cache.delete(metric) }
+        @sporadics.clear
+        # reset all continuous source/metric pairs to 0
+        @cache.each_key { |key| @cache[key] = 0 }
+      end
+
+    end
+
+  end
+end