librato-rails 0.6.0 → 0.7.0

data/README.md

@@ -7,9 +7,7 @@ Report key statistics for your Rails app to [Librato Metrics](https://metrics.li
 
 **NOTE: This is currently in alpha development and is not yet officially supported**
 
-**NOTES FOR ALPHA TESTERS:**
- * If you are upgrading from a version prior to the rename to librato-rails, note that *the env variable names for configuration and the name of the config files have changed*. See the new names in configuration, below.
- * Starting with 0.4.0 *all metrics are now submitted as gauges*. If you were using a prior version you will need to manually remove any librato-rails generated metrics which are counters.
+You may want to read the [notes on upgrading](https://github.com/librato/librato-rails/wiki/Alpha-Tester-Upgrade-Notes) if you are an alpha tester.
 
 ## Installation
 
@@ -23,25 +21,45 @@ Then run `bundle install`.
 
 If you don't have a Metrics account already, [sign up](https://metrics.librato.com/). In order to send measurements to Metrics you need to provide your account credentials to `librato-rails`. You can provide these one of two ways:
 
+##### Use a config file
+
 Create a `config/librato.yml` like the following:
 
     production:
       user: <your-email>
       token: <your-api-key>
 
-(the file is parsed via ERB in case you need to add some magic in there - useful in some cloud environments)
+The `librato.yml` file is parsed via ERB in case you need to add some host or environment-specific magic. 
+
+Note that using a configuration file allows you to specify different configurations per-environment. Submission will be disabled in any environment without credentials. 
+
+##### Use environment variables
+
+Alternately you can provide `LIBRATO_METRICS_USER` and `LIBRATO_METRICS_TOKEN` environment variables. Unlike config file settings, environment variables will be used in all non-test environments (development, production, etc).  
+
+Note that if a config file is present, _all environment variables will be ignored_.
 
-OR provide `LIBRATO_METRICS_USER` and `LIBRATO_METRICS_TOKEN` environment variables. If both env variables and a config file are present, environment variables will take precendence.
+For more information on combining config files and environment variables, see the [full configuration docs](https://github.com/librato/librato-rails/wiki/Configuration).
 
-Note that using a configuration file allows you to specify configurations per-environment. Submission will be disabled in any environment without credentials. However, if environment variables are set they will be used in all environments. 
+##### Running on Heroku
+
+If you are using the Librato Metrics Heroku addon, your user and token environment variables will already be set in your Heroku environment. If you are running without the addon you will need to provide them yourself.
+
+In either case you will need to specify a custom source for your app to track properly. If `librato-rails` does not detect an explicit source it will not start. You can set the source in your environment:
+
+    heroku config:add LIBRATO_METRICS_SOURCE=myappname
+    
+If you are using a config file, add your source entry to that instead.
 
 Full information on configuration options is available on the [configuration wiki page](https://github.com/librato/librato-rails/wiki/Configuration).
 
+Note that if Heroku idles your application measurements will not be sent until it receives another request and is restarted. If you see intermittent gaps in your measurements during periods of low traffic this is the most likely cause.
+
 ## Automatic Measurements
 
-After installing `librato-rails` and restarting your app and you will see a number of new metrics appear in your Metrics account. These track request performance, sql queries, mail handling, and other key stats. All built-in performance metrics start with the prefix `rails` by convention &mdash; for example: `rails.request.total` is the total number of requests received during an interval. 
+After installing `librato-rails` and restarting your app and you will see a number of new metrics appear in your Metrics account. These track request performance, sql queries, mail handling, and other key stats. 
 
-If you have multiple apps reporting to the same Metrics account you can change this prefix in your [configuration](https://github.com/librato/librato-rails/wiki/Configuration).
+Built-in performance metrics will start with either `rack` or `rails`, depending on the level they are being sampled from. For example: `rails.request.total` is the total number of requests rails has received each minute. 
 
 ## Custom Measurements
 
@@ -111,7 +129,11 @@ Can also be written as:
 
 Symbols can be used interchangably with strings for metric names.
 
-## Cross-process Aggregation
+## Custom Prefix
+
+You can set an optional prefix to all metrics reported by `librato-rails` in your [configuration](https://github.com/librato/librato-rails/wiki/Configuration). This can be helpful for isolating test data or forcing different apps to use different metric names.
+
+## Cross-Process Aggregation
 
 `librato-rails` submits measurements back to the Librato platform on a _per-process_ basis. By default these measurements are then combined into a single measurement per source (default is your hostname) before persisting the data. 
 
@@ -121,6 +143,16 @@ Current pricing applies after aggregation, so in this case you will be charged f
 If you want to report per-process instead, you can set `source_pids` to `true` in
 your config, which will append the process id to the source name used by each thread. 
 
+## Troubleshooting
+
+Note that it may take 2-3 minutes for the first results to show up in your Metrics account after you have started your servers with `librato-rails` enabled and the first request has been received. 
+
+If you want to get more information about `librato-rails` submissions to the Metrics service you can set your `log_level` to `debug` (see [configuration](https://github.com/librato/librato-rails/wiki/Configuration)) to get detailed information added to your logs about the settings `librato-rails` is seeing at startup and when it is submitting.
+
+If you are having an issue with a specific metric, using a `log_level` of `trace` will add the exact measurements being sent to your logs along with lots of other information about `librato-rails` as it executes. Neither of these modes are recommended long-term in production as they will add quite a bit of volume to your log file and will slow operation somewhat. Note that submission I/O is non-blocking, submission times are total time - your process will continue to handle requests during submissions.
+
+If you are debugging setting up `librato-rails` locally you can set `flush_interval` to something shorter (say 10s) to force submission more frequently. Don't change your `flush_interval` in production as it will not result in measurements showing up more quickly, but may affect performance.
+
 ## Contribution
 
 * Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.

data/lib/librato-rails.rb

@@ -1 +1 @@
-require 'librato/rails'
+require 'librato/rails'

data/lib/librato/rails.rb

@@ -19,8 +19,9 @@ module Librato
 
   module Rails
     extend SingleForwardable
-    CONFIG_SETTABLE = %w{user token flush_interval prefix source source_pids}
+    CONFIG_SETTABLE = %w{user token flush_interval log_level prefix source source_pids}
     FORKING_SERVERS = [:unicorn, :passenger]
+    LOG_LEVELS = [:off, :error, :warn, :info, :debug, :trace]
 
     mattr_accessor :config_file
     self.config_file = 'config/librato.yml'
@@ -34,9 +35,14 @@ module Librato
     # config defaults
     self.flush_interval = 60 # seconds
     self.source_pids = false # append process id to the source?
+    # log_level (default :info)
+    # source (default: your machine's hostname)
+
+    # handy introspection
+    mattr_accessor :explicit_source
 
     # a collector instance handles all measurement addition/storage
-    def_delegators :collector, :aggregate, :counters, :delete_all, :group, :increment, 
+    def_delegators :collector, :aggregate, :counters, :delete_all, :group, :increment,
                                :measure, :prefix, :prefix=, :timing
 
     class << self
@@ -49,15 +55,16 @@ module Librato
       # detect / update configuration
       def check_config
         if self.config_file && File.exists?(self.config_file)
-          logger.debug "[librato-rails] configuration file present, ignoring ENV variables"
+          log :debug, "configuration file present, ignoring ENV variables"
           env_specific = YAML.load(ERB.new(File.read(config_file)).result)[::Rails.env]
           settable = CONFIG_SETTABLE & env_specific.keys
           settable.each { |key| self.send("#{key}=", env_specific[key]) }
         else
-          logger.debug "[librato-rails] no configuration file present, using ENV variables"
-          self.token = ENV['LIBRATO_METRICS_TOKEN'] if ENV['LIBRATO_METRICS_TOKEN']
-          self.user = ENV['LIBRATO_METRICS_USER'] if ENV['LIBRATO_METRICS_USER']
-          self.source = ENV['LIBRATO_METRICS_SOURCE'] if ENV['LIBRATO_METRICS_SOURCE']
+          log :debug, "no configuration file present, using ENV variables"
+          %w{user token source log_level}.each do |settable|
+            env_var = "LIBRATO_METRICS_#{settable.upcase}"
+            send("#{settable}=", ENV[env_var]) if ENV[env_var]
+          end
         end
       end
 
@@ -75,7 +82,7 @@ module Librato
       def client
         @client ||= prepare_client
       end
-      
+
       # collector instance which is tracking all measurement additions
       def collector
         @collector ||= Collector.new
@@ -83,19 +90,45 @@ module Librato
 
       # send all current data to Metrics
       def flush
-        logger.debug "[librato-rails] flushing #{Process.pid} (#{Time.now}):"
-        queue = client.new_queue(:source => qualified_source, :prefix => self.prefix)
+        log :debug, "flushing pid #{@pid} (#{Time.now}).."
+        start = Time.now
+        queue = client.new_queue(:source => qualified_source, 
+          :prefix => self.prefix, :skip_measurement_times => true)
         # thread safety is handled internally for both stores
         counters.flush_to(queue)
         aggregate.flush_to(queue)
-        logger.debug queue.queued
+        trace_queued(queue.queued) if should_log?(:trace)
         queue.submit unless queue.empty?
+        log :trace, "flushed pid #{@pid} in #{(Time.now - start)*1000.to_f}ms"
       rescue Exception => error
-        logger.error "[librato-rails] submission failed permanently: #{error}"
+        log :error, "submission failed permanently: #{error}"
       end
 
-      def logger
-        @logger ||= ::Rails.logger
+      def log(level, message)
+        return unless should_log?(level)
+        case level
+        when :error, :warn
+          method = level
+        else
+          method = :info
+        end
+        message = '[librato-rails] ' << message
+        logger.send(method, message)
+      end
+
+      # set log level to any of LOG_LEVELS
+      def log_level=(level)
+        level = level.to_sym
+        if LOG_LEVELS.index(level)
+          @log_level = level
+          require 'pp' if should_log?(:debug)
+        else
+          raise "Invalid log level '#{level}'"
+        end
+      end
+
+      def log_level
+        @log_level ||= :info
       end
 
       # source including process pid
@@ -106,19 +139,27 @@ module Librato
       # run once during Rails startup sequence
       def setup(app)
         check_config
-        return unless credentials_present?
-        logger.info "[librato-rails] starting up with #{app_server}..."
+        trace_settings if should_log?(:debug)
+        return unless should_start?
+        if app_server == :other
+          log :info, "starting up..."
+        else
+          log :info, "starting up with #{app_server}..."
+        end
         @pid = $$
         app.middleware.use Librato::Rack::Middleware
         start_worker unless forking_server?
       end
 
       def source
-        @source ||= Socket.gethostname
+        return @source if @source
+        self.explicit_source = false
+        @source = Socket.gethostname
       end
 
       # set a custom source
       def source=(src)
+        self.explicit_source = true
         @source = src
       end
 
@@ -129,7 +170,7 @@ module Librato
       def start_worker
         return if @worker # already running
         @pid = $$
-        logger.debug "[librato-rails] >> starting up worker for pid #{@pid}..."
+        log :debug, ">> starting up worker for pid #{@pid}..."
         @worker = Thread.new do
           worker = Worker.new
           worker.run_periodically(self.flush_interval) do
@@ -151,21 +192,34 @@ module Librato
           :other
         end
       end
-      
-      def credentials_present?
-        self.user && self.token
-      end
 
       def forking_server?
         FORKING_SERVERS.include?(app_server)
       end
 
-      def install_worker_check
-        ::ApplicationController.prepend_before_filter do |c|
-          Librato::Rails.check_worker
+      # there isn't anything in the environment before the
+      # first request to know if we're running on heroku, but
+      # they set all hostnames to UUIDs.
+      def implicit_source_on_heroku?
+        !explicit_source && on_heroku
+      end
+
+      def logger
+        @logger ||= if on_heroku
+          logger = Logger.new(STDOUT)
+          logger.level = Logger::INFO
+          logger
+        else
+          ::Rails.logger
         end
       end
 
+      def on_heroku
+        # would be nice to have something more specific here,
+        # but nothing characteristic in ENV, etc.
+        @on_heroku ||= source_is_uuid?(Socket.gethostname)
+      end
+
       def prepare_client
         check_config
         client = Librato::Metrics::Client.new
@@ -180,6 +234,44 @@ module Librato
         RUBY_DESCRIPTION.split[0]
       end
 
+      def should_log?(level)
+        LOG_LEVELS.index(self.log_level) >= LOG_LEVELS.index(level)
+      end
+
+      def should_start?
+        return false if implicit_source_on_heroku?
+        self.user && self.token # are credentials present?
+      end
+
+      def source_is_uuid?(source)
+        source =~ /[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}/i
+      end
+
+      # trace current environment
+      def trace_environment
+        log :info, "Environment: " + ENV.pretty_inspect
+      end
+
+      # trace metrics being sent
+      def trace_queued(queued)
+        log :trace, "Queued: " + queued.pretty_inspect
+      end
+
+      def trace_settings
+        settings = {
+          :user => self.user,
+          :token => self.token,
+          :source => source,
+          :explicit_source => self.explicit_source ? 'true' : 'false',
+          :source_pids => self.source_pids ? 'true' : 'false',
+          :qualified_source => qualified_source,
+          :log_level => log_level,
+          :prefix => prefix,
+          :flush_interval => self.flush_interval
+        }
+        log :info, 'Settings: ' + settings.pretty_inspect
+      end
+
       def user_agent
         ua_chunks = []
         ua_chunks << "librato-rails/#{Librato::Rails::VERSION}"

data/lib/librato/rails/aggregator.rb

@@ -13,7 +13,7 @@ module Librato
       def [](key)
         fetch(key)
       end
-      
+
       def fetch(key, options={})
         return nil if @cache.empty?
         gauges = nil
@@ -61,7 +61,7 @@ module Librato
         options = {}
         event = args[0].to_s
         returned = nil
-        
+
         # handle block or specified argument
         if block_given?
           start = Time.now
@@ -72,13 +72,13 @@ module Librato
         else
           raise "no value provided"
         end
-        
+
         # detect options hash if present
         if args.length > 1 and args[-1].respond_to?(:each)
           options = args[-1]
         end
         source = options[:source]
-        
+
         @lock.synchronize do
           if source
             @cache.add event => {:source => source, :value => value}

data/lib/librato/rails/collector.rb

@@ -6,37 +6,37 @@ module Librato
   module Rails
     class Collector
       extend Forwardable
-    
+
       def_delegators :counters, :increment
       def_delegators :aggregate, :measure, :timing
-    
+
       # access to internal aggregator object
       def aggregate
         @aggregator_cache ||= Aggregator.new(:prefix => @prefix)
       end
-      
+
       # access to internal counters object
       def counters
         @counter_cache ||= CounterCache.new
       end
-      
+
       # remove any accumulated but unsent metrics
       def delete_all
         aggregate.delete_all
         counters.delete_all
       end
-      
+
       def group(prefix)
         group = Group.new(prefix)
         yield group
       end
-      
+
       # update prefix
       def prefix=(new_prefix)
         @prefix = new_prefix
         aggregate.prefix = @prefix
       end
-      
+
       def prefix
         @prefix
       end

data/lib/librato/rails/counter_cache.rb

@@ -1,19 +1,19 @@
 module Librato
   module Rails
-    
+
     class CounterCache
       DEFAULT_SOURCE = '%%'
-      
+
       extend Forwardable
-    
+
       def_delegators :@cache, :empty?
-    
+
       def initialize
         @cache = {}
         @lock = Mutex.new
         @sporadics = {}
       end
-    
+
       # Retrieve the current value for a given metric. This is a short
       # form for convenience which only retrieves metrics with no custom
       # source specified. For more options see #fetch.
@@ -21,30 +21,30 @@ module Librato
       # @param [String|Symbol] key metric name
       # @return [Integer|Float] current value
       def [](key)
-        @lock.synchronize do 
-          @cache[key.to_s][DEFAULT_SOURCE] 
+        @lock.synchronize do
+          @cache[key.to_s][DEFAULT_SOURCE]
         end
       end
-      
+
       # removes all tracked metrics. note this removes all measurement
       # data AND metric names any continuously tracked metrics will not
       # report until they get another measurement
       def delete_all
         @lock.synchronize { @cache.clear }
       end
-      
-      
+
+
       def fetch(key, options={})
         source = DEFAULT_SOURCE
         if options[:source]
           source = options[:source].to_s
         end
-        @lock.synchronize do 
+        @lock.synchronize do
           return nil unless @cache[key.to_s]
           @cache[key.to_s][source]
         end
       end
-      
+
       # transfer all measurements to queue and reset internal status
       def flush_to(queue)
         counts = nil
@@ -64,7 +64,7 @@ module Librato
           end
         end
       end
-    
+
       # Increment a given metric
       #
       # @example Increment metric 'foo' by 1
@@ -78,7 +78,7 @@ module Librato
       #
       def increment(counter, options={})
         counter = counter.to_s
-        if options.is_a?(Fixnum) 
+        if options.is_a?(Fixnum)
           # suppport legacy style
           options = {:by => options}
         end
@@ -96,14 +96,14 @@ module Librato
           @cache[counter][source] += by
         end
       end
-      
+
       private
-    
+
       def make_sporadic(metric, source)
         @sporadics[metric] ||= Set.new
         @sporadics[metric] << source
       end
-    
+
       def reset_cache
         # remove any source/metric pairs that aren't continuous
         @sporadics.each do |key, sources|
@@ -115,8 +115,8 @@ module Librato
           @cache[key].each_key { |source| @cache[key][source] = 0 }
         end
       end
-    
+
     end
-    
+
   end
 end