mouth 0.8.1 → 0.8.2

data/.gitignore

@@ -5,3 +5,4 @@ log/*.log
 log/*.pid
 Gemfile.lock
 *.gem
+Capfile

data/MIT-LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2011 Jonathan Novak - http://github.com/cypriss
+Copyright (c) 2012 Jonathan Novak - http://github.com/cypriss
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,13 +1,13 @@
 # Mouth
 
-Mouth is a Ruby daemon that collects stats via UDP and stores them in Mongo.  It comes with a modern UI that allows you to view graphs and create dashboards of these statistics.  Mouth is very similar to [StatsD](https://github.com/etsy/statsd) + [Graphite](http://graphite.wikidot.com/) + [Graphene](http://jondot.github.com/graphene/).
+Mouth is a Ruby daemon that collects metrics via UDP and stores them in Mongo.  It comes with a modern UI that allows you to view graphs and create dashboards of these statistics.  Mouth is very similar to [StatsD](https://github.com/etsy/statsd) + [Graphite](http://graphite.wikidot.com/) + [Graphene](http://jondot.github.com/graphene/).
 
+![Mouth Screenshot](http://cypriss.github.com/mouth/images/ss/mouth-screenshot1.png)
 
 ## Why
 
 Why duplicate effort of the excellent StatsD / Graphite packages?  I wanted a graphing and monitoring tool that offered:
 
-
 * **Accessible data**.  Because the metrics are stored in Mongo, they are very accessible to any scripts you want to write to read and act on your data.
 * **Easy installation**.  Mouth depends on just Ruby and Mongo, both incredibly easy to install.
 * **Modern, friendly UI**.  Mouth graphs are beautiful and easy to work with.  Exploring data and building dashboards are a joy.
@@ -17,20 +17,24 @@ Why duplicate effort of the excellent StatsD / Graphite packages?  I wanted a gr
 There are two kinds of metrics currently: counters and timers.  Both are stored in a time series with minute granularity.
 
 * **Counters**:
-  * how many occurances of something happen per minute?
+  * How many occurrences of something happen per minute?
   * Can be sampled
   * Standard usage: Mouth.increment("myapp.happenings")
   * Advanced usage: Mouth.increment("myapp.happenings", 1, 0.1) # 1/10 sample rate.  UDP packets are sent 1 in 10 times, but count for 1 * 10 each.
   * Gives you a time series of happenings by minute: {"myapp.happenings" => [1, 2, 5, 20, ...]}
 * **Timers**:
   * How long does something take?
-  * Standard usage: Mouth.measure("myapp.occurances", 3.3) # This occurance took 3.3 ms to occur
-  * Standard usage 2: Mouth.measure("myapp.occurances") { do_occurance() }
-  * Gives you a time series of occurance timings by minute: {"myapp.happenings" => [1, 2, 5, 20, ...]}
-* **Gauges**: (Coming soon)
+  * Standard usage: Mouth.measure("myapp.occurrences", 3.3) # This occurrence took 3.3 ms to occur
+  * Standard usage 2: Mouth.measure("myapp.occurrences") { do_occurance() }
+  * Gives you a time series of occurrence timings by minute: {"myapp.happenings" => [1, 2, 5, 20, ...]}
+* **Gauges**:
+  * Record the value of something at a specific time
+  * Standard usage: Mouth.gauge("myapp.subscriber_count", 3000) # At this time, you have 3000 subscribers.
 
 ## Installation on OSX
 
+Make sure you're using Ruby 1.9.2+ or compatible.
+
 Install mouth:
 
     gem install mouth
@@ -39,7 +43,7 @@ Install MongoDB if you haven't:
 
     brew install mongodb
 
-Start collector:
+Start collector daemon:
 
     mouth
 
@@ -73,21 +77,23 @@ Mouth comes with a built-in facility to instrument your apps:
     require 'mouth'
     require 'mouth/instrument'
     
-    Mouth.server = "0.0.0.0:8889"
+    Mouth.daemon_hostport = "0.0.0.0:8889"
     Mouth.increment('hello.world')
-    Mouth.measure('hello.happening', 42.9)
+    Mouth.measure('hello.happening') { happen! }
+    Mouth.gauge('hello.level', 1000)
     
 ### Using mouth-instrument
   
-mouth-instrument is a lightweight gem that doesn't have the baggage of the various gems that come with mouth. Its usage is nearly identical:
+[mouth-instrument](http://www.github.com/cypriss/mouth-instrument) is a lightweight gem that doesn't have the baggage of the various gems that come with mouth. Its usage is nearly identical:
     
     gem install mouth-instrument
 
     require 'mouth-instrument'
     
-    Mouth.server = "0.0.0.0:8889"
+    Mouth.daemon_hostport = "0.0.0.0:8889"
     Mouth.increment('hello.world')
-    Mouth.measure('hello.happening', 42.9)
+    Mouth.measure('hello.happening') { happen! }
+    Mouth.gauge('hello.level', 1000)
     
 ### Using any StatsD instrumentation
   
@@ -97,19 +103,27 @@ Mouth is StatsD compatible -- if you've instrumented your application to record
 
 You can access and act on your metrics quite easily.
 
-  require 'mouth'
-  require 'mouth/sequence'
-  
-  Mouth.host = "0.0.0.0:8889"
-  Sequence.new("exceptions.app", :kind => :counter).sequence
-  # => [4, 9, 0, ...]
-  
-  Sequence.new("app.requests", :kind => :timer, :granularity_in_minutes => 15, :start_time => Time.now - 86400, :end_time => Time.now).sequence
-  # => [{:count => 3, :min => 1, :max => 30, :mean => 17.0, :sum => 51.0, :median => 20, :stddev => 12.02}, ...]
+    require 'mouth'
+    require 'mouth/sequence_query'
+    
+    Mouth::SequenceQuery.new("exceptions.app", :kind => :counter).sequence
+    # => [4, 9, 0, ...]
+    
+    Mouth::SequenceQuery.new("app.requests", :kind => :timer, :granularity_in_minutes => 15, :start_time => Time.now - 86400, :end_time => Time.now).sequence
+    # => [{:count => 3, :min => 1, :max => 30, :mean => 17.0, :sum => 51.0, :median => 20, :stddev => 12.02}, ...]
+
+Additionally, you can insert metrics directly into the Mongo store, without sending UDP packets.  You might want to do this if you need guarantees UDP can't provide.
+
+    require 'mouth'
+    require 'mouth/recorder'
+    
+    Mouth::Recorder.increment("app.happening")
+    Mouth::Recorder.gauge("app.level", 10)
+    # Mouth::Recorder.measure("app.occurrence") { occur! } # Currently unsupported
 
 ## Tech
 
-* **Ruby** - 1.9.2+ is required.  Ruby was chosen because many Ruby shops already have it deployed as part of their infrastucture.  By putting everything in Ruby, node + python aren't needed.
+* **Ruby** - 1.9.2+ is required.  Ruby was chosen because many Ruby shops already have it deployed as part of their infrastructure.  By putting everything in Ruby, node + python aren't needed.
 * **MongoDB** -  Mouth stores metrics in Mongo.  Mongo was chosen for 3 reasons:
   * It's very easy to install and get going
   * It has drivers for everything, so getting at your data is super easy
@@ -119,12 +133,11 @@ You can access and act on your metrics quite easily.
 * **Backbone.js** - The web UI is powered by Backbone.js
 * **D3.js** - The graphs are powered by D3.js
 
-
 ## Contributing
 
-You're interested in contributing to Mouth? *AWESOME*. Here are the basic steps:
+You're interested in contributing to Mouth? *AWESOME*. Both bug reports and pull requests are welcome!
 
-fork Mouth from here: http://github.com/cypriss/mouth
+Fork Mouth from here: http://github.com/cypriss/mouth
 
 1. Clone your fork
 2. Hack away
@@ -135,4 +148,4 @@ fork Mouth from here: http://github.com/cypriss/mouth
 
 ## Thanks
 
-Thanks to UserVoice.com for sponsoring this project.  Thanks to the [StatsD](https://github.com/etsy/statsd) project for massive inspiration.  Other contributors: https://github.com/cypriss/mouth/graphs/contributors
+Thanks to [UserVoice.com](http://developer.uservoice.com) for sponsoring this project.  Thanks to the [StatsD](https://github.com/etsy/statsd) project for massive inspiration.  Other contributors: https://github.com/cypriss/mouth/graphs/contributors

data/Rakefile

@@ -7,13 +7,13 @@ rescue Bundler::BundlerError => e
   $stderr.puts "Run `bundle install` to install missing gems"
   exit e.status_code
 end
-require 'rake'
 
 require 'rake/testtask'
 Rake::TestTask.new(:test) do |test|
+  test.ruby_opts << '-rubygems'
   test.libs << 'lib' << 'test'
   test.pattern = 'test/*_test.rb'
   test.verbose = true
 end
 
-task :default => :test
+task :default => :test

data/TODO

@@ -1,7 +1,6 @@
-Now
- - Build a caching system so that ticks don't pull in old data
-
 Next
+ - Logging before/after daemonization
+ - Build a caching system so that ticks don't pull in old data
  - Cookie-ize the timespan settings
  - Re implement graph autoplacement
  - Better backend impl of source list
@@ -10,6 +9,8 @@ Next
  - Build in some mechanisms to diagnose mongo connection issues
  - Auto refreshing source list (every minute?)
  - More Tests
+   - sequence: generate sample
+   - source
  - Figure out time zones
  - replicaset connection for endoscope
 
@@ -29,7 +30,6 @@ Someday
 
 Bugs:
  - When picker is open, changing dashboards should hide picker
- - tooltip spawns multiple copies on large datasets
 
 Refactorings:
  - reconcile naming of 'sources' sometimes being array of strings and sometimes array of objects

data/bin/mouth

@@ -60,9 +60,9 @@ parser = OptionParser.new do |op|
   end
   
   # NOTE: this option can be given multiple times for a replica set
-  op.on("--mongohost HOSTPORT", "STORE SUCKINGS IN THIS MONGO") do |mongo_host|
-    options[:mongo_hosts] ||= []
-    options[:mongo_hosts] << mongo_host
+  op.on("--mongohost HOSTPORT", "STORE SUCKINGS IN THIS MONGO [eg, localhost or localhost:27017]") do |mongo_hostport|
+    options[:mongo_hostports] ||= []
+    options[:mongo_hostports] << mongo_hostport
   end
 
   op.on("-h", "--help", "I WANT MOAR HALP") do

data/bin/mouth-console

@@ -8,7 +8,7 @@ require 'irb/completion'
 $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
 
 require 'mouth'
-require 'mouth/sequence'
+require 'mouth/sequence_query'
 require 'mouth/record'
 require 'mouth/graph'
 require 'mouth/dashboard'

data/bin/mouth-endoscope

@@ -12,11 +12,8 @@ rescue LoadError
 end 
 
 Vegas::Runner.new(Mouth::Endoscope, 'mouth-endoscope') do |runner, opts, app|
-  opts.on("--mongohost HOST", "STORE SUCKINGS IN THIS MONGO HOST") do |mongo_host|
-    Mouth.mongo_host = mongo_host
-  end
-  
-  opts.on("--mongoport PORT", "STORE SUCKINGS IN THIS MONGO PORT") do |mongo_port|
-    Mouth.mongo_port = mongo_port
+  opts.on("--mongohost HOSTPORT", "READ FROM THIS MONGO  [eg, localhost or localhost:27017]") do |mongo_hostport|
+    Mouth.mongo_hostports ||= []
+    Mouth.mongo_hostports << mongo_hostport
   end
 end

data/lib/mouth.rb

@@ -4,14 +4,35 @@ require 'mouth/version'
 module Mouth
   class << self
     attr_accessor :logger
-    attr_accessor :mongo_host
-    attr_accessor :mongo_port
+    
+    # Mongo connection
+    attr_accessor :mongo
+    
+    # Info to connect to mongo
+    attr_accessor :mongo_db_name
+    attr_accessor :mongo_hostports
 
     # Returns a mongo connection (NOT an em-mongo connection)
     def mongo
       @mongo ||= begin
         require 'mongo'   # require mongo here, as opposed to the top, because we don't want mongo included in the reactor (use em-mongo for that)
-        Mongo::Connection.new(self.mongo_host || "localhost", self.mongo_port || 27017, :pool_size => 5, :pool_timeout => 20).db("mouth")
+        
+        hostports = self.mongo_hostports || [["localhost", Mongo::Connection::DEFAULT_PORT]]
+        self.mongo_hostports = hostports.collect do |hp|
+          if hp.is_a?(String)
+            host, port = hp.split(":")
+            [host, port || Mongo::Connection::DEFAULT_PORT]
+          else
+            hp
+          end
+        end
+        
+        if self.mongo_hostports.length == 1
+          hostport = self.mongo_hostports.first
+          Mongo::Connection.new(hostport[0], hostport[1], :pool_size => 5, :pool_timeout => 20).db(self.mongo_db_name || "mouth")
+        else
+          raise "repls set con not impl"
+        end
       end
     end
     
@@ -28,6 +49,10 @@ module Mouth
       "mouth_#{namespace}"
     end
     
+    def collection_for(namespace)
+      collection(mongo_collection_name(namespace))
+    end
+    
     def sanitize_namespace(key)
       key.gsub(/\s+/, '_').gsub(/\//, '-').gsub(/[^a-zA-Z_\-0-9]/, '')
     end
@@ -55,8 +80,15 @@ module Mouth
       end
       
       [sanitize_namespace(namespace), sanitize_metric(metric)]
-    end 
+    end
+    
+    def current_timestamp
+      timestamp_for(Time.now)
+    end
     
+    def timestamp_for(time)
+      time.to_i / 60
+    end
 
-  end  
+  end
 end

data/lib/mouth/endoscope.rb

@@ -2,7 +2,7 @@ require 'sinatra/base'
 require 'erb'
 require 'yajl'
 
-require 'mouth/sequence'
+require 'mouth/sequence_query'
 require 'mouth/record'
 require 'mouth/graph'
 require 'mouth/dashboard'

data/lib/mouth/endoscope/public/application.js

@@ -41,7 +41,7 @@ var Dashboard = Backbone.Model.extend({
 
 var Graph = Backbone.Model.extend({
   defaults: {
-    kind: 'counters'
+    kind: 'counter'
   },
     
   name: function() {
@@ -538,6 +538,7 @@ var GraphView = Backbone.View.extend({
     params.end_time = Math.floor(+range.endDate / 1000);
     
     $.getJSON('/graphs/' + self.model.get('id') + '/data', params, function(data) {
+      $('#' + self.graphId).empty().unbind();
       var seven = null;
       _.each(data, function(d) {
         if (!seven) {
@@ -665,7 +666,7 @@ var SourceList = Backbone.View.extend({
     
     html = [
       '<div class="head">',
-        '<h2>Choose multiple counters or one timer:</h2>',
+        '<h2>Choose multiple counters, multiple gaugues, or one timer:</h2>',
         // '<input type="text" class="idea" />',
         '<a href="#" class="cancel">X</a>',
       '</div>',
@@ -774,9 +775,14 @@ var SourceList = Backbone.View.extend({
         target.prop('checked', true);
       }
       
-      // If you check a counter, all timers are unchecked
+      // If you check a counter, all non-counters are unchecked
       if (source.kind === 'counter') {
-        el.find('input:checkbox.timer').prop('checked', false);
+        el.find('input:checkbox.timer, input:checkbox.gauge').prop('checked', false);
+      }
+      
+      // If you check a gauge, all non-gauges are unchecked
+      if (source.kind === 'gauge') {
+        el.find('input:checkbox.timer, input:checkbox.counter').prop('checked', false);
       }
     }
     

data/lib/mouth/endoscope/public/seven.js

@@ -31,6 +31,7 @@ window.Seven = function(selector, opts) {
   this.points = opts.points;
   this.endDate = dateFor(this.startDate, this.granularityInMinutes, this.points);
   this.kind = opts.kind || 'counter';
+  if (this.kind == 'gauge') this.kind = 'counter';   // Gauges and counters and graphed the same
   this.dateFormat = d3.time.format("%Y-%m-%d %H:%M:%S");
 }
 

data/lib/mouth/graph.rb

@@ -8,7 +8,7 @@ module Mouth
    #     :height => 7,
    #     :width => 20
    #   },
-   #   :kind => 'counters' || 'timer',
+   #   :kind => 'counter' || 'timer' || 'gauge',
    #   :sources => ["auth.authentications"]
    # }
   class Graph < Record
@@ -27,7 +27,7 @@ module Mouth
     def data(opts = {})
       sources = self.attributes[:sources] || []
       seq_opts = {:kind => self.attributes[:kind].to_sym}.merge(opts)
-      sequence = Sequence.new(sources, seq_opts)
+      sequence = SequenceQuery.new(sources, seq_opts)
       seqs = sequence.sequences
       seqs.map do |seq|
         {

data/lib/mouth/instrument.rb

@@ -4,20 +4,20 @@ require 'benchmark'
 module Mouth
   unless self.respond_to?(:measure)
     class << self
-      attr_accessor :host, :port, :disabled
+      attr_accessor :daemon_host, :daemon_port, :disabled
       
       # Mouth.server = 'localhost:1234'
-      def server=(conn)
-        self.host, port = conn.split(':')
-        self.port = port.to_i
+      def daemon_hostport=(hostport)
+        self.daemon_host, port = hostport.split(':')
+        self.daemon_port = port.to_i
       end
       
-      def host
-        @host || "localhost"
+      def daemon_host
+        @daemon_host || "localhost"
       end
       
-      def port
-        @port || 8889
+      def daemon_port
+        @daemon_port || 8889
       end
 
       def measure(key, milli = nil)
@@ -32,6 +32,10 @@ module Mouth
       def increment(key, delta = 1, sample_rate = nil)
         write(key, delta, :c, sample_rate)
       end
+      
+      def gauge(key, value)
+        write(key, value, :g)
+      end
 
       protected
 
@@ -49,7 +53,7 @@ module Mouth
         command = "#{k}:#{v}|#{op}"
         command << "|@#{sample_rate}" if sample_rate
         
-        socket.send(command, 0, self.host, self.port) 
+        socket.send(command, 0, self.daemon_host, self.daemon_port) 
       end
     end
   end

data/lib/mouth/record.rb

@@ -1,5 +1,6 @@
 module Mouth
-  
+
+  # Simple base class to serve as an ORM for the mongo driver, without importing a large ORM like mongomapper.
   class Record
     
     # Keys are symbols

data/lib/mouth/recorder.rb

@@ -0,0 +1,39 @@
+module Mouth
+  
+  # If you don't want to use the Mouth daemon to collect UDP packets and write digests, you can use this to write metrics directly into mongo.
+  # IMPORTANT: You can't use this + the daemon simultaneously for a given metric namespace.
+  module Recorder
+    class << self
+      
+      def increment(key, delta = 1, sample_rate = nil)
+        factor = 1.0
+        if sample_rate
+          sample_rate = 1 if sample_rate > 1
+          return if rand > sample_rate
+          factor = (sample_rate <= 0.0 || sample_rate > 1.0) ? 1.0 : 1.0 / sample_rate
+        end
+        
+        ns, metric = Mouth.parse_key(key)
+        collection = Mouth.collection_for(ns)
+        t = Mouth.current_timestamp
+        
+        
+        collection.update({"t" => t}, {"$inc" => {"c.#{metric}" => delta * factor}}, :upsert => true)
+      end
+      
+      def gauge(key, value)
+        ns, metric = Mouth.parse_key(key)
+        collection = Mouth.collection_for(ns)
+        t = Mouth.current_timestamp
+        
+        
+        collection.update({"t" => t}, {"$set" => {"g.#{metric}" => value}}, :upsert => true)
+      end
+      
+      def measure(key, milli = nil)
+        raise "Not implemented"
+      end
+      
+    end
+  end
+end