graphite-metric 0.1.0 → 0.2.0

data/README.md

@@ -2,96 +2,143 @@ If you're sending data to Graphite from Ruby, GraphiteMetric will help you get
 it in the right format. For now, it only supports the plaintext protocol, but
 pickle should be relatively easy to add. Contributions are welcome.
 
-At [GoSquared](http://www.gosquared.com/), we're using graphite extensively.
-Some services use it directly via TCP, others via UDP (direct or via
-aggregators such as statsd). When metrics are critical, we push them into our
-RabbitMQ cluster. Carbon agents support AMQP natively, thus an efficient and
-fault-tolerant setup is achievable with little effort.
+If you're loading data from graphite-web, you can use the raw protocol
+and apply a few transformations before consuming it.
+
+At [GoSquared] [1], we're using graphite extensively.  Some services use
+it directly via TCP, others via UDP (direct or via aggregators such as
+statsd). When metrics delivery is critical, we push them into our
+RabbitMQ cluster. Carbon agent supports AMQP natively.
 
 ## USAGE
 
-What is a `GraphiteMetric::Plaintext`?
+### GraphiteMetric::Plaintext
+
+#### From values
+
+```ruby
+graphite_metric = GraphiteMetric::Plaintext.new("visitors", 2)
+
+graphite_metric.key
+=> "visitors"
+
+graphite_metric.value
+=> 2
+
+graphite_metric.timestamp
+=> 1331043095
+
+graphite_metric.to_s
+=> "visitors 2 1331043095"
+
+"#{graphite_metric}"
+=> "visitors 2 1331043095"
+```
+
+#### From hash
+
+```ruby
+gmp = GraphiteMetric::Plaintext.from_hash(
+  :key       => "visitors",
+  :value     => 2,
+  :timestamp => Time.now.to_i
+)
 
-    GraphiteMetric::Plaintext.ancestors
-    => [GraphiteMetric::Plaintext,
-     Struct,
-     Enumerable,
-     Object,
-     TestHelpers,
-     MiniTest::Expectations,
-     PP::ObjectMixin,
-     Kernel,
-     BasicObject]
+"#{gmp}"
+=> "visitors 2 1331039853"
+```
 
-### From values
+#### From array (of hashes)
 
-    graphite_metric = GraphiteMetric::Plaintext.new("visitors", 2)
-    => #<struct GraphiteMetric::Plaintext
-     key="visitors",
-     value=2,
-     timestamp=1331043095>
+```ruby
+gmps = GraphiteMetric::Plaintext.from_array([
+  {
+    :key       => "visitors",
+    :value     => 1,
+    :timestamp => Time.now.to_i
+  },
+  {
+    :key   => "visitors",
+    :value => 5
+  }
+]
 
-    graphite_metric.key
-    => "visitors"
-    graphite_metric.value
-    => 2
-    graphite_metric.timestamp
-    => 1331043095
+gmps.map(&:to_s)
+["visitors 1 1331039914", "visitors 5 1331043514"]
+```
 
-    graphite_metric.to_s
-    => "visitors 2 1331043095"
+### GraphiteMetric::Raw
 
-    "#{graphite_metric}"
-    => "visitors 2 1331043095"
+It supports single as well as multiple raw metrics (multiple targets).
+For simplicity's, all examples will use single metrics:
 
-`GraphiteMetric::Plaintext` is the same as `GMP`. The following examples will
-use the abbreviated form.
+```ruby
+gmr = GraphiteMetric::Raw.new("my.metric,1336559725,1336559845,60|34.999,35.10,33.0")
 
-### From hash
+gmr.metrics
+=> [{:key=>"my.metric", :timestamp=>1336559725, :value=>34.999},
+ {:key=>"my.metric", :timestamp=>1336559785, :value=>35.1},
+ {:key=>"my.metric", :timestamp=>1336559845, :value=>33.0}]
+```
 
-    gmp = GMP.from_hash(
-      :key       => "visitors",
-      :value     => 2,
-      :timestamp => Time.now.to_i
-    )
-    => #<struct GraphiteMetric::Plaintext
-     key="visitors",
-     value=2,
-     timestamp=1331039853>
+#### Round all values
 
-    "#{gmp}"
-    => "visitors 2 1331039853"
+```ruby
+gmr.round
+=> #<GraphiteMetric::Raw:0x00000104054fb0
+ @metrics=
+  [{:key=>"my.metric", :timestamp=>1336559725, :value=>35},
+   {:key=>"my.metric", :timestamp=>1336559785, :value=>35},
+   {:key=>"my.metric", :timestamp=>1336559845, :value=>33}],
+ @raw="my.metric,1336559725,1336559845,60|34.999,35.10,33.0\n">
+```
 
-### From array (of hashes)
+#### Timeshift timestamps
 
-    gmps = GMP.from_array([
-      {
-        :key       => "visitors",
-        :value     => 1,
-        :timestamp => Time.now.to_i
-      },
-      {
-        :key   => "visitors",
-        :value => 5
-      }
-    ]
-    => [#<struct GraphiteMetric::Plaintext
-      key="visitors",
-      value=1,
-      timestamp=1331039914>,
-     #<struct GraphiteMetric::Plaintext
-      key="visitors",
-      value=5,
-      timestamp=1331043514>]
+[Graphite 0.9.9 didn't support positive timeShifts] [2]. Until 0.9.10 is
+released:
 
-    gmps.map(&:to_s)
-    ["visitors 1 1331039914", "visitors 5 1331043514"]
+```ruby
+gmr.timeshift(3600)
+=> #<GraphiteMetric::Raw:0x00000104054fb0
+ @metrics=
+  [{:key=>"my.metric", :timestamp=>1336563325, :value=>35},
+   {:key=>"my.metric", :timestamp=>1336563385, :value=>35},
+   {:key=>"my.metric", :timestamp=>1336563445, :value=>33}],
+ @raw="my.metric,1336559725,1336559845,60|34.999,35.10,33.0\n">
+```
 
+#### Group metrics by key
 
+```ruby
+gmr.grouped_metrics
+=> {"my.metric"=>
+  [{:timestamp=>1336563325, :value=>35},
+   {:timestamp=>1336563385, :value=>35},
+   {:timestamp=>1336563445, :value=>33}]}
+```
 
-## COMPATIBILITY
+#### Reset metrics
 
-Tested against Ruby 1.9.2. Used in production on both 1.9.2 & 1.9.3.
+Both timeshifting and rounding modify the internal @metrics. If you need
+to reset them back to their original state:
+
+```ruby
+gmr.populate_from_raw
+=> #<GraphiteMetric::Raw:0x00000101fc0ab0
+ @metrics=
+  [{:key=>"my.metric", :timestamp=>1336559725, :value=>34.999},
+   {:key=>"my.metric", :timestamp=>1336559785, :value=>35.1},
+   {:key=>"my.metric", :timestamp=>1336559845, :value=>33.0}],
+ @raw="my.metric,1336559725,1336559845,60|34.999,35.10,33.0\n">
+```
+
+
+
+## Ruby 1.8
+
+All our development and production systems run 1.9. We have no intention
+of making this 1.8 compatible.
 
 
 
@@ -118,3 +165,6 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
+
+[1]: http://www.gosquared.com/
+[2]: https://bugs.launchpad.net/graphite/+bug/903675

data/graphite-metric.gemspec

@@ -8,8 +8,8 @@ Gem::Specification.new do |s|
   s.authors     = ["Gerhard Lazu"]
   s.email       = ["gerhard@lazu.co.uk"]
   s.homepage    = ""
-  s.summary     = %q{Generates strings that graphite understands}
-  s.description = %q{Converts hashes and arrays into graphite plaintext format}
+  s.summary     = %q{Generates strings that graphite understands. Loads raw graphite data.}
+  s.description = %q{Converts hashes and arrays into graphite plaintext format. Creates hashes from graphite raw format.}
 
   #s.rubyforge_project = "graphite-metric"
 
@@ -18,8 +18,8 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_development_dependency 'guard-minitest'
-  s.add_development_dependency 'minitest'
-  s.add_development_dependency 'pry'
-  s.add_development_dependency 'turn'
+  s.add_development_dependency 'guard-minitest', '~> 0.5'
+  s.add_development_dependency 'minitest', '~> 3.0'
+  s.add_development_dependency 'pry', '~> 0.9'
+  s.add_development_dependency 'turn', '~> 0.9'
 end

data/lib/graphite-metric/raw.rb

@@ -0,0 +1,53 @@
+module GraphiteMetric
+  class Raw
+    attr_reader :raw, :metrics
+
+    def initialize(raw)
+      @raw = raw
+      populate_from_raw
+    end
+
+    def timeshift(offset)
+      @metrics.each { |metric| metric[:timestamp] += offset }
+      self
+    end
+
+    def round
+      @metrics.each { |metric| metric[:value] = metric[:value].round }
+      self
+    end
+
+    # Cannot be chained unlike the previous methods because the
+    # @metrics structure gets changed. Not memoizing on purpose, the
+    # other methods might modify @metrics.
+    def grouped_metrics
+      @metrics.inject({}) do |result, metric|
+        (result[metric[:key]] ||= []) << {
+          :timestamp => metric[:timestamp],
+          :value     => metric[:value]
+        }
+        result
+      end
+    end
+
+    def populate_from_raw
+      @metrics = []
+      @raw.split("\n").each do |raw|
+        raw_headers, raw_metrics = raw.split("|")
+
+        metric_name, from, to, step  = raw_headers.split(",")
+        
+        current_step = 0
+        raw_metrics.split(",").each do |raw_metric|
+          @metrics << {
+            :key       => metric_name,
+            :timestamp => from.to_i + current_step,
+            :value     => Float(raw_metric)
+          }
+          current_step += step.to_i
+        end
+      end
+      self
+    end
+  end
+end

data/lib/graphite-metric/version.rb

@@ -1,3 +1,3 @@
 module GraphiteMetric
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/lib/graphite-metric.rb

@@ -1,7 +1,6 @@
 require 'graphite-metric/version'
 require 'graphite-metric/plaintext'
+require 'graphite-metric/raw'
 
 module GraphiteMetric
 end
-
-GMP = GraphiteMetric::Plaintext

data/test/fixtures/multiple_metrics.txt

@@ -0,0 +1,2 @@
+my.first.metric,1336559725,1336559845,60|10.0,9.21,8.333
+my.second.metric,1336559725,1336559845,60|1.1,2.999,3.50

data/test/fixtures/single_metric.txt

@@ -0,0 +1 @@
+my.metric,1336559725,1336559845,60|34.999,35.10,33.0

data/test/graphite-metric/raw_test.rb

@@ -0,0 +1,38 @@
+require_relative '../test_helper'
+require 'graphite-metric/raw'
+
+include TestHelpers
+
+module GraphiteMetric
+  describe Raw do
+    it "converts raw graphite single metric to hashes" do
+      gmr = GraphiteMetric::Raw.new(single_metric_raw)
+      gmr.metrics.must_equal single_metric_converted
+    end
+
+    it "converts raw graphite multiple metrics to hashes" do
+      gmr = GraphiteMetric::Raw.new(multiple_metrics_raw)
+      gmr.metrics.must_equal multiple_metrics_converted
+    end
+
+    it "timeshifts with a positive offset" do
+      gmr = GraphiteMetric::Raw.new(single_metric_raw)
+      gmr.timeshift(3600).metrics.first[:timestamp].must_equal 1336559725 + 3600
+    end
+
+    it "timeshifts with a negatice offset" do
+      gmr = GraphiteMetric::Raw.new(single_metric_raw)
+      gmr.timeshift(-3600).metrics.first[:timestamp].must_equal 1336559725 - 3600
+    end
+
+    it "rounds values" do
+      gmr = GraphiteMetric::Raw.new(single_metric_raw)
+      gmr.round.metrics.first[:value].must_equal 35
+    end
+
+    it "groups metrics by keys" do
+      gmr = GraphiteMetric::Raw.new(multiple_metrics_raw)
+      gmr.grouped_metrics.must_equal multiple_metrics_grouped
+    end
+  end
+end

data/test/test_helper.rb

@@ -1,10 +1,106 @@
-require 'bundler'
-Bundler.setup
 require 'pry'
 require 'turn/autorun'
 
+BASE_PATH = File.expand_path('../../', __FILE__)
+
 module TestHelpers
   def utc_now
     Time.now.utc.to_i
   end
+
+  def single_metric_raw
+    @single_metric_raw ||= File.read("#{BASE_PATH}/test/fixtures/single_metric.txt")
+  end
+
+  def single_metric_converted
+    [
+      {
+        :key       => "my.metric",
+        :timestamp => 1336559725,
+        :value     => 34.999
+      },
+      {
+        :key       => "my.metric",
+        :timestamp => 1336559785,
+        :value     => 35.10
+      },
+      {
+        :key       => "my.metric",
+        :timestamp => 1336559845,
+        :value     => 33.0
+      }
+    ]
+  end
+
+  def multiple_metrics_raw
+    @multiple_metrics_raw ||= File.read("#{BASE_PATH}/test/fixtures/multiple_metrics.txt")
+  end
+
+  def multiple_metrics_converted
+    [
+      {
+        :key       => "my.first.metric",
+        :timestamp => 1336559725,
+        :value     => 10.0
+      },
+      {
+        :key       => "my.first.metric",
+        :timestamp => 1336559785,
+        :value     => 9.21
+      },
+      {
+        :key       => "my.first.metric",
+        :timestamp => 1336559845,
+        :value     => 8.333
+      },
+      {
+        :key       => "my.second.metric",
+        :timestamp => 1336559725,
+        :value     => 1.1
+      },
+      {
+        :key       => "my.second.metric",
+        :timestamp => 1336559785,
+        :value     => 2.999
+      },
+      {
+        :key       => "my.second.metric",
+        :timestamp => 1336559845,
+        :value     => 3.50
+      }
+    ]
+  end
+
+  def multiple_metrics_grouped
+    {
+      "my.first.metric" => [
+        {
+          :timestamp => 1336559725,
+          :value     => 10.0
+        },
+        {
+          :timestamp => 1336559785,
+          :value     => 9.21
+        },
+        {
+          :timestamp => 1336559845,
+          :value     => 8.333
+        }
+      ],
+      "my.second.metric" => [
+        {
+          :timestamp => 1336559725,
+          :value     => 1.1
+        },
+        {
+          :timestamp => 1336559785,
+          :value     => 2.999
+        },
+        {
+          :timestamp => 1336559845,
+          :value     => 3.50
+        }
+      ]
+    }
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: graphite-metric
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,53 +9,74 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-03-08 00:00:00.000000000Z
+date: 2012-05-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: guard-minitest
-  requirement: &2158012620 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.5'
   type: :development
   prerelease: false
-  version_requirements: *2158012620
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.5'
 - !ruby/object:Gem::Dependency
   name: minitest
-  requirement: &2158012200 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '3.0'
   type: :development
   prerelease: false
-  version_requirements: *2158012200
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: pry
-  requirement: &2158011780 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.9'
   type: :development
   prerelease: false
-  version_requirements: *2158011780
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.9'
 - !ruby/object:Gem::Dependency
   name: turn
-  requirement: &2158011360 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '0.9'
   type: :development
   prerelease: false
-  version_requirements: *2158011360
-description: Converts hashes and arrays into graphite plaintext format
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.9'
+description: Converts hashes and arrays into graphite plaintext format. Creates hashes
+  from graphite raw format.
 email:
 - gerhard@lazu.co.uk
 executables: []
@@ -71,10 +92,13 @@ files:
 - graphite-metric.gemspec
 - lib/graphite-metric.rb
 - lib/graphite-metric/plaintext.rb
+- lib/graphite-metric/raw.rb
 - lib/graphite-metric/util.rb
 - lib/graphite-metric/version.rb
-- test/graphite-metric-test.rb
+- test/fixtures/multiple_metrics.txt
+- test/fixtures/single_metric.txt
 - test/graphite-metric/plaintext_test.rb
+- test/graphite-metric/raw_test.rb
 - test/graphite-metric/util_test.rb
 - test/test_helper.rb
 homepage: ''
@@ -97,12 +121,14 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.10
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
-summary: Generates strings that graphite understands
+summary: Generates strings that graphite understands. Loads raw graphite data.
 test_files:
-- test/graphite-metric-test.rb
+- test/fixtures/multiple_metrics.txt
+- test/fixtures/single_metric.txt
 - test/graphite-metric/plaintext_test.rb
+- test/graphite-metric/raw_test.rb
 - test/graphite-metric/util_test.rb
 - test/test_helper.rb

data/test/graphite-metric-test.rb

@@ -1,8 +0,0 @@
-require_relative('./test_helper')
-require 'graphite-metric'
-
-describe GraphiteMetric do
-  it "Plainext has an abbreviated form" do
-    GMP.new.must_be_instance_of GraphiteMetric::Plaintext
-  end
-end