graphite-api 0.0.3.beta3 → 0.0.3

data/README.md

@@ -1,4 +1,4 @@
-# GraphiteAPI ( [![Gem Version](https://fury-badge.herokuapp.com/rb/graphite-api.png)](http://badge.fury.io/rb/graphite-api) )
+# GraphiteAPI
 A Ruby API toolkit for [Graphite](http://graphite.wikidot.com/).
 
 ## Description
@@ -7,12 +7,27 @@ A Ruby API toolkit for [Graphite](http://graphite.wikidot.com/).
 ## Package Content
 * Includes a **simple** client for ruby.
 * Ships with a **GraphiteAPI-Middleware**, which is a lightweight, event-driven, aggregator daemon.
-* only one dependency (EventMachine).
+* Only one gem dependency ( EventMachine ).
 * Utilities like scheduling and caching.
 
 ## Key Features
-* **Multiple Graphite Servers Support** - GraphiteAPI-Middleware supports sending aggregated data to multiple graphite servers, useful for large data centers and backup purposes
+* **Multiple Graphite Servers Support** - GraphiteAPI-Middleware supports sending aggregated data to multiple graphite servers, in a multiplex fashion, useful for large data centers and backup purposes
 * **Reanimation mode** - support cases which the same keys (same timestamps as well) can be received simultaneously and asynchronously from multiple input sources, in these cases GraphiteAPI-Middleware will "reanimate" old records (records that were already sent to Graphite server), and will send the sum of the reanimated record value + the value of the record that was just received to the graphite server; this new summed record should override the key with the new value on Graphite database.
+* **non-blocking I/O** ( EventMachine aware ).
+* **Thread-Safe** client.
+
+## Status
+<table>
+  <tr>
+    <td> Version </td>
+    <td><a href="https://rubygems.org/gems/graphite-api"><img src=https://fury-badge.herokuapp.com/rb/graphite-api.png></a> </td>
+  </tr>
+  <tr>
+    <td> Build </td>
+    <td><a href="https://travis-ci.org/kontera-technologies/graphite-api"><img src=https://travis-ci.org/kontera-technologies/graphite-api.png?branch=master></a>
+      </td>
+  </tr>
+</table>
 
 ## Installation
 Install stable version
@@ -30,55 +45,125 @@ rake install
 ```
 
 ## Client Usage
+Creating a new client instance
+
+```ruby
+require 'graphite-api'
+
+GraphiteAPI::Client.new(
+  graphite: "graphite.example.com:2003", # not optional
+  prefix: ["example","prefix"], # add example.prefix to each key
+  slice: 60 # results are aggregated in 60 seconds slices
+  interval: 60 # send to graphite every 60 seconds
+  cache: 4 * 60 * 60 # set the max age in seconds for records reanimation
+)
+```
+
+Adding simple metrics
 ```ruby
-  require 'graphite-api'
-  require 'logger'
-  
-  # Turn on the logging ( optional )
-  GraphiteAPI::Logger.logger = ::Logger.new(STDOUT)
-  GraphiteAPI::Logger.logger.level = ::Logger::DEBUG
-  
-  # Setup client
-  client = GraphiteAPI::Client.new(  
-   :graphite => "graphite.example.com:2003",
-   :prefix   => ["example","prefix"], # add example.prefix to each key
-   :slice    => 60.seconds            # results are aggregated in 60 seconds slices
-   :interval => 60.seconds            # send to graphite every 60 seconds
-  )
-  
-  # Simple
-  client.webServer.web01.loadAvg 10.7 
-  # => example.prefix.webServer.web01.loadAvg 10.7 time.now.to_i
-  
-  # "Same Same But Different" ( http://en.wikipedia.org/wiki/Tinglish )
-  client.metrics "webServer.web01.loadAvg" => 10.7
-  # => example.prefix.webServer.web01.loadAvg 10.7 time.now.to_i
-  
-  # With event time
-  client.webServer.web01.blaBlaBla(29.1, Time.at(9999999999))
-  # => example.prefix.webServer.web01.blaBlaBla 29.1 9999999999
-  
-  # Multiple with event time
-  client.metrics({
-    "webServer.web01.loadAvg"  => 10.7,
-    "webServer.web01.memUsage" => 40
-  },Time.at(1326067060))
-  # => example.prefix.webServer.web01.loadAvg  10.7 1326067060
-  # => example.prefix.webServer.web01.memUsage 40 1326067060
-  
-  # Timers
-  client.every 10.seconds do |c|
-    c.webServer.web01.uptime `uptime`.split.first.to_i
-    # => example.prefix.webServer.web01.uptime 40 1326067060
-  end
-  
-  client.every 52.minutes do |c|
-    c.abcd.efghi.jklmnop.qrst 12 
-    # => example.prefix.abcd.efghi.jklmnop.qrst 12 1326067060
-  end
-  
-  client.join # wait...
-```	
+require 'graphite-api'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+client.metrics "webServer.web01.loadAvg" => 10.7
+# => webServer.web01.loadAvg 10.7 time.now.to_i
+
+client.metrics(
+  "webServer.web01.loadAvg"  => 10.7,
+  "webServer.web01.memUsage" => 40
+)
+# => webServer.web01.loadAvg  10.7 1326067060
+# => webServer.web01.memUsage 40 1326067060
+```
+
+Adding metrics with timestamp
+```ruby
+require 'graphite-api'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+client.metrics({
+  "webServer.web01.loadAvg"  => 10.7,
+  "webServer.web01.memUsage" => 40
+},Time.at(1326067060))
+# => webServer.web01.loadAvg  10.7 1326067060
+# => webServer.web01.memUsage 40 1326067060
+```
+
+Some DSL sweetness
+```ruby
+require 'graphite-api'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+client.webServer.web01.loadAvg 10.7 
+# => webServer.web01.loadAvg 10.7 time.now.to_i
+
+client.webServer.web01.blaBlaBla(29.1, Time.at(9999999999))
+# => webServer.web01.blaBlaBla 29.1 9999999999
+```
+
+Built-in timers support
+```ruby
+require 'graphite-api'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+# lets send the metric every 120 seconds
+client.every(120) do |c|
+  c.webServer.web01.uptime `uptime`.split.first.to_i
+end
+```
+
+Built-in extension for time declarations stuff, like 2.minutes, 3.hours etc...
+```ruby
+require 'graphite-api'
+require 'graphite-api/core_ext/numeric'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+client.every 10.seconds do |c|
+  c.webServer.web01.uptime `uptime`.split.first.to_i
+end
+
+client.every 52.minutes do |c|
+  c.just.fake 12
+end
+```
+
+Make your own custom metrics daemons, using `client#join`
+```ruby
+require 'graphite-api'
+require 'graphite-api/core_ext/numeric'
+
+client = GraphiteAPI::Client.new( graphite: 'graphite:2003' )
+
+client.every 26.minutes do |c|
+  c.webServer.shuki.stats 10
+  c.webServer.shuki.x 97
+  c.webServer.shuki.y 121
+end
+
+client.join # wait for ever...
+```
+
+Logging support
+
+```ruby
+# Provide an external logger
+require 'graphite-api'
+require 'logger'
+
+GraphiteAPI::Logger.logger = ::Logger.new(STDOUT)
+GraphiteAPI::Logger.logger.level = ::Logger::DEBUG
+
+# Or use the built-in one
+GraphiteAPI::Logger.init(
+  :level => :debug,
+  :std   => 'logger.out' # or STDOUT | STDERR
+)
+```
+
 > more examples can be found [here](https://github.com/kontera-technologies/graphite-api/tree/master/examples).
 
 ## GraphiteAPI-Middleware Usage
@@ -142,21 +227,13 @@ client.bla.bla.value2 27
 > more examples can be found [here](https://github.com/kontera-technologies/graphite-api/tree/master/examples).
 
 
-## Recommended Topologies
+## Example Setup
 <br/>
-
-<img src="https://raw.github.com/kontera-technologies/graphite-api/master/examples/graphite-middleware-star.jpg" align="center">
-
-<hr/>
-<br/>
-
-<img src="https://raw.github.com/kontera-technologies/graphite-api/master/examples/graphite-middleware-mesh.jpg" align="center">
-
-<hr/>
+<img src="https://raw.github.com/kontera-technologies/graphite-api/master/examples/middleware_t1.png" align="center">
 
 ## TODO:
-* Documentation
-* Use Redis
+* Better documentation
+* Use Redis for caching
 * Multiple backends via client as well
 
 ## Bugs

data/Rakefile

@@ -3,4 +3,8 @@ Dir.chdir File.dirname __FILE__
 
 require 'graphite-api'
 
+def message msg
+  puts "*** #{msg} ***"
+end
+
 Dir['tasks/**/*.rake'].each { |rake| load rake }

data/lib/graphite-api.rb

@@ -19,5 +19,4 @@ module GraphiteAPI
     GraphiteAPI::Version::VERSION
   end
   
-  Dir.glob( "#{ROOT}/core-extensions/**" ).each &method( :require )
 end

data/lib/graphite-api/cache/memory.rb

@@ -8,11 +8,11 @@ module GraphiteAPI
       end
       
       def get time, key
-        cache[time][key]
+        cache[time.to_i][key]
       end
       
       def set time, key, value
-        cache[time][key] = value.to_f 
+        cache[time.to_i][key] = value.to_f 
       end
       
       def incr time, key, value
@@ -22,12 +22,12 @@ module GraphiteAPI
       private
       
       def cache
-        @cache ||= Hash.new {|h,k| h[k] = Hash.new {|h1,k1| h1[k1] = 0}}
+        @cache ||= nested_zero_hash
       end
       
-      def clean age
+      def clean max_age
         debug [:MemoryCache, :before_clean, cache]
-        cache.delete_if {|t,k| Time.now.to_i - t > age}
+        cache.delete_if {|t,k| Time.now.to_i - t > max_age }
         debug [:MemoryCache, :after_clean, cache]
       end
       

data/lib/graphite-api/client.rb

@@ -11,15 +11,12 @@
 #   :interval => 60.seconds            # send to graphite every 60 seconds
 #  )
 #  
-#  # Simple
 #  client.webServer.web01.loadAvg 10.7 
 #  # => example.prefix.webServer.web01.loadAvg 10.7 time.now.to_i
 
-#  # "Same Same But Different" ( http://en.wikipedia.org/wiki/Tinglish )
 #  client.metrics "webServer.web01.loadAvg" => 10.7
 #  # => example.prefix.webServer.web01.loadAvg 10.7 time.now.to_i
 #  
-#  # Multiple with event time
 #  client.metrics({
 #   "webServer.web01.loadAvg"  => 10.7,
 #   "webServer.web01.memUsage" => 40

data/lib/graphite-api/connector.rb

@@ -23,8 +23,7 @@ module GraphiteAPI
     
     def puts message
       begin
-        Logger.debug [:connector,:puts,[host,port].join(":"),message]
-        return
+        debug [:connector,:puts,[host,port].join(":"),message]
         socket.puts message + "\n"
       rescue Errno::EPIPE, Errno::EINVAL
         @socket = nil

data/lib/graphite-api/safe_buffer.rb

@@ -16,7 +16,6 @@
 #    ["mem.usage", 190.0, 1326842520]
 #    ["shuki.tuki", 999.0, 1326842520]
 # -----------------------------------------------------
-
 require 'thread'
 require 'set'
 
@@ -48,7 +47,7 @@ module GraphiteAPI
         streamer[client_id] += char 
         
         if closed_stream? streamer[client_id]
-          if valid_stream_message streamer[client_id]
+          if valid_stream_message? streamer[client_id]
             push stream_message_to_obj streamer[client_id]
           end
           streamer.delete client_id
@@ -62,16 +61,16 @@ module GraphiteAPI
       debug [:buffer,:add, obj]
       queue.push obj
       nil
-    end    
+    end
 
     alias_method :<<, :push
 
     def pull format = nil
-      data = Hash.new {|h,k| h[k] = Hash.new {|h,k| h[k] = 0} }
+      data = nested_zero_hash
 
       counter = 0
       while new_records?
-        break if ( counter += 1 ) > 10_000
+        break if ( counter += 1 ) > 1_000_000 # TODO: fix this
         hash = queue.pop
         time = normalize_time(hash[:time],options[:slice])
         hash[:metric].each { |k,v| data[time][k] += v.to_f }
@@ -91,7 +90,8 @@ module GraphiteAPI
     end
     
     def inspect
-      "#<GraphiteAPI::SafeBuffer:#{object_id} @quque#size=#{queue.size} @streamer=#{streamer.inspect}>"
+      "#<GraphiteAPI::SafeBuffer:%s @quque#size=%s @streamer=%s>" % 
+        [object_id,queue.size,streamer]
     end
     
     private
@@ -109,7 +109,7 @@ module GraphiteAPI
       string[-1,1] == END_OF_STREAM
     end
     
-    def valid_stream_message message
+    def valid_stream_message? message
       message =~ VALID_MESSAGE
     end
 

data/lib/graphite-api/utils.rb

@@ -28,7 +28,11 @@ module GraphiteAPI
       slice = 60 if slice.nil?
       ((time || Time.now).to_i / slice * slice).to_i
     end
- 
+    
+    def nested_zero_hash
+      Hash.new {|h,k| h[k] = Hash.new {|h,k| h[k] = 0} }
+    end
+    
     module_function
  
     def expand_host host

data/lib/graphite-api/version.rb

@@ -1,5 +1,5 @@
 module GraphiteAPI
   class Version
-    VERSION = "0.0.3.beta3"
+    VERSION = "0.0.3"
   end
 end

data/tasks/build.rake

@@ -19,7 +19,8 @@ GraphiteAPI::GemSpec = Gem::Specification.new do |s|
   s.add_dependency 'eventmachine','>= 0.3.3'
 end
 
-task :gem => [:clobber_package]
+task :gem => [:test,:clobber_package]
+
 Gem::PackageTask.new(GraphiteAPI::GemSpec) do |p|
   p.gem_spec = GraphiteAPI::GemSpec
 end

data/tasks/tests.rake

@@ -1,10 +1,42 @@
 require "rake/testtask"
 
-task(:test) { ENV['with_coverage'] = "true" }
+task(:test => :functional) { ENV['with_coverage'] = "true" }
 
 Rake::TestTask.new(:test) do |t|
   t.libs << "tests"
   t.pattern = "tests/**/*_test.rb"
 end
 
+
+task :functional do
+  some_failed = false
+  
+  next unless ENV['SKIP_FUNC'].nil?
+
+  unless RUBY_COPYRIGHT.end_with?("Matsumoto")
+    puts("Functional tests are enabled only on MRI...")
+    next
+  end
+  
+  message "Executing GraphiteAPI Functional Tests"
+  message "( You can skip them by passing SKIP_FUNC=true )"
+
+  Dir[File.expand_path("../../tests/functional/*",__FILE__)].each do |file|
+    next unless file.end_with?(".rb")
+    now = Time.now.to_i
+    name = File.basename(file)
+    message "Executing #{name}"
+    Process.waitpid(Process.spawn("ruby", File.expand_path(file)))
+    took = "took #{Time.now.to_i - now} seconds"
+    if $?.success?
+      message "[PASS] #{name}, #{took}"
+    else
+      message "[FAIL] #{name}, #{took}"
+      some_failed = true
+    end
+  end
+  message "Done Executing GraphiteAPI Functional Tests"
+  abort "Some functional tests failed..." if some_failed
+end
+
 task :default => :test

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: graphite-api
 version: !ruby/object:Gem::Version
-  version: 0.0.3.beta3
-  prerelease: 6
+  version: 0.0.3
+  prerelease: 
 platform: ruby
 authors:
 - Eran Barak Levi
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-03-06 00:00:00.000000000 Z
+date: 2013-03-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: eventmachine
@@ -37,14 +37,13 @@ files:
 - README.md
 - Rakefile
 - bin/graphite-middleware
-- lib/core-extensions/numeric.rb
-- lib/graphite-api/buffer.rb
 - lib/graphite-api/cache/memory.rb
 - lib/graphite-api/cache.rb
 - lib/graphite-api/cli.rb
 - lib/graphite-api/client.rb
 - lib/graphite-api/connector.rb
 - lib/graphite-api/connector_group.rb
+- lib/graphite-api/core_ext/numeric.rb
 - lib/graphite-api/logger.rb
 - lib/graphite-api/middleware.rb
 - lib/graphite-api/reactor.rb
@@ -70,9 +69,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>'
+  - - ! '>='
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project: graphite-api
 rubygems_version: 1.8.25

data/lib/graphite-api/buffer.rb

@@ -1,137 +0,0 @@
-# -----------------------------------------------------
-# Buffer Object
-# Handle Socket & Client data streams
-# -----------------------------------------------------
-# Usage:
-#     buff = GraphiteAPI::Buffer.new(GraphiteAPI::Utils.default_options)
-#     buff << {:metric => {"load_avg" => 10},:time => Time.now}
-#     buff << {:metric => {"load_avg" => 30},:time => Time.now}
-#     buff.stream "mem.usage 1"
-#     buff.stream "90 1326842563\n"
-#     buff.stream "shuki.tuki 999 1326842563\n"
-#     buff.pull.each {|o| p o} 
-#
-# Produce:
-#    ["load_avg", 40.0, 1326881160]
-#    ["mem.usage", 190.0, 1326842520]
-#    ["shuki.tuki", 999.0, 1326842520]
-# -----------------------------------------------------
-require 'set'
-
-module GraphiteAPI
-  class Buffer
-    include Utils    
-
-    CLOSING_STREAM  = "\n"   # end of message - when streaming to buffer obj
-    CHARS_TO_IGNORE = %w(\r) # skip these chars when parsing new message
-    FLOATS_ROUND_BY = 2      # round(x) after summing floats 
-
-    VALID_MESSAGE   = /^[\w|\.]+ \d+(?:\.|\d)* \d+$/
-    
-    def initialize options
-      @options = options
-      @keys_to_sync = Hash.new { |h,k| h[k] = Set.new }
-      @streamer_buff = Hash.new {|h,k| h[k] = ""}
-      @reanimation_mode = !options[:cache].nil?
-      start_cleaner if reanimation_mode
-    end
-
-    private_reader :options, :keys_to_sync, :reanimation_mode, :streamer_buff
-
-    def push hash
-      debug [:buffer,:add, hash]
-      time = normalize_time(hash[:time],options[:slice])
-      hash[:metric].each { |k,v| cache_set(time,k,v) }
-    end
-
-    alias_method :<<, :push
-    
-    def stream message, client_id = nil
-      message.gsub(/\t/,' ').each_char do |char|
-        next if invalid_char? char
-        streamer_buff[client_id] += char 
-        
-        if closed_stream? streamer_buff[client_id]
-          if valid streamer_buff[client_id]
-            push build_metric *streamer_buff[client_id].split
-          end
-          streamer_buff.delete client_id
-        end
-      end
-    end
-    
-    def pull as = nil
-      Array.new.tap do |data|
-        keys_to_sync.each do |time,keys|
-          keys.each do |key|
-            data.push cache_get(time, key, as)
-          end
-        end
-        clear
-      end
-      
-    end
-    
-    def new_records?
-      !keys_to_sync.empty?
-    end
-    
-    private
-    
-    def closed_stream? string
-      string[-1,1] == CLOSING_STREAM
-    end
-    
-    def invalid_char? char
-      CHARS_TO_IGNORE.include? char
-    end
-    
-    def cache_set time, key, value
-      buffer_cache[time][key] = sum buffer_cache[time][key], value.to_f 
-      keys_to_sync[time].add key
-    end
-    
-    def sum float1, float2
-      ("%.#{FLOATS_ROUND_BY}f" % (float1 + float2)).to_f
-    end
-        
-    def cache_get time, key, as
-      metric = [prefix + key,buffer_cache[time][key],time]
-      as == :string ? metric.join(" ") : metric
-    end
-
-    def build_metric key, value, time
-      { :metric => { key => value },:time => Time.at(time.to_i) }
-    end
-
-    def clear
-      keys_to_sync.clear
-      buffer_cache.clear unless reanimation_mode
-    end
-    
-    def valid message
-      message =~ VALID_MESSAGE
-    end
-    
-    def prefix
-      @prefix ||= options[:prefix].empty? ? '' : prefix_to_s 
-    end
-    
-    def prefix_to_s
-      Array(options[:prefix]).join('.') << '.'
-    end
-    
-    def buffer_cache
-      @buffer_cache ||= Hash.new {|h,k| h[k] = Hash.new {|h1,k1| h1[k1] = 0}}
-    end
-
-    def clean age
-      [buffer_cache,keys_to_sync].each {|o| o.delete_if {|t,k| Time.now.to_i - t > age}}
-    end
-    
-    def start_cleaner
-      Reactor::every(options[:cleaner_interval]) { clean(options[:cache]) }
-    end
-
-  end
-end