tabs 0.9.1 → 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 5925ee887e482c08b4ae405b4463162a90b41494
+  data.tar.gz: 878be73e42a4c32a58853c61385f71460bbafa60
+SHA512:
+  metadata.gz: 531f0554860cb6143ee9d8cb29d7362d2111c63324a52ca5cc773f1767a80da58e2bae4680479abf791b024c185275334cc9b07fa7014dd459099af49e2cb4cd
+  data.tar.gz: 62bc646ae94b276fa7a089fe7e88824473b65e9080244be8a54a5ea0f5b064114226d06e82cbf918320c748bbedbfecc545280156c7b0542d2f3f4466f8c7be7

data/.gitignore

@@ -16,3 +16,4 @@ test/tmp
 test/version_tmp
 tmp
 dump.rdb
+tags

data/.ruby-version

@@ -1 +1 @@
-1.9.3
+2.0.0-p247

data/.travis.yml

@@ -1,3 +1,6 @@
 language: ruby
 rvm:
   - 1.9.3
+  - 2.0.0
+services:
+  - redis-server

data/README.md

@@ -2,7 +2,7 @@
 
 # Tabs
 
-Tabs is a redis-backed metrics tracker that supports counts, sums,
+Tabs is a redis-backed metrics tracker for time-based events that supports counts, sums,
 averages, and min/max, and task based stats sliceable by the minute, hour, day, week, month, and year.
 
 ## Installation
@@ -208,11 +208,15 @@ that's necessary is a module that conforms to the following protocol:
 
 ```ruby
 module SecondResolution
-  extend Tabs::Resolutionable
+  include Tabs::Resolutionable
   extend self
 
   PATTERN = "%Y-%m-%d-%H-%M-%S"
 
+  def name
+    :seconds
+  end
+  
   def serialize(timestamp)
     timestamp.strftime(PATTERN)
   end
@@ -225,6 +229,10 @@ module SecondResolution
   def from_seconds(s)
     s / 1
   end
+  
+  def to_seconds
+    1
+  end
 
   def add(timestamp, num_of_seconds)
     timestamp + num_of_seconds.seconds
@@ -239,6 +247,8 @@ end
 
 A little description on each of the above methods:
 
+*`name`*: unique symbol used to reference registered resolution
+
 *`serialize`*: converts the timestamp to a string.  The return value
 here will be used as part of the Redis key storing values associated
 with a given metric.
@@ -249,6 +259,8 @@ into an actual DateTime value.
 *`from_seconds`*: should return the number of periods in the given
 number of seconds.  For example, there are 60 seconds in a minute.
 
+*`to_seconds`*: should return the number of seconds in '1' of these time periods.  For example, there are 3600 seconds in an hour.
+
 *`add`*: should add the number of seconds in the given resolution to the
 supplied timestamp.
 
@@ -256,7 +268,7 @@ supplied timestamp.
 For example, the week resolution returns the first hour of the first day
 of the week.
 
-*NOTE: If you're doing a custom resolution you should probably look into
+*NOTE: If you're doing a custom resolution, you should probably look into
 the code a bit.*
 
 Once you have a module that conforms to the resolution protocol you need
@@ -264,11 +276,19 @@ to register it with Tabs.  You can do this in one of two ways:
 
 ```ruby
 # This call can be anywhere before you start using tabs
-Tabs::Resolution.register(:second, SecondResolution)
+Tabs::Resolution.register(SecondResolution)
 
 # or, you can use the config block described below
 ```
 
+#### Removing a Resolution
+
+You can also remove any resolution (custom or built-in) by calling the `unregister_resolutions` method in the config block (see config section below).  Or, you can remove manually by calling:
+
+```ruby
+Tabs::Resolution.unregister(:minute, :hour)
+```
+
 ### Inspecting Metrics
 
 You can list all metrics using `list_metrics`:
@@ -300,11 +320,31 @@ Tabs.drop_metric!("website-visits")
 
 This will drop all recorded values for the metric so it may not be un-done...be careful.
 
+To drop only a specific resolution for a metric, just call `Tabs#drop_resolution_for_metric!`
+
+```ruby
+Tabs.drop_resolution_for_metric!("website-visits", :minute)
+```
+
 Even more dangerous, you can drop all metrics...be very careful.
 
 ```ruby
 Tabs.drop_all_metrics!
 ```
+### Aging Out Old Metrics
+
+You can use the expiration features to age out old metrics that may no longer be in your operational data set.  For example, you may want to keep monthly or yearly data around but the minute or day level data isn't necessary past a certain date.  You can set expirations for any resolution:
+
+```ruby
+Tabs.configure do |config|
+	config.set_expirations(minute: 1.day, day: 1.week)
+end
+```
+
+The expiration date will start counting at the beginning at the end of the given resolution.  Meaning that for a month resolution the given expiration time would start at the end of a given month.  A month resolution metric recorded in January with an expiration of 2 weeks would expire after the 2nd week of February.
+
+*NOTE: You cannot expire task metrics at this time, only counter and
+values.*
 
 ### Configuration
 
@@ -318,6 +358,10 @@ Tabs.configure do |config|
 
   # pass a config hash that will be passed to Redis.new
   config.redis = { :host => 'localhost', :port => 6379 }
+  
+  # pass a prefix that will be used in addition to the "tabs" prefix with Redis keys
+  # Example: "tabs:my_app:metric_name"
+  config.prefix = "my_app"
 
   # override default decimal precision (5)
   # affects stat averages and task completion rate
@@ -326,10 +370,31 @@ Tabs.configure do |config|
   # registers a custom resolution
   config.register_resolution :second, SecondResolution
 
+  # unregisters any resolution
+  config.unregister_resolutions(:minute, :hour)
+  
+  # sets TTL for redis keys of specific resolutions
+  config.set_expirations({ minute: 1.hour, hour: 1.day })
+
 end
 ```
 
-## Breaking Changes
+#### Prefixing
+
+Many applications use a single Redis instance for a number of uses:
+background jobs, ephemeral data, Tabs, etc.  To avoid key collisions,
+and to make it easier to drop all of your tabs data without affecting
+other parts of your system (or if more than one app shares the Redis
+instance) you can prefix a given 'instance'.
+
+Setting the prefix config option will cause all of the keys that tabs
+stores to use this format:
+
+```
+tabs:#{prefix}:#{key}..."
+```
+
+## Change Log & Breaking Changes
 
 ### v0.6.0
 
@@ -361,6 +426,21 @@ statement to simulate a transaction.  Value data that was recorded prior
 to v0.8.2 will not be accessible in this or future versions so please
 continue to use v0.8.1 or lower if that is an issue.
 
+### v1.0.0
+
+_WARNING: Version 1.0.0 is not compatible with previous versions of
+Tabs_
+
+We have made a number of changes related to hour metric keys are stored
+in Redis.  At this point we'll be following semantec versioning and will
+support backwards compatability between major versions.  In this release
+we've added a number of major features:
+
+* Metric expiration
+* Key prefixes
+* Support for unregistering resolutions
+* A number of small numeric fixes
+
 ## Contributing
 
 1. Fork it

data/lib/tabs/config.rb

@@ -22,8 +22,43 @@ module Tabs
       @redis ||= Redis.new
     end
 
-    def register_resolution(resolution, klass)
-      Tabs::Resolution.register(resolution, klass)
+    def prefix=(arg)
+      @prefix = arg
+    end
+
+    def prefix
+      @prefix
+    end
+
+    def register_resolution(klass)
+      Tabs::Resolution.register(klass)
+    end
+
+    def unregister_resolutions(*resolutions)
+      Tabs::Resolution.unregister(resolutions)
+    end
+
+    def expiration_settings
+      @expiration_settings ||= {}
+    end
+
+    def set_expirations(resolution_hash)
+      resolution_hash.each do |resolution, expires_in_seconds|
+        raise Tabs::ResolutionMissingError.new(resolution) unless Tabs::Resolution.all.include? resolution
+        expiration_settings[resolution] = expires_in_seconds
+      end
+    end
+
+    def expires?(resolution)
+      expiration_settings.has_key?(resolution)
+    end
+
+    def expires_in(resolution)
+      expiration_settings[resolution]
+    end
+
+    def reset_expirations
+      @expiration_settings = {}
     end
 
   end

data/lib/tabs/metrics/counter.rb

@@ -21,8 +21,8 @@ module Tabs
 
       def stats(period, resolution)
         timestamps = timestamp_range period, resolution
-        keys = timestamps.map do |ts|
-          "stat:counter:#{key}:count:#{Tabs::Resolution.serialize(resolution, ts)}"
+        keys = timestamps.map do |timestamp|
+          storage_key(resolution, timestamp)
         end
 
         values = mget(*keys).map do |v|
@@ -43,12 +43,21 @@ module Tabs
         del_by_prefix("stat:counter:#{key}")
       end
 
+      def drop_by_resolution!(resolution)
+        del_by_prefix("stat:counter:#{key}:count:#{resolution}")
+      end
+
+      def storage_key(resolution, timestamp)
+        formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
+        "stat:counter:#{key}:count:#{resolution}:#{formatted_time}"
+      end
+
       private
 
       def increment_resolution(resolution, timestamp)
-        formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
-        stat_key = "stat:counter:#{key}:count:#{formatted_time}"
-        incr(stat_key)
+        store_key = storage_key(resolution, timestamp)
+        incr(store_key)
+        Tabs::Resolution.expire(resolution, store_key, timestamp)
       end
 
       def increment_total

data/lib/tabs/metrics/task.rb

@@ -37,7 +37,7 @@ module Tabs
         matching_tokens = started_tokens.select { |token| completed_tokens.include? token }
         completion_rate = (matching_tokens.size.to_f / range.size).round(Config.decimal_precision)
         elapsed_times = matching_tokens.map { |t| t.time_elapsed(resolution) }
-        average_completion_time = (elapsed_times.sum) / matching_tokens.size
+        average_completion_time = matching_tokens.blank? ? 0.0 : (elapsed_times.sum) / matching_tokens.size
         Stats.new(
           started_tokens.size,
           completed_tokens.size,
@@ -51,18 +51,20 @@ module Tabs
         del_by_prefix("stat:task:#{key}")
       end
 
+      def storage_key(resolution, timestamp, type)
+        formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
+        "stat:task:#{key}:#{type}:#{resolution}:#{formatted_time}"
+      end
+
       private
 
       def tokens_for_period(range, resolution, type)
         keys = keys_for_range(range, resolution, type)
-        mget(*keys).compact.map(&:to_a).flatten.map { |t| Token.new(t, key) }
+        smembers_all(*keys).compact.map(&:to_a).flatten.map { |t| Token.new(t, key) }
       end
 
       def keys_for_range(range, resolution, type)
-        range.map do |date|
-          formatted_time = Tabs::Resolution.serialize(resolution, date)
-          "stat:task:#{key}:#{type}:#{formatted_time}"
-        end
+        range.map { |timestamp| storage_key(resolution, timestamp, type) }
       end
 
     end

data/lib/tabs/metrics/task/token.rb

@@ -14,13 +14,13 @@ module Tabs
 
         def start(timestamp=Time.now)
           self.start_time = timestamp.utc
-          sadd("stat:task:#{key}:tokens", token)
+          sadd(tokens_storage_key, token)
           Tabs::Resolution.all.each { |res| record_start(res, start_time) }
         end
 
         def complete(timestamp=Time.now)
           self.complete_time = timestamp.utc
-          unless sismember("stat:task:#{key}:tokens", token)
+          unless sismember(tokens_storage_key, token)
             raise UnstartedTaskMetricError.new("No task for metric '#{key}' was started with token '#{token}'")
           end
           Tabs::Resolution.all.each { |res| record_complete(res, complete_time) }
@@ -40,32 +40,47 @@ module Tabs
 
         private
 
-        def record_start(resolution, timestamp)
+        def storage_key(resolution, timestamp, type)
           formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
-          sadd("stat:task:#{key}:started:#{formatted_time}", token)
+          "stat:task:#{key}:#{type}:#{resolution}:#{formatted_time}"
+        end
+
+        def started_storage_key
+          "stat:task:#{key}:#{token}:started_time"
+        end
+
+        def completed_storage_key
+          "stat:task:#{key}:#{token}:completed_time"
+        end
+
+        def tokens_storage_key
+          "stat:task:#{key}:tokens"
+        end
+
+        def record_start(resolution, timestamp)
+          sadd(storage_key(resolution, timestamp, "started"), token)
         end
 
         def record_complete(resolution, timestamp)
-          formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
-          sadd("stat:task:#{key}:completed:#{formatted_time}", token)
+          sadd(storage_key(resolution, timestamp, "completed"), token)
         end
 
         def start_time=(timestamp)
-          set("stat:task:#{key}:#{token}:started_time", timestamp)
+          set(started_storage_key, timestamp)
           @start_time = timestamp
         end
 
         def start_time
-          Time.parse(get("stat:task:#{key}:#{token}:started_time"))
+          Time.parse(get(started_storage_key))
         end
 
         def complete_time=(timestamp)
-          set("stat:task:#{key}:#{token}:completed_time", timestamp)
+          set(completed_storage_key, timestamp)
           @complete_time = timestamp
         end
 
         def complete_time
-          Time.parse(get("stat:task:#{key}:#{token}:completed_time"))
+          Time.parse(get(completed_storage_key))
         end
 
       end

data/lib/tabs/metrics/value.rb

@@ -13,23 +13,21 @@ module Tabs
       def record(value, timestamp=Time.now)
         timestamp.utc
         Tabs::Resolution.all.each do |resolution|
-          formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
-          stat_key = "stat:value:#{key}:data:#{formatted_time}"
-          update_values(stat_key, value)
+          store_key = storage_key(resolution, timestamp)
+          update_values(store_key, value)
+          Tabs::Resolution.expire(resolution, store_key, timestamp)
         end
         true
       end
 
       def stats(period, resolution)
         timestamps = timestamp_range period, resolution
-        keys = timestamps.map do |ts|
-          formatted_time = Tabs::Resolution.serialize(resolution, ts)
-          "stat:value:#{key}:data:#{formatted_time}"
+        keys = timestamps.map do |timestamp|
+          storage_key(resolution, timestamp)
         end
 
         values = mget(*keys).map do |v|
-          value = v.nil? ? default_value(0) : v
-          value = Hash[value.map { |k, i| [k, to_numeric(i)] }]
+          value = v.nil? ? default_value(0) : JSON.parse(v)
           value["timestamp"] = timestamps.shift
           value.with_indifferent_access
         end
@@ -41,41 +39,51 @@ module Tabs
         del_by_prefix("stat:value:#{key}")
       end
 
+      def drop_by_resolution!(resolution)
+        del_by_prefix("stat:value:#{key}:data:#{resolution}")
+      end
+
+      def storage_key(resolution, timestamp)
+        formatted_time = Tabs::Resolution.serialize(resolution, timestamp)
+        "stat:value:#{key}:data:#{resolution}:#{formatted_time}"
+      end
+
       private
 
       def update_values(stat_key, value)
-        count = update_count(stat_key)
-        sum = update_sum(stat_key, value)
-        update_min(stat_key, value)
-        update_max(stat_key, value)
-        update_avg(stat_key, sum, count)
+        hash = get_current_hash(stat_key)
+        increment(hash, value)
+        update_min(hash, value)
+        update_max(hash, value)
+        update_avg(hash)
+        set(stat_key, JSON.generate(hash))
       end
 
-      def update_count(stat_key)
-        hincrby(stat_key, "count", 1)
+      def get_current_hash(stat_key)
+        hash = get(stat_key)
+        return JSON.parse(hash) if hash
+        default_value
       end
 
-      def update_sum(stat_key, value)
-        hincrby(stat_key, "sum", value)
+      def increment(hash, value)
+        hash["count"] += 1
+        hash["sum"] += value.to_f
       end
 
-      def update_min(stat_key, value)
-        min = (hget(stat_key, "min") || 0).to_i
-        hset(stat_key, "min", value) if value < min || min == 0
+      def update_min(hash, value)
+        hash["min"] = value.to_f if hash["min"].nil? || value.to_f < hash["min"]
       end
 
-      def update_max(stat_key, value)
-        max = (hget(stat_key, "max") || 0).to_i
-        hset(stat_key, "max", value) if value > max || max == 0
+      def update_max(hash, value)
+        hash["max"] = value.to_f if hash["max"].nil? || value.to_f > hash["max"]
       end
 
-      def update_avg(stat_key, sum, count)
-        avg = sum.to_f / count.to_f
-        hset(stat_key, "avg", avg)
+      def update_avg(hash)
+        hash["avg"] = hash["sum"].to_f / hash["count"]
       end
 
       def default_value(nil_value=nil)
-        { "count" => 0, "min" => nil_value, "max" => nil_value, "sum" => 0, "avg" => 0 }
+        { "count" => 0, "min" => nil_value, "max" => nil_value, "sum" => 0.0, "avg" => 0.0 }
       end
 
     end