trackoid 0.2.0 → 0.3.0

data/.rspec

@@ -0,0 +1,3 @@
+--color
+--format documentation
+

data/README.rdoc

@@ -2,6 +2,15 @@
 
 Trackoid is an analytics tracking system made specifically for MongoDB using Mongoid as ORM.
 
+= *** IMPORTANT ***
+
+Trackoid Version 0.3.0 changes the internal representation of tracking data. So <b>YOU WILL NOT SEE PREVIOUS DATA</b> when you update.
+
+Hopefully, due to the magic of MongoDB, data is <b>NOT LOST</b>. In fact it's never lost unless you delete it. :-) Just it's not visible right away.
+
+See <b>Changes for TZ support</b> below for an explanation of changes. If you absolutely, desperately, dead or alive, need a migration, leave me a message and we can arrange a migration script.
+
+
 = Requirements
 
 Trackoid requires Mongoid, which obviously in turn requires MongoDB. Although you can only use Trackoid in Rails projects using Mongoid, it can easily be ported to MongoMapper or other ORM. You can also port it to work directly using MongoDB.
@@ -186,8 +195,129 @@ A year full of statistical data takes only 2.8Kb, if you store integers. If your
 For comparison, this README is already 8.5Kb in size...
 
 
+= Changes for TZ support
+
+Well, this is the time (no pun intended) to add TZ support to Trackoid.
+
+The problem is that "today" is not the same "today" for everyone, so unless you live in UTC or don't care about time zones, you probably should stay in 0.2.0 version and live long and prosper...
+
+But... Okay, given the fact that "today" is not the same "today" for everyone, this is the brand new Trackoid, with TZ support.
+
+== What has changed?
+
+In the surface, almost nothing, but internally there has been a major rewrite of the tracking code (the 'inc', 'set' methods) and the readers ('today', 'yesterday', etc). This is due to the changes I've made to the MongoDB structure of the tracking data.
+
+<b>YOU WILL NEED TO MIGRATE THE EXISTING DATA IF YOU WANT TO KEEP IT</b>
+
+This is very important, so I will repeat:
+
+<b>YOU WILL NEED TO MIGRATE THE EXISTING DATA IF YOU WANT TO KEEP IT</b>
+
+The internal JSON structure of a tracking field was like that.
+
+    {
+      ... some other fields in the model...,
+      "tracking_data" : {
+        "2011" : {
+          "01" : {
+            "01" : 10,
+            "02" : 20,
+            "03" : 30,
+            ...
+          },
+          "02" : {
+            "01" : 10,
+            "02" : 20,
+            "03" : 30,
+            ...
+          }
+        }
+      }
+    }
+
+That is, years, months and days numbers created a nested hash in which the final data (leaves) was the amount tracked. You see? There was no trace of hours... That's the problem.
+
+This is the new, TZ aware version of the internal JSON structure:
+
+    {
+      ... some other fields in the model...,
+      "tracking_data" : {
+        "14975" : {
+          "00" : 10,
+          "01" : 20,
+          "02" : 30,
+          ...
+          "22" : 88,
+          "23" : 99
+        },
+        "14976" : {
+          "00" : 10,
+          "01" : 20,
+          "02" : 30,
+          ...
+          "22" : 88,
+          "23" : 99
+        }
+      }
+    }
+
+So, instead of a nested array with keys like year/month/day, I now use the timestamp of the date. Well, a cooked timestamp. "14975" is the numbers of days since the epoch, which is the number of seconds elapsed since midnight Coordinated Universal Time (UTC) of January 1, 1970, and blah, blah, blah... You know what's this all about (http://en.wikipedia.org/wiki/Unix_time)
+
+The exact formula is like this (Ruby):
+
+    date_index = Time.now.utc.to_i / 60 / 60 / 24
+
+The contents of every "day record" is another hash with 24 keys, one for each hour. This MUST be a hash, not an array (which might be more natural) sice Trackoid uses "upserts" operations to be atomic. Reading the array, modifying it and saving it back is not an option. The exact MongoDB operation to support this is as follows:
+
+    db.update(
+      { search_criteria },
+      { "$inc" => {"track_data.14976.10" => 1} },
+      { :upsert => true, :safe => false }
+    )
+
+== What "today" is it?
+
+All dates are saved in UTC. That means Trackoid returns a whole 24 hour block for "today" only where the TZ is exactly UTC/GMT (no offset). If you live in a country where there is an offset into UTC, Trackoid must read a whole block and some hours from the block before or after to build "your today".
+
+Example: I live in GMT+0200 (Daylight saving in effect, or summer time), then if I request data for "today" as of 2011-04-14, Trackoid must first read the block for 15078 (UTC index for 2011-04-14), shift up 2 hours and then fill the missing 2 hours from the day before (15078). The entire block will be like this:
+
+    "tracking_data" : {
+      "15078" : {
+        "22" : 88,    # Last two hours from 2011-04-13 UTC
+        "23" : 99
+      },
+      "15079" : {
+        "00" : 10,
+        "01" : 20,
+        "02" : 30,
+        ""
+      }
+    
+This is a more graphical representation:
+
+    Hours  00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
+    ------ -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --        
+    GMT+2: 00 00 00 XX 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+      UTC: --->  00 XX 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+           Shift up ---> 2 hours.
+
+
+For timezones with a negative offset from UTC (Like PDT/PST) the process is reversed: UTC values are shifted down and holes filled with the following day.
+
+
+== How should I tell Trackoid how TZ to use?
+
+Piece of cake: Use the reader methods "today", "yesterday", "last_days(N)" and Trackoid will use the effective Time Zone of your Rails/Ruby application.
+
+Trackoid will correctly translate dates for you (hopefully) if you pass a date to any of those methods.
+
+
+
 = Revision History
 
+  0.3.0  -  Biggest change ever. Read <b>Changes for TZ support</b> above.
+            <b>YOU WILL NEED A MIGRATION FOR EXISTING DATA</b>
+
   0.2.0  -  Added 'reset' and 'erase' methods to tracker fields:
             * Reset does the same as "set" but also sets aggregate fields.
 

data/VERSION

@@ -1 +1 @@
-0.2.0
+0.3.0

data/lib/trackoid/aggregates.rb

@@ -2,6 +2,9 @@
 module Mongoid  #:nodoc:
   module Tracking #:nodoc:
     module Aggregates
+      
+      DEPRECATED_TOKENS = ['hour', 'hours']
+      
       # This module includes aggregate data extensions to Trackoid instances
       def self.included(base)
         base.class_eval do
@@ -55,8 +58,8 @@ module Mongoid  #:nodoc:
         # But you are encouraged to use Trackoid methods whenever possible.
         #
         def aggregate(name, &block)
-          raise Errors::AggregationAlreadyDefined.new(self.name, name) if
-              aggregate_fields.has_key? name
+          raise Errors::AggregationAlreadyDefined.new(self.name, name) if aggregate_fields.has_key? name
+          raise Errors::AggregationNameDeprecated.new(name) if DEPRECATED_TOKENS.include? name.to_s
 
           define_aggregate_model if aggregate_klass.nil?
           has_many_related internal_accessor_name(name), :class_name => aggregate_klass.to_s
@@ -83,6 +86,7 @@ module Mongoid  #:nodoc:
         # Defines the aggregation model. It checks for class name conflicts
         def define_aggregate_model
           raise Errors::ClassAlreadyDefined.new(internal_aggregates_name) if foreign_class_defined?
+
           parent = self
           define_klass do
             include Mongoid::Document

data/lib/trackoid/core_ext/range.rb

@@ -0,0 +1,53 @@
+# encoding: utf-8
+class Range
+  # Adds some enumerable capabilities to Time ranges
+  # (Normally they raise a "Can't iterate time ranges")
+  #
+  # It works by assuming days while iterating the time range, but you can
+  # pass an optional parameter
+
+  HOURS = 3600
+  DAYS = 24*HOURS
+  DEFAULT_TIME_GRANULARITY = DAYS
+
+  # Map / Collect over a Time range.
+  # A better implementation would be redefining 'succ' on Time. However,
+  # the ruby source code (At least 1.9.2-p0) hardcodes a check for Type,
+  # so it would not work even if we provide our own 'succ' for Time.
+  def collect(step = DEFAULT_TIME_GRANULARITY)
+    return super() unless first.is_a?(Time)
+
+    return collect(step) {|c| c} unless block_given?
+
+    # Pretty much a standard implementation of Map/Collect here
+    ary, current, op = [], first, (exclude_end? ? :< : :<=)
+    while current.send(op, last)
+      ary << yield(current)
+      current = current + step
+    end 
+    ary
+  end
+  alias :map :collect
+
+  # Diff returns the number of elements in the Range, much like 'count'.
+  # Again, redefining 'succ' would be a better idea (see above).
+  # However, I think redefining 'succ' makes this O(n) while this is O(1)
+  def diff(granularity = DEFAULT_TIME_GRANULARITY)
+    if first.is_a?(Time)
+      @diff ||= (last - first) / granularity + (exclude_end? ? 0 : 1)
+      @diff.to_i
+    else
+      @diff ||= count
+    end
+  end
+
+  # Helper methods for non default parameters
+  def hour_diff
+    diff(HOURS)
+  end
+
+  def hour_collect(&block)
+    collect(HOURS, &block)
+  end
+  alias :hour_map :hour_collect
+end

data/lib/trackoid/core_ext/time.rb

@@ -0,0 +1,46 @@
+# encoding: utf-8
+class Time
+  # Functions to construct the MongoDB field key for trackers
+  #
+  # to_i_timestamp returns the computed UTC timestamp regardless of the
+  # timezone.
+  #
+  # Examples:
+  #    2011-01-01 00:00:00 UTC  ===> 14975
+  #    2011-01-01 23:59:59 UTC  ===> 14975
+  #    2011-01-02 00:00:00 UTC  ===> 14976
+  # 
+  # to_i_hour returns the hour for the date, again regardless of TZ
+  #
+  #    2011-01-01 00:00:00 UTC  ===> 0
+  #    2011-01-01 23:59:59 UTC  ===> 23
+  #
+  ONEHOUR = 60 * 60
+  ONEDAY = 24 * ONEHOUR
+
+  def to_i_timestamp
+    self.dup.utc.to_i / ONEDAY
+  end
+
+  def to_i_hour
+    self.dup.utc.hour
+  end
+
+  # Returns an integer to use as MongoDB key
+  def to_key
+    "#{to_i_timestamp}.#{to_i_hour}"
+  end
+
+  def self.from_key(ts, h)
+    Time.at(ts.to_i * ONEDAY + h.to_i * ONEHOUR)
+  end
+
+  # Returns a range to be enumerated using hours for the whole day
+  def whole_day
+    # We could have used 'beginning_of_day' from ActiveSupport, but don't
+    # want to introduce a dependency (I've tried to avoid ActiveSupport
+    # although you will be using it since it's introduced by Mongoid)
+    midnight = Time.send(utc? ? :utc : :local, year, month, day)
+    midnight...(midnight + ::Range::DAYS)
+  end
+end

data/lib/trackoid/core_ext.rb

@@ -0,0 +1,3 @@
+# encoding: utf-8
+require 'trackoid/core_ext/time'
+require 'trackoid/core_ext/range'

data/lib/trackoid/errors.rb

@@ -13,10 +13,20 @@ module Mongoid #:nodoc
 
     class AggregationAlreadyDefined < RuntimeError
       def initialize(klass, token)
+        @klass = klass
+        @token = token
+      end
+      def message
+        "Aggregation '#{@token}' already defined for model #{@klass}"
+      end
+    end
+
+    class AggregationNameDeprecated < RuntimeError
+      def initialize(token)
         @token = token
       end
       def message
-        "Aggregation '#{@token}' already defined for model #{klass}"
+        "Ussing aggregation name '#{@klass}' is deprecated. Please select another name."
       end
     end
 

data/lib/trackoid/reader_extender.rb

@@ -0,0 +1,64 @@
+# encoding: utf-8
+module Mongoid  #:nodoc:
+  module Tracking
+
+    # ReaderExtender is used in cases where we need to return an integer
+    # (class Numeric) while extending their contents. It would allow to
+    # perform advanced calculations in some situations:
+    #
+    # Example:
+    #
+    #   a = visits.today   # Would return a number, but "extended" so that
+    #                      # we can make a.hourly to get a detailed, hourly
+    #                      # array of the visits.
+    #
+    #   b = visits.yesterday
+    #   c = a + b          # Here, in c, normally we would have a FixNum with
+    #                      # the sum of a plus b, but if we extend the sum
+    #                      # operation, we can additionaly sum the hourly
+    #                      # array and return a new ReaderExtender c.
+    #
+    class ReaderExtender
+      def initialize(number, hours)
+        @number = number
+        @hours = hours
+      end
+
+      def hourly
+        @hours
+      end
+
+      def to_s
+        @number.to_s
+      end
+
+      def ==(other)
+        @number == other
+      end
+
+      def <=>(other)
+        @number <=> other
+      end
+
+      def +(other)
+        return @number + other unless other.is_a?(ReaderExtender)
+
+        @number = @number + other
+        @hours = @hours.zip(other.hourly).map!(&:sum)
+        self
+      end
+
+      # Solution proposed by Yehuda Katz in the following Stack Overflow:
+      # http://stackoverflow.com/questions/1095789/sub-classing-fixnum-in-ruby
+      #
+      # Basically we override our methods while proxying all missing methods
+      # to the underliying FixNum
+      #
+      def method_missing(name, *args, &blk)
+        ret = @number.send(name, *args, &blk)
+        ret.is_a?(Numeric) ? ReaderExtender.new(ret, @hours) : ret
+      end
+    end
+
+  end
+end

data/lib/trackoid/readers.rb

@@ -6,11 +6,11 @@ module Mongoid  #:nodoc:
 
       # Access methods
       def today
-        data_for(Date.today)
+        whole_data_for(Time.now)
       end
 
       def yesterday
-        data_for(Date.today - 1)
+        whole_data_for(Time.now - 1.day)
       end
 
       def first_value
@@ -20,17 +20,16 @@ module Mongoid  #:nodoc:
       def last_value
         data_for(last_date)
       end
-      
+
       def last_days(how_much = 7)
         return [today] unless how_much > 0
-        date, values = Date.today, []
-        (date - how_much.abs + 1).step(date) {|d| values << data_for(d) }
-        values
+        now, hmd = Time.now, (how_much - 1)
+        on( now.ago(hmd.days)..now )
       end
 
       def on(date)
-        return date.collect {|d| data_for(d)} if date.is_a?(Range)
-        data_for(date)
+        return date.collect {|d| whole_data_for(d)} if date.is_a?(Range)
+        whole_data_for(date)
       end
 
       def all_values
@@ -39,25 +38,38 @@ module Mongoid  #:nodoc:
 
       # Utility methods
       def first_date
-        # We are guaranteed _m and _d to exists unless @data is a malformed
-        # hash, so we need to do this nasty "return nil", sorry...
-        # TODO: I'm open to change this to a cleaner algorithm :-)
-        return nil unless _y = @data.keys.min
-        return nil unless _m = @data[_y].keys.min
-        return nil unless _d = @data[_y][_m].keys.min
-        Date.new(_y.to_i, _m.to_i, _d.to_i)
-      end
-      
+        date_cleanup
+        return nil unless _ts = @data.keys.min
+        return nil unless _h = @data[_ts].keys.min
+        Time.from_key(_ts, _h)
+      end
+
       def last_date
-        # We are guaranteed _m and _d to exists unless @data is a malformed
-        # hash, so we need to do this nasty "return nil", sorry...
-        # TODO: I'm open to change this to a cleaner algorithm :-)
-        return nil unless _y = @data.keys.max
-        return nil unless _m = @data[_y].keys.max
-        return nil unless _d = @data[_y][_m].keys.max
-        Date.new(_y.to_i, _m.to_i, _d.to_i)
+        date_cleanup
+        return nil unless _ts = @data.keys.max
+        return nil unless _h = @data[_ts].keys.max
+        Time.from_key(_ts, _h)
       end
 
+      # We need the cleanup method only for methods who rely on date indexes
+      # to be valid (well formed) like first/last_date. This is because
+      # Mongo update operations cleans up the last key, which in our case
+      # left the array in an inconsistent state.
+      #
+      # Example:
+      # Before update:
+      #
+      #  { :visits_data => {"14803" => {"22" => 1} } }
+      #
+      # After updating with:  {"$unset"=>{"visits_data.14803.22"=>1}
+      #
+      #  { :visits_data => {"14803" => {} } }
+      #
+      # We can NOT retrieve the first date with visits_data.keys.min
+      #
+      def date_cleanup
+        @data.reject! {|k,v| v.count == 0}
+      end
     end
   end
 end

data/lib/trackoid/tracker.rb

@@ -34,7 +34,7 @@ module Mongoid  #:nodoc:
       end
 
       # Update methods
-      def add(how_much = 1, date = Date.today)
+      def add(how_much = 1, date = Time.now)
         raise Errors::ModelNotSaved, "Can't update a new record. Save first!" if @owner.new_record?
         return if how_much == 0
 
@@ -64,15 +64,15 @@ module Mongoid  #:nodoc:
         end
       end
 
-      def inc(date = Date.today)
+      def inc(date = Time.now)
         add(1, date)
       end
 
-      def dec(date = Date.today)
+      def dec(date = Time.now)
         add(-1, date)
       end
 
-      def set(how_much, date = Date.today)
+      def set(how_much, date = Time.now)
         raise Errors::ModelNotSaved, "Can't update a new record" if @owner.new_record?
         update_data(how_much, date)
         @owner.collection.update(
@@ -92,7 +92,7 @@ module Mongoid  #:nodoc:
         end
       end
 
-      def reset(how_much, date = Date.today)
+      def reset(how_much, date = Time.now)
         return erase(date) if how_much.nil?
 
         # First, we use the default "set" for the tracking field
@@ -111,11 +111,10 @@ module Mongoid  #:nodoc:
         end
       end
 
-      def erase(date = Date.today)
+      def erase(date = Time.now)
         raise Errors::ModelNotSaved, "Can't update a new record" if @owner.new_record?
 
-        # For the in memory data, we just need to set it to nil
-        update_data(nil, date)
+        remove_data(date)
         @owner.collection.update(
             @owner._selector,
             { "$unset" => update_hash(1, date) },
@@ -137,35 +136,58 @@ module Mongoid  #:nodoc:
 
       private
       def data_for(date)
-        return nil if date.nil?
-        date = normalize_date(date)
-        @data.try(:[], date.year.to_s).try(:[], date.month.to_s).try(:[], date.day.to_s) || 0
+        unless date.nil?
+          date = normalize_date(date)
+          @data.try(:[], date.to_i_timestamp.to_s).try(:[], date.to_i_hour.to_s) || 0
+        end
+      end
+
+      def whole_data_for(date)
+        unless date.nil?
+          date = normalize_date(date)
+          h = date.whole_day.hour_collect {|d| data_for(d)}
+          ReaderExtender.new(h.sum, h)
+        end
       end
 
       def update_data(value, date)
-        return nil if date.nil?
-        date = normalize_date(date)
-        [:year, :month].inject(@data) { |data, period|
-          data[date.send(period).to_s] ||= {}
-        }
-        @data[date.year.to_s][date.month.to_s][date.day.to_s] = value
+        unless date.nil?
+          return remove_data(date) unless value
+          date = normalize_date(date)
+          dk, hk = date.to_i_timestamp.to_s, date.to_i_hour.to_s
+          unless ts = @data[dk]
+            ts = (@data[dk] = {})
+          end
+          ts[hk] = value
+        end
       end
 
-      def year_literal(d);  "#{d.year}"; end
-      def month_literal(d); "#{d.year}.#{d.month}"; end
-      def date_literal(d);  "#{d.year}.#{d.month}.#{d.day}"; end
+      def remove_data(date)
+        unless date.nil?
+          date = normalize_date(date)
+          dk, hk = date.to_i_timestamp.to_s, date.to_i_hour.to_s
+          if ts = @data[dk]
+            ts.delete(hk)
+            unless ts.count > 0
+              @data.delete(dk)
+            end
+          end
+        end
+      end
 
       def update_hash(num, date)
         date = normalize_date(date)
         {
-          "#{@for_data}.#{date_literal(date)}" => num
+          "#{@for_data}.#{date.to_key}" => num
         }
       end
 
       def normalize_date(date)
         case date
         when String
-          Date.parse(date)
+          Time.parse(date)
+        when Date
+          date.to_time
         else
           date
         end

data/lib/trackoid/tracker_aggregates.rb

@@ -12,6 +12,7 @@ module Mongoid  #:nodoc:
         @accessor = @owner.class.send(:internal_accessor_name, @token)
         @selector = {:ns => @token}
         @selector.merge!(:key => @key) if @key
+
         @criteria = @owner.send(@accessor).where(@selector)
       end
 

data/lib/trackoid/tracking.rb

@@ -40,8 +40,11 @@ module Mongoid #:nodoc:
       # an index for the internal tracking field.
       def set_tracking_field(name)
         field internal_track_name(name), :type => Hash    # , :default => {}
-        # Should we make an index for this field?
-        index internal_track_name(name)
+
+        # DONT make an index for this field. MongoDB indexes have limited
+        # size and seems that this is not a good target for indexing.
+        # index internal_track_name(name)
+
         tracked_fields << name
       end
       

data/lib/trackoid.rb

@@ -1,8 +1,11 @@
+# encoding: utf-8
 require 'rubygems'
 
 gem "mongoid", ">= 1.9.0"
 
 require 'trackoid/errors'
+require 'trackoid/core_ext'
+require 'trackoid/reader_extender'
 require 'trackoid/readers'
 require 'trackoid/tracker'
 require 'trackoid/aggregates'

data/spec/aggregates_spec.rb

@@ -84,6 +84,16 @@ describe Mongoid::Tracking::Aggregates do
     }.should raise_error Mongoid::Errors::AggregationAlreadyDefined
   end
 
+  it "should raise error if we try to use 'hours' as aggregate" do
+    lambda {
+      class TestModel
+        aggregate :hours do
+          "(none)"
+        end
+      end
+    }.should raise_error Mongoid::Errors::AggregationNameDeprecated
+  end
+
   it "should have Mongoid accessors defined" do
     tm = TestModel.create(:name => "Dummy")
     tm.send(tm.class.send(:internal_accessor_name, "browsers")).class.should == Mongoid::Criteria
@@ -311,7 +321,6 @@ describe Mongoid::Tracking::Aggregates do
 
     it "should delete the values when using nil" do
       @mock.visits.reset(nil, "2010-07-14")
-
       @mock.visits.on("2010-07-14").should == 0
       @mock.visits.browsers.all_values.should =~ [
         ["mozilla",  [1]],
@@ -401,17 +410,17 @@ describe Mongoid::Tracking::Aggregates do
 
     it "should return the correct first_date for every aggregate" do
       @mock.visits.browsers.first_date.should =~ [
-        ["mozilla",  Date.parse("2010-07-11")],
-        ["google",   Date.parse("2010-07-12")],
-        ["internet", Date.parse("2010-07-13")]
+        ["mozilla",  Time.parse("2010-07-11")],
+        ["google",   Time.parse("2010-07-12")],
+        ["internet", Time.parse("2010-07-13")]
       ]
     end
 
     it "should return the correct last_date for every aggregate" do
       @mock.visits.browsers.last_date.should =~ [
-        ["mozilla",  Date.parse("2010-07-14")],
-        ["google",   Date.parse("2010-07-15")],
-        ["internet", Date.parse("2010-07-16")]
+        ["mozilla",  Time.parse("2010-07-14")],
+        ["google",   Time.parse("2010-07-15")],
+        ["internet", Time.parse("2010-07-16")]
       ]
     end