cubicle 0.1.29 → 0.1.30

data/CHANGELOG.rdoc

@@ -1,122 +1,125 @@
-==0.1.29
-  *Enabled custom configuring cubicle's mongo connection even if MongoMapper is present, to allow running map reduce jobs
-  on a slave while MongoMapper operates on the master.
-
-==0.1.28
-  *Updated to latest versions of MongoMapper, Mongo & BSON
-==0.1.26,0.1.27
-  *Included mustache dependency in gem (whoops) and fixed a bug where an error in cube processing would replace
-   useful error information with nonsensical error information.
-
-==0.1.25
-  *Modified data extraction to ensure that member and query level aliases are respected in the final result. Also
-  stopped automatically dropping temporary map reduce tables during query execution for performance reasons. Instead,
-  added a static method to the Cubicle module, clear_temp_tables, which can be called at application startup or teardown:
-  Cubicle.clear_temp_tables()
-
-==0.1.24
-  *Added more detail to the profile for the 'find' action
-
-==0.1.23
- *Fixed profiler's capped collection sizing bug (off by x1000)
-
-==0.1.22
-  *Added profiler, which is off by default, that will profile every operation performed by the aggregation engine
-  in a collection called cubicle.profiler. It uses a capped collection size at, by default, 250mb, but this can
-  be configured. Also added basic metadata table for overall aggregation definitions that will record the last processing
-  time, along with the duration of the processing job, in the collection cubicle.metadata
-
-==0.1.21
-  *Added metadata tables in the database for cubicle to manage aggregation info. This was necessary because previously
-  I was trying to overload the collection name with metadata, which was making the names longer than MongoDb could support
-  and causing errors. This change will enable richer monitoring and profiling and optimization in the near future.
-
-==0.1.20
-  *Updated to work with mongo driver 1.0 (and therefore latest versions of MongoMapper)
-
-==0.1.19
-   *Fixed bug that caused cubicle to hang when grouping by days
-
-==0.1.18
-  *Bug fixes
-
-==0.1.16
-  *Added a set of default constants for expression templates, such as date_today, date_today_iso, date_today_utc_iso, etc.
-  Also added the 'define' method to the query DSL (instead of simply the aggregation DSL) so that you can either define new
-  named expressions or override an existing named expression or date constant in a query context, instead of only at the
-  global level
-
-==0.1.15
-  *Added 'difference' calculated measure to aggregation DSL, used in the same was as 'ratio' - so, in your aggregation
-  definition, you can do 'difference :big_diff, :measure_one, :measure_two'
-  and the value of :big_diff will be measure_one - measure_two. Also monkey patched Mustache (per suggestion in the
-  mailing list) to turn of HTML escaping, so that the triple-curly syntax should not be required when using operators
-  in templated expressions.
-
-==0.1.14
-  *Incorporated the Mustache templating library into aggregation definitions allowing expression to reference previously
-   defined dimensions, measures, or named expressions (using the syntax: define :expression_name, 'this.my_expression != "that"')
-   Allows for definitions such as this:  avg :average_happy_defects, :expression=>'{{defects}} && this.type=="Happy"'
-   Also available is 'time_now' to represent the Ruby's Time.now in the form of a javascript date, so '{{time_now}}' will
-   become 'new Date(2342342)' where 2342342 is Time.now.to_i
-
-==0.1.13
-  *Previous feature broke ratios. Fixed now.
-
-==0.1.12
-  *Added distinct' option to 'count' measure, to allow measures which represent distinct counts of  given field/expression value.
-  ONLY works when the value being distinctly counted is a string.
-
-==0.1.11
-  *Fixed a bug when filtering categorized dimensions in transient queries
-
-==0.1.10
-  *enhanced cubicle aggregation manager to transparently convert several of the basic mongo operators (such as $ne,
-  $gt, $gte, etc) into javascript for use in the generated $where clause for transient queries. In practice, this
-  means that you can define a measure as an expression, say dimension :month, :expression=>"'2010-01'", and then
-  use that dimension name in a standard mongo driver compatible filter expression, without concern for whether
-  your query is a transient query or a regular query - cubicle will do the appropriate conversion to javascript
-  in the former case, translating :month=>{"$gte"=>"'2009-12'", "$lt"=>"'2010-02'"} into the functional equivalent of
-  "$where"=>"'2010-01' >= '2009-12' && '2010-01' < '2010-02'". Actually, an inline function is used to reduce IO
-  as repeating the expression for each comparison can get out of hand if your expression is large.
-
-==0.1.9
-  * Fixed problems with missing number in bucket intervals. Moved resolution of source measure javascript to
-  * last moment, to ensure dynamic values don't get accidentally cached too early.
-
-==0.1.8
- * Added 'bucketize' aliased as 'categorize' to cubicle DSL to allow dimensions to be created from bucketizing
-   (categorizing based on name) a measure value.
-
-==0.1.7
- * Minor bug fixes
-
-==0.1.6
- *Added generic 'condition' argument to member definitions, which allows the specification of a JavaScript expression
-  which will determine if the measure (or dimension) expression will be evaluated. This capability was available
-  before via the :expression option, but using a :condition modifier allows for more straightforward (readable) measure
-  definitions and separates the definition of the actual value from the conditions under which the value is valid.
-
-==0.1.5
- * Added duration_since (aliased as elapsed, age_since) to calculate the time elapsed since any
-   given timestamp and the current execution time of the query.
-
-== 0.1.4
- * Cached aggregations were being stored on disk, but not properly utilized. This is now fixed.
-
-== 0.1.3
-* Formalized flat (Cubicle::Data::Table) and hierarchical (Cubicle::Data::Hierarchy) data results from cubicle queries,
-  and added client side support for rolling up measure values in hierarchical data so that no matter how a given
-  query is organized hierarchically, a summary of aggregated measure values for each level of data will be available. 
-* Re-organized codebase
-* Bug fixes
-
-== 0.1.2
-* Added ability to calculate durations between two timestamps
-* Bug fixes
-
-== 0.1.1
-* Fixed a bug that required a logger to be initialized for the thing to work
-
-== 0.1.0
-* Initial release
+==0.1.30
+  *Added duration in weeks.
+  
+==0.1.29
+  *Enabled custom configuring cubicle's mongo connection even if MongoMapper is present, to allow running map reduce jobs
+  on a slave while MongoMapper operates on the master.
+
+==0.1.28
+  *Updated to latest versions of MongoMapper, Mongo & BSON
+==0.1.26,0.1.27
+  *Included mustache dependency in gem (whoops) and fixed a bug where an error in cube processing would replace
+   useful error information with nonsensical error information.
+
+==0.1.25
+  *Modified data extraction to ensure that member and query level aliases are respected in the final result. Also
+  stopped automatically dropping temporary map reduce tables during query execution for performance reasons. Instead,
+  added a static method to the Cubicle module, clear_temp_tables, which can be called at application startup or teardown:
+  Cubicle.clear_temp_tables()
+
+==0.1.24
+  *Added more detail to the profile for the 'find' action
+
+==0.1.23
+ *Fixed profiler's capped collection sizing bug (off by x1000)
+
+==0.1.22
+  *Added profiler, which is off by default, that will profile every operation performed by the aggregation engine
+  in a collection called cubicle.profiler. It uses a capped collection size at, by default, 250mb, but this can
+  be configured. Also added basic metadata table for overall aggregation definitions that will record the last processing
+  time, along with the duration of the processing job, in the collection cubicle.metadata
+
+==0.1.21
+  *Added metadata tables in the database for cubicle to manage aggregation info. This was necessary because previously
+  I was trying to overload the collection name with metadata, which was making the names longer than MongoDb could support
+  and causing errors. This change will enable richer monitoring and profiling and optimization in the near future.
+
+==0.1.20
+  *Updated to work with mongo driver 1.0 (and therefore latest versions of MongoMapper)
+
+==0.1.19
+   *Fixed bug that caused cubicle to hang when grouping by days
+
+==0.1.18
+  *Bug fixes
+
+==0.1.16
+  *Added a set of default constants for expression templates, such as date_today, date_today_iso, date_today_utc_iso, etc.
+  Also added the 'define' method to the query DSL (instead of simply the aggregation DSL) so that you can either define new
+  named expressions or override an existing named expression or date constant in a query context, instead of only at the
+  global level
+
+==0.1.15
+  *Added 'difference' calculated measure to aggregation DSL, used in the same was as 'ratio' - so, in your aggregation
+  definition, you can do 'difference :big_diff, :measure_one, :measure_two'
+  and the value of :big_diff will be measure_one - measure_two. Also monkey patched Mustache (per suggestion in the
+  mailing list) to turn of HTML escaping, so that the triple-curly syntax should not be required when using operators
+  in templated expressions.
+
+==0.1.14
+  *Incorporated the Mustache templating library into aggregation definitions allowing expression to reference previously
+   defined dimensions, measures, or named expressions (using the syntax: define :expression_name, 'this.my_expression != "that"')
+   Allows for definitions such as this:  avg :average_happy_defects, :expression=>'{{defects}} && this.type=="Happy"'
+   Also available is 'time_now' to represent the Ruby's Time.now in the form of a javascript date, so '{{time_now}}' will
+   become 'new Date(2342342)' where 2342342 is Time.now.to_i
+
+==0.1.13
+  *Previous feature broke ratios. Fixed now.
+
+==0.1.12
+  *Added distinct' option to 'count' measure, to allow measures which represent distinct counts of  given field/expression value.
+  ONLY works when the value being distinctly counted is a string.
+
+==0.1.11
+  *Fixed a bug when filtering categorized dimensions in transient queries
+
+==0.1.10
+  *enhanced cubicle aggregation manager to transparently convert several of the basic mongo operators (such as $ne,
+  $gt, $gte, etc) into javascript for use in the generated $where clause for transient queries. In practice, this
+  means that you can define a measure as an expression, say dimension :month, :expression=>"'2010-01'", and then
+  use that dimension name in a standard mongo driver compatible filter expression, without concern for whether
+  your query is a transient query or a regular query - cubicle will do the appropriate conversion to javascript
+  in the former case, translating :month=>{"$gte"=>"'2009-12'", "$lt"=>"'2010-02'"} into the functional equivalent of
+  "$where"=>"'2010-01' >= '2009-12' && '2010-01' < '2010-02'". Actually, an inline function is used to reduce IO
+  as repeating the expression for each comparison can get out of hand if your expression is large.
+
+==0.1.9
+  * Fixed problems with missing number in bucket intervals. Moved resolution of source measure javascript to
+  * last moment, to ensure dynamic values don't get accidentally cached too early.
+
+==0.1.8
+ * Added 'bucketize' aliased as 'categorize' to cubicle DSL to allow dimensions to be created from bucketizing
+   (categorizing based on name) a measure value.
+
+==0.1.7
+ * Minor bug fixes
+
+==0.1.6
+ *Added generic 'condition' argument to member definitions, which allows the specification of a JavaScript expression
+  which will determine if the measure (or dimension) expression will be evaluated. This capability was available
+  before via the :expression option, but using a :condition modifier allows for more straightforward (readable) measure
+  definitions and separates the definition of the actual value from the conditions under which the value is valid.
+
+==0.1.5
+ * Added duration_since (aliased as elapsed, age_since) to calculate the time elapsed since any
+   given timestamp and the current execution time of the query.
+
+== 0.1.4
+ * Cached aggregations were being stored on disk, but not properly utilized. This is now fixed.
+
+== 0.1.3
+* Formalized flat (Cubicle::Data::Table) and hierarchical (Cubicle::Data::Hierarchy) data results from cubicle queries,
+  and added client side support for rolling up measure values in hierarchical data so that no matter how a given
+  query is organized hierarchically, a summary of aggregated measure values for each level of data will be available. 
+* Re-organized codebase
+* Bug fixes
+
+== 0.1.2
+* Added ability to calculate durations between two timestamps
+* Bug fixes
+
+== 0.1.1
+* Fixed a bug that required a logger to be initialized for the thing to work
+
+== 0.1.0
+* Initial release

data/cubicle.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{cubicle}
-  s.version = "0.1.29"
+  s.version = "0.1.30"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Nathan Stults"]
-  s.date = %q{2010-07-06}
+  s.date = %q{2010-07-07}
   s.description = %q{Cubicle provides a dsl and aggregation caching framework for automating the generation, execution and caching of map reduce queries when using MongoDB in Ruby. Cubicle also includes a MongoMapper plugin for quickly performing ad-hoc, multi-level group-by queries against a MongoMapper model.}
   s.email = %q{hereiam@sonic.net}
   s.extra_rdoc_files = [

data/lib/cubicle/duration.rb

@@ -1,52 +1,53 @@
-module Cubicle
-  class Duration < Measure
-    attr_accessor :duration_unit, :begin_milestone, :end_milestone, :timestamp_prefix
-
-    def initialize(*args)
-      super
-      self.duration_unit = options(:in) || :seconds
-      self.timestamp_prefix = options :timestamp_prefix, :prefix
-      self.expression_type = :javascript
-      #only one item should be left in the hash, the duration map
-      raise "duration must be provided with a hash with a single entry, where the key represents the starting milestone of a duration and the value represents the ending milestone." if options.length != 1
-
-      self.begin_milestone, self.end_milestone = options.to_a[0]
-      self.name ||= "#{begin_milestone}_to_#{end_milestone}_#{aggregation_method}".to_sym
-    end
-
-    def default_aggregation_method
-      :average
-    end
-
-    def condition
-      cond = " && (#{super})" unless super.blank?
-      "#{milestone_js(:begin)} && #{milestone_js(:end)}#{cond}"
-    end
-
-    def expression
-      #prefix these names for the expression
-#      prefix = "#{self.timestamp_prefix}#{self.timestamp_prefix.blank? ? '' : '.'}"
-#      ms1,ms2 = [self.begin_milestone,self.end_milestone].map{|ms|ms.to_s=='now' ? "new Date(#{Time.now.to_i*1000})" : "this.#{prefix}#{ms}"}
-      @expression = "(#{milestone_js(:end)}-#{milestone_js(:begin)})/#{denominator}" 
-    end
-
-    private
-
-    def milestone_js(which)
-     prefix = "#{self.timestamp_prefix}#{self.timestamp_prefix.blank? ? '' : '.'}"
-     ms = self.send("#{which.to_s}_milestone")
-     ms.to_s=='now' ? "new Date(#{Time.now.to_i*1000})" : "this.#{prefix}#{ms}"
-    end
-
-    def denominator
-      #Date math results in milliseconds in javascript
-      case self.duration_unit || :seconds
-        when :days    then "1000/60/60/24.0"
-        when :hours   then "1000/60/60.0"
-        when :minutes then "1000/60.0"
-        else               "1000.0"
-      end
-    end
-
-  end
+module Cubicle
+  class Duration < Measure
+    attr_accessor :duration_unit, :begin_milestone, :end_milestone, :timestamp_prefix
+
+    def initialize(*args)
+      super
+      self.duration_unit = options(:in) || :seconds
+      self.timestamp_prefix = options :timestamp_prefix, :prefix
+      self.expression_type = :javascript
+      #only one item should be left in the hash, the duration map
+      raise "duration must be provided with a hash with a single entry, where the key represents the starting milestone of a duration and the value represents the ending milestone." if options.length != 1
+
+      self.begin_milestone, self.end_milestone = options.to_a[0]
+      self.name ||= "#{begin_milestone}_to_#{end_milestone}_#{aggregation_method}".to_sym
+    end
+
+    def default_aggregation_method
+      :average
+    end
+
+    def condition
+      cond = " && (#{super})" unless super.blank?
+      "#{milestone_js(:begin)} && #{milestone_js(:end)}#{cond}"
+    end
+
+    def expression
+      #prefix these names for the expression
+#      prefix = "#{self.timestamp_prefix}#{self.timestamp_prefix.blank? ? '' : '.'}"
+#      ms1,ms2 = [self.begin_milestone,self.end_milestone].map{|ms|ms.to_s=='now' ? "new Date(#{Time.now.to_i*1000})" : "this.#{prefix}#{ms}"}
+      @expression = "(#{milestone_js(:end)}-#{milestone_js(:begin)})/#{denominator}" 
+    end
+
+    private
+
+    def milestone_js(which)
+     prefix = "#{self.timestamp_prefix}#{self.timestamp_prefix.blank? ? '' : '.'}"
+     ms = self.send("#{which.to_s}_milestone")
+     ms.to_s=='now' ? "new Date(#{Time.now.to_i*1000})" : "this.#{prefix}#{ms}"
+    end
+
+    def denominator
+      #Date math results in milliseconds in javascript
+      case self.duration_unit || :seconds
+        when :weeks   then "1000/60/60/24/7.0"
+        when :days    then "1000/60/60/24.0"
+        when :hours   then "1000/60/60.0"
+        when :minutes then "1000/60.0"
+        else               "1000.0"
+      end
+    end
+
+  end
 end

data/lib/cubicle/version.rb

@@ -1,3 +1,3 @@
-module Cubicle
-  VERSION = '0.1.29'
+module Cubicle
+  VERSION = '0.1.30'
 end

data/test/cubicle/duration_test.rb

@@ -3,8 +3,8 @@ require "test_helper"
 class DurationTest < ActiveSupport::TestCase
   context "Querying a cubicle with durations" do
     setup do
-      #record 1, ms1 = 1/1/2000, ms2 = 1/2/2000, ms3=1/4/2000
-      #record 2, ms1 = 1/1/2000, ms2 = 1/3/2000, ms3=1/5/2000
+      #record 1, ms1 = 1/1/2000, ms2 = 1/2/2000, ms3=1/4/2000, ms4=1/23/2000
+      #record 2, ms1 = 1/1/2000, ms2 = 1/3/2000, ms3=1/5/2000, ms4=1/29/2000
       Defect.create_duration_test_data
     end
     should "correctly calculate durations for each activity" do
@@ -18,10 +18,12 @@ class DurationTest < ActiveSupport::TestCase
       assert_equal 1 * 60 * 60 * 24, a1["ms1_to_ms2_average"]
       assert_equal 2 * 60 * 60 * 24, a1["ms2_to_ms3_sum"]
       assert_equal 3,                a1["total_duration"]
+      assert_equal 22/7.0,           a1["total_duration_in_weeks"]
 
       assert_equal 2 * 60 * 60 * 24, a2["ms1_to_ms2_average"]
       assert_equal 2 * 60 * 60 * 24, a2["ms2_to_ms3_sum"]
       assert_equal 4,                a2["total_duration"]
+      assert_equal 28/7.0,           a2["total_duration_in_weeks"]
     end
     should "correctly aggregate durations" do
       results = DefectCubicle.query do

data/test/cubicles/defect_cubicle.rb

@@ -1,57 +1,58 @@
-class DefectCubicle
-  extend Cubicle::Aggregation
-
-  define     :preventable, "this.root_cause != 'act_of_god'"
-  define     :product_name, "this.product.name"
-  define     :current_year, "'{{date_today_iso}}'.substring(0,4)"
-
-  date       :manufacture_date,  :field_name=>'manufacture_date', :alias=>:date
-  dimension  :month,             :expression=>'this.manufacture_date.substring(0,7)'
-  dimension  :year,              :expression=>'this.manufacture_date.substring(0,4)'
-
-  dimension  :manufacture_time
-
-  dimension  :product,           :expression=>'{{product_name}}'
-  dimension  :region,            :field_name=>'plant.address.region'
-
-  dimensions :operator, :outcome
-  
-  count :total_defects,          :field_name=>'defect_id'
-  count :distinct_products,      :expression=>'{{product}}', :distinct=>true
-  count :preventable_defects,    :expression=>'{{preventable}}'
-  count :conditioned_preventable,:expression=>'1.0', :condition=>'{{preventable}}'
-  count :defects_this_year,      :expression=>'{{year}} == {{current_year}}'
-  sum   :total_cost,             :field_name=>'cost'
-  avg   :avg_cost,               :field_name=>'cost'
-
-  #calculated fields
-  ratio      :preventable_pct,  :preventable_defects, :total_defects
-  ratio      :distinct_ratio,   :distinct_products,   :total_defects
-  difference :inevitable_defects, :total_defects, :preventable_defects
-  #durations
-  average_duration :ms1 => :ms2
-  total_duration :ms2 => :ms3
-  duration  :total_duration, :ms1 => :ms3, :in=>:days
-  duration  :conditional_duration, :ms1 => :ms3, :in=>:days, :condition=>"this.defect_id != 2"
-  elapsed   :ms3, :in=>:days
-  age_since :avg_time_since_ms3,:ms3, :in=>:days
-
-  #bucketized fields
-  categorize :avg_cost_category,
-             :avg_cost, 1..5, :bucket_size=>0.5, :range_start_bump=>0.01 do |bucket_start,bucket_end|
-    if bucket_start == :begin
-      '< $1'
-    elsif bucket_end == :end
-      '> $5'
-    else
-      "$#{bucket_start} - $#{bucket_end}"
-    end
-  end
-
-
-
-  #pre-cached aggregations
-  aggregation :month, :year, :product
-  aggregation :month, :region
-
+class DefectCubicle
+  extend Cubicle::Aggregation
+
+  define     :preventable, "this.root_cause != 'act_of_god'"
+  define     :product_name, "this.product.name"
+  define     :current_year, "'{{date_today_iso}}'.substring(0,4)"
+
+  date       :manufacture_date,  :field_name=>'manufacture_date', :alias=>:date
+  dimension  :month,             :expression=>'this.manufacture_date.substring(0,7)'
+  dimension  :year,              :expression=>'this.manufacture_date.substring(0,4)'
+
+  dimension  :manufacture_time
+
+  dimension  :product,           :expression=>'{{product_name}}'
+  dimension  :region,            :field_name=>'plant.address.region'
+
+  dimensions :operator, :outcome
+  
+  count :total_defects,          :field_name=>'defect_id'
+  count :distinct_products,      :expression=>'{{product}}', :distinct=>true
+  count :preventable_defects,    :expression=>'{{preventable}}'
+  count :conditioned_preventable,:expression=>'1.0', :condition=>'{{preventable}}'
+  count :defects_this_year,      :expression=>'{{year}} == {{current_year}}'
+  sum   :total_cost,             :field_name=>'cost'
+  avg   :avg_cost,               :field_name=>'cost'
+
+  #calculated fields
+  ratio      :preventable_pct,  :preventable_defects, :total_defects
+  ratio      :distinct_ratio,   :distinct_products,   :total_defects
+  difference :inevitable_defects, :total_defects, :preventable_defects
+  #durations
+  average_duration :ms1 => :ms2
+  total_duration :ms2 => :ms3
+  duration  :total_duration, :ms1 => :ms3, :in=>:days
+  duration  :total_duration_in_weeks, :ms1 => :ms4, :in=>:weeks
+  duration  :conditional_duration, :ms1 => :ms3, :in=>:days, :condition=>"this.defect_id != 2"
+  elapsed   :ms3, :in=>:days
+  age_since :avg_time_since_ms3,:ms3, :in=>:days
+
+  #bucketized fields
+  categorize :avg_cost_category,
+             :avg_cost, 1..5, :bucket_size=>0.5, :range_start_bump=>0.01 do |bucket_start,bucket_end|
+    if bucket_start == :begin
+      '< $1'
+    elsif bucket_end == :end
+      '> $5'
+    else
+      "$#{bucket_start} - $#{bucket_end}"
+    end
+  end
+
+
+
+  #pre-cached aggregations
+  aggregation :month, :year, :product
+  aggregation :month, :region
+
 end