wvanbergen-active_olap 0.0.2

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Willem van Bergen
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,59 @@
+h1. Active OLAP
+
+This Rails plugin makes it easy to add an OLAP interface to your application, which is 
+great for administration interfaces. Its main uses are collection information about the
+usage of your application and detecting inconsistencies and problems in your data.
+
+This plugin provides:
+* The main functions for OLAP querying: olap_query and olap_drilldown. These functions
+   must be enabled for your model by calling enable_active_olap on your model class.
+* Functions to easily define dimension, categories and aggregates to use in your
+   OLAP queries.
+
+In the future, the following functionality is planned to be included:
+* A helper module to generate tables and charts for the query results. The gchartrb gem
+   is needed for charts, as they are generated using the Google charts API.
+* A controller that can be included in your Rails projects to get started quickly.
+
+More information about the concepts and usage of this plugin, see the Active OLAP Wiki on
+GitHub: http://github.com/wvanbergen/active_olap/wikis. I have blogged about this plugin 
+on the Floorplanner tech blog: http://techblog.floorplanner.com/tag/active_olap/. Finally,
+if you want to get involved or tinker with the code, you can access the repository at
+http://github.com/wvanbergen/active_olap/tree.
+
+
+h2. Why use this plugin?
+
+This plugin simply runs SQL queries using the find-method of ActiveRecord. You might be 
+wondering why you would need a plugin for that. 
+
+First of all, it makes your life as a developer easier:
+* This plugin generates the nasty SQL expressions for you using standard compliant SQL, 
+   handles issues with SQL NULL values and makes sure the results have a consistent format.
+* You can define dimensions and aggregates that are "safe to use" or known to yield useful
+   results. Once dimensions and aggregates are defined, they can be combined at will safely 
+   and without any coding effort, so it is suitable for management. :-)
+
+
+h2. Requirements
+
+This plugin is usable for any ActiveRecord-based model. Because named_scope is used for the 
+implementation, Rails 2.1 is required for it to work. It is tested to work with MySQL 5 and
+SQLite 3 but should work with other databases as well, as it only generates standard 
+compliant SQL queries.
+
+Warning: OLAP queries can be heavy on the database. They can impact the performance of your
+application if you perform them on the same server or database. Setting good indices is 
+helpful, but it may be a good idea to use a copy of the production database on another 
+server for these heavy queries.
+
+Another warning: while this plugin makes it easy to perform OLAP queries and play around
+with it, interpreting the results is hard and mistakes are easily made. At least, make sure
+to validate the results before they are used for decision making.
+
+
+h2. About this plugin
+
+The plugin is written by Willem van Bergen for Floorplanner.com. It is MIT-licensed (see 
+MIT-LICENSE). If you have any questions or want to help out with the development of this plugin,
+please contact me on willem AT vanbergen DOT org.

data/Rakefile

@@ -0,0 +1,4 @@
+Dir[File.dirname(__FILE__) + "/tasks/*.rake"].each { |file| load(file) }
+ 
+desc 'Default: run unit tests.'
+task :default => :test

data/init.rb

@@ -0,0 +1,2 @@
+$:.unshift(File.dirname(__FILE__) + '/lib')
+require 'active_olap'

data/lib/active_olap.rb

@@ -0,0 +1,116 @@
+module ActiveOLAP
+  
+  def enable_active_olap(config = nil, &block)
+
+    self.send(:extend, ClassMethods)
+    self.named_scope :olap_drilldown, lambda { |hash| self.olap_drilldown_finder_options(hash) }    
+    
+    self.cattr_accessor :active_olap_dimensions, :active_olap_aggregates
+    self.active_olap_dimensions = {}
+    self.active_olap_aggregates = {}    
+    
+    if config.nil? && block_given?
+      conf = Configurator.new(self)
+      yield(conf) 
+    end
+    
+  end
+  
+  
+  module ClassMethods
+    
+    # Performs an OLAP query that counts how many records do occur in given categories.
+    # It can be used for multiple dimensions 
+    # It expects a list of category definitions
+    def olap_query(*args)
+
+      # set aggregates apart if they are given
+      aggregates_given = (args.last.kind_of?(Hash) && args.last.has_key?(:aggregate)) ? args.pop[:aggregate] : nil
+      
+      # parse the dimensions
+      raise "You have to provide at least one dimension for an OLAP query" if args.length == 0    
+      dimensions = args.collect { |d| Dimension.create(self, d) }
+      
+      raise "Overlapping categories only supported in the last dimension" if dimensions[0..-2].any? { |d| d.has_overlap? }
+      raise "Only counting is supported with overlapping categories" if dimensions.last.has_overlap? && aggregates_given
+
+      if !aggregates_given
+        if dimensions.last.has_overlap?
+          aggregates = []
+        else
+          aggregates = [Aggregate.create(self, :the_olap_count_field, :count_distinct)]
+        end
+      else 
+        aggregates = Aggregate.all_from_olap_query_call(self, aggregates_given)        
+      end
+
+      conditions = self.send(:merge_conditions, *dimensions.map(&:conditions))
+      joins = (dimensions.map(&:joins) + aggregates.map(&:joins)).flatten.uniq
+      joins_clause = joins.empty? ? nil : self.send(:merge_joins, *joins)
+
+      selects = aggregates.map { |agg| agg.to_sanitized_sql }
+      groups  = []
+
+      if aggregates.length > 0
+        dimensions_to_group = dimensions.clone
+      else 
+        selects << dimensions.last.to_count_with_overlap_sql
+        dimensions_to_group = dimensions[0, dimensions.length - 1]
+      end
+      
+      dimensions_to_group.each_with_index do |d, index|
+        var_name = "dimension_#{index}"
+        groups  << self.connection.quote_column_name(var_name)
+        selects << d.to_case_expression(var_name)
+      end
+    
+      group_clause = groups.length > 0 ? groups.join(', ') : nil
+      # TODO: having
+      
+      olap_temporarily_set_join_type if joins_clause
+      
+      query_result = self.scoped(:conditions => conditions).find(:all, :select => selects.join(', '), 
+          :joins => joins_clause, :group => group_clause, :order => group_clause)  
+      
+      olap_temporarily_reset_join_type if joins_clause
+      
+      return Cube.new(self, dimensions, aggregates, query_result)
+    end   
+  end
+  
+  protected
+  
+  def olap_drilldown_finder_options(options)
+    raise "You have to provide at least one dimension for an OLAP query" if options.length == 0    
+
+    # returns an options hash to create a scope (the named_scope :olap_drilldown)
+    conditions = options.map { |dim, cat| Dimension.create(self, dim).sanitized_sql_for(cat) }
+    { :select => connection.quote_table_name(table_name) + '.*', :conditions => self.send(:merge_conditions, *conditions) }
+  end
+  
+  # temporarily use LEFT JOINs for specified :joins
+  def olap_temporarily_set_join_type
+    ActiveRecord::Associations::ClassMethods::InnerJoinDependency::InnerJoinAssociation.send(:define_method, :join_type) { "LEFT OUTER JOIN" }
+  end
+
+  # reset to INNER JOINs after query has finished
+  def olap_temporarily_reset_join_type
+    ActiveRecord::Associations::ClassMethods::InnerJoinDependency::InnerJoinAssociation.send(:define_method, :join_type) { "INNER JOIN" }
+  end
+  
+end
+
+
+require 'active_olap/dimension'
+require 'active_olap/category'
+require 'active_olap/aggregate'
+require 'active_olap/cube'
+require 'active_olap/configurator'
+
+require 'active_olap/helpers/display_helper'
+require 'active_olap/helpers/table_helper'
+require 'active_olap/helpers/chart_helper'
+require 'active_olap/helpers/form_helper'
+
+# inlcude the AcvtiveOLAP module in ActiveRecord::Base
+ActiveRecord::Base.send(:extend, ActiveOLAP)

data/lib/active_olap/aggregate.rb

@@ -0,0 +1,148 @@
+module ActiveOLAP
+
+  class Aggregate
+  
+    attr_reader :klass
+    attr_reader :label
+      
+    attr_reader :function
+    attr_reader :distinct
+    attr_reader :expression
+
+    attr_reader :joins
+    attr_reader :info
+
+    def self.all_from_olap_query_call(klass, aggregates_given)
+      aggregates_given = [aggregates_given] unless aggregates_given.kind_of?(Array)
+      
+      return aggregates_given.map do |aggregate_definition|
+        if aggregate_definition.kind_of?(Symbol) && klass.active_olap_aggregates.has_key?(aggregate_definition)
+          Aggregate.from_configuration(klass, aggregate_definition)
+        else
+          Aggregate.create(klass, aggregate_definition.to_sym, aggregate_definition)
+        end
+      end
+    end
+
+    def initialize(klass, label, function, expression = nil, distinct = false)
+      @klass      = klass
+      @label      = label      
+      @function   = function
+      @expression = expression
+      @distinct   = distinct
+      @joins      = []
+      @info       = {}
+    end
+    
+    
+    def self.create(klass, label, definition)
+      case definition
+      when Symbol
+        return from_symbol(klass, label, definition)
+      when String
+        return from_string(klass, label, definition)
+      when Hash
+        return from_hash(klass, label, definition)
+      else
+        raise "Invalid aggregate definition: #{definition.inspect}"
+      end
+    end
+    
+    def self.from_configuration(klass, aggregate_name, label = nil)
+      label = aggregate_name.to_sym if label.nil?
+      if klass.active_olap_aggregates[aggregate_name].respond_to?(:call)
+        return Aggregate.create(klass, label, klass.active_olap_aggregates[aggregate_name].call)
+      else
+        return Aggregate.create(klass, label, klass.active_olap_aggregates[aggregate_name])
+      end     
+    end    
+
+    def self.from_hash(klass, label, hash)
+      hash = hash.clone
+      agg = Aggregate.create(klass, label, hash.delete(:expression))
+      agg.joins.concat(hash[:joins].kind_of?(Array) ? hash.delete(:joins) : [hash.delete(:joins)]) if hash.has_key?(:joins)
+      hash.each { |key, val| agg.info[key] = val }
+      return agg
+    end
+    
+    def self.from_string(klass, label, sql_expression)
+      if sql_expression =~ /^(\w+)\((.+)\)$/
+        return Aggregate.new(klass, label, $1.downcase.to_sym, $2, false)
+      else
+        raise "Invalid aggregate SQL expression: " + sql_expression
+      end
+    end
+    
+    def self.from_symbol(klass, label, aggregate_name)
+      
+      case aggregate_name
+      when :count_all
+        return Aggregate.new(klass, label, :count, '*', false) # with table name?
+      when :count_distinct_all
+        return Aggregate.new(klass, label, :count, '*', true)  # with table name?
+      when :count
+        return Aggregate.new(klass, label, :count, :id, false)        
+      when :count_distinct
+        return Aggregate.new(klass, label, :count, :id, true)
+        
+      else
+        parts = aggregate_name.to_s.split('_')
+        raise "Invalid aggregate name: #{symbol.inspect}" unless parts.length > 1
+       
+        distinct = false
+        if parts[1] == 'distinct'
+          parts.delete_at(1) 
+          distinct = true
+        end
+      
+        raise "Invalid aggregate name: #{symbol.inspect}" unless parts.length >= 2      
+        #TODO: check field name and function name?
+        return Aggregate.new(klass, label, parts[0].to_sym, parts[1..-1].join('_').to_sym, distinct)
+      end
+    end
+    
+    def to_sanitized_sql
+      sql = @function.to_s.upcase! + '('
+      sql << 'DISTINCT ' if @distinct
+      sql << (@expression.kind_of?(Symbol) ? "#{quote_table}.#{quote_column(@expression)}" : @expression.to_s)
+      sql << ") AS #{quote_column(@label)}"
+    end
+    
+    def is_count_with_overlap?
+      @function == :count_with_overlap
+    end
+    
+    def cast_value(source)
+      return nil if source.nil?
+      (@function == :count) ? source.to_i : source.to_f # TODO: better?
+    end
+    
+    def default_value
+      (@function == :count) ? 0 : nil # TODO: better?
+    end
+    
+    def self.values(aggregates, source)
+      result = HashWithIndifferentAccess.new
+      aggregates.each { |agg| result[agg.label] = agg.cast_value(source[agg.label.to_s]) }      
+      return (aggregates.length == 1) ? result[aggregates.first.label] : result
+    end
+    
+    def self.default_values(aggregates)
+      return 0 if aggregates.empty? # count with overlap
+      result = HashWithIndifferentAccess.new
+      aggregates.each { |agg| result[agg.label] = agg.default_value }      
+      return (aggregates.length == 1) ? result[aggregates.first.label] : result
+    end
+    
+    protected
+    
+    def quote_column(column)
+      @klass.connection.send(:quote_column_name, column.to_s)
+    end
+    
+    def quote_table
+      @klass.connection.send(:quote_table_name, @klass.table_name)
+    end
+  end
+  
+end

data/lib/active_olap/category.rb

@@ -0,0 +1,46 @@
+module ActiveOLAP
+  
+  class Category
+    
+    attr_reader :dimension, :label, :conditions, :info  
+    
+    # initializes a category, given the dimension it belongs to, a label,
+    # and a definition. The definition should be a hash with at least the
+    # key expression set to a usable ActiveRecord#find conditions
+    def initialize(dimension, label, definition)
+      @dimension = dimension
+      @label = label
+      @info = {}
+      
+      if definition.kind_of?(Hash) && definition.has_key?(:expression)
+        @conditions = definition[:expression]
+        @info = definition.reject { |k,v| k == :expression } 
+      else
+        @conditions = definition
+      end
+    end
+    
+    # Returns the index of this category in the corresponding dimension
+    def index
+      @dimension.category_index(@label)
+    end
+ 
+    # Returns a santized SQL expression for this category
+    def to_sanitized_sql
+      @dimension.klass.send(:sanitize_sql, @conditions)
+    end
+    
+    def to_count_sql(count_what)
+      "COUNT(DISTINCT CASE WHEN (#{to_sanitized_sql}) THEN #{count_what} ELSE NULL END) 
+              AS #{@dimension.klass.connection.send(:quote_column_name, label.to_s)}"
+    end
+ 
+    # Returns the label of this category as a string
+    def to_s
+      return "nil" if label.nil?
+      label.to_s
+    end
+    
+  end
+  
+end

data/lib/active_olap/configurator.rb

@@ -0,0 +1,32 @@
+module ActiveOLAP
+  
+  class Configurator
+    
+    # initializes a OLAP::Configurator object, which is used in the block 
+    # passed to the call enable_active_olap. It can be used to register
+    # dimensions and classes
+    def initialize(klass)
+      @klass = klass
+    end
+    
+    # registers a dimension for the class it belongs to
+    def dimension(name, definition = nil)
+      definition = name.to_sym if definition.nil?
+      @klass.active_olap_dimensions[name] = definition
+    end
+    
+    def time_dimension(name, field, defaults = {})
+      @klass.active_olap_dimensions[name] = Proc.new do |*options|
+        options = options.empty? ? {} : options.first
+        { :trend => defaults.merge(options).merge(:timestamp_field => field) }
+      end
+    end
+    
+    # registers an aggregate for the class it belongs to
+    def aggregate(name, definition = nil, options = {})
+      definition = name if definition.nil?
+      agg_definition = options.merge(:expression => definition)
+      @klass.active_olap_aggregates[name] = agg_definition
+    end
+  end
+end

data/lib/active_olap/cube.rb

@@ -0,0 +1,215 @@
+module ActiveOLAP
+  
+  class Cube
+    
+    attr_accessor :info
+    attr_accessor :klass
+    attr_accessor :dimensions
+    attr_accessor :aggregates
+    
+    # Initializes a new OLAP cube. 
+    def initialize(klass, dimensions, aggregates, query_result = nil)
+      @klass      = klass
+      @dimensions = dimensions
+      @aggregates = aggregates
+      @info = {}
+      
+      # populates the cube with the query rsult if it is provided.
+      unless query_result.nil?
+        @result = []        
+        populate_result_with(query_result)
+        traverse_result_for_nils(@result)
+      end
+    end
+    
+    # Sums up all the values in this cube
+    def sum(agg = nil)
+      raise "Please provide the aggregate you want to sum." if self.aggregates.length > 1 && agg.nil?
+      total_sum = 0
+      self.each do |cat, value| 
+          total_sum += (value.kind_of?(Cube) ? value.sum : (agg.nil? ? value : value[agg]))
+      end
+      return total_sum
+    end
+    
+    # Returns the total number of cells in this cube. Note that this does not take aggregates into
+    # account, so the result should me multiplied by the number of aggregates if you want to know
+    # the total number of (numeric) values.
+    def cell_count
+      dimensions.inject(1) { |intermediate, dimension| intermediate * dimension.categories.length }
+    end
+    
+    # Returns a reference to the internal array that holds ther raw results of this cube. 
+    # Altering this array will alter the internals of the cube-object, so make sure you know what 
+    # you are doing. Use to_a if you want to obtain a copy of the internal array
+    def raw_results 
+      @result
+    end
+    
+    # Returns a clone of the internal array that holds ther raw results of this cube.
+    def to_a
+      @result.clone
+    end
+    
+    # Returns the array of categories of the current (= first) dimension
+    def categories
+      @dimensions.first.categories
+    end
+    
+    # Returns the current (first) dimension
+    def dimension 
+      @dimensions.first
+    end
+    
+    # Returns the number of dimensions in this cube
+    def depth
+      @dimensions.length
+    end
+    
+    # Returns the number of categories in the current (= first) dimension
+    def breadth
+      @result.length
+    end
+     
+    # Switches the dimensions of a two-dimensional cube   
+    def transpose
+      raise "Can only transpose 2-dimensial results" unless depth == 2
+      result_object = Cube.new(@klass, [@dimensions.last, @dimensions.first], @aggregates)
+      result_object.result = @result.transpose
+      return result_object      
+    end
+    
+    def reorder_dimensions(*order)
+      # IMPLEMENT ME      
+    end
+    
+    def only_aggregate(aggregate_label)
+      # IMPLEMENT ME
+    end
+    
+    def only_dimension(dimension_index)
+      # IMPLEMENT ME
+    end
+    
+    def except_dimension(dimension_index)
+      # IMPLEMENT ME
+    end
+    
+    # Returns a part of the cube or a single cell
+    # If the number of arguments matches the number of dimensions, a single cell is returned;
+    # If the number of arguments is less that the number of dimensons, this function will return 
+    # a cube with (dimensions.length - args.length) dimensions.
+    def [](*args)
+      result = @result.clone
+      args.each_with_index do |cat_label, index|
+        cat_index = @dimensions[index].category_index(cat_label)
+        return nil if cat_index.nil?
+        result = result[cat_index]
+      end
+      
+      if result.kind_of?(Array)
+        # build a new query_result object if not enoug dimensions were provided
+        result_object = Cube.new(@klass, @dimensions[args.length...@dimensions.length], @aggregates)
+        result_object.result = result
+        return result_object
+      else
+        return result
+      end
+    end
+    
+    # Iterates over all the categories of the current dimension and their corresponding values.
+    # The provided block should take two arguments: the first one will be the Category and
+    # the second one will be the sub-cube or cell belonging to that category
+    def each(&block)
+     categories.each { |cat| yield(cat, self[cat.label]) }
+    end
+    
+    # Maps all the categoies of the current dimension and their corresponding values. See each.
+    def map(&block)
+      result = []
+      categories.each { |cat| result << yield(cat, self[cat.label]) }
+      return result
+    end
+    
+    protected
+
+    # Set the result by hand. For internal use
+    def result=(array)
+      @result = array
+    end
+
+    # Walks over all the rows of the resultset to build the cube.
+    def populate_result_with(query_result)
+      
+      query_result.each do |row|
+        
+        result = @result
+        values = row.attributes_before_type_cast
+        discard_data = false
+                
+        (@dimensions.length - 1).times do |dim_index|
+          
+          category_name = values.delete("dimension_#{dim_index}")
+          if @dimensions[dim_index].is_field_dimension?
+            # this field contains the value of the category_field, which should be used as category
+            # this might be the first time this category is seen, so register it in the dimension
+            category_index = @dimensions[dim_index].register_category(category_name)
+            
+          elsif category_name.nil?
+            # this is a record for rows that did not fall in any of the categories of a dimension
+            # therefore, this data can be discarded. This should not happen if an "other"-field is present!
+            discard_data = true
+            break 
+            
+          else
+            # get the index of the category, which should exist
+            category_index = @dimensions[dim_index].category_index(category_name.to_sym)
+          end 
+          
+          # switch the result to the next dimension
+          result[category_index] = [] if result[category_index].nil? # add a new dimension if needed
+          result = result[category_index] # set the result to the next dimension for the next iteration
+        end
+        
+        unless discard_data
+          dim = @dimensions.last # only the last dimension is remaining
+          if dim.is_field_dimension?
+            # the last dimension is a field category.
+            # every category is represented as a single row, with only one count per row
+            dimension_field_value = values["dimension_#{@dimensions.length - 1}"]
+            result[dim.register_category(dimension_field_value)] = Aggregate.values(@aggregates, values)
+            
+          elsif aggregates.length == 0
+            # the last dimension is a category with possible overlap, using SUMs.
+            # every category will have its number on this row
+            result = [] if result.nil?
+            values.each { |key, value| result[dim.category_index(key.to_sym)] = value.to_i }
+            
+          else
+            # the last category is a normal category
+            dimension_field_value = values["dimension_#{@dimensions.length - 1}"]
+            result[dim.category_index(dimension_field_value.to_sym)] = Aggregate.values(@aggregates, values) unless dimension_field_value.nil?
+          end
+        end
+      end
+    end
+    
+    # Makes sure all the values are set in the resulting array
+    def traverse_result_for_nils(result, depth = 0)
+      dim = @dimensions[depth]
+      if dim == @dimensions.last
+        # set all categories to 0 if no value is set
+        dim.categories.length.times do |i|
+          result[i] = Aggregate.default_values(@aggregates) if result[i].nil?
+        end
+      else
+        # if no value set, create an empty array and iterate to the next dimension
+        # so all values will be set to 0
+        dim.categories.length.times do |i|
+          result[i] = [] if result[i].nil?
+          traverse_result_for_nils(result[i], depth + 1)
+        end        
+      end
+    end
+  end
+end