fuzzy_infer 0.0.1 → 0.0.2

data/CHANGELOG

@@ -0,0 +1,11 @@
+== 0.0.2 / 2012-04-02
+
+* Enhancements
+
+  * Allow inferring multiple values at once, e.g. CBECS.fuzzy_infer(:foo, :bar, :baz) for speed
+  * Combine SQL queries where possible for speed
+  * Truly unique column names so we don't step on any toes
+
+== 0.0.1 / 2012-02-21
+
+* Initial release

data/Gemfile

@@ -7,4 +7,4 @@ gem 'minitest'
 gem 'minitest-reporters'
 gem 'mysql2'
 gem 'earth'
-gem 'pg'
+gem 'pg'

data/README.markdown

@@ -4,7 +4,79 @@
 
 * [Brighter Planet CM1 Impact Estimate web service](http://impact.brighterplanet.com) 
 
-## setup
+## What it does
+
+FuzzyInfer predicts one or more unknown characteristics of an input case by comparing its known characteristics to a reference dataset whose records contain both the known and unknown characteristics. It weights these records according to how closely they match the input case on known characteristics, and then performs a weighted average of the records to predict the unknown characteristics.
+
+## Tuning your parameters
+
+As you're iteratively tinkering with the two equations for Sigma and compound weighting scheme, it may be helpful to monitor the effects of various options on the distribution of membership weights, both for each variable and for the final compound weights. It's important to note that the two equations aren't independent, such that they must be developed in tandem to get the desired results across the range of possible input values.
+
+#### Determining Sigma:
+
+The value of Sigma determines the width of the membership function curve for a given variable, and hence the number of records that will be taken into account. Tuning Sigma to fit your desired results for your application is a subjective process based on how wide a net you want to cast around your input value.
+
+Sigma does NOT have to be the same for all variables in the fuzzy analysis -- you can tweak it independently for each variable.
+
+Sigma can be coded as either:
+
+* A constant.
+* A function of all X values.
+  * In this case a reasonable default is to set Sigma equal to the standard deviation of variable X. This ensures that regardless of the range covered by X, your fuzzy membership function will capture a nice subset of the records.
+* A function both of all X values and of Mu.
+
+Remember that if compound fuzzy weighting is being employed to analyze multiple variables (see below), the effect of your weight compounding formula on the width of your final net should be accounted for, as it will either expand or contract the net.
+
+##### Case study
+
+Brighter Planet's [lodging model](http://impact.brighterplanet.com/models/lodging) uses compound fuzzy inference on six variables in the US EIA CBECS data set to predict hotel energy use.
+
+1. To keep things simple we developed an equation for Sigma that works for all six variables, but we could have tweaked them individually if we'd wanted.
+2. We started by setting Sigma at `STDDEV_SAMP(x)`.
+3. For most Mu values this ended up including too many records for our taste, so we experimented with fractions of the standard deviation of x, like `STDDEV_SAMP(x) / 3`.
+4. This worked ok for Mu values close to the mean, but for Mu values near or beyond the range limits of the CBECS data set where records were much more sparse, it wasn't capturing enough records for a reasonable sample size.
+5. We decided to develop a formula that increased the value of Sigma (the width of the net) as Mu got farther away from the mean. This would allow us to cast a wide net when dealing with sparse edge cases but a narrow net when Mu is in the densely-populated center of the data set. We accomplished this by incorporating `ABS(AVG(X) - Mu)` into our equation.
+6. After some more tinkering--including looking at the number of records that would ultimately be highly-weighted for various input combinations after our particular weight compounding formula was applied--we settled on a final formula to use for our Sigma equation for all of our variables: `STDDEV_SAMP(x) / 5 + (ABS(AVG(x) - Mu) / 3)`.
+
+#### Determining compound weighting scheme:
+
+The compound weighting formula comes into play when performing fuzzy inferences based on more than one known variable.
+
+For each record in your reference data set, it takes its calculated normalized membership value for each variable you're using for inference, and consolidates those weights into a single membership value for that record.
+
+Note that these starting weights are all between 0 and 1, and that they've been normalized so that the weight for the heaviest record is exactly 1.
+
+How you want to determine your final membership weights depends on a couple key questions:
+
+* How exclusive you want your fuzzy analysis to be, i.e. how closely a record must match your input case to be worthy of a relatively high weight
+* The relative importance of the variables that are driving your inference scheme, i.e. whether you give more predictive credence to some than others
+
+You could do all kinds of fun formulas, but there are two primary operations for compounding variable weights:
+
+###### Addition
+* This is the more inclusive operation, as all records that resemble the input case in at least one respect will be given some weight.
+* Relative variable weights can be incorporated by multiplying the variable's weight by the record's membership value during addition.
+
+###### Multiplication
+* This is the more exclusive operation, as a record that's dissimilar to the input case in any respect will end up with a low final weight.
+* Relative variable weights can be incorporated by raising the record's membership value to the power of that variable's weight during multiplication.
+  * Note that since membership values are between 0 and 1, raising them to higher powers produces lower values, such that more important variables should be raised to lesser powers than unimportant variables.
+  * Also note that unlike in additive compounding, with multiplicative compounding the absolute (not just relative) values of the variable do affect the final distribution of weights -- raising all your variables to the power of 2 is NOT the same as raising all your variables to the power of 0.5.
+
+These two primary operations can be mixed as needed to meet the needs of your particular application. A combination of addition and multiplication may make sense for many data sets.
+
+##### Case study
+
+Brighter Planet's lodging model uses compound fuzzy inference to predict hotel energy use based on six variables in the CBECS data set: number of rooms, number of floors, heating degree days, cooling degree days, construction year, and percent air conditioned.
+
+1. We wanted to perform our inference based only on records that closely matched our input case for most of these variables, so we started with a straight multiplication scheme rather than addition.
+2. We recognized that number of rooms and number of floors would be highly correlated, as would number of heating and cooling degree days, so we didn't want to treat them independently. For these paired variables, we also wanted records that were similar in one but not both respects to retain some, albeit lesser, weight.
+3. We wanted to weight climate and hotel size more heavily than construction year and air conditioning.
+4. We settled on the equation `(POW(rooms, 0.8) + POW(floors, 0.8)) * (POW(hdd, 0.8) + POW(cdd, 0.8)) * POW(year, 0.8) * POW(ac, 0.8)`
+5. This effectively weights hotel size and climate more heavily than the other variables, because the addition allows weights for these variables to range up to 2 whereas the other two variables have max weights of 1.
+6. It also allows a record to be dissimilar to the input case on half of a variable pair without being disqualified, provided it's a close match on the other half.
+
+## Setup
 
 1) gem install a bleeding edge earth
 
@@ -21,11 +93,11 @@
 
     RUN_DATA_MINER=true rake
 
-## further testing
+## Further testing
 
     rake
 
-## future plans
+## Future plans
 
 **in the future the fuzzy inference machine will make TEMPORARY tables, rather than gum up your db**
 

data/lib/fuzzy_infer/active_record_class_methods.rb

@@ -6,10 +6,8 @@ module FuzzyInfer
     # see test/helper.rb for an example
     def fuzzy_infer(options = {})
       options = ::Hashie::Mash.new options
-      options.target.each do |target|
-        Registry.instance[name] ||= {}
-        Registry.instance[name][target] = options
-      end
+      Registry.instance[name] ||= {}
+      Registry.instance[name][options.target] = options
     end
   end
 end

data/lib/fuzzy_infer/active_record_instance_methods.rb

@@ -1,14 +1,13 @@
 module FuzzyInfer
   module ActiveRecordInstanceMethods
     # Returns a new FuzzyInferenceMachine instance that can infer this target (field)
-    def fuzzy_inference_machine(target)
-      target = target.to_sym
-      FuzzyInferenceMachine.new self, target, Registry.config_for(self.class.name, target)
+    def fuzzy_inference_machine(*targets)
+      FuzzyInferenceMachine.new self, targets, Registry.config_for(self.class.name, targets)
     end
     
     # Shortcut to creating a FIM and immediately calling it
-    def fuzzy_infer(target)
-      fuzzy_inference_machine(target).infer
+    def fuzzy_infer(*targets)
+      fuzzy_inference_machine(*targets).infer
     end
   end
 end

data/lib/fuzzy_infer/fuzzy_inference_machine.rb

@@ -1,126 +1,162 @@
 module FuzzyInfer
   class FuzzyInferenceMachine
-    
+    MYSQL_ADAPTER_NAME = /mysql/i
+
     attr_reader :kernel
-    attr_reader :target
+    attr_reader :targets
     attr_reader :config
 
-    def initialize(kernel, target, config)
+    delegate :execute, :quote_column_name, :to => :connection
+
+    def initialize(kernel, targets, config)
       @kernel = kernel
-      @target = target
+      @targets = targets
       @config = config
     end
-    
+
     def infer
       calculate_table!
-      retval = select_value(%{SELECT SUM(fuzzy_weighted_value)/SUM(fuzzy_membership) FROM #{table_name}}).to_f
+      pieces = targets.map do |target|
+        "SUM(#{qc(target, :v)})/SUM(#{qc(:fuzzy_membership)})"
+      end
+      values = connection.select_rows(%{SELECT #{pieces.join(', ')} FROM #{table_name}}).first.map do |value|
+        value.nil? ? nil : value.to_f
+      end
       execute %{DROP TABLE #{table_name}}
-      retval
+      if targets.length == 1
+        return values.first
+      else
+        return *values
+      end
     end
-    
+
     # TODO technically I could use this to generate the SQL
     def arel_table
       calculate_table!
       Arel::Table.new table_name
     end
-    
+
     def basis
       @basis ||= kernel.attributes.symbolize_keys.slice(*config.basis).reject { |k, v| v.nil? }
     end
-    
-    def sigma
-      @sigma ||= basis.inject({}) do |memo, (k, v)|
-        memo[k] = select_value(%{SELECT #{sigma_sql(k, v)} FROM #{kernel.class.quoted_table_name} WHERE #{target_not_null_sql} AND #{basis_not_null_sql}}).to_f
-        memo
-      end
-    end
-    
+
     def membership
       return @membership if @membership
       sql = kernel.send(config.membership, basis).dup
       basis.keys.each do |k|
-        sql.gsub! ":#{k}_n_w", quote_column_name("#{k}_n_w")
+        sql.gsub! ":#{k}_n_w", qc(k, :n_w)
       end
       @membership = sql
     end
-    
-    # In case you want to `cache_method :infer` with https://github.com/seamusabshere/cache_method
-    def method_cache_hash
-      [kernel.class.name, basis, target, config].hash
-    end
-    
+
     private
-    
+
     def calculate_table!
-      return if table_exists?(table_name)
-      execute %{CREATE TEMPORARY TABLE #{table_name} AS SELECT * FROM #{kernel.class.quoted_table_name} WHERE #{target_not_null_sql} AND #{basis_not_null_sql}}
-      execute %{ALTER TABLE #{table_name} #{weight_create_columns_sql}}
-      execute %{ALTER TABLE #{table_name} ADD COLUMN fuzzy_membership FLOAT default null}
-      execute %{ALTER TABLE #{table_name} ADD COLUMN fuzzy_weighted_value FLOAT default null}
-      execute %{UPDATE #{table_name} SET #{weight_calculate_sql}}
-      weight_normalize_frags.each do |sql|
-        execute sql
-      end
-      execute %{UPDATE #{table_name} SET fuzzy_membership = #{membership_sql}}
-      execute %{UPDATE #{table_name} SET fuzzy_weighted_value = fuzzy_membership * #{quote_column_name(target)}}
+      return if connection.table_exists?(table_name)
+      mysql = connection.adapter_name =~ MYSQL_ADAPTER_NAME
+      execute %{CREATE TEMPORARY TABLE #{table_name} #{'ENGINE=MEMORY' if mysql} AS SELECT * FROM #{kernel.class.quoted_table_name} WHERE #{all_targets_not_null_condition} AND #{basis_not_null_condition}}
+      execute %{ALTER TABLE #{table_name} #{additional_column_definitions.join(', ')}}
+      execute %{ANALYZE #{'TABLE' if mysql} #{table_name}}
+      execute %{UPDATE #{table_name} SET #{weight_calculators.join(', ')}}
+      execute %{UPDATE #{table_name} SET #{weight_normalizers.join(', ')}}
+      execute %{UPDATE #{table_name} SET #{membership_setter}}
+      execute %{UPDATE #{table_name} SET #{target_setters.join(', ')}}
       nil
     end
-    
-    def membership_sql
-      if config.weight
+
+    def membership_setter
+      right = if config.weight
         "(#{membership}) * #{quote_column_name(config.weight.to_s)}"
       else
         membership
       end
+      "#{qc(:fuzzy_membership)} = #{right}"
     end
-    
-    def weight_normalize_frags
+
+    def target_setters
+      targets.map do |target|
+        %{#{qc(target, :v)} = #{qc(:fuzzy_membership)} * #{quote_column_name(target)}}
+      end
+    end
+
+    def weight_calculators
       basis.keys.map do |k|
-        max = select_value("SELECT MAX(#{quote_column_name("#{k}_w")}) FROM #{table_name}").to_f
-        "UPDATE #{table_name} SET #{quote_column_name("#{k}_n_w")} = #{quote_column_name("#{k}_w")} / #{max}"
+        "#{qc(k, :w)} = 1.0 / (#{sigma[k]}*SQRT(2*PI())) * EXP(-(POW(#{quote_column_name(k)} - #{basis[k]},2))/(2*POW(#{sigma[k]},2)))"
       end
     end
-    
-    def weight_calculate_sql
+
+    def weight_normalizers
+      max_exprs = basis.keys.map do |k|
+        "MAX(#{qc(k, :w)}) AS #{qc(k, :w_max)}"
+      end
+      maxes = connection.select_one("SELECT #{max_exprs.join(', ')} FROM #{table_name}")
       basis.keys.map do |k|
-        "#{quote_column_name("#{k}_w")} = 1.0 / (#{sigma[k]}*SQRT(2*PI())) * EXP(-(POW(#{quote_column_name(k)} - #{basis[k]},2))/(2*POW(#{sigma[k]},2)))"
-      end.join(', ')
+        "#{qc(k, :n_w)} = #{qc(k, :w)} / #{maxes[c(k, :w_max)]}"
+      end
     end
-    
-    def sigma_sql(column_name, value)
-      sql = config.sigma.dup
-      sql.gsub! ':column', quote_column_name(column_name)
-      sql.gsub! ':value', value.to_f.to_s
-      sql
+
+    def sigma
+      @sigma ||= begin
+        exprs = basis.map do |column_name, kernel_value|
+          sql = "#{config.sigma} AS #{qc(column_name)}"
+          sql.gsub! ':column', quote_column_name(column_name)
+          sql.gsub! ':value', kernel_value.to_f.to_s
+          sql
+        end
+        row = connection.select_one(%{SELECT #{exprs.join(', ')} FROM #{kernel.class.quoted_table_name} WHERE #{all_targets_not_null_condition} AND #{basis_not_null_condition}})
+        basis.inject({}) do |memo, (column_name, _)|
+          memo[column_name] = row[c(column_name)].to_f
+          memo
+        end
+      end
     end
-    
+
+    def randomness
+      @randomness ||= [Time.now.strftime('%H%M%S'), Kernel.rand(1e5)].join('_')
+    end
+
     def table_name
-      @table_name ||= "fuzzy_infer_#{Time.now.strftime('%Y_%m_%d_%H_%M_%S')}_#{Kernel.rand(1e11)}"
-    end
-    
-    def weight_create_columns_sql
-      basis.keys.inject([]) do |memo, k|
-        memo << "ADD COLUMN #{quote_column_name("#{k}_w")} FLOAT default null"
-        memo << "ADD COLUMN #{quote_column_name("#{k}_n_w")} FLOAT default null"
-        memo
-      end.flatten.join ', '
-    end
-    
-    def basis_not_null_sql
+      @table_name ||= "fuzzy_infer_#{randomness}"
+    end
+
+    def additional_column_definitions
+      cols = []
+      cols << "ADD COLUMN #{qc(:fuzzy_membership)} FLOAT DEFAULT NULL"
+      basis.keys.each do |k|
+        cols << "ADD COLUMN #{qc(k, :w)} FLOAT DEFAULT NULL"
+        cols << "ADD COLUMN #{qc(k, :n_w)} FLOAT DEFAULT NULL"
+      end
+      targets.each do |target|
+        cols << "ADD COLUMN #{qc(target, :v)} FLOAT DEFAULT NULL"
+      end
+      cols
+    end
+
+    def basis_not_null_condition
       basis.keys.map do |basis|
         "#{quote_column_name(basis)} IS NOT NULL"
       end.join ' AND '
     end
-    
-    def target_not_null_sql
-      "#{quote_column_name(target)} IS NOT NULL"
+
+    def all_targets_not_null_condition
+      [config.target].flatten.map do |target|
+        "#{quote_column_name(target)} IS NOT NULL"
+      end.join ' AND '
     end
-    
+
     def connection
       kernel.connection
     end
-    
-    delegate :execute, :quote_column_name, :select_value, :table_exists?, :to => :connection
+
+    # quoted version of #c
+    def qc(column_name, suffix = nil)
+      quote_column_name c(column_name, suffix)
+    end
+
+    # column name that won't step on anybody's toes
+    def c(column_name, suffix = nil)
+      [column_name, suffix, randomness].compact.join '_'
+    end
   end
 end
 

data/lib/fuzzy_infer/registry.rb

@@ -3,10 +3,12 @@ require 'singleton'
 module FuzzyInfer
   class Registry < ::Hash
     class << self
-      def config_for(class_name, target)
+      def config_for(class_name, targets)
         raise %{[fuzzy_infer] Zero machines are defined on #{class_name}.} unless instance.has_key?(class_name)
-        raise %{[fuzzy_infer] Target #{target.inspect} is not available on #{class_name}.} unless instance[class_name].has_key?(target)
-        instance[class_name][target]
+        unless k_v = instance[class_name].detect { |k, _| (targets & k) == targets }
+          raise %{[fuzzy_infer] Target #{targets.inspect} is not available on #{class_name}.}
+        end
+        k_v.last
       end
     end
     

data/lib/fuzzy_infer/version.rb

@@ -1,3 +1,3 @@
 module FuzzyInfer
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/test/helper.rb

@@ -1,5 +1,8 @@
+require 'rubygems'
 require 'bundler/setup'
 
+require 'benchmark'
+
 require 'active_record'
 case ENV['DB_ADAPTER']
 when 'postgresql'

data/test/test_fuzzy_infer.rb

@@ -39,14 +39,14 @@ describe FuzzyInfer do
     end
     describe '#sigma' do
       it "is calculated from the original table, but only those rows that are also in the temp table" do
-        @e.sigma[:heating_degree_days].must_be_close_to 411.9, 0.1
-        @e.sigma[:cooling_degree_days].must_be_close_to 267.6, 0.1
-        @e.sigma[:lodging_rooms].must_be_close_to 55.0, 0.1
+        @e.send(:sigma)[:heating_degree_days].must_be_close_to 411.9, 0.1
+        @e.send(:sigma)[:cooling_degree_days].must_be_close_to 267.6, 0.1
+        @e.send(:sigma)[:lodging_rooms].must_be_close_to 55.0, 0.1
       end
     end
     describe '#membership' do
       it 'depends on the kernel' do
-        @e.membership.must_match %r{\(POW\(.heating_degree_days_n_w.,\ 0\.8\)\ \+\ POW\(.cooling_degree_days_n_w.,\ 0\.8\)\)\ \*\ POW\(.lodging_rooms_n_w.,\ 0\.8\)}
+        @e.membership.must_match %r{\(POW\(.heating_degree_days_n_w_\d+_\d+.,\ 0\.8\)\ \+\ POW\(.cooling_degree_days_n_w_\d+_\d+.,\ 0\.8\)\)\ \*\ POW\(.lodging_rooms_n_w_\d+_\d+.,\ 0\.8\)}
       end
     end
     describe '#infer' do
@@ -54,5 +54,27 @@ describe FuzzyInfer do
         @e.infer.must_be_close_to 17.75, 0.01
       end
     end
+    describe 'optimizations' do
+      it "can run multiple numbers at once" do
+        # dry run
+        @kernel.fuzzy_infer :electricity_per_room_night
+        @kernel.fuzzy_infer :natural_gas_per_room_night
+        @kernel.fuzzy_infer :fuel_oil_per_room_night
+        # end
+        e1 = n1 = f1 = e2 = n2 = f2 = nil
+        uncached_time = Benchmark.realtime do
+          e1 = @kernel.fuzzy_infer :electricity_per_room_night
+          n1 = @kernel.fuzzy_infer :natural_gas_per_room_night
+          f1 = @kernel.fuzzy_infer :fuel_oil_per_room_night
+        end
+        cached_time = Benchmark.realtime do
+          e2, n2, f2 = @kernel.fuzzy_infer :electricity_per_room_night, :natural_gas_per_room_night, :fuel_oil_per_room_night
+        end
+        (uncached_time / cached_time).must_be :>, 2
+        e2.must_equal e1
+        n2.must_equal n1
+        f2.must_equal f1
+      end
+    end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fuzzy_infer
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -11,11 +11,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-02-21 00:00:00.000000000 Z
+date: 2012-04-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
-  requirement: &2153031100 !ruby/object:Gem::Requirement
+  requirement: &2153823060 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -23,10 +23,10 @@ dependencies:
         version: '3'
   type: :runtime
   prerelease: false
-  version_requirements: *2153031100
+  version_requirements: *2153823060
 - !ruby/object:Gem::Dependency
   name: activerecord
-  requirement: &2153029920 !ruby/object:Gem::Requirement
+  requirement: &2153822420 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -34,10 +34,10 @@ dependencies:
         version: '3'
   type: :runtime
   prerelease: false
-  version_requirements: *2153029920
+  version_requirements: *2153822420
 - !ruby/object:Gem::Dependency
   name: hashie
-  requirement: &2153029260 !ruby/object:Gem::Requirement
+  requirement: &2153821960 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -45,7 +45,7 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *2153029260
+  version_requirements: *2153821960
 description: Use fuzzy set analysis to infer missing values. You provide a sigma function,
   a membership function, and a kernel.
 email:
@@ -57,6 +57,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- CHANGELOG
 - Gemfile
 - LICENSE
 - README.markdown