fathom 0.1.0

data/.bundle/config

@@ -0,0 +1,2 @@
+--- 
+BUNDLE_DISABLE_SHARED_GEMS: "1"

data/.document

@@ -0,0 +1,5 @@
+README.rdoc
+lib/**/*.rb
+bin/*
+features/**/*.feature
+LICENSE

data/.gitignore

@@ -0,0 +1,5 @@
+*.sw?
+.DS_Store
+coverage
+rdoc
+pkg

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,5 @@
+source 'http://rubygems.org'
+
+gem "rspec"
+gem "ruby-debug"
+gem "fastercsv"

data/Gemfile.lock

@@ -0,0 +1,30 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    columnize (0.3.1)
+    diff-lcs (1.1.2)
+    fastercsv (1.5.3)
+    linecache (0.43)
+    rspec (2.0.1)
+      rspec-core (~> 2.0.1)
+      rspec-expectations (~> 2.0.1)
+      rspec-mocks (~> 2.0.1)
+    rspec-core (2.0.1)
+    rspec-expectations (2.0.1)
+      diff-lcs (>= 1.1.2)
+    rspec-mocks (2.0.1)
+      rspec-core (~> 2.0.1)
+      rspec-expectations (~> 2.0.1)
+    ruby-debug (0.10.3)
+      columnize (>= 0.1)
+      ruby-debug-base (~> 0.10.3.0)
+    ruby-debug-base (0.10.3)
+      linecache (>= 0.3)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  fastercsv
+  rspec
+  ruby-debug

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 David
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,176 @@
+Fathom
+------
+
+Introduction
+============
+
+This is a library for decision support.  It is useful for recording various types of information, and then combining it in useful ways.  As of right now, it's not very useful, but I'm actively working on it again.  
+
+The ideas for this gem are coming from a lot of places:
+
+* Judea Pearl's work on causal graphs and belief networks.  See [Causality](http://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/052189560X/ref=sr_1_1?s=books&ie=UTF8&qid=1288840948&sr=1-1) and [Probabilistic Reasoning in Intelligent Systems](http://www.amazon.com/Probabilistic-Reasoning-Intelligent-Systems-Plausible/dp/1558604790/ref=ntt_at_ep_dpi_2)
+* Douglas Hubbard's ideas on decision support.  See [How to Measure Anything](http://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/0470539399/ref=sr_1_1?ie=UTF8&qid=1288840870&sr=8-1)
+* Ben Klemens' ideas on data analysis.  See [Modeling with Data](http://modelingwithdata.org/about_the_book.html)
+
+To build useful decision support environments, there are three things that need to be in place:
+
+* Data needs to be gathered or referenced
+* Models need to be developed for the data
+* Data and models need to be presented in context
+
+Setting up the data and models starts with a decoupled Ruby library.  I'll give it a web service API so that a server could be setup for simple systems.  The decoupled library can also be used as consumers on a message queue system for larger installations.
+
+Keeping the data and models in context is more of a user interface question, which I'll build in another library.  I'm considering hosting that solution myself and just making it available publicly.  We'll see after all the core ideas are gathered.
+
+Usage
+=====
+
+Enrico Fermi [said](http://www.lucidcafe.com/library/95sep/fermi.html):
+    There are two possible outcomes: if the result confirms the hypothesis, then you've made a measurement. If the result is contrary to the hypothesis, then you've made a discovery. 
+
+To put together a hypothesis, we gather what we know about our problem: 
+
+* What is the decision we are making?
+* What are the consequences of the decision?
+* What do we know now?  
+* How do we order the data we have?  
+* How can we express this in ranges?
+
+If we have a lot of clarity about what we're after, it's easier to gather data and build worthwhile models.  It's probably a good idea to start with PlausibleRange:
+
+    q1_sales = PlausibleRange.new(:min => 10, :max => 20, :hard_lower_bound => 0, :name => "First Quarter Sales")
+    q1_prices = PlausibleRange.new(:min => 10_000, :max => 12_000, :name => "First Quarter Prices")
+    q1_sales_commissions = PlausibleRange.new(:min => 0.2, :max => 0.2, :name => "Sales Commission Rate")
+    
+We can combine these ranges in a ValueDescription:
+
+    q1_gross_margins = ValueDescription.new(q1_sales, q1_prices, q1_sales_commissions) do |random_sample|
+      revenue = (random_sample.first_quarter_sales * random_sample.first_quarter_prices)
+      commissions_paid = random_sample.sales_commission_rate * revenue
+      gross_margins = revenue - commissions_paid
+      {:revenue => revenue, :commissions_paid => commissions_paid, :gross_margins => gross_margins}
+    end
+    
+A ValueDescription can take the ranges and combine them with a block of code.  Here, we sample sales, prices and commission rates to get revenues, commissions paid, and gross margins.  We can then use Monte Carlo methods to model our system:
+
+    sales_model = MonteCarloSet.new(q1_gross_margins)
+    sales_model.process(10_000)
+    sales_model.revenue.mean
+    sales_model.revenue.sd
+    sales_model.gross_margins.mean
+    sales_model.gross_margins.sd
+    
+Here, we are able to run 10,000 random samples to get an idea of how our system interacts.  Notice how the methods get generated in the different objects:
+
+* The ValueDescription converts the name to a lower case, underscore-joined name (E.g. Sales Commission Rate becomes sales_commission_rate).
+* The MonteCarloSet uses the keys from the return value in the ValueDescription block to generate method names
+
+At this point, everything is using a normal Gaussian distribution.  Since Fathom uses the GNU Scientific Library, there are many other distributions we will incorporate into our library.  
+
+If you start with data instead of data ranges, you can use a DataNode instead:
+
+    q1_sales = DataNode.new(:name => "First Quarter Sales", :values => [10,11,15,9])
+    
+A DataNode can also be used in a ValueDescription.
+
+Sometimes it's easier to load data from other sources, such as a spreadsheet:
+
+    sales_data = CSVImport.new(:content => "path/to/sales_data.csv")
+    sales_data.import
+    
+This reads the sales_data file and imports a DataNode for each column.  The spreadsheet is expected to look something like this:
+
+    First Quarter Sales,First Quarter Prices
+    10,12000
+    11,11500
+    15,10000
+    9,12000
+
+The nodes are then generated and stored in the knowledge base.  Right now, this is just an in-memory hash stored in Fathom.knowledge_base
+
+You can also use YAML files to import data.  Given the following YAML data:
+
+    CO2 Emissions:
+      min: 1_000_000
+      max: 1_000_000_000
+
+    CO2 Readings:
+      - 10
+      - 20
+      - 30
+
+You can load the nodes with:
+
+    yaml_nodes = YAMLImport.new('path/to/yaml/file')
+    yaml_nodes.import
+    
+This will create a PlausibleRange for CO2 Emissions and a DataNode for CO2 Readings.
+
+To use imported data in a ValueDescription, just reference this knowledge base:
+
+    ValueDescription.new(Fathom.knowledge_base['First Quarter Sales'], Fathom.knowledge_base['First Quarter Prices']) do
+      ...
+    end
+
+This code is certainly not production ready.  There are many things I'll want to add just to have basic Monte Carlo methods up to snuff:
+
+* More distributions to choose from
+* More import methods (RDF, relational databases, no SQL data stores)
+* A persisted knowledge base
+* Configuration on the knowledge base and databases
+* Better visualization with plotutils support and possibly other graphics support
+* Project organization: decision descriptions, owners, sharing
+* Measurement values: use Shannon's entropy and some value calculations to point out which measurements have the highest potential ROI
+
+On a bigger level, I still haven't implemented other major ideas:
+
+* Agent-based modeling
+* System dynamics
+* Belief updating in Causal Graphs
+* Fathom as a Web service
+
+Documentation TODO:
+
+* Document using this library from the command line
+* Document these classes as RabbitMQ consumers
+
+Dependencies
+============
+
+This project relies on the GNU Scientific Library and the ruby/gsl bindings for the GSL.      
+
+Note on Patches/Pull Requests
+=============================
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but
+   bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+Copyright
+=========
+
+Copyright (c) 2010 David Richards
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,50 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "fathom"
+    gem.summary = %Q{Decision Support in Ruby}
+    gem.description = %Q{Collecting some decision support tools in a decoupled Ruby library.}
+    gem.email = "davidlamontrichards@gmail.com"
+    gem.homepage = "http://github.com/davidrichards/fathom"
+    gem.authors = ["David"]
+    gem.add_development_dependency "rspec"
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
+end
+
+require "rspec/core/rake_task"
+RSpec::Core::RakeTask.new(:core) do |spec|
+  spec.pattern = 'spec/fathom/*_spec.rb'
+  # spec.rspec_opts = ['--backtrace']
+end
+
+# 
+# Spec::Rake::SpecTask.new(:rcov) do |spec|
+#   spec.libs << 'lib' << 'spec'
+#   spec.pattern = 'spec/**/*_spec.rb'
+#   spec.rcov = true
+# end
+# 
+# task :spec => :check_dependencies
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION')
+    version = File.read('VERSION')
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "fathom #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/autotest/discover.rb

@@ -0,0 +1 @@
+Autotest.add_discovery { "rspec2" }

data/lib/fathom.rb

@@ -0,0 +1,68 @@
+# ================
+# = Dependencies =
+# ================
+
+# Make decoupling easier with an informed LoadPath
+$:.unshift(File.dirname(__FILE__))
+$:.unshift(File.expand_path(File.join(File.dirname(__FILE__), 'fathom')))
+
+require "gsl"
+require 'ostruct'
+require 'options_hash'
+
+# Fix a few bugs in OpenStruct
+class OpenStruct
+  def table
+    @table
+  end
+  
+  def values
+    @table.values
+  end
+  
+  def keys
+    @table.keys
+  end
+end
+
+module Fathom
+  lib = File.expand_path(File.dirname(__FILE__))
+  $LOAD_PATH.unshift(lib)
+  
+  # Autoload classes and modules so that we only load as much of the library as we're using.
+  # This allows us to have a fairly large library without taking up a lot of memory unless we need it.
+  autoload :Inverter, "inverter"
+  autoload :BasicNode, "basic_node"
+  autoload :PlausibleRange, "plausible_range"
+  autoload :R, "plausible_range"
+  # autoload :LowerBound, "lower_bound"
+  # autoload :UpperBound, "upper_bound"
+  # autoload :Distribution, "distribution"
+  # autoload :DependencyGraph, "dependency_graph"
+  autoload :ValueDescription, "value_description"
+  autoload :ValueAggregator, "value_aggregator"
+  autoload :ValueMultiplier, "value_multiplier"
+  autoload :MonteCarloSet, "monte_carlo_set"
+  autoload :CombinedPlausibilities, "combined_plausibilities"
+  autoload :CausalGraph, "causal_graph"
+  autoload :DataNode, "data_node"
+  autoload :KnowledgeBase, "knowledge_base"
+
+  autoload :Import, "import"
+  autoload :YAMLImport, 'import/yaml_import'
+  autoload :CSVImport, 'import/csv_import'
+  autoload :RDFImport, 'import/rdf_import'
+  autoload :SQLiteImport, 'import/sqlite_import'
+  
+  autoload :NodeUtilities, 'node_utilities'
+
+  def knowledge_base
+    @knowledge_base ||= KnowledgeBase.new
+  end
+end
+
+# Temporary
+include Fathom
+def r
+  @r ||= R.new(:min => 1, :max => 10)
+end

data/lib/fathom/archive/conditional_probability_matrix.rb

@@ -0,0 +1,116 @@
+require 'rubygems'
+require 'gsl'
+
+include GSL
+
+class NodeAccessor
+  
+  attr_reader :cpm
+  
+  def initialize(cpm)
+    @cpm = cpm
+  end
+  
+  def is(*labels)
+    ChildAccessor.new(cpm, *labels)
+  end
+  
+  def is_not(*labels)
+    ChildAccessor.new(cpm, *(cpm.child.labels - labels))
+  end
+end
+
+class ChildAccessor
+  
+  attr_reader :cpm, :labels
+  def initialize(cpm, *labels)
+    @cpm, @labels = cpm, labels
+  end
+  
+  def given(parent_name)
+    ParentAccessor.new(cpm, labels)
+  end
+  
+end
+
+class ParentAccessor
+  
+  attr_reader :cpm, :node, :child_labels, :child_indices
+  def initialize(cpm, child_labels)
+    @cpm = cpm
+    @child_labels = child_labels
+    @node = cpm.parent
+    @child_indices = child_labels.map {|label| @cpm.child.labels.index(label)}
+  end
+  
+  def is(*labels)
+    indices = labels.map {|label| get_index(label) }
+    sum_probabilities(indices)
+  end
+  
+  def is_not(*labels)
+    not_indices = labels.map {|label| get_index(label) }
+    indices = (0..node.labels.size).to_a - not_indices
+    sum_probabilities(indices)
+  end
+  
+  protected
+  
+    # TODO: Not right...
+    def sum_probabilities(indices)
+      first_child = child_indices.first
+      cpm.matrix[indices.first, first_child]
+      # indices.inject(0.0) do |s, i|
+      #   s += cpm.matrix[i, first_child]
+      # end
+    end
+    
+    def get_index(label)
+      node.labels.index(label)
+    end
+    
+end
+
+
+class ConditionalProbabilityMatrix
+  
+  class << self
+    def define_node_accessor(node, cpm)
+      define_method(node.name.to_sym) do
+        NodeAccessor.new(cpm)
+      end
+    end
+  end
+  
+  attr_reader :parent, :child, :matrix
+  
+  def initialize(parent, child)
+    @parent, @child = parent, child
+    @matrix = @parent.probabilities.col * @child.probabilities
+    assert_name_access
+  end
+  
+  def probability(opts={})
+    child_label = opts[:child]
+    parent_label = opts[:parent]
+    raise ArgumentError, "Must provide a child and parent label.  E.g., probability(:child => true, :parent => false)" unless child_label and parent_label
+    child_label_index = get_index(child, child_label)
+    parent_label_index = get_index(parent, parent_label)
+    self.matrix[parent_label_index, child_label_index]
+  end
+  alias :p :probability
+  
+  def inspect
+    "ConditionalProbabilityMatrix: #{matrix.to_a.inspect}"
+  end
+  
+  protected
+  
+    def assert_name_access
+      ConditionalProbabilityMatrix.define_node_accessor(child, self)
+    end
+    
+    def get_index(node, label)
+      node.labels.index(label)
+    end
+end