fathom 0.2.2 → 0.2.3

data/Gemfile

@@ -3,3 +3,4 @@ source 'http://rubygems.org'
 gem "rspec"
 gem "ruby-debug"
 gem "fastercsv"
+gem 'uuid'

data/Gemfile.lock

@@ -5,6 +5,7 @@ GEM
     diff-lcs (1.1.2)
     fastercsv (1.5.3)
     linecache (0.43)
+    macaddr (1.0.0)
     rspec (2.0.1)
       rspec-core (~> 2.0.1)
       rspec-expectations (~> 2.0.1)
@@ -20,6 +21,8 @@ GEM
       ruby-debug-base (~> 0.10.3.0)
     ruby-debug-base (0.10.3)
       linecache (>= 0.3)
+    uuid (2.3.1)
+      macaddr (~> 1.0)
 
 PLATFORMS
   ruby
@@ -28,3 +31,4 @@ DEPENDENCIES
   fastercsv
   rspec
   ruby-debug
+  uuid

data/TODO.md

@@ -0,0 +1,140 @@
+TODO
+====
+
+Reorganizing
+------------
+
+I've just made some big refactoring steps regarding the organization of the system and the distributions.  To make sure we're there:
+
+* Go back and test the 4 distributions I decided on
+* Finish the discrete ideas, adding size to the node and automatically using that for stats
+* Create the idea of a labeled, multinomial node
+* Add SQLite3 for in-memory set operations for a labeled, multinomial node
+* Add and remove finder methods on nodes for their parents and children
+
+Also, the general organization of the system could be broken down better:
+
+* agent
+* distributions
+* node
+* import
+* causal_graph
+* belief_network
+* knowledge_base
+* apophenia
+* simulation
+
+MonteCarlo
+----------
+
+This needs to get a few new features:
+
+* combine with ValueDescription into one node
+* generate nodes for the return values
+* consider a more general simulation framework, in case it needs to be extended, or to use some of the tools that will be added to the ABM stuff
+
+Belief Networks
+---------------
+
+To get these delivered, I need to revisit the edge logic, to make sure it's easy to extend each edge with an object.
+
+Then:
+
+* CPM brought back from the archive
+* Network propagation
+* Network testing (polytree)
+
+Agent Based Modeling
+--------------------
+
+* Add parameter-passing standards for callbacks
+* Add EventMachine and async capabilities (Inncluding the cluster idea)
+
+Knowledge Base
+--------------
+
+Probably around here I'll be able to start looking at a persistent knowledge base.  I am not sure which way I'll go, but things I'm considering:
+
+* RDF markup
+* Riak/Redis/Mongo/Couch backend
+* Backend adapters
+* tabular data in an RDBMS (think Apophenia)
+
+One of the key features needs to be search: 
+
+* possibly a Xapian search index for full-text searching
+* still need a standard query language, depending on what I choose above
+
+Apophenia
+---------
+
+I'd like to get Apophenia integrated so that any data model generated there could be combined with the work done here.  That means that most of the "hard" data crunching is using some fairly fast tools: C, SQLite3 in memory, Apophenia and the GSL.  
+
+That would mean that you generally come to Fathom to:
+
+* coordinate the elements of a decision or information discovery project
+* run simulations across their collective knowledge
+* maintain consistent information between information nodes
+
+You would go to Apophenia to:
+
+* build data models from statistical methods
+* generate new sets from grouping, sorting, merging, and dealing with multiple imputation
+
+Fathom could feed the information to Apophenia data models.  Given a fairly robust knowledge base, this makes a lot of sense.
+
+Import
+------
+
+* More robust support for CSV and YAML
+* OpenERP
+* RDF
+* Apophenia
+* Web Crawlers
+
+Publication
+-----------
+
+Turning Fathom into a better tool for publishing knowledge, there are a few major parts to add:
+
+Support for Reports:
+
+* Template-based system (possibly MVC as part of the Web Service)
+* Latex-enabled
+* Graphs and PDFs through Prawn
+* PDFs, CSVs through Ruport
+
+Web Service:
+
+* Basic CRUD for every node type
+* Traversal of the graph with good search semantics
+* Authentication and Authorization
+* HTML-based interface with strong input capabilities (mind map js, auto-build forms, dynamic forms, etc.)
+* Survey system
+
+Meta Data:
+
+* ontological support for systems approach to research (7 nodes / article)
+* ontological support for references (auto generate citations)
+* cleaner approach to the decision framework
+
+Causal Graphs
+-------------
+
+* Add the "DO" operator to a belief network
+* Add the tests for causality using the DO operator
+
+This stuff gets into things I haven't finished reading yet, but it would be very interesting/important to finish that work and bring it into Fathom.  This is all Judea Pearl stuff.
+
+Information Service
+-------------------
+
+Using Seaside, Fathom (through the Web Service), and OpenERP, there are several products I could create using this framework:
+
+* Decision support could become custom integrated to a hosted OpenERP instance, for example.  
+* General domain information could culled and organized and verified so that others could tie in and build only their parts of their decision support framework
+* Think tank support would be a very interesting place to work
+
+All of this could also be coupled with consulting and hosting services.
+
+

data/VERSION

@@ -1 +1 @@
-0.2.2
+0.2.3

data/lib/fathom.rb

@@ -11,6 +11,7 @@ require 'options_hash'
 
 require 'ext/open_struct'
 require 'ext/array'
+require 'ext/string'
 
 module Fathom
   lib = File.expand_path(File.dirname(__FILE__))
@@ -19,18 +20,12 @@ module Fathom
   # Autoload classes and modules so that we only load as much of the library as we're using.
   # This allows us to have a fairly large library without taking up a lot of memory unless we need it.
   autoload :Inverter, "inverter"
-  autoload :BasicNode, "basic_node"
+  autoload :Node, "node"
   autoload :PlausibleRange, "plausible_range"
-  autoload :R, "plausible_range"
-  # autoload :LowerBound, "lower_bound"
-  # autoload :UpperBound, "upper_bound"
-  # autoload :Distribution, "distribution"
-  # autoload :DependencyGraph, "dependency_graph"
   autoload :ValueDescription, "value_description"
   autoload :ValueAggregator, "value_aggregator"
   autoload :ValueMultiplier, "value_multiplier"
   autoload :MonteCarloSet, "monte_carlo_set"
-  autoload :CombinedPlausibilities, "combined_plausibilities"
   autoload :CausalGraph, "causal_graph"
   autoload :DataNode, "data_node"
   autoload :KnowledgeBase, "knowledge_base"
@@ -41,8 +36,6 @@ module Fathom
   autoload :RDFImport, 'import/rdf_import'
   autoload :SQLiteImport, 'import/sqlite_import'
   
-  autoload :NodeUtilities, 'node_utilities'
-  
   autoload :Simulation, 'simulation'
   autoload :TickMethods, 'simulation/tick_methods'
   autoload :TickSimulation, 'simulation/tick_simulation'
@@ -50,6 +43,17 @@ module Fathom
   autoload :Agent, 'agent'
   autoload :Properties, 'agent/properties'
   autoload :AgentCluster, 'agent/agent_cluster'
+  
+  autoload :NumericMethods, 'numeric_methods'
+  autoload :EnforcedName, 'enforced_name'
+  
+  autoload :Distributions, 'distributions'
+  module Distributions
+    autoload :Gaussian, 'distributions/gaussian'
+    autoload :Uniform, 'distributions/uniform'
+    autoload :DiscreteGaussian, 'distributions/discrete_gaussian'
+    autoload :DiscreteUniform, 'distributions/discrete_uniform'
+  end
 
   def knowledge_base
     @knowledge_base ||= KnowledgeBase.new
@@ -59,6 +63,3 @@ end
 
 # Temporary
 include Fathom
-def r
-  @r ||= R.new(:min => 1, :max => 10)
-end

data/lib/fathom/data_node.rb

@@ -4,44 +4,15 @@ require File.expand_path(File.join(File.dirname(__FILE__), '..', 'fathom'))
   A DataNode is a node generated from data itself.  It stores the data and reveals some statistical 
   measurements for the data.  It expects an array or vector of values and generates a vector on demans.
 =end
-class Fathom::DataNode
+class Fathom::DataNode < Node
 
-  include NodeUtilities
-  
-  attr_reader :values, :name, :distribution, :confidence_interval
+  include NumericMethods
   
   def initialize(opts={})
-    @values = opts[:values]
+    super(opts)
     raise ArgumentError, "Must provided values: DataNode.new(:values => [...])" unless self.values
-    @name = opts[:name]
-    @distribution = opts[:distribution]
-  end
-  
-  alias :ci :confidence_interval
-  
-  def vector
-    @vector ||= GSL::Vector.ary_to_gv(self.values)
-  end
-  
-  def standard_deviation
-    @standard_deviation ||= vector.sd
-  end
-  alias :sd :standard_deviation
-  alias :std :standard_deviation
-  
-  def mean
-    @mean ||= vector.mean
-  end
-  
-  def rand
-    rng.gaussian(std) + mean
   end
   
-  protected
-    def rng
-      @rng ||= GSL::Rng.alloc(GSL::Rng::MT19937_1999, Kernel.rand(100_000))
-    end
-    
 end
 
 if __FILE__ == $0

data/lib/fathom/distributions.rb

@@ -0,0 +1,8 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'fathom'))
+module Fathom
+  module Distributions
+    module SharedMethods
+      # TODO: Put helper methods here for sharing some of the distribution functionality
+    end
+  end
+end

data/lib/fathom/distributions/discrete_gaussian.rb

@@ -0,0 +1,44 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'fathom'))
+class Fathom::Distributions::DiscreteGaussian
+  extend Fathom::Distributions::SharedMethods
+  class << self
+    def rng
+      @rng ||= GSL::Rng.alloc(GSL::Rng::MT19937_1999, Kernel.rand(100_000))
+    end
+    
+    def rand(sd)
+      (rng.gaussian(sd) / size).floor + 1
+    end
+    
+    def inverse_cdf(opts={})
+      mean = opts[:mean]
+      sd = opts[:sd]
+      sd ||= opts[:std]
+      sd ||= opts[:standard_deviation]
+      lower = opts.fetch(:lower, true)
+      lower = false if opts[:upper]
+      confidence_interval = opts.fetch(:confidence_interval, 0.05)
+      value = lower ? GSL::Cdf.gaussian_Pinv(confidence_interval, sd) : GSL::Cdf.gaussian_Qinv(confidence_interval, sd)
+      value + mean
+    end
+    alias :lower_bound :inverse_cdf
+
+    def upper_bound(opts={})
+      inverse_cdf(opts.merge(:lower => false))
+    end
+
+    def interval_values(opts={})
+      confidence_interval = opts.fetch(:confidence_interval, 0.9)
+      bound = (1 - confidence_interval) / 2.0
+      [lower_bound(opts.merge(:confidence_interval => bound)), upper_bound(opts.merge(:confidence_interval => bound))]
+    end
+    
+    # If only I had the background to explain what this is....
+    # I want to know how many standard deviations are expressed by the confidence interval
+    # I can then divide  the range by this number to get the standard deviation
+    def standard_deviations_under(confidence_interval)
+      GSL::Cdf.gaussian_Qinv((1 - confidence_interval) / 2) * 2
+    end
+  end
+end
+

data/lib/fathom/distributions/discrete_uniform.rb

@@ -0,0 +1,46 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'fathom'))
+class Fathom::Distributions::DiscreteUniform
+  extend Fathom::Distributions::SharedMethods
+  class << self
+    def rng
+      @rng ||= GSL::Rng.alloc(GSL::Rng::MT19937_1999, Kernel.rand(100_000))
+    end
+    
+    def rand
+      (rng.ugaussian / size).floor + 1
+    end
+    
+    def inverse_cdf(opts={})
+      mean = opts[:mean]
+      sd = opts[:sd]
+      sd ||= opts[:std]
+      sd ||= opts[:standard_deviation]
+      lower = opts.fetch(:lower, true)
+      lower = false if opts[:upper]
+      confidence_interval = opts.fetch(:confidence_interval, 0.05)
+      value = lower ? GSL::Cdf.ugaussian_Pinv(confidence_interval) : GSL::Cdf.ugaussian_Qinv(confidence_interval)
+      value + mean
+    end
+    alias :lower_bound :inverse_cdf
+
+    def upper_bound(opts={})
+      inverse_cdf(opts.merge(:lower => false))
+    end
+
+    def interval_values(opts={})
+      confidence_interval = opts.fetch(:confidence_interval, 0.9)
+      bound = (1 - confidence_interval) / 2.0
+      [lower_bound(opts.merge(:confidence_interval => bound)), upper_bound(opts.merge(:confidence_interval => bound))]
+    end
+    
+    # If only I had the background to explain what this is....
+    # I want to know how many standard deviations are expressed by the confidence interval
+    # I can then divide  the range by this number to get the standard deviation
+    def standard_deviations_under(confidence_interval)
+      GSL::Cdf.ugaussian_Qinv((1 - confidence_interval) / 2) * 2
+    end
+    
+    
+  end
+end
+

data/lib/fathom/distributions/gaussian.rb

@@ -0,0 +1,46 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'fathom'))
+class Fathom::Distributions::Gaussian
+  extend Fathom::Distributions::SharedMethods
+  class << self
+    def rng
+      @rng ||= GSL::Rng.alloc(GSL::Rng::MT19937_1999, Kernel.rand(100_000))
+    end
+    
+    def rand(sd)
+      rng.gaussian(sd)
+    end
+    
+    def inverse_cdf(opts={})
+      mean = opts[:mean]
+      sd = opts[:sd]
+      sd ||= opts[:std]
+      sd ||= opts[:standard_deviation]
+      lower = opts.fetch(:lower, true)
+      lower = false if opts[:upper]
+      confidence_interval = opts.fetch(:confidence_interval, 0.05)
+      value = lower ? GSL::Cdf.gaussian_Pinv(confidence_interval, sd) : GSL::Cdf.gaussian_Qinv(confidence_interval, sd)
+      value + mean
+    end
+    alias :lower_bound :inverse_cdf
+
+    def upper_bound(opts={})
+      inverse_cdf(opts.merge(:lower => false))
+    end
+
+    def interval_values(opts={})
+      confidence_interval = opts.fetch(:confidence_interval, 0.9)
+      bound = (1 - confidence_interval) / 2.0
+      [lower_bound(opts.merge(:confidence_interval => bound)), upper_bound(opts.merge(:confidence_interval => bound))]
+    end
+    
+    # If only I had the background to explain what this is....
+    # I want to know how many standard deviations are expressed by the confidence interval
+    # I can then divide  the range by this number to get the standard deviation
+    def standard_deviations_under(confidence_interval)
+      GSL::Cdf.gaussian_Qinv((1 - confidence_interval) / 2) * 2
+    end
+    
+    
+  end
+end
+

data/lib/fathom/distributions/uniform.rb

@@ -0,0 +1,35 @@
+require File.expand_path(File.join(File.dirname(__FILE__), '..', '..', 'fathom'))
+class Fathom::Distributions::Uniform
+  extend Fathom::Distributions::SharedMethods
+  class << self
+    def rng
+      @rng ||= GSL::Rng.alloc(GSL::Rng::MT19937_1999, Kernel.rand(100_000))
+    end
+    
+    def rand
+      rng.ugaussian
+    end
+    
+    def inverse_cdf(opts={})
+      mean = opts[:mean]
+      lower = opts.fetch(:lower, true)
+      lower = false if opts[:upper]
+      confidence_interval = opts.fetch(:confidence_interval, 0.05)
+      value = lower ? GSL::Cdf.ugaussian_Pinv(confidence_interval) : GSL::Cdf.ugaussian_Qinv(confidence_interval)
+      value + mean
+    end
+    alias :lower_bound :inverse_cdf
+
+    def upper_bound(opts={})
+      inverse_cdf(opts.merge(:lower => false))
+    end
+
+    def interval_values(opts={})
+      confidence_interval = opts.fetch(:confidence_interval, 0.9)
+      bound = (1 - confidence_interval) / 2.0
+      [lower_bound(opts.merge(:confidence_interval => bound)), upper_bound(opts.merge(:confidence_interval => bound))]
+    end
+    
+  end
+end
+