fathom 0.3.7 → 0.5.0

data/.autotest

@@ -1,10 +1,12 @@
-# require 'autotest/fsevent'
-# require 'autotest/growl'
+begin
+  require 'autotest/fsevent'
+  require 'autotest/growl'
+rescue
+  # Don't complain if this OS X-specific functionality isn't available.
+end
 
 Autotest.add_hook :initialize do |autotest|
-  # Keeps me from calling something document_spec.rb
-  # %w{.git .svn .hg .DS_Store ._* vendor tmp log doc}.each do |exception|
   %w{.git .svn .hg .DS_Store ._* vendor tmp log}.each do |exception|
     autotest.add_exception(exception)
   end
-end
+end

data/.document

@@ -1,5 +1,5 @@
-README.rdoc
 lib/**/*.rb
 bin/*
+- 
 features/**/*.feature
-LICENSE
+LICENSE.txt

data/Gemfile

@@ -1,14 +1,13 @@
-source 'http://rubygems.org'
+source "http://rubygems.org"
 
-gem "fastercsv"
 gem 'uuid'
-# gem 'spira'
-# gem 'rdf'
-gem 'sparql-algebra'
 
-group :test do
-  gem "ruby-debug"
-  gem "rspec"
-  gem "ZenTest"
+# Add dependencies to develop your gem here.
+# Include everything needed to run rake, tests, features, etc.
+group :development do
+  gem "rspec", "~> 2.3.0"
+  gem "cucumber", ">= 0"
+  gem "jeweler", "~> 1.6.4"
+  gem "rcov", ">= 0"
+  gem "ruby-debug19"
 end
-

data/{LICENSE → LICENSE.txt}

@@ -1,4 +1,4 @@
-Copyright (c) 2010 David
+Copyright (c) 2011 David Richards
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -10,115 +10,54 @@ Welcome to Fathom.  Fathom is a library for building decision support tools.  Fa
 Stability Note
 ==============
 
-Please note that Fathom is not ready for production at this time.  I happen to be using it in production for a few modeling projects, but it is undergoing some major architectural changes that won't be stabilized until I have release version 0.4, which will contain the Belief Networks, the Knowledge Base API, and a more complete cleanup of the code.  One major change that's already in the system is the simulations and imports are all building a knowledge base as a graph.  This makes it natural to use the output of a simulation as the input of a different model.
+I've recently refactored this library.  That makes the core Fathom quite stable, but much less useful.  The specific functionality (various databases, say, GSL support, various graphical model algorithms) will be in yet-to-be-released gems.  I'm going to stabilize these before releasing them to the public.
 
+The gems I am actively working on are:
+
+* fm-belief-network: a general gem for Bayesian inference
+* fm-amalgalite-beliefs: a specific gem for Bayesian networks using SQLite3 and [Amalgalite](https://github.com/copiousfreetime/amalgalite)
+* fm-goliath: A simple web service API using the wonderful network of gems associated with [Goliath](https://github.com/postrank-labs/goliath)
+
+There are at least another dozen fm-* gems that live in some level of development.  These are gems like fm-redis-beliefs, fm-clinical-decision-support, and fm-apophenia.  I'll release these slowly and carefully as they go into production on real systems.  That helps keep me from getting ahead of myself.  If you're starting to see the power of how these tools can be used for your work, drop me a note and I could see if I can get your kind of analysis out in the Fathom set of gems.
 
 Inspiration for the Project
----------------------------
+===========================
 
 The ideas for this gem are coming from a lot of places:
 
 * Judea Pearl's work on causal graphs and belief networks.  See [Causality](http://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/052189560X/ref=sr_1_1?s=books&ie=UTF8&qid=1288840948&sr=1-1) and [Probabilistic Reasoning in Intelligent Systems](http://www.amazon.com/Probabilistic-Reasoning-Intelligent-Systems-Plausible/dp/1558604790/ref=ntt_at_ep_dpi_2)
 * Douglas Hubbard's ideas on decision support.  See [How to Measure Anything](http://www.amazon.com/How-Measure-Anything-Intangibles-Business/dp/0470539399/ref=sr_1_1?ie=UTF8&qid=1288840870&sr=8-1)
 * Ben Klemens' ideas on data analysis.  See [Modeling with Data](http://modelingwithdata.org/about_the_book.html)
+* Daphne Koller and Nir Friedman's ideas on [Probabilistic Graphical Models](http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=11886)
+
+Goals
+=====
 
 The goals of this project are:
 
-* Build a decoupled library with Ruby and the GSL
+* Build a decoupled library with Ruby and the GSL (currently optional)
 * Make it easy to gather information of all types
 * Add tools to analyze the integration of knowledge
 
-Decoupled Library with Ruby and the GSL
---------------------------------------------------------
-
-I use Ruby because it's very flexible and fast to write.  I use the GSL because it's very fast, robust, and well-tested.  I decouple the library so that it's easy to use the parts that are worthwhile for your problem and ignore the rest.
-
-Most of the library is about coordinating data nodes.  So, an educated guess or the data that came out of a spreadsheet will sit in a particular kind of node.  The system then allows us to relate this information.  For example, a node that defines the income for a company would relate to the nodes that define the revenue and expenses of the company.  All of this is done with some simple Ruby libraries to make things easy to coordinate.
-
-The statistics that we run are usually run on a GSL::Vector using various random number generators coming from the GSL.  This is really the heart of what the GSL does for Fathom.  A lot of the other tools may be used by some plugins for Fathom, and anyone using their computer to do open source data analysis probably has a good installation of the GSL in it.
-
-There are two bindings for Ruby and the GSL that I know of: [Ruby/GSL](http://rb-gsl.rubyforge.org/) and [ruby-gsl](https://github.com/codahale/ruby-gsl).  Ruby/GSL is still maintained and has the syntax that we're using here.
-
-The decoupling is based on making every file as stand-alone as possible.  What that means is we use autoload quite a bit in the library.  Each file references the main lib/fathom.rb file, so that it can autoload any dependencies it may have.  Also, most files have a test to see if they are being run from the command line at the bottom of the file.  The goal is to make it fairly easy to run a single class in a message queue system, say, or as part of another script to get data into our knowledge base.  In this sense, we're trying to follow some of the design principles of Unix: simple scripts that do one thing well and can combine with other scripts.
-
-Gathering Information
------------------------------
-
-Decision support is about making rational choices from the information available.  This means that we do several things:
-
-* Make it fairly easy to load spreadsheet data in a CSV format
-* Add support for listing assumptions in a YAML file
-* Make it possible to link to RDF data and all of the external context that can be useful
-* Create links to richer data, such as ERP installations, databases, and web crawlers
-
-The data we gather is organized as a bunch of nodes in a graph.  We also try to create ranges or probability distributions for everything, so that things become comparable and we can add things together fairly well.  We also always know explicitly what our uncertainty is, so that we're not misleading ourselves.  Importing conflicting data should be manageable if you're careful to document what you're loading.
-
-Also, we keep track of meta data.  So, specific decisions are described in the knowledge base, as are references to other material, or other ways to describe what we're doing.  In this way, we should be able to work on convincing the appropriate people (employees, executives, review boards) that we've made sound decisions and that we should commit resources to execute the plan.
-
-Tools for Analysis and Integration
---------------------------------------------
-
-The integration tools in Fathom are:
-
-* Monte Carlo Simulations
-* Agent Based Models
-* Belief Networks
-* Causal Graphs
-
-These tools allow us to combine information in our knowledge base and run simulations or update beliefs to be able to see the larger perspective.  These tools also offer insight into areas where more information should have the most return on investment.  So, given a fairly limited amount of information, we can draw conclusions about what's going on, and pinpoint areas where we can refine our models and get more certain results.
-
-We are also adding Apophenia to the library.  We'll be able to build data analysis tools outside of Fathom and then bring their results into our knowledge base and integrate it with the bigger picture.  In this way, we'll be able to do all sorts of statistical analysis, machine learning, data mining, and other information-generating tasks without making Fathom too complicated.  
-
-Apophenia is the pragmatic analysts dream.  It's a C-based library that uses the GSL to do its analysis very quickly.  It uses SQLite for data management, which makes set operations in memory optimal.  It stores the models in consistent ways, so that it won't be hard to use this information inside of Fathom.
-
-There are also other kinds of external libraries whose analysis could be brought into Fathom through the import tools.
-
-The ultimate integration tool for Fathom is the web service.  We are exposing access to all of the tools through a RESTful, JSON interface so that Fathom can be part of any sort of application.  We also expect to publish basic HTML support for these same functions so that users can input and read their knowledge base without too much trouble. 
-
-Further Information
--------------------
-
-* You can use our [Wiki](https://github.com/davidrichards/fathom/wiki) to get code examples and see how things are coming along.
-* You can go to the [TODO](https://github.com/davidrichards/fathom/blob/master/TODO.md) page to see how current development is mapped out.
-* You can go to the [Fleet Ventures Blog](http://fleetventures.com) to get more in-depth tutorials and commentary about how to use these types of tools in business as well as a broader perspective of other technologies we use to solve these kinds of problems.
-
-Dependencies and Extensions
----------------------------
-
-This project relies on the [GNU Scientific Library](http://www.gnu.org/software/gsl/) and the [ruby/gsl](http://rb-gsl.rubyforge.org/) bindings for the GSL.  
-
-Fathom has only minimal extensions on external libraries:
-
-* Array responds to rand (so [1,2,3].rand returns a random value from that array)      
-* OpenStruct exposes it's underlying table, keys, and values
-* FasterCSV has a :strip header converter now
-* String has the constantize method added to it from the ActiveSupport library
+Architecture
+============
 
-In the future, more optional dependencies will be introduced for parts of the library:
+I've worked on this system for several years, rewriting it every once in a while.  It has taught me more about analysis and software architecture than any other endeavor of my life.  I've come to appreciate a few things in my work:
 
-* EventMachine is one that I'm sure will be added. 
-* RDF.rb and related gems will be used for some of the KnowledgeBase
-* SQLite will be available for some set operations
-* One of the key/value data stores will be used for the KnowledgeBase (Riak, CouchDB, MongDO, Redis, or similar)
+* Testing: I have thrown away a lot of hastily-written work that I could not come back and test properly.  It's hard enough to have solid software with solid tests, it's insane to work on this kind of software without testing.
+* Modularity: I'm learning a style of development that loosely follows [DCI](http://en.wikipedia.org/wiki/Data,_Context_and_Interaction) philosophies.  I don't know if I've mastered these ideas, but it is a lot easier for me to have thin data models, contexts that support business goals, and interactions that can be well-monitored.
+* Incremental Enhancement: It's nice when an idea can be expressed simply.  If I need to get more complicated, I can mix in more specific code, or require another gem.  In this way, I have Fathom that knows a little about graph algorithms, but fm-belief-networks that knows more about prior and posterior probabilities, conditional probability tables, and the workings of a Bayesian network.  I further enhance these ideas in fm-amalgalite-beliefs with specific implementations of this data, stored in SQLite tables for the flexibility and speed of set operations in this embedded system.  fm-redis-beliefs won't have the benefit of SQL for a query language, but will have the benefit of a federated key-value store for larger networks.
 
-It should be easy to avoid the parts of the library that use dependencies that you don't want to have.  The goal for dependencies is:
+I probably could/should write a book someday about what I'm learning about architecture.  What this means for the practicing analyst is that we should have a fairly easy way of building models with tools that are appropriate to your problem.  There should be little coupling, and lot of graceful experimentation between various tools that are being produced.  
 
-* To use the best tool available for the job
-* Make it easy to avoid those parts of the library that use those dependencies
+Contributing
+============
 
-For example, the in-memory version of the Knowledge Base will remain available for quick and dirty analysis.  
+If you'd like to contribute, you're welcome to do lots of things:
 
-Note on Patches/Pull Requests
------------------------------
- 
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a
-  future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but
-   bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
+* fork a Fathom branch, write code with tests, and submit a merge request.
+* write another fm-* gem that adds these tools with other tools you use.
+* drop me a line and setup a pair programming session.  We can build some tools together and get you on your way quickly.
 
 Copyright
 ---------
@@ -128,7 +67,7 @@ Copyright
 * FasterCSV is released under the [GPL Version 2](http://www.gnu.org/licenses/old-licenses/gpl-2.0.html) license.
 * Ruby is released under [this license](http://www.ruby-lang.org/en/LICENSE.txt).
 
-Copyright (c) 2010 David Richards
+Copyright (c) 2010-2011 Fleet Ventures, Inc.
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the
@@ -147,4 +86,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -1,47 +1,49 @@
+# encoding: utf-8
+
 require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
 require 'rake'
 
-begin
-  require 'jeweler'
-  Jeweler::Tasks.new do |gem|
-    gem.name = "fathom"
-    gem.summary = %Q{Decision Support in Ruby}
-    gem.description = %Q{Collecting some decision support tools in a decoupled Ruby library.}
-    gem.email = "davidlamontrichards@gmail.com"
-    gem.homepage = "http://github.com/davidrichards/fathom"
-    gem.authors = ["David"]
-    gem.add_development_dependency "rspec"
-    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
-  end
-  Jeweler::GemcutterTasks.new
-rescue LoadError
-  puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "fathom"
+  gem.homepage = "http://github.com/davidrichards/fathom"
+  gem.license = "MIT"
+  gem.summary = %Q{Decision support in Ruby}
+  gem.description = %Q{A framework for building analytical models in Ruby and supporting systems.  This is meant to be a flexible tool for many kinds of models, and should grow gently over time.}
+  gem.email = "davidlamontrichards@gmail.com"
+  gem.authors = ["David Richards"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
 end
 
-require "rspec/core/rake_task"
-RSpec::Core::RakeTask.new(:core) do |spec|
-  spec.pattern = 'spec/fathom/*_spec.rb'
-  # spec.rspec_opts = ['--backtrace']
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
 end
 
-# 
-# Spec::Rake::SpecTask.new(:rcov) do |spec|
-#   spec.libs << 'lib' << 'spec'
-#   spec.pattern = 'spec/**/*_spec.rb'
-#   spec.rcov = true
-# end
-# 
-# task :spec => :check_dependencies
+require 'cucumber/rake/task'
+Cucumber::Rake::Task.new(:features)
 
 task :default => :spec
 
 require 'rake/rdoctask'
 Rake::RDocTask.new do |rdoc|
-  if File.exist?('VERSION')
-    version = File.read('VERSION')
-  else
-    version = ""
-  end
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
 
   rdoc.rdoc_dir = 'rdoc'
   rdoc.title = "fathom #{version}"

data/VERSION

@@ -1 +1 @@
-0.3.7
+0.5.0

data/fathom.gemspec

@@ -0,0 +1,105 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = "fathom"
+  s.version = "0.5.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["David Richards"]
+  s.date = "2011-10-31"
+  s.description = "A framework for building analytical models in Ruby and supporting systems.  This is meant to be a flexible tool for many kinds of models, and should grow gently over time."
+  s.email = "davidlamontrichards@gmail.com"
+  s.extra_rdoc_files = [
+    "LICENSE.txt",
+    "README.md"
+  ]
+  s.files = [
+    ".autotest",
+    ".document",
+    ".rspec",
+    "Gemfile",
+    "LICENSE.txt",
+    "README.md",
+    "Rakefile",
+    "VERSION",
+    "fathom.gemspec",
+    "features/fathom.feature",
+    "features/step_definitions/fathom_steps.rb",
+    "features/support/env.rb",
+    "lib/ext/array.rb",
+    "lib/ext/string.rb",
+    "lib/fathom.rb",
+    "lib/fathom/behaviors/attribute_system.rb",
+    "lib/fathom/behaviors/context_behavior.rb",
+    "lib/fathom/behaviors/plugins.rb",
+    "lib/fathom/contexts/network_population.rb",
+    "lib/fathom/contexts/network_traversal.rb",
+    "lib/fathom/data/adjacency_matrix.rb",
+    "lib/fathom/data/definition.rb",
+    "lib/fathom/data/edge.rb",
+    "lib/fathom/data/network.rb",
+    "lib/fathom/data/outcome.rb",
+    "lib/fathom/data/property.rb",
+    "lib/fathom/data/variable.rb",
+    "lib/fathom/roles/general_graph_tools.rb",
+    "lib/fathom/roles/network_builder.rb",
+    "spec/fathom/behaviors/attribute_system_spec.rb",
+    "spec/fathom/behaviors/context_behavior_spec.rb",
+    "spec/fathom/behaviors/plugins_spec.rb",
+    "spec/fathom/contexts/network_population_spec.rb",
+    "spec/fathom/contexts/network_traversal_spec.rb",
+    "spec/fathom/data/adjacency_matrix_spec.rb",
+    "spec/fathom/data/definition_spec.rb",
+    "spec/fathom/data/edge_spec.rb",
+    "spec/fathom/data/network_spec.rb",
+    "spec/fathom/data/outcome_spec.rb",
+    "spec/fathom/data/property_spec.rb",
+    "spec/fathom/data/variable_spec.rb",
+    "spec/fathom/ext/array_spec.rb",
+    "spec/fathom/ext/string_spec.rb",
+    "spec/fathom/roles/general_graph_tools_spec.rb",
+    "spec/fathom/roles/network_builder_spec.rb",
+    "spec/fathom_spec.rb",
+    "spec/spec_helper.rb",
+    "spec/support/context_behavior.rb",
+    "spec/support/custom_matchers.rb",
+    "spec/support/files.rb",
+    "spec/support/network.yml"
+  ]
+  s.homepage = "http://github.com/davidrichards/fathom"
+  s.licenses = ["MIT"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = "1.8.11"
+  s.summary = "Decision support in Ruby"
+
+  if s.respond_to? :specification_version then
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<uuid>, [">= 0"])
+      s.add_development_dependency(%q<rspec>, ["~> 2.3.0"])
+      s.add_development_dependency(%q<cucumber>, [">= 0"])
+      s.add_development_dependency(%q<jeweler>, ["~> 1.6.4"])
+      s.add_development_dependency(%q<rcov>, [">= 0"])
+      s.add_development_dependency(%q<ruby-debug19>, [">= 0"])
+    else
+      s.add_dependency(%q<uuid>, [">= 0"])
+      s.add_dependency(%q<rspec>, ["~> 2.3.0"])
+      s.add_dependency(%q<cucumber>, [">= 0"])
+      s.add_dependency(%q<jeweler>, ["~> 1.6.4"])
+      s.add_dependency(%q<rcov>, [">= 0"])
+      s.add_dependency(%q<ruby-debug19>, [">= 0"])
+    end
+  else
+    s.add_dependency(%q<uuid>, [">= 0"])
+    s.add_dependency(%q<rspec>, ["~> 2.3.0"])
+    s.add_dependency(%q<cucumber>, [">= 0"])
+    s.add_dependency(%q<jeweler>, ["~> 1.6.4"])
+    s.add_dependency(%q<rcov>, [">= 0"])
+    s.add_dependency(%q<ruby-debug19>, [">= 0"])
+  end
+end
+

data/features/fathom.feature

@@ -0,0 +1,26 @@
+Feature: Graph Methods
+  In order to store knowledge in graphs
+  As an analyst
+  I want to be able to build a graph from a complete description
+
+  Scenario: Building with a Hash
+    Given a full description of a graph in a Hash format
+    When I build a network from that description
+    Then I should get a network with variables
+
+  Scenario: Building with YAML
+    Given a full description of a graph in a YAML format
+    When I build a network from that description
+    Then I should get a network with variables
+
+  Scenario: Building with XMLBIF
+    Given a full description of a graph in a XMLBIF format
+    When I build a network from that description
+    Then I should get a network with variables
+
+  Scenario: Building a directed adjacency matrix
+    Given a defined network
+    When I ask it to build the adjacency matrix
+    Then I should have a matrix that marks the edges between variables
+
+  # Possibly I'll build more of the graph features here.  Certainly I'll finish them in the specs.

data/features/step_definitions/fathom_steps.rb

@@ -0,0 +1,23 @@
+Given /^a full description of a graph in a (.+) format$/ do |format|
+  pending # express the regexp above with the code you wish you had
+end
+
+When /^I build a network from that description$/ do
+  pending # express the regexp above with the code you wish you had
+end
+
+Then /^I should get a network with variables$/ do
+  pending # express the regexp above with the code you wish you had
+end
+
+Given /^a defined network$/ do
+  pending # express the regexp above with the code you wish you had
+end
+
+When /^I ask it to build the adjacency matrix$/ do
+  pending # express the regexp above with the code you wish you had
+end
+
+Then /^I should have a matrix that marks the edges between variables$/ do
+  pending # express the regexp above with the code you wish you had
+end