iron_fixture_extractor 1.0.0 → 1.1.0

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 1.0.0 -> 1.1.0
+* Shouldn't be any breaking changes, mostly a switch to rspec for
+testing.
+
+* Added Fe.load_db(...) :only, :except, and :map options
+
 ## 0.1.1 -> 1.0.0
 * Fe.extract must be given a string or array of ActiveRecord queries
   like this

data/Gemfile

@@ -1,12 +1,10 @@
 source 'https://rubygems.org'
 
-# TEMP CRAP FOR DEV, COMMENT OUT OR DELETE
-#
-gem 'debugger'
-
-# USED w ruby 1.9.2
-# http://dirk.net/2010/04/17/ruby-debug-with-ruby-19x-and-rails-3-on-rvm/
-#gem 'ruby-debug19', :require => 'ruby-debug'
-gem 'rake'
-# Specify your gem's dependencies in iron_fixture_extractor.gemspec
+group :test do
+  gem "rspec"
+  gem 'sqlite3'
+  gem 'debugger'
+  gem 'shoulda'
+  gem 'rake'
+end
 gemspec

data/README.md

@@ -1,44 +1,100 @@
 # About Iron Fixture Extractor
 
-For extracting complex data from staging and production databases to be used for automated testing (works great with whatever testing framework your using).
+For extracting complex data from staging and production databases to be used for automated testing.
 
 Its best when:
-* your data is too complex for factories (like when integrating with legacy systems, ERP systems, etc)
-* creating and maintaining manual fixtures is cumbersome and brittle (always, :))
 
-Its good for:
+* your data is too complex for factories
+* creating and maintaining manual fixtures is cumbersome and brittle
 
-  Pulling data from a staging database containing vetted data that has
-  been built up by the development team, users, or business analysts to be used
-  as "archetypical" data structures in test cases, demonstration.
+## Use cases
 
-  Taking snapshots of production data that has triggered unexpected errors to be incorporated into test cases for closer inspection and correction.
+* Pulling data from a staging database containing vetted data that has
+  been built up by the development team, users, or business analysts to be loaded and used
+  as "archetypical" data structures in test cases or demos.
 
-How it works:
+* Taking snapshots of production data that has triggered app exceptions 
+  to be more closely inspected and incorporated into test cases.
 
-  Feed it an array of ActiveRecord objects or ActiveRelation object and it will allow you to extract, load, and rebuild the records associated with your queries.
+## How it works
 
-## Usage (typically in a console)
-### Extract (fixture files)
+  Feed it an array of ActiveRecord objects or ActiveRelation object and
+it will allow you to:
+
+* extract data to .yml fixtures
+* load it into a database or memory
+* rebuild .yml fixtures from a saved ActiveRelation extraction query.
+
+## Usage
+
+### *Extract* fixture set (typically run in a irb console)
 
     Fe.extract 'Post.includes(:comments, :author).limit(1)', :name =>  'first_post_w_comments_and_authors'
+    # or for multi-model extraction something like this:
+    x = '[UserRole.all, Project.includes(:contributors => [:bio])]'
+    Fe.extract(x,:name => :all_permissions_and_all_projects)
 
-### Load (dataset into database)
+### *Load* fixture set into database (typically run in a "setup" test method )
 
     Fe.load_db(:first_post_w_comments_and_authors)
 
-### Rebuild (fixture files associated with the initial extraction)
+If your fixture set is huge, you can avoid loading particular tables with:
 
-    Fe.rebuild(:first_post_w_comments_and_authors)
+    Fe.load_db(:first_post_w_comments_and_authors, :only => 'posts')
+
+Or 
+
+    Fe.load_db(:first_post_w_comments_and_authors, :except => ['comments'])
+
+You can also load to a table name different than the source they were extracted from via a Hash or Proc:
+
+Via Proc: (this will add "a_prefix_" to all target tables)
+
+    Fe.load_db(:first_post_w_comments_and_authors, :map => -> table_name { "a_prefix_#{table_name}" })
 
-### Pull up a hash from a particular fixture file (very handy in test cases)
+Via Hash: (just maps posts to different table, the others stay the same)
+
+    Fe.load_db(:first_post_w_comments_and_authors, :map => {'posts' => 'different_posts'})
+
+### *Load particular fixture into memory* (typically used to instantiate an object or build a factory)
+
+    # 'r1' is the fixture's name, all fixture names start with 'r', 1 is the id
     Fe.get_hash(:first_post_w_comments_and_authors, Post, 'r1')
+
+    # You can specify :first, or :last to the last arg
+    Fe.get_hash(:first_post_w_comments_and_authors, Comment, :first)
+
+    # Get the hash representation of the whole fixture file
+    Fe.get_hash(:first_post_w_comments_and_authors, Comment, :all)
+
+    # Get an array of hashes stored in a fixture file
+    Fe.get_hashes(:first_post_w_comments_and_authors, Comment)
+
+This feature is used to instantiate objects from the hash or define factories like:
+
+    # Create factory from a particular hash within a fixture file
+    Factory.create(:the_post) do
+      h=Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
+      name h.name
+    end
+
     or
-    Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
-    or 
-    Fe.get_hash(:first_post_w_comments_and_authors, Post, :last)
+
+    # Create an instance
+    h=Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
+    ye_old_post=Post.new(h)
+
+### *Rebuild* fixture files associated with the initial extraction (also doable via rake task in Rails)
+
+    Fe.rebuild(:first_post_w_comments_and_authors)
+    # Make sure to `diff` your test/fe_fixtures dir to see what has changed in .yml files
+
+### *Truncate tables* associated with a fixture set (if you're not using DatabaseCleaner)
+
+    Fe.truncate_tables_for(:first_post_w_comments_and_authors)
 
 ## Installation
+
 Add this line to your application's Gemfile:
 
     gem 'iron_fixture_extractor'
@@ -51,47 +107,70 @@ Or install it yourself as:
 
     $ gem install iron_fixture_extractor
 
-### Extract
-The essense of the Fe.extract is dirt simple:
+## Advanced Usage/Changing fe_manifest.yml for fixture set
+
+* Each extracted fixture set has a fe_manifest.yml file that contains
+details about:
+
+  * The ActiveRelation/ActiveRecord query to used to instantiate objects
+to be serialized to .yml fixtures.
+  * The models, table names, and row counts of records in the fixture set
+
+By modifying the :extract_code: field, you can change the extraction
+behavior associated with .rebuild. It can be handy if you want to add
+data to a fixture set.
+
+## Dirt Simple Shiznit
+
+The essense of the Fe.extract "algorithm" is:
 
     for each record given to Fe.extract
       recursively resolve any association pre-loaded in the .association_cache [ActiveRecord] method
       add it to a set of records keyed by model name
     write each set of records as a <TheModel.table_name>.yml fixture
-    write a fe_manifest.yml that will allow you to later change the query, inspect row counts, and rebuild the fixtures by re-executing the originate queries
+    write a fe_manifest.yml containing original query, row counts, etc
+
+## Typical Workflow 
+* Data extracted from a dev, staging, or production db is needed
+* Open `rails console` in the appropriate environment
+* Monkey with ActiveRecord queries to collect the data set you want to use in your test case.
+* Represent the ActiveRecord query code as a string, i.e. `x=[User.all,Project.includes(:author).find(22)]'`
+* Extract the data into fixtures, `Fe.extract(x,:name => :some_fixture_set_name)`
+* Open up test/fe_fixtures/some_fixture_set_name and poke around the yml files to make sure you've captured what you need. Tweak `extract_name` if you need to and `.rebuild`
+* In your test case's setup method:
+
+    Fe.load_db(:some_fixture_set_name)
+    ...then load a instance var to test against:
+    ...in this case 22 is the id of a fixture that has just been loaded
+    @the_project = Project.find(22)
+    or
+    Fe.execute_extract_code(:some_fixture_set_name).first
+    
+* In your test case's teardown method:
 
+    DatabaseCleaner.clean...
+    or
+    Fe.truncate_tables_for(:some_fixture_set_name)
+
+* In your test case `require 'debugger'; debugger; puts 'x'`...inspect @the_project or whatever does the loaded object and db state have the fixtures you want.
+* Once things seem to be working-ish in your tests, add the fixtures to source control + test case that uses them.
 
-## Missing Features
-* If you give a non-string arg to .extract, the manifest should resolve
-  the .extract_code to be a bunch of look-ups by primary key ala [Post.find(1),Comment.find(2)].
-* The output of each of the main commands should be meaningful, aka,
-  make extractor implement a sensible .to_s and .inspect 
-* load_db should error if Rails.env or RAILS_ENV is defined and set to
-  production
-* An :extract_schema option passed to .extract that uses `rake db:structure:dump` functionality 
-  or ActiveRecord::Base.connection.structure_dump to create .sql files containing a "create table" statement
-  for each distinct model class.  This would allow you to completely de-couple your test cases + fixtures
-  from the external databases they were extracted from.
+## Gem Compatibility
+
+* Works on MRI 1.9.3 and 1.9.2
+* Does not work on JRuby, 1.8.7
 
 ## Contributing
-To run test cases:
 
-    # clone or fork repo
-    cd iron_fixture_extractor
-    rake # runs test cases
+In a nutshell:
 
-Help on the missing features above would be much appreciated per the
-usual github approach:
+    git clone     # get the code
+    cd <the dir>
+    rake          # run the tests
+    # make a spec file and hack.
 
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Ensure the test cases run
-4. Copy one of the test cases (like basic_test.rb), rename, rip out the guts, and add some tests + code to the app
-5. Commit your changes (`git commit -am 'Added some feature'`)
-6. Push to the branch (`git push origin my-new-feature`)
-7. Create new Pull Request
+See spec/README_FOR_DEVELOPERS.md for more details.
 
-If you have other ideas for this tool, make a Github Issue.
 
 ## Footnotes
 I used various ideas from the following blog posts, gists, and existing
@@ -109,3 +188,6 @@ ruby gems, thanks to the authors of these pages:
 * http://www.dan-manges.com/blog/38
 * http://www.martinfowler.com/bliki/ObjectMother.html
 * http://asciicasts.com/episodes/158-factories-not-fixtures
+
+## Author
+Joe Goggins

data/Rakefile

@@ -2,15 +2,14 @@
 require "bundler/gem_tasks"
 require 'rake/testtask'
 
-Rake::TestTask.new(:test) do |t|
-  t.libs << 'lib'
-  t.libs << 'test'
-  t.pattern = 'test/**/*_test.rb'
-  t.verbose = true
+desc "Run the rspec test sweet"
+task :test do
+  puts "Running specs in /spec."
+  sh "bundle exec rspec"
 end
 task :default => 'test'
 
 desc "Open an irb session preloaded with this library"
 task :console do
-  sh "irb -rubygems -I lib -I test -r test_helper -r fe"
+  sh "bundle exec irb -rubygems -I lib -I spec -r spec_helper -r fe"
 end

data/iron_fixture_extractor.gemspec

@@ -16,7 +16,4 @@ Gem::Specification.new do |gem|
   gem.version       = Fe::VERSION
   gem.add_runtime_dependency "activerecord", "~> 3.2.1"
   gem.add_runtime_dependency "activesupport", "~> 3.2.1"
-  gem.add_development_dependency "shoulda", "~> 3.0.1"
-  gem.add_development_dependency "sqlite3"
-  gem.add_development_dependency "factory_girl"
 end

data/lib/fe.rb

@@ -3,7 +3,6 @@ module Fe
   class InvalidSourceModelToMapFrom < Exception; end
   extend ActiveSupport::Autoload
   autoload :Extractor
-  autoload :FactoryGirlDslMethods
   require 'fe/railtie' if defined?(Rails)
 
   # global configuration
@@ -28,20 +27,13 @@ module Fe
 
     # Insert fixtures into tables from the yml files in the
     # "extract_name" fixture set
-    # If you specify a map_models hash here for the last arg like
-    #   Post => DifferentPost
-    # It will load the fixtures into a different table than whats listed
-    # in the manifest
     # NOTE: This is destructive, it will delete everything in the target table
     #
-    def load_db(extract_name, map_models={})
+    def load_db(extract_name, options={})
       extractor = Fe::Extractor.new
       extractor.name = extract_name
       extractor.load_from_manifest
-      unless map_models.empty?
-        extractor.map_models_hash = map_models
-      end
-      extractor.load_into_database
+      extractor.load_into_database(options)
       extractor
     end
 
@@ -59,19 +51,13 @@ module Fe
     # Used if you want to get a hash representation of a particular
     # fixture in a fixture set for a given model
     #
-    # Used like
-    #   h = Fe.get_hash(:first_post_w_comments_and_authors, Post, 'r1')
-    # => {:id => 1, :name => 'first post', ....}
-    # in the console
-    # or in a factory declaration like this
+    # Used like:
     #
-    #   FactoryGirl.define do
-    #     factory :fe4,
-    #       :class => Post,
-    #       &Fe.get_hash(:first_post_w_comments_and_authors,Post,"r1").to_factory_girl_string.to_proc
-    #   end
+    #   h = Fe.get_hash(:first_post_w_comments_and_authors, Post, :first)
     #
-    # Kinda meta, but somewhat useful
+    #   # => {:id => 1, :name => 'first post', ....}
+    #
+    # in the console
     #
     def get_hash(extract_name, model_name, fixture_name)
       model_name = model_name.to_s 
@@ -93,34 +79,27 @@ module Fe
           a_hash = h.to_a.first.last
         when :last
           a_hash = h.to_a.last.last
+        when :all
+          a_hash = h
         else
-          raise "symbols can be :first or :last"
+          raise "symbols can be :first, :last, or :all"
         end
       elsif fixture_name.kind_of? String
         raise "Fixture of the name #{fixture_name} did not exist in in #{fixture_path_for_model}" unless h.has_key?(fixture_name)
         a_hash = h[fixture_name]
       else
-        raise "fixture name must be a string or a symbol like :first or :laset"
-      end
-      a_hash.define_singleton_method(:to_factory_girl_string) do
-        s=<<-EOS
-        x = #{model_name}.new(Fe.get_hash(:#{extract_name},#{model_name},"#{fixture_name}"))
-        EOS
-        model_name.constantize.column_names.each do |col|
-          s << "#{col} x.#{col}\n"
-        end
-        s.instance_eval do
-          def to_proc
-            Proc.new {
-              self
-            }
-          end
-        end
-        s
+        raise "fixture name must be a string or a symbol"
       end
       a_hash
     end
 
+
+    # Syntactic sugar for get_hash(extract_name, model_name, :all).values
+    #
+    def get_hashes(extract_name, model_name)
+      self.get_hash(extract_name, model_name, :all).values
+    end
+
     # Execute the ActiveRecord query associated with the extract set
     #
     def execute_extract_code(extract_name)
@@ -150,10 +129,6 @@ module Fe
       end
       true
     end
-
-    def augment_factory_girl!
-      FactoryGirl::Syntax::Default::DSL.send(:include, Fe::FactoryGirlDslMethods)
-    end
   end
 end
 

data/lib/fe/extractor.rb

@@ -1,7 +1,17 @@
 module Fe
   class Extractor
-    attr_accessor :input_array, :extract_code, :name, :row_counts,:table_names, :manifest_hash
-
+    attr_accessor :input_array,
+      :extract_code,
+      :name,
+      :row_counts,
+      :table_names,
+      :table_name_to_model_name_hash,
+      :manifest_hash
+
+    ##################
+    #   PUBLIC API   #
+    ##################
+    #
     def extract
       load_input_array_by_executing_extract_code
       @row_counts = {}
@@ -19,7 +29,8 @@ module Fe
                         :name => self.name,
                         :model_names => self.model_names,
                         :row_counts => self.row_counts,
-                        :table_names => self.models.map {|m| m.table_name}
+                        :table_names => self.models.map {|m| m.table_name},
+                        :table_name_to_model_name_hash => self.models.inject({}) {|h,m| h[m.table_name] = m.to_s; h }
                        }
       File.open(self.manifest_file_path,'w') do |file|
         file.write(@manifest_hash.to_yaml)
@@ -27,68 +38,56 @@ module Fe
       self.write_model_fixtures
     end
 
-    # This is called from 2 types of invocations
-    #   Fe.extract('Post.all', :name => :bla)
-    #   or 
-    #   Fe.extract('[Post.all,Comment.all]', :name => :bla2)
-    #   
-    def load_from_args(active_relation_or_array,*args)
-      options = args.extract_options!
-      @name = (options[:name] || Time.now.strftime("%Y_%m_%d_%H_%M_%S")).to_sym
-      if active_relation_or_array.kind_of? String
-        @extract_code = active_relation_or_array 
-      else
-        raise "Extract code must be a string, so .rebuild can be called"
-      end
-    end
-
-    def load_input_array_by_executing_extract_code
-      @input_array = Array(eval(@extract_code)).to_a
-    end
-
-    def load_from_manifest
-      raise "u gotta set .name to use this method" if self.name.blank?
-      @manifest_hash = YAML.load_file(self.manifest_file_path)
-      @extract_code = @manifest_hash[:extract_code]
-      @name = @manifest_hash[:name]
-      @models = @manifest_hash[:model_names].map {|x| x.constantize}
-    end
-
-    def map_models_hash=(map_models_hash)
-      unless (map_models_hash.keys - self.models).empty?
-        raise InvalidSourceModelToMapFrom.new "your map models hash must contain keys representing class names that exist in the fe_manifest.yml"
-      end
-      @map_models_hash = map_models_hash
-    end
-
-    def map_models_hash
-      if @map_models_hash.nil?
-        {}
-      else
-        @map_models_hash
-      end
-    end
-
-
     # Loads data from each fixture file in the extract set using
     # ActiveRecord::Fixtures
     #
-    def load_into_database
+    def load_into_database(options={})
       # necessary to make multiple invocations possible in a single test
       # case possible
       ActiveRecord::Fixtures.reset_cache
-      self.models.each do |model|
-        if self.map_models_hash.has_key?(model)
-          raise "Unfortunately, this is not implemented...from the implementation of .create_fixtures it seems impossible...keeping this code in case its not"
-          h = {model.table_name => self.map_models_hash[model]}
-          ActiveRecord::Fixtures.create_fixtures(self.target_path, model.table_name, h)
+
+      # Filter down the models to load if specified
+      the_tables = if options.has_key?(:only)
+        self.table_names.select {|x| Array(options[:only]).include?(x) }
+      elsif options.has_key?(:except)
+        self.table_names.select {|x| !Array(options[:except]).include?(x) }
+      else
+        self.table_names
+      end
+      raise "No models to load, relax your :only or :except filters (or don't bother calling this method)" if the_tables.empty?
+
+      the_tables.each do |table_name|
+        if options[:map].nil?
+          # Vanilla create_fixtures will work fine when no mapping is being used
+          ActiveRecord::Fixtures.create_fixtures(self.target_path, table_name)
+          next
         else
-          ActiveRecord::Fixtures.create_fixtures(self.target_path, model.table_name)
+          # Map table_name via a function (great for prefixing)
+          new_table_name = if options[:map].kind_of?(Proc)
+            options[:map].call(table_name)
+          # Map table_name via a Hash table name mapping
+          elsif options[:map][table_name].kind_of? String
+            options[:map][table_name]
+          else
+            table_name # No mapping for this table name
+          end
+          class_name = self.table_name_to_model_name_hash[table_name]
+          fixtures = ActiveRecord::Fixtures.new( ActiveRecord::Base.connection,
+              new_table_name,
+              class_name,
+              ::File.join(self.target_path, table_name))
+          fixtures.table_rows.each do |the_table_name,rows|
+            rows.each do |row|
+              ActiveRecord::Base.connection.insert_fixture(row, the_table_name)
+            end
+          end
         end
+        # FIXME: The right way to do this is to fork the oracle enhanced adapter
+        # and implement a reset_pk_sequence! method, this is what ActiveRecord::Fixtures
+        # calls.  aka this code should be eliminated/live elsewhere.
         case ActiveRecord::Base.connection.adapter_name
         when /oracle/i
           if model.column_names.include? "id"
-            count = model.count
             sequence_name = model.sequence_name.to_s
             max_id = model.maximum(:id)
             next_id = max_id.nil? ? 1 : max_id.to_i + 1
@@ -109,6 +108,7 @@ module Fe
         end
       end
     end
+
     # Returns a hash with model class names for keys and Set's of AR
     # instances for values
     # aka like this
@@ -165,21 +165,49 @@ module Fe
       end
       @fixture_hashes[model_name]
     end
+
+
+    #############################
+    #   OVERLOADED CONSTRUCTORS #
+    #############################
+    #
+    # * These are used by the Fe module to setup the Extractor object
+    # This is called from 2 types of invocations
+    #   Fe.extract('Post.all', :name => :bla)
+    #   or 
+    #   Fe.extract('[Post.all,Comment.all]', :name => :bla2)
+    #   
+    def load_from_args(active_relation_or_array,*args)
+      options = args.extract_options!
+      @name = (options[:name] || Time.now.strftime("%Y_%m_%d_%H_%M_%S")).to_sym
+      if active_relation_or_array.kind_of? String
+        @extract_code = active_relation_or_array 
+      else
+        raise "Extract code must be a string, so .rebuild can be called"
+      end
+    end
+
+    def load_input_array_by_executing_extract_code
+      @input_array = Array(eval(@extract_code)).to_a
+    end
+
+    def load_from_manifest
+      raise "u gotta set .name to use this method" if self.name.blank?
+      @manifest_hash = YAML.load_file(self.manifest_file_path)
+      @extract_code = @manifest_hash[:extract_code]
+      @name = @manifest_hash[:name]
+      @models = @manifest_hash[:model_names].map {|x| x.constantize}
+      @row_counts = @manifest_hash[:row_counts]
+      @table_names = @manifest_hash[:table_names]
+      @table_name_to_model_name_hash = @manifest_hash[:table_name_to_model_name_hash]
+    end
+
     protected
 
     # Recursively goes over all association_cache's from the record and builds the output_hash
     # This is the meat-and-potatoes of this tool (plus the the recurse
     # method) is where something interesting is happening
     #
-    # TODO: To work with ActiveRecord 2.3.x, we'll need to recurse using this logic: (from activerecord-2.3.14/lib/active_record/associations.rb)
-    #    # Clears out the association cache
-    #    def clear_association_cache #:nodoc:
-    #      self.class.reflect_on_all_associations.to_a.each do |assoc|
-    #        instance_variable_set "@#{assoc.name}", nil
-    #      end unless self.new_record?
-    #    end
-    #
-    #
     def recurse(record)
       raise "This gem only knows how to extract stuff w ActiveRecord" unless record.kind_of? ActiveRecord::Base
       key = record.class.base_class.to_s # the base_class is key for correctly handling STI