distyll 0.0.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b258d9e95565fd009d54bc4a70893f7ec7644b2e
-  data.tar.gz: 06714e3d3c272d347c331e27bf9bfbf6d78f1df4
+  metadata.gz: 355593d87b582eb84240e2a7792dbca8f3f28fae
+  data.tar.gz: 68405650d538248c22adeb704d1025198d2f1332
 SHA512:
-  metadata.gz: aa597d98f6dc4ae3848ae092efc3a2e2f3f511be6a05468f89527e74004f53fccd45ceb8b8b25ea053b846b4c7f1fa5149576e20f54aed61bdc3817aaf4d9a28
-  data.tar.gz: 92a4ef0ea98c1ccdbb3b38564e8438aa0b5502e6c3ecdad89046dc3eb038bb4a33a7b1947d388a61d7b10327bc555b0e4312eec2fdaabdc69e7435102f2724bb
+  metadata.gz: a8696cafb27c1076f98a3b11cbc11c2045e6db1970c6ec7ce2373090a874522478f04bdb29bb6bc4de731bcb4b731e230668973244b4ab2dbd0d6ffeb2487a34
+  data.tar.gz: 511082bfadd2ae053c232f501d1838a644f84954de26e472ec1bbecab43cba0bf1976f6833e30778084aaf444594ce0c3a5907a36422926a61e83e1cfa31d280

data/README.rdoc

@@ -11,14 +11,38 @@ However, a record created today may have an associated record (via a foreign key
 years ago.  If you slice the entire database by created_at timestamps, you'll have foreign keys which point
 nowhere.  Not very helpful for ensuring that your new feature works on production data.
 
-Distyll's solution is to start from a set of "core" ActiveRecord models supplied at initialization time
+Distyll's solution is to start from a set of "core" ActiveRecord models supplied at runtime
 (plus a date threshold for these models), and only pull those that have been created since the date
-threshold.  It then traverses all belongs_to relationships from those core models and pulls in all of those
-related records.
+threshold.  It then traverses associations from the core models and pulls in associated records (details
+in the next section).
 
 Consequently, you end up with a data set that is representative of production, is internally consistent,
 and is smaller.
 
+== Associations and associated records
+
+Distyll starts from the core models and stores the ids of all core records with created_at timestamps
+after the date threshold.  It also marks these core models as <code>include_has_many</code>.  This means
+that <code>has_many</code> and <code>has_and_belongs_to_many</code> relationships SHOULD be traversed out
+of the model.  Distyll then counts all the pertinent ids across all pertinent models and starts a loop.
+
+At each step of the loop, for each model in distyll's list of pertinent models, it reaches out via all
+belongs_to associations (and other associations if the model is marked as <code>include_has_many</code>).
+Using these associations, it expands both (a) the set of pertinent models to be copied and (b) the
+pertinent ids for each of the pertinient models.  In each step, it is only reaching out to associated
+models for pertinent ids which are new since the previous loop.  This cuts down on query time.
+
+There is a trick with has_many association traversal. The core models do allow for traversal "down" (into
+has_many and habtm associations), and any direct descendants of the core models continue traversal down.
+However, if a descendant's associations are then followed "up" (through a belongs_to), then that parent
+cannot subsequently traverse down.  In essence, "include_has_many" is contagious down, but not up.
+
+Distyll also ignores "through" relationships.  It assumes that those associations will be covered via the
+multi-step non-through traversal.
+
+The looping stops when the total number of ids has not increased since the previous loop.  Iterating with
+a while was necessary due to self-referrential joins.  They prevent the use of a clean recursive algorithm.
+
 == Using distyll in your project
 
 1. Add <code>gem 'distyll'</code> to your gemfile
@@ -27,9 +51,11 @@ and is smaller.
 1. Run <code>rake db:create RAILS_ENV=distyll</code>
 1. Run <code>rake db:schema:load RAILS_ENV=distyll</code>
 1. Run <code>rails console</code>
-1. Call <code>Distyll.new(model_names, created_since)</code>, passing it an array of strings of the core models and a date after which core records will be copied.
+1. Call <code>Distyll.run(model_names, created_since)</code>, passing it an array of the core models and a
+date after which core records will be copied.
 
-If you need to clear out the distyll database and try again with different parameters, just go back to the <code>schema:load</code> step and continue from there.
+If you need to clear out the distyll database and try again with different parameters, just go back to the
+<code>schema:load</code> step and continue from there.
 
 == Contributing to distyll
 
@@ -39,14 +65,19 @@ If you need to clear out the distyll database and try again with different param
 * Start a feature/bugfix branch.
 * Commit and push until you are happy with your contribution.
 * Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
-* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or
+is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
 
 == Next Steps for distyll
 
-* Distyll only traverses belongs_to associations for now.  Need to consider other association types.
-* Is likely to cause problems with single table inheritance.  Could probably refer to table names rather than model names when traversing relationships... but this would still be an issue for the base models.
-* Currently performs "IN" query.  In Oracle, this is limited to 1000 values, so I would need to chunk them for that DBMS.
-* Tests.  I know.  I just don't yet have my head around how to test something that's SO model- and database-centric, when those models and databases aren't present in the gem.  Any advice would be appreciated.
+* When finding pertinent ids across polymorphic has_many and habtm associations, Distyll currently ignores
+the _type field, which may cause it to save more ids from the associated table than needed (if the foreign
+key matches some other record in a different table but the _type doesn't match the current model.)
+* Currently performs "IN" queries.  In Oracle, this is limited to 1000 values, so I would need to chunk
+them for that DBMS.
+* Tests.  I know.  I just don't yet have my head around how to test something that's SO model- and
+database-centric, when those models and databases aren't present in the gem.  Any advice would be
+appreciated.
 
 == Copyright
 

data/VERSION

@@ -1 +1 @@
-0.0.0
+1.0.0

data/distyll.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "distyll"
-  s.version = "0.0.0"
+  s.version = "1.0.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Mason F. Matthews"]
-  s.date = "2014-06-03"
+  s.date = "2014-06-20"
   s.description = "Have you ever had a 100GB production database and been unable to test on an internally consistent subset of the data?  Distyll is your answer."
   s.email = "mason.f.matthews@gmail.com"
   s.extra_rdoc_files = [

data/lib/distyll.rb

@@ -1,26 +1,33 @@
 class Distyll
 
-  attr_reader :base_models, :model_profiles, :created_since
-
-  def initialize(bms, cs)
-    @created_since = cs.to_date
-    @base_models = bms.map &:constantize
-    set_model_profiles
-  end
+  def self.run(base_models, created_since)
+    @model_profiles = Hash.new
 
-  def run
     base_models.each do |model|
-      @model_profiles[model].load_ids_by_timestamp(@created_since)
+      if @model_profiles[model].nil?
+        base_profile = DistyllModelProfile.new(model, true)
+        base_profile.load_ids_by_timestamp(created_since)
+        @model_profiles[model] = base_profile
+      end
     end
 
     prior_count = -1
     while prior_count != current_count
       prior_count = current_count
-      @model_profiles.each_value &:demote_new_ids
-
-      @model_profiles.each_value do |profile|
-        profile.associations.each do |a|
-          @model_profiles[a.klass].load_ids(profile.get_new_associated_ids(a))
+      @model_profiles.each_value &:demote_current_ids
+
+      # .dup is necessary here because of Ruby 1.9's "RuntimeError: can't add a new
+      #   key into hash during iteration" encountered in self.find_or_store_profile.
+      @model_profiles.dup.each_value do |profile|
+        unless profile.prior_ids.blank?
+          profile.associations.each do |a|
+            # We DO want to make the associated profile continue to traverse has_manies if
+            #   (a) The current profile traverses has_manies AND
+            #   (b) The association we're about to traverse is a has_many.
+            contagious_has_many = profile.include_has_many && !(a.belongs_to? || a.has_and_belongs_to_many?)
+
+            find_or_store_profile(a.klass, contagious_has_many).load_ids(profile.get_new_associated_ids(a))
+          end
         end
       end
     end
@@ -33,24 +40,12 @@ class Distyll
 
   private
 
-  def set_model_profiles
-    @model_profiles = Hash.new
-    base_models.each do |bm|
-      @model_profiles = potentially_add_profiles(bm, @model_profiles)
-    end
-  end
-
-  def potentially_add_profiles(model, profiles)
-    return profiles if profiles.include? model
-    profiles[model] = DistyllModelProfile.new(model)
-    profiles[model].associations.each do |a|
-      profiles = potentially_add_profiles(a.klass, profiles)
-    end
-    profiles
+  def self.current_count
+    @model_profiles.each_value.sum &:get_id_count
   end
 
-  def current_count
-    model_profiles.each_value.sum &:get_id_count
+  def self.find_or_store_profile(model, include_has_many)
+    @model_profiles[model] ||= DistyllModelProfile.new(model, include_has_many)
   end
 
 end
@@ -58,46 +53,51 @@ end
 
 
 class DistyllModelProfile
-  attr_reader :model, :record_count, :associations, :all_ids, :last_ids, :new_ids
+  attr_reader :model, :include_has_many, :record_count, :associations, :all_ids, :prior_ids, :current_ids
 
-  def initialize(m)
+  def initialize(m, include_h_m = false)
     @model = m
+    @include_has_many = include_h_m
     @record_count = m.count
-    @all_ids = Array.new
-    @last_ids = Array.new
-    @new_ids = Array.new
+    @all_ids = Set.new
+    @prior_ids = Set.new
+    @current_ids = Set.new
     set_associations
   end
 
-  def demote_new_ids
-    @last_ids = @new_ids
-    @new_ids = Array.new
+  def demote_current_ids
+    @prior_ids = @current_ids
+    @current_ids = Set.new
   end
 
   def load_ids_by_timestamp(timestamp)
     ids = model.where("created_at >= ?", timestamp).select(:id).map &:id
-    @new_ids += ids
-    @all_ids += ids
+    @current_ids.merge(ids)
+    @all_ids.merge(ids)
   end
 
   def load_ids(ids)
-    @new_ids += ids
-    @all_ids += ids
+    @current_ids.merge(ids)
+    @all_ids.merge(ids)
   end
 
   def get_id_count
-    @all_ids = @all_ids.uniq || []
     @all_ids.count
   end
 
   def get_new_associated_ids(a)
-    model.where(id: last_ids).select(a.foreign_key).map { |r| r.send(a.foreign_key) }
+    if a.belongs_to?
+      model.where(id: prior_ids.to_a).select(a.foreign_key).map { |r| r.send(a.foreign_key) }
+    else
+      # Polymorphism could slow us down here, causing us to pull more records than we want to.
+      a.klass.where(a.foreign_key => prior_ids.to_a).select(:id).map { |r| r.send(:id) }
+    end
   end
 
   def copy_records
     return nil if all_ids.blank?
 
-    records = model.where(id: all_ids).load
+    records = model.where(id: all_ids.to_a).load
 
     model.establish_connection("distyll")
     records.each { |record| model.new(record.attributes).save!(validate: false) }
@@ -111,8 +111,10 @@ class DistyllModelProfile
   def set_associations
     @associations = Array.new
     model.reflect_on_all_associations.each do |association|
-      if association.belongs_to? && association.through_reflection.nil?
-        @associations << association
+      if association.through_reflection.nil?
+        if association.belongs_to? || self.include_has_many
+          @associations << association
+        end
       end
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: distyll
 version: !ruby/object:Gem::Version
-  version: 0.0.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Mason F. Matthews
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-06-03 00:00:00.000000000 Z
+date: 2014-06-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: shoulda