live_fixtures 0.2.0 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 65c289e9334bcc64e494914412b0009ba331df5d
-  data.tar.gz: 1c4e83eb54bc2e270b425e04626fe24d15b0f327
+SHA256:
+  metadata.gz: 4555eba9da64e35ef61d763132a1d2e5a280c785d0b2583fcb9055e6800eae74
+  data.tar.gz: fb54c037ff212fecb9bdbe2fd0c961df6d0acba9a0ed6dab7579c5e487656b0f
 SHA512:
-  metadata.gz: 1ce276cc748f25dd873cd6685da27f96893cba3a1fe254b3dfaf135f46c2ed2db485b9710ad9d45f53f10faa2759a54d419b00fee50c8e9efd147ae83b7a3bae
-  data.tar.gz: 16dafe289086b77e9177766fe059451258c1647d84b1c03249c83f965a176930be4d9d9578ab72ac50a2388040bd37ab24751f551398ddeeb3ccc9f48e1a4ae5
+  metadata.gz: 6f0af7f78d0638a22f2dc607b6a21b0cf24b95497fe974c4f0c1b1d7c4b895048deac44ec55576391ccf67bc297e8aa318e841ed1ef1669eb286b8cdeba3f932
+  data.tar.gz: 12a8488d4e93c45462e7362926f057571f54f4afe2640ff94dc422b8c3bbc9a4ff0a6b4a064962e15ca13b75c2abb60e1f016495189a28e3d14b627504152c32

data/CHANGELOG.md

@@ -2,7 +2,53 @@
 All notable changes to this project will be documented in this file.
 This project adheres to [Semantic Versioning](http://semver.org/).
 
-## [0.2.0] - (Unreleased)
+## [2.0.0] - 2020-11-11
+### Breaking changes
+This is a breaking change because LiveFixtures::Import.new now needs to receive the class_names Hash to be able to correctly compute the insert_order in case there are some unconventional class names associations. And the class_names argument is removed from import_all. But this is the only change.
+
+### Added
+  - compute insert_order automatically (#33)
+
+## [1.0.1] - 2019-04-10
+### Fixed
+  - fixed incompatibility with mysql
+
+### Added
+  - mysql regression test, confirmation that this gem doesn't work with psotgres
+
+## [1.0.0] - 2019-02-15
+### Breaking changes
+ - drop support for rails 4.2, ruby < 2.3
+
+### Fixed
+ - support for rails 5
+
+### Added
+ - None
+
+## [0.3.1] - 2018-03-28
+### Breaking changes
+ - None
+
+### Fixed
+ - None
+
+### Added
+ - It is now possible to export an attribute named "id" when it is included among the [additional attributes](https://github.com/NoRedInk/live_fixtures/tree/3868aaddbeb1c0174261673855610c4f8d9e7842#additional-attributes). #25
+
+## [0.3.0] - 2017-08-10
+### Breaking changes
+ - Imports now raise an error when unable to find a referenced model.
+   To avoid this behavior, pass the option `skip_missing_refs: true`. #24
+
+### Fixed
+ - For models with serialized attributes (for storing a ruby object or json blob in a single column, for example), live_fixtures will now use the coder specified in the model definition to dump the value to a string, rather than serializing it to yaml. #10, #11
+
+### Added
+ - Enhanced documentation #1, #12
+ - Options to suppress progress bar output and import errors. #23
+
+## [0.2.0] - 2017-05-09
 ### Breaking change
 - live_fixtures now depends on activerecord ~> 4.2
 

data/README.md

@@ -1,16 +1,26 @@
 # LiveFixtures
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/live_fixtures`. To experiment with that code, run `bin/console` for an interactive prompt.
+Like ActiveRecord::Fixtures, but for production.
 
-TODO: Delete this and the text above, and describe your gem
+> A test fixture is a fixed state of a set of objects used as a baseline for running tests.
+> The purpose of a test fixture is to ensure that there is a well known and fixed environment in which tests are run so that results are repeatable.
+>
+> [https://github.com/junit-team/junit4/wiki/test-fixtures](https://github.com/junit-team/junit4/wiki/test-fixtures)
+
+[ActiveRecord::Fixtures](http://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html) provide a powerful way to populate a database with test fixtures, but its strategy for handling primary keys and associations is not intended for use with a production db.
+
+LiveFixtures uses a different strategy that means it is safer to use in a live environment.
+
+For more information, see [the motivation section below](#motivation).
+
+## Compatibility
+LiveFixtures is tested against Sqlite3 & mysql. It is known to be incompatible with postgres.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-```ruby
-gem 'live_fixtures'
-```
+    gem 'live_fixtures'
 
 And then execute:
 
@@ -22,7 +32,256 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+This gem provides functionality for both the export and the import of fixtures.
+
+While they work nicely together, it is also possible to import manually generated fixtures.
+
+### Exporting
+
+The `LiveFixtures::Export` module is meant to be included into your export class.
+
+
+    class Export::User
+      include LiveFixtures::Export
+
+      def export(user_id)
+        set_export_dir "#{Rails.root}/data/export/user/#{user_id}/"
+
+        export_user_and_posts(user_id)
+      end
+
+      def export_user_and_posts(user_id)
+        user = User.find(user_id)
+        export_fixtures([user])
+
+        export_fixtures user.posts, :user do |post|
+          { "likes" => post.likes.count,
+            "unique_url" => Template.new("<%= Export::Helper.unique_url %>") }
+        end
+      end
+    end
+
+
+1. Call #set_export_dir to set the dir where files should be created.
+   If the dir does not already exist, it will be created for you.
+
+2. Then call #export_fixtures for each db table, which will produce
+   one yml file for each db table. Do *not* call export_fixtures multiple
+   times for the same db table - that will overwrite the file each time!
+
+3. You can optionally call #set_export_options, passing {show_progress: false}
+   if you'd like to disable the progress bar.
+
+4. For advanced usage, read the sections about Additional Attributes, References, and Templates.
+
+### Importing
+
+The `LiveFixtures::Import` class allows you to specify the location of your fixtures and, optionally, the order in which to import them. If you don't specify an order, the order will be computed from the ActiveRecord models associations. Once you've done that, you can import them directly to your database.
+
+
+    module Seed::User
+      def self.from_fixtures(fixtures_directory)
+        insert_order = %w{users posts}
+
+        importer = LiveFixtures::Import.new fixtures_directory, insert_order
+        importer.import_all
+      end
+    end
+
+Options may be passed when initializing an importer as follow:
+ - show_progress: defaults to true.
+   Pass false to disable the progress bar output.
+ - skip_missing_tables: defaults to false.
+   Pass true to avoid raising an error when a table listed in insert_order has
+   no yml file.
+ - skip_missing_refs: defaults to false.
+   Pass false to raise an error when the importer is unable to re-establish a
+   relation.
+
+## Advanced Usage
+
+The following topics reference this schema and exporter:
+
+    class User < ActiveRecord::Base
+      has_many :posts
+    end
+
+    class Post < ActiveRecord::Base
+      belongs_to :user
+      has_and_belongs_to_many :channels
+    end
+
+    class Channel < ActiveRecord::Base
+      has_and_belongs_to_many :users
+    end
+
+    class YourExporter
+      include LiveFixtures::Export
+      def initialize(fixture_path)
+        set_export_dir fixture_path
+      end
+
+      def export_models(models, references = [], &additional_attributes)
+        export_fixtures(models, references, &additional_attributes)
+      end
+    end
+
+### Additional Attributes
+
+We can use a block to add more attributes to a fixture. Each model is passed to the block as an argument, and the block should return a hash of additional arguments.
+
+    dev_ops_posts = Post.where(topic: "Dev Ops")
+    exporter = YourExporter.new("fixtures/")
+    exporter.export_models(dev_ops_posts) do |post|
+      { summary: PostSummarizer.summarize(post) }
+    end
+
+    # In our fixtures/posts.yml file
+    posts_1234:
+      ...
+      user_id: 5678
+      summary: "Dev ops is cool."
+
+### References
+
+References allow fixtures to capture a model's associations, so they can be correctly re-established on import.
+
+When we export a fixture for a post above, we'd expect to see an attribute `user_id`
+
+    post = Post.find(1234)
+    post.user_id = 5678
+    post.save!
+    exporter = YourExporter.new("fixtures/")
+    exporter.export_models([post])
+
+    # In our posts.yml file
+    posts_1234:
+      ...
+      user_id: 5678
+
+
+If we import this post, it will still have 5678 for its user_id foreign key. This may or may not be the desired outcome.
+
+If we pass `:user` as references, LiveFixtures will replace the foreign key with a reference, so that the association can be correctly re-established on import:
+
+    post = Post.find(1234)
+    post.user_id = 5678
+    post.save!
+    exporter = YourExporter.new("fixtures/")
+    exporter.export_models([post], :user)
+    exporter.export_models([post.user])
+
+    # In our posts.yml file
+    posts_1234:
+      ...
+      user: users_5678
+
+    # In our users.yml file
+    users_5678:
+      ...
+
+When we import these fixtures using the correct `insert_order` (`['users', 'posts']`), the newly imported post will belong to the newly imported user, no matter what their new ids are.
+
+Currently, this works for belongs_to and has_and_belongs_to_many associations.
+
+For has_and_belongs_to_many relation, add a field to one of the records, and the import will populate the join table.
+
+The formatting of the fixture is quite flexible, and the value for this field can be either a list or comma-separated string containing either references or IDs of the associated records. In all cases, though, the value should match the association name. Note that in all cases below the key is `channels` and not `channel_ids`:
+
+    # In our users.yml file
+    users_5678:
+      channels: "1,2,3"
+
+    users_1234:
+      channels:
+        - channel_1
+        - bobs_cool_channel
+
+    # In our channels.yml file
+    channel_1:
+      ...
+
+    bobs_cool_channel:
+      ...
+
+Also note it's not necessary to format your references the way the exporter does - `bobs_cool_channel` is a totally valid reference.
+
+### Templates
+
+Templates allow you to export fixtures containing erb, that will be evaluated at the time of fixture import.
+
+    posts = Post.where(user_id: 5678
+    exporter = YourExporter.new("fixtures/")
+    exporter.export_models([post]) do |post|
+      { unique_promo_code: Template.new("<%= PromoCodeGenerator.unique_promo_code(#{post.id}) %>")
+    end
+
+    # In our fixtures/posts.yml file
+    posts_1234:
+      ...
+      user_id: 5678
+      unique_promo_code: <%= PromoCodeGenerator.unique_promo_code(1234) %>
+
+In the example above, we'd be able to generate a new unique promo code for each post as we import them.
+
+
+## Motivation
+
+One particular challenge when working with fixtures is describing associations between records. When they're in the database, records have unique primary keys, and associations are captured using foreign keys (references to the associated record's primary key). For example, a row in the `posts` table may have a column `user_id`. If it's value is `1234`, it indicated that `Post` belongs to the `User` with id `1234`.
+
+How can we model associations when fixtures are removed from the database? It's not enough to just serialize the foreign keys, as we expect each record to have a different primary key each time it is imported into a database.
+
+ActiveRecord::Fixtures answers this question in a way that is very effective for a test database, but that is not safe for a live database.
+
+### Here's how ActiveRecord::Fixtures work
+
+Each record is assigned a label in its yml file. Primary key values are
+assigned using a guid algorithm that maps a label to a consistent integer
+between 1 and 2^30-1. Primary keys can then be assigned before saving any
+records to the db.
+
+Why would they do this? Because, this enables us to use labels in the
+Fixture yml files to refer to associations. For example:
+
+      <users.yml>
+      bob:
+        username: thebob
+
+      <posts.yml>
+      hello:
+        message: Hello everyone!
+        user: bob
+
+
+
+The ActiveRecord::Fixture system first converts every instance of `bob` and
+`hello` into an integer using ActiveRecord::Fixture#identify, and then can
+save the records IN ANY ORDER and know that all foreign keys will be valid.
+
+There is a big problem with this. In a test db, each table is empty and so the
+odds of inserting a few dozen records causing a primary key collision is
+very small. However, for a production table with a hundred million rows, this
+is no longer the case! Collisions abound and db insertion fails.
+
+Also, auto-increment primary keys will continue from the LARGEST existing
+primary key value. If we insert a record at 1,000,000,000 - we've reduced the
+total number of records we can store in that table in half. Fine for a test db
+but not ideal for production.
+
+
+### LiveFixtures work differently
+
+Since we want to be able to take advantage of normal auto-increment behavior,
+we cannot know the primary keys of each record before saving it to the db.
+Instead, we save each record, and then maintain a mapping (`@label_to_id`)
+from that record's label (`bob`), to its primary key (`213`). Later, when
+another record (`hello`) references `bob`, we can use this mapping to look up
+the primary key for `bob` before saving `hello`.
+
+This means that the order we insert records into the db matters: `bob` must
+be inserted before `hello`! This order is defined in INSERT_ORDER, and
+reflected in the order of the `@table_names` array.
+
 
 ## Development
 
@@ -30,12 +289,13 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+Please remember to update the docs on your PR if you change anything. You can see the YARD docs live while you change them by running `yard server --reload`.
+
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/live_fixtures. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
+Bug reports and pull requests are welcome on GitHub at https://github.com/NoRedInk/live_fixtures. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
 
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
-

data/lib/live_fixtures.rb

@@ -1,9 +1,11 @@
 require "live_fixtures/version"
 require "live_fixtures/import"
 require "live_fixtures/import/fixtures"
+require "live_fixtures/import/insertion_order_computer"
 require "live_fixtures/export"
 require "live_fixtures/export/fixture"
 require "ruby-progressbar"
+require "yaml"
 
 module LiveFixtures
   module_function

data/lib/live_fixtures/export.rb

@@ -8,29 +8,61 @@
 #    times for the same db table - that will overwrite the file each time!
 
 module LiveFixtures::Export
+  # Templates allow you to export fixtures containing erb, that will be evaluated at the time of fixture import.
+  # You should initialize them with a String containing the erb to evaluate, like
+  # @example A template with export and import times.
+  #    Template.new("<%= \"I was exported at #{Time.now} and imported at \" + Time.now.to_s %>")
   Template  = Struct.new(:code)
+
+  # References represent associations between fixtures, in the same way that foreign_keys do for records.
+  # These will be initialized for you based on the contents of `with_references` passed to `export_fixtures`.
+  # They will be initialized with the name of the association (a Symbol) and the particular associated model.
   Reference = Struct.new(:name, :value)
 
   private
 
+  # Specify the directory into which to export the yml files containing your fixtures.
+  # The directory will be created if it does not yet exist.
+  # @param dir [String] a path to a directory into which the fixtures will be exported.
   def set_export_dir(dir)
     @dir = dir
     FileUtils.mkdir_p(@dir) unless File.directory?(@dir)
   end
 
+  # Specify the options to use when exporting your fixtures.
+  # @param [Hash] opts export configuration options
+  # @option opts [Boolean] show_progress whether or not to show the progress bar
+  def set_export_options(**opts)
+    defaults = { show_progress: true }
+    @export_options = defaults.merge(opts)
+  end
+
+  # The options to use when exporting your fixtures.
+  # @return [Hash] export configuration options
+  def export_options
+    @export_options ||= set_export_options
+  end
+
   ##
   # Export models to a yml file named after the corresponding table.
+  # @param models [Enumerable] an Enumerable containing ActiveRecord models.
+  # @param with_references [Array<Symbol>] the associations whose foreign_keys should be replaced with references.
   #
   # Takes an optional block that will be invoked for each model.
   # The block should return a hash of attributes to be merged and
   # saved with the model's attributes.
+  # @yield [model] an optional block that will be invoked for each model.
+  # @yieldparam model [ActiveRecord::Base] each successive model.
+  # @yieldreturn [Hash{String => Object}] a hash of attributes to be merged and saved with the model's attributes.
   def export_fixtures(models, with_references = [])
     return unless models.present?
 
     table_name = models.first.class.table_name
     File.open(File.join(@dir, table_name + '.yml'), 'w') do |file|
 
-      ProgressBarIterator.new(models).each do |model|
+      iterator = export_options[:show_progress] ? ProgressBarIterator : SimpleIterator
+
+      iterator.new(models).each do |model|
         more_attributes = block_given? ? yield(model) : {}
         file.write Fixture.to_yaml(model, with_references, more_attributes)
       end
@@ -54,4 +86,17 @@ module LiveFixtures::Export
       end
     end
   end
+
+  class SimpleIterator
+    def initialize(models)
+      @models = models
+    end
+
+    def each
+      puts @models.first.class.name
+      @models.each do |model|
+        yield model
+      end
+    end
+  end
 end

data/lib/live_fixtures/export/fixture.rb

@@ -1,5 +1,13 @@
+# Exposes functionality to serialize an ActiveRecord record (a model) into a
+# YAML fixture.
 module LiveFixtures::Export::Fixture
   module_function
+  # YAML-Serializes the provided model, including any references and additional
+  # attribtues.
+  # @param model [ActiveRecord::Base] an ActiveRecord record to serialize
+  # @param references [Symbol, Array<Symbol>] the names of associations whose foreign_keys should be replaced with references
+  # @param more_attributes [Hash{String => Time, DateTime, Date, Hash, String, LiveFixtures::Export::Template, LiveFixtures::Export::Reference, #to_s}] a hash of additional attributes to serialize with each record.
+  # @return [String] the model serialized in YAML, with specified foreign_keys replaced by references, including additional attributes.
   def to_yaml(model, references = [], more_attributes = {})
     table_name = model.class.table_name
 
@@ -28,29 +36,44 @@ module LiveFixtures::Export::Fixture
   end
 
   private_class_method def yml_attributes(model, more_attributes)
-    model.attributes.merge(more_attributes).map do |name, value|
-      next if %w{id}.include? name
+    model.attributes.except("id").merge(more_attributes).map do |name, value|
       next if value.nil?
 
-      yml_value ||= case value
-                      when Time, DateTime
-                        value.utc.to_s(:db)
-                      when Date
-                        value.to_s(:db)
-                      when Hash
-                        value.to_yaml.inspect
-                      when String
-                        value.inspect
-                      when LiveFixtures::Export::Template
-                        value.code
-                      when LiveFixtures::Export::Reference
-                        name, value = value.name, value.value
-                        "#{value.class.table_name}_#{value.id}"
-                      else
-                        value.to_s
-                    end
-
-      "#{name}: " + yml_value
+      serialize_attribute(model, name, value)
     end.compact.join("\n  ")
   end
+
+  private_class_method def serialize_attribute(model, name, value)
+    attribute_type = model.class.type_for_attribute(name)
+
+    if attribute_type.is_a?(ActiveRecord::Type::Serialized)
+      value = attribute_type.serialize(value) unless value.is_a?(String)
+
+      "#{name}: |-\n#{value.to_s.indent(4)}" unless value.nil?
+    elsif value.is_a? LiveFixtures::Export::Reference
+      "#{value.name}: #{yml_value(value)}"
+    else
+      "#{name}: #{yml_value(value)}"
+    end
+  end
+
+  private_class_method def yml_value(value)
+    case value
+    when Time, DateTime
+      value.utc.to_s(:db)
+    when Date
+      value.to_s(:db)
+    when Hash
+      value.to_yaml.inspect
+    when String
+      value.inspect
+    when LiveFixtures::Export::Template
+      value.code
+    when LiveFixtures::Export::Reference
+      reference_value = value.value
+      "#{reference_value.class.table_name}_#{reference_value.id}"
+    else
+      value.to_s
+    end
+  end
 end

data/lib/live_fixtures/import.rb

@@ -1,82 +1,58 @@
-# ActiveRecord::Fixtures are a powerful way of populating data in a db;
-# however, its strategy for handling primary keys and associations is
-# UNACCEPTABLE for use with a production db. LiveFixtures works around this.
-#
-#
-########### Here's how ActiveRecord::Fixtures work ###########################
-#
-# Each record is assigned a label in its yml file. Primary key values are
-# assigned using a guid algorithm that maps a label to a consistent integer
-# between 1 and 2^30-1. Primary keys can then be assigned before saving any
-# records to the db.
-#
-# Why would they do this? Because, this enables us to use labels in the
-# Fixture yml files to refer to associations. For example:
-#
-#       <users.yml>
-#       bob:
-#         username: thebob
-#
-#       <posts.yml>
-#       hello:
-#         message: Hello everyone!
-#         user: bob
-#
-# The ActiveRecord::Fixture system first converts every instance of `bob` and
-# `hello` into an integer using ActiveRecord::Fixture#identify, and then can
-# save the records IN ANY ORDER and know that all foreign keys will be valid.
-#
-# There is a big problem with this. In a test db, each table is empty and so the
-# odds of inserting a few dozen records causing a primary key collision is
-# very small. However, for a production table with a hundred million rows, this
-# is no longer the case! Collisions abound and db insertion fails.
-#
-# Also, autoincrement primary keys will continue from the LARGEST existing
-# primary key value. If we insert a record at 1,000,000,000 - we've reduced the
-# total number of records we can store in that table in half. Fine for a test db
-# but not ideal for production.
-#
-#
-########### LiveFixtures work differently ####################################
-#
-# Since we want to be able to take advantage of normal autoincrement behavior,
-# we cannot know the primary keys of each record before saving it to the db.
-# Instead, we save each record, and then maintain a mapping (`@label_to_id`)
-# from that record's label (`bob`), to its primary key (`213`). Later, when
-# another record (`hello`) references `bob`, we can use this mapping to look up
-# the primary key for `bob` before saving `hello`.
-#
-# This means that the order we insert records into the db matters: `bob` must
-# be inserted before `hello`! This order is defined in INSERT_ORDER, and
-# reflected in the order of the `@table_names` array.
-
+# An object that facilitates the import of fixtures into a database.
 class LiveFixtures::Import
   NO_LABEL = nil
 
-  def initialize(root_path, insert_order)
+  # Returns the insert order that was specified in the constructor or
+  # the inferred one if none was specified.
+  attr_reader :insert_order
+
+  # Instantiate a new Import with the directory containing your fixtures, and
+  # the order in which to import them. The order should ensure fixtures
+  # containing references to another fixture are imported AFTER the referenced
+  # fixture.
+  # @raise [ArgumentError] raises an argument error if not every element in the insert_order has a corresponding yml file.
+  # @param root_path [String] path to the directory containing the yml files to import.
+  # @param insert_order [Array<String> | Nil] a list of yml files (without .yml extension) in the order they should be imported, or `nil` if these order is to be inferred by this class.
+  # @param class_names [Hash{Symbol => String}] a mapping table name => Model class, for any that don't follow convention.
+  # @param [Hash] opts export configuration options
+  # @option opts [Boolean] show_progress whether or not to show the progress bar
+  # @option opts [Boolean] skip_missing_tables when false, an error will be raised if a yaml file isn't found for each table in insert_order
+  # @option opts [Boolean] skip_missing_refs when false, an error will be raised if an ID isn't found for a label.
+  # @return [LiveFixtures::Import] an importer
+  # @see LiveFixtures::Export::Reference
+  def initialize(root_path, insert_order = nil, class_names = {}, **opts)
+    defaut_options = { show_progress: true, skip_missing_tables: false, skip_missing_refs: false }
+    @options = defaut_options.merge(opts)
     @root_path = root_path
     @table_names = Dir.glob(File.join(@root_path, '{*,**}/*.yml')).map do |filepath|
       File.basename filepath, ".yml"
     end
-    @table_names = insert_order.select {|table_name| @table_names.include? table_name}
-    if @table_names.size < insert_order.size
-      raise ArgumentError, "table(s) mentioned in `insert_order` which has no yml file to import: #{insert_order - @table_names}"
+
+    @class_names = class_names
+    @table_names.each { |n|
+      @class_names[n.tr('/', '_').to_sym] ||= n.classify if n.include?('/')
+    }
+
+    @insert_order = insert_order
+    @insert_order ||= InsertionOrderComputer.compute(@table_names, @class_names, compute_polymorphic_associations)
+
+    @table_names = @insert_order.select {|table_name| @table_names.include? table_name}
+    if @table_names.size < @insert_order.size && !@options[:skip_missing_tables]
+      raise ArgumentError, "table(s) mentioned in `insert_order` which has no yml file to import: #{@insert_order - @table_names}"
     end
     @label_to_id = {}
   end
 
-  # https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L496
+  # Within a transaction, import all the fixtures into the database.
+  #
   # The very similar method: ActiveRecord::FixtureSet.create_fixtures has the
   # unfortunate side effect of truncating each table!!
   #
   # Therefore, we have reproduced the relevant sections here, without DELETEs,
-  # with calling `LF::Import::Fixtures#each_table_row_with_label` instead of
+  # with calling {LiveFixtures::Import::Fixtures#each_table_row_with_label} instead of
   # `AR::Fixtures#table_rows`, and using those labels to populate `@label_to_id`.
-  def import_all(class_names = {})
-    @table_names.each { |n|
-      class_names[n.tr('/', '_').to_sym] ||= n.classify if n.include?('/')
-    }
-
+  # @see https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L496
+  def import_all
     connection = ActiveRecord::Base.connection
 
     files_to_read = @table_names
@@ -85,22 +61,70 @@ class LiveFixtures::Import
       connection.transaction(requires_new: true) do
         files_to_read.each do |path|
           table_name = path.tr '/', '_'
-          class_name = class_names[table_name.to_sym] || table_name.classify
+          class_name = @class_names[table_name.to_sym] || table_name.classify
 
           ff = Fixtures.new(connection,
                             table_name,
                             class_name,
                             ::File.join(@root_path, path),
-                            @label_to_id)
+                            @label_to_id,
+                            skip_missing_refs: @options[:skip_missing_refs])
 
           conn = ff.model_connection || connection
-          ProgressBarIterator.new(ff).each do |table_name, label, row|
+          iterator = @options[:show_progress] ? ProgressBarIterator : SimpleIterator
+          iterator.new(ff).each do |table_name, label, row|
             conn.insert_fixture(row, table_name)
-            @label_to_id[label] = conn.last_inserted_id(table_name) unless label == NO_LABEL
+            @label_to_id[label] = conn.send(:last_inserted_id, table_name) unless label == NO_LABEL
+          end
+        end
+      end
+    end
+  end
+
+  private
+
+  # Here we go through each of the fixture YAML files to see what polymorphic
+  # dependencies exist for each of the models.
+  # We do this by inspecting the value of any field that ends with `_type`,
+  # for example `author_type`, `assignment_type`, etc.
+  # Becuase we can't know all the possible types of a polymorphic association
+  # we compute them from the YAML file contents.
+  # Returns a Hash[Class => Set[Class]]
+  def compute_polymorphic_associations
+    polymorphic_associations = Hash.new { |h, k| h[k] = Set.new }
+
+    connection = ActiveRecord::Base.connection
+    files_to_read = @table_names
+
+    files_to_read.each do |path|
+      table_name = path.tr '/', '_'
+      class_name = @class_names[table_name.to_sym] || table_name.classify
+
+      # Here we use the yaml file and YAML.load instead of ActiveRecord::FixtureSet.new
+      # because it's faster and we can also check whether we actually need to
+      # load the file: only if it includes "_type" in it, otherwise there will be
+      # no polymorphic types in there.
+
+      filename = ::File.join(@root_path, "#{path}.yml")
+      file = File.read(filename)
+      next unless file =~ /_type/
+
+      yaml = YAML.load(file)
+      yaml.each do |key, object|
+        object.each do |field, value|
+          next unless  field.ends_with?("_type")
+
+          begin
+            polymorphic_associations[class_name.constantize] << value.constantize
+          rescue NameError
+            # It might be the case that the `..._type` field doesn't actually
+            # refer to a type name, so we just ignore it.
           end
         end
       end
     end
+
+    polymorphic_associations
   end
 
   class ProgressBarIterator
@@ -120,4 +144,17 @@ class LiveFixtures::Import
       @bar.finish
     end
   end
+
+  class SimpleIterator
+    def initialize(ff)
+      @ff = ff
+    end
+
+    def each
+      puts @ff.model_class.name
+      @ff.each_table_row_with_label do |*args|
+        yield(*args)
+      end
+    end
+  end
 end

data/lib/live_fixtures/import/fixtures.rb

@@ -1,13 +1,23 @@
 require 'active_record/fixtures'
 
-#rubocop:disable Style/PerlBackrefs
-
 class LiveFixtures::Import
+  # A labeled reference was not found.
+  # Maybe the referenced model was not exported, or the insert order attempted
+  # to import the reference before the referenced model?
+  LiveFixtures::MissingReferenceError = Class.new(KeyError)
+
   class Fixtures
     delegate :model_class, :table_name, :fixtures, to: :ar_fixtures
+    # ActiveRecord::FixtureSet for delegation
     attr_reader :ar_fixtures
 
-    def initialize(connection, table_name, class_name, filepath, label_to_id)
+    # @param connection [ActiveRecord::ConnectionAdapters::AbstractAdapter] connection to the database into which to import the data.
+    # @param table_name [String] name of the database table to populate with models
+    # @param class_name [Constant] the model's class name
+    # @param filepath [String] path to the yml file containing the fixtures
+    # @param label_to_id [Hash{String => Int}] map from a reference's label to its new id.
+    def initialize(connection, table_name, class_name, filepath, label_to_id, skip_missing_refs: false)
+      @skip_missing_refs = skip_missing_refs
       @ar_fixtures = ActiveRecord::FixtureSet.new connection,
         table_name,
         class_name,
@@ -15,12 +25,16 @@ class LiveFixtures::Import
       @label_to_id = label_to_id
     end
 
-    # https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L611
     # Rewritten to take advantage of @label_to_id instead of AR::FixtureSet#identify,
     # and to make an iterator.
     #
+    # @yieldparam table_name [String] the database table's name
+    # @yieldparam label [String] the label for the model being currently imported
+    # @yieldparam row [Hash{String => Value}] the model's attributes to be imported
     # Iterator which yields [table_name, label, row] for each fixture
-    # (and for any implicit join table records)
+    # (and for any implicit join table records). The block is expected to insert
+    # the row and update @label_to_id with the record's newly assigned id.
+    # @see https://github.com/rails/rails/blob/4-2-stable/activerecord/lib/active_record/fixtures.rb#L611
     def each_table_row_with_label
       join_table_rows = Hash.new { |h,table| h[table] = [] }
 
@@ -55,7 +69,7 @@ class LiveFixtures::Import
         rows.each do |targets:, association:, label:|
           targets.each do |target|
             assoc_fk = @label_to_id[target] || target
-            row = { association.foreign_key             => @label_to_id[label],
+            row = { association.foreign_key             => fetch_id_for_label(label),
                     association.association_foreign_key => assoc_fk }
             yield [table_name, NO_LABEL, row]
           end
@@ -95,7 +109,25 @@ class LiveFixtures::Import
         row[association.foreign_type] = $1
       end
 
-      row[fk_name] = @label_to_id[value]
+      row[fk_name] = fetch_id_for_label(value)
+    end
+
+    # Uses the underlying map of labels to return the referenced model's newly
+    # assigned ID.
+    # @raise [LiveFixtures::MissingReferenceError] if the label isn't found.
+    # @param label_to_fetch [String] the label of the referenced model.
+    # @return [Integer] the newly assigned ID of the referenced model.
+    def fetch_id_for_label(label_to_fetch)
+      @label_to_id.fetch(label_to_fetch)
+    rescue KeyError
+      return nil if @skip_missing_refs
+
+      raise LiveFixtures::MissingReferenceError, <<-ERROR.squish
+      Unable to find ID for model referenced by label #{label_to_fetch} while
+      importing #{model_class} from #{table_name}.yml. Perhaps it isn't included
+      in these fixtures or it is too late in the insert_order and has not yet
+      been imported.
+      ERROR
     end
 
     private :ar_fixtures

data/lib/live_fixtures/import/insertion_order_computer.rb

@@ -0,0 +1,127 @@
+class LiveFixtures::Import
+  # :nodoc:
+  class InsertionOrderComputer
+    # :nodoc:
+    class Node
+      attr_reader :path
+      attr_reader :class_name
+      attr_reader :klass
+
+      # The classes this node depends on
+      attr_reader :dependencies
+
+      def initialize(path, class_name, klass)
+        @path = path
+        @class_name = class_name
+        @klass = klass
+        @dependencies = Set.new
+      end
+    end
+
+    def self.compute(table_names, class_names = {}, polymorphic_associations = {})
+      new(table_names, class_names, polymorphic_associations).compute
+    end
+
+    def initialize(table_names, class_names = {}, polymorphic_associations = {})
+      @table_names = table_names
+      @class_names = class_names
+      @polymorphic_associations = polymorphic_associations
+    end
+
+    def compute
+      nodes = build_nodes
+      compute_insert_order(nodes)
+    end
+
+    private
+
+    # Builds an Array of Nodes, each containing dependencies to other nodes
+    # using their class names.
+    def build_nodes
+      # Create a Hash[Class => Node] for each table/class
+      nodes = {}
+      @table_names.each do |path|
+        table_name = path.tr "/", "_"
+        class_name = @class_names[table_name.to_sym] || table_name.classify
+        klass = class_name.constantize
+        nodes[klass] = Node.new(path, class_name, klass)
+      end
+
+      # First iniitalize dependencies from polymorphic associations that we
+      # explicitly found in the yaml files.
+      @polymorphic_associations.each do |klass, associations|
+        associations.each do |association|
+          node = nodes[klass]
+          next unless node
+          next unless nodes.key?(association)
+
+          node.dependencies << association
+        end
+      end
+
+      # Compute dependencies between nodes/classes by reflecting on their
+      # ActiveRecord associations.
+      nodes.each do |_, node|
+        klass = node.klass
+        klass.reflect_on_all_associations.each do |assoc|
+          # We can't handle polymorphic associations, but the concrete types
+          # should have been deduced from the yaml files contents
+          next if assoc.polymorphic?
+
+          # Don't add a dependency if the class is not in the given table names
+          next unless nodes.key?(assoc.klass)
+
+          # A class might depend on itself, but we don't add it as a dependency
+          # because otherwise we'll never make it (the class can probably be created
+          # just fine and these dependencies are optional/nilable)
+          next if klass == assoc.klass
+
+          case assoc.macro
+          when :belongs_to
+            node.dependencies << assoc.klass
+          when :has_one, :has_many
+            # Skip `through` association becuase it will be already computed
+            # for the related `has_one`/`has_many` association
+            next if assoc.options[:through]
+
+            nodes[assoc.klass].dependencies << klass
+          end
+        end
+      end
+
+      # Finally sort all values by name for consistent results
+      nodes.values.sort_by { |node| node.klass.name }
+    end
+
+    def compute_insert_order(nodes)
+      insert_order = []
+
+      until nodes.empty?
+        # Pick a node that has no dependencies
+        free_node = nodes.find { |node| node.dependencies.empty? }
+
+        if free_node.nil?
+          msg = "Can't compute an insert order.\n\n"
+          msg << "These models seem to depend on each other:\n"
+          nodes.each do |node|
+            msg << "  #{node.klass.name}\n"
+            msg << "   - depends on: #{node.dependencies.map(&:name).join(", ")}\n"
+          end
+          raise msg
+        end
+
+        insert_order << free_node.path
+
+        # Delete this node from the other nodes' dependencies
+        nodes.each do |node|
+          node.dependencies.delete(free_node.klass)
+        end
+
+        # And delete this node because we are done with it
+        nodes.delete(free_node)
+      end
+
+      insert_order
+    end
+  end
+end

data/lib/live_fixtures/version.rb

@@ -1,3 +1,3 @@
 module LiveFixtures
-  VERSION = "0.2.0"
+  VERSION = "2.0.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: live_fixtures
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 2.0.0
 platform: ruby
 authors:
 - jleven
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2017-05-09 00:00:00.000000000 Z
+date: 2020-11-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -16,14 +16,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.2'
+        version: '5.2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.2'
+        version: '5.2'
 - !ruby/object:Gem::Dependency
   name: ruby-progressbar
   requirement: !ruby/object:Gem::Requirement
@@ -44,56 +44,56 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.11'
+        version: '2.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.11'
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '12.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '12.3'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.8'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.8'
 - !ruby/object:Gem::Dependency
   name: temping
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.10'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '3.10'
 - !ruby/object:Gem::Dependency
   name: byebug
   requirement: !ruby/object:Gem::Requirement
@@ -110,6 +110,62 @@ dependencies:
         version: '0'
 - !ruby/object:Gem::Dependency
   name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.13
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.3.13
+- !ruby/object:Gem::Dependency
+  name: mysql2
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: reek
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -122,7 +178,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: 
+description:
 email:
 - josh@noredink.com
 executables: []
@@ -137,12 +193,13 @@ files:
 - lib/live_fixtures/export/fixture.rb
 - lib/live_fixtures/import.rb
 - lib/live_fixtures/import/fixtures.rb
+- lib/live_fixtures/import/insertion_order_computer.rb
 - lib/live_fixtures/version.rb
-homepage: 
+homepage:
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -157,9 +214,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.12
-signing_key: 
+rubygems_version: 3.0.3
+signing_key:
 specification_version: 4
 summary: Tools for exporting and importing between databases managed by ActiveRecord.
 test_files: []