test_data 0.2.1 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 84a9fc3fe7b3dda0cf1ebdc775c9191757d21de11e033189c6450b91f63332e7
-  data.tar.gz: 0a47179f0607d9c57f63706e35de176b14d93989c6e5406094524a0fb3b09704
+  metadata.gz: 18d3469e9d0173f81ef3b2d8d082861094d79354b8fb544e983bdcfdd8eb788b
+  data.tar.gz: 983a1645132b22024799670ec463f11fe5f8e0d1b9427b96940dea8c684fbafb
 SHA512:
-  metadata.gz: 4c10de3f9e234ae53e47553d099612dc7474cb3e98022b6a953d60beb8da3f502b00f506f29729dad0a9eb37c8e7f05a662263963300d7c531f48fef660c9173
-  data.tar.gz: d718e34070ca15bd9d0e27c75789e170e42da183fac8efc973404d9cea909ce984453f9393277b37ad36ff42fa4866c66de14d2a82a149ae536f0555c2f5f52d
+  metadata.gz: c7fb1b86f23f074bb185cb1cf11c437d4ba57445d65907509384838993628c5759803ee90f3f290dbb219754806bb1c1133b7e84bf430918c1a23b3fca86b752
+  data.tar.gz: d00e512b415c74ab1650a3b3d58e36759d085056375e07326d8790d0033bc18066a2d82c791a578fb9d11cb7c08f9056d4d5e380b9cc52f97dd467c245469771

data/CHANGELOG.md

@@ -1,4 +1,9 @@
-# unreleased
+# 0.2.2
+
+- Improve performance of Rails fixtures being repeatedly loaded by changing the
+caching strategy
+
+# 0.2.1
 
 - Adds several lifecycle hooks:
   - config.after_test_data_load

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    test_data (0.2.1)
+    test_data (0.2.2)
       railties (~> 6.0)
 
 GEM
@@ -34,14 +34,14 @@ GEM
     erubi (1.10.0)
     i18n (1.8.10)
       concurrent-ruby (~> 1.0)
-    loofah (2.10.0)
+    loofah (2.11.0)
       crass (~> 1.0.2)
       nokogiri (>= 1.5.9)
     method_source (1.0.0)
-    mini_portile2 (2.5.3)
+    mini_portile2 (2.6.1)
     minitest (5.14.4)
-    nokogiri (1.11.7)
-      mini_portile2 (~> 2.5.0)
+    nokogiri (1.12.0)
+      mini_portile2 (~> 2.6.1)
       racc (~> 1.4)
     parallel (1.20.1)
     parser (3.0.1.0)

data/README.md

@@ -375,17 +375,30 @@ RSpec.describe "Kitchen sink", type: :request do
 end
 ```
 
+But wait, there's more! If your test suite switches between multiple modes from
+test-to-test, it's important to be aware of the marginal cost _between_ each of
+those tests. For example, two tests in a row that call `TestData.uses_test_data`
+only need a simple rollback as test setup, but a `TestData.uses_test_data`
+followed by a `TestData.uses_clean_slate` requires a rollback, a truncation, and
+another savepoint. These small costs add up, so consider [speeding up your
+build](#im-worried-my-tests-arent-as-fast-as-they-should-be) by grouping your
+tests into sub-suites based on their source of test data.
+
 #### If your situation is more complicated
 
 If you're adding `test_data` to an existing application, it's likely that you
 won't be able to easily adopt a one-size-fits-all approach to test setup across
 your entire suite. Some points of reference, if that's the situation you're in:
 
-* If your test suite is already using fixtures or factories and the above hooks
-  just broke everything, check out our [interoperability
+* If your test suite is **already using fixtures or factories** and the above
+  hooks just broke everything, check out our [interoperability
   guide](#factory--fixture-interoperability-guide) for help.
-* If you don't want `test_data` managing transactions and cleanup for you and
-  just want to load the SQL dump, you can call
+* If you need to make any changes to the data after it's loaded, truncated, or
+  after Rails fixtures are loaded, you can configure [lifecycle
+  hooks](#lifecycle-hooks) that will help you achieve a **very fast test suite**
+  by including those changes inside the transaction savepoints
+* If you **don't want `test_data` managing transactions** and cleanup for you
+  and just want to load the SQL dump, you can call
   [TestData.insert_test_data_dump](#testdatainsert_test_data_dump)
 * For more information on how all this works, see the [API
   reference](#api-reference).
@@ -489,15 +502,15 @@ suite after following the initial setup guide and see if the suite just passes.
 
 If you find that your test suite is failing after adding
 `TestData.uses_test_data` to your setup, don't panic! Test failures are most
-likely caused by the combination of your `test_data` database with the data
-persisted by your factories.
+likely caused by the combination of your `test_data` SQL dump with the records
+inserted by your factories.
 
 One approach would be to attempt to resolve each such failure one-by-one—usually
 by updating the offending factories or editing your `test_data` database to
 ensure they steer clear of one another. Care should be taken to preserve the
-conceptual encapsulation of each test, however, as naively squashing errors can
-introduce inadvertent coupling between your factories and your `test_data`
-database such that neither can be used independently of the other.
+conceptual encapsulation of each test, however, as naively squashing errors
+risks introducing inadvertent coupling between your factories and your
+`test_data` data such that neither can be used independently of the other.
 
 Another approach that the `test_data` gem provides is an additional mode with
 `TestData.uses_clean_slate`, which—when called at the top of a factory-dependent
@@ -1203,8 +1216,113 @@ run. That said—and especially if you're adding `test_data` to an existing test
 suite—care should be taken to audit everything the suite does between tests in
 order to optimize its overall runtime.
 
-The first and most likely source of unnecessary slowness is redundant test
-cleanup—the speed gained from sandwiching every expensive operation between
+#### Randomized test order leading to data churn
+
+Generally speaking, randomizing the order in which tests run is an unmitigated
+win: randomizing helps you catch any unintended dependency between two tests
+early, when it's still cheap & easy to fix. However, if your tests use different
+sources of test data (e.g. some call `TestData.uses_test_data` and some call
+`TestData.uses_clean_slate`), it's very likely that randomizing your tests will
+result in a significantly slower overall test suite. Instead, if you group tests
+that use the same type of test data together (e.g. by separating them into
+separate suites), you might find profound speed gains.
+
+To illustrate why, suppose you have 5 tests that call `TestData.uses_test_data`
+and 5 that call `TestData.uses_rails_fixtures`. If a test that calls
+`TestData.uses_test_data` is followed by another that calls `uses_test_data`,
+the only operation needed by the second call will be a rollback to the savepoint
+taken after the test data was loaded. If, however, a `uses_test_data` test is
+followed by a `uses_rails_fixtures` test, then a lot more work is required:
+first a rollback, then the truncation of the test data, then a load of the
+fixtures followed by creation of a new savepoint—which would in tunr be undone
+again if the _next_ test happened to call `uses_test_data`. Switching between
+tests that use different sources of test data can cause significant unnecessary
+thrashing.
+
+To illustrate the above, if all of these tests ran in random order (the
+default), you might see:
+
+```
+$ bin/rails test test/example_test.rb
+Run options: --seed 63999
+
+# Running:
+
+   test_data -- loading test_data SQL dump
+.  fixtures  -- truncating tables, loading Rails fixtures
+.  fixtures  -- rolling back to Rails fixtures
+.  test_data -- rolling back to clean test_data
+.  fixtures  -- truncating tables, loading Rails fixtures
+.  test_data -- rolling back to clean test_data
+.  fixtures  -- truncating tables, loading Rails fixtures
+.  test_data -- rolling back to clean test_data
+.  fixtures  -- truncating tables, loading Rails fixtures
+.  test_data -- rolling back to clean test_data
+.
+
+Finished in 2.449957s, 4.0817 runs/s, 4.0817 assertions/s.
+10 runs, 10 assertions, 0 failures, 0 errors, 0 skips
+```
+
+So, what can you do to speed this up? The most effective strategy to avoiding
+this churn is to group the execution of each tests that use each source of test
+data into sub-suites that are run serially, on e after the other.
+
+* If you're using Rails' defualt Minitest, we wrote a gem called
+  [minitest-suite](https://github.com/testdouble/minitest-suite) to accomplish
+  exactly this. Just declare something like `suite :test_data` or `suite
+  :fixtures` at the top of each test class
+* If you're using RSpec, the
+  [tag](https://relishapp.com/rspec/rspec-core/v/3-10/docs/command-line/tag-option)
+  feature can help you organize your tests by type, but you'll likely have to
+  run a separate CLI invocation for each to avoid the tests from being
+  interleaved
+
+Here's what the same example would do at run-time after adding
+[minitest-suite](https://github.com/testdouble/minitest-suite):
+
+```
+$ bin/rails test test/example_test.rb
+Run options: --seed 50105
+
+# Running:
+
+   test_data -- loading test_data SQL dump
+.  test_data -- rolling back to clean test_data
+.  test_data -- rolling back to clean test_data
+.  test_data -- rolling back to clean test_data
+.  test_data -- rolling back to clean test_data
+.  fixtures -- truncating tables, loading Rails fixtures
+.  fixtures -- rolling back to clean fixtures
+.  fixtures -- rolling back to clean fixtures
+.  fixtures -- rolling back to clean fixtures
+.  fixtures -- rolling back to clean fixtures
+.
+
+Finished in 2.377050s, 4.2069 runs/s, 4.2069 assertions/s.
+10 runs, 10 assertions, 0 failures, 0 errors, 0 skips
+```
+
+By grouping the execution in this way, the most expensive operations will
+usually only be run once: at the beginning of the first test in each suite.
+
+#### Expensive data manipulation
+
+If you're doing anything repeatedly that's data-intensive in your test setup
+after calling one of the `TestData.uses_*` methods, that operation is being
+repeated once per test, which could be very slow. Instead, you might consider
+moving that behavior into a [lifecycle hook](#lifecycle-hooks).
+
+Any code passed to a lifecycle hook will only be executed when data is
+_actually_ loaded or truncated and its effect will be included in the
+transaction savepoint that the `test_data` gem rolls back between tests.
+Seriously, appropriately moving data adjustments into these hooks can cut your
+test suite's runtime by an order of magnitude.
+
+#### Redundant test setup tasks
+
+One of the most likely sources of unnecessary slowness is redundant test
+cleanup. The speed gained from sandwiching every expensive operation between
 transaction savepoints can be profound… but can also easily be erased by a
 single before-each hook calling
 [database_cleaner](https://github.com/DatabaseCleaner/database_cleaner) to
@@ -1213,32 +1331,6 @@ time to take stock of everything that's called between tests during setup &
 teardown to ensure multiple tools aren't attempting to clean up the state of the
 database and potentially interfering with one another.
 
-A second opportunity for optimization is to group tests that use the same type
-of test data together, either into separate suites or by preventing them from
-being run in random order across said types. For example, suppose you have 10
-tests that call `TestData.uses_test_data` and 10 that call
-`TestData.uses_rails_fixtures`. If a test that calls `TestData.uses_test_data`
-is followed by another that calls `uses_test_data`, the only operation needed by
-the second call will be a rollback to the savepoint taken after the test data
-was loaded. If, however, a `uses_test_data` test is followed by a
-`uses_rails_fixtures` test, then the test data will be truncated and the
-fixtures loaded and new savepoints created (which would then be undone again if
-the _next_ test happened to call `uses_test_data`).
-
-As a result of the above, the marginal runtime cost for each `TestData.uses_*`
-method depends on which kinds of test precedes and follows it. That means your
-tests will run faster overall if the tests that call `TestData.uses_test_data`
-are run as a group separately from your tests that rely on
-`TestData.uses_clean_slate` or `TestData.uses_rails_fixtures`. Separating your
-tests into logical groups pretty trivial if you're using RSpec, as the
-[tag](https://relishapp.com/rspec/rspec-core/v/3-10/docs/command-line/tag-option)
-feature was built with this sort of need in mind. If you're using Minitest, you
-might consider organizing the tests in different directories and running
-multiple commands to execute them (e.g. `bin/rails test test/test_data_tests`
-and `bin/rails test/factory_tests`). Every CI configuration is different,
-however, and you may find yourself needing to get creative in configuring things
-to achieve the fastest build time.
-
 ## Code of Conduct
 
 This project follows Test Double's [code of

data/example/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: ..
   specs:
-    test_data (0.2.1)
+    test_data (0.2.2)
       railties (~> 6.0)
 
 GEM

data/example/test/integration/fixture_load_count_test.rb

@@ -0,0 +1,82 @@
+# Regression test to make sure we don't load fixtures too many times
+
+class HookCounter
+  def self.count
+    @call_count || 0
+  end
+
+  def self.count!
+    @call_count ||= 0
+    @call_count += 1
+  end
+end
+at_exit do
+  if TestData.statistics.load_rails_fixtures_count > 2 # could be 1 if :all runs first, 2 if :boops only does
+    raise "Rails fixture load was called #{TestData.statistics.load_rails_fixtures_count} times, shouldn't be more than 2!"
+  end
+  if HookCounter.count > 2
+    raise "Rails fixture load hook was called #{HookCounter.count} times, shouldn't be more than 2!"
+  end
+end
+
+require "test_helper"
+
+TestData.prevent_rails_fixtures_from_loading_automatically!
+
+TestData.config do |config|
+  config.after_rails_fixture_load {
+    HookCounter.count!
+  }
+end
+
+class PartialFixtureTest < ActiveSupport::TestCase
+  fixtures :boops
+
+  setup do
+    TestData.uses_rails_fixtures(self)
+  end
+
+  def test_has_only_boops
+    assert boops(:boop_1)
+    assert_raises(NameError) { method(:pants) }
+  end
+end
+
+class AllFixtureTest < ActiveSupport::TestCase
+  fixtures :all
+
+  setup do
+    TestData.uses_rails_fixtures(self)
+  end
+
+  def test_has_both
+    assert boops(:boop_1)
+    assert pants(:pant_1)
+  end
+end
+
+class AllFixtureTest2 < ActiveSupport::TestCase
+  fixtures :all
+
+  setup do
+    TestData.uses_rails_fixtures(self)
+  end
+
+  def test_has_both
+    assert boops(:boop_1)
+    assert pants(:pant_1)
+  end
+end
+
+class AllFixtureTest3 < ActiveSupport::TestCase
+  fixtures :all
+
+  setup do
+    TestData.uses_rails_fixtures(self)
+  end
+
+  def test_has_both
+    assert boops(:boop_1)
+    assert pants(:pant_1)
+  end
+end

data/lib/test_data/custom_loaders/rails_fixtures.rb

@@ -21,19 +21,22 @@ module TestData
 
       def load_requested(test_instance:)
         ActiveRecord::FixtureSet.reset_cache
-        test_instance.instance_variable_set(:@loaded_fixtures, @already_loaded_rails_fixtures[test_instance.class])
+        test_instance.instance_variable_set(:@loaded_fixtures,
+          @already_loaded_rails_fixtures.slice(*test_instance.class.fixture_table_names))
         test_instance.instance_variable_set(:@fixture_cache, {})
       end
 
       def loaded?(test_instance:)
-        @already_loaded_rails_fixtures[test_instance.class].present?
+        test_instance.class.fixture_table_names.all? { |table_name|
+          @already_loaded_rails_fixtures.key?(table_name)
+        }
       end
 
       def load(test_instance:)
         test_instance.pre_loaded_fixtures = false
         test_instance.use_transactional_tests = false
         test_instance.__test_data_gem_setup_fixtures
-        @already_loaded_rails_fixtures[test_instance.class] = test_instance.instance_variable_get(:@loaded_fixtures)
+        @already_loaded_rails_fixtures = test_instance.instance_variable_get(:@loaded_fixtures)
         @statistics.count_load_rails_fixtures!
         @config.after_rails_fixture_load_hook.call
       end

data/lib/test_data/version.rb

@@ -1,3 +1,3 @@
 module TestData
-  VERSION = "0.2.1"
+  VERSION = "0.2.2"
 end

data/script/test

@@ -52,6 +52,7 @@ bin/rails test test/integration/transaction_committing_boops_test.rb
 # Run a test that prevents Rails fixtures for preloading and then loads them in a transaction
 bin/rails test test/integration/rails_fixtures_override_test.rb
 bundle exec rspec spec/requests/rails_fixtures_override_spec.rb
+bin/rails test test/integration/fixture_load_count_test.rb
 
 # Run a test that forgets to prevent Rails fixtures but then tries to load them in a transaction
 bin/rails test test/integration/rails_fixtures_double_load_test.rb

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: test_data
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
 platform: ruby
 authors:
 - Justin Searls
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-07-30 00:00:00.000000000 Z
+date: 2021-08-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: railties
@@ -103,6 +103,7 @@ files:
 - example/test/integration/better_mode_switching_demo_test.rb
 - example/test/integration/boops_that_boop_boops_test.rb
 - example/test/integration/dont_dump_tables_test.rb
+- example/test/integration/fixture_load_count_test.rb
 - example/test/integration/load_rollback_truncate_test.rb
 - example/test/integration/migrated_boops_test.rb
 - example/test/integration/mode_switching_demo_test.rb