constable-rails 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4ee7e9d9303ba3b3e4977f529ab515059faa946e96209dae4e80dbef42d2930c
-  data.tar.gz: feadad150285bf9d68b11cc671b7c1bc13188c39a0b6c76317aef1a7a6ca9c70
+  metadata.gz: 64e582fc3aab8b2c70309a18f5807cc1f00468ba33afa6dd08f4c78f6abde71d
+  data.tar.gz: b34caa421b4c301f5027e9d167c3e6a3ace738ef6668d5a5f81b2cf6711795b2
 SHA512:
-  metadata.gz: 71adc2fab90c1ce4b1239159f7000c2b2200dc7f6becc3d365f6c935dc1702a9c012557b4602c909de880f2728433cfdfc9b4bd1524b8ec25581459c4a4f415a
-  data.tar.gz: 70fd3761fed31d90e365e1b24940025ddd14dc6f80f0761d004cdfe6c69145b4f1806f4fe49753a61af4137677cc2313c9a3f7cd0ce2f284e121ada1e75d0fdf
+  metadata.gz: efd72821dbfff16019e0eeda25eae916753ea5e708d9addec79534002b84693f8d8e0498602b5c20f6793fbd92c5ed57e68dcef9a368026b8e54c5fd567d5af4
+  data.tar.gz: bcae479351ac5a001dec12c0a5e81d337b373998bbd82d57c8588e91db47e94fda99c88abc8c581b860625cc91c44e1e69e38693fca6eb7d1671ee84bc984e1e

data/CHANGELOG.md

@@ -5,6 +5,97 @@ All notable changes to this project are documented here. This project adheres to
 
 ## [Unreleased]
 
+## [2.1.0]
+
+Both changes here come from one afternoon on a real suite: nineteen files, eight forked
+workers, and a run that reported 83 passed and 1 failed on a unique index for a row no
+test had made. The failing test was not the broken one. Nothing was wrong with it.
+
+### A helper named in `.rspec` that will not load is now fatal
+
+It used to be a warning, and the cold cases ran anyway. The reasoning written in the
+source was that Constable's job is not to disappear. That was wrong, and it is worth being
+precise about why, because the failure it produced is the one this project exists to
+prevent.
+
+`rails_helper` is not a convenience. It is where a suite registers DatabaseCleaner's
+per-test transaction, FactoryBot, WebMock, and every `spec/support` hook. A worker that
+fails to finish loading it does not run a slightly degraded suite — it runs an
+**unisolated** one, against a real database, with nothing wrapping any test. The writes
+commit. Some tests pass anyway. Later files die on unique indexes naming rows nobody can
+account for, in a different file, with a stack trace pointing at innocent code.
+
+What produced it: eight workers loading a `spec/support` file that did
+
+```ruby
+Dir.mkdir(cache_directory) unless File.directory?(cache_directory)
+```
+
+Two workers check, both see nothing, both create, and the loser raises `Errno::EEXIST` —
+while `rails_helper` is loading. Six workers were isolated. Two were not, and left
+committed rows behind in their databases.
+
+A green run that was never isolated is worse than no run at all, so this now raises:
+
+```
+could not load what .rspec requires (Errno::EEXIST: File exists @ dir_s_mkdir - tmp/browser_cache_all).
+  Those files set up the suite -- database cleaning, factories, spec/support hooks.
+  Running cold cases without them would not isolate them, so this is fatal rather than a warning.
+```
+
+This is a behaviour change, not a new check: the condition was always detected. If a run
+starts failing here, it was already producing results that did not mean what they said.
+
+### Workers are told which worker they are
+
+`CONSTABLE_WORKER` (the index) and `CONSTABLE_WORKERS` (the count) are set in each forked
+worker, and unset in the parent.
+
+Anything a suite keeps on disk per process needs a name that differs per worker — a
+browser cache, a download directory, a screenshot path, a scratch file — and until now
+there was no way to ask, so every worker computed the same path and raced for it. That is
+the root cause of the bug above rather than an unrelated nicety.
+
+```ruby
+worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
+cache  = Rails.root.join("tmp/browser_cache#{worker}")
+```
+
+Deliberately not `TEST_ENV_NUMBER` or `TEST_SUBCATEGORY`: both are already interpolated
+into some apps' `database.yml`, and setting either here would rename databases behind
+`WorkerDatabases`' back.
+
+### A stale worker database drops the run to serial instead of running wrong
+
+`worker_databases: reuse` keeps the per-worker databases between runs, which is the point
+of it — and it means they do not follow migrations on their own. Run a migration, run the
+suite, and every worker is testing yesterday's schema.
+
+That does not fail cleanly. It fails as a missing column in whichever file happened to
+land on a stale worker, three files from anything you changed, on a different file each
+run because the scheduling moved. Constable now compares what each worker database has
+migrated against the real test database before it forks, and when they disagree:
+
+```
+⚠ worker database 3 has not run the migrations the test database has. `worker_databases:
+  reuse` keeps these between runs, which means they do not follow a migration on their
+  own. Running serially instead -- `constable prepare` rebuilds them.
+```
+
+Serial rather than a refusal, deliberately: the database a serial run uses is the real
+test database, and that one *is* current. The suite still runs, correctly, and says what
+to do to get its speed back. Two integers per database, in the parent, before anything
+forks; a question that cannot be answered (no `schema_migrations`, an adapter that will
+not connect) never blocks a run that would have worked.
+
+### Tests
+
+The fork path had no end-to-end coverage at all, which is where most of this project's
+serious bugs have lived. It has some now: worker identity arriving in the child and not
+leaking into the parent, and both halves of `.rspec` handling — a helper that raises stops
+the run, and a helper that loads is actually loaded before the file.
+
+
 ## [2.0.0]
 
 A tidy-up release. The API you write — `investigate`, `witness`, `briefing`, `docket`,

data/README.md

@@ -414,12 +414,31 @@ Each worker gets **its own database**, built from schema the way `rails test` do
 or kept between runs, with `worker_databases: reuse`, which is both faster and the only
 thing that works for an app whose schema cannot rebuild the database by itself (any app
 with Postgres custom types: `CREATE TYPE` has no `schema.rb` representation). Prepare
-those once with `constable prepare`.
+those once with `constable prepare`, and again after a migration — kept databases do not
+follow one on their own. Forgetting is caught rather than suffered: Constable compares
+what each worker database has migrated against the real test database before it forks, and
+runs serially (which uses the real one, so it is correct) rather than testing yesterday's
+schema.
 Sharing one would not be a speed/safety trade but a correctness bug: on SQLite the run
 dissolves into `database is locked`, and on a client/server database tests quietly see
 each other's rows. If your app has ActiveRecord but cannot shard, Constable runs serially
 and says why — slow is a trade-off, wrong is not.
 
+The database is not the only thing a worker needs to itself. Anything your suite keeps on
+disk per process — a browser cache, a download directory, a screenshot path — needs a name
+that differs per worker, or they race for it. Each worker is told which one it is:
+
+```ruby
+worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
+cache  = Rails.root.join("tmp/browser_cache#{worker}")
+```
+
+`CONSTABLE_WORKER` is the index and `CONSTABLE_WORKERS` the count; both are unset in the
+parent, so serial runs keep whatever name they had. Use `FileUtils.mkdir_p` rather than
+`Dir.mkdir ... unless File.directory?` while you are there — the second is a race, and if
+it runs inside `spec/support` it takes `rails_helper` down with it, which costs the loser
+its database cleaning rather than just its cache directory.
+
 ### Output
 
 stdout is reserved for results — not just Constable's own output, but the app's. Rails

data/lib/constable/cold_case/rspec.rb

@@ -324,10 +324,31 @@ module Constable
         rescue Constable::Error
           raise
         rescue StandardError => e
-          # A helper that will not load is the suite's problem to fix, and it will say so
-          # loudly on the first file. Constable's job here is not to disappear.
-          Constable.warn!("could not load what .rspec requires (#{e.class}: #{e.message}). " \
-                          "Cold cases will run without it.", kind: :cold_case)
+          # Fatal, deliberately.
+          #
+          # This used to be a warning that ran the cold cases anyway, on the reasoning that
+          # Constable's job is not to disappear. That reasoning was wrong, and the way it
+          # was wrong is the exact failure this gem exists to prevent.
+          #
+          # A `rails_helper` that does not finish loading is not a suite missing one
+          # convenience. It is a suite missing its isolation: DatabaseCleaner's per-test
+          # transaction, FactoryBot, WebMock, every `spec/support` hook. The tests then run
+          # against a real database with nothing wrapping them, so their writes commit.
+          # Some pass. Others fail later on a unique index, in a different file, naming a
+          # row that a test three files ago was supposed to have rolled back.
+          #
+          # Observed exactly that: eight forked workers loading a helper that did
+          # `Dir.mkdir` on a shared tmp directory it had just checked for. The workers that
+          # lost the race raised Errno::EEXIST, took this branch, ran their files with no
+          # DatabaseCleaner at all, and left committed rows behind in their databases.
+          #
+          # A green run that was never isolated is worse than no run. Refuse it.
+          raise Constable::Error,
+                "could not load what .rspec requires (#{e.class}: #{e.message}).\n  " \
+                "Those files set up the suite -- database cleaning, factories, " \
+                "spec/support hooks.\n  " \
+                "Running cold cases without them would not isolate them, so this is " \
+                "fatal rather than a warning."
         end
 
         # What `configuration.requires=` does before requiring anything: puts `lib` and the

data/lib/constable/runner.rb

@@ -289,7 +289,7 @@ module Constable
       return [] if items.empty?
 
       count = worker_count(items)
-      if count > 1 && forkable? && parallel_safe?
+      if count > 1 && forkable? && parallel_safe?(count)
         run_parallel(items, count)
       else
         run_serial(items)
@@ -302,17 +302,45 @@ module Constable
     # which is worse. An app with no ActiveRecord has nothing to shard and is always safe.
     #
     # When we cannot shard, we run serially and say why. Slow is a trade-off; wrong is not.
-    def parallel_safe?
+    def parallel_safe?(count)
       # An explicit opt-out. No attempt, and no warning about one -- the user has already
       # told us they know.
       return false if @config.worker_databases == :off
       return true unless WorkerDatabases.active_record?
-      return true if WorkerDatabases.shardable?
+
+      unless WorkerDatabases.shardable?
+        Constable.warn!(
+          "parallel workers need one database per worker, and this app's ActiveRecord " \
+          "cannot provide them (active_record/test_databases did not load). Running " \
+          "serially instead -- pass --workers N once that is available.",
+          kind: :parallel
+        )
+        return false
+      end
+
+      worker_databases_current?(count)
+    end
+
+    # `:reuse` keeps the per-worker databases between runs, so they do not follow
+    # migrations by themselves. A run against stale copies does not fail cleanly -- it
+    # fails as a missing column in whichever file happened to touch it, on a different
+    # worker each run.
+    #
+    # Serial is the right fallback rather than a refusal, because the database a serial
+    # run uses is the real test database, and that one *is* current. So the suite still
+    # runs, correctly, and says exactly what to do to get its speed back.
+    def worker_databases_current?(count)
+      return true unless @config.worker_databases.to_s == "reuse"
+
+      stale = WorkerDatabases.stale_workers(count)
+      return true if stale.empty?
 
       Constable.warn!(
-        "parallel workers need one database per worker, and this app's ActiveRecord " \
-        "cannot provide them (active_record/test_databases did not load). Running " \
-        "serially instead -- pass --workers N once that is available.",
+        "worker database#{"s" if stale.length > 1} #{stale.join(", ")} " \
+        "#{stale.length > 1 ? "have" : "has"} not run the migrations the test database " \
+        "has. `worker_databases: reuse` keeps these between runs, which means they do " \
+        "not follow a migration on their own. Running serially instead -- " \
+        "`constable prepare` rebuilds them.",
         kind: :parallel
       )
       false
@@ -359,6 +387,22 @@ module Constable
         pid = fork do
           reader.close
 
+          # Tell the app which worker it is.
+          #
+          # Anything a suite keeps on disk per process needs this: a browser cache, a
+          # download directory, a screenshot path, a scratch file. Without it every worker
+          # computes the same path and they race -- and the way that surfaces is not a
+          # tidy error. Observed on a real suite: eight workers running
+          # `Dir.mkdir(dir) unless File.directory?(dir)` in a spec/support file, the losers
+          # raising Errno::EEXIST *while loading rails_helper*, so those workers ran their
+          # files with no database cleaning at all.
+          #
+          # Deliberately its own name rather than TEST_ENV_NUMBER or parallel_tests'
+          # TEST_SUBCATEGORY: those are already wired into some apps' database.yml, and
+          # setting one here would rename databases behind WorkerDatabases' back.
+          ENV["CONSTABLE_WORKER"]  = worker_index.to_s
+          ENV["CONSTABLE_WORKERS"] = buckets.length.to_s
+
           # Before a single test runs: build this worker's own database and point the
           # process at it. Never falls back to the shared one -- that is the bug this
           # exists to prevent -- but the failure is reported home rather than raised.

data/lib/constable/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Constable
-  VERSION = "2.0.0"
+  VERSION = "2.1.0"
 end

data/lib/constable/worker_databases.rb

@@ -93,6 +93,84 @@ module Constable
       raise Constable::Error, "could not prepare worker #{index}: #{e.class}: #{e.message}"
     end
 
+    # The one mistake `:reuse` invites, caught before the fork rather than after.
+    #
+    # `:reuse` keeps the per-worker databases between runs, which is the whole point --
+    # and it means they do not follow migrations by themselves. Run a migration, run the
+    # suite, and every worker is now testing yesterday's schema. That does not fail
+    # cleanly: it fails as a missing column in whichever file happened to touch it, three
+    # files away from anything you changed, differently on each run because the file went
+    # to a different worker. Exactly the shape of bug that costs an afternoon.
+    #
+    # So compare what each worker database has migrated against what the real test
+    # database has, and refuse rather than guess. Two integers per database, in the parent,
+    # before anything forks.
+    #
+    # Returns the worker indexes that are out of date, empty when they are all current or
+    # when the question cannot be answered (no schema_migrations table, an adapter that
+    # will not connect) -- an unanswerable check must not block a run that would have
+    # worked.
+    def stale_workers(count)
+      return [] unless shardable?
+
+      expected = {}
+      original = database_names
+      begin
+        each_source_config do |db_config|
+          fingerprint = schema_fingerprint(db_config)
+          expected[db_config.database.to_s] = fingerprint if fingerprint
+        end
+        return [] if expected.empty?
+
+        stale = []
+        (0...count).each do |index|
+          names = database_names
+          begin
+            each_worker_config(index) do |db_config|
+              source = db_config.database.to_s.sub(/_#{index}\z/, "")
+              next unless expected.key?(source)
+
+              actual = schema_fingerprint(db_config)
+              stale << index if actual && actual != expected[source]
+            end
+          ensure
+            restore_database_names(names)
+          end
+        end
+        stale.uniq
+      ensure
+        restore_database_names(original)
+        ::ActiveRecord::Base.establish_connection
+      end
+    rescue StandardError
+      []
+    end
+
+    # What a database has migrated: how many migrations it has run and the latest one.
+    # Cheaper than diffing every version, and a worker that missed a migration differs in
+    # both. nil when the question does not apply.
+    def schema_fingerprint(db_config)
+      ::ActiveRecord::Base.establish_connection(db_config)
+      connection = ::ActiveRecord::Base.connection
+      return nil unless connection.table_exists?("schema_migrations")
+
+      connection.select_rows("SELECT COUNT(*), MAX(version) FROM schema_migrations").first
+    rescue StandardError
+      nil
+    end
+
+    # The same configs `each_worker_config` renames, left under their real names.
+    def each_source_config
+      ::ActiveRecord::Base.configurations
+                          .configs_for(env_name: env_name, include_hidden: true)
+                          .each do |db_config|
+        next unless db_config.database_tasks?
+        next unless shardable_adapter?(db_config)
+
+        yield db_config
+      end
+    end
+
     def database_names
       ::ActiveRecord::Base.configurations
                           .configs_for(env_name: env_name, include_hidden: true)

data/lib/generators/constable/templates/case_helper.rb.tt

@@ -34,8 +34,8 @@ abort("The Rails environment is running in production mode!") if Rails.env.produ
 if defined?(ActiveRecord::Migration)
   begin
     ActiveRecord::Migration.maintain_test_schema!
-  rescue ActiveRecord::PendingMigrationError => e
-    abort("#{e.message}\n\nRun: bin/rails db:migrate RAILS_ENV=test")
+  rescue ActiveRecord::PendingMigrationError => error
+    abort("#{error.message}\n\nRun: bin/rails db:migrate RAILS_ENV=test")
   end
 end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: constable-rails
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.0
 platform: ruby
 authors:
 - Ray Hughes