constable-rails 2.0.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4ee7e9d9303ba3b3e4977f529ab515059faa946e96209dae4e80dbef42d2930c
-  data.tar.gz: feadad150285bf9d68b11cc671b7c1bc13188c39a0b6c76317aef1a7a6ca9c70
+  metadata.gz: '008b9cb5b6a691c81f154c127f924118626c653ddfdf8c5d711503a0fe4531ef'
+  data.tar.gz: 6854d47a266792d2805ad06aae373f3dcfb83150f875d3822c28857ff32fc400
 SHA512:
-  metadata.gz: 71adc2fab90c1ce4b1239159f7000c2b2200dc7f6becc3d365f6c935dc1702a9c012557b4602c909de880f2728433cfdfc9b4bd1524b8ec25581459c4a4f415a
-  data.tar.gz: 70fd3761fed31d90e365e1b24940025ddd14dc6f80f0761d004cdfe6c69145b4f1806f4fe49753a61af4137677cc2313c9a3f7cd0ce2f284e121ada1e75d0fdf
+  metadata.gz: 55b9a3384a28d83fe163e60b853e7e3286d3c03ee6b63cb89b0fc4760c079294ff4d85ba7f6191da8170dea8f13fb4788f071ea6058cbcc64c37adc320dc1fca
+  data.tar.gz: 16e8d3a9e1da126f106cbaff3af7ea34e73025a687fa1c5786fbfc699c6c0b6f4eaa7da1642935beb4607d5b8ea10202216b58a3e135fb426ecaa5b40d89f9f3

data/CHANGELOG.md

@@ -5,6 +5,172 @@ All notable changes to this project are documented here. This project adheres to
 
 ## [Unreleased]
 
+## [2.1.1]
+
+Two bugs found by using 2.1.0's own new feature on a real suite. The first is the most
+serious defect this project has shipped.
+
+### A worker that died took its share of the suite with it, silently
+
+A 192-test suite reporting **"99 passed, 0 failed"**, exit code 0. Ninety-three tests never
+ran, and nothing said so.
+
+One worker died partway through its bucket. The parent collected the results the other
+worker sent, found them non-empty, and reported them as the whole run. `worker_errors` --
+which had the reason, in full -- was only ever read on the path where *nothing* came back,
+so a worker dying beside a healthy one was recorded and then never mentioned.
+
+This is the exact failure mode the runner already had three separate defences against, all
+of which assume total failure. Partial failure walked straight past them.
+
+Each worker now reports its position after every item, so the parent knows what it
+scheduled and how far each worker actually got. Whatever a dead worker abandoned is run in
+the parent -- serially, in the one process that cannot also vanish unnoticed -- and the
+reason comes with it:
+
+```
+| 7 tests did not come back from a parallel worker, so they were run here instead --
+  everything ran, nothing was skipped. A worker died partway through its share:
+  SystemExit: Migrations are pending.
+```
+
+Results for an item are only written once that item finishes, so the boundary is exact and
+re-running from it cannot duplicate a result.
+
+### `establish_connection` was not changing database, and said nothing
+
+`WorkerDatabases` renames a config in place -- `db_config._database = "app_test_3"` --
+which is how Rails' own `TestDatabases` does it. But when the process already holds a pool
+for that same config object, `establish_connection` hands back the pool it has instead of
+opening the database the object now names. The query then succeeds and answers about the
+*source* database.
+
+Verified on a real app: `populated?` returned **true** for a SQLite worker database whose
+file did not exist. That is `constable prepare` reporting "already prepared" for work it
+never did -- and it is why 2.1.0's staleness check could not see a worker that was a
+migration behind. Dropping the pool first is what makes a rename take effect.
+
+A forked worker never hit this, because `before_fork!` clears every connection before the
+fork. It only ever went wrong in the parent, which is why it survived this long.
+
+Looking at a database also no longer borrows `ActiveRecord::Base`'s connection at all. It
+gets a named subclass with a pool of its own, so `constable prepare` and the staleness
+check leave every connection the app holds exactly where they were. Anonymous would not do:
+a class with no name falls back to its superclass's connection specification name, which is
+Base again.
+
+### Databases that cannot be sharded are named out loud
+
+Oracle, and anything else with `database_tasks: false`, cannot be given to each worker.
+Constable already skipped them — but *skipped* and *safe* are different claims, and only
+the first was ever made. Every worker shares that database, and the failures that follow do
+not look like a parallelism problem:
+
+```
+| vacols (database_tasks: false) cannot be given to each worker, so all of them share it.
+  Tests that write to it will interfere with each other, and the failures will not look
+  like a parallelism problem -- they look like rows vanishing mid-test.
+```
+
+Measured on a real app: 163 failures across four workers, every one of them passing
+serially, 53 of them a bare `VacolsRecordNotFound` -- one shared Oracle database that each
+worker's `before(:suite)` deleted from while the others were mid-test. The README has the
+rest, including the harder limit that no warning can fix: an app holding OCI handles is not
+reliably forkable, and aborts a good half of its parallel runs from inside the Oracle
+client.
+
+
+## [2.1.0]
+
+Both changes here come from one afternoon on a real suite: nineteen files, eight forked
+workers, and a run that reported 83 passed and 1 failed on a unique index for a row no
+test had made. The failing test was not the broken one. Nothing was wrong with it.
+
+### A helper named in `.rspec` that will not load is now fatal
+
+It used to be a warning, and the cold cases ran anyway. The reasoning written in the
+source was that Constable's job is not to disappear. That was wrong, and it is worth being
+precise about why, because the failure it produced is the one this project exists to
+prevent.
+
+`rails_helper` is not a convenience. It is where a suite registers DatabaseCleaner's
+per-test transaction, FactoryBot, WebMock, and every `spec/support` hook. A worker that
+fails to finish loading it does not run a slightly degraded suite — it runs an
+**unisolated** one, against a real database, with nothing wrapping any test. The writes
+commit. Some tests pass anyway. Later files die on unique indexes naming rows nobody can
+account for, in a different file, with a stack trace pointing at innocent code.
+
+What produced it: eight workers loading a `spec/support` file that did
+
+```ruby
+Dir.mkdir(cache_directory) unless File.directory?(cache_directory)
+```
+
+Two workers check, both see nothing, both create, and the loser raises `Errno::EEXIST` —
+while `rails_helper` is loading. Six workers were isolated. Two were not, and left
+committed rows behind in their databases.
+
+A green run that was never isolated is worse than no run at all, so this now raises:
+
+```
+could not load what .rspec requires (Errno::EEXIST: File exists @ dir_s_mkdir - tmp/browser_cache_all).
+  Those files set up the suite -- database cleaning, factories, spec/support hooks.
+  Running cold cases without them would not isolate them, so this is fatal rather than a warning.
+```
+
+This is a behaviour change, not a new check: the condition was always detected. If a run
+starts failing here, it was already producing results that did not mean what they said.
+
+### Workers are told which worker they are
+
+`CONSTABLE_WORKER` (the index) and `CONSTABLE_WORKERS` (the count) are set in each forked
+worker, and unset in the parent.
+
+Anything a suite keeps on disk per process needs a name that differs per worker — a
+browser cache, a download directory, a screenshot path, a scratch file — and until now
+there was no way to ask, so every worker computed the same path and raced for it. That is
+the root cause of the bug above rather than an unrelated nicety.
+
+```ruby
+worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
+cache  = Rails.root.join("tmp/browser_cache#{worker}")
+```
+
+Deliberately not `TEST_ENV_NUMBER` or `TEST_SUBCATEGORY`: both are already interpolated
+into some apps' `database.yml`, and setting either here would rename databases behind
+`WorkerDatabases`' back.
+
+### A stale worker database drops the run to serial instead of running wrong
+
+`worker_databases: reuse` keeps the per-worker databases between runs, which is the point
+of it — and it means they do not follow migrations on their own. Run a migration, run the
+suite, and every worker is testing yesterday's schema.
+
+That does not fail cleanly. It fails as a missing column in whichever file happened to
+land on a stale worker, three files from anything you changed, on a different file each
+run because the scheduling moved. Constable now compares what each worker database has
+migrated against the real test database before it forks, and when they disagree:
+
+```
+⚠ worker database 3 has not run the migrations the test database has. `worker_databases:
+  reuse` keeps these between runs, which means they do not follow a migration on their
+  own. Running serially instead -- `constable prepare` rebuilds them.
+```
+
+Serial rather than a refusal, deliberately: the database a serial run uses is the real
+test database, and that one *is* current. The suite still runs, correctly, and says what
+to do to get its speed back. Two integers per database, in the parent, before anything
+forks; a question that cannot be answered (no `schema_migrations`, an adapter that will
+not connect) never blocks a run that would have worked.
+
+### Tests
+
+The fork path had no end-to-end coverage at all, which is where most of this project's
+serious bugs have lived. It has some now: worker identity arriving in the child and not
+leaking into the parent, and both halves of `.rspec` handling — a helper that raises stops
+the run, and a helper that loads is actually loaded before the file.
+
+
 ## [2.0.0]
 
 A tidy-up release. The API you write — `investigate`, `witness`, `briefing`, `docket`,

data/README.md

@@ -414,12 +414,55 @@ Each worker gets **its own database**, built from schema the way `rails test` do
 or kept between runs, with `worker_databases: reuse`, which is both faster and the only
 thing that works for an app whose schema cannot rebuild the database by itself (any app
 with Postgres custom types: `CREATE TYPE` has no `schema.rb` representation). Prepare
-those once with `constable prepare`.
+those once with `constable prepare`, and again after a migration — kept databases do not
+follow one on their own. Forgetting is caught rather than suffered: Constable compares
+what each worker database has migrated against the real test database before it forks, and
+runs serially (which uses the real one, so it is correct) rather than testing yesterday's
+schema.
 Sharing one would not be a speed/safety trade but a correctness bug: on SQLite the run
 dissolves into `database is locked`, and on a client/server database tests quietly see
 each other's rows. If your app has ActiveRecord but cannot shard, Constable runs serially
 and says why — slow is a trade-off, wrong is not.
 
+The database is not the only thing a worker needs to itself. Anything your suite keeps on
+disk per process — a browser cache, a download directory, a screenshot path — needs a name
+that differs per worker, or they race for it. Each worker is told which one it is:
+
+```ruby
+worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
+cache  = Rails.root.join("tmp/browser_cache#{worker}")
+```
+
+Some databases cannot be given to each worker at all — Oracle and anything else Rails does
+not manage (`database_tasks: false`). Constable does not try, and now says so at the start
+of a parallel run, because *skipped* and *safe* are different claims:
+
+```
+⚠ vacols (database_tasks: false) cannot be given to each worker, so all of them share it.
+  Tests that write to it will interfere with each other, and the failures will not look
+  like a parallelism problem -- they look like rows vanishing mid-test.
+```
+
+That warning is worth taking literally. On a real app with a legacy Oracle database, a
+four-worker run produced 163 failures that all passed serially; 53 of them were a bare
+`VacolsRecordNotFound`, because each worker's `before(:suite)` deleted from the one shared
+database while the others were midway through tests that had just written to it.
+
+**And a harder limit, if your app talks to one through a C driver: forking may not be
+possible at all.** The same app aborts roughly half its parallel runs with SIGABRT — no
+output on either stream, the crash report landing inside `libclntsh`, Oracle's client
+library catching a SIGSEGV in its own signal handler. It is not a Constable failure and
+there is nothing Constable can do about it: a process holding OCI handles is not reliably
+forkable. If you see bare exit code 134 and no output, check
+`~/Library/Logs/DiagnosticReports` (or your platform's equivalent) before assuming the test
+runner ate your suite, and run those specs with `worker_databases: off`.
+
+`CONSTABLE_WORKER` is the index and `CONSTABLE_WORKERS` the count; both are unset in the
+parent, so serial runs keep whatever name they had. Use `FileUtils.mkdir_p` rather than
+`Dir.mkdir ... unless File.directory?` while you are there — the second is a race, and if
+it runs inside `spec/support` it takes `rails_helper` down with it, which costs the loser
+its database cleaning rather than just its cache directory.
+
 ### Output
 
 stdout is reserved for results — not just Constable's own output, but the app's. Rails

data/lib/constable/cold_case/rspec.rb

@@ -324,10 +324,31 @@ module Constable
         rescue Constable::Error
           raise
         rescue StandardError => e
-          # A helper that will not load is the suite's problem to fix, and it will say so
-          # loudly on the first file. Constable's job here is not to disappear.
-          Constable.warn!("could not load what .rspec requires (#{e.class}: #{e.message}). " \
-                          "Cold cases will run without it.", kind: :cold_case)
+          # Fatal, deliberately.
+          #
+          # This used to be a warning that ran the cold cases anyway, on the reasoning that
+          # Constable's job is not to disappear. That reasoning was wrong, and the way it
+          # was wrong is the exact failure this gem exists to prevent.
+          #
+          # A `rails_helper` that does not finish loading is not a suite missing one
+          # convenience. It is a suite missing its isolation: DatabaseCleaner's per-test
+          # transaction, FactoryBot, WebMock, every `spec/support` hook. The tests then run
+          # against a real database with nothing wrapping them, so their writes commit.
+          # Some pass. Others fail later on a unique index, in a different file, naming a
+          # row that a test three files ago was supposed to have rolled back.
+          #
+          # Observed exactly that: eight forked workers loading a helper that did
+          # `Dir.mkdir` on a shared tmp directory it had just checked for. The workers that
+          # lost the race raised Errno::EEXIST, took this branch, ran their files with no
+          # DatabaseCleaner at all, and left committed rows behind in their databases.
+          #
+          # A green run that was never isolated is worse than no run. Refuse it.
+          raise Constable::Error,
+                "could not load what .rspec requires (#{e.class}: #{e.message}).\n  " \
+                "Those files set up the suite -- database cleaning, factories, " \
+                "spec/support hooks.\n  " \
+                "Running cold cases without them would not isolate them, so this is " \
+                "fatal rather than a warning."
         end
 
         # What `configuration.requires=` does before requiring anything: puts `lib` and the

data/lib/constable/runner.rb

@@ -289,7 +289,7 @@ module Constable
       return [] if items.empty?
 
       count = worker_count(items)
-      if count > 1 && forkable? && parallel_safe?
+      if count > 1 && forkable? && parallel_safe?(count)
         run_parallel(items, count)
       else
         run_serial(items)
@@ -302,17 +302,67 @@ module Constable
     # which is worse. An app with no ActiveRecord has nothing to shard and is always safe.
     #
     # When we cannot shard, we run serially and say why. Slow is a trade-off; wrong is not.
-    def parallel_safe?
+    def parallel_safe?(count)
       # An explicit opt-out. No attempt, and no warning about one -- the user has already
       # told us they know.
       return false if @config.worker_databases == :off
       return true unless WorkerDatabases.active_record?
-      return true if WorkerDatabases.shardable?
+
+      unless WorkerDatabases.shardable?
+        Constable.warn!(
+          "parallel workers need one database per worker, and this app's ActiveRecord " \
+          "cannot provide them (active_record/test_databases did not load). Running " \
+          "serially instead -- pass --workers N once that is available.",
+          kind: :parallel
+        )
+        return false
+      end
+
+      warn_about_shared_databases!
+      worker_databases_current?(count)
+    end
+
+    # A database Constable could not shard is shared by every worker, and that is worth
+    # saying out loud before the run rather than leaving it to be deduced from the wreckage.
+    #
+    # The failures it causes do not look like a parallelism problem. They look like records
+    # disappearing mid-test: worker 2's `before(:suite)` cleans the shared legacy database
+    # while worker 1 is halfway through a test that just created rows in it. Measured on a
+    # real suite -- 53 `VacolsRecordNotFound` failures across four workers, every one of
+    # them passing serially.
+    def warn_about_shared_databases!
+      shared = WorkerDatabases.unshardable_databases
+      return if shared.empty?
+
+      Constable.warn!(
+        "#{shared.join(", ")} cannot be given to each worker, so all of them share it. " \
+        "Tests that write to it will interfere with each other, and the failures will not " \
+        "look like a parallelism problem -- they look like rows vanishing mid-test. Run " \
+        "specs that touch it serially (`--workers 1`), or `worker_databases: off`.",
+        kind: :parallel
+      )
+    end
+
+    # `:reuse` keeps the per-worker databases between runs, so they do not follow
+    # migrations by themselves. A run against stale copies does not fail cleanly -- it
+    # fails as a missing column in whichever file happened to touch it, on a different
+    # worker each run.
+    #
+    # Serial is the right fallback rather than a refusal, because the database a serial
+    # run uses is the real test database, and that one *is* current. So the suite still
+    # runs, correctly, and says exactly what to do to get its speed back.
+    def worker_databases_current?(count)
+      return true unless @config.worker_databases.to_s == "reuse"
+
+      stale = WorkerDatabases.stale_workers(count)
+      return true if stale.empty?
 
       Constable.warn!(
-        "parallel workers need one database per worker, and this app's ActiveRecord " \
-        "cannot provide them (active_record/test_databases did not load). Running " \
-        "serially instead -- pass --workers N once that is available.",
+        "worker database#{"s" if stale.length > 1} #{stale.join(", ")} " \
+        "#{stale.length > 1 ? "have" : "has"} not run the migrations the test database " \
+        "has. `worker_databases: reuse` keeps these between runs, which means they do " \
+        "not follow a migration on their own. Running serially instead -- " \
+        "`constable prepare` rebuilds them.",
         kind: :parallel
       )
       false
@@ -359,6 +409,22 @@ module Constable
         pid = fork do
           reader.close
 
+          # Tell the app which worker it is.
+          #
+          # Anything a suite keeps on disk per process needs this: a browser cache, a
+          # download directory, a screenshot path, a scratch file. Without it every worker
+          # computes the same path and they race -- and the way that surfaces is not a
+          # tidy error. Observed on a real suite: eight workers running
+          # `Dir.mkdir(dir) unless File.directory?(dir)` in a spec/support file, the losers
+          # raising Errno::EEXIST *while loading rails_helper*, so those workers ran their
+          # files with no database cleaning at all.
+          #
+          # Deliberately its own name rather than TEST_ENV_NUMBER or parallel_tests'
+          # TEST_SUBCATEGORY: those are already wired into some apps' database.yml, and
+          # setting one here would rename databases behind WorkerDatabases' back.
+          ENV["CONSTABLE_WORKER"]  = worker_index.to_s
+          ENV["CONSTABLE_WORKERS"] = buckets.length.to_s
+
           # Before a single test runs: build this worker's own database and point the
           # process at it. Never falls back to the shared one -- that is the bug this
           # exists to prevent -- but the failure is reported home rather than raised.
@@ -377,8 +443,14 @@ module Constable
           # reason, and a run that scheduled nineteen files reports zero tests and exits
           # 0. A worker that dies has to say so.
           begin
-            bucket.each do |item|
+            # The position report after each item is what lets the parent finish the work
+            # if this worker dies partway. Results for an item are all written once the
+            # item is done, so "position n reported" means items 0...n are home and
+            # nothing from item n was ever sent -- the boundary is exact, and re-running
+            # from it cannot duplicate a result.
+            bucket.each_with_index do |item, position|
               run_item(item).each { |result| write_message(writer, :result, result.to_h) }
+              write_message(writer, :progress, position + 1)
             end
           rescue Exception => e # rubocop:disable Lint/RescueException
             write_message(writer, :worker_error, "#{e.class}: #{e.message}")
@@ -427,9 +499,45 @@ module Constable
       # results carry them home. Nothing that bends the rules is allowed to go missing
       # just because it happened in a subprocess.
       collected.each { |result| Constable.warnings.concat(Array(result.warnings)) }
-      collected
+      collected + finish_abandoned_work(buckets)
     end
 
+    # One worker dying used to cost its whole remaining bucket, silently.
+    #
+    # `worker_errors` was only ever read on the path where *nothing* came back, so a
+    # worker that died beside living ones was collected and never mentioned. The parent
+    # reported the results it happened to receive, called them the whole suite, and
+    # exited 0. Observed: a 192-test suite reporting "99 passed, 0 failed" -- green, with
+    # 93 tests that never ran.
+    #
+    # Now the parent knows what it scheduled and how far each worker actually got, so it
+    # can just run the rest itself. Serial, in this process, which is the one place that
+    # cannot also die without anyone noticing.
+    def finish_abandoned_work(buckets)
+      abandoned = buckets.each_with_index.flat_map do |bucket, index|
+        done = worker_progress[index]
+        done < bucket.size ? bucket[done..] : []
+      end
+      return [] if abandoned.empty?
+
+      Constable.warn!(
+        "#{abandoned.size} test#{"s" unless abandoned.size == 1} did not come back from " \
+        "a parallel worker, so #{abandoned.size == 1 ? "it was" : "they were"} run here " \
+        "instead -- everything ran, nothing was skipped. A worker died partway through " \
+        "its share#{worker_death_reason}.",
+        kind: :parallel
+      )
+      run_serial(abandoned)
+    end
+
+    def worker_death_reason
+      return "" if worker_errors.empty?
+
+      ": #{worker_errors.first}"
+    end
+
+    def worker_progress = (@worker_progress ||= Hash.new(0))
+
     def worker_errors = (@worker_errors ||= [])
 
     # A worker can die below Ruby: a segfault, an OOM kill, a signal. No `rescue` reaches
@@ -530,6 +638,9 @@ module Constable
               @reporter.record(result)
             when :coverage
               @worker_coverage = Constable::Coverage.merge_raw(@worker_coverage, body)
+            when :progress
+              index = readers.index(reader)
+              worker_progress[index] = body.to_i if index
             when :worker_error
               # A worker that could not start. Collected rather than raised, so the parent
               # decides what to do once it knows whether any worker got going at all.

data/lib/constable/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Constable
-  VERSION = "2.0.0"
+  VERSION = "2.1.1"
 end

data/lib/constable/worker_databases.rb

@@ -37,6 +37,13 @@ module Constable
 
     # Parent side, before the fork. A child inheriting a live connection is a corruption
     # risk in exactly the way an inherited SQLite handle is.
+    #
+    # `clear_all_connections!` and not `disconnect!`. Closing the pools outright was tried
+    # here, on the theory that returning a connection to the pool leaves the socket and the
+    # driver's C-side state for `fork` to copy into every child. It changed nothing
+    # measurable, and the crash it was meant to prevent turned out to predate it -- see
+    # the note on native drivers in the README. Rails does the same thing before its own
+    # fork, and matching it is the conservative choice.
     def before_fork!
       return false unless active_record?
 
@@ -93,6 +100,124 @@ module Constable
       raise Constable::Error, "could not prepare worker #{index}: #{e.class}: #{e.message}"
     end
 
+    # The one mistake `:reuse` invites, caught before the fork rather than after.
+    #
+    # `:reuse` keeps the per-worker databases between runs, which is the whole point --
+    # and it means they do not follow migrations by themselves. Run a migration, run the
+    # suite, and every worker is now testing yesterday's schema. That does not fail
+    # cleanly: it fails as a missing column in whichever file happened to touch it, three
+    # files away from anything you changed, differently on each run because the file went
+    # to a different worker. Exactly the shape of bug that costs an afternoon.
+    #
+    # So compare what each worker database has migrated against what the real test
+    # database has, and refuse rather than guess. Two integers per database, in the parent,
+    # before anything forks.
+    #
+    # Returns the worker indexes that are out of date, empty when they are all current or
+    # when the question cannot be answered (no schema_migrations table, an adapter that
+    # will not connect) -- an unanswerable check must not block a run that would have
+    # worked.
+    def stale_workers(count)
+      return [] unless shardable?
+
+      expected = {}
+      original = database_names
+      begin
+        each_source_config do |db_config|
+          fingerprint = schema_fingerprint(db_config)
+          expected[db_config.database.to_s] = fingerprint if fingerprint
+        end
+        return [] if expected.empty?
+
+        stale = []
+        (0...count).each do |index|
+          names = database_names
+          begin
+            each_worker_config(index) do |db_config|
+              source = db_config.database.to_s.sub(/_#{index}\z/, "")
+              next unless expected.key?(source)
+
+              actual = schema_fingerprint(db_config)
+              stale << index if actual && actual != expected[source]
+            end
+          ensure
+            restore_database_names(names)
+          end
+        end
+        stale.uniq
+      ensure
+        # Names only. Nothing here ever repointed ActiveRecord::Base, so there is no
+        # connection to put back -- which is the point.
+        restore_database_names(original)
+      end
+    rescue StandardError
+      []
+    end
+
+    # What a database has migrated: how many migrations it has run and the latest one.
+    # Cheaper than diffing every version, and a worker that missed a migration differs in
+    # both. nil when the question does not apply.
+    def schema_fingerprint(db_config)
+      with_probe_connection(db_config) do |connection|
+        next nil unless connection.table_exists?("schema_migrations")
+
+        connection.select_rows("SELECT COUNT(*), MAX(version) FROM schema_migrations").first
+      end
+    rescue StandardError
+      nil
+    end
+
+    # Databases this environment declares that cannot be given to each worker, so every
+    # worker shares the one copy.
+    #
+    # Constable skips them on purpose -- appending `_3` to an Oracle TNS service name names
+    # nothing -- but "skipped" and "safe" are different claims, and only the first was ever
+    # made. A legacy database that tests write to is shared mutable state across processes,
+    # and the failures it produces do not look like a parallelism problem: they look like
+    # records vanishing mid-test, in whichever spec happened to be running when another
+    # worker's suite hook cleaned the database they were both using.
+    #
+    # Measured on a real suite: 53 `VacolsRecordNotFound` failures across a four-worker
+    # run, every one of them passing serially, all from one shared Oracle database that
+    # each worker deleted from at startup.
+    def unshardable_databases
+      return [] unless active_record?
+
+      ::ActiveRecord::Base.configurations
+                          .configs_for(env_name: env_name, include_hidden: true)
+                          .filter_map { |db_config| share_reason(db_config) }
+    rescue StandardError
+      []
+    end
+
+    # Why a database stays shared, in the words of the setting that caused it. Both reasons
+    # matter and only one of them is about the adapter: `database_tasks: false` is how an
+    # app says "Rails does not manage this one", which is the usual way a legacy database is
+    # declared -- and it is exactly the database most likely to be shared, written to by
+    # tests, and cleaned by a suite hook in every worker at once.
+    #
+    # The config's *name* rather than its database, because a TNS descriptor is four lines
+    # of connection string and "vacols" is what anyone reading the warning calls it.
+    def share_reason(db_config)
+      name = db_config.respond_to?(:name) ? db_config.name : db_config.database
+      return "#{name} (database_tasks: false)" unless db_config.database_tasks?
+      return nil if shardable_adapter?(db_config)
+
+      "#{name} (#{db_config.adapter})"
+    end
+
+    # The same configs `each_worker_config` renames, left under their real names.
+    def each_source_config
+      ::ActiveRecord::Base.configurations
+                          .configs_for(env_name: env_name, include_hidden: true)
+                          .each do |db_config|
+        next unless db_config.database_tasks?
+        next unless shardable_adapter?(db_config)
+
+        yield db_config
+      end
+    end
+
     def database_names
       ::ActiveRecord::Base.configurations
                           .configs_for(env_name: env_name, include_hidden: true)
@@ -269,12 +394,52 @@ module Constable
     # Present and holding tables. A database that exists but is empty is not prepared, and
     # silently running a suite against no tables is the worst of the available outcomes.
     def populated?(db_config)
-      ::ActiveRecord::Base.establish_connection(db_config)
-      ::ActiveRecord::Base.connection.tables.any?
+      with_probe_connection(db_config) { |connection| connection.tables.any? }
     rescue StandardError
       false
     end
 
+    # A connection class of its own, so looking at a database never disturbs the app's.
+    #
+    # Asking "is this worker database prepared, and has it run our migrations?" needs a
+    # connection, and the obvious way to get one is to point ActiveRecord::Base at it and
+    # then point it back. That works, right up until the app has a native driver attached.
+    #
+    # Measured on a real app with a legacy Oracle database: repointing Base in the parent
+    # before forking killed the whole run with SIGABRT, no output on either stream, the
+    # crash report landing inside libclntsh -- Oracle's client catching a SIGSEGV in its
+    # own handler and calling abort. Nothing about it says "your test runner opened a
+    # connection it did not need".
+    #
+    # A named subclass gets its own `connection_specification_name`, so its pool is its
+    # own: establishing and removing it leaves ActiveRecord::Base, and every other class
+    # with a connection, untouched. Anonymous would not do -- a class with no name falls
+    # back to its superclass's specification name, which is Base again.
+    def probe_class
+      base = ::ActiveRecord::Base
+      return @probe_class if defined?(@probe_class) && @probe_base.equal?(base)
+
+      klass = Class.new(base)
+      klass.abstract_class = true if klass.respond_to?(:abstract_class=)
+      # Naming it is not decoration: an unnamed class falls back to its superclass's
+      # connection specification name, which would put us right back on Base's pool.
+      remove_const(:ProbeConnection) if const_defined?(:ProbeConnection, false)
+      const_set(:ProbeConnection, klass)
+      @probe_base = base
+      @probe_class = klass
+    end
+
+    def with_probe_connection(db_config)
+      probe_class.establish_connection(db_config)
+      yield probe_class.connection
+    ensure
+      begin
+        probe_class.remove_connection
+      rescue StandardError
+        nil
+      end
+    end
+
     def env_name
       if defined?(::ActiveRecord::ConnectionHandling::DEFAULT_ENV)
         ::ActiveRecord::ConnectionHandling::DEFAULT_ENV.call

data/lib/generators/constable/templates/case_helper.rb.tt

@@ -34,8 +34,8 @@ abort("The Rails environment is running in production mode!") if Rails.env.produ
 if defined?(ActiveRecord::Migration)
   begin
     ActiveRecord::Migration.maintain_test_schema!
-  rescue ActiveRecord::PendingMigrationError => e
-    abort("#{e.message}\n\nRun: bin/rails db:migrate RAILS_ENV=test")
+  rescue ActiveRecord::PendingMigrationError => error
+    abort("#{error.message}\n\nRun: bin/rails db:migrate RAILS_ENV=test")
   end
 end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: constable-rails
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.1.1
 platform: ruby
 authors:
 - Ray Hughes