constable-rails 2.1.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 64e582fc3aab8b2c70309a18f5807cc1f00468ba33afa6dd08f4c78f6abde71d
-  data.tar.gz: b34caa421b4c301f5027e9d167c3e6a3ace738ef6668d5a5f81b2cf6711795b2
+  metadata.gz: '008b9cb5b6a691c81f154c127f924118626c653ddfdf8c5d711503a0fe4531ef'
+  data.tar.gz: 6854d47a266792d2805ad06aae373f3dcfb83150f875d3822c28857ff32fc400
 SHA512:
-  metadata.gz: efd72821dbfff16019e0eeda25eae916753ea5e708d9addec79534002b84693f8d8e0498602b5c20f6793fbd92c5ed57e68dcef9a368026b8e54c5fd567d5af4
-  data.tar.gz: bcae479351ac5a001dec12c0a5e81d337b373998bbd82d57c8588e91db47e94fda99c88abc8c581b860625cc91c44e1e69e38693fca6eb7d1671ee84bc984e1e
+  metadata.gz: 55b9a3384a28d83fe163e60b853e7e3286d3c03ee6b63cb89b0fc4760c079294ff4d85ba7f6191da8170dea8f13fb4788f071ea6058cbcc64c37adc320dc1fca
+  data.tar.gz: 16e8d3a9e1da126f106cbaff3af7ea34e73025a687fa1c5786fbfc699c6c0b6f4eaa7da1642935beb4607d5b8ea10202216b58a3e135fb426ecaa5b40d89f9f3

data/CHANGELOG.md

@@ -5,6 +5,81 @@ All notable changes to this project are documented here. This project adheres to
 
 ## [Unreleased]
 
+## [2.1.1]
+
+Two bugs found by using 2.1.0's own new feature on a real suite. The first is the most
+serious defect this project has shipped.
+
+### A worker that died took its share of the suite with it, silently
+
+A 192-test suite reporting **"99 passed, 0 failed"**, exit code 0. Ninety-three tests never
+ran, and nothing said so.
+
+One worker died partway through its bucket. The parent collected the results the other
+worker sent, found them non-empty, and reported them as the whole run. `worker_errors` --
+which had the reason, in full -- was only ever read on the path where *nothing* came back,
+so a worker dying beside a healthy one was recorded and then never mentioned.
+
+This is the exact failure mode the runner already had three separate defences against, all
+of which assume total failure. Partial failure walked straight past them.
+
+Each worker now reports its position after every item, so the parent knows what it
+scheduled and how far each worker actually got. Whatever a dead worker abandoned is run in
+the parent -- serially, in the one process that cannot also vanish unnoticed -- and the
+reason comes with it:
+
+```
+| 7 tests did not come back from a parallel worker, so they were run here instead --
+  everything ran, nothing was skipped. A worker died partway through its share:
+  SystemExit: Migrations are pending.
+```
+
+Results for an item are only written once that item finishes, so the boundary is exact and
+re-running from it cannot duplicate a result.
+
+### `establish_connection` was not changing database, and said nothing
+
+`WorkerDatabases` renames a config in place -- `db_config._database = "app_test_3"` --
+which is how Rails' own `TestDatabases` does it. But when the process already holds a pool
+for that same config object, `establish_connection` hands back the pool it has instead of
+opening the database the object now names. The query then succeeds and answers about the
+*source* database.
+
+Verified on a real app: `populated?` returned **true** for a SQLite worker database whose
+file did not exist. That is `constable prepare` reporting "already prepared" for work it
+never did -- and it is why 2.1.0's staleness check could not see a worker that was a
+migration behind. Dropping the pool first is what makes a rename take effect.
+
+A forked worker never hit this, because `before_fork!` clears every connection before the
+fork. It only ever went wrong in the parent, which is why it survived this long.
+
+Looking at a database also no longer borrows `ActiveRecord::Base`'s connection at all. It
+gets a named subclass with a pool of its own, so `constable prepare` and the staleness
+check leave every connection the app holds exactly where they were. Anonymous would not do:
+a class with no name falls back to its superclass's connection specification name, which is
+Base again.
+
+### Databases that cannot be sharded are named out loud
+
+Oracle, and anything else with `database_tasks: false`, cannot be given to each worker.
+Constable already skipped them — but *skipped* and *safe* are different claims, and only
+the first was ever made. Every worker shares that database, and the failures that follow do
+not look like a parallelism problem:
+
+```
+| vacols (database_tasks: false) cannot be given to each worker, so all of them share it.
+  Tests that write to it will interfere with each other, and the failures will not look
+  like a parallelism problem -- they look like rows vanishing mid-test.
+```
+
+Measured on a real app: 163 failures across four workers, every one of them passing
+serially, 53 of them a bare `VacolsRecordNotFound` -- one shared Oracle database that each
+worker's `before(:suite)` deleted from while the others were mid-test. The README has the
+rest, including the harder limit that no warning can fix: an app holding OCI handles is not
+reliably forkable, and aborts a good half of its parallel runs from inside the Oracle
+client.
+
+
 ## [2.1.0]
 
 Both changes here come from one afternoon on a real suite: nineteen files, eight forked

data/README.md

@@ -433,6 +433,30 @@ worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
 cache  = Rails.root.join("tmp/browser_cache#{worker}")
 ```
 
+Some databases cannot be given to each worker at all — Oracle and anything else Rails does
+not manage (`database_tasks: false`). Constable does not try, and now says so at the start
+of a parallel run, because *skipped* and *safe* are different claims:
+
+```
+⚠ vacols (database_tasks: false) cannot be given to each worker, so all of them share it.
+  Tests that write to it will interfere with each other, and the failures will not look
+  like a parallelism problem -- they look like rows vanishing mid-test.
+```
+
+That warning is worth taking literally. On a real app with a legacy Oracle database, a
+four-worker run produced 163 failures that all passed serially; 53 of them were a bare
+`VacolsRecordNotFound`, because each worker's `before(:suite)` deleted from the one shared
+database while the others were midway through tests that had just written to it.
+
+**And a harder limit, if your app talks to one through a C driver: forking may not be
+possible at all.** The same app aborts roughly half its parallel runs with SIGABRT — no
+output on either stream, the crash report landing inside `libclntsh`, Oracle's client
+library catching a SIGSEGV in its own signal handler. It is not a Constable failure and
+there is nothing Constable can do about it: a process holding OCI handles is not reliably
+forkable. If you see bare exit code 134 and no output, check
+`~/Library/Logs/DiagnosticReports` (or your platform's equivalent) before assuming the test
+runner ate your suite, and run those specs with `worker_databases: off`.
+
 `CONSTABLE_WORKER` is the index and `CONSTABLE_WORKERS` the count; both are unset in the
 parent, so serial runs keep whatever name they had. Use `FileUtils.mkdir_p` rather than
 `Dir.mkdir ... unless File.directory?` while you are there — the second is a race, and if

data/lib/constable/runner.rb

@@ -318,9 +318,31 @@ module Constable
         return false
       end
 
+      warn_about_shared_databases!
       worker_databases_current?(count)
     end
 
+    # A database Constable could not shard is shared by every worker, and that is worth
+    # saying out loud before the run rather than leaving it to be deduced from the wreckage.
+    #
+    # The failures it causes do not look like a parallelism problem. They look like records
+    # disappearing mid-test: worker 2's `before(:suite)` cleans the shared legacy database
+    # while worker 1 is halfway through a test that just created rows in it. Measured on a
+    # real suite -- 53 `VacolsRecordNotFound` failures across four workers, every one of
+    # them passing serially.
+    def warn_about_shared_databases!
+      shared = WorkerDatabases.unshardable_databases
+      return if shared.empty?
+
+      Constable.warn!(
+        "#{shared.join(", ")} cannot be given to each worker, so all of them share it. " \
+        "Tests that write to it will interfere with each other, and the failures will not " \
+        "look like a parallelism problem -- they look like rows vanishing mid-test. Run " \
+        "specs that touch it serially (`--workers 1`), or `worker_databases: off`.",
+        kind: :parallel
+      )
+    end
+
     # `:reuse` keeps the per-worker databases between runs, so they do not follow
     # migrations by themselves. A run against stale copies does not fail cleanly -- it
     # fails as a missing column in whichever file happened to touch it, on a different
@@ -421,8 +443,14 @@ module Constable
           # reason, and a run that scheduled nineteen files reports zero tests and exits
           # 0. A worker that dies has to say so.
           begin
-            bucket.each do |item|
+            # The position report after each item is what lets the parent finish the work
+            # if this worker dies partway. Results for an item are all written once the
+            # item is done, so "position n reported" means items 0...n are home and
+            # nothing from item n was ever sent -- the boundary is exact, and re-running
+            # from it cannot duplicate a result.
+            bucket.each_with_index do |item, position|
               run_item(item).each { |result| write_message(writer, :result, result.to_h) }
+              write_message(writer, :progress, position + 1)
             end
           rescue Exception => e # rubocop:disable Lint/RescueException
             write_message(writer, :worker_error, "#{e.class}: #{e.message}")
@@ -471,9 +499,45 @@ module Constable
       # results carry them home. Nothing that bends the rules is allowed to go missing
       # just because it happened in a subprocess.
       collected.each { |result| Constable.warnings.concat(Array(result.warnings)) }
-      collected
+      collected + finish_abandoned_work(buckets)
     end
 
+    # One worker dying used to cost its whole remaining bucket, silently.
+    #
+    # `worker_errors` was only ever read on the path where *nothing* came back, so a
+    # worker that died beside living ones was collected and never mentioned. The parent
+    # reported the results it happened to receive, called them the whole suite, and
+    # exited 0. Observed: a 192-test suite reporting "99 passed, 0 failed" -- green, with
+    # 93 tests that never ran.
+    #
+    # Now the parent knows what it scheduled and how far each worker actually got, so it
+    # can just run the rest itself. Serial, in this process, which is the one place that
+    # cannot also die without anyone noticing.
+    def finish_abandoned_work(buckets)
+      abandoned = buckets.each_with_index.flat_map do |bucket, index|
+        done = worker_progress[index]
+        done < bucket.size ? bucket[done..] : []
+      end
+      return [] if abandoned.empty?
+
+      Constable.warn!(
+        "#{abandoned.size} test#{"s" unless abandoned.size == 1} did not come back from " \
+        "a parallel worker, so #{abandoned.size == 1 ? "it was" : "they were"} run here " \
+        "instead -- everything ran, nothing was skipped. A worker died partway through " \
+        "its share#{worker_death_reason}.",
+        kind: :parallel
+      )
+      run_serial(abandoned)
+    end
+
+    def worker_death_reason
+      return "" if worker_errors.empty?
+
+      ": #{worker_errors.first}"
+    end
+
+    def worker_progress = (@worker_progress ||= Hash.new(0))
+
     def worker_errors = (@worker_errors ||= [])
 
     # A worker can die below Ruby: a segfault, an OOM kill, a signal. No `rescue` reaches
@@ -574,6 +638,9 @@ module Constable
               @reporter.record(result)
             when :coverage
               @worker_coverage = Constable::Coverage.merge_raw(@worker_coverage, body)
+            when :progress
+              index = readers.index(reader)
+              worker_progress[index] = body.to_i if index
             when :worker_error
               # A worker that could not start. Collected rather than raised, so the parent
               # decides what to do once it knows whether any worker got going at all.

data/lib/constable/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Constable
-  VERSION = "2.1.0"
+  VERSION = "2.1.1"
 end

data/lib/constable/worker_databases.rb

@@ -37,6 +37,13 @@ module Constable
 
     # Parent side, before the fork. A child inheriting a live connection is a corruption
     # risk in exactly the way an inherited SQLite handle is.
+    #
+    # `clear_all_connections!` and not `disconnect!`. Closing the pools outright was tried
+    # here, on the theory that returning a connection to the pool leaves the socket and the
+    # driver's C-side state for `fork` to copy into every child. It changed nothing
+    # measurable, and the crash it was meant to prevent turned out to predate it -- see
+    # the note on native drivers in the README. Rails does the same thing before its own
+    # fork, and matching it is the conservative choice.
     def before_fork!
       return false unless active_record?
 
@@ -139,8 +146,9 @@ module Constable
         end
         stale.uniq
       ensure
+        # Names only. Nothing here ever repointed ActiveRecord::Base, so there is no
+        # connection to put back -- which is the point.
         restore_database_names(original)
-        ::ActiveRecord::Base.establish_connection
       end
     rescue StandardError
       []
@@ -150,15 +158,54 @@ module Constable
     # Cheaper than diffing every version, and a worker that missed a migration differs in
     # both. nil when the question does not apply.
     def schema_fingerprint(db_config)
-      ::ActiveRecord::Base.establish_connection(db_config)
-      connection = ::ActiveRecord::Base.connection
-      return nil unless connection.table_exists?("schema_migrations")
+      with_probe_connection(db_config) do |connection|
+        next nil unless connection.table_exists?("schema_migrations")
 
-      connection.select_rows("SELECT COUNT(*), MAX(version) FROM schema_migrations").first
+        connection.select_rows("SELECT COUNT(*), MAX(version) FROM schema_migrations").first
+      end
     rescue StandardError
       nil
     end
 
+    # Databases this environment declares that cannot be given to each worker, so every
+    # worker shares the one copy.
+    #
+    # Constable skips them on purpose -- appending `_3` to an Oracle TNS service name names
+    # nothing -- but "skipped" and "safe" are different claims, and only the first was ever
+    # made. A legacy database that tests write to is shared mutable state across processes,
+    # and the failures it produces do not look like a parallelism problem: they look like
+    # records vanishing mid-test, in whichever spec happened to be running when another
+    # worker's suite hook cleaned the database they were both using.
+    #
+    # Measured on a real suite: 53 `VacolsRecordNotFound` failures across a four-worker
+    # run, every one of them passing serially, all from one shared Oracle database that
+    # each worker deleted from at startup.
+    def unshardable_databases
+      return [] unless active_record?
+
+      ::ActiveRecord::Base.configurations
+                          .configs_for(env_name: env_name, include_hidden: true)
+                          .filter_map { |db_config| share_reason(db_config) }
+    rescue StandardError
+      []
+    end
+
+    # Why a database stays shared, in the words of the setting that caused it. Both reasons
+    # matter and only one of them is about the adapter: `database_tasks: false` is how an
+    # app says "Rails does not manage this one", which is the usual way a legacy database is
+    # declared -- and it is exactly the database most likely to be shared, written to by
+    # tests, and cleaned by a suite hook in every worker at once.
+    #
+    # The config's *name* rather than its database, because a TNS descriptor is four lines
+    # of connection string and "vacols" is what anyone reading the warning calls it.
+    def share_reason(db_config)
+      name = db_config.respond_to?(:name) ? db_config.name : db_config.database
+      return "#{name} (database_tasks: false)" unless db_config.database_tasks?
+      return nil if shardable_adapter?(db_config)
+
+      "#{name} (#{db_config.adapter})"
+    end
+
     # The same configs `each_worker_config` renames, left under their real names.
     def each_source_config
       ::ActiveRecord::Base.configurations
@@ -347,12 +394,52 @@ module Constable
     # Present and holding tables. A database that exists but is empty is not prepared, and
     # silently running a suite against no tables is the worst of the available outcomes.
     def populated?(db_config)
-      ::ActiveRecord::Base.establish_connection(db_config)
-      ::ActiveRecord::Base.connection.tables.any?
+      with_probe_connection(db_config) { |connection| connection.tables.any? }
     rescue StandardError
       false
     end
 
+    # A connection class of its own, so looking at a database never disturbs the app's.
+    #
+    # Asking "is this worker database prepared, and has it run our migrations?" needs a
+    # connection, and the obvious way to get one is to point ActiveRecord::Base at it and
+    # then point it back. That works, right up until the app has a native driver attached.
+    #
+    # Measured on a real app with a legacy Oracle database: repointing Base in the parent
+    # before forking killed the whole run with SIGABRT, no output on either stream, the
+    # crash report landing inside libclntsh -- Oracle's client catching a SIGSEGV in its
+    # own handler and calling abort. Nothing about it says "your test runner opened a
+    # connection it did not need".
+    #
+    # A named subclass gets its own `connection_specification_name`, so its pool is its
+    # own: establishing and removing it leaves ActiveRecord::Base, and every other class
+    # with a connection, untouched. Anonymous would not do -- a class with no name falls
+    # back to its superclass's specification name, which is Base again.
+    def probe_class
+      base = ::ActiveRecord::Base
+      return @probe_class if defined?(@probe_class) && @probe_base.equal?(base)
+
+      klass = Class.new(base)
+      klass.abstract_class = true if klass.respond_to?(:abstract_class=)
+      # Naming it is not decoration: an unnamed class falls back to its superclass's
+      # connection specification name, which would put us right back on Base's pool.
+      remove_const(:ProbeConnection) if const_defined?(:ProbeConnection, false)
+      const_set(:ProbeConnection, klass)
+      @probe_base = base
+      @probe_class = klass
+    end
+
+    def with_probe_connection(db_config)
+      probe_class.establish_connection(db_config)
+      yield probe_class.connection
+    ensure
+      begin
+        probe_class.remove_connection
+      rescue StandardError
+        nil
+      end
+    end
+
     def env_name
       if defined?(::ActiveRecord::ConnectionHandling::DEFAULT_ENV)
         ::ActiveRecord::ConnectionHandling::DEFAULT_ENV.call

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: constable-rails
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 2.1.1
 platform: ruby
 authors:
 - Ray Hughes