constable-rails 1.4.0 → 1.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5fd3b3f867d958ed8f1306a51fb6a001145d88bd4c79839dd6d603f0b565248b
-  data.tar.gz: 1dee6cf2bece623c102744cb44e271461907d386715ed435fcc871ee131dd92c
+  metadata.gz: 807163563a2d149b1f1267c2dd873ebc19e9948664b8311f4185676ac9698753
+  data.tar.gz: d7c59c367bf69908c0909b2bda248f2d5d0c320a8238bbea628ac203134105a5
 SHA512:
-  metadata.gz: 1f6c418f0dd90a2e6e45733b237e8aa3c8e2e0d7c62b55d145dc7596bf047569983f243ba92558050612328f78834971cf5fef19e52efe4eb4deee0f5a751506
-  data.tar.gz: 74e6fd7f76215036de95d36ccdd84a068a71b999368e97f998e5e832c2230d3624eaf6aba4aafe0ff5edf2aa0245b15a050e85fb5784347c6407a01d72842711
+  metadata.gz: 16458c866e5975ed485a03793c52c4d918b1191f8760ee45fd814d824857c943f5a848d268888e2d29334850b360d6f1df56e4cc1f619ad347f79c94a2bec0ae
+  data.tar.gz: 0e7fc08cf6105a1fea100ebbda9dc3d316444982fe5ed467b09e45c27f8d36577711f8f175dc020773724f1973f2748b9a0fd76b0d791d24f48e1afb15a31ed7

data/CHANGELOG.md

@@ -5,6 +5,54 @@ All notable changes to this project are documented here. This project adheres to
 
 ## [Unreleased]
 
+## [1.4.1]
+
+### A parallel run that ran nothing is no longer a pass
+
+Pointing a real app at `worker_databases: reuse` produced this:
+
+```
+CONSTABLE            0 tests · 0 cases · 7.9s
+✓ 0 passed   ✗ 0 failed
+```
+
+Nineteen files scheduled. None ran. Exit 0. The third time this shape has appeared in a
+week, and the most dangerous instance of it.
+
+Two causes. A forked worker only reported `Constable::Error`, so anything else killed it
+silently — the parent saw a closed pipe, no results and no reason. Workers now report
+whatever they die of, including the exit status when they die below Ruby (a signal, a
+segfault, an OOM kill; forking an app that already holds native database connections can
+do exactly that). And the fallback to a serial run was conditional on a worker having
+managed to *explain* itself; it now triggers on the fact that matters — work was
+scheduled and nothing came back. If the serial fallback also produces nothing, that
+raises rather than being summarised as a clean zero.
+
+Same command now: **84 tests run**, serially, with a warning saying why.
+
+### Workers no longer build databases; `constable prepare` does
+
+`reuse` had each worker build its own missing databases after forking. On Postgres the
+clone must disconnect everything attached to the template first — and the template is the
+shared test database *every other worker* is cloning from at the same moment. Twelve
+workers terminated each other's connections and died mid-run.
+
+Preparation happens once, in the parent, through `constable prepare`. A worker that finds
+nothing to connect to says so and names the command. `constable prepare` also boots the
+app first, which it was not doing: it reported "this app has no ActiveRecord test
+databases to prepare" on an app with three of them.
+
+### Configuration is documented where you read it
+
+Every setting in `.constable/config.yml` now carries its explanation, and the ones with
+fixed choices name and describe each value inline — `worker_databases`, `output`, the
+storage adapter. Two tests enforce it: a setting cannot arrive without an explanation, and
+one with fixed choices has to name them.
+
+The blotter section now says outright that it is **not** your application's database.
+Two people read it the other way, which is a naming problem, not a reading problem.
+
+
 ## [1.4.0]
 
 Two problems from the same 1,277-file suite: a docket nobody asked for, and 110 minutes.
@@ -664,7 +712,8 @@ Initial release.
 - Diff-based coverage gate — only lines changed in the current diff are held to the
   threshold. `constable beat` for the full picture, `--html` for a browsable report.
 
-[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.4.0...HEAD
+[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.4.1...HEAD
+[1.4.1]: https://github.com/Ray-Hughes/constable/compare/v1.4.0...v1.4.1
 [1.4.0]: https://github.com/Ray-Hughes/constable/compare/v1.3.3...v1.4.0
 [1.3.3]: https://github.com/Ray-Hughes/constable/compare/v1.3.2...v1.3.3
 [1.3.2]: https://github.com/Ray-Hughes/constable/compare/v1.3.1...v1.3.2

data/lib/constable/cli.rb

@@ -428,6 +428,10 @@ module Constable
     option :workers, type: :numeric, desc: "How many to prepare (default: the configured worker count)"
     def prepare
       config = load_config
+      # The app has to be up before we can ask ActiveRecord anything about it.
+      Runner.boot!
+      config.apply_overrides!(Constable.configuration.overrides)
+
       unless WorkerDatabases.shardable?
         CLI.complain("This app has no ActiveRecord test databases to prepare.")
         exit(EXIT_USAGE)

data/lib/constable/runner.rb

@@ -64,6 +64,20 @@ module Constable
     def jail_run?   = @jail_run
     def coverage?   = @coverage_requested
 
+    # Boots the app the way a run does -- test/case_helper.rb, which requires
+    # config/environment -- without selecting or running anything.
+    #
+    # `constable prepare` needs this: it asks ActiveRecord what databases exist, and
+    # before the helper has run there is no ActiveRecord to ask. It reported "this app has
+    # no test databases to prepare" on an app with three of them.
+    def self.boot!
+      helper = %w[test/case_helper.rb spec/case_helper.rb]
+               .map { |p| File.join(Constable.root, p) }
+               .find { |p| File.exist?(p) }
+      require helper if helper
+      helper
+    end
+
     # Loads every case file and hands back the identities the suite actually defines,
     # without running anything. `constable prune` needs this: which tests still exist is
     # only knowable once the whole suite has been loaded.
@@ -352,8 +366,18 @@ module Constable
             exit!(0)
           end
 
-          bucket.each do |item|
-            run_item(item).each { |result| write_message(writer, :result, result.to_h) }
+          # Anything at all, not just Constable::Error. An uncaught exception in a forked
+          # child kills it silently: the parent sees a closed pipe, no results and no
+          # reason, and a run that scheduled nineteen files reports zero tests and exits
+          # 0. A worker that dies has to say so.
+          begin
+            bucket.each do |item|
+              run_item(item).each { |result| write_message(writer, :result, result.to_h) }
+            end
+          rescue Exception => e # rubocop:disable Lint/RescueException
+            write_message(writer, :worker_error, "#{e.class}: #{e.message}")
+            writer.close
+            exit!(0)
           end
 
           # A worker owns its own cold-case session, and it dies here. Fire the engine's
@@ -374,7 +398,7 @@ module Constable
       end
 
       collected = drain(readers)
-      pids.each { |pid| Process.waitpid(pid) rescue nil } # rubocop:disable Style/RescueModifier
+      record_worker_exits(pids)
 
       # No worker could build itself a database, so no test ran. Not every app can be
       # sharded: an app whose schema.rb cannot rebuild the database on its own -- Postgres
@@ -385,7 +409,13 @@ module Constable
       #
       # Nothing has run yet, so falling back to a serial run costs a restart, not
       # correctness.
-      return run_serially_after_worker_failure(items) if collected.empty? && worker_errors.any?
+      #
+      # The condition is deliberately "nothing came back", not "a worker said why". A
+      # child can die without managing to report -- and then a run that scheduled
+      # nineteen files says "0 tests, 0 failed" and exits 0, which is the worst thing a
+      # test runner can do. If work was scheduled and no result arrived, something is
+      # wrong whether or not anyone explained it.
+      return run_serially_after_worker_failure(items) if collected.empty? && !items.empty?
 
       # A warning raised inside a worker only ever reached that worker's memory, so the
       # results carry them home. Nothing that bends the rules is allowed to go missing
@@ -396,22 +426,58 @@ module Constable
 
     def worker_errors = (@worker_errors ||= [])
 
+    # A worker can die below Ruby: a segfault, an OOM kill, a signal. No `rescue` reaches
+    # that, so the only evidence is the exit status, and without it the run can only say
+    # "the workers exited without reporting anything" -- true, and useless.
+    #
+    # Forking a process that already holds database connections is where this comes from.
+    # An app with a native driver -- Oracle's OCI, for instance -- can have a child die
+    # the moment it touches an inherited handle.
+    def record_worker_exits(pids)
+      pids.each do |pid|
+        _, status = Process.waitpid2(pid)
+        next if status.nil? || status.success?
+
+        worker_errors << if status.signaled?
+                           "a worker was killed by SIG#{Signal.signame(status.termsig)} " \
+                             "-- forking an app that already holds native database " \
+                             "connections can do this"
+                         else
+                           "a worker exited with status #{status.exitstatus}"
+                         end
+      rescue StandardError
+        nil
+      end
+    end
+
     def run_serially_after_worker_failure(items)
       reason = worker_errors.first.to_s
+      reason = "the workers exited without reporting anything" if reason.empty?
 
       Constable.warn!(
-        "no parallel worker could build its own test database, so the suite ran serially " \
-        "instead. This usually means the app's schema cannot rebuild the database by " \
-        "itself -- Postgres custom types, functions and triggers are the common reason, " \
-        "and `rails test` parallelization fails the same way. Set " \
-        "`worker_databases: reuse` to keep prepared databases between runs instead, or " \
-        "`worker_databases: off` to stop trying. The first worker said: #{reason}",
+        "the parallel workers produced no results, so the suite ran serially instead -- " \
+        "everything ran, nothing was skipped. Two things cause this: the app's schema " \
+        "cannot rebuild a database by itself (Postgres custom types; try " \
+        "`worker_databases: reuse` with `constable prepare`), or forking is unsafe in " \
+        "this app, which happens when a native driver's connections are inherited by a " \
+        "child. `worker_databases: off` stops the attempt. The workers said: #{reason}",
         kind: :parallel
       )
 
       # The blotter handle was closed before forking, and the pool was cleared. Both come
       # back on their next use, so there is nothing to reopen by hand.
-      run_serial(items)
+      results = run_serial(items)
+
+      # Belt and braces. If the serial fallback also produces nothing for work that was
+      # scheduled, the run is broken in a way no summary can honestly describe, and
+      # reporting a clean zero would be a lie.
+      if results.empty? && !items.empty?
+        raise Constable::Error,
+              "#{items.size} test file(s) were scheduled and none of them ran. " \
+              "The first worker said: #{reason}"
+      end
+
+      results
     end
 
     # Every message on the pipe is tagged, because results are not the only thing a worker

data/lib/constable/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Constable
-  VERSION = "1.4.0"
+  VERSION = "1.4.1"
 end

data/lib/constable/worker_databases.rb

@@ -58,7 +58,7 @@ module Constable
     #            that works when the schema cannot rebuild the database by itself.
     def after_fork!(index, mode: :schema)
       return false unless shardable?
-      return reuse!(index) if mode.to_sym == :reuse
+      return connect_worker!(index) if mode.to_sym == :reuse
 
       ::ActiveRecord::TestDatabases.create_and_load_schema(index, env_name: env_name)
       true
@@ -172,8 +172,38 @@ module Constable
       nil
     end
 
-    # The :reuse half. Points every database this environment declares at its `_<index>`
-    # sibling, and only builds the ones that are not there yet.
+    # The :reuse half, inside a worker: connect to `<database>_<index>` and nothing else.
+    #
+    # Building is deliberately not done here. Twelve workers forked at once would each try
+    # to build the same missing databases simultaneously, and on Postgres the clone has to
+    # disconnect everything attached to the template first -- which is the shared test
+    # database every other worker is also cloning from. They terminate each other's
+    # connections and die, silently, mid-run. Observed exactly that: nineteen files
+    # scheduled, zero results, workers gone without a word.
+    #
+    # So preparation happens once, in the parent, through `constable prepare`. A worker
+    # that finds nothing to connect to says so, and the runner falls back to serial.
+    def connect_worker!(index)
+      missing = []
+
+      each_worker_config(index) do |db_config|
+        missing << db_config.database unless populated?(db_config)
+      end
+
+      unless missing.empty?
+        raise Constable::Error,
+              "worker #{index} has no database to use (#{missing.join(", ")}). " \
+              "`worker_databases: reuse` expects them to exist already -- run " \
+              "`constable prepare` once, then run the suite."
+      end
+
+      ::ActiveRecord::Base.establish_connection
+      []
+    end
+
+    # The building half, run from the parent by `constable prepare`. Points every database
+    # this environment declares at its `_<index>` sibling, and builds the ones that are
+    # not there yet.
     #
     # "There" means present *and* populated: an empty database is not a prepared one, and
     # connecting to it would hand the worker a suite with no tables. Deciding that per

data/lib/generators/constable/templates/config.yml.tt

@@ -43,6 +43,11 @@ cold_cases: []
 
 # The blotter: flake history, the jail docket, warrants.
 #
+# NOT your application's database, and nothing to do with it. Your tests keep using
+# whatever config/database.yml says -- Postgres stays Postgres. This is a separate file
+# holding Constable's own bookkeeping, and it never contains a row of your data. Leaving
+# it on SQLite is right even for a Postgres app.
+#
 # Constable owns this store outright. It is deliberately not the app's own
 # database, for two reasons. Native cases run inside a transaction that gets
 # rolled back, so writing "this test just failed" through that connection would
@@ -54,7 +59,10 @@ cold_cases: []
 # need one queryable store shared across many CI machines -- and always at a
 # separate database from the app's own, never its test connection.
 storage:
-  adapter: sqlite                      # sqlite (default) | postgres | mysql
+  adapter: sqlite                      # sqlite (default, and right for almost everyone)
+                                       #   postgres / mysql: only when many CI machines
+                                       #   need to share ONE blotter. Always a separate
+                                       #   database from your app's own.
   path: .constable/constable.sqlite3   # sqlite only
   url:                                 # postgres/mysql only, e.g.
                                        #   postgres://user:pass@host/constable_metadata
@@ -81,29 +89,45 @@ fail_on_warnings: false  # CI: fail the build when the warning count doesn't tre
 
 parallel_workers: auto   # or an explicit integer
 
-# How a parallel worker gets a database of its own. Sharing one is not a speed/safety
-# trade but a correctness bug: on SQLite the run dissolves into "database is locked", and
-# on a client/server database tests quietly see each other's rows.
-#
-#   schema  rebuild <database>_<index> from schema on every run. What Rails does for
-#           `rails test`, and correct by construction -- no drift is possible.
-#   reuse   connect to <database>_<index> when it is already there, and build it from
-#           schema only when it is not. Faster, because a large schema is not reloaded
-#           every run -- and the only option that works at all when the schema cannot
-#           rebuild the database by itself, which is true of any app with Postgres custom
-#           types (`CREATE TYPE` has no schema.rb representation, so a from-scratch load
-#           fails). Keeping those databases current becomes your job.
-#   off     do not shard, so do not fork. An explicit serial run: no attempt, no warning.
-worker_databases: schema   # schema | reuse | off
+# How a parallel worker gets a database of its own.
+#
+# Sharing one is not a speed/safety trade, it is a correctness bug: on SQLite the run
+# dissolves into "database is locked", and on a client/server database tests quietly see
+# each other's rows, which is worse because it looks like it worked.
+#
+# Three options:
+#
+#   schema   Rebuild <database>_<index> from schema.rb on every run.
+#            What Rails itself does for `rails test`. Correct by construction -- the
+#            databases cannot drift, because they are thrown away and rebuilt. The right
+#            default, and the wrong choice only when it is slow or impossible.
+#
+#   reuse    Keep <database>_<index> between runs, building it only when it is missing.
+#            Two reasons to want this:
+#              * Speed. A large schema is not replayed once per worker per run. On
+#                Postgres the databases are cloned in one statement
+#                (CREATE DATABASE ... TEMPLATE), which is far faster than a schema load.
+#              * It is the ONLY option that works when your schema cannot rebuild the
+#                database by itself -- true of any app with Postgres custom types, since
+#                CREATE TYPE has no schema.rb representation and a from-scratch load
+#                fails on a dump that references types it never defines.
+#            The cost: keeping those databases current is now your job. Run
+#            `constable prepare` after a migration.
+#
+#   off      Do not shard, so do not fork. An explicit serial run: no attempt to build
+#            anything, and no warning about it. Correct, and as slow as one core.
+#
+worker_databases: schema   # schema (rebuild each run) | reuse (keep them) | off (serial)
 
-# How much the live stream says while the suite is running. The summary is identical
-# either way -- this only changes what you watch on the way there.
-#
-#   concise   one glyph per test, grouped into a run per case. A thousand tests stay
-#             inside one screen, and a wall of green is the point.
-#   expanded  a line per test: glyph, name, duration. Slower to read in bulk, but you
-#             can see which test is hanging while it hangs, rather than after.
-output: concise          # concise | expanded
+# How much the live stream says while the suite is running. The summary at the end is
+# identical either way -- this only changes what you watch on the way there.
+#
+#   concise    One glyph per test, grouped into a run per case. A thousand tests stay
+#              inside one screen, and a wall of green is the point.
+#   expanded   A line per test: glyph, description, duration. Slower to read in bulk, but
+#              you can see which test is hanging while it hangs, rather than afterwards.
+#
+output: concise          # concise (one glyph per test) | expanded (a line per test)
 
 # Fallback path-based tier inference, used only when a case doesn't inherit from
 # a tiered base class. The base classes in test/case_helper.rb are the primary

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: constable-rails
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.4.1
 platform: ruby
 authors:
 - Ray Hughes