constable-rails 1.0.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 05e7e65ec7f1f096713bc5cb7b891ce2297275876c01cf9151130dc13608befe
-  data.tar.gz: db0b094a1e70a0052f02bb343cd0d41c53b224e56df089638ad6f5be60025e36
+  metadata.gz: 65a31ecd5db11b141e18762227af70518749d77042d45fad1626cc1713e06289
+  data.tar.gz: 6fe6e92175d4d306e47626a4d66e6e1ed029817c80af644f6c70f88f4c15c9df
 SHA512:
-  metadata.gz: 19e12d780b73ec73830eab6290b27d61f1fb8c9cd40894ff0c2832120883f06265e58b405ec107c1dab002d6bf484fb166a99cad273507d1b05e0da1234ae0df
-  data.tar.gz: 789ba56df5a2720e25ddbadec56d9916776f809c95b812748468addbdf05276292e76ee37686a40d8489c5678121d44d35e41a0b2b09ab6e72af82b016dc682a
+  metadata.gz: 717a3d9604ef554b4216a0fc497626b7bbff207767904c7c73d66805d70b7e3e60c932d030c728123e414823a346040817ddeb715fe84260ff46c8038bb02181
+  data.tar.gz: 4f22e2278306e93af27767a71ae63931ea4b1752dc403ef5b33d604cb6ca8b4c136267f3c3d95f78e74bfdaa581eab2d8f343a65bc49bd8e1180a434bbf1ebeb

data/CHANGELOG.md

@@ -5,6 +5,173 @@ All notable changes to this project are documented here. This project adheres to
 
 ## [Unreleased]
 
+## [1.2.0]
+
+Two bugs found installing 1.1.0 into a large real Postgres app (~3,000-line schema,
+custom types, a factory directory). Both are the same shape as the ones 1.0.0 fixed:
+Constable was confidently wrong and said nothing useful about it.
+
+### A factory named `*_case.rb` was loaded as a test
+
+`spec/**/*_case.rb` is a generous net, and a real app has things in it that merely share
+the suffix. Caseflow has a FactoryBot factory at `spec/factories/distributed_case.rb`.
+Constable loaded it as a case file, FactoryBot raised `DuplicateDefinitionError` because
+the factory was already registered, and the run reported a failing test in a file that
+contains no tests:
+
+```
+✗ spec/factories/distributed_case.rb
+  "could not be loaded"
+  FactoryBot::DuplicateDefinitionError: Factory already registered: distributed_case
+```
+
+A filename is not evidence. A file outside the conventional `test/cases/` and
+`spec/cases/` directories now has to look like a case before it is loaded — a class
+declaration, or the DSL. Files under those directories are still taken at their word,
+since that is what they are for and an empty one there is a case somebody is part-way
+through writing.
+
+### An app that cannot be sharded now runs anyway
+
+Per-worker databases (1.0.0) are built by loading `schema.rb` into `<database>_<index>`.
+Not every app can do that: one with Postgres custom types, functions or triggers cannot
+rebuild itself from `schema.rb` at all, which is exactly why such apps keep a
+`structure.sql`. Rails' own `parallelize` fails the same way.
+
+Constable handled it about as badly as possible. Each worker raised, printing a full
+stack trace — four workers, four traces, several hundred lines — and the parent then
+reported a run that had never happened:
+
+```
+CONSTABLE            1 test · 1 case · 10.8s
+✓ 0 passed   ✗ 1 failed
+```
+
+A worker that cannot build its database now reports that home rather than raising. If no
+worker got started, nothing has run yet, so the parent simply runs the suite serially and
+says why in one sentence — including the real error and how to skip the attempt
+(`parallel_workers: 1`). The rule from 1.0.0 is unchanged: a worker never falls back to
+sharing the parent's database, because that is the corruption this whole mechanism
+exists to prevent.
+
+
+## [1.1.0]
+
+Three things that were documented and did not work, plus the command for a docket
+that has gone stale.
+
+### `Constable.configure` actually configures things now
+
+The generated `test/case_helper.rb` told you to write `c.parallel_workers = 4`, explained
+when you would want to, and then **nothing in the codebase ever read it**. Four accessors,
+all inert.
+
+They work now, and every setting `.constable/config.yml` understands is settable in Ruby
+alongside them — `cold_cases`, `storage`, `warrants`, `warrant_retries`, `auto_relink`,
+`parole_period`, `coverage`, `coverage_threshold`, `coverage_html`, `fail_on_warnings`,
+`parallel_workers`, `output`, `tiers`. A test asserts the two halves stay in step, so a
+setting cannot be added to one and forgotten in the other.
+
+Precedence, and the reasoning:
+
+```
+a CLI flag              --workers 4, --expanded      one run, most specific
+Constable.configure     test/case_helper.rb          code you deliberately ran
+.constable/config.yml   the project's declared default
+Constable's defaults
+```
+
+Which to use? A setting that differs per machine or per branch belongs in the YAML, where
+it is obvious and greppable. A setting that has to be *computed* belongs in Ruby, because
+YAML cannot do this:
+
+```ruby
+Constable.configure do |c|
+  c.parallel_workers = ENV.fetch("CI_WORKERS", 4).to_i
+  c.coverage         = ENV["CI"] == "true"
+end
+```
+
+Values set in Ruby go through the same clamping as values set in the file, so a typo is no
+more dangerous in one than the other.
+
+**`storage` is the one exception, and it now says so.** The blotter is opened before
+`case_helper.rb` loads, so that `constable jail`, `warrants`, `watchlist` and `status` can
+read the docket without booting the app — a broken app should not stop you reading the
+docket. Setting it in Ruby would have been silently ignored, which is the exact failure
+this release exists to stop, so it raises and explains why.
+
+The docs now lead with Ruby. The generated `case_helper.rb` lists **every** setting at its
+default in one compact block — a test asserts the list stays complete, and that it never
+offers `storage`. The long-form explanation of each stays in `config.yml`, so the two files
+are a reference and a place to write code rather than two competing references.
+
+Neither is `config/initializers/`, and the reason is concrete: initializers run on every
+boot including production, where a test-only gem is not in the bundle, so an initializer
+calling `Constable.configure` takes the app down with a NoMethodError. Same reason RSpec,
+SimpleCov, WebMock and Capybara all configure from the test helper.
+
+### `.constable/config.yml` is now genuinely optional
+
+You can delete it. `rails generate constable:install --skip-config` never writes it.
+
+The one thing keeping it mandatory was `storage`, which cannot be set in Ruby for the
+ordering reason above. It now reads from the environment as well, which is the right shape
+for CI anyway, where the value is a secret and differs per machine:
+
+```
+CONSTABLE_STORAGE_URL=postgres://user:pass@host/constable_metadata
+CONSTABLE_STORAGE_PATH=/var/lib/constable/blotter.sqlite3
+CONSTABLE_STORAGE_ADAPTER=postgres
+```
+
+An adapter is inferred from the URL scheme when it is not given, and an empty variable
+means unset rather than "connect to the empty string". The `.constable/` **directory**
+still exists to hold the blotter — deliberately not `tmp/`, which `rails tmp:clear` and
+most deploys would wipe, taking weeks of flake history with it.
+
+One bug found while checking this end to end, in the override layer added above: settings
+were applied *after* the selection had already been asked for its targets, so
+`c.cold_cases` in Ruby was silently ignored and a suite of 192 ran 35. Overrides are now
+applied between requiring the helper and asking the selection anything.
+
+### Settings added after you install are no longer invisible
+
+`.constable/config.yml` doubles as the reference — every key at its default, with the
+reasoning above it — which only works if it stays current. A gem upgrade cannot rewrite it
+without clobbering your settings, and Thor's only other answer is to skip the file, so
+**`output` shipped in 1.0.0 and never appeared in any existing config.** The first anyone
+knew was running `bundle update` and finding the setting missing.
+
+`rails generate constable:install` now appends only the settings your file does not
+mention, each with its explanation, under a header saying where they came from. Your
+values and your own comments are never touched. Re-run it after any upgrade.
+
+### `constable prune`
+
+A test's key is a content hash of its body, so editing a jailed test gives it a new
+identity and leaves the old row behind — pointing at a `file:line` that may now hold
+something else. That is identity working as designed, and it was the one known limitation
+left open in 1.0.0. This is the broom:
+
+```console
+$ constable prune --dry-run   # list what would go
+$ constable prune             # forget it
+```
+
+It loads the whole suite first, because which tests still exist is only knowable once
+every case file has been read, and it is deliberately conservative in two directions:
+
+- **A known identity is never pruned**, even when the file recorded beside it is gone. The
+  path is a display label; the identity is the truth. A test that moved file has an
+  out-of-date label, not a missing test.
+- **A cold case is never pruned while its file exists.** Cold-case tests cannot be
+  enumerated without running their own engine, so their absence says nothing.
+
+Flake history is left alone either way — only the docket and outstanding warrants are
+touched.
+
+
 ## [1.0.0]
 
 The first release anybody should install.
@@ -241,6 +408,8 @@ Initial release.
 - Diff-based coverage gate — only lines changed in the current diff are held to the
   threshold. `constable beat` for the full picture, `--html` for a browsable report.
 
-[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.0.0...HEAD
+[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.2.0...HEAD
+[1.2.0]: https://github.com/Ray-Hughes/constable/compare/v1.1.0...v1.2.0
+[1.1.0]: https://github.com/Ray-Hughes/constable/compare/v1.0.0...v1.1.0
 [1.0.0]: https://github.com/Ray-Hughes/constable/compare/v0.1.0...v1.0.0
 [0.1.0]: https://github.com/Ray-Hughes/constable/releases/tag/v0.1.0

data/README.md

@@ -399,6 +399,7 @@ worse than one that resets.
 | `constable status` | How the suite is doing over time |
 | `constable beat [--html]` | Coverage: overall %, per-file, the unpatrolled list |
 | `constable history relink OLD NEW` | Carry history across a real body change |
+| `constable prune [--dry-run]` | Forget docket rows and warrants for tests that no longer exist |
 | `constable import --from=rspec` | Adopt an existing suite as cold cases |
 | `constable modernize PATH` | Opt-in AST rewrite into the native DSL |
 
@@ -557,6 +558,50 @@ storage:
 
 ### Configuration
 
+Settings can be written in Ruby, in `test/case_helper.rb` — the same place RSpec puts
+`RSpec.configure` — or in `.constable/config.yml`, or on the command line. The most
+specific wins:
+
+```
+a CLI flag              --workers 4, --expanded      one run
+Constable.configure     test/case_helper.rb          code you deliberately ran
+.constable/config.yml   the project's declared default
+Constable's defaults
+```
+
+Every key below can be set in either place. Put settings that differ per machine or per
+branch in the YAML, where they are obvious and greppable; put settings that have to be
+*computed* in Ruby, because YAML cannot:
+
+```ruby
+# test/case_helper.rb
+Constable.configure do |c|
+  c.parallel_workers = ENV.fetch("CI_WORKERS", 4).to_i
+  c.coverage         = ENV["CI"] == "true"
+  c.output           = :expanded
+end
+```
+
+Not `config/initializers/`, which is where a runtime gem like Devise goes. Initializers
+run on **every** boot including production, where a test-only gem is not in the bundle, so
+an initializer calling `Constable.configure` takes the app down. It is the same reason
+RSpec, SimpleCov, WebMock and Capybara all configure from the test helper.
+
+`config.yml` is optional, and one setting is the reason it exists: **`storage` can only be
+set there.** The blotter is opened before `case_helper.rb` loads, so that `constable jail`,
+`warrants`, `watchlist` and `status` can read the docket without booting the app — a
+broken app should not stop you reading the docket. Setting it in Ruby raises rather than
+being quietly ignored.
+
+Beyond that it is a preference. A settings file is greppable and diffable without running
+anything, which suits values that differ per project or per branch; Ruby suits anything
+computed.
+
+`config.yml` doubles as the reference, so re-run
+`rails generate constable:install --skip` after an upgrade: it appends any settings your
+file does not mention and leaves your own values and comments alone. (`--skip` so the
+other generated files, which you have probably edited, are left as they are.)
+
 ```yaml
 # .constable/config.yml
 cold_cases:

data/lib/constable/cli.rb

@@ -399,6 +399,48 @@ module Constable
       end
     end
 
+    desc "prune", "Forget docket rows and warrants for tests that no longer exist"
+    long_desc <<~DESC
+      A test's key is a content hash of its body, so editing a jailed test gives it a new
+      identity and leaves the old row behind -- pointing at a file:line that may now hold
+      something else. That is identity working as designed; this is the broom.
+
+      Loads the whole suite first, because which tests still exist is only knowable once
+      every case file has been read. Cold cases are never pruned while their file exists:
+      their tests cannot be enumerated without running their own engine, so silence about
+      them means nothing.
+    DESC
+    option :dry_run, type: :boolean, default: false, desc: "List what would go, change nothing"
+    def prune
+      config = load_config
+      known  = Runner.identities(config: config)
+      cold   = ->(path) { config.cold_case?(path) }
+
+      jail     = Jail.new(config: config, storage: Constable.storage)
+      warrants = Warrants.new(config: config, storage: Constable.storage)
+
+      stale_rows     = jail.stale_entries(known, cold_case: cold)
+      stale_warrants = warrants.stale_entries(known, cold_case: cold)
+
+      if stale_rows.empty? && stale_warrants.empty?
+        say "Nothing to prune -- every row on the docket still names a test that exists."
+        return 0
+      end
+
+      report_prunable("docket", stale_rows)
+      report_prunable("warrants", stale_warrants)
+
+      if options[:dry_run]
+        say "Dry run: nothing was changed."
+        return 0
+      end
+
+      stale_rows.each { |entry| jail.forget(entry.identity) }
+      stale_warrants.each { |entry| warrants.release(entry.identity) }
+      say "Pruned #{stale_rows.size + stale_warrants.size} row(s). Flake history is left alone."
+      0
+    end
+
     desc "jail SUBCOMMAND", "The jail docket"
     subcommand "jail", JailCommand
 
@@ -409,6 +451,16 @@ module Constable
     subcommand "history", HistoryCommand
 
     no_commands do
+      def report_prunable(heading, entries)
+        return if entries.empty?
+
+        say "#{heading} (#{entries.size}):"
+        entries.sort_by { |entry| [entry.file.to_s, entry.line.to_i] }.each do |entry|
+          say "  #{entry.location}  #{entry.label}"
+        end
+        say ""
+      end
+
       def load_config
         Constable.config
       end

data/lib/constable/config.rb

@@ -63,6 +63,16 @@ module Constable
       @raw  = deep_merge(@raw, stringify(overrides || {}))
     end
 
+    # Merges values set in Ruby (Constable.configure) over the ones read from the file.
+    # Called once, after case_helper.rb has been loaded -- which is the first moment those
+    # values exist.
+    def apply_overrides!(overrides)
+      return self if overrides.nil? || overrides.empty?
+
+      @raw = deep_merge(@raw, stringify(overrides))
+      self
+    end
+
     def cold_cases         = Array(@raw["cold_cases"])
     def warrants?          = truthy(@raw["warrants"])
     # Negative retries are a typo for "off", not an instruction to count backwards.
@@ -103,11 +113,33 @@ module Constable
     def expanded_output? = output_mode == :expanded
     def storage            = @raw["storage"] || {}
 
-    def storage_adapter = (storage["adapter"] || "sqlite").to_s
-    def storage_url     = storage["url"]
+    # Storage is the one setting that cannot be written in Ruby -- the blotter is opened
+    # before test/case_helper.rb loads, so `constable jail` and `constable status` can read
+    # the docket without booting the app. That would leave .constable/config.yml mandatory
+    # for anyone not on the default SQLite, so the environment can say it instead:
+    #
+    #   CONSTABLE_STORAGE_URL=postgres://user:pass@host/constable_metadata
+    #   CONSTABLE_STORAGE_PATH=/var/lib/constable/blotter.sqlite3
+    #
+    # Which is also the right shape for CI, where the value is a secret and differs per
+    # machine. The environment wins over the file, as an environment usually should.
+    def storage_url = env_or("CONSTABLE_STORAGE_URL", storage["url"])
+
+    def storage_adapter
+      explicit = env_or("CONSTABLE_STORAGE_ADAPTER", storage["adapter"])
+      return explicit.to_s if explicit
+
+      # A URL with no adapter names its own: postgres://... can only mean postgres.
+      scheme = storage_url.to_s[%r{\A([a-z][a-z0-9+.-]*)://}, 1]
+      return "postgres" if %w[postgres postgresql].include?(scheme)
+      return "mysql" if %w[mysql mysql2].include?(scheme)
+
+      "sqlite"
+    end
 
     def storage_path
-      path = storage["path"] || DEFAULTS["storage"]["path"]
+      path = env_or("CONSTABLE_STORAGE_PATH", storage["path"]) ||
+             DEFAULTS["storage"]["path"]
       File.absolute_path?(path) ? path : File.join(@root, path)
     end
 
@@ -145,6 +177,13 @@ module Constable
 
     private
 
+    # An empty environment variable is not a value. `CONSTABLE_STORAGE_URL=` in a CI
+    # config means "unset", not "connect to the empty string".
+    def env_or(name, fallback)
+      value = ENV.fetch(name, nil)
+      value.nil? || value.strip.empty? ? fallback : value.strip
+    end
+
     def truthy(value)
       return false if value.nil? || value == false
       return false if value.to_s.strip.downcase == "false"

data/lib/constable/jail.rb

@@ -241,6 +241,11 @@ module Constable
     # `constable jail release PATH:LINE`. Off the docket entirely, no supervision.
     def release(identity) = @storage.release(identity.to_s) ? true : false
 
+    # Pruning is not release. Release says "this test is trusted again"; prune says "this
+    # row is about a test that no longer exists". The distinction matters because release
+    # is a judgement someone made and prune is bookkeeping.
+    def forget(identity) = @storage.release(identity.to_s) ? true : false
+
     # `jail run` never auto-releases and never auto-paroles. One green run proves nothing;
     # it only earns a mention. A human reads this list and decides.
     #
@@ -360,6 +365,39 @@ module Constable
       matches
     end
 
+    # Rows whose test no longer exists.
+    #
+    # A test's key is a content hash of its body, so editing a jailed test gives it a new
+    # identity and leaves the old row behind -- pointing at a file:line that may now hold
+    # something else entirely. That is content-hash identity working as designed, but over
+    # a few months the docket fills with tests nobody can find.
+    #
+    # `known` is every identity the loaded suite registered, so this is only meaningful
+    # after a full load. Two things are deliberately conservative about it:
+    #
+    #   * A known identity is never stale, even when the file recorded beside it is gone.
+    #     The path is a display label; the identity is the truth. A test that moved file
+    #     has an out-of-date label, not a missing test.
+    #   * A cold case is never pruned while its file exists. Cold-case tests cannot be
+    #     enumerated without running their own engine, so absence from `known` says
+    #     nothing about them.
+    def stale_entries(known, cold_case: nil)
+      known = Array(known)
+      entries.reject { |entry| known.include?(entry.identity) }
+             .select { |entry| gone?(entry, cold_case) }
+    end
+
+    def gone?(entry, cold_case)
+      relative = entry.file.to_s
+      return true if relative.empty?
+
+      path = File.absolute_path?(relative) ? relative : File.join(Constable.root, relative)
+      return true unless File.exist?(path)
+      return false if cold_case&.call(relative)
+
+      true
+    end
+
     # An identity String, or nil when nothing matches -- or when more than one does.
     # Ambiguity is the caller's to report, with the candidates in hand.
     def resolve(target)

data/lib/constable/reporter.rb

@@ -100,14 +100,16 @@ module Constable
       0x1F900..0x1F9FF, 0x20000..0x3FFFD
     ].freeze
 
-    attr_reader :io, :config, :seed, :total, :mode
+    attr_reader :io, :config, :seed, :total
 
     def initialize(io: $stdout, config: nil, color: nil, slowest: DEFAULT_SLOWEST, mode: nil)
       @io      = io
       @config  = config || Constable.config
       @color   = resolve_color(color)
       @slowest = slowest.to_i
-      @mode    = resolve_mode(mode)
+      # Resolved lazily: the reporter is built before case_helper.rb has run, so reading
+      # the config now would miss anything Constable.configure sets.
+      @requested_mode = mode
       @io.set_encoding(Encoding::UTF_8) if @io.respond_to?(:set_encoding)
 
       reset_stream!
@@ -210,7 +212,8 @@ module Constable
     def failed?  = !success?
     def color?   = @color
     def finished? = @finished
-    def expanded? = @mode == :expanded
+    def mode      = @mode ||= resolve_mode(@requested_mode)
+    def expanded? = mode == :expanded
 
     private
 

data/lib/constable/runner.rb

@@ -64,6 +64,19 @@ module Constable
     def jail_run?   = @jail_run
     def coverage?   = @coverage_requested
 
+    # Loads every case file and hands back the identities the suite actually defines,
+    # without running anything. `constable prune` needs this: which tests still exist is
+    # only knowable once the whole suite has been loaded.
+    def self.identities(config: Constable.config)
+      selection = Selection.new([], config: config, root: Constable.root, full: true)
+      runner = new(selection: selection, config: config,
+                   reporter: Reporter.new(io: StringIO.new, config: config, color: false),
+                   storage: Constable.storage, workers: 1)
+      runner.send(:load_suite!)
+      Constable.registry.disambiguate_identities!
+      Constable.registry.investigations.map(&:identity)
+    end
+
     # => Integer exit status (0 clean, 1 failures)
     def call
       # Before anything is loaded: Coverage only counts files required after it starts, so
@@ -143,6 +156,15 @@ module Constable
                                                           .find { |p| File.exist?(p) }
       require helper if helper
 
+      # Between requiring the helper and asking the selection anything.
+      #
+      # Ordering is the whole point. case_helper.rb is where Constable.configure runs, so
+      # its settings do not exist until the line above. But Selection memoizes its targets
+      # the first time it is asked for them, and `cold_cases` is one of the settings people
+      # will most want to set in Ruby -- ask first and the override arrives too late to
+      # matter, silently. Requiring the helper needs no selection, so this fits between.
+      @config.apply_overrides!(Constable.configuration.overrides)
+
       @selection.native_targets.each { |target| load_case_file(target.path) }
       helper
     end
@@ -315,9 +337,17 @@ module Constable
           reader.close
 
           # Before a single test runs: build this worker's own database and point the
-          # process at it. Raises rather than falling back to the shared one, because a
-          # silent fallback is the bug we are here to prevent.
-          WorkerDatabases.after_fork!(worker_index)
+          # process at it. Never falls back to the shared one -- that is the bug this
+          # exists to prevent -- but the failure is reported home rather than raised.
+          # A raise here dumps a full stack trace per worker and leaves the parent
+          # reporting a run that never happened.
+          begin
+            WorkerDatabases.after_fork!(worker_index)
+          rescue Constable::Error => e
+            write_message(writer, :worker_error, e.message)
+            writer.close
+            exit!(0)
+          end
 
           bucket.each do |item|
             run_item(item).each { |result| write_message(writer, :result, result.to_h) }
@@ -343,6 +373,17 @@ module Constable
       collected = drain(readers)
       pids.each { |pid| Process.waitpid(pid) rescue nil } # rubocop:disable Style/RescueModifier
 
+      # No worker could build itself a database, so no test ran. Not every app can be
+      # sharded: an app whose schema.rb cannot rebuild the database on its own -- Postgres
+      # custom types, functions and triggers are the usual reason, and are exactly why
+      # such apps use structure.sql -- will fail here every time. Rails' own `parallelize`
+      # fails the same way; the difference is that this is not the user's fault and they
+      # should not have to read four stack traces to find that out.
+      #
+      # Nothing has run yet, so falling back to a serial run costs a restart, not
+      # correctness.
+      return run_serially_after_worker_failure(items) if collected.empty? && worker_errors.any?
+
       # A warning raised inside a worker only ever reached that worker's memory, so the
       # results carry them home. Nothing that bends the rules is allowed to go missing
       # just because it happened in a subprocess.
@@ -350,6 +391,25 @@ module Constable
       collected
     end
 
+    def worker_errors = (@worker_errors ||= [])
+
+    def run_serially_after_worker_failure(items)
+      reason = worker_errors.first.to_s
+
+      Constable.warn!(
+        "no parallel worker could build its own test database, so the suite ran serially " \
+        "instead. This usually means the app's schema cannot rebuild the database by " \
+        "itself -- Postgres custom types, functions and triggers are the common reason, " \
+        "and `rails test` parallelization fails the same way. Set `parallel_workers: 1` " \
+        "to skip the attempt. The first worker said: #{reason}",
+        kind: :parallel
+      )
+
+      # The blotter handle was closed before forking, and the pool was cleared. Both come
+      # back on their next use, so there is nothing to reopen by hand.
+      run_serial(items)
+    end
+
     # Every message on the pipe is tagged, because results are not the only thing a worker
     # has to send home.
     def write_message(writer, kind, body)
@@ -394,6 +454,10 @@ module Constable
               @reporter.record(result)
             when :coverage
               @worker_coverage = Constable::Coverage.merge_raw(@worker_coverage, body)
+            when :worker_error
+              # A worker that could not start. Collected rather than raised, so the parent
+              # decides what to do once it knows whether any worker got going at all.
+              worker_errors << body
             end
           end
         end

data/lib/constable/selection.rb

@@ -17,6 +17,22 @@ module Constable
       def to_s    = line ? "#{path}:#{line}" : path.to_s
     end
 
+    # A filename is not evidence. `spec/**/*_case.rb` is a generous net, and in a real app
+    # it catches things that merely share the suffix -- a FactoryBot factory named
+    # spec/factories/distributed_case.rb, say. Loading one of those runs somebody's code
+    # twice and reports the resulting explosion as a failing test in a file that contains
+    # no tests.
+    #
+    # So the file has to look like a case before we load it: a class declaration, or the
+    # DSL. Deliberately a cheap read rather than a parse -- this runs over every candidate
+    # on every run, and anything a parse would catch that this misses would also have to
+    # be a file that defines a case without mentioning one.
+    NATIVE_MARKERS = /
+      <\s*(?:Constable::Case|\w*Case)\b   # class FooCase < UnitCase
+      | ^\s*investigate\s*[("]             # or the DSL, for a reopened class
+      | ^\s*tier\s+:
+    /x
+
     NATIVE_GLOBS = [
       "test/cases/**/*.rb",
       "spec/cases/**/*.rb",
@@ -183,6 +199,17 @@ module Constable
 
     def native_files
       @native_files ||= glob(NATIVE_GLOBS).reject { |f| @config.cold_case?(f) }
+                                          .select { |f| native_by_content?(f) }
+    end
+
+    def native_by_content?(path)
+      # Files under the conventional case directories are taken at their word: that is
+      # what the directory is for, and an empty one there is a case file being written.
+      return true if path.match?(%r{/(?:test|spec)/cases/})
+
+      File.read(path).match?(NATIVE_MARKERS)
+    rescue StandardError
+      false
     end
 
     def cold_files

data/lib/constable/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Constable
-  VERSION = "1.0.0"
+  VERSION = "1.2.0"
 end

data/lib/constable/warrants.rb

@@ -244,6 +244,39 @@ module Constable
       matches
     end
 
+    # Rows whose test no longer exists.
+    #
+    # A test's key is a content hash of its body, so editing a jailed test gives it a new
+    # identity and leaves the old row behind -- pointing at a file:line that may now hold
+    # something else entirely. That is content-hash identity working as designed, but over
+    # a few months the docket fills with tests nobody can find.
+    #
+    # `known` is every identity the loaded suite registered, so this is only meaningful
+    # after a full load. Two things are deliberately conservative about it:
+    #
+    #   * A known identity is never stale, even when the file recorded beside it is gone.
+    #     The path is a display label; the identity is the truth. A test that moved file
+    #     has an out-of-date label, not a missing test.
+    #   * A cold case is never pruned while its file exists. Cold-case tests cannot be
+    #     enumerated without running their own engine, so absence from `known` says
+    #     nothing about them.
+    def stale_entries(known, cold_case: nil)
+      known = Array(known)
+      entries.reject { |entry| known.include?(entry.identity) }
+             .select { |entry| gone?(entry, cold_case) }
+    end
+
+    def gone?(entry, cold_case)
+      relative = entry.file.to_s
+      return true if relative.empty?
+
+      path = File.absolute_path?(relative) ? relative : File.join(Constable.root, relative)
+      return true unless File.exist?(path)
+      return false if cold_case&.call(relative)
+
+      true
+    end
+
     def resolve(target)
       matches = candidates(target)
       return matches.first.identity if matches.size == 1

data/lib/constable.rb

@@ -130,14 +130,60 @@ module Constable
   end
 
   # Code-level configuration set from test/case_helper.rb.
+  # The Ruby half of configuration, set from test/case_helper.rb.
+  #
+  # Anything that is *code* -- custom matchers, tier base classes, one-time global setup --
+  # can only live here. Everything that is merely a *setting* can live in either place, and
+  # this is the one that wins:
+  #
+  #   a CLI flag        for one run
+  #   Constable.configure   in case_helper.rb, because it is code and ran deliberately
+  #   .constable/config.yml the declared default for the project
+  #   Constable's defaults
+  #
+  # Until 1.1.0 the four accessors here were read by nothing at all: `c.parallel_workers = 4`
+  # was in the generated case_helper.rb, documented, and silently ignored.
   class Configuration
-    attr_accessor :parallel_workers, :seed, :coverage, :warrants
+    # Every setting .constable/config.yml understands, settable in Ruby as well. A nil is
+    # "not set here" rather than "set to nothing", so leaving one alone defers to the file.
+    SETTINGS = %i[
+      cold_cases warrants warrant_retries auto_relink parole_period
+      coverage coverage_threshold coverage_html fail_on_warnings parallel_workers
+      output tiers
+    ].freeze
+
+    # `storage` is the one setting that cannot live here, and the reason is ordering, not
+    # preference: the blotter handle is opened before test/case_helper.rb is loaded,
+    # because `constable jail`, `warrants`, `watchlist` and `status` all read the docket
+    # without booting the app at all. By the time this block runs, it is already open.
+    #
+    # Raising beats accepting the value and quietly using the old path -- silently
+    # ignoring a setting somebody wrote is the failure mode this whole class was fixed for.
+    SETTINGS_ONLY_IN_YAML = %i[storage].freeze
+
+    attr_accessor(*SETTINGS, :seed)
+
+    def storage=(_value)
+      raise ConfigurationError,
+            "storage must be set in .constable/config.yml, not Constable.configure. The " \
+            "blotter is opened before case_helper.rb loads, so that `constable jail` and " \
+            "`constable status` can read the docket without booting the app -- by the " \
+            "time this block runs the connection is already open."
+    end
 
     def initialize
       @before_suite_hooks = []
       @after_suite_hooks  = []
     end
 
+    # Only the settings actually assigned, ready to merge over the file's values.
+    def overrides
+      SETTINGS.each_with_object({}) do |name, out|
+        value = public_send(name)
+        out[name.to_s] = value unless value.nil?
+      end
+    end
+
     def before_suite(&block) = @before_suite_hooks << block
     def after_suite(&block)  = @after_suite_hooks << block
 

data/lib/generators/constable/install_generator.rb

@@ -45,6 +45,8 @@ module Constable
                                   desc: "Don't write the example case under test/cases/"
       class_option :skip_support, type: :boolean, default: false,
                                   desc: "Don't write the example files under test/support/"
+      class_option :skip_config, type: :boolean, default: false,
+                                 desc: "Don't write .constable/config.yml -- configure in Ruby instead"
 
       COLD_CASE_HEADER = <<~RUBY
         # Cold cases: your existing RSpec/Minitest files, run verbatim through their own
@@ -72,6 +74,15 @@ module Constable
         /.constable/*.sqlite3-*
       TEXT
 
+      NEW_SETTINGS_HEADER = <<~TEXT
+        # ---------------------------------------------------------------------------
+        # Added by `rails generate constable:install` on a later upgrade. These settings
+        # did not exist when this file was written; each is shown at its default, so
+        # deleting any of them changes nothing.
+        # ---------------------------------------------------------------------------
+
+      TEXT
+
       RUBOCOP_EXTENSION = "rubocop-constable"
 
       def create_case_helper
@@ -85,8 +96,32 @@ module Constable
         template "authenticatable.rb.tt", "test/support/authenticatable.rb"
       end
 
+      # The config file doubles as the reference -- every key at its default, with the
+      # reasoning above it -- which only works if it stays current. A gem upgrade cannot
+      # rewrite it (that would clobber your settings) and Thor's only other answer is to
+      # skip the file entirely, so before 1.1.0 a setting added after you installed was
+      # invisible: `output` shipped in 1.0.0 and never appeared in an existing config.
+      #
+      # So: create it if it is missing, and otherwise append only the settings it does not
+      # already mention. Your edits and comments are never touched.
       def create_config
-        template "config.yml.tt", ".constable/config.yml"
+        if options[:skip_config]
+          say_status :skip, ".constable/config.yml (configure in test/case_helper.rb instead)", :blue
+          return
+        end
+
+        path = File.join(destination_root, ".constable/config.yml")
+        return template("config.yml.tt", ".constable/config.yml") unless File.exist?(path)
+
+        missing = missing_config_blocks(File.read(path))
+        if missing.empty?
+          say_status :identical, ".constable/config.yml (every setting is documented)", :blue
+          return
+        end
+
+        names = missing.flat_map { |block| config_keys_in(block) }
+        say_status :append, ".constable/config.yml (#{names.join(", ")})", :green
+        append_to_file ".constable/config.yml", "\n#{NEW_SETTINGS_HEADER}#{missing.join("\n\n")}\n"
       end
 
       def create_example_case
@@ -204,6 +239,31 @@ module Constable
         end
       end
 
+      # Blocks in the shipped reference whose settings the existing file never mentions.
+      # A "block" is a run of lines between blank ones: the comment and the setting it
+      # explains travel together, because a bare key with no reasoning is not a reference.
+      def missing_config_blocks(existing)
+        present = config_keys_in(existing)
+
+        reference_blocks.select do |block|
+          keys = config_keys_in(block)
+          keys.any? && (keys - present) == keys
+        end
+      end
+
+      def reference_blocks
+        File.read(find_in_source_paths("config.yml.tt")).split(/\n{2,}/).map(&:rstrip).reject(&:empty?)
+      end
+
+      # Top-level YAML keys only, ignoring comments and nested ones -- a commented-out
+      # example is a suggestion, not a declaration.
+      def config_keys_in(text)
+        text.lines.filter_map do |line|
+          match = line.match(/\A([a-z][a-z0-9_]*):/)
+          match && match[1]
+        end.uniq
+      end
+
       def gemfile_contents
         File.read(File.join(destination_root, "Gemfile"))
       end

data/lib/generators/constable/templates/case_helper.rb.tt

@@ -139,16 +139,49 @@ end
 #  Code-level configuration.
 # -----------------------------------------------------------------------------
 
+#  Code only, by default: matchers and suite hooks can be written nowhere else.
+#
+#  Settings can be written here too, and this file wins over
+#  .constable/config.yml -- a CLI flag wins over both, for one run. Use the file
+#  for values that differ per project or per branch, where being greppable
+#  without running anything is worth something. Use Ruby when the value has to be
+#  computed, which YAML cannot do:
+#
+#      c.parallel_workers = ENV.fetch("CI_WORKERS", 4).to_i
+#      c.coverage         = ENV["CI"] == "true"
+#
+#  Every setting, at its default -- uncomment what you want and delete the rest.
+#  `.constable/config.yml` explains each one at length and is optional; between
+#  the two, this file can be the only place you configure Constable.
+#
+#      c.cold_cases         = []      # RSpec/Minitest globs to run as cold cases
+#      c.parallel_workers   = "auto"  # or an integer. Each worker gets its own database
+#      c.output             = :concise    # or :expanded -- a line per test, with timings
+#      c.fail_on_warnings   = false   # CI: fail when the warning count is not trending down
+#      c.warrants           = false   # rerun a failure in isolation before believing it
+#      c.warrant_retries    = 5
+#      c.parole_period      = 10      # clean runs before a paroled test releases itself
+#      c.auto_relink        = false   # auto-confirm rename detection
+#      c.coverage           = false
+#      c.coverage_threshold = 90      # diff-based: only lines changed in this diff
+#      c.coverage_html      = false
+#      c.tiers              = { "unit" => "test/cases/models/**/*" }
+#
+#  `storage` is the one setting that cannot go here: the blotter is opened before
+#  this file loads, so `constable jail` and `constable status` can read the docket
+#  without booting the app. Setting it here raises rather than being ignored. Use
+#  .constable/config.yml, or CONSTABLE_STORAGE_URL / _PATH / _ADAPTER.
+#
+#  Not config/initializers/: initializers run on every boot including production,
+#  where a test-only gem is not in the bundle. Same reason RSpec, SimpleCov and
+#  WebMock all configure from the test helper rather than an initializer.
+# -----------------------------------------------------------------------------
+
 Constable.configure do |c|
-  # Workers default to `auto` -- processor count minus a little headroom, so the
-  # machine stays usable while the suite runs. Pin it only when a CI container
-  # lies about its core count.
-  # c.parallel_workers = 4
-
-  # One-time global setup, run once per process before the whole suite. This is
-  # for configuring the world -- drivers, adapters, formats -- and deliberately
-  # not for creating records that tests then share. See the note at the bottom
-  # of this file about why that distinction is the whole ballgame.
+  # One-time global setup, run once per process before the whole suite. For
+  # configuring the world -- drivers, adapters, formats -- and deliberately not
+  # for creating records that tests then share. See the note at the bottom of
+  # this file about why that distinction is the whole ballgame.
   #
   # c.before_suite do
   #   Capybara.default_driver    = :rack_test

data/lib/generators/constable/templates/config.yml.tt

@@ -1,13 +1,29 @@
 # .constable/config.yml
 #
-# Settings, as opposed to code. Everything here is a number, a flag or a glob;
-# anything that is Ruby -- custom matchers, tier base classes, one-time global
-# setup -- lives in test/case_helper.rb instead.
+# Optional. Everything here can also be written in Ruby, in test/case_helper.rb:
 #
-# Every key Constable understands is present below at its default value, so this
-# file doubles as the complete reference. Delete anything you haven't changed;
-# the behavior is identical either way. CLI flags win over this file for the
-# duration of a single run.
+#     Constable.configure do |c|
+#       c.parallel_workers = ENV.fetch("CI_WORKERS", 4).to_i
+#       c.output           = :expanded
+#     end
+#
+# ...which is the better home for anything computed, and the more familiar one if you
+# come from RSpec's spec_helper.rb. Ruby wins over this file; a CLI flag wins over both.
+#
+# Two reasons this file exists at all:
+#
+#   1. `storage` cannot go anywhere else. The blotter is opened before case_helper.rb
+#      loads, so that `constable jail`, `warrants`, `watchlist` and `status` can read the
+#      docket without booting the app -- a broken app should not stop you reading the
+#      docket. Setting it in Ruby raises rather than being quietly ignored.
+#
+#   2. A settings file is greppable and diffable without executing anything, which is
+#      what you want for the values that differ per project or per branch.
+#
+# Every key Constable understands is below at its default, so this doubles as the
+# reference. Delete anything you have not changed; the behavior is identical either way.
+# Re-run `rails generate constable:install --skip` after upgrading and any settings added
+# since will be appended here, leaving your own values and comments alone.
 
 # Glob paths to run as cold cases: original RSpec/Minitest files, driven verbatim
 # through their own real engine, with pass/fail/timing fed into Constable's

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: constable-rails
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Ray Hughes