deflake 0.2.1 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9af8f08f69b7cef40caa8ec740811275d5d32ae606d44f038bd34d5865a9378a
4
- data.tar.gz: 3d718355bf548526b415419f925703a1a0ab9d52f451c168fcaf8cf46d366a2b
3
+ metadata.gz: c313183f1186641fa3aa7604e07d53871f5d7b6299b3987d72c2aa1ced56179c
4
+ data.tar.gz: 16d9f33ae4939adab4919dfd2f737c308e7d19d230f9d6e46c4c9ace52e57212
5
5
  SHA512:
6
- metadata.gz: 3baf1b38bf0cc1e6d97d2bd6ce4c02bad7f4e8f6f9437ad6c2a33d5517849b9719f3567cd7be236cf758a3863c9cfe06d02bd9d1f770cdc07291632e5059bc6a
7
- data.tar.gz: a389907125c1f66b9c77cf01f10267b3fea760df4972258499c19b9cfd27fca05b1b00874ca3740459a05ce8c3237be219a627f24159e5a76950d5c7db8b32c1
6
+ metadata.gz: ced157cbbd367522e0c24cd6c755b8920a7f303eb0f633d365a27c436efde7b16618f68e5c23c0d0b40045a08c48afa6efbfb80eefd6606f1b94ec58c722f0d7
7
+ data.tar.gz: cb425720892e8c3cf1bf016334fa3e5bbda4183d27717d6a473fcacf20bd088d40967841bc889079c8904859b00b06dbd06b93fa868544dacfb69e7faaebd9fb
data/CHANGELOG.md CHANGED
@@ -1,6 +1,11 @@
1
1
  # Changelog
2
2
 
3
- ## [Unreleased]
3
+ ## [0.2.2] - 2026-11-11
4
+
5
+ - The Rows detector counts rows with queries named `SCHEMA`, so Rails' `assert_no_queries` and the log leave them out. A test that loads cached fixtures inside `assert_no_queries` no longer fails under deflake.
6
+ - A test that fails at random on its own no longer gets culprits: errors that differ only in numbers count as the same error, so they are no reason to bisect.
7
+ - In RSpec, a bisect runs only the examples it asks for, also when a file brings in examples of another one with `require_relative`.
8
+ - With `--jobs`, deflake says so when test processes share a database: it names the database when ActiveRecord connected them to the same one, and also warns when the errors show them waiting for each other on one.
4
9
 
5
10
  ## [0.2.1] - 2026-11-09
6
11
 
data/README.md CHANGED
@@ -1,6 +1,18 @@
1
+ <div align="center">
2
+
1
3
  # deflake
2
4
 
3
- Find flaky tests in Minitest and RSpec and explain why they fail.
5
+ **Find flaky tests in Minitest and RSpec and explain why they fail.**
6
+
7
+ Repeat · Isolate · Bisect · Detect leaked state
8
+
9
+ [![Gem Version](https://img.shields.io/gem/v/deflake?logo=rubygems&logoColor=white&color=cc342d)](https://rubygems.org/gems/deflake)
10
+ [![Downloads](https://img.shields.io/gem/dt/deflake?logo=rubygems&logoColor=white)](https://rubygems.org/gems/deflake)
11
+ [![CI](https://github.com/SlyDeath/ruby-deflake/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SlyDeath/ruby-deflake/actions/workflows/ci.yml)
12
+ [![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.3-cc342d?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
13
+ [![License: MIT](https://img.shields.io/github/license/SlyDeath/ruby-deflake?color=blue)](LICENSE.txt)
14
+
15
+ </div>
4
16
 
5
17
  A flaky test fails once in a while, someone adds a retry, and the cause stays.
6
18
  deflake runs a suspicious test many times and answers three questions:
@@ -46,7 +58,7 @@ What happens:
46
58
 
47
59
  1. **The whole file runs `--runs` times** (10 by default), every time in a fresh `ruby` process with a new random seed. Running the whole file, not just one test, is what makes order-dependent failures show up. A test that takes its process down (`exit`, a crash, a signal) counts as failed in that run.
48
60
  2. **Every test that failed runs again on its own**, as many times. The verdict is *fails on its own* if it still fails, which points at time, randomness or a race inside the test, or *fails only after other tests* if it does not, or if it skips itself every time it runs alone. When it cannot run alone at all, as with errors outside of tests or a file that breaks before any test whenever one test runs on its own, the verdict is *failed, could not run it alone*. The errors of the runs alone are shown too when the runs of the file did not have them.
49
- 3. **Order-dependent failures are bisected** with one of the failing seeds, down to a smallest set of tests that break the victim; when two tests break it independently, one set is named. deflake has the test framework run just those tests in the recorded order, so hooks around a whole class still run, and watches only the victim, so other failing tests do not get in the way. A failure counts only when it happens twice in a row, and the set found must break the victim again at the end, so a test that fails at random gets no culprits: when the failure does not happen again with the same seed and order, the verdict is *fails at random, not after particular tests*, with the command that runs that seed. Tests in a class without a name cannot be put in an order, so when the file has some and the failure does not happen again without them, the verdict stays *fails only after other tests*, and deflake says it could not look among them. When the victim fails with that seed with no test before it, the verdict is *fails on its own*, with the command that shows it. Up to three failing seeds are tried, the first seed of every error first. A test that fails on its own is bisected too when the runs of the file have an error it never had alone. A culprit that two other tests of its group replace, because the state comes from a hook of the group such as `before(:context)` or `before_all`, is named as such. A victim stopped by `--timeout` does not count as failing in a bisect, as fewer tests before it would not reach the limit. This works on Minitest 5 and 6 alike, `Minitest::Spec` included, and on RSpec, which keeps the relative order of any subset of examples for a seed.
61
+ 3. **Order-dependent failures are bisected** with one of the failing seeds, down to a smallest set of tests that break the victim; when two tests break it independently, one set is named. deflake has the test framework run just those tests in the recorded order, so hooks around a whole class still run, and watches only the victim, so other failing tests do not get in the way. A failure counts only when it happens twice in a row, and the set found must break the victim again at the end, so a test that fails at random gets no culprits: when the failure does not happen again with the same seed and order, the verdict is *fails at random, not after particular tests*, with the command that runs that seed. Tests in a class without a name cannot be put in an order, so when the file has some and the failure does not happen again without them, the verdict stays *fails only after other tests*, and deflake says it could not look among them. When the victim fails with that seed with no test before it, the verdict is *fails on its own*, with the command that shows it. Up to three failing seeds are tried, the first seed of every error first. A test that fails on its own is bisected too when the runs of the file have an error it never had alone; errors that differ only in numbers, as those of a test that fails at random do, count as the same error. A culprit that two other tests of its group replace, because the state comes from a hook of the group such as `before(:context)` or `before_all`, is named as such. A victim stopped by `--timeout` does not count as failing in a bisect, as fewer tests before it would not reach the limit. This works on Minitest 5 and 6 alike, `Minitest::Spec` included, and on RSpec, which keeps the relative order of any subset of examples for a seed.
50
62
  4. **Every test is checked for leaks**, tests that skip themselves included: a snapshot of global state before the test, setup included, and after it, teardown, transaction rollback and whatever Rails does around a test (such as the executor clearing `Current`) included. Whatever changed is reported. So is what a whole Minitest class or top-level RSpec group leaves behind beyond the leaks of its tests, such as the state a `before_all` or `before(:context)` hook sets up and never puts back: it shows up under *errors outside of tests*.
51
63
 
52
64
  A process that fails while every test in it passed is reported as *errors outside of tests*, with the end of its output. An error RSpec reports outside of examples, such as in an `after(:context)` hook, is reported with its class and message, even when an example failed in the same run. So is a run that breaks before its first test, such as a `before(:suite)` hook that fails now and then: the other runs still count. A failure deflake cannot run again by name, such as a class-level hook (minitest-hooks' `before_all`) that reports its error as a result, shows up there too. Only when no run gets to a test does deflake give up with exit code 2, and say how the test process ended, for example that it timed out.
@@ -80,6 +92,8 @@ test:
80
92
  database: myapp_test<%= ENV["TEST_ENV_NUMBER"] %>
81
93
  ```
82
94
 
95
+ When processes share a database, their rows get in each other's way and the results say more about the database than about the tests, so deflake says so: it names the database when ActiveRecord connected the processes to the same one, and otherwise warns when the errors show them waiting for each other on one, such as `database is locked` from SQLite.
96
+
83
97
  Every test process runs in a session of its own, without a terminal, as in CI: its input is empty and its output is not a TTY, so a test that skips itself unless `$stdin.tty?` always skips under deflake. In RSpec, deflake runs every example even when the project asks for `fail_fast`, and leaves out the profile; options in `SPEC_OPTS` that set the seed, the order, the formatters or `--fail-fast` are left out too, as deflake sets them itself.
84
98
 
85
99
  deflake sets `PARALLEL_WORKERS=1` and `MT_CPU=1` for every test process, so neither Rails' `parallelize` nor Minitest's `parallelize_me!` runs tests at once: the order a seed gives stays the same and the leaks of one test do not show up in another. To rerun a seed by hand, set them too, as the `reproduce:` command does; without `PARALLEL_WORKERS=1` Rails gives the same seed a different order.
data/lib/deflake/check.rb CHANGED
@@ -7,6 +7,8 @@ module Deflake
7
7
  PATTERN = "**/*_{test,spec}.rb"
8
8
  # Folders of installed gems and packages, whose tests are not the project's
9
9
  SKIPPED = %w[vendor node_modules].freeze
10
+ # Errors of test processes that wait for each other on one database
11
+ LOCKED = /database is locked|BusyException|database table is locked|deadlock/i
10
12
 
11
13
  def initialize(report, err:, runs:, jobs:)
12
14
  @report = report
@@ -84,6 +86,7 @@ module Deflake
84
86
  end
85
87
  # As with a plain run, tests that all skipped themselves are not an error
86
88
  @err.puts "deflake: #{skipped(path, line)}, nothing to report" if diagnoses.empty? && not_run.empty?
89
+ warn_shared_database(diagnoses, investigation)
87
90
  output { @report.print(diagnoses, file: path, not_run: not_run, shared: investigation.shared_names) }
88
91
  diagnoses.all?(&:clean?) && not_run.empty? ? 0 : 1
89
92
  end
@@ -98,6 +101,26 @@ module Deflake
98
101
  nil
99
102
  end
100
103
 
104
+ # Once per run of deflake, as the files of a folder would all repeat it. Databases that
105
+ # ActiveRecord connects to are known; errors of waiting for a lock tell about any other one
106
+ def warn_shared_database(diagnoses, investigation)
107
+ return if @jobs < 2 || @warned_shared
108
+
109
+ shared = investigation.shared_databases.to_a
110
+ errors = diagnoses.flat_map { |diagnosis| diagnosis.file_runs.errors.keys }
111
+ if shared.any?
112
+ warn_shared("test processes running at once used the same database (#{shared.join(", ")})")
113
+ elsif errors.any? { |error| LOCKED.match?(error.to_s) }
114
+ warn_shared("test processes running at once seem to wait for each other on one database")
115
+ end
116
+ end
117
+
118
+ def warn_shared(problem)
119
+ @warned_shared = true
120
+ @err.puts "deflake: #{problem}, so they change each other's rows and the results may be wrong; " \
121
+ "give each its own database with TEST_ENV_NUMBER, or run with --jobs 1"
122
+ end
123
+
101
124
  def skipped(path, line)
102
125
  return "the test at line #{line} skipped itself in every run" if line
103
126
 
@@ -30,6 +30,9 @@ module Deflake
30
30
  summaries.values.sort_by { |summary| [summary.name, summary.id] }.map { |summary| diagnose(summary) }
31
31
  end
32
32
 
33
+ # Databases that the test processes running at once shared in the runs of the whole file
34
+ attr_reader :shared_databases
35
+
33
36
  # Names that more than one test of the file has, so a report tells them apart by their ids
34
37
  def shared_names
35
38
  @names.values.tally.select { |_, count| count > 1 }.keys
@@ -41,6 +44,7 @@ module Deflake
41
44
  def focused(focus)
42
45
  file = runner
43
46
  summaries = file.call
47
+ @shared_databases = file.shared_databases
44
48
  # Of every test that ran, including those that skipped themselves
45
49
  @names = file.names
46
50
  @not_run = focus ? focus - file.seen : []
@@ -70,11 +74,22 @@ module Deflake
70
74
  errors = diagnosis.file_runs.errors
71
75
  case diagnosis.verdict
72
76
  when :order_dependent then seeds_of(errors)
73
- when :fails_on_its_own then seeds_of(errors.except(*diagnosis.isolated_runs.errors.keys))
77
+ when :fails_on_its_own then seeds_of(new_errors(errors, diagnosis.isolated_runs.errors))
74
78
  else []
75
79
  end
76
80
  end
77
81
 
82
+ # Numbers left out: a test that fails at random tells other values every time. Bisecting it would
83
+ # name whatever test draws random numbers before it, as Minitest seeds them once for the whole run
84
+ def new_errors(errors, alone)
85
+ known = alone.keys.map { |message| shape(message) }
86
+ errors.reject { |message, _| known.include?(shape(message)) }
87
+ end
88
+
89
+ def shape(message)
90
+ message.to_s.gsub(/0x\h+|\d+(?:\.\d+)?/, "N")
91
+ end
92
+
78
93
  # The first seed of every error first, as a single odd failure may be the first one
79
94
  def seeds_of(errors)
80
95
  firsts = errors.values.map(&:first)
@@ -0,0 +1,41 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Deflake
4
+ module Leaks
5
+ # When several test processes run at once, each tells deflake which databases it uses, so that
6
+ # deflake can say when two of them share one: their rows would get in each other's way
7
+ module Databases
8
+ module_function
9
+
10
+ def record(pools)
11
+ folder = ENV.fetch(ResultFile::DATABASES_KEY, nil)
12
+ return unless folder
13
+
14
+ names = pools.filter_map { |pool| name(pool) } - recorded
15
+ return if names.empty?
16
+
17
+ recorded.concat(names)
18
+ File.open(File.join(folder, "worker#{ENV.fetch("TEST_ENV_NUMBER", "")}"), "a") { |file| file.puts(names) }
19
+ end
20
+
21
+ def recorded
22
+ @recorded ||= []
23
+ end
24
+
25
+ # A database in memory belongs to its process alone
26
+ def name(pool)
27
+ config = config(pool)
28
+ database = config[:database].to_s
29
+ return if database.empty? || database.include?(":memory:") || database.include?("mode=memory")
30
+
31
+ database = File.expand_path(database) if config[:adapter].to_s.include?("sqlite")
32
+ [config[:adapter], config[:host], config[:port], database].compact.join(" ")
33
+ end
34
+
35
+ # Rails before 6.1 has no db_config
36
+ def config(pool)
37
+ pool.respond_to?(:db_config) ? pool.db_config.configuration_hash : pool.spec.config
38
+ end
39
+ end
40
+ end
41
+ end
@@ -68,6 +68,7 @@ module Deflake
68
68
  pools = connected_pools
69
69
  return if pools.empty?
70
70
 
71
+ Leaks.quietly { Databases.record(pools.map(&:last)) }
71
72
  Rows.depth = Rows.open_transactions
72
73
  State.new(pools.to_h { |owner, pool| [owner, count(pool)] }, Rows.loads.to_i)
73
74
  end
@@ -134,12 +135,14 @@ module Deflake
134
135
  end
135
136
 
136
137
  # Outside the query cache: a count the test makes itself, such as User.count, must still go to
137
- # the database, and must not find the cache emptied by deflake either
138
+ # the database, and must not find the cache emptied by deflake either. Named SCHEMA, as Rails'
139
+ # own internal queries are, so that assert_no_queries and the log leave the counts out
138
140
  def count(pool)
139
141
  pool.with_connection do |connection|
140
142
  uncached(connection) do
141
143
  (connection.tables - INTERNAL_TABLES).to_h do |table|
142
- [table, connection.select_value("SELECT COUNT(*) FROM #{connection.quote_table_name(table)}").to_i]
144
+ sql = "SELECT COUNT(*) FROM #{connection.quote_table_name(table)}"
145
+ [table, connection.select_value(sql, "SCHEMA").to_i]
143
146
  end
144
147
  end
145
148
  end
data/lib/deflake/leaks.rb CHANGED
@@ -6,6 +6,7 @@ require_relative "leaks/globals"
6
6
  require_relative "leaks/cache"
7
7
  require_relative "leaks/current"
8
8
  require_relative "leaks/rows"
9
+ require_relative "leaks/databases"
9
10
  require_relative "leaks/jobs"
10
11
 
11
12
  module Deflake
@@ -13,9 +13,11 @@ module Deflake
13
13
  ONLY_KEY = "DEFLAKE_ONLY"
14
14
  # Path of the file whose tests to list, with where each is defined, instead of running them (Minitest)
15
15
  LIST_KEY = "DEFLAKE_LIST"
16
+ # Folder where each test process running alongside others writes which databases it used
17
+ DATABASES_KEY = "DEFLAKE_DATABASES"
16
18
  # Every variable that tells a test process what to do: a test that starts deflake itself
17
19
  # must not pass them on
18
- KEYS = [ENV_KEY, ORDER_KEY, ONLY_KEY, LIST_KEY].freeze
20
+ KEYS = [ENV_KEY, ORDER_KEY, ONLY_KEY, LIST_KEY, DATABASES_KEY].freeze
19
21
  # Characters JSON escapes with a short form; other control characters become \u00XX
20
22
  ESCAPES = { '"' => '\\"', "\\" => "\\\\", "\n" => "\\n", "\r" => "\\r", "\t" => "\\t" }.freeze
21
23
 
data/lib/deflake/rspec.rb CHANGED
@@ -33,11 +33,20 @@ module Deflake
33
33
 
34
34
  module_function
35
35
 
36
- # The examples deflake asked for through a file, which holds more of them than a command line could
36
+ # The examples deflake asked for through a file, which holds more of them than a command line could.
37
+ # RSpec filters by ids only the files it has ids for, so the examples of a file brought in with
38
+ # require_relative would all run: the others are excluded by their ids too. An example's :id is
39
+ # not in its metadata until asked for, so the filter builds it from the parts RSpec builds it from
37
40
  def only(config, path)
38
- ResultFile.unpack_ids(File.read(path)).group_by { |id| id[/\A(.+)\[[^\]]+\]\z/, 1] }.each do |file, ids|
41
+ wanted = ResultFile.unpack_ids(File.read(path))
42
+ wanted.group_by { |id| id[/\A(.+)\[[^\]]+\]\z/, 1] }.each do |file, ids|
39
43
  config.filter_manager.add_ids(file, ids.map { |id| id[/\[([^\]]+)\]\z/, 1] }) if file
40
44
  end
45
+ exclude_others(config, wanted.to_set)
46
+ end
47
+
48
+ def exclude_others(config, wanted)
49
+ config.filter_run_excluding(rerun_file_path: ->(file, meta) { !wanted.include?("#{file}[#{meta[:scoped_id]}]") })
41
50
  end
42
51
 
43
52
  # The leaks of a top-level group that none of its examples left: the state a before(:context)
@@ -22,12 +22,18 @@ module Deflake
22
22
  attr_reader :seen
23
23
  # Every summary of the last call, the quiet ones too, and every test's name by its id
24
24
  attr_reader :all, :names
25
+ # Databases that test processes running at once used together in the last call
26
+ attr_reader :shared_databases
25
27
 
26
28
  # Returns a hash of test id => Summary. Runs where a test skipped itself count only for its
27
29
  # leaks, and a test that skipped itself every time and leaked nothing is left out.
28
30
  # Errors outside of any test, in a run of the whole file, show up as Execution::OUTSIDE
29
31
  def call
30
- rows = in_parallel(seeds) { |seed, worker| execute(seed, worker_env(worker)) }
32
+ databases = SharedDatabases.new
33
+ rows = databases.watch(@jobs) do |folder|
34
+ in_parallel(seeds) { |seed, worker| execute(seed, worker_env(worker, folder)) }
35
+ end
36
+ @shared_databases = databases.shared
31
37
  check_some_ran(rows)
32
38
  check_some_tests(rows)
33
39
  # The id, not the name: RSpec examples can share a description
@@ -110,10 +116,10 @@ module Deflake
110
116
  end
111
117
 
112
118
  # Same convention as parallel_tests: "", "2", "3"... so each process can use its own database
113
- def worker_env(worker)
119
+ def worker_env(worker, folder)
114
120
  return {} if @jobs == 1
115
121
 
116
- { "TEST_ENV_NUMBER" => worker.zero? ? "" : (worker + 1).to_s }
122
+ { "TEST_ENV_NUMBER" => worker.zero? ? "" : (worker + 1).to_s, ResultFile::DATABASES_KEY => folder }
117
123
  end
118
124
 
119
125
  # Leaks outside of tests, such as those of a before(:all) hook, are not quiet either
@@ -0,0 +1,33 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "tmpdir"
4
+
5
+ module Deflake
6
+ # Databases that test processes running at once both used: each process writes the names
7
+ # of its databases to a file of its own in a folder, see Leaks::Databases
8
+ class SharedDatabases
9
+ attr_reader :shared
10
+
11
+ def initialize
12
+ @shared = []
13
+ end
14
+
15
+ # Yields the folder, nil with one process at a time, when no database can be shared
16
+ def watch(jobs)
17
+ return yield nil if jobs == 1
18
+
19
+ Dir.mktmpdir do |folder|
20
+ result = yield folder
21
+ @shared = shared_in(folder)
22
+ result
23
+ end
24
+ end
25
+
26
+ private
27
+
28
+ def shared_in(folder)
29
+ used = Dir.children(folder).map { |worker| File.readlines(File.join(folder, worker), chomp: true).uniq }
30
+ used.flatten.tally.select { |_, workers| workers > 1 }.keys.sort
31
+ end
32
+ end
33
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Deflake
4
- VERSION = "0.2.1"
4
+ VERSION = "0.2.2"
5
5
  end
data/lib/deflake.rb CHANGED
@@ -18,6 +18,7 @@ require_relative "deflake/execution"
18
18
  require_relative "deflake/culprits"
19
19
  require_relative "deflake/bisect"
20
20
  require_relative "deflake/adapters"
21
+ require_relative "deflake/shared_databases"
21
22
  require_relative "deflake/runner"
22
23
  require_relative "deflake/diagnosis"
23
24
  require_relative "deflake/investigation"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: deflake
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.2.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Roman Belyaev
@@ -43,6 +43,7 @@ files:
43
43
  - lib/deflake/leaks/cache.rb
44
44
  - lib/deflake/leaks/clock.rb
45
45
  - lib/deflake/leaks/current.rb
46
+ - lib/deflake/leaks/databases.rb
46
47
  - lib/deflake/leaks/env.rb
47
48
  - lib/deflake/leaks/globals.rb
48
49
  - lib/deflake/leaks/jobs.rb
@@ -56,6 +57,7 @@ files:
56
57
  - lib/deflake/rspec_formatter.rb
57
58
  - lib/deflake/run.rb
58
59
  - lib/deflake/runner.rb
60
+ - lib/deflake/shared_databases.rb
59
61
  - lib/deflake/summary.rb
60
62
  - lib/deflake/tail.rb
61
63
  - lib/deflake/text_report.rb