constable-rails 1.4.2 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +177 -1
- data/README.md +22 -3
- data/lib/constable/cli.rb +40 -2
- data/lib/constable/cold_case/rspec.rb +25 -4
- data/lib/constable/importer/modernizer.rb +52 -1
- data/lib/constable/result.rb +1 -1
- data/lib/constable/runner.rb +56 -6
- data/lib/constable/version.rb +1 -1
- data/lib/constable/worker_databases.rb +78 -0
- data/lib/generators/constable/templates/case_helper.rb.tt +2 -2
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 64e582fc3aab8b2c70309a18f5807cc1f00468ba33afa6dd08f4c78f6abde71d
|
|
4
|
+
data.tar.gz: b34caa421b4c301f5027e9d167c3e6a3ace738ef6668d5a5f81b2cf6711795b2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: efd72821dbfff16019e0eeda25eae916753ea5e708d9addec79534002b84693f8d8e0498602b5c20f6793fbd92c5ed57e68dcef9a368026b8e54c5fd567d5af4
|
|
7
|
+
data.tar.gz: bcae479351ac5a001dec12c0a5e81d337b373998bbd82d57c8588e91db47e94fda99c88abc8c581b860625cc91c44e1e69e38693fca6eb7d1671ee84bc984e1e
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,181 @@ All notable changes to this project are documented here. This project adheres to
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [2.1.0]
|
|
9
|
+
|
|
10
|
+
Both changes here come from one afternoon on a real suite: nineteen files, eight forked
|
|
11
|
+
workers, and a run that reported 83 passed and 1 failed on a unique index for a row no
|
|
12
|
+
test had made. The failing test was not the broken one. Nothing was wrong with it.
|
|
13
|
+
|
|
14
|
+
### A helper named in `.rspec` that will not load is now fatal
|
|
15
|
+
|
|
16
|
+
It used to be a warning, and the cold cases ran anyway. The reasoning written in the
|
|
17
|
+
source was that Constable's job is not to disappear. That was wrong, and it is worth being
|
|
18
|
+
precise about why, because the failure it produced is the one this project exists to
|
|
19
|
+
prevent.
|
|
20
|
+
|
|
21
|
+
`rails_helper` is not a convenience. It is where a suite registers DatabaseCleaner's
|
|
22
|
+
per-test transaction, FactoryBot, WebMock, and every `spec/support` hook. A worker that
|
|
23
|
+
fails to finish loading it does not run a slightly degraded suite — it runs an
|
|
24
|
+
**unisolated** one, against a real database, with nothing wrapping any test. The writes
|
|
25
|
+
commit. Some tests pass anyway. Later files die on unique indexes naming rows nobody can
|
|
26
|
+
account for, in a different file, with a stack trace pointing at innocent code.
|
|
27
|
+
|
|
28
|
+
What produced it: eight workers loading a `spec/support` file that did
|
|
29
|
+
|
|
30
|
+
```ruby
|
|
31
|
+
Dir.mkdir(cache_directory) unless File.directory?(cache_directory)
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Two workers check, both see nothing, both create, and the loser raises `Errno::EEXIST` —
|
|
35
|
+
while `rails_helper` is loading. Six workers were isolated. Two were not, and left
|
|
36
|
+
committed rows behind in their databases.
|
|
37
|
+
|
|
38
|
+
A green run that was never isolated is worse than no run at all, so this now raises:
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
could not load what .rspec requires (Errno::EEXIST: File exists @ dir_s_mkdir - tmp/browser_cache_all).
|
|
42
|
+
Those files set up the suite -- database cleaning, factories, spec/support hooks.
|
|
43
|
+
Running cold cases without them would not isolate them, so this is fatal rather than a warning.
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
This is a behaviour change, not a new check: the condition was always detected. If a run
|
|
47
|
+
starts failing here, it was already producing results that did not mean what they said.
|
|
48
|
+
|
|
49
|
+
### Workers are told which worker they are
|
|
50
|
+
|
|
51
|
+
`CONSTABLE_WORKER` (the index) and `CONSTABLE_WORKERS` (the count) are set in each forked
|
|
52
|
+
worker, and unset in the parent.
|
|
53
|
+
|
|
54
|
+
Anything a suite keeps on disk per process needs a name that differs per worker — a
|
|
55
|
+
browser cache, a download directory, a screenshot path, a scratch file — and until now
|
|
56
|
+
there was no way to ask, so every worker computed the same path and raced for it. That is
|
|
57
|
+
the root cause of the bug above rather than an unrelated nicety.
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
|
|
61
|
+
cache = Rails.root.join("tmp/browser_cache#{worker}")
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Deliberately not `TEST_ENV_NUMBER` or `TEST_SUBCATEGORY`: both are already interpolated
|
|
65
|
+
into some apps' `database.yml`, and setting either here would rename databases behind
|
|
66
|
+
`WorkerDatabases`' back.
|
|
67
|
+
|
|
68
|
+
### A stale worker database drops the run to serial instead of running wrong
|
|
69
|
+
|
|
70
|
+
`worker_databases: reuse` keeps the per-worker databases between runs, which is the point
|
|
71
|
+
of it — and it means they do not follow migrations on their own. Run a migration, run the
|
|
72
|
+
suite, and every worker is testing yesterday's schema.
|
|
73
|
+
|
|
74
|
+
That does not fail cleanly. It fails as a missing column in whichever file happened to
|
|
75
|
+
land on a stale worker, three files from anything you changed, on a different file each
|
|
76
|
+
run because the scheduling moved. Constable now compares what each worker database has
|
|
77
|
+
migrated against the real test database before it forks, and when they disagree:
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
⚠ worker database 3 has not run the migrations the test database has. `worker_databases:
|
|
81
|
+
reuse` keeps these between runs, which means they do not follow a migration on their
|
|
82
|
+
own. Running serially instead -- `constable prepare` rebuilds them.
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Serial rather than a refusal, deliberately: the database a serial run uses is the real
|
|
86
|
+
test database, and that one *is* current. The suite still runs, correctly, and says what
|
|
87
|
+
to do to get its speed back. Two integers per database, in the parent, before anything
|
|
88
|
+
forks; a question that cannot be answered (no `schema_migrations`, an adapter that will
|
|
89
|
+
not connect) never blocks a run that would have worked.
|
|
90
|
+
|
|
91
|
+
### Tests
|
|
92
|
+
|
|
93
|
+
The fork path had no end-to-end coverage at all, which is where most of this project's
|
|
94
|
+
serious bugs have lived. It has some now: worker identity arriving in the child and not
|
|
95
|
+
leaking into the parent, and both halves of `.rspec` handling — a helper that raises stops
|
|
96
|
+
the run, and a helper that loads is actually loaded before the file.
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
## [2.0.0]
|
|
100
|
+
|
|
101
|
+
A tidy-up release. The API you write — `investigate`, `witness`, `briefing`, `docket`,
|
|
102
|
+
`attest` — has not changed, and no case file or config key needs editing. The major
|
|
103
|
+
version is for one removed accessor and for the behaviour change in 1.4.0, both below.
|
|
104
|
+
|
|
105
|
+
### `constable modernize --cold`
|
|
106
|
+
|
|
107
|
+
A conversion that comes back flagged is **not runnable**: the flagged constructs are left
|
|
108
|
+
verbatim, so `let!` stays `let!` and the class body raises the moment it loads. That is
|
|
109
|
+
deliberate — what a `let!` should become is a decision, not a rewrite — but it leaves a
|
|
110
|
+
file stranded in `spec/` when the goal is one tree.
|
|
111
|
+
|
|
112
|
+
```console
|
|
113
|
+
$ constable modernize spec/models/tag_spec.rb --cold
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
writes `spec/models/tag_case.rb` as a cold case: one line at the top, the body byte for
|
|
117
|
+
byte, run through real RSpec. Move it into `test/cases/` and the file is ported without
|
|
118
|
+
being converted. Verified end to end — a `let!`-using spec that cannot convert runs 9/9
|
|
119
|
+
from `test/` untouched.
|
|
120
|
+
|
|
121
|
+
So a port is now: `modernize` a file, convert it if it comes back clean, `--cold` it if it
|
|
122
|
+
does not, and revisit later. Nothing has to stay behind.
|
|
123
|
+
|
|
124
|
+
### A case file that will not load says why
|
|
125
|
+
|
|
126
|
+
A half-converted case still containing `let!` reported:
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
no tests matched test/cases/tmp
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
A file that raises while loading registers no investigations, so the selection came back
|
|
133
|
+
empty and the empty-selection guard — added in 1.0.0 to catch mistyped paths — claimed
|
|
134
|
+
the path was wrong. For a file that was right there and broken. It now reports the real
|
|
135
|
+
error:
|
|
136
|
+
|
|
137
|
+
```
|
|
138
|
+
NoMethodError: undefined method `let!' for TagCase:Class
|
|
139
|
+
This file never ran. Nothing in it was tested.
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### `constable jail release --all`
|
|
143
|
+
|
|
144
|
+
Emptying the docket took one command per test, which is unusable at the scale a docket
|
|
145
|
+
reaches — before 1.4.0 a pass/fail flip jailed a test by itself, and one real suite put 29
|
|
146
|
+
on the docket from a single run.
|
|
147
|
+
|
|
148
|
+
Releasing is safe, and that is the point: a jailed test is *skipped*, so a docket full of
|
|
149
|
+
tests nobody nominated is silently reducing what the suite covers. A bare `release` with
|
|
150
|
+
no locator stays a usage error rather than an implicit "everything".
|
|
151
|
+
|
|
152
|
+
### Removed: `Result#coverage`
|
|
153
|
+
|
|
154
|
+
**Breaking, and why the major.** It was never written or read anywhere in the gem, and it
|
|
155
|
+
did not survive `to_h`/`from_h` — so a value set on it vanished crossing the pipe from a
|
|
156
|
+
parallel worker, silently, and only in parallel. An attribute that loses data is worse
|
|
157
|
+
than no attribute. Coverage has always travelled separately and still does.
|
|
158
|
+
|
|
159
|
+
### Also breaking, from 1.4.0
|
|
160
|
+
|
|
161
|
+
`jail_flakes` defaults to **false**: a test that passed last run and failed this one is no
|
|
162
|
+
longer jailed automatically. Jailing skips the test on every later run, and doing that to
|
|
163
|
+
a test nobody nominated is how a suite quietly stops testing things. `--jail` is
|
|
164
|
+
unaffected.
|
|
165
|
+
|
|
166
|
+
### Test coverage
|
|
167
|
+
|
|
168
|
+
The suite is now **1,025 runs**, up from 738 at 1.0.0. This release adds permanent tests
|
|
169
|
+
for the three areas that had none and carried the most risk:
|
|
170
|
+
|
|
171
|
+
- **`Result`** — the wire format between a worker and the parent. Every attribute is
|
|
172
|
+
asserted to survive a real `Marshal` round trip, because anything that does not is data
|
|
173
|
+
lost only in parallel runs, which is the worst way to find out. That is how the
|
|
174
|
+
`coverage` bug above was found.
|
|
175
|
+
- **Cold-case shapes** — 17 real-world files run end to end through both engines: pending
|
|
176
|
+
and `xit`, `before`/`around` hooks, tags, aggregate failures, a raising hook, shared
|
|
177
|
+
examples, `RSpec.configure` inside a spec, deep nesting, the same file twice, and seven
|
|
178
|
+
Minitest arrangements. Every serious bug this project has had lived in these adapters.
|
|
179
|
+
- **`Identity`** — stability under reformatting and comments, distinctness on real change,
|
|
180
|
+
unicode, cold-case keys, and the disambiguation added in 1.0.0.
|
|
181
|
+
|
|
182
|
+
|
|
8
183
|
## [1.4.2]
|
|
9
184
|
|
|
10
185
|
### Cold cases run inside a parallel worker
|
|
@@ -745,7 +920,8 @@ Initial release.
|
|
|
745
920
|
- Diff-based coverage gate — only lines changed in the current diff are held to the
|
|
746
921
|
threshold. `constable beat` for the full picture, `--html` for a browsable report.
|
|
747
922
|
|
|
748
|
-
[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.4.
|
|
923
|
+
[Unreleased]: https://github.com/Ray-Hughes/constable/compare/v1.4.3...HEAD
|
|
924
|
+
[1.4.3]: https://github.com/Ray-Hughes/constable/compare/v1.4.2...v1.4.3
|
|
749
925
|
[1.4.2]: https://github.com/Ray-Hughes/constable/compare/v1.4.1...v1.4.2
|
|
750
926
|
[1.4.1]: https://github.com/Ray-Hughes/constable/compare/v1.4.0...v1.4.1
|
|
751
927
|
[1.4.0]: https://github.com/Ray-Hughes/constable/compare/v1.3.3...v1.4.0
|
data/README.md
CHANGED
|
@@ -393,7 +393,7 @@ worse than one that resets.
|
|
|
393
393
|
| `constable test PATH[:LINE]` | One file, or one investigation at that line |
|
|
394
394
|
| `constable test --unsafe` | Cold cases only |
|
|
395
395
|
| `constable test --jail` | The full run, in jail mode |
|
|
396
|
-
| `constable jail [run\|parole\|release]` | The docket |
|
|
396
|
+
| `constable jail [run\|parole\|release]` | The docket. `release --all` empties it |
|
|
397
397
|
| `constable warrants [release]` | Outstanding warrants |
|
|
398
398
|
| `constable watchlist` | Everything under supervision right now |
|
|
399
399
|
| `constable status` | How the suite is doing over time |
|
|
@@ -402,7 +402,7 @@ worse than one that resets.
|
|
|
402
402
|
| `constable prepare [--workers N]` | Build the per-worker test databases `worker_databases: reuse` needs |
|
|
403
403
|
| `constable prune [--dry-run]` | Forget docket rows and warrants for tests that no longer exist |
|
|
404
404
|
| `constable import --from=rspec` | Adopt an existing suite as cold cases |
|
|
405
|
-
| `constable modernize PATH` | Opt-in AST rewrite into the native DSL |
|
|
405
|
+
| `constable modernize PATH [--cold]` | Opt-in AST rewrite into the native DSL. `--cold` moves it verbatim instead |
|
|
406
406
|
|
|
407
407
|
Flags: `--full --unsafe --jail --warrants --coverage --seed N --workers N --verbose --tier T\n--expanded --concise --output MODE --no-color`.
|
|
408
408
|
|
|
@@ -414,12 +414,31 @@ Each worker gets **its own database**, built from schema the way `rails test` do
|
|
|
414
414
|
or kept between runs, with `worker_databases: reuse`, which is both faster and the only
|
|
415
415
|
thing that works for an app whose schema cannot rebuild the database by itself (any app
|
|
416
416
|
with Postgres custom types: `CREATE TYPE` has no `schema.rb` representation). Prepare
|
|
417
|
-
those once with `constable prepare
|
|
417
|
+
those once with `constable prepare`, and again after a migration — kept databases do not
|
|
418
|
+
follow one on their own. Forgetting is caught rather than suffered: Constable compares
|
|
419
|
+
what each worker database has migrated against the real test database before it forks, and
|
|
420
|
+
runs serially (which uses the real one, so it is correct) rather than testing yesterday's
|
|
421
|
+
schema.
|
|
418
422
|
Sharing one would not be a speed/safety trade but a correctness bug: on SQLite the run
|
|
419
423
|
dissolves into `database is locked`, and on a client/server database tests quietly see
|
|
420
424
|
each other's rows. If your app has ActiveRecord but cannot shard, Constable runs serially
|
|
421
425
|
and says why — slow is a trade-off, wrong is not.
|
|
422
426
|
|
|
427
|
+
The database is not the only thing a worker needs to itself. Anything your suite keeps on
|
|
428
|
+
disk per process — a browser cache, a download directory, a screenshot path — needs a name
|
|
429
|
+
that differs per worker, or they race for it. Each worker is told which one it is:
|
|
430
|
+
|
|
431
|
+
```ruby
|
|
432
|
+
worker = ENV["CONSTABLE_WORKER"] ? "_w#{ENV['CONSTABLE_WORKER']}" : ""
|
|
433
|
+
cache = Rails.root.join("tmp/browser_cache#{worker}")
|
|
434
|
+
```
|
|
435
|
+
|
|
436
|
+
`CONSTABLE_WORKER` is the index and `CONSTABLE_WORKERS` the count; both are unset in the
|
|
437
|
+
parent, so serial runs keep whatever name they had. Use `FileUtils.mkdir_p` rather than
|
|
438
|
+
`Dir.mkdir ... unless File.directory?` while you are there — the second is a race, and if
|
|
439
|
+
it runs inside `spec/support` it takes `rails_helper` down with it, which costs the loser
|
|
440
|
+
its database cleaning rather than just its cache directory.
|
|
441
|
+
|
|
423
442
|
### Output
|
|
424
443
|
|
|
425
444
|
stdout is reserved for results — not just Constable's own output, but the app's. Rails
|
data/lib/constable/cli.rb
CHANGED
|
@@ -197,6 +197,8 @@ module Constable
|
|
|
197
197
|
DESC
|
|
198
198
|
option :alongside, type: :boolean, default: false, desc: "Write PATH_case.rb beside the original"
|
|
199
199
|
option :"in-place", type: :boolean, default: false, desc: "Overwrite the file"
|
|
200
|
+
option :cold, type: :boolean, default: false,
|
|
201
|
+
desc: "Move it verbatim as a cold case instead of converting"
|
|
200
202
|
option :"show-source", type: :boolean, default: false, desc: "Print the rewritten source"
|
|
201
203
|
def modernize(*paths)
|
|
202
204
|
if paths.empty?
|
|
@@ -205,6 +207,7 @@ module Constable
|
|
|
205
207
|
end
|
|
206
208
|
|
|
207
209
|
mode = if options[:"in-place"] then :in_place
|
|
210
|
+
elsif options[:cold] then :cold
|
|
208
211
|
elsif options[:alongside] then :alongside
|
|
209
212
|
else :none
|
|
210
213
|
end
|
|
@@ -312,13 +315,48 @@ module Constable
|
|
|
312
315
|
say "Paroled. It runs normally now; one failure sends it straight back."
|
|
313
316
|
end
|
|
314
317
|
|
|
315
|
-
desc "release PATH:LINE", "Fully release a test, no supervision"
|
|
316
|
-
|
|
318
|
+
desc "release [PATH:LINE]", "Fully release a test, no supervision"
|
|
319
|
+
long_desc <<~DESC
|
|
320
|
+
Takes one test off the docket. `--all` empties it.
|
|
321
|
+
|
|
322
|
+
Emptying is the right move after a docket that filled up on its own -- before
|
|
323
|
+
1.4.0 a pass/fail flip jailed a test automatically, and a suite with
|
|
324
|
+
order-dependent tests could put dozens on it that nobody chose. Releasing is safe:
|
|
325
|
+
anything genuinely broken fails again on the next run, in the open.
|
|
326
|
+
DESC
|
|
327
|
+
option :all, type: :boolean, default: false, desc: "Release every test on the docket"
|
|
328
|
+
def release(locator = nil)
|
|
329
|
+
return release_all if options[:all]
|
|
330
|
+
|
|
331
|
+
if locator.nil?
|
|
332
|
+
CLI.complain("release needs a PATH:LINE, or --all to empty the docket")
|
|
333
|
+
exit(EXIT_USAGE)
|
|
334
|
+
end
|
|
335
|
+
|
|
317
336
|
act(locator) { |jail, identity| jail.release(identity) }
|
|
318
337
|
say "Released."
|
|
319
338
|
end
|
|
320
339
|
|
|
321
340
|
no_commands do
|
|
341
|
+
# Emptying the docket in one go. Right after a docket that filled up on its own --
|
|
342
|
+
# before 1.4.0 a pass/fail flip jailed a test automatically, and a suite with
|
|
343
|
+
# order-dependent tests could put dozens on it that nobody chose. Releasing is
|
|
344
|
+
# safe: anything genuinely broken fails again on the next run, in the open.
|
|
345
|
+
def release_all
|
|
346
|
+
jail = Jail.new(config: Constable.config, storage: Constable.storage)
|
|
347
|
+
entries = jail.entries
|
|
348
|
+
|
|
349
|
+
if entries.empty?
|
|
350
|
+
say "The docket is already empty."
|
|
351
|
+
return 0
|
|
352
|
+
end
|
|
353
|
+
|
|
354
|
+
entries.each { |entry| jail.release(entry.identity) }
|
|
355
|
+
say "Released #{entries.size} #{entries.size == 1 ? "test" : "tests"}. The docket is empty."
|
|
356
|
+
say "Anything genuinely broken will fail on the next run, where you can see it."
|
|
357
|
+
0
|
|
358
|
+
end
|
|
359
|
+
|
|
322
360
|
def act(locator)
|
|
323
361
|
jail = Jail.new(config: Constable.config, storage: Constable.storage)
|
|
324
362
|
refuse_ambiguous(locator, jail.candidates(locator), "on the docket")
|
|
@@ -324,10 +324,31 @@ module Constable
|
|
|
324
324
|
rescue Constable::Error
|
|
325
325
|
raise
|
|
326
326
|
rescue StandardError => e
|
|
327
|
-
#
|
|
328
|
-
#
|
|
329
|
-
|
|
330
|
-
|
|
327
|
+
# Fatal, deliberately.
|
|
328
|
+
#
|
|
329
|
+
# This used to be a warning that ran the cold cases anyway, on the reasoning that
|
|
330
|
+
# Constable's job is not to disappear. That reasoning was wrong, and the way it
|
|
331
|
+
# was wrong is the exact failure this gem exists to prevent.
|
|
332
|
+
#
|
|
333
|
+
# A `rails_helper` that does not finish loading is not a suite missing one
|
|
334
|
+
# convenience. It is a suite missing its isolation: DatabaseCleaner's per-test
|
|
335
|
+
# transaction, FactoryBot, WebMock, every `spec/support` hook. The tests then run
|
|
336
|
+
# against a real database with nothing wrapping them, so their writes commit.
|
|
337
|
+
# Some pass. Others fail later on a unique index, in a different file, naming a
|
|
338
|
+
# row that a test three files ago was supposed to have rolled back.
|
|
339
|
+
#
|
|
340
|
+
# Observed exactly that: eight forked workers loading a helper that did
|
|
341
|
+
# `Dir.mkdir` on a shared tmp directory it had just checked for. The workers that
|
|
342
|
+
# lost the race raised Errno::EEXIST, took this branch, ran their files with no
|
|
343
|
+
# DatabaseCleaner at all, and left committed rows behind in their databases.
|
|
344
|
+
#
|
|
345
|
+
# A green run that was never isolated is worse than no run. Refuse it.
|
|
346
|
+
raise Constable::Error,
|
|
347
|
+
"could not load what .rspec requires (#{e.class}: #{e.message}).\n " \
|
|
348
|
+
"Those files set up the suite -- database cleaning, factories, " \
|
|
349
|
+
"spec/support hooks.\n " \
|
|
350
|
+
"Running cold cases without them would not isolate them, so this is " \
|
|
351
|
+
"fatal rather than a warning."
|
|
331
352
|
end
|
|
332
353
|
|
|
333
354
|
# What `configuration.requires=` does before requiring anything: puts `lib` and the
|
|
@@ -28,7 +28,7 @@ module Constable
|
|
|
28
28
|
# :none -- dry run. Report only. The default.
|
|
29
29
|
# :alongside -- write foo_spec.rb's conversion to foo_case.rb, never clobbering.
|
|
30
30
|
# :in_place -- overwrite the original file.
|
|
31
|
-
WRITE_MODES = %i[none alongside in_place].freeze
|
|
31
|
+
WRITE_MODES = %i[none alongside in_place cold].freeze
|
|
32
32
|
|
|
33
33
|
GROUP_METHODS = %i[describe context xdescribe xcontext fdescribe fcontext feature].freeze
|
|
34
34
|
EXAMPLE_METHODS = %i[it specify example scenario].freeze
|
|
@@ -175,9 +175,60 @@ module Constable
|
|
|
175
175
|
File.join(dir, "#{base}_case.rb")
|
|
176
176
|
end
|
|
177
177
|
|
|
178
|
+
# `--cold`: move the file into the native tree without converting a line of it.
|
|
179
|
+
#
|
|
180
|
+
# A conversion that comes back flagged is not runnable -- the flagged constructs
|
|
181
|
+
# are left verbatim, so `let!` stays `let!` and the class body raises the moment it
|
|
182
|
+
# loads. That is deliberate: what a `let!` should become is a decision, not a
|
|
183
|
+
# rewrite. But it leaves a file stuck in spec/ when the goal is one tree.
|
|
184
|
+
#
|
|
185
|
+
# A cold case is the answer that already exists: one line at the top, the body
|
|
186
|
+
# untouched, run through real RSpec, results folded into the same report. This
|
|
187
|
+
# writes exactly that, so a port can move every file and convert the ones worth
|
|
188
|
+
# converting on its own schedule.
|
|
189
|
+
def cold_wrap(result, root)
|
|
190
|
+
target = alongside_path(result.path)
|
|
191
|
+
if File.exist?(target)
|
|
192
|
+
result.error = "refusing to overwrite #{target.delete_prefix("#{root}/")}"
|
|
193
|
+
return
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
FileUtils.mkdir_p(File.dirname(target))
|
|
197
|
+
File.write(target, cold_source(result))
|
|
198
|
+
result.written_to = target.delete_prefix("#{root}/")
|
|
199
|
+
end
|
|
200
|
+
|
|
201
|
+
# The original bytes, between a header line and a final `end`. Nothing inside is
|
|
202
|
+
# parsed, reindented or touched -- that is the whole promise of a cold case.
|
|
203
|
+
def cold_source(result)
|
|
204
|
+
<<~RUBY
|
|
205
|
+
# frozen_string_literal: true
|
|
206
|
+
|
|
207
|
+
# Moved verbatim from #{result.relative_path}. Runs through real RSpec, with its
|
|
208
|
+
# results folded into Constable's reporting, flake history and CI gate.
|
|
209
|
+
#
|
|
210
|
+
# Nothing inside has been converted, so every RSpec feature still works --
|
|
211
|
+
# `let!`, `before(:all)`, shared examples, rspec-mocks. Convert it with
|
|
212
|
+
# `constable modernize` when it is worth doing; there is no deadline.
|
|
213
|
+
class #{cold_class_name(result)} < Constable::ColdCase::RSpec
|
|
214
|
+
#{indent(result.original.to_s.rstrip)}
|
|
215
|
+
end
|
|
216
|
+
RUBY
|
|
217
|
+
end
|
|
218
|
+
|
|
219
|
+
def cold_class_name(result)
|
|
220
|
+
base = File.basename(result.path, ".rb").sub(/_(?:spec|test)\z/, "")
|
|
221
|
+
"Legacy#{base.split(%r{[_/]}).map(&:capitalize).join}Spec"
|
|
222
|
+
end
|
|
223
|
+
|
|
224
|
+
def indent(source)
|
|
225
|
+
source.lines.map { |line| line.strip.empty? ? line : " #{line}" }.join
|
|
226
|
+
end
|
|
227
|
+
|
|
178
228
|
private
|
|
179
229
|
|
|
180
230
|
def persist(result, root, write)
|
|
231
|
+
return cold_wrap(result, root) if write == :cold
|
|
181
232
|
return unless result.ok? && result.changed?
|
|
182
233
|
|
|
183
234
|
case write
|
data/lib/constable/result.rb
CHANGED
|
@@ -18,7 +18,7 @@ module Constable
|
|
|
18
18
|
|
|
19
19
|
attr_reader :identity, :case_name, :description, :file, :line, :kind, :tier
|
|
20
20
|
attr_accessor :status, :duration, :failure, :warnings, :retries, :jail_reason,
|
|
21
|
-
:parole_day, :times_jailed, :seed
|
|
21
|
+
:parole_day, :times_jailed, :seed
|
|
22
22
|
|
|
23
23
|
def initialize(identity:, case_name:, description:, file:, line:, kind: :native, tier: nil,
|
|
24
24
|
status: :passed, duration: 0.0, failure: nil, warnings: [], retries: [])
|
data/lib/constable/runner.rb
CHANGED
|
@@ -231,6 +231,12 @@ module Constable
|
|
|
231
231
|
def refuse_empty_selection!(items)
|
|
232
232
|
return unless items.empty?
|
|
233
233
|
return unless @selection.explicit?
|
|
234
|
+
# A file that raised while loading registers no investigations, so the selection
|
|
235
|
+
# comes back empty and this guard would report "no tests matched" -- burying the
|
|
236
|
+
# actual error, which is already captured and about to be reported as a failure.
|
|
237
|
+
# "Your path is wrong" is the wrong thing to say about a file that is right there
|
|
238
|
+
# and broken.
|
|
239
|
+
return if load_errors.any?
|
|
234
240
|
|
|
235
241
|
raise Constable::Error, @selection.empty_selection_message
|
|
236
242
|
end
|
|
@@ -283,7 +289,7 @@ module Constable
|
|
|
283
289
|
return [] if items.empty?
|
|
284
290
|
|
|
285
291
|
count = worker_count(items)
|
|
286
|
-
if count > 1 && forkable? && parallel_safe?
|
|
292
|
+
if count > 1 && forkable? && parallel_safe?(count)
|
|
287
293
|
run_parallel(items, count)
|
|
288
294
|
else
|
|
289
295
|
run_serial(items)
|
|
@@ -296,17 +302,45 @@ module Constable
|
|
|
296
302
|
# which is worse. An app with no ActiveRecord has nothing to shard and is always safe.
|
|
297
303
|
#
|
|
298
304
|
# When we cannot shard, we run serially and say why. Slow is a trade-off; wrong is not.
|
|
299
|
-
def parallel_safe?
|
|
305
|
+
def parallel_safe?(count)
|
|
300
306
|
# An explicit opt-out. No attempt, and no warning about one -- the user has already
|
|
301
307
|
# told us they know.
|
|
302
308
|
return false if @config.worker_databases == :off
|
|
303
309
|
return true unless WorkerDatabases.active_record?
|
|
304
|
-
|
|
310
|
+
|
|
311
|
+
unless WorkerDatabases.shardable?
|
|
312
|
+
Constable.warn!(
|
|
313
|
+
"parallel workers need one database per worker, and this app's ActiveRecord " \
|
|
314
|
+
"cannot provide them (active_record/test_databases did not load). Running " \
|
|
315
|
+
"serially instead -- pass --workers N once that is available.",
|
|
316
|
+
kind: :parallel
|
|
317
|
+
)
|
|
318
|
+
return false
|
|
319
|
+
end
|
|
320
|
+
|
|
321
|
+
worker_databases_current?(count)
|
|
322
|
+
end
|
|
323
|
+
|
|
324
|
+
# `:reuse` keeps the per-worker databases between runs, so they do not follow
|
|
325
|
+
# migrations by themselves. A run against stale copies does not fail cleanly -- it
|
|
326
|
+
# fails as a missing column in whichever file happened to touch it, on a different
|
|
327
|
+
# worker each run.
|
|
328
|
+
#
|
|
329
|
+
# Serial is the right fallback rather than a refusal, because the database a serial
|
|
330
|
+
# run uses is the real test database, and that one *is* current. So the suite still
|
|
331
|
+
# runs, correctly, and says exactly what to do to get its speed back.
|
|
332
|
+
def worker_databases_current?(count)
|
|
333
|
+
return true unless @config.worker_databases.to_s == "reuse"
|
|
334
|
+
|
|
335
|
+
stale = WorkerDatabases.stale_workers(count)
|
|
336
|
+
return true if stale.empty?
|
|
305
337
|
|
|
306
338
|
Constable.warn!(
|
|
307
|
-
"
|
|
308
|
-
"
|
|
309
|
-
"
|
|
339
|
+
"worker database#{"s" if stale.length > 1} #{stale.join(", ")} " \
|
|
340
|
+
"#{stale.length > 1 ? "have" : "has"} not run the migrations the test database " \
|
|
341
|
+
"has. `worker_databases: reuse` keeps these between runs, which means they do " \
|
|
342
|
+
"not follow a migration on their own. Running serially instead -- " \
|
|
343
|
+
"`constable prepare` rebuilds them.",
|
|
310
344
|
kind: :parallel
|
|
311
345
|
)
|
|
312
346
|
false
|
|
@@ -353,6 +387,22 @@ module Constable
|
|
|
353
387
|
pid = fork do
|
|
354
388
|
reader.close
|
|
355
389
|
|
|
390
|
+
# Tell the app which worker it is.
|
|
391
|
+
#
|
|
392
|
+
# Anything a suite keeps on disk per process needs this: a browser cache, a
|
|
393
|
+
# download directory, a screenshot path, a scratch file. Without it every worker
|
|
394
|
+
# computes the same path and they race -- and the way that surfaces is not a
|
|
395
|
+
# tidy error. Observed on a real suite: eight workers running
|
|
396
|
+
# `Dir.mkdir(dir) unless File.directory?(dir)` in a spec/support file, the losers
|
|
397
|
+
# raising Errno::EEXIST *while loading rails_helper*, so those workers ran their
|
|
398
|
+
# files with no database cleaning at all.
|
|
399
|
+
#
|
|
400
|
+
# Deliberately its own name rather than TEST_ENV_NUMBER or parallel_tests'
|
|
401
|
+
# TEST_SUBCATEGORY: those are already wired into some apps' database.yml, and
|
|
402
|
+
# setting one here would rename databases behind WorkerDatabases' back.
|
|
403
|
+
ENV["CONSTABLE_WORKER"] = worker_index.to_s
|
|
404
|
+
ENV["CONSTABLE_WORKERS"] = buckets.length.to_s
|
|
405
|
+
|
|
356
406
|
# Before a single test runs: build this worker's own database and point the
|
|
357
407
|
# process at it. Never falls back to the shared one -- that is the bug this
|
|
358
408
|
# exists to prevent -- but the failure is reported home rather than raised.
|
data/lib/constable/version.rb
CHANGED
|
@@ -93,6 +93,84 @@ module Constable
|
|
|
93
93
|
raise Constable::Error, "could not prepare worker #{index}: #{e.class}: #{e.message}"
|
|
94
94
|
end
|
|
95
95
|
|
|
96
|
+
# The one mistake `:reuse` invites, caught before the fork rather than after.
|
|
97
|
+
#
|
|
98
|
+
# `:reuse` keeps the per-worker databases between runs, which is the whole point --
|
|
99
|
+
# and it means they do not follow migrations by themselves. Run a migration, run the
|
|
100
|
+
# suite, and every worker is now testing yesterday's schema. That does not fail
|
|
101
|
+
# cleanly: it fails as a missing column in whichever file happened to touch it, three
|
|
102
|
+
# files away from anything you changed, differently on each run because the file went
|
|
103
|
+
# to a different worker. Exactly the shape of bug that costs an afternoon.
|
|
104
|
+
#
|
|
105
|
+
# So compare what each worker database has migrated against what the real test
|
|
106
|
+
# database has, and refuse rather than guess. Two integers per database, in the parent,
|
|
107
|
+
# before anything forks.
|
|
108
|
+
#
|
|
109
|
+
# Returns the worker indexes that are out of date, empty when they are all current or
|
|
110
|
+
# when the question cannot be answered (no schema_migrations table, an adapter that
|
|
111
|
+
# will not connect) -- an unanswerable check must not block a run that would have
|
|
112
|
+
# worked.
|
|
113
|
+
def stale_workers(count)
|
|
114
|
+
return [] unless shardable?
|
|
115
|
+
|
|
116
|
+
expected = {}
|
|
117
|
+
original = database_names
|
|
118
|
+
begin
|
|
119
|
+
each_source_config do |db_config|
|
|
120
|
+
fingerprint = schema_fingerprint(db_config)
|
|
121
|
+
expected[db_config.database.to_s] = fingerprint if fingerprint
|
|
122
|
+
end
|
|
123
|
+
return [] if expected.empty?
|
|
124
|
+
|
|
125
|
+
stale = []
|
|
126
|
+
(0...count).each do |index|
|
|
127
|
+
names = database_names
|
|
128
|
+
begin
|
|
129
|
+
each_worker_config(index) do |db_config|
|
|
130
|
+
source = db_config.database.to_s.sub(/_#{index}\z/, "")
|
|
131
|
+
next unless expected.key?(source)
|
|
132
|
+
|
|
133
|
+
actual = schema_fingerprint(db_config)
|
|
134
|
+
stale << index if actual && actual != expected[source]
|
|
135
|
+
end
|
|
136
|
+
ensure
|
|
137
|
+
restore_database_names(names)
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
stale.uniq
|
|
141
|
+
ensure
|
|
142
|
+
restore_database_names(original)
|
|
143
|
+
::ActiveRecord::Base.establish_connection
|
|
144
|
+
end
|
|
145
|
+
rescue StandardError
|
|
146
|
+
[]
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
# What a database has migrated: how many migrations it has run and the latest one.
|
|
150
|
+
# Cheaper than diffing every version, and a worker that missed a migration differs in
|
|
151
|
+
# both. nil when the question does not apply.
|
|
152
|
+
def schema_fingerprint(db_config)
|
|
153
|
+
::ActiveRecord::Base.establish_connection(db_config)
|
|
154
|
+
connection = ::ActiveRecord::Base.connection
|
|
155
|
+
return nil unless connection.table_exists?("schema_migrations")
|
|
156
|
+
|
|
157
|
+
connection.select_rows("SELECT COUNT(*), MAX(version) FROM schema_migrations").first
|
|
158
|
+
rescue StandardError
|
|
159
|
+
nil
|
|
160
|
+
end
|
|
161
|
+
|
|
162
|
+
# The same configs `each_worker_config` renames, left under their real names.
|
|
163
|
+
def each_source_config
|
|
164
|
+
::ActiveRecord::Base.configurations
|
|
165
|
+
.configs_for(env_name: env_name, include_hidden: true)
|
|
166
|
+
.each do |db_config|
|
|
167
|
+
next unless db_config.database_tasks?
|
|
168
|
+
next unless shardable_adapter?(db_config)
|
|
169
|
+
|
|
170
|
+
yield db_config
|
|
171
|
+
end
|
|
172
|
+
end
|
|
173
|
+
|
|
96
174
|
def database_names
|
|
97
175
|
::ActiveRecord::Base.configurations
|
|
98
176
|
.configs_for(env_name: env_name, include_hidden: true)
|
|
@@ -34,8 +34,8 @@ abort("The Rails environment is running in production mode!") if Rails.env.produ
|
|
|
34
34
|
if defined?(ActiveRecord::Migration)
|
|
35
35
|
begin
|
|
36
36
|
ActiveRecord::Migration.maintain_test_schema!
|
|
37
|
-
rescue ActiveRecord::PendingMigrationError =>
|
|
38
|
-
abort("#{
|
|
37
|
+
rescue ActiveRecord::PendingMigrationError => error
|
|
38
|
+
abort("#{error.message}\n\nRun: bin/rails db:migrate RAILS_ENV=test")
|
|
39
39
|
end
|
|
40
40
|
end
|
|
41
41
|
|