suture 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.codeclimate.yml +28 -0
- data/.gitignore +1 -0
- data/.rubocop.yml +1156 -0
- data/.travis.yml +4 -0
- data/README.md +143 -9
- data/Rakefile +22 -1
- data/lib/suture/adapter/dictaphone.rb +41 -23
- data/lib/suture/adapter/log.rb +31 -0
- data/lib/suture/builds_plan.rb +2 -2
- data/lib/suture/chooses_surgeon.rb +10 -0
- data/lib/suture/comparator.rb +7 -0
- data/lib/suture/error/observation_conflict.rb +26 -13
- data/lib/suture/error/verification_failed.rb +160 -5
- data/lib/suture/interprets_results.rb +2 -2
- data/lib/suture/prescribes_test_plan.rb +4 -4
- data/lib/suture/surgeon/observer.rb +11 -1
- data/lib/suture/tests_patient.rb +31 -7
- data/lib/suture/util/env.rb +9 -4
- data/lib/suture/util/shuffle.rb +13 -0
- data/lib/suture/util/timer.rb +14 -0
- data/lib/suture/value/plan.rb +2 -1
- data/lib/suture/value/test_plan.rb +35 -6
- data/lib/suture/value/test_results.rb +5 -1
- data/lib/suture/version.rb +1 -1
- data/lib/suture/wrap/logger.rb +40 -0
- data/lib/suture/wrap/sqlite.rb +8 -1
- data/lib/suture.rb +24 -2
- data/suture.gemspec +8 -1
- metadata +65 -2
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# Suture
|
2
2
|
|
3
|
-
[](https://travis-ci.org/testdouble/suture)
|
3
|
+
[](https://travis-ci.org/testdouble/suture) [](https://codeclimate.com/github/testdouble/suture) [](https://codeclimate.com/github/testdouble/suture/coverage)
|
4
4
|
|
5
5
|
A refactoring tool for Ruby, designed to make it safe to change code you don't
|
6
6
|
confidently understand. In fact, changing untrustworthy code is so fraught,
|
@@ -97,7 +97,7 @@ implementations behave the same way.
|
|
97
97
|
First, we tell Suture to start recording calls by setting the environment
|
98
98
|
variable `SUTURE_RECORD_CALLS` to something truthy (e.g.
|
99
99
|
`SUTURE_RECORD_CALLS=true bundle exec rails s`). So long as this variable is set,
|
100
|
-
any calls to our
|
100
|
+
any calls to our seam will record the arguments passed to the legacy code path
|
101
101
|
and the return value.
|
102
102
|
|
103
103
|
As you use the application (whether it's a queue system, a web app, or a CLI),
|
@@ -304,8 +304,29 @@ you're yet unsure that your refactor or reimplementation is complete.
|
|
304
304
|
|
305
305
|
While your application's logs aren't affected by Suture, it may be helpful for
|
306
306
|
Suture to maintain a separate log file for any errors that are raised by the
|
307
|
-
refactored code path.
|
308
|
-
|
307
|
+
refactored code path.
|
308
|
+
|
309
|
+
Suture has a handful of process-wide logging settings that can be set at any
|
310
|
+
point as your app starts up (if you're using Rails, then your
|
311
|
+
environment-specific (e.g. `config/environments/production.rb`) config file
|
312
|
+
is a good choice).
|
313
|
+
|
314
|
+
``` ruby
|
315
|
+
Suture.config({
|
316
|
+
:log_level => "WARN", #<-- defaults to "INFO"
|
317
|
+
:log_stdout => false, #<-- defaults to true
|
318
|
+
:log_file => "log/suture.log" #<-- defaults to nil
|
319
|
+
})
|
320
|
+
```
|
321
|
+
|
322
|
+
When your new code path raises an error with the above settings, it will
|
323
|
+
propogate and log the error to the specified file.
|
324
|
+
|
325
|
+
### Custom error handlers
|
326
|
+
|
327
|
+
Additionally, you may have some idea of what you want to do (i.e. phone home to
|
328
|
+
a reporting service) in the event that your new code path fails. To add custom
|
329
|
+
error handling before, set the `:on_error` option to a callable.
|
309
330
|
|
310
331
|
``` ruby
|
311
332
|
class MyWorker
|
@@ -314,7 +335,7 @@ class MyWorker
|
|
314
335
|
old: LegacyWorker.new,
|
315
336
|
new: NewWorker.new,
|
316
337
|
args: [id],
|
317
|
-
|
338
|
+
on_error: -> (name, args) { PhonesHome.new.phone(name, args) }
|
318
339
|
}))
|
319
340
|
end
|
320
341
|
end
|
@@ -323,9 +344,9 @@ end
|
|
323
344
|
### Retrying failures
|
324
345
|
|
325
346
|
Since the legacy code path hasn't been deleted yet, there's no reason to leave
|
326
|
-
users hanging if the new code path explodes. By setting the `:
|
327
|
-
`true`, Suture will rescue any errors raised from the new code path and
|
328
|
-
to invoke the legacy code path instead.
|
347
|
+
users hanging if the new code path explodes. By setting the `:fallback_to_old`
|
348
|
+
entry to `true`, Suture will rescue any errors raised from the new code path and
|
349
|
+
attempt to invoke the legacy code path instead.
|
329
350
|
|
330
351
|
``` ruby
|
331
352
|
class MyWorker
|
@@ -334,7 +355,7 @@ class MyWorker
|
|
334
355
|
old: LegacyWorker.new,
|
335
356
|
new: NewWorker.new,
|
336
357
|
args: [id],
|
337
|
-
|
358
|
+
fallback_to_old: true
|
338
359
|
}))
|
339
360
|
end
|
340
361
|
end
|
@@ -344,3 +365,116 @@ Since this approach rescues errors, it's possible that errors in the new code
|
|
344
365
|
path will go unnoticed, so it's best used in conjunction with Suture's logging
|
345
366
|
feature. Before ultimately deciding to finally delete the legacy code path,
|
346
367
|
double-check that the logs aren't full of rescued errors!
|
368
|
+
|
369
|
+
## Configuration
|
370
|
+
|
371
|
+
Legacy code is, necessarily, complex and hard-to-wrangle. That's why Suture comes
|
372
|
+
with a bunch of configuration options to modify its behavior, particularly for
|
373
|
+
hard-to-compare objects.
|
374
|
+
|
375
|
+
### Setting configuration options
|
376
|
+
|
377
|
+
In general, most configuration options can be set in several places:
|
378
|
+
|
379
|
+
* Globally, via an environment variable. The flag `record_calls` will translate
|
380
|
+
to an expected ENV var named `SUTURE_RECORD_CALLS` and can be set from the
|
381
|
+
command line like so: `SUTURE_RECORD_CALLS=true bundle exec rails server`, to
|
382
|
+
tell Suture to record all your interactions with your seams without touching the
|
383
|
+
source code.
|
384
|
+
|
385
|
+
* Globally, via the top-level `Suture.config` method. Most variables can be set
|
386
|
+
via this top-level configuration, like
|
387
|
+
`Suture.config(:database_path => 'my.db')`. Once set, this will apply to all your
|
388
|
+
interactions with Suture for the life of the process until you call
|
389
|
+
`Suture.reset!`.
|
390
|
+
|
391
|
+
* At a `Suture.create` or `Suture.verify` call-site as part of its options hash.
|
392
|
+
If you have several seams, you'll probably want to set most options locally
|
393
|
+
where you call Suture, like `Suture.create(:foo, { :comparator => my_thing })`
|
394
|
+
|
395
|
+
### Supported options
|
396
|
+
|
397
|
+
#### Suture.create
|
398
|
+
|
399
|
+
TODO
|
400
|
+
|
401
|
+
#### Suture.verify
|
402
|
+
|
403
|
+
TODO
|
404
|
+
|
405
|
+
### Creating a custom comparator
|
406
|
+
|
407
|
+
Out-of-the-box, Suture will do its best to compare your recorded & actual results
|
408
|
+
to ensure that things are equivalent to one another, but reality is often less
|
409
|
+
tidy than a gem can predict up-front. When the built-in equivalency comparator
|
410
|
+
fails you, you can define a custom one—globally or at each `Suture.create` or
|
411
|
+
`Suture.verify` call-site.
|
412
|
+
|
413
|
+
#### Extending the built-in comparator class
|
414
|
+
|
415
|
+
If you have a bunch of value types that require special equivalency checks, it
|
416
|
+
makes sense to invest the time to extend built-in one:
|
417
|
+
|
418
|
+
``` ruby
|
419
|
+
class MyComparator < Suture::Comparator
|
420
|
+
def call(recorded, actual)
|
421
|
+
if recorded.kind_of?(MyType)
|
422
|
+
recorded.data_stuff == actual.data_stuff
|
423
|
+
else
|
424
|
+
super
|
425
|
+
end
|
426
|
+
end
|
427
|
+
end
|
428
|
+
```
|
429
|
+
|
430
|
+
So long as you return `super` for non-special cases, it should be safe to set an
|
431
|
+
instance of your custom comparator globally for the life of the process with:
|
432
|
+
|
433
|
+
``` ruby
|
434
|
+
Suture.config({
|
435
|
+
:comparator => MyComparator.new
|
436
|
+
})
|
437
|
+
```
|
438
|
+
|
439
|
+
#### Creating a one-off comparator
|
440
|
+
|
441
|
+
If a particular seam requires a custom comparator and will always return
|
442
|
+
sufficiently homogeneous types, it may be good enough to set a custom comparator
|
443
|
+
inline at the `Suture.create` or `Suture.verify` call-site, like so:
|
444
|
+
|
445
|
+
``` ruby
|
446
|
+
Suture.create(:my_type, {
|
447
|
+
:old => method(:old_method),
|
448
|
+
:args => [42],
|
449
|
+
:comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
|
450
|
+
})
|
451
|
+
```
|
452
|
+
|
453
|
+
Just be sure to set it the same way if you want `Suture.verify` to be able to
|
454
|
+
test your recorded values!
|
455
|
+
|
456
|
+
``` ruby
|
457
|
+
Suture.verify(:my_type, {
|
458
|
+
:subject => method(:old_method),
|
459
|
+
:comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
|
460
|
+
})
|
461
|
+
```
|
462
|
+
|
463
|
+
## Troubleshooting
|
464
|
+
|
465
|
+
Some ideas if you can't get a particular verification to work or if you keep
|
466
|
+
seeing false negatives:
|
467
|
+
|
468
|
+
* There may be a side effect in your code that you haven't found, extracted,
|
469
|
+
replicated, or controlled for. Consider contributing to [this
|
470
|
+
milestone](https://github.com/testdouble/suture/milestone/3), which specifies
|
471
|
+
a side-effect detector to be paired with Suture to make it easier to see
|
472
|
+
when observable database, network, and in-memory changes are made during a
|
473
|
+
Suture operation
|
474
|
+
* Consider writing a [custom comparator](#creating-a-custom-comparator) with
|
475
|
+
a relaxed conception of equivalence between the recorded and observed results
|
476
|
+
* If a recording was made in error, you can always delete it, either by
|
477
|
+
dropping Suture's database (which is, by default, stored in
|
478
|
+
`db/suture.sqlite3`) or by observing the ID of the recording from an error
|
479
|
+
message and invoking `Suture.delete(42)`
|
480
|
+
|
data/Rakefile
CHANGED
@@ -13,4 +13,25 @@ Rake::TestTask.new(:safe) do |t|
|
|
13
13
|
t.test_files = FileList['safe/helper.rb', 'safe/**/*_test.rb']
|
14
14
|
end
|
15
15
|
|
16
|
-
|
16
|
+
Rake::TestTask.new(:everything) do |t|
|
17
|
+
t.libs << "test"
|
18
|
+
t.libs << "safe"
|
19
|
+
t.libs << "lib"
|
20
|
+
t.test_files = FileList[
|
21
|
+
'safe/support/code_climate',
|
22
|
+
'test/helper.rb',
|
23
|
+
'test/**/*_test.rb',
|
24
|
+
'safe/helper.rb',
|
25
|
+
'safe/**/*_test.rb'
|
26
|
+
]
|
27
|
+
end
|
28
|
+
|
29
|
+
require 'github_changelog_generator/task'
|
30
|
+
|
31
|
+
GitHubChangelogGenerator::RakeTask.new :changelog do |config|
|
32
|
+
config.since_tag = '0.1.14'
|
33
|
+
config.future_release = '0.2.0'
|
34
|
+
end
|
35
|
+
task :release => :changelog
|
36
|
+
|
37
|
+
task :default => :everything
|
@@ -1,50 +1,68 @@
|
|
1
1
|
require "suture/wrap/sqlite"
|
2
|
+
require "suture/adapter/log"
|
2
3
|
require "suture/value/observation"
|
3
4
|
require "suture/error/observation_conflict"
|
4
5
|
|
5
6
|
module Suture::Adapter
|
6
7
|
class Dictaphone
|
8
|
+
include Suture::Adapter::Log
|
9
|
+
|
7
10
|
def initialize(plan)
|
8
11
|
@db = Suture::Wrap::Sqlite.init(plan.database_path)
|
9
12
|
@name = plan.name
|
10
|
-
@
|
13
|
+
@comparator = plan.comparator
|
14
|
+
if plan.respond_to?(:args) # does not apply to TestPlan objects
|
15
|
+
@args_inspect = plan.args.inspect
|
16
|
+
@args_dump = Marshal.dump(plan.args)
|
17
|
+
end
|
11
18
|
end
|
12
19
|
|
13
20
|
def record(result)
|
14
21
|
Suture::Wrap::Sqlite.insert(@db, :observations, [:name, :args, :result],
|
15
|
-
[@name.to_s,
|
16
|
-
|
17
|
-
rescue SQLite3::ConstraintException
|
18
|
-
|
19
|
-
if
|
20
|
-
|
22
|
+
[@name.to_s, @args_dump, Marshal.dump(result)])
|
23
|
+
log_info("recorded call for seam #{@name.inspect} with args `#{@args_inspect}` and result `#{result.inspect}`")
|
24
|
+
rescue SQLite3::ConstraintException
|
25
|
+
old_observation = known_observation
|
26
|
+
if @comparator.call(old_observation.result, result)
|
27
|
+
log_debug("skipped recording of duplicate call for seam #{@name.inspect} with args `#{@args_inspect}` and result `#{result.inspect}`")
|
21
28
|
else
|
22
|
-
|
29
|
+
raise Suture::Error::ObservationConflict.new(@name, @args_inspect, result, old_observation)
|
23
30
|
end
|
24
31
|
end
|
25
32
|
|
26
|
-
def play
|
27
|
-
rows = Suture::Wrap::Sqlite.select(
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
33
|
+
def play(only_id = nil)
|
34
|
+
rows = Suture::Wrap::Sqlite.select(
|
35
|
+
@db, :observations,
|
36
|
+
"where name = ? #{"and id = ?" if only_id}",
|
37
|
+
[@name.to_s, only_id].compact
|
38
|
+
)
|
39
|
+
log_debug("found #{rows.size} recorded calls for seam #{@name.inspect}.")
|
40
|
+
rows.map { |row| row_to_observation(row) }
|
41
|
+
end
|
42
|
+
|
43
|
+
def delete(id)
|
44
|
+
log_info("deleting call with ID: #{id}")
|
45
|
+
Suture::Wrap::Sqlite.delete(@db, :observations, "where id = ?", [id])
|
36
46
|
end
|
37
47
|
|
38
48
|
private
|
39
49
|
|
40
|
-
def
|
41
|
-
|
50
|
+
def row_to_observation(row)
|
51
|
+
Suture::Value::Observation.new(
|
52
|
+
row[0],
|
53
|
+
row[1].to_sym,
|
54
|
+
Marshal.load(row[2]),
|
55
|
+
Marshal.load(row[3])
|
56
|
+
)
|
57
|
+
end
|
58
|
+
|
59
|
+
def known_observation
|
60
|
+
row_to_observation(Suture::Wrap::Sqlite.select(
|
42
61
|
@db,
|
43
62
|
:observations,
|
44
63
|
"where name = ? and args = ?",
|
45
|
-
[name.to_s,
|
46
|
-
)
|
47
|
-
Marshal.load(rows.first[3])
|
64
|
+
[@name.to_s, @args_dump]
|
65
|
+
).first)
|
48
66
|
end
|
49
67
|
end
|
50
68
|
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
require "suture/wrap/logger"
|
2
|
+
require "suture/util/env"
|
3
|
+
|
4
|
+
module Suture::Adapter
|
5
|
+
module Log
|
6
|
+
def self.logger
|
7
|
+
if !@setup
|
8
|
+
@logger = Suture::Wrap::Logger.init(Suture.config.merge(Suture::Util::Env.to_map))
|
9
|
+
@setup = true
|
10
|
+
end
|
11
|
+
@logger
|
12
|
+
end
|
13
|
+
|
14
|
+
def self.reset!
|
15
|
+
@setup = nil
|
16
|
+
@logger = nil
|
17
|
+
end
|
18
|
+
|
19
|
+
def log_debug(*args, &blk)
|
20
|
+
Log.logger.debug(*args, &blk)
|
21
|
+
end
|
22
|
+
|
23
|
+
def log_info(*args, &blk)
|
24
|
+
Log.logger.info(*args, &blk)
|
25
|
+
end
|
26
|
+
|
27
|
+
def log_warn(*args, &blk)
|
28
|
+
Log.logger.warn(*args, &blk)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
data/lib/suture/builds_plan.rb
CHANGED
@@ -3,11 +3,11 @@ require "suture/util/env"
|
|
3
3
|
|
4
4
|
module Suture
|
5
5
|
class BuildsPlan
|
6
|
-
UN_ENV_IABLE_OPTIONS = [:name, :old, :new, :args]
|
6
|
+
UN_ENV_IABLE_OPTIONS = [:name, :old, :new, :args, :comparator]
|
7
7
|
|
8
8
|
def build(name, options = {})
|
9
9
|
Value::Plan.new(
|
10
|
-
|
10
|
+
Suture.config.
|
11
11
|
merge(options).
|
12
12
|
merge(:name => name).
|
13
13
|
merge(Suture::Util::Env.to_map(UN_ENV_IABLE_OPTIONS))
|
@@ -1,10 +1,20 @@
|
|
1
|
+
require "suture/adapter/log"
|
1
2
|
require "suture/surgeon/observer"
|
2
3
|
require "suture/surgeon/no_op"
|
3
4
|
|
4
5
|
module Suture
|
5
6
|
class ChoosesSurgeon
|
7
|
+
include Suture::Adapter::Log
|
8
|
+
|
6
9
|
def choose(plan)
|
7
10
|
if plan.record_calls
|
11
|
+
if plan.new
|
12
|
+
log_warn <<-MSG.gsub(/^ {12}/,'')
|
13
|
+
Seam #{plan.name.inspect} has a :new code path defined, but because
|
14
|
+
it is set to :record_calls, we will invoke the :old code path
|
15
|
+
instead. If this is not what you intend, set :record_calls to false.
|
16
|
+
MSG
|
17
|
+
end
|
8
18
|
Surgeon::Observer.new
|
9
19
|
else
|
10
20
|
Surgeon::NoOp.new
|
@@ -1,16 +1,28 @@
|
|
1
1
|
module Suture::Error
|
2
2
|
class ObservationConflict < StandardError
|
3
|
-
def initialize(name,
|
3
|
+
def initialize(name, args_inspect, new_result, old_observation)
|
4
4
|
@name = name
|
5
|
-
@
|
5
|
+
@args_inspect = args_inspect
|
6
6
|
@new_result = new_result
|
7
|
-
@
|
7
|
+
@old_id = old_observation.id
|
8
|
+
@old_result = old_observation.result
|
8
9
|
end
|
9
10
|
|
10
11
|
def message
|
11
|
-
<<-MSG.gsub(
|
12
|
-
At
|
13
|
-
|
12
|
+
<<-MSG.gsub(/^ {8}/,'')
|
13
|
+
At seam #{@name.inspect}, we just recorded a duplicate call, but the same arguments
|
14
|
+
resulted in a different output. Read on for details:
|
15
|
+
|
16
|
+
Arguments: ```
|
17
|
+
#{@args_inspect}
|
18
|
+
```
|
19
|
+
Previously-observed return value: ```
|
20
|
+
#{@old_result.inspect}
|
21
|
+
```
|
22
|
+
|
23
|
+
Newly-observed return value: ```
|
24
|
+
#{@new_result.inspect}
|
25
|
+
```
|
14
26
|
|
15
27
|
That's not good! Here are a few ideas of what may have happened:
|
16
28
|
|
@@ -22,21 +34,22 @@ module Suture::Error
|
|
22
34
|
different timestamp) or side effects (e.g. saving to a database
|
23
35
|
resulting in a different GUID value) mean that Suture is detecting two
|
24
36
|
different results for the same inputs. This can be worked around by
|
25
|
-
providing a custom comparator
|
26
|
-
|
27
|
-
https://github.com/testdouble/suture
|
37
|
+
providing a custom comparator to Suture. For more info, see the README:
|
38
|
+
|
39
|
+
https://github.com/testdouble/suture#creating-a-custom-comparator
|
28
40
|
|
29
41
|
3. If neither of the above are true, it's possible that the old code path
|
30
42
|
was changed while still in the early stage of recording characterization
|
31
43
|
calls (presumably by mistake). If such a change may have occurred in
|
32
44
|
error, check your git history. Otherwise, perhaps you `record_calls` is
|
33
|
-
accidentally still enabled and should be turned off for this
|
45
|
+
accidentally still enabled and should be turned off for this seam
|
34
46
|
(either with SUTURE_RECORD_CALLS=false or :record_calls => false).
|
35
47
|
|
36
|
-
4. If
|
37
|
-
|
38
|
-
|
48
|
+
4. If, after exhausting the possibilities above, you're pretty sure the
|
49
|
+
recorded result is in error, you can delete it from Suture's database
|
50
|
+
with:
|
39
51
|
|
52
|
+
Suture.delete(#{@old_id})
|
40
53
|
MSG
|
41
54
|
end
|
42
55
|
end
|
@@ -1,12 +1,167 @@
|
|
1
1
|
module Suture::Error
|
2
2
|
class VerificationFailed < StandardError
|
3
|
-
def initialize(
|
4
|
-
|
5
|
-
@
|
3
|
+
def initialize(plan, results)
|
4
|
+
@plan = plan
|
5
|
+
@results = results
|
6
6
|
end
|
7
7
|
|
8
|
-
|
9
|
-
|
8
|
+
def message
|
9
|
+
[
|
10
|
+
intro,
|
11
|
+
describe_failures(@results.failed, @plan),
|
12
|
+
configuration(@plan),
|
13
|
+
summarize(@results)
|
14
|
+
].join("\n")
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def intro
|
20
|
+
<<-MSG.gsub(/^ {8}/,'')
|
21
|
+
|
22
|
+
# Verification of your seam failed!
|
23
|
+
|
24
|
+
Descriptions of each unsuccessful verification follows:
|
25
|
+
MSG
|
26
|
+
end
|
27
|
+
|
28
|
+
def summarize(results)
|
29
|
+
<<-MSG.gsub(/^ {8}/,'')
|
30
|
+
# Result Summary
|
31
|
+
|
32
|
+
- Passed........#{results.passed_count}
|
33
|
+
- Failed........#{results.failed_count}
|
34
|
+
- with error..#{results.errored_count}
|
35
|
+
- Skipped.......#{results.skipped_count}
|
36
|
+
- Total calls...#{results.total_count}
|
37
|
+
MSG
|
38
|
+
end
|
39
|
+
|
40
|
+
def configuration(plan)
|
41
|
+
<<-MSG.gsub(/^ {8}/,'')
|
42
|
+
# Configuration
|
43
|
+
|
44
|
+
This is the configuration used by this test run:
|
45
|
+
|
46
|
+
```
|
47
|
+
{
|
48
|
+
:comparator => #{describe_comparator(plan.comparator)}
|
49
|
+
:database_path => #{plan.database_path.inspect},
|
50
|
+
:fail_fast => #{plan.fail_fast},
|
51
|
+
:call_limit => #{plan.call_limit.inspect},#{" # (no limit)" if plan.call_limit.nil?}
|
52
|
+
:time_limit => #{plan.time_limit.inspect},#{plan.time_limit.nil? ? " # (no limit)" : " # (in seconds)"}
|
53
|
+
:error_message_limit => #{plan.error_message_limit.inspect},#{" # (no limit)" if plan.error_message_limit.nil?}
|
54
|
+
:random_seed => #{plan.random_seed.inspect}#{" # (insertion order)" if plan.random_seed.nil?}
|
55
|
+
}
|
56
|
+
```
|
57
|
+
MSG
|
58
|
+
end
|
59
|
+
|
60
|
+
def describe_comparator(comparator)
|
61
|
+
if comparator.kind_of?(Proc)
|
62
|
+
"Proc, # (in: `#{describe_source_location(*comparator.source_location)}`)"
|
63
|
+
elsif comparator.respond_to?(:method) && comparator.method(:call)
|
64
|
+
"#{comparator.class}.new, # (in: `#{describe_source_location(*comparator.method(:call).source_location)}`)"
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
def describe_source_location(file, line)
|
69
|
+
root = File.join(Dir.getwd, "/")
|
70
|
+
path = file.start_with?(root) ? file.gsub(root, '') : file
|
71
|
+
"#{path}:#{line}"
|
72
|
+
end
|
73
|
+
|
74
|
+
def describe_failures(failures, plan)
|
75
|
+
return if failures.empty?
|
76
|
+
[
|
77
|
+
"## Failures\n",
|
78
|
+
failures.each_with_index.map { |failure, index|
|
79
|
+
describe_failure(failure, index) if plan.error_message_limit.nil? || index < plan.error_message_limit
|
80
|
+
},
|
81
|
+
describe_squelched_failure_messages(failures.size, plan.error_message_limit),
|
82
|
+
describe_general_failure_advice(plan)
|
83
|
+
].flatten.compact.join("\n")
|
84
|
+
end
|
85
|
+
|
86
|
+
def describe_failure(failure, index)
|
87
|
+
expected = failure[:observation]
|
88
|
+
return <<-MSG.gsub(/^ {8}/,'')
|
89
|
+
#{index + 1}.) Recorded call for seam #{expected.name.inspect} (ID: #{expected.id}) ran and #{failure[:error] ? "raised an error" : "failed comparison"}.
|
90
|
+
|
91
|
+
Arguments: ```
|
92
|
+
#{expected.args.inspect}
|
93
|
+
```
|
94
|
+
Expected result: ```
|
95
|
+
#{expected.result.inspect}
|
96
|
+
```
|
97
|
+
#{failure[:error] ? "Error raised" : "Actual result"}: ```
|
98
|
+
#{if failure[:error]
|
99
|
+
stringify_error(failure[:error])
|
100
|
+
else
|
101
|
+
failure[:new_result].inspect
|
102
|
+
end
|
103
|
+
}
|
104
|
+
```
|
105
|
+
|
106
|
+
Ideas to fix this:
|
107
|
+
* Focus on this test by setting ENV var `SUTURE_VERIFY_ONLY=#{expected.id}`
|
108
|
+
* Is the recording wrong? Delete it! `Suture.delete(#{expected.id})`
|
109
|
+
MSG
|
110
|
+
end
|
111
|
+
|
112
|
+
def describe_squelched_failure_messages(failure_count, error_message_limit)
|
113
|
+
return if error_message_limit.nil? || error_message_limit >= failure_count
|
114
|
+
"(#{failure_count - error_message_limit} more failure messages were hidden because :error_message_limit was set to #{error_message_limit}.)"
|
115
|
+
end
|
116
|
+
|
117
|
+
def describe_general_failure_advice(plan)
|
118
|
+
<<-MSG.gsub(/^ {8}/,'')
|
119
|
+
### Fixing these failures
|
120
|
+
|
121
|
+
#### Custom comparator
|
10
122
|
|
123
|
+
If any comparison is failing and you believe the results are
|
124
|
+
equivalent, we suggest you look into creating a custom comparator.
|
125
|
+
See more details here:
|
126
|
+
|
127
|
+
https://github.com/testdouble/suture#creating-a-custom-comparator
|
128
|
+
|
129
|
+
#### Random seed
|
130
|
+
|
131
|
+
Suture runs all verifications in random order by default. If you're
|
132
|
+
seeing an erratic failure, it's possibly due to order-dependent
|
133
|
+
behavior somewhere in your subject's code.
|
134
|
+
|
135
|
+
#{if !plan.random_seed.nil?
|
136
|
+
<<-MOAR.gsub(/^ {14}/,'')
|
137
|
+
To re-run the tests with the same random seed as was used in this run,
|
138
|
+
set the env var `SUTURE_RANDOM_SEED=#{plan.random_seed}` or the config entry
|
139
|
+
`:random_seed => #{plan.random_seed}`.
|
140
|
+
|
141
|
+
To re-run the tests without added shuffling (that is, in the order the
|
142
|
+
calls were recorded in), then set the random seed explicitly to nil
|
143
|
+
with env var `SUTURE_RANDOM_SEED=nil` or the config entry
|
144
|
+
`:random_seed => nil`.
|
145
|
+
MOAR
|
146
|
+
else
|
147
|
+
<<-MOAR.gsub(/^ {14}/,'')
|
148
|
+
This test was run in insertion order (by the primary key of the table
|
149
|
+
that stores calls, ascending). This is sometimes necessary when the
|
150
|
+
code has an order-dependent side effect, but shouldn't be set unless it's
|
151
|
+
clearly necessary, so as not to incidentally encourage _porting over_
|
152
|
+
that temporal side effect to the new code path. To restore random
|
153
|
+
ordering, unset the env var `SUTURE_RANDOM_SEED` and/or the config entry
|
154
|
+
`:random_seed`.
|
155
|
+
MOAR
|
156
|
+
end.chomp}
|
157
|
+
MSG
|
158
|
+
end
|
159
|
+
|
160
|
+
|
161
|
+
def stringify_error(error)
|
162
|
+
s = error.inspect
|
163
|
+
s += "\n" + error.backtrace.join("\n") if error.backtrace
|
164
|
+
s
|
165
|
+
end
|
11
166
|
end
|
12
167
|
end
|
@@ -2,9 +2,9 @@ require "suture/error/verification_failed"
|
|
2
2
|
|
3
3
|
module Suture
|
4
4
|
class InterpretsResults
|
5
|
-
def interpret(test_results)
|
5
|
+
def interpret(test_plan, test_results)
|
6
6
|
return unless test_results.failed?
|
7
|
-
raise Suture::Error::VerificationFailed.new(test_results)
|
7
|
+
raise Suture::Error::VerificationFailed.new(test_plan, test_results)
|
8
8
|
end
|
9
9
|
end
|
10
10
|
end
|
@@ -3,14 +3,14 @@ require "suture/util/env"
|
|
3
3
|
|
4
4
|
module Suture
|
5
5
|
class PrescribesTestPlan
|
6
|
-
UN_ENV_IABLE_OPTIONS = [:name, :subject, :
|
6
|
+
UN_ENV_IABLE_OPTIONS = [:name, :subject, :comparator]
|
7
7
|
DEFAULT_TEST_OPTIONS = {
|
8
|
-
:fail_fast =>
|
8
|
+
:fail_fast => false
|
9
9
|
}
|
10
10
|
|
11
11
|
def prescribe(name, options = {})
|
12
|
-
Value::TestPlan.new(
|
13
|
-
merge(
|
12
|
+
Value::TestPlan.new(DEFAULT_TEST_OPTIONS.
|
13
|
+
merge(Suture.config).
|
14
14
|
merge(options).
|
15
15
|
merge(:name => name).
|
16
16
|
merge(Suture::Util::Env.to_map(UN_ENV_IABLE_OPTIONS)))
|