scientist 1.1.1 → 1.5.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.travis.yml +4 -6
- data/CONTRIBUTING.md +26 -0
- data/README.md +68 -15
- data/doc/changelog.md +23 -0
- data/lib/scientist/experiment.rb +59 -29
- data/lib/scientist/observation.rb +4 -6
- data/lib/scientist/version.rb +1 -1
- data/scientist.gemspec +2 -0
- data/script/test +1 -1
- data/test/scientist/experiment_test.rb +75 -0
- data/test/scientist_test.rb +10 -8
- metadata +8 -8
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c2831c4fc4f76eb1e8e822a21e613a07e33229bdfe47528cd4555b72ecb05a7c
|
4
|
+
data.tar.gz: e0e003dbbe7ecb82749c41aff85ff6b4174e6169dce34081d20f325f62959f8d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 07f99bcd0d9619205f42f15f16cc00b31fff3fab127344ca70360c51eea28d3eb15bc21e9471edbccfb06ddf2f07bc93ac0024af3899371486757fb53dbb5710
|
7
|
+
data.tar.gz: cb7a3cf609e501a5d4a7894cf62400dacde1dc654ec7caf5d9beb8d115bb4f0c1e217cdeddd7c72863385459bfd7bb929221df74725312c3dfe21af315598486
|
data/.travis.yml
CHANGED
data/CONTRIBUTING.md
ADDED
@@ -0,0 +1,26 @@
|
|
1
|
+
## Contributing
|
2
|
+
|
3
|
+
[fork]: https://github.com/github/scientist/fork
|
4
|
+
[pr]: https://github.com/github/scientist/compare
|
5
|
+
|
6
|
+
Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
|
7
|
+
|
8
|
+
Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE.txt).
|
9
|
+
|
10
|
+
## Submitting a pull request
|
11
|
+
|
12
|
+
0. [Fork][fork] and clone the repository
|
13
|
+
0. Create a new branch: `git checkout -b my-branch-name`
|
14
|
+
0. Make your change, push to your fork and [submit a pull request][pr]
|
15
|
+
0. Pat your self on the back and wait for your pull request to be reviewed and merged.
|
16
|
+
|
17
|
+
Here are a few things you can do that will increase the likelihood of your pull request being accepted:
|
18
|
+
|
19
|
+
- Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
|
20
|
+
- Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
|
21
|
+
|
22
|
+
## Resources
|
23
|
+
|
24
|
+
- [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
|
25
|
+
- [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
|
26
|
+
- [GitHub Help](https://help.github.com)
|
data/README.md
CHANGED
@@ -26,7 +26,7 @@ Wrap a `use` block around the code's original behavior, and wrap `try` around th
|
|
26
26
|
* Randomizes the order in which `use` and `try` blocks are run,
|
27
27
|
* Measures the durations of all behaviors,
|
28
28
|
* Compares the result of `try` to the result of `use`,
|
29
|
-
*
|
29
|
+
* Swallow and record exceptions raised in the `try` block when overriding `raised`, and
|
30
30
|
* Publishes all this information.
|
31
31
|
|
32
32
|
The `use` block is called the **control**. The `try` block is called the **candidate**.
|
@@ -52,7 +52,7 @@ If you don't declare any `try` blocks, none of the Scientist machinery is invoke
|
|
52
52
|
|
53
53
|
## Making science useful
|
54
54
|
|
55
|
-
The examples above will run, but they're not really *doing* anything. The `try` blocks run
|
55
|
+
The examples above will run, but they're not really *doing* anything. The `try` blocks don't run yet and none of the results get published. Replace the default experiment implementation to control execution and reporting:
|
56
56
|
|
57
57
|
```ruby
|
58
58
|
require "scientist/experiment"
|
@@ -62,7 +62,7 @@ class MyExperiment
|
|
62
62
|
|
63
63
|
attr_accessor :name
|
64
64
|
|
65
|
-
def initialize(name
|
65
|
+
def initialize(name)
|
66
66
|
@name = name
|
67
67
|
end
|
68
68
|
|
@@ -71,18 +71,17 @@ class MyExperiment
|
|
71
71
|
true
|
72
72
|
end
|
73
73
|
|
74
|
+
def raised(operation, error)
|
75
|
+
# see "In a Scientist callback" below
|
76
|
+
p "Operation '#{operation}' failed with error '#{error.inspect}'"
|
77
|
+
super # will re-raise
|
78
|
+
end
|
79
|
+
|
74
80
|
def publish(result)
|
75
81
|
# see "Publishing results" below
|
76
82
|
p result
|
77
83
|
end
|
78
84
|
end
|
79
|
-
|
80
|
-
# replace `Scientist::Default` as the default implementation
|
81
|
-
module Scientist::Experiment
|
82
|
-
def self.new(name)
|
83
|
-
MyExperiment.new(name: name)
|
84
|
-
end
|
85
|
-
end
|
86
85
|
```
|
87
86
|
|
88
87
|
Now calls to the `science` helper will load instances of `MyExperiment`.
|
@@ -206,6 +205,8 @@ class MyExperiment
|
|
206
205
|
end
|
207
206
|
```
|
208
207
|
|
208
|
+
Note that the `#clean` method will discard the previous cleaner block if you call it again. If for some reason you need to access the currently configured cleaner block, `Scientist::Experiment#cleaner` will return the block without further ado. _(This probably won't come up in normal usage, but comes in handy if you're writing, say, a custom experiment runner that provides default cleaners.)_
|
209
|
+
|
209
210
|
### Ignoring mismatches
|
210
211
|
|
211
212
|
During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the `ignore` method. You may include more than one block if needed:
|
@@ -254,7 +255,7 @@ class MyExperiment
|
|
254
255
|
|
255
256
|
attr_accessor :name, :percent_enabled
|
256
257
|
|
257
|
-
def initialize(name
|
258
|
+
def initialize(name)
|
258
259
|
@name = name
|
259
260
|
@percent_enabled = 100
|
260
261
|
end
|
@@ -327,7 +328,7 @@ class MyExperiment
|
|
327
328
|
}
|
328
329
|
else
|
329
330
|
{
|
330
|
-
# see "Keeping it clean"
|
331
|
+
# see "Keeping it clean" above
|
331
332
|
:value => observation.cleaned_value
|
332
333
|
}
|
333
334
|
end
|
@@ -352,6 +353,35 @@ MyExperiment.raise_on_mismatches = true
|
|
352
353
|
|
353
354
|
Scientist will raise a `Scientist::Experiment::MismatchError` exception if any observations don't match.
|
354
355
|
|
356
|
+
#### Custom mismatch errors
|
357
|
+
|
358
|
+
To instruct Scientist to raise a custom error instead of the default `Scientist::Experiment::MismatchError`:
|
359
|
+
|
360
|
+
```ruby
|
361
|
+
class CustomMismatchError < Scientist::Experiment::MismatchError
|
362
|
+
def to_s
|
363
|
+
message = "There was a mismatch! Here's the diff:"
|
364
|
+
|
365
|
+
diffs = result.candidates.map do |candidate|
|
366
|
+
Diff.new(result.control, candidate)
|
367
|
+
end.join("\n")
|
368
|
+
|
369
|
+
"#{message}\n#{diffs}"
|
370
|
+
end
|
371
|
+
end
|
372
|
+
```
|
373
|
+
|
374
|
+
```ruby
|
375
|
+
science "widget-permissions" do |e|
|
376
|
+
e.use { Report.find(id) }
|
377
|
+
e.try { ReportService.new.fetch(id) }
|
378
|
+
|
379
|
+
e.raise_with CustomMismatchError
|
380
|
+
end
|
381
|
+
```
|
382
|
+
|
383
|
+
This allows for pre-processing on mismatch error exception messages.
|
384
|
+
|
355
385
|
### Handling errors
|
356
386
|
|
357
387
|
#### In candidate code
|
@@ -462,6 +492,22 @@ science "various-ways", run: "first-way" do |e|
|
|
462
492
|
end
|
463
493
|
```
|
464
494
|
|
495
|
+
#### Providing fake timing data
|
496
|
+
|
497
|
+
If you're writing tests that depend on specific timing values, you can provide canned durations using the `fabricate_durations_for_testing_purposes` method, and Scientist will report these in `Scientist::Observation#duration` instead of the actual execution times.
|
498
|
+
|
499
|
+
```ruby
|
500
|
+
science "absolutely-nothing-suspicious-happening-here" do |e|
|
501
|
+
e.use { ... } # "control"
|
502
|
+
e.try { ... } # "candidate"
|
503
|
+
e.fabricate_durations_for_testing_purposes( "control" => 1.0, "candidate" => 0.5 )
|
504
|
+
end
|
505
|
+
```
|
506
|
+
|
507
|
+
`fabricate_durations_for_testing_purposes` takes a Hash of duration values, keyed by behavior names. (By default, Scientist uses `"control"` and `"candidate"`, but if you override these as shown in [Trying more than one thing](#trying-more-than-one-thing) or [No control, just candidates](#no-control-just-candidates), use matching names here.) If a name is not provided, the actual execution time will be reported instead.
|
508
|
+
|
509
|
+
_Like `Scientist::Experiment#cleaner`, this probably won't come up in normal usage. It's here to make it easier to test code that extends Scientist._
|
510
|
+
|
465
511
|
### Without including Scientist
|
466
512
|
|
467
513
|
If you need to use Scientist in a place where you aren't able to include the Scientist module, you can call `Scientist.run`:
|
@@ -475,17 +521,23 @@ end
|
|
475
521
|
|
476
522
|
## Hacking
|
477
523
|
|
478
|
-
Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically.
|
524
|
+
Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically. Scientist requires Ruby 2.3 or newer.
|
525
|
+
|
526
|
+
## Wrappers
|
527
|
+
|
528
|
+
- [RealGeeks/lab_tech](https://github.com/RealGeeks/lab_tech) is a Rails engine for using this library by controlling, storing, and analyzing experiment results with ActiveRecord.
|
479
529
|
|
480
530
|
## Alternatives
|
481
531
|
|
482
532
|
- [daylerees/scientist](https://github.com/daylerees/scientist) (PHP)
|
483
|
-
- [
|
533
|
+
- [scientistproject/scientist.net](https://github.com/scientistproject/Scientist.net) (.NET)
|
484
534
|
- [joealcorn/laboratory](https://github.com/joealcorn/laboratory) (Python)
|
485
535
|
- [rawls238/Scientist4J](https://github.com/rawls238/Scientist4J) (Java)
|
486
536
|
- [tomiaijo/scientist](https://github.com/tomiaijo/scientist) (C++)
|
487
537
|
- [trello/scientist](https://github.com/trello/scientist) (node.js)
|
488
538
|
- [ziyasal/scientist.js](https://github.com/ziyasal/scientist.js) (node.js, ES6)
|
539
|
+
- [TrueWill/tzientist](https://github.com/TrueWill/tzientist) (node.js, TypeScript)
|
540
|
+
- [TrueWill/paleontologist](https://github.com/TrueWill/paleontologist) (Deno, TypeScript)
|
489
541
|
- [yeller/laboratory](https://github.com/yeller/laboratory) (Clojure)
|
490
542
|
- [lancew/Scientist](https://github.com/lancew/Scientist) (Perl 5)
|
491
543
|
- [lancew/ScientistP6](https://github.com/lancew/ScientistP6) (Perl 6)
|
@@ -494,7 +546,8 @@ Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs t
|
|
494
546
|
- [calavera/go-scientist](https://github.com/calavera/go-scientist) (Go)
|
495
547
|
- [jelmersnoeck/experiment](https://github.com/jelmersnoeck/experiment) (Go)
|
496
548
|
- [spoptchev/scientist](https://github.com/spoptchev/scientist) (Kotlin / Java)
|
497
|
-
|
549
|
+
- [junkpiano/scientist](https://github.com/junkpiano/scientist) (Swift)
|
550
|
+
- [serverless scientist](http://serverlessscientist.com/) (AWS Lambda)
|
498
551
|
|
499
552
|
## Maintainers
|
500
553
|
|
data/doc/changelog.md
CHANGED
@@ -1,5 +1,28 @@
|
|
1
1
|
# Changes
|
2
2
|
|
3
|
+
## v1.3.0 (2 April 2019)
|
4
|
+
|
5
|
+
- New: Drop support for ruby <2.3
|
6
|
+
- Fix: Build new strings instead of modifying frozen ones
|
7
|
+
- New: Add an accessor for the configured clean block
|
8
|
+
- New: Add a hook to use fabricated durations instead of actual timing data.
|
9
|
+
|
10
|
+
## v1.2.0 (5 July 2018)
|
11
|
+
|
12
|
+
- New: Use monotonic clock for duration calculations
|
13
|
+
- New: Drop support for ruby <2.1 to support monotonic clock
|
14
|
+
- New: Run CI on Ruby 2.5
|
15
|
+
|
16
|
+
## v1.1.2 (9 May 2018)
|
17
|
+
|
18
|
+
- New: Add `raise_with` option to allow for custom mismatch errors to be raised
|
19
|
+
|
20
|
+
## v1.1.1 (6 February 2018)
|
21
|
+
|
22
|
+
- Fix: default experiment no longer runs all `try` paths
|
23
|
+
- New: Add `Scientist.run` module method for running experiments when an included module isn't available
|
24
|
+
- New: Add [Noise and error rates](https://github.com/github/scientist#noise-and-error-rates) to `README.md`
|
25
|
+
|
3
26
|
## v1.1.0 (29 August 2017)
|
4
27
|
|
5
28
|
- New: [Specify which exception types to rescue](https://github.com/github/scientist#in-candidate-code)
|
data/lib/scientist/experiment.rb
CHANGED
@@ -9,16 +9,26 @@ module Scientist::Experiment
|
|
9
9
|
# If this is nil, raise_on_mismatches class attribute is used instead.
|
10
10
|
attr_accessor :raise_on_mismatches
|
11
11
|
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
12
|
+
def self.included(base)
|
13
|
+
self.set_default(base)
|
14
|
+
base.extend RaiseOnMismatch
|
15
|
+
end
|
16
|
+
|
17
|
+
# Instantiate a new experiment (using the class given to the .set_default method).
|
16
18
|
def self.new(name)
|
17
|
-
Scientist::Default.new(name)
|
19
|
+
(@experiment_klass || Scientist::Default).new(name)
|
20
|
+
end
|
21
|
+
|
22
|
+
# Configure Scientist to use the given class for all future experiments
|
23
|
+
# (must implement the Scientist::Experiment interface).
|
24
|
+
#
|
25
|
+
# Called automatically when new experiments are defined.
|
26
|
+
def self.set_default(klass)
|
27
|
+
@experiment_klass = klass
|
18
28
|
end
|
19
29
|
|
20
30
|
# A mismatch, raised when raise_on_mismatches is enabled.
|
21
|
-
class MismatchError <
|
31
|
+
class MismatchError < Exception
|
22
32
|
attr_reader :name, :result
|
23
33
|
|
24
34
|
def initialize(name, result)
|
@@ -41,10 +51,10 @@ module Scientist::Experiment
|
|
41
51
|
def format_observation(observation)
|
42
52
|
observation.name + ":\n" +
|
43
53
|
if observation.raised?
|
44
|
-
observation.exception.
|
45
|
-
|
54
|
+
lines = observation.exception.backtrace.map { |line| " #{line}" }.join("\n")
|
55
|
+
" #{observation.exception.inspect}" + "\n" + lines
|
46
56
|
else
|
47
|
-
observation.
|
57
|
+
" #{observation.cleaned_value.inspect}"
|
48
58
|
end
|
49
59
|
end
|
50
60
|
end
|
@@ -67,10 +77,6 @@ module Scientist::Experiment
|
|
67
77
|
end
|
68
78
|
end
|
69
79
|
|
70
|
-
def self.included(base)
|
71
|
-
base.extend RaiseOnMismatch
|
72
|
-
end
|
73
|
-
|
74
80
|
# Define a block of code to run before an experiment begins, if the experiment
|
75
81
|
# is enabled.
|
76
82
|
#
|
@@ -96,6 +102,13 @@ module Scientist::Experiment
|
|
96
102
|
@_scientist_cleaner = block
|
97
103
|
end
|
98
104
|
|
105
|
+
# Accessor for the clean block, if one is available.
|
106
|
+
#
|
107
|
+
# Returns the configured block, or nil.
|
108
|
+
def cleaner
|
109
|
+
@_scientist_cleaner
|
110
|
+
end
|
111
|
+
|
99
112
|
# Internal: Clean a value with the configured clean block, or return the value
|
100
113
|
# if no clean block is configured.
|
101
114
|
#
|
@@ -176,6 +189,10 @@ module Scientist::Experiment
|
|
176
189
|
false
|
177
190
|
end
|
178
191
|
|
192
|
+
def raise_with(exception)
|
193
|
+
@_scientist_custom_mismatch_error = exception
|
194
|
+
end
|
195
|
+
|
179
196
|
# Called when an exception is raised while running an internal operation,
|
180
197
|
# like :publish. Override this method to track these exceptions. The
|
181
198
|
# default implementation re-raises the exception.
|
@@ -205,16 +222,7 @@ module Scientist::Experiment
|
|
205
222
|
@_scientist_before_run.call
|
206
223
|
end
|
207
224
|
|
208
|
-
|
209
|
-
|
210
|
-
behaviors.keys.shuffle.each do |key|
|
211
|
-
block = behaviors[key]
|
212
|
-
observations << Scientist::Observation.new(key, self, &block)
|
213
|
-
end
|
214
|
-
|
215
|
-
control = observations.detect { |o| o.name == name }
|
216
|
-
|
217
|
-
result = Scientist::Result.new self, observations, control
|
225
|
+
result = generate_result(name)
|
218
226
|
|
219
227
|
begin
|
220
228
|
publish(result)
|
@@ -223,14 +231,16 @@ module Scientist::Experiment
|
|
223
231
|
end
|
224
232
|
|
225
233
|
if raise_on_mismatches? && result.mismatched?
|
226
|
-
|
234
|
+
if @_scientist_custom_mismatch_error
|
235
|
+
raise @_scientist_custom_mismatch_error.new(self.name, result)
|
236
|
+
else
|
237
|
+
raise MismatchError.new(self.name, result)
|
238
|
+
end
|
227
239
|
end
|
228
240
|
|
229
|
-
|
230
|
-
|
231
|
-
|
232
|
-
control.value
|
233
|
-
end
|
241
|
+
control = result.control
|
242
|
+
raise control.exception if control.raised?
|
243
|
+
control.value
|
234
244
|
end
|
235
245
|
|
236
246
|
# Define a block that determines whether or not the experiment should run.
|
@@ -282,4 +292,24 @@ module Scientist::Experiment
|
|
282
292
|
!!raise_on_mismatches
|
283
293
|
end
|
284
294
|
end
|
295
|
+
|
296
|
+
# Provide predefined durations to use instead of actual timing data.
|
297
|
+
# This is here solely as a convenience for developers of libraries that extend Scientist.
|
298
|
+
def fabricate_durations_for_testing_purposes(fabricated_durations = {})
|
299
|
+
@_scientist_fabricated_durations = fabricated_durations
|
300
|
+
end
|
301
|
+
|
302
|
+
# Internal: Generate the observations and create the result from those and the control.
|
303
|
+
def generate_result(name)
|
304
|
+
observations = []
|
305
|
+
|
306
|
+
behaviors.keys.shuffle.each do |key|
|
307
|
+
block = behaviors[key]
|
308
|
+
fabricated_duration = @_scientist_fabricated_durations && @_scientist_fabricated_durations[key]
|
309
|
+
observations << Scientist::Observation.new(key, self, fabricated_duration: fabricated_duration, &block)
|
310
|
+
end
|
311
|
+
|
312
|
+
control = observations.detect { |o| o.name == name }
|
313
|
+
Scientist::Result.new(self, observations, control)
|
314
|
+
end
|
285
315
|
end
|
@@ -8,9 +8,6 @@ class Scientist::Observation
|
|
8
8
|
# The experiment this observation is for
|
9
9
|
attr_reader :experiment
|
10
10
|
|
11
|
-
# The instant observation began.
|
12
|
-
attr_reader :now
|
13
|
-
|
14
11
|
# The String name of the behavior.
|
15
12
|
attr_reader :name
|
16
13
|
|
@@ -23,18 +20,19 @@ class Scientist::Observation
|
|
23
20
|
# The Float seconds elapsed.
|
24
21
|
attr_reader :duration
|
25
22
|
|
26
|
-
def initialize(name, experiment, &block)
|
23
|
+
def initialize(name, experiment, fabricated_duration: nil, &block)
|
27
24
|
@name = name
|
28
25
|
@experiment = experiment
|
29
|
-
@now = Time.now
|
30
26
|
|
27
|
+
starting = Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second) unless fabricated_duration
|
31
28
|
begin
|
32
29
|
@value = block.call
|
33
30
|
rescue *RESCUES => e
|
34
31
|
@exception = e
|
35
32
|
end
|
36
33
|
|
37
|
-
@duration =
|
34
|
+
@duration = fabricated_duration ||
|
35
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second) - starting
|
38
36
|
|
39
37
|
freeze
|
40
38
|
end
|
data/lib/scientist/version.rb
CHANGED
data/scientist.gemspec
CHANGED
@@ -10,6 +10,8 @@ Gem::Specification.new do |gem|
|
|
10
10
|
gem.homepage = "https://github.com/github/scientist"
|
11
11
|
gem.license = "MIT"
|
12
12
|
|
13
|
+
gem.required_ruby_version = '>= 2.3'
|
14
|
+
|
13
15
|
gem.files = `git ls-files`.split($/)
|
14
16
|
gem.executables = []
|
15
17
|
gem.test_files = gem.files.grep(/^test/)
|
data/script/test
CHANGED
@@ -2,6 +2,9 @@ describe Scientist::Experiment do
|
|
2
2
|
class Fake
|
3
3
|
include Scientist::Experiment
|
4
4
|
|
5
|
+
# Undo auto-config magic / preserve default behavior of Scientist::Experiment.new
|
6
|
+
Scientist::Experiment.set_default(nil)
|
7
|
+
|
5
8
|
def initialize(*args)
|
6
9
|
end
|
7
10
|
|
@@ -239,6 +242,13 @@ describe Scientist::Experiment do
|
|
239
242
|
assert_equal 10, @ex.clean_value(10)
|
240
243
|
end
|
241
244
|
|
245
|
+
it "provides the clean block when asked for it, in case subclasses wish to override and provide defaults" do
|
246
|
+
assert_nil @ex.cleaner
|
247
|
+
cleaner = ->(value) { value.upcase }
|
248
|
+
@ex.clean(&cleaner)
|
249
|
+
assert_equal cleaner, @ex.cleaner
|
250
|
+
end
|
251
|
+
|
242
252
|
it "calls the configured clean block with a value when configured" do
|
243
253
|
@ex.clean do |value|
|
244
254
|
value.upcase
|
@@ -262,6 +272,19 @@ describe Scientist::Experiment do
|
|
262
272
|
assert_equal "kaboom", exception.message
|
263
273
|
end
|
264
274
|
|
275
|
+
describe "#raise_with" do
|
276
|
+
it "raises custom error if provided" do
|
277
|
+
CustomError = Class.new(Scientist::Experiment::MismatchError)
|
278
|
+
|
279
|
+
@ex.use { 1 }
|
280
|
+
@ex.try { 2 }
|
281
|
+
@ex.raise_with(CustomError)
|
282
|
+
@ex.raise_on_mismatches = true
|
283
|
+
|
284
|
+
assert_raises(CustomError) { @ex.run }
|
285
|
+
end
|
286
|
+
end
|
287
|
+
|
265
288
|
describe "#run_if" do
|
266
289
|
it "does not run the experiment if the given block returns false" do
|
267
290
|
candidate_ran = false
|
@@ -392,6 +415,16 @@ describe Scientist::Experiment do
|
|
392
415
|
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
|
393
416
|
end
|
394
417
|
|
418
|
+
it "cleans values when raising on observation mismatch" do
|
419
|
+
Fake.raise_on_mismatches = true
|
420
|
+
@ex.use { "fine" }
|
421
|
+
@ex.try { "not fine" }
|
422
|
+
@ex.clean { "So Clean" }
|
423
|
+
|
424
|
+
err = assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
|
425
|
+
assert_match /So Clean/, err.message
|
426
|
+
end
|
427
|
+
|
395
428
|
it "doesn't raise when there is a mismatch if raise on mismatches is disabled" do
|
396
429
|
Fake.raise_on_mismatches = false
|
397
430
|
@ex.use { "fine" }
|
@@ -414,6 +447,20 @@ describe Scientist::Experiment do
|
|
414
447
|
assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
|
415
448
|
end
|
416
449
|
|
450
|
+
it "allows MismatchError to bubble up through bare rescues" do
|
451
|
+
Fake.raise_on_mismatches = true
|
452
|
+
@ex.use { "control" }
|
453
|
+
@ex.try { "candidate" }
|
454
|
+
runner = -> {
|
455
|
+
begin
|
456
|
+
@ex.run
|
457
|
+
rescue
|
458
|
+
# StandardError handled
|
459
|
+
end
|
460
|
+
}
|
461
|
+
assert_raises(Scientist::Experiment::MismatchError) { runner.call }
|
462
|
+
end
|
463
|
+
|
417
464
|
describe "#raise_on_mismatches?" do
|
418
465
|
it "raises when there is a mismatch if the experiment instance's raise on mismatches is enabled" do
|
419
466
|
Fake.raise_on_mismatches = false
|
@@ -536,4 +583,32 @@ candidate:
|
|
536
583
|
refute before, "before_run should not have run"
|
537
584
|
end
|
538
585
|
end
|
586
|
+
|
587
|
+
describe "testing hooks for extending code" do
|
588
|
+
it "allows a user to provide fabricated durations for testing purposes" do
|
589
|
+
@ex.use { true }
|
590
|
+
@ex.try { true }
|
591
|
+
@ex.fabricate_durations_for_testing_purposes( "control" => 0.5, "candidate" => 1.0 )
|
592
|
+
|
593
|
+
@ex.run
|
594
|
+
|
595
|
+
cont = @ex.published_result.control
|
596
|
+
cand = @ex.published_result.candidates.first
|
597
|
+
assert_in_delta 0.5, cont.duration, 0.01
|
598
|
+
assert_in_delta 1.0, cand.duration, 0.01
|
599
|
+
end
|
600
|
+
|
601
|
+
it "returns actual durations if fabricated ones are omitted for some blocks" do
|
602
|
+
@ex.use { true }
|
603
|
+
@ex.try { sleep 0.1; true }
|
604
|
+
@ex.fabricate_durations_for_testing_purposes( "control" => 0.5 )
|
605
|
+
|
606
|
+
@ex.run
|
607
|
+
|
608
|
+
cont = @ex.published_result.control
|
609
|
+
cand = @ex.published_result.candidates.first
|
610
|
+
assert_in_delta 0.5, cont.duration, 0.01
|
611
|
+
assert_in_delta 0.1, cand.duration, 0.01
|
612
|
+
end
|
613
|
+
end
|
539
614
|
end
|
data/test/scientist_test.rb
CHANGED
@@ -56,27 +56,29 @@ describe Scientist do
|
|
56
56
|
obj = Object.new
|
57
57
|
obj.extend(Scientist)
|
58
58
|
|
59
|
-
|
60
|
-
experiment = e
|
59
|
+
behaviors_executed = []
|
61
60
|
|
62
|
-
|
63
|
-
e.try("
|
61
|
+
result = obj.science "test", run: "first-way" do |e|
|
62
|
+
e.try("first-way") { behaviors_executed << "first-way" ; true }
|
63
|
+
e.try("second-way") { behaviors_executed << "second-way" ; true }
|
64
64
|
end
|
65
65
|
|
66
66
|
assert_equal true, result
|
67
|
+
assert_equal [ "first-way" ], behaviors_executed
|
67
68
|
end
|
68
69
|
|
69
70
|
it "runs control when there is a nil named test" do
|
70
71
|
obj = Object.new
|
71
72
|
obj.extend(Scientist)
|
72
73
|
|
73
|
-
|
74
|
-
experiment = e
|
74
|
+
behaviors_executed = []
|
75
75
|
|
76
|
-
|
77
|
-
e.
|
76
|
+
result = obj.science "test", nil do |e|
|
77
|
+
e.use { behaviors_executed << "control" ; true }
|
78
|
+
e.try("second-way") { behaviors_executed << "second-way" ; true }
|
78
79
|
end
|
79
80
|
|
80
81
|
assert_equal true, result
|
82
|
+
assert_equal [ "control" ], behaviors_executed
|
81
83
|
end
|
82
84
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: scientist
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.5.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- GitHub Open Source
|
@@ -9,10 +9,10 @@ authors:
|
|
9
9
|
- Rick Bradley
|
10
10
|
- Jesse Toth
|
11
11
|
- Nathan Witmer
|
12
|
-
autorequire:
|
12
|
+
autorequire:
|
13
13
|
bindir: bin
|
14
14
|
cert_chain: []
|
15
|
-
date:
|
15
|
+
date: 2020-09-08 00:00:00.000000000 Z
|
16
16
|
dependencies:
|
17
17
|
- !ruby/object:Gem::Dependency
|
18
18
|
name: minitest
|
@@ -55,6 +55,7 @@ extra_rdoc_files: []
|
|
55
55
|
files:
|
56
56
|
- ".gitignore"
|
57
57
|
- ".travis.yml"
|
58
|
+
- CONTRIBUTING.md
|
58
59
|
- Gemfile
|
59
60
|
- LICENSE.txt
|
60
61
|
- README.md
|
@@ -79,7 +80,7 @@ homepage: https://github.com/github/scientist
|
|
79
80
|
licenses:
|
80
81
|
- MIT
|
81
82
|
metadata: {}
|
82
|
-
post_install_message:
|
83
|
+
post_install_message:
|
83
84
|
rdoc_options: []
|
84
85
|
require_paths:
|
85
86
|
- lib
|
@@ -87,16 +88,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
87
88
|
requirements:
|
88
89
|
- - ">="
|
89
90
|
- !ruby/object:Gem::Version
|
90
|
-
version: '
|
91
|
+
version: '2.3'
|
91
92
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
92
93
|
requirements:
|
93
94
|
- - ">="
|
94
95
|
- !ruby/object:Gem::Version
|
95
96
|
version: '0'
|
96
97
|
requirements: []
|
97
|
-
|
98
|
-
|
99
|
-
signing_key:
|
98
|
+
rubygems_version: 3.1.2
|
99
|
+
signing_key:
|
100
100
|
specification_version: 4
|
101
101
|
summary: Carefully test, measure, and track refactored code.
|
102
102
|
test_files:
|