scientist 1.1.1 → 1.5.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2ad0ca94921ed85fa82b13057d4c68c842d42ea256df543b32196e665c75519c
4
- data.tar.gz: 02f8948568c02e57d003b42a7ca4964e78e1ca14df57b6f204c2db31be5098a9
3
+ metadata.gz: c2831c4fc4f76eb1e8e822a21e613a07e33229bdfe47528cd4555b72ecb05a7c
4
+ data.tar.gz: e0e003dbbe7ecb82749c41aff85ff6b4174e6169dce34081d20f325f62959f8d
5
5
  SHA512:
6
- metadata.gz: 671d170da1739b46f102a898a32b522438f45323cc1d2b788865566bfc2c16d16c6b364c4610a97ee30e4da597642bb4ad9d2c422384eddbab278ae170469f9f
7
- data.tar.gz: 5334f5ae2d735cdc3b230bfefd469bd51c9e9363ec5cec83d45e99cbb2b3b55fa14f057dc59dc7909828dc0182c9168c9a86d47efe5529971206b9ff5c14a52a
6
+ metadata.gz: 07f99bcd0d9619205f42f15f16cc00b31fff3fab127344ca70360c51eea28d3eb15bc21e9471edbccfb06ddf2f07bc93ac0024af3899371486757fb53dbb5710
7
+ data.tar.gz: cb7a3cf609e501a5d4a7894cf62400dacde1dc654ec7caf5d9beb8d115bb4f0c1e217cdeddd7c72863385459bfd7bb929221df74725312c3dfe21af315598486
@@ -3,12 +3,10 @@ language: ruby
3
3
  cache: bundler
4
4
  script: script/test
5
5
  rvm:
6
- - 1.9.3
7
- - 2.0.0
8
- - 2.1.8
9
- - 2.2.4
10
- - 2.3.0
11
- - 2.4.0
6
+ - 2.3.8
7
+ - 2.4.5
8
+ - 2.5.3
9
+ - 2.6.1
12
10
  before_install: gem install bundler
13
11
  addons:
14
12
  apt:
@@ -0,0 +1,26 @@
1
+ ## Contributing
2
+
3
+ [fork]: https://github.com/github/scientist/fork
4
+ [pr]: https://github.com/github/scientist/compare
5
+
6
+ Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
7
+
8
+ Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE.txt).
9
+
10
+ ## Submitting a pull request
11
+
12
+ 0. [Fork][fork] and clone the repository
13
+ 0. Create a new branch: `git checkout -b my-branch-name`
14
+ 0. Make your change, push to your fork and [submit a pull request][pr]
15
+ 0. Pat your self on the back and wait for your pull request to be reviewed and merged.
16
+
17
+ Here are a few things you can do that will increase the likelihood of your pull request being accepted:
18
+
19
+ - Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
20
+ - Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
21
+
22
+ ## Resources
23
+
24
+ - [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
25
+ - [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
26
+ - [GitHub Help](https://help.github.com)
data/README.md CHANGED
@@ -26,7 +26,7 @@ Wrap a `use` block around the code's original behavior, and wrap `try` around th
26
26
  * Randomizes the order in which `use` and `try` blocks are run,
27
27
  * Measures the durations of all behaviors,
28
28
  * Compares the result of `try` to the result of `use`,
29
- * Swallows (but records) any exceptions raised in the `try` block, and
29
+ * Swallow and record exceptions raised in the `try` block when overriding `raised`, and
30
30
  * Publishes all this information.
31
31
 
32
32
  The `use` block is called the **control**. The `try` block is called the **candidate**.
@@ -52,7 +52,7 @@ If you don't declare any `try` blocks, none of the Scientist machinery is invoke
52
52
 
53
53
  ## Making science useful
54
54
 
55
- The examples above will run, but they're not really *doing* anything. The `try` blocks run every time and none of the results get published. Replace the default experiment implementation to control execution and reporting:
55
+ The examples above will run, but they're not really *doing* anything. The `try` blocks don't run yet and none of the results get published. Replace the default experiment implementation to control execution and reporting:
56
56
 
57
57
  ```ruby
58
58
  require "scientist/experiment"
@@ -62,7 +62,7 @@ class MyExperiment
62
62
 
63
63
  attr_accessor :name
64
64
 
65
- def initialize(name:)
65
+ def initialize(name)
66
66
  @name = name
67
67
  end
68
68
 
@@ -71,18 +71,17 @@ class MyExperiment
71
71
  true
72
72
  end
73
73
 
74
+ def raised(operation, error)
75
+ # see "In a Scientist callback" below
76
+ p "Operation '#{operation}' failed with error '#{error.inspect}'"
77
+ super # will re-raise
78
+ end
79
+
74
80
  def publish(result)
75
81
  # see "Publishing results" below
76
82
  p result
77
83
  end
78
84
  end
79
-
80
- # replace `Scientist::Default` as the default implementation
81
- module Scientist::Experiment
82
- def self.new(name)
83
- MyExperiment.new(name: name)
84
- end
85
- end
86
85
  ```
87
86
 
88
87
  Now calls to the `science` helper will load instances of `MyExperiment`.
@@ -206,6 +205,8 @@ class MyExperiment
206
205
  end
207
206
  ```
208
207
 
208
+ Note that the `#clean` method will discard the previous cleaner block if you call it again. If for some reason you need to access the currently configured cleaner block, `Scientist::Experiment#cleaner` will return the block without further ado. _(This probably won't come up in normal usage, but comes in handy if you're writing, say, a custom experiment runner that provides default cleaners.)_
209
+
209
210
  ### Ignoring mismatches
210
211
 
211
212
  During the early stages of an experiment, it's possible that some of your code will always generate a mismatch for reasons you know and understand but haven't yet fixed. Instead of these known cases always showing up as mismatches in your metrics or analysis, you can tell an experiment whether or not to ignore a mismatch using the `ignore` method. You may include more than one block if needed:
@@ -254,7 +255,7 @@ class MyExperiment
254
255
 
255
256
  attr_accessor :name, :percent_enabled
256
257
 
257
- def initialize(name:)
258
+ def initialize(name)
258
259
  @name = name
259
260
  @percent_enabled = 100
260
261
  end
@@ -327,7 +328,7 @@ class MyExperiment
327
328
  }
328
329
  else
329
330
  {
330
- # see "Keeping it clean" below
331
+ # see "Keeping it clean" above
331
332
  :value => observation.cleaned_value
332
333
  }
333
334
  end
@@ -352,6 +353,35 @@ MyExperiment.raise_on_mismatches = true
352
353
 
353
354
  Scientist will raise a `Scientist::Experiment::MismatchError` exception if any observations don't match.
354
355
 
356
+ #### Custom mismatch errors
357
+
358
+ To instruct Scientist to raise a custom error instead of the default `Scientist::Experiment::MismatchError`:
359
+
360
+ ```ruby
361
+ class CustomMismatchError < Scientist::Experiment::MismatchError
362
+ def to_s
363
+ message = "There was a mismatch! Here's the diff:"
364
+
365
+ diffs = result.candidates.map do |candidate|
366
+ Diff.new(result.control, candidate)
367
+ end.join("\n")
368
+
369
+ "#{message}\n#{diffs}"
370
+ end
371
+ end
372
+ ```
373
+
374
+ ```ruby
375
+ science "widget-permissions" do |e|
376
+ e.use { Report.find(id) }
377
+ e.try { ReportService.new.fetch(id) }
378
+
379
+ e.raise_with CustomMismatchError
380
+ end
381
+ ```
382
+
383
+ This allows for pre-processing on mismatch error exception messages.
384
+
355
385
  ### Handling errors
356
386
 
357
387
  #### In candidate code
@@ -462,6 +492,22 @@ science "various-ways", run: "first-way" do |e|
462
492
  end
463
493
  ```
464
494
 
495
+ #### Providing fake timing data
496
+
497
+ If you're writing tests that depend on specific timing values, you can provide canned durations using the `fabricate_durations_for_testing_purposes` method, and Scientist will report these in `Scientist::Observation#duration` instead of the actual execution times.
498
+
499
+ ```ruby
500
+ science "absolutely-nothing-suspicious-happening-here" do |e|
501
+ e.use { ... } # "control"
502
+ e.try { ... } # "candidate"
503
+ e.fabricate_durations_for_testing_purposes( "control" => 1.0, "candidate" => 0.5 )
504
+ end
505
+ ```
506
+
507
+ `fabricate_durations_for_testing_purposes` takes a Hash of duration values, keyed by behavior names. (By default, Scientist uses `"control"` and `"candidate"`, but if you override these as shown in [Trying more than one thing](#trying-more-than-one-thing) or [No control, just candidates](#no-control-just-candidates), use matching names here.) If a name is not provided, the actual execution time will be reported instead.
508
+
509
+ _Like `Scientist::Experiment#cleaner`, this probably won't come up in normal usage. It's here to make it easier to test code that extends Scientist._
510
+
465
511
  ### Without including Scientist
466
512
 
467
513
  If you need to use Scientist in a place where you aren't able to include the Scientist module, you can call `Scientist.run`:
@@ -475,17 +521,23 @@ end
475
521
 
476
522
  ## Hacking
477
523
 
478
- Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically. Science requires Ruby 1.9 or newer.
524
+ Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs the unit tests. All development dependencies are installed automatically. Scientist requires Ruby 2.3 or newer.
525
+
526
+ ## Wrappers
527
+
528
+ - [RealGeeks/lab_tech](https://github.com/RealGeeks/lab_tech) is a Rails engine for using this library by controlling, storing, and analyzing experiment results with ActiveRecord.
479
529
 
480
530
  ## Alternatives
481
531
 
482
532
  - [daylerees/scientist](https://github.com/daylerees/scientist) (PHP)
483
- - [github/scientist.net](https://github.com/github/scientist.net) (.NET)
533
+ - [scientistproject/scientist.net](https://github.com/scientistproject/Scientist.net) (.NET)
484
534
  - [joealcorn/laboratory](https://github.com/joealcorn/laboratory) (Python)
485
535
  - [rawls238/Scientist4J](https://github.com/rawls238/Scientist4J) (Java)
486
536
  - [tomiaijo/scientist](https://github.com/tomiaijo/scientist) (C++)
487
537
  - [trello/scientist](https://github.com/trello/scientist) (node.js)
488
538
  - [ziyasal/scientist.js](https://github.com/ziyasal/scientist.js) (node.js, ES6)
539
+ - [TrueWill/tzientist](https://github.com/TrueWill/tzientist) (node.js, TypeScript)
540
+ - [TrueWill/paleontologist](https://github.com/TrueWill/paleontologist) (Deno, TypeScript)
489
541
  - [yeller/laboratory](https://github.com/yeller/laboratory) (Clojure)
490
542
  - [lancew/Scientist](https://github.com/lancew/Scientist) (Perl 5)
491
543
  - [lancew/ScientistP6](https://github.com/lancew/ScientistP6) (Perl 6)
@@ -494,7 +546,8 @@ Be on a Unixy box. Make sure a modern Bundler is available. `script/test` runs t
494
546
  - [calavera/go-scientist](https://github.com/calavera/go-scientist) (Go)
495
547
  - [jelmersnoeck/experiment](https://github.com/jelmersnoeck/experiment) (Go)
496
548
  - [spoptchev/scientist](https://github.com/spoptchev/scientist) (Kotlin / Java)
497
-
549
+ - [junkpiano/scientist](https://github.com/junkpiano/scientist) (Swift)
550
+ - [serverless scientist](http://serverlessscientist.com/) (AWS Lambda)
498
551
 
499
552
  ## Maintainers
500
553
 
@@ -1,5 +1,28 @@
1
1
  # Changes
2
2
 
3
+ ## v1.3.0 (2 April 2019)
4
+
5
+ - New: Drop support for ruby <2.3
6
+ - Fix: Build new strings instead of modifying frozen ones
7
+ - New: Add an accessor for the configured clean block
8
+ - New: Add a hook to use fabricated durations instead of actual timing data.
9
+
10
+ ## v1.2.0 (5 July 2018)
11
+
12
+ - New: Use monotonic clock for duration calculations
13
+ - New: Drop support for ruby <2.1 to support monotonic clock
14
+ - New: Run CI on Ruby 2.5
15
+
16
+ ## v1.1.2 (9 May 2018)
17
+
18
+ - New: Add `raise_with` option to allow for custom mismatch errors to be raised
19
+
20
+ ## v1.1.1 (6 February 2018)
21
+
22
+ - Fix: default experiment no longer runs all `try` paths
23
+ - New: Add `Scientist.run` module method for running experiments when an included module isn't available
24
+ - New: Add [Noise and error rates](https://github.com/github/scientist#noise-and-error-rates) to `README.md`
25
+
3
26
  ## v1.1.0 (29 August 2017)
4
27
 
5
28
  - New: [Specify which exception types to rescue](https://github.com/github/scientist#in-candidate-code)
@@ -9,16 +9,26 @@ module Scientist::Experiment
9
9
  # If this is nil, raise_on_mismatches class attribute is used instead.
10
10
  attr_accessor :raise_on_mismatches
11
11
 
12
- # Create a new instance of a class that implements the Scientist::Experiment
13
- # interface.
14
- #
15
- # Override this method directly to change the default implementation.
12
+ def self.included(base)
13
+ self.set_default(base)
14
+ base.extend RaiseOnMismatch
15
+ end
16
+
17
+ # Instantiate a new experiment (using the class given to the .set_default method).
16
18
  def self.new(name)
17
- Scientist::Default.new(name)
19
+ (@experiment_klass || Scientist::Default).new(name)
20
+ end
21
+
22
+ # Configure Scientist to use the given class for all future experiments
23
+ # (must implement the Scientist::Experiment interface).
24
+ #
25
+ # Called automatically when new experiments are defined.
26
+ def self.set_default(klass)
27
+ @experiment_klass = klass
18
28
  end
19
29
 
20
30
  # A mismatch, raised when raise_on_mismatches is enabled.
21
- class MismatchError < StandardError
31
+ class MismatchError < Exception
22
32
  attr_reader :name, :result
23
33
 
24
34
  def initialize(name, result)
@@ -41,10 +51,10 @@ module Scientist::Experiment
41
51
  def format_observation(observation)
42
52
  observation.name + ":\n" +
43
53
  if observation.raised?
44
- observation.exception.inspect.prepend(" ") + "\n" +
45
- observation.exception.backtrace.map { |line| line.prepend(" ") }.join("\n")
54
+ lines = observation.exception.backtrace.map { |line| " #{line}" }.join("\n")
55
+ " #{observation.exception.inspect}" + "\n" + lines
46
56
  else
47
- observation.value.inspect.prepend(" ")
57
+ " #{observation.cleaned_value.inspect}"
48
58
  end
49
59
  end
50
60
  end
@@ -67,10 +77,6 @@ module Scientist::Experiment
67
77
  end
68
78
  end
69
79
 
70
- def self.included(base)
71
- base.extend RaiseOnMismatch
72
- end
73
-
74
80
  # Define a block of code to run before an experiment begins, if the experiment
75
81
  # is enabled.
76
82
  #
@@ -96,6 +102,13 @@ module Scientist::Experiment
96
102
  @_scientist_cleaner = block
97
103
  end
98
104
 
105
+ # Accessor for the clean block, if one is available.
106
+ #
107
+ # Returns the configured block, or nil.
108
+ def cleaner
109
+ @_scientist_cleaner
110
+ end
111
+
99
112
  # Internal: Clean a value with the configured clean block, or return the value
100
113
  # if no clean block is configured.
101
114
  #
@@ -176,6 +189,10 @@ module Scientist::Experiment
176
189
  false
177
190
  end
178
191
 
192
+ def raise_with(exception)
193
+ @_scientist_custom_mismatch_error = exception
194
+ end
195
+
179
196
  # Called when an exception is raised while running an internal operation,
180
197
  # like :publish. Override this method to track these exceptions. The
181
198
  # default implementation re-raises the exception.
@@ -205,16 +222,7 @@ module Scientist::Experiment
205
222
  @_scientist_before_run.call
206
223
  end
207
224
 
208
- observations = []
209
-
210
- behaviors.keys.shuffle.each do |key|
211
- block = behaviors[key]
212
- observations << Scientist::Observation.new(key, self, &block)
213
- end
214
-
215
- control = observations.detect { |o| o.name == name }
216
-
217
- result = Scientist::Result.new self, observations, control
225
+ result = generate_result(name)
218
226
 
219
227
  begin
220
228
  publish(result)
@@ -223,14 +231,16 @@ module Scientist::Experiment
223
231
  end
224
232
 
225
233
  if raise_on_mismatches? && result.mismatched?
226
- raise MismatchError.new(self.name, result)
234
+ if @_scientist_custom_mismatch_error
235
+ raise @_scientist_custom_mismatch_error.new(self.name, result)
236
+ else
237
+ raise MismatchError.new(self.name, result)
238
+ end
227
239
  end
228
240
 
229
- if control.raised?
230
- raise control.exception
231
- else
232
- control.value
233
- end
241
+ control = result.control
242
+ raise control.exception if control.raised?
243
+ control.value
234
244
  end
235
245
 
236
246
  # Define a block that determines whether or not the experiment should run.
@@ -282,4 +292,24 @@ module Scientist::Experiment
282
292
  !!raise_on_mismatches
283
293
  end
284
294
  end
295
+
296
+ # Provide predefined durations to use instead of actual timing data.
297
+ # This is here solely as a convenience for developers of libraries that extend Scientist.
298
+ def fabricate_durations_for_testing_purposes(fabricated_durations = {})
299
+ @_scientist_fabricated_durations = fabricated_durations
300
+ end
301
+
302
+ # Internal: Generate the observations and create the result from those and the control.
303
+ def generate_result(name)
304
+ observations = []
305
+
306
+ behaviors.keys.shuffle.each do |key|
307
+ block = behaviors[key]
308
+ fabricated_duration = @_scientist_fabricated_durations && @_scientist_fabricated_durations[key]
309
+ observations << Scientist::Observation.new(key, self, fabricated_duration: fabricated_duration, &block)
310
+ end
311
+
312
+ control = observations.detect { |o| o.name == name }
313
+ Scientist::Result.new(self, observations, control)
314
+ end
285
315
  end
@@ -8,9 +8,6 @@ class Scientist::Observation
8
8
  # The experiment this observation is for
9
9
  attr_reader :experiment
10
10
 
11
- # The instant observation began.
12
- attr_reader :now
13
-
14
11
  # The String name of the behavior.
15
12
  attr_reader :name
16
13
 
@@ -23,18 +20,19 @@ class Scientist::Observation
23
20
  # The Float seconds elapsed.
24
21
  attr_reader :duration
25
22
 
26
- def initialize(name, experiment, &block)
23
+ def initialize(name, experiment, fabricated_duration: nil, &block)
27
24
  @name = name
28
25
  @experiment = experiment
29
- @now = Time.now
30
26
 
27
+ starting = Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second) unless fabricated_duration
31
28
  begin
32
29
  @value = block.call
33
30
  rescue *RESCUES => e
34
31
  @exception = e
35
32
  end
36
33
 
37
- @duration = (Time.now - @now).to_f
34
+ @duration = fabricated_duration ||
35
+ Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_second) - starting
38
36
 
39
37
  freeze
40
38
  end
@@ -1,3 +1,3 @@
1
1
  module Scientist
2
- VERSION = "1.1.1"
2
+ VERSION = "1.5.0"
3
3
  end
@@ -10,6 +10,8 @@ Gem::Specification.new do |gem|
10
10
  gem.homepage = "https://github.com/github/scientist"
11
11
  gem.license = "MIT"
12
12
 
13
+ gem.required_ruby_version = '>= 2.3'
14
+
13
15
  gem.files = `git ls-files`.split($/)
14
16
  gem.executables = []
15
17
  gem.test_files = gem.files.grep(/^test/)
@@ -4,7 +4,7 @@
4
4
  set -e
5
5
 
6
6
  cd $(dirname "$0")/..
7
- script/bootstrap && ruby -I lib \
7
+ script/bootstrap && bundle exec ruby -I lib \
8
8
  -e 'require "bundler/setup"' \
9
9
  -e 'require "coveralls"; Coveralls.wear!{ add_filter ".bundle" }' \
10
10
  -e 'require "minitest/autorun"' \
@@ -2,6 +2,9 @@ describe Scientist::Experiment do
2
2
  class Fake
3
3
  include Scientist::Experiment
4
4
 
5
+ # Undo auto-config magic / preserve default behavior of Scientist::Experiment.new
6
+ Scientist::Experiment.set_default(nil)
7
+
5
8
  def initialize(*args)
6
9
  end
7
10
 
@@ -239,6 +242,13 @@ describe Scientist::Experiment do
239
242
  assert_equal 10, @ex.clean_value(10)
240
243
  end
241
244
 
245
+ it "provides the clean block when asked for it, in case subclasses wish to override and provide defaults" do
246
+ assert_nil @ex.cleaner
247
+ cleaner = ->(value) { value.upcase }
248
+ @ex.clean(&cleaner)
249
+ assert_equal cleaner, @ex.cleaner
250
+ end
251
+
242
252
  it "calls the configured clean block with a value when configured" do
243
253
  @ex.clean do |value|
244
254
  value.upcase
@@ -262,6 +272,19 @@ describe Scientist::Experiment do
262
272
  assert_equal "kaboom", exception.message
263
273
  end
264
274
 
275
+ describe "#raise_with" do
276
+ it "raises custom error if provided" do
277
+ CustomError = Class.new(Scientist::Experiment::MismatchError)
278
+
279
+ @ex.use { 1 }
280
+ @ex.try { 2 }
281
+ @ex.raise_with(CustomError)
282
+ @ex.raise_on_mismatches = true
283
+
284
+ assert_raises(CustomError) { @ex.run }
285
+ end
286
+ end
287
+
265
288
  describe "#run_if" do
266
289
  it "does not run the experiment if the given block returns false" do
267
290
  candidate_ran = false
@@ -392,6 +415,16 @@ describe Scientist::Experiment do
392
415
  assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
393
416
  end
394
417
 
418
+ it "cleans values when raising on observation mismatch" do
419
+ Fake.raise_on_mismatches = true
420
+ @ex.use { "fine" }
421
+ @ex.try { "not fine" }
422
+ @ex.clean { "So Clean" }
423
+
424
+ err = assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
425
+ assert_match /So Clean/, err.message
426
+ end
427
+
395
428
  it "doesn't raise when there is a mismatch if raise on mismatches is disabled" do
396
429
  Fake.raise_on_mismatches = false
397
430
  @ex.use { "fine" }
@@ -414,6 +447,20 @@ describe Scientist::Experiment do
414
447
  assert_raises(Scientist::Experiment::MismatchError) { @ex.run }
415
448
  end
416
449
 
450
+ it "allows MismatchError to bubble up through bare rescues" do
451
+ Fake.raise_on_mismatches = true
452
+ @ex.use { "control" }
453
+ @ex.try { "candidate" }
454
+ runner = -> {
455
+ begin
456
+ @ex.run
457
+ rescue
458
+ # StandardError handled
459
+ end
460
+ }
461
+ assert_raises(Scientist::Experiment::MismatchError) { runner.call }
462
+ end
463
+
417
464
  describe "#raise_on_mismatches?" do
418
465
  it "raises when there is a mismatch if the experiment instance's raise on mismatches is enabled" do
419
466
  Fake.raise_on_mismatches = false
@@ -536,4 +583,32 @@ candidate:
536
583
  refute before, "before_run should not have run"
537
584
  end
538
585
  end
586
+
587
+ describe "testing hooks for extending code" do
588
+ it "allows a user to provide fabricated durations for testing purposes" do
589
+ @ex.use { true }
590
+ @ex.try { true }
591
+ @ex.fabricate_durations_for_testing_purposes( "control" => 0.5, "candidate" => 1.0 )
592
+
593
+ @ex.run
594
+
595
+ cont = @ex.published_result.control
596
+ cand = @ex.published_result.candidates.first
597
+ assert_in_delta 0.5, cont.duration, 0.01
598
+ assert_in_delta 1.0, cand.duration, 0.01
599
+ end
600
+
601
+ it "returns actual durations if fabricated ones are omitted for some blocks" do
602
+ @ex.use { true }
603
+ @ex.try { sleep 0.1; true }
604
+ @ex.fabricate_durations_for_testing_purposes( "control" => 0.5 )
605
+
606
+ @ex.run
607
+
608
+ cont = @ex.published_result.control
609
+ cand = @ex.published_result.candidates.first
610
+ assert_in_delta 0.5, cont.duration, 0.01
611
+ assert_in_delta 0.1, cand.duration, 0.01
612
+ end
613
+ end
539
614
  end
@@ -56,27 +56,29 @@ describe Scientist do
56
56
  obj = Object.new
57
57
  obj.extend(Scientist)
58
58
 
59
- result = obj.science "test", run: "first-way" do |e|
60
- experiment = e
59
+ behaviors_executed = []
61
60
 
62
- e.try("first-way") { true }
63
- e.try("second-way") { true }
61
+ result = obj.science "test", run: "first-way" do |e|
62
+ e.try("first-way") { behaviors_executed << "first-way" ; true }
63
+ e.try("second-way") { behaviors_executed << "second-way" ; true }
64
64
  end
65
65
 
66
66
  assert_equal true, result
67
+ assert_equal [ "first-way" ], behaviors_executed
67
68
  end
68
69
 
69
70
  it "runs control when there is a nil named test" do
70
71
  obj = Object.new
71
72
  obj.extend(Scientist)
72
73
 
73
- result = obj.science "test", nil do |e|
74
- experiment = e
74
+ behaviors_executed = []
75
75
 
76
- e.use { true }
77
- e.try("second-way") { true }
76
+ result = obj.science "test", nil do |e|
77
+ e.use { behaviors_executed << "control" ; true }
78
+ e.try("second-way") { behaviors_executed << "second-way" ; true }
78
79
  end
79
80
 
80
81
  assert_equal true, result
82
+ assert_equal [ "control" ], behaviors_executed
81
83
  end
82
84
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: scientist
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.1.1
4
+ version: 1.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - GitHub Open Source
@@ -9,10 +9,10 @@ authors:
9
9
  - Rick Bradley
10
10
  - Jesse Toth
11
11
  - Nathan Witmer
12
- autorequire:
12
+ autorequire:
13
13
  bindir: bin
14
14
  cert_chain: []
15
- date: 2018-02-06 00:00:00.000000000 Z
15
+ date: 2020-09-08 00:00:00.000000000 Z
16
16
  dependencies:
17
17
  - !ruby/object:Gem::Dependency
18
18
  name: minitest
@@ -55,6 +55,7 @@ extra_rdoc_files: []
55
55
  files:
56
56
  - ".gitignore"
57
57
  - ".travis.yml"
58
+ - CONTRIBUTING.md
58
59
  - Gemfile
59
60
  - LICENSE.txt
60
61
  - README.md
@@ -79,7 +80,7 @@ homepage: https://github.com/github/scientist
79
80
  licenses:
80
81
  - MIT
81
82
  metadata: {}
82
- post_install_message:
83
+ post_install_message:
83
84
  rdoc_options: []
84
85
  require_paths:
85
86
  - lib
@@ -87,16 +88,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
87
88
  requirements:
88
89
  - - ">="
89
90
  - !ruby/object:Gem::Version
90
- version: '0'
91
+ version: '2.3'
91
92
  required_rubygems_version: !ruby/object:Gem::Requirement
92
93
  requirements:
93
94
  - - ">="
94
95
  - !ruby/object:Gem::Version
95
96
  version: '0'
96
97
  requirements: []
97
- rubyforge_project:
98
- rubygems_version: 2.7.3
99
- signing_key:
98
+ rubygems_version: 3.1.2
99
+ signing_key:
100
100
  specification_version: 4
101
101
  summary: Carefully test, measure, and track refactored code.
102
102
  test_files: