lab_coat 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +3 -0
- data/README.md +91 -30
- data/lib/lab_coat/observation.rb +5 -0
- data/lib/lab_coat/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d819fd05745d82590e7d0f8c2e05f415b3ec6bea1cdc5def244fcfee17b969c7
|
4
|
+
data.tar.gz: 27819f2af7099608845196b00ae89a0ba59e9bdb8224814585a659601a261e76
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 711d1510f5d1f4d59a57830567988d74fad50bfe7c4e548cd76e67d3f46bb2dfbbff503914e6129f7023032f7986c2153a1359e600d1f7d2e6aa443d98f27c0d
|
7
|
+
data.tar.gz: 66ed5879e1e803045806464c6b708b1a355c0d997965e1a6ea43e42b789d4f2904b7c39223b60101c08a1857c35dba72003d9463102962da27d9f39b181c1636
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -7,7 +7,7 @@ A simple experiment library to safely test new code paths.
|
|
7
7
|
This library is heavily inspired by [Scientist](https://github.com/github/scientist), with some key differences:
|
8
8
|
- `Experiments` are `classes`, not `modules` which means they are stateful by default.
|
9
9
|
- There is no app wide default experiment that gets magically set.
|
10
|
-
- The `Result` only supports one comparison at a time, i.e. only 1 `candidate` is allowed.
|
10
|
+
- The `Result` only supports one comparison at a time, i.e. only 1 `candidate` is allowed per run.
|
11
11
|
|
12
12
|
## Installation
|
13
13
|
|
@@ -23,7 +23,26 @@ If bundler is not being used to manage dependencies, install the gem by executin
|
|
23
23
|
|
24
24
|
### Create an `Experiment`
|
25
25
|
|
26
|
-
To do some science, i.e. test out a new code path, start by defining an `Experiment`. An experiment is any class that inherits from `LabCoat::Experiment` and implements the required methods.
|
26
|
+
To do some science, i.e. test out a new code path, start by defining an `Experiment`. An experiment is any class that inherits from `LabCoat::Experiment` and implements the [required methods](#required-methods).
|
27
|
+
|
28
|
+
```ruby
|
29
|
+
# your_experiment.rb
|
30
|
+
class YourExperiment < LabCoat::Experiment
|
31
|
+
def control
|
32
|
+
expensive_query.first
|
33
|
+
end
|
34
|
+
|
35
|
+
def candidate
|
36
|
+
refactored_version_of_the_query.first
|
37
|
+
end
|
38
|
+
|
39
|
+
def enabled?
|
40
|
+
true
|
41
|
+
end
|
42
|
+
end
|
43
|
+
```
|
44
|
+
|
45
|
+
The base initializer for an `Experiment` requires a `name` argument; it's a good idea to name your experiments.
|
27
46
|
|
28
47
|
#### Required methods
|
29
48
|
|
@@ -31,11 +50,14 @@ See the [`Experiment`](lib/lab_coat/experiment.rb) class for more details.
|
|
31
50
|
|
32
51
|
|Method|Description|
|
33
52
|
|---|---|
|
34
|
-
|`enabled?`|Returns a `Boolean` that controls whether or not the experiment runs.|
|
35
|
-
|`control`|The existing or default behavior. This will always be returned from `#run!`.|
|
36
53
|
|`candidate`|The new behavior you want to test.|
|
54
|
+
|`control`|The existing or default behavior. This will always be returned from `#run!`.|
|
55
|
+
|`enabled?`|Returns a `Boolean` that controls whether or not the experiment runs.|
|
37
56
|
|`publish!`|This is not _technically_ required, but `Experiments` are not useful unless you can analyze the results. Override this method to record the `Result` however you wish.|
|
38
57
|
|
58
|
+
> [!TIP]
|
59
|
+
> The `#run!` method accepts arbitrary arguments and forwards them to `enabled?`, `control`, and `candidate` in case you need to provide data at runtime.
|
60
|
+
|
39
61
|
#### Additional methods
|
40
62
|
|
41
63
|
|Method|Description|
|
@@ -44,13 +66,8 @@ See the [`Experiment`](lib/lab_coat/experiment.rb) class for more details.
|
|
44
66
|
|`ignore?`|Whether or not the result should be ignored. Ignored `Results` are still passed to `#publish!`|
|
45
67
|
|`raised`|Callback method that's called when an `Observation` raises.|
|
46
68
|
|
47
|
-
|
48
|
-
|
49
|
-
```ruby
|
50
|
-
# application_experiment.rb
|
51
|
-
class ApplicationExperiment < LabCoat::Experiment
|
52
|
-
end
|
53
|
-
```
|
69
|
+
> [!TIP]
|
70
|
+
> You should create a shared base class(es) to maintain consistency across experiments within your app.
|
54
71
|
|
55
72
|
You may want to give your experiment some context, or state. You can do this via an initializer or writer methods just like any other Ruby class.
|
56
73
|
|
@@ -66,7 +83,7 @@ class ApplicationExperiment < LabCoat::Experiment
|
|
66
83
|
end
|
67
84
|
```
|
68
85
|
|
69
|
-
You likely want to `publish!` all experiments in a
|
86
|
+
You likely want to `publish!` all experiments in a consistent way, so that you can analyze the data and make decisions. New `Experiment` authors should not have to redo the "plumbing" between your experimentation framework (e.g. `LabCoat`) and your observability (o11y) process.
|
70
87
|
|
71
88
|
```ruby
|
72
89
|
# application_experiment.rb
|
@@ -84,7 +101,7 @@ class ApplicationExperiment < LabCoat::Experiment
|
|
84
101
|
end
|
85
102
|
```
|
86
103
|
|
87
|
-
You might also have a common way to enable experiments such as a feature flag system and/or common guards you want to enforce application wide.
|
104
|
+
You might also have a common way to enable experiments such as a feature flag system and/or common guards you want to enforce application wide. These might come from a mix of services and the `Experiment`'s state.
|
88
105
|
|
89
106
|
```ruby
|
90
107
|
# application_experiment.rb
|
@@ -97,17 +114,18 @@ end
|
|
97
114
|
|
98
115
|
### Make some `Observations` via `run!`
|
99
116
|
|
100
|
-
You don't have to create an `Observation` yourself; that happens automatically when you call `Experiment#run!`.
|
117
|
+
You don't have to create an `Observation` yourself; that happens automatically when you call `Experiment#run!`. The control and candidate `Observations` are packaged into a `Result` and [passed to `Experiment#publish!`](#publish-the-result).
|
101
118
|
|
102
119
|
|Attribute|Description|
|
103
120
|
|---|---|
|
104
|
-
|`name`|Either `"control"` or `"candidate"`.|
|
105
|
-
|`experiment`|The `Experiment` instance this `Result` is for.|
|
106
121
|
|`duration`|The duration of the run in `float` seconds.|
|
107
|
-
|`
|
122
|
+
|`error`|If the code path raised, the thrown exception is stored here.|
|
123
|
+
|`experiment`|The `Experiment` instance this `Result` is for.|
|
124
|
+
|`name`|Either `"control"` or `"candidate"`.|
|
108
125
|
|`publishable_value`|A publishable representation of the `value`, as defined by `Experiment#publishable_value`.|
|
109
126
|
|`raised?`|Whether or not the code path raised.|
|
110
|
-
|`
|
127
|
+
|`slug`|A combination of the `Experiment#name` and `Observation#name`, e.g. `"experiment_name.control"`|
|
128
|
+
|`value`|The return value of the observed code path.|
|
111
129
|
|
112
130
|
`Observation` instances are passed to many of the `Experiment` methods that you may override.
|
113
131
|
|
@@ -151,11 +169,11 @@ A `Result` represents a single run of an `Experiment`.
|
|
151
169
|
|
152
170
|
|Attribute|Description|
|
153
171
|
|---|---|
|
154
|
-
|`experiment`|The `Experiment` instance this `Result` is for.|
|
155
|
-
|`control`|An `Observation` instance representing the `Experiment#control` behavior|
|
156
172
|
|`candidate`|An `Observation` instance representing the `Experiment#candidate` behavior|
|
157
|
-
|`
|
173
|
+
|`control`|An `Observation` instance representing the `Experiment#control` behavior|
|
174
|
+
|`experiment`|The `Experiment` instance this `Result` is for.|
|
158
175
|
|`ignored?`|Whether or not the result should be ignored, as defined by `Experiment#ignore?`|
|
176
|
+
|`matched?`|Whether or not the `control` and `candidate` match, as defined by `Experiment#compare`|
|
159
177
|
|
160
178
|
The `Result` is passed to your implementation of `#publish!` when an `Experiment` is finished running.
|
161
179
|
|
@@ -169,22 +187,65 @@ def publish!(result)
|
|
169
187
|
if result.matched?
|
170
188
|
puts "😎"
|
171
189
|
else
|
190
|
+
control = result.control
|
191
|
+
candidate = result.candidate
|
172
192
|
puts <<~MISMATCH
|
173
193
|
😮
|
174
194
|
|
175
|
-
|
176
|
-
Value: #{
|
177
|
-
Duration: #{
|
178
|
-
Error: #{
|
195
|
+
#{control.slug}
|
196
|
+
Value: #{control.publishable_value}
|
197
|
+
Duration: #{control.duration}
|
198
|
+
Error: #{control.error&.message}
|
179
199
|
|
180
|
-
|
181
|
-
Value: #{
|
182
|
-
Duration: #{
|
183
|
-
Error: #{
|
200
|
+
#{candidate.slug}
|
201
|
+
Value: #{candidate.publishable_value}
|
202
|
+
Duration: #{candidate.duration}
|
203
|
+
Error: #{candidate.error&.message}
|
184
204
|
MISMATCH
|
185
205
|
end
|
186
206
|
end
|
187
207
|
```
|
208
|
+
Running a mismatched experiment with this implementation of `publish!` would produce:
|
209
|
+
|
210
|
+
```
|
211
|
+
😮
|
212
|
+
|
213
|
+
my_experiment.control
|
214
|
+
Value: 420
|
215
|
+
Duration: 12.934
|
216
|
+
Error:
|
217
|
+
|
218
|
+
my_experiment.candidate
|
219
|
+
Value: 69
|
220
|
+
Duration: 9.702
|
221
|
+
Error:
|
222
|
+
```
|
223
|
+
|
224
|
+
### Standalone `Observations`
|
225
|
+
|
226
|
+
The `Observation` class can be used as a standalone wrapper for any code that you want to experiment with. Instantiating an `Observation` automatically:
|
227
|
+
- measures the duration of the code block
|
228
|
+
- captures the return value of the code block
|
229
|
+
- rescues and stores any errors raised by the code block
|
230
|
+
|
231
|
+
```ruby
|
232
|
+
10.times do |i|
|
233
|
+
observation = Observation.new("test-#{i}", nil) do
|
234
|
+
some_code_path
|
235
|
+
end
|
236
|
+
|
237
|
+
puts "#{observation.name} results:"
|
238
|
+
if observation.raised?
|
239
|
+
puts "error: #{observation.error.message}"
|
240
|
+
else
|
241
|
+
puts "duration: #{observation.duration}"
|
242
|
+
puts "succeeded: #{!observation.raised?}"
|
243
|
+
end
|
244
|
+
end
|
245
|
+
```
|
246
|
+
|
247
|
+
> [!WARNING]
|
248
|
+
> Be careful when using `Observation` instances without an `Experiment` set. Some methods like `#publishable_value` and `#slug` depend on an `experiment` and may raise an error when called.
|
188
249
|
|
189
250
|
## Development
|
190
251
|
|
@@ -194,7 +255,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
|
|
194
255
|
|
195
256
|
## Contributing
|
196
257
|
|
197
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
258
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/omkarmoghe/lab_coat.
|
198
259
|
|
199
260
|
## License
|
200
261
|
|
data/lib/lab_coat/observation.rb
CHANGED
data/lib/lab_coat/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: lab_coat
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Omkar Moghe
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-04-
|
11
|
+
date: 2024-04-09 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: minitest
|