field_test 0.2.4 → 0.4.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +46 -8
- data/LICENSE.txt +1 -1
- data/README.md +242 -22
- data/app/controllers/field_test/base_controller.rb +1 -1
- data/app/controllers/field_test/memberships_controller.rb +2 -2
- data/app/controllers/field_test/participants_controller.rb +10 -2
- data/app/helpers/field_test/base_helper.rb +12 -0
- data/app/models/field_test/membership.rb +2 -1
- data/app/views/field_test/experiments/_experiments.html.erb +3 -2
- data/app/views/field_test/experiments/show.html.erb +1 -1
- data/app/views/layouts/field_test/application.html.erb +4 -0
- data/config/routes.rb +2 -1
- data/lib/field_test.rb +47 -12
- data/lib/field_test/controller.rb +76 -0
- data/lib/field_test/experiment.rb +81 -48
- data/lib/field_test/helpers.rb +24 -53
- data/lib/field_test/mailer.rb +20 -0
- data/lib/field_test/participant.rb +33 -2
- data/lib/field_test/version.rb +1 -1
- data/lib/generators/field_test/events_generator.rb +3 -18
- data/lib/generators/field_test/install_generator.rb +3 -18
- data/lib/generators/field_test/templates/events.rb.tt +2 -4
- data/lib/generators/field_test/templates/memberships.rb.tt +5 -4
- metadata +59 -19
- data/.gitignore +0 -9
- data/Gemfile +0 -4
- data/Rakefile +0 -10
- data/field_test.gemspec +0 -30
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 25a516eac00c5ded25aa203d9e3feb89196c7641868e5575938bf223ee8d20d2
|
4
|
+
data.tar.gz: 89dc335addf645088c5251ccd7a99b0d5348857a210c520350a3d39a9244fc3f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 97c5900ff54996e5b31250b882300104c991e93ab4582b30e1a2ada4fb0c8f191ddbac9240941fce9d2160bf01a66ca1ed73cd3e4a5a734de99e487b9c0c3772
|
7
|
+
data.tar.gz: 765b5fbb1928889ff40b576b710b93ffe14adc38d6153715b89d806cbcfa46021fb4d0cc780754a1e794d3ad2ccbcb8f939dff72a148392ae95a0b968cb7d443
|
data/CHANGELOG.md
CHANGED
@@ -1,33 +1,71 @@
|
|
1
|
-
## 0.
|
1
|
+
## 0.4.1 (2020-09-07)
|
2
|
+
|
3
|
+
- Use `datetime` type in migration
|
4
|
+
|
5
|
+
## 0.4.0 (2020-08-04)
|
6
|
+
|
7
|
+
- Fixed CSRF vulnerability with non-session based authentication - [more info](https://github.com/ankane/field_test/issues/28)
|
8
|
+
- Fixed cache key for requests
|
9
|
+
|
10
|
+
## 0.3.2 (2020-04-16)
|
11
|
+
|
12
|
+
- Added support for excluding IP addresses
|
13
|
+
|
14
|
+
## 0.3.1 (2019-07-01)
|
15
|
+
|
16
|
+
- Added `closed` and `keep_variant`
|
17
|
+
- Added `field_test_upgrade_memberships` method
|
18
|
+
- Fixed API controller error
|
19
|
+
- Fixed bug where conversions were recorded after winner
|
20
|
+
|
21
|
+
Security
|
22
|
+
|
23
|
+
- Fixed arbitrary variants via query parameters - [more info](https://github.com/ankane/field_test/issues/17)
|
24
|
+
|
25
|
+
## 0.3.0 (2019-06-02)
|
26
|
+
|
27
|
+
- Added support for native apps
|
28
|
+
- Added `cookies` option
|
29
|
+
- Added `precision` option
|
30
|
+
- Fixed bug in results with multiple goals
|
31
|
+
- Fixed issue where metrics disappeared from dashboard when moving to multiple goals
|
32
|
+
- Dropped support for Rails < 5
|
33
|
+
|
34
|
+
Breaking changes
|
35
|
+
|
36
|
+
- Split out participant id and type
|
37
|
+
- Changed participant logic for emails
|
38
|
+
|
39
|
+
## 0.2.4 (2019-01-03)
|
2
40
|
|
3
41
|
- Fixed `PG::AmbiguousColumn` error
|
4
42
|
|
5
|
-
## 0.2.3
|
43
|
+
## 0.2.3 (2018-01-28)
|
6
44
|
|
7
45
|
- Fixed participant reporting for multiple goals
|
8
46
|
|
9
|
-
## 0.2.2
|
47
|
+
## 0.2.2 (2017-05-01)
|
10
48
|
|
11
49
|
- Added support for Rails 5.1
|
12
50
|
|
13
|
-
## 0.2.1
|
51
|
+
## 0.2.1 (2016-12-18)
|
14
52
|
|
15
53
|
- Added support for multiple goals
|
16
54
|
|
17
|
-
## 0.2.0
|
55
|
+
## 0.2.0 (2016-12-17)
|
18
56
|
|
19
57
|
- Better web UI
|
20
58
|
- Removed `cookie:` prefix for unknown participants
|
21
59
|
|
22
|
-
## 0.1.2
|
60
|
+
## 0.1.2 (2016-12-17)
|
23
61
|
|
24
62
|
- Exclude bots
|
25
63
|
- Mailer improvements
|
26
64
|
|
27
|
-
## 0.1.1
|
65
|
+
## 0.1.1 (2016-12-15)
|
28
66
|
|
29
67
|
- Added basic web UI
|
30
68
|
|
31
|
-
## 0.1.0
|
69
|
+
## 0.1.0 (2016-12-14)
|
32
70
|
|
33
71
|
- First release
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -3,11 +3,14 @@
|
|
3
3
|
:maple_leaf: A/B testing for Rails
|
4
4
|
|
5
5
|
- Designed for web and email
|
6
|
-
- Comes with a [
|
6
|
+
- Comes with a [dashboard](https://fieldtest.dokkuapp.com/) to view results and update variants
|
7
|
+
- Uses your database for storage
|
7
8
|
- Seamlessly handles the transition from anonymous visitor to logged in user
|
8
9
|
|
9
10
|
Uses [Bayesian statistics](https://www.evanmiller.org/bayesian-ab-testing.html) to evaluate results so you don’t need to choose a sample size ahead of time.
|
10
11
|
|
12
|
+
[![Build Status](https://travis-ci.org/ankane/field_test.svg?branch=master)](https://travis-ci.org/ankane/field_test)
|
13
|
+
|
11
14
|
## Installation
|
12
15
|
|
13
16
|
Add this line to your application’s Gemfile:
|
@@ -19,7 +22,8 @@ gem "field_test"
|
|
19
22
|
Run:
|
20
23
|
|
21
24
|
```sh
|
22
|
-
rails
|
25
|
+
rails generate field_test:install
|
26
|
+
rails db:migrate
|
23
27
|
```
|
24
28
|
|
25
29
|
And mount the dashboard in your `config/routes.rb`:
|
@@ -28,7 +32,7 @@ And mount the dashboard in your `config/routes.rb`:
|
|
28
32
|
mount FieldTest::Engine, at: "field_test"
|
29
33
|
```
|
30
34
|
|
31
|
-
Be sure to [secure the dashboard](#security) in production.
|
35
|
+
Be sure to [secure the dashboard](#dashboard-security) in production.
|
32
36
|
|
33
37
|
![Screenshot](https://ankane.github.io/field_test/screenshot6.png)
|
34
38
|
|
@@ -45,12 +49,18 @@ experiments:
|
|
45
49
|
- blue
|
46
50
|
```
|
47
51
|
|
48
|
-
Refer to it in
|
52
|
+
Refer to it in controllers, views, and mailers.
|
49
53
|
|
50
54
|
```ruby
|
51
55
|
button_color = field_test(:button_color)
|
52
56
|
```
|
53
57
|
|
58
|
+
To make testing easier, you can specify a variant with query parameters
|
59
|
+
|
60
|
+
```
|
61
|
+
http://localhost:3000/?field_test[button_color]=green
|
62
|
+
```
|
63
|
+
|
54
64
|
When someone converts, record it with:
|
55
65
|
|
56
66
|
```ruby
|
@@ -67,24 +77,90 @@ experiments:
|
|
67
77
|
|
68
78
|
All calls to `field_test` will now return the winner, and metrics will stop being recorded.
|
69
79
|
|
70
|
-
|
80
|
+
You can keep returning the variant for existing participants after a winner is declared:
|
71
81
|
|
72
|
-
|
82
|
+
```yml
|
83
|
+
experiments:
|
84
|
+
button_color:
|
85
|
+
winner: green
|
86
|
+
keep_variant: true
|
87
|
+
```
|
88
|
+
|
89
|
+
You can also close an experiment to new participants without declaring a winner while still recording metrics for existing participants:
|
73
90
|
|
91
|
+
```yml
|
92
|
+
experiments:
|
93
|
+
button_color:
|
94
|
+
closed: true
|
74
95
|
```
|
75
|
-
|
96
|
+
|
97
|
+
Calls to `field_test` for new participants will return the control, and they won’t be added to the experiment.
|
98
|
+
|
99
|
+
You can get the list of experiments and variants for a user with:
|
100
|
+
|
101
|
+
```ruby
|
102
|
+
field_test_experiments
|
76
103
|
```
|
77
104
|
|
78
|
-
|
105
|
+
## JavaScript and Native Apps
|
106
|
+
|
107
|
+
For JavaScript and native apps, add calls to your normal endpoints.
|
79
108
|
|
80
109
|
```ruby
|
81
|
-
|
82
|
-
|
110
|
+
class CheckoutController < ActionController::API
|
111
|
+
def start
|
112
|
+
render json: {button_color: field_test(:button_color)}
|
113
|
+
end
|
114
|
+
|
115
|
+
def finish
|
116
|
+
field_test_converted(:button_color)
|
117
|
+
# ...
|
118
|
+
end
|
119
|
+
end
|
83
120
|
```
|
84
121
|
|
85
|
-
|
122
|
+
For anonymous visitors in native apps, pass a `Field-Test-Visitor` header with a unique identifier.
|
86
123
|
|
87
|
-
##
|
124
|
+
## Participants
|
125
|
+
|
126
|
+
Any model or string can be a participant in an experiment.
|
127
|
+
|
128
|
+
For web requests, it uses `current_user` (if it exists) and an anonymous visitor id to determine the participant. Set your own with:
|
129
|
+
|
130
|
+
```ruby
|
131
|
+
class ApplicationController < ActionController::Base
|
132
|
+
def field_test_participant
|
133
|
+
current_company
|
134
|
+
end
|
135
|
+
end
|
136
|
+
```
|
137
|
+
|
138
|
+
For mailers, it tries `@user` then `params[:user]` to determine the participant. Set your own with:
|
139
|
+
|
140
|
+
```ruby
|
141
|
+
class ApplicationMailer < ActionMailer::Base
|
142
|
+
def field_test_participant
|
143
|
+
@company
|
144
|
+
end
|
145
|
+
end
|
146
|
+
```
|
147
|
+
|
148
|
+
You can also manually pass a participant with:
|
149
|
+
|
150
|
+
```ruby
|
151
|
+
field_test(:button_color, participant: company)
|
152
|
+
```
|
153
|
+
|
154
|
+
## Jobs
|
155
|
+
|
156
|
+
To get variants in jobs, models, and other contexts, use:
|
157
|
+
|
158
|
+
```ruby
|
159
|
+
experiment = FieldTest::Experiment.find(:button_color)
|
160
|
+
button_color = experiment.variant(user)
|
161
|
+
```
|
162
|
+
|
163
|
+
## Exclusions
|
88
164
|
|
89
165
|
By default, bots are returned the first variant and excluded from metrics. Change this with:
|
90
166
|
|
@@ -93,6 +169,23 @@ exclude:
|
|
93
169
|
bots: false
|
94
170
|
```
|
95
171
|
|
172
|
+
Exclude certain IP addresses with:
|
173
|
+
|
174
|
+
```yml
|
175
|
+
exclude:
|
176
|
+
ips:
|
177
|
+
- 127.0.0.1
|
178
|
+
- 10.0.0.0/8
|
179
|
+
```
|
180
|
+
|
181
|
+
You can also use custom logic:
|
182
|
+
|
183
|
+
```ruby
|
184
|
+
field_test(:button_color, exclude: request.user_agent == "Test")
|
185
|
+
```
|
186
|
+
|
187
|
+
## Config
|
188
|
+
|
96
189
|
Keep track of when experiments started and ended. Use any format `Time.parse` accepts. Variants assigned outside this window are not included in metrics.
|
97
190
|
|
98
191
|
```yml
|
@@ -126,6 +219,14 @@ experiments:
|
|
126
219
|
- 15
|
127
220
|
```
|
128
221
|
|
222
|
+
To help with GDPR compliance, you can switch from cookies to [anonymity sets](https://privacypatterns.org/patterns/Anonymity-set) for anonymous visitors. Visitors with the same IP mask and user agent are grouped together.
|
223
|
+
|
224
|
+
```yml
|
225
|
+
cookies: false
|
226
|
+
```
|
227
|
+
|
228
|
+
## Dashboard Config
|
229
|
+
|
129
230
|
If the dashboard gets slow, you can make it faster with:
|
130
231
|
|
131
232
|
```yml
|
@@ -134,12 +235,19 @@ cache: true
|
|
134
235
|
|
135
236
|
This will use the Rails cache to speed up winning probability calculations.
|
136
237
|
|
137
|
-
|
238
|
+
If you need more precision, set:
|
239
|
+
|
240
|
+
```yml
|
241
|
+
precision: 1
|
242
|
+
```
|
243
|
+
|
244
|
+
## Multiple Goals
|
138
245
|
|
139
246
|
You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:
|
140
247
|
|
141
248
|
```sh
|
142
|
-
rails
|
249
|
+
rails generate field_test:events
|
250
|
+
rails db:migrate
|
143
251
|
```
|
144
252
|
|
145
253
|
And add to your config:
|
@@ -162,20 +270,30 @@ The results for all goals will appear on the dashboard.
|
|
162
270
|
|
163
271
|
## Analytics Platforms
|
164
272
|
|
165
|
-
You
|
273
|
+
You may also want to send experiment data as properties to other analytics platforms like [Segment](https://segment.com), [Amplitude](https://amplitude.com), and [Ahoy](https://github.com/ankane/ahoy). Get the list of experiments and variants with:
|
166
274
|
|
167
275
|
```ruby
|
168
276
|
field_test_experiments
|
169
277
|
```
|
170
278
|
|
171
|
-
|
279
|
+
### Ahoy
|
280
|
+
|
281
|
+
You can configure Field Test to use Ahoy’s visitor token instead of creating its own:
|
282
|
+
|
283
|
+
```ruby
|
284
|
+
class ApplicationController < ActionController::Base
|
285
|
+
def field_test_participant
|
286
|
+
[ahoy.user, ahoy.visitor_token]
|
287
|
+
end
|
288
|
+
end
|
289
|
+
```
|
172
290
|
|
173
|
-
## Security
|
291
|
+
## Dashboard Security
|
174
292
|
|
175
293
|
#### Devise
|
176
294
|
|
177
295
|
```ruby
|
178
|
-
authenticate :user, ->
|
296
|
+
authenticate :user, ->(user) { user.admin? } do
|
179
297
|
mount FieldTest::Engine, at: "field_test"
|
180
298
|
end
|
181
299
|
```
|
@@ -189,13 +307,106 @@ ENV["FIELD_TEST_USERNAME"] = "moonrise"
|
|
189
307
|
ENV["FIELD_TEST_PASSWORD"] = "kingdom"
|
190
308
|
```
|
191
309
|
|
192
|
-
##
|
310
|
+
## Updating Variants
|
193
311
|
|
194
|
-
|
312
|
+
Assign a specific variant to a user with:
|
313
|
+
|
314
|
+
```ruby
|
315
|
+
experiment = FieldTest::Experiment.find(:button_color)
|
316
|
+
experiment.variant(participant, variant: "green")
|
317
|
+
```
|
318
|
+
|
319
|
+
You can also change a user’s variant from the dashboard.
|
320
|
+
|
321
|
+
## Associations
|
322
|
+
|
323
|
+
To associate models with field test memberships, use:
|
324
|
+
|
325
|
+
```ruby
|
326
|
+
class User < ApplicationRecord
|
327
|
+
has_many :field_test_memberships, class_name: "FieldTest::Membership", as: :participant
|
328
|
+
end
|
329
|
+
```
|
330
|
+
|
331
|
+
Now you can do:
|
332
|
+
|
333
|
+
```ruby
|
334
|
+
user.field_test_memberships
|
335
|
+
```
|
336
|
+
|
337
|
+
## Upgrading
|
338
|
+
|
339
|
+
### 0.3.0
|
340
|
+
|
341
|
+
Upgrade the gem and add to `config/field_test.yml`:
|
342
|
+
|
343
|
+
```yml
|
344
|
+
legacy_participants: true
|
345
|
+
```
|
346
|
+
|
347
|
+
Also, if you use Field Test in emails, know that the default way participants are determined has changed. Restore the previous way with:
|
348
|
+
|
349
|
+
```ruby
|
350
|
+
class ApplicationMailer < ActionMailer::Base
|
351
|
+
def field_test_participant
|
352
|
+
message.to.first
|
353
|
+
end
|
354
|
+
end
|
355
|
+
```
|
356
|
+
|
357
|
+
We also recommend upgrading participants when you have time.
|
358
|
+
|
359
|
+
#### Upgrading Participants
|
360
|
+
|
361
|
+
Field Test 0.3.0 splits the `field_test_memberships.participant` column into `participant_type` and `participant_id`.
|
362
|
+
|
363
|
+
To upgrade without downtime, create a migration:
|
364
|
+
|
365
|
+
```sh
|
366
|
+
rails generate migration upgrade_field_test_participants
|
367
|
+
```
|
368
|
+
|
369
|
+
with:
|
370
|
+
|
371
|
+
```ruby
|
372
|
+
class UpgradeFieldTestParticipants < ActiveRecord::Migration[6.0]
|
373
|
+
def change
|
374
|
+
add_column :field_test_memberships, :participant_type, :string
|
375
|
+
add_column :field_test_memberships, :participant_id, :string
|
376
|
+
|
377
|
+
add_index :field_test_memberships, [:participant_type, :participant_id, :experiment],
|
378
|
+
unique: true, name: "index_field_test_memberships_on_participant_and_experiment"
|
379
|
+
end
|
380
|
+
end
|
381
|
+
```
|
382
|
+
|
383
|
+
After you run it, writes will go to both the old and new sets of columns.
|
384
|
+
|
385
|
+
Next, backfill data:
|
386
|
+
|
387
|
+
```ruby
|
388
|
+
FieldTest::Membership.where(participant_id: nil).find_each do |membership|
|
389
|
+
participant = membership.participant
|
390
|
+
|
391
|
+
if participant.include?(":")
|
392
|
+
participant_type, _, participant_id = participant.rpartition(":")
|
393
|
+
participant_type = nil if participant_type == "cookie" # legacy
|
394
|
+
else
|
395
|
+
participant_id = participant
|
396
|
+
end
|
397
|
+
|
398
|
+
membership.update!(
|
399
|
+
participant_type: participant_type,
|
400
|
+
participant_id: participant_id
|
401
|
+
)
|
402
|
+
end
|
403
|
+
```
|
404
|
+
|
405
|
+
Finally, remove `legacy_participants: true` from the config file. Once you confirm it’s working, you can drop the `participant` column (you can rename it first just to be extra safe).
|
195
406
|
|
196
|
-
##
|
407
|
+
## Credits
|
197
408
|
|
198
|
-
|
409
|
+
A huge thanks to [Evan Miller](https://www.evanmiller.org/) for deriving the Bayesian formulas.
|
199
410
|
|
200
411
|
## History
|
201
412
|
|
@@ -209,3 +420,12 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
|
|
209
420
|
- Fix bugs and [submit pull requests](https://github.com/ankane/field_test/pulls)
|
210
421
|
- Write, clarify, or fix documentation
|
211
422
|
- Suggest or add new features
|
423
|
+
|
424
|
+
To get started with development:
|
425
|
+
|
426
|
+
```sh
|
427
|
+
git clone https://github.com/ankane/field_test.git
|
428
|
+
cd field_test
|
429
|
+
bundle install
|
430
|
+
bundle exec rake test
|
431
|
+
```
|