field_test 0.2.3 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/CHANGELOG.md +45 -7
- data/LICENSE.txt +1 -1
- data/README.md +234 -22
- data/app/controllers/field_test/base_controller.rb +1 -1
- data/app/controllers/field_test/memberships_controller.rb +2 -2
- data/app/controllers/field_test/participants_controller.rb +10 -2
- data/app/helpers/field_test/base_helper.rb +12 -0
- data/app/models/field_test/membership.rb +2 -1
- data/app/views/field_test/experiments/_experiments.html.erb +3 -2
- data/app/views/field_test/experiments/show.html.erb +1 -1
- data/app/views/layouts/field_test/application.html.erb +4 -0
- data/config/routes.rb +2 -1
- data/lib/field_test.rb +47 -12
- data/lib/field_test/calculations.rb +1 -1
- data/lib/field_test/controller.rb +76 -0
- data/lib/field_test/experiment.rb +83 -50
- data/lib/field_test/helpers.rb +24 -53
- data/lib/field_test/mailer.rb +20 -0
- data/lib/field_test/participant.rb +33 -2
- data/lib/field_test/version.rb +1 -1
- data/lib/generators/field_test/events_generator.rb +3 -18
- data/lib/generators/field_test/install_generator.rb +3 -18
- data/lib/generators/field_test/templates/{config.yml → config.yml.tt} +0 -0
- data/lib/generators/field_test/templates/{events.rb → events.rb.tt} +1 -3
- data/lib/generators/field_test/templates/{memberships.rb → memberships.rb.tt} +4 -3
- metadata +62 -22
- data/.gitignore +0 -9
- data/Gemfile +0 -4
- data/Rakefile +0 -10
- data/field_test.gemspec +0 -30
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 96def1c6805d880a03c141a421570a3e4b9f912e9427c385b132ffae646bc74b
|
4
|
+
data.tar.gz: b6eb3262d522544a40db7169364ebef0f6d9f4d2f1f4c82712c91fec5a617a4f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 35765af19297887bae1e807aeff9f4f741687b4c3fc961b6b91a6aa9c3060098f42bb677d1055309b2698377dd6694c24770e8ab0418fa4c675146c1661559c0
|
7
|
+
data.tar.gz: 786390b24efece6a99b5db602b6b0162a2b398b319f42e61c8b0db4aca021fb4b80fba65ecfa5c5d94ebb451b7b1c852c25983e72246a3aa4640dc41613963a1
|
data/CHANGELOG.md
CHANGED
@@ -1,29 +1,67 @@
|
|
1
|
-
## 0.
|
1
|
+
## 0.4.0 (2020-08-04)
|
2
|
+
|
3
|
+
- Fixed CSRF vulnerability with non-session based authentication
|
4
|
+
- Fixed cache key for requests
|
5
|
+
|
6
|
+
## 0.3.2 (2020-04-16)
|
7
|
+
|
8
|
+
- Added support for excluding IP addresses
|
9
|
+
|
10
|
+
## 0.3.1 (2019-07-01)
|
11
|
+
|
12
|
+
- Added `closed` and `keep_variant`
|
13
|
+
- Added `field_test_upgrade_memberships` method
|
14
|
+
- Fixed API controller error
|
15
|
+
- Fixed bug where conversions were recorded after winner
|
16
|
+
|
17
|
+
Security
|
18
|
+
|
19
|
+
- Fixed arbitrary variants via query parameters - see [#17](https://github.com/ankane/field_test/issues/17)
|
20
|
+
|
21
|
+
## 0.3.0 (2019-06-02)
|
22
|
+
|
23
|
+
- Added support for native apps
|
24
|
+
- Added `cookies` option
|
25
|
+
- Added `precision` option
|
26
|
+
- Fixed bug in results with multiple goals
|
27
|
+
- Fixed issue where metrics disappeared from dashboard when moving to multiple goals
|
28
|
+
- Dropped support for Rails < 5
|
29
|
+
|
30
|
+
Breaking changes
|
31
|
+
|
32
|
+
- Split out participant id and type
|
33
|
+
- Changed participant logic for emails
|
34
|
+
|
35
|
+
## 0.2.4 (2019-01-03)
|
36
|
+
|
37
|
+
- Fixed `PG::AmbiguousColumn` error
|
38
|
+
|
39
|
+
## 0.2.3 (2018-01-28)
|
2
40
|
|
3
41
|
- Fixed participant reporting for multiple goals
|
4
42
|
|
5
|
-
## 0.2.2
|
43
|
+
## 0.2.2 (2017-05-01)
|
6
44
|
|
7
45
|
- Added support for Rails 5.1
|
8
46
|
|
9
|
-
## 0.2.1
|
47
|
+
## 0.2.1 (2016-12-18)
|
10
48
|
|
11
49
|
- Added support for multiple goals
|
12
50
|
|
13
|
-
## 0.2.0
|
51
|
+
## 0.2.0 (2016-12-17)
|
14
52
|
|
15
53
|
- Better web UI
|
16
54
|
- Removed `cookie:` prefix for unknown participants
|
17
55
|
|
18
|
-
## 0.1.2
|
56
|
+
## 0.1.2 (2016-12-17)
|
19
57
|
|
20
58
|
- Exclude bots
|
21
59
|
- Mailer improvements
|
22
60
|
|
23
|
-
## 0.1.1
|
61
|
+
## 0.1.1 (2016-12-15)
|
24
62
|
|
25
63
|
- Added basic web UI
|
26
64
|
|
27
|
-
## 0.1.0
|
65
|
+
## 0.1.0 (2016-12-14)
|
28
66
|
|
29
67
|
- First release
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -3,10 +3,13 @@
|
|
3
3
|
:maple_leaf: A/B testing for Rails
|
4
4
|
|
5
5
|
- Designed for web and email
|
6
|
-
- Comes with a [
|
6
|
+
- Comes with a [dashboard](https://fieldtest.dokkuapp.com/) to view results and update variants
|
7
|
+
- Uses your database for storage
|
7
8
|
- Seamlessly handles the transition from anonymous visitor to logged in user
|
8
9
|
|
9
|
-
Uses [Bayesian statistics](
|
10
|
+
Uses [Bayesian statistics](https://www.evanmiller.org/bayesian-ab-testing.html) to evaluate results so you don’t need to choose a sample size ahead of time.
|
11
|
+
|
12
|
+
[![Build Status](https://travis-ci.org/ankane/field_test.svg?branch=master)](https://travis-ci.org/ankane/field_test)
|
10
13
|
|
11
14
|
## Installation
|
12
15
|
|
@@ -19,7 +22,8 @@ gem "field_test"
|
|
19
22
|
Run:
|
20
23
|
|
21
24
|
```sh
|
22
|
-
rails
|
25
|
+
rails generate field_test:install
|
26
|
+
rails db:migrate
|
23
27
|
```
|
24
28
|
|
25
29
|
And mount the dashboard in your `config/routes.rb`:
|
@@ -28,7 +32,7 @@ And mount the dashboard in your `config/routes.rb`:
|
|
28
32
|
mount FieldTest::Engine, at: "field_test"
|
29
33
|
```
|
30
34
|
|
31
|
-
Be sure to [secure the dashboard](#security) in production.
|
35
|
+
Be sure to [secure the dashboard](#dashboard-security) in production.
|
32
36
|
|
33
37
|
![Screenshot](https://ankane.github.io/field_test/screenshot6.png)
|
34
38
|
|
@@ -45,12 +49,18 @@ experiments:
|
|
45
49
|
- blue
|
46
50
|
```
|
47
51
|
|
48
|
-
Refer to it in
|
52
|
+
Refer to it in controllers, views, and mailers.
|
49
53
|
|
50
54
|
```ruby
|
51
55
|
button_color = field_test(:button_color)
|
52
56
|
```
|
53
57
|
|
58
|
+
To make testing easier, you can specify a variant with query parameters
|
59
|
+
|
60
|
+
```
|
61
|
+
http://localhost:3000/?field_test[button_color]=green
|
62
|
+
```
|
63
|
+
|
54
64
|
When someone converts, record it with:
|
55
65
|
|
56
66
|
```ruby
|
@@ -67,22 +77,88 @@ experiments:
|
|
67
77
|
|
68
78
|
All calls to `field_test` will now return the winner, and metrics will stop being recorded.
|
69
79
|
|
70
|
-
|
80
|
+
You can keep returning the variant for existing participants after a winner is declared:
|
71
81
|
|
72
|
-
|
82
|
+
```yml
|
83
|
+
experiments:
|
84
|
+
button_color:
|
85
|
+
winner: green
|
86
|
+
keep_variant: true
|
87
|
+
```
|
73
88
|
|
89
|
+
You can also close an experiment to new participants without declaring a winner while still recording metrics for existing participants:
|
90
|
+
|
91
|
+
```yml
|
92
|
+
experiments:
|
93
|
+
button_color:
|
94
|
+
closed: true
|
74
95
|
```
|
75
|
-
|
96
|
+
|
97
|
+
Calls to `field_test` for new participants will return the control, and they won’t be added to the experiment.
|
98
|
+
|
99
|
+
You can get the list of experiments and variants for a user with:
|
100
|
+
|
101
|
+
```ruby
|
102
|
+
field_test_experiments
|
76
103
|
```
|
77
104
|
|
78
|
-
|
105
|
+
## JavaScript and Native Apps
|
106
|
+
|
107
|
+
For JavaScript and native apps, add calls to your normal endpoints.
|
79
108
|
|
80
109
|
```ruby
|
81
|
-
|
82
|
-
|
110
|
+
class CheckoutController < ActionController::API
|
111
|
+
def start
|
112
|
+
render json: {button_color: field_test(:button_color)}
|
113
|
+
end
|
114
|
+
|
115
|
+
def finish
|
116
|
+
field_test_converted(:button_color)
|
117
|
+
# ...
|
118
|
+
end
|
119
|
+
end
|
83
120
|
```
|
84
121
|
|
85
|
-
|
122
|
+
For anonymous visitors in native apps, pass a `Field-Test-Visitor` header with a unique identifier.
|
123
|
+
|
124
|
+
## Participants
|
125
|
+
|
126
|
+
Any model or string can be a participant in an experiment.
|
127
|
+
|
128
|
+
For web requests, it uses `current_user` (if it exists) and an anonymous visitor id to determine the participant. Set your own with:
|
129
|
+
|
130
|
+
```ruby
|
131
|
+
class ApplicationController < ActionController::Base
|
132
|
+
def field_test_participant
|
133
|
+
current_company
|
134
|
+
end
|
135
|
+
end
|
136
|
+
```
|
137
|
+
|
138
|
+
For mailers, it tries `@user` then `params[:user]` to determine the participant. Set your own with:
|
139
|
+
|
140
|
+
```ruby
|
141
|
+
class ApplicationMailer < ActionMailer::Base
|
142
|
+
def field_test_participant
|
143
|
+
@company
|
144
|
+
end
|
145
|
+
end
|
146
|
+
```
|
147
|
+
|
148
|
+
You can also manually pass a participant with:
|
149
|
+
|
150
|
+
```ruby
|
151
|
+
field_test(:button_color, participant: company)
|
152
|
+
```
|
153
|
+
|
154
|
+
## Jobs
|
155
|
+
|
156
|
+
To get variants in jobs, models, and other contexts, use:
|
157
|
+
|
158
|
+
```ruby
|
159
|
+
experiment = FieldTest::Experiment.find(:button_color)
|
160
|
+
button_color = experiment.variant(user)
|
161
|
+
```
|
86
162
|
|
87
163
|
## Config
|
88
164
|
|
@@ -93,6 +169,15 @@ exclude:
|
|
93
169
|
bots: false
|
94
170
|
```
|
95
171
|
|
172
|
+
Exclude certain IP addresses with:
|
173
|
+
|
174
|
+
```yml
|
175
|
+
exclude:
|
176
|
+
ips:
|
177
|
+
- 127.0.0.1
|
178
|
+
- 10.0.0.0/8
|
179
|
+
```
|
180
|
+
|
96
181
|
Keep track of when experiments started and ended. Use any format `Time.parse` accepts. Variants assigned outside this window are not included in metrics.
|
97
182
|
|
98
183
|
```yml
|
@@ -126,6 +211,14 @@ experiments:
|
|
126
211
|
- 15
|
127
212
|
```
|
128
213
|
|
214
|
+
To help with GDPR compliance, you can switch from cookies to [anonymity sets](https://privacypatterns.org/patterns/Anonymity-set) for anonymous visitors. Visitors with the same IP mask and user agent are grouped together.
|
215
|
+
|
216
|
+
```yml
|
217
|
+
cookies: false
|
218
|
+
```
|
219
|
+
|
220
|
+
## Dashboard Config
|
221
|
+
|
129
222
|
If the dashboard gets slow, you can make it faster with:
|
130
223
|
|
131
224
|
```yml
|
@@ -134,12 +227,19 @@ cache: true
|
|
134
227
|
|
135
228
|
This will use the Rails cache to speed up winning probability calculations.
|
136
229
|
|
137
|
-
|
230
|
+
If you need more precision, set:
|
231
|
+
|
232
|
+
```yml
|
233
|
+
precision: 1
|
234
|
+
```
|
235
|
+
|
236
|
+
## Multiple Goals
|
138
237
|
|
139
238
|
You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:
|
140
239
|
|
141
240
|
```sh
|
142
|
-
rails
|
241
|
+
rails generate field_test:events
|
242
|
+
rails db:migrate
|
143
243
|
```
|
144
244
|
|
145
245
|
And add to your config:
|
@@ -162,20 +262,30 @@ The results for all goals will appear on the dashboard.
|
|
162
262
|
|
163
263
|
## Analytics Platforms
|
164
264
|
|
165
|
-
You
|
265
|
+
You may also want to send experiment data as properties to other analytics platforms like [Segment](https://segment.com), [Amplitude](https://amplitude.com), and [Ahoy](https://github.com/ankane/ahoy). Get the list of experiments and variants with:
|
166
266
|
|
167
267
|
```ruby
|
168
268
|
field_test_experiments
|
169
269
|
```
|
170
270
|
|
171
|
-
|
271
|
+
### Ahoy
|
272
|
+
|
273
|
+
You can configure Field Test to use Ahoy’s visitor token instead of creating its own:
|
274
|
+
|
275
|
+
```ruby
|
276
|
+
class ApplicationController < ActionController::Base
|
277
|
+
def field_test_participant
|
278
|
+
[ahoy.user, ahoy.visitor_token]
|
279
|
+
end
|
280
|
+
end
|
281
|
+
```
|
172
282
|
|
173
|
-
## Security
|
283
|
+
## Dashboard Security
|
174
284
|
|
175
285
|
#### Devise
|
176
286
|
|
177
287
|
```ruby
|
178
|
-
authenticate :user, ->
|
288
|
+
authenticate :user, ->(user) { user.admin? } do
|
179
289
|
mount FieldTest::Engine, at: "field_test"
|
180
290
|
end
|
181
291
|
```
|
@@ -189,13 +299,106 @@ ENV["FIELD_TEST_USERNAME"] = "moonrise"
|
|
189
299
|
ENV["FIELD_TEST_PASSWORD"] = "kingdom"
|
190
300
|
```
|
191
301
|
|
192
|
-
##
|
302
|
+
## Updating Variants
|
303
|
+
|
304
|
+
Assign a specific variant to a user with:
|
305
|
+
|
306
|
+
```ruby
|
307
|
+
experiment = FieldTest::Experiment.find(:button_color)
|
308
|
+
experiment.variant(participant, variant: "green")
|
309
|
+
```
|
310
|
+
|
311
|
+
You can also change a user’s variant from the dashboard.
|
312
|
+
|
313
|
+
## Associations
|
314
|
+
|
315
|
+
To associate models with field test memberships, use:
|
316
|
+
|
317
|
+
```ruby
|
318
|
+
class User < ApplicationRecord
|
319
|
+
has_many :field_test_memberships, class_name: "FieldTest::Membership", as: :participant
|
320
|
+
end
|
321
|
+
```
|
322
|
+
|
323
|
+
Now you can do:
|
324
|
+
|
325
|
+
```ruby
|
326
|
+
user.field_test_memberships
|
327
|
+
```
|
328
|
+
|
329
|
+
## Upgrading
|
330
|
+
|
331
|
+
### 0.3.0
|
332
|
+
|
333
|
+
Upgrade the gem and add to `config/field_test.yml`:
|
334
|
+
|
335
|
+
```yml
|
336
|
+
legacy_participants: true
|
337
|
+
```
|
338
|
+
|
339
|
+
Also, if you use Field Test in emails, know that the default way participants are determined has changed. Restore the previous way with:
|
340
|
+
|
341
|
+
```ruby
|
342
|
+
class ApplicationMailer < ActionMailer::Base
|
343
|
+
def field_test_participant
|
344
|
+
message.to.first
|
345
|
+
end
|
346
|
+
end
|
347
|
+
```
|
348
|
+
|
349
|
+
We also recommend upgrading participants when you have time.
|
350
|
+
|
351
|
+
#### Upgrading Participants
|
352
|
+
|
353
|
+
Field Test 0.3.0 splits the `field_test_memberships.participant` column into `participant_type` and `participant_id`.
|
354
|
+
|
355
|
+
To upgrade without downtime, create a migration:
|
356
|
+
|
357
|
+
```sh
|
358
|
+
rails generate migration upgrade_field_test_participants
|
359
|
+
```
|
360
|
+
|
361
|
+
with:
|
193
362
|
|
194
|
-
|
363
|
+
```ruby
|
364
|
+
class UpgradeFieldTestParticipants < ActiveRecord::Migration[6.0]
|
365
|
+
def change
|
366
|
+
add_column :field_test_memberships, :participant_type, :string
|
367
|
+
add_column :field_test_memberships, :participant_id, :string
|
368
|
+
|
369
|
+
add_index :field_test_memberships, [:participant_type, :participant_id, :experiment],
|
370
|
+
unique: true, name: "index_field_test_memberships_on_participant_and_experiment"
|
371
|
+
end
|
372
|
+
end
|
373
|
+
```
|
374
|
+
|
375
|
+
After you run it, writes will go to both the old and new sets of columns.
|
195
376
|
|
196
|
-
|
377
|
+
Next, backfill data:
|
378
|
+
|
379
|
+
```ruby
|
380
|
+
FieldTest::Membership.where(participant_id: nil).find_each do |membership|
|
381
|
+
participant = membership.participant
|
382
|
+
|
383
|
+
if participant.include?(":")
|
384
|
+
participant_type, _, participant_id = participant.rpartition(":")
|
385
|
+
participant_type = nil if participant_type == "cookie" # legacy
|
386
|
+
else
|
387
|
+
participant_id = participant
|
388
|
+
end
|
389
|
+
|
390
|
+
membership.update!(
|
391
|
+
participant_type: participant_type,
|
392
|
+
participant_id: participant_id
|
393
|
+
)
|
394
|
+
end
|
395
|
+
```
|
197
396
|
|
198
|
-
|
397
|
+
Finally, remove `legacy_participants: true` from the config file. Once you confirm it’s working, you can drop the `participant` column (you can rename it first just to be extra safe).
|
398
|
+
|
399
|
+
## Credits
|
400
|
+
|
401
|
+
A huge thanks to [Evan Miller](https://www.evanmiller.org/) for deriving the Bayesian formulas.
|
199
402
|
|
200
403
|
## History
|
201
404
|
|
@@ -209,3 +412,12 @@ Everyone is encouraged to help improve this project. Here are a few ways you can
|
|
209
412
|
- Fix bugs and [submit pull requests](https://github.com/ankane/field_test/pulls)
|
210
413
|
- Write, clarify, or fix documentation
|
211
414
|
- Suggest or add new features
|
415
|
+
|
416
|
+
To get started with development:
|
417
|
+
|
418
|
+
```sh
|
419
|
+
git clone https://github.com/ankane/field_test.git
|
420
|
+
cd field_test
|
421
|
+
bundle install
|
422
|
+
bundle exec rake test
|
423
|
+
```
|