@delmaredigital/payload-experiments 0.4.0-rc.1 → 0.4.0-rc.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +283 -773
- package/dist/admin/TestsDashboardCards.d.ts +1 -0
- package/dist/admin/TestsDashboardCards.js +4 -8
- package/dist/admin/TestsNavLink.d.ts +1 -0
- package/dist/admin/TestsNavLink.js +4 -10
- package/dist/admin/api.d.ts +7 -0
- package/dist/admin/api.js +27 -0
- package/dist/admin/blocks/BlockPanel.js +65 -19
- package/dist/admin/blocks/formApi.d.ts +3 -0
- package/dist/admin/blocks/formApi.js +9 -3
- package/dist/admin/blocks/statusCache.d.ts +2 -0
- package/dist/admin/blocks/statusCache.js +11 -2
- package/dist/admin/copy/blocks.d.ts +16 -0
- package/dist/admin/copy/blocks.js +27 -1
- package/dist/admin/copy/common.d.ts +1 -0
- package/dist/admin/copy/common.js +1 -1
- package/dist/admin/copy/goals.d.ts +1 -0
- package/dist/admin/copy/goals.js +1 -0
- package/dist/admin/copy/pages.d.ts +2 -4
- package/dist/admin/copy/pages.js +3 -5
- package/dist/admin/copy/readiness.d.ts +1 -0
- package/dist/admin/copy/readiness.js +1 -0
- package/dist/admin/goals/XpGoalField.d.ts +2 -0
- package/dist/admin/goals/XpGoalField.js +51 -35
- package/dist/admin/goals/xpGoalField.css +48 -0
- package/dist/admin/paths.d.ts +3 -0
- package/dist/admin/paths.js +3 -0
- package/dist/admin/surface/TestsHome.js +2 -4
- package/dist/admin/surface/TestsSurface.js +2 -5
- package/dist/admin/surface/goals/GoalForm.js +18 -18
- package/dist/admin/surface/goals/GoalsList.js +2 -1
- package/dist/admin/surface/home.model.d.ts +2 -2
- package/dist/admin/surface/newTest/NewTest.js +2 -4
- package/dist/admin/surface/newTest/PageWizard.js +7 -2
- package/dist/admin/surface/pagePicker/CopyRoute.d.ts +1 -1
- package/dist/admin/surface/pagePicker/{PagePicker.d.ts → PageCombobox.d.ts} +3 -3
- package/dist/admin/surface/pagePicker/PageCombobox.js +395 -0
- package/dist/admin/surface/pagePicker/pagePicker.css +10 -0
- package/dist/admin/surface/pagePicker/pagePicker.model.d.ts +2 -0
- package/dist/admin/surface/readiness/readiness.model.js +1 -0
- package/dist/admin/surface/results/ResultsHeader.js +3 -8
- package/dist/admin/surface/results/useResults.d.ts +10 -0
- package/dist/admin/surface/test/GoalSection.js +4 -7
- package/dist/admin/surface/test/Section.js +9 -8
- package/dist/admin/surface/test/TestPage.js +3 -8
- package/dist/admin/surface/test/pageParts.js +10 -39
- package/dist/admin/surface/test/testPage.css +0 -5
- package/dist/admin/surface/useTestDocument.d.ts +4 -0
- package/dist/admin/surfaceUrls.d.ts +7 -0
- package/dist/admin/surfaceUrls.js +19 -0
- package/dist/admin/theme.css +2 -0
- package/dist/admin/useApi.d.ts +10 -0
- package/dist/admin/views/CollectionListRedirect.d.ts +1 -0
- package/dist/admin/views/CollectionListRedirect.js +1 -1
- package/dist/admin/views/ExperimentsRootView.js +18 -165
- package/dist/admin/views/GoalEditRedirect.d.ts +4 -0
- package/dist/admin/views/GoalEditRedirect.js +14 -0
- package/dist/admin/views/GoalsCollectionListView.d.ts +2 -0
- package/dist/admin/views/GoalsCollectionListView.js +4 -0
- package/dist/admin/views/TestsCollectionListView.d.ts +2 -0
- package/dist/admin/views/TestsCollectionListView.js +4 -0
- package/dist/admin/views/TestsNavLinkServer.d.ts +1 -0
- package/dist/admin/views/TestsNavLinkServer.js +1 -1
- package/dist/admin/views/buildSurfaceProps.d.ts +8 -0
- package/dist/admin/views/buildSurfaceProps.js +143 -0
- package/dist/admin/views/renderCollectionList.d.ts +2 -0
- package/dist/admin/views/renderCollectionList.js +34 -0
- package/dist/admin/views/renderTestsSurface.d.ts +3 -0
- package/dist/admin/views/renderTestsSurface.js +34 -0
- package/dist/collections/experiments/config.js +9 -5
- package/dist/collections/metrics.js +16 -4
- package/dist/exports/adminRsc.d.ts +3 -0
- package/dist/exports/adminRsc.js +3 -0
- package/dist/fields/xpGoal.js +2 -14
- package/dist/goals/picker/copy.d.ts +2 -0
- package/dist/goals/picker/copy.js +7 -0
- package/dist/goals/validate.d.ts +3 -0
- package/dist/goals/validate.js +15 -0
- package/dist/hooks/puckGoals.d.ts +5 -0
- package/dist/hooks/puckGoals.js +62 -0
- package/dist/plugin.js +12 -18
- package/dist/puck/editor/editor.chrome.css +13 -0
- package/dist/puck/editor/editor.css +105 -0
- package/dist/puck/editor/xpGoalPuckEditorField.js +28 -20
- package/dist/sanitizeReadinessDetail.js +2 -0
- package/dist/transitions/checkStart.d.ts +2 -1
- package/dist/transitions/checkStart.js +3 -1
- package/dist/transitions/decide.js +1 -0
- package/dist/transitions/start.js +2 -1
- package/dist/types/endpoints.d.ts +1 -1
- package/dist/ui/ui.css +4 -0
- package/package.json +1 -1
- package/dist/admin/surface/pagePicker/PagePicker.js +0 -172
package/README.md
CHANGED
|
@@ -1,54 +1,56 @@
|
|
|
1
1
|
# @delmaredigital/payload-experiments
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A/B testing for Payload CMS and Next.js, from the content you already manage. Test a block on a page, a whole page against another page, or a setting your developer makes testable. Pick a goal, including click goals you name and attach to buttons and links, then compare the Original with your new versions. Results appear in the admin: they count only people who actually saw the content, warn when traffic looks wrong, and check guardrail goals (goals you don't want to hurt). You pick a winner and can make it permanent. Your data stays in your own Postgres database. There is no vendor.
|
|
4
4
|
|
|
5
|
-
> ⚠️ Upgrading to 0.
|
|
5
|
+
> ⚠️ Upgrading to 0.4? Run `payload migrate:create` for four nullable plugin columns (`xp_experiments.page_collection` and its `_xp_experiments_v` counterpart, `xp_phases.placement_docs`, `xp_state.retired_slugs`), plus the Experiment block and `xpGoal` columns if you adopt them, then `payload generate:importmap`. Saves on pages, blocks and Puck collections can now be refused while a test runs on them (`page-busy`, `interacting-tests`); Puck tests whose slot is only on a draft can't start until the page is published; host `metricSources` ids may not start with `click:`; `admin.editorUrl` now receives a required `editor` argument. See the [CHANGELOG's BREAKING section](https://github.com/delmaredigital/payload-experiments/blob/main/CHANGELOG.md).
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
## Install
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
in your own Postgres database. Editors can test blocks, whole pages and
|
|
11
|
-
registered flag points, choose a winner, and make it the permanent content.
|
|
12
|
-
Results count people who actually saw the content, with checks for traffic
|
|
13
|
-
imbalances and metrics that must not get worse. Puck support is optional.
|
|
14
|
-
|
|
15
|
-
```sh
|
|
9
|
+
```bash
|
|
16
10
|
pnpm add @delmaredigital/payload-experiments
|
|
17
11
|
```
|
|
18
12
|
|
|
19
|
-
|
|
13
|
+
### Requirements
|
|
20
14
|
|
|
21
|
-
|
|
22
|
-
`@payloadcms/db-postgres` with transactions enabled. Other database adapters
|
|
23
|
-
are not supported. Install the peers used by your host:
|
|
15
|
+
Use Node.js `>=22.23.4`, the Next.js App Router, and Postgres through `@payloadcms/db-postgres` with transactions enabled. Puck is optional. Install the peers your application uses:
|
|
24
16
|
|
|
25
|
-
| Peer |
|
|
17
|
+
| Peer | Version |
|
|
26
18
|
| --- | --- |
|
|
27
19
|
| `payload`, `@payloadcms/db-postgres`, `@payloadcms/next`, `@payloadcms/ui` | `>=3.90.0 <4` |
|
|
28
20
|
| `next` | `>=16.4.1 <17` |
|
|
29
21
|
| `react`, `react-dom` | `>=19.2.1 <20` |
|
|
30
22
|
| `pg` | `>=8.13.0 <9` |
|
|
31
|
-
| `@puckeditor/core` (optional
|
|
23
|
+
| `@puckeditor/core` (optional) | `>=0.24.0` |
|
|
24
|
+
|
|
25
|
+
## Quick Start
|
|
26
|
+
|
|
27
|
+
These examples use your existing Payload app and Pages collection. Imports starting with `@/` are files you supply; the rendering example uses Puck.
|
|
28
|
+
|
|
29
|
+
### 1. Set a secret and create the runtimes
|
|
32
30
|
|
|
33
|
-
|
|
34
|
-
and server readers. Keep Payload imports lazy so the proxy stays independent
|
|
35
|
-
of Payload, React and Puck:
|
|
31
|
+
Set a private `XP_SECRET` in your server environment. Create a shared `pg` pool for the same database as Payload, with a finite connection timeout:
|
|
36
32
|
|
|
37
33
|
```ts
|
|
38
|
-
// experiments/proxyRuntime.ts
|
|
34
|
+
// src/experiments/proxyRuntime.ts
|
|
35
|
+
import { Pool } from 'pg'
|
|
39
36
|
import {
|
|
40
37
|
defineExperimentsProxyRuntime,
|
|
41
38
|
createPostgresManifestLoader,
|
|
42
39
|
} from '@delmaredigital/payload-experiments/runtime/proxy'
|
|
43
|
-
import { manifestPool } from '@/db/manifestPool' // host-owned pg Pool
|
|
44
40
|
|
|
41
|
+
const pool = new Pool({
|
|
42
|
+
connectionString: process.env.DATABASE_URI,
|
|
43
|
+
connectionTimeoutMillis: 5_000,
|
|
44
|
+
})
|
|
45
45
|
export const proxyRuntime = defineExperimentsProxyRuntime({
|
|
46
46
|
secret: process.env.XP_SECRET!,
|
|
47
|
-
apiRoute: '/api', // match Payload's
|
|
48
|
-
loadManifest: createPostgresManifestLoader({ pool
|
|
47
|
+
apiRoute: '/api', // match Payload's routes.api
|
|
48
|
+
loadManifest: createPostgresManifestLoader({ pool }),
|
|
49
49
|
})
|
|
50
|
+
```
|
|
50
51
|
|
|
51
|
-
|
|
52
|
+
```ts
|
|
53
|
+
// src/experiments/serverRuntime.ts
|
|
52
54
|
import { defineExperimentsServerRuntime } from
|
|
53
55
|
'@delmaredigital/payload-experiments/runtime/server'
|
|
54
56
|
import { proxyRuntime } from './proxyRuntime'
|
|
@@ -60,851 +62,359 @@ export const serverRuntime = defineExperimentsServerRuntime(proxyRuntime, {
|
|
|
60
62
|
return getPayload({ config })
|
|
61
63
|
},
|
|
62
64
|
})
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Keep the secret private and stable across deployments and instances. To rotate it, pass `secret: [newSecret, previousSecret]`, newest first. For a custom Postgres schema, pass the loader's `schemaName`.
|
|
68
|
+
|
|
69
|
+
### 2. Add the plugin
|
|
63
70
|
|
|
64
|
-
|
|
71
|
+
Add this to your existing `payload.config.ts` plugins array. Your permission check must return a boolean; grant it to the staff who will manage tests.
|
|
72
|
+
|
|
73
|
+
```ts
|
|
65
74
|
import { createExperimentsPlugin } from '@delmaredigital/payload-experiments'
|
|
66
75
|
import { serverRuntime } from '@/experiments/serverRuntime'
|
|
67
|
-
import {
|
|
76
|
+
import { canManageTests } from '@/access/tests'
|
|
77
|
+
import { sharedRateLimit } from '@/experiments/rateLimit'
|
|
68
78
|
|
|
69
|
-
// Add to buildConfig({ plugins: [...] })
|
|
70
79
|
createExperimentsPlugin({
|
|
71
80
|
runtime: serverRuntime,
|
|
72
|
-
access: { manage:
|
|
73
|
-
|
|
74
|
-
enabled: true,
|
|
75
|
-
collections: ['pages', { slug: 'templates', field: 'content' }],
|
|
76
|
-
},
|
|
81
|
+
access: { manage: canManageTests },
|
|
82
|
+
events: { rateLimit: sharedRateLimit },
|
|
77
83
|
})
|
|
78
84
|
```
|
|
79
85
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
callback receives `{ req }` and must return a boolean (or a promise of one);
|
|
90
|
-
permission is granted only for `true`. Optional `access.admin` and
|
|
91
|
-
`access.readResults` default to `manage`; `access.readIdentities` defaults to
|
|
92
|
-
`admin`. Use `admin` for code experiments and changing the analysis method.
|
|
93
|
-
Other plugin options are `flagPoints`, `metricSources`, `eventSources`,
|
|
94
|
-
`attributeKeys`, `pages`, `puck` and `events`:
|
|
95
|
-
|
|
96
|
-
- `attributeKeys` declares targeting attributes as `{ type, values? }`, with
|
|
97
|
-
`type` one of `string`, `number`, `boolean`; optional string `values` supplies
|
|
98
|
-
an admin select. Pass only non-identifying attributes.
|
|
99
|
-
- `puck.collections` registers existing host collections for placement discovery,
|
|
100
|
-
content-change markers and making winners permanent. A slug string uses the
|
|
101
|
-
`puckData` JSON field; `{ slug, field }` names another field. `puck.enabled`
|
|
102
|
-
defaults to `true` when this list is non-empty; set it explicitly in TypeScript
|
|
103
|
-
config (the current public type requires it). `false` disables the installed
|
|
104
|
-
content-change hook.
|
|
105
|
-
- `events.retentionDays` defaults to `180`; `events.maxBodyBytes` to `8192`.
|
|
106
|
-
Both must be positive integers. `events.rateLimit` is an optional host adapter,
|
|
107
|
-
called with `{ req, unitId }`; return `true` to permit or `false` to drop.
|
|
108
|
-
- `events.dimensions` (optional, default none) names up to 8 request attributes
|
|
109
|
-
to record on each exposure for later analysis, for example `['loggedIn']`.
|
|
110
|
-
Values come only from the attributes the server computes for that request
|
|
111
|
-
(`path`, `loggedIn`, `staff` and your `getUnit` attributes), never from what a
|
|
112
|
-
browser sends; only strings (first 64 characters), numbers and booleans are
|
|
113
|
-
kept. Names that identify a person (such as `userId`, `email` or `ip`) are
|
|
114
|
-
refused at startup, but whether an attribute you add yourself identifies
|
|
115
|
-
someone is your responsibility. For client-reported exposures `path` is the
|
|
116
|
-
page path from the request's `Referer`. Reporting by dimension is not yet
|
|
117
|
-
available.
|
|
118
|
-
Supply a shared limiter suitable for your deployment; there is no built-in
|
|
119
|
-
in-memory limiter.
|
|
120
|
-
|
|
121
|
-
The plugin adds `xp-experiments`, `xp-phases`, `xp-layers`, `xp-metrics`,
|
|
122
|
-
`xp-events`, `xp-identities`, `xp-results-snapshots` and the `xp-state` global.
|
|
123
|
-
The host owns their migrations. After registering the plugin, generate and
|
|
124
|
-
review a migration, then apply it in the host:
|
|
125
|
-
|
|
126
|
-
```sh
|
|
86
|
+
Supply a shared rate limiter for your deployment: `events.rateLimit` receives
|
|
87
|
+
`{ req, unitId }` and returns `true` to permit or `false` to drop a request.
|
|
88
|
+
Keep identifying attributes out of `events.dimensions`. Use the proxy runtime's `cookie.consent` to enrol only after consent (`ExperimentsProxyRuntimeOptions`); `eraseUser(payload, { userId })` from `/rsc` handles privacy erasure (typed by that exported function).
|
|
89
|
+
|
|
90
|
+
### 3. Run migrations and generate the import map
|
|
91
|
+
|
|
92
|
+
Generate and review the migration in your host app, then apply it. Regenerate the import map so the Tests screens and goal pickers are available.
|
|
93
|
+
|
|
94
|
+
```bash
|
|
127
95
|
pnpm exec payload migrate:create
|
|
128
96
|
pnpm exec payload migrate
|
|
97
|
+
pnpm exec payload generate:importmap
|
|
129
98
|
```
|
|
130
99
|
|
|
131
|
-
|
|
100
|
+
### 4. Wire the Next.js proxy
|
|
132
101
|
|
|
133
|
-
|
|
102
|
+
Create your app's `proxy.ts`. This assigns the version to show and handles
|
|
103
|
+
whole-page rewrites.
|
|
134
104
|
|
|
135
105
|
```ts
|
|
106
|
+
// src/proxy.ts
|
|
136
107
|
import { createExperimentProxy } from '@delmaredigital/payload-experiments/proxy'
|
|
137
108
|
import { proxyRuntime } from '@/experiments/proxyRuntime'
|
|
138
109
|
|
|
139
110
|
export const proxy = createExperimentProxy(proxyRuntime)
|
|
140
111
|
```
|
|
141
112
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
113
|
+
### 5. Add a results cron route
|
|
114
|
+
|
|
115
|
+
Your host schedules and authenticates this route. Choose a budget below your platform's route timeout, and schedule regular calls so results refresh.
|
|
145
116
|
|
|
146
117
|
```ts
|
|
147
|
-
|
|
148
|
-
import {
|
|
149
|
-
|
|
150
|
-
import {
|
|
118
|
+
// src/app/api/cron/xp-results/route.ts
|
|
119
|
+
import { runResults } from '@delmaredigital/payload-experiments/rsc'
|
|
120
|
+
import { serverRuntime } from '@/experiments/serverRuntime'
|
|
121
|
+
import { isAuthorizedCronRequest } from '@/auth/cron'
|
|
151
122
|
|
|
152
|
-
export async function
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
return
|
|
123
|
+
export async function GET(request: Request) {
|
|
124
|
+
if (!(await isAuthorizedCronRequest(request))) {
|
|
125
|
+
return new Response('Unauthorized', { status: 401 })
|
|
126
|
+
}
|
|
127
|
+
const payload = await serverRuntime.getPayload()
|
|
128
|
+
const result = await runResults(payload, { budgetMs: 20_000 })
|
|
129
|
+
return Response.json(result, { headers: { 'Cache-Control': 'no-store' } })
|
|
159
130
|
}
|
|
160
131
|
```
|
|
161
132
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
is forwarded into the same request, so server readers see it immediately.
|
|
166
|
-
Inbound `x-xp-*` headers are removed before any skip check. The proxy writes
|
|
167
|
-
signed `x-xp-assignments` (at most 1 KB) and `x-xp-request` (method, path/search
|
|
168
|
-
and consent, at most 2048 bytes). Server readers verify these; unsigned or
|
|
169
|
-
changed values are ignored, with a safe fallback when context is missing.
|
|
170
|
-
|
|
171
|
-
`cookie.consent(request)` must read **cookies and headers only**: server
|
|
172
|
-
conversion and identity helpers call it with a placeholder URL. When it returns
|
|
173
|
-
`false`, nobody is enrolled, including signed-in users; no visitor id is minted
|
|
174
|
-
or honoured, and an existing unit cookie is removed from the forwarded request.
|
|
175
|
-
Running tests show the Original. Decided winners without an active holdout
|
|
176
|
-
round still serve the decided content subject to targeting. Archived page
|
|
177
|
-
rewrites serve shipped content by path alone. Neither records exposure.
|
|
178
|
-
`linkIdentity` and `trackConversion`
|
|
179
|
-
also become no-ops when consent is declined. Omitting `consent` permits cookies
|
|
180
|
-
and enrolment.
|
|
181
|
-
|
|
182
|
-
Optional runtime `getUnit(request)` returns `{ userId?, attributes,
|
|
183
|
-
sessionUnknown? }`, synchronously or asynchronously. Use opaque user ids. With
|
|
184
|
-
no session token, return known logged-out attributes such as
|
|
185
|
-
`{ loggedIn: false, staff: false }`. With a verified session cache, return the
|
|
186
|
-
known user and attributes; with an unresolved session, set `sessionUnknown:
|
|
187
|
-
true`. User-unit experiments and session-dependent targeting can then be
|
|
188
|
-
resolved on the server through `getUser`. Server runtime options `userIdOf`
|
|
189
|
-
(default `String(user.id)`) and `attributesOf` map the host user to the unit id
|
|
190
|
-
and targeting attributes; server attributes take precedence. `attributesOf`
|
|
191
|
-
defaults to `{ loggedIn: Boolean(user) }`. If you declare `staff` in
|
|
192
|
-
`attributeKeys`, pass your own `attributesOf` that identifies staff to
|
|
193
|
-
`defineExperimentsServerRuntime`; the plugin refuses to start otherwise. The plugin owns the
|
|
194
|
-
normalized `path` attribute (no search, no trailing slash except `/`).
|
|
195
|
-
|
|
196
|
-
Staff, bots detected by the built-in user-agent list, preview renders and `HEAD`
|
|
197
|
-
requests produce no exposures. A signed `xp_preview` token can select a group
|
|
198
|
-
for review without counting it; preview responses are not cached. The plugin
|
|
199
|
-
skips admin, API, Next internals and asset paths. Host editor renders must also
|
|
200
|
-
remain outside exposure recording.
|
|
201
|
-
|
|
202
|
-
Whole-page experiments use visitor units only. Requests to any target path are
|
|
203
|
-
rewritten to the assigned group's page; a holdout sees the Original page. After
|
|
204
|
-
a winner is made permanent, losing paths keep rewriting to the winner while
|
|
205
|
-
the archived experiment record exists. A live experiment targeting a path takes
|
|
206
|
-
precedence. Decided winners remain subject to targeting.
|
|
207
|
-
|
|
208
|
-
## Server components (`/rsc`)
|
|
209
|
-
|
|
210
|
-
Import server helpers from `@delmaredigital/payload-experiments/rsc`.
|
|
211
|
-
`getAssignments(serverRuntime, { user? })` reads and verifies request context,
|
|
212
|
-
resolves deferred users, and recomputes stale or missing assignments using the
|
|
213
|
-
signed cookie and manifest. It is memoised per request. It reads dynamic Next
|
|
214
|
-
headers: call it inside `<Suspense>` under Cache Components, outside
|
|
215
|
-
`'use cache'`.
|
|
216
|
-
|
|
217
|
-
For Puck content, resolve before rendering and record only the returned
|
|
218
|
-
`rendered` list after the response. Capture all request values before `after()`:
|
|
133
|
+
### 6. Render the assigned version
|
|
134
|
+
|
|
135
|
+
For Puck pages, resolve the saved content before rendering it. Your page loader and render config stay in your app; add the [Puck integration](#puck-optional). For a Payload blocks field, see [Payload blocks (without Puck)](#payload-blocks-without-puck).
|
|
219
136
|
|
|
220
137
|
```tsx
|
|
138
|
+
// src/components/TestedPage.tsx
|
|
221
139
|
import { after } from 'next/server'
|
|
222
|
-
import {
|
|
223
|
-
|
|
140
|
+
import { Render } from '@puckeditor/core'
|
|
141
|
+
import {
|
|
142
|
+
getAssignments, resolveExperimentData, recordExposures,
|
|
143
|
+
} from '@delmaredigital/payload-experiments/rsc'
|
|
224
144
|
import { serverRuntime } from '@/experiments/serverRuntime'
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
145
|
+
import { loadPage } from '@/pages/loadPage'
|
|
146
|
+
import { renderConfig } from '@/puck/renderConfig'
|
|
147
|
+
|
|
148
|
+
export async function TestedPage({ path }: { path: string }) {
|
|
149
|
+
const page = await loadPage(path)
|
|
150
|
+
const assignments = await getAssignments(serverRuntime)
|
|
151
|
+
const { data, rendered } = resolveExperimentData(page.puckData, assignments)
|
|
152
|
+
after(() => recordExposures(serverRuntime, rendered, assignments))
|
|
153
|
+
return <Render config={renderConfig} data={data} />
|
|
154
|
+
}
|
|
231
155
|
```
|
|
232
156
|
|
|
233
|
-
|
|
234
|
-
`
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
assignment map. Only schedule it when you render that resolved content.
|
|
238
|
-
Viewport and pageview exposures come from the client; manual flag/code
|
|
239
|
-
exposures require an explicit host call. Non-enrolled units see the Original
|
|
240
|
-
without being counted. Same-unit exposures to multiple groups in a round are
|
|
241
|
-
excluded and reported.
|
|
242
|
-
|
|
243
|
-
Next's default prefetch does not render the dynamic subtree in `<Suspense>`.
|
|
244
|
-
Full prefetch (`<Link prefetch={true}>` or full `router.prefetch`) of pages with
|
|
245
|
-
`render`-policy slots is unsupported: it would record content before a visit.
|
|
246
|
-
|
|
247
|
-
`toClientAssignments(assignments, { pathname, apiRoute?, flagPoints?,
|
|
248
|
-
experiments? })` creates the provider value without visitor identifiers. Both
|
|
249
|
-
allow-lists default to empty. Use the served page's pathname for pageview
|
|
250
|
-
logging; see the client integration below.
|
|
251
|
-
|
|
252
|
-
`getFlagPoint(serverRuntime, id, { user? })` returns `arm`, `params`, `source`
|
|
253
|
-
(`experiment` or `default`), optional `experiment`/`experimentId`, and
|
|
254
|
-
`expose(renderedArm)`. Bind host code to the registered point id. It uses the
|
|
255
|
-
live experiment, then saved winner defaults, then the point's `defaultParams`.
|
|
256
|
-
Render the returned settings and call `await point.expose(point.arm)` only
|
|
257
|
-
when that arm's content is actually delivered; this records only the manual
|
|
258
|
-
exposure policy. Nothing is exposed by reading the point.
|
|
259
|
-
|
|
260
|
-
Other server integrations:
|
|
261
|
-
|
|
262
|
-
- `trackConversion(serverRuntime, { eventName, value?, idempotencyKey, userId?,
|
|
263
|
-
headers? })` records a server event (default value `1`). Use a stable, globally
|
|
264
|
-
unique idempotency key for each business event. Pass request headers when
|
|
265
|
-
outside a Next request; a user-only server conversion can supply `userId`.
|
|
266
|
-
Record revenue through trusted server code or a metric source, never a client
|
|
267
|
-
beacon.
|
|
268
|
-
- `linkIdentity(serverRuntime, { headers, userId, verified })` belongs on every
|
|
269
|
-
user-creation and proving sign-in path; see Identity linking below.
|
|
270
|
-
- `eraseUser(payload, { userId })` deletes the user's identities and associated
|
|
271
|
-
events transactionally and returns `{ identities, events }` counts. Call it
|
|
272
|
-
from the host's authorized user-erasure flow.
|
|
273
|
-
- `recordContentChange({ doc, previousDoc, req, collection })` can be called
|
|
274
|
-
from a host collection's `afterChange` for content tagged with the arm-select
|
|
275
|
-
field; for a collection with drafts, also add `captureContentBaseline` to its
|
|
276
|
-
`beforeChange` hooks so the previously published content can be compared.
|
|
277
|
-
Collections without drafts can also add `captureContentBaseline` for an exact
|
|
278
|
-
raw comparison; without it, the helper normalises relationships to ids. No host
|
|
279
|
-
change is required for collections without drafts. Supply that hook's transaction request. Puck collections get the hook
|
|
280
|
-
automatically when enabled. Markers compare published content; drafts and
|
|
281
|
-
autosaves do not add markers (collections without drafts record on save).
|
|
282
|
-
Neither the helper nor the installed hooks ever throw into your save: a
|
|
283
|
-
failure is logged and the content save proceeds.
|
|
284
|
-
- `experimentMetadata(serverRuntime, path)` returns metadata to merge into the
|
|
285
|
-
page's Next metadata: live variant targets get noindex and an Original
|
|
286
|
-
canonical; archived losing paths point canonically to the winner. It reads
|
|
287
|
-
the signed request context via `headers()`, so it decides from the URL the
|
|
288
|
-
visitor requested, not the rewritten route, and the Original's URL is never
|
|
289
|
-
noindex. While a test is live the Original's URL also gets a canonical to
|
|
290
|
-
itself, even when it is served with a variant's route: merge this result after
|
|
291
|
-
your own metadata so its `alternates.canonical` wins over one derived from the
|
|
292
|
-
rendered route. Call it only from a dynamic `generateMetadata`, never inside
|
|
293
|
-
`'use cache'`; `path` is the fallback when the proxy did not run.
|
|
294
|
-
`experimentSitemapFilter(path, manifest)` is a synchronous build-time filter:
|
|
295
|
-
pass a privately server-loaded manifest and keep paths for which it returns
|
|
296
|
-
`true`, excluding live variants and archived losing paths.
|
|
297
|
-
|
|
298
|
-
## Identity linking
|
|
299
|
-
|
|
300
|
-
Call `linkIdentity(serverRuntime, { headers, userId, verified })` on every
|
|
301
|
-
user-creation and proving sign-in path. It links the signed anonymous cookie
|
|
302
|
-
to an opaque user id on its own connection and never throws into sign-in.
|
|
303
|
-
`verified` is required. Pass `true` only when **this request proves, in this
|
|
304
|
-
browser, that the person owns the account**: a magic-link or one-time-code
|
|
305
|
-
callback, password sign-in, or OAuth callback. It does not mean "a session
|
|
306
|
-
exists" or "the account's email is verified". Registration alone uses `false`.
|
|
157
|
+
Render this async component inside `<Suspense>` and outside `'use cache'`.
|
|
158
|
+
Only record the returned `rendered` list when you deliver that content. Keep
|
|
159
|
+
editor and preview renders out of this path. Use Next's default prefetch;
|
|
160
|
+
full prefetch of pages that count content on server render is unsupported.
|
|
307
161
|
|
|
308
|
-
|
|
309
|
-
// Registration, before proof of account ownership:
|
|
310
|
-
await linkIdentity(serverRuntime, { headers, userId, verified: false })
|
|
311
|
-
// After successful magic-link/code verification in this browser:
|
|
312
|
-
await linkIdentity(serverRuntime, { headers, userId, verified: true })
|
|
313
|
-
// After a successful password sign-in (or OAuth callback):
|
|
314
|
-
await linkIdentity(serverRuntime, { headers, userId, verified: true })
|
|
315
|
-
```
|
|
162
|
+
## Running a test
|
|
316
163
|
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
link
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
Each event goes to the earliest exposed eligible visitor. Eligible links
|
|
330
|
-
spanning groups are reported as a diagnostic. User-keyed CUPED covariates use
|
|
331
|
-
the same eligibility rule at each visitor's first exposure and credit one
|
|
332
|
-
aggregate value to the earliest eligible visitor per user.
|
|
333
|
-
|
|
334
|
-
Accepted limits:
|
|
335
|
-
|
|
336
|
-
- Cross-device: register in A, verify on B, keep using A. A's events after B's
|
|
337
|
-
proof are not credited until a proving sign-in on A upgrades A. Events before
|
|
338
|
-
B's proof, including the registration funnel, remain credited. Once A is
|
|
339
|
-
upgraded, its link is verified and eligible for all its events, so the next
|
|
340
|
-
results run also credits A's events from between B's proof and the upgrade.
|
|
341
|
-
- Password set before verification: a later password sign-in is proof the
|
|
342
|
-
plugin accepts. Whether unverified registration may set a password is a host
|
|
343
|
-
authentication concern.
|
|
344
|
-
- Shared device: a different user's verified sign-in never relinks an already
|
|
345
|
-
linked browser and returns `'skipped'`; that user's events get no credit
|
|
346
|
-
through that browser.
|
|
347
|
-
- If someone registers another person's email from their own browser and that
|
|
348
|
-
person never signs in, a later user-keyed event the host records against that
|
|
349
|
-
account (for example an offline gift) is credited to the registering browser.
|
|
350
|
-
It needs a deliberate registration of someone else's email plus an event
|
|
351
|
-
matched to that account, and it moves one unit's value, never assignment.
|
|
352
|
-
There is no time window, so recurring gifts from never-signed-in donors keep
|
|
353
|
-
counting.
|
|
354
|
-
|
|
355
|
-
## Results
|
|
356
|
-
|
|
357
|
-
The host owns scheduling and authentication. Call
|
|
358
|
-
`runResults(payload, { budgetMs })` in-process from your cron route; no Payload
|
|
359
|
-
jobs queue or HTTP call back into the application is needed:
|
|
164
|
+
1. Start from the content: add an **Experiment block** (in Puck's **A/B tests** category),
|
|
165
|
+
or open **Tests** in the admin to test whole pages or a registered setting.
|
|
166
|
+
2. Keep the **Original**, use **Copy Original here** in Puck to make a version,
|
|
167
|
+
and make your changes. For whole pages, use Payload's **Duplicate** action, give the
|
|
168
|
+
copy its own address, publish it, then choose it as a version in the test.
|
|
169
|
+
3. Pick a **goal**: a signup, purchase, or a named click goal on a button or
|
|
170
|
+
link. Add guardrail goals for outcomes you don't want to hurt.
|
|
171
|
+
4. Save, follow the setup checks, preview each version ([wire preview links](#preview-and-editor-links)), and **Start**. The
|
|
172
|
+
results explain whether to keep running or choose a winner.
|
|
173
|
+
5. Use **Decide** to pick the winner, then **Make it the content** to make it permanent.
|
|
174
|
+
You can also **Stop** to show everyone the Original; starting again begins
|
|
175
|
+
a new round with fresh results.
|
|
360
176
|
|
|
361
|
-
|
|
362
|
-
// app/api/cron/xp-results/route.ts
|
|
363
|
-
import { runResults } from '@delmaredigital/payload-experiments/rsc'
|
|
364
|
-
import { serverRuntime } from '@/experiments/serverRuntime'
|
|
365
|
-
import { isAuthorizedCronRequest } from '@/auth/cron' // host-owned auth
|
|
177
|
+
A click tells you someone took an action; it doesn't prove they completed a purchase or signup. Use the outcome itself as a goal when you can measure it.
|
|
366
178
|
|
|
367
|
-
|
|
368
|
-
if (!(await isAuthorizedCronRequest(request))) {
|
|
369
|
-
return new Response('Unauthorized', { status: 401 })
|
|
370
|
-
}
|
|
371
|
-
const payload = await serverRuntime.getPayload()
|
|
372
|
-
const result = await runResults(payload, { budgetMs: 20_000 })
|
|
373
|
-
return Response.json(result, { headers: { 'Cache-Control': 'no-store' } })
|
|
374
|
-
}
|
|
375
|
-
```
|
|
179
|
+
## What your app wires
|
|
376
180
|
|
|
377
|
-
|
|
378
|
-
The runner prioritizes the oldest snapshots, checks the remaining budget before
|
|
379
|
-
metrics, bounds waiting for host `query`/`covariate` promises and database
|
|
380
|
-
connections, and applies a remaining-budget SQL statement timeout. Hung work
|
|
381
|
-
cannot keep the runner waiting beyond that budget. Source failures mark that
|
|
382
|
-
metric unavailable. A timed-out source is reported in `unavailableMetrics`;
|
|
383
|
-
that experiment's new snapshots are not written, its previous snapshots remain,
|
|
384
|
-
and `budgetExhausted` signals the cutoff. Racing a host promise does not cancel
|
|
385
|
-
its underlying work: give your own source clients appropriate timeouts too.
|
|
386
|
-
The returned summary also includes `ran`, `skipped`, `swept`, `sweepAttempted` and `processedIds`.
|
|
387
|
-
Optional `experimentId` limits analysis to one experiment; `sweep: false`
|
|
388
|
-
disables retention for that call.
|
|
389
|
-
|
|
390
|
-
Analysis uses each round's frozen config and actual exposures. Its window ends
|
|
391
|
-
at `endedAt`; later exposures and conversions are excluded. Finished rounds
|
|
392
|
-
have a **24-hour late-beacon grace** before their final snapshot freezes: once
|
|
393
|
-
a snapshot has been computed after that grace, later runs leave it unchanged.
|
|
394
|
-
This does not extend the attribution window beyond the round's end.
|
|
395
|
-
|
|
396
|
-
Event retention defaults to **180 days after the experiment's final round
|
|
397
|
-
ends**, configurable with `events.retentionDays`. No experiment events are
|
|
398
|
-
swept while any of its rounds is open, including a long-lived holdout round.
|
|
399
|
-
The runner reserves part of its budget for the batched retention sweep.
|
|
400
|
-
|
|
401
|
-
Routes below are relative to Payload's `config.routes.api` (normally `/api`):
|
|
402
|
-
|
|
403
|
-
| Endpoint | Permission and response |
|
|
404
|
-
| --- | --- |
|
|
405
|
-
| `GET /xp/experiments/:id/results?phase=1&series=1` | `access.readResults`; round summaries, newest selected-round `snapshot` (`null` before a run), optional `decisionSnapshot` and requested time series. Without `phase`, a decided experiment opens on its decision round. |
|
|
406
|
-
| `GET /xp/experiments/summary?ids=...` | `access.readResults`; map of experiment ids to exposures, days and verdict summary. Encode each id individually, comma-join, and request at most 100 ids. |
|
|
407
|
-
| `POST /xp/experiments/:id/recompute` | `access.manage`; runs results for one experiment without sweeping and returns `{ snapshotId }`. Failed or budget-exhausted work returns an error. Frozen rounds stay frozen. |
|
|
181
|
+
### Proxy
|
|
408
182
|
|
|
409
|
-
|
|
183
|
+
Keep the [proxy from step 4](#4-wire-the-nextjs-proxy) on your public routes.
|
|
184
|
+
If you already have a proxy, compose with `resolveExperiments` and
|
|
185
|
+
`applyExperiments` from `/proxy`: forward the returned `requestHeaders` on
|
|
186
|
+
both normal responses and rewrites, then apply the returned cookies and headers.
|
|
187
|
+
|
|
188
|
+
### Whole-page tests
|
|
410
189
|
|
|
411
|
-
|
|
412
|
-
`defineMetricSource` are optional typed identity helpers from the root export;
|
|
413
|
-
the root also exports the `FlagPoint`, `ParamField` and `MetricSource` types.
|
|
414
|
-
|
|
415
|
-
A `FlagPoint` has `id`, `label`, `params` and `defaultParams`. `params` maps
|
|
416
|
-
setting names to `ParamField` definitions: `{ type, label, required?, options?,
|
|
417
|
-
min?, max? }`. Supported types are `text`, `textarea`, `number`, `checkbox`,
|
|
418
|
-
`select`; select options are `{ label, value }`. Keep defaults valid for that
|
|
419
|
-
schema. Optional callbacks integrate host-owned content:
|
|
420
|
-
|
|
421
|
-
- `placements({ experimentId, arms, req })` returns
|
|
422
|
-
`[{ arm, ok, detail? }]` for the required content groups at Start. Holdouts
|
|
423
|
-
use Original content and are not passed as required placements.
|
|
424
|
-
- `makeContent({ experimentId, winner: { arm, params }, req })` runs when the
|
|
425
|
-
chosen settings are made permanent. Match host tags on both experiment id
|
|
426
|
-
and arm. Use the manager's `req` and `overrideAccess: false` for host-document
|
|
427
|
-
writes. The plugin saves the winning settings as defaults for the point.
|
|
428
|
-
|
|
429
|
-
A `MetricSource`, registered in `metricSources`, has `id`, `label`,
|
|
430
|
-
`kind: 'binomial' | 'count' | 'revenue'`, and `keyedBy: 'visitor' | 'user'`.
|
|
431
|
-
Source ids starting with `click:` are reserved for click goals and refused at boot.
|
|
432
|
-
`clientWritable` optionally permits client conversions (revenue is always
|
|
433
|
-
server-only). `query({ from, to, unitIds? })` returns a promise of
|
|
434
|
-
`[{ unitId, value, ts }]`, **one row per event**, with `ts` a `Date`; use the
|
|
435
|
-
same opaque ids as assignment or identity linking. A user experiment requires
|
|
436
|
-
a user-keyed source. Visitor experiments can use a user-keyed source through
|
|
437
|
-
linked identities.
|
|
438
|
-
|
|
439
|
-
Optional `covariate({ units: [{ unitId, firstExposureTs }], lookbackDays })`
|
|
440
|
-
returns `[{ unitId, value }]`, one aggregate value per unit for pre-exposure
|
|
441
|
-
CUPED adjustment (the plugin asks for a 14-day lookback). Percentile
|
|
442
|
-
revenue capping reuses `query` over the 90 days before Start; the resolved cap
|
|
443
|
-
is frozen for the round. Capping affects decision statistics, while results
|
|
444
|
-
also show uncapped totals. Count and binomial metrics are never capped.
|
|
445
|
-
|
|
446
|
-
For simple yes/no outcomes, list names in `eventSources`, for example
|
|
447
|
-
`eventSources: ['signup']`. A bare string allows client writes; the object form
|
|
448
|
-
sets that permission explicitly:
|
|
190
|
+
Add this to the plugin from step 2 for a drafts-enabled Pages collection:
|
|
449
191
|
|
|
450
192
|
```ts
|
|
451
|
-
|
|
193
|
+
pages: { collections: [{
|
|
194
|
+
slug: 'pages',
|
|
195
|
+
pathOf: doc => typeof doc.slug === 'string' ? `/${doc.slug}` : null,
|
|
196
|
+
findByPath: async (path, req) => {
|
|
197
|
+
const { docs } = await req.payload.find({
|
|
198
|
+
collection: 'pages', req, overrideAccess: false, draft: false, depth: 0, limit: 1,
|
|
199
|
+
where: { and: [
|
|
200
|
+
{ slug: { equals: path.slice(1) } },
|
|
201
|
+
{ _status: { equals: 'published' } },
|
|
202
|
+
] },
|
|
203
|
+
})
|
|
204
|
+
return docs[0] ? { id: docs[0].id, published: true } : null
|
|
205
|
+
},
|
|
206
|
+
}] }
|
|
452
207
|
```
|
|
453
208
|
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
by stored conversions with the corresponding `eventName`. Writable sources
|
|
457
|
-
accept both `trackConversion` and `trackConversionBeacon`; server-only sources
|
|
458
|
-
are recorded with `trackConversion`. Beacon conversions for a non-writable name
|
|
459
|
-
are dropped silently. Only listed names become `events:<name>` sources. They
|
|
460
|
-
support visitor and user units and do not represent counts or revenue in v1.
|
|
209
|
+
Adapt both callbacks for your homepage. Add `contentOf: doc => doc.layout` to select content for comparison. Keep shipped winner pages published at the same address.
|
|
210
|
+
In dynamic `generateMetadata`, merge `await experimentMetadata(serverRuntime, path)` from `/rsc` **after** your own metadata so its canonical and robots settings win; keep it outside `'use cache'`. Use `experimentSitemapFilter` from `/rsc` in your sitemap.
|
|
461
211
|
|
|
462
|
-
###
|
|
212
|
+
### Preview and editor links
|
|
463
213
|
|
|
464
|
-
|
|
465
|
-
returns `{ manifest, rows }`. States are `running`, `decided` and `archived`;
|
|
466
|
-
provide a winner path for the latter two. Build the host's test runtime with
|
|
467
|
-
`loadManifest: async () => manifest` and the same secret. For each row, set
|
|
468
|
-
`next/headers` to its signed `headers`, render metadata for `servedPath`, and
|
|
469
|
-
assert `expect.robots`, `expect.canonical` and `expect.inSitemap` against the
|
|
470
|
-
host's metadata and sitemap functions. This includes the Original served as
|
|
471
|
-
each version, including the homepage.
|
|
214
|
+
The setup checks in [Running a test, step 4](#running-a-test) get their preview links from `admin.previewUrl`. Add these options to the plugin, adapting the routes to your app:
|
|
472
215
|
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
216
|
+
```ts
|
|
217
|
+
admin: {
|
|
218
|
+
previewUrl: ({ collection, slug }) => collection === 'pages' && slug !== undefined
|
|
219
|
+
? (slug === 'home' ? '/' : `/${slug}`) : null,
|
|
220
|
+
editorUrl: ({ collection, id, editor }) => editor === 'puck'
|
|
221
|
+
? `/admin/puck-editor/${encodeURIComponent(collection)}/${encodeURIComponent(String(id))}`
|
|
222
|
+
: `/admin/collections/${encodeURIComponent(collection)}/${encodeURIComponent(String(id))}`,
|
|
223
|
+
}
|
|
224
|
+
```
|
|
476
225
|
|
|
477
|
-
|
|
226
|
+
`previewUrl` receives `PreviewPlacement`: `collection`, `id`, `title`, `published`, and optional `slug` from the published page (absent for unpublished pages). Return `null` to hide preview. `editorUrl` receives `collection`, `id`, optional `draft` and `doc`, and `editor: 'puck' | 'blocks' | 'page'` to choose the editor route.
|
|
478
227
|
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
228
|
+
### Results schedule
|
|
229
|
+
|
|
230
|
+
The [route in step 5](#5-add-a-results-cron-route) runs results in your app.
|
|
231
|
+
On Vercel, add it to your cron configuration and check its authentication.
|
|
232
|
+
The host owns the schedule; the plugin does not use Payload jobs.
|
|
482
233
|
|
|
483
|
-
|
|
234
|
+
### Migrations
|
|
484
235
|
|
|
485
|
-
|
|
486
|
-
|
|
236
|
+
The schema belongs to your app. Repeat [step 3](#3-run-migrations-and-generate-the-import-map) after every upgrade that changes it, including fields you add to host blocks.
|
|
237
|
+
|
|
238
|
+
### Signing users in
|
|
239
|
+
|
|
240
|
+
If your site has logins, extend the [server runtime in step 1](#1-set-a-secret-and-create-the-runtimes)
|
|
241
|
+
with `getUser` for your session lookup and `attributesOf` for non-identifying
|
|
242
|
+
attributes such as `loggedIn` and `staff`. Identify staff so their visits don't
|
|
243
|
+
count. In the proxy runtime, supply `getUnit`: return `{ userId, attributes }`
|
|
244
|
+
from a verified session, with `loggedIn` and `staff` in `attributes`; return
|
|
245
|
+
known logged-out attributes when there is no session token, or set
|
|
246
|
+
`sessionUnknown: true` when a token still needs checking on the server.
|
|
247
|
+
Declare `staff: { type: 'boolean' }` in the plugin's `attributeKeys`.
|
|
248
|
+
Call `linkIdentity` at registration and after a successful sign-in:
|
|
487
249
|
|
|
488
250
|
```ts
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
const { docs } = await req.payload.find({
|
|
495
|
-
collection: 'pages', req, overrideAccess: false, draft: false, depth: 0, limit: 1,
|
|
496
|
-
where: { and: [
|
|
497
|
-
{ slug: { equals: path.slice(1) } },
|
|
498
|
-
{ _status: { equals: 'published' } },
|
|
499
|
-
] },
|
|
500
|
-
})
|
|
501
|
-
return docs[0] ? { id: docs[0].id, published: true } : null
|
|
502
|
-
},
|
|
503
|
-
}],
|
|
504
|
-
},
|
|
251
|
+
import { linkIdentity } from '@delmaredigital/payload-experiments/rsc'
|
|
252
|
+
import { serverRuntime } from '@/experiments/serverRuntime'
|
|
253
|
+
|
|
254
|
+
// In your successful password, magic-link/code, or OAuth callback:
|
|
255
|
+
await linkIdentity(serverRuntime, { headers: request.headers, userId, verified: true })
|
|
505
256
|
```
|
|
506
257
|
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
Whole-page registrations also install the sidebar test status and content-change hooks.
|
|
511
|
-
For 0.4.0, run `payload migrate:create` for `xp_state.retired_slugs`,
|
|
512
|
-
`xp_experiments.page_collection` and its version column, then
|
|
513
|
-
`payload generate:importmap`.
|
|
258
|
+
`verified` is required: use `true` only when this request proves ownership of
|
|
259
|
+
the account in this browser. Registration without that proof uses `false`.
|
|
260
|
+
A session refresh or an account's verified-email flag is not proof here.
|
|
514
261
|
|
|
515
|
-
|
|
516
|
-
`doc.layout`). Published copies are compared with nested `id` and `blockName`
|
|
517
|
-
keys removed. Without it, content checks and page content-change comparisons
|
|
518
|
-
are skipped. `copyHint` (up to 200 characters), `copyUrl` (a root-relative path)
|
|
519
|
-
and `copyLabel` (up to 40 characters, requires `copyUrl`) customize the guidance
|
|
520
|
-
for making version pages. The default copy link opens the registered collection's
|
|
521
|
-
list under the configured admin route.
|
|
262
|
+
### Erasing a user
|
|
522
263
|
|
|
523
|
-
|
|
264
|
+
When you erase a user, call `eraseUser(payload, { userId })` from `/rsc` so their test data goes too; see the [full reference](https://github.com/delmaredigital/payload-experiments/blob/main/packages/plugin/REFERENCE.md#server-components-rsc).
|
|
524
265
|
|
|
525
|
-
|
|
526
|
-
editor route, public pages and a `puck-templates` collection:
|
|
266
|
+
### Puck (optional)
|
|
527
267
|
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
268
|
+
For [steps 2](#2-add-the-plugin) and [6](#6-render-the-assigned-version), add
|
|
269
|
+
`puck: { enabled: true, collections: ['pages'] }` to the plugin. A collection
|
|
270
|
+
slug uses its `puckData` field; use `{ slug: 'pages', field: 'content' }` for another.
|
|
271
|
+
Works with `@delmaredigital/payload-puck >=0.10.3` and experiments `>=0.4.0`.
|
|
531
272
|
|
|
532
|
-
|
|
273
|
+
In your render config, import `experimentsPuck` from `/puck` and pass
|
|
274
|
+
`integrations: [experimentsPuck()]` to payload-puck's `extendConfig`. In a
|
|
275
|
+
`'use client'` editor module, import `experimentsPuckEditor` from `/puck/editor`,
|
|
276
|
+
create `const integrations = [experimentsPuckEditor()]` once at module scope,
|
|
277
|
+
and pass it to `PuckConfigProvider`. Pass your custom `routes` to both helpers.
|
|
278
|
+
Load this in your iframe stylesheet, after Tailwind if you use it, and list that stylesheet's URL in payload-puck's `editorStylesheets`:
|
|
533
279
|
|
|
534
|
-
```
|
|
535
|
-
|
|
536
|
-
editorUrl: ({ collection, id, draft, editor, doc }) =>
|
|
537
|
-
`/admin/puck-editor/${encodeURIComponent(collection)}/${encodeURIComponent(String(id))}`,
|
|
538
|
-
previewUrl: async ({ collection, slug }) => {
|
|
539
|
-
if (collection !== 'pages' || slug === undefined) return null
|
|
540
|
-
// Match the site's public page routing, including its homepage convention.
|
|
541
|
-
return slug === 'home' ? '/' : `/${slug}`
|
|
542
|
-
},
|
|
543
|
-
descriptions: {
|
|
544
|
-
block: 'Try different content on a landing page.',
|
|
545
|
-
document: 'Try different versions of a donation ask.',
|
|
546
|
-
setting: 'Try a different signup or donation prompt.',
|
|
547
|
-
},
|
|
548
|
-
}
|
|
280
|
+
```css
|
|
281
|
+
@import "@delmaredigital/payload-experiments/puck/editor.css";
|
|
549
282
|
```
|
|
550
283
|
|
|
551
|
-
|
|
552
|
-
the published main row's string slug, including for a placement found in a newer
|
|
553
|
-
draft with a different slug. It is absent for unpublished documents. `published`
|
|
554
|
-
describes the placement's version; the callback receives no other document fields.
|
|
555
|
-
An async callback can also look up the public path by `collection` and `id` using
|
|
556
|
-
the host's own data access. The old `({ collection, id }) => string` form still works.
|
|
557
|
-
|
|
558
|
-
Preflight resolves the URL callbacks on the server for block placements and sends
|
|
559
|
-
only URL strings to the client. A null/undefined result, a non-string, an exception,
|
|
560
|
-
a rejected promise or a timeout leaves preview hidden for that placement. When an
|
|
561
|
-
editor URL exists, the admin offers "Edit on page"; when neither URL exists, it
|
|
562
|
-
omits the row. Timed-out host work cannot be cancelled and keeps its concurrency
|
|
563
|
-
slot until it settles; if all 8 slots remain occupied, further preview lookups
|
|
564
|
-
are skipped for that preflight. Kind descriptions are available to the server views through
|
|
565
|
-
`config.custom.xp`.
|
|
566
|
-
|
|
567
|
-
A flag point can also declare `description?: string` to explain the setting below its label.
|
|
568
|
-
|
|
569
|
-
## Security
|
|
570
|
-
|
|
571
|
-
The unit cookie, assignments header, request-context header and preview tokens
|
|
572
|
-
are HMAC-SHA256 signed with purpose-separated inputs and constant-time
|
|
573
|
-
verification. Signing prevents a browser from choosing a trusted identity,
|
|
574
|
-
assigned arm, request context or preview group by editing a value. The manifest
|
|
575
|
-
is private. Store no PII beyond opaque unit ids: keep names, emails and other
|
|
576
|
-
identifying values out of attributes, params and event data.
|
|
577
|
-
|
|
578
|
-
Every collection/global access operation is explicit: create, read, update,
|
|
579
|
-
delete, readVersions and unlock. Managers operate experiments, layers and
|
|
580
|
-
metrics; results and identity reads have separate permissions. Internal event,
|
|
581
|
-
identity, round, snapshot and state writes are denied through ordinary
|
|
582
|
-
collection/global access. Every Payload Local API call declares
|
|
583
|
-
`overrideAccess`; request handlers and views pass `req` and
|
|
584
|
-
`overrideAccess: false`. The enumerated privileged operations are the events
|
|
585
|
-
insert and assignment lookup, `recordExposures`, server `trackConversion`,
|
|
586
|
-
`linkIdentity`, manifest reads, transition writes to plugin collections/state,
|
|
587
|
-
content-change markers, results writes and retention sweep, and `eraseUser`.
|
|
588
|
-
Host content writes when making a winner permanent still use the manager's
|
|
589
|
-
access. Privileged side effects follow their authorized operation in the same
|
|
590
|
-
transaction; identity linking deliberately uses its own insert-ignore connection
|
|
591
|
-
so duplicate links cannot abort sign-in.
|
|
592
|
-
|
|
593
|
-
`POST /xp/events` requires a valid signed unit cookie, reads a streamed body
|
|
594
|
-
with an **8192-byte default limit**, and accepts at most **20 events** per body.
|
|
595
|
-
It parses JSON regardless of content type for beacon compatibility. Exposure
|
|
596
|
-
beacons must name the current round, policy and the arm actually rendered;
|
|
597
|
-
the server recomputes assignment and drops mismatches while counting them
|
|
598
|
-
separately. Only manual, viewport and pageview exposure policies are accepted.
|
|
599
|
-
Conversions require a client-writable source, never revenue; client values are
|
|
600
|
-
stored as `1`, attributes are allow-listed, and the host's `events.rateLimit`
|
|
601
|
-
adapter runs when supplied. Accepted and dropped events both return `202 {}`;
|
|
602
|
-
malformed requests can return errors. `click:<key>` conversions are also accepted
|
|
603
|
-
for defined click-goal keys matching `[a-z0-9][a-z0-9_-]{0,63}`; undefined or
|
|
604
|
-
invalid keys are dropped with `202`. Clicks carry no attributes, and defined
|
|
605
|
-
keys may be cached for up to 30 seconds.
|
|
606
|
-
|
|
607
|
-
**The real bound on slow or method-override bodies is the host firewall/platform
|
|
608
|
-
timeout.** A stream under the byte limit can remain open, and Payload can read
|
|
609
|
-
an `X-HTTP-Method-Override` body before the plugin's streamed cap runs. Apply
|
|
610
|
-
host firewall rules and platform request timeouts at that boundary.
|
|
611
|
-
|
|
612
|
-
Static checks enforce explicit access and keep raw SQL in the database modules,
|
|
613
|
-
but have accepted gaps: extracting a Payload method with a computed property
|
|
614
|
-
in destructuring; reflection through `Object.getOwnPropertyNames`,
|
|
615
|
-
`Object.getOwnPropertyDescriptor` or `Reflect.ownKeys`; capturing a computed
|
|
616
|
-
method from an unknown receiver and calling it later; `payload.jobs.*` calls
|
|
617
|
-
(the jobs queue is not used or supported here); lowercase SQL through `pg`;
|
|
618
|
-
and aliases changed through `let` reassignment. **Code review is the backstop**
|
|
619
|
-
for these patterns.
|
|
620
|
-
|
|
621
|
-
## Admin and client
|
|
622
|
-
|
|
623
|
-
The sidebar includes a **Tests** group with **All tests** and **Goals**. The
|
|
624
|
-
dashboard also includes **Tests** and **Goals** cards above its widgets. Goals
|
|
625
|
-
appears only for users who can manage tests. Both additions follow the host's
|
|
626
|
-
admin route and keep host-provided navigation and dashboard entries first.
|
|
627
|
-
After upgrading, run `payload generate:importmap` to register the dashboard card
|
|
628
|
-
component and the server nav wrapper.
|
|
629
|
-
A host nav group also labelled "Tests" shares Payload's NavGroup DOM id and open-state preference with the plugin's group, so pick another label.
|
|
630
|
-
|
|
631
|
-
### Client assignments and exposure
|
|
632
|
-
|
|
633
|
-
Read assignments on the server and pass them to `ExperimentsProvider` from
|
|
634
|
-
`@delmaredigital/payload-experiments/client`. Build its `value` with
|
|
635
|
-
`toClientAssignments` from `@delmaredigital/payload-experiments/rsc`:
|
|
284
|
+
### Payload blocks (without Puck)
|
|
636
285
|
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
286
|
+
Add `blocks: { collections: [{ slug: 'pages', field: 'layout' }] }` to the plugin from step 2. Add the wrapper to your host's blocks field, then repeat step 3:
|
|
287
|
+
|
|
288
|
+
```ts
|
|
289
|
+
import { createExperimentBlock } from '@delmaredigital/payload-experiments'
|
|
290
|
+
import { contentBlocks } from '@/blocks'
|
|
291
|
+
|
|
292
|
+
// In the Pages collection's fields array:
|
|
293
|
+
{ name: 'layout', type: 'blocks', blocks: [
|
|
294
|
+
...contentBlocks, createExperimentBlock({ blocks: contentBlocks }),
|
|
295
|
+
] }
|
|
645
296
|
```
|
|
646
297
|
|
|
647
|
-
|
|
648
|
-
only the flag points and experiments that the client needs. The provider value
|
|
649
|
-
contains no visitor identifier. Use **one `apiRoute` for every provider on a
|
|
650
|
-
page**, including nested providers; it is the API base, such as `/api`. When the
|
|
651
|
-
API isn't at `/api`, set that shared route through
|
|
652
|
-
`toClientAssignments(assignments, { pathname, apiRoute })`.
|
|
653
|
-
|
|
654
|
-
`useFlagPoint(id)` reads the current group and settings at a registered flag
|
|
655
|
-
point. `useExperiment(slug)` reads an allow-listed experiment. An unlisted flag
|
|
656
|
-
point or experiment slug throws in development and returns the Original with
|
|
657
|
-
one console error in production. Neither hook records exposure automatically.
|
|
658
|
-
For manual exposure, call
|
|
659
|
-
`expose(renderedArm)` only when that group's content actually showed, passing
|
|
660
|
-
the group key you rendered. Reading an assignment alone does not count as
|
|
661
|
-
seeing content. `expose()` sends only for experiments using the manual exposure
|
|
662
|
-
policy.
|
|
663
|
-
|
|
664
|
-
On each whole-page experiment target, render a nested `ExperimentsProvider`
|
|
665
|
-
whose value is built for **that page's pathname**, and put
|
|
666
|
-
`<ExperimentPageview />` inside it. This component reads from the nearest
|
|
667
|
-
provider and records the page that appeared, including after navigation.
|
|
668
|
-
|
|
669
|
-
Every page that renders Puck viewport slots also needs a provider. A
|
|
670
|
-
layout-level provider with empty allow-lists is enough:
|
|
671
|
-
`toClientAssignments(assignments, { pathname })`. Without a provider, viewport
|
|
672
|
-
exposures cannot be sent.
|
|
673
|
-
|
|
674
|
-
For a client-writable conversion, call
|
|
675
|
-
`trackConversionBeacon(eventName, { idempotencyKey })`. Pass a stable key for
|
|
676
|
-
conversions that matter, so retries count as the same conversion. It sends
|
|
677
|
-
nothing without a mounted provider. The server accepts only conversion sources
|
|
678
|
-
configured to allow client writes; record revenue on the server.
|
|
679
|
-
|
|
680
|
-
### Puck render and editor configs
|
|
681
|
-
|
|
682
|
-
Works with: payload-puck ≥0.10.3 (experiments ≥0.4.0).
|
|
683
|
-
|
|
684
|
-
For payload-puck hosts, add one integration object on each side. In the render
|
|
685
|
-
config, pass `integrations` to `extendConfig`:
|
|
298
|
+
Resolve before rendering, as in step 6:
|
|
686
299
|
|
|
687
300
|
```ts
|
|
688
|
-
import {
|
|
689
|
-
import {
|
|
690
|
-
|
|
301
|
+
import { after } from 'next/server'
|
|
302
|
+
import { getAssignments, resolveExperimentBlocks, recordExposures } from
|
|
303
|
+
'@delmaredigital/payload-experiments/rsc'
|
|
691
304
|
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
305
|
+
const assignments = await getAssignments(serverRuntime)
|
|
306
|
+
const { blocks, rendered } = resolveExperimentBlocks(page.layout, assignments)
|
|
307
|
+
after(() => recordExposures(serverRuntime, rendered, assignments))
|
|
308
|
+
// Render `blocks` with your host's block renderer.
|
|
696
309
|
```
|
|
697
310
|
|
|
698
|
-
|
|
699
|
-
then pass that array to `PuckConfigProvider`. A new object per render rebuilds
|
|
700
|
-
Puck's config and plugin list.
|
|
311
|
+
Use the same Suspense, preview and prefetch rules as step 6; record only `rendered` when delivering the content.
|
|
701
312
|
|
|
702
|
-
|
|
703
|
-
'use client'
|
|
313
|
+
### Click goals on your blocks
|
|
704
314
|
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
import { editorConfig } from './editorConfig'
|
|
315
|
+
After [step 2](#2-add-the-plugin), add a goal picker to your button or link block,
|
|
316
|
+
then repeat [step 3](#3-run-migrations-and-generate-the-import-map):
|
|
708
317
|
|
|
709
|
-
|
|
318
|
+
```tsx
|
|
319
|
+
import { xpGoalField } from '@delmaredigital/payload-experiments'
|
|
710
320
|
|
|
711
|
-
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
</PuckConfigProvider>
|
|
715
|
-
}
|
|
321
|
+
// Add xpGoalField() to the host block's fields array.
|
|
322
|
+
// In that block's renderer, put the saved value on the actual button or link:
|
|
323
|
+
<a href={href} data-xp-goal={xpGoal}>{label}</a>
|
|
716
324
|
```
|
|
717
325
|
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
326
|
+
For Puck components, use `xpGoalPuckField()` from `/puck` in the render fields
|
|
327
|
+
and `xpGoalPuckEditorField()` from `/puck/editor` in the editor fields; pass custom `routes` as
|
|
328
|
+
`xpGoalPuckEditorField({ routes })`. Editors name click goals in **Goals** or the picker. Mount the provider described below
|
|
329
|
+
to count clicks; put the main goal and guardrail goals in every version or
|
|
330
|
+
outside all tested content.
|
|
723
331
|
|
|
724
|
-
|
|
332
|
+
## How it fits a typical site
|
|
725
333
|
|
|
726
|
-
|
|
334
|
+
For a site with Puck Pages, signed-in users, a Vercel cron and staff who manage tests, the pieces usually live here:
|
|
727
335
|
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
336
|
+
| File | What belongs there |
|
|
337
|
+
| --- | --- |
|
|
338
|
+
| `src/experiments/proxyRuntime.ts` | Private secret, manifest pool and API route |
|
|
339
|
+
| `src/experiments/serverRuntime.ts` | Lazy Payload loader, session lookup and staff attributes |
|
|
340
|
+
| `src/payload.config.ts` | Plugin, staff permissions, shared limiter and Pages registration |
|
|
341
|
+
| `src/proxy.ts` | Version selection and whole-page rewrites |
|
|
342
|
+
| `src/app/api/cron/xp-results/route.ts`, `vercel.json` | Authenticated results route and schedule |
|
|
343
|
+
| `src/components/TestedPage.tsx` | Resolve Puck content and render the assigned version |
|
|
344
|
+
| `src/puck/renderConfig.ts`, `src/puck/EditorProvider.tsx` | Render/editor integrations and iframe CSS |
|
|
345
|
+
| `src/auth/callbacks.ts` | Identity linking after registration and proving sign-in |
|
|
346
|
+
| `src/blocks/Button.ts` and its renderer | Click-goal picker and button/link attribute |
|
|
732
347
|
|
|
733
|
-
|
|
734
|
-
add a sidebar category listing it, and add `experimentOutlinePlugin()` to Puck's
|
|
735
|
-
`plugins`. It composes with the host's outline override.
|
|
348
|
+
## Flag points and metric sources
|
|
736
349
|
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
defaults. Hosts on non-default routes switch to the factories. The standalone
|
|
740
|
-
`createExperimentSlotField({ routes })` also accepts the host's routes.
|
|
350
|
+
Register a stable point once; editors can run new tests on it without changing
|
|
351
|
+
your code. Add this option to the plugin from step 2:
|
|
741
352
|
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
353
|
+
```ts
|
|
354
|
+
flagPoints: [{
|
|
355
|
+
id: 'signup-prompt', label: 'Signup prompt',
|
|
356
|
+
params: { heading: { type: 'text', label: 'Heading', required: true } },
|
|
357
|
+
defaultParams: { heading: 'Start learning' },
|
|
358
|
+
}]
|
|
359
|
+
```
|
|
745
360
|
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
361
|
+
Read it with `getFlagPoint(serverRuntime, 'signup-prompt')` from `/rsc`, render
|
|
362
|
+
its `params`, and call `await point.expose(point.arm)` only when you deliver
|
|
363
|
+
that version. Bind code to the point id. For simple outcome goals, register
|
|
364
|
+
`eventSources: [{ name: 'signup', clientWritable: false }]` and call
|
|
365
|
+
`trackConversion` from `/rsc` with `eventName`, a stable `idempotencyKey` and,
|
|
366
|
+
when needed, `userId` or request `headers`. Editors see **signup** in the goal picker.
|
|
367
|
+
For existing business data, supply `metricSources` adapters with `id`, `label`,
|
|
368
|
+
`kind`, `keyedBy` and `query`, returning one `{ unitId, value, ts: Date }` row
|
|
369
|
+
per event. Revenue belongs in trusted server code, never a browser beacon.
|
|
749
370
|
|
|
750
|
-
|
|
751
|
-
@import "tailwindcss";
|
|
752
|
-
@import "@delmaredigital/payload-experiments/puck/editor.css";
|
|
753
|
-
```
|
|
371
|
+
## Client counting for viewport and pageview tests
|
|
754
372
|
|
|
755
|
-
|
|
756
|
-
append that URL to payload-puck's `editorStylesheets`, after the host stylesheet.
|
|
757
|
-
The editor CSS includes its own light and dark palette, selected by
|
|
758
|
-
`data-theme="dark"` or the `dark` class on the iframe's `<html>`. The client component's CSS import alone
|
|
759
|
-
does not reach an iframe configured with `syncHostStyles: false`.
|
|
373
|
+
Pass server assignments through a provider around the rendered content:
|
|
760
374
|
|
|
761
|
-
|
|
375
|
+
```tsx
|
|
376
|
+
import { ExperimentsProvider, ExperimentPageview } from
|
|
377
|
+
'@delmaredigital/payload-experiments/client'
|
|
378
|
+
import { toClientAssignments } from '@delmaredigital/payload-experiments/rsc'
|
|
762
379
|
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
@import "@delmaredigital/payload-experiments/admin/ui.css";
|
|
380
|
+
<ExperimentsProvider value={toClientAssignments(assignments, { pathname })}>
|
|
381
|
+
<ExperimentPageview />
|
|
382
|
+
{children}
|
|
383
|
+
</ExperimentsProvider>
|
|
768
384
|
```
|
|
769
385
|
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
from `/puck` if you need the experiment picker in another custom component.
|
|
776
|
-
|
|
777
|
-
The picker lists the 100 most recently updated block experiments.
|
|
778
|
-
Viewport exposure watches a fixed, invisible **100 px sentinel**
|
|
779
|
-
at the top of the slot and counts it when at least half is visible. The
|
|
780
|
-
sentinel stays the same size for short and tall content.
|
|
781
|
-
|
|
782
|
-
### Tag host content with a group
|
|
783
|
-
|
|
784
|
-
Add `createArmSelectField({ flagPoint: 'signup-form' })` from
|
|
785
|
-
`@delmaredigital/payload-experiments` to a host collection's fields. By default
|
|
786
|
-
the field is named `experimentArm`; it stores
|
|
787
|
-
`{ experiment: String(experimentId), arm }`.
|
|
788
|
-
|
|
789
|
-
Match host content on **`String(experimentId)` plus the arm key**, never on
|
|
790
|
-
the experiment slug. The field shows group labels, leaves out the holdout,
|
|
791
|
-
warns about tags from an earlier experiment, and lets you clear them. A running
|
|
792
|
-
experiment shows a warning before you change its content.
|
|
793
|
-
|
|
794
|
-
### Inside the admin
|
|
795
|
-
|
|
796
|
-
The experiment's edit tabs guide setup:
|
|
797
|
-
|
|
798
|
-
- **What:** what you are testing and why, including the flag point when needed.
|
|
799
|
-
- **Who:** who takes part, targeting, allocation, and space in a layer.
|
|
800
|
-
- **Groups:** the Original, variants and optional holdout, their percentages,
|
|
801
|
-
settings, or page paths.
|
|
802
|
-
- **Metrics:** the main outcome, supporting metrics, guardrails, and planned
|
|
803
|
-
duration.
|
|
804
|
-
- **Review:** saved setup checks, placements and preview links, recent traffic,
|
|
805
|
-
and a duration estimate when enough information is available. Apply copies
|
|
806
|
-
the suggested duration and sensitivity into the form; save before you start.
|
|
807
|
-
|
|
808
|
-
The **Results** view lets you select a round, read the verdict, compare groups,
|
|
809
|
-
check data quality, and see results over time with content-change markers.
|
|
810
|
-
Holdouts are measured without being compared during the test. Revenue results
|
|
811
|
-
show uncapped totals alongside any capped decision statistic. Recompute
|
|
812
|
-
requests fresh results. The experiment list shows status and a results summary;
|
|
813
|
-
round history and layer usage provide context in the admin.
|
|
814
|
-
|
|
815
|
-
Actions depend on the experiment's status and your permissions. **Unsaved
|
|
816
|
-
changes block actions**; save first so they use the saved experiment.
|
|
817
|
-
|
|
818
|
-
- **Start** runs a draft. **Start a new round** after stopping warns:
|
|
819
|
-
“Starting again begins a new round; results restart.”
|
|
820
|
-
- **Stop** says: “Everyone will see the Original. Starting again begins a new
|
|
821
|
-
round; results restart.”
|
|
822
|
-
- **Raise allocation** adds people without changing the group of anyone
|
|
823
|
-
already taking part. Lowering allocation needs a new round.
|
|
824
|
-
- **Add a metric** adds a supporting metric or guardrail to a running test.
|
|
825
|
-
- **Decide** starts with the recommended group. Choosing the Original says
|
|
826
|
-
“Everyone will see the Original.” Choosing a variant says everyone will see
|
|
827
|
-
it, except an existing holdout, which keeps seeing the Original so you can
|
|
828
|
-
measure the shipped change. When results need caution or you choose against
|
|
829
|
-
the recommendation, enter a note and type the chosen group's label.
|
|
830
|
-
- **Release holdout** says the holdout will see the winner too and measuring
|
|
831
|
-
since shipping ends.
|
|
832
|
-
- **Make it the content** copies the winner into page content or saves its
|
|
833
|
-
settings as defaults, then archives the experiment. Failed page updates are
|
|
834
|
-
listed with Retry. **Finish archiving** completes archiving when the content
|
|
835
|
-
has already been updated.
|
|
836
|
-
- **Archive** closes a stopped experiment. **Discard winner** says: “Everyone
|
|
837
|
-
goes back to the Original. The decision note is kept.” It requires typed
|
|
838
|
-
confirmation. For code experiments, the action confirming that the developer
|
|
839
|
-
has hard-coded the winner also requires typed confirmation before archiving.
|
|
840
|
-
|
|
841
|
-
### Known limitations (v1)
|
|
842
|
-
|
|
843
|
-
- Preview links open after a token request, so Safari's popup blocker may block
|
|
844
|
-
them. Allow popups for the admin.
|
|
845
|
-
- “Raise allocation” is offered even at 100%.
|
|
846
|
-
- The metric picker in “Add a metric” lists 100 metrics.
|
|
847
|
-
- The Puck picker lists 100 experiments.
|
|
848
|
-
- The Puck experiment picker always calls `/api`; hosts with a custom API route
|
|
849
|
-
see an empty picker.
|
|
850
|
-
- Content-change markers are recorded on publish only (on save for collections
|
|
851
|
-
without drafts).
|
|
852
|
-
- Results cover the whole round; breakdowns by segment or dimension are not
|
|
853
|
-
available in v1.
|
|
854
|
-
|
|
855
|
-
### Limits
|
|
856
|
-
|
|
857
|
-
| Capability | Limit |
|
|
858
|
-
|---|---|
|
|
859
|
-
| Make it the content: localized collections | not supported yet |
|
|
860
|
-
| Make it the content: field defaults | An update-denied field empty in both versions may publish its default because Payload applies defaults when reading. |
|
|
386
|
+
Use the served page's own pathname, including after a rewrite. Puck viewport
|
|
387
|
+
blocks count when visible; whole-page tests need `ExperimentPageview` on each
|
|
388
|
+
target page. The provider also enables click goals. Pass `apiRoute` if yours
|
|
389
|
+
isn't `/api`. Client readers need explicit `flagPoints` or `experiments`
|
|
390
|
+
allow-lists in `toClientAssignments`; reading alone never counts a visit.
|
|
861
391
|
|
|
862
392
|
## Theme
|
|
863
393
|
|
|
864
|
-
The admin
|
|
394
|
+
The admin follows Payload's light and dark themes. To use your own accent, add:
|
|
865
395
|
|
|
866
396
|
```ts
|
|
867
397
|
theme: { accent: '#1f3a68' }
|
|
868
398
|
```
|
|
869
399
|
|
|
870
|
-
|
|
871
|
-
|
|
872
|
-
|
|
873
|
-
|
|
874
|
-
|
|
875
|
-
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
`
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
884
|
-
|
|
885
|
-
|
|
886
|
-
|
|
887
|
-
|
|
888
|
-
|
|
889
|
-
|
|
890
|
-
|
|
891
|
-
`data-xp-goal="join"` directly. Define a click goal with that short name first.
|
|
892
|
-
For managers, an undefined key blocks the save through the Payload field's
|
|
893
|
-
server validation, refreshing a cached miss immediately.
|
|
894
|
-
Draft saves are also refused when the collection validates drafts
|
|
895
|
-
(`versions.drafts.validate`). Editors without goal-management access see a
|
|
896
|
-
plain text input. Goals in
|
|
897
|
-
related documents and keys placed only in code are invisible to the content
|
|
898
|
-
fairness check.
|
|
899
|
-
|
|
900
|
-
`ExperimentsProvider` records native clicks when its server assignments arm
|
|
901
|
-
click goals. Nested providers share one document listener; the deepest armed
|
|
902
|
-
provider supplies the route and pageview id. Enter on links/buttons and Space
|
|
903
|
-
on buttons count through the browser's native click. A focusable `div` needs
|
|
904
|
-
`role="button"` and the host's own keyboard handler. Beacons use `sendBeacon`
|
|
905
|
-
(with a keepalive fallback) so navigation/unload does not discard the click.
|
|
906
|
-
Click and exposure beacons share `events.rateLimit`; size the adapter for both.
|
|
907
|
-
New click goals are accepted within 30 seconds. A click goal may be primary;
|
|
908
|
-
primary and guardrail goals must appear in every tested version or outside all
|
|
909
|
-
of them. Secondary asymmetry warns, and results health checks published rows
|
|
910
|
-
again on every run.
|
|
400
|
+
Use a hex colour with three or six digits; colours without enough contrast are refused. Pass the same `accent` to `experimentsPuckEditor` for Puck.
|
|
401
|
+
|
|
402
|
+
## Access options
|
|
403
|
+
|
|
404
|
+
Callbacks receive `{ req }` and return a boolean or a promise of one.
|
|
405
|
+
|
|
406
|
+
| Option | Who it permits | Default |
|
|
407
|
+
| --- | --- | --- |
|
|
408
|
+
| `manage` | Create and run tests, manage goals and choose winners | Required |
|
|
409
|
+
| `admin` | Code tests and analysis-method changes | `manage` |
|
|
410
|
+
| `readResults` | Read results | `manage` |
|
|
411
|
+
| `readIdentities` | Read identity links | `admin` |
|
|
412
|
+
|
|
413
|
+
## Full reference (until the docs site)
|
|
414
|
+
|
|
415
|
+
The longer option-by-option reference from before this rewrite is in [REFERENCE.md](https://github.com/delmaredigital/payload-experiments/blob/main/packages/plugin/REFERENCE.md) until the documentation site exists.
|
|
416
|
+
|
|
417
|
+
## License
|
|
418
|
+
|
|
419
|
+
[MIT](https://github.com/delmaredigital/payload-experiments/blob/main/LICENSE).
|
|
420
|
+
See the [statistics engine README](https://github.com/delmaredigital/payload-experiments/blob/main/packages/stats/README.md) for the methods behind the results.
|