@thinkingai/ae-cli 6.1.9 → 6.1.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -3
- package/README.zh.md +25 -3
- package/dist/{auth-56Z45UVR.js → auth-GDV3H5I4.js} +18 -8
- package/dist/{auth-ENM3FE6L.js → auth-NN55553T.js} +3 -3
- package/dist/{capability-2H6PAOA3.js → capability-TPORIKRQ.js} +19 -15
- package/dist/{capability-YPOQX6PL.js → capability-VQNC5CF7.js} +19 -15
- package/dist/chunk-4NN5IWVN.js +26 -0
- package/dist/{chunk-DB4Q3ANU.js → chunk-6A2FUCIS.js} +3 -3
- package/dist/{chunk-PVBYJWC2.js → chunk-7KQWSBSL.js} +4 -4
- package/dist/{chunk-24BAVOX3.js → chunk-AXDXJTPC.js} +3 -1
- package/dist/{chunk-CPVTECJ3.js → chunk-DT6Y3TD7.js} +5 -5
- package/dist/{chunk-EGEIXA2Z.js → chunk-DWO43OIB.js} +0 -248
- package/dist/{chunk-V6FR6WTW.js → chunk-GJJA4CQZ.js} +34 -2
- package/dist/{chunk-RGXCNC4N.js → chunk-GS2P7LFD.js} +50 -17
- package/dist/{chunk-LYVNONC4.js → chunk-JHENBQ5B.js} +35 -0
- package/dist/{chunk-3P3562ZX.js → chunk-MVDZ7DBQ.js} +5 -5
- package/dist/{chunk-LCXU3AAT.js → chunk-NBPKWKRA.js} +2 -2
- package/dist/{chunk-HQ2A7ITL.js → chunk-RBNKI5ZW.js} +50 -17
- package/dist/{chunk-LHVM35J4.js → chunk-TS6BUGUY.js} +4 -4
- package/dist/chunk-TUKQZTMI.js +250 -0
- package/dist/{chunk-J7MZHDHQ.js → chunk-VKD5WQKN.js} +5 -5
- package/dist/{chunk-3KI3RRXX.js → chunk-VTXHDCBW.js} +3 -3
- package/dist/{chunk-KTYR3U6D.js → chunk-YTG6Q75E.js} +10 -6
- package/dist/{chunk-IR4ZLVPW.js → chunk-YV52FB5G.js} +25 -3
- package/dist/{chunk-E7UXXHO3.js → chunk-Z3OXWCIA.js} +3 -3
- package/dist/{cli-token-4SBMXAUK.js → cli-token-GL5MS5FK.js} +4 -4
- package/dist/{client-PP5FETMW.js → client-DAIPF7XN.js} +4 -4
- package/dist/{community-report-client-KK2QBANO.js → community-report-client-M2RW4MXD.js} +2 -2
- package/dist/config-4VZNLBKF.js +489 -0
- package/dist/index.js +94 -35
- package/dist/memory-RWJW4XFO.js +892 -0
- package/dist/memory-VO2ZJCRT.js +892 -0
- package/dist/{metadata-JIQ77HFY.js → metadata-YGTHR2XJ.js} +8 -8
- package/dist/{metadata-AN3YFZEV.js → metadata-ZRN2GHPN.js} +8 -8
- package/dist/{model-JASQVOFD.js → model-CLUIK3K5.js} +7 -5
- package/dist/{raw-XJCAT3HX.js → raw-52B4UKO4.js} +8 -7
- package/dist/sync-3REDHGY6.js +10259 -0
- package/dist/{te-agent-JHUG6DVV.js → te-agent-XNPELAKX.js} +580 -84
- package/dist/{te-analysis-IWQZO423.js → te-analysis-GJI5FZUL.js} +1611 -25
- package/dist/{te-analysis-X222IRYR.js → te-analysis-N2BRDJZ5.js} +1611 -25
- package/dist/{te-community-QOYIYEJI.js → te-community-SQXKE5OO.js} +7 -7
- package/dist/{te-community-UFKI6ONP.js → te-community-TYSNU3NQ.js} +7 -7
- package/dist/{te-dataops-PZQ5NQLY.js → te-dataops-5TM7WZDI.js} +370 -189
- package/dist/{te-dataops-XTWVTJCA.js → te-dataops-OWIADNSM.js} +370 -189
- package/dist/{te-engage-NLZUPSBK.js → te-engage-F7V55KVW.js} +333 -105
- package/dist/{te-engage-E7F4HTXU.js → te-engage-FMYAYCNV.js} +333 -105
- package/dist/{te-experiment-N63WF7XA.js → te-experiment-JWXOYJ3W.js} +83 -10
- package/dist/{te-experiment-D32TB6ZB.js → te-experiment-UPDMHCAJ.js} +83 -10
- package/dist/{te-kb-E7NSCBRB.js → te-kb-VRMEY3D4.js} +6 -6
- package/dist/{te-meta-GBDTMPEL.js → te-meta-53BVXPFI.js} +7 -7
- package/dist/{te-meta-ZTLTSHXC.js → te-meta-TOCBPBXI.js} +7 -7
- package/dist/{te-system-AH7DMCAQ.js → te-system-XGS5EQIQ.js} +4 -4
- package/dist/{te-team-BQ3SKSZV.js → te-team-BZRDV2CM.js} +7 -7
- package/dist/{update-DKG6UXEM.js → update-HEDXGOJH.js} +8 -6
- package/package.json +5 -2
- package/skills/ae-agent/SKILL.md +178 -16
- package/skills/ae-agent/references/add-skill.md +22 -10
- package/skills/ae-agent/references/edit-skill.md +24 -14
- package/skills/ae-agent/references/find-archived-conversations.md +82 -0
- package/skills/ae-agent/references/restore-conversation.md +54 -0
- package/skills/ae-agent/references/upload-skill.md +24 -14
- package/skills/ae-analysis/SKILL.md +1 -1
- package/skills/ae-analysis/references/command_index.md +109 -42
- package/skills/ae-analysis/references/debug_data_list.md +28 -0
- package/skills/ae-analysis/references/debug_device_add.md +23 -0
- package/skills/ae-analysis/references/debug_device_list.md +19 -0
- package/skills/ae-analysis/references/debug_device_select.md +22 -0
- package/skills/ae-analysis/references/entity_id_import_options.md +1 -1
- package/skills/ae-analysis/references/plan_sync_display_names.md +28 -0
- package/skills/ae-analysis/references/project_access_detail_get.md +3 -3
- package/skills/ae-analysis/references/project_data_power_delete.md +3 -3
- package/skills/ae-analysis/references/project_data_power_get.md +3 -3
- package/skills/ae-analysis/references/project_data_power_list.md +3 -3
- package/skills/ae-analysis/references/project_data_power_upsert.md +3 -3
- package/skills/ae-analysis/references/project_entity_create.md +3 -3
- package/skills/ae-analysis/references/project_entity_delete.md +3 -3
- package/skills/ae-analysis/references/project_entity_event_list.md +3 -3
- package/skills/ae-analysis/references/project_entity_get.md +3 -3
- package/skills/ae-analysis/references/project_entity_list.md +3 -3
- package/skills/ae-analysis/references/project_entity_update.md +3 -3
- package/skills/ae-analysis/references/project_function_list.md +3 -3
- package/skills/ae-analysis/references/project_info_create.md +25 -0
- package/skills/ae-analysis/references/project_info_delete.md +24 -0
- package/skills/ae-analysis/references/project_info_get.md +3 -3
- package/skills/ae-analysis/references/project_info_list.md +3 -3
- package/skills/ae-analysis/references/project_info_update.md +3 -3
- package/skills/ae-analysis/references/project_mark_time_create.md +3 -3
- package/skills/ae-analysis/references/project_mark_time_delete.md +3 -3
- package/skills/ae-analysis/references/project_mark_time_list.md +3 -3
- package/skills/ae-analysis/references/project_mark_time_update.md +3 -3
- package/skills/ae-analysis/references/project_member_add.md +3 -3
- package/skills/ae-analysis/references/project_member_batch_update.md +3 -3
- package/skills/ae-analysis/references/project_member_candidate_list.md +3 -3
- package/skills/ae-analysis/references/project_member_handover_export.md +3 -3
- package/skills/ae-analysis/references/project_member_handover_run.md +3 -3
- package/skills/ae-analysis/references/project_member_import.md +3 -3
- package/skills/ae-analysis/references/project_member_list.md +3 -3
- package/skills/ae-analysis/references/project_member_receiver_list.md +3 -3
- package/skills/ae-analysis/references/project_member_remove.md +3 -3
- package/skills/ae-analysis/references/project_member_update.md +3 -3
- package/skills/ae-analysis/references/project_owner_update.md +3 -3
- package/skills/ae-analysis/references/project_permission_binding_list.md +3 -3
- package/skills/ae-analysis/references/project_receive_status_update.md +3 -3
- package/skills/ae-analysis/references/project_role_delete.md +3 -3
- package/skills/ae-analysis/references/project_role_function_list.md +3 -3
- package/skills/ae-analysis/references/project_role_get.md +3 -3
- package/skills/ae-analysis/references/project_role_list.md +3 -3
- package/skills/ae-analysis/references/project_role_upsert.md +3 -3
- package/skills/ae-analysis/references/project_role_user_list.md +3 -3
- package/skills/ae-analysis/references/project_space_list.md +1 -1
- package/skills/ae-analysis/references/project_timezone_get.md +3 -3
- package/skills/ae-analysis/references/project_timezone_overview.md +3 -3
- package/skills/ae-analysis/references/project_timezone_update.md +3 -3
- package/skills/ae-analysis/references/project_user_id_items_update.md +3 -3
- package/skills/ae-analysis/references/system_admin_function_list.md +22 -0
- package/skills/ae-analysis/references/system_admin_function_update.md +26 -0
- package/skills/ae-analysis/references/system_admin_list.md +21 -0
- package/skills/ae-analysis/references/system_admin_remove.md +25 -0
- package/skills/ae-analysis/references/system_admin_upsert.md +25 -0
- package/skills/ae-analysis/references/system_function_list.md +21 -0
- package/skills/ae-analysis/references/system_member_add.md +25 -0
- package/skills/ae-analysis/references/system_member_candidate_list.md +22 -0
- package/skills/ae-analysis/references/system_member_delete.md +25 -0
- package/skills/ae-analysis/references/system_member_list.md +24 -0
- package/skills/ae-analysis/references/system_member_mfa_unbind.md +25 -0
- package/skills/ae-analysis/references/system_member_password_reset.md +32 -0
- package/skills/ae-analysis/references/system_member_project_batch_update.md +27 -0
- package/skills/ae-analysis/references/system_member_status_update.md +26 -0
- package/skills/ae-analysis/references/system_member_update.md +23 -0
- package/skills/ae-analysis/references/system_mfa_get.md +21 -0
- package/skills/ae-analysis/references/system_mfa_update.md +25 -0
- package/skills/ae-analysis/references/system_node_monitor_list.md +24 -0
- package/skills/ae-analysis/references/system_oauth2_update.md +22 -0
- package/skills/ae-analysis/references/system_ops_alert_contact_delete.md +25 -0
- package/skills/ae-analysis/references/system_ops_alert_contact_list.md +23 -0
- package/skills/ae-analysis/references/system_ops_alert_contact_test.md +29 -0
- package/skills/ae-analysis/references/system_ops_alert_contact_upsert.md +39 -0
- package/skills/ae-analysis/references/system_preference_get.md +21 -0
- package/skills/ae-analysis/references/system_preference_update.md +22 -0
- package/skills/ae-analysis/references/system_project_usage_list.md +28 -0
- package/skills/ae-analysis/references/system_query_alert_rule_list.md +21 -0
- package/skills/ae-analysis/references/system_query_alert_rule_update.md +26 -0
- package/skills/ae-analysis/references/system_query_monitor_overview.md +26 -0
- package/skills/ae-analysis/references/system_query_task_cancel.md +25 -0
- package/skills/ae-analysis/references/system_query_task_export.md +49 -0
- package/skills/ae-analysis/references/system_query_task_get.md +23 -0
- package/skills/ae-analysis/references/system_query_task_list.md +35 -0
- package/skills/ae-analysis/references/system_query_task_options.md +27 -0
- package/skills/ae-analysis/references/system_receiver_address_delete.md +26 -0
- package/skills/ae-analysis/references/system_receiver_address_overview.md +21 -0
- package/skills/ae-analysis/references/system_receiver_address_project_list.md +21 -0
- package/skills/ae-analysis/references/system_receiver_address_promote.md +24 -0
- package/skills/ae-analysis/references/system_receiver_address_upsert.md +27 -0
- package/skills/ae-analysis/references/system_receiver_detection_get.md +22 -0
- package/skills/ae-analysis/references/system_receiver_detection_run.md +22 -0
- package/skills/ae-analysis/references/system_receiver_detection_update.md +25 -0
- package/skills/ae-analysis/references/system_role_delete.md +26 -0
- package/skills/ae-analysis/references/system_role_function_list.md +22 -0
- package/skills/ae-analysis/references/system_role_get.md +22 -0
- package/skills/ae-analysis/references/system_role_list.md +24 -0
- package/skills/ae-analysis/references/system_role_upsert.md +27 -0
- package/skills/ae-analysis/references/system_role_user_list.md +22 -0
- package/skills/ae-analysis/references/system_seat_list.md +25 -0
- package/skills/ae-analysis/references/system_seat_update.md +26 -0
- package/skills/ae-analysis/references/system_smtp_delete.md +24 -0
- package/skills/ae-analysis/references/system_smtp_get.md +21 -0
- package/skills/ae-analysis/references/system_smtp_test.md +22 -0
- package/skills/ae-analysis/references/system_smtp_upsert.md +31 -0
- package/skills/ae-analysis/references/system_third_party_login_disable.md +25 -0
- package/skills/ae-analysis/references/system_third_party_login_list.md +21 -0
- package/skills/ae-analysis/references/system_third_party_login_upsert.md +33 -0
- package/skills/ae-analysis/references/system_usage_overview.md +21 -0
- package/skills/ae-analysis/references/system_usage_trend_export.md +44 -0
- package/skills/ae-analysis/references/system_usage_trend_query.md +28 -0
- package/skills/ae-dataops/SKILL.md +1 -1
- package/skills/ae-dataops/references/dataops-flow-create.md +51 -16
- package/skills/ae-engage/SKILL.md +56 -26
- package/skills/ae-engage/references/activity-activity.md +3 -0
- package/skills/ae-engage/references/activity-approval.md +12 -4
- package/skills/ae-engage/references/activity-task.md +21 -6
- package/skills/ae-engage/references/activity-topic.md +31 -13
- package/skills/ae-engage/references/add-channel.md +170 -41
- package/skills/ae-engage/references/build-task-save-guide.md +40 -27
- package/skills/ae-engage/references/channel-update-config.md +3 -2
- package/skills/ae-engage/references/common-metric.md +48 -120
- package/skills/ae-engage/references/flow-detail.md +13 -0
- package/skills/ae-engage/references/preset-event.md +12 -32
- package/skills/ae-engage/references/save-flow.md +127 -64
- package/skills/ae-engage/references/save-task.md +197 -59
- package/skills/ae-engage/references/scene-config-metric.md +3 -0
- package/skills/ae-engage/references/scene-preset-metric.md +8 -37
- package/skills/ae-engage/references/scene-strategy-audience.md +51 -643
- package/skills/ae-engage/references/scene-strategy.md +6 -6
- package/skills/ae-engage/references/task-detail.md +10 -0
- package/skills/ae-engage/references/task-submit-approval.md +45 -0
- package/skills/ae-engage/references/validate-flow-node-config.md +1 -1
- package/skills/ae-experiment/SKILL.md +21 -1
- package/skills/ae-experiment/references/check_experiment_ready.md +2 -0
- package/skills/ae-experiment/references/query_experiment_detail.md +6 -0
- package/skills/ae-experiment/references/query_experiment_list.md +4 -0
- package/skills/ae-experiment/references/query_experiment_list_archived.md +3 -0
- package/skills/ae-experiment/references/query_metric_detail.md +5 -0
- package/skills/ae-experiment/references/query_metric_list.md +3 -0
- package/skills/ae-experiment/references/save_experiment.md +46 -3
- package/skills/ae-experiment/references/save_metric.md +62 -0
- package/skills/ae-experiment/references/save_submit_experiment.md +3 -0
- package/skills/ae-experiment-design/SKILL.md +149 -0
- package/skills/ae-experiment-design/agents/openai.yaml +4 -0
- package/skills/ae-experiment-design/references/client-experiment-sdk.md +147 -0
- package/skills/ae-experiment-design/references/experiment-creation.md +108 -0
- package/skills/ae-experiment-design/references/experiment-sdk-contract.md +100 -0
- package/skills/ae-experiment-design/references/exposure-contract.md +91 -0
- package/skills/ae-experiment-design/references/hybrid-experiment-sdk.md +74 -0
- package/skills/ae-experiment-design/references/metric-readiness.md +143 -0
- package/skills/ae-experiment-design/references/platform-operations.md +99 -0
- package/skills/ae-experiment-design/references/sdk-index.md +76 -0
- package/skills/ae-experiment-design/references/sdk-integration.md +114 -0
- package/skills/ae-experiment-design/references/sdk-troubleshooting.md +139 -0
- package/skills/ae-experiment-design/references/server-experiment-sdk.md +78 -0
- package/skills/ae-experiment-design/scripts/calculate_experiment_plan.py +450 -0
- package/skills/ae-experiment-insight/SKILL.md +149 -0
- package/skills/ae-experiment-insight/agents/openai.yaml +4 -0
- package/skills/ae-experiment-insight/references/decision-framework.md +69 -0
- package/skills/ae-experiment-insight/references/diagnostic-playbook.md +225 -0
- package/skills/ae-experiment-insight/references/platform-operations.md +76 -0
- package/skills/ae-experiment-insight/scripts/analyze_experiment.py +478 -0
- package/skills/ae-generate-tracking-code/SKILL.md +24 -8
- package/skills/ae-generate-tracking-code/references/debug-script.md +46 -37
- package/skills/ae-generate-tracking-plan/SKILL.md +26 -1
- package/skills/ae-metadata/SKILL.md +1 -1
- package/dist/config-BSSALXEN.js +0 -128
- package/dist/sync-QFP4XFN3.js +0 -485
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Client Experiment SDK
|
|
2
|
+
|
|
3
|
+
Read this reference for Android, iOS, or browser JavaScript experiment integration. Read `sdk-index.md` first and verify the current main document before producing production code.
|
|
4
|
+
|
|
5
|
+
## Shared client model
|
|
6
|
+
|
|
7
|
+
All three verified client pages describe the same model:
|
|
8
|
+
|
|
9
|
+
- analytics SDK handles data collection;
|
|
10
|
+
- Remote Config retrieves AE configuration;
|
|
11
|
+
- experiment SDK provides typed Feature getters and exposure;
|
|
12
|
+
- automatic exposure is enabled through `automaticExposureTracking`;
|
|
13
|
+
- manual exposure is available when activation does not coincide with the getter;
|
|
14
|
+
- a custom bucket map can override the default assignment subject;
|
|
15
|
+
- custom fetch parameters can be attached to configuration requests;
|
|
16
|
+
- an explicit fetch can refresh experiment information.
|
|
17
|
+
|
|
18
|
+
## Android
|
|
19
|
+
|
|
20
|
+
Verified document state: `TDExperiment` 1.0.1, updated 2026-07-24.
|
|
21
|
+
|
|
22
|
+
Dependencies:
|
|
23
|
+
|
|
24
|
+
- `TDAnalytics` >= 3.3.6
|
|
25
|
+
- `TDRemoteConfig` >= 1.3.0
|
|
26
|
+
- automatic package: `cn.thinkingdata.android:TDExperiment:1.0.1`
|
|
27
|
+
|
|
28
|
+
Initialization order:
|
|
29
|
+
|
|
30
|
+
```java
|
|
31
|
+
TDAnalytics.init(context, "APP_ID", "https://YOUR_SERVER_URL");
|
|
32
|
+
|
|
33
|
+
TDExperimentConfig config =
|
|
34
|
+
new TDExperimentConfig("APP_ID", "https://YOUR_SERVER_URL");
|
|
35
|
+
config.automaticExposureTracking = true;
|
|
36
|
+
TDExperiment.init(context, config);
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Verified operations:
|
|
40
|
+
|
|
41
|
+
```java
|
|
42
|
+
TDExperiment.getValueAsString(key);
|
|
43
|
+
TDExperiment.getValueAsDouble(key);
|
|
44
|
+
TDExperiment.getValueAsBoolean(key);
|
|
45
|
+
TDExperiment.getValueAsJson(key);
|
|
46
|
+
TDExperiment.exposure(key);
|
|
47
|
+
TDExperiment.fetch();
|
|
48
|
+
TDExperiment.setCustomBucketId(bucketId);
|
|
49
|
+
TDExperiment.setCustomFetchParams(params);
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Use the initialization callback for first-fetch success or error. `TDExperiment.enableLog(true)` enables experiment logging.
|
|
53
|
+
|
|
54
|
+
The verified page does not show typed-default overloads for Android getters. Keep an application-owned typed control default and verify whether a newer SDK adds overloads.
|
|
55
|
+
|
|
56
|
+
## iOS
|
|
57
|
+
|
|
58
|
+
Verified document state: `TDExperiment` 1.0.2, updated 2026-07-24.
|
|
59
|
+
|
|
60
|
+
Dependencies:
|
|
61
|
+
|
|
62
|
+
- `ThinkingSDK` >= 3.1.6
|
|
63
|
+
- `TDRemoteConfig` >= 1.3.1
|
|
64
|
+
- CocoaPods: `pod 'TDExperiment', '1.0.2'`
|
|
65
|
+
- minimum deployment target shown by the experiment page: iOS 9.0
|
|
66
|
+
|
|
67
|
+
Initialization order:
|
|
68
|
+
|
|
69
|
+
```objective-c
|
|
70
|
+
[TDAnalytics startAnalyticsWithAppId:@"APP_ID"
|
|
71
|
+
serverUrl:@"https://YOUR_SERVER_URL"];
|
|
72
|
+
|
|
73
|
+
TDExperimentConfig *config =
|
|
74
|
+
[[TDExperimentConfig alloc] initWithAppId:@"APP_ID"
|
|
75
|
+
serverUrl:@"https://YOUR_SERVER_URL"];
|
|
76
|
+
config.automaticExposureTracking = YES;
|
|
77
|
+
[TDExperiment startWithConfig:config];
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Verified getters support no-argument defaults and explicit defaults:
|
|
81
|
+
|
|
82
|
+
```objective-c
|
|
83
|
+
[TDExperiment getValueAsString:key defaultValue:@"default"];
|
|
84
|
+
[TDExperiment getValueAsNumber:key defaultValue:@0];
|
|
85
|
+
[TDExperiment getValueAsBoolean:key defaultValue:NO];
|
|
86
|
+
[TDExperiment getValueAsJson:key defaultValue:@{}];
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Other verified operations:
|
|
90
|
+
|
|
91
|
+
```objective-c
|
|
92
|
+
[TDExperiment exposure:key];
|
|
93
|
+
[TDExperiment fetch];
|
|
94
|
+
[TDExperiment setCustomBucketId:bucketId];
|
|
95
|
+
[TDExperiment setCustomFetchParams:params];
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
Use `TDExperimentTask` listeners for startup or fetch success and failure. `[TDExperiment enableLog:YES]` enables experiment logging.
|
|
99
|
+
|
|
100
|
+
## JavaScript
|
|
101
|
+
|
|
102
|
+
Verified document state: experiment package 1.0.0, updated 2026-07-24.
|
|
103
|
+
|
|
104
|
+
Dependencies:
|
|
105
|
+
|
|
106
|
+
- `TDAnalytics` >= 2.6.0
|
|
107
|
+
- `TDRemoteconfig` >= 1.3.0
|
|
108
|
+
- files shown by the page: `thinkingdata.umd.min.js`, `tdremoteconfig.umd.min.js`, `tdexperiment.umd.min.js`
|
|
109
|
+
|
|
110
|
+
The page initializes analytics first and then the experiment global with:
|
|
111
|
+
|
|
112
|
+
- `appId`
|
|
113
|
+
- `serverUrl`
|
|
114
|
+
- `automaticExposureTracking`
|
|
115
|
+
- `customBucketId`
|
|
116
|
+
- `customFetchParams`
|
|
117
|
+
- `enableLog`
|
|
118
|
+
- fetch success and failure callbacks
|
|
119
|
+
|
|
120
|
+
The page shows typed getters, manual exposure, fetch, custom bucket ID, and custom fetch parameters.
|
|
121
|
+
|
|
122
|
+
### Spelling gate
|
|
123
|
+
|
|
124
|
+
The verified page spells the global object `TDExpriment`. Treat this as unresolved until the downloaded 1.0.0 package or a newer verified main document confirms the export. Do not silently change it, and do not publish exact JavaScript code based only on the page.
|
|
125
|
+
|
|
126
|
+
## Remote Config companion behavior
|
|
127
|
+
|
|
128
|
+
The verified Remote Config pages show:
|
|
129
|
+
|
|
130
|
+
- local defaults when no remote value is available;
|
|
131
|
+
- value order: remote, local default, then empty;
|
|
132
|
+
- a successful-fetch update listener;
|
|
133
|
+
- status information for strategies changed to `suspend` or `force_offline`;
|
|
134
|
+
- debug/test mode polling every five seconds for test strategies;
|
|
135
|
+
- test-device selection for client send testing.
|
|
136
|
+
|
|
137
|
+
These are Remote Config behaviors. Do not imply that the experiment SDK itself polls every five seconds in production.
|
|
138
|
+
|
|
139
|
+
## Client implementation checklist
|
|
140
|
+
|
|
141
|
+
- Verify all three SDK versions together.
|
|
142
|
+
- Initialize on the documented thread or lifecycle point.
|
|
143
|
+
- Set assignment identity before first fetch or getter.
|
|
144
|
+
- Use a typed control default.
|
|
145
|
+
- Choose automatic or manual exposure, not both.
|
|
146
|
+
- Freeze behavior when a mid-session update would cause flicker.
|
|
147
|
+
- Test control, treatment, no-network, timeout, account switch, and stale-cache cases.
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Experiment Draft Creation
|
|
2
|
+
|
|
3
|
+
Use this reference when the user asks to create, save, or reuse an A/B experiment draft.
|
|
4
|
+
|
|
5
|
+
## Execution boundary
|
|
6
|
+
|
|
7
|
+
- Perform every platform read or write through `ae-cli`.
|
|
8
|
+
- Discover and inspect the current capability before constructing input.
|
|
9
|
+
- Do not assume Feature, layer, metric, group, targeting, duration, or traffic fields before inspecting the current schema.
|
|
10
|
+
|
|
11
|
+
## Creation contract gate
|
|
12
|
+
|
|
13
|
+
Before writing a draft, establish the minimum semantic contract:
|
|
14
|
+
|
|
15
|
+
1. verified project ID and host;
|
|
16
|
+
2. experiment variable and the user-visible or system behavior that changes;
|
|
17
|
+
3. experiment goal and one primary metric with an exact feasible definition;
|
|
18
|
+
4. target population and material exclusions;
|
|
19
|
+
5. stable assignment unit and identity join path;
|
|
20
|
+
6. one control group and at least one treatment group, with typed Feature values and allocations totaling `1.0`;
|
|
21
|
+
7. experiment traffic, targeting, and a resolved layer or explicit new-layer plan;
|
|
22
|
+
8. Feature ownership, key, type, and typed default;
|
|
23
|
+
9. when the draft requires a duration, a confirmed sample target and
|
|
24
|
+
formula-derived duration.
|
|
25
|
+
|
|
26
|
+
Do not create a prose-only shell that leaves the experiment variable, group behavior, primary metric, target population, or assignment identity undefined. Ask one focused question at a time when a missing answer materially changes the draft.
|
|
27
|
+
|
|
28
|
+
## Design heuristics are not platform facts
|
|
29
|
+
|
|
30
|
+
Use these only as recommendations and verify that the identity exists and remains stable:
|
|
31
|
+
|
|
32
|
+
- pre-login experience: usually `#distinct_id` or a stable device identity;
|
|
33
|
+
- authenticated user experience: usually `#user_id`;
|
|
34
|
+
- account-wide B2B behavior: an account identity;
|
|
35
|
+
- device-specific rendering or performance: a device identity.
|
|
36
|
+
|
|
37
|
+
Do not silently choose an assignment identity. Explain the material identity risk and obtain confirmation when more than one viable identity changes who receives a consistent experience.
|
|
38
|
+
|
|
39
|
+
## Authorization gate
|
|
40
|
+
|
|
41
|
+
A draft write is authorized when either condition holds:
|
|
42
|
+
|
|
43
|
+
- the user reviewed the proposed design and then explicitly asked to create or confirmed creation; or
|
|
44
|
+
- the user's current request explicitly says to create and already provides an exact, complete creation contract.
|
|
45
|
+
|
|
46
|
+
If the design contains recommended or inferred choices that the user has not reviewed and those choices materially affect assignment, audience, behavior, metrics, or traffic, show a compact plan summary and ask for confirmation before writing.
|
|
47
|
+
|
|
48
|
+
Do not ask for redundant confirmation when the user has already confirmed the unchanged plan. Design-only language is not authorization to create. Draft authorization never authorizes submit, start, live traffic changes, pause, end, or delete.
|
|
49
|
+
|
|
50
|
+
## Resolution and idempotency
|
|
51
|
+
|
|
52
|
+
Before creating assets:
|
|
53
|
+
|
|
54
|
+
1. resolve the project;
|
|
55
|
+
2. search for exact or likely duplicate experiment drafts;
|
|
56
|
+
3. inspect any likely duplicate before deciding to reuse it;
|
|
57
|
+
4. resolve the exact Feature, layer, and metrics;
|
|
58
|
+
5. verify definitions, types, defaults, ownership, status, and remaining traffic;
|
|
59
|
+
6. identify existing active or draft experiments that may interact with the same Feature, layer, audience, or identity.
|
|
60
|
+
|
|
61
|
+
Reuse only an exact semantic match. Similar names are not enough. If an exact-match draft already exists, reuse it and report that outcome instead of creating a duplicate.
|
|
62
|
+
|
|
63
|
+
An empty list is not proof of no conflict. Only a supported conflict check can justify a user-facing "no conflict" result.
|
|
64
|
+
|
|
65
|
+
## Materialization sequence
|
|
66
|
+
|
|
67
|
+
Use the inspected schemas and execute only the steps required for this design:
|
|
68
|
+
|
|
69
|
+
1. create or reuse an exact metric when its definition is complete;
|
|
70
|
+
2. create or reuse an inactive Feature with the correct type and default;
|
|
71
|
+
3. create or reuse a compatible layer with verified assignment identity and sufficient traffic;
|
|
72
|
+
4. assemble experiment name, falsifiable hypothesis, Feature bindings, groups, allocations, traffic, targeting, metrics, and any required schedule fields;
|
|
73
|
+
5. validate or dry-run once when appropriate under `platform-operations.md`;
|
|
74
|
+
6. save the experiment as a draft;
|
|
75
|
+
7. read back the draft and verify its persisted fields;
|
|
76
|
+
8. run supported readiness and conflict checks;
|
|
77
|
+
9. return a concise creation receipt and experiment-detail link.
|
|
78
|
+
|
|
79
|
+
Do not claim atomic success when supporting assets were created but the experiment save failed. Report created, reused, failed, and unresolved assets separately.
|
|
80
|
+
Immediately before saving, revalidate the creation contract against the inspected schema and ensure no proposed ID is being presented as an existing platform ID.
|
|
81
|
+
|
|
82
|
+
## Post-save verification
|
|
83
|
+
|
|
84
|
+
A successful write response is not sufficient by itself. Read back the saved or reused draft and verify, when returned by the platform:
|
|
85
|
+
|
|
86
|
+
- actual experiment ID and name;
|
|
87
|
+
- project ID;
|
|
88
|
+
- draft status;
|
|
89
|
+
- Feature and layer bindings;
|
|
90
|
+
- group names, values, and allocations;
|
|
91
|
+
- experiment traffic and targeting;
|
|
92
|
+
- primary event metric and calculation code.
|
|
93
|
+
|
|
94
|
+
Run readiness and conflict checks only when supported. Report each check separately and do not translate "not run" into "passed".
|
|
95
|
+
|
|
96
|
+
## User-visible completion
|
|
97
|
+
|
|
98
|
+
Return an operation receipt, not the discovery log. Include:
|
|
99
|
+
|
|
100
|
+
- created, reused, partially completed, blocked, or failed;
|
|
101
|
+
- experiment name, actual ID, and draft status;
|
|
102
|
+
- control and treatment values and allocations;
|
|
103
|
+
- experiment traffic and primary metric;
|
|
104
|
+
- readiness and conflict results that actually ran;
|
|
105
|
+
- at most three material blockers or semantic risks;
|
|
106
|
+
- the clickable experiment-detail link required by `SKILL.md`.
|
|
107
|
+
|
|
108
|
+
Do not expose raw candidate lists, capability discovery, request schemas, commands, or internal execution narration unless the user explicitly asks for an audit or debugging view.
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Experiment SDK Contract
|
|
2
|
+
|
|
3
|
+
Read this reference when defining Feature retrieval, assignment, defaults, caching, or cross-platform behavior.
|
|
4
|
+
|
|
5
|
+
## Required sequence
|
|
6
|
+
|
|
7
|
+
1. Initialize the analytics SDK.
|
|
8
|
+
2. Initialize the Remote Config dependency.
|
|
9
|
+
3. Initialize the experiment SDK with the same verified app and server environment.
|
|
10
|
+
4. Establish the stable assignment identity before the first Feature evaluation.
|
|
11
|
+
5. Fetch or read the Feature using the expected value type.
|
|
12
|
+
6. Apply the assigned behavior.
|
|
13
|
+
7. Record exposure only when the behavior becomes visible or effective.
|
|
14
|
+
8. Report outcome events with an identity that can join to exposure.
|
|
15
|
+
|
|
16
|
+
The verified Android and iOS experiment documents explicitly require analytics initialization before experiment initialization. Apply the same ordering to JavaScript unless a newer verified main document states otherwise.
|
|
17
|
+
|
|
18
|
+
## Feature contract
|
|
19
|
+
|
|
20
|
+
Define the following before implementation:
|
|
21
|
+
|
|
22
|
+
| Field | Requirement |
|
|
23
|
+
|---|---|
|
|
24
|
+
| Feature key | Stable, environment-correct, and resolved from the real platform asset |
|
|
25
|
+
| Value type | String, number, Boolean, or JSON; match the platform Feature |
|
|
26
|
+
| Default | Typed, safe for control behavior, and owned by the application |
|
|
27
|
+
| Assignment unit | Stable device, account, role, or another verified identifier |
|
|
28
|
+
| Evaluation owner | Client, server, or one side of a hybrid architecture |
|
|
29
|
+
| Exposure owner | Exactly one component |
|
|
30
|
+
| Outcome join | Same stable identity or a verified merge path |
|
|
31
|
+
|
|
32
|
+
Do not use an empty or null fallback when a safe control behavior is required. A default is product behavior, not only an SDK parameter.
|
|
33
|
+
|
|
34
|
+
## Assignment identity
|
|
35
|
+
|
|
36
|
+
Resolve:
|
|
37
|
+
|
|
38
|
+
- pre-login identity;
|
|
39
|
+
- post-login identity;
|
|
40
|
+
- account switching;
|
|
41
|
+
- multiple roles under one account;
|
|
42
|
+
- device-to-account merge or alias behavior;
|
|
43
|
+
- server/client identity consistency.
|
|
44
|
+
|
|
45
|
+
If a custom bucket ID is used, set it before fetching or reading the Feature. The verified client SDKs expose a custom bucket map, but the exact key and value semantics must match the experiment configuration. Do not assume that the literal example `bucket_id -> account_id` is the only supported schema.
|
|
46
|
+
|
|
47
|
+
Do not use custom request parameters as a hidden substitute for assignment identity unless the platform contract explicitly defines that behavior.
|
|
48
|
+
|
|
49
|
+
## Fetch and cache
|
|
50
|
+
|
|
51
|
+
Define:
|
|
52
|
+
|
|
53
|
+
- initial fetch timing;
|
|
54
|
+
- request timeout;
|
|
55
|
+
- retry and backoff;
|
|
56
|
+
- last-known-good cache;
|
|
57
|
+
- cache freshness;
|
|
58
|
+
- whether a stale value may be used;
|
|
59
|
+
- whether a mid-session update can change behavior;
|
|
60
|
+
- control fallback when no value is available.
|
|
61
|
+
|
|
62
|
+
Avoid UI flicker and treatment changes after the user has already seen control. For a session-scoped experience, freeze the evaluated value for the session unless the product requirement explicitly allows live changes.
|
|
63
|
+
|
|
64
|
+
Remote Config documentation establishes the general fallback order for configuration keys:
|
|
65
|
+
|
|
66
|
+
1. remote value;
|
|
67
|
+
2. local default;
|
|
68
|
+
3. empty when neither exists.
|
|
69
|
+
|
|
70
|
+
For an experiment Feature, prefer an explicit typed application default even when the platform SDK offers an overload or local default store.
|
|
71
|
+
|
|
72
|
+
## Exposure
|
|
73
|
+
|
|
74
|
+
Configuration retrieval is not exposure. A getter that automatically reports exposure is valid only if calling the getter coincides with applying the assigned behavior. If the application reads early, preloads, branches later, or may discard the result, disable automatic exposure and use manual exposure at the true activation point.
|
|
75
|
+
|
|
76
|
+
Read `exposure-contract.md` before finalizing exposure behavior.
|
|
77
|
+
|
|
78
|
+
## Custom fetch parameters
|
|
79
|
+
|
|
80
|
+
Use custom fetch parameters only for fields supported by the client configuration channel. Document:
|
|
81
|
+
|
|
82
|
+
- field name and type;
|
|
83
|
+
- source of truth;
|
|
84
|
+
- whether it affects targeting, diagnostics, or payload enrichment;
|
|
85
|
+
- privacy classification;
|
|
86
|
+
- behavior when absent.
|
|
87
|
+
|
|
88
|
+
Never place secrets or unstable session values in custom fetch parameters.
|
|
89
|
+
|
|
90
|
+
## Acceptance checks
|
|
91
|
+
|
|
92
|
+
- Analytics, Remote Config, and experiment SDK versions are compatible.
|
|
93
|
+
- Initialization uses the intended app and environment.
|
|
94
|
+
- The control default is returned when configuration is unavailable.
|
|
95
|
+
- A test identity receives a stable group across sessions.
|
|
96
|
+
- Client and server agree for the same assignment identity.
|
|
97
|
+
- Exposure occurs once at treatment activation, not at preload.
|
|
98
|
+
- Outcome events join to exposure.
|
|
99
|
+
- Account switching does not leak the previous account’s assignment.
|
|
100
|
+
- Debug logging and test mode are disabled or production-safe before release.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Experiment Exposure Contract
|
|
2
|
+
|
|
3
|
+
Read this reference whenever exposure affects assignment, metrics, SDK code, or rollout readiness.
|
|
4
|
+
|
|
5
|
+
## Exposure definition
|
|
6
|
+
|
|
7
|
+
Exposure means the assigned behavior was actually rendered or became effective for the analysis unit. It is not:
|
|
8
|
+
|
|
9
|
+
- SDK initialization;
|
|
10
|
+
- configuration fetch;
|
|
11
|
+
- Feature preload;
|
|
12
|
+
- getter execution when the result may be discarded;
|
|
13
|
+
- assignment calculation without treatment activation.
|
|
14
|
+
|
|
15
|
+
## Automatic versus manual exposure
|
|
16
|
+
|
|
17
|
+
The verified Android, iOS, and JavaScript experiment pages state that a typed getter automatically uploads exposure when `automaticExposureTracking` is enabled.
|
|
18
|
+
|
|
19
|
+
Use automatic exposure only when the getter is called at the same decision point where the value is applied.
|
|
20
|
+
|
|
21
|
+
Use manual exposure when:
|
|
22
|
+
|
|
23
|
+
- values are prefetched;
|
|
24
|
+
- a getter is called before eligibility is final;
|
|
25
|
+
- the result may not render;
|
|
26
|
+
- the server assigns but the client activates;
|
|
27
|
+
- one value is read multiple times before a single activation;
|
|
28
|
+
- the product needs a later, explicit effective point.
|
|
29
|
+
|
|
30
|
+
Do not combine automatic getter exposure and manual exposure for the same activation.
|
|
31
|
+
|
|
32
|
+
## Exposure owner
|
|
33
|
+
|
|
34
|
+
Choose exactly one:
|
|
35
|
+
|
|
36
|
+
- client rendering owner;
|
|
37
|
+
- server behavior owner;
|
|
38
|
+
- another verified activation service.
|
|
39
|
+
|
|
40
|
+
Document why that component knows the treatment became effective.
|
|
41
|
+
|
|
42
|
+
## Deduplication contract
|
|
43
|
+
|
|
44
|
+
Define an application-level deduplication key from stable facts such as:
|
|
45
|
+
|
|
46
|
+
- project and environment;
|
|
47
|
+
- experiment or Feature;
|
|
48
|
+
- assignment identity;
|
|
49
|
+
- assignment or group;
|
|
50
|
+
- activation scope such as session, page instance, or decision instance.
|
|
51
|
+
|
|
52
|
+
Do not invent AE event property names. Let the verified SDK emit its supported schema, or resolve the exposure event contract through `ae-cli`.
|
|
53
|
+
|
|
54
|
+
Retry must be idempotent at the chosen activation scope.
|
|
55
|
+
|
|
56
|
+
## Identity and join
|
|
57
|
+
|
|
58
|
+
The identity used for assignment, exposure, and outcome must either be the same or have a verified merge path.
|
|
59
|
+
|
|
60
|
+
Block launch when:
|
|
61
|
+
|
|
62
|
+
- anonymous and logged-in identities can cross groups;
|
|
63
|
+
- server and client use different assignment subjects;
|
|
64
|
+
- exposure uses device identity while outcomes use an unlinked account identity;
|
|
65
|
+
- account switching can inherit a cached assignment;
|
|
66
|
+
- the analysis unit cannot be reconstructed from platform evidence.
|
|
67
|
+
|
|
68
|
+
## Outcome contract
|
|
69
|
+
|
|
70
|
+
For the confirmed primary event metric, define:
|
|
71
|
+
|
|
72
|
+
- event or measure;
|
|
73
|
+
- identity;
|
|
74
|
+
- timestamp;
|
|
75
|
+
- attribution window;
|
|
76
|
+
- denominator or eligible population;
|
|
77
|
+
- relation to exposure;
|
|
78
|
+
- duplicate and late-event treatment.
|
|
79
|
+
|
|
80
|
+
Do not report an exposure merely to force a user into the denominator. Fix the metric or eligibility contract instead.
|
|
81
|
+
|
|
82
|
+
## Acceptance tests
|
|
83
|
+
|
|
84
|
+
- Getter without rendering does not produce exposure in manual mode.
|
|
85
|
+
- Rendering produces one exposure.
|
|
86
|
+
- Re-render behavior matches the pre-registered exposure scope.
|
|
87
|
+
- Retry does not create duplicate logical exposure.
|
|
88
|
+
- Control and treatment exposure are both observable.
|
|
89
|
+
- Assignment proportions match configured allocation.
|
|
90
|
+
- Exposure and outcome join for anonymous, login, logout, and account-switch paths.
|
|
91
|
+
- Offline and delayed-upload behavior is understood before launch.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Hybrid Experiment Architecture
|
|
2
|
+
|
|
3
|
+
Read this reference when the server assigns or prepares a variant but the client renders or activates the treatment.
|
|
4
|
+
|
|
5
|
+
## Ownership model
|
|
6
|
+
|
|
7
|
+
Assign one owner for each responsibility:
|
|
8
|
+
|
|
9
|
+
| Responsibility | Typical owner |
|
|
10
|
+
|---|---|
|
|
11
|
+
| Stable identity resolution | Server or shared identity service |
|
|
12
|
+
| Assignment or Feature evaluation | Server |
|
|
13
|
+
| Typed default | Both sides, with one canonical value |
|
|
14
|
+
| Rendering or activation | Client |
|
|
15
|
+
| Exposure | Client when it knows rendering succeeded |
|
|
16
|
+
| Outcome events | Side that owns the business action |
|
|
17
|
+
|
|
18
|
+
These are design defaults, not fixed product requirements.
|
|
19
|
+
|
|
20
|
+
## Server-to-client envelope
|
|
21
|
+
|
|
22
|
+
Define an application envelope such as:
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
{
|
|
26
|
+
featureKey,
|
|
27
|
+
value,
|
|
28
|
+
valueType,
|
|
29
|
+
assignmentOrGroupId,
|
|
30
|
+
configVersion,
|
|
31
|
+
evaluatedAt,
|
|
32
|
+
exposureToken?
|
|
33
|
+
}
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
This is architecture-level pseudocode. Verify any AE-specific schema through `ae-cli`.
|
|
37
|
+
|
|
38
|
+
## Consistency rules
|
|
39
|
+
|
|
40
|
+
- Server and client must use the same project, environment, Feature key, and stable identity.
|
|
41
|
+
- The client must validate the value type before applying it.
|
|
42
|
+
- A missing, invalid, or expired envelope falls back to the typed control behavior.
|
|
43
|
+
- Do not re-evaluate independently on the client after a server assignment unless the product explicitly supports that mode.
|
|
44
|
+
- Do not allow account switching to reuse another account’s cached envelope.
|
|
45
|
+
- Include a configuration version or equivalent diagnostic marker when the verified platform provides one.
|
|
46
|
+
|
|
47
|
+
## Exposure handoff
|
|
48
|
+
|
|
49
|
+
The server should not report exposure merely because it returned a treatment envelope when the client may never render it. Prefer:
|
|
50
|
+
|
|
51
|
+
1. server evaluates;
|
|
52
|
+
2. client receives and validates;
|
|
53
|
+
3. client applies behavior;
|
|
54
|
+
4. client reports exposure once.
|
|
55
|
+
|
|
56
|
+
If exposure must be server-side, define a client acknowledgment or another verified activation signal and make retry idempotent.
|
|
57
|
+
|
|
58
|
+
## Failure scenarios
|
|
59
|
+
|
|
60
|
+
- server evaluation succeeds, client render fails: no exposure;
|
|
61
|
+
- server times out: client uses control;
|
|
62
|
+
- client starts offline with cached assignment: use only within the documented stale window;
|
|
63
|
+
- server and client identity disagree: block rollout;
|
|
64
|
+
- both sides report exposure: disable one path before launch;
|
|
65
|
+
- value arrives after control rendered: freeze control for the session or use an explicitly approved transition.
|
|
66
|
+
|
|
67
|
+
## Acceptance checks
|
|
68
|
+
|
|
69
|
+
- Server and client return or apply the same value type.
|
|
70
|
+
- Treatment is not visible before identity is stable.
|
|
71
|
+
- Exposure is reported by one owner.
|
|
72
|
+
- Refresh and retry do not duplicate exposure.
|
|
73
|
+
- Cached assignment is isolated by identity and environment.
|
|
74
|
+
- Outcome events remain joinable across client and server.
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# Metric Readiness
|
|
2
|
+
|
|
3
|
+
Use this checklist to decide whether a candidate metric can support an experiment decision.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Supported experiment metric contract
|
|
8
|
+
- Preregistered planning policy
|
|
9
|
+
- Evidence levels
|
|
10
|
+
- Feasibility gates
|
|
11
|
+
- Role selection
|
|
12
|
+
- Common blockers
|
|
13
|
+
- Required output
|
|
14
|
+
|
|
15
|
+
## Supported experiment metric contract
|
|
16
|
+
|
|
17
|
+
Use event metrics only. A platform enum or API field is not evidence that
|
|
18
|
+
another metric kind is implemented in the report calculation path.
|
|
19
|
+
|
|
20
|
+
Use only these event calculation codes:
|
|
21
|
+
|
|
22
|
+
| Code | Calculation | User-level value | Planning type |
|
|
23
|
+
| --- | --- | --- | --- |
|
|
24
|
+
| `A101` | Triggered users | Whether each user triggered the event; compare group trigger rates | `proportion` |
|
|
25
|
+
| `A100` | Total times | Each user's total event count; compare group user-level means | `continuous` |
|
|
26
|
+
| `A201` | Days | Each user's distinct active-event days; compare group user-level means | `continuous` |
|
|
27
|
+
| `A103` | Sum | Each user's sum of a numeric event property; compare group user-level means | `continuous` |
|
|
28
|
+
| `A104` | Average | Each user's average numeric event-property value; compare group user-level means | `continuous` |
|
|
29
|
+
| `A106` | Maximum | Each user's maximum numeric event-property value; compare group user-level means | `continuous` |
|
|
30
|
+
| `A108` | Distinct count | Each user's distinct count of one event property; compare group user-level means | `continuous` |
|
|
31
|
+
|
|
32
|
+
Require a numeric event property for `A103`, `A104`, and `A106`, and an exact
|
|
33
|
+
event property for `A108`.
|
|
34
|
+
|
|
35
|
+
The current report calculation uses a two-sided Z test for both planning
|
|
36
|
+
types, returns p-value, lift, and a 95% confidence interval, and uses
|
|
37
|
+
`p < 0.05` plus confidence-interval direction and goal direction to label a
|
|
38
|
+
positive or negative significant result.
|
|
39
|
+
|
|
40
|
+
## Preregistered planning policy
|
|
41
|
+
|
|
42
|
+
Fix the family-wise significance level at `alpha=0.05`, the confidence level
|
|
43
|
+
at `0.95`, and the test direction at two-sided to match current reporting.
|
|
44
|
+
Use `power=0.80` as the explicit planning-policy default when the user does not
|
|
45
|
+
provide power, and label its source as `policy_default`.
|
|
46
|
+
|
|
47
|
+
Require the MDE before definitive planning. Record whether it is `relative` or
|
|
48
|
+
`absolute` and whether the desired effect is an `increase` or `decrease`; never
|
|
49
|
+
silently substitute a 5% lift. Require observed user-level standard deviation
|
|
50
|
+
for continuous metrics; never substitute an assumed coefficient of variation.
|
|
51
|
+
|
|
52
|
+
For one control and multiple treatments, treat each treatment-versus-control
|
|
53
|
+
comparison as one member of the same family. Use Bonferroni planning:
|
|
54
|
+
`alpha_per_comparison = 0.05 / number_of_treatments`. Return the family alpha,
|
|
55
|
+
adjusted alpha, method, and comparison count. The current native report still
|
|
56
|
+
labels significance with unadjusted `p < 0.05` and a 95% confidence interval,
|
|
57
|
+
so use raw p-values against the preregistered adjusted threshold for the final
|
|
58
|
+
decision and disclose that native-label mismatch.
|
|
59
|
+
|
|
60
|
+
Use a minimum runtime of seven days and align recommendations to full weeks by
|
|
61
|
+
default; permit explicit overrides when the decision context justifies them.
|
|
62
|
+
When traffic is not fixed, evaluate absolute experiment-traffic candidates no
|
|
63
|
+
greater than the verified layer capacity and choose the smallest candidate
|
|
64
|
+
that reaches the sample target within the maximum runtime. If none is feasible,
|
|
65
|
+
return the true duration at maximum available traffic and mark it infeasible;
|
|
66
|
+
never truncate the duration to the maximum.
|
|
67
|
+
|
|
68
|
+
Do not add ad hoc missing-value imputation, winsorization, truncation, or
|
|
69
|
+
post-treatment exclusions. Use the exact platform metric definition and
|
|
70
|
+
preregister any supported filtering or data-quality rule before the run.
|
|
71
|
+
|
|
72
|
+
Call `calculate_experiment_plan.py` with `metric_type`, `baseline`, `mde`,
|
|
73
|
+
`mde_type`, `effect_direction`, `allocations`, and `daily_eligible_units`.
|
|
74
|
+
Pass `standard_deviation` for a continuous metric. Pass `experiment_traffic`
|
|
75
|
+
when traffic is confirmed; otherwise pass `max_available_traffic` and
|
|
76
|
+
optionally `traffic_candidates` to request a recommendation. `power`,
|
|
77
|
+
`min_runtime_days`, `max_runtime_days`, and `align_to_full_weeks` may override
|
|
78
|
+
their policy defaults. Do not pass `confidence_level` other than `0.95`.
|
|
79
|
+
|
|
80
|
+
Do not create or recommend a metric outside this contract. If the requested
|
|
81
|
+
business outcome cannot be expressed by one supported event calculation,
|
|
82
|
+
state that it is unavailable and propose the closest valid event metric
|
|
83
|
+
without representing it as equivalent.
|
|
84
|
+
|
|
85
|
+
## Evidence levels
|
|
86
|
+
|
|
87
|
+
Label every statement:
|
|
88
|
+
|
|
89
|
+
- `verified`: supported by current `ae-cli` platform output;
|
|
90
|
+
- `provided`: supplied by the user but not verified on the platform;
|
|
91
|
+
- `assumed`: introduced for design or sensitivity analysis;
|
|
92
|
+
- `missing`: required evidence is unavailable.
|
|
93
|
+
|
|
94
|
+
Only `verified` or explicitly accepted `provided` evidence can support a final readiness decision.
|
|
95
|
+
|
|
96
|
+
## Feasibility gates
|
|
97
|
+
|
|
98
|
+
A metric is feasible only when all applicable gates pass:
|
|
99
|
+
|
|
100
|
+
1. **Contract**: event, supported calculation code, required property, goal
|
|
101
|
+
direction, filters, and attribution window are explicit.
|
|
102
|
+
2. **Source**: required events and properties exist or are supplied as a concrete schema.
|
|
103
|
+
3. **Identity**: assignment, exposure, and outcome share a stable join path.
|
|
104
|
+
4. **Ordering**: outcome is attributed after exposure or treatment activation.
|
|
105
|
+
5. **Population**: pre-treatment eligibility and exclusions are reproducible.
|
|
106
|
+
6. **Planning evidence**: baseline, MDE and its type and direction, eligible
|
|
107
|
+
daily units, verified layer capacity when recommending traffic, and—except
|
|
108
|
+
for `A101`—user-level standard deviation are available before a definitive
|
|
109
|
+
sample or duration calculation.
|
|
110
|
+
7. **Stability**: collection and definition are not expected to change during the experiment.
|
|
111
|
+
8. **Sensitivity**: the metric can plausibly respond to the treatment.
|
|
112
|
+
|
|
113
|
+
Block or mark unverified when any required gate lacks evidence.
|
|
114
|
+
|
|
115
|
+
## Role selection
|
|
116
|
+
|
|
117
|
+
### Primary
|
|
118
|
+
|
|
119
|
+
Choose one supported event metric closest to the hypothesis and decision.
|
|
120
|
+
Confirm its calculation code and direction, then establish its baseline and
|
|
121
|
+
minimum business-relevant effect. Do not calculate a definitive duration
|
|
122
|
+
before the metric contract is confirmed.
|
|
123
|
+
|
|
124
|
+
## Common blockers
|
|
125
|
+
|
|
126
|
+
- missing exposure event or treatment-activation marker;
|
|
127
|
+
- assignment uses `user_id` while outcomes only have `device_id`;
|
|
128
|
+
- anonymous-to-login identity cannot be reconciled;
|
|
129
|
+
- cross-device identity is unstable;
|
|
130
|
+
- outcome precedes exposure;
|
|
131
|
+
- denominator is filtered by post-treatment behavior;
|
|
132
|
+
- event volume is too low;
|
|
133
|
+
- metric definition or instrumentation changed during the run;
|
|
134
|
+
- cross-platform events use incompatible identities or semantics.
|
|
135
|
+
|
|
136
|
+
## Required output
|
|
137
|
+
|
|
138
|
+
For each candidate, return:
|
|
139
|
+
|
|
140
|
+
| Metric | Status | Event | Calculation | Property | Direction | Join identity | Window | Evidence | Reason or remediation |
|
|
141
|
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
142
|
+
|
|
143
|
+
When blocked, give the smallest concrete instrumentation or identity change that would make the metric measurable.
|