@thinkingai/ae-cli 6.0.48-customer.1 → 6.1.0-project-role-upsert.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +96 -36
- package/README.zh.md +96 -36
- package/dist/{auth-QPEVVBKT.js → auth-FUM37MHF.js} +236 -127
- package/dist/{capability-RUSV77CO.js → capability-AXFQW5WM.js} +49 -33
- package/dist/{chunk-FR2EDAQM.js → chunk-4P355ZWB.js} +69 -1
- package/dist/chunk-6ZIQV5GW.js +26 -0
- package/dist/chunk-ATSM5XAW.js +623 -0
- package/dist/{chunk-6QK7SYYY.js → chunk-BBEFP4SB.js} +29 -37
- package/dist/{chunk-KEWVSO64.js → chunk-CZU3V3DQ.js} +4 -15
- package/dist/chunk-ECBLHAZO.js +15 -0
- package/dist/{chunk-B5HMQAOM.js → chunk-I4WQAEYB.js} +1 -1
- package/dist/{chunk-NSIVXKJH.js → chunk-JDPEVVHR.js} +5 -4
- package/dist/{chunk-BE73PUFX.js → chunk-LNZBEQXW.js} +2 -2
- package/dist/{chunk-GJXRF7Y6.js → chunk-QJQZH5GQ.js} +47 -78
- package/dist/{chunk-BVDST5YT.js → chunk-RSG4ONOI.js} +7 -7
- package/dist/{chunk-JYMGHGCE.js → chunk-VXNMYUXU.js} +24 -13
- package/dist/{chunk-KZ6ZHF5J.js → chunk-WZ6YVQSF.js} +14 -14
- package/dist/{community-report-client-6OL6GBQW.js → community-report-client-XXWGSBSD.js} +2 -4
- package/dist/{config-XNMACCKX.js → config-EXUGQN5T.js} +8 -9
- package/dist/{data-integration-FZQQC3KW.js → data-integration-4NZ53OMT.js} +7 -9
- package/dist/index.js +61 -78
- package/dist/{local-data-upload-client-EFHDLXFG.js → local-data-upload-client-KYOKVYZV.js} +2 -4
- package/dist/{memory-HBJQ2GC5.js → memory-ATNKZNW5.js} +4 -6
- package/dist/{metadata-PGDJE5RQ.js → metadata-SOALPEA7.js} +7 -9
- package/dist/{model-BHLJ2QMT.js → model-E4JMQ4V2.js} +4 -6
- package/dist/{personal-semantic-preference-E5NZ56DN.js → personal-semantic-preference-LOF33N5X.js} +6 -8
- package/dist/project-semantic-RV6NA5LN.js +1113 -0
- package/dist/{sync-MUWPRM5I.js → sync-YV3E66IF.js} +6 -8
- package/dist/{te-agent-WQZTMEK4.js → te-agent-JB5T3PO7.js} +393 -90
- package/dist/{te-analysis-YWCSSLYL.js → te-analysis-TV2YZ4IC.js} +112 -30
- package/dist/{te-community-BNE6KREL.js → te-community-UDBI672N.js} +9 -32
- package/dist/{te-dataops-TXLKM2W6.js → te-dataops-ZLYOCXZ4.js} +19 -20
- package/dist/{te-engage-VEW4CJHK.js → te-engage-EUVMTUDJ.js} +67 -11
- package/dist/te-experiment-ESEYE2SK.js +1059 -0
- package/dist/{te-kb-BUD6N3NI.js → te-kb-WYQWHFSC.js} +595 -85
- package/dist/{te-system-44LZJDTQ.js → te-system-7G6F2LJA.js} +6 -8
- package/dist/{te-team-XMP47YXE.js → te-team-E7FBBXMQ.js} +6 -8
- package/dist/{update-UODFOZ4P.js → update-D47BUG25.js} +7 -8
- package/package.json +13 -9
- package/skills/ae-agent/SKILL.md +30 -13
- package/skills/ae-agent/references/agent-distribution.md +94 -0
- package/skills/ae-agent/references/approval-request.md +4 -0
- package/skills/ae-agent/references/command_index.md +9 -2
- package/skills/ae-agent/references/create-automation.md +20 -2
- package/skills/ae-agent/references/get-agent-context.md +70 -0
- package/skills/ae-agent/references/list-automations.md +18 -3
- package/skills/ae-agent/references/list-mcp-credentials.md +1 -1
- package/skills/ae-agent/references/mcp-token.md +3 -3
- package/skills/ae-agent/references/set-mcp-credential.md +0 -1
- package/skills/ae-agent/references/update-automation.md +18 -0
- package/skills/ae-analysis/SKILL.md +28 -3
- package/skills/ae-analysis/references/adhoc_run.md +2 -0
- package/skills/ae-analysis/references/ai_models.md +23 -3
- package/skills/ae-analysis/references/analysis_gateway_assets.md +2 -2
- package/skills/ae-analysis/references/command_index.md +8 -8
- package/skills/ae-analysis/references/cross_source_config.md +84 -0
- package/skills/ae-analysis/references/dashboard_update.md +1 -1
- package/skills/ae-analysis/references/personal_semantic_preference_add.md +1 -1
- package/skills/ae-analysis/references/personal_semantic_preference_get.md +3 -1
- package/skills/ae-analysis/references/personal_semantic_preference_list.md +3 -1
- package/skills/ae-analysis/references/personal_semantic_preference_update.md +1 -1
- package/skills/ae-analysis/references/project_role_upsert.md +32 -2
- package/skills/ae-analysis/references/user_tag_create.md +23 -1
- package/skills/ae-analysis/references/user_tag_models.md +9 -1
- package/skills/ae-analysis/references/user_tag_update.md +14 -2
- package/skills/ae-data-integration/SKILL.md +2 -1
- package/skills/ae-data-integration/references/dimension-routing.md +36 -0
- package/skills/ae-data-integration/references/local-analysis.md +2 -0
- package/skills/ae-data-integration/references/source-inspect.md +1 -1
- package/skills/ae-data-integration/references/ue-routing.md +17 -1
- package/skills/ae-engage/SKILL.md +2 -0
- package/skills/ae-engage/references/save-flow.md +37 -1
- package/skills/ae-experiment/SKILL.md +117 -0
- package/skills/ae-experiment/references/batch_delete_experiment.md +11 -0
- package/skills/ae-experiment/references/batch_delete_feature.md +11 -0
- package/skills/ae-experiment/references/batch_delete_traffic_layer.md +11 -0
- package/skills/ae-experiment/references/cancel_experiment_query_by_request_id.md +11 -0
- package/skills/ae-experiment/references/check_experiment_ready.md +17 -0
- package/skills/ae-experiment/references/check_experiment_traffic_layer_conflict.md +37 -0
- package/skills/ae-experiment/references/delete_metric.md +13 -0
- package/skills/ae-experiment/references/manage_experiment.md +17 -0
- package/skills/ae-experiment/references/manage_feature_status.md +11 -0
- package/skills/ae-experiment/references/manage_feature_whitelist.md +66 -0
- package/skills/ae-experiment/references/manage_guardrail_metrics.md +26 -0
- package/skills/ae-experiment/references/query_bucket_list.md +12 -0
- package/skills/ae-experiment/references/query_experiment_detail.md +19 -0
- package/skills/ae-experiment/references/query_experiment_list.md +22 -0
- package/skills/ae-experiment/references/query_experiment_list_archived.md +18 -0
- package/skills/ae-experiment/references/query_experiment_metric_trend.md +12 -0
- package/skills/ae-experiment/references/query_experiment_operation_log.md +17 -0
- package/skills/ae-experiment/references/query_experiment_report_summary.md +12 -0
- package/skills/ae-experiment/references/query_experiment_sample_size_report.md +13 -0
- package/skills/ae-experiment/references/query_feature_detail.md +14 -0
- package/skills/ae-experiment/references/query_feature_list.md +12 -0
- package/skills/ae-experiment/references/query_feature_operation_log.md +19 -0
- package/skills/ae-experiment/references/query_feature_version_list.md +20 -0
- package/skills/ae-experiment/references/query_metric_detail.md +18 -0
- package/skills/ae-experiment/references/query_metric_list.md +15 -0
- package/skills/ae-experiment/references/query_traffic_layer_detail.md +13 -0
- package/skills/ae-experiment/references/query_traffic_layer_list.md +12 -0
- package/skills/ae-experiment/references/save_build_guide.md +39 -0
- package/skills/ae-experiment/references/save_experiment.md +210 -0
- package/skills/ae-experiment/references/save_feature.md +13 -0
- package/skills/ae-experiment/references/save_metric.md +91 -0
- package/skills/ae-experiment/references/save_submit_experiment.md +16 -0
- package/skills/ae-experiment/references/save_traffic_layer.md +13 -0
- package/skills/ae-experiment/references/save_validate.md +33 -0
- package/skills/ae-experiment/references/update_experiment_group.md +20 -0
- package/skills/ae-experiment-design/SKILL.md +149 -0
- package/skills/ae-experiment-design/agents/openai.yaml +4 -0
- package/skills/ae-experiment-design/references/client-experiment-sdk.md +147 -0
- package/skills/ae-experiment-design/references/experiment-creation.md +108 -0
- package/skills/ae-experiment-design/references/experiment-sdk-contract.md +100 -0
- package/skills/ae-experiment-design/references/exposure-contract.md +91 -0
- package/skills/ae-experiment-design/references/hybrid-experiment-sdk.md +74 -0
- package/skills/ae-experiment-design/references/metric-readiness.md +143 -0
- package/skills/ae-experiment-design/references/platform-operations.md +105 -0
- package/skills/ae-experiment-design/references/sdk-index.md +76 -0
- package/skills/ae-experiment-design/references/sdk-integration.md +114 -0
- package/skills/ae-experiment-design/references/sdk-troubleshooting.md +139 -0
- package/skills/ae-experiment-design/references/server-experiment-sdk.md +78 -0
- package/skills/ae-experiment-design/scripts/calculate_experiment_plan.py +450 -0
- package/skills/ae-experiment-insight/SKILL.md +149 -0
- package/skills/ae-experiment-insight/agents/openai.yaml +4 -0
- package/skills/ae-experiment-insight/references/decision-framework.md +69 -0
- package/skills/ae-experiment-insight/references/diagnostic-playbook.md +225 -0
- package/skills/ae-experiment-insight/references/platform-operations.md +82 -0
- package/skills/ae-experiment-insight/scripts/analyze_experiment.py +478 -0
- package/skills/ae-generate-tracking-plan/SKILL.md +16 -5
- package/skills/ae-kb/SKILL.md +107 -36
- package/skills/ae-kb/references/query-workflow.md +59 -29
- package/skills/ae-kb/references/versions.md +46 -0
- package/skills/ae-project-semantic/SKILL.md +193 -0
- package/skills/ae-project-semantic/references/query-routing-v5.md +165 -0
- package/skills/ae-project-semantic/references/recommendation-quality.md +68 -0
- package/skills/ae-system/SKILL.md +1 -1
- package/skills/ae-use-agent/SKILL.md +42 -0
- package/skills/ae-use-agent/references/local-agent.md +114 -0
- package/dist/auth-UMJAA72O.js +0 -15
- package/dist/chunk-533JK3YX.js +0 -72
- package/dist/chunk-O6K2ODUE.js +0 -318
- package/dist/chunk-RWXIXOZG.js +0 -326
- package/dist/client-N7NGZUNI.js +0 -204
- package/skills/ae-agent/references/auto-provision-mcp-credentials.md +0 -57
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
# Experiment SDK Contract
|
|
2
|
+
|
|
3
|
+
Read this reference when defining Feature retrieval, assignment, defaults, caching, or cross-platform behavior.
|
|
4
|
+
|
|
5
|
+
## Required sequence
|
|
6
|
+
|
|
7
|
+
1. Initialize the analytics SDK.
|
|
8
|
+
2. Initialize the Remote Config dependency.
|
|
9
|
+
3. Initialize the experiment SDK with the same verified app and server environment.
|
|
10
|
+
4. Establish the stable assignment identity before the first Feature evaluation.
|
|
11
|
+
5. Fetch or read the Feature using the expected value type.
|
|
12
|
+
6. Apply the assigned behavior.
|
|
13
|
+
7. Record exposure only when the behavior becomes visible or effective.
|
|
14
|
+
8. Report outcome events with an identity that can join to exposure.
|
|
15
|
+
|
|
16
|
+
The verified Android and iOS experiment documents explicitly require analytics initialization before experiment initialization. Apply the same ordering to JavaScript unless a newer verified main document states otherwise.
|
|
17
|
+
|
|
18
|
+
## Feature contract
|
|
19
|
+
|
|
20
|
+
Define the following before implementation:
|
|
21
|
+
|
|
22
|
+
| Field | Requirement |
|
|
23
|
+
|---|---|
|
|
24
|
+
| Feature key | Stable, environment-correct, and resolved from the real platform asset |
|
|
25
|
+
| Value type | String, number, Boolean, or JSON; match the platform Feature |
|
|
26
|
+
| Default | Typed, safe for control behavior, and owned by the application |
|
|
27
|
+
| Assignment unit | Stable device, account, role, or another verified identifier |
|
|
28
|
+
| Evaluation owner | Client, server, or one side of a hybrid architecture |
|
|
29
|
+
| Exposure owner | Exactly one component |
|
|
30
|
+
| Outcome join | Same stable identity or a verified merge path |
|
|
31
|
+
|
|
32
|
+
Do not use an empty or null fallback when a safe control behavior is required. A default is product behavior, not only an SDK parameter.
|
|
33
|
+
|
|
34
|
+
## Assignment identity
|
|
35
|
+
|
|
36
|
+
Resolve:
|
|
37
|
+
|
|
38
|
+
- pre-login identity;
|
|
39
|
+
- post-login identity;
|
|
40
|
+
- account switching;
|
|
41
|
+
- multiple roles under one account;
|
|
42
|
+
- device-to-account merge or alias behavior;
|
|
43
|
+
- server/client identity consistency.
|
|
44
|
+
|
|
45
|
+
If a custom bucket ID is used, set it before fetching or reading the Feature. The verified client SDKs expose a custom bucket map, but the exact key and value semantics must match the experiment configuration. Do not assume that the literal example `bucket_id -> account_id` is the only supported schema.
|
|
46
|
+
|
|
47
|
+
Do not use custom request parameters as a hidden substitute for assignment identity unless the platform contract explicitly defines that behavior.
|
|
48
|
+
|
|
49
|
+
## Fetch and cache
|
|
50
|
+
|
|
51
|
+
Define:
|
|
52
|
+
|
|
53
|
+
- initial fetch timing;
|
|
54
|
+
- request timeout;
|
|
55
|
+
- retry and backoff;
|
|
56
|
+
- last-known-good cache;
|
|
57
|
+
- cache freshness;
|
|
58
|
+
- whether a stale value may be used;
|
|
59
|
+
- whether a mid-session update can change behavior;
|
|
60
|
+
- control fallback when no value is available.
|
|
61
|
+
|
|
62
|
+
Avoid UI flicker and treatment changes after the user has already seen control. For a session-scoped experience, freeze the evaluated value for the session unless the product requirement explicitly allows live changes.
|
|
63
|
+
|
|
64
|
+
Remote Config documentation establishes the general fallback order for configuration keys:
|
|
65
|
+
|
|
66
|
+
1. remote value;
|
|
67
|
+
2. local default;
|
|
68
|
+
3. empty when neither exists.
|
|
69
|
+
|
|
70
|
+
For an experiment Feature, prefer an explicit typed application default even when the platform SDK offers an overload or local default store.
|
|
71
|
+
|
|
72
|
+
## Exposure
|
|
73
|
+
|
|
74
|
+
Configuration retrieval is not exposure. A getter that automatically reports exposure is valid only if calling the getter coincides with applying the assigned behavior. If the application reads early, preloads, branches later, or may discard the result, disable automatic exposure and use manual exposure at the true activation point.
|
|
75
|
+
|
|
76
|
+
Read `exposure-contract.md` before finalizing exposure behavior.
|
|
77
|
+
|
|
78
|
+
## Custom fetch parameters
|
|
79
|
+
|
|
80
|
+
Use custom fetch parameters only for fields supported by the client configuration channel. Document:
|
|
81
|
+
|
|
82
|
+
- field name and type;
|
|
83
|
+
- source of truth;
|
|
84
|
+
- whether it affects targeting, diagnostics, or payload enrichment;
|
|
85
|
+
- privacy classification;
|
|
86
|
+
- behavior when absent.
|
|
87
|
+
|
|
88
|
+
Never place secrets or unstable session values in custom fetch parameters.
|
|
89
|
+
|
|
90
|
+
## Acceptance checks
|
|
91
|
+
|
|
92
|
+
- Analytics, Remote Config, and experiment SDK versions are compatible.
|
|
93
|
+
- Initialization uses the intended app and environment.
|
|
94
|
+
- The control default is returned when configuration is unavailable.
|
|
95
|
+
- A test identity receives a stable group across sessions.
|
|
96
|
+
- Client and server agree for the same assignment identity.
|
|
97
|
+
- Exposure occurs once at treatment activation, not at preload.
|
|
98
|
+
- Outcome events join to exposure.
|
|
99
|
+
- Account switching does not leak the previous account’s assignment.
|
|
100
|
+
- Debug logging and test mode are disabled or production-safe before release.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
# Experiment Exposure Contract
|
|
2
|
+
|
|
3
|
+
Read this reference whenever exposure affects assignment, metrics, SDK code, or rollout readiness.
|
|
4
|
+
|
|
5
|
+
## Exposure definition
|
|
6
|
+
|
|
7
|
+
Exposure means the assigned behavior was actually rendered or became effective for the analysis unit. It is not:
|
|
8
|
+
|
|
9
|
+
- SDK initialization;
|
|
10
|
+
- configuration fetch;
|
|
11
|
+
- Feature preload;
|
|
12
|
+
- getter execution when the result may be discarded;
|
|
13
|
+
- assignment calculation without treatment activation.
|
|
14
|
+
|
|
15
|
+
## Automatic versus manual exposure
|
|
16
|
+
|
|
17
|
+
The verified Android, iOS, and JavaScript experiment pages state that a typed getter automatically uploads exposure when `automaticExposureTracking` is enabled.
|
|
18
|
+
|
|
19
|
+
Use automatic exposure only when the getter is called at the same decision point where the value is applied.
|
|
20
|
+
|
|
21
|
+
Use manual exposure when:
|
|
22
|
+
|
|
23
|
+
- values are prefetched;
|
|
24
|
+
- a getter is called before eligibility is final;
|
|
25
|
+
- the result may not render;
|
|
26
|
+
- the server assigns but the client activates;
|
|
27
|
+
- one value is read multiple times before a single activation;
|
|
28
|
+
- the product needs a later, explicit effective point.
|
|
29
|
+
|
|
30
|
+
Do not combine automatic getter exposure and manual exposure for the same activation.
|
|
31
|
+
|
|
32
|
+
## Exposure owner
|
|
33
|
+
|
|
34
|
+
Choose exactly one:
|
|
35
|
+
|
|
36
|
+
- client rendering owner;
|
|
37
|
+
- server behavior owner;
|
|
38
|
+
- another verified activation service.
|
|
39
|
+
|
|
40
|
+
Document why that component knows the treatment became effective.
|
|
41
|
+
|
|
42
|
+
## Deduplication contract
|
|
43
|
+
|
|
44
|
+
Define an application-level deduplication key from stable facts such as:
|
|
45
|
+
|
|
46
|
+
- project and environment;
|
|
47
|
+
- experiment or Feature;
|
|
48
|
+
- assignment identity;
|
|
49
|
+
- assignment or group;
|
|
50
|
+
- activation scope such as session, page instance, or decision instance.
|
|
51
|
+
|
|
52
|
+
Do not invent AE event property names. Let the verified SDK emit its supported schema, or resolve the exposure event contract through `ae-cli`.
|
|
53
|
+
|
|
54
|
+
Retry must be idempotent at the chosen activation scope.
|
|
55
|
+
|
|
56
|
+
## Identity and join
|
|
57
|
+
|
|
58
|
+
The identity used for assignment, exposure, and outcome must either be the same or have a verified merge path.
|
|
59
|
+
|
|
60
|
+
Block launch when:
|
|
61
|
+
|
|
62
|
+
- anonymous and logged-in identities can cross groups;
|
|
63
|
+
- server and client use different assignment subjects;
|
|
64
|
+
- exposure uses device identity while outcomes use an unlinked account identity;
|
|
65
|
+
- account switching can inherit a cached assignment;
|
|
66
|
+
- the analysis unit cannot be reconstructed from platform evidence.
|
|
67
|
+
|
|
68
|
+
## Outcome contract
|
|
69
|
+
|
|
70
|
+
For the confirmed primary event metric, define:
|
|
71
|
+
|
|
72
|
+
- event or measure;
|
|
73
|
+
- identity;
|
|
74
|
+
- timestamp;
|
|
75
|
+
- attribution window;
|
|
76
|
+
- denominator or eligible population;
|
|
77
|
+
- relation to exposure;
|
|
78
|
+
- duplicate and late-event treatment.
|
|
79
|
+
|
|
80
|
+
Do not report an exposure merely to force a user into the denominator. Fix the metric or eligibility contract instead.
|
|
81
|
+
|
|
82
|
+
## Acceptance tests
|
|
83
|
+
|
|
84
|
+
- Getter without rendering does not produce exposure in manual mode.
|
|
85
|
+
- Rendering produces one exposure.
|
|
86
|
+
- Re-render behavior matches the pre-registered exposure scope.
|
|
87
|
+
- Retry does not create duplicate logical exposure.
|
|
88
|
+
- Control and treatment exposure are both observable.
|
|
89
|
+
- Assignment proportions match configured allocation.
|
|
90
|
+
- Exposure and outcome join for anonymous, login, logout, and account-switch paths.
|
|
91
|
+
- Offline and delayed-upload behavior is understood before launch.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# Hybrid Experiment Architecture
|
|
2
|
+
|
|
3
|
+
Read this reference when the server assigns or prepares a variant but the client renders or activates the treatment.
|
|
4
|
+
|
|
5
|
+
## Ownership model
|
|
6
|
+
|
|
7
|
+
Assign one owner for each responsibility:
|
|
8
|
+
|
|
9
|
+
| Responsibility | Typical owner |
|
|
10
|
+
|---|---|
|
|
11
|
+
| Stable identity resolution | Server or shared identity service |
|
|
12
|
+
| Assignment or Feature evaluation | Server |
|
|
13
|
+
| Typed default | Both sides, with one canonical value |
|
|
14
|
+
| Rendering or activation | Client |
|
|
15
|
+
| Exposure | Client when it knows rendering succeeded |
|
|
16
|
+
| Outcome events | Side that owns the business action |
|
|
17
|
+
|
|
18
|
+
These are design defaults, not fixed product requirements.
|
|
19
|
+
|
|
20
|
+
## Server-to-client envelope
|
|
21
|
+
|
|
22
|
+
Define an application envelope such as:
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
{
|
|
26
|
+
featureKey,
|
|
27
|
+
value,
|
|
28
|
+
valueType,
|
|
29
|
+
assignmentOrGroupId,
|
|
30
|
+
configVersion,
|
|
31
|
+
evaluatedAt,
|
|
32
|
+
exposureToken?
|
|
33
|
+
}
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
This is architecture-level pseudocode. Verify any AE-specific schema through `ae-cli`.
|
|
37
|
+
|
|
38
|
+
## Consistency rules
|
|
39
|
+
|
|
40
|
+
- Server and client must use the same project, environment, Feature key, and stable identity.
|
|
41
|
+
- The client must validate the value type before applying it.
|
|
42
|
+
- A missing, invalid, or expired envelope falls back to the typed control behavior.
|
|
43
|
+
- Do not re-evaluate independently on the client after a server assignment unless the product explicitly supports that mode.
|
|
44
|
+
- Do not allow account switching to reuse another account’s cached envelope.
|
|
45
|
+
- Include a configuration version or equivalent diagnostic marker when the verified platform provides one.
|
|
46
|
+
|
|
47
|
+
## Exposure handoff
|
|
48
|
+
|
|
49
|
+
The server should not report exposure merely because it returned a treatment envelope when the client may never render it. Prefer:
|
|
50
|
+
|
|
51
|
+
1. server evaluates;
|
|
52
|
+
2. client receives and validates;
|
|
53
|
+
3. client applies behavior;
|
|
54
|
+
4. client reports exposure once.
|
|
55
|
+
|
|
56
|
+
If exposure must be server-side, define a client acknowledgment or another verified activation signal and make retry idempotent.
|
|
57
|
+
|
|
58
|
+
## Failure scenarios
|
|
59
|
+
|
|
60
|
+
- server evaluation succeeds, client render fails: no exposure;
|
|
61
|
+
- server times out: client uses control;
|
|
62
|
+
- client starts offline with cached assignment: use only within the documented stale window;
|
|
63
|
+
- server and client identity disagree: block rollout;
|
|
64
|
+
- both sides report exposure: disable one path before launch;
|
|
65
|
+
- value arrives after control rendered: freeze control for the session or use an explicitly approved transition.
|
|
66
|
+
|
|
67
|
+
## Acceptance checks
|
|
68
|
+
|
|
69
|
+
- Server and client return or apply the same value type.
|
|
70
|
+
- Treatment is not visible before identity is stable.
|
|
71
|
+
- Exposure is reported by one owner.
|
|
72
|
+
- Refresh and retry do not duplicate exposure.
|
|
73
|
+
- Cached assignment is isolated by identity and environment.
|
|
74
|
+
- Outcome events remain joinable across client and server.
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
# Metric Readiness
|
|
2
|
+
|
|
3
|
+
Use this checklist to decide whether a candidate metric can support an experiment decision.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Supported experiment metric contract
|
|
8
|
+
- Preregistered planning policy
|
|
9
|
+
- Evidence levels
|
|
10
|
+
- Feasibility gates
|
|
11
|
+
- Role selection
|
|
12
|
+
- Common blockers
|
|
13
|
+
- Required output
|
|
14
|
+
|
|
15
|
+
## Supported experiment metric contract
|
|
16
|
+
|
|
17
|
+
Use event metrics only. A platform enum or API field is not evidence that
|
|
18
|
+
another metric kind is implemented in the report calculation path.
|
|
19
|
+
|
|
20
|
+
Use only these event calculation codes:
|
|
21
|
+
|
|
22
|
+
| Code | Calculation | User-level value | Planning type |
|
|
23
|
+
| --- | --- | --- | --- |
|
|
24
|
+
| `A101` | Triggered users | Whether each user triggered the event; compare group trigger rates | `proportion` |
|
|
25
|
+
| `A100` | Total times | Each user's total event count; compare group user-level means | `continuous` |
|
|
26
|
+
| `A201` | Days | Each user's distinct active-event days; compare group user-level means | `continuous` |
|
|
27
|
+
| `A103` | Sum | Each user's sum of a numeric event property; compare group user-level means | `continuous` |
|
|
28
|
+
| `A104` | Average | Each user's average numeric event-property value; compare group user-level means | `continuous` |
|
|
29
|
+
| `A106` | Maximum | Each user's maximum numeric event-property value; compare group user-level means | `continuous` |
|
|
30
|
+
| `A108` | Distinct count | Each user's distinct count of one event property; compare group user-level means | `continuous` |
|
|
31
|
+
|
|
32
|
+
Require a numeric event property for `A103`, `A104`, and `A106`, and an exact
|
|
33
|
+
event property for `A108`.
|
|
34
|
+
|
|
35
|
+
The current report calculation uses a two-sided Z test for both planning
|
|
36
|
+
types, returns p-value, lift, and a 95% confidence interval, and uses
|
|
37
|
+
`p < 0.05` plus confidence-interval direction and goal direction to label a
|
|
38
|
+
positive or negative significant result.
|
|
39
|
+
|
|
40
|
+
## Preregistered planning policy
|
|
41
|
+
|
|
42
|
+
Fix the family-wise significance level at `alpha=0.05`, the confidence level
|
|
43
|
+
at `0.95`, and the test direction at two-sided to match current reporting.
|
|
44
|
+
Use `power=0.80` as the explicit planning-policy default when the user does not
|
|
45
|
+
provide power, and label its source as `policy_default`.
|
|
46
|
+
|
|
47
|
+
Require the MDE before definitive planning. Record whether it is `relative` or
|
|
48
|
+
`absolute` and whether the desired effect is an `increase` or `decrease`; never
|
|
49
|
+
silently substitute a 5% lift. Require observed user-level standard deviation
|
|
50
|
+
for continuous metrics; never substitute an assumed coefficient of variation.
|
|
51
|
+
|
|
52
|
+
For one control and multiple treatments, treat each treatment-versus-control
|
|
53
|
+
comparison as one member of the same family. Use Bonferroni planning:
|
|
54
|
+
`alpha_per_comparison = 0.05 / number_of_treatments`. Return the family alpha,
|
|
55
|
+
adjusted alpha, method, and comparison count. The current native report still
|
|
56
|
+
labels significance with unadjusted `p < 0.05` and a 95% confidence interval,
|
|
57
|
+
so use raw p-values against the preregistered adjusted threshold for the final
|
|
58
|
+
decision and disclose that native-label mismatch.
|
|
59
|
+
|
|
60
|
+
Use a minimum runtime of seven days and align recommendations to full weeks by
|
|
61
|
+
default; permit explicit overrides when the decision context justifies them.
|
|
62
|
+
When traffic is not fixed, evaluate absolute experiment-traffic candidates no
|
|
63
|
+
greater than the verified layer capacity and choose the smallest candidate
|
|
64
|
+
that reaches the sample target within the maximum runtime. If none is feasible,
|
|
65
|
+
return the true duration at maximum available traffic and mark it infeasible;
|
|
66
|
+
never truncate the duration to the maximum.
|
|
67
|
+
|
|
68
|
+
Do not add ad hoc missing-value imputation, winsorization, truncation, or
|
|
69
|
+
post-treatment exclusions. Use the exact platform metric definition and
|
|
70
|
+
preregister any supported filtering or data-quality rule before the run.
|
|
71
|
+
|
|
72
|
+
Call `calculate_experiment_plan.py` with `metric_type`, `baseline`, `mde`,
|
|
73
|
+
`mde_type`, `effect_direction`, `allocations`, and `daily_eligible_units`.
|
|
74
|
+
Pass `standard_deviation` for a continuous metric. Pass `experiment_traffic`
|
|
75
|
+
when traffic is confirmed; otherwise pass `max_available_traffic` and
|
|
76
|
+
optionally `traffic_candidates` to request a recommendation. `power`,
|
|
77
|
+
`min_runtime_days`, `max_runtime_days`, and `align_to_full_weeks` may override
|
|
78
|
+
their policy defaults. Do not pass `confidence_level` other than `0.95`.
|
|
79
|
+
|
|
80
|
+
Do not create or recommend a metric outside this contract. If the requested
|
|
81
|
+
business outcome cannot be expressed by one supported event calculation,
|
|
82
|
+
state that it is unavailable and propose the closest valid event metric
|
|
83
|
+
without representing it as equivalent.
|
|
84
|
+
|
|
85
|
+
## Evidence levels
|
|
86
|
+
|
|
87
|
+
Label every statement:
|
|
88
|
+
|
|
89
|
+
- `verified`: supported by current `ae-cli` platform output;
|
|
90
|
+
- `provided`: supplied by the user but not verified on the platform;
|
|
91
|
+
- `assumed`: introduced for design or sensitivity analysis;
|
|
92
|
+
- `missing`: required evidence is unavailable.
|
|
93
|
+
|
|
94
|
+
Only `verified` or explicitly accepted `provided` evidence can support a final readiness decision.
|
|
95
|
+
|
|
96
|
+
## Feasibility gates
|
|
97
|
+
|
|
98
|
+
A metric is feasible only when all applicable gates pass:
|
|
99
|
+
|
|
100
|
+
1. **Contract**: event, supported calculation code, required property, goal
|
|
101
|
+
direction, filters, and attribution window are explicit.
|
|
102
|
+
2. **Source**: required events and properties exist or are supplied as a concrete schema.
|
|
103
|
+
3. **Identity**: assignment, exposure, and outcome share a stable join path.
|
|
104
|
+
4. **Ordering**: outcome is attributed after exposure or treatment activation.
|
|
105
|
+
5. **Population**: pre-treatment eligibility and exclusions are reproducible.
|
|
106
|
+
6. **Planning evidence**: baseline, MDE and its type and direction, eligible
|
|
107
|
+
daily units, verified layer capacity when recommending traffic, and—except
|
|
108
|
+
for `A101`—user-level standard deviation are available before a definitive
|
|
109
|
+
sample or duration calculation.
|
|
110
|
+
7. **Stability**: collection and definition are not expected to change during the experiment.
|
|
111
|
+
8. **Sensitivity**: the metric can plausibly respond to the treatment.
|
|
112
|
+
|
|
113
|
+
Block or mark unverified when any required gate lacks evidence.
|
|
114
|
+
|
|
115
|
+
## Role selection
|
|
116
|
+
|
|
117
|
+
### Primary
|
|
118
|
+
|
|
119
|
+
Choose one supported event metric closest to the hypothesis and decision.
|
|
120
|
+
Confirm its calculation code and direction, then establish its baseline and
|
|
121
|
+
minimum business-relevant effect. Do not calculate a definitive duration
|
|
122
|
+
before the metric contract is confirmed.
|
|
123
|
+
|
|
124
|
+
## Common blockers
|
|
125
|
+
|
|
126
|
+
- missing exposure event or treatment-activation marker;
|
|
127
|
+
- assignment uses `user_id` while outcomes only have `device_id`;
|
|
128
|
+
- anonymous-to-login identity cannot be reconciled;
|
|
129
|
+
- cross-device identity is unstable;
|
|
130
|
+
- outcome precedes exposure;
|
|
131
|
+
- denominator is filtered by post-treatment behavior;
|
|
132
|
+
- event volume is too low;
|
|
133
|
+
- metric definition or instrumentation changed during the run;
|
|
134
|
+
- cross-platform events use incompatible identities or semantics.
|
|
135
|
+
|
|
136
|
+
## Required output
|
|
137
|
+
|
|
138
|
+
For each candidate, return:
|
|
139
|
+
|
|
140
|
+
| Metric | Status | Event | Calculation | Property | Direction | Join identity | Window | Evidence | Reason or remediation |
|
|
141
|
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
142
|
+
|
|
143
|
+
When blocked, give the smallest concrete instrumentation or identity change that would make the metric measurable.
|
|
@@ -0,0 +1,105 @@
|
|
|
1
|
+
# Platform Operations
|
|
2
|
+
|
|
3
|
+
Read this reference before any AE/TE platform interaction.
|
|
4
|
+
|
|
5
|
+
## Hard boundary
|
|
6
|
+
|
|
7
|
+
Use `ae-cli` for all platform operations. Do not call raw APIs, use browser automation, query a database, or fabricate a platform result.
|
|
8
|
+
|
|
9
|
+
Use curated commands when their contract is known. Use the Capability Gateway for experiment capabilities that do not yet have a curated command:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
ae-cli capability search "<terms>" --domain <domain> [--project-id <id>]
|
|
13
|
+
ae-cli capability inspect <capability-id> --project-id <id>
|
|
14
|
+
ae-cli capability validate <capability-id> --input '<json-object>'
|
|
15
|
+
ae-cli capability dry-run <capability-id> --input '<json-object>'
|
|
16
|
+
ae-cli capability run <capability-id> --input '<json-object>'
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Search and inspect before constructing input. Never guess capability IDs, fields, enums, resource IDs, or project IDs. Search with the narrowest useful terms, then inspect every plausible candidate.
|
|
20
|
+
|
|
21
|
+
Use at most one normal pre-check:
|
|
22
|
+
|
|
23
|
+
- use `validate` while correcting a complex nested payload;
|
|
24
|
+
- use `dry-run` for the final request when execution impact or deletion risk must be reviewed;
|
|
25
|
+
- do not routinely stack `validate`, `dry-run`, and `run`.
|
|
26
|
+
|
|
27
|
+
## Host compatibility
|
|
28
|
+
|
|
29
|
+
After every `ae-cli` invocation, inspect stderr and `_notice.host_compat`. If a notice exists, start the user-facing result with a version warning and reproduce its update command lines verbatim. A successful envelope does not cancel the notice.
|
|
30
|
+
|
|
31
|
+
## Project gate
|
|
32
|
+
|
|
33
|
+
When the user supplies a project ID, resolve it directly:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
ae-cli project info get --project-id <id>
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
When the user supplies only a project name, search and disambiguate:
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
ae-cli project info list --query "<project name>" --fields '["project_id","project_name"]' --limit 20 --offset 0
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Reuse a project only when its ID and host were verified in the same continuous conversation. If multiple projects match, stop and ask the user to choose.
|
|
46
|
+
|
|
47
|
+
## Metadata and metric evidence
|
|
48
|
+
|
|
49
|
+
Use these curated reads when applicable:
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
ae-cli analysis-meta event list --project-id <id> --query "<event>" --fields '["event_name","event_desc","authentication_status"]' --limit 20 --offset 0
|
|
53
|
+
ae-cli analysis-meta property list --project-id <id> --scope event --event-name "<event>" --limit 50 --offset 0
|
|
54
|
+
ae-cli analysis-meta property list --project-id <id> --scope user --limit 50 --offset 0
|
|
55
|
+
ae-cli analysis-meta metric list --project-id <id> --query "<metric>" --fields '["metric_id","metric_name","metric_desc","authentication_status"]' --limit 20 --offset 0
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Use `ae-cli analysis-meta metric get` after list/search when the exact metric definition is required. Use `ae-cli analysis adhoc run|export` for historical baseline and eligible-traffic evidence, following the installed `ae-analysis` command reference. Do not infer a baseline from a metric definition.
|
|
59
|
+
|
|
60
|
+
For metric creation, use the curated `ae-cli analysis-meta metric create`
|
|
61
|
+
command only after the supported event-metric definition and calculation code
|
|
62
|
+
are exact. Use `--validate` while fixing complex metric input. Do not create a
|
|
63
|
+
metric from a prose-only definition.
|
|
64
|
+
|
|
65
|
+
## Experiment capability discovery
|
|
66
|
+
|
|
67
|
+
Discover, inspect, and select capabilities for the semantic operation. Typical search concepts include:
|
|
68
|
+
|
|
69
|
+
- experiment list, get, draft save, readiness, conflict, submit, start, pause, end, delete;
|
|
70
|
+
- Feature list, get, create, update;
|
|
71
|
+
- layer or bucket list and available traffic;
|
|
72
|
+
- experiment metric report, summary, trend, sample, group, or exposure.
|
|
73
|
+
|
|
74
|
+
Catalog wording and capability IDs may differ by deployment. The inspected `input_schema`, `risk`, auth requirements, output contract, and supported dry-run behavior are authoritative.
|
|
75
|
+
|
|
76
|
+
If search returns no supported capability, report `capability gap` and continue with an offline design. Do not convert an intended platform write into raw HTTP.
|
|
77
|
+
|
|
78
|
+
## Experiment detail link
|
|
79
|
+
|
|
80
|
+
After a successful experiment save or verified exact-match reuse:
|
|
81
|
+
|
|
82
|
+
1. Read the actual `expId` from the successful `ae-cli` result; do not use a proposed input ID unless the inspected output contract confirms that the save preserved it.
|
|
83
|
+
2. Prefer a canonical detail link returned by the inspected capability output contract.
|
|
84
|
+
3. When the result contains the actual `expId` but no link, use the verified relative frontend route:
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
/#/atlas/experiment/detail?expId=<actual-expId>¤tProjectId=<actual-projectId>
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
The user-facing response must render this as a Markdown link with a localized label. Do not put it in a code block, output it as bare text, add a `{domain}` placeholder, or construct it when either verified ID is missing.
|
|
91
|
+
|
|
92
|
+
## Risk and lifecycle policy
|
|
93
|
+
|
|
94
|
+
- `read`: execute after project and target resolution.
|
|
95
|
+
- `write`: execute when it is directly requested and inputs are exact.
|
|
96
|
+
- `high-risk-write`: dry-run the final input, summarize target and impact, wait for a later explicit confirmation, then execute unchanged input with `--yes`.
|
|
97
|
+
|
|
98
|
+
Regardless of catalog risk, require a separate explicit confirmation before submitting, starting, changing live traffic, pausing, ending, or deleting an experiment. A request to design or create a draft is not authorization to launch.
|
|
99
|
+
|
|
100
|
+
## Envelope handling
|
|
101
|
+
|
|
102
|
+
- `ok: true` with empty data is a successful empty result.
|
|
103
|
+
- `ok: true` with `meta.partial: true` is partial success; preserve `meta.failures`.
|
|
104
|
+
- `ok: false` is failure; preserve error code, message, request ID, invocation ID, stage, and failures.
|
|
105
|
+
- Do not retry an unchanged failed request.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# Experiment SDK Documentation Index
|
|
2
|
+
|
|
3
|
+
Read this reference before giving exact experiment-SDK versions, dependencies, packages, imports, or API calls.
|
|
4
|
+
|
|
5
|
+
## Source policy
|
|
6
|
+
|
|
7
|
+
- Bundled references contain a concise extract for common questions.
|
|
8
|
+
- The local official Wiki is preferred for exact version-specific APIs when it contains the relevant main page.
|
|
9
|
+
- If the Wiki mirror is missing an experiment page, use the verified official source below and label the local mirror gap.
|
|
10
|
+
- Recheck document version and update date before generating production code.
|
|
11
|
+
- Do not use a page merely because a link label says it is an SDK page; verify its title and content.
|
|
12
|
+
|
|
13
|
+
## Experiment SDK matrix
|
|
14
|
+
|
|
15
|
+
Verified from the official Feishu pages on 2026-07-26:
|
|
16
|
+
|
|
17
|
+
| Platform | Experiment SDK | Analytics dependency | Remote Config dependency | Main source |
|
|
18
|
+
|---|---:|---:|---:|---|
|
|
19
|
+
| Android | `TDExperiment` 1.0.1 | `TDAnalytics` >= 3.3.6 | `TDRemoteConfig` >= 1.3.0 | `CQ83wzJjui88V1keGtNcKLxgnXb` |
|
|
20
|
+
| iOS | `TDExperiment` 1.0.2 | `ThinkingSDK` >= 3.1.6 | `TDRemoteConfig` >= 1.3.1 | `RlYIwkcvXizJeFkhp2acbWGwnFf` |
|
|
21
|
+
| JavaScript | experiment package 1.0.0 | `TDAnalytics` >= 2.6.0 | `TDRemoteconfig` >= 1.3.0 | `GlR8w2bFbiJKqUka03pcJEdnndf` |
|
|
22
|
+
|
|
23
|
+
Official source pages:
|
|
24
|
+
|
|
25
|
+
- Client SDK overview: `https://thinkingdata.feishu.cn/wiki/MCy3w6eKjihQsmkZWSQcNlrQnie`
|
|
26
|
+
- Android experiment SDK: `https://thinkingdata.feishu.cn/wiki/CQ83wzJjui88V1keGtNcKLxgnXb`
|
|
27
|
+
- iOS experiment SDK: `https://thinkingdata.feishu.cn/wiki/RlYIwkcvXizJeFkhp2acbWGwnFf`
|
|
28
|
+
- JavaScript experiment SDK: `https://thinkingdata.feishu.cn/wiki/GlR8w2bFbiJKqUka03pcJEdnndf`
|
|
29
|
+
|
|
30
|
+
## Remote Config dependency matrix
|
|
31
|
+
|
|
32
|
+
| Platform | Remote Config version | Minimum OS | Main source |
|
|
33
|
+
|---|---:|---:|---|
|
|
34
|
+
| Android | 1.3.0 | Android 4.0 / API 14 | `Lsdmwhhigi9Z3AkKs2fc9tw3nEd` |
|
|
35
|
+
| iOS | 1.3.1 | iOS 9.0 | `K85DwZetDidpMPkGHWhccKQGnUf` |
|
|
36
|
+
| JavaScript | 1.3.0 | Not stated in the verified page | `Yg3uw8RDii2WEskhz2Qc4zOanwg` |
|
|
37
|
+
|
|
38
|
+
Official source pages:
|
|
39
|
+
|
|
40
|
+
- Android Remote Config: `https://thinkingdata.feishu.cn/wiki/Lsdmwhhigi9Z3AkKs2fc9tw3nEd`
|
|
41
|
+
- iOS Remote Config: `https://thinkingdata.feishu.cn/wiki/K85DwZetDidpMPkGHWhccKQGnUf`
|
|
42
|
+
- JavaScript Remote Config: `https://thinkingdata.feishu.cn/wiki/Yg3uw8RDii2WEskhz2Qc4zOanwg`
|
|
43
|
+
|
|
44
|
+
## Local Wiki discovery
|
|
45
|
+
|
|
46
|
+
The runtime mirror is expected under:
|
|
47
|
+
|
|
48
|
+
```text
|
|
49
|
+
~/.ae-cli/wiki/te-docs/
|
|
50
|
+
├── index.md
|
|
51
|
+
├── synthesis/
|
|
52
|
+
└── raw/
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Before reading a local experiment page:
|
|
56
|
+
|
|
57
|
+
1. Check whether `index.md` exists.
|
|
58
|
+
2. Search the index and `raw/` for `experiment`, `TDExperiment`, `remote config`, `配置中心`, and `实验`.
|
|
59
|
+
3. Choose the latest main page, not a historical/versioned copy.
|
|
60
|
+
4. Read synthesis for orientation and raw for exact API details.
|
|
61
|
+
|
|
62
|
+
At the verification date, the local mirror used to build this reference did not contain experiment or Remote Config pages. Do not assume this remains true after a Wiki refresh.
|
|
63
|
+
|
|
64
|
+
## Known source conflict
|
|
65
|
+
|
|
66
|
+
The client SDK overview page labels `AvX7wAp0zix522kSy9BcBziMnJd` as the JavaScript Remote Config document, but that token currently resolves to a Data Platform Release Note. The JavaScript experiment page links to `Yg3uw8RDii2WEskhz2Qc4zOanwg`, which resolves to the verified JavaScript Remote Config 1.3.0 document.
|
|
67
|
+
|
|
68
|
+
Use `Yg3uw8RDii2WEskhz2Qc4zOanwg` until the overview page is corrected. If either page changes, verify the title, SDK version, and API content again.
|
|
69
|
+
|
|
70
|
+
## JavaScript spelling warning
|
|
71
|
+
|
|
72
|
+
The verified JavaScript experiment page spells the global object `TDExpriment`. This may be an SDK export name or a documentation typo. Copy it only after checking the downloaded package or a newer verified main document. Do not silently “correct” it to `TDExperiment`, and do not ship the documented spelling without verification.
|
|
73
|
+
|
|
74
|
+
## Unsupported exact-code cases
|
|
75
|
+
|
|
76
|
+
The supplied official pages do not establish an exact server-side experiment SDK or server evaluation API. For server or hybrid implementation, follow the architecture references and use pseudocode until `ae-cli` or verified server documentation resolves a concrete API.
|