@thinkingai/ae-cli 6.0.48 → 6.1.0-project-role-upsert.1

package/skills/ae-experiment-design/references/server-experiment-sdk.md

@@ -0,0 +1,78 @@
+# Server Experiment Architecture
+
+Read this reference when assignment, Feature evaluation, or behavior execution belongs on a backend service.
+
+## Evidence boundary
+
+The verified client documentation does not establish an exact AE server experiment SDK, package, endpoint, or method. Do not invent one.
+
+Before exact implementation:
+
+1. Resolve the real project, experiment, Feature, and assignment unit with `ae-cli`.
+2. Search and inspect the Capability Gateway for server experiment evaluation or SDK generation.
+3. Read verified server documentation if returned by the platform or provided by the user.
+4. Use architecture-level pseudocode until the API contract is verified.
+
+## Recommended server contract
+
+```text
+evaluateExperiment(
+  project,
+  featureKey,
+  stableAssignmentId,
+  context
+) -> {
+  value,
+  valueType,
+  assignmentOrGroupId,
+  configVersion,
+  source,
+  evaluatedAt
+}
+```
+
+The field names above describe an application contract, not a verified AE API schema.
+
+## Server responsibilities
+
+- derive a stable assignment ID before evaluation;
+- evaluate once per request, session, or product-defined decision boundary;
+- apply a typed control default on timeout or unavailable configuration;
+- preserve assignment consistency across replicas and regions;
+- prevent cache keys from mixing projects, environments, Features, or identities;
+- expose enough internal metadata to diagnose assignment without leaking it to end users;
+- emit or authorize exposure only when treatment becomes effective;
+- report outcomes with a joinable identity.
+
+## Cache policy
+
+Define separately:
+
+- configuration cache;
+- per-identity assignment cache;
+- response or page cache.
+
+Do not cache a personalized treatment under a shared response key. Include environment, project, Feature key, and assignment identity in any evaluation cache key unless the verified SDK guarantees safe internal caching.
+
+## Failure policy
+
+- timeout or transport failure: use typed control default;
+- invalid value type: reject the value and use control;
+- unknown Feature: fail closed to control and alert;
+- stale configuration: use only within the approved stale window;
+- identity missing: either use a documented anonymous assignment or control; never generate a new unstable ID per request.
+
+## Exposure ownership
+
+If the server fully executes the behavior, server exposure may be appropriate. If the server only sends a variant and the client decides whether it is rendered, client exposure is usually the truthful point.
+
+Choose one owner and read `exposure-contract.md`.
+
+## Verification checklist
+
+- Same identity returns a stable value across replicas.
+- Control fallback is safe and typed.
+- Cache isolation prevents cross-user treatment leakage.
+- Exposure reflects effective behavior, not evaluation alone.
+- Client and server do not both report the same exposure.
+- Outcome identity joins to assignment and exposure.

package/skills/ae-experiment-design/scripts/calculate_experiment_plan.py

@@ -0,0 +1,450 @@
+#!/usr/bin/env python3
+"""Calculate deterministic A/B experiment sample targets and duration."""
+
+from __future__ import annotations
+
+import json
+import math
+import sys
+from typing import Any
+
+
+def normal_ppf(probability: float) -> float:
+    """Return the inverse standard-normal CDF using Acklam's approximation."""
+    if not 0.0 < probability < 1.0:
+        raise ValueError("probability must be between 0 and 1")
+
+    a = (
+        -3.969683028665376e01,
+        2.209460984245205e02,
+        -2.759285104469687e02,
+        1.383577518672690e02,
+        -3.066479806614716e01,
+        2.506628277459239e00,
+    )
+    b = (
+        -5.447609879822406e01,
+        1.615858368580409e02,
+        -1.556989798598866e02,
+        6.680131188771972e01,
+        -1.328068155288572e01,
+    )
+    c = (
+        -7.784894002430293e-03,
+        -3.223964580411365e-01,
+        -2.400758277161838e00,
+        -2.549732539343734e00,
+        4.374664141464968e00,
+        2.938163982698783e00,
+    )
+    d = (
+        7.784695709041462e-03,
+        3.224671290700398e-01,
+        2.445134137142996e00,
+        3.754408661907416e00,
+    )
+
+    low = 0.02425
+    high = 1.0 - low
+    if probability < low:
+        q = math.sqrt(-2.0 * math.log(probability))
+        return (
+            (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
+            / ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0)
+        )
+    if probability <= high:
+        q = probability - 0.5
+        r = q * q
+        return (
+            (((((a[0] * r + a[1]) * r + a[2]) * r + a[3]) * r + a[4]) * r + a[5])
+            * q
+            / (((((b[0] * r + b[1]) * r + b[2]) * r + b[3]) * r + b[4]) * r + 1.0)
+        )
+    q = math.sqrt(-2.0 * math.log(1.0 - probability))
+    return -(
+        (((((c[0] * q + c[1]) * q + c[2]) * q + c[3]) * q + c[4]) * q + c[5])
+        / ((((d[0] * q + d[1]) * q + d[2]) * q + d[3]) * q + 1.0)
+    )
+
+
+def require_number(params: dict[str, Any], key: str) -> float:
+    value = params.get(key)
+    if isinstance(value, bool) or not isinstance(value, (int, float)):
+        raise ValueError(f"{key} must be a number")
+    if not math.isfinite(float(value)):
+        raise ValueError(f"{key} must be finite")
+    return float(value)
+
+
+def optional_integer(params: dict[str, Any], key: str, default: int) -> int:
+    value = params.get(key, default)
+    if isinstance(value, bool) or not isinstance(value, int):
+        raise ValueError(f"{key} must be an integer")
+    return value
+
+
+def optional_boolean(params: dict[str, Any], key: str, default: bool) -> bool:
+    value = params.get(key, default)
+    if not isinstance(value, bool):
+        raise ValueError(f"{key} must be a boolean")
+    return value
+
+
+def pair_sample_proportion(
+    baseline: float,
+    treatment: float,
+    ratio: float,
+    z_alpha: float,
+    z_power: float,
+) -> tuple[int, int]:
+    delta = abs(treatment - baseline)
+    if delta == 0.0:
+        raise ValueError("MDE must produce a non-zero absolute effect")
+    pooled_alternative = (baseline + ratio * treatment) / (1.0 + ratio)
+    first = z_alpha * math.sqrt(
+        pooled_alternative * (1.0 - pooled_alternative) * (1.0 + 1.0 / ratio)
+    )
+    second = z_power * math.sqrt(
+        baseline * (1.0 - baseline) + treatment * (1.0 - treatment) / ratio
+    )
+    control = math.ceil(((first + second) / delta) ** 2)
+    return control, math.ceil(control * ratio)
+
+
+def pair_sample_continuous(
+    absolute_effect: float,
+    ratio: float,
+    z_alpha: float,
+    z_power: float,
+    control_stddev: float,
+    treatment_stddev: float,
+) -> tuple[int, int]:
+    if absolute_effect == 0.0:
+        raise ValueError("MDE must produce a non-zero absolute effect")
+    control = math.ceil(
+        (z_alpha + z_power) ** 2
+        * (control_stddev**2 + treatment_stddev**2 / ratio)
+        / absolute_effect**2
+    )
+    return control, math.ceil(control * ratio)
+
+
+def calculate_duration(
+    total_required: int,
+    daily_eligible_units: float,
+    experiment_traffic: float,
+    min_runtime_days: int,
+    align_to_full_weeks: bool,
+) -> tuple[float, int, int]:
+    """Return effective daily units, raw duration, and policy-adjusted duration."""
+    effective_daily_units = daily_eligible_units * experiment_traffic
+    nearest_integer = round(effective_daily_units)
+    if math.isclose(
+        effective_daily_units,
+        nearest_integer,
+        rel_tol=0.0,
+        abs_tol=1e-9,
+    ):
+        effective_daily_units = float(nearest_integer)
+    else:
+        effective_daily_units = round(effective_daily_units, 12)
+    raw_duration_days = math.ceil(total_required / effective_daily_units)
+    duration_days = max(min_runtime_days, raw_duration_days)
+    if align_to_full_weeks:
+        duration_days = math.ceil(duration_days / 7.0) * 7
+    return effective_daily_units, raw_duration_days, duration_days
+
+
+def traffic_candidates(params: dict[str, Any], max_available_traffic: float) -> list[float]:
+    """Return validated absolute experiment-traffic candidates."""
+    raw_candidates = params.get(
+        "traffic_candidates",
+        [0.05, 0.10, 0.20, 0.30, 0.50, 0.70, 1.0],
+    )
+    if not isinstance(raw_candidates, list) or not raw_candidates:
+        raise ValueError("traffic_candidates must be a non-empty list")
+
+    candidates: list[float] = []
+    for value in raw_candidates:
+        if isinstance(value, bool) or not isinstance(value, (int, float)):
+            raise ValueError("every traffic candidate must be a number")
+        candidate = float(value)
+        if not math.isfinite(candidate) or not 0.0 < candidate <= 1.0:
+            raise ValueError("every traffic candidate must be greater than 0 and at most 1")
+        if candidate <= max_available_traffic + 1e-12:
+            candidates.append(min(candidate, max_available_traffic))
+
+    candidates.append(max_available_traffic)
+    return sorted(set(candidates))
+
+
+def calculate(params: dict[str, Any]) -> dict[str, Any]:
+    metric_type = params.get("metric_type")
+    if metric_type not in {"proportion", "continuous"}:
+        raise ValueError("metric_type must be proportion or continuous")
+
+    baseline = require_number(params, "baseline")
+    mde = require_number(params, "mde")
+    if mde <= 0.0:
+        raise ValueError("mde must be greater than 0")
+    mde_type = params.get("mde_type")
+    if mde_type not in {"relative", "absolute"}:
+        raise ValueError("mde_type must be relative or absolute")
+    effect_direction = params.get("effect_direction")
+    if effect_direction not in {"increase", "decrease"}:
+        raise ValueError("effect_direction must be increase or decrease")
+
+    confidence = (
+        require_number(params, "confidence_level")
+        if "confidence_level" in params
+        else 0.95
+    )
+    power_source = "provided" if "power" in params else "policy_default"
+    power = require_number(params, "power") if "power" in params else 0.80
+    if not math.isclose(confidence, 0.95, rel_tol=0.0, abs_tol=1e-12):
+        raise ValueError("current experiment reporting requires confidence_level=0.95")
+    if not 0.5 < power < 1.0:
+        raise ValueError("power must be between 0.5 and 1")
+
+    allocations_value = params.get("allocations")
+    if not isinstance(allocations_value, list) or len(allocations_value) < 2:
+        raise ValueError("allocations must contain control and at least one treatment")
+    if any(
+        isinstance(value, bool) or not isinstance(value, (int, float))
+        for value in allocations_value
+    ):
+        raise ValueError("every allocation must be a number")
+    allocations = [float(value) for value in allocations_value]
+    if any(not math.isfinite(value) or value <= 0.0 for value in allocations):
+        raise ValueError("every allocation must be a positive finite number")
+    if not math.isclose(sum(allocations), 1.0, rel_tol=0.0, abs_tol=1e-9):
+        raise ValueError("allocations must sum to 1.0")
+
+    daily_eligible_units = require_number(params, "daily_eligible_units")
+    if daily_eligible_units <= 0.0:
+        raise ValueError("daily_eligible_units must be greater than 0")
+
+    min_runtime_days = optional_integer(params, "min_runtime_days", 7)
+    max_runtime_days = optional_integer(params, "max_runtime_days", 60)
+    align_to_full_weeks = optional_boolean(params, "align_to_full_weeks", True)
+    if min_runtime_days < 1 or max_runtime_days < min_runtime_days:
+        raise ValueError("runtime bounds are invalid")
+
+    number_of_comparisons = len(allocations) - 1
+    alpha = 0.05
+    multiple_comparison_method = (
+        "bonferroni" if number_of_comparisons > 1 else "none"
+    )
+    alpha_per_comparison = alpha / number_of_comparisons
+    z_alpha = normal_ppf(1.0 - alpha_per_comparison / 2.0)
+    z_power = normal_ppf(power)
+
+    absolute_effect = mde if mde_type == "absolute" else abs(baseline) * mde
+    signed_effect = absolute_effect if effect_direction == "increase" else -absolute_effect
+    treatment_value = baseline + signed_effect
+    warnings: list[str] = []
+
+    if metric_type == "proportion":
+        if not 0.0 < baseline < 1.0:
+            raise ValueError("proportion baseline must be between 0 and 1")
+        if not 0.0 < treatment_value < 1.0:
+            raise ValueError("proportion MDE produces a treatment rate outside 0..1")
+    else:
+        control_stddev = require_number(params, "standard_deviation")
+        treatment_stddev = (
+            require_number(params, "treatment_standard_deviation")
+            if "treatment_standard_deviation" in params
+            else control_stddev
+        )
+        if control_stddev <= 0.0 or treatment_stddev <= 0.0:
+            raise ValueError("continuous standard deviations must be greater than 0")
+
+    control_allocation = allocations[0]
+    pair_requirements: list[dict[str, Any]] = []
+    total_scale = 0.0
+
+    for index, treatment_allocation in enumerate(allocations[1:], start=1):
+        ratio = treatment_allocation / control_allocation
+        if metric_type == "proportion":
+            control_required, treatment_required = pair_sample_proportion(
+                baseline, treatment_value, ratio, z_alpha, z_power
+            )
+        else:
+            control_required, treatment_required = pair_sample_continuous(
+                absolute_effect,
+                ratio,
+                z_alpha,
+                z_power,
+                control_stddev,
+                treatment_stddev,
+            )
+        pair_scale = max(
+            control_required / control_allocation,
+            treatment_required / treatment_allocation,
+        )
+        total_scale = max(total_scale, pair_scale)
+        pair_requirements.append(
+            {
+                "treatment_index": index,
+                "allocation_ratio_to_control": ratio,
+                "control_required": control_required,
+                "treatment_required": treatment_required,
+            }
+        )
+
+    total_required = math.ceil(total_scale)
+    group_targets = [math.ceil(total_required * allocation) for allocation in allocations]
+
+    max_available_traffic: float | None = None
+    if "max_available_traffic" in params:
+        max_available_traffic = require_number(params, "max_available_traffic")
+        if not 0.0 < max_available_traffic <= 1.0:
+            raise ValueError("max_available_traffic must be greater than 0 and at most 1")
+
+    traffic_evaluations: list[dict[str, Any]] = []
+    if "experiment_traffic" in params:
+        experiment_traffic = require_number(params, "experiment_traffic")
+        if not 0.0 < experiment_traffic <= 1.0:
+            raise ValueError("experiment_traffic must be greater than 0 and at most 1")
+        if (
+            max_available_traffic is not None
+            and experiment_traffic > max_available_traffic + 1e-12
+        ):
+            raise ValueError("experiment_traffic exceeds max_available_traffic")
+        traffic_selection_mode = "provided"
+        effective_daily_units, raw_duration_days, duration_days = calculate_duration(
+            total_required,
+            daily_eligible_units,
+            experiment_traffic,
+            min_runtime_days,
+            align_to_full_weeks,
+        )
+        traffic_evaluations.append(
+            {
+                "experiment_traffic": experiment_traffic,
+                "effective_daily_units": effective_daily_units,
+                "raw_duration_days": raw_duration_days,
+                "recommended_duration_days": duration_days,
+                "feasible_within_max_runtime": duration_days <= max_runtime_days,
+            }
+        )
+    else:
+        if max_available_traffic is None:
+            raise ValueError(
+                "provide experiment_traffic or max_available_traffic for traffic recommendation"
+            )
+        traffic_selection_mode = "recommended"
+        selected: dict[str, Any] | None = None
+        for candidate in traffic_candidates(params, max_available_traffic):
+            effective, raw_days, adjusted_days = calculate_duration(
+                total_required,
+                daily_eligible_units,
+                candidate,
+                min_runtime_days,
+                align_to_full_weeks,
+            )
+            evaluation = {
+                "experiment_traffic": candidate,
+                "effective_daily_units": effective,
+                "raw_duration_days": raw_days,
+                "recommended_duration_days": adjusted_days,
+                "feasible_within_max_runtime": adjusted_days <= max_runtime_days,
+            }
+            traffic_evaluations.append(evaluation)
+            if selected is None and evaluation["feasible_within_max_runtime"]:
+                selected = evaluation
+        if selected is None:
+            selected = traffic_evaluations[-1]
+        experiment_traffic = float(selected["experiment_traffic"])
+        effective_daily_units = float(selected["effective_daily_units"])
+        raw_duration_days = int(selected["raw_duration_days"])
+        duration_days = int(selected["recommended_duration_days"])
+
+    feasible_within_max = duration_days <= max_runtime_days
+    if not feasible_within_max:
+        warnings.append(
+            f"Required duration ({duration_days} days) exceeds max_runtime_days ({max_runtime_days})."
+        )
+    if raw_duration_days < min_runtime_days:
+        warnings.append(
+            "The statistical target is reached before the minimum runtime; keep the full minimum period."
+        )
+    if number_of_comparisons > 1:
+        warnings.append(
+            "Planning uses Bonferroni correction, while the current native report labels significance at unadjusted p < 0.05 with a 95% confidence interval. Use alpha_per_comparison for the preregistered final decision."
+        )
+
+    return {
+        "ok": True,
+        "data": {
+            "metric_type": metric_type,
+            "baseline": baseline,
+            "mde": mde,
+            "mde_type": mde_type,
+            "effect_direction": effect_direction,
+            "absolute_effect": absolute_effect,
+            "signed_effect": signed_effect,
+            "treatment_value": treatment_value,
+            "confidence_level": confidence,
+            "power": power,
+            "power_source": power_source,
+            "alpha": alpha,
+            "family_alpha": alpha,
+            "alpha_per_comparison": alpha_per_comparison,
+            "multiple_comparison_method": multiple_comparison_method,
+            "number_of_treatment_comparisons": number_of_comparisons,
+            "allocations": allocations,
+            "group_sample_targets": group_targets,
+            "total_sample_target": total_required,
+            "pair_requirements": pair_requirements,
+            "daily_eligible_units": daily_eligible_units,
+            "experiment_traffic": experiment_traffic,
+            "selected_experiment_traffic": experiment_traffic,
+            "recommended_experiment_traffic": (
+                experiment_traffic
+                if traffic_selection_mode == "recommended" and feasible_within_max
+                else None
+            ),
+            "traffic_selection_mode": traffic_selection_mode,
+            "max_available_traffic": max_available_traffic,
+            "traffic_candidates_evaluated": traffic_evaluations,
+            "effective_daily_units": effective_daily_units,
+            "raw_duration_days": raw_duration_days,
+            "recommended_duration_days": duration_days,
+            "aligned_to_full_weeks": align_to_full_weeks,
+            "feasible_within_max_runtime": feasible_within_max,
+            "warnings": warnings,
+        },
+    }
+
+
+def main() -> int:
+    try:
+        if len(sys.argv) != 2:
+            raise ValueError("usage: calculate_experiment_plan.py '<json-object>'")
+        raw = sys.argv[1]
+        params = json.load(sys.stdin) if raw == "-" else json.loads(raw)
+        if not isinstance(params, dict):
+            raise ValueError("input must be a JSON object")
+        print(json.dumps(calculate(params), ensure_ascii=False, indent=2))
+        return 0
+    except (ValueError, KeyError, json.JSONDecodeError) as error:
+        print(
+            json.dumps(
+                {
+                    "ok": False,
+                    "error": {
+                        "type": "validation",
+                        "code": "INVALID_INPUT",
+                        "message": str(error),
+                    },
+                },
+                ensure_ascii=False,
+                indent=2,
+            )
+        )
+        return 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/skills/ae-experiment-insight/SKILL.md

@@ -0,0 +1,149 @@
+---
+name: ae-experiment-insight
+description: "Diagnose and interpret AE/TE A/B experiments from configuration and report evidence through a defensible decision. Use when the user asks what an experiment means, whether it can roll out, why a result is not significant, why group sizes or exposure are wrong, why treatment results conflict, whether the report is trustworthy, or what to do next. Covers SRM, duration sufficiency, novelty effects, metric conflicts, missing or anomalous data, design reasonableness, data reliability, metric interpretation, trend and segment analysis, root-cause hypotheses, and rollout recommendations. All platform discovery and reads must use ae-cli."
+---
+
+# AE Experiment Insight and Diagnosis
+
+Validate the evidence before interpreting the effect. Produce a decision only when the experiment is trustworthy enough to support one.
+
+## Platform and reference routing
+
+Use `ae-cli` for every AE/TE platform interaction and follow
+[`references/platform-operations.md`](references/platform-operations.md);
+never substitute another platform access path.
+
+For diagnosis, read
+[`references/diagnostic-playbook.md`](references/diagnostic-playbook.md).
+Before a rollout decision, read
+[`references/decision-framework.md`](references/decision-framework.md).
+
+## Workflow
+
+### 1. Restore experiment context
+
+Resolve the exact project and experiment. Collect:
+
+- hypothesis and intended decision;
+- control and treatment definitions;
+- assignment unit, layer, targeting, traffic, and group allocation;
+- Feature and group values;
+- primary, secondary, and diagnostic metrics;
+- planned MDE, alpha, power, duration, and stopping rule;
+- actual start/end time and material configuration changes.
+
+Without the original hypothesis or success rule, explain observed effects but do not retroactively invent success criteria.
+
+### 2. Check run integrity
+
+Establish the experiment's actual state, complete runtime periods, and material
+mid-run Feature, traffic, audience, metric, or identity changes. Treat
+configuration history as evidence. Evaluate allocation, exposure, sample
+sufficiency, and design quality through the diagnostic playbook rather than
+duplicating its rules here.
+
+### 3. Run data diagnostics
+
+Apply `diagnostic-playbook.md` in causal order and complete its diagnostic
+checklist. Use
+[`scripts/analyze_experiment.py`](scripts/analyze_experiment.py) for SRM and
+group comparisons; do not calculate p-values or confidence intervals mentally.
+
+### 4. Interpret effects
+
+For each pre-registered metric, interpret the verified metric contract from
+the diagnostic playbook and report:
+
+- verified source event, aggregation, analysis unit, denominator or population,
+  filters, and attribution window;
+- control and treatment values;
+- absolute and relative difference;
+- confidence interval and p-value from deterministic calculation or the verified platform report;
+- achieved sample versus planned sample;
+- relation to MDE and business threshold;
+- statistical and practical significance.
+
+Distinguish:
+
+- statistically significant and practically valuable;
+- statistically significant but too small to matter;
+- directionally positive but underpowered;
+- no detectable effect within the current precision;
+- significantly negative;
+- invalid or inconclusive because of data quality.
+
+Do not translate `p > alpha` into “no effect.” State that the current data did not establish an effect and describe the compatible interval.
+
+### 5. Analyze trends and segments
+
+Check trend stability and abnormal dates using complete comparable periods.
+Apply the trend and segment rules in `decision-framework.md`.
+
+### 6. Diagnose causes
+
+For every suspected cause, provide:
+
+- observed evidence;
+- inference and uncertainty;
+- competing explanation;
+- the exact platform query or product check that would distinguish them;
+- remediation if confirmed.
+
+Do not produce a list of generic causes detached from evidence.
+In the final report, express the distinguishing check as a concise user-facing
+verification action. Include the raw `ae-cli` command only when the user
+explicitly asks for commands, an audit trail, or debugging details.
+
+### 7. Make the decision
+
+Apply `decision-framework.md` and return one decision supported by the trust
+assessment, primary effect, duration, trends, segments, and diagnostic results.
+
+Never start, pause, end, change traffic, delete, or roll out an experiment unless the user separately asks for that platform action. Those actions are outside interpretation and require explicit target-and-impact confirmation.
+
+## Output requirements
+
+- Use the explicitly requested language, otherwise the language of the user's latest substantive message. Localize all user-facing prose, headings, labels, statuses, conclusions, warnings, limitations, and next actions; treat section names in this Skill as semantic guidance and remove unintended mixed-language output.
+- Keep code, commands, raw IDs, event/property/metric names, Feature keys, SDK/API names, and official enum values unchanged when translation would alter their technical meaning.
+- Lead with the decision status.
+- Use an adaptive report, not a fixed numbered template. Organize the default
+  user-facing hierarchy as: conclusion, core impact, experiment decision, and
+  product optimization or next action.
+- Keep data reliability, core metrics, and attention metrics inside the
+  **core impact** section:
+  - **Data reliability validation**: summarize the diagnostic checklist and
+    state the overall result as reliable, limited, or unreliable. Keep normal
+    checks compact and expand only warnings, failures, or unverified checks.
+  - **Core metrics**: present the primary metric's control and treatment
+    values, absolute and relative effect, interval and p-value, relation to MDE
+    or the business threshold, and business meaning. Use causal wording only
+    when the trust assessment supports it.
+  - **Attention metrics**: present secondary and diagnostic metrics only when
+    they affect the decision, explain the mechanism, or reveal material risk.
+    Do not promote a post-hoc metric into the success criterion.
+- For a brief answer, collapse these subsections into compact paragraphs under
+  core impact; do not promote them into unrelated top-level sections.
+- Treat labels such as `material blockers`, `trust blockers`, diagnostic
+  severity names, and framework labels such as `Gate 1` or `Gate 2` as
+  internal taxonomy. Never expose them as report headings or append their
+  English forms after localized headings. Use natural
+  user-facing wording such as “数据可靠性验证” or “当前为什么不能下结论” when
+  those concepts need a heading.
+- Place metric trends and metric-related segments under the relevant core or
+  attention metric. Add broader context, cause diagnosis, assumptions, or
+  limitations only when they materially change interpretation.
+- Use a product-optimization section only when trustworthy evidence supports a
+  specific product change. Otherwise use a next-validation or next-action
+  section.
+- Include only content supported by the request and evidence. Do not add a section merely because it appeared in a previous report.
+- Separate platform observations, deterministic calculations, and inferred causes. State uncertainty and evidence gaps explicitly.
+- Do not return an unexplained raw table.
+
+## Failure behavior
+
+- Ambiguous project or experiment: show resolved candidates and ask; do not guess.
+- Missing report capability: preserve the `ae-cli` gap and request a platform export with required fields.
+- Permission failure: stop dependent queries and report the missing permission.
+- Empty successful report: say no matching data; do not relabel it as a transport failure.
+- Partial success: use available evidence and enumerate failures.
+- Conflicting sources: prefer raw assignment/exposure evidence for data-quality diagnosis and explain the discrepancy.

package/skills/ae-experiment-insight/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Experiment Insight & Diagnosis"
+  short_description: "Validate AE experiments and recommend the next decision"
+  default_prompt: "Use $ae-experiment-insight to diagnose an AE experiment and recommend the next decision."