@thinkingai/ae-cli 6.1.17 → 6.1.18

package/skills/ae-data-integration/references/tracking-plan.md

@@ -10,15 +10,15 @@ Generate and confirm the event/property plan **before** any transform or upload.
 ## Sub-steps
 
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
-2. **Column → property draft** — identify system columns (time, `distinct_id`/`account_id`, event-name column, user-property-name column); map the remaining columns to event / user / common properties. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
-3. **Confirmation gate (single, one pass)** — present one summary table: event list + property list (uncertain types highlighted) + field scope (default: plan fields only, with a full-import switch) + unrecognized/dirty-data handling. The user answers once with ok or edits (renames, types, add/drop columns).
+2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
+3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
 4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
-5. **Persist the plan** — draft.json → xlsx → upload with `sdk_integration_mode=none`.
+5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 
 ## CLI
 
-`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out <draft.json>] [--dry-run]` converts the confirmed `ae-local-data-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
+`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out .ae-cli/data-integration/draft.json] [--dry-run]` converts the confirmed `ae-data-integration-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
 
 - `user_set` mode → no events; every mapping property becomes a user property.
 - `track` mode → one event per `--event-name` (or the mapping `default_event_name`); every property becomes an event property.
@@ -26,9 +26,11 @@ Generate and confirm the event/property plan **before** any transform or upload.
 - `exclude_columns` are dropped from the draft.
 - `desc` and `event_tag` flow from the mapping: each property's `desc` (falling back to its source column name) and each event's `event_meta.<name>.desc` / `event_meta.<name>.tag` (falling back to the source event name) are written into the draft. Fill them in the mapping so the plan is never empty.
 - The per-row event-name column cannot be enumerated without a full scan, so when `default_event_name` is absent the CLI requires `--event-name` for each concrete event name.
+- Duplicate `display_name`s across the property pool are auto-deduplicated (the property name is appended) before validation, so same-named source columns under different object paths never collide. Dotted `target`s default each sub-property's `display_name`/`desc` to its leaf segment, which also avoids collisions.
+- The draft is validated locally before any upload: an `object`/`array_row` property with no `parent.child` sub-property, or a sub-property whose type is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` (`--dry-run` fails too). Fix it in the mapping before running `plan`: a mapping `properties` entry with a dotted `target` (`user_info.name`) declares a `parent.child` sub-property whose parent is the `object`/`array_row` entry with the same first segment — so a kept whole container must carry one dotted entry per child (each scalar or `list`) next to the parent entry. A nested object/object-array that cannot stay nested must instead be flattened to scalar leaves: record one `flatten_rules` entry per leaf and one flat scalar (or `list`) `properties` entry per leaf (see [source-inspect.md](source-inspect.md)).
 - `--dry-run` previews events and properties without writing `draft.json`.
 
-Afterward reuse `ae-cli tracking plan draft --in draft.json --out draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
+Afterward reuse `ae-cli tracking plan draft --in .ae-cli/data-integration/draft.json --out .ae-cli/data-integration/draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
 
 ## Reuse
 

package/skills/ae-data-integration/references/transform.md

@@ -1,6 +1,6 @@
 # Transform — column → UE mapping
 
-Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first.
+Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first. The key system fields (steps 1–5 below) were already confirmed during the tracking-plan step (tracking-plan.md step 2); if nothing changed, state the confirmed values once and move to the property set — never skip a confirmation the user has not actually given.
 
 Read [UE mapping](ue-mapping.md) and apply these gates:
 
@@ -12,10 +12,10 @@ Read [UE mapping](ue-mapping.md) and apply these gates:
 Confirm the system fields with the user before touching properties. These are the top-level `#` fields, and users may not understand them, so explain each in plain language before asking; never infer a decision from a column name alone.
 
 1. **`#type` / mode.** Confirm the recommendation's `mode` (`track` / `user_set` / `mixed`). If `record_type_field` is set, confirm each distinct value normalizes to one of the eight record types. Ask whether the data reports events (→ `track`) or modifies user profiles (→ `user_set` or another profile type).
-2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one.
+2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one. `identity_candidates` is name-matched only, so it misses identity columns with arbitrary names (`玩家ID`, `用户账号`, `player_name`); surface any column whose shape is identifier-like (high `unique_ratio`, low `missing_ratio`, a string or numeric key, not a time column) and ask the user what it is. When one file has multiple sheets/data-sets, cross-check identity columns across them before finalizing: a column that appears (or near-matches, e.g. `设备ID` vs `主设备ID`) in more than one sheet likely has the same business meaning, so present a per-sheet `#account_id`/`#distinct_id` view and ask whether the same column maps to the same system field everywhere — a device ID used as `#distinct_id` in the event sheet but left as a differently-named plain property in the profile sheet silently breaks anonymous-user linkage, so surface the mismatch and let the user decide rather than letting it pass silently.
 3. **`#time`.** Confirm the time column and the source timezone (an IANA name from the user/project context). Leave `time_format` unset unless auto-detection failed (US/EU ambiguous dates). Also confirm the data's `#zone_offset` — the whole-hour UTC offset AE applies to those times, emitted inside `properties` (not top-level). The property's value is a number, so ask the user in numeric terms ("is the offset 8, i.e. UTC+8?") and set `zone_offset_value` to that integer; when a column carries the offset per row, set `zone_offset_field` instead. The two are mutually exclusive.
 4. **`#event_name` (track only).** Confirm the event-name column, or a reviewed `default_event_name` when no column exists.
-5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip.
+5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip. `#ip` is event data only and must be a valid IPv4/IPv6 address (a private/LAN IP is kept but reported — AE cannot geolocate it); `#uuid` must be a standard 36-character UUID. A value that violates the spec is dropped from that row only (`INVALID_IP` / `INVALID_UUID`) — the row itself is kept. The program never auto-generates a `#uuid`.
 
 When an `#event_name`, `#account_id`, or `#distinct_id` column's values do not satisfy AE naming rules (pure Chinese, uppercase, spaces), do not stop — scan the distinct values, list them to the user, and ask for one AE-name replacement each; record the pairs in `value_mapping` (see [UE mapping](ue-mapping.md)). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded. The same mechanism applies to a property column via that entry's own `value_mapping`.
 
@@ -24,18 +24,18 @@ Then confirm the property set with the user before saving the mapping. Present e
 Walk the table item by item:
 
 1. **Auto-renamed targets.** Recommendation sanitizes column names (camelCase split, illegal chars → `_`, digit-leading → `field_`, nothing recognizable → `field_N`). Show each source → target; ask for a manual name for every `field_N` fallback and rewrite the `target`.
-2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`). ID-like columns stay `string` even when numeric; `object`/`list` columns carry `transform: 'json'`. A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
+2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`/`array_row`). ID-like columns stay `string` even when numeric; `object`/`array_row`/`list` columns carry `transform: 'json'`. A scalar array is `list` (→ AE `array_string`); an object array is `array_row` (→ AE `array_row`). A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
 3. **Exclusions.** Ask which columns to drop and record them as `exclude_columns`.
 
 Before saving, present the complete mapping for a final sign-off on one grouped page — system fields (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#zone_offset`, `#event_name`, `#ip`/`#uuid`), then the property table (source column → target AE name → type), then excluded columns — and wait for an explicit yes. Never dump the raw mapping JSON at the user.
 
-Save the reviewed `ae-local-data-mapping/v1` JSON locally. Convert into a new output directory:
+Save the reviewed `ae-data-integration-mapping/v1` JSON to `.ae-cli/data-integration/mapping.json` (project workspace). Convert into a new output directory:
 
 ```bash
 ae-cli data-integration convert \
   --input-file '<path>' \
-  --mapping '<mapping.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --mapping '.ae-cli/data-integration/mapping.json' \
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
 For multiple files, repeat `--input-file` and pass a wildcard mapping plus `--type-resolutions` when inspect reported conflicts:
@@ -45,10 +45,10 @@ ae-cli data-integration convert \
   --input-file '<a.csv>' --input-file '<b.csv>' \
   --mapping '<wildcard-mapping.json>' \
   --type-resolutions '<resolutions.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
-The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine.
+The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason. If `manifest.output.skipped_fields` is present, tell the user how many `#ip`/`#uuid` values were invalid and dropped (the rows were otherwise kept); if `manifest.output.lan_ip_records` is present, tell the user that many `#ip` values are private/LAN addresses that AE cannot geolocate. Neither blocks the manifest. If `manifest.output.flatten_misses` is present — or a stderr `Warning: flatten rule "X" did not materialize for N row(s).` fires — a `flatten_rules` path missed some rows: re-check the dot path against the source shape (or confirm the column is legitimately optional); the rows are otherwise kept. A stderr `Warning: … column count different from the header row …` means ragged CSV/TSV rows were tolerated (extra fields dropped, missing fields treated as empty) — surface it as a data-quality note. A blocked manifest whose reason is `The source contained no data rows.` means the file had zero data rows; do not re-run the same command on it. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine. For the full failure taxonomy and how to respond, see [error handling](error-handling.md).
 
 ## Re-report only the failed rows (salvage loop)
 
@@ -59,7 +59,7 @@ ae-cli data-integration convert \
   --input-file '<same-source>' \
   --mapping '<fixed-mapping.json>' \
   --salvage-from '<run-dir>/invalid.rows.jsonl' \
-  --output-dir '.ae-cli/data-integration/<run-id>-salvage'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>-salvage'
 ```
 
 This emits a `valid.ue.jsonl` containing only the rows that now pass, plus a new `invalid.rows.jsonl` with whatever still fails. Fixes are rarely one-shot, so repeat: feed each round's `invalid.rows.jsonl` into the next `--salvage-from` until no rows fail or the user stops. Each round's `valid.ue.jsonl` is disjoint from earlier rounds', so upload each round independently (same confirmation and `--allow-clean-subset` gates as a normal upload). `--salvage-from` is single-file only and the source must be the same file that produced the quarantine.

package/skills/ae-data-integration/references/ue-mapping.md

@@ -1,12 +1,12 @@
 # UE mapping contract
 
-The mapping version is `ae-local-data-mapping/v1`.
+The mapping version is `ae-data-integration-mapping/v1`.
 
 Required structure:
 
 ```json
 {
-  "version": "ae-local-data-mapping/v1",
+  "version": "ae-data-integration-mapping/v1",
   "source": {
     "sha256": "<source-sha256>",
     "format": "csv",
@@ -54,10 +54,25 @@ Top-level `#` fields map from named source columns. The user confirms each mappi
 | `#account_id` | Login/account ID (database `user_id`, phone, member ID) — identifies authenticated users | At least one of `#distinct_id` / `#account_id` |
 | `#time` | Event/profile occurrence time; AE bins data by it | Yes |
 | `#event_name` | What the user did (`purchase`, `login`) | Only for `track` |
-| `#ip` | Client IP; AE resolves geo from it | No |
-| `#uuid` | Short-window deduplication ID | No |
+| `#ip` | Client IP; AE resolves geo from it. Event data only | No |
+| `#uuid` | Standard 36-character UUID uniquely identifying the record (dedup). Both event and user data | No |
 
-`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined.
+`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined. `#zone_offset` is event data only — user profile records never carry it.
+
+## Value specifications
+
+Each mapped system field carries a value spec, enforced at both inspect (warnings) and convert (row handling). A row error quarantines the whole row; a field skip drops only that field and keeps the row.
+
+| Field | Value spec | On violation (convert) |
+| --- | --- | --- |
+| `#account_id` / `#distinct_id` | Non-empty string, at most 128 characters | Row error `MISSING_USER_ID` (absent) / `USER_ID_TOO_LONG` (>128) |
+| `#event_name` | `^[a-z][a-z0-9_]{0,49}$` (lowercase snake_case, letter-leading, ≤50 chars) | Row error `INVALID_EVENT_NAME` |
+| `#time` | One of the supported formats, within 3 years back / 3 days forward | Row error `INVALID_TIME` / `TIME_OUT_OF_RANGE` |
+| `#ip` | Valid IPv4 or IPv6. Event data only | Field skip `INVALID_IP`; a private/LAN IP is kept and reported — AE cannot geolocate it |
+| `#zone_offset` | Integer -12..14 (or an IANA name for `zone_offset_value`). Event data only | Row error `INVALID_ZONE_OFFSET` |
+| `#uuid` | Standard 36-character UUID (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`). Both data kinds | Field skip `INVALID_UUID` |
+
+A field skip is not a row failure: the row is kept with its other fields, and the count is reported in `manifest.output.skipped_fields` so the agent tells the user at the end. `#uuid` is never auto-generated — it comes only from a mapped source column.
 
 Mapping keys: `account_id_field`, `distinct_id_field`, `time.field`, `event_name_field` (or a reviewed `default_event_name`), `ip_field`, `uuid_field`. Inspect surfaces every identity-shaped column in `identity_candidates` (name, `account`/`distinct` kind, unique and missing ratios) so the agent can present all candidates for confirmation. When the required identity column is absent, use an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool` — a user decision, never invented.
 
@@ -86,16 +101,20 @@ All fields below are optional and are explicit user decisions — never invent t
 | `account_id_value` / `distinct_id_value` | `string` | Fixed placeholder identity (≤128 chars). Applies to every row when the corresponding `*_field` is absent; when the field exists, fills only the rows whose column is empty |
 | `random_pool` | `{ account_ids?: string[], distinct_ids?: string[] }` | Synthesizes a random identity when the source field is absent |
 | `exclude_columns` | `string[]` | Source columns skipped when building properties |
-| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it |
+| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV/Excel paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it. `flatten_rules` only materializes the out column in the row; it does **not** emit it — add a `properties` entry with `source` = the out column to actually produce the property. An out column that fails to materialize for some rows (missing path, non-JSON cell, or a path without a `<column>.` prefix) is counted per out-column in `manifest.output.flatten_misses` and named in a convert stderr warning — fix the path or confirm the column is legitimately optional |
 | `headers` | `string[]` | User-confirmed column names for a headerless file; presence means the first row is data. Never use inspect's `col_1..col_N` placeholders — infer names from each column's values, confirm them with the user, then write them here |
 | `missing_time` | `'now'` | Fill a missing/empty `#time` with the current time, for user-profile rows only (explicit user decision; track rows are never filled) |
-| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo) |
-| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (short-window deduplication ID) |
+| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo). Event data only; a value that is not valid IPv4/IPv6 is dropped from that row, and a private/LAN IP is kept but reported |
+| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (standard 36-character UUID, both data kinds). A non-UUID value is dropped from that row |
 | `zone_offset_value` | `number` (integer -12..14) or IANA `string` | Emits the fixed `#zone_offset` preset property inside `properties`. An IANA name resolves to its integer UTC offset |
 | `zone_offset_field` | `string` | Source column whose per-row integer value (-12..14) is emitted as `#zone_offset`; missing/non-integer rows are quarantined. Mutually exclusive with `zone_offset_value` |
 | `event_meta` | `{ <event-name>: { desc?: string, tag?: string } }` | Per-event business description and `event_tag` for the tracking plan, keyed by AE event name. Inferred from the data and user context; the user supplies anything not inferable — never leave them empty |
 
-Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`list`) whose values arrive as JSON text — JSON-encoded CSV cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`array_row`/`list`) whose values arrive as JSON text — JSON-encoded CSV/TSV/Excel cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+
+A `properties` entry whose `target` is dotted (`parent.child`) is a plan-only sub-property declaration: `parent` must name an `object`/`array_row` entry in the same mapping, the child must be scalar or `list`, and the child's `source` must equal the parent's `source` — the nested value lives inside the parent, which conversion emits once and never reads the child's column. A kept whole `object`/`array_row` therefore carries one dotted entry per child alongside the parent entry.
+
+Property types are `string`, `number`, `boolean`, `datetime`, `list`, `object`, and `array_row`. `list` is a scalar array (`["a","b"]`, → AE `array_string`); `object` is a one-to-one object (`{...}`); `array_row` is an object array (`[{...}]`, one-to-many). A mapping expresses an object's/object array's sub-properties as dotted `target`s (`user_info.name`, `orders.sku`); `data-integration plan` maps them one-to-one to `parent.child` plan properties (each child's `display_name`/`desc` default to its leaf segment). In the tracking plan, `object` and `array_row` properties must each have at least one `parent.child` sub-property, and every sub-property must itself be scalar or `array_string` — a nested object/object-array cannot stay nested and must be flattened further to scalar leaves. `list` (→ `array_string`) is a leaf and takes no children. `ae-cli data-integration plan` enforces this locally: a composite property with no children, or a child that is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` before any upload.
 
 ## Time formats
 
@@ -130,13 +149,16 @@ Values with an explicit offset or `Z` are parsed by the JavaScript `Date` constr
 - `#zone_offset`, when set, is a whole-hour integer in -12..14 (or an IANA name/column that resolves to one) and is emitted inside `properties`, never at the top level.
 - Event/property names are lowercase snake_case, begin with a letter, and are at most 50 characters.
 - Target property names are unique and do not collide with UE system fields.
-- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, or `object`.
+- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, `object`, or `array_row`.
 - Text is at most 2 KB; numbers stay within -9E15..9E15.
 - Lists contain at most 500 strings (255 bytes each) or 500 objects.
 - Objects contain at most 100 legal sub-properties; nested values follow the same type limits.
+- No `object`/`array_row` survives into the tracking plan: AE requires each to have a `parent.child` sub-property (scalar or `array_string` only), and a mapping cannot express one, so nested objects/object-arrays are flattened to scalar leaves first (a kept-whole container fails `data-integration plan` fast).
 - A conversion rule does not hide a real type conflict.
 - Event/profile times fall within the receiver window: previous 3 years through next 3 days.
-- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`).
+- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`). `#uuid` is never auto-generated.
+- `#uuid` values are standard 36-character UUIDs and `#ip` values are valid IPv4/IPv6; invalid ones are dropped from the row only (`skipped_fields`), never quarantined.
+- `#ip` and `#zone_offset` appear on `track` records only; user data carries neither.
 - `value_mapping`, `random_pool`, and fixed `account_id_value`/`distinct_id_value` came from an explicit user decision and match the actual distinct values/columns.
 
 `user_set` output for the same identity is ordered by time so receiver application order is deterministic. Conversion applies whole-row quarantine: any error on a row — identity, time, event, record type, or a single property (type coercion, size/limit) — drops the entire row. The row is written to `invalid.rows.jsonl` with its error codes, counted in `manifest.output.invalid_records`, and the manifest is blocked until reviewed. The failed rows are re-reportable without re-sending valid rows: fix the mapping, then run `convert --input-file <same-source> --mapping <fixed-mapping> --salvage-from <invalid.rows.jsonl>`. The salvage run re-processes only the listed row numbers against the same source, emits a `valid.ue.jsonl` containing only the newly fixed rows, and writes a new `invalid.rows.jsonl` with whatever still fails — so the loop repeats (feeding each round's `invalid.rows.jsonl` into the next `--salvage-from`) until no rows fail or the user stops.

package/skills/ae-engage/references/build-task-save-guide.md

@@ -229,6 +229,8 @@ When experiment mode is enabled (`context.enableExp=true` or draft `expConfig.en
 - capability `engage-task.task.build-save-guide` enriches `handoff.reqTemplate.channelConfig.groupContentList`
   so each entry carries `expGroupName`, `expGroupType`, `percentageInExperiment`, `order`, and `contentList`
 - do not drop those association fields when filling content; they must stay aligned with `expConfig.expGroupList`
+- copy the complete group tuple (`expGroupName`, `expGroupType`, `percentageInExperiment`, `order`)
+  into both lists; matching only by list position is not sufficient
 
 ### 4.8 `fieldRules`
 
@@ -300,6 +302,11 @@ Apply this rule only to
 `completionIndicatorDef.completionIndicators[].eventDefinition.filters`. Trigger-event filters have
 their own scenario rules and are not subject to this completion-filter restriction.
 
+For `completionIndicatorType=0`, also read and preserve `requiredMainGoalFields` and
+`touchCycleRule`. A valid main goal includes `touch_cycle_num` and `touch_cycle_num_unit`; use `1`
+and `day` when no custom completion window is requested. Do not rely on static `--validate` alone
+because the save service performs this additional business validation.
+
 ### 4.9 `handoff`
 
 This is the final section before `save_task`.
@@ -336,6 +343,8 @@ Recommended usage pattern:
    definition directly
 7. omit `clientConfig.clientQp`; partial updates preserve the server-authored value
 8. call `engage-task task save`
+9. after an experiment save succeeds, call `engage-task task get` and verify that both
+   `exp_config.exp_group_list` and `group_content_list` contain the expected group tuples
 
 ---
 

package/skills/ae-engage/references/save-task.md

@@ -426,6 +426,11 @@ When the guide points to event-based completion or experiment-driven main-goal r
 Read `fieldRules.blocks.controlConfig.completionIndicatorDef.filterPropertySelectTypes` and exclude
 every property type listed under `excluded` before constructing its `filters`.
 
+For a main goal (`completionIndicatorType=0`), always include `touch_cycle_num` and
+`touch_cycle_num_unit`. If the user does not request another completion window, use
+`touch_cycle_num=1` and `touch_cycle_num_unit="day"`. Static Capability validation may accept a
+main goal without these fields, but the Hermes save service rejects it.
+
 Important constraints that still apply:
 
 - `doNotDisturb.enableDoNotDisturb=true` requires `startTime` and `endTime` in `HH:mm`
@@ -444,6 +449,81 @@ compatibility field, and partial task updates preserve the existing server value
 client-side condition has no semantic field in the current contract, stop and report that it cannot
 be safely authored through this Capability.
 
+### 4.7 Verified A/B Task Template
+
+Use this shape as the starting point for a server-channel split experiment. Replace every
+placeholder with metadata discovered from the current project.
+
+```json
+{
+  "baseInfo": {
+    "taskName": "<taskName>",
+    "taskDesc": "<taskDescription>",
+    "tzOffset": "<projectTimezoneOrDefaultSentinel>"
+  },
+  "channelConfig": {
+    "channelType": 1,
+    "channelId": "<verifiedChannelId>",
+    "groupContentList": [
+      {
+        "expGroupName": "Control",
+        "expGroupType": 1,
+        "percentageInExperiment": 50,
+        "order": 0,
+        "contentList": [{ "pushLanguageCode": "default", "content": "<channelContentJsonString>" }]
+      },
+      {
+        "expGroupName": "Experiment A",
+        "expGroupType": 2,
+        "percentageInExperiment": 50,
+        "order": 1,
+        "contentList": [{ "pushLanguageCode": "default", "content": "<channelContentJsonString>" }]
+      }
+    ]
+  },
+  "targetConfig": {
+    "targetClusterType": 1,
+    "definitionRequest": "<semanticAudienceDefinition>"
+  },
+  "triggerConfig": { "triggerType": 2 },
+  "controlConfig": {
+    "completionIndicatorDef": {
+      "completionIndicators": [
+        {
+          "completionIndicatorType": 0,
+          "touch_cycle_num": 1,
+          "touch_cycle_num_unit": "day",
+          "eventDefinition": {
+            "type": "event",
+            "event": "<verifiedGoalEvent>",
+            "aggregation": "count",
+            "operator": "gte",
+            "value": 1
+          }
+        }
+      ]
+    },
+    "frequencyLimits": { "enableFrequencyLimits": false, "ruleList": [] }
+  },
+  "expConfig": {
+    "enableExp": true,
+    "expType": 1,
+    "percentageInLayer": 100,
+    "expIndicatorBizType": 1,
+    "controlGroupSkipPush": false,
+    "expGroupList": [
+      { "expGroupName": "Control", "expGroupType": 1, "percentageInExperiment": 50, "order": 0 },
+      { "expGroupName": "Experiment A", "expGroupType": 2, "percentageInExperiment": 50, "order": 1 }
+    ]
+  }
+}
+```
+
+The experiment group tuple (`expGroupName`, `expGroupType`, `percentageInExperiment`, `order`)
+must be identical between each `expConfig.expGroupList` entry and its corresponding
+`channelConfig.groupContentList` entry. Validate the final request, save it, then call `task get`
+and verify both persisted lists; a successful task ID alone is not sufficient verification.
+
 ---
 
 ## 5. Final Self-Check Before `engage-task task save`
@@ -461,6 +541,8 @@ Before submission, verify:
 9. No unsupported `triggerType=6` is used.
 10. Ordered steps contain `eventDefinition` and sequence metadata, not persisted aggregate fields.
 11. No placeholder IDs or fabricated resource names remain in the request.
+12. Every main goal includes `touch_cycle_num` and `touch_cycle_num_unit`.
+13. For experiments, `expGroupList` and `groupContentList` contain identical group tuples.
 
 ---
 

package/skills/ae-experiment/SKILL.md

@@ -56,6 +56,7 @@ Naming and response boundary:
    - `experiment feature save`
    - `experiment metric save`
 3. Create or patch the experiment draft with `experiment experiment save`.
+   Use `experiment experiment update-metrics` when replacing metric bindings or assigning guardrail roles.
 4. Check readiness with `experiment experiment ready-check`.
 5. For a non-mutex traffic layer, run `experiment experiment conflict-check` before submit (needs `feature_key_list` from context or `experiment get`).
 6. Move status with `experiment experiment manage`.
@@ -66,6 +67,7 @@ If an experiment save returns `error_code: METRIC_NOT_FOUND`, list metrics for t
 ## Parameter Conventions
 
 - Experiment save payloads distinguish two allocation fields: experiment-level `req.allocation` (**integer only; no decimals**) and group-level `req.groups[].allocation` (**integer only; sum must equal `100` exactly**).
+- Public experiment metric roles are `primary`, `secondary`, and `guardrail`. The internal `observation` role is currently unavailable for saves. Guardrail is a binding role; updating metrics replaces the full binding list.
 
 ```bash
 ae-cli experiment experiment get --project-id 1 --exp-id exp_123
@@ -92,7 +94,9 @@ Read [`save_build_guide.md`](references/save_build_guide.md) and
 
 ### Experiment
 
-`experiment experiment save`, `capability run experiment.experiment.save-submit`, `experiment experiment list`, `experiment experiment list-archived`, `experiment experiment get`, `experiment experiment ready-check`, `experiment experiment conflict-check`, `experiment experiment manage`, `experiment experiment update-group`, `experiment experiment batch-delete`, `experiment operation-log query`
+`experiment experiment save`, `experiment experiment update-metrics`, `capability run experiment.experiment.save-submit`, `experiment experiment list`, `experiment experiment list-archived`, `experiment experiment get`, `experiment experiment ready-check`, `experiment experiment conflict-check`, `experiment experiment manage`, `experiment experiment update-group`, `experiment experiment batch-delete`, `experiment operation-log query`
+
+Read [`manage_guardrail_metrics.md`](references/manage_guardrail_metrics.md) before assigning or replacing metric roles.
 
 ### Traffic Layer and Buckets
 
@@ -104,4 +108,6 @@ Read [`save_build_guide.md`](references/save_build_guide.md) and
 
 ### Metric and Feature
 
-`experiment metric save`, `experiment metric get`, `experiment metric list`, `experiment metric delete`, `experiment feature save`, `experiment feature update-status`, `experiment feature get`, `experiment feature list`, `experiment feature version-list`, `experiment feature operation-log query`, `experiment feature batch-delete`
+`experiment metric save`, `experiment metric get`, `experiment metric list`, `experiment metric delete`, `experiment feature save`, `experiment feature update-status`, `experiment feature get`, `experiment feature list`, `experiment feature version-list`, `experiment feature operation-log query`, `experiment feature batch-delete`, `experiment feature whitelist list`, `experiment feature whitelist save`, `experiment feature whitelist update-status`, `experiment feature whitelist batch-delete`
+
+Read [`manage_feature_whitelist.md`](references/manage_feature_whitelist.md) before querying or changing Feature whitelist rules.

package/skills/ae-experiment/references/manage_feature_whitelist.md

@@ -0,0 +1,66 @@
+# Feature Whitelist Rules
+
+List, create, modify, enable, disable, or delete explicit Feature whitelist rules.
+
+## List
+
+```bash
+ae-cli experiment feature whitelist list \
+  --project-id <id> --feature-key <feature_key>
+```
+
+The result contains only explicit whitelist rules for the Feature. Each item exposes `rule_id`,
+`feature_key`, `priority`, `status`, and a structured snake-case `whitelist` array. Use the returned
+`rule_id` for modification, status changes, or deletion.
+
+## Save
+
+```bash
+ae-cli experiment feature whitelist save \
+  --project-id <id> \
+  --feature-key <feature_key> \
+  --status enable \
+  --whitelist '[{"bucket_id":"#user_id","rules":[{"ids":["u1","u2"],"value":"on"}]}]'
+```
+
+Pass `--rule-id` to modify an existing whitelist rule. Omit it to create a rule. The Hermes
+Capability fixes the Atlas rule type to `targeting` and serializes the supplied buckets into the
+server's explicit whitelist rule configuration.
+
+Rules:
+
+- Resolve the Feature with `experiment feature get` before writing.
+- `bucket_id` is the split subject, such as `#user_id` or `#account_id`.
+- Each bucket contains one or more rows with a non-empty `ids` array and a string `value`.
+- An empty string Feature value is allowed. Bucket IDs cannot be empty or duplicated. IDs must be
+  unique within one bucket, while different buckets may use the same string ID.
+- The server validates each value against the Feature type and limits the total ID count.
+- Only one explicit whitelist rule can be enabled for the same Feature.
+- Enabling, modifying, disabling, or deleting an enabled whitelist rule immediately creates a new
+  version and syncs RCC when the Feature itself is online.
+
+## Status
+
+```bash
+ae-cli experiment feature whitelist update-status \
+  --project-id <id> --rule-id <rule_id> --status enable
+```
+
+Valid status transitions exposed by this command are `enable` and `disable`.
+
+## Delete
+
+```bash
+ae-cli experiment feature whitelist batch-delete \
+  --project-id <id> --rule-ids '["0001"]'
+```
+
+Deletion is high risk and requires confirmation. Enabled whitelist rules may be deleted; Hermes
+resynchronizes affected online Features afterward.
+
+All four commands use Capability Gateway with CLI-token authentication:
+
+- `experiment.feature_whitelist.list`
+- `experiment.feature_whitelist.save`
+- `experiment.feature_whitelist.update_status`
+- `experiment.feature_whitelist.batch_delete`

package/skills/ae-experiment/references/manage_guardrail_metrics.md

@@ -0,0 +1,26 @@
+# experiment experiment update-metrics
+
+Replace the metric bindings of an existing experiment draft and assign metric roles, including
+guardrail metrics.
+
+```bash
+ae-cli experiment experiment update-metrics \
+  --project-id <id> \
+  --exp-id <exp_id> \
+  --metrics '[{"metricId":"conversion","metricRole":"primary"},{"metricId":"error_rate","metricRole":"guardrail"}]'
+```
+
+## Contract
+
+- Discover every `metricId` with `experiment metric list`; never invent metric IDs.
+- `--metrics` must be a non-empty array. Each item requires camelCase `metricId` and `metricRole`.
+- `metricRole` is one of `primary`, `secondary`, or `guardrail`. Do not submit the internal
+  `observation` role; Hermes currently rejects it at save boundaries.
+- This command replaces all saved metric bindings because Hermes treats a non-empty `metrics` list
+  in a draft patch as a replacement. Include bindings that must remain, not only the new guardrail.
+- Guardrail is a binding role, not a separate metric type. Create the underlying metric first with
+  `experiment metric save` when it does not exist.
+- At least one `primary` metric is still required before readiness succeeds.
+
+Run with `--dry-run` first, then verify the persisted roles with `experiment experiment get` and
+run `experiment experiment ready-check`.

package/skills/ae-experiment/references/save_experiment.md

@@ -104,7 +104,7 @@ ae-cli experiment experiment save --project-id 1 --req '{"expId":"exp_123","allo
 - `expCycle.cycleType` can be `day` or `sample`; when `cycleType=day`, `dayNum` is `1..90`. Blank cycle defaults to day + 30.
 - Before readiness, `groups` must be non-empty, contain exactly one control group (`isControl=1`), group allocations must sum to exactly `100`, and each `expGroupValue` must be a non-empty JSON string array.
 - Feature experiments must bind `featureKeyList` before readiness. `featureKeyList` contains Feature key strings, not Feature objects. The MCP currently supports one `featureKey` for feature experiments.
-- Metrics use `metricRole=primary|secondary|guardrail|observation`; before readiness at least one primary metric is required.
+- Metrics use `metricRole=primary|secondary|guardrail`; the internal `observation` role is currently unavailable for saves. Before readiness at least one primary metric is required.
 - `targeting` replaces the saved targeting object when provided.
 - Custom audience QP must be supplied as the semantic object
   `targeting.definitionRequest`. Do not submit the internal `targeting.targetConfig`

package/skills/ae-kb/SKILL.md

@@ -97,7 +97,7 @@ ae-cli kb +url \
   --parse-instruction "Keep headings and code blocks"
 ```
 
-`--url` must be `http(s)`.
+`--url` must be `http(s)`. The server detects the platform from the URL automatically: URLs on a `*.feishu.cn` or `*.larksuite.com` subdomain are parsed with the Feishu pipeline (including sub-documents, using the server's own Feishu parsing instruction — `--parse-instruction` is ignored for them); all other URLs are fetched as regular web pages.
 
 ### Generate Schema and Compile