@thinkingai/ae-cli 6.1.17 → 6.1.19

package/skills/ae-data-integration/references/source-inspect.md

@@ -10,6 +10,8 @@ Confirm:
 
 Accept one or more CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, or XLSX files. CSV/TSV/TXT/JSON/JSONL/XLSX have no hard size limit; files over 1 GB print a stderr warning with an estimated processing time and suggest splitting. XLS over 100 MB prints a memory-risk warning (the legacy parser loads the whole workbook, roughly 5-10x file size); XLS over 1 GB is still rejected — convert it to XLSX or split it first.
 
+XLSX worksheet recognition matches `<sheet>` and namespaced `<x:sheet>` alike: some cleaning/export tools rewrite the default OOXML namespace as a prefix, and the workbook reads the same either way. The worksheet *row* nodes themselves are still read unprefixed, so a workbook whose row data is also namespaced falls back to CSV.
+
 ## Inspect without exposing raw values
 
 Before the full inspection (which streams and profiles the entire file and can take
@@ -39,30 +41,50 @@ If `selection_required=true`, show only the Sheet/JSON Path candidates and ask t
 ae-cli data-integration inspect --input-file '<path>' --data-set '<candidate-id>' --source-timezone '<iana-timezone>'
 ```
 
-Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. Read [UE routing](ue-routing.md) before choosing a branch.
+Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. IP- and UUID-named columns are additionally checked against their value specs: inspect warns how many non-empty values are invalid IPv4/IPv6, private/LAN IPs, or non-UUID strings, so the user can decide whether to map them as `ip_field`/`uuid_field`. Excel columns whose cells carry a date number format infer as `datetime` and are named in a warning (see **Excel date cells** below). Read [UE routing](ue-routing.md) before choosing a branch.
+
+A stderr `Warning: … column count different from the header row …` means the CSV/TSV has ragged rows (extra fields dropped, missing fields treated as empty); report it as a data-quality signal. For how every pipeline failure — abnormal data, parse errors, and program errors — is classified and handled, see [error handling](error-handling.md).
+
+**Value frequency and numeric distribution.** A distinct count says how many different values a column holds, not whether they are worth uploading. Two per-column fields answer that, and both appear in `inspect` output only — never in the convert manifest:
+
+- `value_frequency` — the 10 most frequent values as `{value, count, ratio}`, values truncated the way `samples` are and counted after truncation. It is reported only for a column whose distinct values all fit the tracked budget (200), so the counts are exact and complete when present; a column past the budget reports nothing here and its `unique_count` is the field to read instead. Use it to separate an enum from free text: a `渠道` column with three values is a property worth uploading, and its listed values are also what a `value_mapping` decision is made from. A single value covering every row usually means an export artifact, not data — propose `exclude_columns` and let the user decide.
+- `numeric_summary` — `count`, `min`, `max`, `sum`, `mean`, `p25`, `median`, `p75` for a column that inferred as `number`. `count` is the values that read as numbers, which is below the column's non-missing count when the column is mixed. `count`/`min`/`max`/`sum`/`mean` are always exact; the quantiles come from a bounded sample on large columns and then `quantiles_approximate: true` says so. Use it to check the magnitude before it is locked into an AE property: a mean far below the maximum on a monotonically climbing column is the signature of a cumulative snapshot rather than a per-row measure (see [UE routing](ue-routing.md)), an amount whose values are 100× the expected size is a 分/元 unit mismatch, and a column that is entirely one number carries no signal.
+
+Summarize both — report the shape of the distribution and the names of the values, and do not paste the whole table into the conversation. They are read out of the customer's file like `samples` are.
 
 ## Advanced input
 
 - **Headerless files** — inspect auto-detects a missing header row on CSV/TSV and reports `no_headers: true` with a `header_detection` verdict and `auto_headers` placeholders (`col_1..col_N`); the first row is already treated as data. `--headerless` forces the same behavior without detection. Never keep the `col_1..col_N` placeholders — they carry no business meaning. For each column, read its bounded samples and inferred type and propose a meaningful name, present every proposal to the user (column position, sample summary, suggested name), and let the user confirm or rename each one; record the confirmed names in the mapping's `headers` field. When the user already knows the names, re-run inspect with `--headers 'col1,col2,...'` so the recommended mapping carries them.
+- **Title rows above the header** — an exported report often puts a caption in the first cell (`2026年3月销售明细`) and the real header row underneath. Read as-is, the caption becomes the file's only column name, every real column name is lost, and the header row is counted as a data row. Inspect reports the suspected rows under `leading_title_rows` (row ordinals and non-empty cell counts only, never the cell text) and warns — but it does **not** change what it read: the first row was still used as the header, because Excel exports legitimately carry numeric header rows (`2024`, `2025`) that look like data, and there is no flag that puts a header row back once it has been treated as data. So the report is a question for the user. When they confirm those rows are a title or banner, re-run inspect with `--skip-rows N` (N is exactly the last ordinal listed); the rerun reads the real header row, reports `skipped_rows`, and carries `skip_rows` into the recommended mapping so `convert` reads the same rows inspect profiled. On XLSX, inspect additionally reports `header_signal` when the row it used as the header looks like data — same rule: reported, not applied; resolve it with `--headers` or `--headerless`. This covers CSV/TSV and XLSX; a legacy `.xls` workbook accepts `--skip-rows` but is not scanned for title rows.
+- **Summary and total rows** — an exported report ends with a 合计 row, and a grouped one repeats 小计 after every group. Those rows are not observations: uploaded, each becomes an event that never happened whose amount is the whole group's revenue, and profiled, it doubles the column's `sum` and turns its `max` into the total. Nothing in the row itself says so, so inspect flags them under `summary_rows` and warns. Each entry carries `row` (the data-row ordinal, the same numbering `invalid.rows.jsonl` and `--salvage-from` use) and `signals`: `total_label` when a cell reads as a total label (`合计` / `总计` / `小计` / `汇总` as a prefix, `Total` / `Subtotal` / `Sum` as the whole cell), naming the column in `label_column` but never the cell text; `column_total` when a number on that row equals the total of its column's other rows, listing those columns in `total_columns`. A row can raise one signal or both — a labelled group subtotal holds its group's total, not the column's, so only the label fires. Nothing is removed and nothing is changed: `convert` writes these rows as records, and every number reported for their columns counts them, which is exactly why the finding has to be read. There is no flag that drops a data row, because a row labelled `合计` is sometimes a real business record; when the user confirms a row is a total, ask them to remove it from the source file or re-export without it, then inspect again. `convert` repeats the finding in `manifest.output.summary_rows`, since by upload time a subtotal row with a plausible identity and time is indistinguishable from data. This covers every format, `.xls` included — the check reads rows, not worksheet structure.
+- **Repeated business keys** — a customer re-exports a report whose range overlaps the last export, or pastes two sheets together, and the same observation arrives twice. Uploaded, each repeat is a second event: that user's revenue doubles, every funnel counts them twice, and AE appends accepted events with no way to un-send one — so the only place this is fixable is before the upload. Inspect compares each row's business key against the rows before it and reports repeats under `duplicate_keys`, plus a warning. The report carries `key_columns` (the columns actually compared — always read it, since the key is what the finding means), `checked_rows`, `duplicate_groups`, `extra_rows` (surplus records an upload would carry), and `groups`, each with `count`, the data-row `rows` (the numbering `invalid.rows.jsonl` and `--salvage-from` use), and a `key_hash` prefix that distinguishes groups without revealing values — the key's own text is never reported. `groups_truncated` / `rows_truncated` mean the list is bounded, not that the counts are; `tracking_truncated` means distinct keys outran the scan's budget and there may be more. The key comes from the mapping's identity, time, and event-name columns on `convert`, and from column-name matching on `inspect`; a single column is never a key, so a file with no recognizable time column is not scanned at all (identity alone would call every returning user's second row a repeat). Two limits to state when reporting: values are compared as written, so `2026-03-01 10:00:00` and `2026/03/01 10:00:00` are two different keys and a repeat spelled two ways is missed; and a source's own unique key (an order id) is not compared on unless the mapping names it as identity, time, or event. Nothing is removed — a repeat is sometimes a real pair of records, two order lines in the same checkout second — so ask the user whether the rows are separate observations, and if they are not, have them remove the rows from the source file and inspect again. `convert` repeats the finding in `manifest.output.duplicate_keys`, because by upload time both copies are ordinary valid records; it describes the whole source file even on a `--salvage-from` run, since every valid row of that file ends up in AE. This covers every format, `.xls` included — the check reads rows, not worksheet structure.
 - **TSV / TXT** — `.tsv` and `.tab` use a tab delimiter with no quoting convention; `.txt` and unknown extensions are content-sniffed into CSV, TSV, or NDJSON.
 - **Encoding** — text files are auto-detected (UTF-8, GBK, GB2312, Big5, and others); no flag is needed.
-- **Excel sheets** — `--merge-sheets` streams every worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match.
+- **Excel date cells** — a cell whose number format is a date or date+time is read as the wall-clock timestamp shown in Excel, not as the Excel serial number stored behind it, so the column infers as `datetime` and can serve as the time field. Inspect lists every such column in a warning. Treat that warning as a question to the user, not as a note: the same column profiled as `number` before this behavior existed, so if any part of this file was already sent to AE, the property may have been received as a number and its type is now locked — it cannot be changed to datetime, and the column has to be re-sent under a new property name. Ask whether the column was uploaded before, and only map it once the user answers. Elapsed-duration formats (`[h]:mm:ss` and the equivalent built-ins) are durations rather than points in time and stay `number`.
+- **Excel formula cells** — a spreadsheet stores a formula and, next to it, the result Excel last computed. That cached result is the value: it is read normally, including a result of `0` or `""`, which are real values and not blanks. This tool never evaluates a formula and never guesses a result, so a cell holding a formula the file never computed has nothing to upload; it is read as missing and counted, as is an Excel error value (`#N/A`, `#DIV/0!`, …). Inspect reports the counts per column in a warning and `convert` repeats them in `manifest.output.unreadable_cells`. Report them: the rows are kept and the record count is unchanged, so this is the only explanation for a column that is empty in AE while the spreadsheet looks full. When a column that matters reads as missing this way, ask the user to recalculate and re-export in Excel, or to export values instead of formulas, before uploading. This covers XLSX; a legacy `.xls` workbook goes through a different parser and is not counted here.
+- **Merged cells, hidden rows, and hidden columns** — a sheet maintained by hand merges a label down the rows it covers (`区域` spanning one region's block). Excel keeps that value on the block's first row only and stores every row below it as an empty cell, so a column that looks full on screen arrives mostly missing, and the AE property built from it would be empty for most events. The same worksheet may also hide a row inside a data block or hide a whole column. None of this travels with a row, so inspect scans the worksheet structure separately and reports it under `xlsx_structure`: `merged_ranges` with `merged_range_samples` (references such as `A3:A5`, never cell text), `merged_covered_cells` per column, `hidden_rows` with `hidden_row_samples` (source row numbers as Excel numbers them), and `hidden_columns` by header name. The default read is unchanged, so the report is a question for the user, and each answer is a flag: `--fill-merged-cells` copies each block's value into the cells its own range covers — bounded to the range, never overwriting a value that is there and never inventing one when the block's own cell is empty, so it is not a forward fill; `--exclude-hidden-rows` leaves hidden rows out. Neither is on by default: those cells really are empty in the file, and a row hidden inside a data block may still be real data — unlike a hidden *worksheet* (below), which is excluded by default. Hidden columns have no flag at all; when the user confirms one is not data, list it in the mapping's `exclude_columns`. Both flags are carried into the recommended mapping as `fill_merged_cells` / `exclude_hidden_rows`, which is what makes `convert` read the rows inspect profiled — `convert` has no read flags of its own — and `convert` repeats the findings in `manifest.output.xlsx_structure`, the only record of a layout the converted rows no longer show. This covers XLSX; a legacy `.xls` workbook is not scanned, so ask the user about merged labels and hidden rows there instead of trusting silence.
+- **Hidden worksheets** — a worksheet hidden in the workbook is left out of the `--data-set` candidates and out of `--merge-sheets`, because a sheet the file does not show is usually scratch space, a lookup table, or a superseded draft rather than rows anyone meant to upload. Inspect lists each one under `excluded_sheets` (with `reason: hidden`); report those names to the user, since they are the only explanation for a row count lower than the workbook appears to hold. Their headers are also left out of `header_consistency`, so a stale hidden draft cannot make a mergeable workbook look ragged. A hidden sheet stays readable when the user names it in `--data-set` — the command then warns on stderr that the selected sheet is hidden. Only pass a hidden sheet after the user says that is what they want. When *every* worksheet is hidden there is no candidate left, and inspect fails with `LOCAL_DATA_ALL_DATA_SETS_HIDDEN` whose hint lists the hidden sheets; treat that as a question about which sheet holds the real data, not as an unreadable file. This detection covers XLSX only: a legacy `.xls` workbook's sheet list is unfiltered, so a hidden sheet there still appears as a candidate and is still merged — for `.xls`, ask the user to confirm the sheet list instead of trusting it.
+- **Excel sheets** — `--merge-sheets` streams every visible worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match. Matching headers establish a shared structure, not disjoint rows: a detail sheet and a summary sheet, or `1月` and `1月修订版`, usually carry identical headers and would be merged and reported twice over. Before merging, confirm with the user that the sources are mutually exclusive partitions (one month per sheet, no overlap) rather than overlapping, revised, or derived views of the same rows, and show each sheet's row count and time coverage range in that confirmation so an overlap is visible. The same rule applies to repeated `--input-file`.
 - **Multi-file type conflicts** — when the same column has different inferred types across files, present each conflict and resolve with `--type-resolutions` on `convert` (see [transform](transform.md)).
 
-## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV cells)
+## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV/Excel cells)
 
-Nested data is flattened one level per user decision. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick `Flatten` for a node.
+Nested data is analyzed per field, and the recommended mapping encodes one decision per container: keep whole, flatten one level, or flatten to the leaves. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick a depth for a node.
 
 1. **Locate the nested structure.**
    - NDJSON/JSON: read the top-level `nested_tree` from the inspect result (record-root paths).
-   - CSV/TSV: a JSON-encoded object column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Array cells (`items`-style) are `list` columns and carry no tree.
+   - CSV/TSV/Excel: a JSON-encoded column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Object cells list child keys; array cells (`items`-style) expose an `elementKind` and, for object arrays, the union of the element fields.
    Object nodes list child keys, array nodes carry an `elementKind`, primitive leaves carry an inferred type and bounded samples. Summarize node kinds, never paste samples.
-2. **Judge first, then confirm.** Read the node names and inferred types and propose, per container, whether to flatten (and to what depth), keep as JSON, or keep as a list. Present the proposal as a table and default to it; ask the user to confirm or adjust. Only ask node-by-node for the nodes you cannot judge from the names.
-   - Business-entity names with stable scalar children (`user_info`, `address`, `order`) → propose flattening to the leaves.
-   - Generic container names (`payload`, `data`, `config`, `meta`, `extra`, `attributes`) → propose keeping as JSON, unless the user wants a specific sub-field.
-   - Arrays are always kept as a list property and never split.
-   - Nesting deeper than one level with no clear intent → propose keeping as JSON or flattening only the first level.
-3. **Record the answers in `flatten_rules` (`{ "out_column": "dot.path" }`).**
+2. **Review the recommended mapping's per-field decision, then confirm adjustments.**
+   The recommended mapping already encodes, per container, a data-driven depth decision — the depth is not uniform across fields:
+   - **Keep whole** — a single-level object (every child scalar or a scalar array) is declared `type: 'object'` with `transform: 'json'`, plus one `parent.child` sub-property per child; a single-level object array is declared `type: 'array_row'` with one `parent.child` sub-property per element field. The parent entry carries the native value and conversion emits it once; each child is a plan-only declaration (scalar or `list`) that never reads a column of its own.
+   - **Flatten one level / to the leaves** — an object that itself contains an object or object array cannot be kept whole; it collapses one level and each child is decided independently: scalar children become flat properties materialized through `flatten_rules`, and nested objects/arrays recurse until a single-level container is reached (which is then kept whole).
+   - **Scalar array** — `["a","b"]` stays `list` (→ AE `array_string`), a leaf with no children.
+   - **Nested element field** — an element field that is itself an object/array stays inside the array data and is not declared as a sub-property (array-element flattening is not supported); the mapping warns so it is never silently dropped.
+   Present the resulting property list as a table and default to it; ask the user to confirm or adjust only where you disagree with a node's inferred decision (business-entity names with stable scalar children vs generic containers like `payload`/`data`/`config` vs nesting deeper than one level with no clear intent).
+3. **Record overrides in `flatten_rules` (`{ "out_column": "dot.path" }`) and `exclude_columns`.**
+   The recommended mapping already carries the flatten rules for its collapsed levels. When you override it:
    - NDJSON/JSON: the path is from the record root (`user_info.name`).
-   - CSV/TSV: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
-   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; CSV prefixes the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure.
+   - CSV/TSV/Excel: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
+   - **`flatten_rules` only materializes the out column in the row — it does not emit it.** For every out column, also add a `properties` entry with `source` set to that out-column name (plus `target`/`type`/`desc`); otherwise the flattened value is silently dropped from the output record.
+   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; cell-relative rules prefix the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'array_row'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure; a kept whole `object`/`array_row` also declares its scalar children as dotted `parent.child` sub-properties in `properties`.

package/skills/ae-data-integration/references/tracking-plan.md

@@ -10,15 +10,15 @@ Generate and confirm the event/property plan **before** any transform or upload.
 ## Sub-steps
 
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
-2. **Column → property draft** — identify system columns (time, `distinct_id`/`account_id`, event-name column, user-property-name column); map the remaining columns to event / user / common properties. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
-3. **Confirmation gate (single, one pass)** — present one summary table: event list + property list (uncertain types highlighted) + field scope (default: plan fields only, with a full-import switch) + unrecognized/dirty-data handling. The user answers once with ok or edits (renames, types, add/drop columns).
+2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
+3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
 4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
-5. **Persist the plan** — draft.json → xlsx → upload with `sdk_integration_mode=none`.
+5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 
 ## CLI
 
-`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out <draft.json>] [--dry-run]` converts the confirmed `ae-local-data-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
+`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out .ae-cli/data-integration/draft.json] [--dry-run]` converts the confirmed `ae-data-integration-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
 
 - `user_set` mode → no events; every mapping property becomes a user property.
 - `track` mode → one event per `--event-name` (or the mapping `default_event_name`); every property becomes an event property.
@@ -26,9 +26,11 @@ Generate and confirm the event/property plan **before** any transform or upload.
 - `exclude_columns` are dropped from the draft.
 - `desc` and `event_tag` flow from the mapping: each property's `desc` (falling back to its source column name) and each event's `event_meta.<name>.desc` / `event_meta.<name>.tag` (falling back to the source event name) are written into the draft. Fill them in the mapping so the plan is never empty.
 - The per-row event-name column cannot be enumerated without a full scan, so when `default_event_name` is absent the CLI requires `--event-name` for each concrete event name.
+- Duplicate `display_name`s across the property pool are auto-deduplicated (the property name is appended) before validation, so same-named source columns under different object paths never collide. Dotted `target`s default each sub-property's `display_name`/`desc` to its leaf segment, which also avoids collisions.
+- The draft is validated locally before any upload: an `object`/`array_row` property with no `parent.child` sub-property, or a sub-property whose type is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` (`--dry-run` fails too). Fix it in the mapping before running `plan`: a mapping `properties` entry with a dotted `target` (`user_info.name`) declares a `parent.child` sub-property whose parent is the `object`/`array_row` entry with the same first segment — so a kept whole container must carry one dotted entry per child (each scalar or `list`) next to the parent entry. A nested object/object-array that cannot stay nested must instead be flattened to scalar leaves: record one `flatten_rules` entry per leaf and one flat scalar (or `list`) `properties` entry per leaf (see [source-inspect.md](source-inspect.md)).
 - `--dry-run` previews events and properties without writing `draft.json`.
 
-Afterward reuse `ae-cli tracking plan draft --in draft.json --out draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
+Afterward reuse `ae-cli tracking plan draft --in .ae-cli/data-integration/draft.json --out .ae-cli/data-integration/draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
 
 ## Reuse
 

package/skills/ae-data-integration/references/transform.md

@@ -1,6 +1,6 @@
 # Transform — column → UE mapping
 
-Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first.
+Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first. The key system fields (steps 1–5 below) were already confirmed during the tracking-plan step (tracking-plan.md step 2); if nothing changed, state the confirmed values once and move to the property set — never skip a confirmation the user has not actually given.
 
 Read [UE mapping](ue-mapping.md) and apply these gates:
 
@@ -12,10 +12,10 @@ Read [UE mapping](ue-mapping.md) and apply these gates:
 Confirm the system fields with the user before touching properties. These are the top-level `#` fields, and users may not understand them, so explain each in plain language before asking; never infer a decision from a column name alone.
 
 1. **`#type` / mode.** Confirm the recommendation's `mode` (`track` / `user_set` / `mixed`). If `record_type_field` is set, confirm each distinct value normalizes to one of the eight record types. Ask whether the data reports events (→ `track`) or modifies user profiles (→ `user_set` or another profile type).
-2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one.
+2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one. `identity_candidates` is name-matched only, so it misses identity columns with arbitrary names (`玩家ID`, `用户账号`, `player_name`); surface any column whose shape is identifier-like (high `unique_ratio`, low `missing_ratio`, a string or numeric key, not a time column) and ask the user what it is. When one file has multiple sheets/data-sets, cross-check identity columns across them before finalizing: a column that appears (or near-matches, e.g. `设备ID` vs `主设备ID`) in more than one sheet likely has the same business meaning, so present a per-sheet `#account_id`/`#distinct_id` view and ask whether the same column maps to the same system field everywhere — a device ID used as `#distinct_id` in the event sheet but left as a differently-named plain property in the profile sheet silently breaks anonymous-user linkage, so surface the mismatch and let the user decide rather than letting it pass silently.
 3. **`#time`.** Confirm the time column and the source timezone (an IANA name from the user/project context). Leave `time_format` unset unless auto-detection failed (US/EU ambiguous dates). Also confirm the data's `#zone_offset` — the whole-hour UTC offset AE applies to those times, emitted inside `properties` (not top-level). The property's value is a number, so ask the user in numeric terms ("is the offset 8, i.e. UTC+8?") and set `zone_offset_value` to that integer; when a column carries the offset per row, set `zone_offset_field` instead. The two are mutually exclusive.
 4. **`#event_name` (track only).** Confirm the event-name column, or a reviewed `default_event_name` when no column exists.
-5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip.
+5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip. `#ip` is event data only and must be a valid IPv4/IPv6 address (a private/LAN IP is kept but reported — AE cannot geolocate it); `#uuid` must be a standard 36-character UUID. A value that violates the spec is dropped from that row only (`INVALID_IP` / `INVALID_UUID`) — the row itself is kept. The program never auto-generates a `#uuid`.
 
 When an `#event_name`, `#account_id`, or `#distinct_id` column's values do not satisfy AE naming rules (pure Chinese, uppercase, spaces), do not stop — scan the distinct values, list them to the user, and ask for one AE-name replacement each; record the pairs in `value_mapping` (see [UE mapping](ue-mapping.md)). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded. The same mechanism applies to a property column via that entry's own `value_mapping`.
 
@@ -24,18 +24,18 @@ Then confirm the property set with the user before saving the mapping. Present e
 Walk the table item by item:
 
 1. **Auto-renamed targets.** Recommendation sanitizes column names (camelCase split, illegal chars → `_`, digit-leading → `field_`, nothing recognizable → `field_N`). Show each source → target; ask for a manual name for every `field_N` fallback and rewrite the `target`.
-2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`). ID-like columns stay `string` even when numeric; `object`/`list` columns carry `transform: 'json'`. A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
+2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`/`array_row`). ID-like columns stay `string` even when numeric; `object`/`array_row`/`list` columns carry `transform: 'json'`. A scalar array is `list` (→ AE `array_string`); an object array is `array_row` (→ AE `array_row`). A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
 3. **Exclusions.** Ask which columns to drop and record them as `exclude_columns`.
 
 Before saving, present the complete mapping for a final sign-off on one grouped page — system fields (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#zone_offset`, `#event_name`, `#ip`/`#uuid`), then the property table (source column → target AE name → type), then excluded columns — and wait for an explicit yes. Never dump the raw mapping JSON at the user.
 
-Save the reviewed `ae-local-data-mapping/v1` JSON locally. Convert into a new output directory:
+Save the reviewed `ae-data-integration-mapping/v1` JSON to `.ae-cli/data-integration/mapping.json` (project workspace). Convert into a new output directory:
 
 ```bash
 ae-cli data-integration convert \
   --input-file '<path>' \
-  --mapping '<mapping.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --mapping '.ae-cli/data-integration/mapping.json' \
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
 For multiple files, repeat `--input-file` and pass a wildcard mapping plus `--type-resolutions` when inspect reported conflicts:
@@ -45,10 +45,12 @@ ae-cli data-integration convert \
   --input-file '<a.csv>' --input-file '<b.csv>' \
   --mapping '<wildcard-mapping.json>' \
   --type-resolutions '<resolutions.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
-The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine.
+Merging several sources into one run assumes they are mutually exclusive partitions of the same data set. Confirm that with the user before converting — matching headers do not rule out a detail/summary pair or an original/revised pair — and present each source's row count and time coverage range so an overlap is visible. The same applies to `--merge-sheets` (see [source-inspect.md](source-inspect.md)).
+
+The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason, and check the conservation equation first: `manifest.output.source_rows` must equal `valid_records + invalid_records` — a mismatch means rows were dropped or duplicated between the source and the output, so report the three numbers to the user and do not upload until it is explained. A salvage run's `source_rows` is only the rows it re-processed, not the whole file. If `manifest.output.skipped_fields` is present, tell the user how many `#ip`/`#uuid` values were invalid and dropped (the rows were otherwise kept); if `manifest.output.lan_ip_records` is present, tell the user that many `#ip` values are private/LAN addresses that AE cannot geolocate. Neither blocks the manifest. If `manifest.output.flatten_misses` is present — or a stderr `Warning: flatten rule "X" did not materialize for N row(s).` fires — a `flatten_rules` path missed some rows: re-check the dot path against the source shape (or confirm the column is legitimately optional); the rows are otherwise kept. A stderr `Warning: … column count different from the header row …` means ragged CSV/TSV rows were tolerated (extra fields dropped, missing fields treated as empty) — surface it as a data-quality note. A blocked manifest whose reason is `The source contained no data rows.` means the file had zero data rows; do not re-run the same command on it. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine. For the full failure taxonomy and how to respond, see [error handling](error-handling.md).
 
 ## Re-report only the failed rows (salvage loop)
 
@@ -59,7 +61,7 @@ ae-cli data-integration convert \
   --input-file '<same-source>' \
   --mapping '<fixed-mapping.json>' \
   --salvage-from '<run-dir>/invalid.rows.jsonl' \
-  --output-dir '.ae-cli/data-integration/<run-id>-salvage'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>-salvage'
 ```
 
 This emits a `valid.ue.jsonl` containing only the rows that now pass, plus a new `invalid.rows.jsonl` with whatever still fails. Fixes are rarely one-shot, so repeat: feed each round's `invalid.rows.jsonl` into the next `--salvage-from` until no rows fail or the user stops. Each round's `valid.ue.jsonl` is disjoint from earlier rounds', so upload each round independently (same confirmation and `--allow-clean-subset` gates as a normal upload). `--salvage-from` is single-file only and the source must be the same file that produced the quarantine.

package/skills/ae-data-integration/references/ue-mapping.md

@@ -1,12 +1,12 @@
 # UE mapping contract
 
-The mapping version is `ae-local-data-mapping/v1`.
+The mapping version is `ae-data-integration-mapping/v1`.
 
 Required structure:
 
 ```json
 {
-  "version": "ae-local-data-mapping/v1",
+  "version": "ae-data-integration-mapping/v1",
   "source": {
     "sha256": "<source-sha256>",
     "format": "csv",
@@ -54,10 +54,25 @@ Top-level `#` fields map from named source columns. The user confirms each mappi
 | `#account_id` | Login/account ID (database `user_id`, phone, member ID) — identifies authenticated users | At least one of `#distinct_id` / `#account_id` |
 | `#time` | Event/profile occurrence time; AE bins data by it | Yes |
 | `#event_name` | What the user did (`purchase`, `login`) | Only for `track` |
-| `#ip` | Client IP; AE resolves geo from it | No |
-| `#uuid` | Short-window deduplication ID | No |
+| `#ip` | Client IP; AE resolves geo from it. Event data only | No |
+| `#uuid` | Standard 36-character UUID uniquely identifying the record (dedup). Both event and user data | No |
 
-`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined.
+`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined. `#zone_offset` is event data only — user profile records never carry it.
+
+## Value specifications
+
+Each mapped system field carries a value spec, enforced at both inspect (warnings) and convert (row handling). A row error quarantines the whole row; a field skip drops only that field and keeps the row.
+
+| Field | Value spec | On violation (convert) |
+| --- | --- | --- |
+| `#account_id` / `#distinct_id` | Non-empty string, at most 128 characters | Row error `MISSING_USER_ID` (absent) / `USER_ID_TOO_LONG` (>128) |
+| `#event_name` | `^[a-z][a-z0-9_]{0,49}$` (lowercase snake_case, letter-leading, ≤50 chars) | Row error `INVALID_EVENT_NAME` |
+| `#time` | One of the supported formats, within 3 years back / 3 days forward | Row error `INVALID_TIME` / `TIME_OUT_OF_RANGE` |
+| `#ip` | Valid IPv4 or IPv6. Event data only | Field skip `INVALID_IP`; a private/LAN IP is kept and reported — AE cannot geolocate it |
+| `#zone_offset` | Integer -12..14 (or an IANA name for `zone_offset_value`). Event data only | Row error `INVALID_ZONE_OFFSET` |
+| `#uuid` | Standard 36-character UUID (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`). Both data kinds | Field skip `INVALID_UUID` |
+
+A field skip is not a row failure: the row is kept with its other fields, and the count is reported in `manifest.output.skipped_fields` so the agent tells the user at the end. `#uuid` is never auto-generated — it comes only from a mapped source column.
 
 Mapping keys: `account_id_field`, `distinct_id_field`, `time.field`, `event_name_field` (or a reviewed `default_event_name`), `ip_field`, `uuid_field`. Inspect surfaces every identity-shaped column in `identity_candidates` (name, `account`/`distinct` kind, unique and missing ratios) so the agent can present all candidates for confirmation. When the required identity column is absent, use an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool` — a user decision, never invented.
 
@@ -86,16 +101,20 @@ All fields below are optional and are explicit user decisions — never invent t
 | `account_id_value` / `distinct_id_value` | `string` | Fixed placeholder identity (≤128 chars). Applies to every row when the corresponding `*_field` is absent; when the field exists, fills only the rows whose column is empty |
 | `random_pool` | `{ account_ids?: string[], distinct_ids?: string[] }` | Synthesizes a random identity when the source field is absent |
 | `exclude_columns` | `string[]` | Source columns skipped when building properties |
-| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it |
+| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV/Excel paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it. `flatten_rules` only materializes the out column in the row; it does **not** emit it — add a `properties` entry with `source` = the out column to actually produce the property. An out column that fails to materialize for some rows (missing path, non-JSON cell, or a path without a `<column>.` prefix) is counted per out-column in `manifest.output.flatten_misses` and named in a convert stderr warning — fix the path or confirm the column is legitimately optional |
 | `headers` | `string[]` | User-confirmed column names for a headerless file; presence means the first row is data. Never use inspect's `col_1..col_N` placeholders — infer names from each column's values, confirm them with the user, then write them here |
 | `missing_time` | `'now'` | Fill a missing/empty `#time` with the current time, for user-profile rows only (explicit user decision; track rows are never filled) |
-| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo) |
-| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (short-window deduplication ID) |
+| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo). Event data only; a value that is not valid IPv4/IPv6 is dropped from that row, and a private/LAN IP is kept but reported |
+| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (standard 36-character UUID, both data kinds). A non-UUID value is dropped from that row |
 | `zone_offset_value` | `number` (integer -12..14) or IANA `string` | Emits the fixed `#zone_offset` preset property inside `properties`. An IANA name resolves to its integer UTC offset |
 | `zone_offset_field` | `string` | Source column whose per-row integer value (-12..14) is emitted as `#zone_offset`; missing/non-integer rows are quarantined. Mutually exclusive with `zone_offset_value` |
 | `event_meta` | `{ <event-name>: { desc?: string, tag?: string } }` | Per-event business description and `event_tag` for the tracking plan, keyed by AE event name. Inferred from the data and user context; the user supplies anything not inferable — never leave them empty |
 
-Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`list`) whose values arrive as JSON text — JSON-encoded CSV cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`array_row`/`list`) whose values arrive as JSON text — JSON-encoded CSV/TSV/Excel cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+
+A `properties` entry whose `target` is dotted (`parent.child`) is a plan-only sub-property declaration: `parent` must name an `object`/`array_row` entry in the same mapping, the child must be scalar or `list`, and the child's `source` must equal the parent's `source` — the nested value lives inside the parent, which conversion emits once and never reads the child's column. A kept whole `object`/`array_row` therefore carries one dotted entry per child alongside the parent entry.
+
+Property types are `string`, `number`, `boolean`, `datetime`, `list`, `object`, and `array_row`. `list` is a scalar array (`["a","b"]`, → AE `array_string`); `object` is a one-to-one object (`{...}`); `array_row` is an object array (`[{...}]`, one-to-many). A mapping expresses an object's/object array's sub-properties as dotted `target`s (`user_info.name`, `orders.sku`); `data-integration plan` maps them one-to-one to `parent.child` plan properties (each child's `display_name`/`desc` default to its leaf segment). In the tracking plan, `object` and `array_row` properties must each have at least one `parent.child` sub-property, and every sub-property must itself be scalar or `array_string` — a nested object/object-array cannot stay nested and must be flattened further to scalar leaves. `list` (→ `array_string`) is a leaf and takes no children. `ae-cli data-integration plan` enforces this locally: a composite property with no children, or a child that is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` before any upload.
 
 ## Time formats
 
@@ -130,13 +149,16 @@ Values with an explicit offset or `Z` are parsed by the JavaScript `Date` constr
 - `#zone_offset`, when set, is a whole-hour integer in -12..14 (or an IANA name/column that resolves to one) and is emitted inside `properties`, never at the top level.
 - Event/property names are lowercase snake_case, begin with a letter, and are at most 50 characters.
 - Target property names are unique and do not collide with UE system fields.
-- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, or `object`.
+- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, `object`, or `array_row`.
 - Text is at most 2 KB; numbers stay within -9E15..9E15.
 - Lists contain at most 500 strings (255 bytes each) or 500 objects.
 - Objects contain at most 100 legal sub-properties; nested values follow the same type limits.
+- No `object`/`array_row` survives into the tracking plan: AE requires each to have a `parent.child` sub-property (scalar or `array_string` only), and a mapping cannot express one, so nested objects/object-arrays are flattened to scalar leaves first (a kept-whole container fails `data-integration plan` fast).
 - A conversion rule does not hide a real type conflict.
 - Event/profile times fall within the receiver window: previous 3 years through next 3 days.
-- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`).
+- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`). `#uuid` is never auto-generated.
+- `#uuid` values are standard 36-character UUIDs and `#ip` values are valid IPv4/IPv6; invalid ones are dropped from the row only (`skipped_fields`), never quarantined.
+- `#ip` and `#zone_offset` appear on `track` records only; user data carries neither.
 - `value_mapping`, `random_pool`, and fixed `account_id_value`/`distinct_id_value` came from an explicit user decision and match the actual distinct values/columns.
 
 `user_set` output for the same identity is ordered by time so receiver application order is deterministic. Conversion applies whole-row quarantine: any error on a row — identity, time, event, record type, or a single property (type coercion, size/limit) — drops the entire row. The row is written to `invalid.rows.jsonl` with its error codes, counted in `manifest.output.invalid_records`, and the manifest is blocked until reviewed. The failed rows are re-reportable without re-sending valid rows: fix the mapping, then run `convert --input-file <same-source> --mapping <fixed-mapping> --salvage-from <invalid.rows.jsonl>`. The salvage run re-processes only the listed row numbers against the same source, emits a `valid.ue.jsonl` containing only the newly fixed rows, and writes a new `invalid.rows.jsonl` with whatever still fails — so the loop repeats (feeding each round's `invalid.rows.jsonl` into the next `--salvage-from`) until no rows fail or the user stops.

package/skills/ae-data-integration/references/ue-routing.md

@@ -22,6 +22,28 @@ Classification order:
 
 Aggregated metrics, pivot tables, cross-tabs, model outputs, free-form documents, and records without real identity/time should normally use local analysis.
 
+### Time coverage is not native granularity
+
+A parseable time column establishes only when the rows are stamped, not what period each metric
+covers. A daily report and a cumulative snapshot both look like one row per user per point in time,
+so they satisfy every condition above and are then ingested as per-period events — inflating totals
+in a way that stays invisible in ratios, because numerator and denominator scale together.
+
+Native granularity must come from the user, a data dictionary, or a complete period structure in the
+data itself. Do not infer it from the file name, the first/last date, the interval between rows, or
+the row count; none of those is evidence.
+
+When the data carries numeric columns and any of the following holds, ask the user to state whether
+each row's value is the amount that occurred in that period or the total accumulated up to that
+point, and do not proceed until they answer:
+
+- Paired start/end time columns (`start_date`/`end_date`, `period_begin`/`period_end`).
+- Values for one identity that never decrease over time.
+- Column names carrying a to-date sense (`cumulative`, `total`, `ltv`, `累计`, `总`).
+
+An unanswered question, a cumulative snapshot, or overlapping periods route to local analysis
+instead.
+
 ## Route to local analysis
 
 Choose local analysis when:
@@ -29,6 +51,7 @@ Choose local analysis when:
 - The user wants insights, not project ingestion.
 - UE identity or time prerequisites are missing.
 - Each row is an aggregate rather than a user/event record.
+- The rows are a cumulative snapshot, or their native granularity could not be established.
 - Conversion would invent semantics or discard important structure.
 - The user declines an uncertain mapping or destination.
 

package/skills/ae-dataops/SKILL.md

@@ -20,6 +20,7 @@ The AE Data Development and Operations domain provides capabilities for data war
 | `dataops_flow` | Flow creation, node deletion, and orchestration | `dataops-flow-create` |
 | `dataops_flow` | Flow execution and monitoring | `dataops-flow-monitor` |
 | `dataops_operations` | Operations instance search, details, and task logs | `dataops-flow-monitor` |
+| `dataops_operations` | Backfill job creation, full draft update, deletion, execution, plans, stop, and rerun | `dataops-backfill` |
 | `dataops_ide` | Data exploration and SQL queries | `dataops-query` |
 | `dataops_integration` | Datasource and data integration | `dataops-integration` |
 
@@ -63,6 +64,7 @@ You must understand the following key concepts before use, otherwise errors are
 |----|--------|-------------|
 | **executeId** | Returned by `dataops_flow +execute_flow` | Early stop handle before the scheduler `flowInstanceId` is available |
 | **flowInstanceId** | Returned by `dataops_operations +search_flow_instances` | Operations perspective instance inspection and stop |
+| **jobId** | Returned by `dataops_operations +create_backfill_job` or `+search_backfill_jobs` | Persistent backfill job detail and lifecycle actions |
 
 ### Environment and Defaults
 
@@ -73,6 +75,8 @@ You must understand the following key concepts before use, otherwise errors are
 | `dataops_operations +get_flow_instance_detail` | Instance detail | Inspect one instance DAG and task statuses |
 | `dataops_operations +get_task_instance_detail` | Task detail/logs | Inspect one task and include logs only when needed |
 | `dataops_operations +stop_flow_instance` | Instance stop | Stop by exactly one of `executeId` or `flowInstanceId` |
+| `dataops_operations +list_backfill_flows` | Backfill source discovery | Returns eligible PROD flows and whether ST is required |
+| `dataops_operations +search_backfill_jobs` | Backfill job search | Filter persistent jobs and obtain `jobId` |
 
 ### Schema Naming Rules
 
@@ -92,6 +96,8 @@ You must understand the following key concepts before use, otherwise errors are
 Create DEV Flow → Create/Update DEV SQL, Integration, Workflow Instance Check, or Task Instance Check Tasks → Configure Dependencies/Schedule → Preview Release → Release to PROD → PROD Manual Execution / Operations Troubleshooting
 ```
 
+Backfill lifecycle: Discover eligible PROD flow → Create or fully update DRAFT job → Run explicitly → Search / inspect plans → Stop or rerun the complete job; delete only after target inspection
+
 ### CRON Format (6 fields)
 
 `second minute hour day month weekday` — Note: one more "second" field than standard 5-field format.
@@ -115,6 +121,7 @@ Choose the appropriate scenario skill based on user intent to get complete step-
 | Create flow, add or delete nodes, configure schedule, release | `dataops-flow-create` | create flow, new workflow, configure schedule, add task node, delete task node, release, cron, scheduled execution |
 | View execution status, troubleshoot failures, view logs | `dataops-flow-monitor` | execute flow, running instance, monitor, logs, stop, DAG, troubleshoot |
 | Search operation instances across a space | `dataops-flow-monitor` | operations instance, flow instance search, status statistics, owner statistics |
+| Create or operate a persistent multi-date backfill job | `dataops-backfill` | backfill, fill historical data, base date range, backfill plans, stop backfill, rerun backfill |
 | Create datasource, configure sync solution, execute sync | `dataops-integration` | datasource, sync, integration, field mapping, data ingestion, MySQL, ClickHouse, DatabricksJdbc |
 | Browse metadata, search tables, execute SQL queries | `dataops-query` | query, SQL, data exploration, search tables, view table structure, IDE, catalog, select |
 | Create tables and views | `dataops-table` | create table, table creation, view, data dictionary, table details, DDL |
@@ -155,13 +162,16 @@ Flow orchestration is divided into two scenario skills: **creation and configura
 
 **Lifecycle: DEV configuration and preview → Release to PROD → PROD manual execution and operations troubleshooting**
 
-Detailed creation/configuration commands live in [`references/dataops-flow-create.md`](references/dataops-flow-create.md). Detailed execution, monitoring, operation instance, task log, and stop commands live in [`references/dataops-flow-monitor.md`](references/dataops-flow-monitor.md).
+Detailed creation/configuration commands live in [`references/dataops-flow-create.md`](references/dataops-flow-create.md). Detailed execution, monitoring, operation instance, task log, and stop commands live in [`references/dataops-flow-monitor.md`](references/dataops-flow-monitor.md). Persistent multi-date backfill jobs live in [`references/dataops-backfill.md`](references/dataops-backfill.md).
 
 Key constraints:
 - Create and update tasks in DEV, preview/release before PROD execution.
 - Treat `+delete_task` as high-risk: verify the target with `+get_flow_overview`, preview with `--dry-run`, and use `--yes` only after explicit user confirmation. Deletion affects DEV; release the flow to apply it to PROD.
 - `+execute_flow` always runs PROD; it returns `executeId` for early stop.
 - Prefer `flowInstanceId` from operations search for stable inspection and troubleshooting.
+- A backfill job is persistent and batches multiple base dates; do not emulate it by looping `+execute_flow`.
+- Create and run backfill jobs as separate steps. `+rerun_backfill_job` reruns the complete job, not only failed plans.
+- `+update_backfill_job` replaces a DRAFT job's complete configuration; inspect the job first and do not treat it as a partial patch. Treat `+delete_backfill_job` as high-risk and preview it with `--dry-run` before confirmation.
 - Reference workspace parameters in task SQL as `${paramKey}`.
 
 ---

package/skills/ae-dataops/references/dataops-backfill.md

@@ -0,0 +1,135 @@
+---
+name: dataops-backfill
+version: 1.0.0
+description: "Backfill job lifecycle: discover eligible PROD flows, create or update drafts, delete jobs, run jobs, inspect plans, stop running jobs, and rerun complete jobs."
+metadata:
+  requires:
+    bins: ["ae-cli"]
+---
+
+# DataOps Backfill Jobs
+
+> **Prerequisites:** Read [`ae-dataops/SKILL.md`](../SKILL.md) for authentication, output, and safety rules.
+
+A backfill job is a persistent operations object that runs one PROD task flow for multiple base dates. It is not a single manual flow execution and is not a retry of an existing failed flow instance.
+
+## Lifecycle
+
+### 1. Discover an eligible PROD flow
+
+```bash
+ae-cli dataops_operations +list_backfill_flows --spaceCode "${spaceCode}"
+```
+
+Use only a returned flow whose `completeDataInfo.canRun` is true. When `completeDataInfo.hasSt` is true, pass `--stTime` while creating the job.
+
+### 2. Create a draft
+
+Always pass the backfill date range. For manual selection, also pass `--completeDates`; every selected date must be inside that inclusive range.
+
+```bash
+# Range mode
+ae-cli dataops_operations +create_backfill_job --spaceCode "${spaceCode}" \
+  --jobName "August backfill" --flowCode ${flowCode} \
+  --startDate "2026-08-01" --endDate "2026-08-07"
+
+# Custom-date mode
+ae-cli dataops_operations +create_backfill_job --spaceCode "${spaceCode}" \
+  --jobName "Selected dates" --flowCode ${flowCode} \
+  --startDate "2026-08-01" --endDate "2026-08-07" \
+  --completeDates '["2026-08-01","2026-08-03"]'
+```
+
+Creation returns a `DRAFT` job and does not run it. Defaults are `jobType=TASK_ALL`, `failureStrategy=END`, `parallel=true`, `reverse=false`, `step=1`, and `unit=DAY`. For `TASK_ONLY`, `TASK_PRE`, or `TASK_POST`, also pass `--startNode`.
+
+### 3. Update a draft when needed
+
+Update is a complete replacement, not a partial patch. Inspect the DRAFT job first, then pass `--jobId`, `--jobName`, `--flowCode`, and the complete date and strategy configuration just as for creation. Updating does not run the job.
+
+```bash
+ae-cli dataops_operations +update_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId} --jobName "Revised August backfill" --flowCode ${flowCode} \
+  --startDate "2026-08-01" --endDate "2026-08-10" \
+  --failureStrategy END --parallel true --reverse false
+```
+
+### 4. Run the draft explicitly
+
+```bash
+ae-cli dataops_operations +run_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId}
+```
+
+### 5. Search jobs and inspect plans
+
+```bash
+ae-cli dataops_operations +search_backfill_jobs --spaceCode "${spaceCode}" \
+  --status "RUNNING,FAIL,SUCCESS" --pageNum 1 --pageSize 20
+
+ae-cli dataops_operations +get_backfill_job_detail --spaceCode "${spaceCode}" \
+  --jobId ${jobId}
+```
+
+Detail returns the job and its plans together. A draft has an empty plan list.
+
+### 6. Stop, rerun, or delete
+
+Stopping affects every unfinished plan in the running job. Inspect the job, preview the request, obtain explicit confirmation, and then pass `--yes`.
+
+```bash
+ae-cli dataops_operations +stop_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId} --dry-run
+# After explicit confirmation, execute the same target; the CLI prompts before dispatch.
+ae-cli dataops_operations +stop_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId}
+```
+
+Rerun applies to every plan only when the job is `FAIL` or `STOP`. A `SUCCESS` job cannot be rerun. Rerun reuses the same job and does not create a new backfill job. The CLI does not support rerunning only failed plans.
+
+```bash
+ae-cli dataops_operations +rerun_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId} --dry-run
+ae-cli dataops_operations +rerun_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId}
+```
+
+Deletion is high-risk. The server accepts only supported `DRAFT`, `FAIL`, or `SUCCESS` jobs. Inspect the exact target and preview the scoped request before confirmation; the CLI sends only `spaceCode` and `jobId` and does not pre-query or guess state.
+
+```bash
+ae-cli dataops_operations +delete_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId} --dry-run
+ae-cli dataops_operations +delete_backfill_job --spaceCode "${spaceCode}" \
+  --jobId ${jobId}
+```
+
+## Command Reference
+
+| Command | Purpose | Flags |
+|---|---|---|
+| `+list_backfill_flows` | List eligible PROD flows | `--spaceCode` |
+| `+create_backfill_job` | Create a draft | `--spaceCode` `--jobName` `--flowCode` `--startDate` `--endDate`; optional in-range `--completeDates`, scope, failure, parallel, order, and ST flags |
+| `+update_backfill_job` | Replace a DRAFT job's complete configuration | `--spaceCode` `--jobId` `--jobName` `--flowCode` and the same complete configuration as create |
+| `+delete_backfill_job` | Delete a supported job | `--spaceCode` `--jobId`; high-risk, requires confirmation or `--yes` |
+| `+run_backfill_job` | Run a draft | `--spaceCode` `--jobId` |
+| `+search_backfill_jobs` | Search jobs | `--spaceCode` plus optional keyword, date, type, status, owner, sort, and paging filters |
+| `+get_backfill_job_detail` | Get job and plans | `--spaceCode` `--jobId` |
+| `+stop_backfill_job` | Stop a running job | `--spaceCode` `--jobId`; high-risk, requires confirmation or `--yes` |
+| `+rerun_backfill_job` | Rerun the complete job | `--spaceCode` `--jobId` |
+
+Statuses are `DRAFT`, `RUNNING`, `STOP`, `FAIL`, `SUCCESS`, and `READY_STOP`. Range units are `DAY`, `WEEK`, and `MONTH`. Custom dates must be a non-empty JSON array of unique `yyyy-MM-dd` strings inside the configured date range.
+
+## Transport Status
+
+Transition status: transitional
+
+Owning module: Gaia operations
+
+Current transport: DataOps CLI REST
+
+Covered tools: `operations_list_backfill_flows`, `operations_create_backfill_job`, `operations_update_backfill_job`, `operations_delete_backfill_job`, `operations_run_backfill_job`, `operations_search_backfill_jobs`, `operations_get_backfill_job_detail`, `operations_stop_backfill_job`, and `operations_rerun_backfill_job`
+
+Gateway target: TBD after the DataOps operations Capability Gateway schema review
+
+Review after: 2026-11-20
+
+Exit condition: Migrate these commands after Gaia exposes equivalent capabilities and the command contract tests pass against the Capability Gateway transport.

package/skills/ae-engage/SKILL.md

@@ -398,6 +398,8 @@ When the user wants to "create a flow / generate a flow canvas / save a flow", d
 3. Build condition-related nodes with semantic `targetDefinitionRequest` and
    `triggerDefinition` objects. Resolve real event and property names through Analysis metadata;
    do not create an intermediate cluster merely to obtain persisted QP.
+   For `ab_split_flow` indicators, use `indicatorsDef[].eventDefinition`; always provide the
+   aggregate `operator`/`value` and every event-property filter's `operator`/`values`.
 
 4. Before building touchpoint nodes such as `message_push`, `wechat_push`, or `webhook_push`, you must call:
 
@@ -585,6 +587,9 @@ For task draft creation or update, use this workflow:
    and every-completion events use count/eq/1. Completion target and experiment main-goal event
    filters must not use properties whose metadata `select_type` is `datetime`. Never construct
    persisted QP fields.
+   Select the audience by delivery side: server-side channels allow custom (`1`) or existing (`2`)
+   and reject all users (`3`); `client_push` (`channelType=3`) allows custom (`1`) or all users (`3`)
+   and rejects existing (`2`). Do not infer audience support from `triggerType` alone.
 4. `ae-cli engage-task task save --project-id <projectId> --req '{...}'`
 5. `ae-cli engage-task task submit-approval --project-id <projectId> --task-id <taskId>`