@thinkingai/ae-cli 6.1.16 → 6.1.18

package/skills/ae-data-integration/references/handoff.md

@@ -1,43 +1,102 @@
 # Handoff (reusable package)
 
-After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package freezes the transform logic (the confirmed mapping), the runnable transform script, and the tracking-plan reference. It does **not** re-freeze the raw data or any upload secrets.
+After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package is a DataX-style pipeline — a declarative `source → transform → sink` descriptor plus generic stage executors that dispatch to `ae-cli data-integration` subcommands. It freezes the transform logic (the confirmed mappings) and the tracking-plan reference, and ships a `bin/` directory that a human or agent can run directly. It never re-freezes raw data or upload secrets.
 
 ## When
 
-Run handoff after Transform — or after Sink — once the mapping is confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
+Run handoff after Transform — or after Sink — once the mappings are confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
 
 ## CLI
 
 ```
-ae-cli data-integration handoff --mapping <mapping> [--plan-file <draft.json>] [--out-dir .ae-data-integration]
+ae-cli data-integration handoff --mapping <mapping> [--mapping <mapping2> ...] [--plan-file <draft.json>] [--pushurl <url>] [--project-id <id>] [--out-dir .ae-cli/data-integration]
 ```
 
-- `--mapping` — the confirmed `ae-local-data-mapping/v1` mapping (the frozen transform logic, including `value_mapping` and `flatten_rules`).
-- `--plan-file` — optional tracking-plan `draft.json` to reference inside the package (`plan.json`).
-- `--out-dir` — handoff root. Default `.ae-data-integration/` (travels with the project).
-- `--dry-run` previews the fingerprint, files, and index path without writing.
+- `--mapping` — one or more confirmed `ae-data-integration-mapping/v1` mappings (the frozen transform logic, including `value_mapping` and `flatten_rules`). Repeat it for multi-sheet workbooks: one mapping per sheet.
+- `--plan-file` — optional tracking-plan `draft.json` to reference inside each mapping directory (`plan.json`).
+- `--pushurl` — optional receiver base URL to record as the reuse upload target (the sink endpoint is `pushurl` + `/sync_json`). Record it when the next same-shape file will most likely land at the same receiver.
+- `--project-id` — optional numeric destination project ID to record; `bin/upload.sh` derives the APPID from it via `project info get` at upload time.
+- `--out-dir` — handoff root. Default `.ae-cli/data-integration/` (project workspace; travels with the project).
+- `--dry-run` previews the fingerprints, target, file list, and zip path without writing.
 
 ## Package layout
 
 ```
-.ae-data-integration/
-  index.json               ← shared fingerprint index (reuse detection)
-  <fingerprint[:16]>/
-    mapping.json           ← frozen mapping (re-runnable by convert)
-    transform.mjs          ← node transform.mjs <new-file> [<output-dir>]
-    plan.json              ← optional tracking-plan reference
+.ae-cli/data-integration/    ← handoff root (= out-dir, project workspace)
+  pipeline.json             ← source → transform → sink descriptor (ae-data-integration-pipeline/v1)
+  index.json                ← shared structure-fingerprint index (reuse detection)
+  shape.json                ← per-mapping column baseline (shape gate)
+  <fingerprint[:16]>/       ← one directory per mapping
+    mapping.json            ← frozen mapping
+    transform.mjs           ← node transform.mjs <new-file> [<output-dir>]
+    plan.json               ← optional tracking-plan reference
+  bin/                      ← generic stage executors (read pipeline.json)
+    run.sh                  ← source → transform → plan (never uploads)
+    upload.sh               ← sink (dry-run by default; --confirm uploads; resolves recorded target)
+    bind_mapping.py         ← shape check + rebind sha256/data_set to the new file
+    summarize.py            ← valid/quarantined counts
+    plan_check.py           ← tracking-plan coverage gate (exit 3 on new events/properties)
+    verify.py               ← soft persistence check (submit window vs ingest summary)
+    resolve_appid.py        ← APPID derivation helper (project info get)
+  README.md / RUNBOOK.md    ← how to run, the four gates, persistence verification
+  .local/target.env.example ← destination template (no real secrets)
+  .gitignore                ← inbox/ runs/ .local/target.env
+  inbox/  runs/             ← daily input / per-run outputs
 ```
 
-The `transform.mjs` wrapper shells out to `ae-cli data-integration convert`; it takes the new file path as its first argument, never bakes the source path, and re-stamps `source.sha256` with the new file's fingerprint before converting. Reuse it only for a file of the **same shape** (same header/schema and format) — the transform logic is frozen, but the content guard is re-bound to each specific file.
+A shareable archive is written next to the package root: `<parent>/ae-data-integration-handoff-<fingerprint[:8]>.zip`. The zip carries only this handoff round — the frozen mappings just written, a scoped `index.json` (just this round's entries), and the generic executors/docs — not the accumulated history, which stays in `.ae-cli/data-integration/` for reuse matching.
+
+## Pipeline descriptor
+
+`pipeline.json` declares the three stages and their types. Only `source: local_file` and `sink: restful_sync_json` are implemented today; the `type` fields reserve logbus / datax / mysql for later phases. `bin/run.sh` and `bin/upload.sh` read the descriptor and dispatch each stage by its `type` to `ae-cli data-integration inspect / convert / upload` — they are executors, not a second runtime engine.
+
+## Recorded destination
+
+`pipeline.json` → `sink.params` records `pushurl` and `project_id` when the handoff
+was run with those flags. Reuse defaults to that target, but **`bin/upload.sh`
+never sends without `--confirm`**, so the operator re-confirms the address and
+project on every reuse. Resolution order at upload time:
+
+- endpoint: recorded `pushurl` (+ `/sync_json`), else `AE_ENDPOINT`.
+- APPID: `AE_APPID`, else derived via `ae-cli project info get --project-id <id>`
+  (see `bin/resolve_appid.py`; set `AE_APPID` when that payload lacks `appid`).
+- project id: recorded `project_id`, else `AE_PROJECT_ID`.
+
+`project info get` returns `data.appid` at the top level (verified against the AE
+demo host); `bin/resolve_appid.py` reads that exact field and prints it, falling
+back to `AE_APPID` when the field is absent or not a non-empty string.
+
+## Project custom layers
+
+For projects that need a hard per-event SQL judge or a multi-round salvage loop
+(the reference package's environment-specific workarounds), overlay a custom layer
+next to the package instead of editing it — see [custom-layer.md](custom-layer.md).
+
+## Four confirmation gates
+
+The RUNBOOK and the scripts enforce four gates. The first two run automatically; the last two always need human confirmation:
+
+1. **Shape gate** — `bind_mapping.py` compares the new file's column set against `shape.json` and fails fast on a mismatch. A changed shape means the frozen logic was never reviewed for it: re-run the full pipeline.
+2. **Transform** — `ae-cli data-integration convert` per mapping; quarantined rows go to `invalid.rows.jsonl`, never silently dropped.
+3. **Tracking-plan gate** — `plan_check.py` verifies every produced event/property already exists in `plan.json`; new ones exit 3 and must be merged into the project plan first.
+4. **Sink gate** — `upload.sh` is dry-run by default; `--confirm` is the explicit upload decision.
 
 ## Structure fingerprint and index
 
-Every handoff records one entry in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: columns (source name + type), event model (`mode`, `event_name_field`, `record_type_field`), identity fields, and excluded columns. Business logic (`value_mapping`, transforms, `time_format`, and the fixed `default_event_name`) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+Every handoff records one entry per mapping in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model (`mode`). Business logic (`value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, the fixed `default_event_name`, and the system-field assignments) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+
+Reuse matching is its own step — see [references/reuse.md](reuse.md).
 
-Reuse matching is its own step — see [references/reuse.md](reuse.md). It compares a new file's profile structure against this index and proposes the matching package, which the user confirms before skipping the full pipeline.
+## Completion response
+
+After handoff succeeds, state the **absolute zip path**, the package directory, and the one-line way to run the next same-shape file:
+
+```
+cd <out-dir> && bin/run.sh <new-file>   # then bin/upload.sh runs/<run-id> --confirm
+```
 
 ## Safety rules
 
-- Treat the mapping, plan, generated artifacts, and the `.ae-data-integration/` directory as sensitive.
-- Never write APPID, endpoints, tokens, or raw data values into a handoff package. The package references the plan and the mapping; uploads still require an explicit, confirmed `upload` call.
+- Treat the mappings, plans, generated artifacts, the `.ae-cli/data-integration/` directory, and the zip as sensitive.
+- Never write APPID, tokens, or raw data values into a handoff package. The package records at most a destination `pushurl` and `project_id`; uploads still require an explicit, confirmed `upload` call, so the operator re-confirms the address and project each time.
 - Do not invent a mapping or plan. Handoff only packages what the user already confirmed.

package/skills/ae-data-integration/references/local-analysis.md

@@ -1,6 +1,6 @@
 # Local analysis
 
-Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/<run-id>/` with restrictive permissions.
+Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/runs/<run-id>/` with restrictive permissions.
 Set the directory to `0700` and generated scripts/reports to `0600`.
 
 ## Default report when no question is supplied

package/skills/ae-data-integration/references/reuse.md

@@ -4,21 +4,25 @@ Before walking the full pipeline for a new file, check whether a **handoff packa
 
 ## When
 
-Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-data-integration/index.json` exists. It is read-only and makes no writes.
+Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-cli/data-integration/index.json` exists. It is read-only and makes no writes.
 
 ## CLI
 
 ```
-ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-data-integration]
+ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-cli/data-integration]
 ```
 
 - `--mapping` — the candidate mapping, typically `inspect`'s `recommended_mapping` (the new file's structure, inferred exactly as Source would infer it).
-- `--out-dir` — handoff root. Default `.ae-data-integration/`.
+- `--out-dir` — handoff root. Optional. Without it, `reuse` searches in order: the current directory's `.ae-cli/data-integration/`, then each parent directory upward, then `~/.ae-cli/data-integration/` as a global fallback — so a package written in another directory or another session is still reachable.
 - `--dry-run` previews the fingerprint and match verdict without reading frozen packages.
 
+## Search paths
+
+When `--out-dir` is omitted, the result carries `searched_paths` — the exact `index.json` paths probed, in order. A `matched: false` result with a non-empty `searched_paths` means all of them were checked; a missing global fallback (no `$HOME`) simply omits it from the list.
+
 ## Matching
 
-The command computes the same **structure fingerprint** the handoff index is keyed on — columns (source name + type), event model, identity fields, and excluded columns — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`) is not part of the key, so a same-shape file with different content or a different file name still matches.
+The command computes the same **structure fingerprint** the handoff index is keyed on — the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, and the system-field assignments) is not part of the key, so a same-shape file with different content or a different file name still matches.
 
 ## Result
 
@@ -26,7 +30,7 @@ The command computes the same **structure fingerprint** the handoff index is key
 - **Match** (`matched: true`): the result carries the matched package — `mapping_file`, optional `plan_file`, the frozen `default_event_name` (so the user sees which event name will be reused), and a `run` command:
 
 ```
-node .ae-data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
+node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
 ```
 
 ## Confirmation gate

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,7 +31,7 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
-Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type, grouped into event properties (`track`) and user properties (profile modes) — never a counts-only summary. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
+Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.
 

package/skills/ae-data-integration/references/source-inspect.md

@@ -8,10 +8,23 @@ Confirm:
 2. The desired outcome: AE ingestion, local analysis, or help deciding.
 3. For ingestion, the target environment/project if already known.
 
-Accept one or more CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, or XLSX files. CSV/TSV/TXT/JSON/JSONL/XLSX may be at most 200 MB each; XLS may be at most 50 MB each.
+Accept one or more CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, or XLSX files. CSV/TSV/TXT/JSON/JSONL/XLSX have no hard size limit; files over 1 GB print a stderr warning with an estimated processing time and suggest splitting. XLS over 100 MB prints a memory-risk warning (the legacy parser loads the whole workbook, roughly 5-10x file size); XLS over 1 GB is still rejected — convert it to XLSX or split it first.
 
 ## Inspect without exposing raw values
 
+Before the full inspection (which streams and profiles the entire file and can take
+minutes on large files), pre-check the size and time estimate:
+
+```bash
+ae-cli data-integration inspect --input-file '<path>' --dry-run
+```
+
+For each file that reports a `warning`, `memory_risk: true`, or `rejected: true`,
+surface the `estimated_duration` / `reason` / `warning` to the user verbatim and
+confirm before running the real `inspect`. Never start a multi-minute or memory-risk
+job without the user seeing the estimate and risk first. A rejected file cannot
+proceed; follow its `reason` instead of asking for confirmation.
+
 Run:
 
 ```bash
@@ -26,7 +39,9 @@ If `selection_required=true`, show only the Sheet/JSON Path candidates and ask t
 ae-cli data-integration inspect --input-file '<path>' --data-set '<candidate-id>' --source-timezone '<iana-timezone>'
 ```
 
-Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. Read [UE routing](ue-routing.md) before choosing a branch.
+Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. IP- and UUID-named columns are additionally checked against their value specs: inspect warns how many non-empty values are invalid IPv4/IPv6, private/LAN IPs, or non-UUID strings, so the user can decide whether to map them as `ip_field`/`uuid_field`. Read [UE routing](ue-routing.md) before choosing a branch.
+
+A stderr `Warning: … column count different from the header row …` means the CSV/TSV has ragged rows (extra fields dropped, missing fields treated as empty); report it as a data-quality signal. For how every pipeline failure — abnormal data, parse errors, and program errors — is classified and handled, see [error handling](error-handling.md).
 
 ## Advanced input
 
@@ -36,20 +51,24 @@ Report row/column counts, field types, missing/unique/time-parse ratios, UE elig
 - **Excel sheets** — `--merge-sheets` streams every worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match.
 - **Multi-file type conflicts** — when the same column has different inferred types across files, present each conflict and resolve with `--type-resolutions` on `convert` (see [transform](transform.md)).
 
-## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV cells)
+## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV/Excel cells)
 
-Nested data is flattened one level per user decision. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick `Flatten` for a node.
+Nested data is analyzed per field, and the recommended mapping encodes one decision per container: keep whole, flatten one level, or flatten to the leaves. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick a depth for a node.
 
 1. **Locate the nested structure.**
    - NDJSON/JSON: read the top-level `nested_tree` from the inspect result (record-root paths).
-   - CSV/TSV: a JSON-encoded object column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Array cells (`items`-style) are `list` columns and carry no tree.
+   - CSV/TSV/Excel: a JSON-encoded column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Object cells list child keys; array cells (`items`-style) expose an `elementKind` and, for object arrays, the union of the element fields.
    Object nodes list child keys, array nodes carry an `elementKind`, primitive leaves carry an inferred type and bounded samples. Summarize node kinds, never paste samples.
-2. **Judge first, then confirm.** Read the node names and inferred types and propose, per container, whether to flatten (and to what depth), keep as JSON, or keep as a list. Present the proposal as a table and default to it; ask the user to confirm or adjust. Only ask node-by-node for the nodes you cannot judge from the names.
-   - Business-entity names with stable scalar children (`user_info`, `address`, `order`) → propose flattening to the leaves.
-   - Generic container names (`payload`, `data`, `config`, `meta`, `extra`, `attributes`) → propose keeping as JSON, unless the user wants a specific sub-field.
-   - Arrays are always kept as a list property and never split.
-   - Nesting deeper than one level with no clear intent → propose keeping as JSON or flattening only the first level.
-3. **Record the answers in `flatten_rules` (`{ "out_column": "dot.path" }`).**
+2. **Review the recommended mapping's per-field decision, then confirm adjustments.**
+   The recommended mapping already encodes, per container, a data-driven depth decision — the depth is not uniform across fields:
+   - **Keep whole** — a single-level object (every child scalar or a scalar array) is declared `type: 'object'` with `transform: 'json'`, plus one `parent.child` sub-property per child; a single-level object array is declared `type: 'array_row'` with one `parent.child` sub-property per element field. The parent entry carries the native value and conversion emits it once; each child is a plan-only declaration (scalar or `list`) that never reads a column of its own.
+   - **Flatten one level / to the leaves** — an object that itself contains an object or object array cannot be kept whole; it collapses one level and each child is decided independently: scalar children become flat properties materialized through `flatten_rules`, and nested objects/arrays recurse until a single-level container is reached (which is then kept whole).
+   - **Scalar array** — `["a","b"]` stays `list` (→ AE `array_string`), a leaf with no children.
+   - **Nested element field** — an element field that is itself an object/array stays inside the array data and is not declared as a sub-property (array-element flattening is not supported); the mapping warns so it is never silently dropped.
+   Present the resulting property list as a table and default to it; ask the user to confirm or adjust only where you disagree with a node's inferred decision (business-entity names with stable scalar children vs generic containers like `payload`/`data`/`config` vs nesting deeper than one level with no clear intent).
+3. **Record overrides in `flatten_rules` (`{ "out_column": "dot.path" }`) and `exclude_columns`.**
+   The recommended mapping already carries the flatten rules for its collapsed levels. When you override it:
    - NDJSON/JSON: the path is from the record root (`user_info.name`).
-   - CSV/TSV: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
-   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; CSV prefixes the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure.
+   - CSV/TSV/Excel: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
+   - **`flatten_rules` only materializes the out column in the row — it does not emit it.** For every out column, also add a `properties` entry with `source` set to that out-column name (plus `target`/`type`/`desc`); otherwise the flattened value is silently dropped from the output record.
+   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; cell-relative rules prefix the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'array_row'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure; a kept whole `object`/`array_row` also declares its scalar children as dotted `parent.child` sub-properties in `properties`.

package/skills/ae-data-integration/references/tracking-plan.md

@@ -10,15 +10,15 @@ Generate and confirm the event/property plan **before** any transform or upload.
 ## Sub-steps
 
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
-2. **Column → property draft** — identify system columns (time, `distinct_id`/`account_id`, event-name column, user-property-name column); map the remaining columns to event / user / common properties. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
-3. **Confirmation gate (single, one pass)** — present one summary table: event list + property list (uncertain types highlighted) + field scope (default: plan fields only, with a full-import switch) + unrecognized/dirty-data handling. The user answers once with ok or edits (renames, types, add/drop columns).
+2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
+3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
 4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
-5. **Persist the plan** — draft.json → xlsx → upload with `sdk_integration_mode=none`.
+5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 
 ## CLI
 
-`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out <draft.json>] [--dry-run]` converts the confirmed `ae-local-data-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
+`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out .ae-cli/data-integration/draft.json] [--dry-run]` converts the confirmed `ae-data-integration-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
 
 - `user_set` mode → no events; every mapping property becomes a user property.
 - `track` mode → one event per `--event-name` (or the mapping `default_event_name`); every property becomes an event property.
@@ -26,9 +26,11 @@ Generate and confirm the event/property plan **before** any transform or upload.
 - `exclude_columns` are dropped from the draft.
 - `desc` and `event_tag` flow from the mapping: each property's `desc` (falling back to its source column name) and each event's `event_meta.<name>.desc` / `event_meta.<name>.tag` (falling back to the source event name) are written into the draft. Fill them in the mapping so the plan is never empty.
 - The per-row event-name column cannot be enumerated without a full scan, so when `default_event_name` is absent the CLI requires `--event-name` for each concrete event name.
+- Duplicate `display_name`s across the property pool are auto-deduplicated (the property name is appended) before validation, so same-named source columns under different object paths never collide. Dotted `target`s default each sub-property's `display_name`/`desc` to its leaf segment, which also avoids collisions.
+- The draft is validated locally before any upload: an `object`/`array_row` property with no `parent.child` sub-property, or a sub-property whose type is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` (`--dry-run` fails too). Fix it in the mapping before running `plan`: a mapping `properties` entry with a dotted `target` (`user_info.name`) declares a `parent.child` sub-property whose parent is the `object`/`array_row` entry with the same first segment — so a kept whole container must carry one dotted entry per child (each scalar or `list`) next to the parent entry. A nested object/object-array that cannot stay nested must instead be flattened to scalar leaves: record one `flatten_rules` entry per leaf and one flat scalar (or `list`) `properties` entry per leaf (see [source-inspect.md](source-inspect.md)).
 - `--dry-run` previews events and properties without writing `draft.json`.
 
-Afterward reuse `ae-cli tracking plan draft --in draft.json --out draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
+Afterward reuse `ae-cli tracking plan draft --in .ae-cli/data-integration/draft.json --out .ae-cli/data-integration/draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
 
 ## Reuse
 

package/skills/ae-data-integration/references/transform.md

@@ -1,6 +1,6 @@
 # Transform — column → UE mapping
 
-Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first.
+Precondition: the tracking plan ([tracking-plan.md](tracking-plan.md)) has been generated and confirmed. A field mapping is not a tracking plan; if the plan is missing, return to tracking-plan.md first. The key system fields (steps 1–5 below) were already confirmed during the tracking-plan step (tracking-plan.md step 2); if nothing changed, state the confirmed values once and move to the property set — never skip a confirmation the user has not actually given.
 
 Read [UE mapping](ue-mapping.md) and apply these gates:
 
@@ -12,10 +12,10 @@ Read [UE mapping](ue-mapping.md) and apply these gates:
 Confirm the system fields with the user before touching properties. These are the top-level `#` fields, and users may not understand them, so explain each in plain language before asking; never infer a decision from a column name alone.
 
 1. **`#type` / mode.** Confirm the recommendation's `mode` (`track` / `user_set` / `mixed`). If `record_type_field` is set, confirm each distinct value normalizes to one of the eight record types. Ask whether the data reports events (→ `track`) or modifies user profiles (→ `user_set` or another profile type).
-2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one.
+2. **`#account_id` and `#distinct_id` — ask together; at least one is required.** Explain both: `#account_id` identifies logged-in users (database `user_id`, phone, member ID); `#distinct_id` identifies anonymous visitors (device/cookie/visitor ID). Present every `identity_candidates` entry from the inspect result alongside the recommendation's `account_id_field`/`distinct_id_field` picks, and ask which column maps to `#account_id`, which to `#distinct_id`, and whether both apply. A column named `user_id` can be either an anonymous or a login ID — only the user knows. If a required identity column is missing, offer an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool`; never invent one. `identity_candidates` is name-matched only, so it misses identity columns with arbitrary names (`玩家ID`, `用户账号`, `player_name`); surface any column whose shape is identifier-like (high `unique_ratio`, low `missing_ratio`, a string or numeric key, not a time column) and ask the user what it is. When one file has multiple sheets/data-sets, cross-check identity columns across them before finalizing: a column that appears (or near-matches, e.g. `设备ID` vs `主设备ID`) in more than one sheet likely has the same business meaning, so present a per-sheet `#account_id`/`#distinct_id` view and ask whether the same column maps to the same system field everywhere — a device ID used as `#distinct_id` in the event sheet but left as a differently-named plain property in the profile sheet silently breaks anonymous-user linkage, so surface the mismatch and let the user decide rather than letting it pass silently.
 3. **`#time`.** Confirm the time column and the source timezone (an IANA name from the user/project context). Leave `time_format` unset unless auto-detection failed (US/EU ambiguous dates). Also confirm the data's `#zone_offset` — the whole-hour UTC offset AE applies to those times, emitted inside `properties` (not top-level). The property's value is a number, so ask the user in numeric terms ("is the offset 8, i.e. UTC+8?") and set `zone_offset_value` to that integer; when a column carries the offset per row, set `zone_offset_field` instead. The two are mutually exclusive.
 4. **`#event_name` (track only).** Confirm the event-name column, or a reviewed `default_event_name` when no column exists.
-5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip.
+5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip. `#ip` is event data only and must be a valid IPv4/IPv6 address (a private/LAN IP is kept but reported — AE cannot geolocate it); `#uuid` must be a standard 36-character UUID. A value that violates the spec is dropped from that row only (`INVALID_IP` / `INVALID_UUID`) — the row itself is kept. The program never auto-generates a `#uuid`.
 
 When an `#event_name`, `#account_id`, or `#distinct_id` column's values do not satisfy AE naming rules (pure Chinese, uppercase, spaces), do not stop — scan the distinct values, list them to the user, and ask for one AE-name replacement each; record the pairs in `value_mapping` (see [UE mapping](ue-mapping.md)). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded. The same mechanism applies to a property column via that entry's own `value_mapping`.
 
@@ -24,18 +24,18 @@ Then confirm the property set with the user before saving the mapping. Present e
 Walk the table item by item:
 
 1. **Auto-renamed targets.** Recommendation sanitizes column names (camelCase split, illegal chars → `_`, digit-leading → `field_`, nothing recognizable → `field_N`). Show each source → target; ask for a manual name for every `field_N` fallback and rewrite the `target`.
-2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`). ID-like columns stay `string` even when numeric; `object`/`list` columns carry `transform: 'json'`. A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
+2. **Types.** Confirm each inferred type (`number`/`string`/`boolean`/`datetime`/`list`/`object`/`array_row`). ID-like columns stay `string` even when numeric; `object`/`array_row`/`list` columns carry `transform: 'json'`. A scalar array is `list` (→ AE `array_string`); an object array is `array_row` (→ AE `array_row`). A type is locked on first receipt, so review it before committing. If the user changes a type, warn how many rows would fail coercion — `convert` reports each failing row with its error code, so run it with the tentative mapping and confirm the quarantined count before finalizing.
 3. **Exclusions.** Ask which columns to drop and record them as `exclude_columns`.
 
 Before saving, present the complete mapping for a final sign-off on one grouped page — system fields (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#zone_offset`, `#event_name`, `#ip`/`#uuid`), then the property table (source column → target AE name → type), then excluded columns — and wait for an explicit yes. Never dump the raw mapping JSON at the user.
 
-Save the reviewed `ae-local-data-mapping/v1` JSON locally. Convert into a new output directory:
+Save the reviewed `ae-data-integration-mapping/v1` JSON to `.ae-cli/data-integration/mapping.json` (project workspace). Convert into a new output directory:
 
 ```bash
 ae-cli data-integration convert \
   --input-file '<path>' \
-  --mapping '<mapping.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --mapping '.ae-cli/data-integration/mapping.json' \
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
 For multiple files, repeat `--input-file` and pass a wildcard mapping plus `--type-resolutions` when inspect reported conflicts:
@@ -45,10 +45,10 @@ ae-cli data-integration convert \
   --input-file '<a.csv>' --input-file '<b.csv>' \
   --mapping '<wildcard-mapping.json>' \
   --type-resolutions '<resolutions.json>' \
-  --output-dir '.ae-cli/data-integration/<run-id>'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>'
 ```
 
-The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine.
+The command never modifies the source. Inspect `manifest.json`; summarize valid and quarantined counts and the block reason. If `manifest.output.skipped_fields` is present, tell the user how many `#ip`/`#uuid` values were invalid and dropped (the rows were otherwise kept); if `manifest.output.lan_ip_records` is present, tell the user that many `#ip` values are private/LAN addresses that AE cannot geolocate. Neither blocks the manifest. If `manifest.output.flatten_misses` is present — or a stderr `Warning: flatten rule "X" did not materialize for N row(s).` fires — a `flatten_rules` path missed some rows: re-check the dot path against the source shape (or confirm the column is legitimately optional); the rows are otherwise kept. A stderr `Warning: … column count different from the header row …` means ragged CSV/TSV rows were tolerated (extra fields dropped, missing fields treated as empty) — surface it as a data-quality note. A blocked manifest whose reason is `The source contained no data rows.` means the file had zero data rows; do not re-run the same command on it. Do not expose rows from `invalid.rows.jsonl` unless the user specifically asks to inspect the local quarantine. For the full failure taxonomy and how to respond, see [error handling](error-handling.md).
 
 ## Re-report only the failed rows (salvage loop)
 
@@ -59,7 +59,7 @@ ae-cli data-integration convert \
   --input-file '<same-source>' \
   --mapping '<fixed-mapping.json>' \
   --salvage-from '<run-dir>/invalid.rows.jsonl' \
-  --output-dir '.ae-cli/data-integration/<run-id>-salvage'
+  --output-dir '.ae-cli/data-integration/runs/<run-id>-salvage'
 ```
 
 This emits a `valid.ue.jsonl` containing only the rows that now pass, plus a new `invalid.rows.jsonl` with whatever still fails. Fixes are rarely one-shot, so repeat: feed each round's `invalid.rows.jsonl` into the next `--salvage-from` until no rows fail or the user stops. Each round's `valid.ue.jsonl` is disjoint from earlier rounds', so upload each round independently (same confirmation and `--allow-clean-subset` gates as a normal upload). `--salvage-from` is single-file only and the source must be the same file that produced the quarantine.

package/skills/ae-data-integration/references/ue-mapping.md

@@ -1,12 +1,12 @@
 # UE mapping contract
 
-The mapping version is `ae-local-data-mapping/v1`.
+The mapping version is `ae-data-integration-mapping/v1`.
 
 Required structure:
 
 ```json
 {
-  "version": "ae-local-data-mapping/v1",
+  "version": "ae-data-integration-mapping/v1",
   "source": {
     "sha256": "<source-sha256>",
     "format": "csv",
@@ -54,10 +54,25 @@ Top-level `#` fields map from named source columns. The user confirms each mappi
 | `#account_id` | Login/account ID (database `user_id`, phone, member ID) — identifies authenticated users | At least one of `#distinct_id` / `#account_id` |
 | `#time` | Event/profile occurrence time; AE bins data by it | Yes |
 | `#event_name` | What the user did (`purchase`, `login`) | Only for `track` |
-| `#ip` | Client IP; AE resolves geo from it | No |
-| `#uuid` | Short-window deduplication ID | No |
+| `#ip` | Client IP; AE resolves geo from it. Event data only | No |
+| `#uuid` | Standard 36-character UUID uniquely identifying the record (dedup). Both event and user data | No |
 
-`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined.
+`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined. `#zone_offset` is event data only — user profile records never carry it.
+
+## Value specifications
+
+Each mapped system field carries a value spec, enforced at both inspect (warnings) and convert (row handling). A row error quarantines the whole row; a field skip drops only that field and keeps the row.
+
+| Field | Value spec | On violation (convert) |
+| --- | --- | --- |
+| `#account_id` / `#distinct_id` | Non-empty string, at most 128 characters | Row error `MISSING_USER_ID` (absent) / `USER_ID_TOO_LONG` (>128) |
+| `#event_name` | `^[a-z][a-z0-9_]{0,49}$` (lowercase snake_case, letter-leading, ≤50 chars) | Row error `INVALID_EVENT_NAME` |
+| `#time` | One of the supported formats, within 3 years back / 3 days forward | Row error `INVALID_TIME` / `TIME_OUT_OF_RANGE` |
+| `#ip` | Valid IPv4 or IPv6. Event data only | Field skip `INVALID_IP`; a private/LAN IP is kept and reported — AE cannot geolocate it |
+| `#zone_offset` | Integer -12..14 (or an IANA name for `zone_offset_value`). Event data only | Row error `INVALID_ZONE_OFFSET` |
+| `#uuid` | Standard 36-character UUID (`xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx`). Both data kinds | Field skip `INVALID_UUID` |
+
+A field skip is not a row failure: the row is kept with its other fields, and the count is reported in `manifest.output.skipped_fields` so the agent tells the user at the end. `#uuid` is never auto-generated — it comes only from a mapped source column.
 
 Mapping keys: `account_id_field`, `distinct_id_field`, `time.field`, `event_name_field` (or a reviewed `default_event_name`), `ip_field`, `uuid_field`. Inspect surfaces every identity-shaped column in `identity_candidates` (name, `account`/`distinct` kind, unique and missing ratios) so the agent can present all candidates for confirmation. When the required identity column is absent, use an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool` — a user decision, never invented.
 
@@ -86,16 +101,20 @@ All fields below are optional and are explicit user decisions — never invent t
 | `account_id_value` / `distinct_id_value` | `string` | Fixed placeholder identity (≤128 chars). Applies to every row when the corresponding `*_field` is absent; when the field exists, fills only the rows whose column is empty |
 | `random_pool` | `{ account_ids?: string[], distinct_ids?: string[] }` | Synthesizes a random identity when the source field is absent |
 | `exclude_columns` | `string[]` | Source columns skipped when building properties |
-| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it |
+| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV/Excel paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it. `flatten_rules` only materializes the out column in the row; it does **not** emit it — add a `properties` entry with `source` = the out column to actually produce the property. An out column that fails to materialize for some rows (missing path, non-JSON cell, or a path without a `<column>.` prefix) is counted per out-column in `manifest.output.flatten_misses` and named in a convert stderr warning — fix the path or confirm the column is legitimately optional |
 | `headers` | `string[]` | User-confirmed column names for a headerless file; presence means the first row is data. Never use inspect's `col_1..col_N` placeholders — infer names from each column's values, confirm them with the user, then write them here |
 | `missing_time` | `'now'` | Fill a missing/empty `#time` with the current time, for user-profile rows only (explicit user decision; track rows are never filled) |
-| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo) |
-| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (short-window deduplication ID) |
+| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo). Event data only; a value that is not valid IPv4/IPv6 is dropped from that row, and a private/LAN IP is kept but reported |
+| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (standard 36-character UUID, both data kinds). A non-UUID value is dropped from that row |
 | `zone_offset_value` | `number` (integer -12..14) or IANA `string` | Emits the fixed `#zone_offset` preset property inside `properties`. An IANA name resolves to its integer UTC offset |
 | `zone_offset_field` | `string` | Source column whose per-row integer value (-12..14) is emitted as `#zone_offset`; missing/non-integer rows are quarantined. Mutually exclusive with `zone_offset_value` |
 | `event_meta` | `{ <event-name>: { desc?: string, tag?: string } }` | Per-event business description and `event_tag` for the tracking plan, keyed by AE event name. Inferred from the data and user context; the user supplies anything not inferable — never leave them empty |
 
-Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`list`) whose values arrive as JSON text — JSON-encoded CSV cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`array_row`/`list`) whose values arrive as JSON text — JSON-encoded CSV/TSV/Excel cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+
+A `properties` entry whose `target` is dotted (`parent.child`) is a plan-only sub-property declaration: `parent` must name an `object`/`array_row` entry in the same mapping, the child must be scalar or `list`, and the child's `source` must equal the parent's `source` — the nested value lives inside the parent, which conversion emits once and never reads the child's column. A kept whole `object`/`array_row` therefore carries one dotted entry per child alongside the parent entry.
+
+Property types are `string`, `number`, `boolean`, `datetime`, `list`, `object`, and `array_row`. `list` is a scalar array (`["a","b"]`, → AE `array_string`); `object` is a one-to-one object (`{...}`); `array_row` is an object array (`[{...}]`, one-to-many). A mapping expresses an object's/object array's sub-properties as dotted `target`s (`user_info.name`, `orders.sku`); `data-integration plan` maps them one-to-one to `parent.child` plan properties (each child's `display_name`/`desc` default to its leaf segment). In the tracking plan, `object` and `array_row` properties must each have at least one `parent.child` sub-property, and every sub-property must itself be scalar or `array_string` — a nested object/object-array cannot stay nested and must be flattened further to scalar leaves. `list` (→ `array_string`) is a leaf and takes no children. `ae-cli data-integration plan` enforces this locally: a composite property with no children, or a child that is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` before any upload.
 
 ## Time formats
 
@@ -130,13 +149,16 @@ Values with an explicit offset or `Z` are parsed by the JavaScript `Date` constr
 - `#zone_offset`, when set, is a whole-hour integer in -12..14 (or an IANA name/column that resolves to one) and is emitted inside `properties`, never at the top level.
 - Event/property names are lowercase snake_case, begin with a letter, and are at most 50 characters.
 - Target property names are unique and do not collide with UE system fields.
-- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, or `object`.
+- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, `object`, or `array_row`.
 - Text is at most 2 KB; numbers stay within -9E15..9E15.
 - Lists contain at most 500 strings (255 bytes each) or 500 objects.
 - Objects contain at most 100 legal sub-properties; nested values follow the same type limits.
+- No `object`/`array_row` survives into the tracking plan: AE requires each to have a `parent.child` sub-property (scalar or `array_string` only), and a mapping cannot express one, so nested objects/object-arrays are flattened to scalar leaves first (a kept-whole container fails `data-integration plan` fast).
 - A conversion rule does not hide a real type conflict.
 - Event/profile times fall within the receiver window: previous 3 years through next 3 days.
-- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`).
+- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`). `#uuid` is never auto-generated.
+- `#uuid` values are standard 36-character UUIDs and `#ip` values are valid IPv4/IPv6; invalid ones are dropped from the row only (`skipped_fields`), never quarantined.
+- `#ip` and `#zone_offset` appear on `track` records only; user data carries neither.
 - `value_mapping`, `random_pool`, and fixed `account_id_value`/`distinct_id_value` came from an explicit user decision and match the actual distinct values/columns.
 
 `user_set` output for the same identity is ordered by time so receiver application order is deterministic. Conversion applies whole-row quarantine: any error on a row — identity, time, event, record type, or a single property (type coercion, size/limit) — drops the entire row. The row is written to `invalid.rows.jsonl` with its error codes, counted in `manifest.output.invalid_records`, and the manifest is blocked until reviewed. The failed rows are re-reportable without re-sending valid rows: fix the mapping, then run `convert --input-file <same-source> --mapping <fixed-mapping> --salvage-from <invalid.rows.jsonl>`. The salvage run re-processes only the listed row numbers against the same source, emits a `valid.ue.jsonl` containing only the newly fixed rows, and writes a new `invalid.rows.jsonl` with whatever still fails — so the loop repeats (feeding each round's `invalid.rows.jsonl` into the next `--salvage-from`) until no rows fail or the user stops.

package/skills/ae-engage/references/build-task-save-guide.md

@@ -229,6 +229,8 @@ When experiment mode is enabled (`context.enableExp=true` or draft `expConfig.en
 - capability `engage-task.task.build-save-guide` enriches `handoff.reqTemplate.channelConfig.groupContentList`
   so each entry carries `expGroupName`, `expGroupType`, `percentageInExperiment`, `order`, and `contentList`
 - do not drop those association fields when filling content; they must stay aligned with `expConfig.expGroupList`
+- copy the complete group tuple (`expGroupName`, `expGroupType`, `percentageInExperiment`, `order`)
+  into both lists; matching only by list position is not sufficient
 
 ### 4.8 `fieldRules`
 
@@ -300,6 +302,11 @@ Apply this rule only to
 `completionIndicatorDef.completionIndicators[].eventDefinition.filters`. Trigger-event filters have
 their own scenario rules and are not subject to this completion-filter restriction.
 
+For `completionIndicatorType=0`, also read and preserve `requiredMainGoalFields` and
+`touchCycleRule`. A valid main goal includes `touch_cycle_num` and `touch_cycle_num_unit`; use `1`
+and `day` when no custom completion window is requested. Do not rely on static `--validate` alone
+because the save service performs this additional business validation.
+
 ### 4.9 `handoff`
 
 This is the final section before `save_task`.
@@ -336,6 +343,8 @@ Recommended usage pattern:
    definition directly
 7. omit `clientConfig.clientQp`; partial updates preserve the server-authored value
 8. call `engage-task task save`
+9. after an experiment save succeeds, call `engage-task task get` and verify that both
+   `exp_config.exp_group_list` and `group_content_list` contain the expected group tuples
 
 ---