@thinkingai/ae-cli 6.0.45 → 6.0.46

package/skills/ae-data-integration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-data-integration
-description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it."
+description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it. Trigger words: 本地数据导入 / 离线数据 / 数据文件 / 文件导入 / 文件上报 / CSV 导入 / Excel 导入 / TSV 导入 / JSON 导入 / 导入到 AE / 导入到 ThinkingData / local data import / import local file."
 ---
 
 # AE Data Integration
@@ -23,16 +23,35 @@ Two entrances lead here: the AE Agent dialog (attach / plus-button upload) and `
 - If a batch times out or loses the network, treat that batch as unknown. Stop. Ask the user to verify receiver/AE data before the user chooses `--resume-from`; never resume automatically.
 - Local analysis stays local. AE Agent attachment is a separate, confirmed branch with a 50 MB per-file limit.
 
+## When to use / When NOT to use
+
+Use this skill when the user wants to bring a **local data file** (CSV/TSV/TXT/JSON/JSONL/XLS/XLSX) into AE, or analyze such a file locally without uploading.
+
+| User intent | Use this instead |
+| --- | --- |
+| How to integrate the SDK / tracking code / LogBus2 config / reporting-error triage (usage Q&A, no local file) | ae-data-integration-helper |
+| Database / datasource direct sync (MySQL, DataX, data warehouse, data-dev platform) | ae-dataops |
+| Community content (posts / comments / chat / WeCom groups) insight or submission | ae-community |
+| Generate / upload a project-level tracking plan (source material is PRD / chat / template / code; deliverable is a real platform tracking plan) | ae-generate-tracking-plan |
+| Upload documents / URLs to a knowledge base | ae-kb |
+| Reports / dashboards / queries / governance on data already in AE | ae-analysis |
+
+This skill also produces a tracking-plan draft (`source_type: data`) as a governance prerequisite; that draft is an input to ae-generate-tracking-plan, not a substitute for its five-phase platform plan.
+
 ## Workflow
 
 Walk the four submodules in order. Each submodule is its own reference; follow it and come back here for the next step.
 
 1. **Source — business identification.** Read [references/source-inspect.md](references/source-inspect.md). Profile every file fully, infer its business meaning using business-doc / user-prompt priors, then pick a branch via [references/ue-routing.md](references/ue-routing.md).
-2. **Reuse check.** If a `.ae-data-integration/index.json` exists and the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
-3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). Generate the event/property plan from the profile and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
+2. **Reuse check.** If the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. `reuse` searches the current directory's `.ae-cli/data-integration/` upward, then `~/.ae-cli/data-integration/`, so a package written elsewhere is still found. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
+3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). The plan is generated from the mapping (`plan --mapping`), so confirm the recommended mapping's key system fields with the user first — `mode`, `#account_id`/`#distinct_id`, `#time` + timezone, `#event_name`, `#ip`/`#uuid` (see [references/transform.md](references/transform.md) steps 1–5) — then generate the event/property plan and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
 4. **Transform.** Read [references/transform.md](references/transform.md). Map columns to AE system fields and properties, convert, and quarantine dirty rows per [references/ue-mapping.md](references/ue-mapping.md).
 5. **Sink — upload.** Read [references/sink-upload.md](references/sink-upload.md). Resolve the destination, dry-run, confirm, then upload per [references/sync-json-upload.md](references/sync-json-upload.md). `receiver_accepted` is not persistence: after a ~1-minute ingestion delay, verify the data landed with ae-cli (`tracking live-data list` / `tracking ingest summary` / `tracking ingest-error list`) rather than telling the user to check the console.
-6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (frozen mapping + transform script + plan reference) so the next same-shape file skips the full pipeline.
+6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (pipeline descriptor + frozen mappings + stage executors + docs) and a shareable zip; in the completion response, state the absolute zip path, the package directory, and the one-line way to run the next same-shape file.
+
+## Error handling
+
+When a step fails, classify the failure before acting — see [references/error-handling.md](references/error-handling.md). A quarantined row, a ragged line, and a disk-full are three different problems with three different responses: match on the error `code`, never retry a parse failure by guessing the encoding, and never report a program failure as a data problem.
 
 ## Local analysis branch
 

package/skills/ae-data-integration/references/custom-layer.md

@@ -0,0 +1,93 @@
+# Project custom layer (verify + salvage overlays)
+
+The handoff package is deliberately **standard flow only**: it ships the generic
+stage executors and never embeds environment-specific workarounds such as SQL
+direct queries or multi-round salvage loops. Some projects need those — a hard
+per-event persistence judge, or a repeated salvage loop for a dirty source. Add
+them as a **custom layer next to the package**, never by editing the generic
+package itself.
+
+## When to add a layer
+
+- **Hard persistence judge.** `bin/verify.py` is a soft check: it computes the
+  submit window and expected counts from the local UE output, then shows the
+  `tracking ingest summary` payload before and after for comparison. It does not
+  parse that payload into per-event counts — the capability's `data` shape is
+  server-defined, and a shared project's window delta cannot be attributed to one
+  import. When the project requires an automated per-event ✓/✗ verdict, overlay a
+  SQL layer that reads the project's event/user tables directly.
+- **Multi-round salvage.** `bin/run.sh` already prints a single-round salvage
+  hint when `invalid.rows.jsonl` is non-empty. For sources that fail in layers,
+  wrap that command in a loop that re-feeds each round's quarantine into the next
+  `--salvage-from` until nothing fails or the user stops.
+
+## Rules
+
+1. Put overlays in a sibling directory (e.g. `custom/`) under the package root,
+   and reference `../pipeline.json` / `../index.json` — never modify the frozen
+   mappings, `bin/`, or the generated docs.
+2. Keep every existing confirmation gate. A custom layer may add gates; it must
+   not remove the `--confirm` upload gate or the plan/shape gates.
+3. Keep secrets in `.local/target.env`. Never write APPID, endpoints, tokens, or
+   raw data values into an overlay file.
+4. An overlay is project-specific. Do not copy it into another project's package
+   without re-confirming the environment assumptions (table names, APPID, host).
+
+## Skeleton 1 — SQL verify (hard per-event judge)
+
+The reference deployment queries the ingested tables directly. **Table and
+column names vary by deployment** (`v_event_1` / `v_user_1` and `$part_date` /
+`$part_event` are examples only) — confirm them against the project's receiver
+schema before use, and keep the query read-only.
+
+```bash
+# custom/verify.sh — hard judge; requires the SQL capability to be available.
+# usage: custom/verify.sh <run-dir> [--baseline|--check]
+# Reuses bin/verify.py's window/count computation idea, but answers the question
+# "did exactly these events land?" with a direct count over the event table.
+```
+
+The query shape (adapt table/column names):
+
+```text
+select <event_column>, count(*)
+from <event_table>
+where <date_partition_column> between '<window-start>' and '<window-end>'
+group by 1
+```
+
+Compare the per-event delta (after upload minus before upload) against the
+expected counts from `valid.ue.jsonl`. User-profile rows are overwrite-writes,
+so a zero user-table delta is normal; compare event rows per event, not totals.
+
+Keep this layer read-only and non-blocking: report the verdict, never retransmit
+automatically on a mismatch — first check `tracking ingest-error list` for the
+silent-drop reason.
+
+## Skeleton 2 — multi-round salvage loop
+
+`bin/run.sh` prints the one-round hint. Wrap it into a loop that re-processes each
+round's quarantine against the fixed mapping until clean or the user stops:
+
+```bash
+# custom/salvage.sh <run-dir> — re-process quarantined rows round by round.
+# Each round's valid.ue.jsonl is disjoint from earlier rounds', so upload each
+# round independently (same --confirm and --allow-clean-subset gates).
+round=1
+while [ -s "<run-dir>/invalid.rows.jsonl" ]; do
+  ae-cli data-integration convert \
+    --input-file '<same-source>' \
+    --mapping '<fixed-mapping.json>' \
+    --salvage-from "<run-dir>/invalid.rows.jsonl" \
+    --output-dir "<run-dir>-salvage-$round"
+  round=$((round + 1))
+  # stop condition is the user's: a row that still fails after a fix is a real
+  # data defect, not a code bug.
+done
+```
+
+See [transform.md](transform.md) for the `--salvage-from` semantics (single-file
+only; the source must be the same file that produced the quarantine) and
+[handoff.md](handoff.md) for the package the overlay attaches to. Persistence
+verification always follows [sink-upload.md](sink-upload.md): `receiver_accepted`
+is not durability.

package/skills/ae-data-integration/references/error-handling.md

@@ -0,0 +1,92 @@
+# Error handling
+
+Failures in the data-integration pipeline split into three classes. Each class has a distinct
+response; an agent that mixes them up wastes retries and misleads the user.
+
+| Class | What it is | Response |
+| --- | --- | --- |
+| Abnormal data | Rows or fields that do not meet UE rules | Quarantine the row or skip the field, report counts, salvage |
+| File parsing exception | The source cannot be read as its detected format | Surface the parse error, do not retry by guessing; fix the file or format |
+| Program execution exception | The program itself failed (disk, permissions, output dir, mapping mismatch) | Surface the exact error, fix the environment/inputs, re-run |
+
+Every error leaves the CLI as the standard envelope `{ ok, data, error: { type, code, message, hint } }`
+(JSON on stdout; progress and warnings on stderr). The `code` is the stable identifier an agent
+branches on; `hint` is human-readable guidance.
+
+## Class 1 — Abnormal data
+
+Happens inside `convert`. Two severities:
+
+- **Whole-row quarantine.** Any error on a row — identity, time, event name, record type, or a
+  single property (type coercion, size/limit) — drops the entire row. The row is written to
+  `invalid.rows.jsonl` with its error codes and counted in `manifest.output.invalid_records`; the
+  manifest becomes `blocked`. Row-level codes: `MISSING_USER_ID`, `USER_ID_TOO_LONG`,
+  `INVALID_RECORD_TYPE`, `INVALID_TIME`, `TIME_OUT_OF_RANGE`, `INVALID_EVENT_NAME`,
+  `INVALID_ZONE_OFFSET`, `PROPERTY_TYPE_CONFLICT`, `PROPERTY_LIMIT_EXCEEDED`.
+- **Field skip.** A `#ip` or `#uuid` value that violates its spec drops only that field and keeps
+  the row. Counts land in `manifest.output.skipped_fields`; private/LAN IPs are kept but counted in
+  `manifest.output.lan_ip_records`. Field-skip codes: `INVALID_IP`, `INVALID_UUID`. Never quarantined.
+
+Responses:
+
+- A blocked manifest is not a failure to retry blindly. Fix the mapping, then re-run `convert
+  --salvage-from <invalid.rows.jsonl>` to re-process only the quarantined rows (never re-send the
+  rows that already passed). Repeat the salvage loop until no rows fail or the user stops.
+- An empty source (zero data rows) blocks with the reason `The source contained no data rows.` —
+  a distinct, clearer message than a failed validation run. Do not re-run the same command on the
+  same empty file expecting a different result.
+- Ragged delimited rows (a CSV/TSV row whose column count differs from the header) are tolerated:
+  extra fields are dropped and missing fields are treated as empty, and a stderr warning reports
+  how many rows were ragged. A row that becomes missing its identity/time because a field was
+  absent is then quarantined normally. Treat the warning as a data-quality signal for the user.
+
+## Class 2 — File parsing exceptions
+
+The source cannot be parsed as its detected format. The agent's job is to report precisely and let
+the user decide — never to guess an encoding or structure and retry silently.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_INPUT_NOT_FOUND` | Path is not a readable file | Ask for the correct path |
+| `LOCAL_DATA_FILE_TOO_LARGE` | XLS over 1 GB | Convert to XLSX or split the workbook |
+| `LOCAL_DATA_INPUT_INVALID` | Generic parse failure (malformed CSV/TSV/JSON/XLS) | Verify encoding and structure, then retry without changing the source |
+| `LOCAL_DATA_JSONL_INVALID` | A JSONL line is not valid JSON (`location.record` names the line) | Point at the offending record |
+| `LOCAL_DATA_JSON_ROOT_INVALID` | JSON root is not an object or array | Check the file's top-level shape |
+| `LOCAL_DATA_XLSX_INVALID` | Workbook metadata missing / no readable sheets / worksheet entry missing | Re-export the workbook |
+| `LOCAL_DATA_SET_NOT_FOUND` / `LOCAL_DATA_SET_REQUIRED` | Sheet or JSON Path not found / ambiguous | Ask which `--data-set` to use |
+
+A parse error is never a reason to change the mapping or the tracking plan. Report the code and
+hint verbatim, and ask the user to fix the file (re-export, re-encode, or split).
+
+## Class 3 — Program execution exceptions
+
+The program itself failed; the source data is usually fine. These must never be mislabeled as
+parse errors: a per-row callback failure (for example a disk that filled up mid-convert) propagates
+as itself, not as `LOCAL_DATA_INPUT_INVALID`.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_OUTPUT_NOT_EMPTY` | The output directory must be new or empty | Point convert at a fresh `<run-id>` directory |
+| `LOCAL_DATA_SOURCE_CHANGED` / `LOCAL_DATA_SOURCE_FORMAT_CHANGED` | The source no longer matches the mapping fingerprint/format | Re-run inspect and review a new mapping |
+| `LOCAL_DATA_MAPPING_INVALID` / `LOCAL_DATA_MAPPING_INVALID_JSON` / `LOCAL_DATA_MAPPING_NOT_FOUND` | The mapping cannot be read or validated | Re-read the mapping reference, fix the mapping file |
+| `LOCAL_DATA_SALVAGE_INVALID` / `LOCAL_DATA_SALVAGE_EMPTY` / `LOCAL_DATA_SALVAGE_NO_MATCH` | The salvage file is not a valid quarantine file, is empty, or lists no rows from this source | Point at the correct `invalid.rows.jsonl` from the same source |
+| `LOCAL_DATA_TYPE_CONFLICTS_UNRESOLVED` / `LOCAL_DATA_TYPE_RESOLUTIONS_INVALID` | Cross-file column type conflicts need explicit resolutions | Build `--type-resolutions` |
+| `LOCAL_DATA_PLAN_INVALID_EVENT_NAME` / `LOCAL_DATA_PLAN_EVENT_NAMES_REQUIRED` / `LOCAL_DATA_PLAN_INVALID_LANG` | Tracking-plan draft inputs are invalid | Fix the event name(s) or `--lang` |
+| `LOCAL_DATA_HANDOFF_INDEX_INVALID` / `LOCAL_DATA_HANDOFF_PLAN_NOT_FOUND` / `LOCAL_DATA_HANDOFF_PLAN_INVALID` | Handoff index/plan cannot be read | Point at a valid handoff directory/plan file |
+| `LOCAL_DATA_ENDPOINT_INVALID` / `LOCAL_DATA_APPID_INVALID` / `LOCAL_DATA_BATCH_SIZE_INVALID` / `LOCAL_DATA_COMPRESS_INVALID` / `LOCAL_DATA_RESUME_INVALID` / `LOCAL_DATA_RESUME_OUT_OF_RANGE` | Upload arguments are invalid | Fix the flag before uploading |
+| `LOCAL_DATA_MANIFEST_INVALID` / `LOCAL_DATA_UE_FILE_INVALID` / `LOCAL_DATA_UE_FILE_NOT_FOUND` / `LOCAL_DATA_UE_FILE_CHANGED` / `LOCAL_DATA_UE_COUNT_MISMATCH` / `LOCAL_DATA_MANIFEST_FILE_MISMATCH` | Upload preconditions fail | Re-check the manifest and UE file pairing |
+| `LOCAL_DATA_CLEAN_SUBSET_CONFIRMATION_REQUIRED` | Uploading from a blocked manifest needs a separate clean-subset decision | Confirm the subset explicitly before `--allow-clean-subset` |
+
+Write failures (disk full `ENOSPC`, permission denied `EACCES`) do not hang the command: the
+output streams fail with a clear `Failed to write "<path>"` message telling the user to check disk
+space and directory permissions. Fix the environment, then re-run into a fresh output directory.
+
+## Cross-cutting rules
+
+- Classify first, act second. Match on `code`, not on message text.
+- A parse error is not a data problem; a program error is not a parse error. Do not conflate them
+  when reporting back to the user.
+- Never retry a parse failure by guessing the encoding, delimiter, or header layout. Show the
+  error and ask.
+- After any fix, re-run from the start of the step that failed; do not resume a half-written run
+  or reuse a partially populated output directory.

package/skills/ae-data-integration/references/handoff.md

@@ -1,43 +1,102 @@
 # Handoff (reusable package)
 
-After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package freezes the transform logic (the confirmed mapping), the runnable transform script, and the tracking-plan reference. It does **not** re-freeze the raw data or any upload secrets.
+After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package is a DataX-style pipeline — a declarative `source → transform → sink` descriptor plus generic stage executors that dispatch to `ae-cli data-integration` subcommands. It freezes the transform logic (the confirmed mappings) and the tracking-plan reference, and ships a `bin/` directory that a human or agent can run directly. It never re-freezes raw data or upload secrets.
 
 ## When
 
-Run handoff after Transform — or after Sink — once the mapping is confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
+Run handoff after Transform — or after Sink — once the mappings are confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
 
 ## CLI
 
 ```
-ae-cli data-integration handoff --mapping <mapping> [--plan-file <draft.json>] [--out-dir .ae-data-integration]
+ae-cli data-integration handoff --mapping <mapping> [--mapping <mapping2> ...] [--plan-file <draft.json>] [--pushurl <url>] [--project-id <id>] [--out-dir .ae-cli/data-integration]
 ```
 
-- `--mapping` — the confirmed `ae-local-data-mapping/v1` mapping (the frozen transform logic, including `value_mapping` and `flatten_rules`).
-- `--plan-file` — optional tracking-plan `draft.json` to reference inside the package (`plan.json`).
-- `--out-dir` — handoff root. Default `.ae-data-integration/` (travels with the project).
-- `--dry-run` previews the fingerprint, files, and index path without writing.
+- `--mapping` — one or more confirmed `ae-data-integration-mapping/v1` mappings (the frozen transform logic, including `value_mapping` and `flatten_rules`). Repeat it for multi-sheet workbooks: one mapping per sheet.
+- `--plan-file` — optional tracking-plan `draft.json` to reference inside each mapping directory (`plan.json`).
+- `--pushurl` — optional receiver base URL to record as the reuse upload target (the sink endpoint is `pushurl` + `/sync_json`). Record it when the next same-shape file will most likely land at the same receiver.
+- `--project-id` — optional numeric destination project ID to record; `bin/upload.sh` derives the APPID from it via `project info get` at upload time.
+- `--out-dir` — handoff root. Default `.ae-cli/data-integration/` (project workspace; travels with the project).
+- `--dry-run` previews the fingerprints, target, file list, and zip path without writing.
 
 ## Package layout
 
 ```
-.ae-data-integration/
-  index.json               ← shared fingerprint index (reuse detection)
-  <fingerprint[:16]>/
-    mapping.json           ← frozen mapping (re-runnable by convert)
-    transform.mjs          ← node transform.mjs <new-file> [<output-dir>]
-    plan.json              ← optional tracking-plan reference
+.ae-cli/data-integration/    ← handoff root (= out-dir, project workspace)
+  pipeline.json             ← source → transform → sink descriptor (ae-data-integration-pipeline/v1)
+  index.json                ← shared structure-fingerprint index (reuse detection)
+  shape.json                ← per-mapping column baseline (shape gate)
+  <fingerprint[:16]>/       ← one directory per mapping
+    mapping.json            ← frozen mapping
+    transform.mjs           ← node transform.mjs <new-file> [<output-dir>]
+    plan.json               ← optional tracking-plan reference
+  bin/                      ← generic stage executors (read pipeline.json)
+    run.sh                  ← source → transform → plan (never uploads)
+    upload.sh               ← sink (dry-run by default; --confirm uploads; resolves recorded target)
+    bind_mapping.py         ← shape check + rebind sha256/data_set to the new file
+    summarize.py            ← valid/quarantined counts
+    plan_check.py           ← tracking-plan coverage gate (exit 3 on new events/properties)
+    verify.py               ← soft persistence check (submit window vs ingest summary)
+    resolve_appid.py        ← APPID derivation helper (project info get)
+  README.md / RUNBOOK.md    ← how to run, the four gates, persistence verification
+  .local/target.env.example ← destination template (no real secrets)
+  .gitignore                ← inbox/ runs/ .local/target.env
+  inbox/  runs/             ← daily input / per-run outputs
 ```
 
-The `transform.mjs` wrapper shells out to `ae-cli data-integration convert`; it takes the new file path as its first argument, never bakes the source path, and re-stamps `source.sha256` with the new file's fingerprint before converting. Reuse it only for a file of the **same shape** (same header/schema and format) — the transform logic is frozen, but the content guard is re-bound to each specific file.
+A shareable archive is written next to the package root: `<parent>/ae-data-integration-handoff-<fingerprint[:8]>.zip`. The zip carries only this handoff round — the frozen mappings just written, a scoped `index.json` (just this round's entries), and the generic executors/docs — not the accumulated history, which stays in `.ae-cli/data-integration/` for reuse matching.
+
+## Pipeline descriptor
+
+`pipeline.json` declares the three stages and their types. Only `source: local_file` and `sink: restful_sync_json` are implemented today; the `type` fields reserve logbus / datax / mysql for later phases. `bin/run.sh` and `bin/upload.sh` read the descriptor and dispatch each stage by its `type` to `ae-cli data-integration inspect / convert / upload` — they are executors, not a second runtime engine.
+
+## Recorded destination
+
+`pipeline.json` → `sink.params` records `pushurl` and `project_id` when the handoff
+was run with those flags. Reuse defaults to that target, but **`bin/upload.sh`
+never sends without `--confirm`**, so the operator re-confirms the address and
+project on every reuse. Resolution order at upload time:
+
+- endpoint: recorded `pushurl` (+ `/sync_json`), else `AE_ENDPOINT`.
+- APPID: `AE_APPID`, else derived via `ae-cli project info get --project-id <id>`
+  (see `bin/resolve_appid.py`; set `AE_APPID` when that payload lacks `appid`).
+- project id: recorded `project_id`, else `AE_PROJECT_ID`.
+
+`project info get` returns `data.appid` at the top level (verified against the AE
+demo host); `bin/resolve_appid.py` reads that exact field and prints it, falling
+back to `AE_APPID` when the field is absent or not a non-empty string.
+
+## Project custom layers
+
+For projects that need a hard per-event SQL judge or a multi-round salvage loop
+(the reference package's environment-specific workarounds), overlay a custom layer
+next to the package instead of editing it — see [custom-layer.md](custom-layer.md).
+
+## Four confirmation gates
+
+The RUNBOOK and the scripts enforce four gates. The first two run automatically; the last two always need human confirmation:
+
+1. **Shape gate** — `bind_mapping.py` compares the new file's column set against `shape.json` and fails fast on a mismatch. A changed shape means the frozen logic was never reviewed for it: re-run the full pipeline.
+2. **Transform** — `ae-cli data-integration convert` per mapping; quarantined rows go to `invalid.rows.jsonl`, never silently dropped.
+3. **Tracking-plan gate** — `plan_check.py` verifies every produced event/property already exists in `plan.json`; new ones exit 3 and must be merged into the project plan first.
+4. **Sink gate** — `upload.sh` is dry-run by default; `--confirm` is the explicit upload decision.
 
 ## Structure fingerprint and index
 
-Every handoff records one entry in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: columns (source name + type), event model (`mode`, `event_name_field`, `record_type_field`), identity fields, and excluded columns. Business logic (`value_mapping`, transforms, `time_format`, and the fixed `default_event_name`) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+Every handoff records one entry per mapping in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model (`mode`). Business logic (`value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, the fixed `default_event_name`, and the system-field assignments) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+
+Reuse matching is its own step — see [references/reuse.md](reuse.md).
 
-Reuse matching is its own step — see [references/reuse.md](reuse.md). It compares a new file's profile structure against this index and proposes the matching package, which the user confirms before skipping the full pipeline.
+## Completion response
+
+After handoff succeeds, state the **absolute zip path**, the package directory, and the one-line way to run the next same-shape file:
+
+```
+cd <out-dir> && bin/run.sh <new-file>   # then bin/upload.sh runs/<run-id> --confirm
+```
 
 ## Safety rules
 
-- Treat the mapping, plan, generated artifacts, and the `.ae-data-integration/` directory as sensitive.
-- Never write APPID, endpoints, tokens, or raw data values into a handoff package. The package references the plan and the mapping; uploads still require an explicit, confirmed `upload` call.
+- Treat the mappings, plans, generated artifacts, the `.ae-cli/data-integration/` directory, and the zip as sensitive.
+- Never write APPID, tokens, or raw data values into a handoff package. The package records at most a destination `pushurl` and `project_id`; uploads still require an explicit, confirmed `upload` call, so the operator re-confirms the address and project each time.
 - Do not invent a mapping or plan. Handoff only packages what the user already confirmed.

package/skills/ae-data-integration/references/local-analysis.md

@@ -1,6 +1,6 @@
 # Local analysis
 
-Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/<run-id>/` with restrictive permissions.
+Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/runs/<run-id>/` with restrictive permissions.
 Set the directory to `0700` and generated scripts/reports to `0600`.
 
 ## Default report when no question is supplied

package/skills/ae-data-integration/references/reuse.md

@@ -4,21 +4,25 @@ Before walking the full pipeline for a new file, check whether a **handoff packa
 
 ## When
 
-Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-data-integration/index.json` exists. It is read-only and makes no writes.
+Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-cli/data-integration/index.json` exists. It is read-only and makes no writes.
 
 ## CLI
 
 ```
-ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-data-integration]
+ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-cli/data-integration]
 ```
 
 - `--mapping` — the candidate mapping, typically `inspect`'s `recommended_mapping` (the new file's structure, inferred exactly as Source would infer it).
-- `--out-dir` — handoff root. Default `.ae-data-integration/`.
+- `--out-dir` — handoff root. Optional. Without it, `reuse` searches in order: the current directory's `.ae-cli/data-integration/`, then each parent directory upward, then `~/.ae-cli/data-integration/` as a global fallback — so a package written in another directory or another session is still reachable.
 - `--dry-run` previews the fingerprint and match verdict without reading frozen packages.
 
+## Search paths
+
+When `--out-dir` is omitted, the result carries `searched_paths` — the exact `index.json` paths probed, in order. A `matched: false` result with a non-empty `searched_paths` means all of them were checked; a missing global fallback (no `$HOME`) simply omits it from the list.
+
 ## Matching
 
-The command computes the same **structure fingerprint** the handoff index is keyed on — columns (source name + type), event model, identity fields, and excluded columns — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`) is not part of the key, so a same-shape file with different content or a different file name still matches.
+The command computes the same **structure fingerprint** the handoff index is keyed on — the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, and the system-field assignments) is not part of the key, so a same-shape file with different content or a different file name still matches.
 
 ## Result
 
@@ -26,7 +30,7 @@ The command computes the same **structure fingerprint** the handoff index is key
 - **Match** (`matched: true`): the result carries the matched package — `mapping_file`, optional `plan_file`, the frozen `default_event_name` (so the user sees which event name will be reused), and a `run` command:
 
 ```
-node .ae-data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
+node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
 ```
 
 ## Confirmation gate

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,7 +31,7 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
-Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type, grouped into event properties (`track`) and user properties (profile modes) — never a counts-only summary. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
+Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.
 

package/skills/ae-data-integration/references/source-inspect.md

@@ -39,7 +39,9 @@ If `selection_required=true`, show only the Sheet/JSON Path candidates and ask t
 ae-cli data-integration inspect --input-file '<path>' --data-set '<candidate-id>' --source-timezone '<iana-timezone>'
 ```
 
-Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. Read [UE routing](ue-routing.md) before choosing a branch.
+Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. IP- and UUID-named columns are additionally checked against their value specs: inspect warns how many non-empty values are invalid IPv4/IPv6, private/LAN IPs, or non-UUID strings, so the user can decide whether to map them as `ip_field`/`uuid_field`. Read [UE routing](ue-routing.md) before choosing a branch.
+
+A stderr `Warning: … column count different from the header row …` means the CSV/TSV has ragged rows (extra fields dropped, missing fields treated as empty); report it as a data-quality signal. For how every pipeline failure — abnormal data, parse errors, and program errors — is classified and handled, see [error handling](error-handling.md).
 
 ## Advanced input
 
@@ -49,20 +51,24 @@ Report row/column counts, field types, missing/unique/time-parse ratios, UE elig
 - **Excel sheets** — `--merge-sheets` streams every worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match.
 - **Multi-file type conflicts** — when the same column has different inferred types across files, present each conflict and resolve with `--type-resolutions` on `convert` (see [transform](transform.md)).
 
-## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV cells)
+## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV/Excel cells)
 
-Nested data is flattened one level per user decision. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick `Flatten` for a node.
+Nested data is analyzed per field, and the recommended mapping encodes one decision per container: keep whole, flatten one level, or flatten to the leaves. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick a depth for a node.
 
 1. **Locate the nested structure.**
    - NDJSON/JSON: read the top-level `nested_tree` from the inspect result (record-root paths).
-   - CSV/TSV: a JSON-encoded object column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Array cells (`items`-style) are `list` columns and carry no tree.
+   - CSV/TSV/Excel: a JSON-encoded column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Object cells list child keys; array cells (`items`-style) expose an `elementKind` and, for object arrays, the union of the element fields.
    Object nodes list child keys, array nodes carry an `elementKind`, primitive leaves carry an inferred type and bounded samples. Summarize node kinds, never paste samples.
-2. **Judge first, then confirm.** Read the node names and inferred types and propose, per container, whether to flatten (and to what depth), keep as JSON, or keep as a list. Present the proposal as a table and default to it; ask the user to confirm or adjust. Only ask node-by-node for the nodes you cannot judge from the names.
-   - Business-entity names with stable scalar children (`user_info`, `address`, `order`) → propose flattening to the leaves.
-   - Generic container names (`payload`, `data`, `config`, `meta`, `extra`, `attributes`) → propose keeping as JSON, unless the user wants a specific sub-field.
-   - Arrays are always kept as a list property and never split.
-   - Nesting deeper than one level with no clear intent → propose keeping as JSON or flattening only the first level.
-3. **Record the answers in `flatten_rules` (`{ "out_column": "dot.path" }`).**
+2. **Review the recommended mapping's per-field decision, then confirm adjustments.**
+   The recommended mapping already encodes, per container, a data-driven depth decision — the depth is not uniform across fields:
+   - **Keep whole** — a single-level object (every child scalar or a scalar array) is declared `type: 'object'` with `transform: 'json'`, plus one `parent.child` sub-property per child; a single-level object array is declared `type: 'array_row'` with one `parent.child` sub-property per element field. The parent entry carries the native value and conversion emits it once; each child is a plan-only declaration (scalar or `list`) that never reads a column of its own.
+   - **Flatten one level / to the leaves** — an object that itself contains an object or object array cannot be kept whole; it collapses one level and each child is decided independently: scalar children become flat properties materialized through `flatten_rules`, and nested objects/arrays recurse until a single-level container is reached (which is then kept whole).
+   - **Scalar array** — `["a","b"]` stays `list` (→ AE `array_string`), a leaf with no children.
+   - **Nested element field** — an element field that is itself an object/array stays inside the array data and is not declared as a sub-property (array-element flattening is not supported); the mapping warns so it is never silently dropped.
+   Present the resulting property list as a table and default to it; ask the user to confirm or adjust only where you disagree with a node's inferred decision (business-entity names with stable scalar children vs generic containers like `payload`/`data`/`config` vs nesting deeper than one level with no clear intent).
+3. **Record overrides in `flatten_rules` (`{ "out_column": "dot.path" }`) and `exclude_columns`.**
+   The recommended mapping already carries the flatten rules for its collapsed levels. When you override it:
    - NDJSON/JSON: the path is from the record root (`user_info.name`).
-   - CSV/TSV: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
-   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; CSV prefixes the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure.
+   - CSV/TSV/Excel: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
+   - **`flatten_rules` only materializes the out column in the row — it does not emit it.** For every out column, also add a `properties` entry with `source` set to that out-column name (plus `target`/`type`/`desc`); otherwise the flattened value is silently dropped from the output record.
+   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; cell-relative rules prefix the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'array_row'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure; a kept whole `object`/`array_row` also declares its scalar children as dotted `parent.child` sub-properties in `properties`.

package/skills/ae-data-integration/references/tracking-plan.md

@@ -10,15 +10,15 @@ Generate and confirm the event/property plan **before** any transform or upload.
 ## Sub-steps
 
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
-2. **Column → property draft** — identify system columns (time, `distinct_id`/`account_id`, event-name column, user-property-name column); map the remaining columns to event / user / common properties. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
-3. **Confirmation gate (single, one pass)** — present one summary table: event list + property list (uncertain types highlighted) + field scope (default: plan fields only, with a full-import switch) + unrecognized/dirty-data handling. The user answers once with ok or edits (renames, types, add/drop columns).
+2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
+3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
 4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
-5. **Persist the plan** — draft.json → xlsx → upload with `sdk_integration_mode=none`.
+5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 
 ## CLI
 
-`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out <draft.json>] [--dry-run]` converts the confirmed `ae-local-data-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
+`ae-cli data-integration plan --mapping <mapping> [--event-name <name>...] [--plan-name <name>] [--out .ae-cli/data-integration/draft.json] [--dry-run]` converts the confirmed `ae-data-integration-mapping/v1` mapping into a tracking-plan `draft.json` with `sdk_integration_mode=none` and `source_type=data`.
 
 - `user_set` mode → no events; every mapping property becomes a user property.
 - `track` mode → one event per `--event-name` (or the mapping `default_event_name`); every property becomes an event property.
@@ -26,9 +26,11 @@ Generate and confirm the event/property plan **before** any transform or upload.
 - `exclude_columns` are dropped from the draft.
 - `desc` and `event_tag` flow from the mapping: each property's `desc` (falling back to its source column name) and each event's `event_meta.<name>.desc` / `event_meta.<name>.tag` (falling back to the source event name) are written into the draft. Fill them in the mapping so the plan is never empty.
 - The per-row event-name column cannot be enumerated without a full scan, so when `default_event_name` is absent the CLI requires `--event-name` for each concrete event name.
+- Duplicate `display_name`s across the property pool are auto-deduplicated (the property name is appended) before validation, so same-named source columns under different object paths never collide. Dotted `target`s default each sub-property's `display_name`/`desc` to its leaf segment, which also avoids collisions.
+- The draft is validated locally before any upload: an `object`/`array_row` property with no `parent.child` sub-property, or a sub-property whose type is `object`/`array_row`, fails fast with `LOCAL_DATA_PLAN_INVALID_DRAFT` (`--dry-run` fails too). Fix it in the mapping before running `plan`: a mapping `properties` entry with a dotted `target` (`user_info.name`) declares a `parent.child` sub-property whose parent is the `object`/`array_row` entry with the same first segment — so a kept whole container must carry one dotted entry per child (each scalar or `list`) next to the parent entry. A nested object/object-array that cannot stay nested must instead be flattened to scalar leaves: record one `flatten_rules` entry per leaf and one flat scalar (or `list`) `properties` entry per leaf (see [source-inspect.md](source-inspect.md)).
 - `--dry-run` previews events and properties without writing `draft.json`.
 
-Afterward reuse `ae-cli tracking plan draft --in draft.json --out draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
+Afterward reuse `ae-cli tracking plan draft --in .ae-cli/data-integration/draft.json --out .ae-cli/data-integration/draft.xlsx` and `ae-cli tracking plan validate / upload` for xlsx generation and ingestion.
 
 ## Reuse