@thinkingai/ae-cli 6.1.17 → 6.1.18

package/skills/ae-analysis/references/personal_semantic_preference_add.md

@@ -0,0 +1,23 @@
+# personal-semantic-preference add
+
+Add one personal semantic preference for the authenticated user in one project.
+
+Use this when the user explicitly asks to save a personal semantic preference, or when the current project task contains an explicit stable statement, correction, or confirmation that should become a reusable current-user preference. A current-user working definition remains eligible even when the same content may benefit other users. A second "save" confirmation is not required after that evidence gate is met, unless the target meaning is ambiguous.
+
+Command:
+
+```bash
+ae-cli personal-semantic-preference add --project-id <project_id> --context-type <context_type> --title <title> --summary <summary> --content <content> [--keywords '["keyword"]'] [--resource-refs '[{"resource_type":"report","resource_key":"101","display_name":"Revenue daily report"}]'] [--fresh-until-at "yyyy-MM-dd HH:mm:ss"] [--request-id <id>]
+```
+
+Use `preference` for durable interpretation/output preferences, `asset_context` for durable wording or intent bound to exact assets, `experience` for confirmed reusable work methods, and `background` for stable personal context. `--resource-refs` is required and non-empty only for `asset_context`; for every other type it must be absent or empty.
+
+`--resource-refs` accepts 1 to 50 ordered objects. Each object contains exactly `resource_type`, string `resource_key`, and `display_name`; `(resource_type, resource_key)` must be unique. `resource_type` is generic lower snake_case rather than a report-only enum, so events, properties, metrics, tags, clusters, reports, dashboards, data tables, and later asset types share the same shape. Array order is the user's intended priority.
+
+`--request-id` is an idempotency key; omit it for ordinary interactive use and the CLI will generate one.
+
+This command creates only a current-user preference; it never creates or approves a project semantic. Do not reject an otherwise valid personal preference merely because the same content may benefit other users, and do not imply that the saved preference is shared authority. Keep project-candidate recommendation separate: after personal capture, ask whether to recommend broadly reusable content as a project semantic candidate, and submit nothing without that choice.
+
+Store only the present working preference. Do not append future governance or lifecycle instructions. Do not use this command for company knowledge, standalone metadata facts, reports, dashboards, transient task details, one-off analysis results, or automatic stale/expired preference handling.
+
+Output is the gateway envelope. `data.preference` contains the full created preference, current revision, and complete ordered `resource_refs`. A separate readback is unnecessary unless a later step needs to refresh the record.

package/skills/ae-analysis/references/personal_semantic_preference_delete.md

@@ -0,0 +1,17 @@
+# personal-semantic-preference delete
+
+Soft-delete one personal semantic preference using optimistic locking.
+
+Use this only after explicit user confirmation to remove a saved personal preference.
+
+Command:
+
+```bash
+ae-cli personal-semantic-preference delete --project-id <project_id> --id <preference_id> --expected-revision <revision> [--request-id <id>] --yes
+```
+
+Read the current item first and pass its revision as `--expected-revision`. This is a high-risk write; use `--dry-run` when the target is ambiguous.
+
+Do not use this command for automatic stale/expired preference handling. Stale or low-freshness personal preferences are hidden by list filtering and backend maintenance.
+
+Output is the gateway envelope. `data.deleted` identifies the deleted preference and status. After deletion, it should no longer appear in `personal-semantic-preference list`.

package/skills/ae-analysis/references/personal_semantic_preference_get.md

@@ -0,0 +1,19 @@
+# personal-semantic-preference get
+
+Get one personal semantic preference by ID.
+
+Use this command after `personal-semantic-preference list` identifies a likely current-user preference. Pass `--mark-used` when the preference is adopted for the answer, query path, or as the matched target for an update.
+
+Command:
+
+```bash
+ae-cli personal-semantic-preference get --project-id <project_id> --id <preference_id> [--mark-used]
+```
+
+Input uses `project_id`, `id`, and optional `mark_used`. `id` must be the exact `preference_<id>` value returned by list/add.
+
+Do not use this command as a keyword search, project semantics lookup, or asset catalog lookup. Do not call it repeatedly for every catalog row. Do not use `--mark-used` for a candidate that turns out not to match the user's intent or is only inspected and then rejected.
+
+When a published project semantic conflicts with the personal item, the project semantic is the formal definition. Fetch and mark the personal item only when it materially affects the response, such as an explicitly requested non-formal alternative; never silently use it to override the project semantic.
+
+Output is the gateway envelope. `data.preference` contains the full personal preference, including content, complete ordered `resource_refs`, and revision. When `--mark-used` is set, the backend increments `heat_count` and updates `last_used_at` for that record.

package/skills/ae-analysis/references/personal_semantic_preference_list.md

@@ -0,0 +1,21 @@
+# personal-semantic-preference list
+
+List the authenticated user's active personal semantic preference catalog for one project.
+
+Use this command once per host, authenticated user, project, and conversation after resolving the project. Keep the returned directory in conversation context for later questions where user-specific wording or preferences may change interpretation.
+
+Command:
+
+```bash
+ae-cli personal-semantic-preference list --project-id <project_id>
+```
+
+Input uses `project_id` only. Do not add pagination parameters: the backend returns a compact catalog intended for agent context.
+
+Do not use this command for project semantics, shared knowledge, metadata catalogs, report/dashboard lists, or complete asset discovery. It only returns the current user's personal semantic preferences in the current project.
+
+Output is the gateway envelope. `data.items[]` contains only `id`, `context_type`, `title`, truncated `summary`, limited `keywords`, `resource_ref_count`, distinct `resource_types`, and `revision`; it deliberately omits content, full asset references, heat, and timestamps. `data.returned_count` is at most 200, `data.truncated` says whether entries were omitted, and `data.selection_policy` is `HOT_160_PLUS_RECENT_40`: up to 160 highest-heat items plus up to 40 recently changed items not already selected. The backend may return fewer items to keep the data payload within 64 KiB.
+
+If one returned item is actually adopted to interpret the user's request, call `ae-cli personal-semantic-preference get --project-id <project_id> --id <preference_id> --mark-used` before using its full content. Do not mark an item used when it was only inspected or rejected.
+
+Compare a likely match with the published project semantic catalog. Published project semantics remain the formal project-wide authority; personal items provide current-user defaults and working interpretations. If they conflict, use the project semantic for the formal result, explicitly disclose the personal difference, and do not mark the personal item used unless the user explicitly adopts it as a labeled alternative.

package/skills/ae-analysis/references/personal_semantic_preference_update.md

@@ -0,0 +1,19 @@
+# personal-semantic-preference update
+
+Update one personal semantic preference using optimistic locking.
+
+Use this when the user explicitly asks to change an existing personal preference, or when the current project task provides an explicit stable correction or confirmation that should replace a matching saved current-user preference. A current-user working definition remains eligible even when the same content may benefit other users. First read the matched current item with `personal-semantic-preference get --mark-used` and pass the returned revision as `--expected-revision`.
+
+Command:
+
+```bash
+ae-cli personal-semantic-preference update --project-id <project_id> --id <preference_id> --expected-revision <revision> --context-type <context_type> --title <title> --summary <summary> --content <content> [--keywords '["keyword"]'] [--resource-refs '[{"resource_type":"event","resource_key":"$login","display_name":"Login event"}]'] [--fresh-until-at "yyyy-MM-dd HH:mm:ss"] [--request-id <id>]
+```
+
+The update replaces all editable fields. Include the full intended `title`, `summary`, `content`, keywords, and asset bindings rather than a partial patch. `asset_context` requires 1 to 50 complete ordered `resource_refs`; other context types cannot contain non-empty refs.
+
+This command updates only the current-user preference; it does not update, approve, or publish project semantics. Do not block the update merely because the same content may benefit other users, and do not imply that the saved preference is shared authority. Keep project-candidate recommendation separate and ask before submitting it.
+
+Keep future governance and lifecycle handling out of the stored content. Do not use this command for shared knowledge, standalone metadata facts, reports, dashboards, transient task details, one-off analysis results, or automatic stale/expired preference handling.
+
+Output is the gateway envelope. `data.preference` contains the full updated preference and new revision. If the revision is stale, read the latest record and ask the user how to merge rather than overwriting silently.

package/skills/ae-data-integration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-data-integration
-description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it."
+description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it. Trigger words: 本地数据导入 / 离线数据 / 数据文件 / 文件导入 / 文件上报 / CSV 导入 / Excel 导入 / TSV 导入 / JSON 导入 / 导入到 AE / 导入到 ThinkingData / local data import / import local file."
 ---
 
 # AE Data Integration
@@ -23,16 +23,35 @@ Two entrances lead here: the AE Agent dialog (attach / plus-button upload) and `
 - If a batch times out or loses the network, treat that batch as unknown. Stop. Ask the user to verify receiver/AE data before the user chooses `--resume-from`; never resume automatically.
 - Local analysis stays local. AE Agent attachment is a separate, confirmed branch with a 50 MB per-file limit.
 
+## When to use / When NOT to use
+
+Use this skill when the user wants to bring a **local data file** (CSV/TSV/TXT/JSON/JSONL/XLS/XLSX) into AE, or analyze such a file locally without uploading.
+
+| User intent | Use this instead |
+| --- | --- |
+| How to integrate the SDK / tracking code / LogBus2 config / reporting-error triage (usage Q&A, no local file) | ae-data-integration-helper |
+| Database / datasource direct sync (MySQL, DataX, data warehouse, data-dev platform) | ae-dataops |
+| Community content (posts / comments / chat / WeCom groups) insight or submission | ae-community |
+| Generate / upload a project-level tracking plan (source material is PRD / chat / template / code; deliverable is a real platform tracking plan) | ae-generate-tracking-plan |
+| Upload documents / URLs to a knowledge base | ae-kb |
+| Reports / dashboards / queries / governance on data already in AE | ae-analysis |
+
+This skill also produces a tracking-plan draft (`source_type: data`) as a governance prerequisite; that draft is an input to ae-generate-tracking-plan, not a substitute for its five-phase platform plan.
+
 ## Workflow
 
 Walk the four submodules in order. Each submodule is its own reference; follow it and come back here for the next step.
 
 1. **Source — business identification.** Read [references/source-inspect.md](references/source-inspect.md). Profile every file fully, infer its business meaning using business-doc / user-prompt priors, then pick a branch via [references/ue-routing.md](references/ue-routing.md).
-2. **Reuse check.** If a `.ae-data-integration/index.json` exists and the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
-3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). Generate the event/property plan from the profile and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
+2. **Reuse check.** If the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. `reuse` searches the current directory's `.ae-cli/data-integration/` upward, then `~/.ae-cli/data-integration/`, so a package written elsewhere is still found. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
+3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). The plan is generated from the mapping (`plan --mapping`), so confirm the recommended mapping's key system fields with the user first — `mode`, `#account_id`/`#distinct_id`, `#time` + timezone, `#event_name`, `#ip`/`#uuid` (see [references/transform.md](references/transform.md) steps 1–5) — then generate the event/property plan and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
 4. **Transform.** Read [references/transform.md](references/transform.md). Map columns to AE system fields and properties, convert, and quarantine dirty rows per [references/ue-mapping.md](references/ue-mapping.md).
 5. **Sink — upload.** Read [references/sink-upload.md](references/sink-upload.md). Resolve the destination, dry-run, confirm, then upload per [references/sync-json-upload.md](references/sync-json-upload.md). `receiver_accepted` is not persistence: after a ~1-minute ingestion delay, verify the data landed with ae-cli (`tracking live-data list` / `tracking ingest summary` / `tracking ingest-error list`) rather than telling the user to check the console.
-6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (frozen mapping + transform script + plan reference) so the next same-shape file skips the full pipeline.
+6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (pipeline descriptor + frozen mappings + stage executors + docs) and a shareable zip; in the completion response, state the absolute zip path, the package directory, and the one-line way to run the next same-shape file.
+
+## Error handling
+
+When a step fails, classify the failure before acting — see [references/error-handling.md](references/error-handling.md). A quarantined row, a ragged line, and a disk-full are three different problems with three different responses: match on the error `code`, never retry a parse failure by guessing the encoding, and never report a program failure as a data problem.
 
 ## Local analysis branch
 

package/skills/ae-data-integration/references/custom-layer.md

@@ -0,0 +1,93 @@
+# Project custom layer (verify + salvage overlays)
+
+The handoff package is deliberately **standard flow only**: it ships the generic
+stage executors and never embeds environment-specific workarounds such as SQL
+direct queries or multi-round salvage loops. Some projects need those — a hard
+per-event persistence judge, or a repeated salvage loop for a dirty source. Add
+them as a **custom layer next to the package**, never by editing the generic
+package itself.
+
+## When to add a layer
+
+- **Hard persistence judge.** `bin/verify.py` is a soft check: it computes the
+  submit window and expected counts from the local UE output, then shows the
+  `tracking ingest summary` payload before and after for comparison. It does not
+  parse that payload into per-event counts — the capability's `data` shape is
+  server-defined, and a shared project's window delta cannot be attributed to one
+  import. When the project requires an automated per-event ✓/✗ verdict, overlay a
+  SQL layer that reads the project's event/user tables directly.
+- **Multi-round salvage.** `bin/run.sh` already prints a single-round salvage
+  hint when `invalid.rows.jsonl` is non-empty. For sources that fail in layers,
+  wrap that command in a loop that re-feeds each round's quarantine into the next
+  `--salvage-from` until nothing fails or the user stops.
+
+## Rules
+
+1. Put overlays in a sibling directory (e.g. `custom/`) under the package root,
+   and reference `../pipeline.json` / `../index.json` — never modify the frozen
+   mappings, `bin/`, or the generated docs.
+2. Keep every existing confirmation gate. A custom layer may add gates; it must
+   not remove the `--confirm` upload gate or the plan/shape gates.
+3. Keep secrets in `.local/target.env`. Never write APPID, endpoints, tokens, or
+   raw data values into an overlay file.
+4. An overlay is project-specific. Do not copy it into another project's package
+   without re-confirming the environment assumptions (table names, APPID, host).
+
+## Skeleton 1 — SQL verify (hard per-event judge)
+
+The reference deployment queries the ingested tables directly. **Table and
+column names vary by deployment** (`v_event_1` / `v_user_1` and `$part_date` /
+`$part_event` are examples only) — confirm them against the project's receiver
+schema before use, and keep the query read-only.
+
+```bash
+# custom/verify.sh — hard judge; requires the SQL capability to be available.
+# usage: custom/verify.sh <run-dir> [--baseline|--check]
+# Reuses bin/verify.py's window/count computation idea, but answers the question
+# "did exactly these events land?" with a direct count over the event table.
+```
+
+The query shape (adapt table/column names):
+
+```text
+select <event_column>, count(*)
+from <event_table>
+where <date_partition_column> between '<window-start>' and '<window-end>'
+group by 1
+```
+
+Compare the per-event delta (after upload minus before upload) against the
+expected counts from `valid.ue.jsonl`. User-profile rows are overwrite-writes,
+so a zero user-table delta is normal; compare event rows per event, not totals.
+
+Keep this layer read-only and non-blocking: report the verdict, never retransmit
+automatically on a mismatch — first check `tracking ingest-error list` for the
+silent-drop reason.
+
+## Skeleton 2 — multi-round salvage loop
+
+`bin/run.sh` prints the one-round hint. Wrap it into a loop that re-processes each
+round's quarantine against the fixed mapping until clean or the user stops:
+
+```bash
+# custom/salvage.sh <run-dir> — re-process quarantined rows round by round.
+# Each round's valid.ue.jsonl is disjoint from earlier rounds', so upload each
+# round independently (same --confirm and --allow-clean-subset gates).
+round=1
+while [ -s "<run-dir>/invalid.rows.jsonl" ]; do
+  ae-cli data-integration convert \
+    --input-file '<same-source>' \
+    --mapping '<fixed-mapping.json>' \
+    --salvage-from "<run-dir>/invalid.rows.jsonl" \
+    --output-dir "<run-dir>-salvage-$round"
+  round=$((round + 1))
+  # stop condition is the user's: a row that still fails after a fix is a real
+  # data defect, not a code bug.
+done
+```
+
+See [transform.md](transform.md) for the `--salvage-from` semantics (single-file
+only; the source must be the same file that produced the quarantine) and
+[handoff.md](handoff.md) for the package the overlay attaches to. Persistence
+verification always follows [sink-upload.md](sink-upload.md): `receiver_accepted`
+is not durability.

package/skills/ae-data-integration/references/error-handling.md

@@ -0,0 +1,92 @@
+# Error handling
+
+Failures in the data-integration pipeline split into three classes. Each class has a distinct
+response; an agent that mixes them up wastes retries and misleads the user.
+
+| Class | What it is | Response |
+| --- | --- | --- |
+| Abnormal data | Rows or fields that do not meet UE rules | Quarantine the row or skip the field, report counts, salvage |
+| File parsing exception | The source cannot be read as its detected format | Surface the parse error, do not retry by guessing; fix the file or format |
+| Program execution exception | The program itself failed (disk, permissions, output dir, mapping mismatch) | Surface the exact error, fix the environment/inputs, re-run |
+
+Every error leaves the CLI as the standard envelope `{ ok, data, error: { type, code, message, hint } }`
+(JSON on stdout; progress and warnings on stderr). The `code` is the stable identifier an agent
+branches on; `hint` is human-readable guidance.
+
+## Class 1 — Abnormal data
+
+Happens inside `convert`. Two severities:
+
+- **Whole-row quarantine.** Any error on a row — identity, time, event name, record type, or a
+  single property (type coercion, size/limit) — drops the entire row. The row is written to
+  `invalid.rows.jsonl` with its error codes and counted in `manifest.output.invalid_records`; the
+  manifest becomes `blocked`. Row-level codes: `MISSING_USER_ID`, `USER_ID_TOO_LONG`,
+  `INVALID_RECORD_TYPE`, `INVALID_TIME`, `TIME_OUT_OF_RANGE`, `INVALID_EVENT_NAME`,
+  `INVALID_ZONE_OFFSET`, `PROPERTY_TYPE_CONFLICT`, `PROPERTY_LIMIT_EXCEEDED`.
+- **Field skip.** A `#ip` or `#uuid` value that violates its spec drops only that field and keeps
+  the row. Counts land in `manifest.output.skipped_fields`; private/LAN IPs are kept but counted in
+  `manifest.output.lan_ip_records`. Field-skip codes: `INVALID_IP`, `INVALID_UUID`. Never quarantined.
+
+Responses:
+
+- A blocked manifest is not a failure to retry blindly. Fix the mapping, then re-run `convert
+  --salvage-from <invalid.rows.jsonl>` to re-process only the quarantined rows (never re-send the
+  rows that already passed). Repeat the salvage loop until no rows fail or the user stops.
+- An empty source (zero data rows) blocks with the reason `The source contained no data rows.` —
+  a distinct, clearer message than a failed validation run. Do not re-run the same command on the
+  same empty file expecting a different result.
+- Ragged delimited rows (a CSV/TSV row whose column count differs from the header) are tolerated:
+  extra fields are dropped and missing fields are treated as empty, and a stderr warning reports
+  how many rows were ragged. A row that becomes missing its identity/time because a field was
+  absent is then quarantined normally. Treat the warning as a data-quality signal for the user.
+
+## Class 2 — File parsing exceptions
+
+The source cannot be parsed as its detected format. The agent's job is to report precisely and let
+the user decide — never to guess an encoding or structure and retry silently.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_INPUT_NOT_FOUND` | Path is not a readable file | Ask for the correct path |
+| `LOCAL_DATA_FILE_TOO_LARGE` | XLS over 1 GB | Convert to XLSX or split the workbook |
+| `LOCAL_DATA_INPUT_INVALID` | Generic parse failure (malformed CSV/TSV/JSON/XLS) | Verify encoding and structure, then retry without changing the source |
+| `LOCAL_DATA_JSONL_INVALID` | A JSONL line is not valid JSON (`location.record` names the line) | Point at the offending record |
+| `LOCAL_DATA_JSON_ROOT_INVALID` | JSON root is not an object or array | Check the file's top-level shape |
+| `LOCAL_DATA_XLSX_INVALID` | Workbook metadata missing / no readable sheets / worksheet entry missing | Re-export the workbook |
+| `LOCAL_DATA_SET_NOT_FOUND` / `LOCAL_DATA_SET_REQUIRED` | Sheet or JSON Path not found / ambiguous | Ask which `--data-set` to use |
+
+A parse error is never a reason to change the mapping or the tracking plan. Report the code and
+hint verbatim, and ask the user to fix the file (re-export, re-encode, or split).
+
+## Class 3 — Program execution exceptions
+
+The program itself failed; the source data is usually fine. These must never be mislabeled as
+parse errors: a per-row callback failure (for example a disk that filled up mid-convert) propagates
+as itself, not as `LOCAL_DATA_INPUT_INVALID`.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_OUTPUT_NOT_EMPTY` | The output directory must be new or empty | Point convert at a fresh `<run-id>` directory |
+| `LOCAL_DATA_SOURCE_CHANGED` / `LOCAL_DATA_SOURCE_FORMAT_CHANGED` | The source no longer matches the mapping fingerprint/format | Re-run inspect and review a new mapping |
+| `LOCAL_DATA_MAPPING_INVALID` / `LOCAL_DATA_MAPPING_INVALID_JSON` / `LOCAL_DATA_MAPPING_NOT_FOUND` | The mapping cannot be read or validated | Re-read the mapping reference, fix the mapping file |
+| `LOCAL_DATA_SALVAGE_INVALID` / `LOCAL_DATA_SALVAGE_EMPTY` / `LOCAL_DATA_SALVAGE_NO_MATCH` | The salvage file is not a valid quarantine file, is empty, or lists no rows from this source | Point at the correct `invalid.rows.jsonl` from the same source |
+| `LOCAL_DATA_TYPE_CONFLICTS_UNRESOLVED` / `LOCAL_DATA_TYPE_RESOLUTIONS_INVALID` | Cross-file column type conflicts need explicit resolutions | Build `--type-resolutions` |
+| `LOCAL_DATA_PLAN_INVALID_EVENT_NAME` / `LOCAL_DATA_PLAN_EVENT_NAMES_REQUIRED` / `LOCAL_DATA_PLAN_INVALID_LANG` | Tracking-plan draft inputs are invalid | Fix the event name(s) or `--lang` |
+| `LOCAL_DATA_HANDOFF_INDEX_INVALID` / `LOCAL_DATA_HANDOFF_PLAN_NOT_FOUND` / `LOCAL_DATA_HANDOFF_PLAN_INVALID` | Handoff index/plan cannot be read | Point at a valid handoff directory/plan file |
+| `LOCAL_DATA_ENDPOINT_INVALID` / `LOCAL_DATA_APPID_INVALID` / `LOCAL_DATA_BATCH_SIZE_INVALID` / `LOCAL_DATA_COMPRESS_INVALID` / `LOCAL_DATA_RESUME_INVALID` / `LOCAL_DATA_RESUME_OUT_OF_RANGE` | Upload arguments are invalid | Fix the flag before uploading |
+| `LOCAL_DATA_MANIFEST_INVALID` / `LOCAL_DATA_UE_FILE_INVALID` / `LOCAL_DATA_UE_FILE_NOT_FOUND` / `LOCAL_DATA_UE_FILE_CHANGED` / `LOCAL_DATA_UE_COUNT_MISMATCH` / `LOCAL_DATA_MANIFEST_FILE_MISMATCH` | Upload preconditions fail | Re-check the manifest and UE file pairing |
+| `LOCAL_DATA_CLEAN_SUBSET_CONFIRMATION_REQUIRED` | Uploading from a blocked manifest needs a separate clean-subset decision | Confirm the subset explicitly before `--allow-clean-subset` |
+
+Write failures (disk full `ENOSPC`, permission denied `EACCES`) do not hang the command: the
+output streams fail with a clear `Failed to write "<path>"` message telling the user to check disk
+space and directory permissions. Fix the environment, then re-run into a fresh output directory.
+
+## Cross-cutting rules
+
+- Classify first, act second. Match on `code`, not on message text.
+- A parse error is not a data problem; a program error is not a parse error. Do not conflate them
+  when reporting back to the user.
+- Never retry a parse failure by guessing the encoding, delimiter, or header layout. Show the
+  error and ask.
+- After any fix, re-run from the start of the step that failed; do not resume a half-written run
+  or reuse a partially populated output directory.

package/skills/ae-data-integration/references/handoff.md

@@ -1,43 +1,102 @@
 # Handoff (reusable package)
 
-After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package freezes the transform logic (the confirmed mapping), the runnable transform script, and the tracking-plan reference. It does **not** re-freeze the raw data or any upload secrets.
+After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package is a DataX-style pipeline — a declarative `source → transform → sink` descriptor plus generic stage executors that dispatch to `ae-cli data-integration` subcommands. It freezes the transform logic (the confirmed mappings) and the tracking-plan reference, and ships a `bin/` directory that a human or agent can run directly. It never re-freezes raw data or upload secrets.
 
 ## When
 
-Run handoff after Transform — or after Sink — once the mapping is confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
+Run handoff after Transform — or after Sink — once the mappings are confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
 
 ## CLI
 
 ```
-ae-cli data-integration handoff --mapping <mapping> [--plan-file <draft.json>] [--out-dir .ae-data-integration]
+ae-cli data-integration handoff --mapping <mapping> [--mapping <mapping2> ...] [--plan-file <draft.json>] [--pushurl <url>] [--project-id <id>] [--out-dir .ae-cli/data-integration]
 ```
 
-- `--mapping` — the confirmed `ae-local-data-mapping/v1` mapping (the frozen transform logic, including `value_mapping` and `flatten_rules`).
-- `--plan-file` — optional tracking-plan `draft.json` to reference inside the package (`plan.json`).
-- `--out-dir` — handoff root. Default `.ae-data-integration/` (travels with the project).
-- `--dry-run` previews the fingerprint, files, and index path without writing.
+- `--mapping` — one or more confirmed `ae-data-integration-mapping/v1` mappings (the frozen transform logic, including `value_mapping` and `flatten_rules`). Repeat it for multi-sheet workbooks: one mapping per sheet.
+- `--plan-file` — optional tracking-plan `draft.json` to reference inside each mapping directory (`plan.json`).
+- `--pushurl` — optional receiver base URL to record as the reuse upload target (the sink endpoint is `pushurl` + `/sync_json`). Record it when the next same-shape file will most likely land at the same receiver.
+- `--project-id` — optional numeric destination project ID to record; `bin/upload.sh` derives the APPID from it via `project info get` at upload time.
+- `--out-dir` — handoff root. Default `.ae-cli/data-integration/` (project workspace; travels with the project).
+- `--dry-run` previews the fingerprints, target, file list, and zip path without writing.
 
 ## Package layout
 
 ```
-.ae-data-integration/
-  index.json               ← shared fingerprint index (reuse detection)
-  <fingerprint[:16]>/
-    mapping.json           ← frozen mapping (re-runnable by convert)
-    transform.mjs          ← node transform.mjs <new-file> [<output-dir>]
-    plan.json              ← optional tracking-plan reference
+.ae-cli/data-integration/    ← handoff root (= out-dir, project workspace)
+  pipeline.json             ← source → transform → sink descriptor (ae-data-integration-pipeline/v1)
+  index.json                ← shared structure-fingerprint index (reuse detection)
+  shape.json                ← per-mapping column baseline (shape gate)
+  <fingerprint[:16]>/       ← one directory per mapping
+    mapping.json            ← frozen mapping
+    transform.mjs           ← node transform.mjs <new-file> [<output-dir>]
+    plan.json               ← optional tracking-plan reference
+  bin/                      ← generic stage executors (read pipeline.json)
+    run.sh                  ← source → transform → plan (never uploads)
+    upload.sh               ← sink (dry-run by default; --confirm uploads; resolves recorded target)
+    bind_mapping.py         ← shape check + rebind sha256/data_set to the new file
+    summarize.py            ← valid/quarantined counts
+    plan_check.py           ← tracking-plan coverage gate (exit 3 on new events/properties)
+    verify.py               ← soft persistence check (submit window vs ingest summary)
+    resolve_appid.py        ← APPID derivation helper (project info get)
+  README.md / RUNBOOK.md    ← how to run, the four gates, persistence verification
+  .local/target.env.example ← destination template (no real secrets)
+  .gitignore                ← inbox/ runs/ .local/target.env
+  inbox/  runs/             ← daily input / per-run outputs
 ```
 
-The `transform.mjs` wrapper shells out to `ae-cli data-integration convert`; it takes the new file path as its first argument, never bakes the source path, and re-stamps `source.sha256` with the new file's fingerprint before converting. Reuse it only for a file of the **same shape** (same header/schema and format) — the transform logic is frozen, but the content guard is re-bound to each specific file.
+A shareable archive is written next to the package root: `<parent>/ae-data-integration-handoff-<fingerprint[:8]>.zip`. The zip carries only this handoff round — the frozen mappings just written, a scoped `index.json` (just this round's entries), and the generic executors/docs — not the accumulated history, which stays in `.ae-cli/data-integration/` for reuse matching.
+
+## Pipeline descriptor
+
+`pipeline.json` declares the three stages and their types. Only `source: local_file` and `sink: restful_sync_json` are implemented today; the `type` fields reserve logbus / datax / mysql for later phases. `bin/run.sh` and `bin/upload.sh` read the descriptor and dispatch each stage by its `type` to `ae-cli data-integration inspect / convert / upload` — they are executors, not a second runtime engine.
+
+## Recorded destination
+
+`pipeline.json` → `sink.params` records `pushurl` and `project_id` when the handoff
+was run with those flags. Reuse defaults to that target, but **`bin/upload.sh`
+never sends without `--confirm`**, so the operator re-confirms the address and
+project on every reuse. Resolution order at upload time:
+
+- endpoint: recorded `pushurl` (+ `/sync_json`), else `AE_ENDPOINT`.
+- APPID: `AE_APPID`, else derived via `ae-cli project info get --project-id <id>`
+  (see `bin/resolve_appid.py`; set `AE_APPID` when that payload lacks `appid`).
+- project id: recorded `project_id`, else `AE_PROJECT_ID`.
+
+`project info get` returns `data.appid` at the top level (verified against the AE
+demo host); `bin/resolve_appid.py` reads that exact field and prints it, falling
+back to `AE_APPID` when the field is absent or not a non-empty string.
+
+## Project custom layers
+
+For projects that need a hard per-event SQL judge or a multi-round salvage loop
+(the reference package's environment-specific workarounds), overlay a custom layer
+next to the package instead of editing it — see [custom-layer.md](custom-layer.md).
+
+## Four confirmation gates
+
+The RUNBOOK and the scripts enforce four gates. The first two run automatically; the last two always need human confirmation:
+
+1. **Shape gate** — `bind_mapping.py` compares the new file's column set against `shape.json` and fails fast on a mismatch. A changed shape means the frozen logic was never reviewed for it: re-run the full pipeline.
+2. **Transform** — `ae-cli data-integration convert` per mapping; quarantined rows go to `invalid.rows.jsonl`, never silently dropped.
+3. **Tracking-plan gate** — `plan_check.py` verifies every produced event/property already exists in `plan.json`; new ones exit 3 and must be merged into the project plan first.
+4. **Sink gate** — `upload.sh` is dry-run by default; `--confirm` is the explicit upload decision.
 
 ## Structure fingerprint and index
 
-Every handoff records one entry in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: columns (source name + type), event model (`mode`, `event_name_field`, `record_type_field`), identity fields, and excluded columns. Business logic (`value_mapping`, transforms, `time_format`, and the fixed `default_event_name`) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+Every handoff records one entry per mapping in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model (`mode`). Business logic (`value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, the fixed `default_event_name`, and the system-field assignments) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+
+Reuse matching is its own step — see [references/reuse.md](reuse.md).
 
-Reuse matching is its own step — see [references/reuse.md](reuse.md). It compares a new file's profile structure against this index and proposes the matching package, which the user confirms before skipping the full pipeline.
+## Completion response
+
+After handoff succeeds, state the **absolute zip path**, the package directory, and the one-line way to run the next same-shape file:
+
+```
+cd <out-dir> && bin/run.sh <new-file>   # then bin/upload.sh runs/<run-id> --confirm
+```
 
 ## Safety rules
 
-- Treat the mapping, plan, generated artifacts, and the `.ae-data-integration/` directory as sensitive.
-- Never write APPID, endpoints, tokens, or raw data values into a handoff package. The package references the plan and the mapping; uploads still require an explicit, confirmed `upload` call.
+- Treat the mappings, plans, generated artifacts, the `.ae-cli/data-integration/` directory, and the zip as sensitive.
+- Never write APPID, tokens, or raw data values into a handoff package. The package records at most a destination `pushurl` and `project_id`; uploads still require an explicit, confirmed `upload` call, so the operator re-confirms the address and project each time.
 - Do not invent a mapping or plan. Handoff only packages what the user already confirmed.

package/skills/ae-data-integration/references/local-analysis.md

@@ -1,6 +1,6 @@
 # Local analysis
 
-Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/<run-id>/` with restrictive permissions.
+Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/runs/<run-id>/` with restrictive permissions.
 Set the directory to `0700` and generated scripts/reports to `0600`.
 
 ## Default report when no question is supplied

package/skills/ae-data-integration/references/reuse.md

@@ -4,21 +4,25 @@ Before walking the full pipeline for a new file, check whether a **handoff packa
 
 ## When
 
-Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-data-integration/index.json` exists. It is read-only and makes no writes.
+Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-cli/data-integration/index.json` exists. It is read-only and makes no writes.
 
 ## CLI
 
 ```
-ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-data-integration]
+ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-cli/data-integration]
 ```
 
 - `--mapping` — the candidate mapping, typically `inspect`'s `recommended_mapping` (the new file's structure, inferred exactly as Source would infer it).
-- `--out-dir` — handoff root. Default `.ae-data-integration/`.
+- `--out-dir` — handoff root. Optional. Without it, `reuse` searches in order: the current directory's `.ae-cli/data-integration/`, then each parent directory upward, then `~/.ae-cli/data-integration/` as a global fallback — so a package written in another directory or another session is still reachable.
 - `--dry-run` previews the fingerprint and match verdict without reading frozen packages.
 
+## Search paths
+
+When `--out-dir` is omitted, the result carries `searched_paths` — the exact `index.json` paths probed, in order. A `matched: false` result with a non-empty `searched_paths` means all of them were checked; a missing global fallback (no `$HOME`) simply omits it from the list.
+
 ## Matching
 
-The command computes the same **structure fingerprint** the handoff index is keyed on — columns (source name + type), event model, identity fields, and excluded columns — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`) is not part of the key, so a same-shape file with different content or a different file name still matches.
+The command computes the same **structure fingerprint** the handoff index is keyed on — the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, and the system-field assignments) is not part of the key, so a same-shape file with different content or a different file name still matches.
 
 ## Result
 
@@ -26,7 +30,7 @@ The command computes the same **structure fingerprint** the handoff index is key
 - **Match** (`matched: true`): the result carries the matched package — `mapping_file`, optional `plan_file`, the frozen `default_event_name` (so the user sees which event name will be reused), and a `run` command:
 
 ```
-node .ae-data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
+node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
 ```
 
 ## Confirmation gate

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,7 +31,7 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
-Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type, grouped into event properties (`track`) and user properties (profile modes) — never a counts-only summary. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
+Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.
 

package/skills/ae-data-integration/references/source-inspect.md

@@ -39,7 +39,9 @@ If `selection_required=true`, show only the Sheet/JSON Path candidates and ask t
 ae-cli data-integration inspect --input-file '<path>' --data-set '<candidate-id>' --source-timezone '<iana-timezone>'
 ```
 
-Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. Read [UE routing](ue-routing.md) before choosing a branch.
+Report row/column counts, field types, missing/unique/time-parse ratios, UE eligibility, mapping confidence, and warnings. Samples are bounded (up to 5 distinct, truncated) — summarize, never paste them. ID-like columns (`id`, `*_id`, `*_key`, `*_code`, `*_no`, `*_num`) stay `string` even when every value is numeric; JSON-encoded object/array values inside CSV cells are recognized as `object`/`list`, not `string`. IP- and UUID-named columns are additionally checked against their value specs: inspect warns how many non-empty values are invalid IPv4/IPv6, private/LAN IPs, or non-UUID strings, so the user can decide whether to map them as `ip_field`/`uuid_field`. Read [UE routing](ue-routing.md) before choosing a branch.
+
+A stderr `Warning: … column count different from the header row …` means the CSV/TSV has ragged rows (extra fields dropped, missing fields treated as empty); report it as a data-quality signal. For how every pipeline failure — abnormal data, parse errors, and program errors — is classified and handled, see [error handling](error-handling.md).
 
 ## Advanced input
 
@@ -49,20 +51,24 @@ Report row/column counts, field types, missing/unique/time-parse ratios, UE elig
 - **Excel sheets** — `--merge-sheets` streams every worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match.
 - **Multi-file type conflicts** — when the same column has different inferred types across files, present each conflict and resolve with `--type-resolutions` on `convert` (see [transform](transform.md)).
 
-## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV cells)
+## Nested flattening (NDJSON/JSON records and JSON-encoded CSV/TSV/Excel cells)
 
-Nested data is flattened one level per user decision. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick `Flatten` for a node.
+Nested data is analyzed per field, and the recommended mapping encodes one decision per container: keep whole, flatten one level, or flatten to the leaves. This flow is agent-driven and non-interactive: never pipe pre-filled answers into any prompt, and never silently pick a depth for a node.
 
 1. **Locate the nested structure.**
    - NDJSON/JSON: read the top-level `nested_tree` from the inspect result (record-root paths).
-   - CSV/TSV: a JSON-encoded object column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Array cells (`items`-style) are `list` columns and carry no tree.
+   - CSV/TSV/Excel: a JSON-encoded column carries its own tree at `columns[].nested_tree` (paths relative to that cell). Object cells list child keys; array cells (`items`-style) expose an `elementKind` and, for object arrays, the union of the element fields.
    Object nodes list child keys, array nodes carry an `elementKind`, primitive leaves carry an inferred type and bounded samples. Summarize node kinds, never paste samples.
-2. **Judge first, then confirm.** Read the node names and inferred types and propose, per container, whether to flatten (and to what depth), keep as JSON, or keep as a list. Present the proposal as a table and default to it; ask the user to confirm or adjust. Only ask node-by-node for the nodes you cannot judge from the names.
-   - Business-entity names with stable scalar children (`user_info`, `address`, `order`) → propose flattening to the leaves.
-   - Generic container names (`payload`, `data`, `config`, `meta`, `extra`, `attributes`) → propose keeping as JSON, unless the user wants a specific sub-field.
-   - Arrays are always kept as a list property and never split.
-   - Nesting deeper than one level with no clear intent → propose keeping as JSON or flattening only the first level.
-3. **Record the answers in `flatten_rules` (`{ "out_column": "dot.path" }`).**
+2. **Review the recommended mapping's per-field decision, then confirm adjustments.**
+   The recommended mapping already encodes, per container, a data-driven depth decision — the depth is not uniform across fields:
+   - **Keep whole** — a single-level object (every child scalar or a scalar array) is declared `type: 'object'` with `transform: 'json'`, plus one `parent.child` sub-property per child; a single-level object array is declared `type: 'array_row'` with one `parent.child` sub-property per element field. The parent entry carries the native value and conversion emits it once; each child is a plan-only declaration (scalar or `list`) that never reads a column of its own.
+   - **Flatten one level / to the leaves** — an object that itself contains an object or object array cannot be kept whole; it collapses one level and each child is decided independently: scalar children become flat properties materialized through `flatten_rules`, and nested objects/arrays recurse until a single-level container is reached (which is then kept whole).
+   - **Scalar array** — `["a","b"]` stays `list` (→ AE `array_string`), a leaf with no children.
+   - **Nested element field** — an element field that is itself an object/array stays inside the array data and is not declared as a sub-property (array-element flattening is not supported); the mapping warns so it is never silently dropped.
+   Present the resulting property list as a table and default to it; ask the user to confirm or adjust only where you disagree with a node's inferred decision (business-entity names with stable scalar children vs generic containers like `payload`/`data`/`config` vs nesting deeper than one level with no clear intent).
+3. **Record overrides in `flatten_rules` (`{ "out_column": "dot.path" }`) and `exclude_columns`.**
+   The recommended mapping already carries the flatten rules for its collapsed levels. When you override it:
    - NDJSON/JSON: the path is from the record root (`user_info.name`).
-   - CSV/TSV: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
-   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; CSV prefixes the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure.
+   - CSV/TSV/Excel: the path is `<column>.<cell-relative path>` (`user_profile.name`), and add the source column to `exclude_columns` so the whole object is not also mapped.
+   - **`flatten_rules` only materializes the out column in the row — it does not emit it.** For every out column, also add a `properties` entry with `source` set to that out-column name (plus `target`/`type`/`desc`); otherwise the flattened value is silently dropped from the output record.
+   A leaf path becomes a snake_case out column when not explicitly named (`user_info.address.geo.lat` → `user_info_address_geo_lat`; cell-relative rules prefix the column: `user_profile.level` → `user_profile_level`). A string that looks like a number (phone, zip, ID) is kept as a string unless the user says otherwise. Containers kept whole are declared `type: 'object'`/`'array_row'`/`'list'` **with `transform: 'json'`** so conversion restores the native structure; a kept whole `object`/`array_row` also declares its scalar children as dotted `parent.child` sub-properties in `properties`.