@thinkingai/ae-cli 6.1.17 → 6.1.19

package/skills/ae-analysis/references/user_tag_create.md

@@ -10,10 +10,16 @@ Flags: `--project-id`, `--tag-name`, `--display-name`, `--definition-request` re
 
 `tag_name` is a machine identifier: 1-80 characters, starts with a letter, and contains only letters, digits, or underscores. `display_name` is 1-80 characters. The CLI rejects violations before dispatch.
 
-Read `user_tag_models.md` before constructing `--definition-request`. Create does not accept `--remark`; set it later with `user-tag update` when needed.
+Read `user_tag_models.md` before constructing `--definition-request`. Dynamic first/last ranges use semantic `time_range` values such as `{"mode":"recent","unit":"month","value":1}` for this month or `{"mode":"start_to_today","start_time":"2026-07-01"}` for a fixed start date through today. Create does not accept `--remark`; set it later with `user-tag update` when needed.
 
 The backend validates and compiles the definition inside the create operation; if metadata is ambiguous or missing, creation fails without creating the tag.
 
 ```bash
 ae-cli analysis user-tag create --project-id <project_id> --tag-name high_value --display-name "High Value" --definition-request '{"type":"condition","condition_values":[{"value":"high","events":[{"event":"pay","operator":"gte","value":3,"aggregation":"count","time_range":{"mode":"recent","unit":"day","value":30}}]}]}'
 ```
+
+First/last tag for this month:
+
+```bash
+ae-cli analysis user-tag create --project-id <project_id> --tag-name latest_platform_this_month --display-name "Latest Platform This Month" --definition-request '{"type":"first_last","first_last":{"event":"login","occurrence":"last","property":"platform","time_range":{"mode":"recent","unit":"month","value":1}}}'
+```

package/skills/ae-analysis/references/user_tag_models.md

@@ -32,10 +32,16 @@ Required: `event`, `aggregation`. `property`, `time_range`, and `filters` are op
 
 ## First/last tag
 
-Required: `event`, `occurrence=first|last`, and exactly one value source: `calculation` or `property`. `time_range` and `filters` are optional. Filters support only event properties and user properties. A string `field` is an event property; use `{name,type:"user_property"}` for a user property. Supplying neither or both value sources is rejected before execution.
+Required: `event`, `occurrence=first|last`, and exactly one value source: `calculation` or `property`. `time_range` and `filters` are optional. Filters support only event properties and user properties. A string `field` is an event property; use `{name,type:"user_property"}` for a user property. Supplying neither or both value sources is rejected before execution. Use the semantic time mappings in [`audience_models.md`](audience_models.md) for dynamic ranges such as today, this month, or a fixed start date through today.
 
 ```json
-{"type":"first_last","first_last":{"event":"login","occurrence":"last","property":"platform","time_range":{"mode":"recent","unit":"day","value":30},"filters":{"relation":"and","items":[{"field":{"name":"country","type":"user_property"},"operator":"eq","values":["US"]}]}}}
+{"type":"first_last","first_last":{"event":"login","occurrence":"last","property":"platform","time_range":{"mode":"recent","unit":"month","value":1},"filters":{"relation":"and","items":[{"field":{"name":"country","type":"user_property"},"operator":"eq","values":["US"]}]}}}
+```
+
+From a fixed date through today:
+
+```json
+{"type":"first_last","first_last":{"event":"login","occurrence":"first","calculation":"specific_time","time_range":{"mode":"start_to_today","start_time":"2026-07-01"}}}
 ```
 
 ## SQL tag

package/skills/ae-data-integration/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-data-integration
-description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it."
+description: "Bring local CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, and XLSX files into AE end-to-end: identify the source's business meaning, generate and confirm a tracking plan, transform rows into UE records, and upload. Also supports privacy-preserving local analysis and handing a small file to AE Agent. Use whenever a user wants to import offline/local data into AE or analyze a file without uploading it. Trigger words: 本地数据导入 / 离线数据 / 数据文件 / 文件导入 / 文件上报 / CSV 导入 / Excel 导入 / TSV 导入 / JSON 导入 / 导入到 AE / 导入到 ThinkingData / local data import / import local file."
 ---
 
 # AE Data Integration
@@ -17,22 +17,42 @@ Two entrances lead here: the AE Agent dialog (attach / plus-button upload) and `
 - Do not invent account IDs, distinct IDs, event times, event names, projects, APPIDs, receivers, or timezones.
 - `value_mapping` and `random_pool` are explicit user decisions. Never invent them.
 - Never auto-fill a missing time for `track`/`track_*` rows. A missing time on user-profile rows may be filled with the current time only by setting `missing_time: 'now'` and only after the user explicitly confirms it.
+- The mapping's fill-in options (`missing_time: 'now'`, `account_id_value`/`distinct_id_value`, `random_pool`, `exclude_columns`, `value_mapping`) record a decision the user has already made; they are never to make a validation failure disappear. When `convert` quarantines rows, present the failure first — the error `code`, the row count, its share of the total, and which events or time ranges are affected — and state the consequence in the user's terms ("3,000 events would carry a time that is not when they happened"). Only after the user has seen that may one of these options be set. When the evidence is insufficient, report the batch as not passed and pending customer data; never shrink the upload scope to manufacture a pass.
 - Do not read or send an AE access token or CLI token to `/sync_json`. The receiver request uses only APPID and UE data.
 - Never execute `data-integration upload` until the user has seen the target, mapping, valid/quarantined counts, batches, and dry-run and has explicitly confirmed that upload.
 - A blocked manifest requires a second, explicit clean-subset decision. Never add `--allow-clean-subset` implicitly.
 - If a batch times out or loses the network, treat that batch as unknown. Stop. Ask the user to verify receiver/AE data before the user chooses `--resume-from`; never resume automatically.
 - Local analysis stays local. AE Agent attachment is a separate, confirmed branch with a 50 MB per-file limit.
 
+## When to use / When NOT to use
+
+Use this skill when the user wants to bring a **local data file** (CSV/TSV/TXT/JSON/JSONL/XLS/XLSX) into AE, or analyze such a file locally without uploading.
+
+| User intent | Use this instead |
+| --- | --- |
+| How to integrate the SDK / tracking code / LogBus2 config / reporting-error triage (usage Q&A, no local file) | ae-data-integration-helper |
+| Database / datasource direct sync (MySQL, DataX, data warehouse, data-dev platform) | ae-dataops |
+| Community content (posts / comments / chat / WeCom groups) insight or submission | ae-community |
+| Generate / upload a project-level tracking plan (source material is PRD / chat / template / code; deliverable is a real platform tracking plan) | ae-generate-tracking-plan |
+| Upload documents / URLs to a knowledge base | ae-kb |
+| Reports / dashboards / queries / governance on data already in AE | ae-analysis |
+
+This skill also produces a tracking-plan draft (`source_type: data`) as a governance prerequisite; that draft is an input to ae-generate-tracking-plan, not a substitute for its five-phase platform plan.
+
 ## Workflow
 
 Walk the four submodules in order. Each submodule is its own reference; follow it and come back here for the next step.
 
 1. **Source — business identification.** Read [references/source-inspect.md](references/source-inspect.md). Profile every file fully, infer its business meaning using business-doc / user-prompt priors, then pick a branch via [references/ue-routing.md](references/ue-routing.md).
-2. **Reuse check.** If a `.ae-data-integration/index.json` exists and the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
-3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). Generate the event/property plan from the profile and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
+2. **Reuse check.** If the profile is `ue_eligible`, read [references/reuse.md](references/reuse.md) and match the recommended mapping against the handoff index. `reuse` searches the current directory's `.ae-cli/data-integration/` upward, then `~/.ae-cli/data-integration/`, so a package written elsewhere is still found. A match proposes a frozen package; after one explicit confirmation, run the returned `transform.mjs` command and jump to Sink (step 5). No match → continue.
+3. **Tracking plan.** Read [references/tracking-plan.md](references/tracking-plan.md). The plan is generated from the mapping (`plan --mapping`), so confirm the recommended mapping's key system fields with the user first — `mode`, `#account_id`/`#distinct_id`, `#time` + timezone, `#event_name`, `#ip`/`#uuid` (see [references/transform.md](references/transform.md) steps 1–5) — then generate the event/property plan and get a single explicit confirmation from the user before touching data. The plan is a separate, required deliverable from the transform mapping: a user who supplies a column→field mapping directly has **not** completed this step, so build the plan from the confirmed mapping anyway. `user_set` still requires a plan (no events; every property becomes a user property). This step runs for **every** file: a second or later file merges its new events and properties into the existing project plan (tracking-plan.md step 4) — an existing plan is never a reason to skip it.
 4. **Transform.** Read [references/transform.md](references/transform.md). Map columns to AE system fields and properties, convert, and quarantine dirty rows per [references/ue-mapping.md](references/ue-mapping.md).
 5. **Sink — upload.** Read [references/sink-upload.md](references/sink-upload.md). Resolve the destination, dry-run, confirm, then upload per [references/sync-json-upload.md](references/sync-json-upload.md). `receiver_accepted` is not persistence: after a ~1-minute ingestion delay, verify the data landed with ae-cli (`tracking live-data list` / `tracking ingest summary` / `tracking ingest-error list`) rather than telling the user to check the console.
-6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (frozen mapping + transform script + plan reference) so the next same-shape file skips the full pipeline.
+6. **Handoff.** Read [references/handoff.md](references/handoff.md). Export the reusable package (pipeline descriptor + frozen mappings + stage executors + docs) and a shareable zip; in the completion response, state the absolute zip path, the package directory, and the one-line way to run the next same-shape file.
+
+## Error handling
+
+When a step fails, classify the failure before acting — see [references/error-handling.md](references/error-handling.md). A quarantined row, a ragged line, and a disk-full are three different problems with three different responses: match on the error `code`, never retry a parse failure by guessing the encoding, and never report a program failure as a data problem.
 
 ## Local analysis branch
 

package/skills/ae-data-integration/references/custom-layer.md

@@ -0,0 +1,93 @@
+# Project custom layer (verify + salvage overlays)
+
+The handoff package is deliberately **standard flow only**: it ships the generic
+stage executors and never embeds environment-specific workarounds such as SQL
+direct queries or multi-round salvage loops. Some projects need those — a hard
+per-event persistence judge, or a repeated salvage loop for a dirty source. Add
+them as a **custom layer next to the package**, never by editing the generic
+package itself.
+
+## When to add a layer
+
+- **Hard persistence judge.** `bin/verify.py` is a soft check: it computes the
+  submit window and expected counts from the local UE output, then shows the
+  `tracking ingest summary` payload before and after for comparison. It does not
+  parse that payload into per-event counts — the capability's `data` shape is
+  server-defined, and a shared project's window delta cannot be attributed to one
+  import. When the project requires an automated per-event ✓/✗ verdict, overlay a
+  SQL layer that reads the project's event/user tables directly.
+- **Multi-round salvage.** `bin/run.sh` already prints a single-round salvage
+  hint when `invalid.rows.jsonl` is non-empty. For sources that fail in layers,
+  wrap that command in a loop that re-feeds each round's quarantine into the next
+  `--salvage-from` until nothing fails or the user stops.
+
+## Rules
+
+1. Put overlays in a sibling directory (e.g. `custom/`) under the package root,
+   and reference `../pipeline.json` / `../index.json` — never modify the frozen
+   mappings, `bin/`, or the generated docs.
+2. Keep every existing confirmation gate. A custom layer may add gates; it must
+   not remove the `--confirm` upload gate or the plan/shape gates.
+3. Keep secrets in `.local/target.env`. Never write APPID, endpoints, tokens, or
+   raw data values into an overlay file.
+4. An overlay is project-specific. Do not copy it into another project's package
+   without re-confirming the environment assumptions (table names, APPID, host).
+
+## Skeleton 1 — SQL verify (hard per-event judge)
+
+The reference deployment queries the ingested tables directly. **Table and
+column names vary by deployment** (`v_event_1` / `v_user_1` and `$part_date` /
+`$part_event` are examples only) — confirm them against the project's receiver
+schema before use, and keep the query read-only.
+
+```bash
+# custom/verify.sh — hard judge; requires the SQL capability to be available.
+# usage: custom/verify.sh <run-dir> [--baseline|--check]
+# Reuses bin/verify.py's window/count computation idea, but answers the question
+# "did exactly these events land?" with a direct count over the event table.
+```
+
+The query shape (adapt table/column names):
+
+```text
+select <event_column>, count(*)
+from <event_table>
+where <date_partition_column> between '<window-start>' and '<window-end>'
+group by 1
+```
+
+Compare the per-event delta (after upload minus before upload) against the
+expected counts from `valid.ue.jsonl`. User-profile rows are overwrite-writes,
+so a zero user-table delta is normal; compare event rows per event, not totals.
+
+Keep this layer read-only and non-blocking: report the verdict, never retransmit
+automatically on a mismatch — first check `tracking ingest-error list` for the
+silent-drop reason.
+
+## Skeleton 2 — multi-round salvage loop
+
+`bin/run.sh` prints the one-round hint. Wrap it into a loop that re-processes each
+round's quarantine against the fixed mapping until clean or the user stops:
+
+```bash
+# custom/salvage.sh <run-dir> — re-process quarantined rows round by round.
+# Each round's valid.ue.jsonl is disjoint from earlier rounds', so upload each
+# round independently (same --confirm and --allow-clean-subset gates).
+round=1
+while [ -s "<run-dir>/invalid.rows.jsonl" ]; do
+  ae-cli data-integration convert \
+    --input-file '<same-source>' \
+    --mapping '<fixed-mapping.json>' \
+    --salvage-from "<run-dir>/invalid.rows.jsonl" \
+    --output-dir "<run-dir>-salvage-$round"
+  round=$((round + 1))
+  # stop condition is the user's: a row that still fails after a fix is a real
+  # data defect, not a code bug.
+done
+```
+
+See [transform.md](transform.md) for the `--salvage-from` semantics (single-file
+only; the source must be the same file that produced the quarantine) and
+[handoff.md](handoff.md) for the package the overlay attaches to. Persistence
+verification always follows [sink-upload.md](sink-upload.md): `receiver_accepted`
+is not durability.

package/skills/ae-data-integration/references/error-handling.md

@@ -0,0 +1,145 @@
+# Error handling
+
+Failures in the data-integration pipeline split into three classes. Each class has a distinct
+response; an agent that mixes them up wastes retries and misleads the user.
+
+| Class | What it is | Response |
+| --- | --- | --- |
+| Abnormal data | Rows or fields that do not meet UE rules | Quarantine the row or skip the field, report counts, salvage |
+| File parsing exception | The source cannot be read as its detected format | Surface the parse error, do not retry by guessing; fix the file or format |
+| Program execution exception | The program itself failed (disk, permissions, output dir, mapping mismatch) | Surface the exact error, fix the environment/inputs, re-run |
+
+Every error leaves the CLI as the standard envelope `{ ok, data, error: { type, code, message, hint } }`
+(JSON on stdout; progress and warnings on stderr). The `code` is the stable identifier an agent
+branches on; `hint` is human-readable guidance.
+
+## Class 1 — Abnormal data
+
+Happens inside `convert`. Two severities:
+
+- **Whole-row quarantine.** Any error on a row — identity, time, event name, record type, or a
+  single property (type coercion, size/limit) — drops the entire row. The row is written to
+  `invalid.rows.jsonl` with its error codes and counted in `manifest.output.invalid_records`; the
+  manifest becomes `blocked`. Row-level codes: `MISSING_USER_ID`, `USER_ID_TOO_LONG`,
+  `INVALID_RECORD_TYPE`, `INVALID_TIME`, `TIME_OUT_OF_RANGE`, `INVALID_EVENT_NAME`,
+  `INVALID_ZONE_OFFSET`, `PROPERTY_TYPE_CONFLICT`, `PROPERTY_LIMIT_EXCEEDED`.
+- **Field skip.** A `#ip` or `#uuid` value that violates its spec drops only that field and keeps
+  the row. Counts land in `manifest.output.skipped_fields`; private/LAN IPs are kept but counted in
+  `manifest.output.lan_ip_records`. Field-skip codes: `INVALID_IP`, `INVALID_UUID`. Never quarantined.
+
+Responses:
+
+- A blocked manifest is not a failure to retry blindly. Fix the mapping, then re-run `convert
+  --salvage-from <invalid.rows.jsonl>` to re-process only the quarantined rows (never re-send the
+  rows that already passed). Repeat the salvage loop until no rows fail or the user stops.
+- An empty source (zero data rows) blocks with the reason `The source contained no data rows.` —
+  a distinct, clearer message than a failed validation run. Do not re-run the same command on the
+  same empty file expecting a different result.
+- Ragged delimited rows (a CSV/TSV row whose column count differs from the header) are tolerated:
+  extra fields are dropped and missing fields are treated as empty, and a stderr warning reports
+  how many rows were ragged. A row that becomes missing its identity/time because a field was
+  absent is then quarantined normally. Treat the warning as a data-quality signal for the user.
+
+## Class 2 — File parsing exceptions
+
+The source cannot be parsed as its detected format. The agent's job is to report precisely and let
+the user decide — never to guess an encoding or structure and retry silently.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_INPUT_NOT_FOUND` | Path is not a readable file | Ask for the correct path |
+| `LOCAL_DATA_FILE_TOO_LARGE` | XLS over 1 GB | Convert to XLSX or split the workbook |
+| `LOCAL_DATA_INPUT_INVALID` | Generic parse failure (malformed CSV/TSV/JSON/XLS) | Verify encoding and structure, then retry without changing the source |
+| `LOCAL_DATA_JSONL_INVALID` | A JSONL line is not valid JSON (`location.record` names the line) | Point at the offending record |
+| `LOCAL_DATA_JSON_ROOT_INVALID` | JSON root is not an object or array | Check the file's top-level shape |
+| `LOCAL_DATA_XLSX_INVALID` | Workbook metadata missing / no readable sheets / worksheet entry missing | Re-export the workbook (sheet recognition accepts `<sheet>` and namespaced `<x:sheet>` alike, so this is a real metadata gap) |
+| `LOCAL_DATA_SET_NOT_FOUND` / `LOCAL_DATA_SET_REQUIRED` | Sheet or JSON Path not found / ambiguous | Ask which `--data-set` to use |
+
+A parse error is never a reason to change the mapping or the tracking plan. Report the code and
+hint verbatim, and ask the user to fix the file (re-export, re-encode, or split).
+
+## Class 3 — Program execution exceptions
+
+The program itself failed; the source data is usually fine. These must never be mislabeled as
+parse errors: a per-row callback failure (for example a disk that filled up mid-convert) propagates
+as itself, not as `LOCAL_DATA_INPUT_INVALID`.
+
+| Code | Meaning | What to do |
+| --- | --- | --- |
+| `LOCAL_DATA_OUTPUT_NOT_EMPTY` | The output directory must be new or empty | Point convert at a fresh `<run-id>` directory |
+| `LOCAL_DATA_SOURCE_CHANGED` / `LOCAL_DATA_SOURCE_FORMAT_CHANGED` | The source no longer matches the mapping fingerprint/format | Re-run inspect and review a new mapping |
+| `LOCAL_DATA_MAPPING_INVALID` / `LOCAL_DATA_MAPPING_INVALID_JSON` / `LOCAL_DATA_MAPPING_NOT_FOUND` | The mapping cannot be read or validated | Re-read the mapping reference, fix the mapping file |
+| `LOCAL_DATA_SALVAGE_INVALID` / `LOCAL_DATA_SALVAGE_EMPTY` / `LOCAL_DATA_SALVAGE_NO_MATCH` | The salvage file is not a valid quarantine file, is empty, or lists no rows from this source | Point at the correct `invalid.rows.jsonl` from the same source |
+| `LOCAL_DATA_TYPE_CONFLICTS_UNRESOLVED` / `LOCAL_DATA_TYPE_RESOLUTIONS_INVALID` | Cross-file column type conflicts need explicit resolutions | Build `--type-resolutions` |
+| `LOCAL_DATA_PLAN_INVALID_EVENT_NAME` / `LOCAL_DATA_PLAN_EVENT_NAMES_REQUIRED` / `LOCAL_DATA_PLAN_INVALID_LANG` | Tracking-plan draft inputs are invalid | Fix the event name(s) or `--lang` |
+| `LOCAL_DATA_HANDOFF_INDEX_INVALID` / `LOCAL_DATA_HANDOFF_PLAN_NOT_FOUND` / `LOCAL_DATA_HANDOFF_PLAN_INVALID` | Handoff index/plan cannot be read | Point at a valid handoff directory/plan file |
+| `LOCAL_DATA_ENDPOINT_INVALID` / `LOCAL_DATA_APPID_INVALID` / `LOCAL_DATA_BATCH_SIZE_INVALID` / `LOCAL_DATA_COMPRESS_INVALID` / `LOCAL_DATA_RESUME_INVALID` / `LOCAL_DATA_RESUME_OUT_OF_RANGE` | Upload arguments are invalid | Fix the flag before uploading |
+| `LOCAL_DATA_MANIFEST_INVALID` / `LOCAL_DATA_UE_FILE_INVALID` / `LOCAL_DATA_UE_FILE_NOT_FOUND` / `LOCAL_DATA_UE_FILE_CHANGED` / `LOCAL_DATA_UE_COUNT_MISMATCH` / `LOCAL_DATA_MANIFEST_FILE_MISMATCH` | Upload preconditions fail | Re-check the manifest and UE file pairing |
+| `LOCAL_DATA_CLEAN_SUBSET_CONFIRMATION_REQUIRED` | Uploading from a blocked manifest needs a separate clean-subset decision | Confirm the subset explicitly before `--allow-clean-subset` |
+
+Write failures (disk full `ENOSPC`, permission denied `EACCES`) do not hang the command: the
+output streams fail with a clear `Failed to write "<path>"` message telling the user to check disk
+space and directory permissions. Fix the environment, then re-run into a fresh output directory.
+
+## File-level data-quality severity
+
+The classes above are per row or per field. The user also needs one verdict for the file as a whole,
+and its signals arrive scattered across the manifest and stderr. Grade them before reporting so the
+same file gets the same verdict no matter who reports it.
+
+| Severity | Signal | Handling |
+| --- | --- | --- |
+| Critical | Established with the user to be a cumulative snapshot or an aggregate report (see [ue-routing.md](ue-routing.md)); a required identity or time column empty for the whole file | Do not upload. Resolve with the user first |
+| High | `manifest.output.invalid_records` is a large share of the row count; inspect reported `leading_title_rows` or `header_signal` and the user has not yet said whether those rows are a title/banner; inspect reported `xlsx_structure.merged_covered_cells` and the user has not yet said whether the merged label belongs on the rows the block covers; `summary_rows` was reported and the user has not yet said whether those rows are totals or real records; `duplicate_keys` was reported and the user has not yet said whether the repeated rows are separate observations | Upload is allowed, but name the item explicitly in the confirmation gate and in the completion response |
+| Medium | `manifest.output.skipped_fields`; ragged delimited rows; `manifest.output.flatten_misses`; `manifest.output.unreadable_cells`; `xlsx_structure.hidden_rows` or `hidden_columns` read as data | Report the counts; the gate is unchanged |
+| Low | `manifest.output.lan_ip_records` | Report once |
+
+Rules:
+
+- Severity is reported, never silently applied. A Critical finding stops the pipeline and is stated
+  as a data finding, not as a program error.
+- Grade only what the run actually emitted. Every signal that belongs at High or Critical above now
+  has a detector, so grade it from what the run reported — never tell the user the tool checked
+  something it did not.
+- `unreadable_cells` counts XLSX cells that carried no value the tool may use, grouped by cause:
+  `formula_no_cached_value` (the file stores a formula but not the result Excel last computed),
+  `error_value` (`#N/A`, `#DIV/0!`, …), `unreadable_object` (an unrecognized cell shape). The cells
+  read as missing and their rows are kept, so the record count says nothing about them — a column
+  that is empty in AE while the spreadsheet looks full is this. The tool never evaluates a formula
+  and never guesses a result; the fix is to recalculate and re-export in Excel, or to export values
+  instead of formulas. `inspect` reports the same counts before conversion.
+- `xlsx_structure` records worksheet layout the rows themselves cannot carry: `merged_covered_cells`
+  counts cells that are empty only because a merged block covers them (Excel shows the value on the
+  block's first row), `hidden_rows` and `hidden_columns` name what the worksheet hides. The default
+  read changes none of it, so these counts describe what was uploaded: a column mostly missing in AE
+  while the spreadsheet looks full is the first of them. `merged_cells_filled` and
+  `excluded_hidden_rows` say what the run did about it, which is only ever what the user asked for
+  via `--fill-merged-cells` / `--exclude-hidden-rows` (mapping: `fill_merged_cells` /
+  `exclude_hidden_rows`). XLSX only — a legacy `.xls` workbook is not scanned.
+- `summary_rows` names rows that read as a summary line rather than an observation, by `row` (the
+  data-row ordinal) and `signals`: `total_label` (a cell reads as `合计` / `总计` / `小计` / `汇总` / `Total` /
+  `Subtotal`) and `column_total` (a number equal to the total of its column's other rows). The rows
+  were converted like any other, so this is a finding about what was uploaded: one fabricated event
+  whose amount is the whole group's, and a column whose reported `sum` is twice its real total. There
+  is no flag that drops a data row, because a row labelled `合计` is sometimes a real record; the fix
+  is to remove it from the source file or re-export without it. Any format, `.xls` included.
+- `duplicate_keys` names rows the source repeated under the same business key, by `key_columns` (the
+  columns compared), `duplicate_groups`, `extra_rows` (surplus records an upload would carry), and
+  `groups` with `count`, the data-row `rows`, and a `key_hash` prefix — never the key's own values.
+  Nothing was removed: a repeat is sometimes a real pair of records, two order lines in the same
+  checkout second, and AE appends accepted events with no way to un-send one, so ask the user whether
+  the rows are separate observations; if not, have them remove the rows from the source file. Values
+  are compared as written, so a repeat spelled two ways is missed, and `tracking_truncated` means
+  distinct keys outran the scan's budget and there may be more. Any format, `.xls` included.
+- A large quarantine share has no fixed threshold. State the ratio and the dominant error `code`,
+  and let the user judge.
+
+## Cross-cutting rules
+
+- Classify first, act second. Match on `code`, not on message text.
+- A parse error is not a data problem; a program error is not a parse error. Do not conflate them
+  when reporting back to the user.
+- Never retry a parse failure by guessing the encoding, delimiter, or header layout. Show the
+  error and ask.
+- After any fix, re-run from the start of the step that failed; do not resume a half-written run
+  or reuse a partially populated output directory.

package/skills/ae-data-integration/references/handoff.md

@@ -1,43 +1,102 @@
 # Handoff (reusable package)
 
-After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package freezes the transform logic (the confirmed mapping), the runnable transform script, and the tracking-plan reference. It does **not** re-freeze the raw data or any upload secrets.
+After a successful run, export a **handoff package** so the next file of the same shape skips the full pipeline. The package is a DataX-style pipeline — a declarative `source → transform → sink` descriptor plus generic stage executors that dispatch to `ae-cli data-integration` subcommands. It freezes the transform logic (the confirmed mappings) and the tracking-plan reference, and ships a `bin/` directory that a human or agent can run directly. It never re-freezes raw data or upload secrets.
 
 ## When
 
-Run handoff after Transform — or after Sink — once the mapping is confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
+Run handoff after Transform — or after Sink — once the mappings are confirmed. It is local-only and idempotent: re-running it for the same table refreshes the package in place.
 
 ## CLI
 
 ```
-ae-cli data-integration handoff --mapping <mapping> [--plan-file <draft.json>] [--out-dir .ae-data-integration]
+ae-cli data-integration handoff --mapping <mapping> [--mapping <mapping2> ...] [--plan-file <draft.json>] [--pushurl <url>] [--project-id <id>] [--out-dir .ae-cli/data-integration]
 ```
 
-- `--mapping` — the confirmed `ae-local-data-mapping/v1` mapping (the frozen transform logic, including `value_mapping` and `flatten_rules`).
-- `--plan-file` — optional tracking-plan `draft.json` to reference inside the package (`plan.json`).
-- `--out-dir` — handoff root. Default `.ae-data-integration/` (travels with the project).
-- `--dry-run` previews the fingerprint, files, and index path without writing.
+- `--mapping` — one or more confirmed `ae-data-integration-mapping/v1` mappings (the frozen transform logic, including `value_mapping` and `flatten_rules`). Repeat it for multi-sheet workbooks: one mapping per sheet.
+- `--plan-file` — optional tracking-plan `draft.json` to reference inside each mapping directory (`plan.json`).
+- `--pushurl` — optional receiver base URL to record as the reuse upload target (the sink endpoint is `pushurl` + `/sync_json`). Record it when the next same-shape file will most likely land at the same receiver.
+- `--project-id` — optional numeric destination project ID to record; `bin/upload.sh` derives the APPID from it via `project info get` at upload time.
+- `--out-dir` — handoff root. Default `.ae-cli/data-integration/` (project workspace; travels with the project).
+- `--dry-run` previews the fingerprints, target, file list, and zip path without writing.
 
 ## Package layout
 
 ```
-.ae-data-integration/
-  index.json               ← shared fingerprint index (reuse detection)
-  <fingerprint[:16]>/
-    mapping.json           ← frozen mapping (re-runnable by convert)
-    transform.mjs          ← node transform.mjs <new-file> [<output-dir>]
-    plan.json              ← optional tracking-plan reference
+.ae-cli/data-integration/    ← handoff root (= out-dir, project workspace)
+  pipeline.json             ← source → transform → sink descriptor (ae-data-integration-pipeline/v1)
+  index.json                ← shared structure-fingerprint index (reuse detection)
+  shape.json                ← per-mapping column baseline (shape gate)
+  <fingerprint[:16]>/       ← one directory per mapping
+    mapping.json            ← frozen mapping
+    transform.mjs           ← node transform.mjs <new-file> [<output-dir>]
+    plan.json               ← optional tracking-plan reference
+  bin/                      ← generic stage executors (read pipeline.json)
+    run.sh                  ← source → transform → plan (never uploads)
+    upload.sh               ← sink (dry-run by default; --confirm uploads; resolves recorded target)
+    bind_mapping.py         ← shape check + rebind sha256/data_set to the new file
+    summarize.py            ← valid/quarantined counts
+    plan_check.py           ← tracking-plan coverage gate (exit 3 on new events/properties)
+    verify.py               ← soft persistence check (submit window vs ingest summary)
+    resolve_appid.py        ← APPID derivation helper (project info get)
+  README.md / RUNBOOK.md    ← how to run, the four gates, persistence verification
+  .local/target.env.example ← destination template (no real secrets)
+  .gitignore                ← inbox/ runs/ .local/target.env
+  inbox/  runs/             ← daily input / per-run outputs
 ```
 
-The `transform.mjs` wrapper shells out to `ae-cli data-integration convert`; it takes the new file path as its first argument, never bakes the source path, and re-stamps `source.sha256` with the new file's fingerprint before converting. Reuse it only for a file of the **same shape** (same header/schema and format) — the transform logic is frozen, but the content guard is re-bound to each specific file.
+A shareable archive is written next to the package root: `<parent>/ae-data-integration-handoff-<fingerprint[:8]>.zip`. The zip carries only this handoff round — the frozen mappings just written, a scoped `index.json` (just this round's entries), and the generic executors/docs — not the accumulated history, which stays in `.ae-cli/data-integration/` for reuse matching.
+
+## Pipeline descriptor
+
+`pipeline.json` declares the three stages and their types. Only `source: local_file` and `sink: restful_sync_json` are implemented today; the `type` fields reserve logbus / datax / mysql for later phases. `bin/run.sh` and `bin/upload.sh` read the descriptor and dispatch each stage by its `type` to `ae-cli data-integration inspect / convert / upload` — they are executors, not a second runtime engine.
+
+## Recorded destination
+
+`pipeline.json` → `sink.params` records `pushurl` and `project_id` when the handoff
+was run with those flags. Reuse defaults to that target, but **`bin/upload.sh`
+never sends without `--confirm`**, so the operator re-confirms the address and
+project on every reuse. Resolution order at upload time:
+
+- endpoint: recorded `pushurl` (+ `/sync_json`), else `AE_ENDPOINT`.
+- APPID: `AE_APPID`, else derived via `ae-cli project info get --project-id <id>`
+  (see `bin/resolve_appid.py`; set `AE_APPID` when that payload lacks `appid`).
+- project id: recorded `project_id`, else `AE_PROJECT_ID`.
+
+`project info get` returns `data.appid` at the top level (verified against the AE
+demo host); `bin/resolve_appid.py` reads that exact field and prints it, falling
+back to `AE_APPID` when the field is absent or not a non-empty string.
+
+## Project custom layers
+
+For projects that need a hard per-event SQL judge or a multi-round salvage loop
+(the reference package's environment-specific workarounds), overlay a custom layer
+next to the package instead of editing it — see [custom-layer.md](custom-layer.md).
+
+## Four confirmation gates
+
+The RUNBOOK and the scripts enforce four gates. The first two run automatically; the last two always need human confirmation:
+
+1. **Shape gate** — `bind_mapping.py` compares the new file's column set against `shape.json` and fails fast on a mismatch. A changed shape means the frozen logic was never reviewed for it: re-run the full pipeline.
+2. **Transform** — `ae-cli data-integration convert` per mapping; quarantined rows go to `invalid.rows.jsonl`, never silently dropped.
+3. **Tracking-plan gate** — `plan_check.py` verifies every produced event/property already exists in `plan.json`; new ones exit 3 and must be merged into the project plan first.
+4. **Sink gate** — `upload.sh` is dry-run by default; `--confirm` is the explicit upload decision.
 
 ## Structure fingerprint and index
 
-Every handoff records one entry in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: columns (source name + type), event model (`mode`, `event_name_field`, `record_type_field`), identity fields, and excluded columns. Business logic (`value_mapping`, transforms, `time_format`, and the fixed `default_event_name`) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+Every handoff records one entry per mapping in `index.json` (`ae-data-integration-index/v1`) keyed by a **structure fingerprint** — a SHA-256 over the table shape: the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model (`mode`). Business logic (`value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, the fixed `default_event_name`, and the system-field assignments) is excluded, so re-handing off the same table with new business rules refreshes the existing entry instead of forking a new one.
+
+Reuse matching is its own step — see [references/reuse.md](reuse.md).
 
-Reuse matching is its own step — see [references/reuse.md](reuse.md). It compares a new file's profile structure against this index and proposes the matching package, which the user confirms before skipping the full pipeline.
+## Completion response
+
+After handoff succeeds, state the **absolute zip path**, the package directory, and the one-line way to run the next same-shape file:
+
+```
+cd <out-dir> && bin/run.sh <new-file>   # then bin/upload.sh runs/<run-id> --confirm
+```
 
 ## Safety rules
 
-- Treat the mapping, plan, generated artifacts, and the `.ae-data-integration/` directory as sensitive.
-- Never write APPID, endpoints, tokens, or raw data values into a handoff package. The package references the plan and the mapping; uploads still require an explicit, confirmed `upload` call.
+- Treat the mappings, plans, generated artifacts, the `.ae-cli/data-integration/` directory, and the zip as sensitive.
+- Never write APPID, tokens, or raw data values into a handoff package. The package records at most a destination `pushurl` and `project_id`; uploads still require an explicit, confirmed `upload` call, so the operator re-confirms the address and project each time.
 - Do not invent a mapping or plan. Handoff only packages what the user already confirmed.

package/skills/ae-data-integration/references/local-analysis.md

@@ -1,6 +1,6 @@
 # Local analysis
 
-Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/<run-id>/` with restrictive permissions.
+Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/runs/<run-id>/` with restrictive permissions.
 Set the directory to `0700` and generated scripts/reports to `0600`.
 
 ## Default report when no question is supplied

package/skills/ae-data-integration/references/reuse.md

@@ -4,21 +4,25 @@ Before walking the full pipeline for a new file, check whether a **handoff packa
 
 ## When
 
-Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-data-integration/index.json` exists. It is read-only and makes no writes.
+Run reuse right after `inspect`, only when the profile is `ue_eligible` and a `.ae-cli/data-integration/index.json` exists. It is read-only and makes no writes.
 
 ## CLI
 
 ```
-ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-data-integration]
+ae-cli data-integration reuse --mapping <recommended_mapping> [--out-dir .ae-cli/data-integration]
 ```
 
 - `--mapping` — the candidate mapping, typically `inspect`'s `recommended_mapping` (the new file's structure, inferred exactly as Source would infer it).
-- `--out-dir` — handoff root. Default `.ae-data-integration/`.
+- `--out-dir` — handoff root. Optional. Without it, `reuse` searches in order: the current directory's `.ae-cli/data-integration/`, then each parent directory upward, then `~/.ae-cli/data-integration/` as a global fallback — so a package written in another directory or another session is still reachable.
 - `--dry-run` previews the fingerprint and match verdict without reading frozen packages.
 
+## Search paths
+
+When `--out-dir` is omitted, the result carries `searched_paths` — the exact `index.json` paths probed, in order. A `matched: false` result with a non-empty `searched_paths` means all of them were checked; a missing global fallback (no `$HOME`) simply omits it from the list.
+
 ## Matching
 
-The command computes the same **structure fingerprint** the handoff index is keyed on — columns (source name + type), event model, identity fields, and excluded columns — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`) is not part of the key, so a same-shape file with different content or a different file name still matches.
+The command computes the same **structure fingerprint** the handoff index is keyed on — the raw source columns (by name, reconstructed so flatten, exclude, and account-vs-distinct decisions don't move it), the format, and the event model — and looks it up in `index.json`. Business logic (the frozen event name, `value_mapping`, transforms, `time_format`, `flatten_rules`, `exclude_columns`, and the system-field assignments) is not part of the key, so a same-shape file with different content or a different file name still matches.
 
 ## Result
 
@@ -26,7 +30,7 @@ The command computes the same **structure fingerprint** the handoff index is key
 - **Match** (`matched: true`): the result carries the matched package — `mapping_file`, optional `plan_file`, the frozen `default_event_name` (so the user sees which event name will be reused), and a `run` command:
 
 ```
-node .ae-data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
+node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file> [<output-dir>]
 ```
 
 ## Confirmation gate

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,7 +31,7 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
-Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type, grouped into event properties (`track`) and user properties (profile modes) — never a counts-only summary. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
+Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.