@thinkingai/ae-cli 6.1.14 → 6.1.17

package/skills/ae-data-integration/references/ue-mapping.md

@@ -0,0 +1,142 @@
+# UE mapping contract
+
+The mapping version is `ae-local-data-mapping/v1`.
+
+Required structure:
+
+```json
+{
+  "version": "ae-local-data-mapping/v1",
+  "source": {
+    "sha256": "<source-sha256>",
+    "format": "csv",
+    "data_set": "$"
+  },
+  "mode": "track",
+  "confidence": "high",
+  "account_id_field": "user_id",
+  "time": {
+    "field": "event_time",
+    "format": "auto",
+    "source_timezone": "Asia/Shanghai"
+  },
+  "event_name_field": "event_name",
+  "properties": [
+    { "source": "amount", "target": "amount", "type": "number" }
+  ]
+}
+```
+
+Use `distinct_id_field` when no account field exists. `mixed` requires `record_type_field`. Track requires `event_name_field` or a reviewed `default_event_name`. `source.format` is one of `csv`, `tsv`, `json`, `jsonl`, `xls`, `xlsx`. For a multi-file template mapping, `source.sha256` may be the wildcard `*`.
+
+## Record types
+
+Every record carries a `#type`. `mode: 'mixed'` resolves the type per row via `record_type_field`; otherwise the mode fixes it (`track` or `user_set`). The `record_type_field` values are normalized case- and underscore-insensitively, with `event`→`track` and `user`/`userset`→`user_set` aliases.
+
+| `#type` | Meaning | Requires `#event_name` |
+| --- | --- | :---: |
+| `track` | Report an event into the event table | Yes |
+| `user_set` | Overwrite user properties (create if missing) | No |
+| `user_setOnce` | Initialize a property only when empty | No |
+| `user_add` | Increment numeric user properties | No |
+| `user_unset` | Clear user property values | No |
+| `user_del` | Delete the user from the user table | No |
+| `user_append` | Append elements to list properties | No |
+| `user_uniq_append` | Append elements with deduplication | No |
+
+## System fields
+
+Top-level `#` fields map from named source columns. The user confirms each mapping; never infer it from a column name alone — a `user_id` column can be either an anonymous or a login ID, and only the user knows which.
+
+| Field | Plain-language meaning | Required |
+| --- | --- | :---: |
+| `#distinct_id` | Anonymous visitor ID (device/cookie/visitor) — identifies unauthenticated traffic | At least one of `#distinct_id` / `#account_id` |
+| `#account_id` | Login/account ID (database `user_id`, phone, member ID) — identifies authenticated users | At least one of `#distinct_id` / `#account_id` |
+| `#time` | Event/profile occurrence time; AE bins data by it | Yes |
+| `#event_name` | What the user did (`purchase`, `login`) | Only for `track` |
+| `#ip` | Client IP; AE resolves geo from it | No |
+| `#uuid` | Short-window deduplication ID | No |
+
+`#zone_offset` is a preset property that tells AE the data's UTC offset (whole hours, -12..14) so `#time` is interpreted correctly. Unlike the fields above it lives **inside `properties`**, not at the top level. Provide it via `zone_offset_value` (a whole-hour integer such as `8` for UTC+8; an IANA name is also accepted and resolved to its offset at conversion time — sub-hour zones round to the nearest whole hour and DST zones reflect the offset then in effect, so historical data crossing a DST boundary should use `zone_offset_field`) or `zone_offset_field` (a source column carrying the offset per row); the two are mutually exclusive. Rows whose `zone_offset_field` value is missing or not an integer in -12..14 are quarantined.
+
+Mapping keys: `account_id_field`, `distinct_id_field`, `time.field`, `event_name_field` (or a reviewed `default_event_name`), `ip_field`, `uuid_field`. Inspect surfaces every identity-shaped column in `identity_candidates` (name, `account`/`distinct` kind, unique and missing ratios) so the agent can present all candidates for confirmation. When the required identity column is absent, use an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool` — a user decision, never invented.
+
+## Output record shape
+
+System fields sit at the top level; every mapped property is nested under `properties`.
+
+```json
+{ "#type": "track", "#time": "2026-08-10 10:00:00.000", "#account_id": "u-1", "#event_name": "purchase", "properties": { "amount": 99.9, "channel": "app" } }
+```
+
+```json
+{ "#type": "user_set", "#time": "2026-08-10 10:00:00.000", "#distinct_id": "d-1", "properties": { "user_level": 5, "user_tag": "vip" } }
+```
+
+`track` requires `#event_name`; user-profile types (`user_set`, `user_add`, …) do not.
+
+## Optional overlay fields
+
+All fields below are optional and are explicit user decisions — never invent them.
+
+| Field | Shape | Effect |
+| --- | --- | --- |
+| `time_format` | `string` (strptime pattern) | Overrides auto-detection for `time.field`; used for ambiguous US/EU dates |
+| `value_mapping` | `{ account_id? / distinct_id? / event_name? / record_type?: {original: replacement} }` | Exact-key replacement per system field (business-data keys map to AE-name values). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded |
+| `account_id_value` / `distinct_id_value` | `string` | Fixed placeholder identity (≤128 chars). Applies to every row when the corresponding `*_field` is absent; when the field exists, fills only the rows whose column is empty |
+| `random_pool` | `{ account_ids?: string[], distinct_ids?: string[] }` | Synthesizes a random identity when the source field is absent |
+| `exclude_columns` | `string[]` | Source columns skipped when building properties |
+| `flatten_rules` | `{ outColumn: 'dot.path' }` | Nested flatten map. NDJSON/JSON paths are from the record root (`user_info.name`); CSV/TSV paths are `<column>.<cell-relative path>` into a JSON-encoded object cell (`user_profile.name`) — add the source column to `exclude_columns` when flattening it |
+| `headers` | `string[]` | User-confirmed column names for a headerless file; presence means the first row is data. Never use inspect's `col_1..col_N` placeholders — infer names from each column's values, confirm them with the user, then write them here |
+| `missing_time` | `'now'` | Fill a missing/empty `#time` with the current time, for user-profile rows only (explicit user decision; track rows are never filled) |
+| `ip_field` | `string` | Source column emitted as the top-level `#ip` system field (client IP; AE resolves geo) |
+| `uuid_field` | `string` | Source column emitted as the top-level `#uuid` system field (short-window deduplication ID) |
+| `zone_offset_value` | `number` (integer -12..14) or IANA `string` | Emits the fixed `#zone_offset` preset property inside `properties`. An IANA name resolves to its integer UTC offset |
+| `zone_offset_field` | `string` | Source column whose per-row integer value (-12..14) is emitted as `#zone_offset`; missing/non-integer rows are quarantined. Mutually exclusive with `zone_offset_value` |
+| `event_meta` | `{ <event-name>: { desc?: string, tag?: string } }` | Per-event business description and `event_tag` for the tracking plan, keyed by AE event name. Inferred from the data and user context; the user supplies anything not inferable — never leave them empty |
+
+Each `properties` entry may also carry `value_mapping` (per-property exact-key replacement), `transform` (one of `stringify`, `number`, `boolean`, `json`), `time_format` (only meaningful for `type: 'datetime'`), and `desc` (business description for the tracking plan; inferred, or user-provided when not inferable). Container columns (`object`/`list`) whose values arrive as JSON text — JSON-encoded CSV cells, or flattened NDJSON leaves — must set `transform: 'json'` so conversion parses the text back into a native object/array; it is a safe no-op when the value already arrived native.
+
+## Time formats
+
+Values with an explicit offset or `Z` are parsed by the JavaScript `Date` constructor (authoritative). Other values are matched against 21 formats; a `time_format` field overrides the match.
+
+| Category | Example | Auto-detected |
+| --- | --- | :---: |
+| Standard AE | `2024-01-15 10:30:00.123` / `2024-01-15 10:30:00` | Yes |
+| ISO 8601 | `2024-01-15T10:30:00` / with `.SSS` / with offset | Yes |
+| Unix epoch | `1705314600` (seconds) / `1705314600000` (milliseconds) | Yes |
+| Slash-separated | `2024/01/15 10:30:00` / `2024/01/15` | Yes |
+| Dot-separated | `2024.01.15 10:30:00` / `2024.01.15` | Yes |
+| Compact digits | `20240115103000` / `202401151030` / `20240115` | Yes |
+| English month names | `15 Jan 2024 10:30:00` / `January 15 2024 10:30:00` / `Jan 15 2024 10:30:00` | Yes |
+| Chinese date | `2024年1月15日` / `2024年1月15日 10:30:00` / `2024年1月15日 10时30分00秒` | Yes |
+| US format | `01/15/2024 10:30:00` (MM/DD/YYYY) | No — use `time_format` |
+| EU format | `15/01/2024 10:30:00` (DD/MM/YYYY) | No — use `time_format` |
+
+## Data rules
+
+- **Ingestion time window.** The receiver accepts event/profile times from 3 years before to 3 days after the server time; the CLI enforces this range. Client-side reporting has a tighter window (10 days before to 3 days after); historical data beyond the window needs AE support to extend it.
+- **Property type locking.** A property's type is locked on first receipt, and properties sharing a name across events are one property — later reports must use the same type. Values whose type mismatches the locked type are dropped silently, so review an inferred type before committing it.
+- **Quote stripping.** Paired single or double quotes around a cell value are stripped (`'user_001'` → `user_001`); a single-sided quote is kept as data (SQL exports often quote every value).
+- **Column name sanitization.** Recommended mapping auto-names properties: accents are stripped, camelCase is split to snake_case, everything is lowercased, and illegal characters become `_`; a digit-leading name is prefixed `field_`, and a name with nothing recognizable falls back to `field_N`. Review these targets — especially the `field_N` fallbacks — and rename before converting.
+- **Object sub-property keys.** Object and list-of-object keys follow the same naming rules as property names (lowercase snake_case, letter-leading, at most 50 chars, no `#`). A key that does not — for example a Chinese key inside a JSON cell such as `{"等级":"金牌"}` — quarantines the whole row, so confirm object keys before converting.
+
+## Review checklist
+
+- Source SHA-256 and data-set ID match the inspection result (or the mapping uses the `*` wildcard for multi-file).
+- Identity values remain strings and are at most 128 characters.
+- Source timezone is an IANA name derived from user/project context.
+- `#zone_offset`, when set, is a whole-hour integer in -12..14 (or an IANA name/column that resolves to one) and is emitted inside `properties`, never at the top level.
+- Event/property names are lowercase snake_case, begin with a letter, and are at most 50 characters.
+- Target property names are unique and do not collide with UE system fields.
+- Types are one of `string`, `number`, `boolean`, `datetime`, `list`, or `object`.
+- Text is at most 2 KB; numbers stay within -9E15..9E15.
+- Lists contain at most 500 strings (255 bytes each) or 500 objects.
+- Objects contain at most 100 legal sub-properties; nested values follow the same type limits.
+- A conversion rule does not hide a real type conflict.
+- Event/profile times fall within the receiver window: previous 3 years through next 3 days.
+- Do not fabricate UUIDs, identities, times, or events; `#ip`/`#uuid` map from named source columns only (`ip_field`/`uuid_field`).
+- `value_mapping`, `random_pool`, and fixed `account_id_value`/`distinct_id_value` came from an explicit user decision and match the actual distinct values/columns.
+
+`user_set` output for the same identity is ordered by time so receiver application order is deterministic. Conversion applies whole-row quarantine: any error on a row — identity, time, event, record type, or a single property (type coercion, size/limit) — drops the entire row. The row is written to `invalid.rows.jsonl` with its error codes, counted in `manifest.output.invalid_records`, and the manifest is blocked until reviewed. The failed rows are re-reportable without re-sending valid rows: fix the mapping, then run `convert --input-file <same-source> --mapping <fixed-mapping> --salvage-from <invalid.rows.jsonl>`. The salvage run re-processes only the listed row numbers against the same source, emits a `valid.ue.jsonl` containing only the newly fixed rows, and writes a new `invalid.rows.jsonl` with whatever still fails — so the loop repeats (feeding each round's `invalid.rows.jsonl` into the next `--salvage-from`) until no rows fail or the user stops.

package/skills/ae-data-integration/references/ue-routing.md

@@ -0,0 +1,35 @@
+# UE routing
+
+Use this reference after `data-integration inspect`.
+
+## Route to UE ingestion
+
+All of these must hold:
+
+- A real `#account_id` or `#distinct_id` source is present.
+- A real time column is parseable; no synthetic time is permitted.
+- The records are event facts or user-property snapshots rather than aggregates.
+- Field-level records can be represented without losing essential meaning.
+
+Classification order:
+
+1. A legal explicit `#type` wins (all eight record types — `track`, `user_set`, `user_setOnce`, `user_add`, `user_unset`, `user_del`, `user_append`, `user_uniq_append`).
+2. An event/action field implies `track`.
+3. Repeated users across a time series imply `track`; without an event field, propose a normalized file/Sheet name and require review.
+4. One row per user with snapshot-like fields implies `user_set`.
+5. Rows that mix track and user-profile facts in one file use `mixed` with a `record_type_field`; require explicit review.
+6. Low-confidence output is a proposal, never silent approval.
+
+Aggregated metrics, pivot tables, cross-tabs, model outputs, free-form documents, and records without real identity/time should normally use local analysis.
+
+## Route to local analysis
+
+Choose local analysis when:
+
+- The user wants insights, not project ingestion.
+- UE identity or time prerequisites are missing.
+- Each row is an aggregate rather than a user/event record.
+- Conversion would invent semantics or discard important structure.
+- The user declines an uncertain mapping or destination.
+
+Explain the reason briefly. Do not frame local analysis as an error.

package/skills/ae-data-integration-helper/SKILL.md

@@ -62,7 +62,7 @@ Trigger when a user asks about:
 - **Configuration**: Debug mode / data validation / retry strategy
 - **Troubleshooting**: Upload failures / missing data / format errors
 
-**Do NOT trigger**: Inline Q&A within `ae-generate-tracking-plan` or `ae-generate-tracking-code` sessions (those skills have their own workflows).
+**Do NOT trigger**: Inline Q&A within `ae-generate-tracking-plan` or `ae-generate-tracking-code` sessions (those skills have their own workflows). Offline local-file import (CSV / Excel / JSONL → AE) is owned by the `ae-data-integration` skill — route those requests there instead of answering with LogBus / RESTful guidance.
 
 ## Workflow
 

package/skills/ae-dataops/SKILL.md

@@ -173,7 +173,7 @@ Detailed metadata browsing, SQL query, async download, and cancel workflows live
 Key constraints:
 - IDE is query-only; create/modify/delete tables with `dataops_datatable`.
 - Prefer `dataops_datatable +dict_search_tables` for table discovery unless raw engine metadata or schema browsing is required.
-- SQL query submission creates a download task; rows are not returned inline.
+- Submit exactly one read-only SQL query. It creates a platform-bounded download task; rows are not returned inline and the result is not an unlimited or full export.
 
 ---
 

package/skills/ae-dataops/references/dataops-query.md

@@ -61,7 +61,7 @@ ae-cli dataops_ide +search_tables --spaceCode "${spaceCode}" --searchKey "user"
 
 ## Workflow C: Execute SQL Query (Download-Centered Async Flow)
 
-Use this flow for SQL query execution. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly.
+Use this flow for exactly one read-only SQL query. Result rows are not returned through MCP/CLI; submit creates a Gaia download-center task directly. The query semantics are preserved, but the result remains platform-bounded; when present, `downloadRowLimit` reports that cap. This is not an unlimited or full export.
 
 ```bash
 # Step 1: Submit SQL and create a download task. Defaults: repoCode=te_etl, engineType=TASK_ENGINE_TRINO.
@@ -72,7 +72,7 @@ ae-cli dataops_ide +submit_sql_query --spaceCode "${spaceCode}" --repoCode "te_e
 # Step 2: Poll the download task status by spaceCode/downloadTaskId. Rows are not returned through MCP/CLI.
 ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId}
 
-# Step 3: CLI-only local save after downloadStatus=SUCCESS. Use a .zip suffix.
+# Step 3: CLI-only streaming save after downloadStatus=SUCCESS. The target is replaced only after the stream completes.
 ae-cli dataops_ide +get_sql_query_status --spaceCode "${spaceCode}" --downloadTaskId ${downloadTaskId} --downloadTo "./result.zip"
 
 # (Optional) Cancel the download task.
@@ -104,8 +104,8 @@ ae-cli dataops_ide +cancel_sql_query --spaceCode "${spaceCode}" --downloadTaskId
 - **Table search**: `+search_tables` requires `--spaceCode` and `--searchKey`. `--connType`, `--repoCode`, and `--size` are optional and default to `SPACE`, `te_etl`, and `20`. It returns `items`, `searchKey`, `size`, `totalCount`, `tableCount`, `viewCount`, `returnedCount`, `hasMore`, and `nextAction`.
 - **Table detail**: `+ide_get_table_detail` requires `--spaceCode`, `--catalog`, `--schema`, and `--tableName`. `--connType`, `--repoCode`, `--engineType`, `--entityType`, and `--includeDdl` are optional and default to `SPACE`, `te_etl`, `TASK_ENGINE_TRINO`, auto-detect, and `false`. It returns identity, storage metadata, columns, partitions, partition keys, optional layout fields, and `tableDdl` only when requested.
 - **Schema info**: `+get_schema_info` requires `--spaceCode`, `--catalog`, and `--schema`. `--connType` and `--repoCode` are optional and default to `SPACE` and `te_etl`. It returns only `schema`, `tableNum`, and `viewNum`.
-- **SQL submit**: `+submit_sql_query` requires `--spaceCode` and `--sql`. `--repoCode` and `--engineType` are optional and default to `te_etl` and `TASK_ENGINE_TRINO`. On success it returns `requestId`, `spaceCode`, `repoCode`, `downloadTaskId`, `downloadStatus`, `downloadApi`, `downloadParams`, and `nextAction`; rows are never returned.
-- **SQL status**: `+get_sql_query_status` requires `--spaceCode` and `--downloadTaskId`. `--requestId` is optional trace-only. `--downloadTo` is CLI-only; after `downloadStatus=SUCCESS` it saves the result zip and adds `localFile`. It returns status/progress metadata, `nextAction`, `downloadApi`, and `downloadParams`; rows are never returned.
+- **SQL submit**: `+submit_sql_query` requires `--spaceCode` and exactly one read-only query in `--sql`. `--repoCode` and `--engineType` are optional and default to `te_etl` and `TASK_ENGINE_TRINO`. On success it returns task metadata and, when exposed by Gaia, `downloadRowLimit`; rows are never returned and the result remains platform-bounded.
+- **SQL status**: `+get_sql_query_status` requires `--spaceCode` and `--downloadTaskId`. `--requestId` is optional trace-only. `--downloadTo` is CLI-only; after `downloadStatus=SUCCESS` it streams the result zip to a temporary sibling file, publishes it only after completion, and adds `localFile`. It returns status/progress metadata, `nextAction`, `downloadApi`, and `downloadParams`; rows are never returned.
 - **SQL cancel**: `+cancel_sql_query` requires `--spaceCode` and `--downloadTaskId`. `--requestId` is optional trace-only. It returns cancellation request metadata such as `downloadCancelStatus`.
 - **engineType**: `TASK_ENGINE_TRINO` (default, interactive queries) | `TASK_ENGINE_STARROCKS` (high-concurrency analytics)
 - **entityType**: optional `TABLE` or `VIEW` hint. Omit it unless the target type must be forced.

package/skills/ae-generate-tracking-plan/SKILL.md

@@ -21,7 +21,8 @@ description: "Interactive generation of an AE tracking plan and upload. Trigger
 | 埋点模板 | Tracking Plan Template | Pre-built industry/genre xlsx templates |
 | 方案名称 | Plan Name | User-facing plan identifier |
 | 应用场景 | Application Scenario | One-sentence description of what the app does |
-| 素材来源 | Source Material Type | prd / chat / codebase / template |
+| 素材来源 | Source Material Type | prd / chat / codebase / template / data |
+| 数据样本 / 文件画像 | Data Sample / File Profile | Column→property mapping source from an `ae-data-integration` inspect profile (`source_type: data`) |
 | 业务维度 | Business Dimension | Revenue model, core loop, functional entries, currency system |
 | 收入模型 | Revenue Model | IAA / IAP / mixed / subscription / commission |
 | 核心循环 | Core Loop | Core gameplay loop (e.g. "grind stages → earn coins → gacha for heroes") |
@@ -115,8 +116,9 @@ Choose your source material (up to 2):
 3 - Codebase (local project path; hidden in sandbox) — Analyze source code to extract events and properties
 4 - Pre-built template (built-in industry and game genre templates) — Select a built-in template
 5 - Modify existing tracking plan (local AE format xlsx file) — Import an existing tracking plan xlsx as baseline for modification; can be combined with Product doc / Description / Codebase, but NOT with Pre-built template
+6 - Data sample / file profile (from ae-data-integration inspect) — Map data columns (CSV/Excel/JSONL) to events & properties from an inspect profile; single-source path, not combinable with other options
 
-Reply with number(s), e.g. 1,5 or 4. Select up to 2.
+Reply with number(s), e.g. 1,5 or 4. Select up to 2 (option 6 is single-source).
 ```
 
 If in a sandbox environment, ask exactly:
@@ -128,8 +130,9 @@ Choose your source material (up to 2):
 2 - Detailed description (conversational) — Describe app business flow, core features, user behaviors, monetization model, etc.
 3 - Pre-built template (built-in industry and game genre templates) — Select a built-in template
 4 - Modify existing tracking plan (sandbox workspace path) — Import an existing tracking plan xlsx as baseline for modification; can be combined with Product doc / Description, but NOT with Pre-built template
+5 - Data sample / file profile (from ae-data-integration inspect) — Map data columns (CSV/Excel/JSONL) to events & properties from an inspect profile; single-source path, not combinable with other options
 
-Reply with number(s), e.g. 1,4 or 3. Select up to 2.
+Reply with number(s), e.g. 1,4 or 3. Select up to 2 (option 5 is single-source).
 ```
 
 Do not rewrite this source material list as unnumbered bullets, cards, or prose. The user must be able to reply with the visible numbers.
@@ -153,8 +156,10 @@ Based on user selection, determine source material type and record to `meta.sour
 | Codebase only | `codebase` | Scan source code, extract events/properties from business logic |
 | Template only | `template` | Provide built-in template selection |
 | Existing plan only | `existing_plan` | Import xlsx as baseline (see "Modify Existing Tracking Plan Flow" below) |
+| Data sample only | `data` | Read the inspect profile (`ae-local-data-profile/v1`), map columns → events/properties (see "Data-path (source_type = data)" below) |
 | Any two-item combo | Join two types with `_` | First as baseline, second as supplement (priority: existing_plan → template → codebase → prd → chat) |
 | Existing plan + Pre-built template | **NOT allowed** | Both provide event baselines; semantic conflict |
+| Data sample + any other | **NOT allowed** | Data sample is a standalone single-source path |
 
 **Follow-up questions** (ask in follow-up order defined in Multi-Source Combination Rules below):
 - Product doc → if not in a sandbox environment, ask exactly:
@@ -184,6 +189,7 @@ Based on user selection, determine source material type and record to `meta.sour
 - Codebase → ask **"What is the project directory path?"**, then scan source to extract business logic
 - Pre-built template → display matching templates for user confirmation
 - Modify existing tracking plan → ask **"Please provide the xlsx file path of your existing tracking plan"**, then follow the flow below
+- Data sample → ask **"Please provide the path of the inspect profile JSON (or the run directory containing it)"**, then read the `ae-local-data-profile/v1` product and follow the "Data-path (source_type = data)" flow below
 
 **Modify Existing Tracking Plan Flow** (when user selects this option):
 
@@ -363,6 +369,49 @@ Use the baseline source's inference method as primary. Supplementary sources (es
 }
 ```
 
+### Data-path (source_type = data) — condensed single-gate flow
+
+When the user selects the **Data sample / file profile** option (`source_type = data`), the flow diverges from the standard 5-item anchor. This path **maps table columns → events/properties** instead of inventing events from business understanding, and uses a **single confirmation gate** instead of the 5-segment Refine loop.
+
+**What to skip** (only Item 1 — Application Scenario and Item 2 — Data sample are collected):
+
+- **Item 3 (SDK Integration Config)** → skip. `meta.sdk_integration_mode = "none"` (data ingested via RESTful / LogBus / DataX, no SDK). No SDK auto-track events are injected (Phase 1.4 already skips `none`).
+- **Item 4 (User Identity System)** → skip the visitor-ID strategy question. Derive `meta.user_identity` from the inspect profile's `identity_candidates` instead (e.g. a `distinct_id` / `account_id` column), `account_id_source: "user_account"` when an account column exists, otherwise `"none"`.
+- **Business Dimension confirmation** → skip. Mapping is driven by columns, not by revenue model / core loop. `meta.business_dimension` stays empty.
+
+**Draft construction (replaces Phase 1.1/1.2/1.3 for the data path)**:
+
+1. **Read the inspect profile**: the `ae-local-data-profile/v1` JSON product (columns / types / samples / UE eligibility / mapping confidence). If it is not present or is stale, re-run `ae-cli data-integration inspect` for the source file first.
+2. **Event model determination** (reuse UE routing): single-table single-event `track` / single-table multi-event (event-name column) / single-table `user_set` / mixed. The agent may propose **splitting one table into multiple events** (e.g. an ad table split by `campaign_type` into `ad_show` / `ad_click`); such proposals MUST be confirmed in the gate.
+3. **Column → property mapping draft**:
+   - Identify system columns first: time field, `distinct_id` / `account_id`, event-name column, user-property-name column.
+   - Map the remaining columns to event properties / user properties / super properties.
+   - Naming: `snake_case` event/property names + `display_name` + `desc` + `event_tag` (language follows the user's input).
+   - Type inference: CSV columns default to `string`; infer `number` / `bool` / `datetime` / enum from field name + value distribution + business doc/prompt priors. **Uncertain or conflicting columns are marked "to-confirm" and asked only inside the gate** (do not ask column-by-column beforehand).
+4. **Single confirmation gate** (replaces Phase 2, see below).
+5. **Merge with existing plan** (reuse Phase 4.1/4.2 conflict detection).
+6. **Persist**: `.ae-cli/draft.json` → xlsx → upload (`sdk_integration_mode = none`).
+
+**Single confirmation gate (one round, one summary table)**:
+
+Present ONE merged table covering all of the following in a single message, then wait for a single reply:
+
+- **Event list**: `event_name` / `display_name` / `desc` / `event_tag` / `platform`.
+- **Property list**: `name` / `display_name` / `type` / `desc` / `source`, with uncertain types highlighted (marked "to-confirm").
+- **Field scope**: default to plan-internal fields only; include a **full-import switch** for bringing all source columns in.
+- **Unrecognized / dirty data handling**: how unmapped columns, null values, and unparseable rows are treated.
+- **Same-name property type conflicts**: flagged inline in the gate (see Phase 4.2 Type A).
+
+User replies once: `ok` (accept all), or targeted edits — rename / retype / add / remove individual columns or events. After the gate is confirmed, jump to Phase 3 (project token) then Phase 4 (merge + upload); do NOT enter the 5-segment Refine loop.
+
+**dry-run mode**:
+
+When the user requests dry-run (or the caller passes `data-integration plan --dry-run`), produce the draft preview + column→property mapping summary ONLY:
+
+- Show the single confirmation gate table (events + properties + field scope + unrecognized-data handling) and the mapping result, but do NOT write `.ae-cli/draft.json`.
+- Do NOT generate xlsx, do NOT archive to `plans/`, do NOT upload to AE.
+- State explicitly that nothing was persisted; the user can approve a real run afterwards.
+
 ### Item 3 — SDK Integration Config (client + server combined)
 
 Ask: **"What is your client platform? (multi-select OK, e.g. Android + iOS) Will you integrate a server-side SDK?"**
@@ -822,6 +871,8 @@ AE_LANG=<user_lang> ae-cli tracking code import-template --template-name "<templ
 
 ## Phase 2 — Refine (5-segment loop)
 
+> **Data path (`source_type = data`)**: skip this 5-segment loop entirely — use the single confirmation gate defined in "Data-path (source_type = data)" instead.
+
 In order, one conversation round per segment:
 
 1. **sdk_config (SDK config + User identity, combined)** — Show SDK integration mode, platform/language, visitor ID strategy, account ID source, corresponding SDK calls

package/skills/ae-kb/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: ae-kb
 version: 1.0.0
-description: "AE/TE knowledge base CLI manual for creating, querying, LLM-powered ask, listing accessible knowledge bases, deterministic index/grep/read retrieval, checking status, uploading, compiling, schema generation, URL sources, source deletion, and knowledge base deletion. Use when the user asks to manage TE/AE/ThinkingEngine knowledge bases, upload documents or URLs to a knowledge base, query knowledge, ask knowledge bases with an LLM, list accessible knowledge bases, inspect knowledge base indexes, search knowledge base pages, read a specific knowledge base page, check knowledge base status, generate schemas, compile knowledge, remove sources, or delete a knowledge base. Must use ae-cli kb commands and must not guess knowledge base names, scopes, page paths, source display names, JSON payload shapes, or URL formats."
+description: "AE/TE knowledge base CLI manual for creating, querying, LLM-powered ask, listing accessible knowledge bases, deterministic index/grep/read retrieval, checking status, uploading, compiling, schema generation, URL sources, source deletion, and knowledge base deletion. Use when the user asks to manage TE/AE/ThinkingEngine knowledge bases, upload documents or URLs to a knowledge base, query knowledge, ask knowledge bases with an LLM, list accessible knowledge bases, inspect knowledge base indexes, search knowledge base pages, read a specific knowledge base page, check knowledge base status, generate schemas, compile knowledge, remove sources, or delete a knowledge base. To choose which knowledge base is worth searching, use the ae-kb-discovery skill first; this skill runs the retrieval once a target is chosen. Must use ae-cli kb commands and must not guess knowledge base names, scopes, page paths, source display names, JSON payload shapes, or URL formats."
 ---
 
 # ae-kb
@@ -15,25 +15,27 @@ ae-cli kb +<command> [options]
 ## Global Rules
 
 - Use this skill for TE/AE knowledge base tasks: create, query, ask with LLM, list accessible knowledge bases, inspect indexes, grep pages, read pages, check status, upload sources, add URL sources, generate schema, compile, remove source files, and delete knowledge bases.
+- **Searching a knowledge base for an answer is the most common task. If that is what you are doing, go straight to [Explore Knowledge Base Pages](#explore-knowledge-base-pages) and read [`references/query-workflow.md`](references/query-workflow.md) first — it is the retrieval procedure. The other commands below are for managing knowledge bases, not answering from them.**
 - Read operations can run directly after required inputs are known. Write operations require explicit user intent and normally keep the confirmation prompt unless the user asks to bypass it.
 - Prefer `--dry-run` before destructive or broad writes when the user has not already validated the target.
 - Do not invent knowledge base names, scopes, source display names, or JSON payloads. Ask the user or query known context when values are missing.
+- When building a `--sources` ref (or `+read --source`), copy the exact `scope` and `name` from `+list` output — run `ae-cli kb +list` first when the scope of a named knowledge base is unknown.
 - JSON flags must be valid JSON strings, usually wrapped in single quotes in shell commands.
 - Successful commands return JSON by default. Use `--format table` only when a table is easier for a human to scan. Envelope may include optional `_notice.host_compat`.
 - `--host <url>` overrides the active AE host. It is available on every command and may be placed after the subcommand, e.g. `ae-cli kb +<command> --host <url>`.
 - **CRITICAL — Host compat (do this first):** After each `ae-cli` run, check stderr and `_notice.host_compat`. If either is present, open the user reply with a short ⚠️ version warning and **quote the `npm i -g` / `npx skills add` (or update-cluster) lines verbatim**, then present the business result. Soft tip; `ok: true` can still carry the notice.
-- For external-agent retrieval, prefer the deterministic flow `+index` -> `+grep` -> `+read`: inspect navigation, locate candidate pages, then open the exact page or line window. Use `+ask` only when the user wants an LLM-synthesized answer and accepts token consumption.
+- Retrieval (`+index` / `+grep` / `+read`) is deterministic and server-side LLM-free; use it for simple factual lookups. Use `+ask` when the question requires synthesizing across multiple pages or multi-hop reasoning.
 
 ## Commands
 
 | Command | Risk | Purpose |
 |---|---:|---|
-| `+query` | read | Query one or more knowledge bases with a natural-language question. |
-| `+ask` | read | LLM-powered Q&A over knowledge bases. Consumes platform tokens. |
+| `+ask` | read | LLM-powered Q&A over knowledge bases; for multi-page synthesis or multi-hop questions. |
+| `+ask-status` | read | Query the current status of an ask execution by `--execution-id` without polling. |
 | `+list` | read | List accessible knowledge bases filtered by buildStatus (default: compiled). |
 | `+index` | read | List accessible knowledge bases and their `index.md` navigation maps. |
 | `+grep` | read | Keyword-search knowledge base pages and return matched lines with context. |
-| `+read` | read | Read a full knowledge base page or a line window. |
+| `+read` | read | Read a full knowledge base page, a line window, or (with `--outline`) only the page heading tree. |
 | `+new` | write | Create a new personal or company knowledge base. |
 | `+add` | write | Upload local files, a non-recursive directory, or HTTP(S) pages converted to markdown. |
 | `+url` | write | Upload a URL source directly with optional display name and parsing instruction. |
@@ -123,34 +125,26 @@ Use `+status` to inspect the current status of a knowledge base.
 ae-cli kb +status --name engineering-handbook
 ```
 
-### Query Knowledge
-
-Use `+query` with a natural-language question. Optionally scope to specific knowledge bases or tune result count and locale.
-
-```bash
-ae-cli kb +query \
-  --query "How do we release a dashboard?" \
-  --sources '[{"scope":"company","name":"engineering-handbook"}]' \
-  --top-k 10 \
-  --locale zh
-```
-
-When `--sources` is provided, each entry requires:
-
-- `scope`: knowledge base scope such as `personal` or `company`.
-- `name`: knowledge base name.
-
 ### Ask Knowledge (LLM)
 
-Use `+ask` when the user wants an LLM-synthesized answer. This endpoint calls a large language model and **consumes platform tokens**. Prefer `+index` -> `+grep` -> `+read` when deterministic retrieval is enough.
+Use `+ask` when the question requires synthesizing across multiple pages or multi-hop reasoning — a server-side agent runs the full retrieval loop and returns a synthesized answer with its source paths. Prefer `+index` -> `+grep` -> `+read` when deterministic retrieval is enough.
+
+The `+ask` command uses asynchronous submit/poll: by default, it automatically polls for completion (every 5s, up to 10 minutes) and prints the final answer. The output JSON is isomorphic to the previous synchronous response, so consumers require no changes.
 
 ```bash
+# Default: submit and poll for completion
 ae-cli kb +ask \
   --question "How do we troubleshoot payment alerts?" \
   --sources '[{"scope":"company","name":"engineering-handbook"}]' \
   --model-id claude-sonnet-4-6 \
   --max-turns 50 \
   --locale zh
+
+# Submit only, return executionId immediately (for batch processing)
+ae-cli kb +ask --question "..." --no-wait
+
+# Query execution status later
+ae-cli kb +ask-status --execution-id <id>
 ```
 
 - `--question`, alias `-q`: required natural-language question (1-2000 characters).
@@ -158,10 +152,12 @@ ae-cli kb +ask \
 - `--model-id`: optional LLM model ID. Omit to use the platform default.
 - `--max-turns`: optional agent turn limit (1-100, server default 50).
 - `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
+- `--no-wait`: optional boolean flag. Return immediately after submission with `{executionId, status}`, without polling.
+- **Failure handling**: If execution fails, the command exits non-zero and prints an error message on stderr prefixed with the typed error code, e.g. `[timeout] ...` / `[model_error] ...` / `[invalid_sources] ...` (followed by the executionId). Treat the bracketed code as the machine-readable failure type.
 
 ### List Accessible Knowledge Bases
 
-Use `+list` when you only need accessible knowledge base metadata without loading `index.md` navigation maps. By default it returns knowledge bases with `buildStatus: compiled`:
+Use `+list` when you only need accessible knowledge base metadata without loading `index.md` navigation maps. Omit `--build-status` to default to `compiled`; pass `idle` / `pending` / `compiling` / `compiled` / `failed` to filter by a specific status (system knowledge bases are always listed regardless of status):
 
 ```bash
 ae-cli kb +list
@@ -173,6 +169,8 @@ ae-cli kb +list --build-status compiled
 
 Use the deterministic retrieval primitives when an agent needs to explore knowledge base content like a code repository. These endpoints do not call an LLM on the server side.
 
+**Before running a real query, read [`references/query-workflow.md`](references/query-workflow.md)** — it is the step-by-step procedure for turning a question into an answer without crawling. It covers candidate indexing, copied-path grep, same-page read windows, linked-page re-grep, outline-derived ranges, and coverage assessment. This section below is the per-command reference the workflow draws on.
+
 Start with `+list` or `+index` to discover accessible knowledge bases. Use `+index` when you also need navigation maps:
 
 ```bash
@@ -190,26 +188,30 @@ Then use `+grep` to locate likely pages and line numbers:
 ae-cli kb +grep \
   --query "sandbox configuration" \
   --sources '[{"scope":"company","name":"engineering-handbook"}]' \
+  --paths '["wiki/sandbox.md"]' \
   --top-k 10
 ```
 
-Finally use `+read` to open the exact page, optionally with a line window:
+Each grep hit carries `path`, `line`, `breadcrumb`, a context snippet, and the section range of the matched line (`sectionStartLine` / `sectionEndLine`). `line` is the hit anchor; `sectionStartLine` / `sectionEndLine` are the enclosing heading-section boundaries. Choose the smallest reliable `--offset` / `--limit` window that preserves the needed evidence; use the section range when the answer needs whole-section context.
+
+Use `+read --outline` when the current target page has no reliable grep range and headings are needed to choose a section:
 
 ```bash
 ae-cli kb +read \
   --source '{"scope":"company","name":"engineering-handbook"}' \
   --path "wiki/sandbox.md" \
-  --offset 1 \
-  --limit 200
+  --outline
 ```
 
-Retrieval rules:
+Then use `+read` to open the selected window, using the hit anchor, a section boundary from same-page or linked-page grep, or two adjacent outline headings:
 
-- `+list` accepts optional `--build-status` (default `compiled`) and `--locale`. Returns accessible knowledge base metadata including `buildStatus`, without `index.md` navigation maps.
-- `+index` accepts optional `--sources` and `--locale`; omit `--sources` to list all accessible knowledge bases.
-- `+grep` requires `--query` / `-q`; optional `--sources`, `--top-k` (1-50, default 10), and `--locale`.
-- `+read` requires `--source` pointing to exactly one knowledge base and `--path` relative to the knowledge base root; optional `--offset`, `--limit` (1-10000), and `--locale`.
-- Do not guess a `--path`; get it from `+index` or `+grep` results.
+```bash
+ae-cli kb +read \
+  --source '{"scope":"company","name":"engineering-handbook"}' \
+  --path "wiki/sandbox.md" \
+  --offset 42 \
+  --limit 60
+```
 
 ### Remove One Source
 
@@ -233,29 +235,29 @@ ae-cli kb +remove --name engineering-handbook
 
 ## Command Reference
 
-### `+query`
+### `+ask`
 
 ```bash
-ae-cli kb +query --query "<question>" [--sources '[{"scope":"company","name":"kb-name"}]'] [--top-k 10] [--locale zh|en|ja|ko]
+ae-cli kb +ask --question "<question>" [--sources '[{"scope":"company","name":"kb-name"}]'] [--model-id claude-sonnet-4-6] [--max-turns 50] [--locale zh|en|ja|ko] [--no-wait]
 ```
 
-- `--query`, alias `-q`: required natural-language question.
+- `--question`, alias `-q`: required natural-language question (1-2000 characters).
 - `--sources`: optional JSON array of knowledge base refs. Omit to search all accessible knowledge bases.
-- `--top-k`: optional max number of hits, 1-50, default 10.
+- `--model-id`: optional LLM model ID. Omit to use the platform default.
+- `--max-turns`: optional agent turn limit (1-100, server default 50).
 - `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
+- `--no-wait`: optional. Return immediately with `{executionId, status}` instead of polling.
+- When to use: multi-page synthesis or multi-hop questions. For simple factual lookups, prefer `+index` / `+grep` / `+read`.
+- Output: By default, polls and returns `{executionId, answer, sources, modelUsage, toolCallCount, maxTurns, modelId}` (same fields as the previous synchronous response, plus `executionId`). With `--no-wait`, returns `{executionId, status}` immediately. On failure, exits non-zero with a stderr message prefixed by the typed error code (`[timeout]`, `[model_error]`, `[invalid_sources]`, `[process_restart]`).
 
-### `+ask`
+### `+ask-status`
 
 ```bash
-ae-cli kb +ask --question "<question>" [--sources '[{"scope":"company","name":"kb-name"}]'] [--model-id claude-sonnet-4-6] [--max-turns 50] [--locale zh|en|ja|ko]
+ae-cli kb +ask-status --execution-id <id>
 ```
 
-- `--question`, alias `-q`: required natural-language question (1-2000 characters).
-- `--sources`: optional JSON array of knowledge base refs. Omit to search all accessible knowledge bases.
-- `--model-id`: optional LLM model ID. Omit to use the platform default.
-- `--max-turns`: optional agent turn limit (1-100, server default 50).
-- `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
-- **Token cost**: this command invokes an LLM on the server and consumes platform tokens. Prefer `+index` -> `+grep` -> `+read` for token-free deterministic retrieval.
+- `--execution-id`: required. The execution ID returned by `+ask` submission.
+- Output: Returns the current execution state: `{executionId, status, elapsedMs?, answer?, sources?, modelUsage?, toolCallCount?, error?}`. Does not poll; returns a single snapshot.
 
 ### `+list`
 
@@ -263,7 +265,7 @@ ae-cli kb +ask --question "<question>" [--sources '[{"scope":"company","name":"k
 ae-cli kb +list [--build-status compiled] [--locale zh|en|ja|ko]
 ```
 
-- `--build-status`: optional build status filter (default: `compiled`).
+- `--build-status`: optional; one of `idle` / `pending` / `compiling` / `compiled` / `failed`. Omit to default to `compiled` (system knowledge bases are always listed regardless of status).
 - `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
 - Response items include `buildStatus`.
 
@@ -279,24 +281,27 @@ ae-cli kb +index [--sources '[{"scope":"company","name":"kb-name"}]'] [--locale
 ### `+grep`
 
 ```bash
-ae-cli kb +grep --query "<keywords>" [--sources '[{"scope":"company","name":"kb-name"}]'] [--top-k 10] [--locale zh|en|ja|ko]
+ae-cli kb +grep --query "<keywords>" --sources '[{"scope":"company","name":"kb-name"}]' --paths '["wiki/page.md"]' [--top-k 10] [--locale zh|en|ja|ko]
 ```
 
-- `--query`, alias `-q`: required keywords to search across knowledge bases.
-- `--sources`: optional JSON array of knowledge base refs. Omit to search all accessible knowledge bases.
+- `--query`, alias `-q`: required keywords to search.
+- `--sources`: required JSON array of knowledge base refs.
+- `--paths`: required JSON array of wiki pages or subdirectories **copied** from `+index`. A single page is still an array, e.g. `["wiki/sandbox.md"]`. Distinct from `+read --path` (one string).
 - `--top-k`: optional max number of hits, 1-50, default 10.
 - `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
+- Each hit includes `sectionStartLine` / `sectionEndLine`: the line range of the section (bounded by the nearest headings) containing the matched line. Use it as the `+read` window.
 
 ### `+read`
 
 ```bash
-ae-cli kb +read --source '{"scope":"company","name":"kb-name"}' --path "index.md" [--offset 1] [--limit 200] [--locale zh|en|ja|ko]
+ae-cli kb +read --source '{"scope":"company","name":"kb-name"}' --path "index.md" [--offset 1] [--limit 200] [--outline] [--locale zh|en|ja|ko]
 ```
 
 - `--source`: required JSON object pointing to exactly one knowledge base.
 - `--path`: required page path relative to the knowledge base root, such as `index.md` or `wiki/concepts/data-model.md`.
 - `--offset`: optional 1-based start line.
 - `--limit`: optional max line count, 1-10000.
+- `--outline`: optional. Return only the whole-page heading tree (`{level, heading, line}`) with empty content, independent of `--offset` / `--limit`. Use it on long pages to choose which section to read.
 - `--locale`: optional locale: `zh`, `en`, `ja`, or `ko`.
 
 ### `+new`