@thinkingai/ae-cli 6.1.19 → 6.1.21

package/skills/ae-data-integration/references/lark-bitable-source.md

@@ -0,0 +1,288 @@
+# Source — online document (Feishu Bitable)
+
+## When this branch applies
+
+The input is a **URL**, not a local path. Feishu Bitable (多维表格 / Base) links (`/base/`), and
+wiki links that resolve to a Bitable, route here instead of [source-inspect.md](source-inspect.md).
+Feishu sheet links still route to [lark-sheet-source.md](lark-sheet-source.md); a `/wiki/` link can be
+either, so resolve it first (Step 1) and branch on the resolved type. The pipeline is identical
+after the table is snapshotted to a local JSONL: business identification, tracking plan,
+transform, and sink all reuse the local-file flow unchanged.
+
+Only **Feishu sheets** and **Feishu Bitable (多维表格)** are supported in this phase. Other online
+documents (Tencent Docs, DingTalk, WPS, Google Sheets, Notion) have no CLI read path — route to
+[manual-export-source.md](manual-export-source.md), which has the user export the data themselves
+and then rejoins the local-file flow.
+
+## Safety rules (online-specific)
+
+- The URL, its token, the snapshot file path, and every record value are sensitive. Never print
+  source values while inspecting; summarize types, ratios, counts, and fingerprints only.
+- Pulling records to the local machine moves data across a trust boundary. Before downloading
+  anything, tell the user the table will be snapshotted to a local file, and get their
+  confirmation when the data is not their own.
+- The snapshot is a point-in-time copy. State the export timestamp (and the base timezone the
+  API reports), and note that the import reflects the exported moment only — a table that may be
+  edited mid-run is imported as of the snapshot, not live.
+- lark-cli (Feishu) and ae-cli (AE) are two separate credential domains. Read Feishu with
+  lark-cli; upload to AE with ae-cli. Never forward a token from one to the other, and never
+  send an AE access token or CLI token to any Feishu endpoint.
+
+## Step 0 — Environment self-check (before any pull)
+
+The Feishu read path depends on `lark-cli`, a CLI separate from ae-cli. Check it up front so a
+missing install or a missing login surfaces as an environment problem, not as a data problem
+mid-pipeline.
+
+1. **Detect lark-cli.** Run `lark-cli --version`. If the command is not found, stop and tell the
+   user to install it (`npm i -g @larksuite/cli`). Do not attempt to install it yourself, and do
+   not fall back to another read path — a half-provisioned Feishu client produces worse failures
+   than a clear "not installed" stop.
+2. **Check the login.** Run
+   `LARKSUITE_CLI_NO_UPDATE_NOTIFIER=1 LARKSUITE_CLI_NO_SKILLS_NOTIFIER=1 lark-cli auth status --json --verify`.
+   Reading a Bitable in the user's cloud space needs the **user** identity (`--as user`); the bot
+   identity cannot see a user's drive documents. If the user identity is unavailable or
+   unverified, follow the lark-shared skill's split-flow: start
+   `lark-cli auth login --scope "<scope>" --no-wait --json`, show the QR code / link to the user,
+   and only after the user confirms run `lark-cli auth login --device-code <device_code>`.
+   Authorization requires the user's participation — never bypass or automate it. The Bitable
+   read path needs the scopes `base:table:read`, `base:record:read`, and `base:field:read`.
+3. **Handle permission errors when they occur.** If a pull returns `permission_violations`
+   (missing scopes) together with `console_url` and `hint`, route by identity:
+   - user identity → add the missing scope with
+     `lark-cli auth login --scope "<missing_scope>"` (incremental grant);
+   - bot identity → direct the user to open the scope in the developer console via the returned
+     `console_url`; never run `auth login` for a bot.
+   If the base itself is not accessible to the user's account — a resource-side permission
+   problem, not a scope problem — ask the user to confirm the link is correct and that their
+   Feishu account can open the base. Do not retry against a different base silently.
+
+## Step 1 — Resolve the URL and locate the table
+
+1. **Resolve the URL to a base token.** Primary (richer — returns the title and unwraps wiki
+   links):
+   ```bash
+   lark-cli drive +inspect --url "<url>" --as user
+   ```
+   returns `{ok, identity: "user", data: {token, title, type, url}}`. A Bitable resolves with
+   `type: "bitable"` and its base token in `token`. If `type` is not `bitable`, leave this branch
+   — a sheet routes to [lark-sheet-source.md](lark-sheet-source.md). Domain-native alternative, which also
+   accepts wiki and record-share links:
+   ```bash
+   lark-cli base +url-resolve --url "<url>" --as user
+   ```
+   returns `{ok, identity: "user", data: {base_token, resource_type: "bitable", input_type}}`.
+2. **List the tables.**
+   ```bash
+   lark-cli base +table-list --base-token <base_token>
+   ```
+   returns `{ok, data: {tables: [{id, name, records_count, rev}], total}}`. Never guess a table
+   id or name. Surface the table list to the user with each table's record count; a base with one
+   table proceeds with it, multiple tables ask the user to choose, mirroring `--data-set` on a
+   local workbook.
+
+## Step 2 — Snapshot to a local JSONL
+
+The base domain has **no workbook-export command** (only per-record reads and an attachment
+downloader), so the snapshot is a **JSONL** built by paginating `+record-list` and zipping each
+page into per-record objects — not an XLSX. The snapshot directory convention matches the sheet
+flow: `.ae-cli/data-integration/online/<fingerprint>/snapshot.jsonl`, where `<fingerprint>` is a
+short, collision-resistant name derived from the base token plus a timestamp (e.g. first 8 chars
+of `sha256("<base_token>-<YYYYMMDD-HHMMSS>")`). One online source gets one directory, so a
+re-export never overwrites an unrelated source. Create the directory before writing.
+
+1. **Survey the fields** (type knowledge for the mapping step, and attachment detection):
+   ```bash
+   lark-cli base +field-list --base-token <base_token> --table-id <table_id>
+   ```
+   returns
+   `{ok, data: {fields: [{id, name, type, style?, options?, multiple?, default_value?}], total}}`.
+   `style` and `options` are display metadata (currency code, date format, choice list);
+   `multiple` distinguishes single- from multi-select. Field-list order is NOT the record-list
+   order — do not carry it across to the zip step.
+2. **Paginate the records.** Always pass `--format json` explicitly — `+record-list` defaults to
+   **markdown**, unlike `+table-list`/`+field-list` which already default to json:
+   ```bash
+   lark-cli base +record-list --base-token <base_token> --table-id <table_id> --format json
+   ```
+   The JSON envelope is **column-oriented**, not record-oriented:
+   ```json
+   {
+     "ok": true, "identity": "user",
+     "data": {
+       "data": [["<…>", "<…>", true, 129.5, ["<…>"], ["<…>"], "<ISO8601>", ["<…>"]],
+       "fields": ["<…>", "<…>", "<…>", "<…>", "<…>", "<…>", "<…>", "<…>"],
+       "field_id_list": ["<…>", "…"],
+       "field_type_list": ["<…>", "…"],
+       "record_id_list": ["<…>", "…"],
+       "has_more": true,
+       "query_context": { "field_scope": "all_fields", "record_scope": "all_records" },
+       "timezone": "Asia/Shanghai",
+       "rev": 1,
+       "total": null
+     }
+   }
+   ```
+   (`total` may be absent rather than null — see the pagination contract below.)
+   `data.data` rows are positional arrays aligned with the parallel `fields`, `field_id_list`,
+   `field_type_list`, and `record_id_list` arrays. Pagination contract (verified):
+   - `--offset` is a 0-based row skip; start at 0.
+   - `--limit` is 1–200 (default 100).
+   - After each page, advance `offset` by **the number of rows returned**, not by `limit` — a
+     short final page must not skip rows.
+   - Loop until `has_more: false`. `total` is null or absent in this envelope — never use it to
+     detect the end.
+   - Each page's `fields` order is authoritative for **that page**. Field-list order ≠
+     record-list order, and `--field-id` projection re-aligns the parallel arrays — always zip
+     with the `fields` array from the same response.
+3. **Zip each page into per-record objects.** The flat snapshot reserves `__record_id` for the
+   stable ID returned by the Bitable API. Record `i` becomes
+   `{ __record_id: record_id_list[i], fields[j]: data[i][j] for each j }`. A business field named
+   `record_id` remains an ordinary source column. If the table itself contains a business field
+   named `__record_id`, stop before writing any snapshot instead of overwriting either value or
+   silently renaming the user's field. Reference implementation (paginates, zips, writes
+   `snapshot.jsonl`, prints a summary only — no record values):
+   ```javascript
+   import { execFileSync } from "node:child_process";
+   import { mkdirSync, writeFileSync } from "node:fs";
+   import path from "node:path";
+
+   const [baseToken, tableId, outDir] = process.argv.slice(2);
+   mkdirSync(outDir, { recursive: true });
+
+   function fetchPage(offset) {
+     const stdout = execFileSync(
+       "lark-cli",
+       ["base", "+record-list", "--base-token", baseToken, "--table-id", tableId,
+        "--offset", String(offset), "--limit", "200", "--format", "json"],
+       { encoding: "utf8", timeout: 120_000,
+         env: { ...process.env,
+                LARKSUITE_CLI_NO_UPDATE_NOTIFIER: "1",
+                LARKSUITE_CLI_NO_SKILLS_NOTIFIER: "1" } }
+     );
+     return JSON.parse(stdout);
+   }
+
+   const RECORD_ID_KEY = "__record_id";
+   let offset = 0; const records = []; let meta = null;
+   while (true) {
+     const envelope = fetchPage(offset);
+     if (!envelope.ok) { console.error("record-list failed:", JSON.stringify(envelope.error ?? envelope)); process.exit(1); }
+     const d = envelope.data;
+     const rows = d.data ?? [], ids = d.record_id_list ?? [], fields = d.fields ?? [];
+     if (rows.length !== ids.length) { console.error(`row/id length mismatch: rows=${rows.length} ids=${ids.length}`); process.exit(1); }
+     if (fields.includes(RECORD_ID_KEY)) { console.error(`reserved field name collision: ${RECORD_ID_KEY}`); process.exit(1); }
+     for (let i = 0; i < rows.length; i++) {
+       const rec = { [RECORD_ID_KEY]: ids[i] };
+       for (let j = 0; j < fields.length; j++) rec[fields[j]] = rows[i][j];
+       records.push(rec);
+     }
+     meta = { timezone: d.timezone, rev: d.rev, total: d.total, fieldNames: fields };
+     if (!d.has_more) break;
+     offset += rows.length;
+   }
+
+   const outPath = path.join(outDir, "snapshot.jsonl");
+   writeFileSync(outPath, records.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8");
+   console.log(JSON.stringify({ snapshotPath: outPath, records: records.length,
+     fieldCount: meta.fieldNames.length, fieldNames: meta.fieldNames,
+     timezone: meta.timezone, rev: meta.rev, totalReported: meta.total }));
+   ```
+   Optional scoping flags (all verified; default is all records, all fields): `--field-id
+   <name|id>` (repeatable) projects to specific fields; `--view-id` reads a specific view;
+   `--filter-json` overrides the view's own filter, e.g.
+   `{"logic":"and","conditions":[["Title","==","Launch plan"]]}` (equality),
+   `[["Title","intersects","urgent"]]` (contains), `[["Score","==",95]]` (numbers unquoted);
+   `--sort-json` (max 10 rules) overrides the view's sort, e.g.
+   `[{"field":"Updated","desc":true}]`.
+
+   Field-type mapping and value shapes in the snapshot:
+
+   | Bitable field type | `--format json` value shape | Downstream `inspect` inference |
+   | --- | --- | --- |
+   | text, `style.type: plain/phone/email/barcode` | string | string |
+   | text, `style.type: url` | **markdown link** `[display](url)`, not the raw URL | string — the markdown wrapper is part of the value |
+   | number (plain/currency/progress/rating; `style` is display-only) | plain number | number |
+   | checkbox | boolean | boolean |
+   | select, `multiple: false` (single-select) | **single-element array** (e.g. `["paid"]`) | **`list`**, not string |
+   | select, `multiple: true` (multi-select) | array of strings; **empty selection is `null`, not `[]`** | `list` (empty rows counted as missing) |
+   | datetime (any `style.format`) | full ISO8601 string with offset; envelope `timezone` states the base timezone | datetime |
+   | created_at / updated_at | full ISO8601 string | datetime |
+   | auto_number | string (e.g. `NO.001`) | string |
+   | created_by / updated_by | array of `{id, name}` user objects (single-element for one creator) | `list` |
+   | location | object `{lng, lat, full_address}` | json/object |
+   | link | array of `{id}` record refs (single-element for one link) | `list` |
+   | attachment | `null` when empty (verified); non-empty values are out of scope | all-null column |
+   | formula | **always a string, even for numeric results** (e.g. `"20"`) | string |
+   | lookup | data-dependent: scalar string for a single `raw_value` match; list for multiple matches or a list aggregate | string or list |
+
+   Warnings the mapping step must carry:
+
+   - single-select columns infer as `list` (not `string`) because Bitable returns single-element
+     arrays — surface this when the user confirms the mapping.
+   - `created_by` / `updated_by` are **single-value** fields the API wraps in a one-element array
+     of `{id, name}` user objects — surface this when the user confirms the mapping, and do not
+     treat them as multi-value. The `id` is a Feishu open_id/user_id, not an AE identity; these
+     are audit metadata. Default to excluding them, or import only `name` as a plain `string`
+     property (after confirming with the user); do not map them to a `list` or an `{id, name}`
+     `object` unless the user explicitly asks. A custom **person** field is different:
+     single-person fields wrap the same way, multi-person fields are genuinely multi-value —
+     confirm which before mapping.
+   - `link` (single-link) columns infer as `list` because the API wraps a single link in a
+     one-element array of `{id}` record refs — surface this when the user confirms the mapping,
+     and do not treat them as multi-value. The `{id}` is the linked record's internal
+     `record_id`, not a business value: exclude the column, or flatten the linked record's real
+     fields via a `lookup`/`formula` on the source table instead of importing record_ids.
+     A multi-link field returns multiple `{id}` refs per row; `+field-list` exposes no
+     `multiple` flag for link fields, so distinguish single from multi by the value array
+     length across rows, not by a field-list property.
+   - `location` columns are objects `{lng, lat, full_address}` — mapping the whole column locks
+     an `object` property type. Usually only `full_address` (a plain string) is meaningful;
+     confirm whether to flatten it to that single string (or exclude) rather than default to
+     `object`.
+   - an empty multi-select is `null` (verified), so `inspect` counts those rows as missing — a
+     null multi-select is "no selection", not data loss.
+   - `url`-styled text fields come back as markdown links (`[display](url)`), not raw URLs —
+     strip the wrapper (or re-key from a plain text field) before mapping to a URL property.
+   - formula fields are always strings in the snapshot, even when the expression is numeric
+     (e.g. `"20"`) — inspect infers `string`; do not expect a number column.
+   - lookup value shape depends on the aggregate, the target field type, and the match count —
+     a lookup to a person field returns `{id, name}` objects, to a select field returns option
+     strings (possibly a list), to a scalar text/number field returns a scalar string or list —
+     so confirm the target field type per-table before mapping.
+   - attachment fields are kept as an all-null column when empty; downloading attachments
+     (`base +record-download-attachment`) is out of scope for ingestion.
+4. If pagination fails on a network error or times out mid-loop, treat the snapshot as unknown:
+   do not write or keep a partial file; stop, and ask the user to confirm the base state before
+   retrying — the same rule as a lost upload batch.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the snapshot as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '.ae-cli/data-integration/online/<fingerprint>/snapshot.jsonl'
+```
+
+The zip step prefixes each record with reserved `__record_id` — a stable per-record key from the
+Bitable API, useful as a dedup key downstream. A source `record_id` column remains intact; a source
+`__record_id` column blocks snapshot creation as a reserved-name collision. Expect the Step 2
+inference notes to show up here: single-select columns surface as `list`, empty multi-selects as
+missing rows. Everything downstream — business identification, tracking plan, transform, sink —
+is unchanged. The snapshot lives under
+`.ae-cli/data-integration/`, so [reuse](reuse.md) and [handoff](handoff.md) find it like any other
+pipeline artifact; record in the handoff that the source was an online Feishu Bitable, the base
+and table, and include the snapshot timestamp and base timezone.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Online-source problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| `lark-cli` not installed | Not a parse error, not a data problem | Stop; give the install command |
+| lark-cli not logged in / no user identity | Not a parse error | Split-flow authorization with the user |
+| Missing scope (`permission_violations`) | Not a parse error | Incremental `auth login --scope` (user) or developer console (bot) |
+| Base/table not accessible to the user's account | Not a scope problem | Ask the user to verify the link and their access |
+| `+record-list` network failure / timeout mid-pagination | Same rule as a lost upload batch | Treat the snapshot as unknown; stop and confirm |

package/skills/ae-data-integration/references/lark-sheet-source.md

@@ -0,0 +1,130 @@
+# Source — online document (Feishu sheet)
+
+## When this branch applies
+
+The input is a **URL**, not a local path. Feishu sheet links (`/sheets/`), spreadsheet links
+(`/spreadsheets/`), and wiki links that resolve to a sheet (`/wiki/`) all route here instead of
+[source-inspect.md](source-inspect.md). The pipeline is identical after the sheet is snapshotted
+to a local XLSX: business identification, tracking plan, transform, and sink all reuse the
+local-file flow unchanged.
+
+Only **Feishu sheets** and **Feishu Bitable (多维表格)** are supported in this phase. Bitable links
+(`/base/`) — including wiki links that resolve to a Bitable — route to
+[lark-bitable-source.md](lark-bitable-source.md) instead of this page. Other online documents
+(Tencent Docs, DingTalk, WPS, Google Sheets, Notion) have no CLI read path — route to
+[manual-export-source.md](manual-export-source.md), which has the user export the file themselves
+and then rejoins the local-file flow.
+
+## Safety rules (online-specific)
+
+- The URL, its token, the snapshot file path, and every cell value are sensitive. Never print
+  source values while inspecting; summarize types, ratios, counts, and fingerprints only.
+- Pulling a sheet to the local machine moves data across a trust boundary. Before downloading
+  anything, tell the user the sheet will be snapshotted to a local file, and get their
+  confirmation when the sheet is not their own data.
+- The snapshot is a point-in-time copy. State the export timestamp, and note that the import
+  reflects the exported moment only — a sheet that may be edited mid-run is imported as of the
+  snapshot, not live.
+- lark-cli (Feishu) and ae-cli (AE) are two separate credential domains. Read Feishu with
+  lark-cli; upload to AE with ae-cli. Never forward a token from one to the other, and never
+  send an AE access token or CLI token to any Feishu endpoint.
+
+## Step 0 — Environment self-check (before any pull)
+
+The Feishu read path depends on `lark-cli`, a CLI separate from ae-cli. Check it up front so a
+missing install or a missing login surfaces as an environment problem, not as a data problem
+mid-pipeline.
+
+1. **Detect lark-cli.** Run `lark-cli --version`. If the command is not found, stop and tell the
+   user to install it (`npm i -g @larksuite/cli`). Do not attempt to install it yourself, and do
+   not fall back to another read path — a half-provisioned Feishu client produces worse failures
+   than a clear "not installed" stop.
+2. **Check the login.** Run
+   `LARKSUITE_CLI_NO_UPDATE_NOTIFIER=1 LARKSUITE_CLI_NO_SKILLS_NOTIFIER=1 lark-cli auth status --json --verify`.
+   Reading a sheet in the user's cloud space needs the **user** identity (`--as user`); the bot
+   identity cannot see a user's drive documents. If the user identity is unavailable or
+   unverified, follow the lark-shared skill's split-flow: start
+   `lark-cli auth login --scope "<scope>" --no-wait --json`, show the QR code / link to the user,
+   and only after the user confirms run `lark-cli auth login --device-code <device_code>`.
+   Authorization requires the user's participation — never bypass or automate it.
+3. **Handle permission errors when they occur.** If a pull returns `permission_violations`
+   (missing scopes) together with `console_url` and `hint`, route by identity:
+   - user identity → add the missing scope with
+     `lark-cli auth login --scope "<missing_scope>"` (incremental grant);
+   - bot identity → direct the user to open the scope in the developer console via the returned
+     `console_url`; never run `auth login` for a bot.
+   If the sheet itself is not accessible to the user's account — a resource-side permission
+   problem, not a scope problem — ask the user to confirm the link is correct and that their
+   Feishu account can open the sheet. Do not retry against a different sheet silently.
+
+## Step 1 — Locate the workbook and sheets
+
+1. Run `lark-cli sheets +workbook-info --url "<url>"` to list the sheets (`sheet_id`, `title`,
+   `row_count`, `column_count`, `is_hidden`). Never guess a sheet id or the name `Sheet1`.
+2. Surface the sheet list to the user. A workbook with one visible sheet proceeds with it;
+   multiple sheets ask the user to choose, mirroring `--data-set` on a local workbook. Report any
+   hidden sheet (`is_hidden`) by name — it is usually scratch space or a superseded draft, and
+   the same "ask before reading a hidden sheet" rule as a local XLSX applies. (`inspect` already
+   excludes hidden sheets from discovery, listing them under `excluded_sheets` with reason
+   `hidden` — name them anyway rather than relying on silent exclusion.) Show each sheet's row
+   count and, when available, its time coverage so an overlapping or derived sheet is visible
+   before it is chosen.
+
+## Step 2 — Snapshot to a local XLSX
+
+Export the workbook to an XLSX under the pipeline's working directory (a cwd-relative path, so
+lark-cli's file-path rules accept it):
+
+```bash
+lark-cli sheets +workbook-export --url "<url>" --output-path ".ae-cli/data-integration/online/<fingerprint>/snapshot.xlsx"
+```
+
+- **Export the whole workbook as XLSX**, not CSV: the XLSX export preserves dates, numbers,
+  merged cells, and hidden rows, so the downstream XLSX reader (date-cell detection, merged
+  cells, hidden rows) and the type inference behave exactly as for a local file. One caveat
+  (verified): the export keeps a formula's text but **drops the cached computed result**, so a
+  formula cell reads as missing — `inspect` flags it with a warning naming the affected columns
+  ("a formula whose last computed result is not stored in the file"). Surface that warning when
+  the source sheet contains formulas; the skill never evaluates a formula or guesses its value.
+  Do **not** use `+csv-get` for the snapshot — its display values round long numbers to
+  scientific notation (`1.04E+14`) and flatten types, which would silently corrupt ID-like
+  columns and lock wrong property types in AE.
+- `<fingerprint>` is a short, collision-resistant name derived from the spreadsheet token plus a
+  timestamp (e.g. first 8 chars of the token hash + `YYYYMMDD-HHMMSS`). One online source gets
+  one directory, so a re-export never overwrites an unrelated source. Create the directory before
+  exporting.
+- The export is an async job with built-in polling. If it returns `timed_out=true`, resume with
+  its `next_command` rather than re-exporting blindly.
+- If the export fails on a network error or times out mid-job, treat the snapshot as unknown:
+  stop, and ask the user to confirm the sheet state before retrying — the same rule as a lost
+  upload batch. Never resume a half-written snapshot silently.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the snapshot as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '.ae-cli/data-integration/online/<fingerprint>/snapshot.xlsx'
+```
+
+When the exported workbook holds multiple sheets, pass `--data-set '<chosen-sheet>'` using the
+sheet the user picked in Step 1 — the downstream multi-sheet selection is answered by the same
+choice, not re-asked.
+
+Everything downstream — business identification, tracking plan, transform, sink — is unchanged.
+The snapshot lives under `.ae-cli/data-integration/`, so [reuse](reuse.md) and
+[handoff](handoff.md) find it like any other pipeline artifact; record in the handoff that the
+source was an online Feishu sheet and include the snapshot timestamp.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Online-source problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| `lark-cli` not installed | Not a parse error, not a data problem | Stop; give the install command |
+| lark-cli not logged in / no user identity | Not a parse error | Split-flow authorization with the user |
+| Missing scope (`permission_violations`) | Not a parse error | Incremental `auth login --scope` (user) or developer console (bot) |
+| Sheet not accessible to the user's account | Not a scope problem | Ask the user to verify the link and their access |
+| Export job timed out / network lost | Same rule as a lost upload batch | Treat the snapshot as unknown; stop and confirm |

package/skills/ae-data-integration/references/local-analysis.md

@@ -1,5 +1,7 @@
 # Local analysis
 
+Dimension / dictionary data (a stable-entity lookup with a join key) is not local-analysis material — it routes to ae-metadata as a dimension table. See [ue-routing.md](ue-routing.md) and [dimension-routing.md](dimension-routing.md).
+
 Keep the source on the local machine. Generated scripts and reports belong under `.ae-cli/data-integration/runs/<run-id>/` with restrictive permissions.
 Set the directory to `0700` and generated scripts/reports to `0600`.
 

package/skills/ae-data-integration/references/manual-export-source.md

@@ -0,0 +1,83 @@
+# Source — online document, manual export (Tencent Docs / DingTalk / WPS)
+
+## When this branch applies
+
+The input is a **URL** that points to an online document on a platform other than Feishu:
+Tencent Docs (`docs.qq.com`), DingTalk docs, WPS / Kingsoft Docs (`kdocs.cn`), Google Sheets,
+Notion, and the like. These platforms have **no CLI read path** in this phase — only Feishu has
+one, via lark-cli (see [lark-sheet-source.md](lark-sheet-source.md) and
+[lark-bitable-source.md](lark-bitable-source.md)). There is therefore no automatic snapshot here:
+the user exports the document to a local file themselves, and the pipeline rejoins at the
+local-file flow unchanged.
+
+## Safety rules
+
+- The URL and every exported cell value are sensitive. Never print source values while
+  inspecting; summarize types, ratios, counts, and fingerprints only.
+- The export is a point-in-time copy made by the user on the source platform. State the export
+  timestamp, and note that the import reflects the exported moment only — a document edited
+  mid-run is imported as of the export, not live.
+- The export is the user's own action on the source platform. Do not automate it, and do not ask
+  for that platform's credentials — there is no Feishu-style OAuth to run on this branch.
+
+## Step 1 — Recognize the platform and route
+
+Inspect the URL's host to confirm it is not Feishu. Feishu hosts (`feishu.cn`, `larksuite.com`)
+route to the two lark source pages instead of this one; any other online-document host
+(`docs.qq.com`, `kdocs.cn`, a DingTalk docs host, etc.) lands here. If the host is ambiguous,
+ask the user which platform the link belongs to rather than guessing.
+
+## Step 2 — Guide the user to export
+
+Ask the user to export the document from the source platform's own UI to a local file, and to
+name the file so its origin is traceable. Give only the general route — the exact menu labels
+differ by platform and change over time, so follow the platform's current export UI rather than a
+memorized click path:
+
+- open the document, then use the platform's export / save-as / download action;
+- choose **XLSX** (Excel workbook) where offered, not CSV — CSV flattens types, may round long
+  ID-like numbers to scientific notation, and loses multiple sheets. This is the same reason the
+  Feishu sheet flow snapshots XLSX, not CSV (see
+  [lark-sheet-source.md](lark-sheet-source.md));
+- if the platform only offers CSV, accept it but flag the type-flattening risk to the user, and
+  watch for ID-like columns that must stay text;
+- ask the user to place the file under the pipeline's working directory (e.g.
+  `.ae-cli/data-integration/manual/<fingerprint>/<name>.xlsx`) so [reuse](reuse.md) and
+  [handoff](handoff.md) find it like any other pipeline artifact.
+
+Export-time caveats to surface to the user — platform-dependent, and the same rules as a local
+file:
+
+- **multiple sheets** — the export may contain several sheets; confirm which one is the data set,
+  mirroring `--data-set` on a local workbook.
+- **formulas** — an exported formula cell may carry only its text, not the cached computed result;
+  `inspect` flags such cells as missing. The skill never evaluates a formula or guesses its value.
+- **hidden rows/columns/sheets** — the platform may drop or keep hidden content depending on its
+  export; `inspect` excludes what it can detect (hidden sheets appear under `excluded_sheets`),
+  but ask the user to confirm the export contains what they expect before profiling.
+- **dates / encoding** — prefer a UTF-8 export; if dates arrive as text, say so rather than
+  silently coercing them.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the exported file as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '<exported-file>'
+```
+
+When the export holds multiple sheets, pass `--data-set '<chosen-sheet>'` using the sheet the
+user picked in Step 2. Everything downstream — business identification, tracking plan, transform,
+sink — is unchanged. Record in the handoff that the source was a manually-exported online
+document, name the platform and the export timestamp, and include the exported file path.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Manual-export problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| Platform only exports CSV; types flatten | Not a parse error | Flag the risk; watch ID-like columns |
+| Export is missing sheets / rows the user expects | Not a data problem yet | Ask the user to re-export or confirm the data set |
+| URL host is unrecognized and not Feishu | Not a routing failure | Ask the user which platform the link belongs to |

package/skills/ae-data-integration/references/reuse.md

@@ -37,6 +37,8 @@ node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file>
 
 Do **not** run the returned command on your own. Show the user the proposed package — the frozen event name, the property mapping it implies, and the fact that the confirmed business logic (event name, `value_mapping`, flatten rules) is reused unchanged — and wait for one explicit confirmation. Before confirming, diff the new file's flattened properties against the package's `plan.json`: if the package has no `plan_file`, or the new file produces properties the plan does not cover, run the Tracking plan step to add them (merge into the existing project plan) first. Only then run `transform.mjs` on the new file and continue to Sink.
 
+The frozen mapping's `#type`/`mode` (e.g. `track`) is authoritative after a match — ignore `inspect`'s auto recommendation, which may re-infer `user_set` for a package whose confirmed mapping is `track`. A reuse match must not re-do Source identification or event-model inference.
+
 ## Safety rules
 
 - Reuse only for a file of the **same shape** (same header/schema and format). The `transform.mjs` wrapper re-binds the content fingerprint to the new file, so the content guard still applies per run.

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,10 +31,42 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
+Before the confirmation gate, re-check the project's **runtime-locked** property types. A `reuse` match skips the tracking-plan step, so this check is the one that must not be bypassed on any path — repeat it for every upload:
+
+```bash
+ae-cli analysis-meta property list -p <project-id> --table-type user
+ae-cli analysis-meta property list -p <project-id> --table-type event
+```
+
+Diff each target property name against `select_type` (the locked type). A same-name property whose locked type disagrees with the mapping's type drops those values silently at ingest, even when the plan layer agrees — surface any such conflict in the confirmation gate (rename the target, or change the mapped type to match the locked one) before asking the user to confirm. This is a read-only check; do not run `ae-cli tracking plan sync-from-meta` here, which writes the plan.
+
 Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.
 
+`upload` is **not idempotent** — every execution submits to the receiver again. Never re-run the
+command to inspect a response: read the first run's saved JSON output instead. `--dry-run` only
+previews the request it would send; it neither executes nor previews the receiver's real response.
+
+The execute success response reports the receiver outcome under **different field names than the
+dry-run preview** — do not reuse the dry-run names (`record_count` / `batch_count`) when reading
+it:
+
+| field | meaning |
+| --- | --- |
+| `status` | `receiver_accepted` (or `partially_delivered` when `--retry` salvaged some records) |
+| `delivery_state` | `receiver_accepted` / `partially_delivered` |
+| `submitted_records` | records submitted this run (equals `record_count - resume_from` on a clean run) |
+| `submitted_batches` | batches sent this run |
+| `resume_from` | zero-based offset this run started at |
+| `request_bytes` / `response_bytes` | transport byte counts |
+| `persistence_verified` | always `false` — receiver acceptance is not durable storage |
+| `next_step` | the verification guidance |
+
+`--retry` salvage additionally reports `failed_records` / `failed_count`. A lost batch is reported
+as `delivery_state=unknown` in the error's `meta`, not in the success fields — see
+[sync-json-upload.md](sync-json-upload.md) "Interrupted delivery".
+
 ## Verify the data landed
 
 Ingestion is asynchronous — expect about a 1-minute delay between `receiver_accepted` and the data being queryable. Instead of telling the user to check in the console, offer to verify with ae-cli (control-plane commands; require the CLI to be logged in to the AE host and a project-id):

package/skills/ae-data-integration/references/source-inspect.md

@@ -63,7 +63,7 @@ Summarize both — report the shape of the distribution and the names of the val
 - **Excel date cells** — a cell whose number format is a date or date+time is read as the wall-clock timestamp shown in Excel, not as the Excel serial number stored behind it, so the column infers as `datetime` and can serve as the time field. Inspect lists every such column in a warning. Treat that warning as a question to the user, not as a note: the same column profiled as `number` before this behavior existed, so if any part of this file was already sent to AE, the property may have been received as a number and its type is now locked — it cannot be changed to datetime, and the column has to be re-sent under a new property name. Ask whether the column was uploaded before, and only map it once the user answers. Elapsed-duration formats (`[h]:mm:ss` and the equivalent built-ins) are durations rather than points in time and stay `number`.
 - **Excel formula cells** — a spreadsheet stores a formula and, next to it, the result Excel last computed. That cached result is the value: it is read normally, including a result of `0` or `""`, which are real values and not blanks. This tool never evaluates a formula and never guesses a result, so a cell holding a formula the file never computed has nothing to upload; it is read as missing and counted, as is an Excel error value (`#N/A`, `#DIV/0!`, …). Inspect reports the counts per column in a warning and `convert` repeats them in `manifest.output.unreadable_cells`. Report them: the rows are kept and the record count is unchanged, so this is the only explanation for a column that is empty in AE while the spreadsheet looks full. When a column that matters reads as missing this way, ask the user to recalculate and re-export in Excel, or to export values instead of formulas, before uploading. This covers XLSX; a legacy `.xls` workbook goes through a different parser and is not counted here.
 - **Merged cells, hidden rows, and hidden columns** — a sheet maintained by hand merges a label down the rows it covers (`区域` spanning one region's block). Excel keeps that value on the block's first row only and stores every row below it as an empty cell, so a column that looks full on screen arrives mostly missing, and the AE property built from it would be empty for most events. The same worksheet may also hide a row inside a data block or hide a whole column. None of this travels with a row, so inspect scans the worksheet structure separately and reports it under `xlsx_structure`: `merged_ranges` with `merged_range_samples` (references such as `A3:A5`, never cell text), `merged_covered_cells` per column, `hidden_rows` with `hidden_row_samples` (source row numbers as Excel numbers them), and `hidden_columns` by header name. The default read is unchanged, so the report is a question for the user, and each answer is a flag: `--fill-merged-cells` copies each block's value into the cells its own range covers — bounded to the range, never overwriting a value that is there and never inventing one when the block's own cell is empty, so it is not a forward fill; `--exclude-hidden-rows` leaves hidden rows out. Neither is on by default: those cells really are empty in the file, and a row hidden inside a data block may still be real data — unlike a hidden *worksheet* (below), which is excluded by default. Hidden columns have no flag at all; when the user confirms one is not data, list it in the mapping's `exclude_columns`. Both flags are carried into the recommended mapping as `fill_merged_cells` / `exclude_hidden_rows`, which is what makes `convert` read the rows inspect profiled — `convert` has no read flags of its own — and `convert` repeats the findings in `manifest.output.xlsx_structure`, the only record of a layout the converted rows no longer show. This covers XLSX; a legacy `.xls` workbook is not scanned, so ask the user about merged labels and hidden rows there instead of trusting silence.
-- **Hidden worksheets** — a worksheet hidden in the workbook is left out of the `--data-set` candidates and out of `--merge-sheets`, because a sheet the file does not show is usually scratch space, a lookup table, or a superseded draft rather than rows anyone meant to upload. Inspect lists each one under `excluded_sheets` (with `reason: hidden`); report those names to the user, since they are the only explanation for a row count lower than the workbook appears to hold. Their headers are also left out of `header_consistency`, so a stale hidden draft cannot make a mergeable workbook look ragged. A hidden sheet stays readable when the user names it in `--data-set` — the command then warns on stderr that the selected sheet is hidden. Only pass a hidden sheet after the user says that is what they want. When *every* worksheet is hidden there is no candidate left, and inspect fails with `LOCAL_DATA_ALL_DATA_SETS_HIDDEN` whose hint lists the hidden sheets; treat that as a question about which sheet holds the real data, not as an unreadable file. This detection covers XLSX only: a legacy `.xls` workbook's sheet list is unfiltered, so a hidden sheet there still appears as a candidate and is still merged — for `.xls`, ask the user to confirm the sheet list instead of trusting it.
+- **Hidden worksheets** — a worksheet hidden in the workbook is left out of the `--data-set` candidates and out of `--merge-sheets`, because a sheet the file does not show is usually scratch space or a superseded draft rather than rows anyone meant to upload, but a hidden sheet can also be a dimension / dictionary table that is meant to be loaded, and that goes through dimension routing instead of being dismissed as scratch. Inspect lists each one under `excluded_sheets` (with `reason: hidden`); report those names to the user, since they are the only explanation for a row count lower than the workbook appears to hold. Their headers are also left out of `header_consistency`, so a stale hidden draft cannot make a mergeable workbook look ragged. A hidden sheet stays readable when the user names it in `--data-set` — the command then warns on stderr that the selected sheet is hidden. Only pass a hidden sheet after the user says that is what they want. When *every* worksheet is hidden there is no candidate left, and inspect fails with `LOCAL_DATA_ALL_DATA_SETS_HIDDEN` whose hint lists the hidden sheets; treat that as a question about which sheet holds the real data, not as an unreadable file. This detection covers XLSX only: a legacy `.xls` workbook's sheet list is unfiltered, so a hidden sheet there still appears as a candidate and is still merged — for `.xls`, ask the user to confirm the sheet list instead of trusting it.
 - **Excel sheets** — `--merge-sheets` streams every visible worksheet in file order instead of a single selected sheet; otherwise ask which sheet/`--data-set` to use. Inspect also reports `header_consistency` (`all_same` or `different`) across a workbook's sheets, with `header_details` listing each sheet's header row when they differ; prefer `--merge-sheets` only when headers match. Matching headers establish a shared structure, not disjoint rows: a detail sheet and a summary sheet, or `1月` and `1月修订版`, usually carry identical headers and would be merged and reported twice over. Before merging, confirm with the user that the sources are mutually exclusive partitions (one month per sheet, no overlap) rather than overlapping, revised, or derived views of the same rows, and show each sheet's row count and time coverage range in that confirmation so an overlap is visible. The same rule applies to repeated `--input-file`.
 - **Multi-file type conflicts** — when the same column has different inferred types across files, present each conflict and resolve with `--type-resolutions` on `convert` (see [transform](transform.md)).
 

package/skills/ae-data-integration/references/tracking-plan.md

@@ -10,9 +10,12 @@ Generate and confirm the event/property plan **before** any transform or upload.
 ## Sub-steps
 
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
-2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
+2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case by default (keep an uppercase event name only when the user asks to preserve it; property names stay lowercase-only) and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
 3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
-4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
+4. **Merge with the existing plan — and the project's runtime property metadata.** This runs for **every** file, not just the first, and it checks **two layers**, which are separate stores:
+   - **Plan layer (declarative)** — `ae-cli tracking plan fetch -p <project-id>`: same-name property type conflicts are severe, same-name events are advisory; decide append vs replace.
+   - **Runtime metadata layer (authoritative)** — `ae-cli analysis-meta property list -p <project-id> --table-type user` and `--table-type event`: each property's **locked** type is in `select_type`, and it is what actually gates ingestion. A same-name property whose `select_type` disagrees with the mapping's intended type is a **severe** conflict: those values are dropped silently at ingest even when the plan layer agrees with the mapping. Diff every target property name against this list, translating the mapping type to AE's runtime naming before comparing (`boolean` → `bool`, `list` → `array_string`), and put any mismatch in the confirmation gate (rename the target, or change the mapped type to match the locked one).
+   Diff this file's events and properties against both layers and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user. Do not run `ae-cli tracking plan sync-from-meta` for this check — it is a write (it re-syncs the plan from metadata), not a read-only pre-flight.
 5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 

package/skills/ae-data-integration/references/transform.md

@@ -17,7 +17,7 @@ Confirm the system fields with the user before touching properties. These are th
 4. **`#event_name` (track only).** Confirm the event-name column, or a reviewed `default_event_name` when no column exists.
 5. **`#ip` / `#uuid` (optional).** Ask only when the data has an IP- or UUID-like column; map it via `ip_field`/`uuid_field`, otherwise skip. `#ip` is event data only and must be a valid IPv4/IPv6 address (a private/LAN IP is kept but reported — AE cannot geolocate it); `#uuid` must be a standard 36-character UUID. A value that violates the spec is dropped from that row only (`INVALID_IP` / `INVALID_UUID`) — the row itself is kept. The program never auto-generates a `#uuid`.
 
-When an `#event_name`, `#account_id`, or `#distinct_id` column's values do not satisfy AE naming rules (pure Chinese, uppercase, spaces), do not stop — scan the distinct values, list them to the user, and ask for one AE-name replacement each; record the pairs in `value_mapping` (see [UE mapping](ue-mapping.md)). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded. The same mechanism applies to a property column via that entry's own `value_mapping`.
+When an `#event_name`, `#account_id`, or `#distinct_id` column's values do not satisfy AE naming rules (pure Chinese, spaces), do not stop — scan the distinct values, list them to the user, and ask for one AE-name replacement each; record the pairs in `value_mapping` (see [UE mapping](ue-mapping.md)). Event names are lowercased by default; keep an uppercase event name only when the user asks to preserve it — leave a legal uppercase value unmapped so it passes through, or map an illegal value to an uppercase target (e.g. `购买` → `Purchase`); see the casing rules in [UE mapping](ue-mapping.md). A value with no matching key keeps its original text and fails validation, so confirm every distinct value is covered or excluded. The same mechanism applies to a property column via that entry's own `value_mapping` — property names stay lowercase-only, so uppercase property values still need `value_mapping`.
 
 Then confirm the property set with the user before saving the mapping. Present every property as a readable table — one row per property with source column, target AE name, and type — never dump the raw mapping JSON at the user. Whether the rows are event or user properties follows `mode`: `track` → **event properties**, `user_set` (or another profile mode) → **user properties**, `mixed` → the same set applies to both event and user rows.