@thinkingai/ae-cli 6.1.20 → 6.1.21

package/skills/ae-data-integration/references/lark-bitable-source.md

@@ -0,0 +1,288 @@
+# Source — online document (Feishu Bitable)
+
+## When this branch applies
+
+The input is a **URL**, not a local path. Feishu Bitable (多维表格 / Base) links (`/base/`), and
+wiki links that resolve to a Bitable, route here instead of [source-inspect.md](source-inspect.md).
+Feishu sheet links still route to [lark-sheet-source.md](lark-sheet-source.md); a `/wiki/` link can be
+either, so resolve it first (Step 1) and branch on the resolved type. The pipeline is identical
+after the table is snapshotted to a local JSONL: business identification, tracking plan,
+transform, and sink all reuse the local-file flow unchanged.
+
+Only **Feishu sheets** and **Feishu Bitable (多维表格)** are supported in this phase. Other online
+documents (Tencent Docs, DingTalk, WPS, Google Sheets, Notion) have no CLI read path — route to
+[manual-export-source.md](manual-export-source.md), which has the user export the data themselves
+and then rejoins the local-file flow.
+
+## Safety rules (online-specific)
+
+- The URL, its token, the snapshot file path, and every record value are sensitive. Never print
+  source values while inspecting; summarize types, ratios, counts, and fingerprints only.
+- Pulling records to the local machine moves data across a trust boundary. Before downloading
+  anything, tell the user the table will be snapshotted to a local file, and get their
+  confirmation when the data is not their own.
+- The snapshot is a point-in-time copy. State the export timestamp (and the base timezone the
+  API reports), and note that the import reflects the exported moment only — a table that may be
+  edited mid-run is imported as of the snapshot, not live.
+- lark-cli (Feishu) and ae-cli (AE) are two separate credential domains. Read Feishu with
+  lark-cli; upload to AE with ae-cli. Never forward a token from one to the other, and never
+  send an AE access token or CLI token to any Feishu endpoint.
+
+## Step 0 — Environment self-check (before any pull)
+
+The Feishu read path depends on `lark-cli`, a CLI separate from ae-cli. Check it up front so a
+missing install or a missing login surfaces as an environment problem, not as a data problem
+mid-pipeline.
+
+1. **Detect lark-cli.** Run `lark-cli --version`. If the command is not found, stop and tell the
+   user to install it (`npm i -g @larksuite/cli`). Do not attempt to install it yourself, and do
+   not fall back to another read path — a half-provisioned Feishu client produces worse failures
+   than a clear "not installed" stop.
+2. **Check the login.** Run
+   `LARKSUITE_CLI_NO_UPDATE_NOTIFIER=1 LARKSUITE_CLI_NO_SKILLS_NOTIFIER=1 lark-cli auth status --json --verify`.
+   Reading a Bitable in the user's cloud space needs the **user** identity (`--as user`); the bot
+   identity cannot see a user's drive documents. If the user identity is unavailable or
+   unverified, follow the lark-shared skill's split-flow: start
+   `lark-cli auth login --scope "<scope>" --no-wait --json`, show the QR code / link to the user,
+   and only after the user confirms run `lark-cli auth login --device-code <device_code>`.
+   Authorization requires the user's participation — never bypass or automate it. The Bitable
+   read path needs the scopes `base:table:read`, `base:record:read`, and `base:field:read`.
+3. **Handle permission errors when they occur.** If a pull returns `permission_violations`
+   (missing scopes) together with `console_url` and `hint`, route by identity:
+   - user identity → add the missing scope with
+     `lark-cli auth login --scope "<missing_scope>"` (incremental grant);
+   - bot identity → direct the user to open the scope in the developer console via the returned
+     `console_url`; never run `auth login` for a bot.
+   If the base itself is not accessible to the user's account — a resource-side permission
+   problem, not a scope problem — ask the user to confirm the link is correct and that their
+   Feishu account can open the base. Do not retry against a different base silently.
+
+## Step 1 — Resolve the URL and locate the table
+
+1. **Resolve the URL to a base token.** Primary (richer — returns the title and unwraps wiki
+   links):
+   ```bash
+   lark-cli drive +inspect --url "<url>" --as user
+   ```
+   returns `{ok, identity: "user", data: {token, title, type, url}}`. A Bitable resolves with
+   `type: "bitable"` and its base token in `token`. If `type` is not `bitable`, leave this branch
+   — a sheet routes to [lark-sheet-source.md](lark-sheet-source.md). Domain-native alternative, which also
+   accepts wiki and record-share links:
+   ```bash
+   lark-cli base +url-resolve --url "<url>" --as user
+   ```
+   returns `{ok, identity: "user", data: {base_token, resource_type: "bitable", input_type}}`.
+2. **List the tables.**
+   ```bash
+   lark-cli base +table-list --base-token <base_token>
+   ```
+   returns `{ok, data: {tables: [{id, name, records_count, rev}], total}}`. Never guess a table
+   id or name. Surface the table list to the user with each table's record count; a base with one
+   table proceeds with it, multiple tables ask the user to choose, mirroring `--data-set` on a
+   local workbook.
+
+## Step 2 — Snapshot to a local JSONL
+
+The base domain has **no workbook-export command** (only per-record reads and an attachment
+downloader), so the snapshot is a **JSONL** built by paginating `+record-list` and zipping each
+page into per-record objects — not an XLSX. The snapshot directory convention matches the sheet
+flow: `.ae-cli/data-integration/online/<fingerprint>/snapshot.jsonl`, where `<fingerprint>` is a
+short, collision-resistant name derived from the base token plus a timestamp (e.g. first 8 chars
+of `sha256("<base_token>-<YYYYMMDD-HHMMSS>")`). One online source gets one directory, so a
+re-export never overwrites an unrelated source. Create the directory before writing.
+
+1. **Survey the fields** (type knowledge for the mapping step, and attachment detection):
+   ```bash
+   lark-cli base +field-list --base-token <base_token> --table-id <table_id>
+   ```
+   returns
+   `{ok, data: {fields: [{id, name, type, style?, options?, multiple?, default_value?}], total}}`.
+   `style` and `options` are display metadata (currency code, date format, choice list);
+   `multiple` distinguishes single- from multi-select. Field-list order is NOT the record-list
+   order — do not carry it across to the zip step.
+2. **Paginate the records.** Always pass `--format json` explicitly — `+record-list` defaults to
+   **markdown**, unlike `+table-list`/`+field-list` which already default to json:
+   ```bash
+   lark-cli base +record-list --base-token <base_token> --table-id <table_id> --format json
+   ```
+   The JSON envelope is **column-oriented**, not record-oriented:
+   ```json
+   {
+     "ok": true, "identity": "user",
+     "data": {
+       "data": [["<…>", "<…>", true, 129.5, ["<…>"], ["<…>"], "<ISO8601>", ["<…>"]],
+       "fields": ["<…>", "<…>", "<…>", "<…>", "<…>", "<…>", "<…>", "<…>"],
+       "field_id_list": ["<…>", "…"],
+       "field_type_list": ["<…>", "…"],
+       "record_id_list": ["<…>", "…"],
+       "has_more": true,
+       "query_context": { "field_scope": "all_fields", "record_scope": "all_records" },
+       "timezone": "Asia/Shanghai",
+       "rev": 1,
+       "total": null
+     }
+   }
+   ```
+   (`total` may be absent rather than null — see the pagination contract below.)
+   `data.data` rows are positional arrays aligned with the parallel `fields`, `field_id_list`,
+   `field_type_list`, and `record_id_list` arrays. Pagination contract (verified):
+   - `--offset` is a 0-based row skip; start at 0.
+   - `--limit` is 1–200 (default 100).
+   - After each page, advance `offset` by **the number of rows returned**, not by `limit` — a
+     short final page must not skip rows.
+   - Loop until `has_more: false`. `total` is null or absent in this envelope — never use it to
+     detect the end.
+   - Each page's `fields` order is authoritative for **that page**. Field-list order ≠
+     record-list order, and `--field-id` projection re-aligns the parallel arrays — always zip
+     with the `fields` array from the same response.
+3. **Zip each page into per-record objects.** The flat snapshot reserves `__record_id` for the
+   stable ID returned by the Bitable API. Record `i` becomes
+   `{ __record_id: record_id_list[i], fields[j]: data[i][j] for each j }`. A business field named
+   `record_id` remains an ordinary source column. If the table itself contains a business field
+   named `__record_id`, stop before writing any snapshot instead of overwriting either value or
+   silently renaming the user's field. Reference implementation (paginates, zips, writes
+   `snapshot.jsonl`, prints a summary only — no record values):
+   ```javascript
+   import { execFileSync } from "node:child_process";
+   import { mkdirSync, writeFileSync } from "node:fs";
+   import path from "node:path";
+
+   const [baseToken, tableId, outDir] = process.argv.slice(2);
+   mkdirSync(outDir, { recursive: true });
+
+   function fetchPage(offset) {
+     const stdout = execFileSync(
+       "lark-cli",
+       ["base", "+record-list", "--base-token", baseToken, "--table-id", tableId,
+        "--offset", String(offset), "--limit", "200", "--format", "json"],
+       { encoding: "utf8", timeout: 120_000,
+         env: { ...process.env,
+                LARKSUITE_CLI_NO_UPDATE_NOTIFIER: "1",
+                LARKSUITE_CLI_NO_SKILLS_NOTIFIER: "1" } }
+     );
+     return JSON.parse(stdout);
+   }
+
+   const RECORD_ID_KEY = "__record_id";
+   let offset = 0; const records = []; let meta = null;
+   while (true) {
+     const envelope = fetchPage(offset);
+     if (!envelope.ok) { console.error("record-list failed:", JSON.stringify(envelope.error ?? envelope)); process.exit(1); }
+     const d = envelope.data;
+     const rows = d.data ?? [], ids = d.record_id_list ?? [], fields = d.fields ?? [];
+     if (rows.length !== ids.length) { console.error(`row/id length mismatch: rows=${rows.length} ids=${ids.length}`); process.exit(1); }
+     if (fields.includes(RECORD_ID_KEY)) { console.error(`reserved field name collision: ${RECORD_ID_KEY}`); process.exit(1); }
+     for (let i = 0; i < rows.length; i++) {
+       const rec = { [RECORD_ID_KEY]: ids[i] };
+       for (let j = 0; j < fields.length; j++) rec[fields[j]] = rows[i][j];
+       records.push(rec);
+     }
+     meta = { timezone: d.timezone, rev: d.rev, total: d.total, fieldNames: fields };
+     if (!d.has_more) break;
+     offset += rows.length;
+   }
+
+   const outPath = path.join(outDir, "snapshot.jsonl");
+   writeFileSync(outPath, records.map((r) => JSON.stringify(r)).join("\n") + "\n", "utf8");
+   console.log(JSON.stringify({ snapshotPath: outPath, records: records.length,
+     fieldCount: meta.fieldNames.length, fieldNames: meta.fieldNames,
+     timezone: meta.timezone, rev: meta.rev, totalReported: meta.total }));
+   ```
+   Optional scoping flags (all verified; default is all records, all fields): `--field-id
+   <name|id>` (repeatable) projects to specific fields; `--view-id` reads a specific view;
+   `--filter-json` overrides the view's own filter, e.g.
+   `{"logic":"and","conditions":[["Title","==","Launch plan"]]}` (equality),
+   `[["Title","intersects","urgent"]]` (contains), `[["Score","==",95]]` (numbers unquoted);
+   `--sort-json` (max 10 rules) overrides the view's sort, e.g.
+   `[{"field":"Updated","desc":true}]`.
+
+   Field-type mapping and value shapes in the snapshot:
+
+   | Bitable field type | `--format json` value shape | Downstream `inspect` inference |
+   | --- | --- | --- |
+   | text, `style.type: plain/phone/email/barcode` | string | string |
+   | text, `style.type: url` | **markdown link** `[display](url)`, not the raw URL | string — the markdown wrapper is part of the value |
+   | number (plain/currency/progress/rating; `style` is display-only) | plain number | number |
+   | checkbox | boolean | boolean |
+   | select, `multiple: false` (single-select) | **single-element array** (e.g. `["paid"]`) | **`list`**, not string |
+   | select, `multiple: true` (multi-select) | array of strings; **empty selection is `null`, not `[]`** | `list` (empty rows counted as missing) |
+   | datetime (any `style.format`) | full ISO8601 string with offset; envelope `timezone` states the base timezone | datetime |
+   | created_at / updated_at | full ISO8601 string | datetime |
+   | auto_number | string (e.g. `NO.001`) | string |
+   | created_by / updated_by | array of `{id, name}` user objects (single-element for one creator) | `list` |
+   | location | object `{lng, lat, full_address}` | json/object |
+   | link | array of `{id}` record refs (single-element for one link) | `list` |
+   | attachment | `null` when empty (verified); non-empty values are out of scope | all-null column |
+   | formula | **always a string, even for numeric results** (e.g. `"20"`) | string |
+   | lookup | data-dependent: scalar string for a single `raw_value` match; list for multiple matches or a list aggregate | string or list |
+
+   Warnings the mapping step must carry:
+
+   - single-select columns infer as `list` (not `string`) because Bitable returns single-element
+     arrays — surface this when the user confirms the mapping.
+   - `created_by` / `updated_by` are **single-value** fields the API wraps in a one-element array
+     of `{id, name}` user objects — surface this when the user confirms the mapping, and do not
+     treat them as multi-value. The `id` is a Feishu open_id/user_id, not an AE identity; these
+     are audit metadata. Default to excluding them, or import only `name` as a plain `string`
+     property (after confirming with the user); do not map them to a `list` or an `{id, name}`
+     `object` unless the user explicitly asks. A custom **person** field is different:
+     single-person fields wrap the same way, multi-person fields are genuinely multi-value —
+     confirm which before mapping.
+   - `link` (single-link) columns infer as `list` because the API wraps a single link in a
+     one-element array of `{id}` record refs — surface this when the user confirms the mapping,
+     and do not treat them as multi-value. The `{id}` is the linked record's internal
+     `record_id`, not a business value: exclude the column, or flatten the linked record's real
+     fields via a `lookup`/`formula` on the source table instead of importing record_ids.
+     A multi-link field returns multiple `{id}` refs per row; `+field-list` exposes no
+     `multiple` flag for link fields, so distinguish single from multi by the value array
+     length across rows, not by a field-list property.
+   - `location` columns are objects `{lng, lat, full_address}` — mapping the whole column locks
+     an `object` property type. Usually only `full_address` (a plain string) is meaningful;
+     confirm whether to flatten it to that single string (or exclude) rather than default to
+     `object`.
+   - an empty multi-select is `null` (verified), so `inspect` counts those rows as missing — a
+     null multi-select is "no selection", not data loss.
+   - `url`-styled text fields come back as markdown links (`[display](url)`), not raw URLs —
+     strip the wrapper (or re-key from a plain text field) before mapping to a URL property.
+   - formula fields are always strings in the snapshot, even when the expression is numeric
+     (e.g. `"20"`) — inspect infers `string`; do not expect a number column.
+   - lookup value shape depends on the aggregate, the target field type, and the match count —
+     a lookup to a person field returns `{id, name}` objects, to a select field returns option
+     strings (possibly a list), to a scalar text/number field returns a scalar string or list —
+     so confirm the target field type per-table before mapping.
+   - attachment fields are kept as an all-null column when empty; downloading attachments
+     (`base +record-download-attachment`) is out of scope for ingestion.
+4. If pagination fails on a network error or times out mid-loop, treat the snapshot as unknown:
+   do not write or keep a partial file; stop, and ask the user to confirm the base state before
+   retrying — the same rule as a lost upload batch.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the snapshot as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '.ae-cli/data-integration/online/<fingerprint>/snapshot.jsonl'
+```
+
+The zip step prefixes each record with reserved `__record_id` — a stable per-record key from the
+Bitable API, useful as a dedup key downstream. A source `record_id` column remains intact; a source
+`__record_id` column blocks snapshot creation as a reserved-name collision. Expect the Step 2
+inference notes to show up here: single-select columns surface as `list`, empty multi-selects as
+missing rows. Everything downstream — business identification, tracking plan, transform, sink —
+is unchanged. The snapshot lives under
+`.ae-cli/data-integration/`, so [reuse](reuse.md) and [handoff](handoff.md) find it like any other
+pipeline artifact; record in the handoff that the source was an online Feishu Bitable, the base
+and table, and include the snapshot timestamp and base timezone.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Online-source problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| `lark-cli` not installed | Not a parse error, not a data problem | Stop; give the install command |
+| lark-cli not logged in / no user identity | Not a parse error | Split-flow authorization with the user |
+| Missing scope (`permission_violations`) | Not a parse error | Incremental `auth login --scope` (user) or developer console (bot) |
+| Base/table not accessible to the user's account | Not a scope problem | Ask the user to verify the link and their access |
+| `+record-list` network failure / timeout mid-pagination | Same rule as a lost upload batch | Treat the snapshot as unknown; stop and confirm |

package/skills/ae-data-integration/references/lark-sheet-source.md

@@ -0,0 +1,130 @@
+# Source — online document (Feishu sheet)
+
+## When this branch applies
+
+The input is a **URL**, not a local path. Feishu sheet links (`/sheets/`), spreadsheet links
+(`/spreadsheets/`), and wiki links that resolve to a sheet (`/wiki/`) all route here instead of
+[source-inspect.md](source-inspect.md). The pipeline is identical after the sheet is snapshotted
+to a local XLSX: business identification, tracking plan, transform, and sink all reuse the
+local-file flow unchanged.
+
+Only **Feishu sheets** and **Feishu Bitable (多维表格)** are supported in this phase. Bitable links
+(`/base/`) — including wiki links that resolve to a Bitable — route to
+[lark-bitable-source.md](lark-bitable-source.md) instead of this page. Other online documents
+(Tencent Docs, DingTalk, WPS, Google Sheets, Notion) have no CLI read path — route to
+[manual-export-source.md](manual-export-source.md), which has the user export the file themselves
+and then rejoins the local-file flow.
+
+## Safety rules (online-specific)
+
+- The URL, its token, the snapshot file path, and every cell value are sensitive. Never print
+  source values while inspecting; summarize types, ratios, counts, and fingerprints only.
+- Pulling a sheet to the local machine moves data across a trust boundary. Before downloading
+  anything, tell the user the sheet will be snapshotted to a local file, and get their
+  confirmation when the sheet is not their own data.
+- The snapshot is a point-in-time copy. State the export timestamp, and note that the import
+  reflects the exported moment only — a sheet that may be edited mid-run is imported as of the
+  snapshot, not live.
+- lark-cli (Feishu) and ae-cli (AE) are two separate credential domains. Read Feishu with
+  lark-cli; upload to AE with ae-cli. Never forward a token from one to the other, and never
+  send an AE access token or CLI token to any Feishu endpoint.
+
+## Step 0 — Environment self-check (before any pull)
+
+The Feishu read path depends on `lark-cli`, a CLI separate from ae-cli. Check it up front so a
+missing install or a missing login surfaces as an environment problem, not as a data problem
+mid-pipeline.
+
+1. **Detect lark-cli.** Run `lark-cli --version`. If the command is not found, stop and tell the
+   user to install it (`npm i -g @larksuite/cli`). Do not attempt to install it yourself, and do
+   not fall back to another read path — a half-provisioned Feishu client produces worse failures
+   than a clear "not installed" stop.
+2. **Check the login.** Run
+   `LARKSUITE_CLI_NO_UPDATE_NOTIFIER=1 LARKSUITE_CLI_NO_SKILLS_NOTIFIER=1 lark-cli auth status --json --verify`.
+   Reading a sheet in the user's cloud space needs the **user** identity (`--as user`); the bot
+   identity cannot see a user's drive documents. If the user identity is unavailable or
+   unverified, follow the lark-shared skill's split-flow: start
+   `lark-cli auth login --scope "<scope>" --no-wait --json`, show the QR code / link to the user,
+   and only after the user confirms run `lark-cli auth login --device-code <device_code>`.
+   Authorization requires the user's participation — never bypass or automate it.
+3. **Handle permission errors when they occur.** If a pull returns `permission_violations`
+   (missing scopes) together with `console_url` and `hint`, route by identity:
+   - user identity → add the missing scope with
+     `lark-cli auth login --scope "<missing_scope>"` (incremental grant);
+   - bot identity → direct the user to open the scope in the developer console via the returned
+     `console_url`; never run `auth login` for a bot.
+   If the sheet itself is not accessible to the user's account — a resource-side permission
+   problem, not a scope problem — ask the user to confirm the link is correct and that their
+   Feishu account can open the sheet. Do not retry against a different sheet silently.
+
+## Step 1 — Locate the workbook and sheets
+
+1. Run `lark-cli sheets +workbook-info --url "<url>"` to list the sheets (`sheet_id`, `title`,
+   `row_count`, `column_count`, `is_hidden`). Never guess a sheet id or the name `Sheet1`.
+2. Surface the sheet list to the user. A workbook with one visible sheet proceeds with it;
+   multiple sheets ask the user to choose, mirroring `--data-set` on a local workbook. Report any
+   hidden sheet (`is_hidden`) by name — it is usually scratch space or a superseded draft, and
+   the same "ask before reading a hidden sheet" rule as a local XLSX applies. (`inspect` already
+   excludes hidden sheets from discovery, listing them under `excluded_sheets` with reason
+   `hidden` — name them anyway rather than relying on silent exclusion.) Show each sheet's row
+   count and, when available, its time coverage so an overlapping or derived sheet is visible
+   before it is chosen.
+
+## Step 2 — Snapshot to a local XLSX
+
+Export the workbook to an XLSX under the pipeline's working directory (a cwd-relative path, so
+lark-cli's file-path rules accept it):
+
+```bash
+lark-cli sheets +workbook-export --url "<url>" --output-path ".ae-cli/data-integration/online/<fingerprint>/snapshot.xlsx"
+```
+
+- **Export the whole workbook as XLSX**, not CSV: the XLSX export preserves dates, numbers,
+  merged cells, and hidden rows, so the downstream XLSX reader (date-cell detection, merged
+  cells, hidden rows) and the type inference behave exactly as for a local file. One caveat
+  (verified): the export keeps a formula's text but **drops the cached computed result**, so a
+  formula cell reads as missing — `inspect` flags it with a warning naming the affected columns
+  ("a formula whose last computed result is not stored in the file"). Surface that warning when
+  the source sheet contains formulas; the skill never evaluates a formula or guesses its value.
+  Do **not** use `+csv-get` for the snapshot — its display values round long numbers to
+  scientific notation (`1.04E+14`) and flatten types, which would silently corrupt ID-like
+  columns and lock wrong property types in AE.
+- `<fingerprint>` is a short, collision-resistant name derived from the spreadsheet token plus a
+  timestamp (e.g. first 8 chars of the token hash + `YYYYMMDD-HHMMSS`). One online source gets
+  one directory, so a re-export never overwrites an unrelated source. Create the directory before
+  exporting.
+- The export is an async job with built-in polling. If it returns `timed_out=true`, resume with
+  its `next_command` rather than re-exporting blindly.
+- If the export fails on a network error or times out mid-job, treat the snapshot as unknown:
+  stop, and ask the user to confirm the sheet state before retrying — the same rule as a lost
+  upload batch. Never resume a half-written snapshot silently.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the snapshot as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '.ae-cli/data-integration/online/<fingerprint>/snapshot.xlsx'
+```
+
+When the exported workbook holds multiple sheets, pass `--data-set '<chosen-sheet>'` using the
+sheet the user picked in Step 1 — the downstream multi-sheet selection is answered by the same
+choice, not re-asked.
+
+Everything downstream — business identification, tracking plan, transform, sink — is unchanged.
+The snapshot lives under `.ae-cli/data-integration/`, so [reuse](reuse.md) and
+[handoff](handoff.md) find it like any other pipeline artifact; record in the handoff that the
+source was an online Feishu sheet and include the snapshot timestamp.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Online-source problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| `lark-cli` not installed | Not a parse error, not a data problem | Stop; give the install command |
+| lark-cli not logged in / no user identity | Not a parse error | Split-flow authorization with the user |
+| Missing scope (`permission_violations`) | Not a parse error | Incremental `auth login --scope` (user) or developer console (bot) |
+| Sheet not accessible to the user's account | Not a scope problem | Ask the user to verify the link and their access |
+| Export job timed out / network lost | Same rule as a lost upload batch | Treat the snapshot as unknown; stop and confirm |

package/skills/ae-data-integration/references/manual-export-source.md

@@ -0,0 +1,83 @@
+# Source — online document, manual export (Tencent Docs / DingTalk / WPS)
+
+## When this branch applies
+
+The input is a **URL** that points to an online document on a platform other than Feishu:
+Tencent Docs (`docs.qq.com`), DingTalk docs, WPS / Kingsoft Docs (`kdocs.cn`), Google Sheets,
+Notion, and the like. These platforms have **no CLI read path** in this phase — only Feishu has
+one, via lark-cli (see [lark-sheet-source.md](lark-sheet-source.md) and
+[lark-bitable-source.md](lark-bitable-source.md)). There is therefore no automatic snapshot here:
+the user exports the document to a local file themselves, and the pipeline rejoins at the
+local-file flow unchanged.
+
+## Safety rules
+
+- The URL and every exported cell value are sensitive. Never print source values while
+  inspecting; summarize types, ratios, counts, and fingerprints only.
+- The export is a point-in-time copy made by the user on the source platform. State the export
+  timestamp, and note that the import reflects the exported moment only — a document edited
+  mid-run is imported as of the export, not live.
+- The export is the user's own action on the source platform. Do not automate it, and do not ask
+  for that platform's credentials — there is no Feishu-style OAuth to run on this branch.
+
+## Step 1 — Recognize the platform and route
+
+Inspect the URL's host to confirm it is not Feishu. Feishu hosts (`feishu.cn`, `larksuite.com`)
+route to the two lark source pages instead of this one; any other online-document host
+(`docs.qq.com`, `kdocs.cn`, a DingTalk docs host, etc.) lands here. If the host is ambiguous,
+ask the user which platform the link belongs to rather than guessing.
+
+## Step 2 — Guide the user to export
+
+Ask the user to export the document from the source platform's own UI to a local file, and to
+name the file so its origin is traceable. Give only the general route — the exact menu labels
+differ by platform and change over time, so follow the platform's current export UI rather than a
+memorized click path:
+
+- open the document, then use the platform's export / save-as / download action;
+- choose **XLSX** (Excel workbook) where offered, not CSV — CSV flattens types, may round long
+  ID-like numbers to scientific notation, and loses multiple sheets. This is the same reason the
+  Feishu sheet flow snapshots XLSX, not CSV (see
+  [lark-sheet-source.md](lark-sheet-source.md));
+- if the platform only offers CSV, accept it but flag the type-flattening risk to the user, and
+  watch for ID-like columns that must stay text;
+- ask the user to place the file under the pipeline's working directory (e.g.
+  `.ae-cli/data-integration/manual/<fingerprint>/<name>.xlsx`) so [reuse](reuse.md) and
+  [handoff](handoff.md) find it like any other pipeline artifact.
+
+Export-time caveats to surface to the user — platform-dependent, and the same rules as a local
+file:
+
+- **multiple sheets** — the export may contain several sheets; confirm which one is the data set,
+  mirroring `--data-set` on a local workbook.
+- **formulas** — an exported formula cell may carry only its text, not the cached computed result;
+  `inspect` flags such cells as missing. The skill never evaluates a formula or guesses its value.
+- **hidden rows/columns/sheets** — the platform may drop or keep hidden content depending on its
+  export; `inspect` excludes what it can detect (hidden sheets appear under `excluded_sheets`),
+  but ask the user to confirm the export contains what they expect before profiling.
+- **dates / encoding** — prefer a UTF-8 export; if dates arrive as text, say so rather than
+  silently coercing them.
+
+## Step 3 — Rejoin the local-file pipeline
+
+Feed the exported file as a local file and continue at [source-inspect.md](source-inspect.md):
+
+```bash
+ae-cli data-integration inspect --input-file '<exported-file>'
+```
+
+When the export holds multiple sheets, pass `--data-set '<chosen-sheet>'` using the sheet the
+user picked in Step 2. Everything downstream — business identification, tracking plan, transform,
+sink — is unchanged. Record in the handoff that the source was a manually-exported online
+document, name the platform and the export timestamp, and include the exported file path.
+
+## Error classification
+
+The three pipeline classes in [error-handling.md](error-handling.md) are about rows, files, and
+programs. Manual-export problems land outside those three and must not be mislabeled as them:
+
+| Problem | Class it is NOT | What to do |
+| --- | --- | --- |
+| Platform only exports CSV; types flatten | Not a parse error | Flag the risk; watch ID-like columns |
+| Export is missing sheets / rows the user expects | Not a data problem yet | Ask the user to re-export or confirm the data set |
+| URL host is unrecognized and not Feishu | Not a routing failure | Ask the user which platform the link belongs to |

package/skills/ae-data-integration/references/reuse.md

@@ -37,6 +37,8 @@ node .ae-cli/data-integration/<fingerprint[:16]>/transform.mjs <new-input-file>
 
 Do **not** run the returned command on your own. Show the user the proposed package — the frozen event name, the property mapping it implies, and the fact that the confirmed business logic (event name, `value_mapping`, flatten rules) is reused unchanged — and wait for one explicit confirmation. Before confirming, diff the new file's flattened properties against the package's `plan.json`: if the package has no `plan_file`, or the new file produces properties the plan does not cover, run the Tracking plan step to add them (merge into the existing project plan) first. Only then run `transform.mjs` on the new file and continue to Sink.
 
+The frozen mapping's `#type`/`mode` (e.g. `track`) is authoritative after a match — ignore `inspect`'s auto recommendation, which may re-infer `user_set` for a package whose confirmed mapping is `track`. A reuse match must not re-do Source identification or event-model inference.
+
 ## Safety rules
 
 - Reuse only for a file of the **same shape** (same header/schema and format). The `transform.mjs` wrapper re-binds the content fingerprint to the new file, so the content guard still applies per run.

package/skills/ae-data-integration/references/sink-upload.md

@@ -31,10 +31,42 @@ ae-cli data-integration upload \
   --dry-run
 ```
 
+Before the confirmation gate, re-check the project's **runtime-locked** property types. A `reuse` match skips the tracking-plan step, so this check is the one that must not be bypassed on any path — repeat it for every upload:
+
+```bash
+ae-cli analysis-meta property list -p <project-id> --table-type user
+ae-cli analysis-meta property list -p <project-id> --table-type event
+```
+
+Diff each target property name against `select_type` (the locked type). A same-name property whose locked type disagrees with the mapping's type drops those values silently at ingest, even when the plan layer agrees — surface any such conflict in the confirmation gate (rename the target, or change the mapped type to match the locked one) before asking the user to confirm. This is a read-only check; do not run `ae-cli tracking plan sync-from-meta` here, which writes the plan.
+
 Show the masked target, project, file fingerprint, record count, quarantined count, batch count, and persistence limitation. Re-state the system-field mapping first (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#event_name`), then re-list the final property mapping for every file or sheet being uploaded — source column → target AE name → type (+ `display_name`/`desc` when set), grouped into event properties (`track`) and user properties (profile modes), with each event's attached properties listed — never a counts-only summary. For a multi-sheet workbook, group by sheet. Wait for explicit confirmation. Execute the same command without `--dry-run` only after confirmation. For a blocked manifest, first show the quarantine statistics and separately ask whether the user accepts uploading only valid rows; add `--allow-clean-subset` only after a clear yes.
 
 `status=receiver_accepted` means receiver acceptance only, not durable storage. Say that persistence remains unverified. Never report success on this status alone.
 
+`upload` is **not idempotent** — every execution submits to the receiver again. Never re-run the
+command to inspect a response: read the first run's saved JSON output instead. `--dry-run` only
+previews the request it would send; it neither executes nor previews the receiver's real response.
+
+The execute success response reports the receiver outcome under **different field names than the
+dry-run preview** — do not reuse the dry-run names (`record_count` / `batch_count`) when reading
+it:
+
+| field | meaning |
+| --- | --- |
+| `status` | `receiver_accepted` (or `partially_delivered` when `--retry` salvaged some records) |
+| `delivery_state` | `receiver_accepted` / `partially_delivered` |
+| `submitted_records` | records submitted this run (equals `record_count - resume_from` on a clean run) |
+| `submitted_batches` | batches sent this run |
+| `resume_from` | zero-based offset this run started at |
+| `request_bytes` / `response_bytes` | transport byte counts |
+| `persistence_verified` | always `false` — receiver acceptance is not durable storage |
+| `next_step` | the verification guidance |
+
+`--retry` salvage additionally reports `failed_records` / `failed_count`. A lost batch is reported
+as `delivery_state=unknown` in the error's `meta`, not in the success fields — see
+[sync-json-upload.md](sync-json-upload.md) "Interrupted delivery".
+
 ## Verify the data landed
 
 Ingestion is asynchronous — expect about a 1-minute delay between `receiver_accepted` and the data being queryable. Instead of telling the user to check in the console, offer to verify with ae-cli (control-plane commands; require the CLI to be logged in to the AE host and a project-id):

package/skills/ae-data-integration/references/tracking-plan.md

@@ -12,7 +12,10 @@ Generate and confirm the event/property plan **before** any transform or upload.
 1. **Event-model decision** — reuse the UE routing result: single-table single-event `track`, single-table multi-event (event-name column), `user_set`, or `mixed`. The agent may propose splitting one table into several events (for example an ad table into `ad_show`/`ad_click` by `campaign_type`); that proposal must be confirmed by the user in the confirmation gate.
 2. **Column → property draft** — confirm the recommended mapping's key system fields with the user **before** drafting: `mode` (`#type`), `#account_id`/`#distinct_id` (ask together; at least one is required — a `user_id` column can be either an anonymous or a login ID and only the user knows), `#time` + source timezone + `#zone_offset`, `#event_name` (track only; the event column or a reviewed `default_event_name`), and `#ip`/`#uuid` when the data has such a column. The exact questions and the never-infer-from-a-column-name-alone rule are [transform.md](transform.md) steps 1–5; run them here. The plan is generated from this mapping, so never draft from an unconfirmed mapping. With the system fields settled, map the remaining columns to event and/or user properties (the mapping `mode` decides). Common event properties — project-level super properties attached to every event — are defined by `ae-generate-tracking-plan`, not this import path. Name events and properties in snake_case by default (keep an uppercase event name only when the user asks to preserve it; property names stay lowercase-only) and fill **every** `display_name`, `desc`, and `event_tag` (events also carry `event_desc`). Infer all three from field names, value distribution, samples, and business-doc / prompt priors — never leave them empty: `desc`/`event_desc` state what the item means in plain language (language follows the user), and `event_tag` picks the closest category from the canonical tag list (see the `event_tag` appendix in `../../ae-generate-tracking-plan/references/business-dimension-mapping.md`). When you cannot infer a `desc` or `event_tag`, mark it pending and ask the user for it inside the confirmation gate. Infer types (`number` / `bool` / `datetime` / enum) the same way; CSV defaults to `string`. Columns that stay uncertain or conflicting are marked pending and asked only inside the confirmation gate.
 3. **Confirmation gate (single, one pass)** — present the concrete plan, never a counts-only summary: the confirmed key system-field mapping (`mode`, `#account_id`/`#distinct_id`, `#time` + source timezone, `#event_name`; `#ip`/`#uuid` when present), then a full event table (one row per event: `event_name` + `event_tag`/`event_desc` + the properties attached to it), then a full property table (one row per property: source column → target AE name → type → `display_name`/`desc`, uncertain types highlighted; a kept-whole `object`/`array_row` lists its `parent.child` sub-properties next to the parent), plus field scope (default: plan fields only, with a full-import switch) and unrecognized/dirty-data handling. For a multi-sheet workbook, group the property table by sheet so each sheet's source columns are visible. The user answers once with ok or edits (renames, types, identity/time/event columns, add/drop columns).
-4. **Merge with the existing plan** — fetch the project's current tracking plan; same-name property type conflicts are severe, same-name events are advisory; decide append vs replace. This runs for **every** file, not just the first: when the project already has a plan (an earlier file or run), diff this file's events and properties against it and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user.
+4. **Merge with the existing plan — and the project's runtime property metadata.** This runs for **every** file, not just the first, and it checks **two layers**, which are separate stores:
+   - **Plan layer (declarative)** — `ae-cli tracking plan fetch -p <project-id>`: same-name property type conflicts are severe, same-name events are advisory; decide append vs replace.
+   - **Runtime metadata layer (authoritative)** — `ae-cli analysis-meta property list -p <project-id> --table-type user` and `--table-type event`: each property's **locked** type is in `select_type`, and it is what actually gates ingestion. A same-name property whose `select_type` disagrees with the mapping's intended type is a **severe** conflict: those values are dropped silently at ingest even when the plan layer agrees with the mapping. Diff every target property name against this list, translating the mapping type to AE's runtime naming before comparing (`boolean` → `bool`, `list` → `array_string`), and put any mismatch in the confirmation gate (rename the target, or change the mapped type to match the locked one).
+   Diff this file's events and properties against both layers and put the additions — new events, new properties, new object sub-properties from flattening — in the confirmation gate. An existing plan is never a reason to skip this step; only when every addition is already present may you skip the merge, and even then state and confirm that fact with the user. Do not run `ae-cli tracking plan sync-from-meta` for this check — it is a write (it re-syncs the plan from metadata), not a read-only pre-flight.
 5. **Persist the plan** — `.ae-cli/data-integration/draft.json` → `.ae-cli/data-integration/draft.xlsx` → upload with `sdk_integration_mode=none`.
 6. **Hand off the field mapping** — the confirmed plan plus column→property mapping, `value_mapping`, and `flatten_rules` feed the Transform submodule.
 

package/skills/ae-data-integration/references/ue-mapping.md

@@ -74,7 +74,7 @@ Each mapped system field carries a value spec, enforced at both inspect (warning
 
 Event-name case is decided entirely by the mapping, never by the CLI — the CLI performs no case conversion and has no flag for it. The name AE receives is the source `event_name_field` value (or `default_event_name` when there is no event column), optionally replaced by a `value_mapping.event_name` target; `data-integration plan --event-name` passes each name through unchanged. Lowercase is the default: lowercase the name when writing the mapping (a `value_mapping.event_name` entry such as `Purchase` → `purchase`, or a lowercase `default_event_name`). To keep uppercase, skip that lowercasing: leave a legal uppercase source value unmapped so it passes through, write an uppercase `value_mapping.event_name` target only when the source value is not itself a legal name (e.g. `购买` → `Purchase`), or write an uppercase `default_event_name` / `--event-name`.
 
-A field skip is not a row failure: the row is kept with its other fields, and the count is reported in `manifest.output.skipped_fields` so the agent tells the user at the end. `#uuid` is never auto-generated — it comes only from a mapped source column.
+A field skip is not a row failure: the row is kept with its other fields, and the count is reported in `manifest.output.skipped_fields` so the agent tells the user at the end. `#uuid` is never auto-generated — it comes only from a mapped source column. `#uuid` accepts only a valid 36-character UUID; any other value (a business order/event id, a numeric id) is skipped from that row (`skipped_fields`), never emitted. So propose `uuid_field` only when the source has a column that already holds valid UUIDs — a `#uuid` gives the receiver a dedup key, so a re-submitted record is de-duplicated instead of creating a true duplicate, while without `#uuid` every re-submission appends a real duplicate. Never fabricate UUIDs, and never map a non-UUID unique key to `uuid_field`: the CLI validates the format but does not convert, synthesize, or de-duplicate values.
 
 Mapping keys: `account_id_field`, `distinct_id_field`, `time.field`, `event_name_field` (or a reviewed `default_event_name`), `ip_field`, `uuid_field`. Inspect surfaces every identity-shaped column in `identity_candidates` (name, `account`/`distinct` kind, unique and missing ratios) so the agent can present all candidates for confirmation. When the required identity column is absent, use an explicit `account_id_value`/`distinct_id_value` placeholder or `random_pool` — a user decision, never invented.
 
@@ -138,7 +138,7 @@ Values with an explicit offset or `Z` are parsed by the JavaScript `Date` constr
 ## Data rules
 
 - **Ingestion time window.** The receiver accepts event/profile times from 3 years before to 3 days after the server time; the CLI enforces this range. Client-side reporting has a tighter window (10 days before to 3 days after); historical data beyond the window needs AE support to extend it.
-- **Property type locking.** A property's type is locked on first receipt, and properties sharing a name across events are one property — later reports must use the same type. Values whose type mismatches the locked type are dropped silently, so review an inferred type before committing it.
+- **Property type locking.** A property's type is locked on first receipt, and properties sharing a name across events are one property — later reports must use the same type. Values whose type mismatches the locked type are dropped silently, so review an inferred type before committing it — and before that, check the project's **already-locked** types with `ae-cli analysis-meta property list -p <project-id> --table-type user` / `--table-type event`, diffing each target property name against `select_type` (see tracking-plan.md step 4). The plan layer may still say `string` while the runtime layer has locked `number`; only the runtime `select_type` gates ingestion.
 - **Quote stripping.** Paired single or double quotes around a cell value are stripped (`'user_001'` → `user_001`); a single-sided quote is kept as data (SQL exports often quote every value).
 - **Column name sanitization.** Recommended mapping auto-names properties: accents are stripped, camelCase is split to snake_case, everything is lowercased, and illegal characters become `_`; a digit-leading name is prefixed `field_`, and a name with nothing recognizable falls back to `field_N`. Review these targets — especially the `field_N` fallbacks — and rename before converting.
 - **Object sub-property keys.** Object and list-of-object keys follow the same naming rules as property names (lowercase snake_case, letter-leading, at most 50 chars, no `#`). A key that does not — for example a Chinese key inside a JSON cell such as `{"等级":"金牌"}` — quarantines the whole row, so confirm object keys before converting.