sfmc-dataloader 2.1.0 → 2.3.0

package/README.md

@@ -1,6 +1,6 @@
 # sfmc-dataloader
 
-Command-line tool **`mcdata`** to export and import Salesforce Marketing Cloud Data Extension rows using the same project files as [mcdev](https://github.com/Accenture/sfmc-devtools) (`.mcdevrc.json`, `.mcdev-auth.json`) and [sfmc-sdk](https://www.npmjs.com/package/sfmc-sdk) for REST/SOAP.
+Command-line tool **`mcdata`** to export and import **Salesforce Marketing Cloud Engagement** Data Extension rows using the same project files as [mcdev](https://github.com/Accenture/sfmc-devtools) (`.mcdevrc.json`, `.mcdev-auth.json`) and [sfmc-sdk](https://www.npmjs.com/package/sfmc-sdk) for REST/SOAP.
 
 ## Requirements
 
@@ -41,12 +41,12 @@ Creates one file per BU/DE combination using the same `.mcdata.` naming rules.
 ### Import — single BU
 
 ```bash
-mcdata import MyCred/MyBU --de MyDE_CustomerKey --format csv --mode upsert
+mcdata import MyCred/MyBU --de MyDE_CustomerKey --mode upsert
 ```
 
 Imports use the **asynchronous** bulk row API only: `POST` for `--mode insert`, `PUT` for `--mode upsert` (same endpoint path).
 
-Resolves the latest matching export file under `./data/MyCred/MyBU/` for that DE key.
+Resolves the latest matching export file under `./data/MyCred/MyBU/` for that DE key. The file format is detected automatically from the file extension (`.csv`, `.tsv`, `.json`).
 
 Import from explicit paths (the DE key is taken from the `.mcdata.` basename):
 
@@ -56,8 +56,8 @@ mcdata import MyCred/MyBU --file ./data/MyCred/MyBU/encoded%2Bkey.mcdata.2026-04
 
 #### Upsert vs insert
 
-- **Upsert** follows the platform’s usual behaviour: update when a primary key matches, otherwise insert. For Data Extensions **without** a primary key, upsert may not behave as expected; prefer **`--mode insert`** for those.
-- **Insert** always adds new rows. Running import twice with insert can create **duplicate** rows if the same file is applied again—use upsert when keys are defined and you need idempotent runs.
+- **Upsert** follows the platform's usual behaviour: update when a primary key matches, otherwise insert. For Data Extensions **without** a primary key, upsert **will fail**; use **`--mode insert`** for those.
+- **Insert** always adds new rows. Running import twice with insert can create **duplicate** rows if the same file is applied again—use upsert when primary keys are defined and you need to ensure repeated runs always have the same outcome.
 
 ### Import — one source BU into multiple target BUs (API mode)
 
@@ -67,7 +67,7 @@ Use `--from` (one source) and `--to` (repeatable targets) for a cross-BU import
 mcdata import --from MyCred/Dev --to MyCred/QA --to MyCred/Prod --de Contact_DE
 ```
 
-Before the import starts you will be offered the option to export the current data from each target BU as a timestamped backup. A download file is also written to each target BU's data directory so there is a traceable record of exactly what was imported.
+An optional pre-import backup exports current target BU data as **timestamped** files (backup filenames always include the timestamp, regardless of `--git`). Use `--backup-before-import` to run the backup without a prompt (CI-safe), or `--no-backup-before-import` to skip it entirely. When neither flag is provided the CLI prompts interactively (TTY only). A snapshot file is also written to each target BU's data directory as a record of what was imported.
 
 ### Import — local export files into multiple target BUs (file mode)
 
@@ -80,6 +80,20 @@ mcdata import --to MyCred/QA --to MyCred/Prod \
 
 Multiple files can be supplied to push several DEs in one command. Pass **`--git`** on export if you rely on stable `*.mcdata.<ext>` names for snapshots in this flow.
 
+### Backup target DE before import
+
+Use `--backup-before-import` on any import command (single-BU or cross-BU) to export a timestamped snapshot of the current target DE rows before the import runs. The backup filename always includes a timestamp regardless of whether `--git` is set.
+
+```bash
+mcdata import MyCred/MyBU --de MyKey --backup-before-import
+```
+
+In CI, combine with `--no-backup-before-import` to suppress any TTY prompt:
+
+```bash
+mcdata import MyCred/MyBU --de MyKey --no-backup-before-import
+```
+
 ### Clear all rows before import
 
 **Dangerous:** removes every row in the target Data Extension(s) before uploading.
@@ -101,16 +115,18 @@ Interactive: type `YES` when prompted. In CI, add `--i-accept-clear-data-risk` a
 
 ## Options
 
-| Option                       | Description                                                                                                           |
-| ---------------------------- | --------------------------------------------------------------------------------------------------------------------- |
-| `-p, --project`              | Project root (default: cwd)                                                                                           |
-| `--format`                   | `csv` (default), `tsv`, or `json`                                                                                     |
-| `--git`                      | Stable export filenames: `<key>.mcdata.<ext>` (no timestamp segment)                                                  |
-| `--mode`                     | `upsert` (default) or `insert` — async bulk REST API only                                                             |
-| `--from <cred>/<bu>`         | Export: source BU (repeatable). Import API mode: single source BU (use with `--to` and `--de`)                        |
-| `--to <cred>/<bu>`           | Import: target BU (repeatable). API mode: use with `--from`/`--de`. File mode: use with `--file` (no `--from` needed) |
-| `--clear-before-import`      | SOAP `ClearData` before REST import                                                                                   |
-| `--i-accept-clear-data-risk` | Non-interactive consent for clear                                                                                     |
+| Option                        | Description                                                                                                           |
+| ----------------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| `-p, --project`               | Project root (default: cwd)                                                                                           |
+| `--format`                    | `csv` (default), `tsv`, or `json` — **export only**; import format is detected from the file extension               |
+| `--git`                       | Stable export filenames: `<key>.mcdata.<ext>` (no timestamp segment)                                                  |
+| `--mode`                      | `upsert` (default) or `insert` — async bulk REST API only                                                             |
+| `--from <cred>/<bu>`          | Export: source BU (repeatable). Import API mode: single source BU (use with `--to` and `--de`)                        |
+| `--to <cred>/<bu>`            | Import: target BU (repeatable). API mode: use with `--from`/`--de`. File mode: use with `--file` (no `--from` needed) |
+| `--backup-before-import`      | Export target DE data as a timestamped backup before import (no prompt; always timestamped)                           |
+| `--no-backup-before-import`   | Skip the backup prompt even in interactive (TTY) sessions                                                             |
+| `--clear-before-import`       | SOAP `ClearData` before REST import                                                                                   |
+| `--i-accept-clear-data-risk`  | Non-interactive consent for clear                                                                                     |
 
 Log lines use paths **relative** to the project root (POSIX-style, `./…`) and include **row counts** where applicable.
 

package/lib/cli.mjs

@@ -18,6 +18,7 @@ import { parseExportBasename } from './filename.mjs';
 import { importFromFile } from './import-de.mjs';
 import { clearDataExtensionRows } from './clear-de.mjs';
 import { confirmClearBeforeImport } from './confirm-clear.mjs';
+import { getDeRowCount } from './row-count.mjs';
 import { multiBuExport } from './multi-bu-export.mjs';
 import { crossBuImport } from './cross-bu-import.mjs';
 import { initDebugLogger } from './debug-logger.mjs';
@@ -53,6 +54,8 @@ Options:
 
 Import options:
   --mode <upsert|insert>  Row write mode (default: upsert; async REST bulk API)
+  --backup-before-import  Export target DE data as a timestamped backup before import (no prompt)
+  --no-backup-before-import  Skip the backup prompt even in interactive (TTY) sessions
   --clear-before-import   SOAP ClearData before import (destructive; see below)
   --i-accept-clear-data-risk  Non-interactive acknowledgement for --clear-before-import
 
@@ -96,6 +99,8 @@ export async function main(argv) {
                 to: { type: 'string', multiple: true },
                 git: { type: 'boolean', default: false },
                 mode: { type: 'string' },
+                'backup-before-import': { type: 'boolean', default: false },
+                'no-backup-before-import': { type: 'boolean', default: false },
                 'clear-before-import': { type: 'boolean', default: false },
                 'i-accept-clear-data-risk': { type: 'boolean', default: false },
                 'json-pretty': { type: 'boolean', default: false },
@@ -135,6 +140,12 @@ export async function main(argv) {
         return 1;
     }
 
+    const backupBeforeImport = values['backup-before-import']
+        ? true
+        : values['no-backup-before-import']
+          ? false
+          : undefined;
+
     const useGit = values.git === true;
     const fromFlags = values.from ?? [];
     const toFlags = values.to ?? [];
@@ -264,6 +275,7 @@ export async function main(argv) {
                 targets,
                 format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
                 mode: /** @type {'upsert'|'insert'} */ (mode),
+                backupBeforeImport,
                 clearBeforeImport: clear,
                 acceptRiskFlag: acceptRisk,
                 isTTY: process.stdin.isTTY === true,
@@ -330,6 +342,7 @@ export async function main(argv) {
                 deKeys,
                 format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
                 mode: /** @type {'upsert'|'insert'} */ (mode),
+                backupBeforeImport,
                 clearBeforeImport: clear,
                 acceptRiskFlag: acceptRisk,
                 isTTY: process.stdin.isTTY === true,
@@ -365,15 +378,27 @@ export async function main(argv) {
         if (hasDe) {
             const deKeys = [values.de ?? []].flat();
             const dataDir = dataDirectoryForBu(projectRoot, credential, bu);
+            if (backupBeforeImport === true) {
+                for (const deKey of deKeys) {
+                    const { path: outPath, rowCount } = await exportDataExtensionToFile(sdk, {
+                        projectRoot,
+                        credentialName: credential,
+                        buName: bu,
+                        deKey,
+                        format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
+                        useGit: false,
+                    });
+                    console.error(
+                        `Backup export: ${projectRelativePosix(projectRoot, outPath)} (${rowCount} rows)`,
+                    );
+                }
+            }
             if (clear) {
                 await confirmClearBeforeImport({
                     deKeys,
                     acceptRiskFlag: acceptRisk,
                     isTTY: process.stdin.isTTY === true,
                 });
-                for (const deKey of deKeys) {
-                    await clearDataExtensionRows(sdk.soap, deKey);
-                }
             }
             for (const deKey of deKeys) {
                 const candidates = await findImportCandidates(dataDir, deKey);
@@ -385,6 +410,23 @@ export async function main(argv) {
                 }
                 const filePath =
                     candidates.length === 1 ? candidates[0] : await pickLatestByMtime(candidates);
+
+                const countBefore = await getDeRowCount(sdk, deKey);
+                console.error(
+                    `Row count before import: ${countBefore ?? '(unavailable)'} (DE "${deKey}")`,
+                );
+
+                if (clear) {
+                    if (countBefore === 0) {
+                        console.error(
+                            `Skipping clear-data for DE "${deKey}" — DE is already empty.`,
+                        );
+                    } else {
+                        await clearDataExtensionRows(sdk.soap, deKey);
+                        console.warn(`Cleared data: DE "${deKey}"`);
+                    }
+                }
+
                 const n = await importFromFile(sdk, {
                     filePath,
                     deKey,
@@ -392,6 +434,23 @@ export async function main(argv) {
                 });
                 const rel = projectRelativePosix(projectRoot, filePath);
                 console.error(`Imported: ${rel} (${n} rows) -> DE ${deKey}`);
+
+                const countAfter = await getDeRowCount(sdk, deKey);
+                console.error(
+                    `Row count after import: ${countAfter ?? '(unavailable)'} (DE "${deKey}")`,
+                );
+                if (countAfter === null) {
+                    console.error(`Could not verify import result for DE "${deKey}".`);
+                } else if (countBefore !== null && countAfter < (countBefore ?? 0) + n) {
+                    // Insert: expect countBefore + n; upsert on empty: same; upsert on non-empty: expect >= n
+                    const expected =
+                        mode === 'insert' || countBefore === 0 ? (countBefore ?? 0) + n : n;
+                    if (countAfter < expected) {
+                        console.error(
+                            `Import result for DE "${deKey}" looks unexpected: expected at least ${expected} rows, got ${countAfter}. Note: the async API may not have committed all rows yet.`,
+                        );
+                    }
+                }
             }
             return 0;
         }
@@ -400,19 +459,48 @@ export async function main(argv) {
         const keysFromFiles = fileList.map(
             (fp) => parseExportBasename(path.basename(fp)).customerKey,
         );
+        if (backupBeforeImport === true) {
+            for (const deKey of keysFromFiles) {
+                const { path: outPath, rowCount } = await exportDataExtensionToFile(sdk, {
+                    projectRoot,
+                    credentialName: credential,
+                    buName: bu,
+                    deKey,
+                    format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
+                    useGit: false,
+                });
+                console.error(
+                    `Backup export: ${projectRelativePosix(projectRoot, outPath)} (${rowCount} rows)`,
+                );
+            }
+        }
         if (clear) {
             await confirmClearBeforeImport({
                 deKeys: keysFromFiles,
                 acceptRiskFlag: acceptRisk,
                 isTTY: process.stdin.isTTY === true,
             });
-            for (const deKey of keysFromFiles) {
-                await clearDataExtensionRows(sdk.soap, deKey);
-            }
         }
         for (const filePath of fileList) {
             const base = path.basename(filePath);
             const { customerKey } = parseExportBasename(base);
+
+            const countBefore = await getDeRowCount(sdk, customerKey);
+            console.error(
+                `Row count before import: ${countBefore ?? '(unavailable)'} (DE "${customerKey}")`,
+            );
+
+            if (clear) {
+                if (countBefore === 0) {
+                    console.error(
+                        `Skipping clear-data for DE "${customerKey}" — DE is already empty.`,
+                    );
+                } else {
+                    await clearDataExtensionRows(sdk.soap, customerKey);
+                    console.warn(`Cleared data: DE "${customerKey}"`);
+                }
+            }
+
             const n = await importFromFile(sdk, {
                 filePath,
                 deKey: customerKey,
@@ -420,6 +508,22 @@ export async function main(argv) {
             });
             const rel = projectRelativePosix(projectRoot, filePath);
             console.error(`Imported: ${rel} (${n} rows)`);
+
+            const countAfter = await getDeRowCount(sdk, customerKey);
+            console.error(
+                `Row count after import: ${countAfter ?? '(unavailable)'} (DE "${customerKey}")`,
+            );
+            if (countAfter === null) {
+                console.error(`Could not verify import result for DE "${customerKey}".`);
+            } else {
+                const expected =
+                    mode === 'insert' || countBefore === 0 ? (countBefore ?? 0) + n : n;
+                if (countAfter < expected) {
+                    console.error(
+                        `Import result for DE "${customerKey}" looks unexpected: expected at least ${expected} rows, got ${countAfter}. Note: the async API may not have committed all rows yet.`,
+                    );
+                }
+            }
         }
         return 0;
     }

package/lib/cross-bu-import.mjs

@@ -12,6 +12,7 @@ import { clearDataExtensionRows } from './clear-de.mjs';
 import { confirmClearBeforeImport } from './confirm-clear.mjs';
 import { dataDirectoryForBu, projectRelativePosix } from './paths.mjs';
 import { buildExportBasename, filesystemSafeTimestamp, parseExportBasename } from './filename.mjs';
+import { getDeRowCount } from './row-count.mjs';
 
 /**
  * @typedef {{ credential: string, bu: string }} CredBuTarget
@@ -73,10 +74,11 @@ async function offerPreExportBackup({ targets, deKeys, stdin: stdinStream, stdou
  * @param {CredBuTarget[]} params.targets
  * @param {'csv'|'tsv'|'json'} params.format
  * @param {'upsert'|'insert'} params.mode
+ * @param {boolean} [params.backupBeforeImport] - true=always backup, false=never backup, undefined=TTY prompt
  * @param {boolean} params.clearBeforeImport
  * @param {boolean} params.acceptRiskFlag
  * @param {boolean} params.isTTY
- * @param {boolean} [params.useGit] - stable snapshot basename (no timestamp)
+ * @param {boolean} [params.useGit] - accepted for API compatibility but ignored; snapshot files always use a timestamped name
  * @param {import('./config.mjs').DebugLogger|null} [params.logger] - debug logger for API requests/responses
  * @param {NodeJS.ReadableStream} [params.stdin] Override for testing
  * @param {NodeJS.WritableStream} [params.stdout] Override for testing
@@ -90,10 +92,10 @@ export async function crossBuImport(params) {
         targets,
         format,
         mode,
+        backupBeforeImport,
         clearBeforeImport,
         acceptRiskFlag,
         isTTY,
-        useGit = false,
         logger = null,
     } = params;
     const stdin = params.stdin;
@@ -135,29 +137,29 @@ export async function crossBuImport(params) {
     }
 
     // Optional pre-import backup of target BU data
-    if (isTTY) {
-        const doBackup = await offerPreExportBackup({ targets, deKeys, stdin, stdout });
-        if (doBackup) {
-            for (const { credential, bu } of targets) {
-                const { mid, authCred } = resolveCredentialAndMid(
-                    mcdevrc,
-                    mcdevAuth,
-                    credential,
-                    bu,
-                );
-                const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
-                for (const deKey of deKeys) {
-                    const { path: outPath, rowCount } = await exportDataExtensionToFile(tgtSdk, {
-                        projectRoot,
-                        credentialName: credential,
-                        buName: bu,
-                        deKey,
-                        format,
-                        useGit: false,
-                    });
-                    const rel = projectRelativePosix(projectRoot, outPath);
-                    console.error(`Backup export: ${rel} (${rowCount} rows)`);
-                }
+    const shouldBackup =
+        backupBeforeImport === true
+            ? true
+            : backupBeforeImport === false
+              ? false
+              : isTTY
+                ? await offerPreExportBackup({ targets, deKeys, stdin, stdout })
+                : false;
+    if (shouldBackup) {
+        for (const { credential, bu } of targets) {
+            const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
+            const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
+            for (const deKey of deKeys) {
+                const { path: outPath, rowCount } = await exportDataExtensionToFile(tgtSdk, {
+                    projectRoot,
+                    credentialName: credential,
+                    buName: bu,
+                    deKey,
+                    format,
+                    useGit: false,
+                });
+                const rel = projectRelativePosix(projectRoot, outPath);
+                console.error(`Backup export: ${rel} (${rowCount} rows)`);
             }
         }
     }
@@ -183,29 +185,32 @@ export async function crossBuImport(params) {
             rows = await fetchAllRowObjects(srcSdk, deKey);
         }
 
-        // Clear targets before import (rows already confirmed above)
-        if (clearBeforeImport) {
-            for (const { credential, bu } of targets) {
-                const { mid, authCred } = resolveCredentialAndMid(
-                    mcdevrc,
-                    mcdevAuth,
-                    credential,
-                    bu,
-                );
-                const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
-                await clearDataExtensionRows(tgtSdk.soap, deKey);
-            }
-        }
-
         for (const { credential, bu } of targets) {
             const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
             const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
 
+            const countBefore = await getDeRowCount(tgtSdk, deKey);
+            console.error(
+                `Row count before import: ${countBefore ?? '(unavailable)'} (${credential}/${bu} DE "${deKey}")`,
+            );
+
+            // Clear target before import (already confirmed above); skip if DE is empty
+            if (clearBeforeImport) {
+                if (countBefore === 0) {
+                    console.error(
+                        `Skipping clear-data for ${credential}/${bu} DE "${deKey}" — DE is already empty.`,
+                    );
+                } else {
+                    await clearDataExtensionRows(tgtSdk.soap, deKey);
+                    console.warn(`Cleared data: ${credential}/${bu} DE "${deKey}"`);
+                }
+            }
+
             // Write a snapshot file in the target BU's data directory.
             const dir = dataDirectoryForBu(projectRoot, credential, bu);
             await fs.mkdir(dir, { recursive: true });
             const ts = filesystemSafeTimestamp();
-            const basename = buildExportBasename(deKey, ts, format, useGit);
+            const basename = buildExportBasename(deKey, ts, format, false);
             const snapshotPath = path.join(dir, basename);
             await fs.writeFile(snapshotPath, serializeRows(rows, format, false), 'utf8');
             const snapRel = projectRelativePosix(projectRoot, snapshotPath);
@@ -213,6 +218,24 @@ export async function crossBuImport(params) {
 
             const imported = await importRowsForDe(tgtSdk, { deKey, rows, mode });
             console.error(`Imported: ${credential}/${bu} DE ${deKey} (${imported} rows)`);
+
+            const countAfter = await getDeRowCount(tgtSdk, deKey);
+            console.error(
+                `Row count after import: ${countAfter ?? '(unavailable)'} (${credential}/${bu} DE "${deKey}")`,
+            );
+            if (countAfter === null) {
+                console.error(
+                    `Could not verify import result for ${credential}/${bu} DE "${deKey}".`,
+                );
+            } else {
+                const expected =
+                    mode === 'insert' || countBefore === 0 ? (countBefore ?? 0) + imported : imported;
+                if (countAfter < expected) {
+                    console.error(
+                        `Import result for ${credential}/${bu} DE "${deKey}" looks unexpected: expected at least ${expected} rows, got ${countAfter}. Note: the async API may not have committed all rows yet.`,
+                    );
+                }
+            }
         }
     }
 }

package/lib/index.mjs

@@ -14,4 +14,5 @@ export {
 } from './config.mjs';
 export { multiBuExport } from './multi-bu-export.mjs';
 export { crossBuImport } from './cross-bu-import.mjs';
+export { getDeRowCount } from './row-count.mjs';
 export { projectRelativePosix } from './paths.mjs';

package/lib/row-count.mjs

@@ -0,0 +1,20 @@
+import { rowsetGetPath } from './import-routes.mjs';
+
+/**
+ * Fetch the total row count for a Data Extension without downloading all rows.
+ * Uses a pagesize=1 request so only the `count` metadata is transferred.
+ * Returns `null` if the count cannot be determined (e.g. permissions error).
+ *
+ * @param {{ rest: { get: (path: string) => Promise.<any> } }} sdk
+ * @param {string} deKey - DE external key
+ * @returns {Promise.<number|null>}
+ */
+export async function getDeRowCount(sdk, deKey) {
+    try {
+        const result = await sdk.rest.get(`${rowsetGetPath(deKey)}?$page=1&$pagesize=1`);
+        return result?.count ?? 0;
+    } catch (ex) {
+        console.error(`Could not retrieve row count for DE "${deKey}": ${ex.message}`);
+        return null;
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "sfmc-dataloader",
-    "version": "2.1.0",
+    "version": "2.3.0",
     "description": "SFMC Data Loader CLI (mcdata) to export and import Marketing Cloud Data Extension rows using mcdev project config and sfmc-sdk",
     "author": "Jörn Berkefeld <joern.berkefeld@gmail.com>",
     "license": "MIT",
@@ -44,7 +44,7 @@
         "sfmc-sdk": "3.0.3"
     },
     "scripts": {
-        "test": "node --test test/**/*.test.js",
+        "test": "node --test test/*.test.js",
         "lint": "eslint .",
         "lint:fix": "eslint --fix .",
         "format": "prettier --write .",