sfmc-dataloader 2.1.0 → 2.2.0

package/README.md

@@ -1,6 +1,6 @@
 # sfmc-dataloader
 
-Command-line tool **`mcdata`** to export and import Salesforce Marketing Cloud Data Extension rows using the same project files as [mcdev](https://github.com/Accenture/sfmc-devtools) (`.mcdevrc.json`, `.mcdev-auth.json`) and [sfmc-sdk](https://www.npmjs.com/package/sfmc-sdk) for REST/SOAP.
+Command-line tool **`mcdata`** to export and import **Salesforce Marketing Cloud Engagement** Data Extension rows using the same project files as [mcdev](https://github.com/Accenture/sfmc-devtools) (`.mcdevrc.json`, `.mcdev-auth.json`) and [sfmc-sdk](https://www.npmjs.com/package/sfmc-sdk) for REST/SOAP.
 
 ## Requirements
 
@@ -41,12 +41,12 @@ Creates one file per BU/DE combination using the same `.mcdata.` naming rules.
 ### Import — single BU
 
 ```bash
-mcdata import MyCred/MyBU --de MyDE_CustomerKey --format csv --mode upsert
+mcdata import MyCred/MyBU --de MyDE_CustomerKey --mode upsert
 ```
 
 Imports use the **asynchronous** bulk row API only: `POST` for `--mode insert`, `PUT` for `--mode upsert` (same endpoint path).
 
-Resolves the latest matching export file under `./data/MyCred/MyBU/` for that DE key.
+Resolves the latest matching export file under `./data/MyCred/MyBU/` for that DE key. The file format is detected automatically from the file extension (`.csv`, `.tsv`, `.json`).
 
 Import from explicit paths (the DE key is taken from the `.mcdata.` basename):
 
@@ -56,8 +56,8 @@ mcdata import MyCred/MyBU --file ./data/MyCred/MyBU/encoded%2Bkey.mcdata.2026-04
 
 #### Upsert vs insert
 
-- **Upsert** follows the platform’s usual behaviour: update when a primary key matches, otherwise insert. For Data Extensions **without** a primary key, upsert may not behave as expected; prefer **`--mode insert`** for those.
-- **Insert** always adds new rows. Running import twice with insert can create **duplicate** rows if the same file is applied again—use upsert when keys are defined and you need idempotent runs.
+- **Upsert** follows the platform's usual behaviour: update when a primary key matches, otherwise insert. For Data Extensions **without** a primary key, upsert **will fail**; use **`--mode insert`** for those.
+- **Insert** always adds new rows. Running import twice with insert can create **duplicate** rows if the same file is applied again—use upsert when primary keys are defined and you need to ensure repeated runs always have the same outcome.
 
 ### Import — one source BU into multiple target BUs (API mode)
 
@@ -67,7 +67,7 @@ Use `--from` (one source) and `--to` (repeatable targets) for a cross-BU import
 mcdata import --from MyCred/Dev --to MyCred/QA --to MyCred/Prod --de Contact_DE
 ```
 
-Before the import starts you will be offered the option to export the current data from each target BU as a timestamped backup. A download file is also written to each target BU's data directory so there is a traceable record of exactly what was imported.
+An optional pre-import backup exports current target BU data as **timestamped** files (backup filenames always include the timestamp, regardless of `--git`). Use `--backup-before-import` to run the backup without a prompt (CI-safe), or `--no-backup-before-import` to skip it entirely. When neither flag is provided the CLI prompts interactively (TTY only). A snapshot file is also written to each target BU's data directory as a record of what was imported.
 
 ### Import — local export files into multiple target BUs (file mode)
 
@@ -80,6 +80,20 @@ mcdata import --to MyCred/QA --to MyCred/Prod \
 
 Multiple files can be supplied to push several DEs in one command. Pass **`--git`** on export if you rely on stable `*.mcdata.<ext>` names for snapshots in this flow.
 
+### Backup target DE before import
+
+Use `--backup-before-import` on any import command (single-BU or cross-BU) to export a timestamped snapshot of the current target DE rows before the import runs. The backup filename always includes a timestamp regardless of whether `--git` is set.
+
+```bash
+mcdata import MyCred/MyBU --de MyKey --backup-before-import
+```
+
+In CI, combine with `--no-backup-before-import` to suppress any TTY prompt:
+
+```bash
+mcdata import MyCred/MyBU --de MyKey --no-backup-before-import
+```
+
 ### Clear all rows before import
 
 **Dangerous:** removes every row in the target Data Extension(s) before uploading.
@@ -101,16 +115,18 @@ Interactive: type `YES` when prompted. In CI, add `--i-accept-clear-data-risk` a
 
 ## Options
 
-| Option                       | Description                                                                                                           |
-| ---------------------------- | --------------------------------------------------------------------------------------------------------------------- |
-| `-p, --project`              | Project root (default: cwd)                                                                                           |
-| `--format`                   | `csv` (default), `tsv`, or `json`                                                                                     |
-| `--git`                      | Stable export filenames: `<key>.mcdata.<ext>` (no timestamp segment)                                                  |
-| `--mode`                     | `upsert` (default) or `insert` — async bulk REST API only                                                             |
-| `--from <cred>/<bu>`         | Export: source BU (repeatable). Import API mode: single source BU (use with `--to` and `--de`)                        |
-| `--to <cred>/<bu>`           | Import: target BU (repeatable). API mode: use with `--from`/`--de`. File mode: use with `--file` (no `--from` needed) |
-| `--clear-before-import`      | SOAP `ClearData` before REST import                                                                                   |
-| `--i-accept-clear-data-risk` | Non-interactive consent for clear                                                                                     |
+| Option                        | Description                                                                                                           |
+| ----------------------------- | --------------------------------------------------------------------------------------------------------------------- |
+| `-p, --project`               | Project root (default: cwd)                                                                                           |
+| `--format`                    | `csv` (default), `tsv`, or `json` — **export only**; import format is detected from the file extension               |
+| `--git`                       | Stable export filenames: `<key>.mcdata.<ext>` (no timestamp segment)                                                  |
+| `--mode`                      | `upsert` (default) or `insert` — async bulk REST API only                                                             |
+| `--from <cred>/<bu>`          | Export: source BU (repeatable). Import API mode: single source BU (use with `--to` and `--de`)                        |
+| `--to <cred>/<bu>`            | Import: target BU (repeatable). API mode: use with `--from`/`--de`. File mode: use with `--file` (no `--from` needed) |
+| `--backup-before-import`      | Export target DE data as a timestamped backup before import (no prompt; always timestamped)                           |
+| `--no-backup-before-import`   | Skip the backup prompt even in interactive (TTY) sessions                                                             |
+| `--clear-before-import`       | SOAP `ClearData` before REST import                                                                                   |
+| `--i-accept-clear-data-risk`  | Non-interactive consent for clear                                                                                     |
 
 Log lines use paths **relative** to the project root (POSIX-style, `./…`) and include **row counts** where applicable.
 

package/lib/cli.mjs

@@ -53,6 +53,8 @@ Options:
 
 Import options:
   --mode <upsert|insert>  Row write mode (default: upsert; async REST bulk API)
+  --backup-before-import  Export target DE data as a timestamped backup before import (no prompt)
+  --no-backup-before-import  Skip the backup prompt even in interactive (TTY) sessions
   --clear-before-import   SOAP ClearData before import (destructive; see below)
   --i-accept-clear-data-risk  Non-interactive acknowledgement for --clear-before-import
 
@@ -96,6 +98,8 @@ export async function main(argv) {
                 to: { type: 'string', multiple: true },
                 git: { type: 'boolean', default: false },
                 mode: { type: 'string' },
+                'backup-before-import': { type: 'boolean', default: false },
+                'no-backup-before-import': { type: 'boolean', default: false },
                 'clear-before-import': { type: 'boolean', default: false },
                 'i-accept-clear-data-risk': { type: 'boolean', default: false },
                 'json-pretty': { type: 'boolean', default: false },
@@ -135,6 +139,12 @@ export async function main(argv) {
         return 1;
     }
 
+    const backupBeforeImport = values['backup-before-import']
+        ? true
+        : values['no-backup-before-import']
+          ? false
+          : undefined;
+
     const useGit = values.git === true;
     const fromFlags = values.from ?? [];
     const toFlags = values.to ?? [];
@@ -264,6 +274,7 @@ export async function main(argv) {
                 targets,
                 format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
                 mode: /** @type {'upsert'|'insert'} */ (mode),
+                backupBeforeImport,
                 clearBeforeImport: clear,
                 acceptRiskFlag: acceptRisk,
                 isTTY: process.stdin.isTTY === true,
@@ -330,6 +341,7 @@ export async function main(argv) {
                 deKeys,
                 format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
                 mode: /** @type {'upsert'|'insert'} */ (mode),
+                backupBeforeImport,
                 clearBeforeImport: clear,
                 acceptRiskFlag: acceptRisk,
                 isTTY: process.stdin.isTTY === true,
@@ -365,6 +377,21 @@ export async function main(argv) {
         if (hasDe) {
             const deKeys = [values.de ?? []].flat();
             const dataDir = dataDirectoryForBu(projectRoot, credential, bu);
+            if (backupBeforeImport === true) {
+                for (const deKey of deKeys) {
+                    const { path: outPath, rowCount } = await exportDataExtensionToFile(sdk, {
+                        projectRoot,
+                        credentialName: credential,
+                        buName: bu,
+                        deKey,
+                        format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
+                        useGit: false,
+                    });
+                    console.error(
+                        `Backup export: ${projectRelativePosix(projectRoot, outPath)} (${rowCount} rows)`,
+                    );
+                }
+            }
             if (clear) {
                 await confirmClearBeforeImport({
                     deKeys,
@@ -400,6 +427,21 @@ export async function main(argv) {
         const keysFromFiles = fileList.map(
             (fp) => parseExportBasename(path.basename(fp)).customerKey,
         );
+        if (backupBeforeImport === true) {
+            for (const deKey of keysFromFiles) {
+                const { path: outPath, rowCount } = await exportDataExtensionToFile(sdk, {
+                    projectRoot,
+                    credentialName: credential,
+                    buName: bu,
+                    deKey,
+                    format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
+                    useGit: false,
+                });
+                console.error(
+                    `Backup export: ${projectRelativePosix(projectRoot, outPath)} (${rowCount} rows)`,
+                );
+            }
+        }
         if (clear) {
             await confirmClearBeforeImport({
                 deKeys: keysFromFiles,

package/lib/cross-bu-import.mjs

@@ -73,10 +73,11 @@ async function offerPreExportBackup({ targets, deKeys, stdin: stdinStream, stdou
  * @param {CredBuTarget[]} params.targets
  * @param {'csv'|'tsv'|'json'} params.format
  * @param {'upsert'|'insert'} params.mode
+ * @param {boolean} [params.backupBeforeImport] - true=always backup, false=never backup, undefined=TTY prompt
  * @param {boolean} params.clearBeforeImport
  * @param {boolean} params.acceptRiskFlag
  * @param {boolean} params.isTTY
- * @param {boolean} [params.useGit] - stable snapshot basename (no timestamp)
+ * @param {boolean} [params.useGit] - accepted for API compatibility but ignored; snapshot files always use a timestamped name
  * @param {import('./config.mjs').DebugLogger|null} [params.logger] - debug logger for API requests/responses
  * @param {NodeJS.ReadableStream} [params.stdin] Override for testing
  * @param {NodeJS.WritableStream} [params.stdout] Override for testing
@@ -90,10 +91,10 @@ export async function crossBuImport(params) {
         targets,
         format,
         mode,
+        backupBeforeImport,
         clearBeforeImport,
         acceptRiskFlag,
         isTTY,
-        useGit = false,
         logger = null,
     } = params;
     const stdin = params.stdin;
@@ -135,29 +136,29 @@ export async function crossBuImport(params) {
     }
 
     // Optional pre-import backup of target BU data
-    if (isTTY) {
-        const doBackup = await offerPreExportBackup({ targets, deKeys, stdin, stdout });
-        if (doBackup) {
-            for (const { credential, bu } of targets) {
-                const { mid, authCred } = resolveCredentialAndMid(
-                    mcdevrc,
-                    mcdevAuth,
-                    credential,
-                    bu,
-                );
-                const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
-                for (const deKey of deKeys) {
-                    const { path: outPath, rowCount } = await exportDataExtensionToFile(tgtSdk, {
-                        projectRoot,
-                        credentialName: credential,
-                        buName: bu,
-                        deKey,
-                        format,
-                        useGit: false,
-                    });
-                    const rel = projectRelativePosix(projectRoot, outPath);
-                    console.error(`Backup export: ${rel} (${rowCount} rows)`);
-                }
+    const shouldBackup =
+        backupBeforeImport === true
+            ? true
+            : backupBeforeImport === false
+              ? false
+              : isTTY
+                ? await offerPreExportBackup({ targets, deKeys, stdin, stdout })
+                : false;
+    if (shouldBackup) {
+        for (const { credential, bu } of targets) {
+            const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
+            const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), buildSdkOptions(logger));
+            for (const deKey of deKeys) {
+                const { path: outPath, rowCount } = await exportDataExtensionToFile(tgtSdk, {
+                    projectRoot,
+                    credentialName: credential,
+                    buName: bu,
+                    deKey,
+                    format,
+                    useGit: false,
+                });
+                const rel = projectRelativePosix(projectRoot, outPath);
+                console.error(`Backup export: ${rel} (${rowCount} rows)`);
             }
         }
     }
@@ -205,7 +206,7 @@ export async function crossBuImport(params) {
             const dir = dataDirectoryForBu(projectRoot, credential, bu);
             await fs.mkdir(dir, { recursive: true });
             const ts = filesystemSafeTimestamp();
-            const basename = buildExportBasename(deKey, ts, format, useGit);
+            const basename = buildExportBasename(deKey, ts, format, false);
             const snapshotPath = path.join(dir, basename);
             await fs.writeFile(snapshotPath, serializeRows(rows, format, false), 'utf8');
             const snapRel = projectRelativePosix(projectRoot, snapshotPath);

package/package.json

@@ -1,6 +1,6 @@
 {
     "name": "sfmc-dataloader",
-    "version": "2.1.0",
+    "version": "2.2.0",
     "description": "SFMC Data Loader CLI (mcdata) to export and import Marketing Cloud Data Extension rows using mcdev project config and sfmc-sdk",
     "author": "Jörn Berkefeld <joern.berkefeld@gmail.com>",
     "license": "MIT",
@@ -44,7 +44,7 @@
         "sfmc-sdk": "3.0.3"
     },
     "scripts": {
-        "test": "node --test test/**/*.test.js",
+        "test": "node --test test/*.test.js",
         "lint": "eslint .",
         "lint:fix": "eslint --fix .",
         "format": "prettier --write .",