sfmc-dataloader 1.2.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -24,7 +24,9 @@ Run from your mcdev project root (where `.mcdevrc.json` lives).
24
24
  mcdata export MyCred/MyBU --de MyDE_CustomerKey --format csv
25
25
  ```
26
26
 
27
- Writes to `./data/MyCred/MyBU/<encodedKey>+MCDATA+<timestamp>.csv` (TSV/JSON with `--format`).
27
+ Writes to `./data/MyCred/MyBU/<encodedKey>.mcdata.<timestamp>.csv` (TSV/JSON with `--format`). Timestamps use the same filesystem-safe ISO format as before.
28
+
29
+ Use **`--git`** for a stable name without a timestamp: `<encodedKey>.mcdata.csv` (useful for version control).
28
30
 
29
31
  ### Export — multiple BUs at once
30
32
 
@@ -34,29 +36,29 @@ Use `--from` (repeatable) instead of the positional argument to export the same
34
36
  mcdata export --from MyCred/Dev --from MyCred/QA --de Contact_DE --de Order_DE
35
37
  ```
36
38
 
37
- Creates one timestamped file per BU/DE combination:
38
-
39
- ```
40
- ./data/MyCred/Dev/Contact_DE+MCDATA+<timestamp>.csv
41
- ./data/MyCred/Dev/Order_DE+MCDATA+<timestamp>.csv
42
- ./data/MyCred/QA/Contact_DE+MCDATA+<timestamp>.csv
43
- ./data/MyCred/QA/Order_DE+MCDATA+<timestamp>.csv
44
- ```
39
+ Creates one file per BU/DE combination using the same `.mcdata.` naming rules.
45
40
 
46
41
  ### Import — single BU
47
42
 
48
43
  ```bash
49
- mcdata import MyCred/MyBU --de MyDE_CustomerKey --format csv --api async --mode upsert
44
+ mcdata import MyCred/MyBU --de MyDE_CustomerKey --format csv --mode upsert
50
45
  ```
51
46
 
47
+ Imports use the **asynchronous** bulk row API only: `POST` for `--mode insert`, `PUT` for `--mode upsert` (same endpoint path).
48
+
52
49
  Resolves the latest matching export file under `./data/MyCred/MyBU/` for that DE key.
53
50
 
54
- Import from explicit paths (DE key is recovered from the `+MCDATA+` filename):
51
+ Import from explicit paths (the DE key is taken from the `.mcdata.` basename):
55
52
 
56
53
  ```bash
57
- mcdata import MyCred/MyBU --file ./data/MyCred/MyBU/encoded%2Bkey+MCDATA+2026-04-06T12-00-00.000Z.csv
54
+ mcdata import MyCred/MyBU --file ./data/MyCred/MyBU/encoded%2Bkey.mcdata.2026-04-06T12-00-00.000Z.csv
58
55
  ```
59
56
 
57
+ #### Upsert vs insert
58
+
59
+ - **Upsert** follows the platform’s usual behaviour: update when a primary key matches, otherwise insert. For Data Extensions **without** a primary key, upsert may not behave as expected; prefer **`--mode insert`** for those.
60
+ - **Insert** always adds new rows. Running import twice with insert can create **duplicate** rows if the same file is applied again—use upsert when keys are defined and you need idempotent runs.
61
+
60
62
  ### Import — one source BU into multiple target BUs (API mode)
61
63
 
62
64
  Use `--from` (one source) and `--to` (repeatable targets) for a cross-BU import where rows are fetched live from the source BU:
@@ -65,7 +67,7 @@ Use `--from` (one source) and `--to` (repeatable targets) for a cross-BU import
65
67
  mcdata import --from MyCred/Dev --to MyCred/QA --to MyCred/Prod --de Contact_DE
66
68
  ```
67
69
 
68
- Before the import starts you will be offered the option to export the current data from each target BU as a timestamped backup. A timestamped download file is also written to each target BU's data directory so there is a traceable record of exactly what was imported.
70
+ Before the import starts you will be offered the option to export the current data from each target BU as a timestamped backup. A download file is also written to each target BU's data directory so there is a traceable record of exactly what was imported.
69
71
 
70
72
  ### Import — local export files into multiple target BUs (file mode)
71
73
 
@@ -73,29 +75,23 @@ Use `--to` (repeatable targets) and `--file` (repeatable file paths) to push pre
73
75
 
74
76
  ```bash
75
77
  mcdata import --to MyCred/QA --to MyCred/Prod \
76
- --file ./data/MyCred/Dev/Contact_DE+MCDATA+2026-04-08T10-00-00.000Z.csv
78
+ --file ./data/MyCred/Dev/Contact_DE.mcdata.2026-04-08T10-00-00.000Z.csv
77
79
  ```
78
80
 
79
- Multiple files can be supplied to push several DEs in one command:
80
-
81
- ```bash
82
- mcdata import --to MyCred/QA --to MyCred/Prod \
83
- --file ./data/MyCred/Dev/Contact_DE+MCDATA+2026-04-08T10-00-00.000Z.csv \
84
- --file ./data/MyCred/Dev/Order_DE+MCDATA+2026-04-08T10-00-00.000Z.csv
85
- ```
86
-
87
- A timestamped download file is written to each target BU's data directory, giving a traceable snapshot of exactly what was imported.
81
+ Multiple files can be supplied to push several DEs in one command. Pass **`--git`** on export if you rely on stable `*.mcdata.<ext>` names for snapshots in this flow.
88
82
 
89
83
  ### Clear all rows before import
90
84
 
91
85
  **Dangerous:** removes every row in the target Data Extension(s) before uploading.
92
86
 
93
87
  Single-BU:
88
+
94
89
  ```bash
95
90
  mcdata import MyCred/MyBU --de MyKey --clear-before-import
96
91
  ```
97
92
 
98
93
  Cross-BU (warning lists every affected BU):
94
+
99
95
  ```bash
100
96
  mcdata import --from MyCred/Dev --to MyCred/QA --to MyCred/Prod \
101
97
  --de Contact_DE --clear-before-import
@@ -109,13 +105,15 @@ Interactive: type `YES` when prompted. In CI, add `--i-accept-clear-data-risk` a
109
105
  |--------|-------------|
110
106
  | `-p, --project` | Project root (default: cwd) |
111
107
  | `--format` | `csv` (default), `tsv`, or `json` |
112
- | `--api` | `async` (default) or `sync` |
113
- | `--mode` | `upsert` (default), `insert` and `update` require `--api sync` |
108
+ | `--git` | Stable export filenames: `<key>.mcdata.<ext>` (no timestamp segment) |
109
+ | `--mode` | `upsert` (default) or `insert` async bulk REST API only |
114
110
  | `--from <cred>/<bu>` | Export: source BU (repeatable). Import API mode: single source BU (use with `--to` and `--de`) |
115
111
  | `--to <cred>/<bu>` | Import: target BU (repeatable). API mode: use with `--from`/`--de`. File mode: use with `--file` (no `--from` needed) |
116
112
  | `--clear-before-import` | SOAP `ClearData` before REST import |
117
113
  | `--i-accept-clear-data-risk` | Non-interactive consent for clear |
118
114
 
115
+ Log lines use paths **relative** to the project root (POSIX-style, `./…`) and include **row counts** where applicable.
116
+
119
117
  ## License
120
118
 
121
119
  MIT — Author: Jörn Berkefeld
package/lib/cli.mjs CHANGED
@@ -8,7 +8,7 @@ import {
8
8
  resolveCredentialAndMid,
9
9
  buildSdkAuthObject,
10
10
  } from './config.mjs';
11
- import { dataDirectoryForBu } from './paths.mjs';
11
+ import { dataDirectoryForBu, projectRelativePosix } from './paths.mjs';
12
12
  import { exportDataExtensionToFile } from './export-de.mjs';
13
13
  import { findImportCandidates, pickLatestByMtime } from './file-resolve.mjs';
14
14
  import { parseExportBasename } from './filename.mjs';
@@ -32,10 +32,10 @@ Options:
32
32
  -p, --project <dir> mcdev project root (default: cwd)
33
33
  --format <csv|tsv|json> File format (default: csv)
34
34
  --json-pretty Pretty-print JSON on export
35
+ --git Stable filenames: <key>.mcdata.<ext> (no timestamp)
35
36
 
36
37
  Import options:
37
- --api <async|sync> REST row API family (default: async)
38
- --mode <upsert|insert|update> (default: upsert; insert/update require --api sync)
38
+ --mode <upsert|insert> Row write mode (default: upsert; async REST bulk API)
39
39
  --clear-before-import SOAP ClearData before import (destructive; see below)
40
40
  --i-accept-clear-data-risk Non-interactive acknowledgement for --clear-before-import
41
41
 
@@ -46,10 +46,10 @@ Multi-BU options:
46
46
  Import (file mode): use with --file only (no --from needed)
47
47
 
48
48
  Notes:
49
- Exports are written under ./data/<credential>/<bu>/ with "+MCDATA+" in the filename.
49
+ Exports are written under ./data/<credential>/<bu>/ using ".mcdata." in the filename.
50
50
  Import with --de resolves the latest matching file in that folder (by mtime).
51
- Import with --file parses the DE key from the basename (+MCDATA+ prefix).
52
- Cross-BU import stores a timestamped download file in each target BU's data directory.
51
+ Import with --file parses the DE key from the basename (.mcdata. format).
52
+ Cross-BU import stores a download file in each target BU's data directory.
53
53
 
54
54
  Clear data warning:
55
55
  --clear-before-import deletes ALL existing rows in the target DE(s) before upload.
@@ -74,10 +74,10 @@ export async function main(argv) {
74
74
  format: { type: 'string' },
75
75
  de: { type: 'string', multiple: true },
76
76
  file: { type: 'string', multiple: true },
77
- api: { type: 'string' },
78
- mode: { type: 'string' },
79
77
  from: { type: 'string', multiple: true },
80
78
  to: { type: 'string', multiple: true },
79
+ git: { type: 'boolean', default: false },
80
+ mode: { type: 'string' },
81
81
  'clear-before-import': { type: 'boolean', default: false },
82
82
  'i-accept-clear-data-risk': { type: 'boolean', default: false },
83
83
  'json-pretty': { type: 'boolean', default: false },
@@ -111,6 +111,7 @@ export async function main(argv) {
111
111
  return 1;
112
112
  }
113
113
 
114
+ const useGit = values.git === true;
114
115
  const fromFlags = values.from ?? [];
115
116
  const toFlags = values.to ?? [];
116
117
  const hasFrom = fromFlags.length > 0;
@@ -142,7 +143,6 @@ export async function main(argv) {
142
143
  }
143
144
 
144
145
  if (hasFrom) {
145
- // Multi-BU export
146
146
  let sources;
147
147
  try {
148
148
  sources = fromFlags.map(parseCredBu);
@@ -159,39 +159,36 @@ export async function main(argv) {
159
159
  deKeys: des,
160
160
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
161
161
  jsonPretty: values['json-pretty'],
162
+ useGit,
162
163
  });
163
164
  return 0;
164
165
  }
165
166
 
166
- // Single-BU export (original behavior)
167
167
  const { credential, bu } = parseCredBu(credBuRaw);
168
168
  const { mcdevrc, mcdevAuth } = loadMcdevProject(projectRoot);
169
169
  const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
170
170
  const sdk = new SDK(buildSdkAuthObject(authCred, mid), { requestAttempts: 3 });
171
171
  for (const deKey of des) {
172
- const out = await exportDataExtensionToFile(sdk, {
172
+ const { path: out, rowCount } = await exportDataExtensionToFile(sdk, {
173
173
  projectRoot,
174
174
  credentialName: credential,
175
175
  buName: bu,
176
176
  deKey,
177
177
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
178
178
  jsonPretty: values['json-pretty'],
179
+ useGit,
179
180
  });
180
- console.error(`Exported: ${out}`);
181
+ const rel = projectRelativePosix(projectRoot, out);
182
+ console.error(`Exported: ${rel} (${rowCount} rows)`);
181
183
  }
182
184
  return 0;
183
185
  }
184
186
 
185
187
  // ── import ──────────────────────────────────────────────────────────────
186
188
  if (sub === 'import') {
187
- const api = values.api ?? 'async';
188
189
  const mode = values.mode ?? 'upsert';
189
- if (!['async', 'sync'].includes(api)) {
190
- console.error(`Invalid --api: ${api}`);
191
- return 1;
192
- }
193
- if (!['upsert', 'insert', 'update'].includes(mode)) {
194
- console.error(`Invalid --mode: ${mode}`);
190
+ if (!['upsert', 'insert'].includes(mode)) {
191
+ console.error(`Invalid --mode: ${mode} (use upsert or insert)`);
195
192
  return 1;
196
193
  }
197
194
 
@@ -199,7 +196,6 @@ export async function main(argv) {
199
196
  const acceptRisk = values['i-accept-clear-data-risk'];
200
197
 
201
198
  // ── File-to-multi-BU import: --to + --file (no --from) ─────────────
202
- // Rows are read from local export files; no source BU auth needed.
203
199
  if (hasTo && !hasFrom && values.file?.length > 0) {
204
200
  if (hasPositional) {
205
201
  console.error('Cannot mix a positional <credential>/<bu> with --to/--file. Use one or the other.');
@@ -225,11 +221,11 @@ export async function main(argv) {
225
221
  filePaths,
226
222
  targets,
227
223
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
228
- api: /** @type {'async'|'sync'} */ (api),
229
- mode: /** @type {'upsert'|'insert'|'update'} */ (mode),
224
+ mode: /** @type {'upsert'|'insert'} */ (mode),
230
225
  clearBeforeImport: clear,
231
226
  acceptRiskFlag: acceptRisk,
232
227
  isTTY: process.stdin.isTTY === true,
228
+ useGit,
233
229
  });
234
230
  return 0;
235
231
  }
@@ -280,11 +276,11 @@ export async function main(argv) {
280
276
  targets,
281
277
  deKeys,
282
278
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
283
- api: /** @type {'async'|'sync'} */ (api),
284
- mode: /** @type {'upsert'|'insert'|'update'} */ (mode),
279
+ mode: /** @type {'upsert'|'insert'} */ (mode),
285
280
  clearBeforeImport: clear,
286
281
  acceptRiskFlag: acceptRisk,
287
282
  isTTY: process.stdin.isTTY === true,
283
+ useGit,
288
284
  });
289
285
  return 0;
290
286
  }
@@ -324,19 +320,19 @@ export async function main(argv) {
324
320
  for (const deKey of deKeys) {
325
321
  const candidates = await findImportCandidates(dataDir, deKey, fmt);
326
322
  if (candidates.length === 0) {
327
- console.error(`No ${fmt} file found for DE "${deKey}" under ${dataDir}`);
323
+ console.error(`No ${fmt} file found for DE "${deKey}" under ${projectRelativePosix(projectRoot, dataDir)}`);
328
324
  return 1;
329
325
  }
330
326
  const filePath =
331
327
  candidates.length === 1 ? candidates[0] : await pickLatestByMtime(candidates);
332
- await importFromFile(sdk, {
328
+ const n = await importFromFile(sdk, {
333
329
  filePath,
334
330
  deKey,
335
331
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
336
- api: /** @type {'async'|'sync'} */ (api),
337
- mode: /** @type {'upsert'|'insert'|'update'} */ (mode),
332
+ mode: /** @type {'upsert'|'insert'} */ (mode),
338
333
  });
339
- console.error(`Imported ${filePath} -> DE ${deKey}`);
334
+ const rel = projectRelativePosix(projectRoot, filePath);
335
+ console.error(`Imported: ${rel} (${n} rows) -> DE ${deKey}`);
340
336
  }
341
337
  return 0;
342
338
  }
@@ -356,14 +352,14 @@ export async function main(argv) {
356
352
  for (const filePath of fileList) {
357
353
  const base = path.basename(filePath);
358
354
  const { customerKey } = parseExportBasename(base);
359
- await importFromFile(sdk, {
355
+ const n = await importFromFile(sdk, {
360
356
  filePath,
361
357
  deKey: customerKey,
362
358
  format: /** @type {'csv'|'tsv'|'json'} */ (fmt),
363
- api: /** @type {'async'|'sync'} */ (api),
364
- mode: /** @type {'upsert'|'insert'|'update'} */ (mode),
359
+ mode: /** @type {'upsert'|'insert'} */ (mode),
365
360
  });
366
- console.error(`Imported ${filePath}`);
361
+ const rel = projectRelativePosix(projectRoot, filePath);
362
+ console.error(`Imported: ${rel} (${n} rows)`);
367
363
  }
368
364
  return 0;
369
365
  }
@@ -4,13 +4,12 @@ import readline from 'node:readline/promises';
4
4
  import { stdin as input, stdout as output } from 'node:process';
5
5
  import SDK from 'sfmc-sdk';
6
6
  import { resolveCredentialAndMid, buildSdkAuthObject } from './config.mjs';
7
- import { fetchAllRowObjects, serializeRows } from './export-de.mjs';
8
- import { exportDataExtensionToFile } from './export-de.mjs';
7
+ import { fetchAllRowObjects, serializeRows, exportDataExtensionToFile } from './export-de.mjs';
9
8
  import { importRowsForDe } from './import-de.mjs';
10
9
  import { readRowsFromFile } from './read-rows.mjs';
11
10
  import { clearDataExtensionRows } from './clear-de.mjs';
12
11
  import { confirmClearBeforeImport } from './confirm-clear.mjs';
13
- import { dataDirectoryForBu } from './paths.mjs';
12
+ import { dataDirectoryForBu, projectRelativePosix } from './paths.mjs';
14
13
  import { buildExportBasename, filesystemSafeTimestamp, parseExportBasename } from './filename.mjs';
15
14
 
16
15
  /**
@@ -62,13 +61,6 @@ async function offerPreExportBackup({ targets, deKeys, stdin: stdinStream, stdou
62
61
  * `sourceCred`/`sourceBu` must be omitted. The DE customer key is derived
63
62
  * from each filename via `parseExportBasename`.
64
63
  *
65
- * Before the import each target BU:
66
- * 1. Optionally exports its current DE data as a timestamped backup (TTY only).
67
- * 2. Optionally clears all existing rows (with danger warning covering every target).
68
- * 3. Receives the source rows written to a timestamped "download" file in its
69
- * own `./data/<credential>/<bu>/` directory (mirroring mcdev retrieve).
70
- * 4. Has the rows imported via the REST API.
71
- *
72
64
  * @param {object} params
73
65
  * @param {string} params.projectRoot
74
66
  * @param {import('./config.mjs').Mcdevrc} params.mcdevrc
@@ -79,11 +71,11 @@ async function offerPreExportBackup({ targets, deKeys, stdin: stdinStream, stdou
79
71
  * @param {string[]} [params.filePaths] - File mode only; mutually exclusive with sourceCred/sourceBu/deKeys
80
72
  * @param {CredBuTarget[]} params.targets
81
73
  * @param {'csv'|'tsv'|'json'} params.format
82
- * @param {'async'|'sync'} params.api
83
- * @param {'upsert'|'insert'|'update'} params.mode
74
+ * @param {'upsert'|'insert'} params.mode
84
75
  * @param {boolean} params.clearBeforeImport
85
76
  * @param {boolean} params.acceptRiskFlag
86
77
  * @param {boolean} params.isTTY
78
+ * @param {boolean} [params.useGit] - stable snapshot basename (no timestamp)
87
79
  * @param {NodeJS.ReadableStream} [params.stdin] Override for testing
88
80
  * @param {NodeJS.WritableStream} [params.stdout] Override for testing
89
81
  * @returns {Promise<void>}
@@ -95,11 +87,11 @@ export async function crossBuImport(params) {
95
87
  mcdevAuth,
96
88
  targets,
97
89
  format,
98
- api,
99
90
  mode,
100
91
  clearBeforeImport,
101
92
  acceptRiskFlag,
102
93
  isTTY,
94
+ useGit = false,
103
95
  } = params;
104
96
  const stdin = params.stdin;
105
97
  const stdout = params.stdout;
@@ -144,14 +136,16 @@ export async function crossBuImport(params) {
144
136
  const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
145
137
  const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), { requestAttempts: 3 });
146
138
  for (const deKey of deKeys) {
147
- const outPath = await exportDataExtensionToFile(tgtSdk, {
139
+ const { path: outPath, rowCount } = await exportDataExtensionToFile(tgtSdk, {
148
140
  projectRoot,
149
141
  credentialName: credential,
150
142
  buName: bu,
151
143
  deKey,
152
144
  format,
145
+ useGit: false,
153
146
  });
154
- console.error(`Backup export: ${outPath}`);
147
+ const rel = projectRelativePosix(projectRoot, outPath);
148
+ console.error(`Backup export: ${rel} (${rowCount} rows)`);
155
149
  }
156
150
  }
157
151
  }
@@ -181,19 +175,18 @@ export async function crossBuImport(params) {
181
175
  const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
182
176
  const tgtSdk = new SDK(buildSdkAuthObject(authCred, mid), { requestAttempts: 3 });
183
177
 
184
- // Write a timestamped "download" file in the target BU's data directory.
185
- // This mirrors what mcdev retrieve / mcdata export produces, giving a
186
- // traceable snapshot of exactly what was imported.
178
+ // Write a snapshot file in the target BU's data directory.
187
179
  const dir = dataDirectoryForBu(projectRoot, credential, bu);
188
180
  await fs.mkdir(dir, { recursive: true });
189
181
  const ts = filesystemSafeTimestamp();
190
- const basename = buildExportBasename(deKey, ts, format);
182
+ const basename = buildExportBasename(deKey, ts, format, useGit);
191
183
  const snapshotPath = path.join(dir, basename);
192
184
  await fs.writeFile(snapshotPath, serializeRows(rows, format, false), 'utf8');
193
- console.error(`Download stored: ${snapshotPath}`);
185
+ const snapRel = projectRelativePosix(projectRoot, snapshotPath);
186
+ console.error(`Download stored: ${snapRel} (${rows.length} rows)`);
194
187
 
195
- await importRowsForDe(tgtSdk, { deKey, rows, api, mode });
196
- console.error(`Imported -> ${credential}/${bu} DE ${deKey}`);
188
+ const imported = await importRowsForDe(tgtSdk, { deKey, rows, mode });
189
+ console.error(`Imported: ${credential}/${bu} DE ${deKey} (${imported} rows)`);
197
190
  }
198
191
  }
199
192
  }
package/lib/export-de.mjs CHANGED
@@ -6,37 +6,17 @@ import { buildExportBasename, filesystemSafeTimestamp } from './filename.mjs';
6
6
  import { dataDirectoryForBu } from './paths.mjs';
7
7
 
8
8
  /**
9
- * @param {{ rest: { get: (path: string) => Promise<any> } }} sdk
9
+ * @param {{ rest: { getBulk: (path: string, pageSize?: number) => Promise<any> } }} sdk
10
10
  * @param {string} deKey
11
11
  * @returns {Promise<object[]>}
12
12
  */
13
13
  export async function fetchAllRowObjects(sdk, deKey) {
14
- const pageSize = 2500;
15
- let page = 1;
14
+ const basePath = rowsetGetPath(deKey);
15
+ const data = await sdk.rest.getBulk(basePath, 2500);
16
+ const items = data.items ?? [];
16
17
  const rows = [];
17
- let hasMore = true;
18
- while (hasMore) {
19
- const qs = new URLSearchParams({
20
- page: String(page),
21
- pageSize: String(pageSize),
22
- });
23
- const urlPath = `${rowsetGetPath(deKey)}?${qs.toString()}`;
24
- const data = await sdk.rest.get(urlPath);
25
- const items = data.items ?? [];
26
- for (const item of items) {
27
- rows.push({ ...item.keys, ...item.values });
28
- }
29
- if (items.length === 0) {
30
- hasMore = false;
31
- } else if (data.hasMoreRows === false) {
32
- hasMore = false;
33
- } else if (data.hasMoreRows === true) {
34
- hasMore = true;
35
- page++;
36
- } else {
37
- hasMore = items.length === pageSize;
38
- page++;
39
- }
18
+ for (const item of items) {
19
+ rows.push({ ...item.keys, ...item.values });
40
20
  }
41
21
  return rows;
42
22
  }
@@ -62,7 +42,7 @@ export function serializeRows(rows, format, jsonPretty) {
62
42
  }
63
43
 
64
44
  /**
65
- * @param {{ rest: { get: (path: string) => Promise<any> } }} sdk
45
+ * @param {{ rest: { getBulk: (path: string, pageSize?: number) => Promise<any> } }} sdk
66
46
  * @param {object} params
67
47
  * @param {string} params.projectRoot
68
48
  * @param {string} params.credentialName
@@ -70,17 +50,18 @@ export function serializeRows(rows, format, jsonPretty) {
70
50
  * @param {string} params.deKey
71
51
  * @param {'csv'|'tsv'|'json'} params.format
72
52
  * @param {boolean} [params.jsonPretty]
73
- * @returns {Promise<string>} written file path
53
+ * @param {boolean} [params.useGit]
54
+ * @returns {Promise<{ path: string, rowCount: number }>}
74
55
  */
75
56
  export async function exportDataExtensionToFile(sdk, params) {
76
- const { projectRoot, credentialName, buName, deKey, format, jsonPretty = false } = params;
57
+ const { projectRoot, credentialName, buName, deKey, format, jsonPretty = false, useGit = false } = params;
77
58
  const rows = await fetchAllRowObjects(sdk, deKey);
78
59
  const dir = dataDirectoryForBu(projectRoot, credentialName, buName);
79
60
  await fs.mkdir(dir, { recursive: true });
80
61
  const ts = filesystemSafeTimestamp();
81
- const basename = buildExportBasename(deKey, ts, format);
62
+ const basename = buildExportBasename(deKey, ts, format, useGit);
82
63
  const outPath = path.join(dir, basename);
83
64
  const body = serializeRows(rows, format, jsonPretty);
84
65
  await fs.writeFile(outPath, body, 'utf8');
85
- return outPath;
66
+ return { path: outPath, rowCount: rows.length };
86
67
  }
@@ -1,9 +1,9 @@
1
1
  import fs from 'node:fs/promises';
2
2
  import path from 'node:path';
3
- import { filterIllegalFilenames, MCDATA_SENTINEL } from './filename.mjs';
3
+ import { parseExportBasename } from './filename.mjs';
4
4
 
5
5
  /**
6
- * Find export files under data dir matching encoded DE key prefix and extension.
6
+ * Find export files under data dir matching the DE customer key and extension.
7
7
  *
8
8
  * @param {string} dataDir
9
9
  * @param {string} customerKey
@@ -11,7 +11,6 @@ import { filterIllegalFilenames, MCDATA_SENTINEL } from './filename.mjs';
11
11
  * @returns {Promise<string[]>} full paths
12
12
  */
13
13
  export async function findImportCandidates(dataDir, customerKey, format) {
14
- const prefix = filterIllegalFilenames(customerKey) + MCDATA_SENTINEL;
15
14
  let entries;
16
15
  try {
17
16
  entries = await fs.readdir(dataDir, { withFileTypes: true });
@@ -28,8 +27,13 @@ export async function findImportCandidates(dataDir, customerKey, format) {
28
27
  if (!name.endsWith(`.${ext}`)) {
29
28
  continue;
30
29
  }
31
- if (name.startsWith(prefix)) {
32
- matches.push(path.join(dataDir, name));
30
+ try {
31
+ const { customerKey: parsedKey } = parseExportBasename(name);
32
+ if (parsedKey === customerKey) {
33
+ matches.push(path.join(dataDir, name));
34
+ }
35
+ } catch {
36
+ continue;
33
37
  }
34
38
  }
35
39
  return matches;
package/lib/filename.mjs CHANGED
@@ -4,6 +4,9 @@
4
4
  * @see https://github.com/Accenture/sfmc-devtools (lib/util/file.js)
5
5
  */
6
6
 
7
+ /** Literal segment between encoded DE key and timestamp (or extension in --git mode). */
8
+ export const MCDATA_SEGMENT = '.mcdata.';
9
+
7
10
  /**
8
11
  * @param {string} filename
9
12
  * @returns {string}
@@ -35,17 +38,19 @@ export function reverseFilterIllegalFilenames(filename) {
35
38
  return decodeURIComponent(filename).split('_STAR_').join('*');
36
39
  }
37
40
 
38
- /** Sentinel between encoded DE key and timestamp in export basenames; cannot appear in the key segment after encoding. */
39
- export const MCDATA_SENTINEL = '+MCDATA+';
40
-
41
41
  /**
42
42
  * @param {string} customerKey
43
- * @param {string} safeTs - filesystem-safe UTC timestamp
43
+ * @param {string} safeTs - filesystem-safe UTC timestamp (ignored when useGit is true)
44
44
  * @param {'csv'|'tsv'|'json'} ext
45
+ * @param {boolean} [useGit] - stable `key.mcdata.ext` without timestamp
45
46
  * @returns {string} basename without directory
46
47
  */
47
- export function buildExportBasename(customerKey, safeTs, ext) {
48
- return `${filterIllegalFilenames(customerKey)}${MCDATA_SENTINEL}${safeTs}.${ext}`;
48
+ export function buildExportBasename(customerKey, safeTs, ext, useGit = false) {
49
+ const enc = filterIllegalFilenames(customerKey);
50
+ if (useGit) {
51
+ return `${enc}.mcdata.${ext}`;
52
+ }
53
+ return `${enc}${MCDATA_SEGMENT}${safeTs}.${ext}`;
49
54
  }
50
55
 
51
56
  /**
@@ -57,24 +62,33 @@ export function filesystemSafeTimestamp(d = new Date()) {
57
62
  }
58
63
 
59
64
  /**
60
- * @param {string} basename - e.g. `encodedKey+MCDATA+2026-04-06T15-00-00.000Z.csv`
65
+ * @param {string} basename - e.g. `encodedKey.mcdata.2026-04-06T15-00-00.000Z.csv` or `encodedKey.mcdata.csv`
61
66
  * @returns {{ customerKey: string, timestampPart: string, ext: string }}
62
67
  */
63
68
  export function parseExportBasename(basename) {
64
69
  const lastDot = basename.lastIndexOf('.');
65
70
  const stem = lastDot === -1 ? basename : basename.slice(0, lastDot);
66
- const ext = lastDot === -1 ? '' : basename.slice(lastDot + 1);
67
- const idx = stem.indexOf(MCDATA_SENTINEL);
68
- if (idx === -1) {
69
- throw new Error(
70
- `Filename must contain "${MCDATA_SENTINEL}" between encoded key and timestamp: ${basename}`
71
- );
71
+ const ext = lastDot === -1 ? '' : basename.slice(lastDot + 1).toLowerCase();
72
+
73
+ const idx = stem.indexOf(MCDATA_SEGMENT);
74
+ if (idx !== -1) {
75
+ const encodedKey = stem.slice(0, idx);
76
+ const timestampPart = stem.slice(idx + MCDATA_SEGMENT.length);
77
+ return {
78
+ customerKey: reverseFilterIllegalFilenames(encodedKey),
79
+ timestampPart,
80
+ ext,
81
+ };
72
82
  }
73
- const encodedKey = stem.slice(0, idx);
74
- const timestampPart = stem.slice(idx + MCDATA_SENTINEL.length);
75
- return {
76
- customerKey: reverseFilterIllegalFilenames(encodedKey),
77
- timestampPart,
78
- ext: ext.toLowerCase(),
79
- };
83
+
84
+ if (stem.endsWith('.mcdata')) {
85
+ const encodedKey = stem.slice(0, -'.mcdata'.length);
86
+ return {
87
+ customerKey: reverseFilterIllegalFilenames(encodedKey),
88
+ timestampPart: '',
89
+ ext,
90
+ };
91
+ }
92
+
93
+ throw new Error(`Filename must contain ".mcdata." or end with ".mcdata" before the extension: ${basename}`);
80
94
  }
package/lib/import-de.mjs CHANGED
@@ -8,23 +8,23 @@ import { readRowsFromFile } from './read-rows.mjs';
8
8
  * @param {object} params
9
9
  * @param {string} params.deKey
10
10
  * @param {object[]} params.rows
11
- * @param {'async'|'sync'} params.api
12
- * @param {'upsert'|'insert'|'update'} params.mode
13
- * @returns {Promise<void>}
11
+ * @param {'upsert'|'insert'} params.mode
12
+ * @returns {Promise<number>} number of rows imported
14
13
  */
15
14
  export async function importRowsForDe(sdk, params) {
16
- const { deKey, rows, api, mode } = params;
17
- const route = resolveImportRoute(api, mode);
15
+ const { deKey, rows, mode } = params;
16
+ const route = resolveImportRoute(mode);
18
17
  const chunks = chunkItemsForPayload(rows);
19
18
  for (const chunk of chunks) {
20
- const path = route.path(deKey);
19
+ const p = route.path(deKey);
21
20
  const body = { items: chunk };
22
21
  await withRetry429(() =>
23
22
  route.method === 'PUT'
24
- ? sdk.rest.put(path, body)
25
- : sdk.rest.post(path, body)
23
+ ? sdk.rest.put(p, body)
24
+ : sdk.rest.post(p, body)
26
25
  );
27
26
  }
27
+ return rows.length;
28
28
  }
29
29
 
30
30
  /**
@@ -33,16 +33,14 @@ export async function importRowsForDe(sdk, params) {
33
33
  * @param {string} params.filePath
34
34
  * @param {string} params.deKey - target DE customer key for API
35
35
  * @param {'csv'|'tsv'|'json'} params.format
36
- * @param {'async'|'sync'} params.api
37
- * @param {'upsert'|'insert'|'update'} params.mode
38
- * @returns {Promise<void>}
36
+ * @param {'upsert'|'insert'} params.mode
37
+ * @returns {Promise<number>} number of rows imported
39
38
  */
40
39
  export async function importFromFile(sdk, params) {
41
40
  const rows = await readRowsFromFile(params.filePath, params.format);
42
- await importRowsForDe(sdk, {
41
+ return importRowsForDe(sdk, {
43
42
  deKey: params.deKey,
44
43
  rows,
45
- api: params.api,
46
44
  mode: params.mode,
47
45
  });
48
46
  }
@@ -1,5 +1,5 @@
1
1
  /**
2
- * REST paths for Data Extension row writes (relative to REST base URL).
2
+ * REST paths for Data Extension row operations (relative to REST base URL).
3
3
  * Confirm against current Salesforce reference hubs when adjusting behavior.
4
4
  */
5
5
 
@@ -12,56 +12,24 @@ export function rowsetGetPath(deKey) {
12
12
  }
13
13
 
14
14
  /**
15
- * Async bulk upsert (default for `--api async`).
15
+ * Async bulk row writes: POST insert, PUT upsert (same URL).
16
16
  * @param {string} deKey
17
17
  * @returns {string}
18
18
  */
19
- export function asyncUpsertPath(deKey) {
19
+ export function asyncDataExtensionRowsPath(deKey) {
20
20
  return `/data/v1/async/dataextensions/key:${encodeURIComponent(deKey)}/rows`;
21
21
  }
22
22
 
23
23
  /**
24
- * Synchronous upsert row set.
25
- * @param {string} deKey
26
- * @returns {string}
27
- */
28
- export function syncUpsertPath(deKey) {
29
- return `/data/v1/customobjectdata/key/${encodeURIComponent(deKey)}/rows`;
30
- }
31
-
32
- /**
33
- * Synchronous insert row set (POST).
34
- * @param {string} deKey
35
- * @returns {string}
36
- */
37
- export function syncInsertPath(deKey) {
38
- return `/data/v1/customobjectdata/key/${encodeURIComponent(deKey)}/rows`;
39
- }
40
-
41
- /**
42
- * @param {'async'|'sync'} api
43
- * @param {'upsert'|'insert'|'update'} mode
24
+ * @param {'upsert'|'insert'} mode
44
25
  * @returns {{ method: 'PUT'|'POST', path: (de: string) => string }}
45
26
  */
46
- export function resolveImportRoute(api, mode) {
47
- if (api === 'async') {
48
- if (mode !== 'upsert') {
49
- throw new Error(
50
- `Import mode "${mode}" is not supported with --api async (use --api sync or --mode upsert).`
51
- );
52
- }
53
- return { method: 'PUT', path: asyncUpsertPath };
27
+ export function resolveImportRoute(mode) {
28
+ if (mode === 'upsert') {
29
+ return { method: 'PUT', path: asyncDataExtensionRowsPath };
54
30
  }
55
- if (api === 'sync') {
56
- if (mode === 'upsert') {
57
- return { method: 'PUT', path: syncUpsertPath };
58
- }
59
- if (mode === 'insert') {
60
- return { method: 'POST', path: syncInsertPath };
61
- }
62
- if (mode === 'update') {
63
- return { method: 'PUT', path: syncUpsertPath };
64
- }
31
+ if (mode === 'insert') {
32
+ return { method: 'POST', path: asyncDataExtensionRowsPath };
65
33
  }
66
- throw new Error(`Unsupported --api / --mode combination: ${api} + ${mode}`);
34
+ throw new Error(`Unsupported import mode "${mode}" (use upsert or insert).`);
67
35
  }
package/lib/index.mjs CHANGED
@@ -1,7 +1,8 @@
1
1
  export { main } from './cli.mjs';
2
2
  export { filterIllegalFilenames, reverseFilterIllegalFilenames, parseExportBasename } from './filename.mjs';
3
3
  export { chunkItemsForPayload, DEFAULT_MAX_BODY_BYTES, MAX_OBJECTS_PER_BATCH } from './batch.mjs';
4
- export { resolveImportRoute, rowsetGetPath, asyncUpsertPath } from './import-routes.mjs';
4
+ export { resolveImportRoute, rowsetGetPath, asyncDataExtensionRowsPath } from './import-routes.mjs';
5
5
  export { loadMcdevProject, parseCredBu, resolveCredentialAndMid, buildSdkAuthObject } from './config.mjs';
6
6
  export { multiBuExport } from './multi-bu-export.mjs';
7
7
  export { crossBuImport } from './cross-bu-import.mjs';
8
+ export { projectRelativePosix } from './paths.mjs';
@@ -1,6 +1,7 @@
1
1
  import SDK from 'sfmc-sdk';
2
2
  import { resolveCredentialAndMid, buildSdkAuthObject } from './config.mjs';
3
3
  import { exportDataExtensionToFile } from './export-de.mjs';
4
+ import { projectRelativePosix } from './paths.mjs';
4
5
 
5
6
  /**
6
7
  * @typedef {{ credential: string, bu: string }} CredBuSource
@@ -19,24 +20,27 @@ import { exportDataExtensionToFile } from './export-de.mjs';
19
20
  * @param {string[]} params.deKeys
20
21
  * @param {'csv'|'tsv'|'json'} params.format
21
22
  * @param {boolean} [params.jsonPretty]
23
+ * @param {boolean} [params.useGit]
22
24
  * @returns {Promise<string[]>} Paths of all written files
23
25
  */
24
- export async function multiBuExport({ projectRoot, mcdevrc, mcdevAuth, sources, deKeys, format, jsonPretty = false }) {
26
+ export async function multiBuExport({ projectRoot, mcdevrc, mcdevAuth, sources, deKeys, format, jsonPretty = false, useGit = false }) {
25
27
  /** @type {string[]} */
26
28
  const exported = [];
27
29
  for (const { credential, bu } of sources) {
28
30
  const { mid, authCred } = resolveCredentialAndMid(mcdevrc, mcdevAuth, credential, bu);
29
31
  const sdk = new SDK(buildSdkAuthObject(authCred, mid), { requestAttempts: 3 });
30
32
  for (const deKey of deKeys) {
31
- const outPath = await exportDataExtensionToFile(sdk, {
33
+ const { path: outPath, rowCount } = await exportDataExtensionToFile(sdk, {
32
34
  projectRoot,
33
35
  credentialName: credential,
34
36
  buName: bu,
35
37
  deKey,
36
38
  format,
37
39
  jsonPretty,
40
+ useGit,
38
41
  });
39
- console.error(`Exported: ${outPath}`);
42
+ const rel = projectRelativePosix(projectRoot, outPath);
43
+ console.error(`Exported: ${rel} (${rowCount} rows)`);
40
44
  exported.push(outPath);
41
45
  }
42
46
  }
package/lib/paths.mjs CHANGED
@@ -9,3 +9,19 @@ import path from 'node:path';
9
9
  export function dataDirectoryForBu(projectRoot, credentialName, buName) {
10
10
  return path.join(projectRoot, 'data', credentialName, buName);
11
11
  }
12
+
13
+ /**
14
+ * Path relative to mcdev project root for logs (POSIX-style, `./`-prefixed when needed).
15
+ *
16
+ * @param {string} projectRoot
17
+ * @param {string} absolutePath
18
+ * @returns {string}
19
+ */
20
+ export function projectRelativePosix(projectRoot, absolutePath) {
21
+ const rel = path.relative(path.resolve(projectRoot), path.resolve(absolutePath));
22
+ const norm = rel.split(path.sep).join('/');
23
+ if (norm === '' || norm === '.') {
24
+ return './';
25
+ }
26
+ return norm.startsWith('.') ? norm : `./${norm}`;
27
+ }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "sfmc-dataloader",
3
- "version": "1.2.0",
4
- "description": "CLI (mcdata) to export and import Marketing Cloud Data Extension rows using mcdev project config and sfmc-sdk",
3
+ "version": "2.0.0",
4
+ "description": "SFMC Data Loader CLI (mcdata) to export and import Marketing Cloud Data Extension rows using mcdev project config and sfmc-sdk",
5
5
  "author": "Jörn Berkefeld <joern.berkefeld@gmail.com>",
6
6
  "license": "MIT",
7
7
  "repository": {