@xeplr/actions 1.4.3 → 1.4.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +54 -13
- package/lib/builtins/data-operation.js +12 -2
- package/lib/builtins/email/download-attachments.js +3 -2
- package/lib/builtins/email/download-email.js +2 -1
- package/lib/builtins/export/table.js +51 -11
- package/lib/drivers/email/files.js +68 -0
- package/lib/drivers/email/imap.js +13 -31
- package/lib/fileNames.js +136 -0
- package/package.json +4 -1
package/README.md
CHANGED
|
@@ -123,8 +123,8 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
|
|
|
123
123
|
| `emailRead` | `email-read` | List messages, newest first, with `unread_only`, `limit` (1–1000, default 50), `since`, `from_equals`, `from_contains`, `subject_contains`, `body: 'full' \| 'text' \| 'none'`, optional NDJSON spooling. |
|
|
124
124
|
| `emailMove` | `email-move` | Move a message (`messageId`, `folder`) to `toFolder`. |
|
|
125
125
|
| `emailDelete` | `email-delete` | Flag `\Deleted` **and expunge** — permanent, not a move to Trash. |
|
|
126
|
-
| `emailDownloadEmail` | `email-download-email` | Stream the raw message to an `.eml` file
|
|
127
|
-
| `emailDownloadAttachments` | `email-download-attachments` | Save attachments, optionally filtered by `extension` / `filenameContains
|
|
126
|
+
| `emailDownloadEmail` | `email-download-email` | Stream the raw message to an `.eml` file, named by `fileName`. `{ count, saved[], path, existing[] }`. |
|
|
127
|
+
| `emailDownloadAttachments` | `email-download-attachments` | Save attachments, optionally filtered by `extension` / `filenameContains`, named by `fileName` (see [Saving files](#saving-files-names-and-a-name-already-taken)). `{ count, saved[], path, skipped, existing[] }`. |
|
|
128
128
|
| `httpRequest`, `spawnProgram`, `dbQuery`, `fileMove` | — | **Placeholders.** Empty modules with no `name` or `execute`; `register()` rejects them. |
|
|
129
129
|
|
|
130
130
|
### Connections
|
|
@@ -191,7 +191,7 @@ await runAction({ name: 'data-operation', input: {
|
|
|
191
191
|
sort: '-late',
|
|
192
192
|
format: true, title: 'Overdue'
|
|
193
193
|
} })
|
|
194
|
-
// → { rows, count, html, text }
|
|
194
|
+
// → { rows, count, columns, html, text }
|
|
195
195
|
```
|
|
196
196
|
|
|
197
197
|
**Rows and a rendering, both.** `rows` and `count` are always there; `html`
|
|
@@ -199,6 +199,8 @@ and `text` only when `format` is on, because building a table for ten
|
|
|
199
199
|
thousand rows nobody renders is work done and stored on every run. Neither is
|
|
200
200
|
privileged — the next step binds whichever it wants.
|
|
201
201
|
|
|
202
|
+
**`columns` says the order and the headings** (`[{ key, header }]`, the visible ones), in `export-table`'s column shape, so a workflow writes `document: { columns: "{steps.overdue.output.columns}" }`. Why: a step's output is stored as JSONB, which reorders a row's keys (shortest first), so the next step cannot read the column order off a row. A file exported without it came out as `late, amount, invoice, customer`.
|
|
203
|
+
|
|
202
204
|
**`count` is the point of the step as often as `rows` is**: it is what lets
|
|
203
205
|
the step after it branch on *nothing to report* instead of sending an empty
|
|
204
206
|
email.
|
|
@@ -231,9 +233,42 @@ should not take the other eight hundred and ninety-nine with it.
|
|
|
231
233
|
**No aggregation.** `group by status, count` changes what a row *is*; that is
|
|
232
234
|
a different shape and, when it is wanted, a different step.
|
|
233
235
|
|
|
236
|
+
## Saving files: names, and a name already taken
|
|
237
|
+
|
|
238
|
+
Every action that saves a file follows one rule (`lib/fileNames.js`): `email-download-attachments`, `email-download-email` and `export-table`.
|
|
239
|
+
|
|
240
|
+
**`fileName`** (the email steps) or **`outputPath`** (`export-table`) is a template with two kinds of placeholder:
|
|
241
|
+
|
|
242
|
+
| | filled in by | examples |
|
|
243
|
+
|---|---|---|
|
|
244
|
+
| `{…}` | the workflow, before the step runs | `{trigger.window.from\|date:YYYY-MM}`, `{item.from}` |
|
|
245
|
+
| `[…]` | the action, from the file itself | attachments: `[original]` `[name]` `[ext]` `[n]` `[id]`; a whole email: `[subject]` `[id]` `[ext]` |
|
|
246
|
+
|
|
247
|
+
```
|
|
248
|
+
Bank-{trigger.window.from|date:YYYY-MM:Asia/Kolkata}-[original] → Bank-2026-09-statement.csv
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
- **Why two kinds:** the engine fills every `{…}` first, so a file's own placeholders in `{…}` would be blanked before the action saw them.
|
|
252
|
+
- **A placeholder the file does not have** is refused, listing the ones it has. A name that comes out empty is refused.
|
|
253
|
+
- **A name never leaves its folder:** `/ \ : * ? " < > |` become `_`, and `.` or `..` is refused. Spaces are kept.
|
|
254
|
+
- **No `fileName`:** a random prefix and the original name, as before.
|
|
255
|
+
|
|
256
|
+
**`ifExists`**, when the name is already taken:
|
|
257
|
+
|
|
258
|
+
| | |
|
|
259
|
+
|---|---|
|
|
260
|
+
| `fail` (default) | The step fails with `FILE_EXISTS`, naming the file, and **nothing is written**. A second run for the same period is caught, not repeated. |
|
|
261
|
+
| `skip` | Not written; listed in `existing[]` (`export-table`: `skipped: true`), so the next step can branch on "already downloaded". |
|
|
262
|
+
| `overwrite` | Replaced. Two files of one step with one name still get a suffix, so neither is lost. |
|
|
263
|
+
| `addSuffix` | `Bank-2026-09 (2).csv`, the next free number. |
|
|
264
|
+
|
|
265
|
+
Decided for the whole step before anything is written, so `fail` never leaves half a batch.
|
|
266
|
+
|
|
234
267
|
## Export
|
|
235
268
|
|
|
236
|
-
`
|
|
269
|
+
**From a workflow:** `rows` can be plain records, as a fetch or the data step gives them (`{ ref: 'A1', amount: 500 }`). Columns pick them by `key` (default: the header). With no columns, every field of the first row, in order. `outputPath` follows [Saving files](#saving-files-names-and-a-name-already-taken).
|
|
270
|
+
|
|
271
|
+
`export-table` also writes rows that arrive **already presented**: which columns, in what order, under what heading, and for each cell its raw value and the text a reader saw. The action knows nothing of reports or pivots; the caller decides all that (in xeplr-bi, `report-engine`'s `presentTable` does). Each format takes what it needs:
|
|
237
272
|
|
|
238
273
|
| format | writes | from |
|
|
239
274
|
|---|---|---|
|
|
@@ -484,20 +519,26 @@ await driver.query(pool, userSql, params) // reads only
|
|
|
484
519
|
|
|
485
520
|
## Tests
|
|
486
521
|
|
|
487
|
-
There is no `npm test` script. The tests use `node:test`; run a file directly:
|
|
488
|
-
|
|
489
522
|
```sh
|
|
523
|
+
npm test # every folder of test/, one file at a time
|
|
524
|
+
PG_HOST=localhost PG_USER=$USER npm test # the same, with the Postgres suites running
|
|
490
525
|
node --test test/uploader/reconcile.test.js
|
|
491
526
|
```
|
|
492
527
|
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
|
496
|
-
|
|
497
|
-
| |
|
|
498
|
-
|
|
|
528
|
+
**Databases come from the environment, never from code** (`test/helpers/databases.js`):
|
|
529
|
+
|
|
530
|
+
| database | settings | default port |
|
|
531
|
+
|---|---|---|
|
|
532
|
+
| Postgres | `PG_HOST` `PG_PORT` `PG_USER` `PG_PASSWORD` | 5432 |
|
|
533
|
+
| MySQL | `MYSQL_HOST` `MYSQL_PORT` `MYSQL_USER` `MYSQL_PASSWORD` | 3306 |
|
|
534
|
+
| SQL Server | `MSSQL_HOST` `MSSQL_PORT` `MSSQL_USER` `MSSQL_PASSWORD` | 1433 |
|
|
535
|
+
|
|
536
|
+
- **No `*_HOST`:** that database's tests are **skipped**, and each says which settings run it.
|
|
537
|
+
- **`*_HOST` set:** they run, and a login the database refuses **fails** them. A configured run never passes by not connecting.
|
|
538
|
+
- **One file at a time** (`--test-concurrency=1`): the database suites share the `xeplr_actions_test` database and race to create it when run side by side.
|
|
539
|
+
- **Why:** these files used to carry a fixed host, port and password each (Postgres on `:5435`). On any other machine they all failed, so the pre-commit gate and CI ran only `test/*.test.js` and never this package's sub-folders. That is how `data-operation` shipped returning no rows (fixed in 1.4.2).
|
|
499
540
|
|
|
500
|
-
`test/db-move.live.js` runs against a developer's own `.env` path and is not
|
|
541
|
+
Other tests read their own settings: `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`), `test/readonly-connections.test.js` (`PGHOST` …), and `test/dates-across-zones.test.js` skips a database it cannot reach. `test/db-move.live.js` runs against a developer's own `.env` path and is not part of `npm test`.
|
|
501
542
|
|
|
502
543
|
## License
|
|
503
544
|
|
|
@@ -106,7 +106,9 @@ module.exports = {
|
|
|
106
106
|
{ name: 'rows', type: 'array', description: 'What is left, with the columns asked for.' },
|
|
107
107
|
{ name: 'count', type: 'number', description: 'How many rows — so the next step can branch on none.' },
|
|
108
108
|
{ name: 'html', type: 'string', description: 'A table of the visible columns. Only when `format` is on.' },
|
|
109
|
-
{ name: 'text', type: 'string', description: 'The same, as plain text, for a mail client that refuses HTML.' }
|
|
109
|
+
{ name: 'text', type: 'string', description: 'The same, as plain text, for a mail client that refuses HTML.' },
|
|
110
|
+
{ name: 'columns', type: 'array', description: 'The visible columns, in order: [{ key, header }]. Give it to export-table ' +
|
|
111
|
+
'as document.columns: a step\'s stored output does not keep its fields in order.' }
|
|
110
112
|
],
|
|
111
113
|
|
|
112
114
|
// The same call every action gets from runAction: { input, system }. It
|
|
@@ -183,7 +185,15 @@ module.exports = {
|
|
|
183
185
|
});
|
|
184
186
|
}
|
|
185
187
|
|
|
186
|
-
|
|
188
|
+
// THE ORDER AND THE HEADINGS, said out loud. A step's output is stored
|
|
189
|
+
// as JSONB, which reorders an object's keys (shortest first), so the next
|
|
190
|
+
// step cannot read the column order off a row. This is export-table's
|
|
191
|
+
// column shape, so it is handed over as it is.
|
|
192
|
+
var result = {
|
|
193
|
+
rows: rows,
|
|
194
|
+
count: rows.length,
|
|
195
|
+
columns: computed.filter(function(c) { return !c.hidden; }).map(function(c) { return { key: c.name, header: c.label }; })
|
|
196
|
+
};
|
|
187
197
|
|
|
188
198
|
if (input.format) {
|
|
189
199
|
// THE VISIBLE COLUMNS ONLY. A column kept for the next step but not
|
|
@@ -13,11 +13,12 @@
|
|
|
13
13
|
|
|
14
14
|
var emailDrivers = require('../../drivers/email');
|
|
15
15
|
var emailConfig = require('../../drivers/email/config');
|
|
16
|
+
var fileNames = require('../../fileNames');
|
|
16
17
|
|
|
17
18
|
module.exports = {
|
|
18
19
|
name: 'email-download-attachments',
|
|
19
20
|
description: 'Save a message\'s attachments to disk, optionally filtered by extension or ' +
|
|
20
|
-
'filename substring. Returns { count, saved[], path, skipped }.',
|
|
21
|
+
'filename substring, named as you say. Returns { count, saved[], path, skipped, existing[] }.',
|
|
21
22
|
requires: [],
|
|
22
23
|
|
|
23
24
|
// Which mailbox first — it is the only answer that changes what the rest of
|
|
@@ -39,7 +40,7 @@ module.exports = {
|
|
|
39
40
|
description: 'Only attachments with this extension, e.g. "pdf" or ".pdf".' },
|
|
40
41
|
{ name: 'filenameContains', type: 'string', order: 7, group: 'Filters',
|
|
41
42
|
description: 'Only attachments whose filename contains this substring.' }
|
|
42
|
-
],
|
|
43
|
+
].concat(fileNames.nameInputs(8, '[original], [name], [ext], [n] (1, 2, … per attachment), [id] (the message)')),
|
|
43
44
|
|
|
44
45
|
execute: async function(ctx) {
|
|
45
46
|
var input = ctx.input || {};
|
|
@@ -11,6 +11,7 @@
|
|
|
11
11
|
|
|
12
12
|
var emailDrivers = require('../../drivers/email');
|
|
13
13
|
var emailConfig = require('../../drivers/email/config');
|
|
14
|
+
var fileNames = require('../../fileNames');
|
|
14
15
|
|
|
15
16
|
module.exports = {
|
|
16
17
|
name: 'email-download-email',
|
|
@@ -32,7 +33,7 @@ module.exports = {
|
|
|
32
33
|
description: 'Folder the message is currently in — where its id is valid.' },
|
|
33
34
|
{ name: 'outDir', type: 'string', order: 5,
|
|
34
35
|
description: 'Directory to write into. Defaults to XEPLR_ACTIONS_TMP_DIR.' }
|
|
35
|
-
],
|
|
36
|
+
].concat(fileNames.nameInputs(6, '[subject], [id] (the message), [ext] (eml)')),
|
|
36
37
|
|
|
37
38
|
execute: async function(ctx) {
|
|
38
39
|
var input = ctx.input || {};
|
|
@@ -22,6 +22,7 @@ var path = require('path');
|
|
|
22
22
|
var readline = require('readline');
|
|
23
23
|
var writers = require('../../formats/writers');
|
|
24
24
|
var { checkFormatRequires } = require('../../formats');
|
|
25
|
+
var fileNames = require('../../fileNames');
|
|
25
26
|
|
|
26
27
|
function tmpDir() {
|
|
27
28
|
var dir = process.env.XEPLR_ACTIONS_TMP_DIR || path.join(os.tmpdir(), 'xeplr-actions');
|
|
@@ -35,10 +36,28 @@ function fileNameFor(title, extension) {
|
|
|
35
36
|
return base + '.' + extension;
|
|
36
37
|
}
|
|
37
38
|
|
|
38
|
-
|
|
39
|
-
|
|
39
|
+
/**
|
|
40
|
+
* A ROW AS THE WRITERS TAKE IT: { values: [...] }. A plain record — what a
|
|
41
|
+
* fetch or the data step gives, { ref: 'A1', amount: 500 } — is laid out by
|
|
42
|
+
* the columns' `key` (or header), so a workflow can hand its rows straight
|
|
43
|
+
* over instead of reshaping them first.
|
|
44
|
+
*/
|
|
45
|
+
function asRow(row, columns) {
|
|
46
|
+
if (row && Array.isArray(row.values)) return row;
|
|
47
|
+
var r = row || {};
|
|
48
|
+
return { values: columns.map(function(c) { var k = c.key || c.header; return r[k] === undefined ? null : r[k]; }) };
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
async function* rowsFrom(input, columns) {
|
|
52
|
+
if (Array.isArray(input.rows)) { for (var i = 0; i < input.rows.length; i++) yield asRow(input.rows[i], columns); return; }
|
|
40
53
|
var lines = readline.createInterface({ input: fs.createReadStream(input.rowsFile), crlfDelay: Infinity });
|
|
41
|
-
for await (var line of lines) if (line.trim()) yield JSON.parse(line);
|
|
54
|
+
for await (var line of lines) if (line.trim()) yield asRow(JSON.parse(line), columns);
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/** Columns for plain records when none are given: every key of the first one, as it is. */
|
|
58
|
+
function columnsOf(rows) {
|
|
59
|
+
var first = (rows || []).find(function(r) { return r && typeof r === 'object' && !Array.isArray(r.values); });
|
|
60
|
+
return first ? Object.keys(first).map(function(k) { return { header: k, key: k }; }) : [];
|
|
42
61
|
}
|
|
43
62
|
|
|
44
63
|
module.exports = {
|
|
@@ -50,22 +69,29 @@ module.exports = {
|
|
|
50
69
|
inputSchema: [
|
|
51
70
|
{ name: 'format', type: 'string', required: true, order: 1, enum: writers.SUPPORTED,
|
|
52
71
|
description: 'csv, xlsx or pdf.' },
|
|
53
|
-
{ name: 'document', type: 'object',
|
|
54
|
-
description: '{ title?, subtitle?, footer?, columns: [{ header, kind?: number|date|text, numFmt?, align?, width? }] }. ' +
|
|
55
|
-
'numFmt is an Excel number format ("#,##0.00", "0.0%", "mmm yyyy").'
|
|
72
|
+
{ name: 'document', type: 'object', order: 2,
|
|
73
|
+
description: '{ title?, subtitle?, footer?, columns: [{ header, key?, kind?: number|date|text, numFmt?, align?, width? }] }. ' +
|
|
74
|
+
'numFmt is an Excel number format ("#,##0.00", "0.0%", "mmm yyyy"). key: which field of a plain record ' +
|
|
75
|
+
'goes in the column (default: the header). No columns with plain records: every field of the first row.' },
|
|
56
76
|
{ name: 'rows', type: 'array', order: 3,
|
|
57
|
-
description: '
|
|
77
|
+
description: 'Plain records, as a fetch or the data step gives them ({ ref, amount }), or { values: [], display?: [], ' +
|
|
78
|
+
'kind?: detail|subtotal|total }. Or give rowsFile.' },
|
|
58
79
|
{ name: 'rowsFile', type: 'string', order: 4,
|
|
59
80
|
description: 'An NDJSON file with one row per line, instead of rows.' },
|
|
60
81
|
{ name: 'outputPath', type: 'string', order: 5,
|
|
61
|
-
description: 'Where to write.
|
|
82
|
+
description: 'Where to write, e.g. /data/recon/Recon-{trigger.window.from|date:YYYY-MM}.xlsx. ' +
|
|
83
|
+
'Default: a new file in XEPLR_ACTIONS_TMP_DIR named after the title.' },
|
|
84
|
+
{ name: 'ifExists', type: 'string', order: 7, default: 'fail', options: fileNames.IF_EXISTS,
|
|
85
|
+
description: 'When outputPath is already there. fail: the step fails and nothing is written. skip: left alone ' +
|
|
86
|
+
'(skipped: true). overwrite: replaced. addSuffix: "name (2).xlsx".' },
|
|
62
87
|
{ name: 'maxRows', type: 'number', order: 6,
|
|
63
88
|
description: 'PDF only: rows to draw before stopping with a note (default 5,000).' }
|
|
64
89
|
],
|
|
65
90
|
outputSchema: [
|
|
66
91
|
{ name: 'filePath', type: 'string' }, { name: 'fileName', type: 'string' },
|
|
67
92
|
{ name: 'contentType', type: 'string' }, { name: 'bytes', type: 'number' },
|
|
68
|
-
{ name: 'rows', type: 'number' }, { name: 'truncated', type: 'boolean' }
|
|
93
|
+
{ name: 'rows', type: 'number' }, { name: 'truncated', type: 'boolean' },
|
|
94
|
+
{ name: 'skipped', type: 'boolean' }
|
|
69
95
|
],
|
|
70
96
|
|
|
71
97
|
execute: async function(ctx) {
|
|
@@ -74,6 +100,7 @@ module.exports = {
|
|
|
74
100
|
var writer = writers.getWriter(format);
|
|
75
101
|
checkFormatRequires('export-table', format, writer);
|
|
76
102
|
var doc = Object.assign({}, input.document || {});
|
|
103
|
+
if ((!Array.isArray(doc.columns) || !doc.columns.length) && Array.isArray(input.rows)) doc.columns = columnsOf(input.rows);
|
|
77
104
|
if (!Array.isArray(doc.columns) || !doc.columns.length) {
|
|
78
105
|
var bad = new Error('document.columns must list at least one column');
|
|
79
106
|
bad.status = 400;
|
|
@@ -89,10 +116,22 @@ module.exports = {
|
|
|
89
116
|
var info = writers.INFO[format];
|
|
90
117
|
var fileName = fileNameFor(doc.title, info.extension);
|
|
91
118
|
var filePath = input.outputPath || path.join(fs.mkdtempSync(path.join(tmpDir(), 'export-')), fileName);
|
|
119
|
+
if (input.outputPath) {
|
|
120
|
+
// The same rule as every action that saves a file (lib/fileNames.js):
|
|
121
|
+
// a name already taken is never silently replaced.
|
|
122
|
+
var dir = path.dirname(filePath);
|
|
123
|
+
fs.mkdirSync(dir, { recursive: true });
|
|
124
|
+
var place = fileNames.plan(dir, [path.basename(filePath)], input.ifExists)[0];
|
|
125
|
+
if (!place.write) {
|
|
126
|
+
return { filePath: place.path, fileName: place.name, contentType: info.contentType,
|
|
127
|
+
bytes: fs.statSync(place.path).size, rows: 0, truncated: false, skipped: true };
|
|
128
|
+
}
|
|
129
|
+
filePath = place.path;
|
|
130
|
+
}
|
|
92
131
|
var open = writer.open(filePath, doc);
|
|
93
132
|
var signal = ctx.signal;
|
|
94
133
|
try {
|
|
95
|
-
for await (var row of rowsFrom(input)) {
|
|
134
|
+
for await (var row of rowsFrom(input, doc.columns)) {
|
|
96
135
|
if (signal && signal.aborted) throw new Error('Stopped by request');
|
|
97
136
|
var more = await open.writeRow(row);
|
|
98
137
|
// Excel's own limit: the rest are only counted. A PDF keeps counting
|
|
@@ -106,7 +145,8 @@ module.exports = {
|
|
|
106
145
|
contentType: info.contentType,
|
|
107
146
|
bytes: fs.statSync(filePath).size,
|
|
108
147
|
rows: done.rows,
|
|
109
|
-
truncated: done.truncated
|
|
148
|
+
truncated: done.truncated,
|
|
149
|
+
skipped: false
|
|
110
150
|
};
|
|
111
151
|
} catch (err) {
|
|
112
152
|
try { fs.rmSync(filePath, { force: true }); } catch (e) { /* nothing to clean */ }
|
|
@@ -12,6 +12,7 @@ var path = require('path');
|
|
|
12
12
|
var fs = require('fs');
|
|
13
13
|
var fsp = require('fs/promises');
|
|
14
14
|
var os = require('os');
|
|
15
|
+
var fileNames = require('../../fileNames');
|
|
15
16
|
|
|
16
17
|
// Short random prefix. Attachment names collide constantly in mail —
|
|
17
18
|
// "invoice.pdf" from fifty senders — and a download that silently overwrites
|
|
@@ -65,7 +66,74 @@ function emlFilename(subject, uid) {
|
|
|
65
66
|
return shortId() + '_' + base + '.eml';
|
|
66
67
|
}
|
|
67
68
|
|
|
69
|
+
/**
|
|
70
|
+
* Where each attachment goes. Named (input.fileName): every name decided
|
|
71
|
+
* before any file is written (fileNames.plan, with ifExists). Unnamed: a
|
|
72
|
+
* random prefix, as always — attachment names collide.
|
|
73
|
+
* @returns [{ name, path, write }] in the order of `attachments`
|
|
74
|
+
*/
|
|
75
|
+
function attachmentPlaces(dir, attachments, uid, input) {
|
|
76
|
+
if (!input.fileName) {
|
|
77
|
+
return attachments.map(function(a) {
|
|
78
|
+
var n = shortId() + '_' + sanitizeFilename(a.filename);
|
|
79
|
+
return { name: n, path: path.join(dir, n), write: true };
|
|
80
|
+
});
|
|
81
|
+
}
|
|
82
|
+
return fileNames.plan(dir, attachments.map(function(a, i) {
|
|
83
|
+
var p = fileNames.partsOf(a.filename);
|
|
84
|
+
return fileNames.nameFrom(input.fileName, { original: p.original, name: p.name, ext: p.ext, n: i + 1, id: uid });
|
|
85
|
+
}), input.ifExists);
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/** Where a whole message's .eml goes: named by input.fileName, else the subject. */
|
|
89
|
+
function emlPlace(dir, subject, uid, input) {
|
|
90
|
+
if (!input.fileName) {
|
|
91
|
+
var n = emlFilename(subject, uid);
|
|
92
|
+
return { name: n, path: path.join(dir, n), write: true };
|
|
93
|
+
}
|
|
94
|
+
var named = fileNames.nameFrom(input.fileName, { subject: fileNames.safeName(subject) || 'message', id: uid, ext: 'eml' });
|
|
95
|
+
return fileNames.plan(dir, [named], input.ifExists)[0];
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
/**
|
|
99
|
+
* Save a parsed message's attachments that pass the filters, named as asked.
|
|
100
|
+
* @returns { count, saved[], existing[], path, skipped }
|
|
101
|
+
* saved[] entries are { filename, path, size, contentType } — nodemailer's
|
|
102
|
+
* attachment shape, so email-send can forward them as they are.
|
|
103
|
+
*/
|
|
104
|
+
function saveAttachments(dir, all, uid, input) {
|
|
105
|
+
var wanted = all.filter(function(a) { return attachmentMatches(a.filename, input.extension, input.filenameContains); });
|
|
106
|
+
var places = attachmentPlaces(dir, wanted, uid, input);
|
|
107
|
+
var saved = [];
|
|
108
|
+
var existing = [];
|
|
109
|
+
wanted.forEach(function(a, i) {
|
|
110
|
+
var place = places[i];
|
|
111
|
+
if (!place.write) { existing.push({ filename: place.name, path: place.path }); return; }
|
|
112
|
+
fs.writeFileSync(place.path, a.content);
|
|
113
|
+
saved.push({
|
|
114
|
+
filename: input.fileName ? place.name : (a.filename || place.name),
|
|
115
|
+
path: place.path,
|
|
116
|
+
size: a.size || (a.content && a.content.length) || null,
|
|
117
|
+
contentType: a.contentType || null
|
|
118
|
+
});
|
|
119
|
+
});
|
|
120
|
+
return {
|
|
121
|
+
count: saved.length,
|
|
122
|
+
saved: saved,
|
|
123
|
+
// Already there and left alone (ifExists: skip), so the next step can
|
|
124
|
+
// tell "nothing new" from "nothing attached".
|
|
125
|
+
existing: existing,
|
|
126
|
+
path: saved.length ? saved[0].path : null,
|
|
127
|
+
// Reported rather than silent: "0 attachments" and "6 attachments, none
|
|
128
|
+
// matched your filter" are different problems with the same count.
|
|
129
|
+
skipped: all.length - wanted.length
|
|
130
|
+
};
|
|
131
|
+
}
|
|
132
|
+
|
|
68
133
|
module.exports = {
|
|
134
|
+
saveAttachments: saveAttachments,
|
|
135
|
+
attachmentPlaces: attachmentPlaces,
|
|
136
|
+
emlPlace: emlPlace,
|
|
69
137
|
shortId: shortId,
|
|
70
138
|
sanitizeFilename: sanitizeFilename,
|
|
71
139
|
resolveOutDir: resolveOutDir,
|
|
@@ -317,12 +317,17 @@ async function downloadEmail(connection, input) {
|
|
|
317
317
|
subject = (meta && meta.envelope && meta.envelope.subject) || null;
|
|
318
318
|
} catch (_) { /* filename falls back to the uid — never fatal */ }
|
|
319
319
|
|
|
320
|
-
var dl = await client.download(uid, undefined, { uid: true });
|
|
321
|
-
|
|
322
320
|
var dir = await files.resolveOutDir(input.outDir);
|
|
323
|
-
|
|
324
|
-
|
|
321
|
+
// Decided before the download: a name already taken under `fail` costs
|
|
322
|
+
// nothing, and under `skip` nothing is fetched at all.
|
|
323
|
+
var where = files.emlPlace(dir, subject, uid, input);
|
|
324
|
+
if (!where.write) {
|
|
325
|
+
return { count: 0, saved: [], path: null, existing: [{ filename: where.name, path: where.path }], subject: subject, messageId: uid };
|
|
326
|
+
}
|
|
327
|
+
var filename = where.name;
|
|
328
|
+
var filepath = where.path;
|
|
325
329
|
|
|
330
|
+
var dl = await client.download(uid, undefined, { uid: true });
|
|
326
331
|
await pipeline(dl.content, files.fs.createWriteStream(filepath));
|
|
327
332
|
|
|
328
333
|
var size = files.fs.statSync(filepath).size;
|
|
@@ -332,6 +337,7 @@ async function downloadEmail(connection, input) {
|
|
|
332
337
|
// `path` / `saved[]` identically whichever produced the file.
|
|
333
338
|
saved: [{ filename: filename, path: filepath, size: size, contentType: 'message/rfc822' }],
|
|
334
339
|
path: filepath,
|
|
340
|
+
existing: [],
|
|
335
341
|
subject: subject,
|
|
336
342
|
messageId: uid
|
|
337
343
|
};
|
|
@@ -353,35 +359,11 @@ async function downloadAttachments(connection, input) {
|
|
|
353
359
|
return withMailbox('email-download-attachments', connection, input.folder || 'INBOX', async function(client) {
|
|
354
360
|
var dl = await client.download(uid, undefined, { uid: true });
|
|
355
361
|
var parsed = await simpleParser(dl.content);
|
|
356
|
-
var all = parsed.attachments || [];
|
|
357
362
|
|
|
358
363
|
var dir = await files.resolveOutDir(input.outDir);
|
|
359
|
-
var
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
for (var i = 0; i < all.length; i++) {
|
|
363
|
-
var a = all[i];
|
|
364
|
-
if (!files.attachmentMatches(a.filename, input.extension, input.filenameContains)) { skipped++; continue; }
|
|
365
|
-
var filename = files.shortId() + '_' + files.sanitizeFilename(a.filename);
|
|
366
|
-
var filepath = files.path.join(dir, filename);
|
|
367
|
-
files.fs.writeFileSync(filepath, a.content);
|
|
368
|
-
saved.push({
|
|
369
|
-
filename: a.filename || filename,
|
|
370
|
-
path: filepath,
|
|
371
|
-
size: a.size || (a.content && a.content.length) || null,
|
|
372
|
-
contentType: a.contentType || null
|
|
373
|
-
});
|
|
374
|
-
}
|
|
375
|
-
|
|
376
|
-
return {
|
|
377
|
-
count: saved.length,
|
|
378
|
-
saved: saved,
|
|
379
|
-
path: saved.length ? saved[0].path : null,
|
|
380
|
-
// Reported rather than silent: "0 attachments" and "6 attachments, none
|
|
381
|
-
// matched your filter" are different problems with the same count.
|
|
382
|
-
skipped: skipped,
|
|
383
|
-
messageId: uid
|
|
384
|
-
};
|
|
364
|
+
var out = files.saveAttachments(dir, parsed.attachments || [], uid, input);
|
|
365
|
+
out.messageId = uid;
|
|
366
|
+
return out;
|
|
385
367
|
});
|
|
386
368
|
}
|
|
387
369
|
|
package/lib/fileNames.js
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
// WHAT A SAVED FILE IS CALLED, AND WHAT HAPPENS WHEN ONE IS ALREADY THERE.
|
|
2
|
+
//
|
|
3
|
+
// One rule for every action that writes a file, so a workflow can say
|
|
4
|
+
// "Bank-2026-09.csv" instead of getting "3f9a1c02e4_statement.csv".
|
|
5
|
+
//
|
|
6
|
+
// A name is a template with two kinds of placeholder:
|
|
7
|
+
// {…} from the WORKFLOW, filled in by the engine before the step runs:
|
|
8
|
+
// Bank-{trigger.window.from|date:YYYY-MM}
|
|
9
|
+
// […] from the FILE ITSELF, filled in here: [original] [name] [ext] [n]
|
|
10
|
+
// and whatever the action adds (an email's [subject], [id]).
|
|
11
|
+
// Square brackets because the engine fills every {…} first, and would blank
|
|
12
|
+
// a file's own placeholders before the action ever saw them.
|
|
13
|
+
//
|
|
14
|
+
// ifExists, when the name is already taken:
|
|
15
|
+
// fail (default) the step fails, naming the file: nothing is written,
|
|
16
|
+
// so a second run for the same period is caught, not repeated
|
|
17
|
+
// skip nothing is written for it, and it is listed in `existing`, so
|
|
18
|
+
// the next step can branch on "already downloaded"
|
|
19
|
+
// overwrite replaced
|
|
20
|
+
// addSuffix "Bank-2026-09 (2).csv"
|
|
21
|
+
// Decided for the whole step before anything is written, so `fail` never
|
|
22
|
+
// leaves half a batch behind.
|
|
23
|
+
|
|
24
|
+
var fs = require('fs');
|
|
25
|
+
var path = require('path');
|
|
26
|
+
|
|
27
|
+
var IF_EXISTS = ['fail', 'skip', 'overwrite', 'addSuffix'];
|
|
28
|
+
|
|
29
|
+
function nameError(message, code) {
|
|
30
|
+
var err = new Error(message);
|
|
31
|
+
err.code = code || 'FILE_NAME_INVALID';
|
|
32
|
+
err.status = 400;
|
|
33
|
+
return err;
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* A name that is safe to join to a directory: no separators, no characters
|
|
38
|
+
* a filesystem refuses, not "." or "..". Spaces are kept — "Bank Sep 2026.csv"
|
|
39
|
+
* is a name a person wants.
|
|
40
|
+
*/
|
|
41
|
+
function safeName(name) {
|
|
42
|
+
var out = String(name == null ? '' : name)
|
|
43
|
+
.replace(/[\/\\:*?"<>|\u0000-\u001f]+/g, '_')
|
|
44
|
+
.replace(/\s+/g, ' ')
|
|
45
|
+
.trim()
|
|
46
|
+
.slice(0, 200);
|
|
47
|
+
if (!out || /^\.+$/.test(out)) return '';
|
|
48
|
+
return out;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
/** The parts of an original file name: [original], [name], [ext]. */
|
|
52
|
+
function partsOf(original) {
|
|
53
|
+
var o = safeName(original) || 'file';
|
|
54
|
+
var dot = o.lastIndexOf('.');
|
|
55
|
+
var hasExt = dot > 0 && dot < o.length - 1;
|
|
56
|
+
return { original: o, name: hasExt ? o.slice(0, dot) : o, ext: hasExt ? o.slice(dot + 1) : '' };
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/**
|
|
60
|
+
* The name from `template` and the file's own `tokens`. A placeholder the
|
|
61
|
+
* file does not have is refused, listing the ones it does — a typo must not
|
|
62
|
+
* quietly drop part of a name.
|
|
63
|
+
*/
|
|
64
|
+
function nameFrom(template, tokens) {
|
|
65
|
+
var unknown = [];
|
|
66
|
+
var out = String(template).replace(/\[([A-Za-z]+)\]/g, function(whole, key) {
|
|
67
|
+
if (!Object.prototype.hasOwnProperty.call(tokens, key)) { unknown.push(key); return whole; }
|
|
68
|
+
var v = tokens[key];
|
|
69
|
+
return v === null || v === undefined ? '' : String(v);
|
|
70
|
+
});
|
|
71
|
+
if (unknown.length) {
|
|
72
|
+
throw nameError('The file name "' + template + '" uses [' + unknown.join('], [') + '], which this file does not have. ' +
|
|
73
|
+
'It has: [' + Object.keys(tokens).join('], [') + '].');
|
|
74
|
+
}
|
|
75
|
+
var safe = safeName(out);
|
|
76
|
+
if (!safe) throw nameError('The file name "' + template + '" comes out empty. Check the values it is made from.');
|
|
77
|
+
return safe;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
function withSuffix(name, k) {
|
|
81
|
+
var dot = name.lastIndexOf('.');
|
|
82
|
+
return dot > 0 ? name.slice(0, dot) + ' (' + k + ')' + name.slice(dot) : name + ' (' + k + ')';
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
/**
|
|
86
|
+
* Where each file goes, decided for the whole step before anything is
|
|
87
|
+
* written. `names` in order; returns [{ name, path, write }] where `write`
|
|
88
|
+
* is false for one skipped. Throws FILE_EXISTS for `fail`.
|
|
89
|
+
*/
|
|
90
|
+
function plan(dir, names, ifExists) {
|
|
91
|
+
var mode = ifExists || 'fail';
|
|
92
|
+
if (IF_EXISTS.indexOf(mode) === -1) {
|
|
93
|
+
throw nameError('ifExists is one of ' + IF_EXISTS.join(', ') + ', not "' + mode + '".');
|
|
94
|
+
}
|
|
95
|
+
var taken = {};
|
|
96
|
+
var exists = function(n) { return taken[n] || fs.existsSync(path.join(dir, n)); };
|
|
97
|
+
return names.map(function(name) {
|
|
98
|
+
var out = name;
|
|
99
|
+
if (exists(out)) {
|
|
100
|
+
if (mode === 'fail') {
|
|
101
|
+
throw nameError('"' + out + '" is already in ' + dir + '. Nothing was saved. ' +
|
|
102
|
+
'Set ifExists to skip, overwrite or addSuffix to save anyway.', 'FILE_EXISTS');
|
|
103
|
+
}
|
|
104
|
+
if (mode === 'skip') return { name: out, path: path.join(dir, out), write: false };
|
|
105
|
+
if (mode === 'addSuffix') {
|
|
106
|
+
var k = 2;
|
|
107
|
+
while (exists(withSuffix(name, k))) k++;
|
|
108
|
+
out = withSuffix(name, k);
|
|
109
|
+
}
|
|
110
|
+
// overwrite: the same name, written again. Two files of THIS step with
|
|
111
|
+
// one name still get a suffix: overwriting its own first file would lose it.
|
|
112
|
+
if (mode === 'overwrite' && taken[out]) {
|
|
113
|
+
var j = 2;
|
|
114
|
+
while (taken[withSuffix(name, j)]) j++;
|
|
115
|
+
out = withSuffix(name, j);
|
|
116
|
+
}
|
|
117
|
+
}
|
|
118
|
+
taken[out] = true;
|
|
119
|
+
return { name: out, path: path.join(dir, out), write: true };
|
|
120
|
+
});
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
/** The inputs every file-writing action takes, to spread into its inputSchema. */
|
|
124
|
+
function nameInputs(order, tokensHelp) {
|
|
125
|
+
return [
|
|
126
|
+
{ name: 'fileName', type: 'string', order: order, group: 'Saving',
|
|
127
|
+
description: 'What to call the file. {…} comes from the workflow, e.g. Bank-{trigger.window.from|date:YYYY-MM}; ' +
|
|
128
|
+
'[…] from the file: ' + tokensHelp + '. Empty: a random prefix and the original name.' },
|
|
129
|
+
{ name: 'ifExists', type: 'string', order: order + 1, group: 'Saving', default: 'fail',
|
|
130
|
+
options: IF_EXISTS,
|
|
131
|
+
description: 'When the name is taken. fail: the step fails and nothing is saved. skip: listed in `existing`, not saved. ' +
|
|
132
|
+
'overwrite: replaced. addSuffix: "name (2).ext". Only with a file name.' }
|
|
133
|
+
];
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
module.exports = { safeName: safeName, partsOf: partsOf, nameFrom: nameFrom, plan: plan, nameInputs: nameInputs, IF_EXISTS: IF_EXISTS };
|
package/package.json
CHANGED
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@xeplr/actions",
|
|
3
|
-
"version": "1.4.
|
|
3
|
+
"version": "1.4.5",
|
|
4
4
|
"description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
|
|
5
5
|
"main": "index.js",
|
|
6
|
+
"scripts": {
|
|
7
|
+
"test": "node --test --test-concurrency=1 \"test/**/*.test.js\""
|
|
8
|
+
},
|
|
6
9
|
"files": [
|
|
7
10
|
"index.js",
|
|
8
11
|
"lib/"
|