@xeplr/imports 0.0.0-stage → 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Xeplr
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,125 @@
1
- # Temporary Holding Version
1
+ # @xeplr/imports
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Data imports as a drop-in module. Upload a CSV, Excel or JSON file and check it against the table it is going into. Then load it as a **New table**, an **Append** or a **Replace**. Afterwards you can see and download the rows the database refused, and roll the import back. One package holds the routes, the access catalog and the settings. The screen is [`ImportsPage` in `@xeplr/ui-utils`](https://www.npmjs.com/package/@xeplr/ui-utils).
4
+
5
+ ## What you get
6
+
7
+ - **Big files, no waiting.** CSV, Excel and JSON files of hundreds of megabytes upload and load in the background, with live progress and a stop button.
8
+ - **New table, append or replace.** You choose what happens to the table. A typo in a table's name is caught, so it never turns into a new table.
9
+ - **Replace without risk.** The table is only swapped once every row has loaded. An import that fails or is stopped leaves your data exactly as it was.
10
+ - **Checked before it loads.** The file's columns are compared with the table first. You see what will be refused, what will be left empty and what matches, before anything is touched.
11
+ - **Copy-paste damage repaired.** Rows that a paste from another system folded into one cell are found and put back.
12
+ - **Nothing silently lost.** Every row the database refuses is kept with the reason. View it, copy it, or download it as Excel to fix and load again.
13
+ - **Full history and undo.** Every import is recorded with who ran it, where it went and how it went, and can be rolled back.
14
+ - **Drop-in.** One router and one screen, and the app supplies only its database connections.
15
+
16
+ Every step underneath is [`@xeplr/actions`](https://www.npmjs.com/package/@xeplr/actions): its file readers, the uploader, the error table and staged loads. This package puts those steps behind HTTP and leaves the app only what is its own.
17
+
18
+ ## Using it
19
+
20
+ ```js
21
+ var imports = require('@xeplr/imports');
22
+
23
+ // routes — mounted at /imports (the access catalog assumes that path)
24
+ router.use('/imports', imports.router({
25
+ auth: auth.mtMembershipGate, // the app's gate, run before every route
26
+ service: 'my-app', // names this app's import records
27
+ resolveTarget: async function(connectionId, dbInfoId, req) {
28
+ // the database a file loads into, as a login the @xeplr/actions drivers take
29
+ return { dbType: 'postgres', dbConnection: { host, port, user, password, database } }; // or null
30
+ },
31
+ listTargets: async function(req) {
32
+ // the databases this caller may import into, for the picker and for history's labels
33
+ return [{ connectionId: 'c1', label: 'Warehouse', databases: [{ id: 'd1', label: 'sales' }] }];
34
+ }
35
+ }));
36
+ ```
37
+
38
+ ```js
39
+ // env.required.js
40
+ module.exports = [].concat(imports.requiredEnv /* , … */);
41
+ ```
42
+
43
+ ```
44
+ # .env
45
+ UPLOAD_DIR=/data/my-app/uploads # required, absolute
46
+ XCFG_DB_NAME=xeplr_configs # required: where import records live
47
+ XEPLR_AUTH_MIGRATIONS=…,/…/node_modules/@xeplr/imports/migrations-auth
48
+ ```
49
+
50
+ On the screen side, add one route for `<ImportsPage/>` and a drawer entry. That is the whole of it.
51
+
52
+ ### What the app supplies, and why only that
53
+
54
+ | | why it is the app's |
55
+ |---|---|
56
+ | `auth` | who may call these routes is decided by the app's sign-in, as for every other route |
57
+ | `resolveTarget(connectionId, dbInfoId, req)` | where the app keeps its connections, and which of them a caller may use, is its own business. Return `null` for one the caller may not use, and the route answers 404 |
58
+ | `listTargets(req)` | the same connections, as a list for the picker |
59
+ | `service` | names this app's rows in the shared `import_meta` table. Or pass `config`, an `attachConfig()` instance the app already has |
60
+ | `userNames(ids)` (optional) | names for records made before the importer's name was stored on them |
61
+
62
+ Tenancy comes from the app's `mtMiddleware`, as it does for its other routes. Each import record carries `mtId1`/`mtId2`, and history reads them back through the tenant filter, so another company's or workspace's imports are simply not found. Import records live in `xeplr_configs` and are application-scoped (`registerApplication()` at startup).
63
+
64
+ Database drivers are the app's own: install `pg`, `mysql2`, `mssql` or `@duckdb/node-api` for each kind of database it imports into. The file readers (CSV, Excel) come with this package.
65
+
66
+ ## Settings
67
+
68
+ | Variable | Required | Meaning |
69
+ |---|---|---|
70
+ | `UPLOAD_DIR` | **yes**, absolute | Where uploaded files are staged until imported, and where a dropped-rows download is written for the moment it is sent. Never defaulted: a missing or relative one would make the same install read different files depending on where it was started. `router()` refuses to start without it. |
71
+ | `XCFG_DB_NAME` | **yes** | The `xeplr_configs` database holding `import_meta`. The login is `XCFG_DB_CONNECTION_INFO_ENCRYPTED`, or the shared `XEPLR_DB_CONNECTION` (`@xeplr/db`) |
72
+ | `UPLOAD_MAX_MB` | no | The largest file accepted, default 500. Files go to disk and are streamed, so this costs disk, not memory. A proxy in front (nginx `client_max_body_size`) must allow at least as much |
73
+ | `IMPORT_TIME_LIMIT_MINUTES` | no | How long an import may run before it is stopped, default 60 |
74
+
75
+ ## Routes
76
+
77
+ Mounted at `/imports`:
78
+
79
+ | Method | Path | |
80
+ |---|---|---|
81
+ | POST | `/files` | The file (multipart, field `file`) → `{ filePath, fileName, originalName, size }` |
82
+ | POST | `/inspect` | `{ filePath }` → `{ format, sheets?, checks }`. `checks` lists what is wrong before anything loads: rows pasted into one cell, with "Recover them" as a choice |
83
+ | POST | `/check-target` | `{ filePath, sheet?, connectionId, dbInfoId, targetTable }` → `{ exists, matched, extra, missing: [{ name, required }] }` |
84
+ | POST | `/commit` | `{ filePath, connectionId, dbInfoId, originalFileName?, mappings: [{ sheet?, targetTable, mode, repairPastedRows?, primaryKeys? }] }` → **202** `{ runId }`. The import runs in the background. The same file is never imported twice at once (409) |
85
+ | GET | `/runs/:id` | `{ status: running \| complete \| error \| stopped, progress, result: { results }, error }`. A run is shown only to the person who started it. Kept an hour after it ends |
86
+ | POST | `/runs/:id/stop` | Stops it within a batch |
87
+ | GET | `/targets` | `listTargets` |
88
+ | GET | `/history` | Past imports, newest first: `{ id, originalFileName, part, targetTable, destination, status, mode, totalRows, rowsLoaded, rowsDropped, errorMessage, createdByName, startedAt }` |
89
+ | GET | `/history/:id/dropped` | The rows the database refused, from `<table>_import_errors`: `{ columns, rows: [{ rowNum, reason, values }], total, dropped }`, the first 1,000 |
90
+ | GET | `/history/:id/dropped/file` | The same rows as .xlsx: the file's columns only, header in row 1, so it imports again once fixed. The reason is not a column, because it would be refused on the way back in |
91
+ | POST | `/history/:id/rollback` | Deletes what the import loaded |
92
+ | POST | `/history/:id/delete` | Removes the record only. The data stays |
93
+
94
+ A commit result per part is `{ targetTable, sheet, success, rows, dropped, jobId, recovered, error }`.
95
+
96
+ ## The rules
97
+
98
+ - **Modes.** A **New table** must not exist yet. **Append** and **Replace** need one that does, so a typo is never a fresh table. A **Replace** is staged: the rows load into a staging table, then one transaction empties the table and moves them in. A Replace that fails or is stopped leaves the table as it was. See [`@xeplr/actions` Import modes](https://www.npmjs.com/package/@xeplr/actions#import-modes).
99
+ - **Rollback is refused for a Replace**, because the rows it replaced are gone. It is also refused for an upsert (`primaryKeys`), which may only have updated rows that already existed.
100
+ - **Refused rows are kept.** Each row the database refuses goes to `<table>_import_errors` with its place in the file and the reason, so it can be seen, fixed and loaded again. An import gives up after five refusals in a row, and rows after that are counted but not kept.
101
+ - **A filePath is checked.** It comes from the client, so only a path inside `UPLOAD_DIR` is accepted. A file the import cannot read is removed at inspect, and an imported file is removed when its import ends.
102
+
103
+ ## Access
104
+
105
+ `migrations-auth/0001_imports_access.sql` registers every route by group: `imports:view` (history, dropped rows, runs, targets), `imports:create` (upload, inspect, check, commit, stop) and `imports:edit` (rollback, delete). It maps all of them to Super Admin and CompanyAdmin. Loading data into a company's databases is an administrator's job, so Creator and Viewer get none. An app that wants them to can map those roles in its own migration. Inserts are idempotent.
106
+
107
+ ## Tests
108
+
109
+ ```
110
+ PG_HOST=… PG_PORT=… PG_USER=… PG_PASSWORD=… PG_DATABASE=… npm test
111
+ ```
112
+
113
+ The real router runs in an Express app, against real Postgres and a throwaway `xeplr_config_imports_test`. The tests cover:
114
+
115
+ - refused paths and types
116
+ - pasted rows found and recovered
117
+ - each mode and its refusals
118
+ - refused rows shown and downloaded
119
+ - history scoped to a workspace
120
+ - rollback, with a Replace refused
121
+ - delete
122
+ - a stopped Replace leaving the table untouched
123
+ - runs that only their importer can see
124
+
125
+ Without `PG_HOST`, only the startup checks run.
package/index.js ADDED
@@ -0,0 +1,26 @@
1
+ // @xeplr/imports — data imports as a drop-in module. See README.md.
2
+ //
3
+ // var imports = require('@xeplr/imports');
4
+ // router.use('/imports', imports.router({ auth, resolveTarget, listTargets, service: 'my-app' }));
5
+ //
6
+ // plus `imports.requiredEnv` in the app's env.required.js and
7
+ // `imports.authMigrationsDir` in XEPLR_AUTH_MIGRATIONS.
8
+
9
+ var path = require('path');
10
+ var { requiredEnv: configRequiredEnv } = require('@xeplr/actions/lib/uploader/attach');
11
+
12
+ // What this package needs from the environment, for the app's checkEnv.
13
+ // UPLOAD_DIR: where uploaded files are staged — absolute, never defaulted (a
14
+ // wrong-but-present location reads as "no file", not as an error). The
15
+ // import records live in xeplr_configs (XCFG_DB_NAME, and the login in
16
+ // XCFG_DB_CONNECTION_INFO_ENCRYPTED or the shared XEPLR_DB_CONNECTION).
17
+ var requiredEnv = ['UPLOAD_DIR'].concat(configRequiredEnv);
18
+
19
+ module.exports = {
20
+ router: require('./lib/router'),
21
+ requiredEnv: requiredEnv,
22
+ // This package's access catalog (apis, role mappings) — add it to the app's
23
+ // XEPLR_AUTH_MIGRATIONS so its auth database knows these routes.
24
+ authMigrationsDir: path.join(__dirname, 'migrations-auth'),
25
+ MODES: require('./lib/imports').MODES
26
+ };
package/lib/imports.js ADDED
@@ -0,0 +1,232 @@
1
+ // WHAT AN IMPORT DOES, apart from HTTP: what a file holds, how it compares
2
+ // with the table it names, loading it, and reading back what it refused.
3
+ // Every step is @xeplr/actions — this file only puts them in order. The
4
+ // routes (router.js) call these; nothing here knows about a request.
5
+
6
+ var fs = require('fs');
7
+ var path = require('path');
8
+ var { runAction, builtins, formats, uploader, drivers } = require('@xeplr/actions');
9
+
10
+ var EXT_TO_FORMAT = { '.csv': 'csv', '.json': 'json', '.xlsx': 'excel', '.xls': 'excel' };
11
+
12
+ // Rows the dropped-rows view shows at once, and the most a download carries
13
+ // (Excel's own limit — past it the sheet would be cut anyway).
14
+ var DROPPED_VIEW_ROWS = 1000;
15
+ var DROPPED_FILE_ROWS = 1048575;
16
+
17
+ /**
18
+ * Where uploads are staged and how big one may be, from the environment.
19
+ * UPLOAD_DIR is required and absolute: a missing one would default to
20
+ * somewhere, and a relative one resolves against wherever the process was
21
+ * started — the same install reading different files depending on that.
22
+ */
23
+ function settings() {
24
+ var dir = process.env.UPLOAD_DIR;
25
+ if (!dir) throw new Error('@xeplr/imports: UPLOAD_DIR is not set — the absolute directory uploaded files are staged in');
26
+ if (!path.isAbsolute(dir)) throw new Error('@xeplr/imports: UPLOAD_DIR must be an absolute path, got "' + dir + '"');
27
+ fs.mkdirSync(dir, { recursive: true });
28
+ return {
29
+ uploadDir: path.resolve(dir),
30
+ // Client exports run to a few hundred MB (a 773k-row ERP CSV is 270 MB);
31
+ // the file goes to disk and is streamed, so the size costs disk, not memory.
32
+ maxUploadMb: Number(process.env.UPLOAD_MAX_MB) || 500,
33
+ timeLimitMs: (Number(process.env.IMPORT_TIME_LIMIT_MINUTES) || 60) * 60 * 1000
34
+ };
35
+ }
36
+
37
+ function formatOf(filename) {
38
+ return EXT_TO_FORMAT[path.extname(filename).toLowerCase()] || null;
39
+ }
40
+
41
+ /**
42
+ * A filePath comes from the client, relaying what the upload answered — so
43
+ * only one inside the upload directory is accepted, never an arbitrary file
44
+ * on the server.
45
+ */
46
+ function isWithin(dir, filePath) {
47
+ var resolved = path.resolve(String(filePath || ''));
48
+ return resolved === dir || resolved.indexOf(dir + path.sep) === 0;
49
+ }
50
+
51
+ function listSheets(filePath) {
52
+ return formats.getFormat('excel').listSheets(filePath);
53
+ }
54
+
55
+ /**
56
+ * What is wrong with the file before anything is loaded — today, rows a paste
57
+ * folded into one cell (see @xeplr/actions formats/pastedRows.js), with the
58
+ * fix offered as a choice. Per sheet for Excel, once ('') for a CSV.
59
+ *
60
+ * @returns {{ [part]: [{ key, message, fixLabel, examples }] }}
61
+ */
62
+ async function inspectChecks(filePath, format, sheets) {
63
+ var checks = {};
64
+ if (format !== 'excel' && format !== 'csv') return checks;
65
+ var parts = format === 'excel' ? (sheets || []) : [null];
66
+ for (var i = 0; i < parts.length; i++) {
67
+ var found = await formats.scanPastedRows(format, filePath, parts[i] ? { sheet: parts[i], examples: 3 } : { examples: 3 });
68
+ if (!found.cells) continue;
69
+ checks[parts[i] == null ? '' : parts[i]] = [{
70
+ key: 'pastedRows',
71
+ message: plural(found.rows, 'row') + (found.rows === 1 ? ' was' : ' were') + ' pasted into ' + plural(found.cells, 'cell') +
72
+ ' — they exist nowhere else in the file. This happens when data copied out of another system is pasted into Excel and a value starts with a quote mark (an inch mark).',
73
+ fixLabel: 'Recover them as ' + plural(found.rows, 'separate row'),
74
+ examples: found.examples.map(function(e) {
75
+ return 'Row ' + e.row + ', ' + e.column + ': holds ' + plural(e.rows, 'more row') + ' — begins with ' + e.first;
76
+ })
77
+ }];
78
+ }
79
+ return checks;
80
+ }
81
+
82
+ /**
83
+ * One part of a file against the table it names — said before loading: does
84
+ * the table exist, which file columns does it lack (the import would be
85
+ * refused), which of its columns does the file lack (left empty, or every row
86
+ * refused when required).
87
+ *
88
+ * @returns {Promise<{ exists, matched, extra, missing: [{ name, required }] }>}
89
+ */
90
+ async function checkTarget(opts) {
91
+ var columns = await formats.columnsOf(opts.format, opts.filePath, opts.sheet ? { sheet: opts.sheet } : {});
92
+ return uploader.compareToTable({
93
+ driver: drivers.db.getDriver(opts.dbType),
94
+ connection: opts.dbConnection,
95
+ targetTable: opts.targetTable,
96
+ columns: columns
97
+ });
98
+ }
99
+
100
+ /**
101
+ * Load one part of a file into one table: the file-upload action, which
102
+ * records the import (import_meta) before writing a row. Never throws — the
103
+ * caller reports each part's outcome without stopping the others.
104
+ */
105
+ async function loadPart(opts, metaStore) {
106
+ var outcome = await runAction(builtins.fileUpload, {
107
+ sourceType: 'local',
108
+ sourcePath: opts.filePath,
109
+ format: opts.format,
110
+ formatConfig: Object.assign(opts.sheet ? { sheet: opts.sheet } : {}, opts.repairPastedRows ? { repairPastedRows: true } : {}),
111
+ dbType: opts.dbType,
112
+ dbConnection: opts.dbConnection,
113
+ targetTable: opts.targetTable,
114
+ primaryKeys: opts.primaryKeys,
115
+ mode: opts.mode || undefined,
116
+ movementId: opts.movementId
117
+ }, {
118
+ system: { mtId1: opts.mtId1, mtId2: opts.mtId2, details: opts.details, onProgress: opts.onProgress || null, signal: opts.signal || null },
119
+ metaStore: metaStore
120
+ });
121
+ // An action that failed before its own ending was recorded would leave the
122
+ // import 'running' forever. recordEnd is an upsert, so this is harmless when
123
+ // the uploader already closed it.
124
+ if (outcome.status !== 'success') {
125
+ await metaStore.recordEnd(opts.movementId, {
126
+ status: 'failed',
127
+ error: outcome.error && outcome.error.message,
128
+ targetTable: opts.targetTable, connectionKey: opts.targetTable, dbType: opts.dbType,
129
+ mtId1: opts.mtId1, mtId2: opts.mtId2, details: opts.details
130
+ });
131
+ }
132
+ return outcome;
133
+ }
134
+
135
+ /** Undo an import: every row it loaded, and its refused rows. See uploader.rollback for what it refuses. */
136
+ function rollback(opts, metaStore) {
137
+ return uploader.rollback({
138
+ movementId: opts.movementId,
139
+ driver: drivers.db.getDriver(opts.dbType),
140
+ connection: opts.dbConnection,
141
+ targetTable: opts.targetTable,
142
+ metaStore: metaStore
143
+ });
144
+ }
145
+
146
+ /**
147
+ * The rows an import dropped — refused by the database — from
148
+ * `<table>_import_errors`, in the file's column order (`columns`, the
149
+ * import's own record). An import made before refused rows were kept has
150
+ * only its count: total 0.
151
+ *
152
+ * @returns {Promise<{ columns: string[], rows: [{ rowNum, reason, values }], total }>}
153
+ */
154
+ async function droppedRows(opts) {
155
+ var driver = drivers.db.getDriver(opts.dbType);
156
+ var pool = await driver.connect(opts.dbConnection);
157
+ try {
158
+ var errTable = opts.targetTable + '_import_errors';
159
+ var columns = (opts.columns || []).map(function(c) { return typeof c === 'string' ? c : c.name; })
160
+ .filter(function(n) { return n && n.indexOf('__xeplr_') !== 0; });
161
+ var schema = await driver.getTableSchema(pool, errTable);
162
+ if (!schema || !schema.length) return { columns: columns, rows: [], total: 0 };
163
+
164
+ var mark = require('@xeplr/actions/lib/drivers/db/procedure').placeholders(opts.dbType)(0);
165
+ var t = driver.quoteIdent(errTable);
166
+ var counted = await driver.query(pool, 'SELECT COUNT(*) AS n FROM ' + t + ' WHERE movement_id = ' + mark, [opts.movementId]);
167
+ var total = Number(counted.rows[0].n || counted.rows[0].N || 0);
168
+
169
+ var limit = Math.max(1, Math.min(Number(opts.limit) || DROPPED_VIEW_ROWS, DROPPED_FILE_ROWS));
170
+ var top = opts.dbType === 'mssql' ? 'TOP (' + limit + ') ' : '';
171
+ var tail = opts.dbType === 'mssql' ? '' : ' LIMIT ' + limit;
172
+ var found = await driver.query(pool, 'SELECT ' + top + 'row_num, error_description, raw_row FROM ' + t +
173
+ ' WHERE movement_id = ' + mark + ' ORDER BY row_num' + tail, [opts.movementId]);
174
+
175
+ var rows = found.rows.map(function(r) {
176
+ var values = r.raw_row;
177
+ if (typeof values === 'string') { try { values = JSON.parse(values); } catch (_) { values = {}; } }
178
+ values = values || {};
179
+ // A value the import's record does not name is shown, not lost.
180
+ Object.keys(values).forEach(function(k) { if (columns.indexOf(k) === -1) columns.push(k); });
181
+ return { rowNum: r.row_num == null ? null : Number(r.row_num), reason: r.error_description || '', values: values };
182
+ });
183
+ return { columns: columns, rows: rows, total: total };
184
+ } finally {
185
+ await driver.close(pool);
186
+ }
187
+ }
188
+
189
+ /**
190
+ * droppedRows' answer as an .xlsx: the file's columns only, header in row 1,
191
+ * no title — the shape an import reads, so the rows go back in once fixed.
192
+ * The reason is not a column: it would be refused on the way back in.
193
+ */
194
+ async function droppedRowsFile(found, outputPath) {
195
+ var result = await runAction(builtins.exportTable, {
196
+ format: 'xlsx',
197
+ document: { columns: found.columns.map(function(c) { return { header: c }; }) },
198
+ rows: found.rows.map(function(r) {
199
+ return { values: found.columns.map(function(c) { return r.values[c] == null ? null : r.values[c]; }) };
200
+ }),
201
+ outputPath: outputPath
202
+ });
203
+ if (result.status !== 'success') throw new Error((result.error && result.error.message) || 'The file could not be written');
204
+ return result.output.filePath;
205
+ }
206
+
207
+ /** How an import went in — 'new' | 'append' | 'replace', or null for one from before modes. */
208
+ function modeOf(job) {
209
+ var meta = job && job.meta;
210
+ if (typeof meta === 'string') { try { meta = JSON.parse(meta); } catch (e) { meta = null; } }
211
+ return (meta && meta.mode) || null;
212
+ }
213
+
214
+ function plural(n, word) {
215
+ return n + ' ' + word + (n === 1 ? '' : 's');
216
+ }
217
+
218
+ module.exports = {
219
+ MODES: uploader.MODES,
220
+ DROPPED_FILE_ROWS: DROPPED_FILE_ROWS,
221
+ settings: settings,
222
+ formatOf: formatOf,
223
+ isWithin: isWithin,
224
+ listSheets: listSheets,
225
+ inspectChecks: inspectChecks,
226
+ checkTarget: checkTarget,
227
+ loadPart: loadPart,
228
+ rollback: rollback,
229
+ droppedRows: droppedRows,
230
+ droppedRowsFile: droppedRowsFile,
231
+ modeOf: modeOf
232
+ };
package/lib/router.js ADDED
@@ -0,0 +1,317 @@
1
+ // THE IMPORT ROUTES, mounted whole by an app:
2
+ //
3
+ // router.use('/imports', imports.router({ auth, resolveTarget, listTargets, service }))
4
+ //
5
+ // POST /files the file (multipart "file") → { filePath, … }
6
+ // POST /inspect { filePath } → { format, sheets?, checks }
7
+ // POST /check-target { filePath, sheet?, connectionId, dbInfoId, targetTable } → { exists, matched, extra, missing }
8
+ // POST /commit { filePath, connectionId, dbInfoId, originalFileName?, mappings } → 202 { runId }
9
+ // GET /runs/:id { status, progress, result, error } — the importer's own runs only
10
+ // POST /runs/:id/stop
11
+ // GET /targets the databases a person may import into (the app's listTargets)
12
+ // GET /history past imports, newest first
13
+ // GET /history/:id/dropped the rows the database refused, with reasons
14
+ // GET /history/:id/dropped/file the same rows as .xlsx, to fix and import again
15
+ // POST /history/:id/rollback delete what the import loaded (not a Replace, not an upsert)
16
+ // POST /history/:id/delete remove the record only
17
+ //
18
+ // The app supplies only what is its own: who may call these (auth), and its
19
+ // connections (resolveTarget, listTargets). Tenancy comes from the app's
20
+ // mtMiddleware, as for every other route: import records carry mtId1/mtId2 and
21
+ // are read back through the tenant filter.
22
+
23
+ var fs = require('fs');
24
+ var path = require('path');
25
+ var crypto = require('crypto');
26
+ var express = require('express');
27
+ var { uploadRoute } = require('@xeplr/base-apis');
28
+ var { getMtContext } = require('@xeplr/db');
29
+ var { attachConfig } = require('@xeplr/actions');
30
+ var imports = require('./imports');
31
+ var runs = require('./runs');
32
+
33
+ /**
34
+ * @param {object} options
35
+ * @param {Function|Function[]} options.auth - middleware run before every route (the app's gate)
36
+ * @param {(connectionId, dbInfoId, req) => Promise<{ dbType, dbConnection, label? } | null>} options.resolveTarget
37
+ * the database a part loads into, as a login the drivers take; null when it is not one this caller may use
38
+ * @param {(req) => Promise<[{ connectionId, label, databases: [{ id, label }] }]>} options.listTargets
39
+ * the databases this caller may import into, for the picker and for history's labels
40
+ * @param {string} [options.service] - names this app's import records (import_meta.service). Required without `config`.
41
+ * @param {object} [options.config] - an @xeplr/actions attachConfig() instance the app already has; else one is made
42
+ * @param {(userIds: string[]) => Promise<{ [id]: string }>} [options.userNames] - names for records made before
43
+ * the importer's name was kept on them
44
+ */
45
+ function createRouter(options) {
46
+ options = options || {};
47
+ if (!options.resolveTarget) throw new Error('@xeplr/imports: router() needs resolveTarget(connectionId, dbInfoId, req)');
48
+ if (!options.listTargets) throw new Error('@xeplr/imports: router() needs listTargets(req)');
49
+ var config = options.config || (options.service ? attachConfig({ service: options.service }) : null);
50
+ if (!config) throw new Error('@xeplr/imports: router() needs `service` (names this app\'s import records) or `config`');
51
+ var s = imports.settings();
52
+ var router = express.Router();
53
+ var auth = options.auth ? (Array.isArray(options.auth) ? options.auth : [options.auth]) : [];
54
+ var inFlight = new Set();
55
+
56
+ if (auth.length) router.use(auth);
57
+ router.use(function(req, res, next) { config.ready().then(function() { next(); }, next); });
58
+ router.use(express.json({ limit: '1mb' }));
59
+
60
+ // The file itself — @xeplr/base-apis' generic receive-and-store.
61
+ router.use('/files', uploadRoute({ destination: s.uploadDir, maxSize: s.maxUploadMb * 1024 * 1024 }));
62
+
63
+ /** The uploaded file a body names, checked: inside the upload directory, there, and a format we read. */
64
+ function fileOf(body) {
65
+ var filePath = body && body.filePath;
66
+ if (!filePath) return { status: 400, error: 'filePath is required' };
67
+ if (!imports.isWithin(s.uploadDir, filePath)) return { status: 400, error: 'Invalid filePath' };
68
+ if (!fs.existsSync(filePath)) return { status: 404, error: 'Upload not found — it may have expired. Please upload the file again.' };
69
+ var format = imports.formatOf(filePath);
70
+ if (!format) return { status: 400, error: 'Unsupported file type — use CSV, JSON, or Excel (.xlsx/.xls)' };
71
+ return { filePath: path.resolve(filePath), format: format };
72
+ }
73
+
74
+ async function targetOf(connectionId, dbInfoId, req) {
75
+ if (!connectionId || !dbInfoId) return null;
76
+ return options.resolveTarget(connectionId, dbInfoId, req);
77
+ }
78
+
79
+ router.post('/inspect', handle(async function(req, res) {
80
+ var file = fileOf(req.body);
81
+ if (file.error) {
82
+ if (file.status === 400 && req.body && req.body.filePath && /Unsupported/.test(file.error)) {
83
+ await fs.promises.unlink(req.body.filePath).catch(function() {});
84
+ }
85
+ return res.status(file.status).json({ error: file.error });
86
+ }
87
+ var sheets = file.format === 'excel' ? await imports.listSheets(file.filePath) : undefined;
88
+ res.json({ format: file.format, sheets: sheets, checks: await imports.inspectChecks(file.filePath, file.format, sheets) });
89
+ }));
90
+
91
+ router.post('/check-target', handle(async function(req, res) {
92
+ var b = req.body || {};
93
+ var file = fileOf(b);
94
+ if (file.error) return res.status(file.status).json({ error: file.error });
95
+ if (!b.targetTable) return res.status(400).json({ error: 'targetTable is required' });
96
+ var target = await targetOf(b.connectionId, b.dbInfoId, req);
97
+ if (!target) return res.status(404).json({ error: 'Database not found' });
98
+ res.json(await imports.checkTarget({ filePath: file.filePath, format: file.format, sheet: b.sheet || null,
99
+ dbType: target.dbType, dbConnection: target.dbConnection, targetTable: String(b.targetTable).trim() }));
100
+ }));
101
+
102
+ router.post('/commit', handle(async function(req, res) {
103
+ var b = req.body || {};
104
+ var mappings = b.mappings;
105
+ if (!Array.isArray(mappings) || !mappings.length) return res.status(400).json({ error: 'At least one mapping is required' });
106
+ var file = fileOf(b);
107
+ if (file.error) return res.status(file.status).json({ error: file.error });
108
+ var badMode = mappings.filter(function(m) { return m.mode != null && imports.MODES.indexOf(m.mode) === -1; })[0];
109
+ if (badMode) return res.status(400).json({ error: 'mode must be one of ' + imports.MODES.join(', ') });
110
+ if (mappings.some(function(m) { return !m.targetTable || !String(m.targetTable).trim(); })) {
111
+ return res.status(400).json({ error: 'Every mapping needs a targetTable' });
112
+ }
113
+ var target = await targetOf(b.connectionId, b.dbInfoId, req);
114
+ if (!target) return res.status(404).json({ error: 'Database not found' });
115
+ // ONE IMPORT PER FILE AT A TIME: a second press, or a retry after the
116
+ // browser gave up waiting, would load the same rows twice.
117
+ if (inFlight.has(file.filePath)) {
118
+ return res.status(409).json({ error: 'This file is already being imported. Wait for it to finish.' });
119
+ }
120
+ inFlight.add(file.filePath);
121
+
122
+ var mt = getMtContext() || {};
123
+ var user = req.user || {};
124
+ var count = mappings.length;
125
+ var runId = runs.startRun(async function(onProgress, signal) {
126
+ try {
127
+ var results = [];
128
+ for (var i = 0; i < mappings.length; i++) {
129
+ if (signal.aborted) break;
130
+ var m = mappings[i];
131
+ var movementId = crypto.randomBytes(12).toString('hex');
132
+ var outcome = await imports.loadPart({
133
+ filePath: file.filePath, format: file.format, sheet: m.sheet || null,
134
+ dbType: target.dbType, dbConnection: target.dbConnection,
135
+ targetTable: String(m.targetTable).trim(), primaryKeys: m.primaryKeys, mode: m.mode,
136
+ repairPastedRows: Boolean(m.repairPastedRows),
137
+ movementId: movementId, signal: signal,
138
+ onProgress: progressFor(onProgress, i, count, m.sheet),
139
+ mtId1: mt.mtId1, mtId2: mt.mtId2,
140
+ details: {
141
+ connectionId: b.connectionId, dbInfoId: b.dbInfoId,
142
+ // The person's own name for the file — never the stored one, a
143
+ // random id that means nothing to anybody.
144
+ originalFileName: b.originalFileName || null,
145
+ part: m.sheet || null,
146
+ recordCreatedBy: user.id || null,
147
+ // Kept on the record, so history can say who without asking
148
+ // the app's user store.
149
+ recordCreatedByName: user.name || user.email || null
150
+ }
151
+ }, config.metaStore);
152
+ results.push({
153
+ targetTable: String(m.targetTable).trim(),
154
+ sheet: m.sheet || null,
155
+ success: outcome.status === 'success',
156
+ rows: outcome.output ? outcome.output.totalRows : null,
157
+ dropped: outcome.output ? (outcome.output.dropped || 0) : null,
158
+ jobId: movementId,
159
+ recovered: outcome.output && outcome.output.repairedPastedRows ? outcome.output.repairedPastedRows.rows : null,
160
+ error: outcome.error ? outcome.error.message : null
161
+ });
162
+ }
163
+ return { results: results };
164
+ } finally {
165
+ inFlight.delete(file.filePath);
166
+ await fs.promises.unlink(file.filePath).catch(function() {});
167
+ }
168
+ }, { userId: user.id, timeLimitMs: s.timeLimitMs });
169
+ res.status(202).json({ runId: runId });
170
+ }));
171
+
172
+ router.get('/runs/:id', function(req, res) {
173
+ var run = runs.view(req.params.id, req.user && req.user.id);
174
+ if (!run) return res.status(404).json({ error: 'No such import run' });
175
+ res.json(run);
176
+ });
177
+
178
+ router.post('/runs/:id/stop', function(req, res) {
179
+ var run = runs.view(req.params.id, req.user && req.user.id);
180
+ if (!run) return res.status(404).json({ error: 'No such import run' });
181
+ res.json({ stopped: runs.stop(req.params.id, 'Stopped') });
182
+ });
183
+
184
+ router.get('/targets', handle(async function(req, res) {
185
+ res.json({ dataArray: (await options.listTargets(req)) || [] });
186
+ }));
187
+
188
+ /** One import record of this app, through the tenant filter — another company's is simply not found. */
189
+ async function jobOf(id) {
190
+ return config.ImportMeta.query().where('service', config.SERVICE_NAME).findById(id);
191
+ }
192
+
193
+ router.get('/history', handle(async function(req, res) {
194
+ var jobs = await config.ImportMeta.query().where('service', config.SERVICE_NAME).orderBy('started_at', 'desc').limit(200);
195
+ var targets = (await options.listTargets(req)) || [];
196
+ var labels = {};
197
+ targets.forEach(function(t) {
198
+ (t.databases || []).forEach(function(d) { labels[t.connectionId + '\u0000' + d.id] = [t.label, d.label].filter(Boolean).join(' / '); });
199
+ });
200
+ var unnamed = jobs.map(function(j) { var d = j.details || {}; return !d.recordCreatedByName && d.recordCreatedBy; }).filter(Boolean);
201
+ var names = unnamed.length && options.userNames ? (await options.userNames(Array.from(new Set(unnamed)))) || {} : {};
202
+ res.json({ dataArray: jobs.map(function(job) {
203
+ var d = job.details || {};
204
+ return {
205
+ id: job.id,
206
+ originalFileName: d.originalFileName,
207
+ part: d.part,
208
+ targetTable: job.target_table,
209
+ destination: labels[d.connectionId + '\u0000' + d.dbInfoId] || null,
210
+ status: job.status,
211
+ mode: imports.modeOf(job),
212
+ totalRows: job.total_rows == null ? null : Number(job.total_rows),
213
+ rowsLoaded: job.completed == null ? null : Number(job.completed),
214
+ rowsDropped: job.dropped == null ? null : Number(job.dropped),
215
+ errorMessage: job.error,
216
+ createdByName: d.recordCreatedByName || names[d.recordCreatedBy] || null,
217
+ startedAt: job.started_at
218
+ };
219
+ }) });
220
+ }));
221
+
222
+ /** A record and the database it loaded into, or { status, error }. */
223
+ async function jobAndTarget(req) {
224
+ var job = await jobOf(req.params.id);
225
+ if (!job) return { status: 404, error: 'Import not found' };
226
+ var d = job.details || {};
227
+ var target = await targetOf(d.connectionId, d.dbInfoId, req);
228
+ if (!target) return { status: 404, error: 'The database this import loaded into is no longer available' };
229
+ return { job: job, target: target };
230
+ }
231
+
232
+ async function droppedOf(req, limit) {
233
+ var got = await jobAndTarget(req);
234
+ if (got.error) return got;
235
+ got.found = await imports.droppedRows({
236
+ dbType: got.target.dbType, dbConnection: got.target.dbConnection,
237
+ targetTable: got.job.target_table, movementId: got.job.id, columns: got.job.columns, limit: limit
238
+ });
239
+ return got;
240
+ }
241
+
242
+ router.get('/history/:id/dropped', handle(async function(req, res) {
243
+ var got = await droppedOf(req);
244
+ if (got.error) return res.status(got.status).json({ error: got.error });
245
+ res.json({ columns: got.found.columns, rows: got.found.rows, total: got.found.total, dropped: Number(got.job.dropped || 0) });
246
+ }));
247
+
248
+ router.get('/history/:id/dropped/file', handle(async function(req, res) {
249
+ var got = await droppedOf(req, imports.DROPPED_FILE_ROWS);
250
+ if (got.error) return res.status(got.status).json({ error: got.error });
251
+ if (!got.found.rows.length) return res.status(404).json({ error: 'No dropped rows were kept for this import' });
252
+ var d = got.job.details || {};
253
+ var base = String(d.originalFileName || got.job.target_table).replace(/\.[^.]+$/, '');
254
+ var name = (base + ' - dropped rows').replace(/[\\/:*?"<>|\u0000-\u001F]+/g, ' ').trim().slice(0, 120) + '.xlsx';
255
+ // Beside the uploads — these are customer rows too — and gone once sent.
256
+ // No leading dot in the name: Express will not send a file under one.
257
+ var dir = fs.mkdtempSync(path.join(s.uploadDir, 'dropped-'));
258
+ try {
259
+ var filePath = await imports.droppedRowsFile(got.found, path.join(dir, name));
260
+ // With a callback Express leaves a failed send to us — answer it, or
261
+ // the request hangs.
262
+ res.download(filePath, name, function(err) {
263
+ fs.rm(dir, { recursive: true, force: true }, function() {});
264
+ if (err && !res.headersSent) res.status(500).json({ error: 'The file could not be sent: ' + err.message });
265
+ });
266
+ } catch (err) {
267
+ fs.rm(dir, { recursive: true, force: true }, function() {});
268
+ throw err;
269
+ }
270
+ }));
271
+
272
+ router.post('/history/:id/rollback', handle(async function(req, res) {
273
+ var job = await jobOf(req.params.id);
274
+ if (!job) return res.status(404).json({ error: 'Import not found' });
275
+ if (job.status !== 'completed') return res.status(400).json({ error: 'Only a completed import can be rolled back' });
276
+ if (imports.modeOf(job) === 'replace') {
277
+ return res.status(400).json({ error: 'This import replaced the table\'s rows, and the rows it replaced are gone — ' +
278
+ 'deleting the new ones would leave the table empty, not as it was. Import the earlier data again to restore it.' });
279
+ }
280
+ if (job.primary_keys && job.primary_keys.length) {
281
+ return res.status(400).json({ error: 'This import used upsert (matched on ' + job.primary_keys.join(', ') + ') — rollback is ' +
282
+ 'disabled because it may have only updated rows that already existed, not created them.' });
283
+ }
284
+ var got = await jobAndTarget(req);
285
+ if (got.error) return res.status(got.status).json({ error: got.error });
286
+ var result = await imports.rollback({ movementId: job.id, dbType: got.target.dbType, dbConnection: got.target.dbConnection,
287
+ targetTable: job.target_table }, config.metaStore);
288
+ res.json({ success: true, mainDeleted: result.mainDeleted, errorDeleted: result.errorDeleted });
289
+ }));
290
+
291
+ router.post('/history/:id/delete', handle(async function(req, res) {
292
+ var job = await jobOf(req.params.id);
293
+ if (!job) return res.status(404).json({ error: 'Import not found' });
294
+ await config.ImportMeta.query().findById(job.id).patch({ isActive: false });
295
+ res.json({ success: true });
296
+ }));
297
+
298
+ return router;
299
+ }
300
+
301
+ /** Which part of how many, with the uploader's own row counts. */
302
+ function progressFor(onProgress, index, count, sheet) {
303
+ return function(p) {
304
+ onProgress(Object.assign({}, p, { mapping: index + 1, mappings: count, sheet: sheet || null }));
305
+ };
306
+ }
307
+
308
+ /** An async route whose failure answers 500 (or the error's own status) with its message. */
309
+ function handle(fn) {
310
+ return function(req, res) {
311
+ Promise.resolve(fn(req, res)).catch(function(err) {
312
+ if (!res.headersSent) res.status(err.status || 500).json({ error: err.message });
313
+ });
314
+ };
315
+ }
316
+
317
+ module.exports = createRouter;
package/lib/runs.js ADDED
@@ -0,0 +1,93 @@
1
+ // IMPORTS RUN IN THE BACKGROUND. A 300 MB file takes minutes to load, far
2
+ // past any gateway's timeout — so the commit answers at once with a run id,
3
+ // and the page asks how far it has got (GET /runs/:id) until it ends.
4
+ //
5
+ // Held in this process's memory: a run belongs to the process doing the work,
6
+ // and dies with it. The import's durable record is import_meta (written by
7
+ // @xeplr/actions' uploader before the first row), so a run lost to a restart
8
+ // is still in the history, with its rows to roll back.
9
+
10
+ var crypto = require('crypto');
11
+
12
+ // A finished run is kept this long for its page to read the ending, then gone.
13
+ var KEEP_FINISHED_MS = 60 * 60 * 1000;
14
+
15
+ var runs = new Map();
16
+
17
+ /**
18
+ * Start `work(onProgress, signal)` in the background.
19
+ *
20
+ * @param {object} opts - { userId, timeLimitMs? }
21
+ * @returns {string} the run id
22
+ */
23
+ function startRun(work, opts) {
24
+ opts = opts || {};
25
+ var run = {
26
+ id: 'imp_' + crypto.randomBytes(10).toString('hex'),
27
+ userId: opts.userId == null ? null : String(opts.userId),
28
+ status: 'running',
29
+ progress: null,
30
+ result: null,
31
+ error: null,
32
+ startedAt: Date.now(),
33
+ finishedAt: null,
34
+ controller: new AbortController(),
35
+ timer: null
36
+ };
37
+ runs.set(run.id, run);
38
+
39
+ if (opts.timeLimitMs) {
40
+ run.timer = setTimeout(function() {
41
+ stop(run.id, 'The import took longer than ' + Math.round(opts.timeLimitMs / 60000) + ' minutes and was stopped.');
42
+ }, opts.timeLimitMs);
43
+ if (run.timer.unref) run.timer.unref();
44
+ }
45
+
46
+ Promise.resolve().then(function() {
47
+ return work(function(progress) { run.progress = progress; }, run.controller.signal);
48
+ }).then(function(result) {
49
+ settle(run, run.controller.signal.aborted ? 'stopped' : 'complete', result, null);
50
+ }, function(err) {
51
+ settle(run, run.controller.signal.aborted ? 'stopped' : 'error', null, (err && err.message) || String(err));
52
+ });
53
+ return run.id;
54
+ }
55
+
56
+ function settle(run, status, result, error) {
57
+ if (run.timer) clearTimeout(run.timer);
58
+ run.status = status;
59
+ run.result = result;
60
+ run.error = status === 'stopped' ? (run.stopReason || 'Stopped') : error;
61
+ run.finishedAt = Date.now();
62
+ var forget = setTimeout(function() { runs.delete(run.id); }, KEEP_FINISHED_MS);
63
+ if (forget.unref) forget.unref();
64
+ }
65
+
66
+ /** Ask a running import to stop: it stops reading within a batch. */
67
+ function stop(id, reason) {
68
+ var run = runs.get(id);
69
+ if (!run || run.status !== 'running') return false;
70
+ run.stopReason = reason || 'Stopped';
71
+ run.controller.abort();
72
+ return true;
73
+ }
74
+
75
+ /**
76
+ * A run as its page sees it — or null when there is none for this user. A
77
+ * run is only ever shown to the person who started it.
78
+ */
79
+ function view(id, userId) {
80
+ var run = runs.get(id);
81
+ if (!run || run.userId !== (userId == null ? null : String(userId))) return null;
82
+ return {
83
+ id: run.id,
84
+ status: run.status,
85
+ progress: run.progress,
86
+ result: run.result,
87
+ error: run.error,
88
+ startedAt: run.startedAt,
89
+ finishedAt: run.finishedAt
90
+ };
91
+ }
92
+
93
+ module.exports = { startRun: startRun, stop: stop, view: view };
@@ -0,0 +1,49 @@
1
+ -- 0001_imports_access.sql
2
+ -- @xeplr/imports' own access catalog: every route lib/router.js serves, by
3
+ -- group, and the roles that reach them. The package never runs this itself —
4
+ -- a consuming app lists this directory in XEPLR_AUTH_MIGRATIONS
5
+ -- (imports.authMigrationsDir), so its auth database knows these routes exist.
6
+ --
7
+ -- The names assume the app mounts the router at /imports, as the README says.
8
+ --
9
+ -- Groups:
10
+ -- imports:view see past imports, their dropped rows, a run's progress, the databases
11
+ -- imports:create upload, inspect, check against a table, import, stop one's own import
12
+ -- imports:edit roll an import back, delete its record
13
+ --
14
+ -- Roles are the shared vocabulary every app's auth database uses (as in
15
+ -- @xeplr/jobs' 0001): Super Admin gets everything; CompanyAdmin gets
16
+ -- imports:* (its rows stay within its own company through mtId1). Loading
17
+ -- data into a company's databases is an administrator's job, so Creator and
18
+ -- Viewer get none — an app that wants them to can map them in its own
19
+ -- migration. Idempotent: insert-if-absent on name + group, and on each mapping.
20
+
21
+ INSERT INTO "roles" (id, name, "isActive", "mtId1", "recordCreatedDate", "recordModifiedDate")
22
+ SELECT encode(gen_random_bytes(12), 'hex'), name, true, '*', now(), now()
23
+ FROM (VALUES ('Super Admin'), ('CompanyAdmin')) AS v(name)
24
+ WHERE NOT EXISTS (SELECT 1 FROM "roles" r WHERE r.name = v.name);
25
+
26
+ INSERT INTO "apis" (id, name, "apiGroup", "isPublic", "isActive", "mtId1", "recordCreatedDate", "recordModifiedDate")
27
+ SELECT encode(gen_random_bytes(12), 'hex'), v.name, v.grp, false, true, '*', now(), now()
28
+ FROM (VALUES
29
+ ('POST /imports/files', 'imports:create'),
30
+ ('POST /imports/inspect', 'imports:create'),
31
+ ('POST /imports/check-target', 'imports:create'),
32
+ ('POST /imports/commit', 'imports:create'),
33
+ ('POST /imports/runs/:id/stop', 'imports:create'),
34
+ ('GET /imports/runs/:id', 'imports:view'),
35
+ ('GET /imports/targets', 'imports:view'),
36
+ ('GET /imports/history', 'imports:view'),
37
+ ('GET /imports/history/:id/dropped', 'imports:view'),
38
+ ('GET /imports/history/:id/dropped/file', 'imports:view'),
39
+ ('POST /imports/history/:id/rollback', 'imports:edit'),
40
+ ('POST /imports/history/:id/delete', 'imports:edit')
41
+ ) AS v(name, grp)
42
+ WHERE NOT EXISTS (SELECT 1 FROM "apis" a WHERE a.name = v.name AND a."apiGroup" = v.grp);
43
+
44
+ INSERT INTO "apisRolesMapping" (id, "roleId", "apiId", "isActive", "mtId1", "recordCreatedDate", "recordModifiedDate")
45
+ SELECT encode(gen_random_bytes(12), 'hex'), r.id, a.id, true, '*', now(), now()
46
+ FROM "roles" r CROSS JOIN "apis" a
47
+ WHERE r.name IN ('Super Admin', 'CompanyAdmin')
48
+ AND a."apiGroup" LIKE 'imports:%'
49
+ AND NOT EXISTS (SELECT 1 FROM "apisRolesMapping" m WHERE m."roleId" = r.id AND m."apiId" = a.id);
package/package.json CHANGED
@@ -1,6 +1,54 @@
1
1
  {
2
2
  "name": "@xeplr/imports",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "1.0.1",
4
+ "description": "Data imports as a drop-in module: upload a CSV/Excel/JSON file, check it against the table, load it as a New table, Append or Replace, see and download the rows the database refused, and roll back — routes, access catalog and env in one package",
5
+ "main": "index.js",
6
+ "files": [
7
+ "index.js",
8
+ "lib/",
9
+ "migrations-auth/"
10
+ ],
11
+ "exports": {
12
+ ".": "./index.js",
13
+ "./package.json": "./package.json"
14
+ },
15
+ "keywords": [
16
+ "import",
17
+ "upload",
18
+ "csv",
19
+ "excel",
20
+ "express"
21
+ ],
22
+ "author": "xeplr",
23
+ "license": "MIT",
24
+ "repository": {
25
+ "type": "git",
26
+ "url": "https://github.com/Xeplr/imports"
27
+ },
28
+ "publishConfig": {
29
+ "access": "public"
30
+ },
31
+ "dependencies": {
32
+ "@xeplr/actions": "^1.1.23",
33
+ "@xeplr/base-apis": "^2.0.3",
34
+ "@xeplr/db": "^1.0.8",
35
+ "@xeplr/utils": "^1.0.13",
36
+ "csv-parse": "^7.0.3",
37
+ "fflate": "^0.8.3",
38
+ "multer": "^2.4.0",
39
+ "saxes": "^6.0.0",
40
+ "unzipper": "^0.12.5"
41
+ },
42
+ "peerDependencies": {
43
+ "express": "^4.0.0 || ^5.0.0"
44
+ },
45
+ "scripts": {
46
+ "test": "node --test test/*.test.js"
47
+ },
48
+ "devDependencies": {
49
+ "express": "^5.3.0",
50
+ "knex": "^3.3.0",
51
+ "mysql2": "^3.24.5",
52
+ "pg": "^8.23.1"
53
+ }
54
+ }