@xeplr/actions 1.1.19 → 1.1.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -4
- package/lib/builtins/file/upload.js +18 -1
- package/lib/drivers/db/dateValues.js +106 -0
- package/lib/drivers/db/duckdb.js +32 -20
- package/lib/drivers/db/mssql.js +21 -32
- package/lib/drivers/db/mysql.js +23 -32
- package/lib/drivers/db/postgres.js +32 -23
- package/lib/formats/_xlsx-stream.js +14 -6
- package/lib/formats/index.js +20 -1
- package/lib/formats/pastedRows.js +146 -0
- package/lib/formats/writers/xlsx.js +8 -2
- package/lib/uploader/index.js +14 -0
- package/lib/uploader/reconcile.js +2 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -300,9 +300,20 @@ Logical types and what each driver creates:
|
|
|
300
300
|
| number | NUMERIC | DECIMAL(38,10) | DECIMAL(38,10) | DOUBLE |
|
|
301
301
|
| boolean | BOOLEAN | TINYINT(1) | BIT | BOOLEAN |
|
|
302
302
|
| date | DATE | DATE | DATE | DATE |
|
|
303
|
-
| datetime | TIMESTAMPTZ | DATETIME(6) | DATETIME2 | TIMESTAMPTZ |
|
|
303
|
+
| datetime (an instant) | TIMESTAMPTZ | DATETIME(6), in UTC | DATETIME2, in UTC | TIMESTAMPTZ |
|
|
304
|
+
| localdatetime (no zone) | TIMESTAMP | DATETIME(6) | DATETIME2 | TIMESTAMP |
|
|
304
305
|
| object / array | JSONB | JSON | NVARCHAR(MAX) | JSON |
|
|
305
306
|
|
|
307
|
+
**Dates and times follow three rules, on every driver** (`lib/drivers/db/dateValues.js`, the one copy):
|
|
308
|
+
|
|
309
|
+
1. **A `date` is copied as the same day.** MySQL/Postgres/SQL Server `DATE` → `date`.
|
|
310
|
+
2. **A date-time with a zone is copied as the same instant**, stored in UTC. MySQL `TIMESTAMP`, Postgres `timestamptz`, SQL Server `datetimeoffset`, DuckDB `TIMESTAMPTZ` → `datetime`.
|
|
311
|
+
3. **A date-time without a zone is copied as its digits**, `10:00` stays `10:00`, and stays zoneless. MySQL `DATETIME`, Postgres `timestamp`, SQL Server `datetime`/`datetime2`/`smalldatetime`, DuckDB `TIMESTAMP` → `localdatetime`. Which zone it was recorded in is declared where it is used (a cube's source zone), never guessed here.
|
|
312
|
+
|
|
313
|
+
**Nothing reads the machine's time zone.** The MySQL session is pinned to UTC (`SET time_zone = '+00:00'` on every connection), SQL Server to `useUTC: true` (after the caller's options), Postgres returns `date` and `timestamp` as text, and DuckDB opens with `TimeZone = 'UTC'`. Every JS `Date` is read with `getUTC*`, and a string with no zone is read as UTC, never handed to `new Date()`. `test/dates-across-zones.test.js` moves a real table from each of the three databases into DuckDB with the process in India time and in New York time, and the two must store the same values.
|
|
314
|
+
|
|
315
|
+
**An existing target column that is zoned but receives a zoneless source column is retyped once** to zoneless, keeping its digits (Postgres and DuckDB; `buildRetypeToLocalDateTimeSql`). Tables created before this rule held those values as instants.
|
|
316
|
+
|
|
306
317
|
Inference (`inferColumns`): any array → `array`; any object → `object`; all `Date` or ISO-date strings → `datetime`; all booleans → `boolean`; all numbers → `number`; anything else, or all null → `string`.
|
|
307
318
|
|
|
308
319
|
### Staged loads
|
|
@@ -337,10 +348,16 @@ Upserts need no index: they are staged and match keys in the move. A read-only o
|
|
|
337
348
|
| format | reads | notes |
|
|
338
349
|
|---|---|---|
|
|
339
350
|
| `csv` | stream | `formatConfig`: `columns` (default first row), `delimiter` (`,`), `cast` (true), `skipEmptyLines`, `trim`. |
|
|
340
|
-
| `excel` | file path | `.xlsx` via `unzipper` + `saxes`; `sheet`, `startRow`. Date-styled cells arrive as `Date`. |
|
|
351
|
+
| `excel` | file path | `.xlsx` via `unzipper` + `saxes`; `sheet`, `startRow`. Date-styled cells arrive as `Date`. Excel's own escapes in cell text are decoded (`_x000D_` → a real line break; `_x005F_` → `_`), and Japanese phonetic runs (`<rPh>`, furigana) are not joined into the value. |
|
|
341
352
|
| `json` | stream | A JSON array or NDJSON; buffers the whole file. |
|
|
342
353
|
| `txt` | — | Placeholder, not implemented. |
|
|
343
354
|
|
|
355
|
+
**Rows pasted into one cell** (`lib/formats/pastedRows.js`). Data copied out of an ERP and pasted into Excel: a value starting with `"` (an inch mark) makes Excel put everything up to the next `"` — tabs, line breaks, whole rows — into that one cell. The rows it swallowed exist nowhere else. Seen for real: 465 rows folded into 11 cells.
|
|
356
|
+
- `formats.scanPastedRows(format, filePath, { sheet })` → `{ cells, rows, examples: [{ row, column, rows, first }] }` — found, nothing changed. For an import's inspect step.
|
|
357
|
+
- `formatConfig.repairPastedRows: true` on `file-upload` puts them back: the cell splits on its line breaks into the rest of its own row, whole rows, and the start of the last row, whose remaining values are the merged row's later cells. The eaten quotes go back where they opened and closed. The action's output says what it did: `repairedPastedRows: { cells, rows, at }`.
|
|
358
|
+
- **Only an exact shape is repaired:** the pieces must have the sheet's column counts (and the cells the last row's values would leave empty must be empty). Ordinary multi-line text never matches, so a genuine note with line breaks is never touched. On the client's real file: 11 of 11 cells, 465 of 465 rows, no recovered entry number duplicating an existing one.
|
|
359
|
+
- `pastedRows.decodeOoxml`, `splitPastedRows(rec, headers)`, `expandPastedRows(rows)`, `scanForPastedRows(rows)` for other readers.
|
|
360
|
+
|
|
344
361
|
File sources: only `local` is implemented (`path`). `sftp`, `sharepoint` and `google` are placeholders.
|
|
345
362
|
|
|
346
363
|
## Movement metadata (`xeplr_configs`)
|
|
@@ -421,7 +438,7 @@ await driver.query(pool, userSql, params) // reads only
|
|
|
421
438
|
| Movement logs carry identifiers, counts, SQL and the error text — never parameter values or row contents; a failed batch shows one row's primary key, not its data. | Logs are copied into tickets and chats. |
|
|
422
439
|
| DuckDB opens **read-only unless the call says `access: 'rw'`**; no environment variable can grant write. A file already open read-only in the process refuses a write. Opening waits on the lock (`openTimeoutMs`, default 60 s). | One read-write process locks every other process out, even readers. Write access belongs at the call site that writes. |
|
|
423
440
|
| **Every DuckDB value comes out plain**, from `query` and `fetchStream` alike: a number, text, boolean, `Date`, array or object, never a BigInt or a DuckDB wrapper object. DECIMAL is a number; DATE and TIMESTAMP are `Date`s; a whole number too big to hold exactly (beyond ±9,007,199,254,740,991) is **text**, never rounded; lists, structs and maps are arrays and objects; TIME, INTERVAL, UUID and any other type are text. `fetchStream` has DuckDB do the conversion inside its engine by wrapping the query (about +3% on a million rows); a query with nothing to convert runs as given. | A streamed read used to hand DECIMAL and TIMESTAMP back as wrapper objects with a BigInt inside, and the first `JSON.stringify` downstream failed with "Do not know how to serialize a BigInt". Text for huge numbers is what the Postgres, MySQL and SQL Server drivers already return. |
|
|
424
|
-
| **A
|
|
441
|
+
| **DuckDB works in UTC** (`TimeZone = 'UTC'` on every instance). A timestamp without a zone is read with its digits in the `Date`'s UTC fields; one with a zone keeps its exact instant. | Left to the machine's zone, `date_trunc('month', …)` of 1 Aug 00:00 UTC on a server in India was 31 Jul 18:30, so August was labelled July, and a filter "to 31 Aug" ended at 30 Aug 18:30. |
|
|
425
442
|
| `meta-store-knex` requires `service` and `applicationId`, and scopes every statement by `applicationId`. | `import_meta` is shared by every app; unattributed rows, or a rollback decided from another app's row, are worse than refusing. |
|
|
426
443
|
| Mail filenames are sanitised (`[^\w.-]` → `_`, all-dot names → `attachment`, max 180 chars) and prefixed with a random id. | Attachment names come from the sender and could escape the output directory; common names collide. |
|
|
427
444
|
| The system mail sender refuses `text`, `bcc`, `from`, `replyTo` rather than dropping them. | A silently lost bcc is a compliance problem found a year later. |
|
|
@@ -437,7 +454,7 @@ node --test test/uploader/reconcile.test.js
|
|
|
437
454
|
|
|
438
455
|
| needs nothing | needs a database |
|
|
439
456
|
|---|---|
|
|
440
|
-
| `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
|
|
457
|
+
| `test/formats/pasted-rows.test.js` (its last case loads into Postgres when `PG_HOST`/`PG_PORT`/`PG_USER`/`PG_PASSWORD`/`PG_DATABASE` are set, else skipped), `test/procedure-call.test.js`, `test/streaming/spool.test.js`, `test/uploader/reconcile.test.js`, `test/write-progress.test.js`, `test/stop-movement.test.js`, `test/formats/csv-gnarly.test.js`, `test/drivers/mysql.test.js`, `test/drivers/mssql.test.js`, `test/drivers/query-columns.test.js`, `test/builtins/email.test.js` (loads the package root, so `@xeplr/db` must be resolvable) | Postgres at `localhost:5435` (`PG_PASSWORD`, default `postgres`; database `xeplr_actions_test`): `test/drivers/postgres.test.js`, `test/builtins/db-push.test.js`, `test/builtins/file-upload.test.js`, `test/uploader/upload.test.js`; plus `@xeplr/db` for `test/uploader/meta-store-knex.test.js` |
|
|
441
458
|
| `test/list-columns-length.test.js` (`XEPLR_TEST_PG_URL`, `MYSQL_TEST_URL`, `MSSQL_TEST_URL`; each skipped when unset), `test/drivers/duckdb.test.js`, `test/drivers/duckdb-values.test.js`, `test/duckdb-index-guard.test.js`, `test/staged-load.test.js` (needs `@duckdb/node-api`) | MySQL `localhost:3306` (`MYSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mysql-integration.test.js` |
|
|
442
459
|
| | SQL Server `localhost:1433` (`MSSQL_HOST/PORT/USER/PASSWORD`): `test/drivers/mssql-integration.test.js` |
|
|
443
460
|
| | Postgres + MySQL + SQL Server (`PG_*`, `MYSQL_*`, `MSSQL_*`): `test/builtins/db-replication.test.js` |
|
|
@@ -11,6 +11,7 @@
|
|
|
11
11
|
var { upload } = require('../../uploader');
|
|
12
12
|
var fileSources = require('../../drivers/file');
|
|
13
13
|
var formats = require('../../formats');
|
|
14
|
+
var { expandPastedRows } = require('../../formats/pastedRows');
|
|
14
15
|
var dbDrivers = require('../../drivers/db');
|
|
15
16
|
var { SqlQueue } = require('@xeplr/utils/lib/queue');
|
|
16
17
|
var { generateId } = require('@xeplr/utils/lib/helpers');
|
|
@@ -82,6 +83,13 @@ module.exports = {
|
|
|
82
83
|
var stream = await source.openStream(sourceCfg);
|
|
83
84
|
rowsIterable = format.parseStream(stream, input.formatConfig || {});
|
|
84
85
|
}
|
|
86
|
+
// Rows that a paste folded into one cell, put back — only when asked
|
|
87
|
+
// (formatConfig.repairPastedRows), after the person saw what was found.
|
|
88
|
+
// See lib/formats/pastedRows.js.
|
|
89
|
+
var repaired = [];
|
|
90
|
+
if (input.formatConfig && input.formatConfig.repairPastedRows) {
|
|
91
|
+
rowsIterable = expandPastedRows(rowsIterable, function(r) { repaired.push(r); });
|
|
92
|
+
}
|
|
85
93
|
|
|
86
94
|
// Set up target
|
|
87
95
|
var movementId = input.movementId || system.occurrenceId || ('mv_' + generateId());
|
|
@@ -97,7 +105,7 @@ module.exports = {
|
|
|
97
105
|
});
|
|
98
106
|
|
|
99
107
|
try {
|
|
100
|
-
|
|
108
|
+
var result = await upload({
|
|
101
109
|
source: rowsIterable,
|
|
102
110
|
driver: driver,
|
|
103
111
|
connection: pool,
|
|
@@ -120,6 +128,15 @@ module.exports = {
|
|
|
120
128
|
onProgress: system.onProgress || null,
|
|
121
129
|
signal: system.signal || null
|
|
122
130
|
});
|
|
131
|
+
// What the repair put back, for the import's report.
|
|
132
|
+
if (repaired.length) {
|
|
133
|
+
result = Object.assign({}, result, { repairedPastedRows: {
|
|
134
|
+
cells: repaired.length,
|
|
135
|
+
rows: repaired.reduce(function(n, r) { return n + r.rows; }, 0),
|
|
136
|
+
at: repaired
|
|
137
|
+
} });
|
|
138
|
+
}
|
|
139
|
+
return result;
|
|
123
140
|
} finally {
|
|
124
141
|
queue.stop();
|
|
125
142
|
await driver.close(pool);
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
// Dates and times as they move between databases — the ONE copy every driver
|
|
2
|
+
// uses, so no driver can quietly read a value in this machine's zone.
|
|
3
|
+
//
|
|
4
|
+
// Three logical types, and three rules:
|
|
5
|
+
//
|
|
6
|
+
// date a calendar day. Copied as the same day, always.
|
|
7
|
+
// datetime an INSTANT (MySQL TIMESTAMP, Postgres timestamptz, SQL Server
|
|
8
|
+
// datetimeoffset, DuckDB TIMESTAMPTZ). Copied as the same
|
|
9
|
+
// instant, in UTC.
|
|
10
|
+
// localdatetime a wall-clock reading with NO zone (MySQL DATETIME, Postgres
|
|
11
|
+
// timestamp, SQL Server datetime/datetime2, DuckDB TIMESTAMP).
|
|
12
|
+
// Copied digit for digit — 10:00 stays 10:00. What zone it was
|
|
13
|
+
// recorded in is declared later, where it is used (a cube's
|
|
14
|
+
// source timezone), never guessed here.
|
|
15
|
+
//
|
|
16
|
+
// NOTHING HERE READS THE MACHINE'S ZONE. Every Date is read with getUTC*, and a
|
|
17
|
+
// string without a zone is read as UTC rather than handed to `new Date()`,
|
|
18
|
+
// which would read it in local time. The drivers deliver values on the same
|
|
19
|
+
// terms: the MySQL session is pinned to UTC, the SQL Server driver to useUTC,
|
|
20
|
+
// Postgres returns its zoneless types as text, and DuckDB works in UTC. So a
|
|
21
|
+
// value here means the same thing on a server in India and one in New York —
|
|
22
|
+
// the tests run under both zones to keep it that way.
|
|
23
|
+
|
|
24
|
+
function pad(n, width) { return String(n).padStart(width || 2, '0'); }
|
|
25
|
+
|
|
26
|
+
function utcDate(d) {
|
|
27
|
+
return d.getUTCFullYear() + '-' + pad(d.getUTCMonth() + 1) + '-' + pad(d.getUTCDate());
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
function utcClock(d) {
|
|
31
|
+
return pad(d.getUTCHours()) + ':' + pad(d.getUTCMinutes()) + ':' + pad(d.getUTCSeconds()) + '.' + pad(d.getUTCMilliseconds(), 3);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
// '2025-08-01', '2025-08-01 10:00', '2025-08-01T10:00:00.123456', optionally with
|
|
35
|
+
// 'Z' or an offset. Captures the parts as written.
|
|
36
|
+
var STAMP = /^(\d{4}-\d{2}-\d{2})(?:[T ](\d{2}):(\d{2})(?::(\d{2})(?:\.(\d+))?)?)?\s*(Z|[+-]\d{2}(?::?\d{2})?)?$/i;
|
|
37
|
+
|
|
38
|
+
/**
|
|
39
|
+
* A calendar day as YYYY-MM-DD. A Date is read in UTC — every driver delivers a
|
|
40
|
+
* date as midnight UTC of that day, or as text.
|
|
41
|
+
*/
|
|
42
|
+
function toDateOnly(v) {
|
|
43
|
+
if (v === null || v === undefined) return v;
|
|
44
|
+
if (v instanceof Date) return isNaN(v.getTime()) ? null : utcDate(v);
|
|
45
|
+
var m = /^(\d{4}-\d{2}-\d{2})/.exec(String(v).trim());
|
|
46
|
+
return m ? m[1] : v;
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* A zoneless wall-clock time as 'YYYY-MM-DD HH:MM:SS.mmm', the digits it had.
|
|
51
|
+
* A Date carries those digits in its UTC fields (that is how the drivers hand
|
|
52
|
+
* one over); a string keeps its own digits, and any zone on it is dropped —
|
|
53
|
+
* the column has none to keep.
|
|
54
|
+
*/
|
|
55
|
+
function toLocalDateTime(v) {
|
|
56
|
+
if (v === null || v === undefined) return v;
|
|
57
|
+
if (v instanceof Date) return isNaN(v.getTime()) ? null : utcDate(v) + ' ' + utcClock(v);
|
|
58
|
+
var m = STAMP.exec(String(v).trim());
|
|
59
|
+
if (!m) return v;
|
|
60
|
+
var ms = m[5] ? (m[5] + '00').slice(0, 3) : '000';
|
|
61
|
+
return m[1] + ' ' + (m[2] || '00') + ':' + (m[3] || '00') + ':' + (m[4] || '00') + '.' + ms;
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/**
|
|
65
|
+
* An instant as a Date. A string that names its zone is read in it; one that
|
|
66
|
+
* does not is read as UTC (a MySQL TIMESTAMP from a UTC session), never in this
|
|
67
|
+
* machine's zone. Unreadable → null.
|
|
68
|
+
*/
|
|
69
|
+
function toInstant(v) {
|
|
70
|
+
if (v === null || v === undefined) return null;
|
|
71
|
+
if (v instanceof Date) return isNaN(v.getTime()) ? null : v;
|
|
72
|
+
var m = STAMP.exec(String(v).trim());
|
|
73
|
+
if (!m) return null;
|
|
74
|
+
var iso = m[1] + 'T' + (m[2] || '00') + ':' + (m[3] || '00') + ':' + (m[4] || '00') +
|
|
75
|
+
(m[5] ? '.' + (m[5] + '00').slice(0, 3) : '') + (m[6] ? normalizeZone(m[6]) : 'Z');
|
|
76
|
+
var d = new Date(iso);
|
|
77
|
+
return isNaN(d.getTime()) ? null : d;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
function normalizeZone(z) {
|
|
81
|
+
if (/^z$/i.test(z)) return 'Z';
|
|
82
|
+
var m = /^([+-])(\d{2}):?(\d{2})?$/.exec(z);
|
|
83
|
+
return m ? m[1] + m[2] + ':' + (m[3] || '00') : 'Z';
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/** An instant as ISO 8601 in UTC ('…Z'), or the value unchanged when unreadable. */
|
|
87
|
+
function toInstantIso(v) {
|
|
88
|
+
if (v === null || v === undefined) return v;
|
|
89
|
+
var d = toInstant(v);
|
|
90
|
+
return d ? d.toISOString() : v;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/** An instant as 'YYYY-MM-DD HH:MM:SS.mmm' in UTC — for a column with no zone of its own (MySQL, SQL Server). */
|
|
94
|
+
function toInstantUtcClock(v) {
|
|
95
|
+
if (v === null || v === undefined) return v;
|
|
96
|
+
var d = toInstant(v);
|
|
97
|
+
return d ? utcDate(d) + ' ' + utcClock(d) : v;
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
module.exports = {
|
|
101
|
+
toDateOnly: toDateOnly,
|
|
102
|
+
toLocalDateTime: toLocalDateTime,
|
|
103
|
+
toInstant: toInstant,
|
|
104
|
+
toInstantIso: toInstantIso,
|
|
105
|
+
toInstantUtcClock: toInstantUtcClock
|
|
106
|
+
};
|
package/lib/drivers/db/duckdb.js
CHANGED
|
@@ -81,7 +81,8 @@ var TYPE_MAP = {
|
|
|
81
81
|
number: 'DOUBLE',
|
|
82
82
|
boolean: 'BOOLEAN',
|
|
83
83
|
date: 'DATE',
|
|
84
|
-
datetime: 'TIMESTAMPTZ',
|
|
84
|
+
datetime: 'TIMESTAMPTZ', // an instant
|
|
85
|
+
localdatetime: 'TIMESTAMP', // no zone; its digits as they were (dateValues.js)
|
|
85
86
|
object: 'JSON',
|
|
86
87
|
array: 'JSON'
|
|
87
88
|
};
|
|
@@ -94,8 +95,10 @@ var DUCK_TO_LOGICAL = {
|
|
|
94
95
|
'HUGEINT': 'number', 'UTINYINT': 'number', 'USMALLINT': 'number', 'UINTEGER': 'number',
|
|
95
96
|
'UBIGINT': 'number', 'FLOAT': 'number', 'DOUBLE': 'number', 'DECIMAL': 'number',
|
|
96
97
|
'DATE': 'date',
|
|
97
|
-
|
|
98
|
-
|
|
98
|
+
// A TIMESTAMP has no zone — a wall-clock reading, copied as its digits. Only
|
|
99
|
+
// the one WITH a zone is an instant.
|
|
100
|
+
'TIMESTAMP': 'localdatetime', 'TIMESTAMP WITH TIME ZONE': 'datetime',
|
|
101
|
+
'TIMESTAMP_S': 'localdatetime', 'TIMESTAMP_MS': 'localdatetime', 'TIMESTAMP_NS': 'localdatetime',
|
|
99
102
|
'TIME': 'datetime', 'TIME WITH TIME ZONE': 'datetime', 'INTERVAL': 'string',
|
|
100
103
|
'JSON': 'object', 'STRUCT': 'object', 'MAP': 'object',
|
|
101
104
|
'LIST': 'array'
|
|
@@ -234,7 +237,15 @@ async function connect(config) {
|
|
|
234
237
|
return existing.pool;
|
|
235
238
|
}
|
|
236
239
|
|
|
237
|
-
|
|
240
|
+
// THE SESSION ZONE IS UTC, ALWAYS. DuckDB otherwise takes the machine's
|
|
241
|
+
// zone, and every date operation on a TIMESTAMPTZ — date_trunc, comparing
|
|
242
|
+
// with a plain date, a cast to DATE — happens in it. On a server in India
|
|
243
|
+
// the month of 1 Aug 00:00 UTC came back as 31 Jul 18:30, a board asked for
|
|
244
|
+
// August labelled its total "Jul-2025", and a filter "to 31 Aug" ended at
|
|
245
|
+
// 30 Aug 18:30. Data here is kept in UTC (and a cube stores its days already
|
|
246
|
+
// cut in its own reporting zone), so UTC is the zone that changes nothing.
|
|
247
|
+
// Set on the instance, so every connection to the file inherits it.
|
|
248
|
+
var options = Object.assign({ TimeZone: 'UTC' }, access === 'ro' ? { access_mode: 'READ_ONLY' } : {});
|
|
238
249
|
// Waits rather than failing — see openWithWait. This is what makes the
|
|
239
250
|
// file's own lock the mutex and removes any need to designate a writer.
|
|
240
251
|
var instance = await openWithWait(api, key, options, access,
|
|
@@ -516,19 +527,8 @@ function buildInsertSql(tableName, rows, columns, movementId, primaryKeys) {
|
|
|
516
527
|
return { sql: sql, params: params };
|
|
517
528
|
}
|
|
518
529
|
|
|
519
|
-
//
|
|
520
|
-
|
|
521
|
-
function toDateOnly(v) {
|
|
522
|
-
if (v === null || v === undefined) return v;
|
|
523
|
-
if (v instanceof Date) {
|
|
524
|
-
if (isNaN(v.getTime())) return null;
|
|
525
|
-
return v.getFullYear() + '-' +
|
|
526
|
-
String(v.getMonth() + 1).padStart(2, '0') + '-' +
|
|
527
|
-
String(v.getDate()).padStart(2, '0');
|
|
528
|
-
}
|
|
529
|
-
var m = /^(\d{4}-\d{2}-\d{2})/.exec(String(v));
|
|
530
|
-
return m ? m[1] : v;
|
|
531
|
-
}
|
|
530
|
+
// Dates and times: one shared rule for every driver (dateValues.js).
|
|
531
|
+
var dateValues = require('./dateValues');
|
|
532
532
|
|
|
533
533
|
// Coerce a JS value to the effective column type. Same contract as the other
|
|
534
534
|
// drivers: a value that cannot fit passes through unchanged so DuckDB rejects
|
|
@@ -567,11 +567,13 @@ function coerceValue(v, type) {
|
|
|
567
567
|
return v;
|
|
568
568
|
|
|
569
569
|
case 'date':
|
|
570
|
-
return toDateOnly(v);
|
|
570
|
+
return dateValues.toDateOnly(v);
|
|
571
571
|
|
|
572
572
|
case 'datetime':
|
|
573
|
-
|
|
574
|
-
|
|
573
|
+
return dateValues.toInstantIso(v);
|
|
574
|
+
|
|
575
|
+
case 'localdatetime':
|
|
576
|
+
return dateValues.toLocalDateTime(v);
|
|
575
577
|
|
|
576
578
|
case 'object':
|
|
577
579
|
case 'array':
|
|
@@ -955,7 +957,17 @@ async function committedMovement(pool, movementId) {
|
|
|
955
957
|
return r.rows[0] || null;
|
|
956
958
|
}
|
|
957
959
|
|
|
960
|
+
/**
|
|
961
|
+
* A zoned date-time column that holds zoneless values, retyped to zoneless with
|
|
962
|
+
* the same digits (read in UTC — how they were written). See the uploader's
|
|
963
|
+
* bootstrap for when this runs.
|
|
964
|
+
*/
|
|
965
|
+
function buildRetypeToLocalDateTimeSql(tableName, column) {
|
|
966
|
+
return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' SET DATA TYPE TIMESTAMP';
|
|
967
|
+
}
|
|
968
|
+
|
|
958
969
|
module.exports = {
|
|
970
|
+
buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
|
|
959
971
|
requires: ['@duckdb/node-api'],
|
|
960
972
|
|
|
961
973
|
connect: connect,
|
package/lib/drivers/db/mssql.js
CHANGED
|
@@ -22,8 +22,9 @@
|
|
|
22
22
|
// number → DECIMAL(38,10) (38 is SQL Server's max precision; > 10
|
|
23
23
|
// fractional digits are rounded)
|
|
24
24
|
// boolean → BIT
|
|
25
|
-
// date
|
|
26
|
-
// datetime→ DATETIME2
|
|
25
|
+
// date → DATE (the same day — dateValues.js)
|
|
26
|
+
// datetime → DATETIME2 (an instant, written in UTC)
|
|
27
|
+
// localdatetime → DATETIME2 (no zone; its digits as they were)
|
|
27
28
|
// object → NVARCHAR(MAX) (SQL Server has no native JSON type)
|
|
28
29
|
// array → NVARCHAR(MAX)
|
|
29
30
|
|
|
@@ -33,6 +34,7 @@ var TYPE_MAP = {
|
|
|
33
34
|
boolean: 'BIT',
|
|
34
35
|
date: 'DATE',
|
|
35
36
|
datetime: 'DATETIME2',
|
|
37
|
+
localdatetime: 'DATETIME2',
|
|
36
38
|
object: 'NVARCHAR(MAX)',
|
|
37
39
|
array: 'NVARCHAR(MAX)'
|
|
38
40
|
};
|
|
@@ -45,30 +47,16 @@ var NATIVE_TO_LOGICAL = {
|
|
|
45
47
|
decimal: 'number', numeric: 'number', float: 'number', real: 'number',
|
|
46
48
|
money: 'number', smallmoney: 'number',
|
|
47
49
|
bit: 'boolean',
|
|
48
|
-
|
|
50
|
+
// No zone: copied as their digits (dateValues.js). datetimeoffset is an
|
|
51
|
+
// instant.
|
|
52
|
+
datetime: 'localdatetime', datetime2: 'localdatetime', smalldatetime: 'localdatetime',
|
|
49
53
|
// 'date' maps to its OWN logical type — see reconcile.js.
|
|
50
54
|
date: 'date', time: 'datetime', datetimeoffset: 'datetime'
|
|
51
55
|
};
|
|
52
56
|
|
|
53
57
|
|
|
54
|
-
//
|
|
55
|
-
|
|
56
|
-
// toISOString() would convert to UTC first, which is exactly the bug: a Date
|
|
57
|
-
// holding 2026-01-05 local becomes 2026-01-04 in UTC anywhere east of
|
|
58
|
-
// Greenwich. The local getters are the right ones here precisely because a
|
|
59
|
-
// business date has no instant to convert.
|
|
60
|
-
function toDateOnly(v) {
|
|
61
|
-
if (v === null || v === undefined) return v;
|
|
62
|
-
if (v instanceof Date) {
|
|
63
|
-
if (isNaN(v.getTime())) return null;
|
|
64
|
-
return v.getFullYear() + '-' +
|
|
65
|
-
String(v.getMonth() + 1).padStart(2, '0') + '-' +
|
|
66
|
-
String(v.getDate()).padStart(2, '0');
|
|
67
|
-
}
|
|
68
|
-
// Already a date string — take the date part and leave the rest alone.
|
|
69
|
-
var m = /^(\d{4}-\d{2}-\d{2})/.exec(String(v));
|
|
70
|
-
return m ? m[1] : v;
|
|
71
|
-
}
|
|
58
|
+
// Dates and times: one shared rule for every driver (dateValues.js).
|
|
59
|
+
var dateValues = require('./dateValues');
|
|
72
60
|
|
|
73
61
|
var INTERNAL_ID_COL = '__xeplr_id__';
|
|
74
62
|
var INTERNAL_MOVEMENT_COL = '__xeplr_movement_id__';
|
|
@@ -92,7 +80,12 @@ async function connect(config) {
|
|
|
92
80
|
// batch and has no session-level read-only mode, so SQL the caller did
|
|
93
81
|
// not write needs a login that can only read (db_datareader). See README.
|
|
94
82
|
config.readOnly ? { readOnlyIntent: true } : {},
|
|
95
|
-
config.options || {}
|
|
83
|
+
config.options || {},
|
|
84
|
+
// PINNED, after the caller's options: a zoneless datetime/date comes
|
|
85
|
+
// back with its digits in the Date's UTC fields, whatever zone this
|
|
86
|
+
// process is in. It was the library's default, which a default change
|
|
87
|
+
// or a caller's `useUTC: false` would have silently undone.
|
|
88
|
+
{ useUTC: true }
|
|
96
89
|
)
|
|
97
90
|
});
|
|
98
91
|
await pool.connect();
|
|
@@ -265,7 +258,7 @@ function coerceValue(v, type) {
|
|
|
265
258
|
|
|
266
259
|
switch (type) {
|
|
267
260
|
case 'string':
|
|
268
|
-
if (v instanceof Date) return
|
|
261
|
+
if (v instanceof Date) return dateValues.toInstantUtcClock(v);
|
|
269
262
|
if (Array.isArray(v)) return JSON.stringify(v);
|
|
270
263
|
if (typeof v === 'object') return JSON.stringify(v);
|
|
271
264
|
return String(v);
|
|
@@ -289,10 +282,13 @@ function coerceValue(v, type) {
|
|
|
289
282
|
return v; // let SQL Server reject
|
|
290
283
|
|
|
291
284
|
case 'date':
|
|
292
|
-
return toDateOnly(v);
|
|
285
|
+
return dateValues.toDateOnly(v);
|
|
293
286
|
|
|
294
287
|
case 'datetime':
|
|
295
|
-
return
|
|
288
|
+
return dateValues.toInstantUtcClock(v);
|
|
289
|
+
|
|
290
|
+
case 'localdatetime':
|
|
291
|
+
return dateValues.toLocalDateTime(v);
|
|
296
292
|
|
|
297
293
|
case 'object':
|
|
298
294
|
case 'array':
|
|
@@ -304,13 +300,6 @@ function coerceValue(v, type) {
|
|
|
304
300
|
}
|
|
305
301
|
}
|
|
306
302
|
|
|
307
|
-
// 'YYYY-MM-DD HH:MM:SS.mmm' in UTC — DATETIME2 parses this unambiguously.
|
|
308
|
-
// Unparseable input passes through so SQL Server rejects it (→ error table).
|
|
309
|
-
function toMssqlDatetime(v) {
|
|
310
|
-
var d = (v instanceof Date) ? v : new Date(v);
|
|
311
|
-
if (isNaN(d.getTime())) return v;
|
|
312
|
-
return d.toISOString().slice(0, 23).replace('T', ' ');
|
|
313
|
-
}
|
|
314
303
|
|
|
315
304
|
async function getTableSchema(pool, tableName, schema) {
|
|
316
305
|
var r = await query(pool,
|
package/lib/drivers/db/mysql.js
CHANGED
|
@@ -19,8 +19,9 @@
|
|
|
19
19
|
// number → DECIMAL(38,10) (exact; > 10 fractional digits are rounded —
|
|
20
20
|
// pre-create a wider column if you need more)
|
|
21
21
|
// boolean → TINYINT(1)
|
|
22
|
-
// date
|
|
23
|
-
// datetime→ DATETIME(6)
|
|
22
|
+
// date → DATE (the same day — dateValues.js)
|
|
23
|
+
// datetime → DATETIME(6) (an instant, written in UTC)
|
|
24
|
+
// localdatetime → DATETIME(6) (no zone; its digits as they were)
|
|
24
25
|
// object → JSON
|
|
25
26
|
// array → JSON
|
|
26
27
|
|
|
@@ -29,7 +30,8 @@ var TYPE_MAP = {
|
|
|
29
30
|
number: 'DECIMAL(38,10)',
|
|
30
31
|
boolean: 'TINYINT(1)',
|
|
31
32
|
date: 'DATE',
|
|
32
|
-
datetime: 'DATETIME(6)',
|
|
33
|
+
datetime: 'DATETIME(6)', // an instant, written in UTC
|
|
34
|
+
localdatetime: 'DATETIME(6)', // a wall-clock reading, written as its digits
|
|
33
35
|
object: 'JSON',
|
|
34
36
|
array: 'JSON'
|
|
35
37
|
};
|
|
@@ -43,30 +45,16 @@ var NATIVE_TO_LOGICAL = {
|
|
|
43
45
|
float: 'number', double: 'number', bit: 'boolean',
|
|
44
46
|
// 'date' maps to its OWN logical type — see reconcile.js. A DATE column
|
|
45
47
|
// created as DATETIME acquires a midnight that shifts under a timezone.
|
|
46
|
-
|
|
48
|
+
// DATETIME has no zone — its digits are copied as they are (dateValues.js).
|
|
49
|
+
// TIMESTAMP is an instant, read in UTC (the session is pinned to it below).
|
|
50
|
+
date: 'date', datetime: 'localdatetime', timestamp: 'datetime',
|
|
47
51
|
time: 'datetime', year: 'datetime',
|
|
48
52
|
json: 'object'
|
|
49
53
|
};
|
|
50
54
|
|
|
51
55
|
|
|
52
|
-
//
|
|
53
|
-
|
|
54
|
-
// toISOString() would convert to UTC first, which is exactly the bug: a Date
|
|
55
|
-
// holding 2026-01-05 local becomes 2026-01-04 in UTC anywhere east of
|
|
56
|
-
// Greenwich. The local getters are the right ones here precisely because a
|
|
57
|
-
// business date has no instant to convert.
|
|
58
|
-
function toDateOnly(v) {
|
|
59
|
-
if (v === null || v === undefined) return v;
|
|
60
|
-
if (v instanceof Date) {
|
|
61
|
-
if (isNaN(v.getTime())) return null;
|
|
62
|
-
return v.getFullYear() + '-' +
|
|
63
|
-
String(v.getMonth() + 1).padStart(2, '0') + '-' +
|
|
64
|
-
String(v.getDate()).padStart(2, '0');
|
|
65
|
-
}
|
|
66
|
-
// Already a date string — take the date part and leave the rest alone.
|
|
67
|
-
var m = /^(\d{4}-\d{2}-\d{2})/.exec(String(v));
|
|
68
|
-
return m ? m[1] : v;
|
|
69
|
-
}
|
|
56
|
+
// Dates and times: one shared rule for every driver (dateValues.js).
|
|
57
|
+
var dateValues = require('./dateValues');
|
|
70
58
|
|
|
71
59
|
var INTERNAL_ID_COL = '__xeplr_id__';
|
|
72
60
|
var INTERNAL_MOVEMENT_COL = '__xeplr_movement_id__';
|
|
@@ -96,6 +84,13 @@ async function connect(config) {
|
|
|
96
84
|
stringifyObjects: true
|
|
97
85
|
// multipleStatements stays at its default (off): one statement per query.
|
|
98
86
|
});
|
|
87
|
+
// THE SESSION IS IN UTC. MySQL hands a TIMESTAMP back in the session's zone
|
|
88
|
+
// (the server's own unless told), so without this the same instant read
|
|
89
|
+
// from a server set to India time came back 5½ hours off. Run first on
|
|
90
|
+
// every connection the pool opens; mysql2 queues it ahead of any query.
|
|
91
|
+
pool.pool.on('connection', function(conn) {
|
|
92
|
+
conn.query("SET time_zone = '+00:00'");
|
|
93
|
+
});
|
|
99
94
|
// READ-ONLY CONNECTIONS — `connect({ ..., readOnly: true })` for SQL the
|
|
100
95
|
// caller did not write itself. Every connection the pool opens is put into
|
|
101
96
|
// read-only transaction mode before it is used, so a write fails.
|
|
@@ -262,7 +257,7 @@ function coerceValue(v, type) {
|
|
|
262
257
|
|
|
263
258
|
switch (type) {
|
|
264
259
|
case 'string':
|
|
265
|
-
if (v instanceof Date) return
|
|
260
|
+
if (v instanceof Date) return dateValues.toInstantUtcClock(v);
|
|
266
261
|
if (Array.isArray(v)) return JSON.stringify(v);
|
|
267
262
|
if (typeof v === 'object') return JSON.stringify(v);
|
|
268
263
|
return String(v);
|
|
@@ -286,10 +281,13 @@ function coerceValue(v, type) {
|
|
|
286
281
|
return v; // let MySQL reject
|
|
287
282
|
|
|
288
283
|
case 'date':
|
|
289
|
-
return toDateOnly(v);
|
|
284
|
+
return dateValues.toDateOnly(v);
|
|
290
285
|
|
|
291
286
|
case 'datetime':
|
|
292
|
-
return
|
|
287
|
+
return dateValues.toInstantUtcClock(v);
|
|
288
|
+
|
|
289
|
+
case 'localdatetime':
|
|
290
|
+
return dateValues.toLocalDateTime(v);
|
|
293
291
|
|
|
294
292
|
case 'object':
|
|
295
293
|
case 'array':
|
|
@@ -301,13 +299,6 @@ function coerceValue(v, type) {
|
|
|
301
299
|
}
|
|
302
300
|
}
|
|
303
301
|
|
|
304
|
-
// 'YYYY-MM-DD HH:MM:SS.mmm' in UTC — the format MySQL DATETIME accepts.
|
|
305
|
-
// Unparseable input passes through so MySQL rejects it (→ error table).
|
|
306
|
-
function toMysqlDatetime(v) {
|
|
307
|
-
var d = (v instanceof Date) ? v : new Date(v);
|
|
308
|
-
if (isNaN(d.getTime())) return v;
|
|
309
|
-
return d.toISOString().slice(0, 23).replace('T', ' ');
|
|
310
|
-
}
|
|
311
302
|
|
|
312
303
|
async function getTableSchema(pool, tableName, schema) {
|
|
313
304
|
var r = await query(pool,
|
|
@@ -22,8 +22,9 @@
|
|
|
22
22
|
// string → TEXT
|
|
23
23
|
// number → NUMERIC (integer + float both handled)
|
|
24
24
|
// boolean → BOOLEAN
|
|
25
|
-
// date
|
|
26
|
-
// datetime→ TIMESTAMPTZ
|
|
25
|
+
// date → DATE (the same day — dateValues.js)
|
|
26
|
+
// datetime → TIMESTAMPTZ (an instant)
|
|
27
|
+
// localdatetime → TIMESTAMP (no zone; its digits as they were)
|
|
27
28
|
// object → JSONB
|
|
28
29
|
// array → JSONB
|
|
29
30
|
|
|
@@ -33,6 +34,7 @@ var TYPE_MAP = {
|
|
|
33
34
|
boolean: 'BOOLEAN',
|
|
34
35
|
date: 'DATE',
|
|
35
36
|
datetime: 'TIMESTAMPTZ',
|
|
37
|
+
localdatetime: 'TIMESTAMP',
|
|
36
38
|
object: 'JSONB',
|
|
37
39
|
array: 'JSONB'
|
|
38
40
|
};
|
|
@@ -60,7 +62,17 @@ async function connect(config) {
|
|
|
60
62
|
user: config.user,
|
|
61
63
|
password: config.password,
|
|
62
64
|
database: config.database,
|
|
63
|
-
max: config.maxConnections || 10
|
|
65
|
+
max: config.maxConnections || 10,
|
|
66
|
+
// THE ZONELESS TYPES COME BACK AS TEXT. node-pg turns a `date` and a
|
|
67
|
+
// `timestamp` into a JS Date read in THIS PROCESS's zone, so the same row
|
|
68
|
+
// read on a server in India and one in New York came back as two
|
|
69
|
+
// different instants — and a date a day apart. As text they are exactly
|
|
70
|
+
// the digits stored. `timestamptz` (1184) is a real instant and stays a
|
|
71
|
+
// Date.
|
|
72
|
+
types: { getTypeParser: function(oid, format) {
|
|
73
|
+
if (oid === 1082 || oid === 1114) return function(text) { return text; };
|
|
74
|
+
return pg.types.getTypeParser(oid, format);
|
|
75
|
+
} }
|
|
64
76
|
});
|
|
65
77
|
if (config.readOnly) pool.__xeplrReadOnly = true;
|
|
66
78
|
return pool;
|
|
@@ -98,24 +110,9 @@ async function query(pool, sql, params) {
|
|
|
98
110
|
|
|
99
111
|
function toPgType(type) { return TYPE_MAP[type] || 'TEXT'; }
|
|
100
112
|
|
|
101
|
-
//
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
// holding 2026-01-05 local becomes 2026-01-04 in UTC anywhere east of
|
|
105
|
-
// Greenwich. The local getters are the right ones here precisely because a
|
|
106
|
-
// business date has no instant to convert.
|
|
107
|
-
function toDateOnly(v) {
|
|
108
|
-
if (v === null || v === undefined) return v;
|
|
109
|
-
if (v instanceof Date) {
|
|
110
|
-
if (isNaN(v.getTime())) return null;
|
|
111
|
-
return v.getFullYear() + '-' +
|
|
112
|
-
String(v.getMonth() + 1).padStart(2, '0') + '-' +
|
|
113
|
-
String(v.getDate()).padStart(2, '0');
|
|
114
|
-
}
|
|
115
|
-
// Already a date string — take the date part and leave the rest alone.
|
|
116
|
-
var m = /^(\d{4}-\d{2}-\d{2})/.exec(String(v));
|
|
117
|
-
return m ? m[1] : v;
|
|
118
|
-
}
|
|
113
|
+
// Dates and times: one shared rule for every driver (dateValues.js).
|
|
114
|
+
var dateValues = require('./dateValues');
|
|
115
|
+
var toDateOnly = dateValues.toDateOnly;
|
|
119
116
|
|
|
120
117
|
|
|
121
118
|
// Double-quote a Postgres identifier, escaping embedded quotes.
|
|
@@ -251,8 +248,10 @@ function coerceValue(v, type) {
|
|
|
251
248
|
return toDateOnly(v);
|
|
252
249
|
|
|
253
250
|
case 'datetime':
|
|
254
|
-
|
|
255
|
-
|
|
251
|
+
return dateValues.toInstantIso(v);
|
|
252
|
+
|
|
253
|
+
case 'localdatetime':
|
|
254
|
+
return dateValues.toLocalDateTime(v);
|
|
256
255
|
|
|
257
256
|
case 'object':
|
|
258
257
|
case 'array':
|
|
@@ -395,7 +394,17 @@ async function rollbackMovement(pool, tableName, movementId) {
|
|
|
395
394
|
return result;
|
|
396
395
|
}
|
|
397
396
|
|
|
397
|
+
/**
|
|
398
|
+
* A zoned date-time column that holds zoneless values, retyped to zoneless with
|
|
399
|
+
* the same digits (read in UTC — how they were written). See the uploader's
|
|
400
|
+
* bootstrap for when this runs.
|
|
401
|
+
*/
|
|
402
|
+
function buildRetypeToLocalDateTimeSql(tableName, column) {
|
|
403
|
+
return 'ALTER TABLE ' + quoteIdent(tableName) + ' ALTER COLUMN ' + quoteIdent(column) + ' TYPE timestamp USING ' + quoteIdent(column) + " AT TIME ZONE 'UTC'";
|
|
404
|
+
}
|
|
405
|
+
|
|
398
406
|
module.exports = {
|
|
407
|
+
buildRetypeToLocalDateTimeSql: buildRetypeToLocalDateTimeSql,
|
|
399
408
|
requires: ['pg'],
|
|
400
409
|
|
|
401
410
|
// Connection lifecycle
|
|
@@ -28,6 +28,7 @@
|
|
|
28
28
|
var unzipper = require('unzipper');
|
|
29
29
|
var { SaxesParser } = require('saxes');
|
|
30
30
|
var { StringDecoder } = require('string_decoder');
|
|
31
|
+
var { decodeOoxml } = require('./pastedRows');
|
|
31
32
|
|
|
32
33
|
// Strip an XML namespace prefix: "x:row" -> "row", "row" -> "row".
|
|
33
34
|
function localName(n) {
|
|
@@ -194,18 +195,25 @@ async function parseSharedStrings(directory) {
|
|
|
194
195
|
var f = findFile(directory, 'xl/sharedStrings.xml');
|
|
195
196
|
if (!f) return null;
|
|
196
197
|
var strings = [];
|
|
197
|
-
|
|
198
|
+
// inRph: inside a phonetic run (<rPh>, the furigana Japanese Excel stores
|
|
199
|
+
// beside a cell's text). Its <t> is a reading aid, not the cell's value —
|
|
200
|
+
// joining it in gave "東京とうきょう" for 東京.
|
|
201
|
+
var cur = null, inT = 0, inRph = 0, buf = '';
|
|
198
202
|
await saxParseStream(f.stream(), {
|
|
199
203
|
opentag: function(tag) {
|
|
200
204
|
var ln = localName(tag.name);
|
|
201
|
-
if (ln === 'si')
|
|
202
|
-
else if (ln === '
|
|
205
|
+
if (ln === 'si') { cur = ''; }
|
|
206
|
+
else if (ln === 'rPh') { inRph++; }
|
|
207
|
+
else if (ln === 't') { inT++; buf = ''; }
|
|
203
208
|
},
|
|
204
209
|
text: function(t) { if (inT > 0) buf += t; },
|
|
205
210
|
closetag: function(tag) {
|
|
206
211
|
var ln = localName(tag.name);
|
|
207
|
-
if (ln === 't')
|
|
208
|
-
else if (ln === '
|
|
212
|
+
if (ln === 't') { inT--; if (!inRph) cur = (cur || '') + buf; buf = ''; }
|
|
213
|
+
else if (ln === 'rPh') { inRph--; }
|
|
214
|
+
// Excel's own escapes (_x000D_ for a line break inside the cell) decoded
|
|
215
|
+
// here, once per distinct string.
|
|
216
|
+
else if (ln === 'si') { strings.push(decodeOoxml(cur || '')); cur = null; }
|
|
209
217
|
}
|
|
210
218
|
});
|
|
211
219
|
return strings;
|
|
@@ -288,7 +296,7 @@ async function stream_xlsx_sheet_rows(filePath, sheet_name, handlers) {
|
|
|
288
296
|
} else if (curType === 'b') {
|
|
289
297
|
val = textBuf === '1';
|
|
290
298
|
} else if (curType === 'str' || curType === 'inlineStr' || closingLocal === 't') {
|
|
291
|
-
val = textBuf;
|
|
299
|
+
val = decodeOoxml(textBuf);
|
|
292
300
|
} else {
|
|
293
301
|
var n = Number(textBuf);
|
|
294
302
|
val = (textBuf !== '' && !Number.isNaN(n)) ? n : (textBuf === '' ? null : textBuf);
|
package/lib/formats/index.js
CHANGED
|
@@ -33,8 +33,27 @@ function checkFormatRequires(actionName, format, mod) {
|
|
|
33
33
|
}
|
|
34
34
|
}
|
|
35
35
|
|
|
36
|
+
/**
|
|
37
|
+
* Which cells of a file hold rows a paste folded into them (pastedRows.js),
|
|
38
|
+
* read without changing anything — for an import's inspect step.
|
|
39
|
+
* opts: the format's own options (`sheet` for excel), plus `examples`.
|
|
40
|
+
*/
|
|
41
|
+
async function scanPastedRows(type, filePath, opts) {
|
|
42
|
+
var format = getFormat(type);
|
|
43
|
+
var config = opts || {};
|
|
44
|
+
var rows = format.inputMode === 'path'
|
|
45
|
+
? format.parsePath(filePath, config)
|
|
46
|
+
: format.parseStream(require('fs').createReadStream(filePath), config);
|
|
47
|
+
return pastedRows.scanForPastedRows(rows, { examples: config.examples });
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
var pastedRows = require('./pastedRows');
|
|
51
|
+
|
|
36
52
|
module.exports = {
|
|
37
53
|
getFormat: getFormat,
|
|
38
54
|
checkFormatRequires: checkFormatRequires,
|
|
39
|
-
SUPPORTED: Object.keys(FORMATS)
|
|
55
|
+
SUPPORTED: Object.keys(FORMATS),
|
|
56
|
+
scanPastedRows: scanPastedRows,
|
|
57
|
+
// decodeOoxml, splitPastedRows, expandPastedRows, scanForPastedRows
|
|
58
|
+
pastedRows: pastedRows
|
|
40
59
|
};
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
// ROWS PASTED INTO ONE CELL — found, and put back.
|
|
2
|
+
//
|
|
3
|
+
// Data copied out of an ERP is tab-separated text, a row per line. Pasted into
|
|
4
|
+
// Excel, a value that STARTS with a double quote (an inch mark: `"4 INCH`)
|
|
5
|
+
// switches Excel into quoted mode, and everything up to the next `"` — tabs,
|
|
6
|
+
// line breaks, whole rows — lands in that one cell. The sheet then holds one
|
|
7
|
+
// row whose cell carries the next N rows, and those rows exist nowhere else.
|
|
8
|
+
// Seen for real: 465 rows folded into 11 Item Description cells.
|
|
9
|
+
//
|
|
10
|
+
// The damage has an exact shape, which is what makes it safe to undo:
|
|
11
|
+
//
|
|
12
|
+
// cell = <its own row's value> \t <its own row's remaining columns>
|
|
13
|
+
// \n <a whole row> \n <a whole row> …
|
|
14
|
+
// \n <the last row's columns up to and including this one>
|
|
15
|
+
//
|
|
16
|
+
// and the merged row's OTHER columns after the cell belong to that last row.
|
|
17
|
+
// So the cell splits on its line breaks into a first piece of (n - c) values,
|
|
18
|
+
// whole rows of n values, and a last piece of L values — n columns, the cell
|
|
19
|
+
// at column c. Usually L is c + 1 (the closing quote was in the same column);
|
|
20
|
+
// when it closed in a later column k, L is k + 1, and the last row's
|
|
21
|
+
// remaining (n - L) values sit in the merged row's cells right after c, the
|
|
22
|
+
// cells beyond them empty. A cell that does not split exactly like that is
|
|
23
|
+
// left alone: ordinary multi-line text never has those counts, so a genuine
|
|
24
|
+
// note with line breaks is never "repaired".
|
|
25
|
+
//
|
|
26
|
+
// The quote that opened the cell and the one that closed it were eaten by the
|
|
27
|
+
// paste; they are put back — at the start of the first row's value, where
|
|
28
|
+
// quoted mode began, and at the end of the last row's value where it ended
|
|
29
|
+
// (an inch mark: `Painting brush 4"`).
|
|
30
|
+
|
|
31
|
+
// Excel's own escape inside cell text: `_xHHHH_` is the character with that
|
|
32
|
+
// hex code. A line break inside a cell is saved as `_x000D_` + a real newline;
|
|
33
|
+
// `_x005F_` is a literal underscore, which is how text that really says
|
|
34
|
+
// "_x000D_" is stored ("_x005F_x000D_") — left to right, that decodes right.
|
|
35
|
+
var ESCAPE = /_x([0-9A-Fa-f]{4})_/g;
|
|
36
|
+
|
|
37
|
+
function decodeOoxml(s) {
|
|
38
|
+
if (typeof s !== 'string' || s.indexOf('_x') === -1) return s;
|
|
39
|
+
return s.replace(ESCAPE, function(_, hex) { return String.fromCharCode(parseInt(hex, 16)); });
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
var LINE = /\r\n|\r|\n/;
|
|
43
|
+
var NUMBER = /^-?\d+(\.\d+)?$/;
|
|
44
|
+
|
|
45
|
+
// A value from inside the cell: text, typed as the sheet's own cells would be.
|
|
46
|
+
function cast(v) {
|
|
47
|
+
if (v === '') return null;
|
|
48
|
+
return NUMBER.test(v) ? Number(v) : v;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
/**
|
|
52
|
+
* The rows folded into one of `rec`'s cells, put back — or null when no cell
|
|
53
|
+
* has that exact shape. `headers` in sheet order.
|
|
54
|
+
*
|
|
55
|
+
* @returns {{ column: string, rows: object[] } | null} rows[0] is `rec` itself, rebuilt
|
|
56
|
+
*/
|
|
57
|
+
function splitPastedRows(rec, headers) {
|
|
58
|
+
var n = headers.length;
|
|
59
|
+
for (var c = 0; c < n; c++) {
|
|
60
|
+
var cell = decodeOoxml(rec[headers[c]]);
|
|
61
|
+
if (typeof cell !== 'string' || cell.indexOf('\t') === -1 || !LINE.test(cell)) continue;
|
|
62
|
+
var lines = cell.split(LINE);
|
|
63
|
+
var first = lines[0].split('\t');
|
|
64
|
+
var last = lines[lines.length - 1].split('\t');
|
|
65
|
+
var middle = lines.slice(1, -1).map(function(l) { return l.split('\t'); });
|
|
66
|
+
var L = last.length;
|
|
67
|
+
if (first.length !== n - c || L < c + 1 || L > n) continue;
|
|
68
|
+
if (middle.some(function(m) { return m.length !== n; })) continue;
|
|
69
|
+
// The last row's values after the closing quote: the merged row's cells
|
|
70
|
+
// right after c — and every cell beyond those must be empty, or this is
|
|
71
|
+
// not the shape a paste leaves.
|
|
72
|
+
var tail = headers.slice(c + 1, c + 1 + (n - L));
|
|
73
|
+
var beyond = headers.slice(c + 1 + (n - L));
|
|
74
|
+
if (beyond.some(function(h) { return rec[h] !== null && rec[h] !== undefined && rec[h] !== ''; })) continue;
|
|
75
|
+
|
|
76
|
+
var rows = [];
|
|
77
|
+
var top = {};
|
|
78
|
+
headers.forEach(function(h, i) { top[h] = i < c ? rec[h] : cast(first[i - c]); });
|
|
79
|
+
top[headers[c]] = '"' + first[0];
|
|
80
|
+
rows.push(top);
|
|
81
|
+
middle.forEach(function(m) {
|
|
82
|
+
var r = {};
|
|
83
|
+
headers.forEach(function(h, i) { r[h] = cast(m[i]); });
|
|
84
|
+
rows.push(r);
|
|
85
|
+
});
|
|
86
|
+
var end = {};
|
|
87
|
+
headers.forEach(function(h, i) { end[h] = i < L ? cast(last[i]) : rec[tail[i - L]]; });
|
|
88
|
+
// The closing quote, at the end of the value it closed.
|
|
89
|
+
end[headers[L - 1]] = String(last[L - 1]) + '"';
|
|
90
|
+
rows.push(end);
|
|
91
|
+
return { column: headers[c], rows: rows };
|
|
92
|
+
}
|
|
93
|
+
return null;
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Rows as they were before the paste folded them: each record whose cell
|
|
98
|
+
* holds pasted rows becomes those rows. Everything else passes through.
|
|
99
|
+
* `onRepair({ row, column, rows })` hears each one (row: 1-based data row).
|
|
100
|
+
*/
|
|
101
|
+
async function* expandPastedRows(records, onRepair) {
|
|
102
|
+
var headers = null;
|
|
103
|
+
var index = 0;
|
|
104
|
+
for await (var rec of records) {
|
|
105
|
+
index++;
|
|
106
|
+
if (!headers) headers = Object.keys(rec);
|
|
107
|
+
var split = splitPastedRows(rec, headers);
|
|
108
|
+
if (!split) { yield rec; continue; }
|
|
109
|
+
if (onRepair) onRepair({ row: index, column: split.column, rows: split.rows.length - 1 });
|
|
110
|
+
for (var r of split.rows) yield r;
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
/**
|
|
115
|
+
* Read a whole file's rows and say which cells hold pasted rows, without
|
|
116
|
+
* changing anything — for the inspect step, before anything is loaded.
|
|
117
|
+
*
|
|
118
|
+
* @returns {{ cells: number, rows: number, examples: [{ row, column, rows, first }] }}
|
|
119
|
+
* rows: how many rows the repair would recover
|
|
120
|
+
*/
|
|
121
|
+
async function scanForPastedRows(records, opts) {
|
|
122
|
+
var limit = (opts && opts.examples) || 5;
|
|
123
|
+
var headers = null;
|
|
124
|
+
var index = 0;
|
|
125
|
+
var found = { cells: 0, rows: 0, examples: [] };
|
|
126
|
+
for await (var rec of records) {
|
|
127
|
+
index++;
|
|
128
|
+
if (!headers) headers = Object.keys(rec);
|
|
129
|
+
var split = splitPastedRows(rec, headers);
|
|
130
|
+
if (!split) continue;
|
|
131
|
+
found.cells++;
|
|
132
|
+
found.rows += split.rows.length - 1;
|
|
133
|
+
if (found.examples.length < limit) {
|
|
134
|
+
found.examples.push({ row: index, column: split.column, rows: split.rows.length - 1,
|
|
135
|
+
first: String(split.rows[0][split.column]).slice(0, 80) });
|
|
136
|
+
}
|
|
137
|
+
}
|
|
138
|
+
return found;
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
module.exports = {
|
|
142
|
+
decodeOoxml: decodeOoxml,
|
|
143
|
+
splitPastedRows: splitPastedRows,
|
|
144
|
+
expandPastedRows: expandPastedRows,
|
|
145
|
+
scanForPastedRows: scanForPastedRows
|
|
146
|
+
};
|
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
// same memory as a hundred. Only the first SAMPLE_ROWS are held, to size the
|
|
18
18
|
// columns, because <cols> must come before the rows in the sheet.
|
|
19
19
|
|
|
20
|
+
var dateValues = require('../../drivers/db/dateValues');
|
|
20
21
|
var fs = require('fs');
|
|
21
22
|
|
|
22
23
|
var MAX_ROWS = 1048576; // Excel's own limit, header and title rows included
|
|
@@ -38,9 +39,14 @@ function colName(i) {
|
|
|
38
39
|
return s;
|
|
39
40
|
}
|
|
40
41
|
|
|
41
|
-
/**
|
|
42
|
+
/**
|
|
43
|
+
* A Date, or an ISO date string, as an Excel serial — or null. A string with
|
|
44
|
+
* no zone is read as UTC (dateValues.toInstant), never in this machine's zone:
|
|
45
|
+
* `new Date('2025-08-01 10:00')` would put a different time in the cell on a
|
|
46
|
+
* server in India than on one in New York.
|
|
47
|
+
*/
|
|
42
48
|
function dateSerial(value) {
|
|
43
|
-
var d = value instanceof Date ? value : (typeof value === 'string'
|
|
49
|
+
var d = value instanceof Date ? value : (typeof value === 'string' ? dateValues.toInstant(value) : null);
|
|
44
50
|
if (!d || isNaN(d.getTime())) return null;
|
|
45
51
|
return d.getTime() / 86400000 + 25569;
|
|
46
52
|
}
|
package/lib/uploader/index.js
CHANGED
|
@@ -410,6 +410,20 @@ async function runBootstrap(driver, pool, targetTable, columns, primaryKeys, sta
|
|
|
410
410
|
var { reconcileColumns } = require('./reconcile');
|
|
411
411
|
var reconciled = reconcileColumns(columns, targetSchema, driver.dataTypeToLogical);
|
|
412
412
|
|
|
413
|
+
// A ZONELESS SOURCE COLUMN IN A ZONED TARGET COLUMN. Tables created before
|
|
414
|
+
// zoneless date-times had their own type hold them as instants (the digits
|
|
415
|
+
// read as UTC). The target's type would win and keep it so forever — so it
|
|
416
|
+
// is retyped to zoneless, once, keeping those same digits. Only where the
|
|
417
|
+
// dialect has the two kinds (Postgres, DuckDB); MySQL DATETIME and SQL
|
|
418
|
+
// Server DATETIME2 are zoneless already.
|
|
419
|
+
var retype = reconciled.filter(function(c) { return c.sourceType === 'localdatetime' && c.targetType === 'datetime'; });
|
|
420
|
+
if (retype.length && typeof driver.buildRetypeToLocalDateTimeSql === 'function') {
|
|
421
|
+
for (var r = 0; r < retype.length; r++) {
|
|
422
|
+
await driver.query(pool, driver.buildRetypeToLocalDateTimeSql(targetTable, retype[r].name));
|
|
423
|
+
}
|
|
424
|
+
reconciled = reconcileColumns(columns, await driver.getTableSchema(pool, targetTable), driver.dataTypeToLogical);
|
|
425
|
+
}
|
|
426
|
+
|
|
413
427
|
var missing = reconciled.filter(function(c) { return c.missing; });
|
|
414
428
|
if (missing.length > 0) {
|
|
415
429
|
throw new Error(
|
|
@@ -29,7 +29,8 @@ var PG_TO_LOGICAL = {
|
|
|
29
29
|
'double precision': 'number',
|
|
30
30
|
'boolean': 'boolean',
|
|
31
31
|
'timestamp with time zone': 'datetime',
|
|
32
|
-
|
|
32
|
+
// No zone: copied as its digits, never as an instant (drivers/db/dateValues.js).
|
|
33
|
+
'timestamp without time zone': 'localdatetime',
|
|
33
34
|
// A DATE IS NOT A TIMESTAMP, and mapping it to one moves it.
|
|
34
35
|
//
|
|
35
36
|
// This said 'datetime', so a source `date` was created as TIMESTAMPTZ in the
|
package/package.json
CHANGED