@xeplr/imports 2.0.0 → 2.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,10 +1,10 @@
1
1
  # @xeplr/imports
2
2
 
3
- Data imports as a drop-in module. Upload a CSV, Excel or JSON file and check it against the table it is going into. Then load it as a **New table**, an **Append** or a **Replace**. Afterwards you can see and download the rows the database refused, and roll the import back. One package holds the routes, the access catalog and the settings. The screen is [`ImportsPage` in `@xeplr/ui-utils`](https://www.npmjs.com/package/@xeplr/ui-utils).
3
+ Data imports as a drop-in module. Upload a CSV, Excel, JSON or Parquet file and check it against the table it is going into. Then load it as a **New table**, an **Append** or a **Replace**. Afterwards you can see and download the rows the database refused, and roll the import back. One package holds the routes, the access catalog and the settings. The screen is [`ImportsPage` in `@xeplr/ui-utils`](https://www.npmjs.com/package/@xeplr/ui-utils).
4
4
 
5
5
  ## What you get
6
6
 
7
- - **Big files, no waiting.** CSV, Excel and JSON files of hundreds of megabytes upload and load in the background, with live progress and a stop button.
7
+ - **Big files, no waiting.** CSV, Excel, JSON and Parquet files of hundreds of megabytes upload and load in the background, with live progress and a stop button.
8
8
  - **New table, append or replace.** You choose what happens to the table. A typo in a table's name is caught, so it never turns into a new table.
9
9
  - **Replace without risk.** The table is only swapped once every row has loaded. An import that fails or is stopped leaves your data exactly as it was.
10
10
  - **Checked before it loads.** The file's columns are compared with the table first. You see what will be refused, what will be left empty and what matches, before anything is touched.
@@ -13,6 +13,10 @@ Data imports as a drop-in module. Upload a CSV, Excel or JSON file and check it
13
13
  - **Full history and undo.** Every import is recorded with who ran it, where it went and how it went, and can be rolled back.
14
14
  - **Drop-in.** One router and one screen, and the app supplies only its database connections.
15
15
 
16
+ **Parquet** needs `@duckdb/node-api` installed in the app (an optional peer of `@xeplr/actions`, which reads Parquet through DuckDB). Its columns keep their own types in the table: a number stays a number, a date a date, a yes/no a boolean. Without DuckDB a Parquet file is refused when it is read, saying what to install.
17
+
18
+ **Files larger than `UPLOAD_MAX_MB`** — gigabytes — need sending in pieces rather than in one request. That is the app's to provide: a file sent another way into the tenant's `uploads/` folder is imported like any other (`POST /inspect` and `/commit` take any `filePath` inside it). `ImportsPage`'s `upload` prop (`@xeplr/ui-utils`) is where an app plugs its own sender in.
19
+
16
20
  Every step underneath is [`@xeplr/actions`](https://www.npmjs.com/package/@xeplr/actions): its file readers, the uploader, the error table and staged loads. This package puts those steps behind HTTP and leaves the app only what is its own.
17
21
 
18
22
  ## Using it
@@ -117,6 +121,7 @@ The real router runs in an Express app, against real Postgres and a throwaway `x
117
121
  - refused paths and types
118
122
  - pasted rows found and recovered
119
123
  - each mode and its refusals
124
+ - a Parquet file inspected, checked and loaded with its own types (a timestamp, a boolean, a number)
120
125
  - refused rows shown and downloaded
121
126
  - history scoped to a workspace
122
127
  - rollback, with a Replace refused
package/lib/imports.js CHANGED
@@ -7,7 +7,9 @@ var fs = require('fs');
7
7
  var path = require('path');
8
8
  var { runAction, builtins, formats, uploader, drivers, dataDir } = require('@xeplr/actions');
9
9
 
10
- var EXT_TO_FORMAT = { '.csv': 'csv', '.json': 'json', '.xlsx': 'excel', '.xls': 'excel' };
10
+ // Parquet: typed columns, read through DuckDB (@xeplr/actions 2.2.7+), so a
11
+ // number stays a number and a date a date in the table it goes into.
12
+ var EXT_TO_FORMAT = { '.csv': 'csv', '.json': 'json', '.xlsx': 'excel', '.xls': 'excel', '.parquet': 'parquet' };
11
13
 
12
14
  // Rows the dropped-rows view shows at once, and the most a download carries
13
15
  // (Excel's own limit — past it the sheet would be cut anyway).
package/lib/router.js CHANGED
@@ -70,7 +70,7 @@ function createRouter(options) {
70
70
  if (!filePath) return { status: 400, error: 'Invalid filePath' };
71
71
  if (!fs.existsSync(filePath)) return { status: 404, error: 'Upload not found — it may have expired. Please upload the file again.' };
72
72
  var format = imports.formatOf(filePath);
73
- if (!format) return { status: 400, error: 'Unsupported file type — use CSV, JSON, or Excel (.xlsx/.xls)' };
73
+ if (!format) return { status: 400, error: 'Unsupported file type — use CSV, JSON, Excel (.xlsx/.xls) or Parquet' };
74
74
  return { filePath: filePath, format: format };
75
75
  }
76
76
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@xeplr/imports",
3
- "version": "2.0.0",
3
+ "version": "2.0.1",
4
4
  "description": "Data imports as a drop-in module: upload a CSV/Excel/JSON file, check it against the table, load it as a New table, Append or Replace, see and download the rows the database refused, and roll back — routes, access catalog and env in one package",
5
5
  "main": "index.js",
6
6
  "files": [
@@ -29,7 +29,7 @@
29
29
  "access": "public"
30
30
  },
31
31
  "dependencies": {
32
- "@xeplr/actions": "^2.0.0",
32
+ "@xeplr/actions": "^2.2.10",
33
33
  "@xeplr/base-apis": "^2.0.3",
34
34
  "@xeplr/db": "^1.0.8",
35
35
  "@xeplr/utils": "^1.1.0",
@@ -46,6 +46,7 @@
46
46
  "test": "node --test test/*.test.js"
47
47
  },
48
48
  "devDependencies": {
49
+ "@duckdb/node-api": "^1.5.5-r.3",
49
50
  "express": "^5.3.0",
50
51
  "knex": "^3.3.0",
51
52
  "mysql2": "^3.24.5",