@forzalabs/remora 2.2.0 → 2.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/index.js +126 -41
- package/package.json +1 -1
- package/workers/ExecutorWorker.js +126 -41
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,14 @@ The format is based on Keep a Changelog, and this project adheres to Semantic Ve
|
|
|
6
6
|
|
|
7
7
|
## Unreleased
|
|
8
8
|
|
|
9
|
+
## V 2.2.1 - 2026-10-15
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
|
|
13
|
+
- **A compressed source file that could not be read failed the run with `unexpected end of file` and nothing else.** zlib reports WHAT went wrong and never WHICH file it was reading, and nothing between it and the CLI added the name back, so a producer staging hundreds of files through a `%` fileKey died with a six-word string that named no file, no producer and no consumer stage — the one thing needed to go and look at it. Worse, that one string covers three different mistakes: a truncated archive, a **zero-byte** file, and (as `incorrect header check`) a file that is not compressed at all. The first two are what a drop folder produces constantly, since a file being written into one exists at zero bytes before it holds anything. Every decompression failure is now re-thrown naming the file and the declared format — `Failed to decompress "/data/claims/claimline_2026_03.csv.gz" as GZ: unexpected end of file. The archive ends mid-stream, so it is truncated...` — and the two cases that are decidable before any decompressor runs are refused ahead of it by their own name: an empty file says it is empty and says that a writer should stage under a temporary name and rename into place, and a file whose first bytes are not the declared format's signature (`1f 8b` for GZ, `50 4b` for ZIP) says it is not that archive whatever its name says. TAR has no signature at offset 0, so it is left to the extractor. ZIP and TAR failures gain the same naming, since yauzl and tar are no more forthcoming than zlib. An unsupported `compressionType` is also now refused as a precondition, naming the three that exist, instead of from inside the switch
|
|
14
|
+
|
|
15
|
+
- **The file log sink could be ended while the records it had accepted were still in flight.** A winston logger is a Transform piped into its transports, and `Logger._final` ends every transport the moment the logger's writable side closes — while records already accepted sit in its readable buffer waiting for the transport to have room. The transport then drains, the pipe resumes on that drain, and the next record is written into a stream that has ended: `ERR_STREAM_WRITE_AFTER_END`, raised from a stream with no line of remora's in its stack. Re-pointing the sink and closing it now drain the in-flight records into the transports first, and a re-point releases the old logger only after the new one is serving, so nothing can be routed to a logger on its way out. `flush()` also writes ONE sentinel record rather than one per transport — every transport sees every record, so a sentinel each only wrote N blank lines into the same file
|
|
16
|
+
|
|
9
17
|
## V 2.2.0 - 2026-10-15
|
|
10
18
|
|
|
11
19
|
**Upgrading from 2.1.** Three changes can alter what an existing project does, despite the minor version. `nullable` on a producer dimension is no longer documentation: it defaults to `true` and keeps a missing value as `null` as before, but a dimension that declares `"nullable": false` now gets the type's own empty value (`""`, `0`, `false`) where it used to get `null`. A `NULL` in a `number` column now stages as `null` rather than as `0`, so a consumer that was summing or filtering over those zeros sees different figures — the fix is in the staging of every CSV-staged source (SQL, Delta Share, local and S3 CSV). And `settings.sql` on a producer is gone from the schema, so a producer still carrying it fails `compile` — the property never did anything; delete it. Also worth a glance: `remora package` writes `./remora.zip` instead of `./remora-config.zip` when no `-o` is given, and the resource folder lists in `project.json` (`sources`, `producers`, `consumers`, `schemas`, `codeSets`) are ignored — the folders are fixed. They still validate, so nothing breaks, but a project pointing them somewhere else was already not being read that way.
|
package/index.js
CHANGED
|
@@ -11114,8 +11114,7 @@ var init_FileLogService = __esm({
|
|
|
11114
11114
|
return;
|
|
11115
11115
|
if (!(0, import_fs.existsSync)(folder))
|
|
11116
11116
|
(0, import_fs.mkdirSync)(folder, { recursive: true });
|
|
11117
|
-
|
|
11118
|
-
this._logger.end();
|
|
11117
|
+
const previous = this._logger;
|
|
11119
11118
|
this._folder = folder;
|
|
11120
11119
|
this._file = file;
|
|
11121
11120
|
this._logger = import_winston.default.createLogger({
|
|
@@ -11143,6 +11142,7 @@ ${stack}` : base;
|
|
|
11143
11142
|
});
|
|
11144
11143
|
this._logger.on("error", (error) => console.error(`File logger error: ${error?.message ?? String(error)}`));
|
|
11145
11144
|
this._enabled = true;
|
|
11145
|
+
if (previous) void this._endSafely(previous);
|
|
11146
11146
|
};
|
|
11147
11147
|
/**
|
|
11148
11148
|
* Where the file sink writes, whether or not it is currently enabled.
|
|
@@ -11168,19 +11168,31 @@ ${stack}` : base;
|
|
|
11168
11168
|
};
|
|
11169
11169
|
this.flush = () => {
|
|
11170
11170
|
if (!this._enabled || !this._logger) return Promise.resolve();
|
|
11171
|
+
return this.drain(this._logger);
|
|
11172
|
+
};
|
|
11173
|
+
/**
|
|
11174
|
+
* Resolves once every record already handed to `target` has reached its transports.
|
|
11175
|
+
*
|
|
11176
|
+
* A winston logger is a Transform piped into each transport, so a record accepted by `log()` is
|
|
11177
|
+
* not a record that has been written: it sits in the logger's readable buffer until the transport
|
|
11178
|
+
* has room. Writing one sentinel and waiting for every transport to report it `logged` is what
|
|
11179
|
+
* establishes that the queue ahead of it has gone — records travel the pipe in order, so the
|
|
11180
|
+
* sentinel landing means everything before it landed.
|
|
11181
|
+
*/
|
|
11182
|
+
this.drain = (target) => {
|
|
11171
11183
|
return new Promise((resolve) => {
|
|
11172
|
-
let pending =
|
|
11184
|
+
let pending = target.transports.length;
|
|
11173
11185
|
if (pending === 0) return resolve();
|
|
11174
|
-
for (const transport of
|
|
11175
|
-
if (typeof transport.on
|
|
11176
|
-
transport.once("logged", () => {
|
|
11177
|
-
if (--pending === 0) resolve();
|
|
11178
|
-
});
|
|
11179
|
-
this._logger.info("");
|
|
11180
|
-
} else {
|
|
11186
|
+
for (const transport of target.transports) {
|
|
11187
|
+
if (typeof transport.on !== "function") {
|
|
11181
11188
|
if (--pending === 0) resolve();
|
|
11189
|
+
continue;
|
|
11182
11190
|
}
|
|
11191
|
+
transport.once("logged", () => {
|
|
11192
|
+
if (--pending === 0) resolve();
|
|
11193
|
+
});
|
|
11183
11194
|
}
|
|
11195
|
+
target.info("");
|
|
11184
11196
|
});
|
|
11185
11197
|
};
|
|
11186
11198
|
/**
|
|
@@ -11191,15 +11203,37 @@ ${stack}` : base;
|
|
|
11191
11203
|
* listening to. Leaving `_enabled` true after a close is what turned "someone logged after
|
|
11192
11204
|
* close" into "the process died". Use `flush()` when the sink is still needed afterwards.
|
|
11193
11205
|
*/
|
|
11194
|
-
this.close = () => {
|
|
11195
|
-
if (!this._enabled || !this._logger) return
|
|
11206
|
+
this.close = async () => {
|
|
11207
|
+
if (!this._enabled || !this._logger) return;
|
|
11196
11208
|
const closing = this._logger;
|
|
11197
11209
|
this._enabled = false;
|
|
11198
11210
|
this._logger = null;
|
|
11199
|
-
|
|
11200
|
-
|
|
11201
|
-
|
|
11202
|
-
|
|
11211
|
+
await this._endSafely(closing);
|
|
11212
|
+
};
|
|
11213
|
+
/**
|
|
11214
|
+
* Ends a winston logger without the race its own `end()` carries.
|
|
11215
|
+
*
|
|
11216
|
+
* `Logger._final` ends EVERY TRANSPORT the moment the logger's writable side closes — while the
|
|
11217
|
+
* records it already accepted are still travelling down the pipe from its readable side. The
|
|
11218
|
+
* transport drains, the pipe resumes on that drain, and the next record is written into a stream
|
|
11219
|
+
* that has ended: ERR_STREAM_WRITE_AFTER_END, raised from a stream no line of ours is standing
|
|
11220
|
+
* next to. In an executor thread under a debug run, where the pipe is never empty, that is an
|
|
11221
|
+
* error per rotation and a stack with nothing of remora's in it. Draining the in-flight records
|
|
11222
|
+
* into the transports first is what removes the race rather than hiding it.
|
|
11223
|
+
*
|
|
11224
|
+
* A failure here is swallowed on purpose: this logger is on its way out, it carries its own
|
|
11225
|
+
* `error` handler, and a sink that cannot close must not take the caller's shutdown with it.
|
|
11226
|
+
*/
|
|
11227
|
+
this._endSafely = async (target) => {
|
|
11228
|
+
try {
|
|
11229
|
+
await this.drain(target);
|
|
11230
|
+
await new Promise((resolve) => {
|
|
11231
|
+
target.once("finish", resolve);
|
|
11232
|
+
target.end();
|
|
11233
|
+
});
|
|
11234
|
+
} catch (error) {
|
|
11235
|
+
console.error(`File logger failed to close: ${error?.message ?? String(error)}`);
|
|
11236
|
+
}
|
|
11203
11237
|
};
|
|
11204
11238
|
}
|
|
11205
11239
|
};
|
|
@@ -11727,7 +11761,7 @@ var CONSTANTS, Constants_default;
|
|
|
11727
11761
|
var init_Constants = __esm({
|
|
11728
11762
|
"../../packages/constants/src/Constants.ts"() {
|
|
11729
11763
|
CONSTANTS = {
|
|
11730
|
-
cliVersion: "2.2.
|
|
11764
|
+
cliVersion: "2.2.1",
|
|
11731
11765
|
backendVersion: 1,
|
|
11732
11766
|
backendPort: 5088,
|
|
11733
11767
|
workerVersion: 2,
|
|
@@ -37732,7 +37766,7 @@ var init_index_min = __esm({
|
|
|
37732
37766
|
});
|
|
37733
37767
|
|
|
37734
37768
|
// ../../packages/parsing/src/ParseCompression.ts
|
|
37735
|
-
var fs10, import_zlib2, import_path9, import_yauzl, import_promises6, ParseCompressionClass, ParseCompression, ParseCompression_default;
|
|
37769
|
+
var fs10, import_zlib2, import_path9, import_yauzl, import_promises6, SUPPORTED_FORMATS, FORMAT_SIGNATURES, TRUNCATED_INPUT_CODE, ParseCompressionClass, ParseCompression, ParseCompression_default;
|
|
37736
37770
|
var init_ParseCompression = __esm({
|
|
37737
37771
|
"../../packages/parsing/src/ParseCompression.ts"() {
|
|
37738
37772
|
fs10 = __toESM(require("fs"), 1);
|
|
@@ -37744,6 +37778,12 @@ var init_ParseCompression = __esm({
|
|
|
37744
37778
|
init_src();
|
|
37745
37779
|
init_src2();
|
|
37746
37780
|
init_src3();
|
|
37781
|
+
SUPPORTED_FORMATS = ["ZIP", "TAR", "GZ"];
|
|
37782
|
+
FORMAT_SIGNATURES = {
|
|
37783
|
+
GZ: { bytes: [31, 139], label: "1f 8b" },
|
|
37784
|
+
ZIP: { bytes: [80, 75], label: '50 4b ("PK")' }
|
|
37785
|
+
};
|
|
37786
|
+
TRUNCATED_INPUT_CODE = "Z_BUF_ERROR";
|
|
37747
37787
|
ParseCompressionClass = class {
|
|
37748
37788
|
constructor() {
|
|
37749
37789
|
/**
|
|
@@ -37762,31 +37802,76 @@ var init_ParseCompression = __esm({
|
|
|
37762
37802
|
Affirm_default(standardPath, "Missing required parameter standardPath");
|
|
37763
37803
|
Affirm_default(executionPath, "Missing required parameter executionPath");
|
|
37764
37804
|
const archivePath = import_path9.default.join(standardPath, fileKey);
|
|
37765
|
-
|
|
37766
|
-
|
|
37767
|
-
|
|
37768
|
-
|
|
37769
|
-
|
|
37770
|
-
|
|
37771
|
-
|
|
37772
|
-
|
|
37773
|
-
|
|
37774
|
-
|
|
37775
|
-
|
|
37776
|
-
|
|
37777
|
-
|
|
37778
|
-
|
|
37779
|
-
|
|
37780
|
-
|
|
37781
|
-
|
|
37782
|
-
|
|
37783
|
-
|
|
37784
|
-
|
|
37785
|
-
|
|
37786
|
-
default: {
|
|
37787
|
-
throw new Error(`this compression type it has not been implemented ${compressionType}`);
|
|
37805
|
+
const format4 = compressionType?.toUpperCase();
|
|
37806
|
+
if (!format4) {
|
|
37807
|
+
if ([".zip", ".tar", ".gz"].some((extension) => fileKey.includes(extension)))
|
|
37808
|
+
Logger_default.warn(`The file ${fileKey} looks compressed while its producer declares no compressionType`);
|
|
37809
|
+
return [archivePath];
|
|
37810
|
+
}
|
|
37811
|
+
Affirm_default(SUPPORTED_FORMATS.includes(format4), `Compression type "${compressionType}" has not been implemented. Declare one of ${SUPPORTED_FORMATS.join(", ")} on the producer, or no compressionType at all`);
|
|
37812
|
+
this._warningPossibleConvertionError(compressionType, fileKey);
|
|
37813
|
+
this._affirmArchive(archivePath, format4);
|
|
37814
|
+
try {
|
|
37815
|
+
switch (format4) {
|
|
37816
|
+
case "ZIP":
|
|
37817
|
+
return await this._extractZip(archivePath, this._prepareTargetDir(executionPath));
|
|
37818
|
+
case "TAR":
|
|
37819
|
+
return await this._extractTar(archivePath, this._prepareTargetDir(executionPath));
|
|
37820
|
+
// GZ, the only format the affirm above leaves
|
|
37821
|
+
default: {
|
|
37822
|
+
fs10.mkdirSync(import_path9.default.dirname(executionPath), { recursive: true });
|
|
37823
|
+
await this._gunzipFile(archivePath, executionPath);
|
|
37824
|
+
return [executionPath];
|
|
37825
|
+
}
|
|
37788
37826
|
}
|
|
37827
|
+
} catch (error) {
|
|
37828
|
+
throw this._asNamedFailure(error, format4, archivePath);
|
|
37829
|
+
}
|
|
37830
|
+
};
|
|
37831
|
+
/**
|
|
37832
|
+
* What has to be true of a file before a decompressor is pointed at it.
|
|
37833
|
+
*
|
|
37834
|
+
* Both checks exist because of what zlib says when they fail: a zero-byte file and a half-written
|
|
37835
|
+
* one are each reported as the bare string "unexpected end of file", and a file that is not
|
|
37836
|
+
* compressed at all as "incorrect header check" — three different configuration mistakes, none of
|
|
37837
|
+
* them named, none of them naming the file. A drop folder produces the first two constantly,
|
|
37838
|
+
* since a file being written into one exists at zero bytes before it holds anything.
|
|
37839
|
+
*/
|
|
37840
|
+
this._affirmArchive = (archivePath, format4) => {
|
|
37841
|
+
Affirm_default(fs10.existsSync(archivePath), `Source file does not exist: ${archivePath}`);
|
|
37842
|
+
const { size } = fs10.statSync(archivePath);
|
|
37843
|
+
Affirm_default(size > 0, `Source file "${archivePath}" is empty (0 bytes), so there is no ${format4} archive in it to read. A file still being written into the source folder looks exactly like this \u2014 have the writer stage it under a temporary name and rename it into place, which is atomic, so a run never sees a partial file`);
|
|
37844
|
+
const signature = FORMAT_SIGNATURES[format4];
|
|
37845
|
+
if (!signature) return;
|
|
37846
|
+
Affirm_default(this._startsWith(archivePath, signature.bytes), `Source file "${archivePath}" is declared as ${format4} but does not start with the ${format4} signature ${signature.label}, so it is not a ${format4} archive whatever its name says. Check the producer's compressionType against what actually lands in the folder`);
|
|
37847
|
+
};
|
|
37848
|
+
/** Whether a file's first bytes are exactly `expected`. Reads only those bytes, never the file. */
|
|
37849
|
+
this._startsWith = (filePath, expected) => {
|
|
37850
|
+
const head = Buffer.alloc(expected.length);
|
|
37851
|
+
const handle = fs10.openSync(filePath, "r");
|
|
37852
|
+
try {
|
|
37853
|
+
fs10.readSync(handle, head, 0, head.length, 0);
|
|
37854
|
+
} finally {
|
|
37855
|
+
fs10.closeSync(handle);
|
|
37789
37856
|
}
|
|
37857
|
+
return head.equals(Buffer.from(expected));
|
|
37858
|
+
};
|
|
37859
|
+
/**
|
|
37860
|
+
* A decompression failure that names the file it happened on.
|
|
37861
|
+
*
|
|
37862
|
+
* Every decompression library reports WHAT went wrong and none of them report WHICH file they
|
|
37863
|
+
* were reading — zlib hands back "unexpected end of file" and nothing else. A producer whose
|
|
37864
|
+
* fileKey carries a `%` stages hundreds of files in one pass, so that bare string was the entire
|
|
37865
|
+
* cause of death of a run: no file, no producer, no way to tell which of the hundreds to go and
|
|
37866
|
+
* look at. Our own failures already name the archive and are passed through untouched.
|
|
37867
|
+
*/
|
|
37868
|
+
this._asNamedFailure = (error, format4, archivePath) => {
|
|
37869
|
+
const reason = error instanceof Error ? error : new Error(String(error));
|
|
37870
|
+
if (reason.message.includes(archivePath)) return reason;
|
|
37871
|
+
const truncated = reason.code === TRUNCATED_INPUT_CODE;
|
|
37872
|
+
const hint = truncated ? " The archive ends mid-stream, so it is truncated: the transfer that wrote it did not finish, or it was read while it was still being written" : "";
|
|
37873
|
+
const message = reason.message.replace(/\s*\.\s*$/, "");
|
|
37874
|
+
return new Error(`Failed to decompress "${archivePath}" as ${format4}: ${message}.${hint}`, { cause: reason });
|
|
37790
37875
|
};
|
|
37791
37876
|
this._warningPossibleConvertionError = (compression, fileKey) => {
|
|
37792
37877
|
if (!fileKey.split(".").includes(compression.toLowerCase()) && compression) {
|
package/package.json
CHANGED
|
@@ -11162,8 +11162,7 @@ var init_FileLogService = __esm({
|
|
|
11162
11162
|
return;
|
|
11163
11163
|
if (!(0, import_fs.existsSync)(folder))
|
|
11164
11164
|
(0, import_fs.mkdirSync)(folder, { recursive: true });
|
|
11165
|
-
|
|
11166
|
-
this._logger.end();
|
|
11165
|
+
const previous = this._logger;
|
|
11167
11166
|
this._folder = folder;
|
|
11168
11167
|
this._file = file;
|
|
11169
11168
|
this._logger = import_winston.default.createLogger({
|
|
@@ -11191,6 +11190,7 @@ ${stack}` : base;
|
|
|
11191
11190
|
});
|
|
11192
11191
|
this._logger.on("error", (error) => console.error(`File logger error: ${error?.message ?? String(error)}`));
|
|
11193
11192
|
this._enabled = true;
|
|
11193
|
+
if (previous) void this._endSafely(previous);
|
|
11194
11194
|
};
|
|
11195
11195
|
/**
|
|
11196
11196
|
* Where the file sink writes, whether or not it is currently enabled.
|
|
@@ -11216,19 +11216,31 @@ ${stack}` : base;
|
|
|
11216
11216
|
};
|
|
11217
11217
|
this.flush = () => {
|
|
11218
11218
|
if (!this._enabled || !this._logger) return Promise.resolve();
|
|
11219
|
+
return this.drain(this._logger);
|
|
11220
|
+
};
|
|
11221
|
+
/**
|
|
11222
|
+
* Resolves once every record already handed to `target` has reached its transports.
|
|
11223
|
+
*
|
|
11224
|
+
* A winston logger is a Transform piped into each transport, so a record accepted by `log()` is
|
|
11225
|
+
* not a record that has been written: it sits in the logger's readable buffer until the transport
|
|
11226
|
+
* has room. Writing one sentinel and waiting for every transport to report it `logged` is what
|
|
11227
|
+
* establishes that the queue ahead of it has gone — records travel the pipe in order, so the
|
|
11228
|
+
* sentinel landing means everything before it landed.
|
|
11229
|
+
*/
|
|
11230
|
+
this.drain = (target) => {
|
|
11219
11231
|
return new Promise((resolve) => {
|
|
11220
|
-
let pending =
|
|
11232
|
+
let pending = target.transports.length;
|
|
11221
11233
|
if (pending === 0) return resolve();
|
|
11222
|
-
for (const transport of
|
|
11223
|
-
if (typeof transport.on
|
|
11224
|
-
transport.once("logged", () => {
|
|
11225
|
-
if (--pending === 0) resolve();
|
|
11226
|
-
});
|
|
11227
|
-
this._logger.info("");
|
|
11228
|
-
} else {
|
|
11234
|
+
for (const transport of target.transports) {
|
|
11235
|
+
if (typeof transport.on !== "function") {
|
|
11229
11236
|
if (--pending === 0) resolve();
|
|
11237
|
+
continue;
|
|
11230
11238
|
}
|
|
11239
|
+
transport.once("logged", () => {
|
|
11240
|
+
if (--pending === 0) resolve();
|
|
11241
|
+
});
|
|
11231
11242
|
}
|
|
11243
|
+
target.info("");
|
|
11232
11244
|
});
|
|
11233
11245
|
};
|
|
11234
11246
|
/**
|
|
@@ -11239,15 +11251,37 @@ ${stack}` : base;
|
|
|
11239
11251
|
* listening to. Leaving `_enabled` true after a close is what turned "someone logged after
|
|
11240
11252
|
* close" into "the process died". Use `flush()` when the sink is still needed afterwards.
|
|
11241
11253
|
*/
|
|
11242
|
-
this.close = () => {
|
|
11243
|
-
if (!this._enabled || !this._logger) return
|
|
11254
|
+
this.close = async () => {
|
|
11255
|
+
if (!this._enabled || !this._logger) return;
|
|
11244
11256
|
const closing = this._logger;
|
|
11245
11257
|
this._enabled = false;
|
|
11246
11258
|
this._logger = null;
|
|
11247
|
-
|
|
11248
|
-
|
|
11249
|
-
|
|
11250
|
-
|
|
11259
|
+
await this._endSafely(closing);
|
|
11260
|
+
};
|
|
11261
|
+
/**
|
|
11262
|
+
* Ends a winston logger without the race its own `end()` carries.
|
|
11263
|
+
*
|
|
11264
|
+
* `Logger._final` ends EVERY TRANSPORT the moment the logger's writable side closes — while the
|
|
11265
|
+
* records it already accepted are still travelling down the pipe from its readable side. The
|
|
11266
|
+
* transport drains, the pipe resumes on that drain, and the next record is written into a stream
|
|
11267
|
+
* that has ended: ERR_STREAM_WRITE_AFTER_END, raised from a stream no line of ours is standing
|
|
11268
|
+
* next to. In an executor thread under a debug run, where the pipe is never empty, that is an
|
|
11269
|
+
* error per rotation and a stack with nothing of remora's in it. Draining the in-flight records
|
|
11270
|
+
* into the transports first is what removes the race rather than hiding it.
|
|
11271
|
+
*
|
|
11272
|
+
* A failure here is swallowed on purpose: this logger is on its way out, it carries its own
|
|
11273
|
+
* `error` handler, and a sink that cannot close must not take the caller's shutdown with it.
|
|
11274
|
+
*/
|
|
11275
|
+
this._endSafely = async (target) => {
|
|
11276
|
+
try {
|
|
11277
|
+
await this.drain(target);
|
|
11278
|
+
await new Promise((resolve) => {
|
|
11279
|
+
target.once("finish", resolve);
|
|
11280
|
+
target.end();
|
|
11281
|
+
});
|
|
11282
|
+
} catch (error) {
|
|
11283
|
+
console.error(`File logger failed to close: ${error?.message ?? String(error)}`);
|
|
11284
|
+
}
|
|
11251
11285
|
};
|
|
11252
11286
|
}
|
|
11253
11287
|
};
|
|
@@ -11775,7 +11809,7 @@ var CONSTANTS, Constants_default;
|
|
|
11775
11809
|
var init_Constants = __esm({
|
|
11776
11810
|
"../../packages/constants/src/Constants.ts"() {
|
|
11777
11811
|
CONSTANTS = {
|
|
11778
|
-
cliVersion: "2.2.
|
|
11812
|
+
cliVersion: "2.2.1",
|
|
11779
11813
|
backendVersion: 1,
|
|
11780
11814
|
backendPort: 5088,
|
|
11781
11815
|
workerVersion: 2,
|
|
@@ -37899,7 +37933,7 @@ var init_index_min = __esm({
|
|
|
37899
37933
|
});
|
|
37900
37934
|
|
|
37901
37935
|
// ../../packages/parsing/src/ParseCompression.ts
|
|
37902
|
-
var fs8, import_zlib2, import_path8, import_yauzl, import_promises6, ParseCompressionClass, ParseCompression, ParseCompression_default;
|
|
37936
|
+
var fs8, import_zlib2, import_path8, import_yauzl, import_promises6, SUPPORTED_FORMATS, FORMAT_SIGNATURES, TRUNCATED_INPUT_CODE, ParseCompressionClass, ParseCompression, ParseCompression_default;
|
|
37903
37937
|
var init_ParseCompression = __esm({
|
|
37904
37938
|
"../../packages/parsing/src/ParseCompression.ts"() {
|
|
37905
37939
|
fs8 = __toESM(require("fs"), 1);
|
|
@@ -37911,6 +37945,12 @@ var init_ParseCompression = __esm({
|
|
|
37911
37945
|
init_src();
|
|
37912
37946
|
init_src2();
|
|
37913
37947
|
init_src3();
|
|
37948
|
+
SUPPORTED_FORMATS = ["ZIP", "TAR", "GZ"];
|
|
37949
|
+
FORMAT_SIGNATURES = {
|
|
37950
|
+
GZ: { bytes: [31, 139], label: "1f 8b" },
|
|
37951
|
+
ZIP: { bytes: [80, 75], label: '50 4b ("PK")' }
|
|
37952
|
+
};
|
|
37953
|
+
TRUNCATED_INPUT_CODE = "Z_BUF_ERROR";
|
|
37914
37954
|
ParseCompressionClass = class {
|
|
37915
37955
|
constructor() {
|
|
37916
37956
|
/**
|
|
@@ -37929,31 +37969,76 @@ var init_ParseCompression = __esm({
|
|
|
37929
37969
|
Affirm_default(standardPath, "Missing required parameter standardPath");
|
|
37930
37970
|
Affirm_default(executionPath, "Missing required parameter executionPath");
|
|
37931
37971
|
const archivePath = import_path8.default.join(standardPath, fileKey);
|
|
37932
|
-
|
|
37933
|
-
|
|
37934
|
-
|
|
37935
|
-
|
|
37936
|
-
|
|
37937
|
-
|
|
37938
|
-
|
|
37939
|
-
|
|
37940
|
-
|
|
37941
|
-
|
|
37942
|
-
|
|
37943
|
-
|
|
37944
|
-
|
|
37945
|
-
|
|
37946
|
-
|
|
37947
|
-
|
|
37948
|
-
|
|
37949
|
-
|
|
37950
|
-
|
|
37951
|
-
|
|
37952
|
-
|
|
37953
|
-
default: {
|
|
37954
|
-
throw new Error(`this compression type it has not been implemented ${compressionType}`);
|
|
37972
|
+
const format4 = compressionType?.toUpperCase();
|
|
37973
|
+
if (!format4) {
|
|
37974
|
+
if ([".zip", ".tar", ".gz"].some((extension) => fileKey.includes(extension)))
|
|
37975
|
+
Logger_default.warn(`The file ${fileKey} looks compressed while its producer declares no compressionType`);
|
|
37976
|
+
return [archivePath];
|
|
37977
|
+
}
|
|
37978
|
+
Affirm_default(SUPPORTED_FORMATS.includes(format4), `Compression type "${compressionType}" has not been implemented. Declare one of ${SUPPORTED_FORMATS.join(", ")} on the producer, or no compressionType at all`);
|
|
37979
|
+
this._warningPossibleConvertionError(compressionType, fileKey);
|
|
37980
|
+
this._affirmArchive(archivePath, format4);
|
|
37981
|
+
try {
|
|
37982
|
+
switch (format4) {
|
|
37983
|
+
case "ZIP":
|
|
37984
|
+
return await this._extractZip(archivePath, this._prepareTargetDir(executionPath));
|
|
37985
|
+
case "TAR":
|
|
37986
|
+
return await this._extractTar(archivePath, this._prepareTargetDir(executionPath));
|
|
37987
|
+
// GZ, the only format the affirm above leaves
|
|
37988
|
+
default: {
|
|
37989
|
+
fs8.mkdirSync(import_path8.default.dirname(executionPath), { recursive: true });
|
|
37990
|
+
await this._gunzipFile(archivePath, executionPath);
|
|
37991
|
+
return [executionPath];
|
|
37992
|
+
}
|
|
37955
37993
|
}
|
|
37994
|
+
} catch (error) {
|
|
37995
|
+
throw this._asNamedFailure(error, format4, archivePath);
|
|
37996
|
+
}
|
|
37997
|
+
};
|
|
37998
|
+
/**
|
|
37999
|
+
* What has to be true of a file before a decompressor is pointed at it.
|
|
38000
|
+
*
|
|
38001
|
+
* Both checks exist because of what zlib says when they fail: a zero-byte file and a half-written
|
|
38002
|
+
* one are each reported as the bare string "unexpected end of file", and a file that is not
|
|
38003
|
+
* compressed at all as "incorrect header check" — three different configuration mistakes, none of
|
|
38004
|
+
* them named, none of them naming the file. A drop folder produces the first two constantly,
|
|
38005
|
+
* since a file being written into one exists at zero bytes before it holds anything.
|
|
38006
|
+
*/
|
|
38007
|
+
this._affirmArchive = (archivePath, format4) => {
|
|
38008
|
+
Affirm_default(fs8.existsSync(archivePath), `Source file does not exist: ${archivePath}`);
|
|
38009
|
+
const { size } = fs8.statSync(archivePath);
|
|
38010
|
+
Affirm_default(size > 0, `Source file "${archivePath}" is empty (0 bytes), so there is no ${format4} archive in it to read. A file still being written into the source folder looks exactly like this \u2014 have the writer stage it under a temporary name and rename it into place, which is atomic, so a run never sees a partial file`);
|
|
38011
|
+
const signature = FORMAT_SIGNATURES[format4];
|
|
38012
|
+
if (!signature) return;
|
|
38013
|
+
Affirm_default(this._startsWith(archivePath, signature.bytes), `Source file "${archivePath}" is declared as ${format4} but does not start with the ${format4} signature ${signature.label}, so it is not a ${format4} archive whatever its name says. Check the producer's compressionType against what actually lands in the folder`);
|
|
38014
|
+
};
|
|
38015
|
+
/** Whether a file's first bytes are exactly `expected`. Reads only those bytes, never the file. */
|
|
38016
|
+
this._startsWith = (filePath, expected) => {
|
|
38017
|
+
const head = Buffer.alloc(expected.length);
|
|
38018
|
+
const handle = fs8.openSync(filePath, "r");
|
|
38019
|
+
try {
|
|
38020
|
+
fs8.readSync(handle, head, 0, head.length, 0);
|
|
38021
|
+
} finally {
|
|
38022
|
+
fs8.closeSync(handle);
|
|
37956
38023
|
}
|
|
38024
|
+
return head.equals(Buffer.from(expected));
|
|
38025
|
+
};
|
|
38026
|
+
/**
|
|
38027
|
+
* A decompression failure that names the file it happened on.
|
|
38028
|
+
*
|
|
38029
|
+
* Every decompression library reports WHAT went wrong and none of them report WHICH file they
|
|
38030
|
+
* were reading — zlib hands back "unexpected end of file" and nothing else. A producer whose
|
|
38031
|
+
* fileKey carries a `%` stages hundreds of files in one pass, so that bare string was the entire
|
|
38032
|
+
* cause of death of a run: no file, no producer, no way to tell which of the hundreds to go and
|
|
38033
|
+
* look at. Our own failures already name the archive and are passed through untouched.
|
|
38034
|
+
*/
|
|
38035
|
+
this._asNamedFailure = (error, format4, archivePath) => {
|
|
38036
|
+
const reason = error instanceof Error ? error : new Error(String(error));
|
|
38037
|
+
if (reason.message.includes(archivePath)) return reason;
|
|
38038
|
+
const truncated = reason.code === TRUNCATED_INPUT_CODE;
|
|
38039
|
+
const hint = truncated ? " The archive ends mid-stream, so it is truncated: the transfer that wrote it did not finish, or it was read while it was still being written" : "";
|
|
38040
|
+
const message = reason.message.replace(/\s*\.\s*$/, "");
|
|
38041
|
+
return new Error(`Failed to decompress "${archivePath}" as ${format4}: ${message}.${hint}`, { cause: reason });
|
|
37957
38042
|
};
|
|
37958
38043
|
this._warningPossibleConvertionError = (compression, fileKey) => {
|
|
37959
38044
|
if (!fileKey.split(".").includes(compression.toLowerCase()) && compression) {
|