@bymax-one/nest-storage 1.1.4 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +72 -1
- package/README.md +25 -4
- package/dist/server/index.cjs +143 -55
- package/dist/server/index.d.cts +20 -5
- package/dist/server/index.d.ts +20 -5
- package/dist/server/index.mjs +144 -56
- package/package.json +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -13,6 +13,75 @@ note on each entry is the compatibility contract — read it, not the number.
|
|
|
13
13
|
|
|
14
14
|
## [Unreleased]
|
|
15
15
|
|
|
16
|
+
## [1.1.6] - 2026-11-06
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- **A failed scan cleanup now names the provider's cause and the key as stored, and a failed abort no
|
|
21
|
+
longer says the upload was aborted.** The 1.1.5 failure reports were too thin to act on:
|
|
22
|
+
- `details.cleanupError` of a rejected post-upload scan whose delete failed was the generic
|
|
23
|
+
"Storage Exception", which lost the provider's message. It is now the error code and the
|
|
24
|
+
provider's message, for example `STORAGE_PROVIDER_ERROR: Access Denied`.
|
|
25
|
+
- `details.key` and the `STORAGE_SCAN_CLEANUP_FAILED` log line named the key you passed, not the
|
|
26
|
+
key under the `keyPrefix`. The details now also carry `storedKey`, the key as the provider
|
|
27
|
+
holds it, and the log line names it; an operator audits by listing the bucket.
|
|
28
|
+
- `STORAGE_MULTIPART_ABORTED` carried the message "Multipart upload was aborted" even when
|
|
29
|
+
`details.abort` was `'failed'`. In that case the message is now "Multipart upload failed and
|
|
30
|
+
could not be aborted".
|
|
31
|
+
|
|
32
|
+
**Apply to a derived backend:** code that matches `details.cleanupError` against the old generic
|
|
33
|
+
text, or the multipart message against "Multipart upload was aborted" when the abort failed, no
|
|
34
|
+
longer matches; read `details.cleanup`, `details.abort` and the error code instead. Log queries for
|
|
35
|
+
`STORAGE_SCAN_CLEANUP_FAILED` now see the key with its `keyPrefix`. `details.storedKey` is new.
|
|
36
|
+
`StorageException` gains an optional fourth constructor parameter, `message`, which replaces the
|
|
37
|
+
default message of the code; existing calls are unchanged.
|
|
38
|
+
|
|
39
|
+
## [1.1.5] - 2026-11-01
|
|
40
|
+
|
|
41
|
+
### Fixed
|
|
42
|
+
|
|
43
|
+
- **With a `keyPrefix`, a post-upload scan never removed the object it rejected.** The cleanup passed
|
|
44
|
+
the already-prefixed key to `delete()`, which prefixes it again, so it deleted `prefix/prefix/key`
|
|
45
|
+
(nothing), succeeded, and the rejected object stayed stored: no error beyond the scan verdict and no
|
|
46
|
+
log line. The cleanup now targets the object's real key.
|
|
47
|
+
- **A failed cleanup or abort is no longer reported as a clean outcome.** These paths told the caller
|
|
48
|
+
one thing while the provider held another:
|
|
49
|
+
- A `post-upload` scan that rejected an object deleted it and rejected with the verdict; when that
|
|
50
|
+
delete failed, the caller saw only the verdict (`STORAGE_SCAN_INFECTED`) while the object stayed
|
|
51
|
+
stored, and the failure was a log line. The verdict is still the error, and its `details` now
|
|
52
|
+
carry `cleanup: 'failed'`, the `key` and the delete failure (`cleanupError`); the log line is at
|
|
53
|
+
error level with the code `STORAGE_SCAN_CLEANUP_FAILED`.
|
|
54
|
+
- A multipart upload whose `CompleteMultipartUpload` failed was reported as aborted but was never
|
|
55
|
+
aborted: the upload and its parts stayed open on the provider. The library now sends
|
|
56
|
+
`AbortMultipartUpload` itself.
|
|
57
|
+
- When the abort also failed, the SDK threw the abort failure in place of the failure that caused
|
|
58
|
+
it. The original failure is now the error, and `details.abort` is `'completed'` or `'failed'`
|
|
59
|
+
(with `abortError`). The same holds for the silent-source abort (`STORAGE_TIMEOUT`,
|
|
60
|
+
`details.phase: 'source-idle'`).
|
|
61
|
+
- After `delete()` or `deleteMany()` removed an object, a repeated `upload()` with the same
|
|
62
|
+
`idempotencyKey` was answered from the cache with a success (`fromIdempotencyCache: true`) for an
|
|
63
|
+
object that no longer existed. Both now drop the cache entries of the objects they delete. The
|
|
64
|
+
cache is per instance, so a delete made by another replica or outside the library still does not
|
|
65
|
+
reach it.
|
|
66
|
+
|
|
67
|
+
**Apply to a derived backend:** if you use a `keyPrefix` together with a `post-upload` scanner,
|
|
68
|
+
**audit your bucket** for objects that failed a scan in an earlier version: they were never removed
|
|
69
|
+
and nothing was logged, so your logs will not list them. Re-scan the prefix, or delete what your
|
|
70
|
+
scanner flagged. Error shapes also change, not codes. Code that reads `error.details` of
|
|
71
|
+
`STORAGE_SCAN_INFECTED`, `STORAGE_SCAN_INCONCLUSIVE`, `STORAGE_MULTIPART_ABORTED` or the
|
|
72
|
+
source-idle `STORAGE_TIMEOUT` gains the fields above: treat `details.cleanup === 'failed'` as
|
|
73
|
+
"rejected but still stored" and alert on it. A failed multipart upload now costs one more request,
|
|
74
|
+
the `AbortMultipartUpload` the SDK used to send (the credentials already needed
|
|
75
|
+
`s3:AbortMultipartUpload`). A repeated `upload()` after a `delete()` now stores the object again
|
|
76
|
+
instead of returning the earlier result. `exists()` is unchanged and still returns `false` for any
|
|
77
|
+
failure that is not a 404; the README now says so next to the method.
|
|
78
|
+
|
|
79
|
+
### Changed
|
|
80
|
+
|
|
81
|
+
- **The Stryker gate is 100.** `stryker.config.json` now fails a mutation run below 100 (`thresholds.break`
|
|
82
|
+
and `low` were 95), and the README states the gate. Equivalent mutants stay documented inline; the gate is
|
|
83
|
+
not lowered. Nothing in the published code changes.
|
|
84
|
+
|
|
16
85
|
## [1.1.4] - 2026-11-01
|
|
17
86
|
|
|
18
87
|
### Changed
|
|
@@ -507,4 +576,6 @@ have regressed from. They are kept because the reasoning is worth having.
|
|
|
507
576
|
[1.1.2]: https://github.com/bymaxone/nest-storage/compare/v1.1.1...v1.1.2
|
|
508
577
|
[1.1.3]: https://github.com/bymaxone/nest-storage/compare/v1.1.2...v1.1.3
|
|
509
578
|
[1.1.4]: https://github.com/bymaxone/nest-storage/compare/v1.1.3...v1.1.4
|
|
510
|
-
[
|
|
579
|
+
[1.1.5]: https://github.com/bymaxone/nest-storage/compare/v1.1.4...v1.1.5
|
|
580
|
+
[1.1.6]: https://github.com/bymaxone/nest-storage/compare/v1.1.5...v1.1.6
|
|
581
|
+
[Unreleased]: https://github.com/bymaxone/nest-storage/compare/v1.1.6...HEAD
|
package/README.md
CHANGED
|
@@ -385,6 +385,12 @@ const r2 = await storage.upload(opts) // returns cached result instantly
|
|
|
385
385
|
console.log(r2.fromIdempotencyCache) // true
|
|
386
386
|
```
|
|
387
387
|
|
|
388
|
+
A repeated upload is answered from the cache only while this instance still believes the object
|
|
389
|
+
exists: `delete()` and `deleteMany()` drop the entries of the objects they remove, so uploading
|
|
390
|
+
again after a delete stores the object again. The cache is **per instance**. A delete made by
|
|
391
|
+
another replica, or outside this library, does not reach it, and there a repeated upload can be
|
|
392
|
+
answered with a success for an object that no longer exists.
|
|
393
|
+
|
|
388
394
|
---
|
|
389
395
|
|
|
390
396
|
## ⬇️ Download
|
|
@@ -571,6 +577,16 @@ arrives, and the scanner and the upload then share that buffer. The read is boun
|
|
|
571
577
|
refused with a reason that says to set it or use `post-upload`. Buffer bodies and `post-upload`
|
|
572
578
|
mode are not affected; `post-upload` hands the scanner no body, only the key.
|
|
573
579
|
|
|
580
|
+
**A rejected object that could not be removed.** In `post-upload` mode the object is already
|
|
581
|
+
stored when the scanner rejects it, so the library deletes it and rejects `upload()` with the
|
|
582
|
+
verdict (`STORAGE_SCAN_INFECTED` or `STORAGE_SCAN_INCONCLUSIVE`). When that delete fails, the
|
|
583
|
+
verdict is still the error, and `details` says so: `cleanup: 'failed'`, the `key` you passed, the
|
|
584
|
+
`storedKey` (the key as the provider holds it, with any `keyPrefix`, which is what a bucket listing
|
|
585
|
+
shows) and the delete failure in `cleanupError` (the error code and the provider's message); the
|
|
586
|
+
failure is also logged at error level with the code `STORAGE_SCAN_CLEANUP_FAILED` and the stored
|
|
587
|
+
key. No `details.cleanup` means the object was removed; `cleanup: 'failed'` means it is **still
|
|
588
|
+
stored** and has to be removed by other means.
|
|
589
|
+
|
|
574
590
|
---
|
|
575
591
|
|
|
576
592
|
## ♻️ Lifecycle Operations
|
|
@@ -736,7 +752,7 @@ All errors are thrown as `StorageException extends HttpException`. The response
|
|
|
736
752
|
| `STORAGE_MIME_NOT_ALLOWED` | 415 | `contentType` outside `mimeWhitelist` |
|
|
737
753
|
| `STORAGE_SIZE_EXCEEDED` | 413 | `size > maxSizeBytes` |
|
|
738
754
|
| `STORAGE_VALIDATION_FAILED` | 400 | `IUploadValidator` rejected the file (`details.reason`) |
|
|
739
|
-
| `STORAGE_SCAN_INFECTED` | 422 | Scanner returned `'infected'` (`details.threat`)
|
|
755
|
+
| `STORAGE_SCAN_INFECTED` | 422 | Scanner returned `'infected'` (`details.threat`); `details.cleanup: 'failed'` means the object is still stored |
|
|
740
756
|
| `STORAGE_SCAN_INCONCLUSIVE` | 422 | Scanner returned `'unknown'` and `rejectOnUnknown: true` |
|
|
741
757
|
| `STORAGE_OBJECT_NOT_FOUND` | 404 | `head()`, `download()`, or `copy()` on a nonexistent key |
|
|
742
758
|
| `STORAGE_PROVIDER_ERROR` | 502 | AWS SDK error (network, 5xx, throttling, `AccessControlListNotSupported`); `details` carries `awsCode`, `awsMessage`, `httpStatus`, `requestId` plus the operation context (`op`, `bucket`, and `key` or `prefix`) |
|
|
@@ -744,7 +760,7 @@ All errors are thrown as `StorageException extends HttpException`. The response
|
|
|
744
760
|
| `STORAGE_PART_TOO_SMALL` | 400 | Reserved: a `multipart.partSizeBytes` below 5 MiB is rejected at init as `STORAGE_INVALID_CONFIG` |
|
|
745
761
|
| `STORAGE_INVALID_PART_COUNT` | 400 | `getMultipartUploadUrls()` with `parts` not an integer from 1 to 10 000 (`details.provided`, `details.reason`) |
|
|
746
762
|
| `STORAGE_BUCKET_UNDEFINED` | 400 | Operation without a bucket and no default in config |
|
|
747
|
-
| `STORAGE_MULTIPART_ABORTED` | 500 | Multipart upload failed and
|
|
763
|
+
| `STORAGE_MULTIPART_ABORTED` | 500 | A multipart upload failed. `details.abort` is `'completed'` when the library aborted it and `'failed'` (with `abortError`) when the abort itself failed (the message then reads "Multipart upload failed and could not be aborted"): that upload stays open on the provider until a lifecycle rule removes it |
|
|
748
764
|
| `STORAGE_INVALID_CONFIG` | 500 | `BymaxStorageModuleOptions` validation failed at initialization |
|
|
749
765
|
| `STORAGE_TIMEOUT` | 504 | An attempt got no response headers within `requestTimeoutMs` (the SDK's `TimeoutError`); a multipart source that stays silent for `uploadIdleTimeoutMs` also surfaces as `STORAGE_TIMEOUT` (`details.phase: 'source-idle'`); any other failure inside a multipart upload surfaces as `STORAGE_MULTIPART_ABORTED` |
|
|
750
766
|
| `STORAGE_LIFECYCLE_RULE_INVALID` | 400 | `BucketLifecycleService` input rejected: empty or over-long `id`, non-positive or non-integer `expirationDays`, bad `prefix`, or an id naming a rule outside `keyPrefix` (`details.reason`) |
|
|
@@ -792,6 +808,11 @@ This is the index.
|
|
|
792
808
|
| `copy` | `(options: CopyOptions) => Promise<{ etag }>` | Server-side; the bytes never reach this process |
|
|
793
809
|
| `checkBucket` | `(options?) => Promise<void>` | One `HeadBucket` for a readiness check; resolves when the bucket is reachable, otherwise rejects `STORAGE_BUCKET_NOT_FOUND`, `STORAGE_BUCKET_ACCESS_DENIED`, `STORAGE_BUCKET_REGION_MISMATCH` or `STORAGE_TIMEOUT`; needs `s3:ListBucket`; never called at boot |
|
|
794
810
|
|
|
811
|
+
> **`exists()` returns `false` for any failure that is not a 404** (a 403, a 5xx, a timeout), after
|
|
812
|
+
> logging a warning, so a transient provider error reads as "absent". Do not use it to decide whether
|
|
813
|
+
> to overwrite or to create (`if (!(await exists(k))) await upload(k)` can overwrite an object
|
|
814
|
+
> that is there); use `head()` and branch on the error code.
|
|
815
|
+
|
|
795
816
|
**A missing bucket and `head()` / `exists()`.** A `HEAD` response has no body, so the provider cannot say which of the bucket and the key is missing: AWS documents that "if the `HEAD` request generates an error, it returns a generic code, such as `400 Bad Request`, `403 Forbidden`, `404 Not Found` ... It's not possible to retrieve the exact exception of these error codes" ([HeadObject][aws-head-object]). `head()` therefore rejects `STORAGE_OBJECT_NOT_FOUND` and `exists()` returns `false` for a missing bucket too. Call `checkBucket()` to tell the two apart, for example from a readiness endpoint.
|
|
796
817
|
|
|
797
818
|
### `SignedUrlService`
|
|
@@ -1030,7 +1051,7 @@ place stays there — so the suite is held to a bar beyond "the tests pass".
|
|
|
1030
1051
|
|
|
1031
1052
|
- ✅ **100% line coverage** — statements, branches, functions and lines, enforced as a gate
|
|
1032
1053
|
- ✅ **100% mutation score** — verified with [Stryker](https://stryker-mutator.io/) at
|
|
1033
|
-
`break:
|
|
1054
|
+
`break: 100`; every survivor was killed by a strengthened assertion, and the provable
|
|
1034
1055
|
equivalents are documented inline
|
|
1035
1056
|
- ✅ **Real object storage in e2e** — MinIO through Testcontainers, so multipart, presigning
|
|
1036
1057
|
and listing are exercised against an actual S3 API rather than a mock
|
|
@@ -1047,7 +1068,7 @@ place stays there — so the suite is held to a bar beyond "the tests pass".
|
|
|
1047
1068
|
pnpm test # unit tests (Jest)
|
|
1048
1069
|
pnpm test:cov # unit tests with the 100% coverage gate
|
|
1049
1070
|
pnpm test:e2e # end-to-end against MinIO (requires Docker)
|
|
1050
|
-
pnpm mutation # Stryker mutation testing (break:
|
|
1071
|
+
pnpm mutation # Stryker mutation testing (break: 100)
|
|
1051
1072
|
pnpm typecheck # tsc strict check
|
|
1052
1073
|
pnpm lint # ESLint
|
|
1053
1074
|
```
|
package/dist/server/index.cjs
CHANGED
|
@@ -117,12 +117,12 @@ var STORAGE_ERROR_STATUS = {
|
|
|
117
117
|
// src/server/errors/storage-exception.ts
|
|
118
118
|
var StorageException = class extends common.HttpException {
|
|
119
119
|
code;
|
|
120
|
-
constructor(code, statusCode = STORAGE_ERROR_STATUS[code], details) {
|
|
120
|
+
constructor(code, statusCode = STORAGE_ERROR_STATUS[code], details, message = STORAGE_ERROR_MESSAGES[code]) {
|
|
121
121
|
super(
|
|
122
122
|
{
|
|
123
123
|
error: {
|
|
124
124
|
code,
|
|
125
|
-
message
|
|
125
|
+
message,
|
|
126
126
|
...details ? { details } : {}
|
|
127
127
|
}
|
|
128
128
|
},
|
|
@@ -1091,6 +1091,7 @@ function guardIdleBody(body, idleMs) {
|
|
|
1091
1091
|
}
|
|
1092
1092
|
|
|
1093
1093
|
// src/server/errors/multipart-error-mapper.ts
|
|
1094
|
+
var ABORT_FAILED_MESSAGE = "Multipart upload failed and could not be aborted";
|
|
1094
1095
|
function mapMultipartError(err, context) {
|
|
1095
1096
|
if (err instanceof SourceIdleError) {
|
|
1096
1097
|
return new StorageException(STORAGE_ERROR_CODES.STORAGE_TIMEOUT, common.HttpStatus.GATEWAY_TIMEOUT, {
|
|
@@ -1103,11 +1104,87 @@ function mapMultipartError(err, context) {
|
|
|
1103
1104
|
if (bucketMissing.code === STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND) {
|
|
1104
1105
|
return bucketMissing;
|
|
1105
1106
|
}
|
|
1106
|
-
return new StorageException(
|
|
1107
|
-
|
|
1108
|
-
|
|
1107
|
+
return new StorageException(
|
|
1108
|
+
STORAGE_ERROR_CODES.STORAGE_MULTIPART_ABORTED,
|
|
1109
|
+
void 0,
|
|
1110
|
+
{ ...context, awsMessage: describeError(err) },
|
|
1111
|
+
// The code's message says the upload was aborted; it is not, when the abort itself failed.
|
|
1112
|
+
context.abort === "failed" ? ABORT_FAILED_MESSAGE : void 0
|
|
1113
|
+
);
|
|
1114
|
+
}
|
|
1115
|
+
function knownStatus(error) {
|
|
1116
|
+
const name = common.HttpStatus[error.getStatus()];
|
|
1117
|
+
return name === void 0 ? void 0 : common.HttpStatus[name];
|
|
1118
|
+
}
|
|
1119
|
+
function describeCleanupError(error) {
|
|
1120
|
+
if (!(error instanceof StorageException)) {
|
|
1121
|
+
return describeError(error);
|
|
1122
|
+
}
|
|
1123
|
+
const body = error.getResponse();
|
|
1124
|
+
const awsMessage = body.error?.details?.awsMessage;
|
|
1125
|
+
return typeof awsMessage === "string" ? `${error.code}: ${awsMessage}` : error.code;
|
|
1126
|
+
}
|
|
1127
|
+
function withCleanupFailure(scanError, keys, cleanupError) {
|
|
1128
|
+
if (!(scanError instanceof StorageException)) {
|
|
1129
|
+
return scanError;
|
|
1130
|
+
}
|
|
1131
|
+
const body = scanError.getResponse();
|
|
1132
|
+
return new StorageException(scanError.code, knownStatus(scanError), {
|
|
1133
|
+
...body.error?.details ?? {},
|
|
1134
|
+
key: keys.key,
|
|
1135
|
+
storedKey: keys.storedKey,
|
|
1136
|
+
cleanup: "failed",
|
|
1137
|
+
cleanupError: describeCleanupError(cleanupError)
|
|
1109
1138
|
});
|
|
1110
1139
|
}
|
|
1140
|
+
async function closeMultipartUpload(client, bucket, key, uploadId) {
|
|
1141
|
+
if (uploadId === void 0) {
|
|
1142
|
+
return {};
|
|
1143
|
+
}
|
|
1144
|
+
try {
|
|
1145
|
+
await client.send(
|
|
1146
|
+
new clientS3.AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })
|
|
1147
|
+
);
|
|
1148
|
+
return { abort: "completed" };
|
|
1149
|
+
} catch (err) {
|
|
1150
|
+
if (err?.name === "NoSuchUpload") {
|
|
1151
|
+
return { abort: "completed" };
|
|
1152
|
+
}
|
|
1153
|
+
return { abort: "failed", abortError: describeError(err) };
|
|
1154
|
+
}
|
|
1155
|
+
}
|
|
1156
|
+
async function deleteObjectsChunk(client, bucket, chunk, stripPrefix) {
|
|
1157
|
+
const deleted = [];
|
|
1158
|
+
const failed = [];
|
|
1159
|
+
try {
|
|
1160
|
+
const response = await client.send(
|
|
1161
|
+
new clientS3.DeleteObjectsCommand({
|
|
1162
|
+
Bucket: bucket,
|
|
1163
|
+
// `Quiet: false` returns both successes and per-key errors.
|
|
1164
|
+
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1165
|
+
})
|
|
1166
|
+
);
|
|
1167
|
+
for (const ok of response.Deleted ?? []) {
|
|
1168
|
+
if (ok.Key !== void 0) {
|
|
1169
|
+
deleted.push(stripPrefix(ok.Key));
|
|
1170
|
+
}
|
|
1171
|
+
}
|
|
1172
|
+
for (const failure of response.Errors ?? []) {
|
|
1173
|
+
if (failure.Key !== void 0) {
|
|
1174
|
+
failed.push({
|
|
1175
|
+
key: stripPrefix(failure.Key),
|
|
1176
|
+
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1177
|
+
});
|
|
1178
|
+
}
|
|
1179
|
+
}
|
|
1180
|
+
} catch (err) {
|
|
1181
|
+
const message = describeError(err);
|
|
1182
|
+
for (const key of chunk) {
|
|
1183
|
+
failed.push({ key: stripPrefix(key), error: message });
|
|
1184
|
+
}
|
|
1185
|
+
}
|
|
1186
|
+
return { deleted, failed };
|
|
1187
|
+
}
|
|
1111
1188
|
function mapBucketCheckError(err, context) {
|
|
1112
1189
|
const failure = err ?? {};
|
|
1113
1190
|
const status = failure.$metadata?.httpStatusCode;
|
|
@@ -1349,12 +1426,16 @@ exports.StorageService = class StorageService {
|
|
|
1349
1426
|
...options.size !== void 0 ? { size: options.size } : {}
|
|
1350
1427
|
});
|
|
1351
1428
|
} catch (scanErr) {
|
|
1352
|
-
|
|
1429
|
+
const key = this.keyResolver.stripPrefix(finalKey);
|
|
1430
|
+
try {
|
|
1431
|
+
await this.delete(key, { bucket });
|
|
1432
|
+
} catch (deleteErr) {
|
|
1353
1433
|
this.logger.error(
|
|
1354
|
-
`
|
|
1434
|
+
`STORAGE_SCAN_CLEANUP_FAILED: the rejected object is still stored: ${finalKey}`,
|
|
1355
1435
|
describeError(deleteErr)
|
|
1356
1436
|
);
|
|
1357
|
-
|
|
1437
|
+
throw withCleanupFailure(scanErr, { key, storedKey: finalKey }, deleteErr);
|
|
1438
|
+
}
|
|
1358
1439
|
throw scanErr;
|
|
1359
1440
|
}
|
|
1360
1441
|
}
|
|
@@ -1475,12 +1556,12 @@ exports.StorageService = class StorageService {
|
|
|
1475
1556
|
await this.s3Provider.getClient().send(new clientS3.DeleteObjectCommand({ Bucket: bucket, Key: finalKey }));
|
|
1476
1557
|
} catch (err) {
|
|
1477
1558
|
const mapped = mapAwsError(err, { key: finalKey, bucket, op: "delete" });
|
|
1478
|
-
if (mapped.code
|
|
1479
|
-
|
|
1480
|
-
return;
|
|
1559
|
+
if (mapped.code !== STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND) {
|
|
1560
|
+
throw mapped;
|
|
1481
1561
|
}
|
|
1482
|
-
|
|
1562
|
+
this.logger.warn(`delete() \u2014 key not found (idempotent no-op): ${finalKey}`);
|
|
1483
1563
|
}
|
|
1564
|
+
this.idempotencyCache.invalidate(bucket, this.keyResolver.stripPrefix(finalKey));
|
|
1484
1565
|
}
|
|
1485
1566
|
/**
|
|
1486
1567
|
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
@@ -1590,8 +1671,10 @@ exports.StorageService = class StorageService {
|
|
|
1590
1671
|
}
|
|
1591
1672
|
}
|
|
1592
1673
|
/**
|
|
1593
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
1594
|
-
*
|
|
1674
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
1675
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
1676
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
1677
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
1595
1678
|
*/
|
|
1596
1679
|
async uploadMultipart(options, finalKey, bucket) {
|
|
1597
1680
|
const params = {
|
|
@@ -1613,14 +1696,9 @@ exports.StorageService = class StorageService {
|
|
|
1613
1696
|
params,
|
|
1614
1697
|
queueSize: this.options.multipart.queueSize,
|
|
1615
1698
|
partSize: this.options.multipart.partSizeBytes,
|
|
1616
|
-
leavePartsOnError:
|
|
1699
|
+
leavePartsOnError: true
|
|
1617
1700
|
});
|
|
1618
|
-
|
|
1619
|
-
const onProgress = options.onProgress;
|
|
1620
|
-
uploader.on("httpUploadProgress", (event) => {
|
|
1621
|
-
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1622
|
-
});
|
|
1623
|
-
}
|
|
1701
|
+
this.forwardProgress(uploader, options.onProgress);
|
|
1624
1702
|
try {
|
|
1625
1703
|
const response = await uploader.done();
|
|
1626
1704
|
return this.buildUploadResult({
|
|
@@ -1633,9 +1711,26 @@ exports.StorageService = class StorageService {
|
|
|
1633
1711
|
multipart: true
|
|
1634
1712
|
});
|
|
1635
1713
|
} catch (err) {
|
|
1636
|
-
|
|
1714
|
+
return this.failMultipart(err, finalKey, bucket, uploader.uploadId);
|
|
1637
1715
|
}
|
|
1638
1716
|
}
|
|
1717
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
1718
|
+
forwardProgress(uploader, onProgress) {
|
|
1719
|
+
if (!onProgress) return;
|
|
1720
|
+
uploader.on("httpUploadProgress", (event) => {
|
|
1721
|
+
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1722
|
+
});
|
|
1723
|
+
}
|
|
1724
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
1725
|
+
async failMultipart(err, finalKey, bucket, uploadId) {
|
|
1726
|
+
const abort = await closeMultipartUpload(
|
|
1727
|
+
this.s3Provider.getClient(),
|
|
1728
|
+
bucket,
|
|
1729
|
+
finalKey,
|
|
1730
|
+
uploadId
|
|
1731
|
+
);
|
|
1732
|
+
throw mapMultipartError(err, { key: finalKey, bucket, ...abort });
|
|
1733
|
+
}
|
|
1639
1734
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
1640
1735
|
normalizeSingleShotBody(body) {
|
|
1641
1736
|
if (isBufferLike(body)) {
|
|
@@ -1748,42 +1843,17 @@ exports.StorageService = class StorageService {
|
|
|
1748
1843
|
const failed = [];
|
|
1749
1844
|
for (let i = 0; i < normalized.length; i += DELETE_BATCH_LIMIT) {
|
|
1750
1845
|
const chunk = normalized.slice(i, i + DELETE_BATCH_LIMIT);
|
|
1751
|
-
const outcome = await
|
|
1846
|
+
const outcome = await deleteObjectsChunk(
|
|
1847
|
+
this.s3Provider.getClient(),
|
|
1848
|
+
bucket,
|
|
1849
|
+
chunk,
|
|
1850
|
+
(key) => this.keyResolver.stripPrefix(key)
|
|
1851
|
+
);
|
|
1752
1852
|
deleted.push(...outcome.deleted);
|
|
1753
1853
|
failed.push(...outcome.failed);
|
|
1754
1854
|
}
|
|
1755
|
-
|
|
1756
|
-
|
|
1757
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
1758
|
-
async deleteChunk(bucket, chunk) {
|
|
1759
|
-
const deleted = [];
|
|
1760
|
-
const failed = [];
|
|
1761
|
-
try {
|
|
1762
|
-
const response = await this.s3Provider.getClient().send(
|
|
1763
|
-
new clientS3.DeleteObjectsCommand({
|
|
1764
|
-
Bucket: bucket,
|
|
1765
|
-
// `Quiet: false` returns both successes and per-key errors.
|
|
1766
|
-
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1767
|
-
})
|
|
1768
|
-
);
|
|
1769
|
-
for (const ok of response.Deleted ?? []) {
|
|
1770
|
-
if (ok.Key !== void 0) {
|
|
1771
|
-
deleted.push(this.keyResolver.stripPrefix(ok.Key));
|
|
1772
|
-
}
|
|
1773
|
-
}
|
|
1774
|
-
for (const failure of response.Errors ?? []) {
|
|
1775
|
-
if (failure.Key !== void 0) {
|
|
1776
|
-
failed.push({
|
|
1777
|
-
key: this.keyResolver.stripPrefix(failure.Key),
|
|
1778
|
-
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1779
|
-
});
|
|
1780
|
-
}
|
|
1781
|
-
}
|
|
1782
|
-
} catch (err) {
|
|
1783
|
-
const message = describeError(err);
|
|
1784
|
-
for (const key of chunk) {
|
|
1785
|
-
failed.push({ key: this.keyResolver.stripPrefix(key), error: message });
|
|
1786
|
-
}
|
|
1855
|
+
for (const key of deleted) {
|
|
1856
|
+
this.idempotencyCache.invalidate(bucket, key);
|
|
1787
1857
|
}
|
|
1788
1858
|
return { deleted, failed };
|
|
1789
1859
|
}
|
|
@@ -2084,6 +2154,24 @@ var IdempotencyCache = class {
|
|
|
2084
2154
|
}
|
|
2085
2155
|
}
|
|
2086
2156
|
}
|
|
2157
|
+
/**
|
|
2158
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
2159
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
2160
|
+
*
|
|
2161
|
+
* @param bucket - The bucket of the deleted object.
|
|
2162
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
2163
|
+
* @returns How many entries were dropped.
|
|
2164
|
+
*/
|
|
2165
|
+
invalidate(bucket, key) {
|
|
2166
|
+
let dropped = 0;
|
|
2167
|
+
for (const [cacheKey, entry] of this.entries) {
|
|
2168
|
+
if (entry.value.bucket === bucket && entry.value.key === key) {
|
|
2169
|
+
this.entries.delete(cacheKey);
|
|
2170
|
+
dropped += 1;
|
|
2171
|
+
}
|
|
2172
|
+
}
|
|
2173
|
+
return dropped;
|
|
2174
|
+
}
|
|
2087
2175
|
/**
|
|
2088
2176
|
* Reports the current number of cached entries.
|
|
2089
2177
|
*
|
package/dist/server/index.d.cts
CHANGED
|
@@ -691,6 +691,15 @@ declare class IdempotencyCache {
|
|
|
691
691
|
* @param value - The upload result to cache.
|
|
692
692
|
*/
|
|
693
693
|
set(cacheKey: string, value: UploadResult): void;
|
|
694
|
+
/**
|
|
695
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
696
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
697
|
+
*
|
|
698
|
+
* @param bucket - The bucket of the deleted object.
|
|
699
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
700
|
+
* @returns How many entries were dropped.
|
|
701
|
+
*/
|
|
702
|
+
invalidate(bucket: string, key: string): number;
|
|
694
703
|
/**
|
|
695
704
|
* Reports the current number of cached entries.
|
|
696
705
|
*
|
|
@@ -842,10 +851,16 @@ declare class StorageService {
|
|
|
842
851
|
/** Single `PutObject` upload for bodies below the multipart threshold. */
|
|
843
852
|
private uploadSingleShot;
|
|
844
853
|
/**
|
|
845
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
846
|
-
*
|
|
854
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
855
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
856
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
857
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
847
858
|
*/
|
|
848
859
|
private uploadMultipart;
|
|
860
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
861
|
+
private forwardProgress;
|
|
862
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
863
|
+
private failMultipart;
|
|
849
864
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
850
865
|
private normalizeSingleShotBody;
|
|
851
866
|
/** Assembles an `UploadResult`, omitting optional fields that are absent. */
|
|
@@ -884,8 +899,6 @@ declare class StorageService {
|
|
|
884
899
|
* @returns The deleted keys and the per-key failures.
|
|
885
900
|
*/
|
|
886
901
|
deleteMany(keys: string[], options?: DeleteManyOptions): Promise<DeleteManyResult>;
|
|
887
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
888
|
-
private deleteChunk;
|
|
889
902
|
}
|
|
890
903
|
|
|
891
904
|
/**
|
|
@@ -1382,7 +1395,9 @@ declare class StorageException extends HttpException {
|
|
|
1382
1395
|
readonly code: StorageErrorCode;
|
|
1383
1396
|
constructor(code: StorageErrorCode,
|
|
1384
1397
|
/** Defaults to `STORAGE_ERROR_STATUS[code]`. Pass only to override. */
|
|
1385
|
-
statusCode?: HttpStatus, details?: Record<string, unknown
|
|
1398
|
+
statusCode?: HttpStatus, details?: Record<string, unknown>,
|
|
1399
|
+
/** Defaults to the message of `code`. Pass only when the situation is more specific than the code. */
|
|
1400
|
+
message?: string);
|
|
1386
1401
|
}
|
|
1387
1402
|
|
|
1388
1403
|
export { type AbortMultipartRuleInput, BYMAX_STORAGE_FILE_SCANNER, BYMAX_STORAGE_OPTIONS, BYMAX_STORAGE_S3_CLIENT, BYMAX_STORAGE_UPLOAD_VALIDATORS, type BucketLifecycleRule, type BucketLifecycleRuleView, BucketLifecycleService, BymaxStorageModule, type BymaxStorageModuleAsyncOptions, type BymaxStorageModuleOptions, type BymaxStorageModuleOptionsFactory, type CopyOptions, type DeleteManyOptions, type DeleteManyResult, type DownloadOptions, type ExpirationRuleInput, type FailedDeletion, type FileScanResult, type IFileScanner, type IUploadValidator, type LifecycleBucketOptions, type LifecycleRuleInput, type ListOptions, type ListResult, type ListedObject, type MultipartUploadUrlsOptions, type MultipartUploadUrlsResult, NoOpFileScanner, NoOpUploadValidator, type ObjectMetadata, type ProviderRecipe, STORAGE_ERROR_CODES, type SignedGetUrlOptions, type SignedPutUrlOptions, type SignedUrlResult, SignedUrlService, type StorageErrorCode, StorageException, StorageService, type UploadOptions, type UploadResult, providerRecipes };
|
package/dist/server/index.d.ts
CHANGED
|
@@ -691,6 +691,15 @@ declare class IdempotencyCache {
|
|
|
691
691
|
* @param value - The upload result to cache.
|
|
692
692
|
*/
|
|
693
693
|
set(cacheKey: string, value: UploadResult): void;
|
|
694
|
+
/**
|
|
695
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
696
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
697
|
+
*
|
|
698
|
+
* @param bucket - The bucket of the deleted object.
|
|
699
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
700
|
+
* @returns How many entries were dropped.
|
|
701
|
+
*/
|
|
702
|
+
invalidate(bucket: string, key: string): number;
|
|
694
703
|
/**
|
|
695
704
|
* Reports the current number of cached entries.
|
|
696
705
|
*
|
|
@@ -842,10 +851,16 @@ declare class StorageService {
|
|
|
842
851
|
/** Single `PutObject` upload for bodies below the multipart threshold. */
|
|
843
852
|
private uploadSingleShot;
|
|
844
853
|
/**
|
|
845
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
846
|
-
*
|
|
854
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
855
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
856
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
857
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
847
858
|
*/
|
|
848
859
|
private uploadMultipart;
|
|
860
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
861
|
+
private forwardProgress;
|
|
862
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
863
|
+
private failMultipart;
|
|
849
864
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
850
865
|
private normalizeSingleShotBody;
|
|
851
866
|
/** Assembles an `UploadResult`, omitting optional fields that are absent. */
|
|
@@ -884,8 +899,6 @@ declare class StorageService {
|
|
|
884
899
|
* @returns The deleted keys and the per-key failures.
|
|
885
900
|
*/
|
|
886
901
|
deleteMany(keys: string[], options?: DeleteManyOptions): Promise<DeleteManyResult>;
|
|
887
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
888
|
-
private deleteChunk;
|
|
889
902
|
}
|
|
890
903
|
|
|
891
904
|
/**
|
|
@@ -1382,7 +1395,9 @@ declare class StorageException extends HttpException {
|
|
|
1382
1395
|
readonly code: StorageErrorCode;
|
|
1383
1396
|
constructor(code: StorageErrorCode,
|
|
1384
1397
|
/** Defaults to `STORAGE_ERROR_STATUS[code]`. Pass only to override. */
|
|
1385
|
-
statusCode?: HttpStatus, details?: Record<string, unknown
|
|
1398
|
+
statusCode?: HttpStatus, details?: Record<string, unknown>,
|
|
1399
|
+
/** Defaults to the message of `code`. Pass only when the situation is more specific than the code. */
|
|
1400
|
+
message?: string);
|
|
1386
1401
|
}
|
|
1387
1402
|
|
|
1388
1403
|
export { type AbortMultipartRuleInput, BYMAX_STORAGE_FILE_SCANNER, BYMAX_STORAGE_OPTIONS, BYMAX_STORAGE_S3_CLIENT, BYMAX_STORAGE_UPLOAD_VALIDATORS, type BucketLifecycleRule, type BucketLifecycleRuleView, BucketLifecycleService, BymaxStorageModule, type BymaxStorageModuleAsyncOptions, type BymaxStorageModuleOptions, type BymaxStorageModuleOptionsFactory, type CopyOptions, type DeleteManyOptions, type DeleteManyResult, type DownloadOptions, type ExpirationRuleInput, type FailedDeletion, type FileScanResult, type IFileScanner, type IUploadValidator, type LifecycleBucketOptions, type LifecycleRuleInput, type ListOptions, type ListResult, type ListedObject, type MultipartUploadUrlsOptions, type MultipartUploadUrlsResult, NoOpFileScanner, NoOpUploadValidator, type ObjectMetadata, type ProviderRecipe, STORAGE_ERROR_CODES, type SignedGetUrlOptions, type SignedPutUrlOptions, type SignedUrlResult, SignedUrlService, type StorageErrorCode, StorageException, StorageService, type UploadOptions, type UploadResult, providerRecipes };
|
package/dist/server/index.mjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { HttpStatus, Injectable, Inject, Global, Module, Logger, HttpException } from '@nestjs/common';
|
|
2
|
-
import { S3Client, GetBucketLifecycleConfigurationCommand, DeleteBucketLifecycleCommand, PutBucketLifecycleConfigurationCommand, HeadObjectCommand, GetObjectCommand, DeleteObjectCommand, HeadBucketCommand, PutObjectCommand, ListObjectsV2Command, CopyObjectCommand,
|
|
2
|
+
import { S3Client, GetBucketLifecycleConfigurationCommand, DeleteBucketLifecycleCommand, PutBucketLifecycleConfigurationCommand, HeadObjectCommand, GetObjectCommand, DeleteObjectCommand, HeadBucketCommand, PutObjectCommand, ListObjectsV2Command, CopyObjectCommand, CreateMultipartUploadCommand, UploadPartCommand, CompleteMultipartUploadCommand, AbortMultipartUploadCommand, DeleteObjectsCommand } from '@aws-sdk/client-s3';
|
|
3
3
|
import { Upload } from '@aws-sdk/lib-storage';
|
|
4
4
|
import { PassThrough, Readable } from 'stream';
|
|
5
5
|
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
|
|
@@ -115,12 +115,12 @@ var STORAGE_ERROR_STATUS = {
|
|
|
115
115
|
// src/server/errors/storage-exception.ts
|
|
116
116
|
var StorageException = class extends HttpException {
|
|
117
117
|
code;
|
|
118
|
-
constructor(code, statusCode = STORAGE_ERROR_STATUS[code], details) {
|
|
118
|
+
constructor(code, statusCode = STORAGE_ERROR_STATUS[code], details, message = STORAGE_ERROR_MESSAGES[code]) {
|
|
119
119
|
super(
|
|
120
120
|
{
|
|
121
121
|
error: {
|
|
122
122
|
code,
|
|
123
|
-
message
|
|
123
|
+
message,
|
|
124
124
|
...details ? { details } : {}
|
|
125
125
|
}
|
|
126
126
|
},
|
|
@@ -1089,6 +1089,7 @@ function guardIdleBody(body, idleMs) {
|
|
|
1089
1089
|
}
|
|
1090
1090
|
|
|
1091
1091
|
// src/server/errors/multipart-error-mapper.ts
|
|
1092
|
+
var ABORT_FAILED_MESSAGE = "Multipart upload failed and could not be aborted";
|
|
1092
1093
|
function mapMultipartError(err, context) {
|
|
1093
1094
|
if (err instanceof SourceIdleError) {
|
|
1094
1095
|
return new StorageException(STORAGE_ERROR_CODES.STORAGE_TIMEOUT, HttpStatus.GATEWAY_TIMEOUT, {
|
|
@@ -1101,11 +1102,87 @@ function mapMultipartError(err, context) {
|
|
|
1101
1102
|
if (bucketMissing.code === STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND) {
|
|
1102
1103
|
return bucketMissing;
|
|
1103
1104
|
}
|
|
1104
|
-
return new StorageException(
|
|
1105
|
-
|
|
1106
|
-
|
|
1105
|
+
return new StorageException(
|
|
1106
|
+
STORAGE_ERROR_CODES.STORAGE_MULTIPART_ABORTED,
|
|
1107
|
+
void 0,
|
|
1108
|
+
{ ...context, awsMessage: describeError(err) },
|
|
1109
|
+
// The code's message says the upload was aborted; it is not, when the abort itself failed.
|
|
1110
|
+
context.abort === "failed" ? ABORT_FAILED_MESSAGE : void 0
|
|
1111
|
+
);
|
|
1112
|
+
}
|
|
1113
|
+
function knownStatus(error) {
|
|
1114
|
+
const name = HttpStatus[error.getStatus()];
|
|
1115
|
+
return name === void 0 ? void 0 : HttpStatus[name];
|
|
1116
|
+
}
|
|
1117
|
+
function describeCleanupError(error) {
|
|
1118
|
+
if (!(error instanceof StorageException)) {
|
|
1119
|
+
return describeError(error);
|
|
1120
|
+
}
|
|
1121
|
+
const body = error.getResponse();
|
|
1122
|
+
const awsMessage = body.error?.details?.awsMessage;
|
|
1123
|
+
return typeof awsMessage === "string" ? `${error.code}: ${awsMessage}` : error.code;
|
|
1124
|
+
}
|
|
1125
|
+
function withCleanupFailure(scanError, keys, cleanupError) {
|
|
1126
|
+
if (!(scanError instanceof StorageException)) {
|
|
1127
|
+
return scanError;
|
|
1128
|
+
}
|
|
1129
|
+
const body = scanError.getResponse();
|
|
1130
|
+
return new StorageException(scanError.code, knownStatus(scanError), {
|
|
1131
|
+
...body.error?.details ?? {},
|
|
1132
|
+
key: keys.key,
|
|
1133
|
+
storedKey: keys.storedKey,
|
|
1134
|
+
cleanup: "failed",
|
|
1135
|
+
cleanupError: describeCleanupError(cleanupError)
|
|
1107
1136
|
});
|
|
1108
1137
|
}
|
|
1138
|
+
async function closeMultipartUpload(client, bucket, key, uploadId) {
|
|
1139
|
+
if (uploadId === void 0) {
|
|
1140
|
+
return {};
|
|
1141
|
+
}
|
|
1142
|
+
try {
|
|
1143
|
+
await client.send(
|
|
1144
|
+
new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })
|
|
1145
|
+
);
|
|
1146
|
+
return { abort: "completed" };
|
|
1147
|
+
} catch (err) {
|
|
1148
|
+
if (err?.name === "NoSuchUpload") {
|
|
1149
|
+
return { abort: "completed" };
|
|
1150
|
+
}
|
|
1151
|
+
return { abort: "failed", abortError: describeError(err) };
|
|
1152
|
+
}
|
|
1153
|
+
}
|
|
1154
|
+
async function deleteObjectsChunk(client, bucket, chunk, stripPrefix) {
|
|
1155
|
+
const deleted = [];
|
|
1156
|
+
const failed = [];
|
|
1157
|
+
try {
|
|
1158
|
+
const response = await client.send(
|
|
1159
|
+
new DeleteObjectsCommand({
|
|
1160
|
+
Bucket: bucket,
|
|
1161
|
+
// `Quiet: false` returns both successes and per-key errors.
|
|
1162
|
+
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1163
|
+
})
|
|
1164
|
+
);
|
|
1165
|
+
for (const ok of response.Deleted ?? []) {
|
|
1166
|
+
if (ok.Key !== void 0) {
|
|
1167
|
+
deleted.push(stripPrefix(ok.Key));
|
|
1168
|
+
}
|
|
1169
|
+
}
|
|
1170
|
+
for (const failure of response.Errors ?? []) {
|
|
1171
|
+
if (failure.Key !== void 0) {
|
|
1172
|
+
failed.push({
|
|
1173
|
+
key: stripPrefix(failure.Key),
|
|
1174
|
+
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1175
|
+
});
|
|
1176
|
+
}
|
|
1177
|
+
}
|
|
1178
|
+
} catch (err) {
|
|
1179
|
+
const message = describeError(err);
|
|
1180
|
+
for (const key of chunk) {
|
|
1181
|
+
failed.push({ key: stripPrefix(key), error: message });
|
|
1182
|
+
}
|
|
1183
|
+
}
|
|
1184
|
+
return { deleted, failed };
|
|
1185
|
+
}
|
|
1109
1186
|
function mapBucketCheckError(err, context) {
|
|
1110
1187
|
const failure = err ?? {};
|
|
1111
1188
|
const status = failure.$metadata?.httpStatusCode;
|
|
@@ -1347,12 +1424,16 @@ var StorageService = class {
|
|
|
1347
1424
|
...options.size !== void 0 ? { size: options.size } : {}
|
|
1348
1425
|
});
|
|
1349
1426
|
} catch (scanErr) {
|
|
1350
|
-
|
|
1427
|
+
const key = this.keyResolver.stripPrefix(finalKey);
|
|
1428
|
+
try {
|
|
1429
|
+
await this.delete(key, { bucket });
|
|
1430
|
+
} catch (deleteErr) {
|
|
1351
1431
|
this.logger.error(
|
|
1352
|
-
`
|
|
1432
|
+
`STORAGE_SCAN_CLEANUP_FAILED: the rejected object is still stored: ${finalKey}`,
|
|
1353
1433
|
describeError(deleteErr)
|
|
1354
1434
|
);
|
|
1355
|
-
|
|
1435
|
+
throw withCleanupFailure(scanErr, { key, storedKey: finalKey }, deleteErr);
|
|
1436
|
+
}
|
|
1356
1437
|
throw scanErr;
|
|
1357
1438
|
}
|
|
1358
1439
|
}
|
|
@@ -1473,12 +1554,12 @@ var StorageService = class {
|
|
|
1473
1554
|
await this.s3Provider.getClient().send(new DeleteObjectCommand({ Bucket: bucket, Key: finalKey }));
|
|
1474
1555
|
} catch (err) {
|
|
1475
1556
|
const mapped = mapAwsError(err, { key: finalKey, bucket, op: "delete" });
|
|
1476
|
-
if (mapped.code
|
|
1477
|
-
|
|
1478
|
-
return;
|
|
1557
|
+
if (mapped.code !== STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND) {
|
|
1558
|
+
throw mapped;
|
|
1479
1559
|
}
|
|
1480
|
-
|
|
1560
|
+
this.logger.warn(`delete() \u2014 key not found (idempotent no-op): ${finalKey}`);
|
|
1481
1561
|
}
|
|
1562
|
+
this.idempotencyCache.invalidate(bucket, this.keyResolver.stripPrefix(finalKey));
|
|
1482
1563
|
}
|
|
1483
1564
|
/**
|
|
1484
1565
|
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
@@ -1588,8 +1669,10 @@ var StorageService = class {
|
|
|
1588
1669
|
}
|
|
1589
1670
|
}
|
|
1590
1671
|
/**
|
|
1591
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
1592
|
-
*
|
|
1672
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
1673
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
1674
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
1675
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
1593
1676
|
*/
|
|
1594
1677
|
async uploadMultipart(options, finalKey, bucket) {
|
|
1595
1678
|
const params = {
|
|
@@ -1611,14 +1694,9 @@ var StorageService = class {
|
|
|
1611
1694
|
params,
|
|
1612
1695
|
queueSize: this.options.multipart.queueSize,
|
|
1613
1696
|
partSize: this.options.multipart.partSizeBytes,
|
|
1614
|
-
leavePartsOnError:
|
|
1697
|
+
leavePartsOnError: true
|
|
1615
1698
|
});
|
|
1616
|
-
|
|
1617
|
-
const onProgress = options.onProgress;
|
|
1618
|
-
uploader.on("httpUploadProgress", (event) => {
|
|
1619
|
-
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1620
|
-
});
|
|
1621
|
-
}
|
|
1699
|
+
this.forwardProgress(uploader, options.onProgress);
|
|
1622
1700
|
try {
|
|
1623
1701
|
const response = await uploader.done();
|
|
1624
1702
|
return this.buildUploadResult({
|
|
@@ -1631,9 +1709,26 @@ var StorageService = class {
|
|
|
1631
1709
|
multipart: true
|
|
1632
1710
|
});
|
|
1633
1711
|
} catch (err) {
|
|
1634
|
-
|
|
1712
|
+
return this.failMultipart(err, finalKey, bucket, uploader.uploadId);
|
|
1635
1713
|
}
|
|
1636
1714
|
}
|
|
1715
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
1716
|
+
forwardProgress(uploader, onProgress) {
|
|
1717
|
+
if (!onProgress) return;
|
|
1718
|
+
uploader.on("httpUploadProgress", (event) => {
|
|
1719
|
+
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1720
|
+
});
|
|
1721
|
+
}
|
|
1722
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
1723
|
+
async failMultipart(err, finalKey, bucket, uploadId) {
|
|
1724
|
+
const abort = await closeMultipartUpload(
|
|
1725
|
+
this.s3Provider.getClient(),
|
|
1726
|
+
bucket,
|
|
1727
|
+
finalKey,
|
|
1728
|
+
uploadId
|
|
1729
|
+
);
|
|
1730
|
+
throw mapMultipartError(err, { key: finalKey, bucket, ...abort });
|
|
1731
|
+
}
|
|
1637
1732
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
1638
1733
|
normalizeSingleShotBody(body) {
|
|
1639
1734
|
if (isBufferLike(body)) {
|
|
@@ -1746,42 +1841,17 @@ var StorageService = class {
|
|
|
1746
1841
|
const failed = [];
|
|
1747
1842
|
for (let i = 0; i < normalized.length; i += DELETE_BATCH_LIMIT) {
|
|
1748
1843
|
const chunk = normalized.slice(i, i + DELETE_BATCH_LIMIT);
|
|
1749
|
-
const outcome = await
|
|
1844
|
+
const outcome = await deleteObjectsChunk(
|
|
1845
|
+
this.s3Provider.getClient(),
|
|
1846
|
+
bucket,
|
|
1847
|
+
chunk,
|
|
1848
|
+
(key) => this.keyResolver.stripPrefix(key)
|
|
1849
|
+
);
|
|
1750
1850
|
deleted.push(...outcome.deleted);
|
|
1751
1851
|
failed.push(...outcome.failed);
|
|
1752
1852
|
}
|
|
1753
|
-
|
|
1754
|
-
|
|
1755
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
1756
|
-
async deleteChunk(bucket, chunk) {
|
|
1757
|
-
const deleted = [];
|
|
1758
|
-
const failed = [];
|
|
1759
|
-
try {
|
|
1760
|
-
const response = await this.s3Provider.getClient().send(
|
|
1761
|
-
new DeleteObjectsCommand({
|
|
1762
|
-
Bucket: bucket,
|
|
1763
|
-
// `Quiet: false` returns both successes and per-key errors.
|
|
1764
|
-
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1765
|
-
})
|
|
1766
|
-
);
|
|
1767
|
-
for (const ok of response.Deleted ?? []) {
|
|
1768
|
-
if (ok.Key !== void 0) {
|
|
1769
|
-
deleted.push(this.keyResolver.stripPrefix(ok.Key));
|
|
1770
|
-
}
|
|
1771
|
-
}
|
|
1772
|
-
for (const failure of response.Errors ?? []) {
|
|
1773
|
-
if (failure.Key !== void 0) {
|
|
1774
|
-
failed.push({
|
|
1775
|
-
key: this.keyResolver.stripPrefix(failure.Key),
|
|
1776
|
-
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1777
|
-
});
|
|
1778
|
-
}
|
|
1779
|
-
}
|
|
1780
|
-
} catch (err) {
|
|
1781
|
-
const message = describeError(err);
|
|
1782
|
-
for (const key of chunk) {
|
|
1783
|
-
failed.push({ key: this.keyResolver.stripPrefix(key), error: message });
|
|
1784
|
-
}
|
|
1853
|
+
for (const key of deleted) {
|
|
1854
|
+
this.idempotencyCache.invalidate(bucket, key);
|
|
1785
1855
|
}
|
|
1786
1856
|
return { deleted, failed };
|
|
1787
1857
|
}
|
|
@@ -2082,6 +2152,24 @@ var IdempotencyCache = class {
|
|
|
2082
2152
|
}
|
|
2083
2153
|
}
|
|
2084
2154
|
}
|
|
2155
|
+
/**
|
|
2156
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
2157
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
2158
|
+
*
|
|
2159
|
+
* @param bucket - The bucket of the deleted object.
|
|
2160
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
2161
|
+
* @returns How many entries were dropped.
|
|
2162
|
+
*/
|
|
2163
|
+
invalidate(bucket, key) {
|
|
2164
|
+
let dropped = 0;
|
|
2165
|
+
for (const [cacheKey, entry] of this.entries) {
|
|
2166
|
+
if (entry.value.bucket === bucket && entry.value.key === key) {
|
|
2167
|
+
this.entries.delete(cacheKey);
|
|
2168
|
+
dropped += 1;
|
|
2169
|
+
}
|
|
2170
|
+
}
|
|
2171
|
+
return dropped;
|
|
2172
|
+
}
|
|
2085
2173
|
/**
|
|
2086
2174
|
* Reports the current number of cached entries.
|
|
2087
2175
|
*
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-storage",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.6",
|
|
4
4
|
"description": "Provider-agnostic S3-compatible object storage for NestJS. Works with AWS S3, DigitalOcean Spaces, Cloudflare R2, Backblaze B2, MinIO, Wasabi.",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
},
|
|
46
46
|
"scripts": {
|
|
47
47
|
"build": "pnpm clean && tsup",
|
|
48
|
+
"check:api": "node scripts/check-api-surface.mjs",
|
|
48
49
|
"check:exports": "attw --pack . --profile strict",
|
|
49
50
|
"check:mutants": "node scripts/check-mutation-directives.mjs",
|
|
50
51
|
"check:published": "node scripts/check-published-surface.mjs",
|