@bymax-one/nest-storage 1.1.3 → 1.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +74 -1
- package/README.md +31 -5
- package/dist/server/index.cjs +217 -75
- package/dist/server/index.d.cts +32 -8
- package/dist/server/index.d.ts +32 -8
- package/dist/server/index.mjs +218 -76
- package/dist/shared/index.cjs +4 -1
- package/dist/shared/index.d.cts +3 -0
- package/dist/shared/index.d.ts +3 -0
- package/dist/shared/index.mjs +4 -1
- package/package.json +2 -1
package/CHANGELOG.md
CHANGED
|
@@ -13,6 +13,77 @@ note on each entry is the compatibility contract — read it, not the number.
|
|
|
13
13
|
|
|
14
14
|
## [Unreleased]
|
|
15
15
|
|
|
16
|
+
## [1.1.5] - 2026-11-01
|
|
17
|
+
|
|
18
|
+
### Fixed
|
|
19
|
+
|
|
20
|
+
- **With a `keyPrefix`, a post-upload scan never removed the object it rejected.** The cleanup passed
|
|
21
|
+
the already-prefixed key to `delete()`, which prefixes it again, so it deleted `prefix/prefix/key`
|
|
22
|
+
(nothing), succeeded, and the rejected object stayed stored: no error beyond the scan verdict and no
|
|
23
|
+
log line. The cleanup now targets the object's real key.
|
|
24
|
+
- **A failed cleanup or abort is no longer reported as a clean outcome.** These paths told the caller
|
|
25
|
+
one thing while the provider held another:
|
|
26
|
+
- A `post-upload` scan that rejected an object deleted it and rejected with the verdict; when that
|
|
27
|
+
delete failed, the caller saw only the verdict (`STORAGE_SCAN_INFECTED`) while the object stayed
|
|
28
|
+
stored, and the failure was a log line. The verdict is still the error, and its `details` now
|
|
29
|
+
carry `cleanup: 'failed'`, the `key` and the delete failure (`cleanupError`); the log line is at
|
|
30
|
+
error level with the code `STORAGE_SCAN_CLEANUP_FAILED`.
|
|
31
|
+
- A multipart upload whose `CompleteMultipartUpload` failed was reported as aborted but was never
|
|
32
|
+
aborted: the upload and its parts stayed open on the provider. The library now sends
|
|
33
|
+
`AbortMultipartUpload` itself.
|
|
34
|
+
- When the abort also failed, the SDK threw the abort failure in place of the failure that caused
|
|
35
|
+
it. The original failure is now the error, and `details.abort` is `'completed'` or `'failed'`
|
|
36
|
+
(with `abortError`). The same holds for the silent-source abort (`STORAGE_TIMEOUT`,
|
|
37
|
+
`details.phase: 'source-idle'`).
|
|
38
|
+
- After `delete()` or `deleteMany()` removed an object, a repeated `upload()` with the same
|
|
39
|
+
`idempotencyKey` was answered from the cache with a success (`fromIdempotencyCache: true`) for an
|
|
40
|
+
object that no longer existed. Both now drop the cache entries of the objects they delete. The
|
|
41
|
+
cache is per instance, so a delete made by another replica or outside the library still does not
|
|
42
|
+
reach it.
|
|
43
|
+
|
|
44
|
+
**Apply to a derived backend:** if you use a `keyPrefix` together with a `post-upload` scanner,
|
|
45
|
+
**audit your bucket** for objects that failed a scan in an earlier version: they were never removed
|
|
46
|
+
and nothing was logged, so your logs will not list them. Re-scan the prefix, or delete what your
|
|
47
|
+
scanner flagged. Error shapes also change, not codes. Code that reads `error.details` of
|
|
48
|
+
`STORAGE_SCAN_INFECTED`, `STORAGE_SCAN_INCONCLUSIVE`, `STORAGE_MULTIPART_ABORTED` or the
|
|
49
|
+
source-idle `STORAGE_TIMEOUT` gains the fields above: treat `details.cleanup === 'failed'` as
|
|
50
|
+
"rejected but still stored" and alert on it. A failed multipart upload now costs one more request,
|
|
51
|
+
the `AbortMultipartUpload` the SDK used to send (the credentials already needed
|
|
52
|
+
`s3:AbortMultipartUpload`). A repeated `upload()` after a `delete()` now stores the object again
|
|
53
|
+
instead of returning the earlier result. `exists()` is unchanged and still returns `false` for any
|
|
54
|
+
failure that is not a 404; the README now says so next to the method.
|
|
55
|
+
|
|
56
|
+
### Changed
|
|
57
|
+
|
|
58
|
+
- **The Stryker gate is 100.** `stryker.config.json` now fails a mutation run below 100 (`thresholds.break`
|
|
59
|
+
and `low` were 95), and the README states the gate. Equivalent mutants stay documented inline; the gate is
|
|
60
|
+
not lowered. Nothing in the published code changes.
|
|
61
|
+
|
|
62
|
+
## [1.1.4] - 2026-11-01
|
|
63
|
+
|
|
64
|
+
### Changed
|
|
65
|
+
|
|
66
|
+
- **A missing bucket is no longer reported as an absent object, and `delete()` no longer hides it.** When
|
|
67
|
+
the provider answers `NoSuchBucket`, `download()`, `upload()`, `list()`, `copy()` and `delete()` now
|
|
68
|
+
reject with the new `STORAGE_BUCKET_NOT_FOUND` (404) instead of `STORAGE_OBJECT_NOT_FOUND`, and
|
|
69
|
+
`delete()`, which used to resolve successfully against a bucket that does not exist, rejects it.
|
|
70
|
+
|
|
71
|
+
**Apply to a derived backend:** a behaviour change for a deployment whose bucket is missing or
|
|
72
|
+
misnamed. Code that treated `STORAGE_OBJECT_NOT_FOUND` after a failed `download()`, `upload()`, `list()`
|
|
73
|
+
or `copy()` as "the bucket or key is not there" must also handle `STORAGE_BUCKET_NOT_FOUND`, and a
|
|
74
|
+
`delete()` that used to succeed silently now fails until the bucket exists. `head()` and `exists()` are
|
|
75
|
+
unchanged: a `HEAD` response has no body, so the provider cannot say which of the bucket and the key is
|
|
76
|
+
missing, and a missing bucket still reads as an absent object there (use `checkBucket()` to tell).
|
|
77
|
+
|
|
78
|
+
### Added
|
|
79
|
+
|
|
80
|
+
- **`StorageService.checkBucket()`.** One `HeadBucket` that resolves when the bucket is reachable with these
|
|
81
|
+
credentials and otherwise rejects with a typed code: `STORAGE_BUCKET_NOT_FOUND` (404),
|
|
82
|
+
`STORAGE_BUCKET_ACCESS_DENIED` (403, which a provider also answers for a bucket the caller cannot list),
|
|
83
|
+
`STORAGE_BUCKET_REGION_MISMATCH` (301 or `PermanentRedirect`), `STORAGE_TIMEOUT`, or
|
|
84
|
+
`STORAGE_PROVIDER_ERROR`. Nothing calls it at boot; wire it into a readiness endpoint so a misconfigured
|
|
85
|
+
bucket keeps the instance out of rotation. It needs `s3:ListBucket`.
|
|
86
|
+
|
|
16
87
|
## [1.1.3] - 2026-11-01
|
|
17
88
|
|
|
18
89
|
### Fixed
|
|
@@ -481,4 +552,6 @@ have regressed from. They are kept because the reasoning is worth having.
|
|
|
481
552
|
[1.1.1]: https://github.com/bymaxone/nest-storage/compare/v1.1.0...v1.1.1
|
|
482
553
|
[1.1.2]: https://github.com/bymaxone/nest-storage/compare/v1.1.1...v1.1.2
|
|
483
554
|
[1.1.3]: https://github.com/bymaxone/nest-storage/compare/v1.1.2...v1.1.3
|
|
484
|
-
[
|
|
555
|
+
[1.1.4]: https://github.com/bymaxone/nest-storage/compare/v1.1.3...v1.1.4
|
|
556
|
+
[1.1.5]: https://github.com/bymaxone/nest-storage/compare/v1.1.4...v1.1.5
|
|
557
|
+
[Unreleased]: https://github.com/bymaxone/nest-storage/compare/v1.1.5...HEAD
|
package/README.md
CHANGED
|
@@ -285,6 +285,7 @@ The full configuration reference lives in `docs/technical_specification.md` §4.
|
|
|
285
285
|
|
|
286
286
|
[aws-presigned-expiry]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
|
|
287
287
|
[aws-get-object]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
|
|
288
|
+
[aws-head-object]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
|
|
288
289
|
[aws-mpu-limits]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
|
|
289
290
|
[aws-mpu-abort-rule]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpu-abort-incomplete-mpu-lifecycle-config.html
|
|
290
291
|
[sdk-request-timeout]: https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/node-configuring-maxsockets.html
|
|
@@ -384,6 +385,12 @@ const r2 = await storage.upload(opts) // returns cached result instantly
|
|
|
384
385
|
console.log(r2.fromIdempotencyCache) // true
|
|
385
386
|
```
|
|
386
387
|
|
|
388
|
+
A repeated upload is answered from the cache only while this instance still believes the object
|
|
389
|
+
exists: `delete()` and `deleteMany()` drop the entries of the objects they remove, so uploading
|
|
390
|
+
again after a delete stores the object again. The cache is **per instance**. A delete made by
|
|
391
|
+
another replica, or outside this library, does not reach it, and there a repeated upload can be
|
|
392
|
+
answered with a success for an object that no longer exists.
|
|
393
|
+
|
|
387
394
|
---
|
|
388
395
|
|
|
389
396
|
## ⬇️ Download
|
|
@@ -570,6 +577,14 @@ arrives, and the scanner and the upload then share that buffer. The read is boun
|
|
|
570
577
|
refused with a reason that says to set it or use `post-upload`. Buffer bodies and `post-upload`
|
|
571
578
|
mode are not affected; `post-upload` hands the scanner no body, only the key.
|
|
572
579
|
|
|
580
|
+
**A rejected object that could not be removed.** In `post-upload` mode the object is already
|
|
581
|
+
stored when the scanner rejects it, so the library deletes it and rejects `upload()` with the
|
|
582
|
+
verdict (`STORAGE_SCAN_INFECTED` or `STORAGE_SCAN_INCONCLUSIVE`). When that delete fails, the
|
|
583
|
+
verdict is still the error, and `details` says so: `cleanup: 'failed'`, the `key` and the delete
|
|
584
|
+
failure in `cleanupError`; the failure is also logged at error level with the code
|
|
585
|
+
`STORAGE_SCAN_CLEANUP_FAILED`. No `details.cleanup` means the object was removed; `cleanup:
|
|
586
|
+
'failed'` means it is **still stored** and has to be removed by other means.
|
|
587
|
+
|
|
573
588
|
---
|
|
574
589
|
|
|
575
590
|
## ♻️ Lifecycle Operations
|
|
@@ -735,7 +750,7 @@ All errors are thrown as `StorageException extends HttpException`. The response
|
|
|
735
750
|
| `STORAGE_MIME_NOT_ALLOWED` | 415 | `contentType` outside `mimeWhitelist` |
|
|
736
751
|
| `STORAGE_SIZE_EXCEEDED` | 413 | `size > maxSizeBytes` |
|
|
737
752
|
| `STORAGE_VALIDATION_FAILED` | 400 | `IUploadValidator` rejected the file (`details.reason`) |
|
|
738
|
-
| `STORAGE_SCAN_INFECTED` | 422 | Scanner returned `'infected'` (`details.threat`)
|
|
753
|
+
| `STORAGE_SCAN_INFECTED` | 422 | Scanner returned `'infected'` (`details.threat`); `details.cleanup: 'failed'` means the object is still stored |
|
|
739
754
|
| `STORAGE_SCAN_INCONCLUSIVE` | 422 | Scanner returned `'unknown'` and `rejectOnUnknown: true` |
|
|
740
755
|
| `STORAGE_OBJECT_NOT_FOUND` | 404 | `head()`, `download()`, or `copy()` on a nonexistent key |
|
|
741
756
|
| `STORAGE_PROVIDER_ERROR` | 502 | AWS SDK error (network, 5xx, throttling, `AccessControlListNotSupported`); `details` carries `awsCode`, `awsMessage`, `httpStatus`, `requestId` plus the operation context (`op`, `bucket`, and `key` or `prefix`) |
|
|
@@ -743,12 +758,15 @@ All errors are thrown as `StorageException extends HttpException`. The response
|
|
|
743
758
|
| `STORAGE_PART_TOO_SMALL` | 400 | Reserved: a `multipart.partSizeBytes` below 5 MiB is rejected at init as `STORAGE_INVALID_CONFIG` |
|
|
744
759
|
| `STORAGE_INVALID_PART_COUNT` | 400 | `getMultipartUploadUrls()` with `parts` not an integer from 1 to 10 000 (`details.provided`, `details.reason`) |
|
|
745
760
|
| `STORAGE_BUCKET_UNDEFINED` | 400 | Operation without a bucket and no default in config |
|
|
746
|
-
| `STORAGE_MULTIPART_ABORTED` | 500 |
|
|
761
|
+
| `STORAGE_MULTIPART_ABORTED` | 500 | A multipart upload failed. `details.abort` is `'completed'` when the library aborted it and `'failed'` (with `abortError`) when the abort itself failed: that upload stays open on the provider until a lifecycle rule removes it |
|
|
747
762
|
| `STORAGE_INVALID_CONFIG` | 500 | `BymaxStorageModuleOptions` validation failed at initialization |
|
|
748
763
|
| `STORAGE_TIMEOUT` | 504 | An attempt got no response headers within `requestTimeoutMs` (the SDK's `TimeoutError`); a multipart source that stays silent for `uploadIdleTimeoutMs` also surfaces as `STORAGE_TIMEOUT` (`details.phase: 'source-idle'`); any other failure inside a multipart upload surfaces as `STORAGE_MULTIPART_ABORTED` |
|
|
749
764
|
| `STORAGE_LIFECYCLE_RULE_INVALID` | 400 | `BucketLifecycleService` input rejected: empty or over-long `id`, non-positive or non-integer `expirationDays`, bad `prefix`, or an id naming a rule outside `keyPrefix` (`details.reason`) |
|
|
750
765
|
| `STORAGE_LIFECYCLE_UNSUPPORTED` | 501 | The provider does not implement the lifecycle API |
|
|
751
766
|
| `STORAGE_LIFECYCLE_REJECTED` | 422 | The provider refused the lifecycle configuration |
|
|
767
|
+
| `STORAGE_BUCKET_NOT_FOUND` | 404 | The provider answered `NoSuchBucket` (any operation that gets an error body: `download()`, `upload()`, `list()`, `copy()`, `delete()`), or `checkBucket()` got a 404 |
|
|
768
|
+
| `STORAGE_BUCKET_ACCESS_DENIED` | 403 | `checkBucket()` got a 403: the credentials cannot use the bucket, or it does not exist and the credentials cannot list (S3 answers 403 to a caller without `s3:ListBucket`) |
|
|
769
|
+
| `STORAGE_BUCKET_REGION_MISMATCH` | 500 | `checkBucket()` got a 301 or `PermanentRedirect`: the bucket lives in another region than the configured one |
|
|
752
770
|
| `STORAGE_NOT_MODIFIED` | 304 | `download()` with `ifNoneMatch` equal to the current ETag |
|
|
753
771
|
| `STORAGE_PRECONDITION_FAILED` | 412 | `download()` with `ifMatch` different from the current ETag |
|
|
754
772
|
|
|
@@ -782,10 +800,18 @@ This is the index.
|
|
|
782
800
|
| `head` | `(key, options?) => Promise<ObjectMetadata>` | Metadata without transferring the body |
|
|
783
801
|
| `exists` | `(key, options?) => Promise<boolean>` | `head` reduced to a boolean; any non-404 failure is logged and returns `false` |
|
|
784
802
|
| `getPublicUrl` | `(key, options?) => string` | Composed, not signed — for buckets that are public by policy |
|
|
785
|
-
| `delete` | `(key, options?) => Promise<void>` | Idempotent, as S3 delete is
|
|
803
|
+
| `delete` | `(key, options?) => Promise<void>` | Idempotent for a missing object, as S3 delete is; a missing bucket rejects `STORAGE_BUCKET_NOT_FOUND` |
|
|
786
804
|
| `deleteMany` | `(keys: string[], options?) => Promise<DeleteManyResult>` | Partial failures are reported, not thrown |
|
|
787
805
|
| `list` | `(options: ListOptions) => Promise<ListResult>` | Paginated by continuation token |
|
|
788
806
|
| `copy` | `(options: CopyOptions) => Promise<{ etag }>` | Server-side; the bytes never reach this process |
|
|
807
|
+
| `checkBucket` | `(options?) => Promise<void>` | One `HeadBucket` for a readiness check; resolves when the bucket is reachable, otherwise rejects `STORAGE_BUCKET_NOT_FOUND`, `STORAGE_BUCKET_ACCESS_DENIED`, `STORAGE_BUCKET_REGION_MISMATCH` or `STORAGE_TIMEOUT`; needs `s3:ListBucket`; never called at boot |
|
|
808
|
+
|
|
809
|
+
> **`exists()` returns `false` for any failure that is not a 404** (a 403, a 5xx, a timeout), after
|
|
810
|
+
> logging a warning, so a transient provider error reads as "absent". Do not use it to decide whether
|
|
811
|
+
> to overwrite or to create (`if (!(await exists(k))) await upload(k)` can overwrite an object
|
|
812
|
+
> that is there); use `head()` and branch on the error code.
|
|
813
|
+
|
|
814
|
+
**A missing bucket and `head()` / `exists()`.** A `HEAD` response has no body, so the provider cannot say which of the bucket and the key is missing: AWS documents that "if the `HEAD` request generates an error, it returns a generic code, such as `400 Bad Request`, `403 Forbidden`, `404 Not Found` ... It's not possible to retrieve the exact exception of these error codes" ([HeadObject][aws-head-object]). `head()` therefore rejects `STORAGE_OBJECT_NOT_FOUND` and `exists()` returns `false` for a missing bucket too. Call `checkBucket()` to tell the two apart, for example from a readiness endpoint.
|
|
789
815
|
|
|
790
816
|
### `SignedUrlService`
|
|
791
817
|
|
|
@@ -1023,7 +1049,7 @@ place stays there — so the suite is held to a bar beyond "the tests pass".
|
|
|
1023
1049
|
|
|
1024
1050
|
- ✅ **100% line coverage** — statements, branches, functions and lines, enforced as a gate
|
|
1025
1051
|
- ✅ **100% mutation score** — verified with [Stryker](https://stryker-mutator.io/) at
|
|
1026
|
-
`break:
|
|
1052
|
+
`break: 100`; every survivor was killed by a strengthened assertion, and the provable
|
|
1027
1053
|
equivalents are documented inline
|
|
1028
1054
|
- ✅ **Real object storage in e2e** — MinIO through Testcontainers, so multipart, presigning
|
|
1029
1055
|
and listing are exercised against an actual S3 API rather than a mock
|
|
@@ -1040,7 +1066,7 @@ place stays there — so the suite is held to a bar beyond "the tests pass".
|
|
|
1040
1066
|
pnpm test # unit tests (Jest)
|
|
1041
1067
|
pnpm test:cov # unit tests with the 100% coverage gate
|
|
1042
1068
|
pnpm test:e2e # end-to-end against MinIO (requires Docker)
|
|
1043
|
-
pnpm mutation # Stryker mutation testing (break:
|
|
1069
|
+
pnpm mutation # Stryker mutation testing (break: 100)
|
|
1044
1070
|
pnpm typecheck # tsc strict check
|
|
1045
1071
|
pnpm lint # ESLint
|
|
1046
1072
|
```
|
package/dist/server/index.cjs
CHANGED
|
@@ -50,7 +50,10 @@ var STORAGE_ERROR_CODES = {
|
|
|
50
50
|
STORAGE_LIFECYCLE_UNSUPPORTED: "STORAGE_LIFECYCLE_UNSUPPORTED",
|
|
51
51
|
STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED",
|
|
52
52
|
STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED",
|
|
53
|
-
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED"
|
|
53
|
+
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED",
|
|
54
|
+
STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND",
|
|
55
|
+
STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED",
|
|
56
|
+
STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH"
|
|
54
57
|
};
|
|
55
58
|
|
|
56
59
|
// src/server/errors/storage-error-messages.ts
|
|
@@ -77,7 +80,10 @@ var STORAGE_ERROR_MESSAGES = {
|
|
|
77
80
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_UNSUPPORTED]: "The storage provider does not support bucket lifecycle configuration",
|
|
78
81
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_REJECTED]: "The storage provider rejected the lifecycle configuration",
|
|
79
82
|
[STORAGE_ERROR_CODES.STORAGE_NOT_MODIFIED]: "The object has not changed since the supplied ETag",
|
|
80
|
-
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: "The object does not satisfy the supplied precondition"
|
|
83
|
+
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: "The object does not satisfy the supplied precondition",
|
|
84
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND]: "The bucket does not exist",
|
|
85
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED]: "The bucket cannot be accessed with these credentials (it may also not exist)",
|
|
86
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH]: "The bucket belongs to a different region than the configured one"
|
|
81
87
|
};
|
|
82
88
|
var STORAGE_ERROR_STATUS = {
|
|
83
89
|
[STORAGE_ERROR_CODES.STORAGE_NOT_CONFIGURED]: common.HttpStatus.SERVICE_UNAVAILABLE,
|
|
@@ -102,7 +108,10 @@ var STORAGE_ERROR_STATUS = {
|
|
|
102
108
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_UNSUPPORTED]: common.HttpStatus.NOT_IMPLEMENTED,
|
|
103
109
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_REJECTED]: common.HttpStatus.UNPROCESSABLE_ENTITY,
|
|
104
110
|
[STORAGE_ERROR_CODES.STORAGE_NOT_MODIFIED]: common.HttpStatus.NOT_MODIFIED,
|
|
105
|
-
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: common.HttpStatus.PRECONDITION_FAILED
|
|
111
|
+
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: common.HttpStatus.PRECONDITION_FAILED,
|
|
112
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND]: common.HttpStatus.NOT_FOUND,
|
|
113
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED]: common.HttpStatus.FORBIDDEN,
|
|
114
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH]: common.HttpStatus.INTERNAL_SERVER_ERROR
|
|
106
115
|
};
|
|
107
116
|
|
|
108
117
|
// src/server/errors/storage-exception.ts
|
|
@@ -415,6 +424,13 @@ function mapAwsError(err, context) {
|
|
|
415
424
|
requestId,
|
|
416
425
|
awsMessage: e.message
|
|
417
426
|
};
|
|
427
|
+
if (awsCode === "NoSuchBucket") {
|
|
428
|
+
return new StorageException(
|
|
429
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND,
|
|
430
|
+
common.HttpStatus.NOT_FOUND,
|
|
431
|
+
details
|
|
432
|
+
);
|
|
433
|
+
}
|
|
418
434
|
if (e.name === "NotFound" || httpStatus === 404) {
|
|
419
435
|
return new StorageException(
|
|
420
436
|
STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND,
|
|
@@ -1083,11 +1099,127 @@ function mapMultipartError(err, context) {
|
|
|
1083
1099
|
idleMs: err.idleMs
|
|
1084
1100
|
});
|
|
1085
1101
|
}
|
|
1102
|
+
const bucketMissing = mapAwsError(err, context);
|
|
1103
|
+
if (bucketMissing.code === STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND) {
|
|
1104
|
+
return bucketMissing;
|
|
1105
|
+
}
|
|
1086
1106
|
return new StorageException(STORAGE_ERROR_CODES.STORAGE_MULTIPART_ABORTED, void 0, {
|
|
1087
1107
|
...context,
|
|
1088
1108
|
awsMessage: describeError(err)
|
|
1089
1109
|
});
|
|
1090
1110
|
}
|
|
1111
|
+
function knownStatus(error) {
|
|
1112
|
+
const name = common.HttpStatus[error.getStatus()];
|
|
1113
|
+
return name === void 0 ? void 0 : common.HttpStatus[name];
|
|
1114
|
+
}
|
|
1115
|
+
function withCleanupFailure(scanError, key, cleanupError) {
|
|
1116
|
+
if (!(scanError instanceof StorageException)) {
|
|
1117
|
+
return scanError;
|
|
1118
|
+
}
|
|
1119
|
+
const body = scanError.getResponse();
|
|
1120
|
+
return new StorageException(scanError.code, knownStatus(scanError), {
|
|
1121
|
+
...body.error?.details ?? {},
|
|
1122
|
+
key,
|
|
1123
|
+
cleanup: "failed",
|
|
1124
|
+
cleanupError: describeError(cleanupError)
|
|
1125
|
+
});
|
|
1126
|
+
}
|
|
1127
|
+
async function closeMultipartUpload(client, bucket, key, uploadId) {
|
|
1128
|
+
if (uploadId === void 0) {
|
|
1129
|
+
return {};
|
|
1130
|
+
}
|
|
1131
|
+
try {
|
|
1132
|
+
await client.send(
|
|
1133
|
+
new clientS3.AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })
|
|
1134
|
+
);
|
|
1135
|
+
return { abort: "completed" };
|
|
1136
|
+
} catch (err) {
|
|
1137
|
+
if (err?.name === "NoSuchUpload") {
|
|
1138
|
+
return { abort: "completed" };
|
|
1139
|
+
}
|
|
1140
|
+
return { abort: "failed", abortError: describeError(err) };
|
|
1141
|
+
}
|
|
1142
|
+
}
|
|
1143
|
+
async function deleteObjectsChunk(client, bucket, chunk, stripPrefix) {
|
|
1144
|
+
const deleted = [];
|
|
1145
|
+
const failed = [];
|
|
1146
|
+
try {
|
|
1147
|
+
const response = await client.send(
|
|
1148
|
+
new clientS3.DeleteObjectsCommand({
|
|
1149
|
+
Bucket: bucket,
|
|
1150
|
+
// `Quiet: false` returns both successes and per-key errors.
|
|
1151
|
+
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1152
|
+
})
|
|
1153
|
+
);
|
|
1154
|
+
for (const ok of response.Deleted ?? []) {
|
|
1155
|
+
if (ok.Key !== void 0) {
|
|
1156
|
+
deleted.push(stripPrefix(ok.Key));
|
|
1157
|
+
}
|
|
1158
|
+
}
|
|
1159
|
+
for (const failure of response.Errors ?? []) {
|
|
1160
|
+
if (failure.Key !== void 0) {
|
|
1161
|
+
failed.push({
|
|
1162
|
+
key: stripPrefix(failure.Key),
|
|
1163
|
+
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1164
|
+
});
|
|
1165
|
+
}
|
|
1166
|
+
}
|
|
1167
|
+
} catch (err) {
|
|
1168
|
+
const message = describeError(err);
|
|
1169
|
+
for (const key of chunk) {
|
|
1170
|
+
failed.push({ key: stripPrefix(key), error: message });
|
|
1171
|
+
}
|
|
1172
|
+
}
|
|
1173
|
+
return { deleted, failed };
|
|
1174
|
+
}
|
|
1175
|
+
function mapBucketCheckError(err, context) {
|
|
1176
|
+
const failure = err ?? {};
|
|
1177
|
+
const status = failure.$metadata?.httpStatusCode;
|
|
1178
|
+
const details = { ...context, httpStatus: status, awsCode: failure.name };
|
|
1179
|
+
if (status === 301 || failure.name === "PermanentRedirect") {
|
|
1180
|
+
return new StorageException(
|
|
1181
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH,
|
|
1182
|
+
void 0,
|
|
1183
|
+
details
|
|
1184
|
+
);
|
|
1185
|
+
}
|
|
1186
|
+
if (status === 403) {
|
|
1187
|
+
return new StorageException(
|
|
1188
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED,
|
|
1189
|
+
void 0,
|
|
1190
|
+
details
|
|
1191
|
+
);
|
|
1192
|
+
}
|
|
1193
|
+
if (status === 404) {
|
|
1194
|
+
return new StorageException(
|
|
1195
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND,
|
|
1196
|
+
common.HttpStatus.NOT_FOUND,
|
|
1197
|
+
details
|
|
1198
|
+
);
|
|
1199
|
+
}
|
|
1200
|
+
return mapAwsError(err, context);
|
|
1201
|
+
}
|
|
1202
|
+
|
|
1203
|
+
// src/server/utils/list-result.ts
|
|
1204
|
+
function toListedObject(obj, stripPrefix) {
|
|
1205
|
+
return {
|
|
1206
|
+
key: stripPrefix(obj.Key ?? ""),
|
|
1207
|
+
size: obj.Size ?? 0,
|
|
1208
|
+
etag: obj.ETag ?? "",
|
|
1209
|
+
lastModified: obj.LastModified ?? /* @__PURE__ */ new Date(0),
|
|
1210
|
+
...obj.StorageClass !== void 0 ? { storageClass: obj.StorageClass } : {}
|
|
1211
|
+
};
|
|
1212
|
+
}
|
|
1213
|
+
function toListResult(response, stripPrefix) {
|
|
1214
|
+
const objects = (response.Contents ?? []).map((obj) => toListedObject(obj, stripPrefix));
|
|
1215
|
+
const commonPrefixes = (response.CommonPrefixes ?? []).map((cp) => cp.Prefix ?? "").filter((cp) => cp.length > 0).map((cp) => stripPrefix(cp));
|
|
1216
|
+
return {
|
|
1217
|
+
objects,
|
|
1218
|
+
commonPrefixes,
|
|
1219
|
+
isTruncated: response.IsTruncated ?? false,
|
|
1220
|
+
...response.NextContinuationToken !== void 0 ? { nextContinuationToken: response.NextContinuationToken } : {}
|
|
1221
|
+
};
|
|
1222
|
+
}
|
|
1091
1223
|
|
|
1092
1224
|
// src/server/utils/header-utils.ts
|
|
1093
1225
|
function buildContentDisposition(perCall, defaultValue) {
|
|
@@ -1281,12 +1413,16 @@ exports.StorageService = class StorageService {
|
|
|
1281
1413
|
...options.size !== void 0 ? { size: options.size } : {}
|
|
1282
1414
|
});
|
|
1283
1415
|
} catch (scanErr) {
|
|
1284
|
-
|
|
1416
|
+
const key = this.keyResolver.stripPrefix(finalKey);
|
|
1417
|
+
try {
|
|
1418
|
+
await this.delete(key, { bucket });
|
|
1419
|
+
} catch (deleteErr) {
|
|
1285
1420
|
this.logger.error(
|
|
1286
|
-
`
|
|
1421
|
+
`STORAGE_SCAN_CLEANUP_FAILED: the rejected object is still stored: ${key}`,
|
|
1287
1422
|
describeError(deleteErr)
|
|
1288
1423
|
);
|
|
1289
|
-
|
|
1424
|
+
throw withCleanupFailure(scanErr, key, deleteErr);
|
|
1425
|
+
}
|
|
1290
1426
|
throw scanErr;
|
|
1291
1427
|
}
|
|
1292
1428
|
}
|
|
@@ -1407,11 +1543,31 @@ exports.StorageService = class StorageService {
|
|
|
1407
1543
|
await this.s3Provider.getClient().send(new clientS3.DeleteObjectCommand({ Bucket: bucket, Key: finalKey }));
|
|
1408
1544
|
} catch (err) {
|
|
1409
1545
|
const mapped = mapAwsError(err, { key: finalKey, bucket, op: "delete" });
|
|
1410
|
-
if (mapped.code
|
|
1411
|
-
|
|
1412
|
-
return;
|
|
1546
|
+
if (mapped.code !== STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND) {
|
|
1547
|
+
throw mapped;
|
|
1413
1548
|
}
|
|
1414
|
-
|
|
1549
|
+
this.logger.warn(`delete() \u2014 key not found (idempotent no-op): ${finalKey}`);
|
|
1550
|
+
}
|
|
1551
|
+
this.idempotencyCache.invalidate(bucket, this.keyResolver.stripPrefix(finalKey));
|
|
1552
|
+
}
|
|
1553
|
+
/**
|
|
1554
|
+
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
1555
|
+
* is reachable with these credentials and otherwise rejects with a typed code. Nothing in the
|
|
1556
|
+
* module calls it on its own, so booting costs no request. Needs `s3:ListBucket`.
|
|
1557
|
+
*
|
|
1558
|
+
* @param options - Optional per-call bucket override.
|
|
1559
|
+
* @throws StorageException `STORAGE_BUCKET_NOT_FOUND` (404), `STORAGE_BUCKET_ACCESS_DENIED` (403,
|
|
1560
|
+
* which a provider also answers for a bucket the caller cannot see), `STORAGE_BUCKET_REGION_MISMATCH`
|
|
1561
|
+
* (301), `STORAGE_TIMEOUT`, or `STORAGE_PROVIDER_ERROR`; `STORAGE_NOT_CONFIGURED` and
|
|
1562
|
+
* `STORAGE_BUCKET_UNDEFINED` before any request.
|
|
1563
|
+
*/
|
|
1564
|
+
async checkBucket(options) {
|
|
1565
|
+
this.assertConfigured();
|
|
1566
|
+
const bucket = this.resolveBucket(options?.bucket);
|
|
1567
|
+
try {
|
|
1568
|
+
await this.s3Provider.getClient().send(new clientS3.HeadBucketCommand({ Bucket: bucket }));
|
|
1569
|
+
} catch (err) {
|
|
1570
|
+
throw mapBucketCheckError(err, { bucket, op: "checkBucket" });
|
|
1415
1571
|
}
|
|
1416
1572
|
}
|
|
1417
1573
|
/** Throws when the S3 client was never built (missing credentials). */
|
|
@@ -1502,8 +1658,10 @@ exports.StorageService = class StorageService {
|
|
|
1502
1658
|
}
|
|
1503
1659
|
}
|
|
1504
1660
|
/**
|
|
1505
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
1506
|
-
*
|
|
1661
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
1662
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
1663
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
1664
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
1507
1665
|
*/
|
|
1508
1666
|
async uploadMultipart(options, finalKey, bucket) {
|
|
1509
1667
|
const params = {
|
|
@@ -1525,14 +1683,9 @@ exports.StorageService = class StorageService {
|
|
|
1525
1683
|
params,
|
|
1526
1684
|
queueSize: this.options.multipart.queueSize,
|
|
1527
1685
|
partSize: this.options.multipart.partSizeBytes,
|
|
1528
|
-
leavePartsOnError:
|
|
1686
|
+
leavePartsOnError: true
|
|
1529
1687
|
});
|
|
1530
|
-
|
|
1531
|
-
const onProgress = options.onProgress;
|
|
1532
|
-
uploader.on("httpUploadProgress", (event) => {
|
|
1533
|
-
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1534
|
-
});
|
|
1535
|
-
}
|
|
1688
|
+
this.forwardProgress(uploader, options.onProgress);
|
|
1536
1689
|
try {
|
|
1537
1690
|
const response = await uploader.done();
|
|
1538
1691
|
return this.buildUploadResult({
|
|
@@ -1545,9 +1698,26 @@ exports.StorageService = class StorageService {
|
|
|
1545
1698
|
multipart: true
|
|
1546
1699
|
});
|
|
1547
1700
|
} catch (err) {
|
|
1548
|
-
|
|
1701
|
+
return this.failMultipart(err, finalKey, bucket, uploader.uploadId);
|
|
1549
1702
|
}
|
|
1550
1703
|
}
|
|
1704
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
1705
|
+
forwardProgress(uploader, onProgress) {
|
|
1706
|
+
if (!onProgress) return;
|
|
1707
|
+
uploader.on("httpUploadProgress", (event) => {
|
|
1708
|
+
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1709
|
+
});
|
|
1710
|
+
}
|
|
1711
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
1712
|
+
async failMultipart(err, finalKey, bucket, uploadId) {
|
|
1713
|
+
const abort = await closeMultipartUpload(
|
|
1714
|
+
this.s3Provider.getClient(),
|
|
1715
|
+
bucket,
|
|
1716
|
+
finalKey,
|
|
1717
|
+
uploadId
|
|
1718
|
+
);
|
|
1719
|
+
throw mapMultipartError(err, { key: finalKey, bucket, ...abort });
|
|
1720
|
+
}
|
|
1551
1721
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
1552
1722
|
normalizeSingleShotBody(body) {
|
|
1553
1723
|
if (isBufferLike(body)) {
|
|
@@ -1601,32 +1771,11 @@ exports.StorageService = class StorageService {
|
|
|
1601
1771
|
Delimiter: options.delimiter
|
|
1602
1772
|
})
|
|
1603
1773
|
);
|
|
1604
|
-
return
|
|
1774
|
+
return toListResult(response, (key) => this.keyResolver.stripPrefix(key));
|
|
1605
1775
|
} catch (err) {
|
|
1606
1776
|
throw mapAwsError(err, { bucket, prefix, op: "list" });
|
|
1607
1777
|
}
|
|
1608
1778
|
}
|
|
1609
|
-
/** Maps a `ListObjectsV2` response onto the public `ListResult` shape. */
|
|
1610
|
-
toListResult(response) {
|
|
1611
|
-
const objects = (response.Contents ?? []).map((obj) => this.toListedObject(obj));
|
|
1612
|
-
const commonPrefixes = (response.CommonPrefixes ?? []).map((cp) => cp.Prefix ?? "").filter((cp) => cp.length > 0).map((cp) => this.keyResolver.stripPrefix(cp));
|
|
1613
|
-
return {
|
|
1614
|
-
objects,
|
|
1615
|
-
commonPrefixes,
|
|
1616
|
-
isTruncated: response.IsTruncated ?? false,
|
|
1617
|
-
...response.NextContinuationToken !== void 0 ? { nextContinuationToken: response.NextContinuationToken } : {}
|
|
1618
|
-
};
|
|
1619
|
-
}
|
|
1620
|
-
/** Maps one `Contents` entry onto `ListedObject`, stripping the global prefix. */
|
|
1621
|
-
toListedObject(obj) {
|
|
1622
|
-
return {
|
|
1623
|
-
key: this.keyResolver.stripPrefix(obj.Key ?? ""),
|
|
1624
|
-
size: obj.Size ?? 0,
|
|
1625
|
-
etag: obj.ETag ?? "",
|
|
1626
|
-
lastModified: obj.LastModified ?? /* @__PURE__ */ new Date(0),
|
|
1627
|
-
...obj.StorageClass !== void 0 ? { storageClass: obj.StorageClass } : {}
|
|
1628
|
-
};
|
|
1629
|
-
}
|
|
1630
1779
|
/**
|
|
1631
1780
|
* Server-side copy of an object (no bytes flow through the app). Supports both
|
|
1632
1781
|
* same-bucket and cross-bucket copies and preserves the source metadata.
|
|
@@ -1681,42 +1830,17 @@ exports.StorageService = class StorageService {
|
|
|
1681
1830
|
const failed = [];
|
|
1682
1831
|
for (let i = 0; i < normalized.length; i += DELETE_BATCH_LIMIT) {
|
|
1683
1832
|
const chunk = normalized.slice(i, i + DELETE_BATCH_LIMIT);
|
|
1684
|
-
const outcome = await
|
|
1833
|
+
const outcome = await deleteObjectsChunk(
|
|
1834
|
+
this.s3Provider.getClient(),
|
|
1835
|
+
bucket,
|
|
1836
|
+
chunk,
|
|
1837
|
+
(key) => this.keyResolver.stripPrefix(key)
|
|
1838
|
+
);
|
|
1685
1839
|
deleted.push(...outcome.deleted);
|
|
1686
1840
|
failed.push(...outcome.failed);
|
|
1687
1841
|
}
|
|
1688
|
-
|
|
1689
|
-
|
|
1690
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
1691
|
-
async deleteChunk(bucket, chunk) {
|
|
1692
|
-
const deleted = [];
|
|
1693
|
-
const failed = [];
|
|
1694
|
-
try {
|
|
1695
|
-
const response = await this.s3Provider.getClient().send(
|
|
1696
|
-
new clientS3.DeleteObjectsCommand({
|
|
1697
|
-
Bucket: bucket,
|
|
1698
|
-
// `Quiet: false` returns both successes and per-key errors.
|
|
1699
|
-
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1700
|
-
})
|
|
1701
|
-
);
|
|
1702
|
-
for (const ok of response.Deleted ?? []) {
|
|
1703
|
-
if (ok.Key !== void 0) {
|
|
1704
|
-
deleted.push(this.keyResolver.stripPrefix(ok.Key));
|
|
1705
|
-
}
|
|
1706
|
-
}
|
|
1707
|
-
for (const failure of response.Errors ?? []) {
|
|
1708
|
-
if (failure.Key !== void 0) {
|
|
1709
|
-
failed.push({
|
|
1710
|
-
key: this.keyResolver.stripPrefix(failure.Key),
|
|
1711
|
-
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1712
|
-
});
|
|
1713
|
-
}
|
|
1714
|
-
}
|
|
1715
|
-
} catch (err) {
|
|
1716
|
-
const message = describeError(err);
|
|
1717
|
-
for (const key of chunk) {
|
|
1718
|
-
failed.push({ key: this.keyResolver.stripPrefix(key), error: message });
|
|
1719
|
-
}
|
|
1842
|
+
for (const key of deleted) {
|
|
1843
|
+
this.idempotencyCache.invalidate(bucket, key);
|
|
1720
1844
|
}
|
|
1721
1845
|
return { deleted, failed };
|
|
1722
1846
|
}
|
|
@@ -2017,6 +2141,24 @@ var IdempotencyCache = class {
|
|
|
2017
2141
|
}
|
|
2018
2142
|
}
|
|
2019
2143
|
}
|
|
2144
|
+
/**
|
|
2145
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
2146
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
2147
|
+
*
|
|
2148
|
+
* @param bucket - The bucket of the deleted object.
|
|
2149
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
2150
|
+
* @returns How many entries were dropped.
|
|
2151
|
+
*/
|
|
2152
|
+
invalidate(bucket, key) {
|
|
2153
|
+
let dropped = 0;
|
|
2154
|
+
for (const [cacheKey, entry] of this.entries) {
|
|
2155
|
+
if (entry.value.bucket === bucket && entry.value.key === key) {
|
|
2156
|
+
this.entries.delete(cacheKey);
|
|
2157
|
+
dropped += 1;
|
|
2158
|
+
}
|
|
2159
|
+
}
|
|
2160
|
+
return dropped;
|
|
2161
|
+
}
|
|
2020
2162
|
/**
|
|
2021
2163
|
* Reports the current number of cached entries.
|
|
2022
2164
|
*
|
package/dist/server/index.d.cts
CHANGED
|
@@ -691,6 +691,15 @@ declare class IdempotencyCache {
|
|
|
691
691
|
* @param value - The upload result to cache.
|
|
692
692
|
*/
|
|
693
693
|
set(cacheKey: string, value: UploadResult): void;
|
|
694
|
+
/**
|
|
695
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
696
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
697
|
+
*
|
|
698
|
+
* @param bucket - The bucket of the deleted object.
|
|
699
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
700
|
+
* @returns How many entries were dropped.
|
|
701
|
+
*/
|
|
702
|
+
invalidate(bucket: string, key: string): number;
|
|
694
703
|
/**
|
|
695
704
|
* Reports the current number of cached entries.
|
|
696
705
|
*
|
|
@@ -812,6 +821,18 @@ declare class StorageService {
|
|
|
812
821
|
* @param options - Optional per-call bucket override.
|
|
813
822
|
*/
|
|
814
823
|
delete(key: string, options?: BucketScopedOptions): Promise<void>;
|
|
824
|
+
/**
|
|
825
|
+
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
826
|
+
* is reachable with these credentials and otherwise rejects with a typed code. Nothing in the
|
|
827
|
+
* module calls it on its own, so booting costs no request. Needs `s3:ListBucket`.
|
|
828
|
+
*
|
|
829
|
+
* @param options - Optional per-call bucket override.
|
|
830
|
+
* @throws StorageException `STORAGE_BUCKET_NOT_FOUND` (404), `STORAGE_BUCKET_ACCESS_DENIED` (403,
|
|
831
|
+
* which a provider also answers for a bucket the caller cannot see), `STORAGE_BUCKET_REGION_MISMATCH`
|
|
832
|
+
* (301), `STORAGE_TIMEOUT`, or `STORAGE_PROVIDER_ERROR`; `STORAGE_NOT_CONFIGURED` and
|
|
833
|
+
* `STORAGE_BUCKET_UNDEFINED` before any request.
|
|
834
|
+
*/
|
|
835
|
+
checkBucket(options?: BucketScopedOptions): Promise<void>;
|
|
815
836
|
/** Throws when the S3 client was never built (missing credentials). */
|
|
816
837
|
private assertConfigured;
|
|
817
838
|
/** Resolves the target bucket from a per-call override or the module default. */
|
|
@@ -830,10 +851,16 @@ declare class StorageService {
|
|
|
830
851
|
/** Single `PutObject` upload for bodies below the multipart threshold. */
|
|
831
852
|
private uploadSingleShot;
|
|
832
853
|
/**
|
|
833
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
834
|
-
*
|
|
854
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
855
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
856
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
857
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
835
858
|
*/
|
|
836
859
|
private uploadMultipart;
|
|
860
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
861
|
+
private forwardProgress;
|
|
862
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
863
|
+
private failMultipart;
|
|
837
864
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
838
865
|
private normalizeSingleShotBody;
|
|
839
866
|
/** Assembles an `UploadResult`, omitting optional fields that are absent. */
|
|
@@ -850,10 +877,6 @@ declare class StorageService {
|
|
|
850
877
|
* @throws StorageException mapped from any AWS failure.
|
|
851
878
|
*/
|
|
852
879
|
list(options: ListOptions): Promise<ListResult>;
|
|
853
|
-
/** Maps a `ListObjectsV2` response onto the public `ListResult` shape. */
|
|
854
|
-
private toListResult;
|
|
855
|
-
/** Maps one `Contents` entry onto `ListedObject`, stripping the global prefix. */
|
|
856
|
-
private toListedObject;
|
|
857
880
|
/**
|
|
858
881
|
* Server-side copy of an object (no bytes flow through the app). Supports both
|
|
859
882
|
* same-bucket and cross-bucket copies and preserves the source metadata.
|
|
@@ -876,8 +899,6 @@ declare class StorageService {
|
|
|
876
899
|
* @returns The deleted keys and the per-key failures.
|
|
877
900
|
*/
|
|
878
901
|
deleteMany(keys: string[], options?: DeleteManyOptions): Promise<DeleteManyResult>;
|
|
879
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
880
|
-
private deleteChunk;
|
|
881
902
|
}
|
|
882
903
|
|
|
883
904
|
/**
|
|
@@ -1349,6 +1370,9 @@ declare const STORAGE_ERROR_CODES: {
|
|
|
1349
1370
|
readonly STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED";
|
|
1350
1371
|
readonly STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED";
|
|
1351
1372
|
readonly STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED";
|
|
1373
|
+
readonly STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND";
|
|
1374
|
+
readonly STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED";
|
|
1375
|
+
readonly STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH";
|
|
1352
1376
|
};
|
|
1353
1377
|
/** Union of every stable storage error code. */
|
|
1354
1378
|
type StorageErrorCode = (typeof STORAGE_ERROR_CODES)[keyof typeof STORAGE_ERROR_CODES];
|
package/dist/server/index.d.ts
CHANGED
|
@@ -691,6 +691,15 @@ declare class IdempotencyCache {
|
|
|
691
691
|
* @param value - The upload result to cache.
|
|
692
692
|
*/
|
|
693
693
|
set(cacheKey: string, value: UploadResult): void;
|
|
694
|
+
/**
|
|
695
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
696
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
697
|
+
*
|
|
698
|
+
* @param bucket - The bucket of the deleted object.
|
|
699
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
700
|
+
* @returns How many entries were dropped.
|
|
701
|
+
*/
|
|
702
|
+
invalidate(bucket: string, key: string): number;
|
|
694
703
|
/**
|
|
695
704
|
* Reports the current number of cached entries.
|
|
696
705
|
*
|
|
@@ -812,6 +821,18 @@ declare class StorageService {
|
|
|
812
821
|
* @param options - Optional per-call bucket override.
|
|
813
822
|
*/
|
|
814
823
|
delete(key: string, options?: BucketScopedOptions): Promise<void>;
|
|
824
|
+
/**
|
|
825
|
+
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
826
|
+
* is reachable with these credentials and otherwise rejects with a typed code. Nothing in the
|
|
827
|
+
* module calls it on its own, so booting costs no request. Needs `s3:ListBucket`.
|
|
828
|
+
*
|
|
829
|
+
* @param options - Optional per-call bucket override.
|
|
830
|
+
* @throws StorageException `STORAGE_BUCKET_NOT_FOUND` (404), `STORAGE_BUCKET_ACCESS_DENIED` (403,
|
|
831
|
+
* which a provider also answers for a bucket the caller cannot see), `STORAGE_BUCKET_REGION_MISMATCH`
|
|
832
|
+
* (301), `STORAGE_TIMEOUT`, or `STORAGE_PROVIDER_ERROR`; `STORAGE_NOT_CONFIGURED` and
|
|
833
|
+
* `STORAGE_BUCKET_UNDEFINED` before any request.
|
|
834
|
+
*/
|
|
835
|
+
checkBucket(options?: BucketScopedOptions): Promise<void>;
|
|
815
836
|
/** Throws when the S3 client was never built (missing credentials). */
|
|
816
837
|
private assertConfigured;
|
|
817
838
|
/** Resolves the target bucket from a per-call override or the module default. */
|
|
@@ -830,10 +851,16 @@ declare class StorageService {
|
|
|
830
851
|
/** Single `PutObject` upload for bodies below the multipart threshold. */
|
|
831
852
|
private uploadSingleShot;
|
|
832
853
|
/**
|
|
833
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
834
|
-
*
|
|
854
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
855
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
856
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
857
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
835
858
|
*/
|
|
836
859
|
private uploadMultipart;
|
|
860
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
861
|
+
private forwardProgress;
|
|
862
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
863
|
+
private failMultipart;
|
|
837
864
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
838
865
|
private normalizeSingleShotBody;
|
|
839
866
|
/** Assembles an `UploadResult`, omitting optional fields that are absent. */
|
|
@@ -850,10 +877,6 @@ declare class StorageService {
|
|
|
850
877
|
* @throws StorageException mapped from any AWS failure.
|
|
851
878
|
*/
|
|
852
879
|
list(options: ListOptions): Promise<ListResult>;
|
|
853
|
-
/** Maps a `ListObjectsV2` response onto the public `ListResult` shape. */
|
|
854
|
-
private toListResult;
|
|
855
|
-
/** Maps one `Contents` entry onto `ListedObject`, stripping the global prefix. */
|
|
856
|
-
private toListedObject;
|
|
857
880
|
/**
|
|
858
881
|
* Server-side copy of an object (no bytes flow through the app). Supports both
|
|
859
882
|
* same-bucket and cross-bucket copies and preserves the source metadata.
|
|
@@ -876,8 +899,6 @@ declare class StorageService {
|
|
|
876
899
|
* @returns The deleted keys and the per-key failures.
|
|
877
900
|
*/
|
|
878
901
|
deleteMany(keys: string[], options?: DeleteManyOptions): Promise<DeleteManyResult>;
|
|
879
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
880
|
-
private deleteChunk;
|
|
881
902
|
}
|
|
882
903
|
|
|
883
904
|
/**
|
|
@@ -1349,6 +1370,9 @@ declare const STORAGE_ERROR_CODES: {
|
|
|
1349
1370
|
readonly STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED";
|
|
1350
1371
|
readonly STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED";
|
|
1351
1372
|
readonly STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED";
|
|
1373
|
+
readonly STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND";
|
|
1374
|
+
readonly STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED";
|
|
1375
|
+
readonly STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH";
|
|
1352
1376
|
};
|
|
1353
1377
|
/** Union of every stable storage error code. */
|
|
1354
1378
|
type StorageErrorCode = (typeof STORAGE_ERROR_CODES)[keyof typeof STORAGE_ERROR_CODES];
|
package/dist/server/index.mjs
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { HttpStatus, Injectable, Inject, Global, Module, Logger, HttpException } from '@nestjs/common';
|
|
2
|
-
import { S3Client, GetBucketLifecycleConfigurationCommand, DeleteBucketLifecycleCommand, PutBucketLifecycleConfigurationCommand, HeadObjectCommand, GetObjectCommand, DeleteObjectCommand, PutObjectCommand, ListObjectsV2Command, CopyObjectCommand,
|
|
2
|
+
import { S3Client, GetBucketLifecycleConfigurationCommand, DeleteBucketLifecycleCommand, PutBucketLifecycleConfigurationCommand, HeadObjectCommand, GetObjectCommand, DeleteObjectCommand, HeadBucketCommand, PutObjectCommand, ListObjectsV2Command, CopyObjectCommand, CreateMultipartUploadCommand, UploadPartCommand, CompleteMultipartUploadCommand, AbortMultipartUploadCommand, DeleteObjectsCommand } from '@aws-sdk/client-s3';
|
|
3
3
|
import { Upload } from '@aws-sdk/lib-storage';
|
|
4
4
|
import { PassThrough, Readable } from 'stream';
|
|
5
5
|
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
|
|
@@ -48,7 +48,10 @@ var STORAGE_ERROR_CODES = {
|
|
|
48
48
|
STORAGE_LIFECYCLE_UNSUPPORTED: "STORAGE_LIFECYCLE_UNSUPPORTED",
|
|
49
49
|
STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED",
|
|
50
50
|
STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED",
|
|
51
|
-
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED"
|
|
51
|
+
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED",
|
|
52
|
+
STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND",
|
|
53
|
+
STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED",
|
|
54
|
+
STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH"
|
|
52
55
|
};
|
|
53
56
|
|
|
54
57
|
// src/server/errors/storage-error-messages.ts
|
|
@@ -75,7 +78,10 @@ var STORAGE_ERROR_MESSAGES = {
|
|
|
75
78
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_UNSUPPORTED]: "The storage provider does not support bucket lifecycle configuration",
|
|
76
79
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_REJECTED]: "The storage provider rejected the lifecycle configuration",
|
|
77
80
|
[STORAGE_ERROR_CODES.STORAGE_NOT_MODIFIED]: "The object has not changed since the supplied ETag",
|
|
78
|
-
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: "The object does not satisfy the supplied precondition"
|
|
81
|
+
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: "The object does not satisfy the supplied precondition",
|
|
82
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND]: "The bucket does not exist",
|
|
83
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED]: "The bucket cannot be accessed with these credentials (it may also not exist)",
|
|
84
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH]: "The bucket belongs to a different region than the configured one"
|
|
79
85
|
};
|
|
80
86
|
var STORAGE_ERROR_STATUS = {
|
|
81
87
|
[STORAGE_ERROR_CODES.STORAGE_NOT_CONFIGURED]: HttpStatus.SERVICE_UNAVAILABLE,
|
|
@@ -100,7 +106,10 @@ var STORAGE_ERROR_STATUS = {
|
|
|
100
106
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_UNSUPPORTED]: HttpStatus.NOT_IMPLEMENTED,
|
|
101
107
|
[STORAGE_ERROR_CODES.STORAGE_LIFECYCLE_REJECTED]: HttpStatus.UNPROCESSABLE_ENTITY,
|
|
102
108
|
[STORAGE_ERROR_CODES.STORAGE_NOT_MODIFIED]: HttpStatus.NOT_MODIFIED,
|
|
103
|
-
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: HttpStatus.PRECONDITION_FAILED
|
|
109
|
+
[STORAGE_ERROR_CODES.STORAGE_PRECONDITION_FAILED]: HttpStatus.PRECONDITION_FAILED,
|
|
110
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND]: HttpStatus.NOT_FOUND,
|
|
111
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED]: HttpStatus.FORBIDDEN,
|
|
112
|
+
[STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH]: HttpStatus.INTERNAL_SERVER_ERROR
|
|
104
113
|
};
|
|
105
114
|
|
|
106
115
|
// src/server/errors/storage-exception.ts
|
|
@@ -413,6 +422,13 @@ function mapAwsError(err, context) {
|
|
|
413
422
|
requestId,
|
|
414
423
|
awsMessage: e.message
|
|
415
424
|
};
|
|
425
|
+
if (awsCode === "NoSuchBucket") {
|
|
426
|
+
return new StorageException(
|
|
427
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND,
|
|
428
|
+
HttpStatus.NOT_FOUND,
|
|
429
|
+
details
|
|
430
|
+
);
|
|
431
|
+
}
|
|
416
432
|
if (e.name === "NotFound" || httpStatus === 404) {
|
|
417
433
|
return new StorageException(
|
|
418
434
|
STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND,
|
|
@@ -1081,11 +1097,127 @@ function mapMultipartError(err, context) {
|
|
|
1081
1097
|
idleMs: err.idleMs
|
|
1082
1098
|
});
|
|
1083
1099
|
}
|
|
1100
|
+
const bucketMissing = mapAwsError(err, context);
|
|
1101
|
+
if (bucketMissing.code === STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND) {
|
|
1102
|
+
return bucketMissing;
|
|
1103
|
+
}
|
|
1084
1104
|
return new StorageException(STORAGE_ERROR_CODES.STORAGE_MULTIPART_ABORTED, void 0, {
|
|
1085
1105
|
...context,
|
|
1086
1106
|
awsMessage: describeError(err)
|
|
1087
1107
|
});
|
|
1088
1108
|
}
|
|
1109
|
+
function knownStatus(error) {
|
|
1110
|
+
const name = HttpStatus[error.getStatus()];
|
|
1111
|
+
return name === void 0 ? void 0 : HttpStatus[name];
|
|
1112
|
+
}
|
|
1113
|
+
function withCleanupFailure(scanError, key, cleanupError) {
|
|
1114
|
+
if (!(scanError instanceof StorageException)) {
|
|
1115
|
+
return scanError;
|
|
1116
|
+
}
|
|
1117
|
+
const body = scanError.getResponse();
|
|
1118
|
+
return new StorageException(scanError.code, knownStatus(scanError), {
|
|
1119
|
+
...body.error?.details ?? {},
|
|
1120
|
+
key,
|
|
1121
|
+
cleanup: "failed",
|
|
1122
|
+
cleanupError: describeError(cleanupError)
|
|
1123
|
+
});
|
|
1124
|
+
}
|
|
1125
|
+
async function closeMultipartUpload(client, bucket, key, uploadId) {
|
|
1126
|
+
if (uploadId === void 0) {
|
|
1127
|
+
return {};
|
|
1128
|
+
}
|
|
1129
|
+
try {
|
|
1130
|
+
await client.send(
|
|
1131
|
+
new AbortMultipartUploadCommand({ Bucket: bucket, Key: key, UploadId: uploadId })
|
|
1132
|
+
);
|
|
1133
|
+
return { abort: "completed" };
|
|
1134
|
+
} catch (err) {
|
|
1135
|
+
if (err?.name === "NoSuchUpload") {
|
|
1136
|
+
return { abort: "completed" };
|
|
1137
|
+
}
|
|
1138
|
+
return { abort: "failed", abortError: describeError(err) };
|
|
1139
|
+
}
|
|
1140
|
+
}
|
|
1141
|
+
async function deleteObjectsChunk(client, bucket, chunk, stripPrefix) {
|
|
1142
|
+
const deleted = [];
|
|
1143
|
+
const failed = [];
|
|
1144
|
+
try {
|
|
1145
|
+
const response = await client.send(
|
|
1146
|
+
new DeleteObjectsCommand({
|
|
1147
|
+
Bucket: bucket,
|
|
1148
|
+
// `Quiet: false` returns both successes and per-key errors.
|
|
1149
|
+
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1150
|
+
})
|
|
1151
|
+
);
|
|
1152
|
+
for (const ok of response.Deleted ?? []) {
|
|
1153
|
+
if (ok.Key !== void 0) {
|
|
1154
|
+
deleted.push(stripPrefix(ok.Key));
|
|
1155
|
+
}
|
|
1156
|
+
}
|
|
1157
|
+
for (const failure of response.Errors ?? []) {
|
|
1158
|
+
if (failure.Key !== void 0) {
|
|
1159
|
+
failed.push({
|
|
1160
|
+
key: stripPrefix(failure.Key),
|
|
1161
|
+
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1162
|
+
});
|
|
1163
|
+
}
|
|
1164
|
+
}
|
|
1165
|
+
} catch (err) {
|
|
1166
|
+
const message = describeError(err);
|
|
1167
|
+
for (const key of chunk) {
|
|
1168
|
+
failed.push({ key: stripPrefix(key), error: message });
|
|
1169
|
+
}
|
|
1170
|
+
}
|
|
1171
|
+
return { deleted, failed };
|
|
1172
|
+
}
|
|
1173
|
+
function mapBucketCheckError(err, context) {
|
|
1174
|
+
const failure = err ?? {};
|
|
1175
|
+
const status = failure.$metadata?.httpStatusCode;
|
|
1176
|
+
const details = { ...context, httpStatus: status, awsCode: failure.name };
|
|
1177
|
+
if (status === 301 || failure.name === "PermanentRedirect") {
|
|
1178
|
+
return new StorageException(
|
|
1179
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_REGION_MISMATCH,
|
|
1180
|
+
void 0,
|
|
1181
|
+
details
|
|
1182
|
+
);
|
|
1183
|
+
}
|
|
1184
|
+
if (status === 403) {
|
|
1185
|
+
return new StorageException(
|
|
1186
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_ACCESS_DENIED,
|
|
1187
|
+
void 0,
|
|
1188
|
+
details
|
|
1189
|
+
);
|
|
1190
|
+
}
|
|
1191
|
+
if (status === 404) {
|
|
1192
|
+
return new StorageException(
|
|
1193
|
+
STORAGE_ERROR_CODES.STORAGE_BUCKET_NOT_FOUND,
|
|
1194
|
+
HttpStatus.NOT_FOUND,
|
|
1195
|
+
details
|
|
1196
|
+
);
|
|
1197
|
+
}
|
|
1198
|
+
return mapAwsError(err, context);
|
|
1199
|
+
}
|
|
1200
|
+
|
|
1201
|
+
// src/server/utils/list-result.ts
|
|
1202
|
+
function toListedObject(obj, stripPrefix) {
|
|
1203
|
+
return {
|
|
1204
|
+
key: stripPrefix(obj.Key ?? ""),
|
|
1205
|
+
size: obj.Size ?? 0,
|
|
1206
|
+
etag: obj.ETag ?? "",
|
|
1207
|
+
lastModified: obj.LastModified ?? /* @__PURE__ */ new Date(0),
|
|
1208
|
+
...obj.StorageClass !== void 0 ? { storageClass: obj.StorageClass } : {}
|
|
1209
|
+
};
|
|
1210
|
+
}
|
|
1211
|
+
function toListResult(response, stripPrefix) {
|
|
1212
|
+
const objects = (response.Contents ?? []).map((obj) => toListedObject(obj, stripPrefix));
|
|
1213
|
+
const commonPrefixes = (response.CommonPrefixes ?? []).map((cp) => cp.Prefix ?? "").filter((cp) => cp.length > 0).map((cp) => stripPrefix(cp));
|
|
1214
|
+
return {
|
|
1215
|
+
objects,
|
|
1216
|
+
commonPrefixes,
|
|
1217
|
+
isTruncated: response.IsTruncated ?? false,
|
|
1218
|
+
...response.NextContinuationToken !== void 0 ? { nextContinuationToken: response.NextContinuationToken } : {}
|
|
1219
|
+
};
|
|
1220
|
+
}
|
|
1089
1221
|
|
|
1090
1222
|
// src/server/utils/header-utils.ts
|
|
1091
1223
|
function buildContentDisposition(perCall, defaultValue) {
|
|
@@ -1279,12 +1411,16 @@ var StorageService = class {
|
|
|
1279
1411
|
...options.size !== void 0 ? { size: options.size } : {}
|
|
1280
1412
|
});
|
|
1281
1413
|
} catch (scanErr) {
|
|
1282
|
-
|
|
1414
|
+
const key = this.keyResolver.stripPrefix(finalKey);
|
|
1415
|
+
try {
|
|
1416
|
+
await this.delete(key, { bucket });
|
|
1417
|
+
} catch (deleteErr) {
|
|
1283
1418
|
this.logger.error(
|
|
1284
|
-
`
|
|
1419
|
+
`STORAGE_SCAN_CLEANUP_FAILED: the rejected object is still stored: ${key}`,
|
|
1285
1420
|
describeError(deleteErr)
|
|
1286
1421
|
);
|
|
1287
|
-
|
|
1422
|
+
throw withCleanupFailure(scanErr, key, deleteErr);
|
|
1423
|
+
}
|
|
1288
1424
|
throw scanErr;
|
|
1289
1425
|
}
|
|
1290
1426
|
}
|
|
@@ -1405,11 +1541,31 @@ var StorageService = class {
|
|
|
1405
1541
|
await this.s3Provider.getClient().send(new DeleteObjectCommand({ Bucket: bucket, Key: finalKey }));
|
|
1406
1542
|
} catch (err) {
|
|
1407
1543
|
const mapped = mapAwsError(err, { key: finalKey, bucket, op: "delete" });
|
|
1408
|
-
if (mapped.code
|
|
1409
|
-
|
|
1410
|
-
return;
|
|
1544
|
+
if (mapped.code !== STORAGE_ERROR_CODES.STORAGE_OBJECT_NOT_FOUND) {
|
|
1545
|
+
throw mapped;
|
|
1411
1546
|
}
|
|
1412
|
-
|
|
1547
|
+
this.logger.warn(`delete() \u2014 key not found (idempotent no-op): ${finalKey}`);
|
|
1548
|
+
}
|
|
1549
|
+
this.idempotencyCache.invalidate(bucket, this.keyResolver.stripPrefix(finalKey));
|
|
1550
|
+
}
|
|
1551
|
+
/**
|
|
1552
|
+
* Probes the bucket with one `HeadBucket`, for a readiness check: it resolves when the bucket
|
|
1553
|
+
* is reachable with these credentials and otherwise rejects with a typed code. Nothing in the
|
|
1554
|
+
* module calls it on its own, so booting costs no request. Needs `s3:ListBucket`.
|
|
1555
|
+
*
|
|
1556
|
+
* @param options - Optional per-call bucket override.
|
|
1557
|
+
* @throws StorageException `STORAGE_BUCKET_NOT_FOUND` (404), `STORAGE_BUCKET_ACCESS_DENIED` (403,
|
|
1558
|
+
* which a provider also answers for a bucket the caller cannot see), `STORAGE_BUCKET_REGION_MISMATCH`
|
|
1559
|
+
* (301), `STORAGE_TIMEOUT`, or `STORAGE_PROVIDER_ERROR`; `STORAGE_NOT_CONFIGURED` and
|
|
1560
|
+
* `STORAGE_BUCKET_UNDEFINED` before any request.
|
|
1561
|
+
*/
|
|
1562
|
+
async checkBucket(options) {
|
|
1563
|
+
this.assertConfigured();
|
|
1564
|
+
const bucket = this.resolveBucket(options?.bucket);
|
|
1565
|
+
try {
|
|
1566
|
+
await this.s3Provider.getClient().send(new HeadBucketCommand({ Bucket: bucket }));
|
|
1567
|
+
} catch (err) {
|
|
1568
|
+
throw mapBucketCheckError(err, { bucket, op: "checkBucket" });
|
|
1413
1569
|
}
|
|
1414
1570
|
}
|
|
1415
1571
|
/** Throws when the S3 client was never built (missing credentials). */
|
|
@@ -1500,8 +1656,10 @@ var StorageService = class {
|
|
|
1500
1656
|
}
|
|
1501
1657
|
}
|
|
1502
1658
|
/**
|
|
1503
|
-
* Multipart upload via `@aws-sdk/lib-storage`.
|
|
1504
|
-
*
|
|
1659
|
+
* Multipart upload via `@aws-sdk/lib-storage`. The SDK is told to leave the parts
|
|
1660
|
+
* (`leavePartsOnError: true`) because its own abort runs inside its failure path: a failing
|
|
1661
|
+
* abort replaced the failure that caused it, and a failed `CompleteMultipartUpload` was never
|
|
1662
|
+
* aborted. This method aborts the upload itself and reports the outcome in the error details.
|
|
1505
1663
|
*/
|
|
1506
1664
|
async uploadMultipart(options, finalKey, bucket) {
|
|
1507
1665
|
const params = {
|
|
@@ -1523,14 +1681,9 @@ var StorageService = class {
|
|
|
1523
1681
|
params,
|
|
1524
1682
|
queueSize: this.options.multipart.queueSize,
|
|
1525
1683
|
partSize: this.options.multipart.partSizeBytes,
|
|
1526
|
-
leavePartsOnError:
|
|
1684
|
+
leavePartsOnError: true
|
|
1527
1685
|
});
|
|
1528
|
-
|
|
1529
|
-
const onProgress = options.onProgress;
|
|
1530
|
-
uploader.on("httpUploadProgress", (event) => {
|
|
1531
|
-
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1532
|
-
});
|
|
1533
|
-
}
|
|
1686
|
+
this.forwardProgress(uploader, options.onProgress);
|
|
1534
1687
|
try {
|
|
1535
1688
|
const response = await uploader.done();
|
|
1536
1689
|
return this.buildUploadResult({
|
|
@@ -1543,9 +1696,26 @@ var StorageService = class {
|
|
|
1543
1696
|
multipart: true
|
|
1544
1697
|
});
|
|
1545
1698
|
} catch (err) {
|
|
1546
|
-
|
|
1699
|
+
return this.failMultipart(err, finalKey, bucket, uploader.uploadId);
|
|
1547
1700
|
}
|
|
1548
1701
|
}
|
|
1702
|
+
/** Relays the uploader's progress events to the caller's callback, when there is one. */
|
|
1703
|
+
forwardProgress(uploader, onProgress) {
|
|
1704
|
+
if (!onProgress) return;
|
|
1705
|
+
uploader.on("httpUploadProgress", (event) => {
|
|
1706
|
+
this.emitProgress(onProgress, event.loaded, event.total, event.part);
|
|
1707
|
+
});
|
|
1708
|
+
}
|
|
1709
|
+
/** Aborts the multipart upload that failed, then throws the original failure with the outcome. */
|
|
1710
|
+
async failMultipart(err, finalKey, bucket, uploadId) {
|
|
1711
|
+
const abort = await closeMultipartUpload(
|
|
1712
|
+
this.s3Provider.getClient(),
|
|
1713
|
+
bucket,
|
|
1714
|
+
finalKey,
|
|
1715
|
+
uploadId
|
|
1716
|
+
);
|
|
1717
|
+
throw mapMultipartError(err, { key: finalKey, bucket, ...abort });
|
|
1718
|
+
}
|
|
1549
1719
|
/** Converts a `Uint8Array` body to a `Buffer` for SDK safety; passes others through. */
|
|
1550
1720
|
normalizeSingleShotBody(body) {
|
|
1551
1721
|
if (isBufferLike(body)) {
|
|
@@ -1599,32 +1769,11 @@ var StorageService = class {
|
|
|
1599
1769
|
Delimiter: options.delimiter
|
|
1600
1770
|
})
|
|
1601
1771
|
);
|
|
1602
|
-
return
|
|
1772
|
+
return toListResult(response, (key) => this.keyResolver.stripPrefix(key));
|
|
1603
1773
|
} catch (err) {
|
|
1604
1774
|
throw mapAwsError(err, { bucket, prefix, op: "list" });
|
|
1605
1775
|
}
|
|
1606
1776
|
}
|
|
1607
|
-
/** Maps a `ListObjectsV2` response onto the public `ListResult` shape. */
|
|
1608
|
-
toListResult(response) {
|
|
1609
|
-
const objects = (response.Contents ?? []).map((obj) => this.toListedObject(obj));
|
|
1610
|
-
const commonPrefixes = (response.CommonPrefixes ?? []).map((cp) => cp.Prefix ?? "").filter((cp) => cp.length > 0).map((cp) => this.keyResolver.stripPrefix(cp));
|
|
1611
|
-
return {
|
|
1612
|
-
objects,
|
|
1613
|
-
commonPrefixes,
|
|
1614
|
-
isTruncated: response.IsTruncated ?? false,
|
|
1615
|
-
...response.NextContinuationToken !== void 0 ? { nextContinuationToken: response.NextContinuationToken } : {}
|
|
1616
|
-
};
|
|
1617
|
-
}
|
|
1618
|
-
/** Maps one `Contents` entry onto `ListedObject`, stripping the global prefix. */
|
|
1619
|
-
toListedObject(obj) {
|
|
1620
|
-
return {
|
|
1621
|
-
key: this.keyResolver.stripPrefix(obj.Key ?? ""),
|
|
1622
|
-
size: obj.Size ?? 0,
|
|
1623
|
-
etag: obj.ETag ?? "",
|
|
1624
|
-
lastModified: obj.LastModified ?? /* @__PURE__ */ new Date(0),
|
|
1625
|
-
...obj.StorageClass !== void 0 ? { storageClass: obj.StorageClass } : {}
|
|
1626
|
-
};
|
|
1627
|
-
}
|
|
1628
1777
|
/**
|
|
1629
1778
|
* Server-side copy of an object (no bytes flow through the app). Supports both
|
|
1630
1779
|
* same-bucket and cross-bucket copies and preserves the source metadata.
|
|
@@ -1679,42 +1828,17 @@ var StorageService = class {
|
|
|
1679
1828
|
const failed = [];
|
|
1680
1829
|
for (let i = 0; i < normalized.length; i += DELETE_BATCH_LIMIT) {
|
|
1681
1830
|
const chunk = normalized.slice(i, i + DELETE_BATCH_LIMIT);
|
|
1682
|
-
const outcome = await
|
|
1831
|
+
const outcome = await deleteObjectsChunk(
|
|
1832
|
+
this.s3Provider.getClient(),
|
|
1833
|
+
bucket,
|
|
1834
|
+
chunk,
|
|
1835
|
+
(key) => this.keyResolver.stripPrefix(key)
|
|
1836
|
+
);
|
|
1683
1837
|
deleted.push(...outcome.deleted);
|
|
1684
1838
|
failed.push(...outcome.failed);
|
|
1685
1839
|
}
|
|
1686
|
-
|
|
1687
|
-
|
|
1688
|
-
/** Deletes one chunk (≤ 1000 keys); a whole-chunk failure fails every key. */
|
|
1689
|
-
async deleteChunk(bucket, chunk) {
|
|
1690
|
-
const deleted = [];
|
|
1691
|
-
const failed = [];
|
|
1692
|
-
try {
|
|
1693
|
-
const response = await this.s3Provider.getClient().send(
|
|
1694
|
-
new DeleteObjectsCommand({
|
|
1695
|
-
Bucket: bucket,
|
|
1696
|
-
// `Quiet: false` returns both successes and per-key errors.
|
|
1697
|
-
Delete: { Objects: chunk.map((key) => ({ Key: key })), Quiet: false }
|
|
1698
|
-
})
|
|
1699
|
-
);
|
|
1700
|
-
for (const ok of response.Deleted ?? []) {
|
|
1701
|
-
if (ok.Key !== void 0) {
|
|
1702
|
-
deleted.push(this.keyResolver.stripPrefix(ok.Key));
|
|
1703
|
-
}
|
|
1704
|
-
}
|
|
1705
|
-
for (const failure of response.Errors ?? []) {
|
|
1706
|
-
if (failure.Key !== void 0) {
|
|
1707
|
-
failed.push({
|
|
1708
|
-
key: this.keyResolver.stripPrefix(failure.Key),
|
|
1709
|
-
error: `${failure.Code ?? "Unknown"}: ${failure.Message ?? ""}`
|
|
1710
|
-
});
|
|
1711
|
-
}
|
|
1712
|
-
}
|
|
1713
|
-
} catch (err) {
|
|
1714
|
-
const message = describeError(err);
|
|
1715
|
-
for (const key of chunk) {
|
|
1716
|
-
failed.push({ key: this.keyResolver.stripPrefix(key), error: message });
|
|
1717
|
-
}
|
|
1840
|
+
for (const key of deleted) {
|
|
1841
|
+
this.idempotencyCache.invalidate(bucket, key);
|
|
1718
1842
|
}
|
|
1719
1843
|
return { deleted, failed };
|
|
1720
1844
|
}
|
|
@@ -2015,6 +2139,24 @@ var IdempotencyCache = class {
|
|
|
2015
2139
|
}
|
|
2016
2140
|
}
|
|
2017
2141
|
}
|
|
2142
|
+
/**
|
|
2143
|
+
* Drops every entry whose cached result is this object, so a repeated upload after the object
|
|
2144
|
+
* was deleted runs instead of being answered with a success that no longer holds.
|
|
2145
|
+
*
|
|
2146
|
+
* @param bucket - The bucket of the deleted object.
|
|
2147
|
+
* @param key - The object's key, relative to `keyPrefix`, as `UploadResult.key` carries it.
|
|
2148
|
+
* @returns How many entries were dropped.
|
|
2149
|
+
*/
|
|
2150
|
+
invalidate(bucket, key) {
|
|
2151
|
+
let dropped = 0;
|
|
2152
|
+
for (const [cacheKey, entry] of this.entries) {
|
|
2153
|
+
if (entry.value.bucket === bucket && entry.value.key === key) {
|
|
2154
|
+
this.entries.delete(cacheKey);
|
|
2155
|
+
dropped += 1;
|
|
2156
|
+
}
|
|
2157
|
+
}
|
|
2158
|
+
return dropped;
|
|
2159
|
+
}
|
|
2018
2160
|
/**
|
|
2019
2161
|
* Reports the current number of cached entries.
|
|
2020
2162
|
*
|
package/dist/shared/index.cjs
CHANGED
|
@@ -26,7 +26,10 @@ var STORAGE_ERROR_CODES = {
|
|
|
26
26
|
STORAGE_LIFECYCLE_UNSUPPORTED: "STORAGE_LIFECYCLE_UNSUPPORTED",
|
|
27
27
|
STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED",
|
|
28
28
|
STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED",
|
|
29
|
-
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED"
|
|
29
|
+
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED",
|
|
30
|
+
STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND",
|
|
31
|
+
STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED",
|
|
32
|
+
STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH"
|
|
30
33
|
};
|
|
31
34
|
|
|
32
35
|
// src/shared/constants/mime-types.constants.ts
|
package/dist/shared/index.d.cts
CHANGED
|
@@ -95,6 +95,9 @@ declare const STORAGE_ERROR_CODES: {
|
|
|
95
95
|
readonly STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED";
|
|
96
96
|
readonly STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED";
|
|
97
97
|
readonly STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED";
|
|
98
|
+
readonly STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND";
|
|
99
|
+
readonly STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED";
|
|
100
|
+
readonly STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH";
|
|
98
101
|
};
|
|
99
102
|
/** Union of every stable storage error code. */
|
|
100
103
|
type StorageErrorCode = (typeof STORAGE_ERROR_CODES)[keyof typeof STORAGE_ERROR_CODES];
|
package/dist/shared/index.d.ts
CHANGED
|
@@ -95,6 +95,9 @@ declare const STORAGE_ERROR_CODES: {
|
|
|
95
95
|
readonly STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED";
|
|
96
96
|
readonly STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED";
|
|
97
97
|
readonly STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED";
|
|
98
|
+
readonly STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND";
|
|
99
|
+
readonly STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED";
|
|
100
|
+
readonly STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH";
|
|
98
101
|
};
|
|
99
102
|
/** Union of every stable storage error code. */
|
|
100
103
|
type StorageErrorCode = (typeof STORAGE_ERROR_CODES)[keyof typeof STORAGE_ERROR_CODES];
|
package/dist/shared/index.mjs
CHANGED
|
@@ -24,7 +24,10 @@ var STORAGE_ERROR_CODES = {
|
|
|
24
24
|
STORAGE_LIFECYCLE_UNSUPPORTED: "STORAGE_LIFECYCLE_UNSUPPORTED",
|
|
25
25
|
STORAGE_LIFECYCLE_REJECTED: "STORAGE_LIFECYCLE_REJECTED",
|
|
26
26
|
STORAGE_NOT_MODIFIED: "STORAGE_NOT_MODIFIED",
|
|
27
|
-
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED"
|
|
27
|
+
STORAGE_PRECONDITION_FAILED: "STORAGE_PRECONDITION_FAILED",
|
|
28
|
+
STORAGE_BUCKET_NOT_FOUND: "STORAGE_BUCKET_NOT_FOUND",
|
|
29
|
+
STORAGE_BUCKET_ACCESS_DENIED: "STORAGE_BUCKET_ACCESS_DENIED",
|
|
30
|
+
STORAGE_BUCKET_REGION_MISMATCH: "STORAGE_BUCKET_REGION_MISMATCH"
|
|
28
31
|
};
|
|
29
32
|
|
|
30
33
|
// src/shared/constants/mime-types.constants.ts
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-storage",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.5",
|
|
4
4
|
"description": "Provider-agnostic S3-compatible object storage for NestJS. Works with AWS S3, DigitalOcean Spaces, Cloudflare R2, Backblaze B2, MinIO, Wasabi.",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
},
|
|
46
46
|
"scripts": {
|
|
47
47
|
"build": "pnpm clean && tsup",
|
|
48
|
+
"check:api": "node scripts/check-api-surface.mjs",
|
|
48
49
|
"check:exports": "attw --pack . --profile strict",
|
|
49
50
|
"check:mutants": "node scripts/check-mutation-directives.mjs",
|
|
50
51
|
"check:published": "node scripts/check-published-surface.mjs",
|