@bymax-one/nest-storage 1.1.10 → 1.1.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -1
- package/README.md +259 -350
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -13,6 +13,22 @@ note on each entry is the compatibility contract — read it, not the number.
|
|
|
13
13
|
|
|
14
14
|
## [Unreleased]
|
|
15
15
|
|
|
16
|
+
## [1.1.11] - 2026-11-13
|
|
17
|
+
|
|
18
|
+
### Changed
|
|
19
|
+
|
|
20
|
+
- **Development and CI tooling updated; the published code does not change.** The AWS SDK
|
|
21
|
+
development dependencies and the pinned GitHub Actions are bumped to newer patch releases.
|
|
22
|
+
No file under `src/` changed.
|
|
23
|
+
|
|
24
|
+
### Documentation
|
|
25
|
+
|
|
26
|
+
- **The README is shorter and corrected.** The API table and the export list now match the
|
|
27
|
+
published surface, text that repeated itself or did not belong is removed, and the security
|
|
28
|
+
controls are restored as one compact table with a row per control.
|
|
29
|
+
|
|
30
|
+
**Apply to a derived backend:** nothing to change; this release carries no runtime change.
|
|
31
|
+
|
|
16
32
|
## [1.1.10] - 2026-11-09
|
|
17
33
|
|
|
18
34
|
### Fixed
|
|
@@ -667,4 +683,5 @@ have regressed from. They are kept because the reasoning is worth having.
|
|
|
667
683
|
[1.1.8]: https://github.com/bymaxone/nest-storage/compare/v1.1.7...v1.1.8
|
|
668
684
|
[1.1.9]: https://github.com/bymaxone/nest-storage/compare/v1.1.8...v1.1.9
|
|
669
685
|
[1.1.10]: https://github.com/bymaxone/nest-storage/compare/v1.1.9...v1.1.10
|
|
670
|
-
[
|
|
686
|
+
[1.1.11]: https://github.com/bymaxone/nest-storage/compare/v1.1.10...v1.1.11
|
|
687
|
+
[Unreleased]: https://github.com/bymaxone/nest-storage/compare/v1.1.11...HEAD
|
package/README.md
CHANGED
|
@@ -46,74 +46,41 @@ minimal.
|
|
|
46
46
|
|
|
47
47
|
### Why nest-storage?
|
|
48
48
|
|
|
49
|
-
- **One engine, six providers.** `providerRecipes`
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
`KeyResolverService` before it reaches S3 — `..` segments, a leading `/`, null bytes and empty
|
|
55
|
-
keys are refused, and only then is `keyPrefix` prepended. A tenant cannot climb out of its
|
|
56
|
-
prefix by naming a key.
|
|
49
|
+
- **One engine, six providers.** `providerRecipes` only produce module options (endpoint, region,
|
|
50
|
+
addressing style, public URLs and, for the non-AWS providers, the checksum opt-out). There is no per-provider code
|
|
51
|
+
path, so switching provider is a configuration change.
|
|
52
|
+
- **One key chokepoint.** Every caller key passes through `KeyResolverService` before it reaches S3,
|
|
53
|
+
and `keyPrefix` is prepended only after it ([Security Model](#-security-model)).
|
|
57
54
|
- **Uploads are refused before they are stored.** MIME allowlist and size limit first, then any
|
|
58
|
-
`IUploadValidator` you register, then the `IFileScanner
|
|
59
|
-
or after it.
|
|
60
|
-
- **Presigned URLs stay inside a configured ceiling.** A requested TTL above
|
|
61
|
-
`signedUrls.maxTtlSeconds` (604 800 s, the SigV4 limit, by default) is clamped locally, rather
|
|
62
|
-
than handed to a signature the provider rejects at use time.
|
|
55
|
+
`IUploadValidator` you register, then the `IFileScanner`, before or after the upload.
|
|
63
56
|
|
|
64
57
|
---
|
|
65
58
|
|
|
66
59
|
## 🔥 Features
|
|
67
60
|
|
|
68
|
-
### ⬆️ Uploading
|
|
69
|
-
|
|
70
61
|
- ✅ **Multipart upload** — `@aws-sdk/lib-storage` with automatic abort on failure
|
|
71
62
|
(`leavePartsOnError: false`), progress events, and a configurable threshold and part size
|
|
72
|
-
- ✅ **Content validation** — MIME allowlist with wildcards (`image
|
|
73
|
-
|
|
74
|
-
- ✅ **
|
|
75
|
-
|
|
76
|
-
- ✅ **
|
|
77
|
-
|
|
78
|
-
- ✅ **
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
- ✅ **
|
|
84
|
-
|
|
85
|
-
upload directly without the bytes passing through your process
|
|
86
|
-
- ✅ **TTL clamping** — a requested TTL above `signedUrls.maxTtlSeconds` (default 604 800 s, the
|
|
87
|
-
7-day SigV4 limit) is clamped locally, not discovered when the provider rejects the signature
|
|
88
|
-
- ✅ **Public URL composition** — `getPublicUrl` builds from `cdnBaseUrl` or `publicBaseUrl` for
|
|
89
|
-
buckets that are public by policy; it composes, it does not sign
|
|
90
|
-
|
|
91
|
-
### 🗄️ Objects & Lifecycle
|
|
92
|
-
|
|
93
|
-
- ✅ **Streaming download** — `download` returns a stream; the body is never buffered unless you
|
|
94
|
-
ask for `downloadBuffer`
|
|
95
|
-
- ✅ **Metadata without transfer** — `head` and `exists` answer from object metadata alone
|
|
96
|
-
- ✅ **Paginated listing** — `list` over continuation tokens, with the prefix stripped back off
|
|
97
|
-
the keys it returns
|
|
98
|
-
- ✅ **Server-side copy** — `copy` moves bytes inside the provider; they never reach this process
|
|
99
|
-
- ✅ **Bulk delete with partial results** — `deleteMany` reports what failed instead of throwing
|
|
100
|
-
away the successes
|
|
101
|
-
- ✅ **Bucket lifecycle rules** — read the rules, add or replace an expiration rule by id, remove
|
|
102
|
-
one, and leave every other rule exactly as the provider returned it
|
|
103
|
-
|
|
104
|
-
### 🧩 Developer Experience
|
|
105
|
-
|
|
63
|
+
- ✅ **Content validation** — MIME allowlist with wildcards (`image/*`) and a size limit, then any
|
|
64
|
+
number of `IUploadValidator`s, all refused before a byte is stored
|
|
65
|
+
- ✅ **Virus scanning hook** — `IFileScanner` pre-upload or post-upload, for ClamAV, AWS Macie or
|
|
66
|
+
anything that answers `'clean' | 'infected' | 'unknown'`
|
|
67
|
+
- ✅ **Idempotent retries** — a bounded in-memory LRU with a TTL returns the cached
|
|
68
|
+
`UploadResult` for a repeated `idempotencyKey`
|
|
69
|
+
- ✅ **Presigned GET / PUT / multipart** — the PUT signs the `Content-Type` (and ACL and metadata, when set) as required headers, so a PUT that sends another type is refused and stores nothing; multipart gets one signed URL per part, so a browser uploads directly
|
|
70
|
+
- ✅ **Streaming download, metadata without transfer, paginated listing, server-side copy, bulk delete**
|
|
71
|
+
— `deleteMany` reports what failed instead of throwing away the successes
|
|
72
|
+
- ✅ **Public URL composition** — `getPublicUrl` builds from `cdnBaseUrl` or `publicBaseUrl`; it
|
|
73
|
+
composes, it does not sign
|
|
74
|
+
- ✅ **Bucket lifecycle rules** — read the rules, add or replace an expiration or a multipart-abort
|
|
75
|
+
rule by id, remove one, and leave every other rule as the provider returned it
|
|
106
76
|
- ✅ **Six provider recipes** — `awsS3`, `cloudflareR2`, `backblazeB2`, `digitalOceanSpaces`,
|
|
107
77
|
`minio`, `wasabi`
|
|
108
|
-
- ✅ **Zero runtime dependencies** — `@aws-sdk/*` and `@nestjs/*` all arrive as peer
|
|
109
|
-
|
|
110
|
-
- ✅ **Two subpaths** — the server surface, and `./shared` carrying types and the error catalog
|
|
111
|
-
that pull in nothing at all
|
|
78
|
+
- ✅ **Zero runtime dependencies** — `@aws-sdk/*` and `@nestjs/*` all arrive as peer dependencies,
|
|
79
|
+
so you pin the versions
|
|
112
80
|
- ✅ **Escape hatch** — `BYMAX_STORAGE_S3_CLIENT` injects the raw `S3Client` for the
|
|
113
|
-
provider-specific operations this surface
|
|
114
|
-
- ✅ **Key prefix** — `keyPrefix` is prepended after normalization
|
|
115
|
-
keys the library returns are relative to it
|
|
116
|
-
- ✅ **Server-side encryption** — AES256 or `aws:kms`, globally or per upload
|
|
81
|
+
provider-specific operations this surface does not wrap
|
|
82
|
+
- ✅ **Key prefix and server-side encryption** — `keyPrefix` is prepended after normalization and the
|
|
83
|
+
keys the library returns are relative to it; AES256 or `aws:kms`, globally or per upload
|
|
117
84
|
- ✅ **Typed end to end** — TypeScript `strict` with `exactOptionalPropertyTypes` and
|
|
118
85
|
`noUncheckedIndexedAccess`; zero `any`
|
|
119
86
|
|
|
@@ -126,11 +93,7 @@ minimal.
|
|
|
126
93
|
| `.` | Server runtime: `BymaxStorageModule`, `StorageService`, `SignedUrlService`, `BucketLifecycleService`, `providerRecipes`, DI tokens, interfaces, `StorageException`, `NoOpUploadValidator`, `NoOpFileScanner` |
|
|
127
94
|
| `./shared` | Framework-free types (`UploadResult`, `ObjectMetadata`, `ListedObject`, `SignedUrlResult`, `StorageErrorResponse`) + `STORAGE_ERROR_CODES` + `StorageErrorCode` + the default MIME whitelists and the TTL and multipart default constants |
|
|
128
95
|
|
|
129
|
-
|
|
130
|
-
@bymax-one/nest-storage (server — NestJS + @aws-sdk/client-s3)
|
|
131
|
-
│
|
|
132
|
-
└── re-exports ──▶ @bymax-one/nest-storage/shared (zero dependencies)
|
|
133
|
-
```
|
|
96
|
+
The root re-exports some `./shared` names for convenience; import the rest from the subpath.
|
|
134
97
|
|
|
135
98
|
Both subpaths ship ESM **and** CommonJS with declarations for each format, so a
|
|
136
99
|
`require()` consumer receives CommonJS declarations rather than ESM ones. The
|
|
@@ -145,12 +108,6 @@ Both subpaths ship ESM **and** CommonJS with declarations for each format, so a
|
|
|
145
108
|
|
|
146
109
|
---
|
|
147
110
|
|
|
148
|
-
> [!TIP]
|
|
149
|
-
> A reference application lives in
|
|
150
|
-
> [`bymaxone/nest-storage-example`](https://github.com/bymaxone/nest-storage-example).
|
|
151
|
-
|
|
152
|
-
---
|
|
153
|
-
|
|
154
111
|
## 🚀 Quick Start
|
|
155
112
|
|
|
156
113
|
### 1 — AWS S3
|
|
@@ -247,7 +204,7 @@ BymaxStorageModule.forRoot({
|
|
|
247
204
|
|
|
248
205
|
## ⚙️ Configuration
|
|
249
206
|
|
|
250
|
-
|
|
207
|
+
Options accepted by `forRoot` and `forRootAsync`:
|
|
251
208
|
|
|
252
209
|
| Option | Type | Default | Notes |
|
|
253
210
|
| ------------------------------ | ------------------------------------------------- | ----------------- | ---------------------------------------------- |
|
|
@@ -270,7 +227,7 @@ The full configuration reference lives in `docs/technical_specification.md` §4.
|
|
|
270
227
|
| `uploadIdleTimeoutMs` | `number` | `30000` | Longest silence from a stream body during a multipart upload; follows `requestTimeoutMs` unless set, `0` disables; see below the table |
|
|
271
228
|
| `signedUrls.defaultGetTtlSeconds` | `number` | `300` | Default TTL for `getDownloadUrl` |
|
|
272
229
|
| `signedUrls.defaultPutTtlSeconds` | `number` | `300` | Default TTL for `getUploadUrl` |
|
|
273
|
-
| `signedUrls.maxTtlSeconds` | `number` | `604800` | Per-request TTLs above this are clamped; a value
|
|
230
|
+
| `signedUrls.maxTtlSeconds` | `number` | `604800` | Per-request TTLs above this are clamped; a larger value is reduced to 604800 (the 7-day SigV4 limit) at init |
|
|
274
231
|
| `multipart.thresholdBytes` | `number` | `5 MiB` | Switch to multipart at or above this size |
|
|
275
232
|
| `multipart.partSizeBytes` | `number` | `5 MiB` | Part size (S3 minimum: 5 MiB) |
|
|
276
233
|
| `multipart.queueSize` | `number` | `4` | Concurrent parts |
|
|
@@ -281,7 +238,8 @@ The full configuration reference lives in `docs/technical_specification.md` §4.
|
|
|
281
238
|
|
|
282
239
|
**`requestTimeoutMs`** is passed to the S3 client's request handler as `requestTimeout` with `throwOnRequestTimeout` ([AWS SDK for JavaScript v3: `requestHandler: { requestTimeout: 3_000, ... }`][sdk-request-timeout]). It bounds one attempt, from sending the request until the response headers arrive, and an upload body counts toward it, so a large single-shot upload on a slow link needs a larger value. It does not limit reading a download stream. With `maxAttempts` above 1 a stalled request is retried, so the total wait is the timeout times the attempts.
|
|
283
240
|
|
|
284
|
-
**`uploadIdleTimeoutMs`** covers
|
|
241
|
+
**`uploadIdleTimeoutMs`** covers what `requestTimeoutMs` cannot: a multipart upload waiting for the next bytes of a stream has no request in flight, so a source that stops delivering would hold the upload open on the provider indefinitely. After this much silence the upload is aborted (the provider is sent `AbortMultipartUpload`), your stream is destroyed and `upload()` rejects with `STORAGE_TIMEOUT` and `details.phase` set to `'source-idle'`. The clock restarts on every chunk and runs only while the library is waiting for data, so a slow but live source succeeds and a slow provider does not trip it. It defaults to the value of `requestTimeoutMs`; `0` turns it off. It applies only to a stream on the multipart path: a single-shot upload is bounded by `requestTimeoutMs`, and a `Buffer` cannot stall.
|
|
242
|
+
|
|
285
243
|
|
|
286
244
|
[aws-presigned-expiry]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
|
|
287
245
|
[aws-get-object]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
|
|
@@ -350,18 +308,17 @@ const result = await storage.upload({
|
|
|
350
308
|
console.log(result.publicUrl) // https://...
|
|
351
309
|
```
|
|
352
310
|
|
|
353
|
-
**Header values.**
|
|
311
|
+
**Header values.** Every caller string that becomes an HTTP header or a signed query value —
|
|
354
312
|
`contentType`, `cacheControl`, `contentDisposition`, `kmsKeyId`, the keys and values of `metadata`,
|
|
355
313
|
`range`, `ifMatch`, `ifNoneMatch`, and `responseContentDisposition` / `responseContentType` on a
|
|
356
|
-
signed GET — may hold only tab, space and printable ASCII.
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
`
|
|
364
|
-
non-ASCII `metadata` values yourself (`encodeURIComponent`) and decode them when you read them.
|
|
314
|
+
signed GET — may hold only tab, space and printable ASCII. Anything else, a line break included, fails
|
|
315
|
+
with `STORAGE_VALIDATION_FAILED` before any request, whether or not `validation.mimeWhitelist` is set;
|
|
316
|
+
`details` names the field and the reason, never the value. With a whitelist set it is consulted first,
|
|
317
|
+
so a `contentType` it does not allow, a malformed one included, fails as `STORAGE_MIME_NOT_ALLOWED`. A
|
|
318
|
+
`metadata` key must also be a non-empty header name: letters, digits and the characters
|
|
319
|
+
``!#$%&'*+-.^_`|~``. To send a non-ASCII filename, percent-encode it as RFC 6266 `filename*`:
|
|
320
|
+
`contentDisposition: "attachment; filename*=UTF-8''caf%C3%A9.pdf"`. Percent-encode non-ASCII
|
|
321
|
+
`metadata` values yourself (`encodeURIComponent`) and decode them when you read them.
|
|
365
322
|
|
|
366
323
|
### Stream with Progress
|
|
367
324
|
|
|
@@ -379,7 +336,7 @@ await storage.upload({
|
|
|
379
336
|
|
|
380
337
|
Uploads are automatically routed to multipart when `size >= multipart.thresholdBytes` (default 5 MiB). A `Readable` without a `size` always uses multipart. Multipart aborts automatically on failure (`leavePartsOnError: false`).
|
|
381
338
|
|
|
382
|
-
**The library owns a `Readable` once you pass it to `upload()`**,
|
|
339
|
+
**The library owns a `Readable` once you pass it to `upload()`**, as with `stream.pipeline`. When the upload is rejected for any reason, the stream is destroyed without an error, which releases the file handle or socket behind it: a listener on the stream sees `'close'`, and the failure is the rejection of `upload()`. A call answered from the idempotency cache destroys the stream unread. Do not reuse a stream you passed in; open a new one to retry. No listener on the stream is needed to avoid an unhandled `'error'` event.
|
|
383
340
|
|
|
384
341
|
### Idempotency
|
|
385
342
|
|
|
@@ -582,23 +539,19 @@ BymaxStorageModule.forRoot({
|
|
|
582
539
|
})
|
|
583
540
|
```
|
|
584
541
|
|
|
585
|
-
**Streams in pre-upload mode.** A stream can be read once, so
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
`
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
shows) and the delete failure in `cleanupError` (the error code and the provider's message); the
|
|
599
|
-
failure is also logged at error level with the code `STORAGE_SCAN_CLEANUP_FAILED` and the stored
|
|
600
|
-
key. No `details.cleanup` means the object was removed; `cleanup: 'failed'` means it is **still
|
|
601
|
-
stored** and has to be removed by other means.
|
|
542
|
+
**Streams in pre-upload mode.** A stream can be read once, so in `pre-upload` mode it is first read
|
|
543
|
+
into memory, bounded by `validation.maxSizeBytes`, and the scanner and the upload share that buffer. A
|
|
544
|
+
stream longer than the limit is refused with `STORAGE_VALIDATION_FAILED` as soon as it passes it, and
|
|
545
|
+
with no `maxSizeBytes` it is refused with a reason that says to set it or use `post-upload`. Buffer
|
|
546
|
+
bodies are not affected, and `post-upload` hands the scanner the key only, no body.
|
|
547
|
+
|
|
548
|
+
**A rejected object that could not be removed.** In `post-upload` mode the object is already stored
|
|
549
|
+
when the scanner rejects it, so the library deletes it and rejects `upload()` with the verdict
|
|
550
|
+
(`STORAGE_SCAN_INFECTED` or `STORAGE_SCAN_INCONCLUSIVE`). If that delete fails the verdict is still the
|
|
551
|
+
error, and `details` carries `cleanup: 'failed'`, the `key` you passed, the `storedKey` (with any
|
|
552
|
+
`keyPrefix`, as a bucket listing shows it) and the delete failure in `cleanupError`; the failure is
|
|
553
|
+
also logged at error level as `STORAGE_SCAN_CLEANUP_FAILED`. With no `details.cleanup` the object was
|
|
554
|
+
removed; with `cleanup: 'failed'` it is **still stored** and has to be removed by other means.
|
|
602
555
|
|
|
603
556
|
---
|
|
604
557
|
|
|
@@ -637,9 +590,9 @@ const { deleted, failed } = await storage.deleteMany(['a.png', 'b.png', 'c.png']
|
|
|
637
590
|
|
|
638
591
|
### Bucket Lifecycle Rules
|
|
639
592
|
|
|
640
|
-
`BucketLifecycleService`
|
|
641
|
-
|
|
642
|
-
|
|
593
|
+
`BucketLifecycleService` reads, adds, replaces and removes one expiration rule or one multipart-abort
|
|
594
|
+
rule by id. Every other rule in the bucket, and the `TransitionDefaultMinimumObjectSize` setting when
|
|
595
|
+
the provider reports it, is sent back unchanged.
|
|
643
596
|
|
|
644
597
|
```typescript
|
|
645
598
|
import { BucketLifecycleService } from '@bymax-one/nest-storage'
|
|
@@ -655,54 +608,43 @@ const rules = await lifecycle.get() // [] when the bucket has no configuration
|
|
|
655
608
|
const removed = await lifecycle.remove('expire-tmp') // true; false when no rule has that id
|
|
656
609
|
```
|
|
657
610
|
|
|
658
|
-
- **Read-modify-write.** `PutBucketLifecycleConfiguration` replaces the whole
|
|
659
|
-
([AWS: "this will overwrite an existing lifecycle configuration"][aws-put-lifecycle]),
|
|
660
|
-
`upsert` and `remove` read the rules, change the one with the given id
|
|
661
|
-
back.
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
-
|
|
665
|
-
|
|
666
|
-
not
|
|
667
|
-
-
|
|
668
|
-
|
|
669
|
-
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
whole bucket — is returned exactly as stored with `isOutsideKeyPrefix: true`. `upsert` and
|
|
675
|
-
`remove` refuse an id that names such a rule (`STORAGE_LIFECYCLE_RULE_INVALID`), and every write
|
|
676
|
-
sends it back field for field. Without a `keyPrefix`, no rule is flagged.
|
|
677
|
-
- **Two rule kinds.** An input has exactly one of `expirationDays` (delete objects under the prefix after
|
|
678
|
-
N days) and `abortIncompleteMultipartDays` (abort uploads not completed within N days of starting;
|
|
679
|
-
[AWS: "Specifies the number of days after which Amazon S3 aborts an incomplete multipart
|
|
611
|
+
- **Read-modify-write, not atomic.** `PutBucketLifecycleConfiguration` replaces the whole
|
|
612
|
+
configuration ([AWS: "this will overwrite an existing lifecycle configuration"][aws-put-lifecycle]),
|
|
613
|
+
so `upsert` and `remove` read the rules, change the one with the given id (replaced in place, or
|
|
614
|
+
appended when new) and write them all back. S3 has no conditional write for it, so a change made by
|
|
615
|
+
another writer between the read and the write is overwritten: run lifecycle changes from one place.
|
|
616
|
+
- **Removing the last rule deletes the configuration**, because a put with no rules is invalid.
|
|
617
|
+
- **`prefix` is relative to `keyPrefix`**, the way `list()` keys are, and `get()` returns filter
|
|
618
|
+
prefixes relative too, so `upsert({ prefix: rule.Filter.Prefix, ... })` on a value read from `get()`
|
|
619
|
+
does not prefix twice. The deprecated rule-level `Prefix` is treated like `Filter.Prefix`.
|
|
620
|
+
- **Rules outside `keyPrefix` are read-only.** With a `keyPrefix` set, a rule whose filter prefix does
|
|
621
|
+
not start with it (another application's, tag-only, or whole-bucket) is returned as stored with
|
|
622
|
+
`isOutsideKeyPrefix: true`; `upsert` and `remove` refuse its id with `STORAGE_LIFECYCLE_RULE_INVALID`,
|
|
623
|
+
and every write sends it back field for field.
|
|
624
|
+
- **Two rule kinds.** An input has exactly one of `expirationDays` (delete objects under the prefix
|
|
625
|
+
after N days) and `abortIncompleteMultipartDays` (abort uploads not completed within N days of
|
|
626
|
+
starting; [AWS: "Specifies the number of days after which Amazon S3 aborts an incomplete multipart
|
|
680
627
|
upload"][aws-abort-mpu]). A rule with the same id is replaced, whichever kind it was.
|
|
681
|
-
- **
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
`STORAGE_LIFECYCLE_UNSUPPORTED` (501); a configuration the provider refuses throws
|
|
689
|
-
`STORAGE_LIFECYCLE_REJECTED` (422). `details` carries the provider's error code, status and
|
|
690
|
-
request id, never the caught error.
|
|
691
|
-
- **`TransitionDefaultMinimumObjectSize`.** When the provider reports this setting, a write sends it
|
|
692
|
-
back unchanged so existing transition behaviour does not reset.
|
|
628
|
+
- **Validated before any request.** `id` is 1 to 255 characters, the day count an integer of at least
|
|
629
|
+
1, and `prefix` passes the key guard. An expiration rule needs a non-empty prefix; a multipart-abort
|
|
630
|
+
rule may have an empty one, meaning every upload in the module's namespace, because aborting deletes
|
|
631
|
+
no object. A bad input throws `STORAGE_LIFECYCLE_RULE_INVALID`.
|
|
632
|
+
- **Typed provider errors.** `STORAGE_LIFECYCLE_UNSUPPORTED` (501) when the provider lacks the API,
|
|
633
|
+
`STORAGE_LIFECYCLE_REJECTED` (422) when it refuses the configuration; `details` carries the
|
|
634
|
+
provider's code, status and request id, never the caught error.
|
|
693
635
|
|
|
694
636
|
**Provider support**
|
|
695
637
|
|
|
696
|
-
| Provider | Status | Basis
|
|
697
|
-
| ------------------------------------------ | --------------------- |
|
|
698
|
-
| MinIO (tested with the `pgsty/silo` image) | **Verified** | The end-to-end suite applies an expiration rule
|
|
699
|
-
| Cloudflare R2 | **Verified (docs)** | R2 documents the
|
|
700
|
-
| AWS S3 | Documented, not run |
|
|
701
|
-
| Backblaze B2, DigitalOcean Spaces, Wasabi | **Unverified** | Neither documented nor tested here. A provider without the API surfaces as `STORAGE_LIFECYCLE_UNSUPPORTED`.
|
|
638
|
+
| Provider | Status | Basis |
|
|
639
|
+
| ------------------------------------------ | --------------------- | ----- |
|
|
640
|
+
| MinIO (tested with the `pgsty/silo` image) | **Verified** | The end-to-end suite applies an expiration rule and keeps unrelated rules through upsert, replace and remove. It does not accept a multipart-abort rule: the server answers HTTP 400 `InvalidArgument`, typed as `STORAGE_LIFECYCLE_REJECTED`, and the existing rules stay as they were. |
|
|
641
|
+
| Cloudflare R2 | **Verified (docs)** | R2 documents the lifecycle calls and multipart-abort rules ([R2 object lifecycles][r2-lifecycles]); this repository does not run them against R2. |
|
|
642
|
+
| AWS S3 | Documented, not run | Follows the S3 API reference ([Put][aws-put-lifecycle], [Get][aws-get-lifecycle], [Delete][aws-delete-lifecycle]); no test here hits AWS. |
|
|
643
|
+
| Backblaze B2, DigitalOcean Spaces, Wasabi | **Unverified** | Neither documented nor tested here. A provider without the API surfaces as `STORAGE_LIFECYCLE_UNSUPPORTED`. |
|
|
702
644
|
|
|
703
|
-
Expiry timing is the provider's
|
|
704
|
-
|
|
705
|
-
|
|
645
|
+
Expiry timing is the provider's: R2 states that objects "will typically be removed from a bucket
|
|
646
|
+
within 24 hours" of their expiry, and MinIO's scanner runs at low priority, so an object can outlive
|
|
647
|
+
its rule period.
|
|
706
648
|
|
|
707
649
|
[aws-put-lifecycle]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html
|
|
708
650
|
[aws-abort-mpu]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortIncompleteMultipartUpload.html
|
|
@@ -710,6 +652,7 @@ object can outlive its rule period.
|
|
|
710
652
|
[aws-delete-lifecycle]: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketLifecycle.html
|
|
711
653
|
[r2-lifecycles]: https://developers.cloudflare.com/r2/buckets/object-lifecycles/
|
|
712
654
|
|
|
655
|
+
|
|
713
656
|
### Raw S3Client (Advanced)
|
|
714
657
|
|
|
715
658
|
For provider-specific operations not covered by the public API, inject the raw client:
|
|
@@ -770,13 +713,13 @@ All errors are thrown as `StorageException extends HttpException`. The response
|
|
|
770
713
|
| `STORAGE_OBJECT_NOT_FOUND` | 404 | `head()`, `download()`, or `copy()` on a nonexistent key |
|
|
771
714
|
| `STORAGE_PROVIDER_ERROR` | 502 | AWS SDK error (network, 5xx, throttling, `AccessControlListNotSupported`); `details` carries `awsCode`, `awsMessage`, `httpStatus`, `requestId` plus the operation context (`op`, `bucket`, and `key` or `prefix`) |
|
|
772
715
|
| `STORAGE_SIGNED_URL_TTL_INVALID` | 400 | Per-request `ttlSeconds` ≤ 0 |
|
|
773
|
-
| `STORAGE_PART_TOO_SMALL` | 400 |
|
|
716
|
+
| `STORAGE_PART_TOO_SMALL` | 400 | Never thrown: a `multipart.partSizeBytes` below 5 MiB fails at init as `STORAGE_INVALID_CONFIG` |
|
|
774
717
|
| `STORAGE_INVALID_PART_COUNT` | 400 | `getMultipartUploadUrls()` with `parts` not an integer from 1 to 10 000 (`details.provided`, `details.reason`) |
|
|
775
718
|
| `STORAGE_BUCKET_UNDEFINED` | 400 | Operation without a bucket and no default in config |
|
|
776
|
-
| `STORAGE_MULTIPART_ABORTED` | 500 | A multipart upload failed. `details.abort` is `'completed'` when the library aborted it
|
|
719
|
+
| `STORAGE_MULTIPART_ABORTED` | 500 | A multipart upload failed. `details.abort` is `'completed'` when the library aborted it, `'failed'` (with `abortError`) when the abort itself failed: that upload then stays open on the provider until a lifecycle rule removes it |
|
|
777
720
|
| `STORAGE_INVALID_CONFIG` | 500 | `BymaxStorageModuleOptions` validation failed at initialization |
|
|
778
|
-
| `STORAGE_TIMEOUT` | 504 | An attempt got no response headers within `requestTimeoutMs
|
|
779
|
-
| `STORAGE_LIFECYCLE_RULE_INVALID` | 400 | `BucketLifecycleService` input rejected:
|
|
721
|
+
| `STORAGE_TIMEOUT` | 504 | An attempt got no response headers within `requestTimeoutMs`, or a multipart source stayed silent for `uploadIdleTimeoutMs` (`details.phase: 'source-idle'`) |
|
|
722
|
+
| `STORAGE_LIFECYCLE_RULE_INVALID` | 400 | `BucketLifecycleService` input rejected: bad `id`, day count or `prefix`, or an id naming a rule outside `keyPrefix` (`details.reason`) |
|
|
780
723
|
| `STORAGE_LIFECYCLE_UNSUPPORTED` | 501 | The provider does not implement the lifecycle API |
|
|
781
724
|
| `STORAGE_LIFECYCLE_REJECTED` | 422 | The provider refused the lifecycle configuration |
|
|
782
725
|
| `STORAGE_BUCKET_NOT_FOUND` | 404 | The provider answered `NoSuchBucket` (any operation that gets an error body: `download()`, `upload()`, `list()`, `copy()`, `delete()`), or `checkBucket()` got a 404 |
|
|
@@ -800,9 +743,168 @@ try {
|
|
|
800
743
|
|
|
801
744
|
---
|
|
802
745
|
|
|
746
|
+
## 🚫 What This Library Does NOT Do
|
|
747
|
+
|
|
748
|
+
- **Detect a file's real type.** The MIME check reads the declared `Content-Type` only; plug in an
|
|
749
|
+
`IUploadValidator` with a `readBytes(n)` magic-byte check for untrusted sources ([Validation](#-validation)).
|
|
750
|
+
- **Bound a signed PUT.** A signed URL skips local size validation and custom validators, and no size
|
|
751
|
+
limit is signed into it ([Presigned PUT](#presigned-put-direct-browser-upload)).
|
|
752
|
+
- **Isolate tenants.** `keyPrefix` is a namespace, not an access boundary ([Security Model](#-security-model)).
|
|
753
|
+
- **Share state between replicas.** The idempotency cache is per instance.
|
|
754
|
+
- **Complete or abort a presigned multipart upload.** The client sends the part ETags to `completeUrl`
|
|
755
|
+
and aborts with its own signed request ([Presigned Multipart](#presigned-multipart)).
|
|
756
|
+
- **Make ACLs work everywhere.** `publicRead` fails on modern AWS S3 and is a no-op on R2
|
|
757
|
+
([Public Access / ACLs](#public-access--acls)).
|
|
758
|
+
- **Lock lifecycle writes.** Changes are read-modify-write and not atomic.
|
|
759
|
+
- **Wrap every S3 operation.** Provider-specific calls go through the raw `S3Client`
|
|
760
|
+
([Raw S3Client](#raw-s3client-advanced)).
|
|
761
|
+
|
|
762
|
+
---
|
|
763
|
+
|
|
764
|
+
## 🏗️ Architecture
|
|
765
|
+
|
|
766
|
+
```
|
|
767
|
+
BymaxStorageModule.forRoot / forRootAsync
|
|
768
|
+
│
|
|
769
|
+
validateOptions (fail fast)
|
|
770
|
+
credentials tolerated empty on purpose
|
|
771
|
+
│
|
|
772
|
+
┌───────────────┴────────────────┐
|
|
773
|
+
│ S3ClientProvider │
|
|
774
|
+
│ one @aws-sdk/client-s3 client │
|
|
775
|
+
│ built from a provider recipe │
|
|
776
|
+
└───────────────┬────────────────┘
|
|
777
|
+
│
|
|
778
|
+
BYMAX_STORAGE_S3_CLIENT
|
|
779
|
+
(escape hatch — the raw S3Client)
|
|
780
|
+
│
|
|
781
|
+
┌─────────────────────────┴─────────────────────────┐
|
|
782
|
+
│ │
|
|
783
|
+
StorageService SignedUrlService
|
|
784
|
+
upload · download GET · PUT · multipart
|
|
785
|
+
head · exists · list TTL-clamped
|
|
786
|
+
copy · delete · deleteMany SigV4 ceiling 7 days
|
|
787
|
+
│ │
|
|
788
|
+
└──────────────┬────────────────────────────────────┘
|
|
789
|
+
│
|
|
790
|
+
KeyResolverService
|
|
791
|
+
every caller key passes here before S3:
|
|
792
|
+
refuse `..`, leading `/`, NUL, empty → then keyPrefix
|
|
793
|
+
│
|
|
794
|
+
┌────────────────────┼────────────────────┐
|
|
795
|
+
│ │ │
|
|
796
|
+
ValidationService IFileScanner IdempotencyCache
|
|
797
|
+
MIME + size, then pre- and/or LRU, in-process,
|
|
798
|
+
IUploadValidator[] post-upload collapses a retry
|
|
799
|
+
(yours) (yours) within one instance
|
|
800
|
+
```
|
|
801
|
+
|
|
802
|
+
`SignedUrlService` stops at `KeyResolverService`: it signs a URL and never sees the body, so
|
|
803
|
+
validation, scanning and the idempotency cache apply to `StorageService` only. See the signed-PUT
|
|
804
|
+
caveat above.
|
|
805
|
+
|
|
806
|
+
Options are validated before any client is built. Credentials are the deliberate exception: empty
|
|
807
|
+
values are tolerated so a development workflow boots, and operations then fail with
|
|
808
|
+
`STORAGE_NOT_CONFIGURED`.
|
|
809
|
+
|
|
810
|
+
---
|
|
811
|
+
|
|
812
|
+
## 🔐 Security Model
|
|
813
|
+
|
|
814
|
+
This library turns caller input into object keys, signs URLs that grant access on their own, and holds
|
|
815
|
+
the credentials for a bucket. Its contract covers what a key can reach, how long a signature lives,
|
|
816
|
+
and what leaves in an error.
|
|
817
|
+
|
|
818
|
+
### The object key is the attack surface
|
|
819
|
+
|
|
820
|
+
Everything a caller sends becomes part of a key, and a key is a path. `KeyResolverService.normalize`
|
|
821
|
+
is the single chokepoint: it refuses an empty key, a null byte, a leading `/` and any `..` segment,
|
|
822
|
+
collapses duplicate slashes, and only then prepends `keyPrefix`, so a tenant cannot climb out of its prefix by naming one.
|
|
823
|
+
|
|
824
|
+
### Presigned URLs are bearer credentials
|
|
825
|
+
|
|
826
|
+
Anyone holding one has the access it encodes until it expires, so treat it like a password with an
|
|
827
|
+
expiry, not like a link. Its lifetime is capped by `signedUrls.maxTtlSeconds` (see
|
|
828
|
+
[Signed URLs](#-signed-urls-get--put--multipart)), and it must never be logged.
|
|
829
|
+
|
|
830
|
+
### Credentials stay where they were put
|
|
831
|
+
|
|
832
|
+
They arrive through module options and are handed to the SDK. This library never reads `process.env`,
|
|
833
|
+
never logs them, and never places them — or a signed URL — in an exception.
|
|
834
|
+
|
|
835
|
+
They are also not reachable by serializing the objects that hold them. The resolved options are
|
|
836
|
+
injected into every service, so `credentials` is attached as a non-enumerable accessor:
|
|
837
|
+
`JSON.stringify`, object spread and `util.inspect` (with or without `showHidden`) all omit it, which
|
|
838
|
+
is what a structured logger or an error reporter capturing a scope would otherwise pick up. Reading it
|
|
839
|
+
on purpose is unchanged.
|
|
840
|
+
|
|
841
|
+
### Error payloads carry the operation, and that includes the key
|
|
842
|
+
|
|
843
|
+
`details` holds the provider's error code and message, the HTTP status, the request id, and the
|
|
844
|
+
operation context the call site supplies: the operation, the bucket, and the resolved key or prefix.
|
|
845
|
+
That is enough to diagnose a failure, and it means an object key reaches whatever consumes the
|
|
846
|
+
exception, so if a key is itself sensitive in your deployment, do not log the envelope verbatim.
|
|
847
|
+
|
|
848
|
+
### `keyPrefix` is a namespace, not an access boundary
|
|
849
|
+
|
|
850
|
+
It scopes the keys this library composes; it does not restrict what the credentials can reach. Anything
|
|
851
|
+
holding the bucket's credentials can read every prefix in it. Isolation that must survive a compromised
|
|
852
|
+
client belongs in IAM policies or separate buckets.
|
|
853
|
+
|
|
854
|
+
It is also fixed when the module is registered: `forRoot` resolves one prefix, and the module is
|
|
855
|
+
`@Global()` with one `StorageService` behind fixed injection tokens, so it is not designed to be
|
|
856
|
+
registered twice in one application with different prefixes. To keep tenants apart in one application,
|
|
857
|
+
build the tenant segment into the key on each call from a trusted source (`tenants/${tenantId}/${name}`,
|
|
858
|
+
never from request input) or give each tenant its own bucket and credentials.
|
|
859
|
+
|
|
860
|
+
---
|
|
861
|
+
|
|
862
|
+
## 🛡️ Security Table
|
|
863
|
+
|
|
864
|
+
| Layer | Control |
|
|
865
|
+
| ---------------- | ---------------------------------------------------------------------------------------------------- |
|
|
866
|
+
| Object keys | One guard: `..`, a leading `/`, null bytes and empty keys refused before `keyPrefix` is prepended |
|
|
867
|
+
| Multi-tenancy | `keyPrefix` is a namespace, not an access boundary; isolate with IAM policies or separate buckets |
|
|
868
|
+
| Credentials | Injected options only; never read from `process.env`, never logged, never in an exception |
|
|
869
|
+
| Error payloads | Provider code and message, status, request id and operation context; never credentials or a signed URL |
|
|
870
|
+
| Presign lifetime | TTL capped by `signedUrls.maxTtlSeconds` |
|
|
871
|
+
| Header values | Tab, space and printable ASCII only; refused with `STORAGE_VALIDATION_FAILED` before any request |
|
|
872
|
+
| Encryption | Server-side (AES256 / `aws:kms`), globally or per upload |
|
|
873
|
+
| Content | MIME allowlist and size limit, then registered validators, then the scanner |
|
|
874
|
+
| Supply chain | `dependencies: {}`; see the repository's CI workflows for the scanners that run on it |
|
|
875
|
+
|
|
876
|
+
---
|
|
877
|
+
|
|
878
|
+
## 🧱 Tech Stack
|
|
879
|
+
|
|
880
|
+

|
|
881
|
+

|
|
882
|
+
-FF9900?style=flat-square&logo=amazons3&logoColor=white)
|
|
883
|
+

|
|
884
|
+

|
|
885
|
+

|
|
886
|
+

|
|
887
|
+

|
|
888
|
+
|
|
889
|
+
---
|
|
890
|
+
|
|
891
|
+
## 🧪 Testing & Quality
|
|
892
|
+
|
|
893
|
+
- ✅ **100% coverage** — statements, branches, functions and lines, enforced as a gate
|
|
894
|
+
- ✅ **100% mutation score** — [Stryker](https://stryker-mutator.io/) at `break: 100`
|
|
895
|
+
- ✅ **Real object storage in e2e** — MinIO through Testcontainers, so multipart, presigning and
|
|
896
|
+
listing are exercised against an actual S3 API rather than a mock
|
|
897
|
+
- ✅ **Published-artifact gates** — `check:exports` resolves the types the way each module system
|
|
898
|
+
does, `check:runtime` loads every subpath from the packed tarball in ESM and CommonJS, and
|
|
899
|
+
`check:published` compiles this README's snippets against `dist/`
|
|
900
|
+
- ✅ **Every suppression is justified** — no coverage directives anywhere; each `// Stryker disable`
|
|
901
|
+
comment in the production source names why its mutant is provably equivalent, and `check:mutants`
|
|
902
|
+
proves the reason is written in the grammar Stryker parses, so the mutation report carries it in full
|
|
903
|
+
|
|
904
|
+
---
|
|
905
|
+
|
|
803
906
|
## 📖 API Reference
|
|
804
907
|
|
|
805
|
-
Every operation below is documented with a runnable example in the sections above.
|
|
806
908
|
This is the index.
|
|
807
909
|
|
|
808
910
|
### `StorageService`
|
|
@@ -841,7 +943,7 @@ This is the index.
|
|
|
841
943
|
| Method | Signature | Notes |
|
|
842
944
|
| -------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------- |
|
|
843
945
|
| `get` | `(options?) => Promise<BucketLifecycleRuleView[]>` | `[]` when the bucket has no configuration; filter prefixes are relative to `keyPrefix`; rules outside it carry `isOutsideKeyPrefix` |
|
|
844
|
-
| `upsert` | `(rule:
|
|
946
|
+
| `upsert` | `(rule: LifecycleRuleInput, options?) => Promise<void>` | Read-modify-write; replaces the rule with the same id, keeps the rest |
|
|
845
947
|
| `remove` | `(id: string, options?) => Promise<boolean>` | `false` and no write for an unknown id; deletes the configuration when it removes the last rule |
|
|
846
948
|
|
|
847
949
|
### Key normalization
|
|
@@ -859,7 +961,6 @@ is the one full value: it is a URL, and the prefix is part of the object's real
|
|
|
859
961
|
|
|
860
962
|
### Exported types and constants
|
|
861
963
|
|
|
862
|
-
Every name the package exports is listed here, so none is a surprise.
|
|
863
964
|
|
|
864
965
|
- **Module configuration:** `BymaxStorageModuleOptions`, `BymaxStorageModuleAsyncOptions` (the
|
|
865
966
|
`forRootAsync` shape: `imports`, `inject`, and one of `useFactory`, `useClass`, `useExisting`) and
|
|
@@ -871,11 +972,12 @@ Every name the package exports is listed here, so none is a surprise.
|
|
|
871
972
|
`CopyOptions`, `DeleteManyOptions`, `DeleteManyResult`, `FailedDeletion`, `SignedGetUrlOptions`,
|
|
872
973
|
`SignedPutUrlOptions`, `MultipartUploadUrlsOptions`, `MultipartUploadUrlsResult`.
|
|
873
974
|
- **Lifecycle:** `BucketLifecycleRule` (the provider's rule as stored), `BucketLifecycleRuleView` (what
|
|
874
|
-
`get()` returns), `
|
|
975
|
+
`get()` returns), `LifecycleRuleInput` (`ExpirationRuleInput` or `AbortMultipartRuleInput`) and
|
|
976
|
+
`LifecycleBucketOptions` (`{ bucket? }`).
|
|
875
977
|
- **Extension points:** `IUploadValidator`, `IFileScanner`, `FileScanResult`, and the helpers
|
|
876
978
|
`NoOpUploadValidator` and `NoOpFileScanner`.
|
|
877
979
|
- **From `./shared`:** `UploadResult`, `ObjectMetadata`, `ListedObject`, `SignedUrlResult`,
|
|
878
|
-
`StorageErrorResponse`, `StorageErrorCode`, `STORAGE_ERROR_CODES`, the
|
|
980
|
+
`StorageErrorResponse`, `StorageErrorCode`, `STORAGE_ERROR_CODES`, the MIME presets
|
|
879
981
|
`DEFAULT_IMAGE_MIME_WHITELIST`, `DEFAULT_VIDEO_MIME_WHITELIST`, `DEFAULT_DOC_MIME_WHITELIST`, and the
|
|
880
982
|
defaults `DEFAULT_SIGNED_URL_TTL_SECONDS`, `MAX_SIGNED_URL_TTL_SECONDS`,
|
|
881
983
|
`DEFAULT_MULTIPART_THRESHOLD_BYTES`, `DEFAULT_MULTIPART_PART_SIZE_BYTES`,
|
|
@@ -889,212 +991,19 @@ that object, but they are readable on it: treat the token as sensitive.
|
|
|
889
991
|
|
|
890
992
|
`BYMAX_STORAGE_OPTIONS` · `BYMAX_STORAGE_S3_CLIENT` · `BYMAX_STORAGE_FILE_SCANNER` ·
|
|
891
993
|
`BYMAX_STORAGE_UPLOAD_VALIDATORS` — all `Symbol()`, so no string token can collide with
|
|
892
|
-
them, and
|
|
893
|
-
`UnknownDependenciesException`.
|
|
994
|
+
them, and they are in the module's `exports`, so injection resolves rather than raising
|
|
995
|
+
`UnknownDependenciesException`. Some are nullable by design: `BYMAX_STORAGE_S3_CLIENT`
|
|
894
996
|
is `null` until credentials are configured, and `BYMAX_STORAGE_FILE_SCANNER` is `null` when no
|
|
895
997
|
scanner is set, which is the default. Type them as `S3Client | null` and `IFileScanner | null`.
|
|
896
998
|
|
|
897
999
|
---
|
|
898
1000
|
|
|
899
|
-
## 🏗️ Architecture
|
|
900
|
-
|
|
901
|
-
```
|
|
902
|
-
BymaxStorageModule.forRoot / forRootAsync
|
|
903
|
-
│
|
|
904
|
-
validateOptions (fail fast)
|
|
905
|
-
credentials tolerated empty on purpose
|
|
906
|
-
│
|
|
907
|
-
┌───────────────┴────────────────┐
|
|
908
|
-
│ S3ClientProvider │
|
|
909
|
-
│ one @aws-sdk/client-s3 client │
|
|
910
|
-
│ built from a provider recipe │
|
|
911
|
-
└───────────────┬────────────────┘
|
|
912
|
-
│
|
|
913
|
-
BYMAX_STORAGE_S3_CLIENT
|
|
914
|
-
(escape hatch — the raw S3Client)
|
|
915
|
-
│
|
|
916
|
-
┌─────────────────────────┴─────────────────────────┐
|
|
917
|
-
│ │
|
|
918
|
-
StorageService SignedUrlService
|
|
919
|
-
upload · download GET · PUT · multipart
|
|
920
|
-
head · exists · list TTL-clamped
|
|
921
|
-
copy · delete · deleteMany SigV4 ceiling 7 days
|
|
922
|
-
│ │
|
|
923
|
-
└──────────────┬────────────────────────────────────┘
|
|
924
|
-
│
|
|
925
|
-
KeyResolverService
|
|
926
|
-
every caller key passes here before S3:
|
|
927
|
-
refuse `..`, leading `/`, NUL, empty → then keyPrefix
|
|
928
|
-
│
|
|
929
|
-
┌────────────────────┼────────────────────┐
|
|
930
|
-
│ │ │
|
|
931
|
-
ValidationService IFileScanner IdempotencyCache
|
|
932
|
-
MIME + size, then pre- and/or LRU, in-process,
|
|
933
|
-
IUploadValidator[] post-upload collapses a retry
|
|
934
|
-
(yours) (yours) within one instance
|
|
935
|
-
```
|
|
936
|
-
|
|
937
|
-
`SignedUrlService` stops at `KeyResolverService`: it signs a URL and never sees the body, so
|
|
938
|
-
validation, scanning and the idempotency cache apply to `StorageService` only. See the signed-PUT
|
|
939
|
-
caveat above.
|
|
940
|
-
|
|
941
|
-
One engine, six recipes. `providerRecipes` differ only in endpoint, region and the
|
|
942
|
-
handful of flags each provider needs (`forcePathStyle` for MinIO, `auto` region for
|
|
943
|
-
R2) — there is no per-provider code path, so a provider swap is a configuration
|
|
944
|
-
change and every operation behaves the same way afterwards.
|
|
945
|
-
|
|
946
|
-
The idempotency cache is **in-process**. It collapses a retried `upload` inside one
|
|
947
|
-
instance; it does not coordinate between replicas.
|
|
948
|
-
|
|
949
|
-
### Design Principles
|
|
950
|
-
|
|
951
|
-
| Principle | Description |
|
|
952
|
-
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
953
|
-
| 🔑 **One key chokepoint** | Every caller-supplied key passes through `KeyResolverService` before it reaches S3. `..`, a leading `/`, null bytes and empty keys are refused, and `keyPrefix` is prepended after normalization — so it cannot be escaped by the key |
|
|
954
|
-
| 🧩 **One engine, six recipes** | `providerRecipes` differ in endpoint, region and per-provider flags only. There is no per-provider code path, so a provider swap is configuration and every operation behaves the same afterwards |
|
|
955
|
-
| 🚫 **Refuse before storing** | MIME allowlist and size limit, then registered `IUploadValidator`s, then the `IFileScanner`. An upload that will be rejected is rejected before a byte is written |
|
|
956
|
-
| ⏱️ **Bounded signatures** | Presign TTLs above `signedUrls.maxTtlSeconds` (default 604 800 s, the SigV4 limit) are clamped locally, not discovered when the provider rejects the signature |
|
|
957
|
-
| 💥 **Fail at boot, mostly** | Options are validated before any client is built. Credentials are the deliberate exception: empty values are tolerated so a development workflow boots, and operations then fail with `STORAGE_NOT_CONFIGURED` |
|
|
958
|
-
| 🧊 **Zero runtime dependencies** | `dependencies` is `{}`. You choose the exact `@aws-sdk/*` versions and the supply-chain surface stays yours |
|
|
959
|
-
|
|
960
|
-
---
|
|
961
|
-
|
|
962
|
-
## 🔐 Security Model
|
|
963
|
-
|
|
964
|
-
This library turns caller input into object keys, signs URLs that grant access on their
|
|
965
|
-
own, and holds the credentials for a bucket. Its security contract is about what a key can
|
|
966
|
-
reach, how long a signature lives, and what leaves in an error.
|
|
967
|
-
|
|
968
|
-
### The object key is the attack surface
|
|
969
|
-
|
|
970
|
-
Everything a caller sends becomes part of a key, and a key is a path.
|
|
971
|
-
`KeyResolverService.normalize` is the single chokepoint: it refuses an empty key, a key
|
|
972
|
-
containing a null byte, a key starting with `/`, and any key with a `..` segment; it
|
|
973
|
-
collapses duplicate slashes; and only then does it prepend `keyPrefix`. Nothing in this
|
|
974
|
-
library composes a key any other way, so a tenant cannot climb out of its prefix by naming
|
|
975
|
-
one.
|
|
976
|
-
|
|
977
|
-
### Presigned URLs are bearer credentials
|
|
978
|
-
|
|
979
|
-
Anyone holding one has the access it encodes, for as long as it lives. A requested TTL
|
|
980
|
-
above `signedUrls.maxTtlSeconds` (604 800 s, SigV4's limit, by default) is clamped here rather
|
|
981
|
-
than passed through to a signature the provider would reject at use time. Treat a signed URL like a password
|
|
982
|
-
with an expiry, not like a link.
|
|
983
|
-
|
|
984
|
-
### Credentials stay where they were put
|
|
985
|
-
|
|
986
|
-
They arrive through module options and are handed to the SDK. This library never reads
|
|
987
|
-
`process.env`, never logs them, and never places them — or a signed URL — in an exception.
|
|
988
|
-
|
|
989
|
-
They are also not reachable by serializing the objects that hold them. The resolved
|
|
990
|
-
options are injected into every service, so `credentials` is attached as a non-enumerable
|
|
991
|
-
accessor: `JSON.stringify`, object spread, `util.inspect` and `util.inspect` with
|
|
992
|
-
`showHidden` all omit it. Those are the paths taken by code that renders a provider it was
|
|
993
|
-
handed incidentally — a structured logger formatting its arguments, an error reporter
|
|
994
|
-
capturing the scope of a throw. Reading on purpose is unchanged.
|
|
995
|
-
|
|
996
|
-
### Error payloads carry the operation, and that includes the key
|
|
997
|
-
|
|
998
|
-
What `aws-error-mapper` puts in `details` is the provider's error code and message, the
|
|
999
|
-
HTTP status, the request id, and the operation context the call site supplies: which
|
|
1000
|
-
operation, which bucket, and the resolved key or prefix. That is enough to diagnose a
|
|
1001
|
-
failure, and it means an object key reaches whatever consumes the exception — so if a key
|
|
1002
|
-
is itself sensitive in your deployment, do not log the envelope verbatim.
|
|
1003
|
-
|
|
1004
|
-
### Uploads are refused before they are stored
|
|
1005
|
-
|
|
1006
|
-
MIME allowlist (wildcards included) and size limit run first, then any `IUploadValidator`
|
|
1007
|
-
you register, then the `IFileScanner` if configured. A scanner runs either pre-upload or
|
|
1008
|
-
post-upload — the post-upload position exists because some scanners only accept an
|
|
1009
|
-
object they can fetch.
|
|
1010
|
-
|
|
1011
|
-
### `keyPrefix` is a namespace, not an access boundary
|
|
1012
|
-
|
|
1013
|
-
It scopes the keys this library composes; it does not restrict what the credentials can
|
|
1014
|
-
reach. Anything holding the bucket's credentials can read every prefix in it. Isolation
|
|
1015
|
-
that must survive a compromised client belongs in IAM policies or separate buckets.
|
|
1016
|
-
|
|
1017
|
-
It is also fixed when the module is registered: `forRoot` resolves one prefix, and the
|
|
1018
|
-
module is `@Global()` with one `StorageService` behind fixed injection tokens, so it is not
|
|
1019
|
-
designed to be registered twice in one application with different prefixes. To keep tenants
|
|
1020
|
-
apart in one application, build the tenant segment into the key on each call from a trusted
|
|
1021
|
-
source (`tenants/${tenantId}/${name}`, never from request input) or give each tenant its own
|
|
1022
|
-
bucket and credentials.
|
|
1023
|
-
|
|
1024
|
-
---
|
|
1025
|
-
|
|
1026
|
-
## 🛡️ Security Table
|
|
1027
|
-
|
|
1028
|
-
| Layer | Implementation |
|
|
1029
|
-
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
1030
|
-
| Object keys | Guarded and normalized in one place; `..`, leading `/`, null bytes and empty keys refused |
|
|
1031
|
-
| Multi-tenancy | `keyPrefix` prepended after normalization, so it cannot be escaped by the key |
|
|
1032
|
-
| Credentials | Injected options only; never read from `process.env`, never logged, never in an exception; held in a non-enumerable accessor, so serializing a service omits them |
|
|
1033
|
-
| Error payloads | Provider code and message, HTTP status, request id, and the operation context (`op`, `bucket`, `key`/`prefix`) — never credentials, never a signed URL |
|
|
1034
|
-
| Presign lifetime | TTL clamped to `signedUrls.maxTtlSeconds` (default: SigV4's 7-day limit) |
|
|
1035
|
-
| Encryption | Server-side (AES256 / `aws:kms`) configurable globally or per upload |
|
|
1036
|
-
| Content | MIME allowlist + size limit, then registered validators, then the scanner |
|
|
1037
|
-
| Supply chain | `dependencies: {}`; third-party Actions pinned by commit SHA (org-internal reusables by tag); CodeQL, OSV-Scanner and OpenSSF Scorecard |
|
|
1038
|
-
|
|
1039
|
-
> [!IMPORTANT]
|
|
1040
|
-
> **`keyPrefix` is not an access boundary.** It scopes the keys this library composes;
|
|
1041
|
-
> it does not restrict what the credentials can reach. Anything holding the bucket's
|
|
1042
|
-
> credentials can read every prefix in it. Enforce tenant isolation with IAM policies
|
|
1043
|
-
> or separate buckets when the boundary has to hold against the application itself.
|
|
1044
|
-
|
|
1045
|
-
---
|
|
1046
|
-
|
|
1047
|
-
## 🧱 Tech Stack
|
|
1048
|
-
|
|
1049
|
-
- **Runtime:** Node.js 24+
|
|
1050
|
-
- **Framework:** NestJS 11 (a hand-written `forRoot` / `forRootAsync` dynamic module, `@Global()`, `Symbol()` tokens)
|
|
1051
|
-
- **Storage engine:** `@aws-sdk/client-s3 ^3.700` (peer), with `@aws-sdk/lib-storage`
|
|
1052
|
-
for multipart and `@aws-sdk/s3-request-presigner` for presigns — all peers
|
|
1053
|
-
- **Providers:** AWS S3, Cloudflare R2, Backblaze B2, DigitalOcean Spaces, MinIO, Wasabi
|
|
1054
|
-
- **Build:** tsup — ESM + CJS per subpath, with `.d.ts` _and_ `.d.cts` declarations
|
|
1055
|
-
- **Tests:** Jest + Testcontainers (MinIO, E2E) + Stryker (mutation)
|
|
1056
|
-
- **TypeScript:** 5.x strict (`noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`), zero `any`
|
|
1057
|
-
|
|
1058
|
-
---
|
|
1059
|
-
|
|
1060
|
-
## 🧪 Testing & Quality
|
|
1061
|
-
|
|
1062
|
-
Object storage is where a bug is expensive to discover later — a key written to the wrong
|
|
1063
|
-
place stays there — so the suite is held to a bar beyond "the tests pass".
|
|
1064
|
-
|
|
1065
|
-
- ✅ **100% line coverage** — statements, branches, functions and lines, enforced as a gate
|
|
1066
|
-
- ✅ **100% mutation score** — verified with [Stryker](https://stryker-mutator.io/) at
|
|
1067
|
-
`break: 100`; every survivor was killed by a strengthened assertion, and the provable
|
|
1068
|
-
equivalents are documented inline
|
|
1069
|
-
- ✅ **Real object storage in e2e** — MinIO through Testcontainers, so multipart, presigning
|
|
1070
|
-
and listing are exercised against an actual S3 API rather than a mock
|
|
1071
|
-
- ✅ **Published-artifact gates** — `check:exports` resolves the types the way each module
|
|
1072
|
-
system does, `check:runtime` loads every subpath from the packed tarball in ESM and
|
|
1073
|
-
CommonJS, and `check:published` compiles this README's snippets against `dist/`
|
|
1074
|
-
- ✅ **Every suppression is justified** — no coverage directives anywhere; the six
|
|
1075
|
-
`// Stryker disable` comments in the production source each name why the mutant they
|
|
1076
|
-
silence is provably equivalent, and `check:mutants` proves each reason is written in the
|
|
1077
|
-
grammar Stryker parses, so the mutation report carries it in full rather than the
|
|
1078
|
-
`Ignored using a comment` fallback
|
|
1079
|
-
|
|
1080
|
-
```bash
|
|
1081
|
-
pnpm test # unit tests (Jest)
|
|
1082
|
-
pnpm test:cov # unit tests with the 100% coverage gate
|
|
1083
|
-
pnpm test:e2e # end-to-end against MinIO (requires Docker)
|
|
1084
|
-
pnpm mutation # Stryker mutation testing (break: 100)
|
|
1085
|
-
pnpm typecheck # tsc strict check
|
|
1086
|
-
pnpm lint # ESLint
|
|
1087
|
-
```
|
|
1088
|
-
|
|
1089
|
-
---
|
|
1090
|
-
|
|
1091
1001
|
## 🤝 Contributing
|
|
1092
1002
|
|
|
1093
1003
|
Pull requests are welcome. Please open an issue first for significant changes.
|
|
1094
1004
|
|
|
1095
|
-
- Read [`docs/technical_specification.md`](docs/technical_specification.md) for architecture decisions.
|
|
1096
1005
|
- Run `pnpm test:cov` and `pnpm lint` before opening a PR.
|
|
1097
|
-
-
|
|
1006
|
+
- Use Conventional Commits for commit messages.
|
|
1098
1007
|
|
|
1099
1008
|
---
|
|
1100
1009
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@bymax-one/nest-storage",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.11",
|
|
4
4
|
"description": "Provider-agnostic S3-compatible object storage for NestJS. Works with AWS S3, DigitalOcean Spaces, Cloudflare R2, Backblaze B2, MinIO, Wasabi.",
|
|
5
5
|
"author": "Bymax One <support@bymax.one>",
|
|
6
6
|
"license": "MIT",
|