@lenne.tech/nest-server 11.41.6 → 11.41.7

package/src/core/modules/file/INTEGRATION-CHECKLIST.md

@@ -88,6 +88,31 @@ If some files are genuinely public and others are not, do not solve it with a ro
 record a visibility flag in the metadata and branch in `checkRights()`, or expose a separate public
 route for exactly the public files and leave the core routes gated.
 
+### 6. Switching to the S3 driver — in this order
+
+Each step fails on its own if skipped, and only the first one fails loudly:
+
+1. **Forward the services to `super()`.** `FileService` must call
+   `super(connection, 'fs', { configService, s3Service })`, as the reference implementation does. With
+   `super(connection)` the S3 store counts as unavailable and `file.storage: 's3'` fails the boot
+   with "that storage is not available".
+2. **Install the S3 packages as `dependencies`, not `devDependencies`:** `@aws-sdk/client-s3` and
+   `@aws-sdk/lib-storage` (same version), plus `@aws-sdk/s3-request-presigner` for
+   `s3.presignedDownloads`. A production image installed with `--prod` otherwise boots without them.
+   Vendor mode has them installed already (the vendored core references their types) — check which
+   section they are in.
+3. **Count stored files across all three metadata collections.** `fs.files` is GridFS only; the
+   filesystem driver records in `filesystem-files` and S3 in `s3-files`. Project code that
+   aggregates sizes or counts directly — a quota, a statistic — must cover all three, or it reads 0
+   after the switch and the quota never refuses anything. `findFileInfo()` already merges every
+   store.
+4. **Then switch the driver** — `file.storage: 's3'`, or leave it unset and let `s3.bucket` derive
+   it. Files written before stay readable: reads consult every store.
+
+An `s3` block with a `bucket` turns `CoreS3Service` on whatever `file.storage` says (TUS, where enabled, stages
+uploads in S3 as soon as it is on), so it needs `@aws-sdk/client-s3` at boot. To prepare the block
+without activating it, set `enabled: false`.
+
 ## Pick your project class first — the whole model is one dial with five settings
 
 `file.access` is the per-file rule as a DECLARATION instead of code. Until 11.35.0 the framework shipped
@@ -201,14 +226,17 @@ personal or medical, a per-file rule is not optional.
 
 ## Common Mistakes
 
-| Mistake                                            | Symptom                                  | Fix                                                                                                                    |
-| -------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
-| Overriding `getFileById` to add Swagger decorators | `file.downloadRoles` silently ignored    | Don't override; document at class level                                                                                |
-| Overriding without re-declaring `@Get(...)`        | Route 404s for everyone, no error logged | Don't override                                                                                                         |
-| `downloadRoles: []`                                | Warning logged, default applied          | Use a non-empty array; `[]` would read as "no roles required" and open the route                                       |
-| Expecting `metadata` back from `getFileInfo()`     | `undefined`                              | Use `getRawFileInfo()` inside `checkRights()`                                                                          |
-| `downloadRoles: ['member']` with multiTenancy      | Works from code, fails from `<img>`      | Both file classes carry `@SkipTenantCheck()`; roles resolve against `user.roles`. Use `checkRights()` for tenant rules |
-| Signed-in user can upload via TUS but not download | 403 on their own file                    | `tus.roles` and `file.downloadRoles` are separate. Add an owner to the metadata and authorize per file                 |
-| Rule narrows only `'id'` / `'filename'`            | `findFileInfo()` returns every file      | Refuse `'filterArgs'`; force a per-user filter server-side with `{ force: true }`                                      |
-| Treating the file id as unguessable                | Enumerable inventory                     | Ids share a per-process random part and an incrementing counter — authorize every read, do not rely on the id          |
-| Approving the caller's `filterArgs` as "narrowed"  | Bypass via a different filter shape      | `filterArgs` is client-controlled. Override the filter; never approve it                                               |
+| Mistake                                                        | Symptom                                     | Fix                                                                                                                    |
+| -------------------------------------------------------------- | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
+| Overriding `getFileById` to add Swagger decorators             | `file.downloadRoles` silently ignored       | Don't override; document at class level                                                                                |
+| Overriding without re-declaring `@Get(...)`                    | Route 404s for everyone, no error logged    | Don't override                                                                                                         |
+| `downloadRoles: []`                                            | Warning logged, default applied             | Use a non-empty array; `[]` would read as "no roles required" and open the route                                       |
+| Expecting `metadata` back from `getFileInfo()`                 | `undefined`                                 | Use `getRawFileInfo()` inside `checkRights()`                                                                          |
+| `downloadRoles: ['member']` with multiTenancy                  | Works from code, fails from `<img>`         | Both file classes carry `@SkipTenantCheck()`; roles resolve against `user.roles`. Use `checkRights()` for tenant rules |
+| Signed-in user can upload via TUS but not download             | 403 on their own file                       | `tus.roles` and `file.downloadRoles` are separate. Add an owner to the metadata and authorize per file                 |
+| Rule narrows only `'id'` / `'filename'`                        | `findFileInfo()` returns every file         | Refuse `'filterArgs'`; force a per-user filter server-side with `{ force: true }`                                      |
+| Treating the file id as unguessable                            | Enumerable inventory                        | Ids share a per-process random part and an incrementing counter — authorize every read, do not rely on the id          |
+| Approving the caller's `filterArgs` as "narrowed"              | Bypass via a different filter shape         | `filterArgs` is client-controlled. Override the filter; never approve it                                               |
+| `super(connection)` in `FileService` with `file.storage: 's3'` | Boot fails: "that storage is not available" | Forward `{ configService, s3Service }` — step 6                                                                        |
+| S3 packages in `devDependencies`                               | Production image cannot load S3             | Move them to `dependencies` — step 6                                                                                   |
+| A quota aggregating only `fs.files`                            | Reads 0 on the S3 / filesystem driver       | Aggregate `fs.files`, `filesystem-files` and `s3-files` — step 6                                                       |

package/src/core/modules/file/README.md

@@ -45,11 +45,11 @@ file: {
 
 Three equivalent options. They differ only in where the bytes end up:
 
-| Driver         | Bytes                     | Survives a restart       | Shared between replicas | Needs                              |
-| -------------- | ------------------------- | ------------------------ | ----------------------- | ---------------------------------- |
-| `'s3'`         | S3-compatible bucket      | yes                      | **yes**                 | `s3` config + `@aws-sdk/client-s3` |
-| `'gridfs'`     | MongoDB GridFS            | yes                      | yes                     | nothing beyond the database        |
-| `'filesystem'` | local disk (`storageDir`) | only on a mounted volume | **no**                  | nothing                            |
+| Driver         | Bytes                     | Survives a restart       | Shared between replicas | Needs                                                                    |
+| -------------- | ------------------------- | ------------------------ | ----------------------- | ------------------------------------------------------------------------ |
+| `'s3'`         | S3-compatible bucket      | yes                      | **yes**                 | `s3` config + `@aws-sdk/client-s3` (+ `@aws-sdk/lib-storage`, see below) |
+| `'gridfs'`     | MongoDB GridFS            | yes                      | yes                     | nothing beyond the database                                              |
+| `'filesystem'` | local disk (`storageDir`) | only on a mounted volume | **no**                  | nothing                                                                  |
 
 **Metadata always lives in the database**, whichever driver holds the bytes. Filename, content type,
 length and the custom `metadata` a per-file rule reads have to be queryable — `findFileInfo()` filters
@@ -57,6 +57,31 @@ and pages over them, `checkRights()` reads them per request. A directory listing
 that, and sidecar files would reinvent an index the database already is. So `'filesystem'` moves the
 bytes off the database, not the bookkeeping.
 
+### How an upload reaches S3
+
+No upload is read into memory on its way to S3. Which request carries it depends on whether its
+length is known:
+
+| Length                                                            | Request                                                                             | Memory                   |
+| ----------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------ |
+| known, up to 5 GiB — `size` on the upload, a TUS upload, a buffer | one streamed `PutObject`                                                            | constant                 |
+| known, above 5 GiB                                                | multipart upload via `@aws-sdk/lib-storage` (AWS S3 refuses larger single requests) | at most four parts       |
+| unknown — every GraphQL and streamed REST upload                  | multipart upload via `@aws-sdk/lib-storage`                                         | at most four 5 MiB parts |
+| unknown, and `@aws-sdk/lib-storage` not installed                 | one `PutObject` of the whole file, read into memory first                           | **the file's size**      |
+
+Install the optional peer next to `@aws-sdk/client-s3` (`pnpm add @aws-sdk/lib-storage`, same
+version). Without it uploads still work, but a large one of unknown length costs its own size in
+memory — a multi-GB file then gets the container OOM-killed — and a known length above 5 GiB goes up
+as one request, which AWS S3 refuses (MinIO and RustFS accept it). The boot log says so once when
+the package is missing.
+
+Pass `size` on a `FileUploadSource` only when it is the file's EXACT length — a `stat()` of a file on
+disk. Never pass the `Content-Length` of a multipart/form-data request: it counts the whole envelope
+and is always larger than the file, and a wrong length breaks the upload. Leaving `size` out costs
+nothing but the multipart round trips, and for data already in memory it is the better choice: the
+SDK retries the Buffer parts of a multipart upload after a transient S3 error, but not one streamed
+request.
+
 ### Choosing the driver
 
 **Set it explicitly and it is enforced.** If the chosen store is not available, the boot **fails**:

package/src/core/modules/file/core-file.service.ts

@@ -31,7 +31,7 @@ import {
   FilesystemFileHelper,
   FilesystemFileInfo,
 } from './filesystem-file.helper';
-import { S3_FILES_COLLECTION, S3FileHelper, S3FileInfo, streamToBuffer } from './s3-file.helper';
+import { S3_FILES_COLLECTION, S3FileHelper, S3FileInfo } from './s3-file.helper';
 
 /**
  * Type for checking input
@@ -160,7 +160,7 @@ export abstract class CoreFileService {
     if (!(await this.checkRights(file, { ...serviceOptions, checkInputType: 'file' }))) {
       return null;
     }
-    const { createReadStream, filename, mimetype } = await file;
+    const { createReadStream, filename, mimetype, size } = await file;
     const readStream = createReadStream();
     // Resolved ONCE for all three driver branches: under an `'owner'` / `'tenant'` preset this adds
     // the very fields the preset decides on, so an upload through this service is authorizable
@@ -176,11 +176,20 @@ export abstract class CoreFileService {
       return this.prepareOutput(fsFileInfo as unknown as CoreFileInfo, serviceOptions);
     }
     if (this.storageDriver === 's3') {
-      // Only buffer when the size is unknown: a GraphQL upload stream carries no length, and the
-      // S3 SDK needs one. Callers that know it (TUS) pass `body` + `contentLength` instead and
-      // stream straight through — see S3FileHelper.writeFile.
+      // The stream is handed to S3 as it is, never read into memory here. With a `size` the object
+      // goes up as one streamed PutObject; without one, `CoreS3Service.putObject()` uses a multipart
+      // upload, which needs no total up front (and only buffers when its optional peer dependency
+      // is missing). Reading the stream into a Buffer at this point — what this branch used to do
+      // for every upload — materialised the whole file in one process: a multi-GB upload then
+      // threw "Array buffer allocation failed" or got the container OOM-killed at 100 % progress,
+      // taking every other in-flight request with it.
+      //
+      // A GraphQL or streamed REST upload carries no trustworthy length (a multipart request's
+      // `Content-Length` counts the whole envelope), so the unknown-length case is the common one,
+      // not a fallback.
       const s3FileInfo = await S3FileHelper.writeFile(this.options.s3Service, this.s3Files, {
-        buffer: await streamToBuffer(readStream),
+        body: readStream,
+        contentLength: size, // validated in writeFile(): anything but a positive integer counts as unknown
         contentType: mimetype,
         filename,
         ...(metadata ? { metadata } : {}),

package/src/core/modules/file/interfaces/file-upload.interface.ts

@@ -15,6 +15,24 @@ export interface FileUploadSource {
   encoding?: string;
   filename: string;
   mimetype: string;
+
+  /**
+   * EXACT size of the file in bytes, when the caller knows it — otherwise leave it out.
+   *
+   * Only the S3 driver reads it: with a size the object goes up as one streamed request (as a
+   * multipart upload above 5 GiB, which AWS S3 refuses in one request), without one it goes up as a
+   * multipart upload (`@aws-sdk/lib-storage`, an optional peer — without it the
+   * file is read into memory). Omitting it is not a fallback: a GraphQL or streamed REST upload never
+   * knows its length, and that is the common case.
+   *
+   * A WRONG size breaks the upload instead of slowing it down. So never pass the `Content-Length` of a
+   * multipart/form-data request: it counts the whole envelope (boundaries, part headers, other
+   * fields) and is always larger than the file. Pass it only when it is the file's own length — a
+   * TUS upload's `upload.size`, a `stat()` of a file on disk. For data already in memory leave it
+   * out: the SDK retries the Buffer parts of a multipart upload after a transient S3 error, but not
+   * one streamed request. Anything but a positive integer is treated as unknown.
+   */
+  size?: number;
 }
 
 /**

package/src/core/modules/file/s3-file.helper.ts

@@ -30,6 +30,20 @@ export const S3_FILES_COLLECTION = 's3-files';
  */
 export type S3FileInfo = FileMetadataInfo;
 
+/**
+ * Whether an upload size can be handed to S3 as the object's `Content-Length`.
+ *
+ * Only a positive safe integer qualifies. Anything else — `undefined`, `0`, a negative number,
+ * `NaN`, a fraction — is treated as "length unknown" and goes through the multipart path, which
+ * needs no total. The direction matters: a WRONG length is not a slower upload but a broken one,
+ * whereas an unknown length only costs the multipart round trips. `0` is included in "unknown" because it is
+ * what an unset numeric field defaults to far more often than it is a real empty file — and an
+ * empty file stores correctly through the multipart path too.
+ */
+export function isKnownUploadSize(size: unknown): size is number {
+  return typeof size === 'number' && Number.isSafeInteger(size) && size > 0;
+}
+
 /**
  * Read a stream completely into a buffer
  */
@@ -70,21 +84,22 @@ export class S3FileHelper {
     const _id = new Types.ObjectId();
     const key = _id.toHexString();
 
-    // A stream with a known length streams straight through. Reading it into a Buffer first —
-    // which every caller used to do — materialises the WHOLE file in one process: a 4 GB
-    // resumable upload (well inside the 50 GB default cap) then either throws
-    // "Array buffer allocation failed" or gets the container OOM-killed at 100% progress,
-    // taking every other in-flight request with it.
+    // A stream is never read into a Buffer here. Doing so — which every caller used to do —
+    // materialises the WHOLE file in one process: a 4 GB resumable upload (well inside the 50 GB
+    // default cap) then either throws "Array buffer allocation failed" or gets the container
+    // OOM-killed at 100% progress, taking every other in-flight request with it. With a known
+    // length the stream goes up as one PutObject; without one, `putObject()` uploads it in parts.
     const body = options.body ?? options.buffer;
     if (!body) {
       throw new Error('S3FileHelper.writeFile needs either a body stream or a buffer');
     }
-    await s3Service.putObject(key, body, options.contentType, options.contentLength ?? options.buffer?.length);
+    const contentLength = isKnownUploadSize(options.contentLength) ? options.contentLength : options.buffer?.length;
+    await s3Service.putObject(key, body, options.contentType, contentLength);
 
     return S3FileHelper.recordFile(s3Service, collection, _id, {
       contentType: options.contentType,
       filename: options.filename,
-      length: options.contentLength ?? options.buffer?.length,
+      length: contentLength,
       metadata: options.metadata,
     });
   }
@@ -134,7 +149,11 @@ export class S3FileHelper {
     _id: Types.ObjectId,
     options: { contentType?: string; filename: string; length?: number; metadata?: Record<string, any> },
   ): Promise<S3FileInfo> {
-    if (!(await s3Service.objectExists(_id.toHexString()))) {
+    // One HEAD answers both questions: whether the bytes arrived, and — for a stream whose length
+    // nobody knew up front — how many there are. The stored object is the authority on its size;
+    // a length guessed by the caller (a multipart request's `Content-Length`, say) is not.
+    const stat = await s3Service.statObject(_id.toHexString());
+    if (!stat) {
       throw new Error('File uploaded but not found in S3');
     }
 
@@ -147,7 +166,7 @@ export class S3FileHelper {
       _id,
       contentType: options.contentType,
       filename: options.filename,
-      length: options.length,
+      length: options.length ?? stat.contentLength,
       metadata: options.metadata,
       // Records WHERE the bytes went, so a reader never has to probe all three
       // stores to find out. Legacy documents lack it and are handled by probing.