pompelmi 1.4.0 → 1.6.0

package/wiki/local-vs-tcp-mode.md

@@ -0,0 +1,179 @@
+# Local vs TCP Mode
+
+pompelmi supports two scanning modes. Which one you use is controlled entirely by the options you pass — no configuration files, no environment flags in pompelmi itself.
+
+---
+
+## Summary
+
+| | Local mode | TCP mode |
+|---|---|---|
+| **How it works** | Spawns `clamscan` as a child process | Streams file to `clamd` daemon over TCP |
+| **ClamAV requirement** | `clamscan` binary in PATH | Running `clamd` daemon reachable over TCP |
+| **Startup time** | Slow — loads virus DB each invocation | Fast — daemon keeps DB in memory |
+| **Throughput** | Low — one process per scan | High — persistent connection |
+| **Disk I/O** | Reads file from disk | Reads file from disk (or buffer/stream with no disk) |
+| **Docker** | Requires ClamAV in app container | Use ClamAV as a sidecar |
+| **Zero-copy scan** | Not possible | `scanBuffer()` and `scanStream()` with no disk I/O |
+
+---
+
+## Enabling local mode
+
+Do not pass `host` or `port`. pompelmi spawns `clamscan --no-summary <filePath>`:
+
+```js
+const { scan, Verdict } = require('pompelmi');
+
+// Local mode — no options, or empty options
+const result = await scan('/uploads/file.pdf');
+const result = await scan('/uploads/file.pdf', {});
+```
+
+`clamscan` must be in `PATH`. Install it with:
+
+```bash
+# macOS
+brew install clamav && freshclam
+
+# Linux
+sudo apt-get install -y clamav && sudo freshclam
+```
+
+---
+
+## Enabling TCP mode
+
+Pass `host` (and optionally `port`) to any scan function:
+
+```js
+const result = await scan('/uploads/file.pdf', {
+  host:    '127.0.0.1',
+  port:    3310,           // default 3310
+  timeout: 30_000,         // socket idle timeout ms, default 15000
+});
+```
+
+Setting `host` switches all four functions — `scan`, `scanBuffer`, `scanStream`, `scanDirectory` — to TCP mode.
+
+---
+
+## How local mode works
+
+```
+pompelmi              OS
+   │                   │
+   ├── spawn clamscan ─┤
+   │                   ├── load virus DB (~300 MB) into memory
+   │                   ├── scan file
+   │                   ├── exit 0 / 1 / 2
+   ├── read exit code ─┘
+   │
+   └── resolve Verdict
+```
+
+Each scan call:
+1. Spawns a new `clamscan` process
+2. `clamscan` loads the full virus database into memory
+3. Scans the file
+4. Exits with code 0 (clean), 1 (malicious), or 2 (error)
+5. pompelmi maps the exit code to a Verdict Symbol
+
+**Typical latency:** 400–800 ms per scan (dominated by database load time).
+
+---
+
+## How TCP mode works
+
+```
+pompelmi              clamd daemon
+   │                   │
+   ├── TCP connect ───►│ (keep-alive daemon)
+   ├── INSTREAM ───────┤
+   ├── stream chunks ──┤ scan in memory
+   │                   ├── "stream: OK" / "stream: X FOUND"
+   ├── read response ──┘
+   │
+   └── resolve Verdict
+```
+
+Each scan call:
+1. Opens a TCP connection to clamd
+2. Sends `zINSTREAM\0` command
+3. Streams the file in 64 KB chunks, each prefixed with a 4-byte big-endian length header
+4. Sends 4 zero bytes to signal end of stream
+5. Reads the response line
+6. Maps response to Verdict Symbol
+
+**Typical latency:** 5–50 ms per scan (clamd keeps DB in memory; network is the bottleneck).
+
+---
+
+## Performance comparison
+
+| Metric | Local mode | TCP mode |
+|--------|-----------|----------|
+| First scan latency | ~600 ms | ~10 ms |
+| Subsequent scan latency | ~600 ms | ~10 ms |
+| Concurrent scans (4-core) | ~4 (CPU-bound) | ~50+ |
+| Memory per scan | ~300 MB (DB load) | ~0 (clamd holds DB) |
+
+Local mode is fine for low-traffic applications (< 10 uploads/minute). TCP mode is required for any sustained upload throughput.
+
+---
+
+## Switching modes without changing application code
+
+Structure your options from environment variables so the same code runs in local mode during development and TCP mode in production:
+
+```js
+const SCAN_OPTS = process.env.CLAMAV_HOST
+  ? {
+      host:    process.env.CLAMAV_HOST,
+      port:    Number(process.env.CLAMAV_PORT) || 3310,
+      timeout: Number(process.env.CLAMAV_TIMEOUT) || 15_000,
+    }
+  : {}; // local mode — empty options
+
+const result = await scan('/uploads/file.pdf', SCAN_OPTS);
+```
+
+Set `CLAMAV_HOST=clamav` in your Docker environment; leave it unset in local development.
+
+---
+
+## Timeout differences
+
+| | Local mode | TCP mode |
+|---|---|---|
+| **`timeout` option** | Ignored | Socket idle timeout in ms |
+| **Default timeout** | OS process timeout | 15 000 ms |
+| **Timeout error** | `Process killed by signal: SIGTERM` | `clamd connection timed out after Nms` |
+
+In local mode, the process runs until `clamscan` finishes or the OS kills it. In TCP mode, pompelmi sets a socket idle timeout — if clamd stops sending data for longer than `timeout` ms, the connection is closed and the promise rejects.
+
+---
+
+## Error behaviour differences
+
+| Condition | Local mode error | TCP mode error |
+|-----------|-----------------|----------------|
+| Service unavailable | `ENOENT` (clamscan not found) | `ECONNREFUSED` |
+| Service slow | Process runs to completion | `clamd connection timed out` |
+| File not scannable | `Verdict.ScanError` (exit code 2) | `Verdict.ScanError` (error response) |
+
+---
+
+## When to use local mode
+
+- Development and testing on a developer's machine
+- Low-traffic applications (< a few uploads per minute)
+- Environments where Docker is unavailable
+- Simple scripts and one-off scans
+
+## When to use TCP mode
+
+- Production applications with concurrent uploads
+- Docker or Kubernetes deployments
+- Scanning in-memory buffers or streams with zero disk I/O
+- Environments where the application container cannot install ClamAV

package/wiki/multer-memory-storage.md

@@ -0,0 +1,166 @@
+# multer Memory Storage
+
+When you use `multer({ storage: multer.memoryStorage() })`, the uploaded file is never written to disk. It lives entirely in `req.file.buffer` as a Node.js `Buffer`. This page explains why `scan()` does not work in this case and how `scanBuffer()` solves it.
+
+---
+
+## Why `scan()` doesn't work with memoryStorage
+
+`scan()` requires a file path — it calls `clamscan <filePath>` or streams the file at that path to clamd. With `memoryStorage`, there is no path:
+
+```js
+const upload = multer({ storage: multer.memoryStorage() });
+
+app.post('/upload', upload.single('file'), async (req, res) => {
+  console.log(req.file.path);   // undefined — no file on disk
+  console.log(req.file.buffer); // <Buffer 25 50 44 ...> — file is here
+
+  // This throws "filePath must be a string"
+  await scan(req.file.path);
+});
+```
+
+---
+
+## Use `scanBuffer()` instead
+
+```js
+const express = require('express');
+const multer  = require('multer');
+const { scanBuffer, Verdict } = require('pompelmi');
+
+const app    = express();
+const upload = multer({
+  storage: multer.memoryStorage(),
+  limits:  { fileSize: 10 * 1024 * 1024 }, // 10 MB
+});
+
+const SCAN_OPTS = {
+  host: process.env.CLAMAV_HOST,
+  port: Number(process.env.CLAMAV_PORT) || 3310,
+  timeout: 30_000,
+};
+
+app.post('/upload', upload.single('file'), async (req, res) => {
+  if (!req.file) {
+    return res.status(400).json({ error: 'No file uploaded.' });
+  }
+
+  let result;
+  try {
+    result = await scanBuffer(req.file.buffer, SCAN_OPTS);
+  } catch (err) {
+    return res.status(500).json({ error: `Scan failed: ${err.message}` });
+  }
+
+  if (result === Verdict.Malicious) {
+    return res.status(422).json({ error: 'Malicious file rejected.' });
+  }
+
+  if (result === Verdict.ScanError) {
+    return res.status(422).json({ error: 'Scan incomplete — file rejected.' });
+  }
+
+  // Verdict.Clean — buffer is available for forwarding to S3, DB, etc.
+  return res.json({ ok: true, size: req.file.size });
+});
+
+app.listen(3000);
+```
+
+---
+
+## How `scanBuffer()` works in each mode
+
+| Mode | Behaviour |
+|------|-----------|
+| **TCP** (`host` set) | Buffer is streamed directly to clamd via INSTREAM — zero disk I/O on the application host. |
+| **Local** (no `host`) | Buffer is written to a temp file in `os.tmpdir()`, scanned with `clamscan`, then deleted in a `finally` block. |
+
+For `memoryStorage` workloads, TCP mode is strongly recommended: the whole point of keeping the file in memory is to avoid touching disk, and TCP mode preserves that guarantee.
+
+---
+
+## Multiple files with `upload.array()`
+
+```js
+app.post('/upload-many', upload.array('files', 10), async (req, res) => {
+  if (!req.files || req.files.length === 0) {
+    return res.status(400).json({ error: 'No files uploaded.' });
+  }
+
+  const scanResults = await Promise.allSettled(
+    req.files.map(async (file) => {
+      const verdict = await scanBuffer(file.buffer, SCAN_OPTS);
+      return { originalname: file.originalname, verdict };
+    })
+  );
+
+  const accepted = [];
+  const rejected = [];
+
+  for (const r of scanResults) {
+    if (r.status === 'rejected') {
+      rejected.push({ originalname: '?', reason: 'scan_failed' });
+      continue;
+    }
+    const { originalname, verdict } = r.value;
+    if (verdict === Verdict.Clean) {
+      accepted.push(originalname);
+    } else {
+      rejected.push({ originalname, reason: verdict.description });
+    }
+  }
+
+  if (rejected.length > 0) {
+    return res.status(422).json({ accepted, rejected });
+  }
+  return res.json({ ok: true, accepted });
+});
+```
+
+---
+
+## Memory usage considerations
+
+With `memoryStorage`, every uploaded file occupies memory for the duration of the request. For large files or high concurrency, this can exhaust the Node.js heap. Options:
+
+1. **Set a file size limit** — always set `limits.fileSize` on multer.
+2. **Use disk storage for large files** — fall back to `diskStorage` for files above a threshold.
+3. **Use `scanStream()` instead** — pipe the multipart stream directly to `scanStream()` without buffering the entire file. This requires bypassing multer and parsing multipart manually (e.g. with `busboy`).
+
+```js
+// scanStream with busboy — no full buffering
+const busboy = require('busboy');
+
+app.post('/upload-stream', (req, res) => {
+  const bb = busboy({ headers: req.headers });
+
+  bb.on('file', async (_name, fileStream, _info) => {
+    const result = await scanStream(fileStream, SCAN_OPTS);
+
+    if (result !== Verdict.Clean) {
+      return res.status(422).json({ error: result.description });
+    }
+    return res.json({ ok: true });
+  });
+
+  req.pipe(bb);
+});
+```
+
+---
+
+## After scanning: forward to S3
+
+```js
+const { PutObjectCommand } = require('@aws-sdk/client-s3');
+
+// After confirmed Verdict.Clean
+await s3.send(new PutObjectCommand({
+  Bucket:      process.env.S3_BUCKET,
+  Key:         `uploads/${Date.now()}-${req.file.originalname}`,
+  Body:        req.file.buffer,
+  ContentType: req.file.mimetype,
+}));
+```

package/wiki/nestjs-integration.md

@@ -0,0 +1,228 @@
+# NestJS Integration
+
+Integrate pompelmi into a NestJS application using a custom `PipeTransform` or `Interceptor` to scan uploaded files before they reach your controller.
+
+---
+
+## Setup
+
+```bash
+npm install pompelmi @nestjs/platform-express multer
+npm install -D @types/multer
+```
+
+---
+
+## Custom pipe: `FileScanPipe`
+
+A `PipeTransform` that receives the `Express.Multer.File` object from `FileInterceptor` and rejects it if ClamAV detects malware. Throw `BadRequestException` or `UnprocessableEntityException` as appropriate.
+
+```ts
+// file-scan.pipe.ts
+import { PipeTransform, Injectable, BadRequestException } from '@nestjs/common';
+import { scan, Verdict } from 'pompelmi';
+
+@Injectable()
+export class FileScanPipe implements PipeTransform {
+  private readonly opts = {
+    host: process.env.CLAMAV_HOST,
+    port: Number(process.env.CLAMAV_PORT) || 3310,
+    timeout: 30_000,
+  };
+
+  async transform(file: Express.Multer.File): Promise<Express.Multer.File> {
+    if (!file) throw new BadRequestException('No file uploaded.');
+
+    let result: symbol;
+    try {
+      result = await scan(file.path, this.opts);
+    } catch (err) {
+      throw new BadRequestException(`Scan failed: ${(err as Error).message}`);
+    }
+
+    if (result === Verdict.Malicious) {
+      throw new BadRequestException('Malicious file rejected.');
+    }
+
+    if (result === Verdict.ScanError) {
+      throw new BadRequestException('Scan incomplete — file rejected.');
+    }
+
+    return file;
+  }
+}
+```
+
+---
+
+## Using the pipe in a controller
+
+```ts
+// upload.controller.ts
+import {
+  Controller,
+  Post,
+  UploadedFile,
+  UseInterceptors,
+} from '@nestjs/common';
+import { FileInterceptor } from '@nestjs/platform-express';
+import { FileScanPipe } from './file-scan.pipe';
+import * as diskStorage from 'multer';
+
+@Controller('upload')
+export class UploadController {
+  @Post()
+  @UseInterceptors(
+    FileInterceptor('file', {
+      dest: './uploads',
+      limits: { fileSize: 10 * 1024 * 1024 },
+    }),
+  )
+  async uploadFile(
+    @UploadedFile(FileScanPipe) file: Express.Multer.File,
+  ) {
+    return { ok: true, filename: file.filename };
+  }
+}
+```
+
+`FileInterceptor` writes the file to `dest` before the pipe runs, so `file.path` is available.
+
+---
+
+## Memory storage with `scanBuffer()`
+
+If you use `multer.memoryStorage()` the file is in `file.buffer`. Update the pipe:
+
+```ts
+// file-scan-buffer.pipe.ts
+import { PipeTransform, Injectable, BadRequestException } from '@nestjs/common';
+import { scanBuffer, Verdict } from 'pompelmi';
+
+@Injectable()
+export class FileScanBufferPipe implements PipeTransform {
+  private readonly opts = {
+    host: process.env.CLAMAV_HOST,
+    port: Number(process.env.CLAMAV_PORT) || 3310,
+  };
+
+  async transform(file: Express.Multer.File): Promise<Express.Multer.File> {
+    if (!file) throw new BadRequestException('No file uploaded.');
+
+    const result = await scanBuffer(file.buffer, this.opts).catch((err) => {
+      throw new BadRequestException(`Scan failed: ${err.message}`);
+    });
+
+    if (result !== Verdict.Clean) {
+      throw new BadRequestException(`Upload rejected: ${result.description}`);
+    }
+
+    return file;
+  }
+}
+```
+
+Controller stays the same; swap `FileScanPipe` for `FileScanBufferPipe`:
+
+```ts
+@UseInterceptors(
+  FileInterceptor('file', {
+    storage: multer.memoryStorage(),
+    limits: { fileSize: 10 * 1024 * 1024 },
+  }),
+)
+async uploadFile(@UploadedFile(FileScanBufferPipe) file: Express.Multer.File) {
+  // file.buffer is the scanned content
+  return { ok: true, size: file.size };
+}
+```
+
+---
+
+## Module setup
+
+Register the pipe as a provider if you want it injectable via DI:
+
+```ts
+// upload.module.ts
+import { Module } from '@nestjs/common';
+import { UploadController } from './upload.controller';
+import { FileScanPipe } from './file-scan.pipe';
+
+@Module({
+  controllers: [UploadController],
+  providers: [FileScanPipe],
+})
+export class UploadModule {}
+```
+
+Or use it directly inline without DI — the `new FileScanPipe()` form works equally well:
+
+```ts
+@UploadedFile(new FileScanPipe())
+```
+
+---
+
+## Interceptor approach
+
+An `NestInterceptor` runs around the entire handler. Use this if you want to clean up the file after the handler completes regardless of outcome.
+
+```ts
+// scan.interceptor.ts
+import {
+  Injectable,
+  NestInterceptor,
+  ExecutionContext,
+  CallHandler,
+  BadRequestException,
+} from '@nestjs/common';
+import { Observable } from 'rxjs';
+import { scan, Verdict } from 'pompelmi';
+
+@Injectable()
+export class ScanInterceptor implements NestInterceptor {
+  async intercept(ctx: ExecutionContext, next: CallHandler): Promise<Observable<any>> {
+    const req = ctx.switchToHttp().getRequest();
+    const file: Express.Multer.File | undefined = req.file;
+
+    if (!file) return next.handle();
+
+    const result = await scan(file.path, {
+      host: process.env.CLAMAV_HOST,
+      port: 3310,
+    });
+
+    if (result !== Verdict.Clean) {
+      throw new BadRequestException(`Upload rejected: ${result.description}`);
+    }
+
+    return next.handle();
+  }
+}
+```
+
+Apply it at the controller or handler level:
+
+```ts
+@UseInterceptors(FileInterceptor('file', { dest: './uploads' }), ScanInterceptor)
+@Post()
+async uploadFile(@UploadedFile() file: Express.Multer.File) {
+  return { ok: true, filename: file.filename };
+}
+```
+
+---
+
+## Cleanup on rejection
+
+When the file is rejected, delete it to avoid accumulating rejected uploads on disk:
+
+```ts
+import * as fs from 'fs';
+
+if (result !== Verdict.Clean) {
+  try { fs.unlinkSync(file.path); } catch {}
+  throw new BadRequestException('Malicious file rejected.');
+}
+```