@patrickdk77/archive-defuse 0.1.0 → 0.1.1

package/README.md

@@ -247,6 +247,35 @@ A finding from another package scores at its `weight`. An archive's findings car
 
 archive-defuse follows the contract of pdf-defuse. `ContainedFilePlugin`, `PluginContext`, `ContainedFile`, `ContainedFileResult` and `DefuseFinding` have pdf-defuse's shapes, and `context.depth` is the depth of the file itself, with the upload at 0. `archivePlugin(options)` is the plugin the router builds, and the router puts itself in `filePlugins`, so an entry of any format comes back to it. The same `password` list goes to every package.
 
+### Detection
+
+`detectArchive(bytes, options)` and `detectArchiveSource(source, options)` tell a router whether archive-defuse reads a file, and in which format. They return `{ mimeType, extension, offset }`, or undefined. The answer comes from the same format selection the inspection runs, so routing and reading cannot disagree. The content decides, apart from Brotli below. `options.name` and `options.declaredType` change nothing else, so a .docx comes back as a ZIP, and the router asks office-defuse before it takes that answer.
+
+| Format | `mimeType` | `extension` |
+|---|---|---|
+| ZIP and ZIPX | application/zip | zip |
+| tar | application/x-tar | tar |
+| gzip | application/gzip | gz |
+| bzip2 | application/x-bzip2 | bz2 |
+| zstd | application/zstd | zst |
+| Brotli | application/x-brotli | br |
+| cabinet | application/vnd.ms-cab-compressed | cab |
+| 7z | application/x-7z-compressed | 7z |
+| xz | application/x-xz | xz |
+| .lzma | application/x-lzma | lzma |
+
+ZIPX has no format of its own here. The ZIP reader reads it, so it comes back as a ZIP.
+
+`offset` is where the archive's data starts. It is 0 when a signature starts the file. For a ZIP found from its end record, it is where the ZIP reader finds the first entry, the same number STRUCTURE/LEADING_BYTES reports, or 0 when the reader cannot open the archive. For a 7z archive after a self-extracting program, it is the signature header's offset.
+
+Three claims rest on no signature, and those answers carry `heuristic`, which says what the claim rests on. An .lzma file passes liblzma's header test. A Brotli file decodes as one stream that ends with the file, or past 32 MiB without an error. A file of zeros, at least 1024 bytes long and a multiple of 512, is an empty tar. Other runs of zeros, from 14 bytes, pass the .lzma test instead.
+
+A file the selection picks and the reader then refuses, such as a truncated ZIP, still counts, and the inspection rejects it with its own finding. RAR is detected only to be rejected, so detection does not claim it.
+
+Brotli has no signature, and a file of one or two bytes can decode as a Brotli stream. A file holding just `5` is an empty one. So detection claims Brotli only when `options.name` ends in `.br` or `options.declaredType` is application/x-brotli, and the whole file decodes. `archivePlugin` accepts exactly what detection claims, with the file's own name and declared type.
+
+`detectArchiveSource` leaves the source open. It passes read errors on, and throws with the `DEFUSE_LIMIT` code once `options.deadline` passes. Detection uses no temporary files and ignores `tempDir` and `memoryThreshold`, which the family's options carry for other packages.
+
 ## Making and reading archives
 
 archive-defuse also makes archives and reads them, with no defusing involved, for code that would otherwise use archiver, adm-zip or tar-stream.
@@ -278,6 +307,10 @@ const result = JSON.parse(await zip.text('result.json'));
 
 ZIP, tar and 7z need an entry's size before its data, so an entry given as a stream is read first, into memory up to `memoryThreshold` (8 MiB) and into a temporary file past it. ZIP entries are deflated at `level` 6. ZIP and 7z store an entry that compression gains nothing on, or that would compress past `ratioLimit`, 1,000 to 1 by default, so their archives read back under the default limits. A compressed tar and the output of `compress` have no stored form. Content that compresses past 1,000 to 1, such as a long run of zeros, stops `openArchive`, `decompress` and the inspection with LIMIT/RATIO. Raise `limits.compressionRatio` to read it.
 
+`streaming: true` writes a ZIP archive the way archiver does, with no temporary file. Each file is deflated as its data arrives and goes straight to the sink, and a data descriptor after the data carries its CRC and sizes. Only zlib's buffers are held. A file with `store: true` and a link's target are held in memory, because a stored entry keeps its sizes in its header, and an empty file is stored. Two things of the default mode are lost. A file deflate cannot shrink is deflated anyway, and `ratioLimit` does not apply, since the entry is out before its ratio is known. An entry of 4 GiB or more gets eight-byte sizes in its descriptor. tar and 7z need each entry's size before its data, so `streaming` writes ZIP only. A 26 MB sheet given one row per chunk takes 0.75 seconds this way, and 1.2 in the default mode.
+
+Small chunks of an entry are gathered into 64 KiB pieces before they are compressed, so data given one row at a time compresses as fast as data given in large pieces.
+
 `createArchive` throws a PackError, before it writes the entry, for a name the inspection would change. That covers an empty name, a NUL, an absolute path, a drive letter, a `..` part with or without dots and spaces after it, an empty or `.` part, a backslash, a control character or one that reorders text, and a part Windows cannot create, such as `con.txt`, `report:v2.txt` or a part that ends in a dot or a space. It also refuses a name that leaves its folder once NFKC normalization or Windows best fit changes it. NFKC turns two fullwidth full stops, U+FF0E, into `..`. Best fit turns a yen sign, U+00A5, into a backslash when a program on Japanese Windows converts the name to code page 932. A name with a yen sign is still written unless that conversion takes it out of its folder. Duplicate names are written as given.
 
 When an entry's stream fails, the sink is aborted instead of closed, and the Readable is destroyed with the error, so an upload or an HTTP response never ends with a partial archive that looks whole.

package/dist/detect.d.ts

@@ -0,0 +1,10 @@
+import type { ByteSource, DetectOptions, FormatDetection } from './types';
+/**
+ * Whether archive-defuse reads `source`, and in which format: what the inspection's own format selection picks, from
+ * the content alone, except that Brotli, which has no signature, counts only when the name or the declared type says
+ * Brotli. A format detected only to be rejected, such as RAR, is not claimed. A ZIP archive found from its end record starts where the ZIP reader finds its first entry, or at
+ * 0 when the reader cannot open it. The source stays open.
+ */
+export declare function detectArchiveSource(source: ByteSource, options?: DetectOptions): Promise<FormatDetection | undefined>;
+/** `detectArchiveSource` over bytes in memory. */
+export declare function detectArchive(bytes: Uint8Array, options?: DetectOptions): Promise<FormatDetection | undefined>;

package/dist/detect.js

@@ -0,0 +1,95 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.detectArchiveSource = detectArchiveSource;
+exports.detectArchive = detectArchive;
+const engine_1 = require("./engine");
+const errors_1 = require("./errors");
+const formats_1 = require("./formats");
+const io_1 = require("./io");
+const sniff_1 = require("./sniff");
+const zip_1 = require("./zip");
+/** Each format's MIME type and extension, the names sniff.ts gives them. */
+const TYPES = {
+    zip: { mimeType: 'application/zip', extension: 'zip' },
+    tar: { mimeType: 'application/x-tar', extension: 'tar' },
+    gzip: { mimeType: 'application/gzip', extension: 'gz' },
+    bzip2: { mimeType: 'application/x-bzip2', extension: 'bz2' },
+    zstd: { mimeType: 'application/zstd', extension: 'zst' },
+    brotli: { mimeType: 'application/x-brotli', extension: 'br' },
+    cab: { mimeType: 'application/vnd.ms-cab-compressed', extension: 'cab' },
+    '7z': { mimeType: 'application/x-7z-compressed', extension: '7z' },
+    xz: { mimeType: 'application/x-xz', extension: 'xz' },
+    lzma: { mimeType: 'application/x-lzma', extension: 'lzma' },
+};
+/**
+ * `source` without its close(), checking the deadline before every call and keeping the first error any call throws.
+ * The Brotli test catches every error, so the kept one is how a failed read or the deadline still gets out.
+ */
+function watch(source, deadline) {
+    const failed = {};
+    const keep = async (op) => {
+        try {
+            (0, errors_1.checkTime)(deadline);
+            return await op();
+        }
+        catch (e) {
+            failed.error ??= e;
+            throw e;
+        }
+    };
+    return { source: { size: () => keep(() => source.size()), read: (offset, length) => keep(() => source.read(offset, length)) }, failed };
+}
+const BROTLI_TYPES = new Set(['application/x-brotli', 'application/brotli']);
+const saysBrotli = (o) => (0, sniff_1.typeFromName)(o.name) === 'application/x-brotli' || BROTLI_TYPES.has((o.declaredType ?? '').toLowerCase().split(';')[0].trim());
+/**
+ * Whether archive-defuse reads `source`, and in which format: what the inspection's own format selection picks, from
+ * the content alone, except that Brotli, which has no signature, counts only when the name or the declared type says
+ * Brotli. A format detected only to be rejected, such as RAR, is not claimed. A ZIP archive found from its end record starts where the ZIP reader finds its first entry, or at
+ * 0 when the reader cannot open it. The source stays open.
+ */
+async function detectArchiveSource(source, options = {}) {
+    const watched = watch(source, options.deadline);
+    const pass = (e) => {
+        if (watched.failed.error !== undefined)
+            throw watched.failed.error;
+        if ((0, errors_1.stopsTree)(e) || (0, errors_1.isIoError)(e))
+            throw e;
+    };
+    let picked;
+    try {
+        (0, errors_1.checkTime)(options.deadline);
+        picked = await (0, formats_1.select)(watched.source);
+        pass(undefined);
+    }
+    catch (e) {
+        pass(e);
+        // Any other failure, as in an inspection, means archive-defuse does not read the file.
+        return undefined;
+    }
+    if (picked === undefined)
+        return undefined;
+    // Brotli has no signature, and a one-byte file such as "7" decodes as an empty stream. Without a name or declared
+    // type that says Brotli, archive-defuse cannot know the file is one, so it does not claim it.
+    if (picked.format === 'brotli' && !saysBrotli(options))
+        return undefined;
+    let offset = picked.offset;
+    if (offset === undefined) {
+        try {
+            offset = (await zip_1.ZipArchive.open(watched.source, engine_1.DEFAULT_LIMITS, options.deadline)).start;
+            pass(undefined);
+        }
+        catch (e) {
+            pass(e);
+            // The selection picked ZIP, so the file is one, which the inspection rejects with its own finding.
+            offset = 0;
+        }
+    }
+    const found = { ...TYPES[picked.format], offset };
+    if (picked.heuristic !== undefined)
+        found.heuristic = picked.heuristic;
+    return found;
+}
+/** `detectArchiveSource` over bytes in memory. */
+function detectArchive(bytes, options = {}) {
+    return detectArchiveSource((0, io_1.bufferSource)(bytes), options);
+}

package/dist/formats.d.ts

@@ -2,10 +2,20 @@ import type { TempDir } from './io';
 import type { Archive, Entry } from './model';
 import type { Limits } from './stream-meter';
 import { type ArchiveFormat, type ByteSource } from './types';
+/** The format `select` picks, where its data starts, and what a pick without a signature at the start rests on. */
+export interface Selection {
+    format: ArchiveFormat;
+    /** Where the format's data starts. Absent for a ZIP archive found from its end record, whose start the ZIP reader finds. */
+    offset?: number;
+    /** Set when no signature picked the format, and says what did. */
+    heuristic?: string;
+}
 /**
  * The format the content has, from its first bytes. A file that starts with no signature of ours can still be a ZIP
  * archive with data in front, which its end record shows, or Brotli, which has no signature and is tried last.
  */
+export declare function select(source: ByteSource): Promise<Selection | undefined>;
+/** The format `select` picks. */
 export declare function detect(source: ByteSource): Promise<ArchiveFormat | undefined>;
 /** Opens the archive in the format `detect` found. A file in no format of ours throws the UNPARSEABLE refusal. */
 export declare function openArchive(source: ByteSource, limits: Limits, deadline: number | undefined, passwords: string[], temp?: {

package/dist/formats.js

@@ -1,5 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
+exports.select = select;
 exports.detect = detect;
 exports.openArchive = openArchive;
 const cab_1 = require("./cab");
@@ -24,42 +25,50 @@ function otherFormat(b) {
  * The format the content has, from its first bytes. A file that starts with no signature of ours can still be a ZIP
  * archive with data in front, which its end record shows, or Brotli, which has no signature and is tried last.
  */
-async function detect(source) {
+async function select(source) {
     const size = await source.size();
     const head = await source.read(0, Math.min(1024, size));
+    const atStart = (format) => ({ format, offset: 0 });
     if (isZipStart(head))
-        return 'zip';
+        return atStart('zip');
     if (startsWith(head, lzma_sniff_1.XZ_MAGIC))
-        return 'xz';
+        return atStart('xz');
     if (startsWith(head, lzma_sniff_1.SEVENZIP_MAGIC))
-        return '7z';
+        return atStart('7z');
     if ((0, single_1.isGzip)(head))
-        return 'gzip';
+        return atStart('gzip');
     if ((0, single_1.isBzip2)(head))
-        return 'bzip2';
+        return atStart('bzip2');
     if ((0, zstd_1.isZstdMagic)(head))
-        return 'zstd';
+        return atStart('zstd');
     if ((0, cab_1.isCab)(head))
-        return 'cab';
+        return atStart('cab');
     if ((0, tar_1.isTar)(head))
-        return 'tar';
+        return atStart('tar');
     // An archive with no entries is its two end blocks, and a tar writer pads it with more zeros.
-    if (size >= 1024 && size % 512 === 0 && head.every(b => b === 0))
-        return 'tar';
+    if (size >= 1024 && size % 512 === 0 && head.every(b => b === 0)) {
+        return { format: 'tar', offset: 0, heuristic: 'no tar header: the file is zeros, at least 1024 bytes and a multiple of 512, as an empty tar is' };
+    }
     if (otherFormat(head) !== undefined)
         return undefined;
     // An .lzma header has no magic, and 14 zero bytes pass liblzma's own test, so a ZIP found from its end comes first.
     if (await (0, zip_1.hasZipEnd)(source))
-        return 'zip';
+        return { format: 'zip' };
     if ((0, lzma_sniff_1.looksLikeLzma)(head))
-        return 'lzma';
+        return { format: 'lzma', offset: 0, heuristic: "no signature: the 13-byte header and the first data byte pass liblzma's test for an .lzma file" };
     // A 7z archive after other data, such as the program of a self-extracting archive.
-    if ((0, lzma_sniff_1.detectLzmaFormat)(await source.read(0, Math.min(size, lzma_sniff_1.SEVENZIP_SEARCH)))?.format === '7z')
-        return '7z';
-    if (await (0, single_1.looksLikeBrotli)(source))
-        return 'brotli';
+    const sfx = (0, lzma_sniff_1.detectLzmaFormat)(await source.read(0, Math.min(size, lzma_sniff_1.SEVENZIP_SEARCH)));
+    if (sfx?.format === '7z')
+        return { format: '7z', offset: sfx.offset };
+    if (await (0, single_1.looksLikeBrotli)(source)) {
+        return { format: 'brotli', offset: 0, heuristic: 'no signature: the file decodes as one Brotli stream that ends where the file does, or past 32 MiB without an error' };
+    }
     return undefined;
 }
+/** The format `select` picks. */
+async function detect(source) {
+    return (await select(source))?.format;
+}
 /** Opens the archive in the format `detect` found. A file in no format of ours throws the UNPARSEABLE refusal. */
 async function openArchive(source, limits, deadline, passwords, temp) {
     const format = await detect(source);

package/dist/index.d.ts

@@ -1,3 +1,4 @@
+export { detectArchive, detectArchiveSource } from './detect';
 export { disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource } from './engine';
 export { allFindingSpecs } from './findings';
 export { bufferSink, bufferSource, fileSink, fileSource, writableSink } from './io';

package/dist/index.js

@@ -14,7 +14,10 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
     for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.openArchive = exports.decompress = exports.ArchiveReadError = exports.scoreFindings = exports.DEFAULT_BANDS = exports.COMBINATIONS = exports.passThrough = exports.archivePlugin = exports.PackError = exports.createArchiveStream = exports.createArchiveBytes = exports.createArchive = exports.compressTo = exports.compressStream = exports.compress = exports.writableSink = exports.fileSource = exports.fileSink = exports.bufferSource = exports.bufferSink = exports.allFindingSpecs = exports.inspectArchiveSource = exports.inspectArchive = exports.disarmArchiveSource = exports.disarmArchive = void 0;
+exports.openArchive = exports.decompress = exports.ArchiveReadError = exports.scoreFindings = exports.DEFAULT_BANDS = exports.COMBINATIONS = exports.passThrough = exports.archivePlugin = exports.PackError = exports.createArchiveStream = exports.createArchiveBytes = exports.createArchive = exports.compressTo = exports.compressStream = exports.compress = exports.writableSink = exports.fileSource = exports.fileSink = exports.bufferSource = exports.bufferSink = exports.allFindingSpecs = exports.inspectArchiveSource = exports.inspectArchive = exports.disarmArchiveSource = exports.disarmArchive = exports.detectArchiveSource = exports.detectArchive = void 0;
+var detect_1 = require("./detect");
+Object.defineProperty(exports, "detectArchive", { enumerable: true, get: function () { return detect_1.detectArchive; } });
+Object.defineProperty(exports, "detectArchiveSource", { enumerable: true, get: function () { return detect_1.detectArchiveSource; } });
 var engine_1 = require("./engine");
 Object.defineProperty(exports, "disarmArchive", { enumerable: true, get: function () { return engine_1.disarmArchive; } });
 Object.defineProperty(exports, "disarmArchiveSource", { enumerable: true, get: function () { return engine_1.disarmArchiveSource; } });

package/dist/index.mjs

@@ -1,3 +1,3 @@
 import cjs from './index.js';
-export const { disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource, allFindingSpecs, bufferSink, bufferSource, fileSink, fileSource, writableSink, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, PackError, archivePlugin, passThrough, COMBINATIONS, DEFAULT_BANDS, scoreFindings, ArchiveReadError, decompress, openArchive, ArchiveCategory, ArchiveDetail, ArchiveRisk } = cjs;
+export const { detectArchive, detectArchiveSource, disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource, allFindingSpecs, bufferSink, bufferSource, fileSink, fileSource, writableSink, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, PackError, archivePlugin, passThrough, COMBINATIONS, DEFAULT_BANDS, scoreFindings, ArchiveReadError, decompress, openArchive, ArchiveCategory, ArchiveDetail, ArchiveRisk } = cjs;
 export default cjs;

package/dist/pack.d.ts

@@ -50,6 +50,13 @@ export interface PackOptions {
     memoryThreshold?: number;
     /** Stops the work when aborted: the sink is aborted and the call throws the signal's reason. */
     signal?: AbortSignal;
+    /**
+     * ZIP only: write each file as its data arrives, deflated on the way, with its CRC and sizes in a data descriptor
+     * after the data. Nothing goes to a temporary file. An entry with `store: true` is held in memory, since a stored
+     * entry keeps its sizes in its header. `ratioLimit` does not apply, since an entry is written before its ratio is
+     * known.
+     */
+    streaming?: boolean;
 }
 /** Thrown for an entry this package will not write, before anything of the entry is written. */
 export declare class PackError extends Error {

package/dist/pack.js

@@ -28,6 +28,8 @@ const enc = new TextEncoder();
 /** The reading default of `limits.compressionRatio`. */
 const DEFAULT_RATIO = 1000;
 const toBytes = (c) => (typeof c === 'string' ? enc.encode(c) : c);
+/** Bytes that small chunks of an entry are gathered into before they go on. */
+const BATCH = 64 * 1024;
 /** `chunks`, with `check` before each one, so an aborted signal stops a long read. */
 async function* checked(chunks, check) {
     for await (const c of chunks) {
@@ -45,11 +47,28 @@ async function* chunksOf(data) {
             yield b;
         return;
     }
+    // Small chunks, such as one row of a sheet each, go to the compressor in pieces of BATCH bytes. zlib does the same
+    // work either way, and each write costs as much as a large one.
+    let pending = [];
+    let held = 0;
     for await (const c of data) {
         const b = toBytes(c);
-        if (b.length)
+        if (!b.length)
+            continue;
+        if (held === 0 && b.length >= BATCH) {
             yield b;
+            continue;
+        }
+        pending.push(b);
+        held += b.length;
+        if (held >= BATCH) {
+            yield Buffer.concat(pending, held);
+            pending = [];
+            held = 0;
+        }
     }
+    if (held)
+        yield Buffer.concat(pending, held);
 }
 /** Why `createArchive` refuses a name, for each name rule of the inspection that changes a name. */
 const REFUSED = {
@@ -180,6 +199,10 @@ async function writeZipEntries(entries, sink, options, temp, check) {
             await out.entry(info, () => chunksOf(undefined), s.name);
             continue;
         }
+        if (options.streaming) {
+            await streamZipEntry(out, s, info, level, check);
+            continue;
+        }
         // The content is read once, into memory or a temporary file, and deflated from there, so it can be stored instead
         // when deflate gains nothing or passes the ratio limit.
         const { spill: raw, crc } = await spillOf(s, temp, threshold, check);
@@ -206,6 +229,56 @@ async function writeZipEntries(entries, sink, options, temp, check) {
     }
     await out.end(enc.encode(options.comment ?? ''));
 }
+/**
+ * Writes one entry of a streaming ZIP archive. A file is deflated as its data arrives and gets a data descriptor; at
+ * level 0, deflate writes stored blocks. An empty file is stored with its sizes. A link's target and a file the caller
+ * stores are held in memory, so their headers carry their sizes, since a stored entry with a data descriptor is how a
+ * second, hidden entry gets past a reader of the central directory.
+ */
+async function streamZipEntry(out, s, info, level, check) {
+    const it = checked(await contentOf(s), check)[Symbol.asyncIterator]();
+    let first;
+    try {
+        first = await it.next();
+    }
+    catch (e) {
+        await it.return?.(undefined);
+        throw e;
+    }
+    if (s.kind === 'symlink' || s.entry.store === true || first.done) {
+        const parts = [];
+        let crc = 0;
+        try {
+            for (let r = first; !r.done; r = await it.next()) {
+                crc = (0, crc32_1.crc32)(r.value, crc);
+                parts.push(r.value);
+            }
+        }
+        finally {
+            await it.return?.(undefined);
+        }
+        const data = Buffer.concat(parts);
+        Object.assign(info, { method: 0, versionNeeded: 10, crc, size: data.length, compressedSize: data.length });
+        await out.entry(info, () => chunksOf(data), s.name);
+        return;
+    }
+    let crc = 0;
+    let size = 0;
+    const raw = async function* () {
+        try {
+            for (let r = first; !r.done; r = await it.next()) {
+                crc = (0, crc32_1.crc32)(r.value, crc);
+                size += r.value.length;
+                yield r.value;
+            }
+        }
+        finally {
+            await it.return?.(undefined);
+        }
+    };
+    Object.assign(info, { method: 8, versionNeeded: 20 });
+    await out.streamEntry(info, (0, codecs_1.deflateRaw)(raw(), level), () => ({ crc, size }));
+}
 // ---------------------------------------------------------------------------------------------------------------
 // tar
 const TAR_TYPES = { file: '0', directory: '5', symlink: '2' };
@@ -427,6 +500,9 @@ async function deliver(produce, sink) {
 async function createArchive(format, entries, sink, options = {}) {
     if (!ARCHIVE_FORMATS.has(format))
         throw new PackError(`Unknown archive format ${String(format)}`);
+    // tar needs each entry's size before its data, and 7z writes its header from every entry after the data.
+    if (options.streaming && format !== 'zip')
+        throw new PackError(`streaming writes ZIP archives only, not ${format}`);
     const now = new Date();
     const temp = new io_1.TempDir(options.tempDir);
     const check = () => options.signal?.throwIfAborted();

package/dist/plugins.js

@@ -2,8 +2,8 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.passThrough = passThrough;
 exports.archivePlugin = archivePlugin;
+const detect_1 = require("./detect");
 const engine_1 = require("./engine");
-const formats_1 = require("./formats");
 const score_1 = require("./score");
 const sniff_1 = require("./sniff");
 /**
@@ -36,7 +36,8 @@ function archivePlugin(options) {
     return {
         kind: 'file',
         name: 'archive',
-        accepts: async (f) => (await (0, formats_1.detect)(f.source)) !== undefined,
+        // It takes what detectArchive claims, which is what the inspection reads, Brotli included: the content decides.
+        accepts: async (f) => (await (0, detect_1.detectArchiveSource)(f.source, { name: f.name, declaredType: f.declaredType })) !== undefined,
         async process(file, sink, context) {
             const run = options ?? context[engine_1.RUN_OPTIONS] ?? {};
             const r = await (0, engine_1.disarmAtDepth)(file.source, sink, run, context.depth, context.deadline);

package/dist/types.d.ts

@@ -205,6 +205,33 @@ export interface ArchiveLimits {
     /** Milliseconds for the whole run. Not enforced unless set. */
     timeMs?: number;
 }
+/**
+ * A file's format as a package of the family finds it, among the formats that package reads. The family's router asks
+ * each package and routes the file by the answers.
+ */
+export interface FormatDetection {
+    /** The format's MIME type, such as application/zip. */
+    mimeType: string;
+    /** The format's usual extension, lowercase and without the dot. */
+    extension: string;
+    /** Where the format starts in the file: 0, or the offset of its data when other data comes first. */
+    offset: number;
+    /** Set when the match rests on more than the content, and says what it rests on. */
+    heuristic?: string;
+}
+/** What a detect function may know of a file besides its content. */
+export interface DetectOptions {
+    /** The file's name, as the upload or the container that holds it gives it. */
+    name?: string;
+    /** The MIME type the upload or the container declares. */
+    declaredType?: string;
+    /** The Date.now() value detection must finish by, as `PluginContext.deadline`. */
+    deadline?: number;
+    /** Folder for temporary files. Defaults to `os.tmpdir()`. */
+    tempDir?: string;
+    /** Bytes held in memory before a temporary file takes over. */
+    memoryThreshold?: number;
+}
 /** What a contained file plugin learns of the run, with nothing format-specific, so any package of the family can make one. */
 export interface PluginContext {
     /** The depth of the file itself: 0 for the upload, 1 for a file inside it, and so on, whatever the formats. */

package/dist/zip-writer.d.ts

@@ -34,6 +34,16 @@ export declare class ZipOutput {
     put(b: Uint8Array): Promise<void>;
     /** Writes one entry: its local header, then `chunks`, which must hold exactly `info.compressedSize` bytes. */
     entry(info: Written, chunks: () => AsyncIterable<Uint8Array>, label: string): Promise<void>;
+    /**
+     * Writes one entry whose sizes are known only once its data has gone out: a local header with the data descriptor
+     * flag and no CRC or sizes, then `chunks`, the compressed data, then a data descriptor with the CRC and the sizes.
+     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended. An entry of 4 GiB or more gets
+     * eight-byte sizes in its descriptor and a ZIP64 field in the central directory, as archiver writes one.
+     */
+    streamEntry(info: Written, chunks: AsyncIterable<Uint8Array>, totals: () => {
+        crc: number;
+        size: number;
+    }): Promise<void>;
     /** Writes the central directory and the end records, with the archive's comment. */
     end(comment: Uint8Array): Promise<void>;
 }

package/dist/zip-writer.js

@@ -84,39 +84,45 @@ class ZipOutput {
         }
         if (written !== info.compressedSize)
             throw new Error(`Entry ${label} has ${written} compressed bytes, not ${info.compressedSize}`);
-        const far = localOffset >= MAX32;
-        const z64 = new Builder();
-        if (info.size >= MAX32)
-            z64.u64(info.size);
-        if (info.compressedSize >= MAX32)
-            z64.u64(info.compressedSize);
-        if (far)
-            z64.u64(localOffset);
-        const z64Bytes = z64.build();
-        const needed = z64Bytes.length ? Math.max(45, info.versionNeeded) : info.versionNeeded;
-        const c = new Builder()
-            .u32(0x02014b50)
-            .u16(info.versionMadeBy)
-            .u16(needed)
-            .u16(info.flags)
+        this.central.push(centralRecord(info, localOffset));
+    }
+    /**
+     * Writes one entry whose sizes are known only once its data has gone out: a local header with the data descriptor
+     * flag and no CRC or sizes, then `chunks`, the compressed data, then a data descriptor with the CRC and the sizes.
+     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended. An entry of 4 GiB or more gets
+     * eight-byte sizes in its descriptor and a ZIP64 field in the central directory, as archiver writes one.
+     */
+    async streamEntry(info, chunks, totals) {
+        const flags = info.flags | 0x0008;
+        const localOffset = this.offset;
+        const local = new Builder()
+            .u32(0x04034b50)
+            .u16(info.versionNeeded)
+            .u16(flags)
             .u16(info.method)
             .u16(info.time)
             .u16(info.date)
-            .u32(info.crc)
-            .u32(info.compressedSize >= MAX32 ? MAX32 : info.compressedSize)
-            .u32(info.size >= MAX32 ? MAX32 : info.size)
+            .u32(0)
+            .u32(0)
+            .u32(0)
             .u16(info.name.length)
-            .u16(info.extras.length + (z64Bytes.length ? z64Bytes.length + 4 : 0))
-            .u16(info.comment.length)
-            .u16(0)
-            .u16(info.internalAttr)
-            .u32(info.externalAttr)
-            .u32(far ? MAX32 : localOffset)
-            .bytes(info.name);
-        if (z64Bytes.length)
-            c.u16(0x0001).u16(z64Bytes.length).bytes(z64Bytes);
-        c.bytes(info.extras).bytes(info.comment);
-        this.central.push(c.build());
+            .u16(info.extras.length)
+            .bytes(info.name)
+            .bytes(info.extras);
+        await this.put(local.build());
+        let compressedSize = 0;
+        for await (const c of chunks) {
+            compressedSize += c.length;
+            await this.put(c);
+        }
+        const { crc, size } = totals();
+        const descriptor = new Builder().u32(0x08074b50).u32(crc);
+        if (size >= MAX32 || compressedSize >= MAX32)
+            descriptor.u64(compressedSize).u64(size);
+        else
+            descriptor.u32(compressedSize).u32(size);
+        await this.put(descriptor.build());
+        this.central.push(centralRecord({ ...info, flags, crc, size, compressedSize }, localOffset));
     }
     /** Writes the central directory and the end records, with the archive's comment. */
     async end(comment) {
@@ -145,6 +151,42 @@ class ZipOutput {
     }
 }
 exports.ZipOutput = ZipOutput;
+/** The central directory record of an entry whose local header starts at `localOffset`. */
+function centralRecord(info, localOffset) {
+    const far = localOffset >= MAX32;
+    const z64 = new Builder();
+    if (info.size >= MAX32)
+        z64.u64(info.size);
+    if (info.compressedSize >= MAX32)
+        z64.u64(info.compressedSize);
+    if (far)
+        z64.u64(localOffset);
+    const z64Bytes = z64.build();
+    const needed = z64Bytes.length ? Math.max(45, info.versionNeeded) : info.versionNeeded;
+    const c = new Builder()
+        .u32(0x02014b50)
+        .u16(info.versionMadeBy)
+        .u16(needed)
+        .u16(info.flags)
+        .u16(info.method)
+        .u16(info.time)
+        .u16(info.date)
+        .u32(info.crc)
+        .u32(info.compressedSize >= MAX32 ? MAX32 : info.compressedSize)
+        .u32(info.size >= MAX32 ? MAX32 : info.size)
+        .u16(info.name.length)
+        .u16(info.extras.length + (z64Bytes.length ? z64Bytes.length + 4 : 0))
+        .u16(info.comment.length)
+        .u16(0)
+        .u16(info.internalAttr)
+        .u32(info.externalAttr)
+        .u32(far ? MAX32 : localOffset)
+        .bytes(info.name);
+    if (z64Bytes.length)
+        c.u16(0x0001).u16(z64Bytes.length).bytes(z64Bytes);
+    c.bytes(info.extras).bytes(info.comment);
+    return c.build();
+}
 /** Writes `zip` again with the entries `plan` keeps. */
 async function writeZip(zip, plan, sink, context) {
     const out = new ZipOutput(sink);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@patrickdk77/archive-defuse",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Sanitize zip, tar, gzip, bzip2, zstd, Brotli, cab, 7z, xz and lzma archives. Detects path traversal, links out of the tree, device files, duplicate names, overlapping entries, hidden and prepended data, encrypted entries and compression bombs, hands every entry to file plugins, scores the risk, and writes a defused archive in the same format with no runtime dependencies.",
   "license": "MIT AND Apache-2.0",
   "author": "patrickdk77",