@patrickdk77/archive-defuse 0.1.1 → 0.1.2

package/README.md

@@ -303,14 +303,34 @@ const result = JSON.parse(await zip.text('result.json'));
 
 - `name`, with "/" between folders. A name that ends in "/" is a folder.
 - `data`: bytes, text written as UTF-8, a stream such as a Node Readable, or a function that returns one of those. A function runs when the entry's turn comes, so a thousand downloads do not all open at once.
+- `size`, a file's length in bytes, for a stream whose length you know, such as a GridFS file's. Content that comes to another size fails the archive.
 - `linkTarget` for a symbolic link, `mode` for the permission bits, `mtime` for the time, and `store: true` to keep a ZIP entry uncompressed, which the `mimetype` entry of an ODF document needs.
 
-ZIP, tar and 7z need an entry's size before its data, so an entry given as a stream is read first, into memory up to `memoryThreshold` (8 MiB) and into a temporary file past it. ZIP entries are deflated at `level` 6. ZIP and 7z store an entry that compression gains nothing on, or that would compress past `ratioLimit`, 1,000 to 1 by default, so their archives read back under the default limits. A compressed tar and the output of `compress` have no stored form. Content that compresses past 1,000 to 1, such as a long run of zeros, stops `openArchive`, `decompress` and the inspection with LIMIT/RATIO. Raise `limits.compressionRatio` to read it.
+ZIP and 7z need a file's CRC and sizes before its data, so a file given as a stream is read first, into memory up to `memoryThreshold` (8 MiB) and into a temporary file past it. tar needs only the size. Bytes, text and a stream with `size` go straight out, and only a stream without one is read first. ZIP entries are deflated at `level` 6. ZIP and 7z store an entry that compression gains nothing on, or that would compress past `ratioLimit`, 1,000 to 1 by default, so their archives read back under the default limits. A compressed tar and the output of `compress` have no stored form. Content that compresses past 1,000 to 1, such as a long run of zeros, stops `openArchive`, `decompress` and the inspection with LIMIT/RATIO. Raise `limits.compressionRatio` to read it.
 
-`streaming: true` writes a ZIP archive the way archiver does, with no temporary file. Each file is deflated as its data arrives and goes straight to the sink, and a data descriptor after the data carries its CRC and sizes. Only zlib's buffers are held. A file with `store: true` and a link's target are held in memory, because a stored entry keeps its sizes in its header, and an empty file is stored. Two things of the default mode are lost. A file deflate cannot shrink is deflated anyway, and `ratioLimit` does not apply, since the entry is out before its ratio is known. An entry of 4 GiB or more gets eight-byte sizes in its descriptor. tar and 7z need each entry's size before its data, so `streaming` writes ZIP only. A 26 MB sheet given one row per chunk takes 0.75 seconds this way, and 1.2 in the default mode.
+`streaming: true` writes a ZIP archive the way archiver does, with no temporary file. Each file is deflated as its data arrives and goes straight to the sink, and a data descriptor after the data carries its CRC and sizes. Only zlib's buffers are held. A file with `store: true` and a link's target are held in memory, because a stored entry keeps its sizes in its header, and an empty file is stored. Two things of the default mode are lost. A file deflate cannot shrink is deflated anyway, and `ratioLimit` does not apply, since the entry is out before its ratio is known. A file whose size is known, from `size` or because it is bytes or text, gets ZIP64 fields in its local header when deflating it could pass 4 GiB, which tells a reader the descriptor's sizes are eight bytes. A stream without a size gets none, because some readers refuse ZIP64 fields an entry does not need. If such a stream reaches 4 GiB, its descriptor holds eight-byte sizes anyway, as archiver writes it. tar needs no option to stream, and 7z writes its header from every entry after the data, so `streaming` writes ZIP only. A 26 MB sheet given one row per chunk takes 0.75 seconds this way, and 1.2 in the default mode.
 
 Small chunks of an entry are gathered into 64 KiB pieces before they are compressed, so data given one row at a time compresses as fast as data given in large pieces.
 
+**Adding entries as you go.** `createArchiveWriter(format, options)` takes entries one at a time, the way archiver's `append` and `finalize` do, for code that finds them as it goes. `add(entry)` queues an entry and returns at once, so a synchronous callback can call it. Entries go out in the order added, and each waits in memory until its turn. `finish()` says no more entries come, and resolves once the end of the archive is written. `abort(reason)` stops the archive. `stream` is the archive as a Readable. Read it while you add entries, because the end of the archive is written only as the stream is read. A refused name, a failed stream or an abort fails the archive. The writer destroys `stream` with the error, `finish` rejects with it, and a later `add` throws a PackError whose `cause` is the error.
+
+```js
+const { createArchiveWriter } = require('@patrickdk77/archive-defuse');
+
+const zip = createArchiveWriter('zip', { streaming: true });
+const sent = pipeline(zip.stream, res);
+try {
+  zip.add({ name: 'mimetype', data: 'application/vnd.oasis.opendocument.text', store: true });
+  // Each picture goes into the archive as layout finds it, before content.xml exists.
+  const xml = render(doc, picture => zip.add({ name: `Pictures/${picture.name}`, data: picture.bytes }));
+  zip.add({ name: 'content.xml', data: xml });
+  zip.finish();
+} catch (e) {
+  zip.abort(e);
+}
+await sent;
+```
+
 `createArchive` throws a PackError, before it writes the entry, for a name the inspection would change. That covers an empty name, a NUL, an absolute path, a drive letter, a `..` part with or without dots and spaces after it, an empty or `.` part, a backslash, a control character or one that reorders text, and a part Windows cannot create, such as `con.txt`, `report:v2.txt` or a part that ends in a dot or a space. It also refuses a name that leaves its folder once NFKC normalization or Windows best fit changes it. NFKC turns two fullwidth full stops, U+FF0E, into `..`. Best fit turns a yen sign, U+00A5, into a backslash when a program on Japanese Windows converts the name to code page 932. A name with a yen sign is still written unless that conversion takes it out of its folder. Duplicate names are written as given.
 
 When an entry's stream fails, the sink is aborted instead of closed, and the Readable is destroyed with the error, so an upload or an HTTP response never ends with a partial archive that looks whole.

package/dist/index.d.ts

@@ -2,7 +2,7 @@ export { detectArchive, detectArchiveSource } from './detect';
 export { disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource } from './engine';
 export { allFindingSpecs } from './findings';
 export { bufferSink, bufferSource, fileSink, fileSource, writableSink } from './io';
-export { type CompressFormat, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, type PackData, type PackEntry, PackError, type PackFormat, type PackOptions, } from './pack';
+export { type ArchiveWriter, type CompressFormat, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, createArchiveWriter, type PackData, type PackEntry, PackError, type PackFormat, type PackOptions, } from './pack';
 export { archivePlugin, passThrough } from './plugins';
 export { COMBINATIONS, DEFAULT_BANDS, scoreFindings } from './score';
 export * from './types';

package/dist/index.js

@@ -14,7 +14,7 @@ var __exportStar = (this && this.__exportStar) || function(m, exports) {
     for (var p in m) if (p !== "default" && !Object.prototype.hasOwnProperty.call(exports, p)) __createBinding(exports, m, p);
 };
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.openArchive = exports.decompress = exports.ArchiveReadError = exports.scoreFindings = exports.DEFAULT_BANDS = exports.COMBINATIONS = exports.passThrough = exports.archivePlugin = exports.PackError = exports.createArchiveStream = exports.createArchiveBytes = exports.createArchive = exports.compressTo = exports.compressStream = exports.compress = exports.writableSink = exports.fileSource = exports.fileSink = exports.bufferSource = exports.bufferSink = exports.allFindingSpecs = exports.inspectArchiveSource = exports.inspectArchive = exports.disarmArchiveSource = exports.disarmArchive = exports.detectArchiveSource = exports.detectArchive = void 0;
+exports.openArchive = exports.decompress = exports.ArchiveReadError = exports.scoreFindings = exports.DEFAULT_BANDS = exports.COMBINATIONS = exports.passThrough = exports.archivePlugin = exports.PackError = exports.createArchiveWriter = exports.createArchiveStream = exports.createArchiveBytes = exports.createArchive = exports.compressTo = exports.compressStream = exports.compress = exports.writableSink = exports.fileSource = exports.fileSink = exports.bufferSource = exports.bufferSink = exports.allFindingSpecs = exports.inspectArchiveSource = exports.inspectArchive = exports.disarmArchiveSource = exports.disarmArchive = exports.detectArchiveSource = exports.detectArchive = void 0;
 var detect_1 = require("./detect");
 Object.defineProperty(exports, "detectArchive", { enumerable: true, get: function () { return detect_1.detectArchive; } });
 Object.defineProperty(exports, "detectArchiveSource", { enumerable: true, get: function () { return detect_1.detectArchiveSource; } });
@@ -38,6 +38,7 @@ Object.defineProperty(exports, "compressTo", { enumerable: true, get: function (
 Object.defineProperty(exports, "createArchive", { enumerable: true, get: function () { return pack_1.createArchive; } });
 Object.defineProperty(exports, "createArchiveBytes", { enumerable: true, get: function () { return pack_1.createArchiveBytes; } });
 Object.defineProperty(exports, "createArchiveStream", { enumerable: true, get: function () { return pack_1.createArchiveStream; } });
+Object.defineProperty(exports, "createArchiveWriter", { enumerable: true, get: function () { return pack_1.createArchiveWriter; } });
 Object.defineProperty(exports, "PackError", { enumerable: true, get: function () { return pack_1.PackError; } });
 var plugins_1 = require("./plugins");
 Object.defineProperty(exports, "archivePlugin", { enumerable: true, get: function () { return plugins_1.archivePlugin; } });

package/dist/index.mjs

@@ -1,3 +1,3 @@
 import cjs from './index.js';
-export const { detectArchive, detectArchiveSource, disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource, allFindingSpecs, bufferSink, bufferSource, fileSink, fileSource, writableSink, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, PackError, archivePlugin, passThrough, COMBINATIONS, DEFAULT_BANDS, scoreFindings, ArchiveReadError, decompress, openArchive, ArchiveCategory, ArchiveDetail, ArchiveRisk } = cjs;
+export const { detectArchive, detectArchiveSource, disarmArchive, disarmArchiveSource, inspectArchive, inspectArchiveSource, allFindingSpecs, bufferSink, bufferSource, fileSink, fileSource, writableSink, compress, compressStream, compressTo, createArchive, createArchiveBytes, createArchiveStream, createArchiveWriter, PackError, archivePlugin, passThrough, COMBINATIONS, DEFAULT_BANDS, scoreFindings, ArchiveReadError, decompress, openArchive, ArchiveCategory, ArchiveDetail, ArchiveRisk } = cjs;
 export default cjs;

package/dist/pack.d.ts

@@ -13,6 +13,13 @@ export interface PackEntry {
      * a caller open a stream only when its turn comes.
      */
     data?: PackData | (() => PackData | Promise<PackData>);
+    /**
+     * A file's size in bytes, for content given as a stream whose length the caller knows, such as a file in a database
+     * that records it. tar writes such a file as its data arrives instead of reading it into memory or a temporary file
+     * first, and streaming ZIP gives it ZIP64 fields only when its size needs them. Content that comes to another size
+     * fails the archive.
+     */
+    size?: number;
     /** The target of a symbolic link. */
     linkTarget?: string;
     /** Permission bits. Defaults to 0644 for a file and 0755 for a folder. */
@@ -44,8 +51,9 @@ export interface PackOptions {
     /** Folder for temporary files. Defaults to `os.tmpdir()`. */
     tempDir?: string;
     /**
-     * Bytes of one entry held in memory before a temporary file takes over, default 8 MiB. ZIP, tar and 7z need an
-     * entry's size before its data, so an entry given as a stream is read into memory or a temporary file first.
+     * Bytes of one entry held in memory before a temporary file takes over, default 8 MiB. An entry given as a stream
+     * is read into memory or a temporary file before it is written when the format needs something of it first: its
+     * CRC and sizes for ZIP without `streaming` and for 7z, and its size for tar when the entry gives no `size`.
      */
     memoryThreshold?: number;
     /** Stops the work when aborted: the sink is aborted and the call throws the signal's reason. */
@@ -54,7 +62,10 @@ export interface PackOptions {
      * ZIP only: write each file as its data arrives, deflated on the way, with its CRC and sizes in a data descriptor
      * after the data. Nothing goes to a temporary file. An entry with `store: true` is held in memory, since a stored
      * entry keeps its sizes in its header. `ratioLimit` does not apply, since an entry is written before its ratio is
-     * known.
+     * known. A file whose size is known, from its `size` or because its content is bytes or text, gets ZIP64 fields in
+     * its local header only when deflating it could pass 4 GiB. A stream without a size gets none, since some readers
+     * refuse ZIP64 fields an entry does not need; should it reach 4 GiB, its data descriptor holds eight-byte sizes, as
+     * archiver writes one. tar writes a file of known size as it arrives without this option.
      */
     streaming?: boolean;
 }
@@ -71,6 +82,31 @@ export declare function createArchive(format: PackFormat, entries: Iterable<Pack
 export declare function createArchiveStream(format: PackFormat, entries: Iterable<PackEntry> | AsyncIterable<PackEntry>, options?: PackOptions): Readable;
 /** `createArchive` into memory. */
 export declare function createArchiveBytes(format: PackFormat, entries: Iterable<PackEntry> | AsyncIterable<PackEntry>, options?: PackOptions): Promise<Uint8Array>;
+/** An archive that entries are added to one at a time, as archiver's `append` and `finalize` work. */
+export interface ArchiveWriter {
+    /**
+     * The archive as it is written, to read or pipe while entries are added. Entries wait in memory until it is read.
+     * An error destroys it, so a reader never takes a partial archive for a whole one.
+     */
+    readonly stream: Readable;
+    /**
+     * Adds an entry after the ones added before it. Its data is read when its turn comes, and a function given as its
+     * data is called then. Throws a PackError once `finish` or `abort` has been called or the archive has failed.
+     */
+    add(entry: PackEntry): void;
+    /**
+     * Says no more entries come. Resolves once the end of the archive is written to `stream`, and rejects with the error
+     * the archive failed on. The end is written only as `stream` is read, so keep reading it while this waits.
+     */
+    finish(): Promise<void>;
+    /** Stops the archive: `stream` is destroyed with `reason`, and `finish` rejects with it. Does nothing once it has finished. */
+    abort(reason?: unknown): void;
+}
+/**
+ * `createArchiveStream` for a caller that finds its entries as it goes. An entry `createArchive` refuses, or data that
+ * fails, fails the archive: `finish` rejects with the error and `stream` is destroyed with it.
+ */
+export declare function createArchiveWriter(format: PackFormat, options?: PackOptions): ArchiveWriter;
 /** Compresses `data` as one stream of `format` to `sink`, then closes it. */
 export declare function compressTo(format: CompressFormat, data: PackData, sink: ByteSink, options?: PackOptions): Promise<void>;
 /** `compressTo` as a Node Readable. */

package/dist/pack.js

@@ -4,6 +4,7 @@ exports.PackError = void 0;
 exports.createArchive = createArchive;
 exports.createArchiveStream = createArchiveStream;
 exports.createArchiveBytes = createArchiveBytes;
+exports.createArchiveWriter = createArchiveWriter;
 exports.compressTo = compressTo;
 exports.compressStream = compressStream;
 exports.compress = compress;
@@ -119,6 +120,12 @@ function settle(entry, options, now) {
     const kind = entry.kind ?? (raw.endsWith('/') ? 'directory' : entry.linkTarget !== undefined ? 'symlink' : 'file');
     if (kind === 'symlink' && entry.linkTarget === undefined)
         throw new PackError(`Symbolic link ${JSON.stringify(raw)} needs a linkTarget`);
+    if (entry.size !== undefined) {
+        if (kind !== 'file')
+            throw new PackError(`Entry ${JSON.stringify(raw)} is a ${kind}, and only a file takes a size`);
+        if (!Number.isSafeInteger(entry.size) || entry.size < 0)
+            throw new PackError(`Entry ${JSON.stringify(raw)} has a size of ${String(entry.size)}, which is not a whole number of bytes`);
+    }
     const name = kind === 'directory' ? (raw.endsWith('/') ? raw : `${raw}/`) : raw.replace(/\/+$/, '');
     for (const detail of (0, rules_1.nameDetails)(name, true)) {
         const why = REFUSED[detail];
@@ -135,21 +142,51 @@ function settle(entry, options, now) {
     }
     return { name, kind, entry, mode: (entry.mode ?? (kind === 'directory' ? 0o755 : kind === 'symlink' ? 0o777 : 0o644)) & 0o7777, mtime: entry.mtime ?? options.mtime ?? now };
 }
-/** The content of a settled entry: a file's data, a link's target, or nothing for a folder. */
+/** `chunks`, failing as soon as they pass `size` bytes, or when they end short of it. */
+async function* sized(chunks, size, name) {
+    let total = 0;
+    for await (const c of chunks) {
+        total += c.length;
+        if (total > size)
+            throw new Error(`Entry ${JSON.stringify(name)} has more than the ${size} bytes its size gives`);
+        yield c;
+    }
+    if (total !== size)
+        throw new Error(`Entry ${JSON.stringify(name)} has ${total} bytes, not the ${size} its size gives`);
+}
+/**
+ * The content of a settled entry: a file's data, a link's target, or nothing for a folder. A file's size is known when
+ * the entry gives one or its content is bytes or text. Content given as bytes or text of another size than the entry
+ * gives throws a PackError here, before the entry is written; a stream of another size fails as it is read.
+ */
 async function contentOf(s) {
     if (s.kind === 'directory')
-        return chunksOf(undefined);
-    if (s.kind === 'symlink')
-        return chunksOf(s.entry.linkTarget);
-    const d = s.entry.data;
-    return chunksOf(typeof d === 'function' ? await d() : d);
+        return { chunks: chunksOf(undefined), size: 0 };
+    if (s.kind === 'symlink') {
+        const target = toBytes(s.entry.linkTarget ?? '');
+        return { chunks: chunksOf(target), size: target.length };
+    }
+    const given = s.entry.data;
+    const raw = typeof given === 'function' ? await given() : given;
+    const data = typeof raw === 'string' ? toBytes(raw) : raw;
+    const own = data === undefined ? 0 : data instanceof Uint8Array ? data.length : undefined;
+    const size = s.entry.size;
+    if (size === undefined)
+        return { chunks: chunksOf(data), size: own };
+    if (own !== undefined && own !== size)
+        throw new PackError(`Entry ${JSON.stringify(s.name)} holds ${own} bytes, not the ${size} its size gives`);
+    return { chunks: sized(chunksOf(data), size, s.name), size };
 }
 /** Reads an entry's content into memory or a temporary file, counting its CRC-32. */
 async function spillOf(s, temp, threshold, check) {
+    return spillChunks((await contentOf(s)).chunks, temp, threshold, check);
+}
+/** Reads chunks into memory or a temporary file, counting their CRC-32. */
+async function spillChunks(chunks, temp, threshold, check) {
     const spill = new io_1.SpillSink(temp, threshold);
     let crc = 0;
     try {
-        for await (const c of checked(await contentOf(s), check)) {
+        for await (const c of checked(chunks, check)) {
             crc = (0, crc32_1.crc32)(c, crc);
             await spill.write(c);
         }
@@ -236,7 +273,8 @@ async function writeZipEntries(entries, sink, options, temp, check) {
  * second, hidden entry gets past a reader of the central directory.
  */
 async function streamZipEntry(out, s, info, level, check) {
-    const it = checked(await contentOf(s), check)[Symbol.asyncIterator]();
+    const content = await contentOf(s);
+    const it = checked(content.chunks, check)[Symbol.asyncIterator]();
     let first;
     try {
         first = await it.next();
@@ -277,7 +315,9 @@ async function streamZipEntry(out, s, info, level, check) {
         }
     };
     Object.assign(info, { method: 8, versionNeeded: 20 });
-    await out.streamEntry(info, (0, codecs_1.deflateRaw)(raw(), level), () => ({ crc, size }));
+    // zlib's deflateBound for any settings: deflate never grows data by more than this.
+    const zip64 = content.size !== undefined && content.size + Math.ceil(content.size / 8) + Math.ceil(content.size / 64) + 5 >= 0xffffffff;
+    await out.streamEntry(info, (0, codecs_1.deflateRaw)(raw(), level), () => ({ crc, size }), zip64);
 }
 // ---------------------------------------------------------------------------------------------------------------
 // tar
@@ -287,9 +327,25 @@ async function* tarChunks(entries, options, temp, check) {
     let written = 0;
     const pad = (n) => Buffer.alloc((tar_1.BLOCK - (n % tar_1.BLOCK)) % tar_1.BLOCK);
     for await (const s of entries) {
-        const { spill } = s.kind === 'file' ? await spillOf(s, temp, threshold, check) : { spill: undefined };
+        let spill;
+        let direct;
+        let first;
+        let size = 0;
         try {
-            const size = spill?.length ?? 0;
+            if (s.kind === 'file') {
+                const content = await contentOf(s);
+                if (content.size === undefined) {
+                    ({ spill } = await spillChunks(content.chunks, temp, threshold, check));
+                    size = spill.length;
+                }
+                else {
+                    // The header can go first and the data straight after it. The first chunk is read before the header, so a
+                    // stream that fails at once fails before its entry is begun, and a stream is open to close should the work stop.
+                    size = content.size;
+                    direct = checked(content.chunks, check)[Symbol.asyncIterator]();
+                    first = await direct.next();
+                }
+            }
             const link = s.kind === 'symlink' ? (s.entry.linkTarget ?? '') : '';
             const mtime = Math.floor(s.mtime.getTime() / 1000);
             // What the ustar header cannot hold goes in a pax header before it.
@@ -344,8 +400,13 @@ async function* tarChunks(entries, options, temp, check) {
             });
             yield h;
             written += h.length;
-            if (spill) {
+            if (spill)
                 yield* checked(spill.read(), check);
+            if (direct && first) {
+                for (let r = first; !r.done; r = await direct.next())
+                    yield r.value;
+            }
+            if (spill || direct) {
                 const p = pad(size);
                 if (p.length)
                     yield p;
@@ -353,6 +414,7 @@ async function* tarChunks(entries, options, temp, check) {
             }
         }
         finally {
+            await direct?.return?.(undefined);
             await spill?.dispose();
         }
     }
@@ -476,6 +538,15 @@ function compressChunks(format, input, options, temp, now, check) {
 // The calls
 const ARCHIVE_FORMATS = new Set(['zip', 'tar', 'tar.gz', 'tar.bz2', 'tar.zst', 'tar.xz', '7z']);
 const TAR_COMPRESSION = { 'tar.gz': 'gzip', 'tar.bz2': 'bzip2', 'tar.zst': 'zstd', 'tar.xz': 'xz' };
+/** Throws for a format `createArchive` does not write, and for `streaming` with any format but ZIP. */
+function checkFormat(format, options) {
+    if (!ARCHIVE_FORMATS.has(format))
+        throw new PackError(`Unknown archive format ${String(format)}`);
+    // tar writes a file of known size as it arrives without the option, and needs the size of any other first. 7z writes
+    // its header from every entry after the data.
+    if (options.streaming && format !== 'zip')
+        throw new PackError(`streaming writes ZIP archives only, not ${format}`);
+}
 /** Copies chunks to the sink, then closes it. On failure it aborts the sink, so a partial output never looks whole. */
 async function deliver(produce, sink) {
     try {
@@ -498,11 +569,7 @@ async function deliver(produce, sink) {
  * sink is aborted instead of closed and the error is thrown.
  */
 async function createArchive(format, entries, sink, options = {}) {
-    if (!ARCHIVE_FORMATS.has(format))
-        throw new PackError(`Unknown archive format ${String(format)}`);
-    // tar needs each entry's size before its data, and 7z writes its header from every entry after the data.
-    if (options.streaming && format !== 'zip')
-        throw new PackError(`streaming writes ZIP archives only, not ${format}`);
+    checkFormat(format, options);
     const now = new Date();
     const temp = new io_1.TempDir(options.tempDir);
     const check = () => options.signal?.throwIfAborted();
@@ -563,6 +630,95 @@ async function createArchiveBytes(format, entries, options = {}) {
     await createArchive(format, entries, sink, options);
     return sink.result();
 }
+/**
+ * `createArchiveStream` for a caller that finds its entries as it goes. An entry `createArchive` refuses, or data that
+ * fails, fails the archive: `finish` rejects with the error and `stream` is destroyed with it.
+ */
+function createArchiveWriter(format, options = {}) {
+    checkFormat(format, options);
+    const queue = [];
+    let head = 0;
+    let closed;
+    let failure;
+    let wake;
+    const drop = () => {
+        queue.length = 0;
+        head = 0;
+    };
+    const stop = new AbortController();
+    const outer = options.signal;
+    const forward = () => stop.abort(outer?.reason);
+    if (outer?.aborted)
+        forward();
+    else
+        outer?.addEventListener('abort', forward, { once: true });
+    // The loop waits here for the next entry. The wait ends when the work stops, so a reader that goes away or an abort
+    // does not leave it waiting for an entry that never comes.
+    const nextEntry = (signal) => new Promise((resolve, reject) => {
+        const stopped = () => {
+            wake = undefined;
+            reject(signal.reason);
+        };
+        signal.addEventListener('abort', stopped, { once: true });
+        wake = () => {
+            signal.removeEventListener('abort', stopped);
+            wake = undefined;
+            resolve();
+        };
+    });
+    async function* entries(signal) {
+        for (;;) {
+            if (head < queue.length) {
+                const e = queue[head];
+                queue[head++] = undefined;
+                if (head === queue.length)
+                    drop();
+                if (e)
+                    yield e;
+                continue;
+            }
+            if (closed === 'finished')
+                return;
+            signal.throwIfAborted();
+            await nextEntry(signal);
+        }
+    }
+    let done = Promise.resolve();
+    const stream = readableOf((sink, signal) => {
+        done = createArchive(format, entries(signal), sink, { ...options, signal });
+        return done;
+    }, stop.signal);
+    done.then(() => outer?.removeEventListener('abort', forward), error => {
+        failure ??= { error };
+        drop();
+        outer?.removeEventListener('abort', forward);
+    });
+    return {
+        stream,
+        add(entry) {
+            if (closed)
+                throw new PackError(`An entry was added after the archive was ${closed}`);
+            if (failure)
+                throw new PackError('An entry was added after the archive failed', { cause: failure.error });
+            queue.push(entry);
+            wake?.();
+        },
+        finish() {
+            if (!closed) {
+                closed = 'finished';
+                wake?.();
+            }
+            return done;
+        },
+        abort(reason) {
+            if (closed === 'aborted')
+                return;
+            closed = 'aborted';
+            drop();
+            stop.abort(reason);
+        },
+    };
+}
 /** Compresses `data` as one stream of `format` to `sink`, then closes it. */
 async function compressTo(format, data, sink, options = {}) {
     const temp = new io_1.TempDir(options.tempDir);

package/dist/zip-writer.d.ts

@@ -37,13 +37,17 @@ export declare class ZipOutput {
     /**
      * Writes one entry whose sizes are known only once its data has gone out: a local header with the data descriptor
      * flag and no CRC or sizes, then `chunks`, the compressed data, then a data descriptor with the CRC and the sizes.
-     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended. An entry of 4 GiB or more gets
-     * eight-byte sizes in its descriptor and a ZIP64 field in the central directory, as archiver writes one.
+     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended.
+     *
+     * With `zip64`, the local header has a ZIP64 field, which tells a reader the descriptor's sizes are eight bytes. Its
+     * sizes are 0xFFFFFFFF and the field's are zero, as Python's zipfile writes a streamed entry. Without it, an entry of
+     * 4 GiB or more still gets eight-byte sizes in its descriptor, as archiver writes one. Either way, the central
+     * directory has a ZIP64 field only when a size or the offset needs one.
      */
     streamEntry(info: Written, chunks: AsyncIterable<Uint8Array>, totals: () => {
         crc: number;
         size: number;
-    }): Promise<void>;
+    }, zip64?: boolean): Promise<void>;
     /** Writes the central directory and the end records, with the archive's comment. */
     end(comment: Uint8Array): Promise<void>;
 }

package/dist/zip-writer.js

@@ -89,26 +89,33 @@ class ZipOutput {
     /**
      * Writes one entry whose sizes are known only once its data has gone out: a local header with the data descriptor
      * flag and no CRC or sizes, then `chunks`, the compressed data, then a data descriptor with the CRC and the sizes.
-     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended. An entry of 4 GiB or more gets
-     * eight-byte sizes in its descriptor and a ZIP64 field in the central directory, as archiver writes one.
+     * `totals` gives the CRC and size of the uncompressed data once `chunks` has ended.
+     *
+     * With `zip64`, the local header has a ZIP64 field, which tells a reader the descriptor's sizes are eight bytes. Its
+     * sizes are 0xFFFFFFFF and the field's are zero, as Python's zipfile writes a streamed entry. Without it, an entry of
+     * 4 GiB or more still gets eight-byte sizes in its descriptor, as archiver writes one. Either way, the central
+     * directory has a ZIP64 field only when a size or the offset needs one.
      */
-    async streamEntry(info, chunks, totals) {
+    async streamEntry(info, chunks, totals, zip64 = false) {
         const flags = info.flags | 0x0008;
+        const versionNeeded = zip64 ? Math.max(45, info.versionNeeded) : info.versionNeeded;
         const localOffset = this.offset;
         const local = new Builder()
             .u32(0x04034b50)
-            .u16(info.versionNeeded)
+            .u16(versionNeeded)
             .u16(flags)
             .u16(info.method)
             .u16(info.time)
             .u16(info.date)
             .u32(0)
-            .u32(0)
-            .u32(0)
+            .u32(zip64 ? MAX32 : 0)
+            .u32(zip64 ? MAX32 : 0)
             .u16(info.name.length)
-            .u16(info.extras.length)
-            .bytes(info.name)
-            .bytes(info.extras);
+            .u16(info.extras.length + (zip64 ? 20 : 0))
+            .bytes(info.name);
+        if (zip64)
+            local.u16(0x0001).u16(16).u64(0).u64(0);
+        local.bytes(info.extras);
         await this.put(local.build());
         let compressedSize = 0;
         for await (const c of chunks) {
@@ -117,12 +124,12 @@ class ZipOutput {
         }
         const { crc, size } = totals();
         const descriptor = new Builder().u32(0x08074b50).u32(crc);
-        if (size >= MAX32 || compressedSize >= MAX32)
+        if (zip64 || size >= MAX32 || compressedSize >= MAX32)
             descriptor.u64(compressedSize).u64(size);
         else
             descriptor.u32(compressedSize).u32(size);
         await this.put(descriptor.build());
-        this.central.push(centralRecord({ ...info, flags, crc, size, compressedSize }, localOffset));
+        this.central.push(centralRecord({ ...info, versionNeeded, flags, crc, size, compressedSize }, localOffset));
     }
     /** Writes the central directory and the end records, with the archive's comment. */
     async end(comment) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@patrickdk77/archive-defuse",
-  "version": "0.1.1",
+  "version": "0.1.2",
   "description": "Sanitize zip, tar, gzip, bzip2, zstd, Brotli, cab, 7z, xz and lzma archives. Detects path traversal, links out of the tree, device files, duplicate names, overlapping entries, hidden and prepended data, encrypted entries and compression bombs, hands every entry to file plugins, scores the risk, and writes a defused archive in the same format with no runtime dependencies.",
   "license": "MIT AND Apache-2.0",
   "author": "patrickdk77",