@gscdump/engine 7.3.2 → 7.5.1

package/README.md

@@ -30,6 +30,146 @@ different publications produce different bytes. It must retain and replay the
 exact encoded bytes for retries; this option does not provide publication or
 recovery coordination itself.
 
+For a bounded, already ordered source, `/hyparquet` also exports
+`encodeParquetRowBatches(batches, options, write)`. It consumes sync or async
+batches, coerces the declared `columns` with the same physical types as the
+array encoder, and emits one Parquet file through an awaited byte callback.
+It does not sort, deduplicate, retry, or publish an object. The callback borrows
+its bytes until its promise settles; copy a chunk if retaining it. The caller
+must abort/discard the entire destination on any error, including an error
+while writing the footer. Reopening the same immutable source with identical
+options produces identical bytes regardless of producer batch boundaries.
+This supports a counting/digest pass followed by a separately coordinated
+fixed-length upload, without the codec owning transport or commit semantics.
+
+`decodeParquetRowGroups({ byteLength, slice }, options)` reads exact byte ranges
+and by default yields one physical row group at a time. The caller must pin every range
+to the same immutable object or ETag. It preserves the existing date, column
+projection, row-window and filter behavior. An absent, corrupt, oversized or
+incompletely read file throws; only valid empty Parquet yields no rows.
+
+The range reader alone accepts an optional `indexedRowWindow` policy:
+
+```ts
+decodeParquetRowGroups(pinnedObject, {
+  indexedRowWindow: {
+    targetRows: 1000,
+    requiredIndexedColumns: ['query', 'url'],
+    rangeCacheBytes: 2 * 1024 * 1024,
+  },
+})
+```
+
+Each group uses windows only when all designated columns are present, selected,
+have usable offset-index descriptors, and have no dictionary encoding. The
+schema must contain only flat scalar/UTF8/date fields; binary, nested and other
+unsupported schemas retain the original group read. Ineligible groups use the
+original uncached path, so `targetRows` is a target rather than a hard batch
+limit. Include every potentially large column in `requiredIndexedColumns`.
+All existing physical-group admission limits still apply.
+
+Before emitting an eligible group's first window, the reader checks index
+structure for projected columns, or all indexed columns when a filter may need
+unprojected fields: row zero, increasing row positions, ordered nonoverlapping
+page ranges and physical column bounds. A failure throws without falling back
+after partial output. This assumes metadata row offsets agree with physical
+page contents; it does not independently verify page-header row counts or
+detect every deliberately consistent but false index.
+
+The optional range cache defaults to zero, allows at most 2 MiB of payload and
+128 entries, and lives only through one eligible group. It is cleared on
+completion, error or iterator return; another pass rereads the pinned object.
+It does not cache decoded rows or provide publication/ETag coordination. Windows
+can otherwise reread overlapping pages and substantially increase range I/O.
+They still decode whole boundary pages, and dictionaries in other columns can
+still allocate in full. These options do not bound decompressed page sizes or
+the total heap; the unchanged fallback also retains its existing memory costs.
+Buffered decoders, encoders and all omitted-option behavior remain unchanged.
+
+The range reader can also opt into `limits.maxReadAndDecompressBytes`. This is a
+**cumulative buffer-work budget for each physical group**, shared across all its
+indexed windows and concurrent column reads. It charges requested source-range
+bytes before dispatch and SNAPPY output bytes before allocating the fixed-size
+output buffer. Cache hits do not allocate another range, while repeated
+decompression still consumes budget. Group indexes are included; the initial
+footer has its separate metadata limit. Raw/uncompressed pages and V2 levels
+are covered by the range charge, so compressed data is deliberately counted
+twice. Earlier charges are never refunded, including after a failed read. Failure
+or iterator return closes the budget, preventing new work and rejecting late
+range results. It cannot physically cancel already dispatched reads: the range
+source interface has no cancellation hook.
+
+ZSTD decompression refuses under this option: the current allocating decoder
+can allocate from its frame header before checking the Parquet output size.
+Without the option, ZSTD and all existing decoding behavior remain unchanged.
+The budget does not measure JavaScript strings, expanded values, row containers,
+internal prefetch or dictionary/page stitching copies, codec working state or
+caller-retained batches. It is
+not a total heap bound or protection against all malformed Parquet contents.
+It can reject valid files that previously decoded, particularly old files whose
+writer understated uncompressed footer sizes. Callers must preserve those files
+and choose their own explicit retry/repair policy; refusal is never empty data.
+
+`decodeParquetRowGroups(source, { boundedValues: {} })` selects a separate strict
+value-cursor path. It supports flat REQUIRED/OPTIONAL scalar columns, explicitly
+annotated UTF8/STRING, PLAIN and dictionary data pages (V1/V2), and SNAPPY or
+UNCOMPRESSED data. Projection, physical row ranges, nulls, date normalization and
+row order are preserved. Nested/binary/other logical types, other codecs or value
+encodings, BOOLEAN dictionaries, filters and `indexedRowWindow` combinations
+refuse; this mode never silently falls back to the eager reader. All selected
+page headers are inspected before the first output to detect unsupported later
+pages. Corrupt value bodies or over-limit individual values may fail after an
+earlier batch; consumers must finish validating their entire input before
+publishing a replacement.
+Explicit row bounds in this strict mode must be nonnegative safe integers, with
+positive Infinity also accepted for an open-ended `rowEnd`. Omitted bounds use
+the full input, and reversed or zero-length ranges are empty.
+
+The cursor retains at most one current data page and one encoded dictionary per
+selected column. UTF8 strings, definition levels and dictionary IDs are consumed
+incrementally. Dictionaries retain encoded bytes and a four-byte offset per
+entry, rather than an array of decoded strings. Final bit-packed padding is
+validated but not expanded. These options have independent meanings:
+
+| `boundedValues` option | Default | Admission rule |
+| --- | --- | --- |
+| `targetRows` | 1000 | Hard emitted batch row limit |
+| `maxBatchBytes` | 1 MiB | Before materialization: 128 per row, 16 per cell slot, 64 per cell plus twice UTF8 byte length or 32 for a fixed scalar |
+| `maxValueBytes` | 256 KiB | Encoded bytes of one string, before decoding |
+| `maxActiveBytes` | 24 MiB | Per group: sum of each column's largest compressed + uncompressed data page, dictionary compressed + uncompressed bytes, dictionary offset tables, plus one maximum header probe |
+| `maxPages` | 16384 | Retained page-descriptor count across the selected input |
+| `maxHeaderBytes` | 64 KiB | Header probe cap; starts at 512 bytes and grows only on truncation |
+| `maxDictionaryEntries` | 100000 | Entry count for each encoded dictionary |
+
+Active-buffer admission uses real page headers, not the often understated
+uncompressed sizes in legacy writer footers. Page transitions release the old
+cursor before reading/decompressing its successor; columns load sequentially.
+Batch bytes are a conservative value-materialization charge, **not a measurement
+of V8 object overhead or an isolate heap guarantee**. The footer (separately
+bounded), page-descriptor objects, fixed cursor/schema state, caller-retained
+batches, codec/runtime overhead and concurrent application work remain separate.
+Setting larger limits deliberately permits a larger working set. A single page
+still decompresses in full, and a consumer that retains all emitted batches
+removes the benefit. This mode does not make arbitrary malicious files safe.
+
+The existing `maxReadAndDecompressBytes` option also applies: a group's header
+probes, physical reads and SNAPPY output share one cumulative budget. Headers
+are scanned once; immutable/pinned object bytes are required across that scan
+and later reads. Iterator return/error releases cursor references; already
+dispatched source reads cannot be physically aborted. Defaults, the buffered
+decoder and indexed-window eligibility/fallback are unchanged.
+
+Both APIs have configurable admission limits. The encoder bounds producer
+batch rows/value bytes, output group rows/value bytes, retained descriptor counts
+and scalar bytes (including statistics), and the current encoded group/footer
+buffer. The decoder checks footer size,
+descriptor count, declared group rows/uncompressed bytes, and every range request.
+The default decoder accepts the existing encoder's 25,000-row groups. These
+limits do not establish a Worker heap guarantee: caller-retained rows, codec
+transient allocations, footer descriptors, and concurrent requests still use
+memory. Metadata limits are not protection against malicious Parquet contents.
+The existing array APIs and their default bytes are unchanged.
+
 ## Subpath exports
 
 | Subpath | Purpose |

package/dist/adapters/hyparquet.d.mts

@@ -1,7 +1,8 @@
 import { ParquetQueryFilter } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/types/index.mjs";
 import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/types/node.mjs";
-import { CodecCtx, DataSource, ParquetCodec, Row, TableName } from "../storage.mjs";
+import { CodecCtx, DataSource, ParquetCodec, Row, TableName, WriteResult } from "../storage.mjs";
 import { ColumnDef } from "../schema.mjs";
+import { BoundedValuesOptions } from "./parquet-bounded-values.mjs";
 export interface EncodeParquetOptions {
   /** File footer metadata, without adding fact columns. Keys are encoded in sorted order. */
   fileMetadata?: Readonly<Record<string, string>>;
@@ -23,6 +24,36 @@ export interface EncodeFlexOptions extends EncodeParquetOptions {
  * merges cleanly with fact-table reads.
  */
 export declare function encodeRowsToParquetFlex(rows: readonly Row[], opts: EncodeFlexOptions): Uint8Array;
+export interface EncodeParquetBatchesOptions extends EncodeParquetOptions {
+  /** Output schema and column order. Input rows must already be ordered/deduplicated. */
+  columns: readonly ColumnDef[];
+  /** Maximum rows per output group. Default 25000, matching the array codec. */
+  rowGroupSize?: number;
+  limits?: {
+    /** Refuse a producer batch with more rows. Default 25000. */
+    maxBatchRows?: number;
+    /** Conservative scalar-value bytes per producer batch. Default 8 MiB. */
+    maxBatchBytes?: number;
+    /** Split groups at this conservative scalar-value budget. Default 2 MiB. */
+    maxRowGroupBytes?: number;
+    /** Bound retained footer/index descriptors. Default 4096. */
+    maxRowGroups?: number;
+    /** Bound retained descriptor/scalar bytes, including statistics. Default 8 MiB. */
+    maxRetainedMetadataBytes?: number;
+    /** Bound the current encoded group or footer buffer. Default 8 MiB. */
+    maxOutputBufferBytes?: number;
+  };
+}
+/**
+ * Encode ordered batches into ONE Parquet file with sequential backpressure.
+ * Group boundaries depend on rows/options, never producer batch boundaries.
+ * `write` borrows its bytes only until its returned promise settles; copy them
+ * if retaining them. This function does not close/commit a destination. On ANY
+ * rejection the caller must abort/discard the entire partial output, including
+ * after a footer callback failure. It performs no sorting, dedupe or retries.
+ * Memory includes one input batch, one group, and bounded footer descriptors.
+ */
+export declare function encodeParquetRowBatches(batches: AsyncIterable<readonly Row[]> | Iterable<readonly Row[]>, opts: EncodeParquetBatchesOptions, write: (chunk: Uint8Array) => Promise<void>): Promise<WriteResult>;
 export interface DecodeParquetOptions {
   /** First physical row to decode, inclusive. */
   rowStart?: number;
@@ -49,6 +80,58 @@ export interface DecodeParquetOptions {
   columns?: readonly string[];
 }
 export declare function decodeParquetToRows(bytes: Uint8Array, opts?: DecodeParquetOptions): Promise<Row[]>;
+/** A caller-owned, immutable/pinned object; ranges must return exactly the requested bytes. */
+export interface ParquetRangeSource {
+  byteLength: number;
+  slice: (start: number, end?: number) => ArrayBuffer | Promise<ArrayBuffer>;
+}
+export interface DecodeParquetGroupsOptions extends DecodeParquetOptions {
+  /**
+   * Strict opt-in value cursors for flat scalar PLAIN/dictionary SNAPPY or raw
+   * pages. Rejects unsupported inputs instead of falling back to eager decoding.
+   * Page buffers and emitted values have separate bounds; not an isolate heap cap.
+   * Cannot be combined with filters or indexedRowWindow.
+   */
+  boundedValues?: BoundedValuesOptions;
+  /**
+   * Opt-in windows for flat scalar groups whose designated, selected columns
+   * have offset indexes and no dictionary encoding. Ineligible groups retain
+   * the original physical-group read, so targetRows is not a hard batch limit.
+   * Windows still decode complete overlapping pages, not a bounded heap size.
+   */
+  indexedRowWindow?: {
+    targetRows: number;
+    requiredIndexedColumns: readonly string[];
+    /** Per eligible group; 0 disables caching. Default 0, maximum 2 MiB/128 entries. */
+    rangeCacheBytes?: number;
+  };
+  limits?: {
+    /** Footer bytes before deserialization. Default 1 MiB. */
+    maxMetadataBytes?: number;
+    /** Includes the existing encoder's 25000-row groups. Default 25000. */
+    maxRowGroupRows?: number;
+    /** Declared uncompressed bytes across all columns in one group. Default 16 MiB. */
+    maxRowGroupBytes?: number;
+    /** Bound metadata descriptors. Default 4096. */
+    maxRowGroups?: number;
+    /** Maximum individual range request. Default 16 MiB. */
+    maxReadBytes?: number;
+    /**
+     * Optional cumulative source-range + SNAPPY-output bytes per physical group,
+     * including all indexed windows. Charged before reads/output allocation;
+     * cache hits do not allocate another range. Deliberately conservative, not
+     * total JS heap. ZSTD refuses only with this option. Omission is unchanged.
+     */
+    maxReadAndDecompressBytes?: number;
+  };
+}
+/**
+ * Pull one physical group at a time without downloading the complete object.
+ * The limits are admission bounds over Parquet metadata, not a memory-safety
+ * guarantee for malicious files or caller-retained batches. No cross-group
+ * read-ahead is started. Missing/corrupt/range-error input throws; it is not empty.
+ */
+export declare function decodeParquetRowGroups(file: ParquetRangeSource, opts?: DecodeParquetGroupsOptions): AsyncGenerator<Row[]>;
 export interface HyparquetCodecOptions {
   /**
    * Override `readRows`. Useful when reads should be delegated to a faster

package/dist/adapters/hyparquet.mjs

@@ -1,12 +1,17 @@
 import { TABLE_METADATA } from "../drizzle-schema.mjs";
 import { SCHEMAS } from "../schema.mjs";
 import { decompress } from "../node_modules/.pnpm/fzstd@0.1.1/node_modules/fzstd/esm/index.mjs";
-import { parquetMetadataAsync, parquetSchema } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/metadata.mjs";
+import { parquetMetadata, parquetMetadataAsync, parquetSchema } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/metadata.mjs";
+import { readOffsetIndex } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/indexes.mjs";
 import { parquetReadObjects } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/read.mjs";
 import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs";
 import { ByteWriter } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/bytewriter.mjs";
+import { ParquetWriter } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/parquet-writer.mjs";
 import { parquetWriteRows } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/write-rows.mjs";
 import "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/node.mjs";
+import { parquetDecodeBudget } from "./parquet-decode-budget.mjs";
+import { boundedValuesPolicy, decodeBoundedValues } from "./parquet-bounded-values.mjs";
+import { parquetRangeCache } from "./parquet-range-cache.mjs";
 const ROW_GROUP_SIZE = 25e3;
 const DEFAULT_COMPACTION_READ_CONCURRENCY = 8;
 const MAX_COMPACTION_READ_CONCURRENCY = 32;
@@ -206,6 +211,139 @@ function encodeRowsToParquetFlex(rows, opts) {
 		return 0;
 	}), columns, rowGroupSize, opts.fileMetadata);
 }
+function streamLimit(value, fallback, name) {
+	const limit = value ?? fallback;
+	if (!Number.isSafeInteger(limit) || limit <= 0) throw new RangeError(`parquet stream ${name} must be a positive safe integer`);
+	return limit;
+}
+function scalarBytes(value) {
+	return typeof value === "string" ? value.length * 3 + 8 : 16;
+}
+function retainedDescriptorBytes(value, remaining) {
+	const seen = /* @__PURE__ */ new Set();
+	let bytes = 0;
+	const visit = (value) => {
+		if (value !== null && typeof value === "object") {
+			if (seen.has(value)) return;
+			seen.add(value);
+			bytes += 64;
+			if (ArrayBuffer.isView(value) || value instanceof ArrayBuffer) bytes += value.byteLength;
+			else for (const key in value) if (Object.hasOwn(value, key)) {
+				bytes += scalarBytes(key);
+				visit(value[key]);
+			}
+		} else bytes += scalarBytes(value);
+		if (bytes > remaining) throw new RangeError("parquet stream exceeds maxRetainedMetadataBytes");
+	};
+	visit(value);
+	return bytes;
+}
+async function encodeParquetRowBatches(batches, opts, write) {
+	const rowGroupSize = streamLimit(opts.rowGroupSize, ROW_GROUP_SIZE, "rowGroupSize");
+	const maxBatchRows = streamLimit(opts.limits?.maxBatchRows, ROW_GROUP_SIZE, "maxBatchRows");
+	const maxBatchBytes = streamLimit(opts.limits?.maxBatchBytes, 8388608, "maxBatchBytes");
+	const maxRowGroupBytes = streamLimit(opts.limits?.maxRowGroupBytes, 2097152, "maxRowGroupBytes");
+	const maxRowGroups = streamLimit(opts.limits?.maxRowGroups, 4096, "maxRowGroups");
+	const maxRetainedMetadataBytes = streamLimit(opts.limits?.maxRetainedMetadataBytes, 8388608, "maxRetainedMetadataBytes");
+	const maxOutputBufferBytes = streamLimit(opts.limits?.maxOutputBufferBytes, 8388608, "maxOutputBufferBytes");
+	if (opts.columns.length === 0 || opts.columns.length > 64 || new Set(opts.columns.map((col) => col.name)).size !== opts.columns.length) throw new RangeError("parquet stream requires 1 to 64 uniquely named columns");
+	const codecs = opts.columns.map((col) => ({
+		name: col.name,
+		isDate: col.type === "DATE",
+		type: basicTypeFor(col.type),
+		nullable: col.nullable
+	}));
+	const schema = buildWriteSchema(opts.columns);
+	const metadataKeys = Object.keys(opts.fileMetadata ?? {}).sort();
+	let metadataBytes = 0;
+	for (const key of metadataKeys) {
+		metadataBytes += scalarBytes(key) + scalarBytes(opts.fileMetadata[key]);
+		if (metadataBytes > maxOutputBufferBytes) throw new RangeError("parquet stream file metadata exceeds maxOutputBufferBytes");
+	}
+	let retainedMetadataBytes = retainedDescriptorBytes([schema, opts.fileMetadata], maxRetainedMetadataBytes);
+	const writer = new ByteWriter(Math.min(65536, maxOutputBufferBytes));
+	writer.ensure = (size) => {
+		const required = writer.index + size;
+		if (required > maxOutputBufferBytes) throw new RangeError("parquet stream encoded group/footer exceeds maxOutputBufferBytes");
+		if (required > writer.buffer.byteLength) {
+			const next = new ArrayBuffer(Math.min(maxOutputBufferBytes, Math.max(required, writer.buffer.byteLength * 2)));
+			new Uint8Array(next).set(new Uint8Array(writer.buffer, 0, writer.index));
+			writer.buffer = next;
+			writer.view = new DataView(next);
+		}
+	};
+	const flush = async () => {
+		if (writer.index > 0) await write(new Uint8Array(writer.buffer, 0, writer.index));
+		writer.index = 0;
+	};
+	const output = Object.assign(writer, {
+		flush,
+		finish: flush
+	});
+	const parquet = new ParquetWriter({
+		writer: output,
+		schema,
+		kvMetadata: metadataKeys.length ? metadataKeys.map((key) => ({
+			key,
+			value: opts.fileMetadata[key]
+		})) : void 0
+	});
+	let values = codecs.map(() => []);
+	let groupRows = 0;
+	let groupBytes = 0;
+	let groups = 0;
+	let rowCount = 0;
+	const flushGroup = async () => {
+		if (!groupRows) return;
+		if (++groups > maxRowGroups) throw new RangeError("parquet stream exceeds maxRowGroups");
+		const firstIndex = parquet.pendingIndexes.length;
+		await parquet.write({
+			columnData: codecs.map((col, index) => ({
+				name: col.name,
+				nullable: col.nullable,
+				columnIndex: true,
+				data: values[index]
+			})),
+			rowGroupSize: groupRows
+		});
+		retainedMetadataBytes += retainedDescriptorBytes([parquet.row_groups[parquet.row_groups.length - 1], parquet.pendingIndexes.slice(firstIndex)], maxRetainedMetadataBytes - retainedMetadataBytes);
+		values = codecs.map(() => []);
+		groupRows = 0;
+		groupBytes = 0;
+	};
+	try {
+		for await (const batch of batches) {
+			if (batch.length > maxBatchRows) throw new RangeError("parquet stream exceeds maxBatchRows");
+			let batchBytes = 0;
+			for (const row of batch) {
+				const converted = codecs.map((col) => col.isDate ? toEpochDays(row[col.name]) : coerceValue(row[col.name], col.type));
+				const bytes = converted.reduce((sum, value) => sum + scalarBytes(value), 0);
+				batchBytes += bytes;
+				if (batchBytes > maxBatchBytes) throw new RangeError("parquet stream exceeds maxBatchBytes");
+				if (bytes > maxRowGroupBytes) throw new RangeError("parquet stream single row exceeds maxRowGroupBytes");
+				if (groupRows && (groupRows >= rowGroupSize || groupBytes + bytes > maxRowGroupBytes)) await flushGroup();
+				for (let index = 0; index < converted.length; index++) values[index].push(converted[index]);
+				groupRows++;
+				groupBytes += bytes;
+				rowCount++;
+				if (groupRows === rowGroupSize) await flushGroup();
+			}
+		}
+		await flushGroup();
+		await parquet.finish();
+		return {
+			bytes: writer.offset,
+			rowCount
+		};
+	} finally {
+		values = [];
+		parquet.row_groups.length = 0;
+		parquet.pendingIndexes.length = 0;
+		writer.buffer = /* @__PURE__ */ new ArrayBuffer(0);
+		writer.view = new DataView(writer.buffer);
+		writer.index = 0;
+	}
+}
 function asyncBufferFromBytes(bytes) {
 	const base = bytes.byteOffset;
 	const buf = bytes.buffer;
@@ -226,7 +364,150 @@ async function decodeParquetToRows(bytes, opts = {}) {
 async function* decodeParquetGroups(bytes, opts = {}) {
 	if (bytes.byteLength === 0) return;
 	const file = asyncBufferFromBytes(bytes);
-	const metadata = await parquetMetadataAsync(file);
+	yield* decodeGroupsFromMetadata(file, await parquetMetadataAsync(file), opts);
+}
+async function* decodeParquetRowGroups(file, opts = {}) {
+	const values = boundedValuesPolicy(opts.boundedValues);
+	const selection = values ? {
+		columns: opts.columns?.slice(),
+		rowStart: opts.rowStart,
+		rowEnd: opts.rowEnd
+	} : void 0;
+	if (values) for (const key of ["rowStart", "rowEnd"]) {
+		const value = selection[key];
+		if (value !== void 0 && !(key === "rowEnd" && value === Infinity) && (!Number.isSafeInteger(value) || value < 0)) throw new RangeError(`invalid boundedValues.${key}`);
+	}
+	if (values && (opts.filter || opts.indexedRowWindow)) throw new Error("boundedValues cannot be combined with filter or indexedRowWindow");
+	const indexedWindow = indexedWindowPolicy(opts.indexedRowWindow);
+	const maxReadAndDecompressBytes = opts.limits?.maxReadAndDecompressBytes === void 0 ? void 0 : streamLimit(opts.limits.maxReadAndDecompressBytes, 0, "maxReadAndDecompressBytes");
+	const maxMetadataBytes = streamLimit(opts.limits?.maxMetadataBytes, 1048576, "maxMetadataBytes");
+	const maxRowGroupRows = streamLimit(opts.limits?.maxRowGroupRows, ROW_GROUP_SIZE, "maxRowGroupRows");
+	const maxRowGroupBytes = streamLimit(opts.limits?.maxRowGroupBytes, 16777216, "maxRowGroupBytes");
+	const maxRowGroups = streamLimit(opts.limits?.maxRowGroups, 4096, "maxRowGroups");
+	const maxReadBytes = streamLimit(opts.limits?.maxReadBytes, 16777216, "maxReadBytes");
+	if (!Number.isSafeInteger(file.byteLength) || file.byteLength < 12) throw new RangeError("parquet stream requires a valid file length");
+	const bounded = {
+		byteLength: file.byteLength,
+		async slice(start, end = file.byteLength) {
+			if (!Number.isSafeInteger(start) || !Number.isSafeInteger(end) || start < 0 || end < start || end > file.byteLength || end - start > maxReadBytes) throw new RangeError("parquet stream range exceeds maxReadBytes or file bounds");
+			const bytes = await file.slice(start, end);
+			if (bytes.byteLength !== end - start) throw new Error("parquet stream returned an incomplete range");
+			return bytes;
+		}
+	};
+	const trailer = new DataView(await bounded.slice(file.byteLength - 8, file.byteLength));
+	if (trailer.getUint32(4, true) !== 827474256) throw new Error("parquet stream invalid footer");
+	const metadataBytes = trailer.getUint32(0, true);
+	if (metadataBytes > maxMetadataBytes || metadataBytes + 12 > file.byteLength) throw new RangeError("parquet stream footer exceeds maxMetadataBytes or file bounds");
+	const metadata = parquetMetadata(await bounded.slice(file.byteLength - metadataBytes - 8, file.byteLength));
+	if (metadata.row_groups.length > maxRowGroups || metadata.schema.length > 65) throw new RangeError("parquet stream exceeds row-group/schema descriptor limits");
+	let totalRows = 0;
+	for (const group of metadata.row_groups) {
+		if (group.columns.length > 64) throw new RangeError("parquet stream exceeds column descriptor limits");
+		const rows = Number(group.num_rows);
+		if (group.columns.some((col) => [col.meta_data?.total_uncompressed_size, col.meta_data?.total_compressed_size].some((size) => !Number.isSafeInteger(Number(size)) || Number(size) < 0))) throw new RangeError("parquet stream invalid declared group bytes");
+		const uncompressed = group.columns.reduce((sum, col) => sum + Number(col.meta_data?.total_uncompressed_size ?? NaN), 0);
+		const compressed = group.columns.reduce((sum, col) => sum + Number(col.meta_data?.total_compressed_size ?? NaN), 0);
+		if (!Number.isSafeInteger(rows) || rows < 0 || rows > maxRowGroupRows) throw new RangeError("parquet stream exceeds maxRowGroupRows");
+		if (!Number.isSafeInteger(uncompressed) || uncompressed < 0 || uncompressed > maxRowGroupBytes || !Number.isSafeInteger(compressed) || compressed < 0 || compressed > maxRowGroupBytes) throw new RangeError("parquet stream declared group bytes exceed maxRowGroupBytes");
+		totalRows += rows;
+	}
+	if (!Number.isSafeInteger(totalRows) || BigInt(totalRows) !== metadata.num_rows) throw new Error("parquet stream row-count metadata mismatch");
+	if (values) {
+		yield* decodeBoundedValues(bounded, metadata, values, maxReadBytes, selection, maxReadAndDecompressBytes);
+		return;
+	}
+	yield* decodeGroupsFromMetadata(bounded, metadata, opts, indexedWindow && {
+		...indexedWindow,
+		maxReadBytes
+	}, maxReadAndDecompressBytes);
+}
+function indexedWindowPolicy(policy) {
+	if (policy === void 0) return void 0;
+	const { targetRows, requiredIndexedColumns, rangeCacheBytes = 0 } = policy;
+	if (!Number.isSafeInteger(targetRows) || targetRows < 1 || !Array.isArray(requiredIndexedColumns) || requiredIndexedColumns.length < 1 || requiredIndexedColumns.length > 64 || requiredIndexedColumns.some((name) => typeof name !== "string" || name.length === 0) || !Number.isSafeInteger(rangeCacheBytes) || rangeCacheBytes < 0 || rangeCacheBytes > 2097152) throw new RangeError("invalid indexedRowWindow policy");
+	return {
+		targetRows,
+		requiredIndexedColumns: [...new Set(requiredIndexedColumns)],
+		rangeCacheBytes
+	};
+}
+function windowScalar(element) {
+	if (element.num_children || element.repetition_type === "REPEATED") return false;
+	const converted = element.converted_type;
+	const logical = element.logical_type?.type;
+	if (element.type === "BYTE_ARRAY") return (!converted || converted === "UTF8") && (!logical || logical === "STRING");
+	return [
+		"BOOLEAN",
+		"INT32",
+		"INT64",
+		"INT96",
+		"FLOAT",
+		"DOUBLE"
+	].includes(element.type ?? "") && (!converted || [
+		"DATE",
+		"TIME_MILLIS",
+		"TIME_MICROS",
+		"TIMESTAMP_MILLIS",
+		"TIMESTAMP_MICROS",
+		"UINT_8",
+		"UINT_16",
+		"UINT_32",
+		"UINT_64",
+		"INT_8",
+		"INT_16",
+		"INT_32",
+		"INT_64"
+	].includes(converted)) && (!logical || [
+		"DATE",
+		"TIME",
+		"TIMESTAMP",
+		"INTEGER"
+	].includes(logical));
+}
+function indexedGroup(group, names, columns, fileBytes, policy) {
+	if (group.columns.length !== names.size || new Set(group.columns.map((col) => col.meta_data?.path_in_schema[0])).size !== names.size || group.columns.some((col) => col.meta_data?.path_in_schema.length !== 1 || !names.has(col.meta_data.path_in_schema[0]))) return false;
+	return policy.requiredIndexedColumns.every((name) => {
+		if (columns && !columns.includes(name)) return false;
+		const matches = group.columns.filter((col) => col.meta_data?.path_in_schema[0] === name);
+		if (matches.length !== 1) return false;
+		const chunk = matches[0];
+		const meta = chunk.meta_data;
+		if (meta.dictionary_page_offset !== void 0 && meta.dictionary_page_offset !== null || meta.encodings.some((encoding) => encoding === "PLAIN_DICTIONARY" || encoding === "RLE_DICTIONARY")) return false;
+		const start = Number(chunk.offset_index_offset);
+		const length = chunk.offset_index_length;
+		return Number.isSafeInteger(start) && start >= 4 && Number.isSafeInteger(length) && length > 0 && length <= policy.maxReadBytes && Number.isSafeInteger(start + length) && start + length <= fileBytes;
+	});
+}
+async function validateWindowIndexes(file, group, columns, hasFilter) {
+	for (const column of group.columns) {
+		const meta = column.meta_data;
+		if (columns && !hasFilter && !columns.includes(meta.path_in_schema[0])) continue;
+		if (!column.offset_index_offset || !column.offset_index_length) continue;
+		const start = Number(column.offset_index_offset);
+		const bytes = await file.slice(start, start + column.offset_index_length);
+		const pages = readOffsetIndex({
+			view: new DataView(bytes),
+			offset: 0
+		}).page_locations;
+		const groupRows = Number(group.num_rows);
+		const dataStart = Number(meta.data_page_offset);
+		const columnStart = Number(meta.dictionary_page_offset ?? meta.data_page_offset);
+		const columnEnd = columnStart + Number(meta.total_compressed_size);
+		if (!pages.length || pages.length > groupRows || !Number.isSafeInteger(dataStart) || dataStart < 4 || !Number.isSafeInteger(columnStart) || columnStart < 4 || !Number.isSafeInteger(columnEnd) || columnEnd > file.byteLength) throw new Error("parquet stream invalid offset index");
+		let previousRow = -1;
+		let previousEnd = dataStart;
+		for (const [index, page] of pages.entries()) {
+			const row = Number(page.first_row_index);
+			const offset = Number(page.offset);
+			const size = page.compressed_page_size;
+			if (!Number.isSafeInteger(row) || row <= previousRow || row >= groupRows || index === 0 && row !== 0 || !Number.isSafeInteger(offset) || offset < previousEnd || !Number.isSafeInteger(size) || size < 1 || !Number.isSafeInteger(offset + size) || offset + size > columnEnd) throw new Error("parquet stream invalid offset index");
+			previousRow = row;
+			previousEnd = offset + size;
+		}
+	}
+}
+async function* decodeGroupsFromMetadata(file, metadata, opts, indexedWindow, maxReadAndDecompressBytes) {
 	const tree = parquetSchema(metadata);
 	const present = /* @__PURE__ */ new Set();
 	const dateColumns = [];
@@ -238,29 +519,46 @@ async function* decodeParquetGroups(bytes, opts = {}) {
 	}
 	const columns = opts.columns?.filter((name) => present.has(name));
 	const normalizeColumns = nested ? null : dateColumns;
+	const windowSchema = indexedWindow && !nested && present.size === tree.children.length && tree.children.every((child) => windowScalar(child.element));
 	let groupStart = 0;
 	for (const group of metadata.row_groups) {
 		const groupEnd = groupStart + Number(group.num_rows);
 		const rowStart = Math.max(groupStart, opts.rowStart ?? 0);
 		const rowEnd = Math.min(groupEnd, opts.rowEnd ?? Infinity);
-		if (rowStart < rowEnd) yield normalizeDecodedDates(await parquetReadObjects({
-			file,
-			compressors: { ZSTD: (input) => decompress(input) },
-			metadata: {
-				...metadata,
-				row_groups: [group],
-				num_rows: group.num_rows
-			},
-			columns,
-			rowStart: rowStart - groupStart,
-			rowEnd: rowEnd - groupStart,
-			useOffsetIndex: rowStart > groupStart || rowEnd < groupEnd,
-			...opts.filter ? {
-				filter: opts.filter,
-				useBloomFilters: true,
-				usePageIndex: true
-			} : {}
-		}), normalizeColumns);
+		if (rowStart < rowEnd) {
+			const window = windowSchema && indexedWindow && indexedGroup(group, present, columns, file.byteLength, indexedWindow) ? indexedWindow : void 0;
+			const budget = maxReadAndDecompressBytes === void 0 ? void 0 : parquetDecodeBudget(file, maxReadAndDecompressBytes);
+			const groupFile = budget?.file ?? file;
+			const cached = window?.rangeCacheBytes ? parquetRangeCache(groupFile, window.rangeCacheBytes) : void 0;
+			try {
+				if (window) await validateWindowIndexes(cached?.file ?? groupFile, group, columns, !!opts.filter);
+				for (let start = rowStart; start < rowEnd;) {
+					const end = Math.min(rowEnd, start + (window?.targetRows ?? rowEnd - rowStart));
+					yield normalizeDecodedDates(await parquetReadObjects({
+						file: cached?.file ?? groupFile,
+						compressors: budget?.compressors ?? { ZSTD: (input) => decompress(input) },
+						metadata: {
+							...metadata,
+							row_groups: [group],
+							num_rows: group.num_rows
+						},
+						columns,
+						rowStart: start - groupStart,
+						rowEnd: end - groupStart,
+						useOffsetIndex: start > groupStart || end < groupEnd,
+						...opts.filter ? {
+							filter: opts.filter,
+							useBloomFilters: true,
+							usePageIndex: true
+						} : {}
+					}), normalizeColumns);
+					start = end;
+				}
+			} finally {
+				budget?.close();
+				cached?.close();
+			}
+		}
 		groupStart = groupEnd;
 	}
 }
@@ -339,4 +637,4 @@ function createHyparquetCodec(options = {}) {
 		}
 	};
 }
-export { createHyparquetCodec, decodeParquetToRows, encodeRowsToParquet, encodeRowsToParquetFlex };
+export { createHyparquetCodec, decodeParquetRowGroups, decodeParquetToRows, encodeParquetRowBatches, encodeRowsToParquet, encodeRowsToParquetFlex };

package/dist/adapters/parquet-bounded-values.d.mts

@@ -0,0 +1,18 @@
+import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/types/node.mjs";
+import "../storage.mjs";
+export interface BoundedValuesOptions {
+  /** Hard emitted row ceiling, default 1000. */
+  targetRows?: number;
+  /** Conservative UTF16 + row/cell charge before materialization. Default 1 MiB. */
+  maxBatchBytes?: number;
+  /** Maximum encoded bytes of one string, checked before UTF8 conversion. Default 256 KiB. */
+  maxValueBytes?: number;
+  /** Aggregate page/dictionary backing-buffer admission, not total JS heap. Default 24 MiB. */
+  maxActiveBytes?: number;
+  /** Maximum retained page descriptors across the selected input. Default 16384. */
+  maxPages?: number;
+  /** Maximum page header probe. Default 64 KiB. */
+  maxHeaderBytes?: number;
+  /** Maximum entries per dictionary; offsets consume 4 bytes per entry. Default 100000. */
+  maxDictionaryEntries?: number;
+}