@gscdump/engine 7.4.0 → 7.6.0

package/README.md

@@ -42,12 +42,136 @@ options produces identical bytes regardless of producer batch boundaries.
 This supports a counting/digest pass followed by a separately coordinated
 fixed-length upload, without the codec owning transport or commit semantics.
 
+The streaming encoder normally refuses when retained footer/index descriptors
+exceed `limits.maxRetainedMetadataBytes` (8 MiB by default). For wide values,
+callers can explicitly choose `metadataOverflow: 'omit-statistics'`. On the
+first overflow it omits column-chunk statistics and ColumnIndexes throughout
+that file, including earlier groups. OffsetIndexes, encoding descriptors,
+schema, row counts, data/dictionary pages and explicit `fileMetadata` remain
+unchanged. Required descriptors must still fit the same budget, and the
+group/footer output-buffer limit still applies. Missing statistics mean unknown
+bounds, so readers may scan more pages or groups; they do not mean empty data.
+If no overflow occurs, this option preserves the normal bytes. It bounds
+retained optional descriptors, not their transient construction or total heap.
+The array encoders and all omitted-option behavior remain unchanged.
+
 `decodeParquetRowGroups({ byteLength, slice }, options)` reads exact byte ranges
-and yields one physical row group at a time. The caller must pin every range
+and by default yields one physical row group at a time. The caller must pin every range
 to the same immutable object or ETag. It preserves the existing date, column
 projection, row-window and filter behavior. An absent, corrupt, oversized or
 incompletely read file throws; only valid empty Parquet yields no rows.
 
+The range reader alone accepts an optional `indexedRowWindow` policy:
+
+```ts
+decodeParquetRowGroups(pinnedObject, {
+  indexedRowWindow: {
+    targetRows: 1000,
+    requiredIndexedColumns: ['query', 'url'],
+    rangeCacheBytes: 2 * 1024 * 1024,
+  },
+})
+```
+
+Each group uses windows only when all designated columns are present, selected,
+have usable offset-index descriptors, and have no dictionary encoding. The
+schema must contain only flat scalar/UTF8/date fields; binary, nested and other
+unsupported schemas retain the original group read. Ineligible groups use the
+original uncached path, so `targetRows` is a target rather than a hard batch
+limit. Include every potentially large column in `requiredIndexedColumns`.
+All existing physical-group admission limits still apply.
+
+Before emitting an eligible group's first window, the reader checks index
+structure for projected columns, or all indexed columns when a filter may need
+unprojected fields: row zero, increasing row positions, ordered nonoverlapping
+page ranges and physical column bounds. A failure throws without falling back
+after partial output. This assumes metadata row offsets agree with physical
+page contents; it does not independently verify page-header row counts or
+detect every deliberately consistent but false index.
+
+The optional range cache defaults to zero, allows at most 2 MiB of payload and
+128 entries, and lives only through one eligible group. It is cleared on
+completion, error or iterator return; another pass rereads the pinned object.
+It does not cache decoded rows or provide publication/ETag coordination. Windows
+can otherwise reread overlapping pages and substantially increase range I/O.
+They still decode whole boundary pages, and dictionaries in other columns can
+still allocate in full. These options do not bound decompressed page sizes or
+the total heap; the unchanged fallback also retains its existing memory costs.
+Buffered decoders, encoders and all omitted-option behavior remain unchanged.
+
+The range reader can also opt into `limits.maxReadAndDecompressBytes`. This is a
+**cumulative buffer-work budget for each physical group**, shared across all its
+indexed windows and concurrent column reads. It charges requested source-range
+bytes before dispatch and SNAPPY output bytes before allocating the fixed-size
+output buffer. Cache hits do not allocate another range, while repeated
+decompression still consumes budget. Group indexes are included; the initial
+footer has its separate metadata limit. Raw/uncompressed pages and V2 levels
+are covered by the range charge, so compressed data is deliberately counted
+twice. Earlier charges are never refunded, including after a failed read. Failure
+or iterator return closes the budget, preventing new work and rejecting late
+range results. It cannot physically cancel already dispatched reads: the range
+source interface has no cancellation hook.
+
+ZSTD decompression refuses under this option: the current allocating decoder
+can allocate from its frame header before checking the Parquet output size.
+Without the option, ZSTD and all existing decoding behavior remain unchanged.
+The budget does not measure JavaScript strings, expanded values, row containers,
+internal prefetch or dictionary/page stitching copies, codec working state or
+caller-retained batches. It is
+not a total heap bound or protection against all malformed Parquet contents.
+It can reject valid files that previously decoded, particularly old files whose
+writer understated uncompressed footer sizes. Callers must preserve those files
+and choose their own explicit retry/repair policy; refusal is never empty data.
+
+`decodeParquetRowGroups(source, { boundedValues: {} })` selects a separate strict
+value-cursor path. It supports flat REQUIRED/OPTIONAL scalar columns, explicitly
+annotated UTF8/STRING, PLAIN and dictionary data pages (V1/V2), and SNAPPY or
+UNCOMPRESSED data. Projection, physical row ranges, nulls, date normalization and
+row order are preserved. Nested/binary/other logical types, other codecs or value
+encodings, BOOLEAN dictionaries, filters and `indexedRowWindow` combinations
+refuse; this mode never silently falls back to the eager reader. All selected
+page headers are inspected before the first output to detect unsupported later
+pages. Corrupt value bodies or over-limit individual values may fail after an
+earlier batch; consumers must finish validating their entire input before
+publishing a replacement.
+Explicit row bounds in this strict mode must be nonnegative safe integers, with
+positive Infinity also accepted for an open-ended `rowEnd`. Omitted bounds use
+the full input, and reversed or zero-length ranges are empty.
+
+The cursor retains at most one current data page and one encoded dictionary per
+selected column. UTF8 strings, definition levels and dictionary IDs are consumed
+incrementally. Dictionaries retain encoded bytes and a four-byte offset per
+entry, rather than an array of decoded strings. Final bit-packed padding is
+validated but not expanded. These options have independent meanings:
+
+| `boundedValues` option | Default | Admission rule |
+| --- | --- | --- |
+| `targetRows` | 1000 | Hard emitted batch row limit |
+| `maxBatchBytes` | 1 MiB | Before materialization: 128 per row, 16 per cell slot, 64 per cell plus twice UTF8 byte length or 32 for a fixed scalar |
+| `maxValueBytes` | 256 KiB | Encoded bytes of one string, before decoding |
+| `maxActiveBytes` | 24 MiB | Per group: sum of each column's largest compressed + uncompressed data page, dictionary compressed + uncompressed bytes, dictionary offset tables, plus one maximum header probe |
+| `maxPages` | 16384 | Retained page-descriptor count across the selected input |
+| `maxHeaderBytes` | 64 KiB | Header probe cap; starts at 512 bytes and grows only on truncation |
+| `maxDictionaryEntries` | 100000 | Entry count for each encoded dictionary |
+
+Active-buffer admission uses real page headers, not the often understated
+uncompressed sizes in legacy writer footers. Page transitions release the old
+cursor before reading/decompressing its successor; columns load sequentially.
+Batch bytes are a conservative value-materialization charge, **not a measurement
+of V8 object overhead or an isolate heap guarantee**. The footer (separately
+bounded), page-descriptor objects, fixed cursor/schema state, caller-retained
+batches, codec/runtime overhead and concurrent application work remain separate.
+Setting larger limits deliberately permits a larger working set. A single page
+still decompresses in full, and a consumer that retains all emitted batches
+removes the benefit. This mode does not make arbitrary malicious files safe.
+
+The existing `maxReadAndDecompressBytes` option also applies: a group's header
+probes, physical reads and SNAPPY output share one cumulative budget. Headers
+are scanned once; immutable/pinned object bytes are required across that scan
+and later reads. Iterator return/error releases cursor references; already
+dispatched source reads cannot be physically aborted. Defaults, the buffered
+decoder and indexed-window eligibility/fallback are unchanged.
+
 Both APIs have configurable admission limits. The encoder bounds producer
 batch rows/value bytes, output group rows/value bytes, retained descriptor counts
 and scalar bytes (including statistics), and the current encoded group/footer

package/dist/adapters/hyparquet.d.mts

@@ -2,6 +2,7 @@ import { ParquetQueryFilter } from "../node_modules/.pnpm/hyparquet@1.31.1/node_
 import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/types/node.mjs";
 import { CodecCtx, DataSource, ParquetCodec, Row, TableName, WriteResult } from "../storage.mjs";
 import { ColumnDef } from "../schema.mjs";
+import { BoundedValuesOptions } from "./parquet-bounded-values.mjs";
 export interface EncodeParquetOptions {
   /** File footer metadata, without adding fact columns. Keys are encoded in sorted order. */
   fileMetadata?: Readonly<Record<string, string>>;
@@ -28,6 +29,13 @@ export interface EncodeParquetBatchesOptions extends EncodeParquetOptions {
   columns: readonly ColumnDef[];
   /** Maximum rows per output group. Default 25000, matching the array codec. */
   rowGroupSize?: number;
+  /**
+   * Opt in to omitting all footer column statistics and ColumnIndexes if they
+   * exceed the retained metadata limit. Default: refuse. OffsetIndexes,
+   * required descriptors and explicit file metadata remain bounded/preserved.
+   * Missing statistics reduce pruning; this does not bound transient codec heap.
+   */
+  metadataOverflow?: 'omit-statistics';
   limits?: {
     /** Refuse a producer batch with more rows. Default 25000. */
     maxBatchRows?: number;
@@ -85,6 +93,25 @@ export interface ParquetRangeSource {
   slice: (start: number, end?: number) => ArrayBuffer | Promise<ArrayBuffer>;
 }
 export interface DecodeParquetGroupsOptions extends DecodeParquetOptions {
+  /**
+   * Strict opt-in value cursors for flat scalar PLAIN/dictionary SNAPPY or raw
+   * pages. Rejects unsupported inputs instead of falling back to eager decoding.
+   * Page buffers and emitted values have separate bounds; not an isolate heap cap.
+   * Cannot be combined with filters or indexedRowWindow.
+   */
+  boundedValues?: BoundedValuesOptions;
+  /**
+   * Opt-in windows for flat scalar groups whose designated, selected columns
+   * have offset indexes and no dictionary encoding. Ineligible groups retain
+   * the original physical-group read, so targetRows is not a hard batch limit.
+   * Windows still decode complete overlapping pages, not a bounded heap size.
+   */
+  indexedRowWindow?: {
+    targetRows: number;
+    requiredIndexedColumns: readonly string[];
+    /** Per eligible group; 0 disables caching. Default 0, maximum 2 MiB/128 entries. */
+    rangeCacheBytes?: number;
+  };
   limits?: {
     /** Footer bytes before deserialization. Default 1 MiB. */
     maxMetadataBytes?: number;
@@ -96,6 +123,13 @@ export interface DecodeParquetGroupsOptions extends DecodeParquetOptions {
     maxRowGroups?: number;
     /** Maximum individual range request. Default 16 MiB. */
     maxReadBytes?: number;
+    /**
+     * Optional cumulative source-range + SNAPPY-output bytes per physical group,
+     * including all indexed windows. Charged before reads/output allocation;
+     * cache hits do not allocate another range. Deliberately conservative, not
+     * total JS heap. ZSTD refuses only with this option. Omission is unchanged.
+     */
+    maxReadAndDecompressBytes?: number;
   };
 }
 /**

package/dist/adapters/hyparquet.mjs

@@ -2,12 +2,16 @@ import { TABLE_METADATA } from "../drizzle-schema.mjs";
 import { SCHEMAS } from "../schema.mjs";
 import { decompress } from "../node_modules/.pnpm/fzstd@0.1.1/node_modules/fzstd/esm/index.mjs";
 import { parquetMetadata, parquetMetadataAsync, parquetSchema } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/metadata.mjs";
+import { readOffsetIndex } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/indexes.mjs";
 import { parquetReadObjects } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/read.mjs";
 import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs";
 import { ByteWriter } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/bytewriter.mjs";
 import { ParquetWriter } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/parquet-writer.mjs";
 import { parquetWriteRows } from "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/write-rows.mjs";
 import "../node_modules/.pnpm/hyparquet-writer@0.16.10_patch_hash_d26d002aede61ad95dd633d6557b56897cdcbc45dfc3a5d93fa0e9075ad5425f/node_modules/hyparquet-writer/src/node.mjs";
+import { parquetDecodeBudget } from "./parquet-decode-budget.mjs";
+import { boundedValuesPolicy, decodeBoundedValues } from "./parquet-bounded-values.mjs";
+import { parquetRangeCache } from "./parquet-range-cache.mjs";
 const ROW_GROUP_SIZE = 25e3;
 const DEFAULT_COMPACTION_READ_CONCURRENCY = 8;
 const MAX_COMPACTION_READ_CONCURRENCY = 32;
@@ -215,6 +219,11 @@ function streamLimit(value, fallback, name) {
 function scalarBytes(value) {
 	return typeof value === "string" ? value.length * 3 + 8 : 16;
 }
+var RetainedMetadataLimitError = class extends RangeError {
+	constructor() {
+		super("parquet stream exceeds maxRetainedMetadataBytes");
+	}
+};
 function retainedDescriptorBytes(value, remaining) {
 	const seen = /* @__PURE__ */ new Set();
 	let bytes = 0;
@@ -229,12 +238,14 @@ function retainedDescriptorBytes(value, remaining) {
 				visit(value[key]);
 			}
 		} else bytes += scalarBytes(value);
-		if (bytes > remaining) throw new RangeError("parquet stream exceeds maxRetainedMetadataBytes");
+		if (bytes > remaining) throw new RetainedMetadataLimitError();
 	};
 	visit(value);
 	return bytes;
 }
 async function encodeParquetRowBatches(batches, opts, write) {
+	const metadataOverflow = opts.metadataOverflow;
+	if (metadataOverflow !== void 0 && metadataOverflow !== "omit-statistics") throw new RangeError("parquet stream metadataOverflow must be omit-statistics when provided");
 	const rowGroupSize = streamLimit(opts.rowGroupSize, ROW_GROUP_SIZE, "rowGroupSize");
 	const maxBatchRows = streamLimit(opts.limits?.maxBatchRows, ROW_GROUP_SIZE, "maxBatchRows");
 	const maxBatchBytes = streamLimit(opts.limits?.maxBatchBytes, 8388608, "maxBatchBytes");
@@ -256,7 +267,8 @@ async function encodeParquetRowBatches(batches, opts, write) {
 		metadataBytes += scalarBytes(key) + scalarBytes(opts.fileMetadata[key]);
 		if (metadataBytes > maxOutputBufferBytes) throw new RangeError("parquet stream file metadata exceeds maxOutputBufferBytes");
 	}
-	let retainedMetadataBytes = retainedDescriptorBytes([schema, opts.fileMetadata], maxRetainedMetadataBytes);
+	const initialMetadataBytes = retainedDescriptorBytes([schema, opts.fileMetadata], maxRetainedMetadataBytes);
+	let retainedMetadataBytes = initialMetadataBytes;
 	const writer = new ByteWriter(Math.min(65536, maxOutputBufferBytes));
 	writer.ensure = (size) => {
 		const required = writer.index + size;
@@ -289,6 +301,14 @@ async function encodeParquetRowBatches(batches, opts, write) {
 	let groupBytes = 0;
 	let groups = 0;
 	let rowCount = 0;
+	let omitStatistics = false;
+	const omitOptionalMetadata = (firstIndex) => {
+		for (let index = firstIndex; index < parquet.pendingIndexes.length; index++) {
+			const pending = parquet.pendingIndexes[index];
+			if (pending.chunk.meta_data) delete pending.chunk.meta_data.statistics;
+			delete pending.columnIndex;
+		}
+	};
 	const flushGroup = async () => {
 		if (!groupRows) return;
 		if (++groups > maxRowGroups) throw new RangeError("parquet stream exceeds maxRowGroups");
@@ -302,7 +322,21 @@ async function encodeParquetRowBatches(batches, opts, write) {
 			})),
 			rowGroupSize: groupRows
 		});
-		retainedMetadataBytes += retainedDescriptorBytes([parquet.row_groups[parquet.row_groups.length - 1], parquet.pendingIndexes.slice(firstIndex)], maxRetainedMetadataBytes - retainedMetadataBytes);
+		if (omitStatistics) omitOptionalMetadata(firstIndex);
+		try {
+			retainedMetadataBytes += retainedDescriptorBytes([parquet.row_groups[parquet.row_groups.length - 1], parquet.pendingIndexes.slice(firstIndex)], maxRetainedMetadataBytes - retainedMetadataBytes);
+		} catch (error) {
+			if (!(error instanceof RetainedMetadataLimitError) || metadataOverflow !== "omit-statistics" || omitStatistics) throw error;
+			omitStatistics = true;
+			omitOptionalMetadata(0);
+			retainedMetadataBytes = initialMetadataBytes;
+			let index = 0;
+			for (const group of parquet.row_groups) {
+				const end = index + group.columns.length;
+				retainedMetadataBytes += retainedDescriptorBytes([group, parquet.pendingIndexes.slice(index, end)], maxRetainedMetadataBytes - retainedMetadataBytes);
+				index = end;
+			}
+		}
 		values = codecs.map(() => []);
 		groupRows = 0;
 		groupBytes = 0;
@@ -363,6 +397,19 @@ async function* decodeParquetGroups(bytes, opts = {}) {
 	yield* decodeGroupsFromMetadata(file, await parquetMetadataAsync(file), opts);
 }
 async function* decodeParquetRowGroups(file, opts = {}) {
+	const values = boundedValuesPolicy(opts.boundedValues);
+	const selection = values ? {
+		columns: opts.columns?.slice(),
+		rowStart: opts.rowStart,
+		rowEnd: opts.rowEnd
+	} : void 0;
+	if (values) for (const key of ["rowStart", "rowEnd"]) {
+		const value = selection[key];
+		if (value !== void 0 && !(key === "rowEnd" && value === Infinity) && (!Number.isSafeInteger(value) || value < 0)) throw new RangeError(`invalid boundedValues.${key}`);
+	}
+	if (values && (opts.filter || opts.indexedRowWindow)) throw new Error("boundedValues cannot be combined with filter or indexedRowWindow");
+	const indexedWindow = indexedWindowPolicy(opts.indexedRowWindow);
+	const maxReadAndDecompressBytes = opts.limits?.maxReadAndDecompressBytes === void 0 ? void 0 : streamLimit(opts.limits.maxReadAndDecompressBytes, 0, "maxReadAndDecompressBytes");
 	const maxMetadataBytes = streamLimit(opts.limits?.maxMetadataBytes, 1048576, "maxMetadataBytes");
 	const maxRowGroupRows = streamLimit(opts.limits?.maxRowGroupRows, ROW_GROUP_SIZE, "maxRowGroupRows");
 	const maxRowGroupBytes = streamLimit(opts.limits?.maxRowGroupBytes, 16777216, "maxRowGroupBytes");
@@ -396,9 +443,101 @@ async function* decodeParquetRowGroups(file, opts = {}) {
 		totalRows += rows;
 	}
 	if (!Number.isSafeInteger(totalRows) || BigInt(totalRows) !== metadata.num_rows) throw new Error("parquet stream row-count metadata mismatch");
-	yield* decodeGroupsFromMetadata(bounded, metadata, opts);
+	if (values) {
+		yield* decodeBoundedValues(bounded, metadata, values, maxReadBytes, selection, maxReadAndDecompressBytes);
+		return;
+	}
+	yield* decodeGroupsFromMetadata(bounded, metadata, opts, indexedWindow && {
+		...indexedWindow,
+		maxReadBytes
+	}, maxReadAndDecompressBytes);
+}
+function indexedWindowPolicy(policy) {
+	if (policy === void 0) return void 0;
+	const { targetRows, requiredIndexedColumns, rangeCacheBytes = 0 } = policy;
+	if (!Number.isSafeInteger(targetRows) || targetRows < 1 || !Array.isArray(requiredIndexedColumns) || requiredIndexedColumns.length < 1 || requiredIndexedColumns.length > 64 || requiredIndexedColumns.some((name) => typeof name !== "string" || name.length === 0) || !Number.isSafeInteger(rangeCacheBytes) || rangeCacheBytes < 0 || rangeCacheBytes > 2097152) throw new RangeError("invalid indexedRowWindow policy");
+	return {
+		targetRows,
+		requiredIndexedColumns: [...new Set(requiredIndexedColumns)],
+		rangeCacheBytes
+	};
 }
-async function* decodeGroupsFromMetadata(file, metadata, opts) {
+function windowScalar(element) {
+	if (element.num_children || element.repetition_type === "REPEATED") return false;
+	const converted = element.converted_type;
+	const logical = element.logical_type?.type;
+	if (element.type === "BYTE_ARRAY") return (!converted || converted === "UTF8") && (!logical || logical === "STRING");
+	return [
+		"BOOLEAN",
+		"INT32",
+		"INT64",
+		"INT96",
+		"FLOAT",
+		"DOUBLE"
+	].includes(element.type ?? "") && (!converted || [
+		"DATE",
+		"TIME_MILLIS",
+		"TIME_MICROS",
+		"TIMESTAMP_MILLIS",
+		"TIMESTAMP_MICROS",
+		"UINT_8",
+		"UINT_16",
+		"UINT_32",
+		"UINT_64",
+		"INT_8",
+		"INT_16",
+		"INT_32",
+		"INT_64"
+	].includes(converted)) && (!logical || [
+		"DATE",
+		"TIME",
+		"TIMESTAMP",
+		"INTEGER"
+	].includes(logical));
+}
+function indexedGroup(group, names, columns, fileBytes, policy) {
+	if (group.columns.length !== names.size || new Set(group.columns.map((col) => col.meta_data?.path_in_schema[0])).size !== names.size || group.columns.some((col) => col.meta_data?.path_in_schema.length !== 1 || !names.has(col.meta_data.path_in_schema[0]))) return false;
+	return policy.requiredIndexedColumns.every((name) => {
+		if (columns && !columns.includes(name)) return false;
+		const matches = group.columns.filter((col) => col.meta_data?.path_in_schema[0] === name);
+		if (matches.length !== 1) return false;
+		const chunk = matches[0];
+		const meta = chunk.meta_data;
+		if (meta.dictionary_page_offset !== void 0 && meta.dictionary_page_offset !== null || meta.encodings.some((encoding) => encoding === "PLAIN_DICTIONARY" || encoding === "RLE_DICTIONARY")) return false;
+		const start = Number(chunk.offset_index_offset);
+		const length = chunk.offset_index_length;
+		return Number.isSafeInteger(start) && start >= 4 && Number.isSafeInteger(length) && length > 0 && length <= policy.maxReadBytes && Number.isSafeInteger(start + length) && start + length <= fileBytes;
+	});
+}
+async function validateWindowIndexes(file, group, columns, hasFilter) {
+	for (const column of group.columns) {
+		const meta = column.meta_data;
+		if (columns && !hasFilter && !columns.includes(meta.path_in_schema[0])) continue;
+		if (!column.offset_index_offset || !column.offset_index_length) continue;
+		const start = Number(column.offset_index_offset);
+		const bytes = await file.slice(start, start + column.offset_index_length);
+		const pages = readOffsetIndex({
+			view: new DataView(bytes),
+			offset: 0
+		}).page_locations;
+		const groupRows = Number(group.num_rows);
+		const dataStart = Number(meta.data_page_offset);
+		const columnStart = Number(meta.dictionary_page_offset ?? meta.data_page_offset);
+		const columnEnd = columnStart + Number(meta.total_compressed_size);
+		if (!pages.length || pages.length > groupRows || !Number.isSafeInteger(dataStart) || dataStart < 4 || !Number.isSafeInteger(columnStart) || columnStart < 4 || !Number.isSafeInteger(columnEnd) || columnEnd > file.byteLength) throw new Error("parquet stream invalid offset index");
+		let previousRow = -1;
+		let previousEnd = dataStart;
+		for (const [index, page] of pages.entries()) {
+			const row = Number(page.first_row_index);
+			const offset = Number(page.offset);
+			const size = page.compressed_page_size;
+			if (!Number.isSafeInteger(row) || row <= previousRow || row >= groupRows || index === 0 && row !== 0 || !Number.isSafeInteger(offset) || offset < previousEnd || !Number.isSafeInteger(size) || size < 1 || !Number.isSafeInteger(offset + size) || offset + size > columnEnd) throw new Error("parquet stream invalid offset index");
+			previousRow = row;
+			previousEnd = offset + size;
+		}
+	}
+}
+async function* decodeGroupsFromMetadata(file, metadata, opts, indexedWindow, maxReadAndDecompressBytes) {
 	const tree = parquetSchema(metadata);
 	const present = /* @__PURE__ */ new Set();
 	const dateColumns = [];
@@ -410,29 +549,46 @@ async function* decodeGroupsFromMetadata(file, metadata, opts) {
 	}
 	const columns = opts.columns?.filter((name) => present.has(name));
 	const normalizeColumns = nested ? null : dateColumns;
+	const windowSchema = indexedWindow && !nested && present.size === tree.children.length && tree.children.every((child) => windowScalar(child.element));
 	let groupStart = 0;
 	for (const group of metadata.row_groups) {
 		const groupEnd = groupStart + Number(group.num_rows);
 		const rowStart = Math.max(groupStart, opts.rowStart ?? 0);
 		const rowEnd = Math.min(groupEnd, opts.rowEnd ?? Infinity);
-		if (rowStart < rowEnd) yield normalizeDecodedDates(await parquetReadObjects({
-			file,
-			compressors: { ZSTD: (input) => decompress(input) },
-			metadata: {
-				...metadata,
-				row_groups: [group],
-				num_rows: group.num_rows
-			},
-			columns,
-			rowStart: rowStart - groupStart,
-			rowEnd: rowEnd - groupStart,
-			useOffsetIndex: rowStart > groupStart || rowEnd < groupEnd,
-			...opts.filter ? {
-				filter: opts.filter,
-				useBloomFilters: true,
-				usePageIndex: true
-			} : {}
-		}), normalizeColumns);
+		if (rowStart < rowEnd) {
+			const window = windowSchema && indexedWindow && indexedGroup(group, present, columns, file.byteLength, indexedWindow) ? indexedWindow : void 0;
+			const budget = maxReadAndDecompressBytes === void 0 ? void 0 : parquetDecodeBudget(file, maxReadAndDecompressBytes);
+			const groupFile = budget?.file ?? file;
+			const cached = window?.rangeCacheBytes ? parquetRangeCache(groupFile, window.rangeCacheBytes) : void 0;
+			try {
+				if (window) await validateWindowIndexes(cached?.file ?? groupFile, group, columns, !!opts.filter);
+				for (let start = rowStart; start < rowEnd;) {
+					const end = Math.min(rowEnd, start + (window?.targetRows ?? rowEnd - rowStart));
+					yield normalizeDecodedDates(await parquetReadObjects({
+						file: cached?.file ?? groupFile,
+						compressors: budget?.compressors ?? { ZSTD: (input) => decompress(input) },
+						metadata: {
+							...metadata,
+							row_groups: [group],
+							num_rows: group.num_rows
+						},
+						columns,
+						rowStart: start - groupStart,
+						rowEnd: end - groupStart,
+						useOffsetIndex: start > groupStart || end < groupEnd,
+						...opts.filter ? {
+							filter: opts.filter,
+							useBloomFilters: true,
+							usePageIndex: true
+						} : {}
+					}), normalizeColumns);
+					start = end;
+				}
+			} finally {
+				budget?.close();
+				cached?.close();
+			}
+		}
 		groupStart = groupEnd;
 	}
 }

package/dist/adapters/parquet-bounded-values.d.mts

@@ -0,0 +1,18 @@
+import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/types/node.mjs";
+import "../storage.mjs";
+export interface BoundedValuesOptions {
+  /** Hard emitted row ceiling, default 1000. */
+  targetRows?: number;
+  /** Conservative UTF16 + row/cell charge before materialization. Default 1 MiB. */
+  maxBatchBytes?: number;
+  /** Maximum encoded bytes of one string, checked before UTF8 conversion. Default 256 KiB. */
+  maxValueBytes?: number;
+  /** Aggregate page/dictionary backing-buffer admission, not total JS heap. Default 24 MiB. */
+  maxActiveBytes?: number;
+  /** Maximum retained page descriptors across the selected input. Default 16384. */
+  maxPages?: number;
+  /** Maximum page header probe. Default 64 KiB. */
+  maxHeaderBytes?: number;
+  /** Maximum entries per dictionary; offsets consume 4 bytes per entry. Default 100000. */
+  maxDictionaryEntries?: number;
+}

package/dist/adapters/parquet-bounded-values.mjs

@@ -0,0 +1,318 @@
+import { snappyUncompress } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/snappy.mjs";
+import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs";
+import { parquetDecodeBudget } from "./parquet-decode-budget.mjs";
+import { BoundedHeaderTruncatedError, boundedPageHeader } from "./parquet-page-header.mjs";
+import { hybridCursor, plainCursor } from "./parquet-value-cursor.mjs";
+function boundedValuesPolicy(options) {
+	if (options === void 0) return void 0;
+	const policy = {
+		targetRows: 1e3,
+		maxBatchBytes: 1048576,
+		maxValueBytes: 262144,
+		maxActiveBytes: 25165824,
+		maxPages: 16384,
+		maxHeaderBytes: 65536,
+		maxDictionaryEntries: 1e5,
+		...options
+	};
+	for (const [key, value] of Object.entries(policy)) if (!Number.isSafeInteger(value) || value < 1) throw new RangeError(`invalid boundedValues.${key}`);
+	return policy;
+}
+const supportedEncodings = /* @__PURE__ */ new Set([
+	0,
+	2,
+	8
+]);
+function natural(value, label) {
+	if (!Number.isSafeInteger(value) || value < 0) throw new Error(`bounded Parquet invalid ${label}`);
+	return value;
+}
+function scalar(element) {
+	if (element.num_children || !["REQUIRED", "OPTIONAL"].includes(element.repetition_type ?? "")) return false;
+	const converted = element.converted_type;
+	const logical = element.logical_type?.type;
+	if (element.type === "BYTE_ARRAY") return (converted === "UTF8" || logical === "STRING") && (!converted || converted === "UTF8") && (!logical || logical === "STRING");
+	return [
+		"BOOLEAN",
+		"INT32",
+		"INT64",
+		"INT96",
+		"FLOAT",
+		"DOUBLE"
+	].includes(element.type ?? "") && (!converted || [
+		"DATE",
+		"TIME_MILLIS",
+		"TIME_MICROS",
+		"TIMESTAMP_MILLIS",
+		"TIMESTAMP_MICROS",
+		"UINT_8",
+		"UINT_16",
+		"UINT_32",
+		"UINT_64",
+		"INT_8",
+		"INT_16",
+		"INT_32",
+		"INT_64"
+	].includes(converted)) && (!logical || [
+		"DATE",
+		"TIME",
+		"TIMESTAMP",
+		"INTEGER"
+	].includes(logical));
+}
+async function readHeader(file, start, end, maxBytes) {
+	const limit = Math.min(end - start, maxBytes);
+	for (let length = Math.min(512, limit);;) try {
+		return boundedPageHeader(await file.slice(start, start + length));
+	} catch (error) {
+		if (!(error instanceof BoundedHeaderTruncatedError) || length === limit) throw error;
+		length = Math.min(limit, length * 2);
+	}
+}
+async function planInput(file, metadata, policy, maxReadBytes, columns, rowStart = 0, rowEnd = Infinity, maxReadAndDecompressBytes) {
+	const schema = metadata.schema.slice(1);
+	if (metadata.schema[0]?.num_children !== schema.length || schema.some((element) => !scalar(element)) || new Set(schema.map((element) => element.name)).size !== schema.length) throw new Error("bounded Parquet requires a flat scalar REQUIRED/OPTIONAL schema");
+	const selected = columns ? schema.filter((element) => columns.includes(element.name)) : schema;
+	const output = [];
+	let start = 0;
+	let pageCount = 0;
+	try {
+		for (const group of metadata.row_groups) {
+			const rows = Number(group.num_rows);
+			const plan = {
+				start,
+				rows,
+				columns: []
+			};
+			start += rows;
+			if (start <= rowStart || plan.start >= rowEnd) continue;
+			plan.budget = maxReadAndDecompressBytes === void 0 ? void 0 : parquetDecodeBudget(file, maxReadAndDecompressBytes);
+			output.push(plan);
+			const groupFile = plan.budget?.file ?? file;
+			let active = Math.min(policy.maxHeaderBytes, maxReadBytes);
+			for (const element of selected) {
+				const matches = group.columns.filter((col) => col.meta_data?.path_in_schema.length === 1 && col.meta_data.path_in_schema[0] === element.name);
+				if (matches.length !== 1) throw new Error("bounded Parquet missing or duplicate column");
+				const chunk = matches[0];
+				const meta = chunk.meta_data;
+				if (chunk.file_path || meta.type !== element.type || Number(meta.num_values) !== rows || !["SNAPPY", "UNCOMPRESSED"].includes(meta.codec)) throw new Error("bounded Parquet unsupported column or codec");
+				const first = Number(meta.dictionary_page_offset ?? meta.data_page_offset);
+				const end = first + Number(meta.total_compressed_size);
+				if (!Number.isSafeInteger(first) || first < 4 || !Number.isSafeInteger(end) || end > file.byteLength || end < first) throw new Error("bounded Parquet invalid column range");
+				const column = {
+					element,
+					meta,
+					pages: []
+				};
+				let pageRows = 0;
+				let maxData = 0;
+				for (let offset = first; offset < end;) {
+					if (++pageCount > policy.maxPages) throw new RangeError("bounded Parquet exceeds maxPages");
+					const header = await readHeader(groupFile, offset, end, Math.min(policy.maxHeaderBytes, maxReadBytes));
+					natural(header.compressed, "compressed page bytes");
+					natural(header.uncompressed, "uncompressed page bytes");
+					natural(header.count, "page values");
+					const page = {
+						...header,
+						offset: offset + header.bytes
+					};
+					offset = page.offset + page.compressed;
+					if (offset > end || page.compressed > maxReadBytes) throw new RangeError("bounded Parquet page exceeds range bounds");
+					if (page.kind === 2) {
+						if (column.dictionary || column.pages.length || page.encoding !== 0 || page.count > policy.maxDictionaryEntries) throw new Error("bounded Parquet unsupported dictionary");
+						column.dictionary = page;
+						active += page.compressed + page.uncompressed + (page.count + 1) * 4;
+					} else {
+						if (!page.count || !supportedEncodings.has(page.encoding) || page.encoding !== 0 && !column.dictionary) throw new Error("bounded Parquet unsupported data encoding");
+						if (page.kind === 0 && (page.definitionEncoding !== 3 || page.repetitionEncoding !== 3)) throw new Error("bounded Parquet unsupported level encoding");
+						if (page.kind === 3) {
+							natural(page.nulls, "null count");
+							natural(page.rows, "page rows");
+							natural(page.definitions, "definition bytes");
+							natural(page.repetitions, "repetition bytes");
+							if (page.rows !== page.count || page.nulls > page.count || page.repetitions !== 0 || page.definitions > page.compressed || page.definitions > page.uncompressed || element.repetition_type === "REQUIRED" && (page.definitions !== 0 || page.nulls !== 0)) throw new Error("bounded Parquet invalid flat V2 page");
+						}
+						pageRows += page.count;
+						if (pageRows > rows) throw new Error("bounded Parquet excess physical rows");
+						maxData = Math.max(maxData, page.compressed + page.uncompressed);
+						column.pages.push(page);
+					}
+				}
+				if (pageRows !== rows) throw new Error("bounded Parquet incomplete physical rows");
+				if (column.dictionary && element.type === "BOOLEAN") throw new Error("bounded Parquet BOOLEAN dictionary is unsupported");
+				active += maxData;
+				if (!Number.isSafeInteger(active) || active > policy.maxActiveBytes) throw new RangeError("bounded Parquet exceeds maxActiveBytes");
+				plan.columns.push(column);
+			}
+		}
+		return output;
+	} catch (error) {
+		for (const plan of output) plan.budget?.close();
+		throw error;
+	}
+}
+function inflate(input, size, compressed, decompress) {
+	if (!compressed) {
+		if (input.length !== size) throw new Error("bounded Parquet uncompressed page size mismatch");
+		return input;
+	}
+	if (decompress) return decompress(input, size);
+	const output = new Uint8Array(size);
+	snappyUncompress(input, output);
+	return output;
+}
+function dataCursor(page, input, plan, policy, dictionary, decompress) {
+	let definitions;
+	let levels;
+	let values;
+	if (page.kind === 0) {
+		const decoded = inflate(input, page.uncompressed, plan.meta.codec === "SNAPPY", decompress);
+		let offset = 0;
+		if (plan.element.repetition_type === "OPTIONAL") {
+			if (decoded.length < 4) throw new Error("bounded Parquet missing definition levels");
+			offset = 4 + new DataView(decoded.buffer, decoded.byteOffset, decoded.byteLength).getUint32(0, true);
+			if (offset > decoded.length) throw new Error("bounded Parquet truncated definition levels");
+			levels = decoded.subarray(4, offset);
+		}
+		values = decoded.subarray(offset);
+	} else {
+		const length = page.definitions;
+		if (plan.element.repetition_type === "OPTIONAL") levels = input.subarray(0, length);
+		values = inflate(input.subarray(length), page.uncompressed - length, page.isCompressed !== false && plan.meta.codec === "SNAPPY", decompress);
+	}
+	let present = page.count;
+	if (levels) {
+		definitions = hybridCursor(levels, 1, page.count);
+		present = 0;
+		for (let i = 0; i < page.count; i++) present += definitions.next();
+		definitions.finish();
+		definitions = hybridCursor(levels, 1, page.count);
+	}
+	if (page.kind === 3 && page.count - present !== page.nulls) throw new Error("bounded Parquet null count mismatch");
+	let plain;
+	let ids;
+	if (page.encoding === 0) plain = plainCursor(values, plan.element, present, policy.maxValueBytes);
+	else {
+		if (!dictionary || !values.length) throw new Error("bounded Parquet missing dictionary IDs");
+		ids = hybridCursor(values.subarray(1), values[0], present);
+	}
+	return {
+		next() {
+			if (definitions && definitions.next() === 0) return {
+				bytes: 64,
+				value: () => null
+			};
+			if (plain) return plain.next();
+			const id = ids.next();
+			if (id >= dictionary.offsets.length - 1) throw new Error("bounded Parquet dictionary ID out of range");
+			const start = dictionary.offsets[id];
+			const end = dictionary.offsets[id + 1];
+			return plainCursor(dictionary.bytes.subarray(start, end), plan.element, 1, policy.maxValueBytes).next();
+		},
+		finish() {
+			definitions?.finish();
+			plain?.finish();
+			ids?.finish();
+		}
+	};
+}
+var ColumnCursor = class {
+	file;
+	plan;
+	policy;
+	decompress;
+	dictionary;
+	current;
+	pageIndex = 0;
+	remaining = 0;
+	constructor(file, plan, policy, decompress) {
+		this.file = file;
+		this.plan = plan;
+		this.policy = policy;
+		this.decompress = decompress;
+	}
+	needsPage() {
+		return this.remaining === 0;
+	}
+	async load() {
+		if (this.remaining) return;
+		this.current?.finish();
+		this.current = void 0;
+		if (!this.dictionary && this.plan.dictionary) {
+			const page = this.plan.dictionary;
+			const bytes = inflate(new Uint8Array(await this.file.slice(page.offset, page.offset + page.compressed)), page.uncompressed, this.plan.meta.codec === "SNAPPY", this.decompress);
+			const offsets = new Uint32Array(page.count + 1);
+			const cursor = plainCursor(bytes, this.plan.element, page.count, this.policy.maxValueBytes);
+			for (let index = 0; index < page.count; index++) {
+				offsets[index] = cursor.position();
+				cursor.skip();
+			}
+			offsets[page.count] = cursor.position();
+			cursor.finish();
+			this.dictionary = {
+				bytes,
+				offsets
+			};
+		}
+		const page = this.plan.pages[this.pageIndex++];
+		if (!page) throw new Error("bounded Parquet exhausted column");
+		const input = new Uint8Array(await this.file.slice(page.offset, page.offset + page.compressed));
+		this.current = dataCursor(page, input, this.plan, this.policy, this.dictionary, this.decompress);
+		this.remaining = page.count;
+	}
+	next() {
+		if (!this.current || this.remaining-- < 1) throw new Error("bounded Parquet column is not ready");
+		return this.current.next();
+	}
+	close() {
+		this.current = void 0;
+		this.dictionary = void 0;
+	}
+	finish() {
+		this.current?.finish();
+		if (this.remaining || this.pageIndex !== this.plan.pages.length) throw new Error("bounded Parquet incomplete column");
+	}
+};
+async function* decodeBoundedValues(file, metadata, policy, maxReadBytes, options, maxReadAndDecompressBytes) {
+	const plans = await planInput(file, metadata, policy, maxReadBytes, options.columns, options.rowStart, options.rowEnd, maxReadAndDecompressBytes);
+	try {
+		for (const plan of plans) {
+			const budget = plan.budget;
+			const cursors = plan.columns.map((column) => new ColumnCursor(budget?.file ?? file, column, policy, budget?.compressors.SNAPPY));
+			let batch = [];
+			let batchBytes = 0;
+			try {
+				for (let index = 0; index < plan.rows; index++) {
+					for (const cursor of cursors) if (cursor.needsPage()) await cursor.load();
+					const tokens = cursors.map((cursor) => cursor.next());
+					const position = plan.start + index;
+					if (position < (options.rowStart ?? 0) || position >= (options.rowEnd ?? Infinity)) continue;
+					const bytes = 128 + tokens.reduce((sum, token) => sum + 16 + token.bytes, 0);
+					if (bytes > policy.maxBatchBytes) throw new RangeError("bounded Parquet row exceeds maxBatchBytes");
+					if (batch.length && (batch.length >= policy.targetRows || batchBytes + bytes > policy.maxBatchBytes)) {
+						yield batch;
+						batch = [];
+						batchBytes = 0;
+					}
+					const row = {};
+					for (let col = 0; col < tokens.length; col++) Object.defineProperty(row, plan.columns[col].element.name, {
+						value: tokens[col].value(),
+						enumerable: true,
+						configurable: true,
+						writable: true
+					});
+					batch.push(row);
+					batchBytes += bytes;
+				}
+				for (const cursor of cursors) cursor.finish();
+				if (batch.length) yield batch;
+			} finally {
+				for (const cursor of cursors) cursor.close();
+				budget?.close();
+			}
+		}
+	} finally {
+		for (const plan of plans) plan.budget?.close();
+	}
+}
+export { boundedValuesPolicy, decodeBoundedValues };

package/dist/adapters/parquet-decode-budget.mjs

@@ -0,0 +1,61 @@
+import { snappyUncompress } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/snappy.mjs";
+import "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs";
+function parquetDecodeBudget(source, maxBytes) {
+	let remaining = maxBytes;
+	let closed = false;
+	let failure;
+	function fail(error) {
+		if (!closed) {
+			closed = true;
+			failure = error;
+		}
+		throw failure;
+	}
+	function assertOpen() {
+		if (closed) throw failure;
+	}
+	function charge(bytes) {
+		assertOpen();
+		if (!Number.isSafeInteger(bytes) || bytes < 0 || bytes > remaining) fail(/* @__PURE__ */ new RangeError("parquet stream exceeds maxReadAndDecompressBytes"));
+		remaining -= bytes;
+	}
+	return {
+		file: {
+			byteLength: source.byteLength,
+			async slice(start, end = source.byteLength) {
+				try {
+					if (!Number.isSafeInteger(start) || !Number.isSafeInteger(end) || start < 0 || end < start || end > source.byteLength) fail(/* @__PURE__ */ new RangeError("parquet stream invalid decode-buffer range"));
+					charge(end - start);
+					const bytes = await source.slice(start, end);
+					assertOpen();
+					if (bytes.byteLength !== end - start) fail(/* @__PURE__ */ new Error("parquet stream returned an incomplete decode-buffer range"));
+					return bytes;
+				} catch (error) {
+					fail(error);
+				}
+			}
+		},
+		compressors: {
+			SNAPPY(input, size) {
+				charge(size);
+				try {
+					const output = new Uint8Array(size);
+					snappyUncompress(input, output);
+					return output;
+				} catch (error) {
+					fail(error);
+				}
+			},
+			ZSTD() {
+				fail(/* @__PURE__ */ new Error("parquet stream maxReadAndDecompressBytes does not support ZSTD"));
+			}
+		},
+		close() {
+			if (!closed) {
+				closed = true;
+				failure = /* @__PURE__ */ new Error("parquet stream decode-buffer budget is closed");
+			}
+		}
+	};
+}
+export { parquetDecodeBudget };

package/dist/adapters/parquet-page-header.mjs

@@ -0,0 +1,136 @@
+var BoundedHeaderTruncatedError = class extends Error {
+	constructor() {
+		super("bounded Parquet truncated page header");
+	}
+};
+function boundedPageHeader(bytes) {
+	const view = new DataView(bytes);
+	let offset = 0;
+	let fields = 0;
+	const take = (length) => {
+		if (!Number.isSafeInteger(length) || length < 0) throw new Error("bounded Parquet invalid page-header size");
+		if (offset + length > view.byteLength) throw new BoundedHeaderTruncatedError();
+		const start = offset;
+		offset += length;
+		return start;
+	};
+	const byte = () => view.getUint8(take(1));
+	const varint = () => {
+		let value = 0;
+		for (let shift = 0; shift < 49; shift += 7) {
+			const b = byte();
+			value += (b & 127) * 2 ** shift;
+			if (!(b & 128) && Number.isSafeInteger(value)) return value;
+		}
+		throw new Error("bounded Parquet invalid header integer");
+	};
+	const integer = () => {
+		const value = varint();
+		return value % 2 ? -(value + 1) / 2 : value / 2;
+	};
+	const skip = (type, depth) => {
+		if (depth > 8 || ++fields > 512) throw new Error("bounded Parquet excessive page-header structure");
+		if (type === 1 || type === 2) return;
+		if (type === 3) {
+			take(1);
+			return;
+		}
+		if (type === 4 || type === 5) {
+			varint();
+			return;
+		}
+		if (type === 6) {
+			for (let i = 0; i < 10; i++) if (!(byte() & 128)) return;
+			throw new Error("bounded Parquet invalid header int64");
+		}
+		if (type === 7) {
+			take(8);
+			return;
+		}
+		if (type === 8) {
+			take(varint());
+			return;
+		}
+		if (type === 12) {
+			struct(depth + 1);
+			return;
+		}
+		throw new Error("bounded Parquet unsupported page-header field");
+	};
+	function struct(depth, select) {
+		let id = 0;
+		const seen = /* @__PURE__ */ new Set();
+		while (true) {
+			const field = byte();
+			if (field === 0) return;
+			const type = field & 15;
+			id = field >> 4 ? id + (field >> 4) : integer();
+			if (id < 1 || seen.has(id)) throw new Error("bounded Parquet invalid page-header field");
+			seen.add(id);
+			if (!select?.(id, type)) skip(type, depth);
+			if (++fields > 512) throw new Error("bounded Parquet excessive page-header fields");
+		}
+	}
+	const root = {};
+	const children = {};
+	struct(0, (id, type) => {
+		if ([
+			1,
+			2,
+			3
+		].includes(id)) {
+			if (type !== 5) throw new Error("bounded Parquet invalid page-header number");
+			root[id] = integer();
+			return true;
+		}
+		if ([
+			5,
+			7,
+			8
+		].includes(id)) {
+			if (type !== 12) throw new Error("bounded Parquet invalid page-header body");
+			const body = {};
+			children[id] = body;
+			struct(1, (field, fieldType) => {
+				if (field > (id === 5 ? 4 : id === 7 ? 2 : 7)) return false;
+				if (id === 8 && field === 7) {
+					if (fieldType !== 1 && fieldType !== 2) throw new Error("bounded Parquet invalid compression boolean");
+					body[field] = fieldType === 1;
+				} else {
+					if (fieldType !== 5) throw new Error("bounded Parquet invalid data-header number");
+					body[field] = integer();
+				}
+				return true;
+			});
+			return true;
+		}
+		return false;
+	});
+	const kind = root[1];
+	const body = children[kind === 0 ? 5 : kind === 2 ? 7 : 8];
+	if (![
+		0,
+		2,
+		3
+	].includes(kind) || !body) throw new Error("bounded Parquet unsupported page type");
+	return {
+		bytes: offset,
+		kind,
+		compressed: root[3],
+		uncompressed: root[2],
+		count: body[1],
+		encoding: body[kind === 3 ? 4 : 2],
+		...kind === 0 ? {
+			definitionEncoding: body[3],
+			repetitionEncoding: body[4]
+		} : {},
+		...kind === 3 ? {
+			nulls: body[2],
+			rows: body[3],
+			definitions: body[5],
+			repetitions: body[6],
+			isCompressed: body[7] === void 0 ? true : body[7]
+		} : {}
+	};
+}
+export { BoundedHeaderTruncatedError, boundedPageHeader };

package/dist/adapters/parquet-range-cache.mjs

@@ -0,0 +1,41 @@
+function parquetRangeCache(source, maxBytes) {
+	const entries = /* @__PURE__ */ new Map();
+	let retainedBytes = 0;
+	let closed = false;
+	return {
+		file: {
+			byteLength: source.byteLength,
+			async slice(start, end = source.byteLength) {
+				const key = `${start}:${end}`;
+				const cached = entries.get(key);
+				if (cached) {
+					entries.delete(key);
+					entries.set(key, cached);
+					return cached;
+				}
+				const bytes = await source.slice(start, end);
+				if (!closed && bytes.byteLength > 0 && bytes.byteLength <= maxBytes) {
+					const previous = entries.get(key);
+					if (previous) {
+						retainedBytes -= previous.byteLength;
+						entries.delete(key);
+					}
+					while (entries.size >= 128 || retainedBytes + bytes.byteLength > maxBytes) {
+						const oldest = entries.keys().next().value;
+						retainedBytes -= entries.get(oldest).byteLength;
+						entries.delete(oldest);
+					}
+					entries.set(key, bytes);
+					retainedBytes += bytes.byteLength;
+				}
+				return bytes;
+			}
+		},
+		close() {
+			closed = true;
+			entries.clear();
+			retainedBytes = 0;
+		}
+	};
+}
+export { parquetRangeCache };

package/dist/adapters/parquet-value-cursor.mjs

@@ -0,0 +1,108 @@
+import { DEFAULT_PARSERS, convert } from "../node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/convert.mjs";
+const utf8 = new TextDecoder();
+function hybridCursor(bytes, width, count) {
+	if (!Number.isInteger(width) || width < 0 || width > 32) throw new Error("bounded Parquet invalid bit width");
+	let offset = 0;
+	let remaining = count;
+	let run = 0;
+	let repeat = 0;
+	let packed = false;
+	let bit = 0;
+	let packedStart = 0;
+	const readByte = () => {
+		if (offset >= bytes.length) throw new Error("bounded Parquet truncated RLE stream");
+		return bytes[offset++];
+	};
+	const next = () => {
+		if (remaining-- <= 0) throw new Error("bounded Parquet too many encoded values");
+		if (width === 0 && bytes.length === 0) return 0;
+		if (!run) {
+			let header = 0;
+			let shift = 0;
+			while (true) {
+				const b = readByte();
+				header += (b & 127) * 2 ** shift;
+				if (!(b & 128)) break;
+				shift += 7;
+				if (shift > 28) throw new Error("bounded Parquet invalid RLE run");
+			}
+			packed = !!(header & 1);
+			run = packed ? Math.floor(header / 2) * 8 : header / 2;
+			if (!Number.isSafeInteger(run) || run < 1 || run > 2147483647 || !packed && run > remaining + 1) throw new Error(`bounded Parquet invalid RLE run length: run=${run}, remaining=${remaining + 1}, packed=${packed}, width=${width}`);
+			if (packed) {
+				packedStart = offset;
+				bit = 0;
+				offset += run * width / 8;
+				if (offset > bytes.length) throw new Error("bounded Parquet truncated bit-packed stream");
+			} else {
+				repeat = 0;
+				for (let b = 0; b < Math.ceil(width / 8); b++) repeat += readByte() * 2 ** (8 * b);
+				if (repeat >= 2 ** width) throw new Error("bounded Parquet RLE value exceeds bit width");
+			}
+		}
+		run--;
+		if (!packed) return repeat;
+		let value = 0;
+		for (let b = 0; b < width; b++, bit++) value += (bytes[packedStart + Math.floor(bit / 8)] >> bit % 8 & 1) * 2 ** b;
+		return value;
+	};
+	return {
+		next,
+		finish() {
+			if (remaining !== 0 || offset !== bytes.length || run && !packed) throw new Error(`bounded Parquet trailing or incomplete RLE data: remaining=${remaining}, offset=${offset}, bytes=${bytes.length}, run=${run}, packed=${packed}, width=${width}`);
+		}
+	};
+}
+function plainCursor(bytes, element, count, maxValueBytes) {
+	const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
+	let offset = 0;
+	let index = 0;
+	const width = element.type === "BOOLEAN" ? 0 : element.type === "INT32" || element.type === "FLOAT" ? 4 : element.type === "INT96" ? 12 : 8;
+	const decoder = {
+		element,
+		type: element.type,
+		parsers: DEFAULT_PARSERS,
+		schemaPath: []
+	};
+	const take = () => {
+		if (index >= count) throw new Error("bounded Parquet too many plain values");
+		let size = width;
+		if (element.type === "BYTE_ARRAY") {
+			if (offset + 4 > bytes.length) throw new Error("bounded Parquet truncated string length");
+			size = view.getUint32(offset, true);
+			offset += 4;
+			if (size > maxValueBytes) throw new RangeError("bounded Parquet exceeds maxValueBytes");
+		}
+		const start = element.type === "BOOLEAN" ? Math.floor(index / 8) : offset;
+		if (start + (element.type === "BOOLEAN" ? 1 : size) > bytes.length) throw new Error("bounded Parquet truncated plain value");
+		offset = element.type === "BOOLEAN" ? Math.ceil((index + 1) / 8) : offset + size;
+		return {
+			start,
+			size,
+			index: index++
+		};
+	};
+	return {
+		next() {
+			const { start, size, index: bitIndex } = take();
+			return {
+				bytes: 64 + (element.type === "BYTE_ARRAY" ? size * 2 : 32),
+				value() {
+					if (element.type === "BYTE_ARRAY") return utf8.decode(bytes.subarray(start, start + size));
+					let raw = element.type === "BOOLEAN" ? !!(view.getUint8(start) & 1 << bitIndex % 8) : element.type === "INT32" ? view.getInt32(start, true) : element.type === "INT64" ? view.getBigInt64(start, true) : element.type === "INT96" ? BigInt(view.getInt32(start + 8, true)) << 64n | view.getBigInt64(start, true) : element.type === "FLOAT" ? view.getFloat32(start, true) : view.getFloat64(start, true);
+					if (element.type === "INT64" && (element.converted_type === "UINT_64" || element.logical_type?.type === "INTEGER" && !element.logical_type.isSigned)) raw = BigInt.asUintN(64, raw);
+					const result = convert([raw], decoder)[0];
+					return result instanceof Date ? `${result.getUTCFullYear()}-${String(result.getUTCMonth() + 1).padStart(2, "0")}-${String(result.getUTCDate()).padStart(2, "0")}` : result;
+				}
+			};
+		},
+		skip() {
+			take();
+		},
+		position: () => offset,
+		finish() {
+			if (index !== count || offset !== bytes.length) throw new Error("bounded Parquet trailing or incomplete plain data");
+		}
+	};
+}
+export { hybridCursor, plainCursor };

package/dist/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/index.mjs

@@ -1,8 +1,8 @@
 import { parquetMetadata, parquetMetadataAsync, parquetSchema } from "./metadata.mjs";
-import "./indexes.mjs";
+import { readOffsetIndex } from "./indexes.mjs";
 import { toJson } from "./utils.mjs";
-import "./snappy.mjs";
+import { snappyUncompress } from "./snappy.mjs";
 import "./scan.mjs";
 import { parquetReadObjects } from "./read.mjs";
 import "./query.mjs";
-export { parquetMetadata, parquetMetadataAsync, parquetReadObjects, parquetSchema, toJson };
+export { parquetMetadata, parquetMetadataAsync, parquetReadObjects, parquetSchema, readOffsetIndex, snappyUncompress, toJson };

package/dist/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs

@@ -1,7 +1,7 @@
 import { parquetMetadata, parquetMetadataAsync, parquetSchema } from "./metadata.mjs";
-import "./indexes.mjs";
+import { readOffsetIndex } from "./indexes.mjs";
 import { toJson } from "./utils.mjs";
-import "./snappy.mjs";
+import { snappyUncompress } from "./snappy.mjs";
 import { parquetReadObjects } from "./read.mjs";
 import "./index.mjs";
-export { parquetMetadata, parquetMetadataAsync, parquetReadObjects, parquetSchema, toJson };
+export { parquetMetadata, parquetMetadataAsync, parquetReadObjects, parquetSchema, readOffsetIndex, snappyUncompress, toJson };

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@gscdump/engine",
   "type": "module",
-  "version": "7.4.0",
+  "version": "7.6.0",
   "description": "Append-only Parquet/DuckDB storage engine + planner + adapters for the gscdump pipeline. Node + edge runtimes; opt-in heavy peers.",
   "author": {
     "name": "Harlan Wilton",
@@ -189,10 +189,10 @@
     }
   },
   "dependencies": {
-    "@gscdump/contracts": "^7.4.0",
-    "@gscdump/lakehouse": "^7.4.0",
+    "@gscdump/contracts": "^7.6.0",
+    "@gscdump/lakehouse": "^7.6.0",
     "drizzle-orm": "1.0.0-rc.4",
-    "gscdump": "^7.4.0",
+    "gscdump": "^7.6.0",
     "pathe": "^2.0.3",
     "proper-lockfile": "^4.1.2"
   },