@gscdump/lakehouse 7.6.0 → 7.8.0

package/README.md

@@ -45,6 +45,109 @@ For retry decisions, import `isCommitRateLimited`, `isCommitServerError`, or
 Use the dataset registry to define datasets.
 Raw Icebird table creation and append functions live under `unsafe-raw` for package adapters and diagnostics.
 
+## Opt-in manifest read limits
+
+`icebergManifests` on `@gscdump/lakehouse/unsafe-raw` accepts an optional
+`readLimits: ManifestReadLimits`. Omission preserves the existing reader and
+caller-owned manifest cache behavior. No file resolver or host automatically
+enables these limits.
+
+```ts
+import { icebergManifests } from '@gscdump/lakehouse/unsafe-raw'
+
+const manifests = await icebergManifests({
+  metadata,
+  resolver,
+  readLimits: {
+    maxObjectBytes: 2 * 1024 * 1024,
+    maxTotalObjectBytes: 8 * 1024 * 1024,
+    maxManifests: 8,
+    maxEntries: 128,
+    maxInflatedBlockBytes: 2 * 1024 * 1024,
+    maxTotalInflatedBytes: 8 * 1024 * 1024,
+    maxValueBytes: 1024 * 1024,
+    maxDecodedValues: 100_000,
+    maxHeaderBytes: 64 * 1024,
+    maxSchemaDepth: 32,
+  },
+})
+```
+
+These are example admission values, not defaults or a total JavaScript heap
+guarantee. Every field is required and must be a positive safe integer;
+`maxSchemaDepth` must also be at most 128. The reader copies and validates the
+policy before its first asynchronous read. Unknown fields are rejected.
+
+The per-object and cumulative source-byte checks happen after `resolver.reader`
+reports `byteLength`, **before** the reader requests `slice(0, byteLength)`.
+Declared manifest lengths and returned slice lengths must match. An injected
+resolver can allocate or fetch inside `reader` itself; those operations are
+outside this boundary and need their own transport limits and immutable-object
+identity checks. No HEAD, catalog JSON, network-request, or freshness bound is
+implied by this option.
+
+Manifest-list counts are admitted before record materialization and pruning;
+entry counts accumulate across all fetched manifests and blocks. V1 inline
+manifest counts are admitted before reads, with each file decoded once in a
+private per-call cache. Supplying `manifestCache` with `readLimits` is rejected:
+previously decoded cache entries carry no verifiable budget provenance. Defaults
+retain their existing cache behavior.
+
+For Avro `null` and raw `deflate` blocks, decoded block bytes and their cumulative
+total are bounded. The existing pure-JavaScript inflater checks output growth
+before allocating or writing past that limit. Bounded DEFLATE returns an exact-size
+backing buffer so byte views do not retain spare inflater capacity; growth and
+final-copy buffers can temporarily overlap. Header framing is byte-bounded
+before JSON parsing; schema depth/work is checked before recursive compilation.
+Variable and fixed values are checked before views or UTF-8 decoding, and
+`maxDecodedValues` bounds decoding work across header/schema nodes and decoded
+value visits, including arrays of zero-byte nulls. Strict reads also reject
+invalid counts, varints, indexes, truncated values and inconsistent block sizes.
+
+These limits do not measure JavaScript object/string overhead, the JSON parser's
+internal allocations, inflater tables, caller-retained output, injected resolver
+caches or total process memory. Output remains an eager array within the chosen
+count/byte/work policy. A refusal rejects the complete call; there is no eager
+fallback or partial success. Already-dispatched resolver reads cannot be
+physically cancelled by the AsyncBuffer interface, but a first failure stops new
+dispatches and prevents late responses from being decoded. Choose compatible
+limits and validate the complete consumer path before enabling them.
+
+The same optional `readLimits` can be passed to `resolveIcebergDataFiles`,
+`dataset.resolveDataFiles(..., { readLimits })`, and the engine's
+`listIcebergDataFiles`. These wrappers capture and validate all ten values before
+catalog reads or lazy imports, then perform one bounded manifest walk. Per-object
+and aggregate limits remain independent upper bounds; a smaller aggregate limit
+does not expand the per-object allowance. Invalid, missing or unknown values are
+rejected even when the table has no current snapshot.
+
+In this opt-in mode, decoded manifest, resolved-file and month caches are neither
+read nor populated: old cache entries cannot establish compliance with a new
+policy. The existing catalog configuration, snapshot pointer and table metadata
+caches remain in use, with their existing identity/freshness rules. Catalog JSON
+and those metadata caches are outside the Avro allocation bound. Existing
+partition, Site, search-type, date pruning and output ordering are unchanged.
+
+This has an explicit warm-read cost. A default exact-range cache hit can avoid
+all Avro reads; a default month hit can need only the manifest list. A bounded
+call reads the list and every selected manifest again (subject to any injected
+resolver's own byte cache).
+
+A connection from `connectIcebergCatalog` keeps the objects it reads in a
+per-connection byte cache. That cache holds at most 16 MiB of fetched bytes.
+If a read takes it over 16 MiB, it evicts the least recently read object
+first. It does not keep an object larger than 16 MiB, and it does not keep a
+failed read. Concurrent reads of one path share one S3 request. A successful
+write or delete of a path evicts that path. A bounded call reads through the uncached
+S3 resolver instead, so the connection keeps none of its manifest bytes after
+the call. It also does not reuse bytes that earlier unbounded calls cached. If
+you replace `connection.resolver`, the bounded call uses your resolver, and your
+resolver owns its retention. Each concurrent bounded call owns its own limits;
+the option does not impose a connection-wide or process-wide budget. There is
+no automatic fallback to cached or unbounded results after refusal. Validate
+retained files and request costs before enabling a policy in a host; this does
+not bound a remote SQL provider's internal manifest planner.
+
 ## Append retries
 
 Supply a durable `commitRetry.appendId` when a dataset append belongs to a job or receipt.

package/dist/_virtual/node_modules/.pnpm/hyparquet-compressors@1.1.2_patch_hash=3f8f44805c034b2563763dcd5d37050ea6b1d5d45b46ea37adc4a721e748a1c5/node_modules/hyparquet-compressors/src/gzip.mjs

@@ -0,0 +1,196 @@
+import { fixedDistance, fixedDistanceExtraBits, fixedDistanceMap, fixedLength, fixedLengthExtraBits, fixedLengthMap, huffMap } from "./gzip.huffman.mjs";
+const codeLengthIndexMap = new Uint8Array([
+	16,
+	17,
+	18,
+	0,
+	8,
+	7,
+	9,
+	6,
+	10,
+	5,
+	11,
+	4,
+	12,
+	3,
+	13,
+	2,
+	14,
+	1,
+	15
+]);
+function bits(input, pos, mask) {
+	const o = pos / 8 | 0;
+	return (input[o] | input[o + 1] << 8) >> (pos & 7) & mask;
+}
+function bits16(d, p) {
+	const o = p / 8 | 0;
+	return (d[o] | d[o + 1] << 8 | d[o + 2] << 16) >> (p & 7);
+}
+function shft(p) {
+	return (p + 7) / 8 | 0;
+}
+function gzipStart(input, i) {
+	if (input[i++] !== 31 || input[i++] !== 139 || input[i++] !== 8) return 0;
+	const flag = input[i++];
+	if (flag & 224) throw new Error("invalid gzip header flags");
+	i += 6;
+	if (flag & 4) i += (input[i] | input[i + 1] << 8) + 2;
+	for (let zs = (flag >> 3 & 1) + (flag >> 4 & 1); zs > 0; zs -= Number(!input[i++]));
+	return i + (flag & 2);
+}
+function gunzip(input, output, inputIndex = 0, outputIndex = 0, limits) {
+	const maximum = limits?.maxOutputBytes;
+	if (limits && (limits.raw !== true || !Number.isSafeInteger(maximum) || maximum < 0 || output || inputIndex !== 0 || outputIndex !== 0)) throw new RangeError("Invalid bounded raw DEFLATE options");
+	let out = output ?? new Uint8Array(limits ? Math.min(1024, maximum) : 1024);
+	if (limits && !input.length) throw new Error("unexpected EOF");
+	if (!(input.length - inputIndex)) return out;
+	const totalBits = input.length * 8;
+	function ensureSize(length) {
+		if (limits && (!Number.isSafeInteger(length) || length > maximum)) throw new RangeError("DEFLATE output byte limit exceeded");
+		if (!output && length > out.length) {
+			const old = out;
+			out = new Uint8Array(limits ? Math.min(maximum, Math.max(old.length * 2, length)) : Math.max(old.length * 2, length));
+			out.set(old);
+		}
+	}
+	let memberStart = inputIndex;
+	for (;;) {
+		const payloadStart = limits ? 0 : gzipStart(input, memberStart);
+		const isGzip = !limits && payloadStart > 0;
+		if (isGzip && payloadStart === input.length - 8) break;
+		if (isGzip && payloadStart > input.length - 8) throw new Error("unexpected EOF");
+		const pos = inflate(payloadStart * 8);
+		if (limits && Math.ceil(pos / 8) !== input.length) throw new Error("trailing raw DEFLATE bytes");
+		if (!isGzip || output && outputIndex >= out.length) break;
+		const nextBlock = Math.ceil(pos / 8) + 8;
+		if (nextBlock > input.length) throw new Error("unexpected EOF");
+		if (nextBlock === input.length) break;
+		if (input[nextBlock] !== 31 || input[nextBlock + 1] !== 139 || input[nextBlock + 2] !== 8) throw new Error("invalid gzip member");
+		memberStart = nextBlock;
+	}
+	function inflate(pos) {
+		let final = 0;
+		let lengthBits = 0;
+		let distBits = 0;
+		let lengthMap;
+		let distMap;
+		do {
+			if (!lengthMap) {
+				final = bits(input, pos, 1);
+				const type = bits(input, pos + 1, 3);
+				pos += 3;
+				if (!type) {
+					const s = shft(pos) + 4;
+					const l = input[s - 4] | input[s - 3] << 8;
+					if (limits && (s > input.length || ((input[s - 2] | input[s - 1] << 8) ^ l) !== 65535)) throw new Error("invalid stored block length");
+					const t = s + l;
+					if (t > input.length) throw new Error("unexpected EOF");
+					ensureSize(outputIndex + l);
+					out.set(input.subarray(s, t), outputIndex);
+					outputIndex += l;
+					pos = t * 8;
+					continue;
+				} else if (type === 1) {
+					lengthMap = fixedLengthMap;
+					distMap = fixedDistanceMap;
+					lengthBits = 9;
+					distBits = 5;
+				} else if (type === 2) {
+					const hLiteral = bits(input, pos, 31) + 257;
+					const hcLengths = bits(input, pos + 10, 15) + 4;
+					const tl = hLiteral + bits(input, pos + 5, 31) + 1;
+					pos += 14;
+					const lengthDistanceTree = new Uint8Array(tl);
+					const codeLengthTree = /* @__PURE__ */ new Uint8Array(19);
+					for (let i = 0; i < hcLengths; ++i) codeLengthTree[codeLengthIndexMap[i]] = bits(input, pos + i * 3, 7);
+					pos += hcLengths * 3;
+					const codeLengthBits = Math.max(...codeLengthTree);
+					const clbMask = (1 << codeLengthBits) - 1;
+					const codeLengthMap = huffMap(codeLengthTree, codeLengthBits, 1);
+					for (let i = 0; i < tl;) {
+						const r = codeLengthMap[bits(input, pos, clbMask)];
+						if (limits && !r) throw new Error("invalid code length");
+						pos += r & 15;
+						const symbol = r >> 4;
+						if (symbol < 16) lengthDistanceTree[i++] = symbol;
+						else {
+							let copy = 0;
+							let n = 0;
+							if (symbol === 16) {
+								if (limits && i === 0) throw new Error("invalid code length repeat");
+								n = 3 + bits(input, pos, 3);
+								pos += 2;
+								copy = lengthDistanceTree[i - 1];
+							} else if (symbol === 17) {
+								n = 3 + bits(input, pos, 7);
+								pos += 3;
+							} else if (symbol === 18) {
+								n = 11 + bits(input, pos, 127);
+								pos += 7;
+							}
+							if (limits && i + n > tl) throw new Error("invalid code length repeat");
+							while (n--) lengthDistanceTree[i++] = copy;
+						}
+					}
+					const lengthTree = lengthDistanceTree.subarray(0, hLiteral);
+					const distanceTree = lengthDistanceTree.subarray(hLiteral);
+					lengthBits = Math.max(...lengthTree);
+					distBits = Math.max(...distanceTree);
+					lengthMap = huffMap(lengthTree, lengthBits, 1);
+					distMap = huffMap(distanceTree, distBits, 1);
+				} else throw new Error("invalid block type");
+				if (pos > totalBits) throw new Error("unexpected EOF");
+			}
+			ensureSize(limits ? Math.min(maximum, outputIndex + 131072) : outputIndex + 131072);
+			const lms = (1 << lengthBits) - 1;
+			const dms = (1 << distBits) - 1;
+			for (;;) {
+				const code = lengthMap[bits16(input, pos) & lms];
+				const sym = code >> 4;
+				pos += code & 15;
+				if (pos > totalBits) throw new Error("unexpected EOF");
+				if (!code) throw new Error("invalid length/literal");
+				if (sym < 256) {
+					if (limits) ensureSize(outputIndex + 1);
+					out[outputIndex++] = sym;
+				} else if (sym === 256) {
+					lengthMap = void 0;
+					break;
+				} else {
+					let add = sym - 254;
+					if (sym > 264) {
+						const index = sym - 257;
+						const b = fixedLengthExtraBits[index];
+						add = bits(input, pos, (1 << b) - 1) + fixedLength[index];
+						pos += b;
+					}
+					if (!distMap) throw new Error("invalid distance map");
+					const d = distMap[bits16(input, pos) & dms];
+					const dsym = d >> 4;
+					if (!d) throw new Error("invalid distance");
+					if (limits && dsym > 29) throw new Error("invalid distance symbol");
+					pos += d & 15;
+					let dt = fixedDistance[dsym];
+					if (dsym > 3) {
+						const b = fixedDistanceExtraBits[dsym];
+						dt += bits16(input, pos) & (1 << b) - 1;
+						pos += b;
+					}
+					if (pos > totalBits) throw new Error("unexpected EOF");
+					const end = outputIndex + add;
+					if (outputIndex < dt) throw new Error("unexpected dictionary case");
+					ensureSize(end);
+					for (; outputIndex < end; outputIndex++) out[outputIndex] = out[outputIndex - dt];
+				}
+			}
+			if (lengthMap) final = 1;
+		} while (!final);
+		return pos;
+	}
+	if (limits && outputIndex !== out.length) return out.slice(0, outputIndex);
+	if (!output) return out.subarray(0, outputIndex);
+	return out;
+}
+export { gunzip };

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/column.mjs

@@ -1,5 +1,5 @@
-import { Encodings, PageTypes } from "./constants.mjs";
 import { convert, convertWithDictionary } from "./convert.mjs";
+import { Encodings, PageTypes } from "./constants.mjs";
 import { isFlatColumn } from "./schema.mjs";
 import { deserializeTCompactProtocol } from "./thrift.mjs";
 import { assembleLists } from "./assemble.mjs";

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/datapage.mjs

@@ -1,8 +1,8 @@
+import { snappyUncompress } from "./snappy.mjs";
 import { getMaxDefinitionLevel, getMaxRepetitionLevel } from "./schema.mjs";
 import { deltaBinaryUnpack, deltaByteArray, deltaLengthByteArray } from "./delta.mjs";
 import { byteStreamSplit, readRleBitPackedHybrid } from "./encoding.mjs";
 import { readPlain } from "./plain.mjs";
-import { snappyUncompress } from "./snappy.mjs";
 function readDataPage(bytes, daph, { type, element, schemaPath }) {
 	const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
 	const reader = {

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/index.mjs

@@ -1,7 +1,7 @@
+import "./snappy.mjs";
 import { parquetMetadataAsync, parquetSchema } from "./metadata.mjs";
 import "./indexes.mjs";
 import { asyncBufferFromUrl, cachedAsyncBuffer, toJson } from "./utils.mjs";
-import "./snappy.mjs";
 import { parquetScan } from "./scan.mjs";
 import { parquetReadObjects } from "./read.mjs";
 import "./query.mjs";

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/indexes.mjs

@@ -1,5 +1,5 @@
-import { BoundaryOrders } from "./constants.mjs";
 import { DEFAULT_PARSERS } from "./convert.mjs";
+import { BoundaryOrders } from "./constants.mjs";
 import { deserializeTCompactProtocol } from "./thrift.mjs";
 import { convertMetadata } from "./metadata.mjs";
 function readColumnIndex(reader, schema, parsers = void 0) {

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/metadata.mjs

@@ -1,5 +1,5 @@
-import { CompressionCodecs, ConvertedTypes, EdgeInterpolationAlgorithms, Encodings, FieldRepetitionTypes, PageTypes, ParquetTypes } from "./constants.mjs";
 import { DEFAULT_PARSERS, parseDecimal, parseFloat16 } from "./convert.mjs";
+import { CompressionCodecs, ConvertedTypes, EdgeInterpolationAlgorithms, Encodings, FieldRepetitionTypes, PageTypes, ParquetTypes } from "./constants.mjs";
 import { getSchemaPath } from "./schema.mjs";
 import { deserializeTCompactProtocol } from "./thrift.mjs";
 import { markGeoColumns } from "./geoparquet.mjs";

package/dist/_virtual/node_modules/.pnpm/hyparquet@1.31.1/node_modules/hyparquet/src/node.mjs

@@ -1,7 +1,7 @@
+import "./snappy.mjs";
 import { parquetMetadataAsync, parquetSchema } from "./metadata.mjs";
 import "./indexes.mjs";
 import { asyncBufferFromUrl, cachedAsyncBuffer, toJson } from "./utils.mjs";
-import "./snappy.mjs";
 import { parquetScan } from "./scan.mjs";
 import { parquetReadObjects } from "./read.mjs";
 import "./index.mjs";

package/dist/_virtual/node_modules/.pnpm/icebird@0.8.30_patch_hash=1359a91da68522a736babe444880ae391f19455fbe47c346f76cec92c5c2e471/node_modules/icebird/src/avro/avro.budget.mjs

@@ -0,0 +1,115 @@
+const fields = [
+	"maxObjectBytes",
+	"maxTotalObjectBytes",
+	"maxManifests",
+	"maxEntries",
+	"maxInflatedBlockBytes",
+	"maxTotalInflatedBytes",
+	"maxValueBytes",
+	"maxDecodedValues",
+	"maxHeaderBytes",
+	"maxSchemaDepth"
+];
+function createAvroBudget(options) {
+	if (options === void 0) return void 0;
+	if (!options || typeof options !== "object" || Array.isArray(options)) throw new TypeError("Invalid manifest read limits");
+	const limits = {};
+	for (const key of Object.keys(options)) if (!fields.includes(key)) throw new TypeError(`Unknown manifest read limit: ${key}`);
+	for (const key of fields) {
+		const value = options[key];
+		if (!Number.isSafeInteger(value) || value <= 0) throw new RangeError(`Invalid manifest read limit: ${key}`);
+		limits[key] = value;
+	}
+	if (limits.maxSchemaDepth > 128) throw new RangeError("Manifest schema depth must not exceed 128");
+	Object.freeze(limits);
+	const used = {
+		objectBytes: 0,
+		inflatedBytes: 0,
+		manifests: 0,
+		entries: 0,
+		values: 0
+	};
+	let failed = false, failure;
+	const check = () => {
+		if (failed) throw failure;
+	};
+	const fail = (error) => {
+		if (!failed) {
+			failed = true;
+			failure = error;
+		}
+		throw failure;
+	};
+	const bound = (value, max, label) => {
+		check();
+		if (!Number.isSafeInteger(value) || value < 0 || value > max) fail(/* @__PURE__ */ new RangeError(`Manifest read limit exceeded: ${label}`));
+	};
+	const charge = (key, value, max) => {
+		bound(value, max - used[key], key);
+		used[key] += value;
+	};
+	return Object.freeze({
+		limits,
+		check,
+		fail,
+		bound,
+		object(bytes) {
+			bound(bytes, limits.maxObjectBytes, "objectBytes");
+			charge("objectBytes", bytes, limits.maxTotalObjectBytes);
+		},
+		records(count, kind) {
+			charge(kind, count, kind === "manifests" ? limits.maxManifests : limits.maxEntries);
+		},
+		values(count) {
+			charge("values", count, limits.maxDecodedValues);
+		},
+		availableValues(count) {
+			bound(count, limits.maxDecodedValues - used.values, "values");
+		},
+		inflateLimit() {
+			check();
+			return Math.min(limits.maxInflatedBlockBytes, limits.maxTotalInflatedBytes - used.inflatedBytes);
+		},
+		inflated(bytes) {
+			bound(bytes, limits.maxInflatedBlockBytes, "inflatedBlockBytes");
+			charge("inflatedBytes", bytes, limits.maxTotalInflatedBytes);
+		},
+		schema(value) {
+			const stack = [{
+				value,
+				depth: 0
+			}];
+			while (stack.length) {
+				const next = stack.pop();
+				bound(next.depth, limits.maxSchemaDepth, "schemaDepth");
+				charge("values", 1, limits.maxDecodedValues);
+				if (next.value && typeof next.value === "object") {
+					const keys = Object.keys(next.value);
+					bound(stack.length + keys.length, limits.maxDecodedValues - used.values, "schemaValues");
+					for (const key of keys) stack.push({
+						value: next.value[key],
+						depth: next.depth + 1
+					});
+				}
+			}
+		}
+	});
+}
+function boundedBytes(reader, length, max) {
+	const budget = reader.budget;
+	budget?.bound(length, max ?? budget.limits.maxValueBytes, "valueBytes");
+	budget?.bound(reader.offset + length, reader.headerEnd ?? reader.view.byteLength, "encodedBytes");
+	return new Uint8Array(reader.view.buffer, reader.view.byteOffset + reader.offset, length);
+}
+function boundedLong(reader) {
+	let value = 0n;
+	for (let shift = 0; shift < 70; shift += 7) {
+		reader.budget.bound(reader.offset + 1, reader.headerEnd ?? reader.view.byteLength, "varintBytes");
+		const byte = reader.view.getUint8(reader.offset++);
+		if (shift === 63 && byte > 1) reader.budget.fail(/* @__PURE__ */ new RangeError("Invalid Avro long"));
+		value |= BigInt(byte & 127) << BigInt(shift);
+		if (!(byte & 128)) return value >> 1n ^ -(value & 1n);
+	}
+	return reader.budget.fail(/* @__PURE__ */ new RangeError("Invalid Avro long"));
+}
+export { boundedBytes, boundedLong, createAvroBudget };

package/dist/_virtual/node_modules/.pnpm/icebird@0.8.30_patch_hash=1359a91da68522a736babe444880ae391f19455fbe47c346f76cec92c5c2e471/node_modules/icebird/src/avro/avro.metadata.mjs

@@ -0,0 +1,97 @@
+import { boundedBytes, boundedLong } from "./avro.budget.mjs";
+const smallLongs = Array.from({ length: 128 }, (_, i) => BigInt(i >>> 1 ^ -(i & 1)));
+function readZigZag(reader) {
+	if (reader.budget) {
+		const value = Number(boundedLong(reader));
+		if (!Number.isSafeInteger(value)) reader.budget.fail(/* @__PURE__ */ new RangeError("Avro integer exceeds safe range"));
+		return value;
+	}
+	let result = 0;
+	let shift = 0;
+	while (true) {
+		const byte = reader.view.getUint8(reader.offset++);
+		result |= (byte & 127) << shift;
+		if (!(byte & 128)) return result >>> 1 ^ -(result & 1);
+		shift += 7;
+	}
+}
+function readZigZagBigInt(reader) {
+	if (reader.budget) return boundedLong(reader);
+	const first = reader.view.getUint8(reader.offset++);
+	if (first < 128) return smallLongs[first];
+	let prefix = first & 127;
+	for (let shift = 7; shift < 28; shift += 7) {
+		const byte = reader.view.getUint8(reader.offset++);
+		prefix |= (byte & 127) << shift;
+		if (!(byte & 128)) return BigInt(prefix >>> 1 ^ -(prefix & 1));
+	}
+	let high = 0;
+	let factor = 1;
+	while (true) {
+		const byte = reader.view.getUint8(reader.offset++);
+		high += (byte & 127) * factor;
+		if (!(byte & 128)) {
+			const result = BigInt(high) << 27n | BigInt(prefix >>> 1);
+			return prefix & 1 ? ~result : result;
+		}
+		factor *= 128;
+	}
+}
+function readBlockEnd(reader) {
+	const size = readZigZag(reader);
+	if (!reader.budget) return void 0;
+	reader.budget.bound(size, reader.view.byteLength, "containerBlockBytes");
+	const end = reader.offset + size;
+	reader.budget.bound(end, reader.headerEnd ?? reader.view.byteLength, "containerBlockEnd");
+	return end;
+}
+function checkBlockEnd(reader, end) {
+	if (end !== void 0 && reader.offset !== end) reader.budget.fail(/* @__PURE__ */ new RangeError("Avro container block length mismatch"));
+}
+function readAvroString(reader) {
+	const length = readZigZag(reader);
+	const bytes = boundedBytes(reader, length);
+	reader.budget?.values(1);
+	reader.offset += length;
+	return new TextDecoder().decode(bytes);
+}
+function avroMetadata(reader) {
+	if (reader.budget) reader.headerEnd = Math.min(reader.view.byteLength, reader.offset + reader.budget.limits.maxHeaderBytes);
+	reader.budget?.bound(reader.offset + 4, reader.headerEnd, "headerBytes");
+	if (reader.view.getUint32(reader.offset) !== 1331849729) throw new Error("avro invalid magic bytes");
+	reader.offset += 4;
+	const metadata = {};
+	let mapCount = readZigZag(reader);
+	while (mapCount !== 0) {
+		let end;
+		if (mapCount < 0) {
+			mapCount = -mapCount;
+			end = readBlockEnd(reader);
+		}
+		reader.budget?.availableValues(mapCount * 2);
+		for (let i = 0; i < mapCount; i++) {
+			const key = readAvroString(reader);
+			metadata[key] = readAvroString(reader);
+		}
+		checkBlockEnd(reader, end);
+		mapCount = readZigZag(reader);
+	}
+	metadata["avro.schema"] = JSON.parse(metadata["avro.schema"]);
+	reader.budget?.schema(metadata["avro.schema"]);
+	if (metadata["schema"]) {
+		metadata["schema"] = JSON.parse(metadata["schema"]);
+		reader.budget?.schema(metadata["schema"]);
+	}
+	if (metadata["iceberg.schema"]) {
+		metadata["iceberg.schema"] = JSON.parse(metadata["iceberg.schema"]);
+		reader.budget?.schema(metadata["iceberg.schema"]);
+	}
+	const syncMarker = boundedBytes(reader, 16, reader.budget?.limits.maxHeaderBytes);
+	reader.offset += 16;
+	delete reader.headerEnd;
+	return {
+		metadata,
+		syncMarker
+	};
+}
+export { avroMetadata, checkBlockEnd, readBlockEnd, readZigZag, readZigZagBigInt };