parquet 0.8.0-aarch64-linux → 0.9.0-aarch64-linux

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89db55543839853aef62e3f11511ba4ff54ee1896c7085b83d9e6e44d2b10335
-  data.tar.gz: 07db9c35d4c4777d9339f92b22112760c9ff0e96091b7dc26940c54ed4df03b8
+  metadata.gz: 06deeee590e4572ff7d4b302597e26804ca0a7dc4a8eccfd7687f926276a5ef7
+  data.tar.gz: 5514690113ee28379aa505d49ee88cd0e806ad8f4fb7e442c3ffef6e0d36305c
 SHA512:
-  metadata.gz: 3f78682dc2b7b4d5aa3fa186b7cf743b1808c93644ee2097d76051c21e900da23578493a3d0e637b4bc61dd9ce06a85865fdf46e33a24acf46ff059519b5199f
-  data.tar.gz: ca2f9c79cfd0faf50567c7df2d73cbbea508e5ade7234ac8d2791288d0f174d1b34f707e63ef29765cdd6573ca34ac8ee0694f21d3ba46d7a8fd0ee49eb2f3cc
+  metadata.gz: e68aced5c5b45ae7dbcd5f3b8215817c10a0cc72379044523895870e3e164a4212fd24d34b9eca9e7913d3f6ff8f43c040ac3d3094a0af955b4353a710db38e2
+  data.tar.gz: 44998090036e3de92c39f2418ccb68369564209e2a3d165405fa56e14d96b1b283aa3e3980cd5880c861e578b1fb90f0332412ee472669e09ad5f2256af11d6c

data/README.md

@@ -145,6 +145,33 @@ puts metadata["row_groups"].size    # Number of row groups
 
 ## Writing Parquet Files
 
+`write_rows` and `write_columns` stream an enumerable to a path or writable IO
+and return `nil`. Each row is an array in schema order. For column writing, each
+yielded batch contains one array per field, all of the same length.
+
+Both methods require `schema:` and `write_to:`. `schema:` accepts the array form
+shown below, a `Parquet::Schema` DSL result, or a `fields` schema hash. Use `nil`
+or `[]` to infer string columns named `f0`, `f1`, ... from the first row or
+batch. Empty input requires an explicit schema.
+
+`compression:` accepts `"none"`, `"uncompressed"`, `"snappy"`, `"gzip"`,
+`"lz4"`, `"zstd"`, and `"brotli"`; `nil` defaults to Snappy.
+`flush_threshold:` defaults to 100 MiB. `logger:` accepts a Ruby logger with
+`debug`, `info`, `warn`, and `error` methods.
+
+`write_rows` also accepts `batch_size:`, `sample_size:`, and `string_cache:`.
+Without `batch_size:`, sizing starts at 1,000 rows and adapts to row size; the
+cap is 1,000,000 rows and lower for wide schemas. Sampling defaults to 100 rows
+and is capped at 10,000. `string_cache: true` uses a capacity of 100, or you can
+pass a capacity up to 65,536; `nil` and `false` disable it.
+
+Path output is staged and atomically published only after the complete file has
+been written. On Unix, replacing an existing path preserves its uid, gid, and
+mode and uses standard last-committer-wins rename semantics; creating a path is
+no-clobber. Extended attributes and ACLs are not preserved. IO output is first
+staged on disk, then copied to the IO; a failed copy may leave the IO partially
+written.
+
 ### Row-wise Writing
 
 Best for: Streaming data, converting from other formats, memory-constrained environments
@@ -159,17 +186,109 @@ schema = [
 ]
 
 # Stream data from any enumerable
-rows = CSV.foreach("input.csv").map do |row|
+rows = CSV.foreach("input.csv").lazy.map do |row|
   [row[0].to_i, row[1], row[2] == "true", row[3].to_f]
 end
 
 Parquet.write_rows(rows,
   schema: schema,
   write_to: "output.parquet",
-  batch_size: 5000  # Positive rows per batch (default: 1000)
+  batch_size: 5000
+)
+```
+
+### Repacking Existing Parquet Files
+
+Concatenate Parquet files and re-split them into differently sized files without
+translating rows through Ruby.
+
+```ruby
+Parquet.repack(
+  ["input-0.parquet", "input-1.parquet"],
+  output_dir: "repacked",
+  rows_per_file: 100_000
 )
+# => [{ "path" => "repacked/batch-0.parquet", "num_rows" => 100_000 },
+#     { "path" => "repacked/batch-1.parquet", "num_rows" => 42_137 }]
+```
+
+The outputs hold exactly the input rows, in input order. Every output but the
+last holds `rows_per_file` rows, and there is always at least one output even
+when the inputs are empty. Omit `rows_per_file:` to concatenate everything into
+a single file.
+
+Each output's Parquet schema is identical to the first input's, and that input's
+file-level key/value metadata (`ARROW:schema`, `pandas`, and so on) is carried
+over. Inputs must agree on leaf column shape — path, physical and logical type,
+nesting — but may differ in key/value metadata and Parquet field ids.
+
+#### Compression and copying
+
+With no `compression:`, each column keeps its own codec — Parquet records one
+per column, and a file may legitimately use several. Naming a codec applies it
+to every column instead:
+
+```ruby
+Parquet.repack("input.parquet", output_dir: "out", compression: "zstd")
 ```
 
+Keeping the inputs' codecs also lets repack copy whole row groups into the
+output byte-for-byte, skipping decompression and re-encoding entirely. A row
+group is copied when it fits the output's remaining row budget, is large enough
+to be worth copying, and the request did not ask for a different codec;
+otherwise its rows are decoded and re-encoded. Both routes produce the same
+rows, so which one runs is not something callers need to reason about — but the
+copy route is dramatically faster, so a plain concatenation is close to an
+I/O-bound copy.
+
+Small row groups are deliberately merged rather than copied: copying them
+one-for-one would make a compaction of many small files reproduce exactly the
+fragmentation it was meant to remove.
+
+#### Output directory ownership
+
+`repack` owns the `{output_file_prefix}-{n}.parquet` names in `output_dir`. If
+any already exist it raises `ArgumentError` rather than mixing two runs' files
+in one directory:
+
+```ruby
+Parquet.repack("input.parquet", output_dir: "out", rows_per_file: 1000, overwrite: true)
+```
+
+`overwrite: true` replaces that set and deletes members left over from a longer
+earlier run, so the returned list always equals what a reader finds in the
+directory. Files outside the set are never touched.
+
+#### Bounds
+
+`max_read_rows_per_chunk:` (default 8192, reduced for wide schemas) bounds rows
+buffered while reading; output row groups are bounded in rows by the same slot
+budget. Both are resource controls: varying them cannot change the returned
+list, the rows, the schema, or the codecs. They do shift compressed byte counts
+and page boundaries, which are representation rather than contract.
+
+Input metadata is read one file at a time, so peak memory is set by the widest
+single file and its row-group size, not by how many files you pass or how many
+rows they hold in total. Note that the row-group bound is in rows, not bytes: a
+schema with very large values still buffers one row group's worth of encoded
+data.
+
+Reading and writing run with the GVL released, so other Ruby threads keep
+running and `Interrupt` / `Timeout` are honoured. An interrupted call leaves no
+output behind.
+
+A repacked output is capped at 32,767 row groups. Merging small groups keeps
+that limit out of reach in practice; if a single output would exceed it,
+repack raises rather than writing an unreadable file, and `rows_per_file:` is
+the way out.
+
+#### Page indexes and other optional structures
+
+Every row group in a Parquet file must agree on whether it carries a page index,
+and a copied row group can only contribute the index its source had. So an
+output carries one exactly when every contributing input does. Bloom filters are
+not carried over on either route.
+
 ### Column-wise Writing
 
 Best for: Pre-columnar data, better compression, higher performance
@@ -196,13 +315,10 @@ schema = [
 Parquet.write_columns(batches.each,
   schema: schema,
   write_to: "output.parquet",
-  compression: "snappy"  # Options: none, snappy, gzip, lz4, zstd
+  compression: "snappy"
 )
 ```
 
-`write_columns` also accepts `logger:` with the same Ruby logger interface as
-row writes.
-
 ## Data Types
 
 ### Basic Types
@@ -385,14 +501,19 @@ Control memory usage with flush thresholds:
 Parquet.write_rows(huge_dataset.each,
   schema: schema,
   write_to: "output.parquet",
-  batch_size: 1000,              # Positive rows before considering flush
-  flush_threshold: 32 * 1024**2  # Flush if batch exceeds 32MB
+  batch_size: 1_000,
+  flush_threshold: 32 * 1024**2 # 32 MiB
 )
 ```
 
-Write batch and sample sizes are bounded before buffer allocation. Very large
-batch sizes are rejected, and wide schemas have a lower effective batch cap so
-the writer cannot reserve unbounded per-column value slots.
+Writer-owned memory stays bounded as the file grows. `flush_threshold` controls
+the converted-value buffer; a single larger row may exceed it temporarily. The
+row-group target is at least 8 MiB, and each file may contain up to 32,768 row
+groups.
+
+Encoded data and completed row-group metadata are staged on disk, so large
+writes need temporary disk space. Ruby still owns the current row or batch, and
+an in-memory destination such as `StringIO` holds the output in memory.
 
 ## Architecture
 

data/lib/parquet/version.rb

@@ -1,3 +1,3 @@
 module Parquet
-  VERSION = "0.8.0"
+  VERSION = "0.9.0"
 end

data/lib/parquet.rbi

@@ -90,7 +90,8 @@ module Parquet
   end
 
   # Options:
-  #   - `read_from`: An Enumerator yielding arrays of values representing each row
+  #   - `read_from`: An Enumerable yielding arrays of values representing each row. The outer
+  #                  enumerable is pulled incrementally and is never materialized with `to_a`.
   #   - `schema`: Array of hashes specifying column names and types. Supported types:
   #     - `int8`, `int16`, `int32`, `int64`
   #     - `uint8`, `uint16`, `uint32`, `uint64`
@@ -103,8 +104,9 @@ module Parquet
   #   - `write_to`: String path or IO object to write the parquet file to
   #   - `batch_size`: Optional positive batch size for writing (defaults to 1000, at most 1_000_000
   #                   for one-column schemas; wide schemas may have a lower safety cap)
-  #   - `flush_threshold`: Optional threshold in bytes for the writer's in-progress (encoded)
-  #                        buffer before a row group is flushed (defaults to 100MB)
+  #   - `flush_threshold`: Optional positive byte quantum for converted native values (defaults
+  #                        to 100MB). One larger row is written alone. Encoded row groups use this
+  #                        value with an 8MB minimum; completed footer metadata is disk-spooled.
   #   - `compression`: Optional compression type to use (defaults to "zstd")
   #                   Supported values: "none", "uncompressed", "snappy", "gzip", "lz4", "zstd"
   #   - `sample_size`: Optional positive number of rows to sample for size estimation
@@ -116,7 +118,7 @@ module Parquet
   #                     cached string content.
   sig do
     params(
-      read_from: T::Enumerator[T::Array[T.untyped]],
+      read_from: T::Enumerable[T::Array[T.untyped]],
       schema: T::Array[T::Hash[String, String]],
       write_to: T.any(String, IO),
       batch_size: T.nilable(Integer),
@@ -139,7 +141,8 @@ module Parquet
   end
 
   # Options:
-  #   - `read_from`: An Enumerator yielding arrays of column batches
+  #   - `read_from`: An Enumerable yielding arrays of column batches. Batches are validated and
+  #                  consumed one at a time; all columns in each batch must have equal length.
   #   - `schema`: Array of hashes specifying column names and types. Supported types:
   #     - `int8`, `int16`, `int32`, `int64`
   #     - `uint8`, `uint16`, `uint32`, `uint64`
@@ -151,14 +154,15 @@ module Parquet
   #     - `timestamp_millis`, `timestamp_micros`
   #     - Looks like [{"column_name" => {"type" => "date32", "format" => "%Y-%m-%d"}}, {"column_name" => "int8"}]
   #   - `write_to`: String path or IO object to write the parquet file to
-  #   - `flush_threshold`: Optional threshold in bytes for the writer's in-progress (encoded)
-  #                        buffer before a row group is flushed (defaults to 100MB)
+  #   - `flush_threshold`: Optional positive byte quantum for converted native values (defaults
+  #                        to 100MB). One larger row is written alone. Encoded row groups use this
+  #                        value with an 8MB minimum; completed footer metadata is disk-spooled.
   #   - `compression`: Optional compression type to use (defaults to "zstd")
   #                   Supported values: "none", "uncompressed", "snappy", "gzip", "lz4", "zstd"
   #   - `logger`: Optional Ruby logger for column-write progress messages
   sig do
     params(
-      read_from: T::Enumerator[T::Array[T::Array[T.untyped]]],
+      read_from: T::Enumerable[T::Array[T::Array[T.untyped]]],
       schema: T::Array[T::Hash[String, String]],
       write_to: T.any(String, IO),
       flush_threshold: T.nilable(Integer),
@@ -175,4 +179,57 @@ module Parquet
     logger: nil
   )
   end
+
+  # Concatenates Parquet files and re-splits them into a new set of files
+  # without translating rows through Ruby. Returns one hash per output file,
+  # `{"path" => String, "num_rows" => Integer}`, in output order.
+  #
+  # The outputs hold exactly the input rows, in input order. Every output but
+  # the last holds `rows_per_file` rows; there is always at least one output,
+  # even when the inputs are empty. Each output's Parquet schema is identical
+  # to the first input's, and that input's file-level key/value metadata (such
+  # as `ARROW:schema` or `pandas`) is carried over.
+  #
+  # Inputs must agree on leaf column shape — path, physical and logical type,
+  # nesting. They may differ in key/value metadata and Parquet field ids.
+  #
+  # Options:
+  #   - `read_from`: String path or array of paths to Parquet files with matching schemas
+  #   - `output_dir`: Directory where {output_file_prefix}-{n}.parquet files will be written
+  #   - `output_file_prefix`: Single filename component used for outputs, default "batch".
+  #     Path separators and `..` are rejected.
+  #   - `rows_per_file`: Optional maximum number of rows per output file. When nil, all input
+  #     rows are concatenated into one file.
+  #   - `max_read_rows_per_chunk`: Optional upper bound for rows read per chunk, default 8192
+  #     and reduced for wide schemas. It bounds memory only; it never changes the returned
+  #     list, the rows, the schema, or the codecs.
+  #   - `compression`: Optional codec for the outputs. When nil each column keeps its own
+  #     codec, which also lets whole row groups be copied without re-encoding.
+  #   - `overwrite`: When false (default), a non-empty `{output_file_prefix}-*.parquet` set in
+  #     `output_dir` raises ArgumentError. When true, that set is replaced and any files left
+  #     over from a longer earlier run are removed. Files outside the set are never touched.
+  #
+  # Raises ArgumentError for an invalid request or mismatched input schemas, and
+  # IOError when an input or output cannot be read or written.
+  sig do
+    params(
+      read_from: T.any(String, T::Array[String]),
+      output_dir: String,
+      output_file_prefix: T.nilable(String),
+      rows_per_file: T.nilable(Integer),
+      max_read_rows_per_chunk: T.nilable(Integer),
+      compression: T.nilable(String),
+      overwrite: T.nilable(T::Boolean)
+    ).returns(T::Array[T::Hash[String, T.any(String, Integer)]])
+  end
+  def self.repack(
+    read_from,
+    output_dir:,
+    output_file_prefix: nil,
+    rows_per_file: nil,
+    max_read_rows_per_chunk: nil,
+    compression: nil,
+    overwrite: nil
+  )
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: parquet
 version: !ruby/object:Gem::Version
-  version: 0.8.0
+  version: 0.9.0
 platform: aarch64-linux
 authors:
 - Nathan Jaremko
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2026-06-25 00:00:00.000000000 Z
+date: 2026-08-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bigdecimal