spark-connect 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ad83f60b70501f49722aa4ae51d768d27accbeb056c7f6fe1be695e010116c52
+  data.tar.gz: 528b67cacc0ba1b516aa93228c242d07efcaa86ae297d75c9c11c9d36905a00f
+SHA512:
+  metadata.gz: ceb4e18bcc75a30a38fe4c2446ae192f07a24446f86b5a76b9992b4f6a70e59bfb3d4709f633988eacd00fa85d174b8904b80eb291855dc943a7c8e10f9c7f4d
+  data.tar.gz: e06a6c98eee8042f56aabc5689655ad8bb7b04c843d90d2093d24fc6c82247993c0e47b002125ab774b1ce87c0e3b4203a88de6864a0c3dde1b45e7236db29f0

data/CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.0] - 2026-06-10
+
+### Added
+
+- **Declarative Pipelines** (Spark 4.1+): `SparkSession#pipeline` returns a
+  `Pipeline` (dataflow graph). Define outputs (`create_table`,
+  `create_materialized_view`, `create_temporary_view`, `create_sink`) and flows
+  (`define_flow`, `define_sql`), then `start_run` (with `full_refresh`/`refresh`/
+  `dry`/`storage`) which streams `PipelineEvent`s; plus `read` and `drop`.
+- **Structured Streaming**: `SparkSession#read_stream` (`DataStreamReader`),
+  `DataFrame#write_stream` (`DataStreamWriter`), `StreamingQuery` (status,
+  recent/last progress, await_termination, process_all_available, stop,
+  exception, explain), and `SparkSession#streams` (`StreamingQueryManager`:
+  active, get, await_any_termination, reset_terminated). Supports triggers
+  (processing-time, once, available-now, continuous), output modes, and
+  file/console/memory/Kafka sinks. (`foreach`/`foreachBatch` and UDFs remain
+  unsupported pending finalized protobuf definitions.)
+- Temporary views: `DataFrame#create_temp_view` /
+  `create_or_replace_temp_view` / `create_global_temp_view` /
+  `create_or_replace_global_temp_view`.
+- `DataFrame#with_watermark`, `repartition_by_range`, `checkpoint` /
+  `local_checkpoint`, `transform`, `col_regex`, `to_json`, and
+  `each` / `to_local_iterator`.
+- `Catalog#create_table` and `Catalog#create_external_table`.
+- `SparkSession#new_session`, `interrupt_all` / `interrupt_tag` /
+  `interrupt_operation`, and operation tags (`add_tag` / `remove_tag` /
+  `get_tags` / `clear_tags`) propagated on every execution.
+- Regenerated the vendored protobuf/gRPC stubs against Spark Connect **4.1.0**
+  (adds `pipelines.proto`).
+- `bin/console` (an IRB session with `spark`/`F`/`T`) and `bin/setup`.
+
+## [0.1.0] - 2026-06-10
+
+Initial release.
+
+### Added
+
+- `SparkConnect::SparkSession` and its fluent `Builder` (`remote`, `app_name`,
+  `config`, `get_or_create`/`create`), plus `range`, `sql`, `table`, `read`,
+  `create_data_frame`, `conf`, `catalog`, `version`, and `stop`.
+- `SparkConnect::DataFrame` with the core transformation and action surface:
+  `select`/`select_expr`, `filter`/`where`, `with_column(s)`,
+  `with_column(s)_renamed`, `drop`, `distinct`/`drop_duplicates`,
+  `order_by`/`sort`/`sort_within_partitions`, `limit`/`offset`,
+  `group_by`/`rollup`/`cube`/`agg`, `join`/`cross_join`,
+  `union`/`union_by_name`/`intersect`/`except`/`subtract`,
+  `repartition`/`coalesce`, `sample`, `alias`, `hint`, `unpivot`, `to`/`to_df`,
+  `collect`/`take`/`head`/`first`/`count`/`show`/`to_arrow`, and plan
+  introspection (`schema`, `columns`, `dtypes`, `print_schema`, `explain`).
+- `SparkConnect::Column` with Ruby operator overloads, aliasing, casting,
+  sort ordering, predicates, `when`/`otherwise`, complex-type access, and `over`.
+- `SparkConnect::Functions` (`F`): a broad PySpark-compatible function library
+  including aggregate, math, string, date/time, collection, JSON, conditional,
+  hashing, and higher-order (lambda) functions.
+- `SparkConnect::Window`/`WindowSpec` for analytic window definitions.
+- `SparkConnect::GroupedData`, `DataFrameNaFunctions` (`na`),
+  `DataFrameStatFunctions` (`stat`), and `Observation`.
+- `SparkConnect::DataFrameReader`, `DataFrameWriter`, and `DataFrameWriterV2`.
+- `SparkConnect::Catalog` and `SparkConnect::RuntimeConfig`.
+- `SparkConnect::Types`: a full Spark SQL type system with proto conversion,
+  `simpleString`/DDL/JSON rendering, and schema trees.
+- `SparkConnect::Row` with positional, by-name, and method-style access.
+- Apache Arrow IPC result decoding and local-relation encoding via `red-arrow`.
+- gRPC client (`SparkConnectClient`, `ChannelBuilder`) with `sc://` connection
+  string parsing, TLS/bearer-token auth, and retry-with-backoff.
+- A structured error hierarchy (`Error`, `ConnectionError`, `SparkConnectError`,
+  `AnalysisError`, `ParseError`, ...).
+- Vendored Spark Connect 4.1 protobuf/gRPC definitions and a regeneration script
+  (`bin/generate-protos`).
+
+[Unreleased]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.2.0...HEAD
+[0.2.0]: https://github.com/HyukjinKwon/spark-connect-ruby/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/HyukjinKwon/spark-connect-ruby/releases/tag/v0.1.0

data/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/NOTICE

@@ -0,0 +1,16 @@
+spark-connect (Ruby client for Apache Spark Connect)
+Copyright 2026 The spark-connect-ruby authors
+
+This product includes software developed as part of the
+Apache Spark project (https://spark.apache.org/).
+
+The Protocol Buffers definitions under proto/spark/connect/ are vendored from
+Apache Spark (https://github.com/apache/spark), which is licensed under the
+Apache License, Version 2.0:
+
+    Apache Spark
+    Copyright 2014 and onwards The Apache Software Foundation.
+
+This product is not affiliated with, endorsed by, or sponsored by the Apache
+Software Foundation. "Apache", "Apache Spark", and "Spark" are trademarks of the
+Apache Software Foundation.

data/README.md

@@ -0,0 +1,166 @@
+# spark-connect (Ruby)
+
+[![CI](https://github.com/HyukjinKwon/spark-connect-ruby/actions/workflows/ci.yml/badge.svg)](https://github.com/HyukjinKwon/spark-connect-ruby/actions/workflows/ci.yml)
+[![Gem Version](https://badge.fury.io/rb/spark-connect.svg)](https://rubygems.org/gems/spark-connect)
+[![Docs](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://hyukjinkwon.github.io/spark-connect-ruby/)
+[![License](https://img.shields.io/badge/license-Apache--2.0-green.svg)](LICENSE)
+
+A pure-Ruby client for **[Apache Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html)** - the gRPC-based, decoupled client/server protocol for Apache Spark.
+
+`spark-connect` lets you build and run Spark DataFrame queries from Ruby against a remote Spark cluster, with an API that closely mirrors PySpark. No JVM, no local Spark installation, no `spark-submit` - just a gRPC connection to a Spark Connect server.
+
+```ruby
+require "spark-connect"
+
+spark = SparkConnect::SparkSession.builder
+                                  .remote("sc://localhost:15002")
+                                  .get_or_create
+
+F = SparkConnect::F
+
+spark.range(1, 1_000)
+     .select(F.col("id"), (F.col("id") % 3).alias("bucket"))
+     .group_by("bucket")
+     .agg(F.count("*").alias("n"), F.sum("id").alias("total"))
+     .order_by("bucket")
+     .show
+
+spark.stop
+```
+
+```
++------+---+------+
+|bucket|  n| total|
++------+---+------+
+|     0|333|166833|
+|     1|333|166167|
+|     2|333|166500|
++------+---+------+
+```
+
+## What it supports
+
+`spark-connect` implements the Spark Connect **DataFrame**, **SQL**,
+**Structured Streaming**, and **Declarative Pipelines** API -- everything
+**except** user-defined functions (UDFs) and the `foreach`/`foreachBatch`
+streaming sinks, whose Spark Connect protobuf definitions are not yet finalized.
+(The separate, experimental MLlib-over-Connect surface is also out of scope.)
+
+Results decode through Apache Arrow into ordered, name-addressable `Row`s.
+Method names are snake_case (idiomatic Ruby) with camelCase aliases for the
+common PySpark names (`groupBy`, `withColumn`, `orderBy`, `createDataFrame`,
+...), so PySpark code ports almost verbatim.
+
+## Requirements
+
+- **Ruby >= 3.1**
+- **Apache Arrow C++/GLib system libraries** (required by the `red-arrow` dependency):
+  - macOS: `brew install apache-arrow apache-arrow-glib`
+  - Ubuntu/Debian: install `libarrow-glib-dev` from the [Apache Arrow APT repository](https://arrow.apache.org/install/)
+- A reachable **Spark Connect server**. This client is generated against the Spark Connect 4.1 protocol and supports **Apache Spark 3.5 and above**.
+
+See the [installation guide](https://hyukjinkwon.github.io/spark-connect-ruby/installation.html) for details.
+
+## Installation
+
+```bash
+gem install spark-connect
+```
+
+Or in a `Gemfile`:
+
+```ruby
+gem "spark-connect"
+```
+
+## Running a local Spark Connect server
+
+```bash
+# Download a Spark distribution (4.1.0 shown here; 3.5+ also works)
+curl -fsSL https://archive.apache.org/dist/spark/spark-4.1.0/spark-4.1.0-bin-hadoop3.tgz | tar xz
+cd spark-4.1.0-bin-hadoop3
+
+# Start the Connect server (requires Java 17+)
+./sbin/start-connect-server.sh --jars "$(pwd)/jars/spark-connect_2.13-4.1.0.jar"
+```
+
+The server listens on `sc://localhost:15002` by default.
+
+## Connecting
+
+Connection strings follow the standard Spark Connect grammar:
+
+```ruby
+# Plaintext, local
+SparkConnect::SparkSession.builder.remote("sc://localhost:15002").get_or_create
+
+# TLS + bearer token (token implies SSL)
+SparkConnect::SparkSession.builder
+  .remote("sc://spark.example.com:443/;token=#{ENV['SPARK_TOKEN']};user_id=alice")
+  .get_or_create
+```
+
+Supported parameters: `token`, `user_id`, `user_agent`, `use_ssl`, `session_id`, and any `x-*` custom gRPC headers.
+
+## A quick tour
+
+```ruby
+F = SparkConnect::F
+T = SparkConnect::Types
+
+# Build a DataFrame from local Ruby data
+df = spark.create_data_frame([
+  { "name" => "alice", "dept" => "eng", "salary" => 120 },
+  { "name" => "bob",   "dept" => "eng", "salary" => 100 },
+  { "name" => "carol", "dept" => "ops", "salary" => 110 },
+])
+
+# Transform and aggregate
+df.where(F.col("salary") >= 105)
+  .group_by("dept")
+  .agg(F.avg("salary").alias("avg_salary"), F.count("*").alias("headcount"))
+  .order_by(F.col("avg_salary").desc)
+  .show
+
+# Window functions
+w = SparkConnect::Window.partition_by("dept").order_by(F.col("salary").desc)
+df.with_column("rank", F.rank.over(w)).show
+
+# Schemas
+df.print_schema
+df.schema.simple_string  #=> "struct<name:string,dept:string,salary:bigint>"
+
+# SQL with parameters
+spark.sql("SELECT * FROM VALUES (1), (2), (3) AS t(x) WHERE x > :min", { min: 1 }).show
+```
+
+## Documentation
+
+Full documentation, including guides for every part of the API, lives at
+**<https://hyukjinkwon.github.io/spark-connect-ruby/>**.
+
+Runnable [`examples/`](examples/) cover quickstart, transformations, aggregations, joins, window functions, SQL, reading/writing, local data, and NA/stat helpers.
+
+## Compatibility
+
+The client is generated against the **Spark Connect 4.1** protocol and supports **Apache Spark 3.5 and above** (the Spark Connect wire protocol is backward compatible across these releases).
+
+## Development
+
+```bash
+git clone https://github.com/HyukjinKwon/spark-connect-ruby
+cd spark-connect-ruby
+bundle install
+
+bundle exec rake spec      # unit specs (no server required)
+bundle exec rake rubocop   # lint
+bundle exec rake yard      # API docs
+
+# Integration specs against a live server
+SPARK_REMOTE=sc://localhost:15002 bundle exec rspec spec/integration
+
+# Regenerate the protobuf/gRPC stubs from the vendored .proto files
+bin/generate-protos
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md).

data/lib/spark-connect.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+# Convenience entry point matching the gem name (`require "spark-connect"`).
+# Delegates to the canonical namespace file.
+require_relative "spark_connect"

data/lib/spark_connect/arrow.rb

@@ -0,0 +1,115 @@
+# frozen_string_literal: true
+
+require "arrow"
+
+module SparkConnect
+  # Decodes the Apache Arrow IPC stream payloads returned by the server into
+  # Ruby {Row}s. Each `ExecutePlanResponse.arrow_batch.data` chunk is a complete,
+  # self-contained Arrow IPC stream (schema + record batches); this converter
+  # reads each chunk and flattens all record batches into rows.
+  #
+  # Named `ArrowConverter` (not `Arrow`) so that references to the `red-arrow`
+  # top-level {Arrow} constant inside the gem are unambiguous.
+  module ArrowConverter
+    module_function
+
+    # Decode IPC stream chunks into rows.
+    #
+    # @param batches [Array<String>] Arrow IPC stream byte chunks.
+    # @return [Array<Row>]
+    def to_rows(batches)
+      rows = []
+      field_names = nil
+      batches.each do |data|
+        next if data.nil? || data.empty?
+
+        reader = ::Arrow::RecordBatchStreamReader.new(::Arrow::BufferInputStream.new(::Arrow::Buffer.new(data)))
+        reader.each do |record_batch|
+          field_names ||= record_batch.schema.fields.map(&:name)
+          record_batch.raw_records.each do |values|
+            rows << Row.new(values, fields: field_names)
+          end
+        end
+      end
+      rows
+    end
+
+    # Decode IPC stream chunks into a single Arrow Table (for advanced/columnar
+    # consumers who want zero-copy access to the underlying data).
+    #
+    # @param batches [Array<String>]
+    # @return [Arrow::Table, nil]
+    def to_table(batches)
+      tables = batches.reject { |b| b.nil? || b.empty? }.map do |data|
+        ::Arrow::RecordBatchStreamReader.new(::Arrow::BufferInputStream.new(::Arrow::Buffer.new(data))).read_all
+      end
+      return nil if tables.empty?
+
+      tables.reduce { |acc, t| acc.concatenate([t]) }
+    end
+
+    # Serialize an array of Ruby hashes into a single Arrow IPC stream given a
+    # Spark {Types::StructType}. Used by {SparkSession#create_data_frame} to ship
+    # local data to the server as a `LocalRelation`.
+    #
+    # @param rows [Array<Hash, Array, Row>]
+    # @param schema [Types::StructType]
+    # @return [String] Arrow IPC stream bytes.
+    def from_rows(rows, schema)
+      arrow_schema = build_arrow_schema(schema)
+      raw_rows = rows.map do |row|
+        schema.fields.each_with_index.map { |field, idx| extract_value(row, field.name, idx) }
+      end
+      record_batch = ::Arrow::RecordBatch.new(arrow_schema, raw_rows)
+      buffer = ::Arrow::ResizableBuffer.new(1024)
+      ::Arrow::BufferOutputStream.open(buffer) do |output|
+        ::Arrow::RecordBatchStreamWriter.open(output, arrow_schema) do |writer|
+          writer.write_record_batch(record_batch)
+        end
+      end
+      buffer.data.to_s
+    end
+
+    def build_arrow_schema(schema)
+      fields = schema.fields.map do |f|
+        ::Arrow::Field.new(f.name, arrow_field_type(f.data_type))
+      end
+      ::Arrow::Schema.new(fields)
+    end
+
+    def extract_value(row, name, idx)
+      case row
+      when Row then row[name]
+      when Hash
+        if row.key?(name) then row[name]
+        elsif row.key?(name.to_sym) then row[name.to_sym]
+        end
+      when Array then row[idx]
+      else row
+      end
+    end
+
+    # Map a Spark type to the corresponding `red-arrow` data type used when
+    # building local relations.
+    def arrow_field_type(data_type)
+      case data_type
+      when Types::BooleanType then :boolean
+      when Types::ByteType then :int8
+      when Types::ShortType then :int16
+      when Types::IntegerType then :int32
+      when Types::LongType then :int64
+      when Types::FloatType then :float
+      when Types::DoubleType then :double
+      when Types::StringType, Types::CharType, Types::VarcharType then :string
+      when Types::BinaryType then :binary
+      when Types::DateType then :date32
+      when Types::TimestampType, Types::TimestampNTZType then { type: :timestamp, unit: :micro }
+      when Types::ArrayType
+        { type: :list, field: { name: "element", type: arrow_field_type(data_type.element_type) } }
+      when Types::StructType
+        { type: :struct, fields: data_type.fields.map { |f| { name: f.name, type: arrow_field_type(f.data_type) } } }
+      else :string # rubocop:disable Lint/DuplicateBranch -- string default for unsupported-locally types
+      end
+    end
+  end
+end