galaaz 2.1.0 → 2.1.1

data/blogs/manual/manual.md

@@ -59,9 +59,11 @@ CRuby when you prefer MRI. R remains the same **GNU R** you use interactively—
 compiled extensions and Bioconductor. Earlier GraalVM / TruffleRuby / FastR experiments
 are no longer the focus.
 
-The bridge handles **communication and typing** between the two worlds; large tables can
-also flow through **Apache Arrow** on the R side when you use the optional helpers described
-later in this manual.
+The bridge handles **communication and typing** between the two worlds. Large tables can use
+**Apache Arrow** in two shipped modes (described later): **Stage A** copies Ruby batches into an
+R-side Arrow table (`R::Arrow.from_ruby_batches`); **Stage B** writes an Arrow IPC file and only
+the **path** crosses NewBridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). Shared-heap
+zero-copy is **Stage C** and is not shipped.
 
 ## R-on-Rails: the one-person app for R scientists
 
@@ -340,7 +342,7 @@ The supported install is **`gem install` + compile the gatekeeper**. You do not
    make -C "${gem_dir}/ext/new_bridge" all
    ```
 
-5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on Java 9+, pass `-J--add-opens=java.base/java.nio=ALL-UNNAMED` to JRuby (from a checkout, `bin/galaaz-jruby` does this; on CRuby this flag is not needed).
+5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on **JRuby** (Java 9+), the child JVM needs `--add-opens=java.base/java.nio=ALL-UNNAMED` via **`JAVA_OPTS`** (from a checkout, `bin/galaaz-jruby` and `mise.toml` set this; a leading `jruby -J... -S bundle exec` does **not** pass `-J` to rspec). On **CRuby**, install Apache Arrow GLib (`libarrow-glib-dev` from the [Apache Arrow APT](https://arrow.apache.org/install/) repo) and `gem install red-arrow` matching `pkg-config --modversion arrow-glib`. Do not install the unrelated Rubygems package named `arrow`.
 
 For **gKnit**, **knitr**, **rmarkdown**, and LaTeX (PDF output), install the corresponding R packages, **Pandoc**, and a TeX distribution if you need PDF; the repository includes helpers such as **`bin/install-tinytex`** where appropriate.
 
@@ -442,13 +444,15 @@ A practical pattern is:
 
 1. Use threads (or a connection pool) to read from **multiple databases or shards** in parallel.
 2. Merge the rows in Ruby under a `Mutex` if you collect into one structure.
-3. Hand the merged table to R **once** (for example with `R::Arrow.from_ruby_batches` and dplyr,
-   or by building a data frame) so heavy statistics run in R with fewer bridge round-trips.
+3. Hand the merged table to R **once**: **`R::Arrow.from_ruby_batches`** (Stage A: copy into R) or
+   **`Galaaz::ArrowIpc.write` / `write_batches`** then **`R::Arrow.open_ipc`** (Stage B: IPC file;
+   only the path crosses the bridge). Then run dplyr in R.
 
 A runnable sketch lives in
 `examples/multithread_shards_to_r/shards_to_r.rb` (simulated shard queries; swap in your DB
 driver). For concurrency tests on the bridge itself, see `specs/bridge_concurrent_spec.rb` and
-`specs/arrow_from_ruby_batches_spec.rb`.
+`specs/arrow_from_ruby_batches_spec.rb`. Stage B IPC tests: `specs/arrow_ipc_handoff_spec.rb`,
+`specs/arrow_ipc_export_spec.rb`.
 
 ## Long-running R calls and a completion block
 
@@ -4025,35 +4029,77 @@ ans = flights[:all, E.list(R[:arr_delay], R[:dep_delay])]
 # Apache Arrow
 
 [Apache Arrow](https://arrow.apache.org/) is a **columnar** in-memory format used heavily in R
-and Python for analytics. In Galaaz, **Ruby does not hold an Arrow C++ table itself**; instead you
-build ordinary Ruby structures (arrays of row hashes), and **`R::Arrow.from_ruby_batches`** creates
-a real **Arrow `Table` inside GNU R**. From there you use R’s **`arrow`** and **`dplyr`** packages
-as usual: **`group_by`** on the Arrow table, **`summarise`** for aggregates, then **`collect()`** to
-materialize a tibble when you need in-memory R rows.
-
-That pattern matches production use: **JRuby threads** (or sequential code) assemble many rows in
-Ruby; you pay **one** bridge-heavy handoff to R; **dplyr** runs vectorised work on the Arrow table
-in R.
+and Python for analytics. GNU R still runs in a **separate process**. Ruby does **not** hold a
+shared Arrow C++ table with R. Stages:
+
+1. **Stage A (copy over the bridge):** Ruby row hashes → **`R::Arrow.from_ruby_batches`** builds
+   an Arrow `Table` **inside GNU R**. You get a **proxy**.
+2. **Stage B1 (Ruby → R IPC file):** **`Galaaz::ArrowIpc.write`** / **`write_batches`** writes an
+   Arrow IPC file (prefer **`/dev/shm`**); **`R::Arrow.open_ipc(path)`** opens it in R. Only the
+   **path** crosses NewBridge. This is **mmap/IPC file handoff**, not a shared heap.
+3. **Stage B2 (R → Ruby IPC file):** **`R::Arrow.write_ipc(obj)`** writes uncompressed IPC; Ruby
+   reads with **`Galaaz::ArrowIpc.read`** (column hash) or **`read_batches`** (row hashes). Call
+   **`Galaaz::ArrowIpc.release(path)`** when finished.
+4. **Stage C (not shipped):** named shared-memory bus. Do not claim 0 ms shared RAM until then.
+   See **`Documentation/ROADMAP_ARROW_RUBY_R.md`**.
+
+After ingest, use R’s **`arrow`** / **`dplyr`** on the proxy (`group_by`, `summarise`, `collect`)
+and unbox only KPIs you need in Ruby.
+
+**Optional Ruby backends for Stage B**
+
+* **CRuby:** Apache **red-arrow** — `gem install red-arrow` pinned to the same major as
+  `pkg-config --modversion arrow-glib`, plus system **Arrow GLib** (`libarrow-glib-dev` from the
+  [Apache Arrow APT](https://arrow.apache.org/install/) repo). Do **not** install the unrelated
+  legacy Rubygems package named `arrow`. `bundle exec` still sees a user-installed `red-arrow`
+  via Galaaz’s load-path helper.
+* **JRuby:** Apache Arrow **Java** JARs — **`GALAAZ_ARROW_JARS`**, `~/arrow_jars`, or
+  `jar-dependencies`. Export **`JAVA_OPTS=--add-opens=java.base/java.nio=ALL-UNNAMED`** on the
+  **child** JVM (`bin/galaaz-jruby`, `mise.toml`). `jruby -J... -S bundle exec rspec` does **not**
+  pass `-J` to rspec.
+
+**R packages:** **`arrow`** and **`dplyr`**. B2 writes IPC with **`compression: 'uncompressed'`**
+so JRuby Arrow Java can read without extra compression JARs.
+
+**Tests:** `specs/arrow_from_ruby_batches_spec.rb` (A);
+`specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb` (B, sync);
+`new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb` (B, async).
+
+## `R::Arrow` and `Galaaz::ArrowIpc`
+
+* **`R::Arrow.from_ruby_batches`** — Stage A ingest.
+* **`R::Arrow.open_ipc(path)`** — Stage B1: IPC file → R Table proxy.
+* **`R::Arrow.write_ipc(obj, path = nil)`** — Stage B2: R Table/tibble → IPC path (scratch if omitted).
+* **`Galaaz::ArrowIpc.write` / `write_batches` / `read` / `read_batches` / `allocate_path` / `release` / `available?`**
+* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
+* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
+  dataset IO on paths visible to R.
 
-**Prerequisites:** install R packages **`arrow`** and **`dplyr`**. Run scripts with
-**`bin/galaaz-jruby`** (or the same JVM flags as in **`docs/testing.md`**) so the Arrow JNI stack is
-available.
+## Example: Stage B round-trip (IPC file)
 
-## Other `R::Arrow` helpers
+Requires `Galaaz::ArrowIpc.available?` (red-arrow or Arrow JARs) and R **`arrow`**. Not knitted
+below so a machine without the optional backend still builds this manual.
 
-The Ruby module **`R::Arrow`** (see `lib/R_interface/r_arrow.rb`) also includes:
+```ruby
+path = Galaaz::ArrowIpc.write(id: [1, 2, 3], grp: %w[a a b], value: [1.0, 2.0, 3.5])
+tbl  = R::Arrow.open_ipc(path)
+Galaaz::ArrowIpc.release(path)
 
-* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
-* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
-  dataset IO on paths visible to R.
+summed   = R.dplyr___summarise(R.dplyr___group_by(tbl, :grp), total: E.sum(:value))
+out_path = R::Arrow.write_ipc(summed)
+rows     = Galaaz::ArrowIpc.read_batches(out_path)
+Galaaz::ArrowIpc.release(out_path)
+# rows => [{:grp=>"a", :total=>3.0}, {:grp=>"b", :total=>3.5}]  (illustrative)
+```
 
 ## Example: many Ruby rows → Arrow in R → grouped statistics
 
 The repository test **`slow-specs/arrow_large_pipeline_spec.rb`** builds **200k rows** in parallel
-(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`**, then checks that
-**dplyr** group summaries match a Ruby reference calculation. The same logic appears below at a
-**smaller scale** so this manual can knit quickly; increase `thread_count` and `rows_per_thread`
-when experimenting locally.
+(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`** (Stage A), then
+checks that **dplyr** group summaries match a Ruby reference calculation. The same logic appears
+below at a **smaller scale** so this manual can knit quickly; increase `thread_count` and
+`rows_per_thread` when experimenting locally. For the same ingest **without** copying every cell
+over NewBridge, use Stage B (`write_batches` + `open_ipc`) instead of `from_ruby_batches`.
 
 
 ``` ruby
@@ -4119,10 +4165,10 @@ end
 ```
 
 **What to notice:** (1) Ruby only sees **`Hash`** rows and Ruby **`Thread`** objects; (2) a single
-**`from_ruby_batches`** call creates the Arrow table in R; (3) **`dplyr___group_by`** /
+**`from_ruby_batches`** call **copies** those columns into an Arrow table in R; (3) **`dplyr___group_by`** /
 **`dplyr___summarise`** / **`dplyr___collect`** mirror **`dplyr::group_by`** /
 **`dplyr::summarise`** / **`dplyr::collect`** on an Arrow-backed table. For a lighter test, see
-**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size benchmark, run
+**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size Stage A benchmark, run
 **`bin/run_slow_rspec slow-specs/arrow_large_pipeline_spec.rb`**.
 
 # Bioconductor and DESeq2
@@ -4260,8 +4306,9 @@ Practical tips:
   glue.
 * **Reuse one process**: running many short scripts cold-starts Ruby, the JVM, and R each time;
   a long-lived process or repeated calls in one run amortize setup (see benchmarks below).
-* **Batch data**: merge shards in Ruby, then call **`R::Arrow.from_ruby_batches`** (or build one
-  data frame) instead of millions of tiny R calls.
+* **Batch data**: merge shards in Ruby, then **`R::Arrow.from_ruby_batches`** (Stage A) or
+  **`Galaaz::ArrowIpc`** + **`R::Arrow.open_ipc`** (Stage B) instead of millions of tiny R calls.
+  When Ruby needs a bulky result table back, **`R::Arrow.write_ipc`** + **`Galaaz::ArrowIpc.read_batches`**.
 
 For measured discussion (including DESeq2-style workloads and warm comparisons), see
 **`docs/performance.md`** and **`docs/deseq2_airway_benchmark.md`** in the Galaaz repository.

data/lib/R_interface/r_arrow.rb

@@ -63,6 +63,43 @@ module R
       R.arrow___as_arrow_table(r_df)
     end
 
+    # Open an Arrow IPC file (written by Galaaz::ArrowIpc or compatible) as an
+    # R-side Arrow Table. Materializes the table in R so the Ruby process may
+    # unlink the path immediately after this call returns (Stage B1 lifetime).
+    #
+    # This is IPC/mmap file handoff — not zero-copy shared heap with Ruby.
+    #
+    # @param path [String] filesystem path visible to the R process
+    # @return [R::Object] Arrow Table proxy
+    def self.open_ipc(path)
+      path = File.expand_path(path.to_s)
+      ok = R::Support.eval("requireNamespace('arrow', quietly=TRUE)")
+      unless ok == true
+        raise LoadError, "R package 'arrow' is required for R::Arrow.open_ipc"
+      end
+
+      R.arrow___read_ipc_file(path, as_data_frame: false)
+    end
+
+    # Write an R Arrow Table / data.frame to an Arrow IPC file Ruby can read
+    # with Galaaz::ArrowIpc.read (Stage B2). Only the path should cross NewBridge.
+    #
+    # @param r_obj [R::Object] Table, RecordBatch, or data.frame/tibble in R
+    # @param path [String, nil] destination; default a new scratch path
+    # @return [String] absolute path (fsync'd from Ruby after R returns)
+    def self.write_ipc(r_obj, path = nil)
+      ok = R::Support.eval("requireNamespace('arrow', quietly=TRUE)")
+      unless ok == true
+        raise LoadError, "R package 'arrow' is required for R::Arrow.write_ipc"
+      end
+
+      path = path.nil? || path.to_s.empty? ? Galaaz::ArrowIpc.allocate_path : File.expand_path(path.to_s)
+      # uncompressed so JRuby Arrow Java can read without arrow-compression JARs
+      R.arrow___write_ipc_file(r_obj, path, compression: 'uncompressed')
+      File.open(path, 'rb') { |f| f.fsync }
+      path
+    end
+
     # Open a Parquet/Feather directory or file as an Arrow Dataset
     # using arrow::open_dataset().
     #

data/lib/galaaz/arrow_ipc/java_arrow_backend.rb

@@ -0,0 +1,250 @@
+# frozen_string_literal: true
+
+##########################################################################################
+# JRuby Arrow IPC writer via Apache Arrow Java.
+#
+# JAR loading order:
+#   1. GALAAZ_ARROW_JARS directory (all *.jar)
+#   2. ~/arrow_jars if present
+#   3. jar-dependencies require_jar for pinned Arrow 18.1.0 artifacts
+##########################################################################################
+
+module Galaaz
+  module ArrowIpc
+    module JavaArrowBackend
+      ARROW_JAVA_VERSION = '18.1.0'
+
+      module_function
+
+      def available?
+        return false unless RUBY_ENGINE == 'jruby'
+
+        load_jars!
+        true
+      rescue LoadError, StandardError
+        false
+      end
+
+      def write_columns(columns_hash, path)
+        load_jars!
+        lengths = columns_hash.values.map(&:length).uniq
+        raise ArgumentError, "all columns must have the same length (got #{lengths.inspect})" if lengths.size != 1
+
+        n = lengths.first
+        alloc = org.apache.arrow.memory.RootAllocator.new(java.lang.Long::MAX_VALUE)
+        fields = []
+        kinds = {}
+
+        columns_hash.each do |name, values|
+          kind = infer_kind(values)
+          kinds[name] = kind
+          fields << org.apache.arrow.vector.types.pojo.Field.nullable(name, arrow_type(kind))
+        end
+
+        schema = org.apache.arrow.vector.types.pojo.Schema.new(fields)
+        root = org.apache.arrow.vector.VectorSchemaRoot.create(schema, alloc)
+
+        begin
+          columns_hash.each do |name, values|
+            fill_vector(root.getVector(name), kinds[name], values, n)
+          end
+          root.setRowCount(n)
+
+          fos = java.io.FileOutputStream.new(path)
+          begin
+            channel = java.nio.channels.Channels.newChannel(fos)
+            writer = org.apache.arrow.vector.ipc.ArrowFileWriter.new(root, nil, channel)
+            begin
+              writer.start
+              writer.writeBatch
+              writer.end
+            ensure
+              writer.close
+            end
+          ensure
+            fos.close
+          end
+        ensure
+          root.close
+          alloc.close
+        end
+      end
+
+      def read_columns(path)
+        load_jars!
+        alloc = org.apache.arrow.memory.RootAllocator.new(java.lang.Long::MAX_VALUE)
+        fis = java.io.FileInputStream.new(path)
+        columns = nil
+        begin
+          reader = org.apache.arrow.vector.ipc.ArrowFileReader.new(fis.getChannel, alloc)
+          begin
+            loop do
+              break unless reader.loadNextBatch
+
+              root = reader.getVectorSchemaRoot
+              columns ||= root.getSchema.getFields.map { |f| f.getName }.each_with_object({}) { |n, h| h[n] = [] }
+              n = root.getRowCount
+              root.getFieldVectors.each do |vec|
+                name = vec.getName
+                n.times { |i| columns[name] << java_value(vec, i) }
+              end
+            end
+          ensure
+            reader.close
+          end
+        ensure
+          fis.close
+          alloc.close
+        end
+        raise ArgumentError, "no record batches in #{path}" if columns.nil?
+
+        columns
+      end
+
+      def load_jars!
+        return if defined?(@jars_loaded) && @jars_loaded
+
+        unless RUBY_ENGINE == 'jruby'
+          raise LoadError, 'Galaaz::ArrowIpc::JavaArrowBackend requires JRuby'
+        end
+
+        loaded = load_jars_from_dir(ENV['GALAAZ_ARROW_JARS'])
+        loaded ||= load_jars_from_dir(File.expand_path('~/arrow_jars'))
+        loaded ||= load_jars_via_jar_dependencies
+
+        unless loaded
+          raise LoadError,
+                'Apache Arrow Java JARs not found. Set GALAAZ_ARROW_JARS to a directory of ' \
+                "Arrow #{ARROW_JAVA_VERSION} JARs, place them in ~/arrow_jars, or install " \
+                'jar-dependencies so require_jar can resolve them.'
+        end
+
+        @jars_loaded = true
+      end
+      private_class_method :load_jars!
+
+      def load_jars_from_dir(dir)
+        return false if dir.nil? || dir.empty?
+        return false unless File.directory?(dir)
+
+        jars = Dir[File.join(dir, '*.jar')]
+        return false if jars.empty?
+
+        jars.each { |jar| require jar }
+        true
+      end
+      private_class_method :load_jars_from_dir
+
+      def load_jars_via_jar_dependencies
+        require 'jar-dependencies'
+        require_jar 'org.apache.arrow', 'arrow-format', ARROW_JAVA_VERSION
+        require_jar 'org.apache.arrow', 'arrow-memory-core', ARROW_JAVA_VERSION
+        require_jar 'org.apache.arrow', 'arrow-memory-netty', ARROW_JAVA_VERSION
+        require_jar 'org.apache.arrow', 'arrow-memory-unsafe', ARROW_JAVA_VERSION
+        require_jar 'org.apache.arrow', 'arrow-vector', ARROW_JAVA_VERSION
+        true
+      rescue LoadError
+        false
+      end
+      private_class_method :load_jars_via_jar_dependencies
+
+      def arrow_type(kind)
+        case kind
+        when :float64
+          org.apache.arrow.vector.types.pojo.ArrowType::FloatingPoint.new(
+            org.apache.arrow.vector.types.FloatingPointPrecision::DOUBLE
+          )
+        when :int32
+          org.apache.arrow.vector.types.pojo.ArrowType::Int.new(32, true)
+        when :utf8
+          org.apache.arrow.vector.types.pojo.ArrowType::Utf8.new
+        else
+          raise ArgumentError, "unsupported kind #{kind.inspect}"
+        end
+      end
+      private_class_method :arrow_type
+
+      def fill_vector(vector, kind, values, n)
+        vector.allocateNew
+        case kind
+        when :float64
+          values.each_with_index do |v, i|
+            if v.nil?
+              vector.setNull(i)
+            else
+              vector.setSafe(i, v.to_f)
+            end
+          end
+        when :int32
+          values.each_with_index do |v, i|
+            if v.nil?
+              vector.setNull(i)
+            else
+              vector.setSafe(i, Integer(v))
+            end
+          end
+        when :utf8
+          values.each_with_index do |v, i|
+            if v.nil?
+              vector.setNull(i)
+            else
+              bytes = v.to_s.to_java_bytes
+              vector.setSafe(i, bytes)
+            end
+          end
+        end
+        vector.setValueCount(n)
+      end
+      private_class_method :fill_vector
+
+      def infer_kind(values)
+        sample = values.find { |v| !v.nil? }
+        return :float64 if sample.nil?
+
+        case sample
+        when Float
+          :float64
+        when Integer
+          values.any? { |v| v.is_a?(Float) } ? :float64 : :int32
+        when String, Symbol
+          :utf8
+        when TrueClass, FalseClass
+          raise ArgumentError, 'boolean columns are not supported in B1 (use int32/float64/utf8)'
+        else
+          if sample.is_a?(Numeric)
+            sample.is_a?(Integer) ? :int32 : :float64
+          else
+            :utf8
+          end
+        end
+      end
+      private_class_method :infer_kind
+
+      def java_value(vector, i)
+        return nil if vector.isNull(i)
+
+        minor = vector.getMinorType.toString
+        case minor
+        when 'FLOAT8', 'FLOAT4'
+          vector.get(i).to_f
+        when 'INT', 'SMALLINT', 'TINYINT', 'UINT1', 'UINT2', 'UINT4'
+          vector.get(i).to_i
+        when 'BIGINT', 'UINT8'
+          vector.get(i).to_i
+        when 'VARCHAR', 'VARBINARY'
+          bytes = vector.get(i)
+          bytes.nil? ? nil : String.from_java_bytes(bytes)
+        else
+          obj = vector.respond_to?(:getObject) ? vector.getObject(i) : vector.get(i)
+          return obj if obj.nil?
+          return obj.to_s if obj.is_a?(String) || obj.java_kind_of?(java.lang.CharSequence)
+          return obj.to_f if obj.is_a?(Float) || obj.java_kind_of?(java.lang.Double) || obj.java_kind_of?(java.lang.Float)
+          return obj.to_i if obj.is_a?(Integer) || obj.java_kind_of?(java.lang.Number)
+
+          obj.to_s
+        end
+      end
+      private_class_method :java_value
+    end
+  end
+end

data/lib/galaaz/arrow_ipc/red_arrow_backend.rb

@@ -0,0 +1,126 @@
+# frozen_string_literal: true
+
+require 'rbconfig'
+
+##########################################################################################
+# CRuby Arrow IPC writer via red-arrow (gem "arrow").
+##########################################################################################
+
+module Galaaz
+  module ArrowIpc
+    module RedArrowBackend
+      module_function
+
+      def available?
+        load!
+        true
+      rescue LoadError, StandardError
+        false
+      end
+
+      def write_columns(columns_hash, path)
+        load!
+        lengths = columns_hash.values.map(&:length).uniq
+        raise ArgumentError, "all columns must have the same length (got #{lengths.inspect})" if lengths.size != 1
+
+        table_hash = {}
+        columns_hash.each do |name, values|
+          table_hash[name] = build_array(name, values)
+        end
+        table = ::Arrow::Table.new(table_hash)
+        table.save(path)
+      end
+
+      def read_columns(path)
+        load!
+        table = ::Arrow::Table.load(path)
+        table.schema.fields.each_with_object({}) do |field, out|
+          col = table[field.name]
+          values = col.respond_to?(:to_a) ? col.to_a : col.data.to_a
+          out[field.name] = values
+        end
+      end
+
+      def load!
+        return if defined?(@loaded) && @loaded
+
+        begin
+          require 'arrow'
+        rescue LoadError
+          prepend_user_red_arrow_load_path!
+          begin
+            require 'arrow'
+          rescue LoadError => e
+            raise LoadError,
+                  "Galaaz::ArrowIpc on CRuby requires the red-arrow gem " \
+                  "(gem install red-arrow, matching pkg-config arrow-glib). #{e.message}"
+          end
+        end
+        @loaded = true
+      end
+      private_class_method :load!
+
+      # bundle exec only exposes Gemfile gems; red-arrow is optional and often
+      # user-installed. Put its lib + native extension on $LOAD_PATH.
+      def prepend_user_red_arrow_load_path!
+        names = '{red-arrow,gobject-introspection,glib2,gio2,native-package-installer,pkg-config,extpp}'
+        roots = []
+        roots << Gem.user_dir if Gem.respond_to?(:user_dir)
+        rubylibdir = RbConfig::CONFIG['rubylibdir']
+        if rubylibdir
+          roots << File.join(File.dirname(rubylibdir), 'gems', RbConfig::CONFIG['ruby_version'])
+        end
+        roots.concat(Array(Gem.path))
+        roots.compact.uniq.each do |root|
+          next unless File.directory?(root)
+
+          Dir.glob(File.join(root, 'gems', "#{names}-*", 'lib')).sort.each do |lib|
+            $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+          end
+          Dir.glob(File.join(root, 'extensions', '**', "#{names}-*")).select { |p| File.directory?(p) }.sort.each do |ext|
+            $LOAD_PATH.unshift(ext) unless $LOAD_PATH.include?(ext)
+          end
+        end
+      end
+      private_class_method :prepend_user_red_arrow_load_path!
+
+      def build_array(name, values)
+        kind = infer_kind(values)
+        case kind
+        when :float64
+          ::Arrow::DoubleArray.new(values.map { |v| v.nil? ? nil : v.to_f })
+        when :int32
+          ::Arrow::Int32Array.new(values.map { |v| v.nil? ? nil : Integer(v) })
+        when :utf8
+          ::Arrow::StringArray.new(values.map { |v| v.nil? ? nil : v.to_s })
+        else
+          raise ArgumentError, "unsupported column type for #{name.inspect}"
+        end
+      end
+      private_class_method :build_array
+
+      def infer_kind(values)
+        sample = values.find { |v| !v.nil? }
+        return :float64 if sample.nil?
+
+        case sample
+        when Float
+          :float64
+        when Integer
+          values.any? { |v| v.is_a?(Float) } ? :float64 : :int32
+        when String, Symbol
+          :utf8
+        when TrueClass, FalseClass
+          raise ArgumentError, 'boolean columns are not supported in B1 (use int32/float64/utf8)'
+        else
+          if sample.respond_to?(:to_f) && sample.is_a?(Numeric)
+            sample.is_a?(Integer) ? :int32 : :float64
+          else
+            :utf8
+          end
+        end
+      end
+      private_class_method :infer_kind
+    end
+  end
+end