galaaz 2.1.0 → 2.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0fc6f307a98f073e4ad49a39f5b4fcfddbd948c9054722caea814c16d6be62fa
-  data.tar.gz: d899ca8013e16d158cb1c035a3e8d1393b3d377782fad502b09d1e81d9eb868a
+  metadata.gz: 671c2fe1fb15dbd12a7f491e52c318d01ebe9625e78d9acdaaa3dffbe3efbd3d
+  data.tar.gz: 2d4d63f915f0b8a09cff53df98802f7bcaff0298f9247ef38b583b2a0e902730
 SHA512:
-  metadata.gz: a73ce81b2b86550af39562adc7d1566746c5239d0b74688c8d1d1cef52bb5371ab5449cb3dbe3d6b14b2608e1c33e3289937f745a9749c7c425f026a4d8f8659
-  data.tar.gz: 3d84d7e16f678ed8b13920585099df2ae641293ae74f65e7920d7bd7ee42c971deaa9c7dd1985343fb2eadf17be582d3922eecf3d20d009bfc45180f762b7b26
+  metadata.gz: 1dc8f434731a05f495909d5f8bc70949f7ec7ae8034fbe8dd6a026f6b9592d426d573dffc0f13326ddfb7c830112bd6922fcdb8ca717e091d8722e219ea43de6
+  data.tar.gz: d594128365977ac52a3a2d15f6e3dd5f1b97ee312c0c6232883f6fb8838fb7c1440f2272c3302e53853026befd7e625f6e4124a8d3ddd32959f7bf665cb8cda7

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## 2.1.1
+
+### Added
+
+- CLI on `bin/galaaz`: `setup`, `blogs init`, `doctor`, `add` (knit / arrow / tex / bio /
+  examples / ledger / demo). Legacy unknown args still forward to rake.
+- Omarchy dogfood helpers under `script/omarchy/` (install/remove scripts + menu overlay).
+- `galaaz doctor` reports both CRuby and JRuby when available (mise-aware).
+- Package lists `r_requires/knit.txt`, `knit-extras.txt`, `arrow.txt` for add-on installs.
+- Cold-install CRuby path uses `galaaz setup` + `galaaz blogs init`.
+
+### Notes
+
+- Omarchy / stranger install remains `gem install galaaz` then `galaaz setup` (no
+  `bundle install` for core use). See `Documentation/PLAN_OMARCHY_INTEGRATION.md`.
+
 ## 2.1.0
 
 Galaaz **2.1** keeps the **galaaz2_0** integration line as the main development branch and marks
@@ -10,14 +26,16 @@ Galaaz **2.1** keeps the **galaaz2_0** integration line as the main development
 - First-class **CRuby** NewBridge path alongside JRuby (engine-aware launchers, cold-install /
   CI coverage for both).
 - Docs: treat JRuby and CRuby as equal supported runtimes for the bridge.
-- [Documentation/ROADMAP_ARROW_RUBY_R.md](Documentation/ROADMAP_ARROW_RUBY_R.md) — roadmap for
-  Ruby↔R Arrow IPC / mmap handoff (stage B) then shared-memory bus (stage C).
+- [Documentation/ROADMAP_ARROW_RUBY_R.md](Documentation/ROADMAP_ARROW_RUBY_R.md) — stages A (copy),
+  B1/B2 (IPC/mmap file; shipped), C (shared-memory bus; future).
+- Stage **B1/B2** APIs: `Galaaz::ArrowIpc`, `R::Arrow.open_ipc`, `R::Arrow.write_ipc` (CRuby
+  red-arrow + system Arrow GLib; JRuby Arrow Java + `JAVA_OPTS` nio opens).
 
 ### Notes
 
 - Git branch **`galaaz2_0`** remains the integration branch name; the gem version is **2.1.0**.
-- Apache Arrow **zero-copy shared RAM** is still future work (see the roadmap); today’s helpers
-  build R-side tables after a handoff, then use proxy / Remote Control calls.
+- Apache Arrow **zero-copy shared RAM** (Stage C) is still future work. Stage A copies into R;
+  Stage B uses an IPC file and only the path crosses NewBridge. See the roadmap.
 
 ## 2.0.0
 

data/README.md

@@ -59,9 +59,11 @@ CRuby when you prefer MRI. R remains the same **GNU R** you use interactively—
 compiled extensions and Bioconductor. Earlier GraalVM / TruffleRuby / FastR experiments
 are no longer the focus.
 
-The bridge handles **communication and typing** between the two worlds; large tables can
-also flow through **Apache Arrow** on the R side when you use the optional helpers described
-later in this manual.
+The bridge handles **communication and typing** between the two worlds. Large tables can use
+**Apache Arrow** in two shipped modes (described later): **Stage A** copies Ruby batches into an
+R-side Arrow table (`R::Arrow.from_ruby_batches`); **Stage B** writes an Arrow IPC file and only
+the **path** crosses NewBridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). Shared-heap
+zero-copy is **Stage C** and is not shipped.
 
 ## R-on-Rails: the one-person app for R scientists
 
@@ -340,7 +342,7 @@ The supported install is **`gem install` + compile the gatekeeper**. You do not
    make -C "${gem_dir}/ext/new_bridge" all
    ```
 
-5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on Java 9+, pass `-J--add-opens=java.base/java.nio=ALL-UNNAMED` to JRuby (from a checkout, `bin/galaaz-jruby` does this; on CRuby this flag is not needed).
+5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on **JRuby** (Java 9+), the child JVM needs `--add-opens=java.base/java.nio=ALL-UNNAMED` via **`JAVA_OPTS`** (from a checkout, `bin/galaaz-jruby` and `mise.toml` set this; a leading `jruby -J... -S bundle exec` does **not** pass `-J` to rspec). On **CRuby**, install Apache Arrow GLib (`libarrow-glib-dev` from the [Apache Arrow APT](https://arrow.apache.org/install/) repo) and `gem install red-arrow` matching `pkg-config --modversion arrow-glib`. Do not install the unrelated Rubygems package named `arrow`.
 
 For **gKnit**, **knitr**, **rmarkdown**, and LaTeX (PDF output), install the corresponding R packages, **Pandoc**, and a TeX distribution if you need PDF; the repository includes helpers such as **`bin/install-tinytex`** where appropriate.
 
@@ -442,13 +444,15 @@ A practical pattern is:
 
 1. Use threads (or a connection pool) to read from **multiple databases or shards** in parallel.
 2. Merge the rows in Ruby under a `Mutex` if you collect into one structure.
-3. Hand the merged table to R **once** (for example with `R::Arrow.from_ruby_batches` and dplyr,
-   or by building a data frame) so heavy statistics run in R with fewer bridge round-trips.
+3. Hand the merged table to R **once**: **`R::Arrow.from_ruby_batches`** (Stage A: copy into R) or
+   **`Galaaz::ArrowIpc.write` / `write_batches`** then **`R::Arrow.open_ipc`** (Stage B: IPC file;
+   only the path crosses the bridge). Then run dplyr in R.
 
 A runnable sketch lives in
 `examples/multithread_shards_to_r/shards_to_r.rb` (simulated shard queries; swap in your DB
 driver). For concurrency tests on the bridge itself, see `specs/bridge_concurrent_spec.rb` and
-`specs/arrow_from_ruby_batches_spec.rb`.
+`specs/arrow_from_ruby_batches_spec.rb`. Stage B IPC tests: `specs/arrow_ipc_handoff_spec.rb`,
+`specs/arrow_ipc_export_spec.rb`.
 
 ## Long-running R calls and a completion block
 
@@ -4025,35 +4029,77 @@ ans = flights[:all, E.list(R[:arr_delay], R[:dep_delay])]
 # Apache Arrow
 
 [Apache Arrow](https://arrow.apache.org/) is a **columnar** in-memory format used heavily in R
-and Python for analytics. In Galaaz, **Ruby does not hold an Arrow C++ table itself**; instead you
-build ordinary Ruby structures (arrays of row hashes), and **`R::Arrow.from_ruby_batches`** creates
-a real **Arrow `Table` inside GNU R**. From there you use R’s **`arrow`** and **`dplyr`** packages
-as usual: **`group_by`** on the Arrow table, **`summarise`** for aggregates, then **`collect()`** to
-materialize a tibble when you need in-memory R rows.
-
-That pattern matches production use: **JRuby threads** (or sequential code) assemble many rows in
-Ruby; you pay **one** bridge-heavy handoff to R; **dplyr** runs vectorised work on the Arrow table
-in R.
+and Python for analytics. GNU R still runs in a **separate process**. Ruby does **not** hold a
+shared Arrow C++ table with R. Stages:
+
+1. **Stage A (copy over the bridge):** Ruby row hashes → **`R::Arrow.from_ruby_batches`** builds
+   an Arrow `Table` **inside GNU R**. You get a **proxy**.
+2. **Stage B1 (Ruby → R IPC file):** **`Galaaz::ArrowIpc.write`** / **`write_batches`** writes an
+   Arrow IPC file (prefer **`/dev/shm`**); **`R::Arrow.open_ipc(path)`** opens it in R. Only the
+   **path** crosses NewBridge. This is **mmap/IPC file handoff**, not a shared heap.
+3. **Stage B2 (R → Ruby IPC file):** **`R::Arrow.write_ipc(obj)`** writes uncompressed IPC; Ruby
+   reads with **`Galaaz::ArrowIpc.read`** (column hash) or **`read_batches`** (row hashes). Call
+   **`Galaaz::ArrowIpc.release(path)`** when finished.
+4. **Stage C (not shipped):** named shared-memory bus. Do not claim 0 ms shared RAM until then.
+   See **`Documentation/ROADMAP_ARROW_RUBY_R.md`**.
+
+After ingest, use R’s **`arrow`** / **`dplyr`** on the proxy (`group_by`, `summarise`, `collect`)
+and unbox only KPIs you need in Ruby.
+
+**Optional Ruby backends for Stage B**
+
+* **CRuby:** Apache **red-arrow** — `gem install red-arrow` pinned to the same major as
+  `pkg-config --modversion arrow-glib`, plus system **Arrow GLib** (`libarrow-glib-dev` from the
+  [Apache Arrow APT](https://arrow.apache.org/install/) repo). Do **not** install the unrelated
+  legacy Rubygems package named `arrow`. `bundle exec` still sees a user-installed `red-arrow`
+  via Galaaz’s load-path helper.
+* **JRuby:** Apache Arrow **Java** JARs — **`GALAAZ_ARROW_JARS`**, `~/arrow_jars`, or
+  `jar-dependencies`. Export **`JAVA_OPTS=--add-opens=java.base/java.nio=ALL-UNNAMED`** on the
+  **child** JVM (`bin/galaaz-jruby`, `mise.toml`). `jruby -J... -S bundle exec rspec` does **not**
+  pass `-J` to rspec.
+
+**R packages:** **`arrow`** and **`dplyr`**. B2 writes IPC with **`compression: 'uncompressed'`**
+so JRuby Arrow Java can read without extra compression JARs.
+
+**Tests:** `specs/arrow_from_ruby_batches_spec.rb` (A);
+`specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb` (B, sync);
+`new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb` (B, async).
+
+## `R::Arrow` and `Galaaz::ArrowIpc`
+
+* **`R::Arrow.from_ruby_batches`** — Stage A ingest.
+* **`R::Arrow.open_ipc(path)`** — Stage B1: IPC file → R Table proxy.
+* **`R::Arrow.write_ipc(obj, path = nil)`** — Stage B2: R Table/tibble → IPC path (scratch if omitted).
+* **`Galaaz::ArrowIpc.write` / `write_batches` / `read` / `read_batches` / `allocate_path` / `release` / `available?`**
+* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
+* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
+  dataset IO on paths visible to R.
 
-**Prerequisites:** install R packages **`arrow`** and **`dplyr`**. Run scripts with
-**`bin/galaaz-jruby`** (or the same JVM flags as in **`docs/testing.md`**) so the Arrow JNI stack is
-available.
+## Example: Stage B round-trip (IPC file)
 
-## Other `R::Arrow` helpers
+Requires `Galaaz::ArrowIpc.available?` (red-arrow or Arrow JARs) and R **`arrow`**. Not knitted
+below so a machine without the optional backend still builds this manual.
 
-The Ruby module **`R::Arrow`** (see `lib/R_interface/r_arrow.rb`) also includes:
+```ruby
+path = Galaaz::ArrowIpc.write(id: [1, 2, 3], grp: %w[a a b], value: [1.0, 2.0, 3.5])
+tbl  = R::Arrow.open_ipc(path)
+Galaaz::ArrowIpc.release(path)
 
-* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
-* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
-  dataset IO on paths visible to R.
+summed   = R.dplyr___summarise(R.dplyr___group_by(tbl, :grp), total: E.sum(:value))
+out_path = R::Arrow.write_ipc(summed)
+rows     = Galaaz::ArrowIpc.read_batches(out_path)
+Galaaz::ArrowIpc.release(out_path)
+# rows => [{:grp=>"a", :total=>3.0}, {:grp=>"b", :total=>3.5}]  (illustrative)
+```
 
 ## Example: many Ruby rows → Arrow in R → grouped statistics
 
 The repository test **`slow-specs/arrow_large_pipeline_spec.rb`** builds **200k rows** in parallel
-(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`**, then checks that
-**dplyr** group summaries match a Ruby reference calculation. The same logic appears below at a
-**smaller scale** so this manual can knit quickly; increase `thread_count` and `rows_per_thread`
-when experimenting locally.
+(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`** (Stage A), then
+checks that **dplyr** group summaries match a Ruby reference calculation. The same logic appears
+below at a **smaller scale** so this manual can knit quickly; increase `thread_count` and
+`rows_per_thread` when experimenting locally. For the same ingest **without** copying every cell
+over NewBridge, use Stage B (`write_batches` + `open_ipc`) instead of `from_ruby_batches`.
 
 
 ``` ruby
@@ -4119,10 +4165,10 @@ end
 ```
 
 **What to notice:** (1) Ruby only sees **`Hash`** rows and Ruby **`Thread`** objects; (2) a single
-**`from_ruby_batches`** call creates the Arrow table in R; (3) **`dplyr___group_by`** /
+**`from_ruby_batches`** call **copies** those columns into an Arrow table in R; (3) **`dplyr___group_by`** /
 **`dplyr___summarise`** / **`dplyr___collect`** mirror **`dplyr::group_by`** /
 **`dplyr::summarise`** / **`dplyr::collect`** on an Arrow-backed table. For a lighter test, see
-**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size benchmark, run
+**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size Stage A benchmark, run
 **`bin/run_slow_rspec slow-specs/arrow_large_pipeline_spec.rb`**.
 
 # Bioconductor and DESeq2
@@ -4260,8 +4306,9 @@ Practical tips:
   glue.
 * **Reuse one process**: running many short scripts cold-starts Ruby, the JVM, and R each time;
   a long-lived process or repeated calls in one run amortize setup (see benchmarks below).
-* **Batch data**: merge shards in Ruby, then call **`R::Arrow.from_ruby_batches`** (or build one
-  data frame) instead of millions of tiny R calls.
+* **Batch data**: merge shards in Ruby, then **`R::Arrow.from_ruby_batches`** (Stage A) or
+  **`Galaaz::ArrowIpc`** + **`R::Arrow.open_ipc`** (Stage B) instead of millions of tiny R calls.
+  When Ruby needs a bulky result table back, **`R::Arrow.write_ipc`** + **`Galaaz::ArrowIpc.read_batches`**.
 
 For measured discussion (including DESeq2-style workloads and warm comparisons), see
 **`docs/performance.md`** and **`docs/deseq2_airway_benchmark.md`** in the Galaaz repository.

data/bin/galaaz

@@ -1,5 +1,13 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
-Dir.chdir(File.dirname(File.expand_path('..', __FILE__)))
+# Galaaz CLI: setup / blogs / doctor / add.
+# Unknown commands still forward to rake (legacy example runners).
 
-system ("rake #{ARGV[0]}")
+root = File.expand_path('..', __dir__)
+lib = File.join(root, 'lib')
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require 'galaaz/cli'
+
+exit Galaaz::CLI.run(ARGV)

data/bin/galaaz_jruby_env.inc.sh

@@ -1,6 +1,22 @@
-# Galaaz required JVM flags for JRuby (Apache Arrow on Java 9+).
+# Galaaz required JVM flags for JRuby (Apache Arrow memory on Java 9+).
 # Source from bin/*.sh:  source "$ROOT/bin/galaaz_jruby_env.inc.sh"
 #
-# Source of truth in Ruby/Rake: lib/galaaz_jruby.rb (GalaazJRuby::REQUIRED_JRUBY_J_ARGS).
+# Source of truth in Ruby/Rake: lib/galaaz_jruby.rb (GalaazJRuby::JAVA_NIO_ADD_OPENS).
 # Optional extra -J flags: export GALAAZ_JRUBY_OPTS="-J-Xmx4g" (space-separated).
-GALAAZ_REQUIRED_JRUBY_J_ARGS='-J--add-opens=java.base/java.nio=ALL-UNNAMED'
+#
+# JAVA_OPTS is exported so *child* JRuby processes (bundle exec rspec, mise exec)
+# get the same opens flag. -J args on the outer command are not inherited by
+# bundle exec's second JVM.
+GALAAZ_JAVA_NIO_ADD_OPENS='--add-opens=java.base/java.nio=ALL-UNNAMED'
+GALAAZ_REQUIRED_JRUBY_J_ARGS="-J${GALAAZ_JAVA_NIO_ADD_OPENS}"
+
+case " ${JAVA_OPTS:-} " in
+  *" ${GALAAZ_JAVA_NIO_ADD_OPENS} "*) ;;
+  *)
+    if [ -n "${JAVA_OPTS:-}" ]; then
+      export JAVA_OPTS="${GALAAZ_JAVA_NIO_ADD_OPENS} ${JAVA_OPTS}"
+    else
+      export JAVA_OPTS="${GALAAZ_JAVA_NIO_ADD_OPENS}"
+    fi
+    ;;
+esac

data/bin/galaaz_ruby_env.inc.sh

@@ -7,6 +7,7 @@
 #
 # JRuby and CRuby are both first-class. Override with GALAAZ_RUBY=jruby, GALAAZ_RUBY=ruby,
 # or a full path. Extra JRuby flags: GALAAZ_JRUBY_OPTS="-J-Xmx4g" (ignored on CRuby).
+# CRuby red-arrow uses system Arrow GLib (Apache Arrow APT / libarrow-glib-dev).
 
 GALAAZ_RUBY_BIN="${GALAAZ_RUBY:-ruby}"
 
@@ -29,6 +30,7 @@ if [[ "$_galaaz_is_jruby" -eq 1 ]]; then
   # shellcheck source=galaaz_jruby_env.inc.sh
   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/galaaz_jruby_env.inc.sh"
   GALAAZ_RUBY_J_ARGS="$GALAAZ_REQUIRED_JRUBY_J_ARGS ${GALAAZ_JRUBY_OPTS:-}"
+  # JAVA_OPTS is exported by galaaz_jruby_env.inc.sh for bundle-exec child JVMs
 else
   GALAAZ_REQUIRED_JRUBY_J_ARGS=''
   GALAAZ_RUBY_J_ARGS=''

data/blogs/R-on-Rails-Planning-Document.md

@@ -68,9 +68,9 @@ Rails never went away—it matured. Several factors make Rails compelling again:
 
 1. **Rails (Ruby)**: Handles web requests, authentication, authorization, database operations, background jobs, email, APIs, caching, sessions, and orchestration
 2. **R**: Handles statistical modeling, machine learning, visualization, and data analysis
-3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—using Apache Arrow for zero-copy data transfer at scale
+3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—proxies for analytics in R, Apache Arrow for bulky tables (Stage A copy, Stage B IPC/mmap file; shared-heap zero-copy is Stage C and not shipped)
 
-**The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications. Move data between languages at memory speed, not serialization speed.
+**The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications. For large tables, prefer Arrow **IPC files** (path across the bridge, bytes on disk or `/dev/shm`) over CSV/JSON serialization. Do **not** claim Ruby and R share one physical Arrow heap until Stage C ships.
 
 ---
 
@@ -86,25 +86,24 @@ Earlier versions of Galaaz explored GraalVM's polyglot capabilities with FastR (
 ├─────────────────────────────────────────────────────────────────┤
 │                                                                 │
 │  ┌──────────────┐         ┌──────────────┐         ┌──────────┐ │
-│  │   JRuby      │ ←─────→ │ Galaaz Bridge│ ←─────→ │ GNU R    │ │
-│  │  (JVM)       │  (Java) │  (JNI/R API) │  (C/R)  │  Process │ │
-│  └──────────────┘         └──────────────┘         └──────────┘ │
-│        │                                                    │   │
+│  │ JRuby/CRuby  │ ←─────→ │ Galaaz Bridge│ ←─────→ │ GNU R    │ │
+│  │              │  path + │  (NewBridge) │  C/R    │  Process │ │
+│  └──────────────┘  cmds   └──────────────┘         └──────────┘ │
 │        │                                                    │   │
 │        ▼                                                    ▼   │
 │  ┌──────────────┐                                  ┌──────────┐│
 │  │ Rails App    │                                  │ Arrow    ││
-│  │ Multi-thread │                                  │ Shared   ││
-│  │ ActiveRecord │                                  │ Memory   ││
+│  │ (threads /   │     bulky tables: IPC file       │ Table in ││
+│  │  processes)  │ ──── mmap / /dev/shm (Stage B) ──│ R (proxy)││
 │  └──────────────┘                                  └──────────┘│
 │                                                                 │
 └─────────────────────────────────────────────────────────────────┘
 ```
 
 **How it works:**
-1. **JRuby** runs on the JVM with true multi-threading, running Rails and application code
-2. **Galaaz Bridge** communicates with a GNU R process via R's C API (or alternative mechanisms)
-3. **Apache Arrow** (optional) provides efficient, shared-memory data transfer for large datasets
+1. **JRuby or CRuby** runs Rails and application code (JRuby for true JVM threads; CRuby for MRI). Both talk the same NewBridge protocol.
+2. **Galaaz Bridge** communicates with a GNU R process (NewBridge / gatekeeper)—commands and small values, not bulk bytes when you use Stage B.
+3. **Apache Arrow** (optional): **Stage A** copies Ruby batches into an R-side Arrow Table (`R::Arrow.from_ruby_batches`). **Stage B** writes an Arrow IPC file; only the **path** crosses the bridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). **Stage C** (shared-heap zero-copy) is not shipped—see `Documentation/ROADMAP_ARROW_RUBY_R.md`.
 4. **GNU R** is the actual R interpreter—full compatibility with all R packages (ggplot2, dplyr, Bioconductor, etc.)
 
 **Why this is better than FastR/GraalVM:**
@@ -448,13 +447,22 @@ end
 
 ---
 
-### Section 6: Apache Arrow - Zero-Copy Data at Scale
+### Section 6: Apache Arrow — bulky tables without CSV/JSON
+
+**Key Message**: Traditional bridges often serialize through text (CSV, JSON). Galaaz uses Apache Arrow in **two shipped modes**. Neither is a shared Ruby+R heap. Maintainer detail: `Documentation/ROADMAP_ARROW_RUBY_R.md`. User manual: `# Apache Arrow` in `README.md` / `blogs/manual/manual.md`.
 
-**Key Message**: Traditional bridges serialize data (JSON, CSV) to pass between languages. Galaaz uses Apache Arrow for zero-copy, memory-mapped data transfer—enabling huge datasets to move from Ruby to R instantly.
+| Stage | API | What actually happens |
+|-------|-----|------------------------|
+| **A** (shipped) | `R::Arrow.from_ruby_batches` / `table_from` | Columns are **copied** into GNU R; Ruby holds a **proxy**. dplyr then runs in R. |
+| **B1** (shipped) | `Galaaz::ArrowIpc.write` → `R::Arrow.open_ipc(path)` | Ruby writes an Arrow **IPC file** (prefer `/dev/shm`); NewBridge carries only the **path**. R memory-maps / reads the file into an Arrow Table. |
+| **B2** (shipped) | `R::Arrow.write_ipc` → `Galaaz::ArrowIpc.read` / `read_batches` | R writes IPC (uncompressed so JRuby Arrow Java can read); Ruby reads columns or row hashes for DB/API. |
+| **C** (future) | not shipped | Named shared segment; do **not** claim 0 ms shared RAM until this exists and is measured. |
+
+**Writers/readers (Stage B):** CRuby uses **red-arrow** (`gem install red-arrow` matching `pkg-config --modversion arrow-glib`, plus Apache Arrow APT / `libarrow-glib-dev`). JRuby uses **Apache Arrow Java** JARs (`GALAAZ_ARROW_JARS` or `~/arrow_jars`) and `JAVA_OPTS` `--add-opens=java.base/java.nio=ALL-UNNAMED` on the **child** JVM (`bin/galaaz-jruby` / `mise.toml`). Do not install the unrelated Rubygems gem named `arrow`.
 
 **The Problem: Data Transfer Overhead**
 
-In traditional polyglot systems (rpy2, reticulate), passing data between languages requires serialization:
+In traditional polyglot systems (rpy2, reticulate), passing data between languages often means a text round-trip:
 
 ```python
 # Python → R via JSON/CSV serialization (SLOW)
@@ -470,36 +478,31 @@ ro.globalenv['r_df'] = pandas2ri.py2rpy(df)
 This approach:
 - Copies all data through a text format
 - Loses type information (factors become strings, dates become strings)
-- Consumes 2-3x memory during transfer
-- Becomes a bottleneck at ~100K+ rows
+- Consumes extra memory during transfer
+- Becomes a bottleneck at large row counts
 
-**The Galaaz Solution: Apache Arrow**
+**Stage A: copy into R, then Remote Control**
 
-Apache Arrow is a columnar, in-memory format designed for zero-copy data sharing across languages. With Galaaz:
+Assemble row hashes in Ruby (threads OK on JRuby), then one ingest. Analytics stay on the R proxy; unbox KPIs only.
 
 ```ruby
 require 'galaaz'
 
-# 1. Build data in Ruby—using parallel threads for multi-source ingestion
 batches = []
 mutex = Mutex.new
 threads = []
 
-# Simulate parallel data ingestion from multiple sources
 [0, 1, 2, 3].each do |tid|
   threads << Thread.new do
-    # Each thread queries its own database/shard/API
-    local_data = fetch_from_source(tid) # Returns array of hashes
+    local_data = fetch_from_source(tid) # array of hashes
     mutex.synchronize { batches << local_data }
   end
 end
 threads.each(&:join)
 
-# 2. Convert to Arrow Table—zero-copy transfer to R
-# This is the key: data moves to R without serialization!
+# Copy into an Arrow Table *inside GNU R* (not shared heap with Ruby)
 table = R::Arrow.from_ruby_batches(batches)
 
-# 3. Use R's dplyr directly on the Arrow table
 R.install_and_loads('dplyr', 'arrow')
 
 grouped = R.dplyr___group_by(table, :region)
@@ -510,57 +513,59 @@ summarised = R.dplyr___summarise(
   total: E.sum(:value)
 )
 results = R.dplyr___collect(summarised)
-
 puts results
-#> # A tibble: 4 × 4
-#>   region count avg_value total
-#>   <chr>  <int>     <dbl> <dbl>
-#> 1 North  25000      5.5  137500
-#> 2 South  25000      5.5  137500
-#> 3 East   25000      5.5  137500
-#> 4 West   25000      5.5  137500
 ```
 
-**Why This Matters:**
+**Stage B: IPC file; path on the bridge**
+
+Use this when the table is large enough that copying every cell over NewBridge/MsgPack is the wrong tax. Scratch files prefer `/dev/shm`.
+
+```ruby
+require 'galaaz'
+
+# B1: Ruby → file → R
+path = Galaaz::ArrowIpc.write_batches(batches)  # or .write(col => array, ...)
+tbl  = R::Arrow.open_ipc(path)
+Galaaz::ArrowIpc.release(path)  # unlink after R has opened (default B1 lifetime)
+
+grouped = R.dplyr___group_by(tbl, :region)
+summarised = R.dplyr___summarise(grouped, total: E.sum(:value))
+
+# B2: R → file → Ruby (e.g. rows for ActiveRecord / JSON)
+out_path = R::Arrow.write_ipc(summarised)
+rows     = Galaaz::ArrowIpc.read_batches(out_path)
+Galaaz::ArrowIpc.release(out_path)
+
+render json: { analytics: rows }
+```
+
+**Why this matters (honest):**
 
-- **No serialization cost**: Data stays in Arrow's columnar format throughout
-- **Type preservation**: Factors, dates, timestamps remain intact
-- **Memory efficiency**: No intermediate copies; Ruby and R share the same memory
-- **Scale**: Tested with millions of rows (see slow-specs/arrow_large_pipeline_spec.rb with 200K+ rows)
-- **Lazy evaluation**: Arrow datasets can be filtered/aggregated before materializing
+- **Columnar Arrow**, not CSV, for the bulky payload
+- **Type-preserving** numeric/string columns on the B1/B2 contract (int32/int64, float64, utf8; nulls kept)
+- **Stage A** still copies; **Stage B** still materializes a file R (or Ruby) then reads—R may mmap the IPC file
+- **Not** “Ruby and R map the same live buffer” until Stage C
+- **Scale**: Stage A pipeline in `slow-specs/arrow_large_pipeline_spec.rb` (200K+ rows); Stage B specs in `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, and the matching `new_bridge_specs/arrow_ipc_*_async_spec.rb`
+- **Parquet/Feather/dataset** stay R-side file APIs (`R::Arrow.write_parquet`, `dataset`)
 
-**Real-World Example: Multi-DB Aggregation**:
+**Real-world sketch: shards in Ruby, analytics in R**
 
 ```ruby
-# Aggregate data from multiple PostgreSQL shards in Ruby,
-# then analyze in R with zero-copy transfer
+# Parallel ingest in Ruby, then one Arrow handoff (A or B), then dplyr in R.
 
 require 'galaaz'
 require 'active_record'
 
-# Connect to multiple database shards
-SHARDS = ['shard_1', 'shard_2', 'shard_3', 'shard_4'].map do |shard_name|
-  ActiveRecord::Base.establish_connection(
-    adapter: 'postgresql',
-    host: "#{shard_name}.db.internal",
-    database: 'analytics'
-  )
-end
-
-# Parallel data collection from all shards
 batches = []
 mutex = Mutex.new
 
 SHARDS.each do |shard|
   Thread.new do
-    # Query this shard
     records = shard.connection.select_all(<<-SQL).cast_values
       SELECT user_id, region, event_type, value, created_at
       FROM events
       WHERE created_at > NOW() - INTERVAL '7 days'
     SQL
-
-    # Convert to array of hashes for Arrow
     batch = records.map do |row|
       {
         user_id: row[0],
@@ -570,60 +575,37 @@ SHARDS.each do |shard|
         created_at: row[4]
       }
     end
-
     mutex.synchronize { batches << batch }
   end
 end
 
-# Zero-copy transfer to R as Arrow Table
-table = R::Arrow.from_ruby_batches(batches)
+R.install_and_loads('dplyr', 'arrow')
 
-# Now use R's full power on the complete dataset
-R.install_and_loads('dplyr', 'arrow', 'lubridate')
+# Prefer B1 when the merged table is large:
+path  = Galaaz::ArrowIpc.write_batches(batches)
+table = R::Arrow.open_ipc(path)
+Galaaz::ArrowIpc.release(path)
 
-# R's dplyr works directly on Arrow tables (lazy evaluation)
-analysis = table \
-  .dplyr___filter(E.created_at > (E.now() - E.days(7))) \
-  .dplyr___group_by(:region, :event_type) \
-  .dplyr___summarise(
+analysis = R.dplyr___collect(
+  R.dplyr___summarise(
+    R.dplyr___group_by(table, :region, :event_type),
     count: E.n(),
-    total_value: E.sum(:value),
-    avg_value: E.mean(:value),
-    unique_users: E.n_distinct(:user_id)
+    total_value: E.sum(:value)
   )
+)
 
-# Materialize results when ready
-results = R.dplyr___collect(analysis)
-
-# Convert back to Ruby objects for API response
-results_hash = results.to_ruby
-render json: { analytics: results_hash }
+out_path = R::Arrow.write_ipc(analysis)
+rows = Galaaz::ArrowIpc.read_batches(out_path)
+Galaaz::ArrowIpc.release(out_path)
+render json: { analytics: rows }
 ```
 
-**Parquet and Feather: Persistence Without Conversion**:
+**Parquet and Feather (R-visible paths):**
 
 ```ruby
-# Save Ruby data as Parquet (columnar, compressed)
-# Then read directly into R without parsing
-
-# 1. Build large dataset in Ruby
-data = (1..1_000_000).map do |i|
-  {
-    id: i,
-    category: ["A", "B", "C", "D"][i % 4],
-    value: rand * 100,
-    timestamp: Time.now - (i % 86400)
-  }
-end
-
-# 2. Convert to R data.frame, then write as Parquet
 df = R.data__frame(data)
 R::Arrow.write_parquet(df, '/data/events.parquet')
-
-# 3. Later, read directly into R as a dataset (lazy, memory-mapped)
 dataset = R::Arrow.dataset('/data/events.parquet')
-
-# Query without loading entire file
 summary = R.dplyr___collect(
   dataset \
      .dplyr___filter(R[:value] > 50) \
@@ -632,28 +614,13 @@ summary = R.dplyr___collect(
 )
 ```
 
-**Large-Scale Test Results** (from `slow-specs/arrow_large_pipeline_spec.rb`):
-
-```ruby
-# Test: 8 threads × 25,000 rows = 200,000 rows
-# With weighted aggregations in R
-thread_count = 8
-rows_per_thread = 25_000
-group_count = 10
-
-# Ruby parallel batch construction... 8 threads
-# Arrow table creation from batches → instant
-# R dplyr group_by + summarise → native speed
-# Results verified accurate against Ruby reference implementation
-```
-
-**Key Insight**: With Arrow, the boundary between Ruby and R disappears for data. You can build data pipelines in Ruby (with its superior concurrency and database libraries) and analyze in R (with its statistical ecosystem)—with **zero overhead** at the language boundary.
+**Key Insight**: Build pipelines in Ruby (concurrency, ActiveRecord); run statistics in R. Pay **one** bulky handoff (copy or IPC file), then **move commands** on proxies. That is the production story today—not a shared Arrow heap.
 
 ---
 
 ### Section 7: Parallelism and Process Orchestration
 
-**Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby's multi-threaded environment—now with Arrow for efficient data distribution.
+**Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby—Arrow for bulky ingest/export, proxies for analytics per worker.
 
 **The Architecture**:
 ```
@@ -892,7 +859,7 @@ Galaaz 2.0 represents the maturation of the Ruby-R bridge:
 - **JRuby + GNU R architecture**: Moved from experimental GraalVM/FastR to battle-tested JRuby and standard GNU R for full package compatibility and production stability
 - **New bridge architecture**: More robust, faster, better error handling
 - **Process management**: Built-in support for R process pools
-- **Apache Arrow integration**: Zero-copy data transfer for large datasets—build data pipelines in Ruby's multi-threaded environment, analyze in R with zero serialization overhead
+- **Apache Arrow**: Stage A copy (`from_ruby_batches`) and Stage B IPC/mmap (`Galaaz::ArrowIpc`, `open_ipc` / `write_ipc`). Shared-heap zero-copy is Stage C. Build pipelines in Ruby, analyze in R, unbox KPIs.
 - **gKnit improvements**: Better R Markdown integration
 - **Rails integration**: First-class support for Rails patterns
 
@@ -907,7 +874,7 @@ For the full blog post, include references to:
 - **NSE/dplyr**: `blogs/nse_dplyr/nse_dplyr.md`
 - **Object-Oriented**: `blogs/oh_my/oh_my.md`
 - **Plotting Tutorial**: `blogs/ruby_plot/ruby_plot.md`
-- **Apache Arrow**: `specs/arrow_semantics_spec.rb`, `specs/arrow_from_ruby_batches_spec.rb`, `slow-specs/arrow_large_pipeline_spec.rb`
+- **Apache Arrow**: `Documentation/ROADMAP_ARROW_RUBY_R.md`; `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, `new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb`; Stage A: `specs/arrow_from_ruby_batches_spec.rb`, `slow-specs/arrow_large_pipeline_spec.rb`
 - **Specs**: `specs/r_nse.spec.rb`, `specs/r_vector_functions.spec.rb`
 
 ---