galaaz 2.1.0 → 2.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0fc6f307a98f073e4ad49a39f5b4fcfddbd948c9054722caea814c16d6be62fa
-  data.tar.gz: d899ca8013e16d158cb1c035a3e8d1393b3d377782fad502b09d1e81d9eb868a
+  metadata.gz: fe7e81157b29ea65c61ca9bb97ab5a7f458b90b601cd1c76d8e1740e95e1f69f
+  data.tar.gz: bfd7e6b8f867ef903cb5119837d0e045b11c57e3aa2b5c9a717447abd0922478
 SHA512:
-  metadata.gz: a73ce81b2b86550af39562adc7d1566746c5239d0b74688c8d1d1cef52bb5371ab5449cb3dbe3d6b14b2608e1c33e3289937f745a9749c7c425f026a4d8f8659
-  data.tar.gz: 3d84d7e16f678ed8b13920585099df2ae641293ae74f65e7920d7bd7ee42c971deaa9c7dd1985343fb2eadf17be582d3922eecf3d20d009bfc45180f762b7b26
+  metadata.gz: 5599509abe5d487fb7e19d8ae3fb1a80a558dcc6928e4a526753253ff816820d41509d2f179d6968ca985bbb19084c60ab0f0fc82e34276d840a49a627ad171d
+  data.tar.gz: 3560c286e71ce88fe5486d71f62c4fb2cba342bf8af03f93e87a2c34ded854fce98eed90b57fb094acd39d333d0d0ea9bb49c35f651a0b33367b2bbc96c09b62

data/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## 2.1.2
+
+### Fixed
+
+- CRAN installs (`galaaz setup` / `galaaz add`) use a writable user library
+  (`R_LIBS_USER`, default `~/.local/lib/R/library`) so Omarchy/Arch no longer fail
+  with `lib = "/usr/lib/R/library" is not writable`.
+- Gatekeeper `ext/new_bridge` Makefile uses `R CMD config --ldflags` instead of
+  hardcoded `-L$(R_HOME)/lib -lR` (breaks when `RHOME` is `/usr/lib64/R`).
+- `galaaz setup` marks profile `core` on success.
+
+### Notes
+
+- Omarchy dogfood: pin `script/omarchy/install-galaaz.sh` to this version; re-copy
+  the installer into `~/.local/bin/omarchy-install-galaaz` after updating.
+
+## 2.1.1
+
+### Added
+
+- CLI on `bin/galaaz`: `setup`, `blogs init`, `doctor`, `add` (knit / arrow / tex / bio /
+  examples / ledger / demo). Legacy unknown args still forward to rake.
+- Omarchy dogfood helpers under `script/omarchy/` (install/remove scripts + menu overlay).
+- `galaaz doctor` reports both CRuby and JRuby when available (mise-aware).
+- Package lists `r_requires/knit.txt`, `knit-extras.txt`, `arrow.txt` for add-on installs.
+- Cold-install CRuby path uses `galaaz setup` + `galaaz blogs init`.
+
+### Notes
+
+- Omarchy / stranger install remains `gem install galaaz` then `galaaz setup` (no
+  `bundle install` for core use). See `Documentation/PLAN_OMARCHY_INTEGRATION.md`.
+
 ## 2.1.0
 
 Galaaz **2.1** keeps the **galaaz2_0** integration line as the main development branch and marks
@@ -10,14 +42,16 @@ Galaaz **2.1** keeps the **galaaz2_0** integration line as the main development
 - First-class **CRuby** NewBridge path alongside JRuby (engine-aware launchers, cold-install /
   CI coverage for both).
 - Docs: treat JRuby and CRuby as equal supported runtimes for the bridge.
-- [Documentation/ROADMAP_ARROW_RUBY_R.md](Documentation/ROADMAP_ARROW_RUBY_R.md) — roadmap for
-  Ruby↔R Arrow IPC / mmap handoff (stage B) then shared-memory bus (stage C).
+- [Documentation/ROADMAP_ARROW_RUBY_R.md](Documentation/ROADMAP_ARROW_RUBY_R.md) — stages A (copy),
+  B1/B2 (IPC/mmap file; shipped), C (shared-memory bus; future).
+- Stage **B1/B2** APIs: `Galaaz::ArrowIpc`, `R::Arrow.open_ipc`, `R::Arrow.write_ipc` (CRuby
+  red-arrow + system Arrow GLib; JRuby Arrow Java + `JAVA_OPTS` nio opens).
 
 ### Notes
 
 - Git branch **`galaaz2_0`** remains the integration branch name; the gem version is **2.1.0**.
-- Apache Arrow **zero-copy shared RAM** is still future work (see the roadmap); today’s helpers
-  build R-side tables after a handoff, then use proxy / Remote Control calls.
+- Apache Arrow **zero-copy shared RAM** (Stage C) is still future work. Stage A copies into R;
+  Stage B uses an IPC file and only the path crosses NewBridge. See the roadmap.
 
 ## 2.0.0
 

data/README.md

@@ -59,9 +59,11 @@ CRuby when you prefer MRI. R remains the same **GNU R** you use interactively—
 compiled extensions and Bioconductor. Earlier GraalVM / TruffleRuby / FastR experiments
 are no longer the focus.
 
-The bridge handles **communication and typing** between the two worlds; large tables can
-also flow through **Apache Arrow** on the R side when you use the optional helpers described
-later in this manual.
+The bridge handles **communication and typing** between the two worlds. Large tables can use
+**Apache Arrow** in two shipped modes (described later): **Stage A** copies Ruby batches into an
+R-side Arrow table (`R::Arrow.from_ruby_batches`); **Stage B** writes an Arrow IPC file and only
+the **path** crosses NewBridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). Shared-heap
+zero-copy is **Stage C** and is not shipped.
 
 ## R-on-Rails: the one-person app for R scientists
 
@@ -340,7 +342,7 @@ The supported install is **`gem install` + compile the gatekeeper**. You do not
    make -C "${gem_dir}/ext/new_bridge" all
    ```
 
-5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on Java 9+, pass `-J--add-opens=java.base/java.nio=ALL-UNNAMED` to JRuby (from a checkout, `bin/galaaz-jruby` does this; on CRuby this flag is not needed).
+5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on **JRuby** (Java 9+), the child JVM needs `--add-opens=java.base/java.nio=ALL-UNNAMED` via **`JAVA_OPTS`** (from a checkout, `bin/galaaz-jruby` and `mise.toml` set this; a leading `jruby -J... -S bundle exec` does **not** pass `-J` to rspec). On **CRuby**, install Apache Arrow GLib (`libarrow-glib-dev` from the [Apache Arrow APT](https://arrow.apache.org/install/) repo) and `gem install red-arrow` matching `pkg-config --modversion arrow-glib`. Do not install the unrelated Rubygems package named `arrow`.
 
 For **gKnit**, **knitr**, **rmarkdown**, and LaTeX (PDF output), install the corresponding R packages, **Pandoc**, and a TeX distribution if you need PDF; the repository includes helpers such as **`bin/install-tinytex`** where appropriate.
 
@@ -442,13 +444,15 @@ A practical pattern is:
 
 1. Use threads (or a connection pool) to read from **multiple databases or shards** in parallel.
 2. Merge the rows in Ruby under a `Mutex` if you collect into one structure.
-3. Hand the merged table to R **once** (for example with `R::Arrow.from_ruby_batches` and dplyr,
-   or by building a data frame) so heavy statistics run in R with fewer bridge round-trips.
+3. Hand the merged table to R **once**: **`R::Arrow.from_ruby_batches`** (Stage A: copy into R) or
+   **`Galaaz::ArrowIpc.write` / `write_batches`** then **`R::Arrow.open_ipc`** (Stage B: IPC file;
+   only the path crosses the bridge). Then run dplyr in R.
 
 A runnable sketch lives in
 `examples/multithread_shards_to_r/shards_to_r.rb` (simulated shard queries; swap in your DB
 driver). For concurrency tests on the bridge itself, see `specs/bridge_concurrent_spec.rb` and
-`specs/arrow_from_ruby_batches_spec.rb`.
+`specs/arrow_from_ruby_batches_spec.rb`. Stage B IPC tests: `specs/arrow_ipc_handoff_spec.rb`,
+`specs/arrow_ipc_export_spec.rb`.
 
 ## Long-running R calls and a completion block
 
@@ -4025,35 +4029,77 @@ ans = flights[:all, E.list(R[:arr_delay], R[:dep_delay])]
 # Apache Arrow
 
 [Apache Arrow](https://arrow.apache.org/) is a **columnar** in-memory format used heavily in R
-and Python for analytics. In Galaaz, **Ruby does not hold an Arrow C++ table itself**; instead you
-build ordinary Ruby structures (arrays of row hashes), and **`R::Arrow.from_ruby_batches`** creates
-a real **Arrow `Table` inside GNU R**. From there you use R’s **`arrow`** and **`dplyr`** packages
-as usual: **`group_by`** on the Arrow table, **`summarise`** for aggregates, then **`collect()`** to
-materialize a tibble when you need in-memory R rows.
-
-That pattern matches production use: **JRuby threads** (or sequential code) assemble many rows in
-Ruby; you pay **one** bridge-heavy handoff to R; **dplyr** runs vectorised work on the Arrow table
-in R.
+and Python for analytics. GNU R still runs in a **separate process**. Ruby does **not** hold a
+shared Arrow C++ table with R. Stages:
+
+1. **Stage A (copy over the bridge):** Ruby row hashes → **`R::Arrow.from_ruby_batches`** builds
+   an Arrow `Table` **inside GNU R**. You get a **proxy**.
+2. **Stage B1 (Ruby → R IPC file):** **`Galaaz::ArrowIpc.write`** / **`write_batches`** writes an
+   Arrow IPC file (prefer **`/dev/shm`**); **`R::Arrow.open_ipc(path)`** opens it in R. Only the
+   **path** crosses NewBridge. This is **mmap/IPC file handoff**, not a shared heap.
+3. **Stage B2 (R → Ruby IPC file):** **`R::Arrow.write_ipc(obj)`** writes uncompressed IPC; Ruby
+   reads with **`Galaaz::ArrowIpc.read`** (column hash) or **`read_batches`** (row hashes). Call
+   **`Galaaz::ArrowIpc.release(path)`** when finished.
+4. **Stage C (not shipped):** named shared-memory bus. Do not claim 0 ms shared RAM until then.
+   See **`Documentation/ROADMAP_ARROW_RUBY_R.md`**.
+
+After ingest, use R’s **`arrow`** / **`dplyr`** on the proxy (`group_by`, `summarise`, `collect`)
+and unbox only KPIs you need in Ruby.
+
+**Optional Ruby backends for Stage B**
+
+* **CRuby:** Apache **red-arrow** — `gem install red-arrow` pinned to the same major as
+  `pkg-config --modversion arrow-glib`, plus system **Arrow GLib** (`libarrow-glib-dev` from the
+  [Apache Arrow APT](https://arrow.apache.org/install/) repo). Do **not** install the unrelated
+  legacy Rubygems package named `arrow`. `bundle exec` still sees a user-installed `red-arrow`
+  via Galaaz’s load-path helper.
+* **JRuby:** Apache Arrow **Java** JARs — **`GALAAZ_ARROW_JARS`**, `~/arrow_jars`, or
+  `jar-dependencies`. Export **`JAVA_OPTS=--add-opens=java.base/java.nio=ALL-UNNAMED`** on the
+  **child** JVM (`bin/galaaz-jruby`, `mise.toml`). `jruby -J... -S bundle exec rspec` does **not**
+  pass `-J` to rspec.
+
+**R packages:** **`arrow`** and **`dplyr`**. B2 writes IPC with **`compression: 'uncompressed'`**
+so JRuby Arrow Java can read without extra compression JARs.
+
+**Tests:** `specs/arrow_from_ruby_batches_spec.rb` (A);
+`specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb` (B, sync);
+`new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb` (B, async).
+
+## `R::Arrow` and `Galaaz::ArrowIpc`
+
+* **`R::Arrow.from_ruby_batches`** — Stage A ingest.
+* **`R::Arrow.open_ipc(path)`** — Stage B1: IPC file → R Table proxy.
+* **`R::Arrow.write_ipc(obj, path = nil)`** — Stage B2: R Table/tibble → IPC path (scratch if omitted).
+* **`Galaaz::ArrowIpc.write` / `write_batches` / `read` / `read_batches` / `allocate_path` / `release` / `available?`**
+* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
+* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
+  dataset IO on paths visible to R.
 
-**Prerequisites:** install R packages **`arrow`** and **`dplyr`**. Run scripts with
-**`bin/galaaz-jruby`** (or the same JVM flags as in **`docs/testing.md`**) so the Arrow JNI stack is
-available.
+## Example: Stage B round-trip (IPC file)
 
-## Other `R::Arrow` helpers
+Requires `Galaaz::ArrowIpc.available?` (red-arrow or Arrow JARs) and R **`arrow`**. Not knitted
+below so a machine without the optional backend still builds this manual.
 
-The Ruby module **`R::Arrow`** (see `lib/R_interface/r_arrow.rb`) also includes:
+```ruby
+path = Galaaz::ArrowIpc.write(id: [1, 2, 3], grp: %w[a a b], value: [1.0, 2.0, 3.5])
+tbl  = R::Arrow.open_ipc(path)
+Galaaz::ArrowIpc.release(path)
 
-* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
-* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
-  dataset IO on paths visible to R.
+summed   = R.dplyr___summarise(R.dplyr___group_by(tbl, :grp), total: E.sum(:value))
+out_path = R::Arrow.write_ipc(summed)
+rows     = Galaaz::ArrowIpc.read_batches(out_path)
+Galaaz::ArrowIpc.release(out_path)
+# rows => [{:grp=>"a", :total=>3.0}, {:grp=>"b", :total=>3.5}]  (illustrative)
+```
 
 ## Example: many Ruby rows → Arrow in R → grouped statistics
 
 The repository test **`slow-specs/arrow_large_pipeline_spec.rb`** builds **200k rows** in parallel
-(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`**, then checks that
-**dplyr** group summaries match a Ruby reference calculation. The same logic appears below at a
-**smaller scale** so this manual can knit quickly; increase `thread_count` and `rows_per_thread`
-when experimenting locally.
+(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`** (Stage A), then
+checks that **dplyr** group summaries match a Ruby reference calculation. The same logic appears
+below at a **smaller scale** so this manual can knit quickly; increase `thread_count` and
+`rows_per_thread` when experimenting locally. For the same ingest **without** copying every cell
+over NewBridge, use Stage B (`write_batches` + `open_ipc`) instead of `from_ruby_batches`.
 
 
 ``` ruby
@@ -4119,10 +4165,10 @@ end
 ```
 
 **What to notice:** (1) Ruby only sees **`Hash`** rows and Ruby **`Thread`** objects; (2) a single
-**`from_ruby_batches`** call creates the Arrow table in R; (3) **`dplyr___group_by`** /
+**`from_ruby_batches`** call **copies** those columns into an Arrow table in R; (3) **`dplyr___group_by`** /
 **`dplyr___summarise`** / **`dplyr___collect`** mirror **`dplyr::group_by`** /
 **`dplyr::summarise`** / **`dplyr::collect`** on an Arrow-backed table. For a lighter test, see
-**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size benchmark, run
+**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size Stage A benchmark, run
 **`bin/run_slow_rspec slow-specs/arrow_large_pipeline_spec.rb`**.
 
 # Bioconductor and DESeq2
@@ -4260,8 +4306,9 @@ Practical tips:
   glue.
 * **Reuse one process**: running many short scripts cold-starts Ruby, the JVM, and R each time;
   a long-lived process or repeated calls in one run amortize setup (see benchmarks below).
-* **Batch data**: merge shards in Ruby, then call **`R::Arrow.from_ruby_batches`** (or build one
-  data frame) instead of millions of tiny R calls.
+* **Batch data**: merge shards in Ruby, then **`R::Arrow.from_ruby_batches`** (Stage A) or
+  **`Galaaz::ArrowIpc`** + **`R::Arrow.open_ipc`** (Stage B) instead of millions of tiny R calls.
+  When Ruby needs a bulky result table back, **`R::Arrow.write_ipc`** + **`Galaaz::ArrowIpc.read_batches`**.
 
 For measured discussion (including DESeq2-style workloads and warm comparisons), see
 **`docs/performance.md`** and **`docs/deseq2_airway_benchmark.md`** in the Galaaz repository.

data/bin/galaaz

@@ -1,5 +1,13 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
-Dir.chdir(File.dirname(File.expand_path('..', __FILE__)))
+# Galaaz CLI: setup / blogs / doctor / add.
+# Unknown commands still forward to rake (legacy example runners).
 
-system ("rake #{ARGV[0]}")
+root = File.expand_path('..', __dir__)
+lib = File.join(root, 'lib')
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require 'galaaz/cli'
+
+exit Galaaz::CLI.run(ARGV)

data/bin/galaaz_jruby_env.inc.sh

@@ -1,6 +1,22 @@
-# Galaaz required JVM flags for JRuby (Apache Arrow on Java 9+).
+# Galaaz required JVM flags for JRuby (Apache Arrow memory on Java 9+).
 # Source from bin/*.sh:  source "$ROOT/bin/galaaz_jruby_env.inc.sh"
 #
-# Source of truth in Ruby/Rake: lib/galaaz_jruby.rb (GalaazJRuby::REQUIRED_JRUBY_J_ARGS).
+# Source of truth in Ruby/Rake: lib/galaaz_jruby.rb (GalaazJRuby::JAVA_NIO_ADD_OPENS).
 # Optional extra -J flags: export GALAAZ_JRUBY_OPTS="-J-Xmx4g" (space-separated).
-GALAAZ_REQUIRED_JRUBY_J_ARGS='-J--add-opens=java.base/java.nio=ALL-UNNAMED'
+#
+# JAVA_OPTS is exported so *child* JRuby processes (bundle exec rspec, mise exec)
+# get the same opens flag. -J args on the outer command are not inherited by
+# bundle exec's second JVM.
+GALAAZ_JAVA_NIO_ADD_OPENS='--add-opens=java.base/java.nio=ALL-UNNAMED'
+GALAAZ_REQUIRED_JRUBY_J_ARGS="-J${GALAAZ_JAVA_NIO_ADD_OPENS}"
+
+case " ${JAVA_OPTS:-} " in
+  *" ${GALAAZ_JAVA_NIO_ADD_OPENS} "*) ;;
+  *)
+    if [ -n "${JAVA_OPTS:-}" ]; then
+      export JAVA_OPTS="${GALAAZ_JAVA_NIO_ADD_OPENS} ${JAVA_OPTS}"
+    else
+      export JAVA_OPTS="${GALAAZ_JAVA_NIO_ADD_OPENS}"
+    fi
+    ;;
+esac

data/bin/galaaz_ruby_env.inc.sh

@@ -7,6 +7,7 @@
 #
 # JRuby and CRuby are both first-class. Override with GALAAZ_RUBY=jruby, GALAAZ_RUBY=ruby,
 # or a full path. Extra JRuby flags: GALAAZ_JRUBY_OPTS="-J-Xmx4g" (ignored on CRuby).
+# CRuby red-arrow uses system Arrow GLib (Apache Arrow APT / libarrow-glib-dev).
 
 GALAAZ_RUBY_BIN="${GALAAZ_RUBY:-ruby}"
 
@@ -29,6 +30,7 @@ if [[ "$_galaaz_is_jruby" -eq 1 ]]; then
   # shellcheck source=galaaz_jruby_env.inc.sh
   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/galaaz_jruby_env.inc.sh"
   GALAAZ_RUBY_J_ARGS="$GALAAZ_REQUIRED_JRUBY_J_ARGS ${GALAAZ_JRUBY_OPTS:-}"
+  # JAVA_OPTS is exported by galaaz_jruby_env.inc.sh for bundle-exec child JVMs
 else
   GALAAZ_REQUIRED_JRUBY_J_ARGS=''
   GALAAZ_RUBY_J_ARGS=''