galaaz 2.0.0 → 2.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +43 -5
- data/README.md +200 -128
- data/Rakefile +17 -10
- data/bin/galaaz +10 -2
- data/bin/galaaz-jruby +3 -6
- data/bin/galaaz-ruby +16 -0
- data/bin/galaaz_jruby_env.inc.sh +19 -3
- data/bin/galaaz_ruby_env.inc.sh +38 -0
- data/bin/gbookdown +2 -3
- data/bin/gknit +2 -3
- data/bin/gstudio +4 -4
- data/bin/run_all_rspec +6 -5
- data/bin/run_example +10 -7
- data/bin/run_old_rspec +6 -5
- data/bin/run_rspec +9 -8
- data/bin/run_slow_rspec +6 -5
- data/blogs/R-on-Rails-Planning-Document.md +79 -112
- data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/midwest_rb.png +0 -0
- data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/scatter_plot_rb.png +0 -0
- data/blogs/gknit/gknit_files/figure-gfm/bubble-1.png +0 -0
- data/blogs/gknit/gknit_files/figure-gfm/diverging_bar.png +0 -0
- data/blogs/manual/manual.Rmd +22 -19
- data/blogs/manual/manual.md +200 -128
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/dose_len.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_delivery.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_dose.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color2.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_decorations.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_jitter.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_points.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_box_plot.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_violin_plot.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/violin_with_jitter.png +0 -0
- data/blogs/test/test.md +10 -0
- data/examples/50Plots_MasterList/ScatterPlot.rb +2 -1
- data/examples/50Plots_MasterList/scatter_plot.rb +1 -0
- data/examples/bioconductor_deseq2_airway/bench_galaaz_three_same_process.rb +1 -0
- data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz.rb +1 -0
- data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz_optimized.rb +1 -0
- data/examples/islr/all.rb +1 -0
- data/examples/islr/ch2.spec.rb +1 -0
- data/examples/islr/ch3.spec.rb +1 -0
- data/examples/islr/ch3_boston.rb +1 -0
- data/examples/islr/ch3_multiple_regression.rb +1 -0
- data/examples/islr/ch6.spec.rb +1 -0
- data/examples/islr/x_y_rnorm.jpg +0 -0
- data/examples/misc/ggplot.rb +2 -1
- data/examples/misc/moneyball.rb +1 -0
- data/examples/misc/subsetting.rb +1 -0
- data/examples/multithread_shards_to_r/shards_to_r.rb +1 -0
- data/examples/sthda_ggplot/all.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/density_gg.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_area.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_density.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_dotplot.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_freqpoly.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_histogram.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/histogram_density.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/stat.rb +1 -0
- data/examples/sthda_ggplot/one_variable_discrete/bar.rb +1 -0
- data/examples/sthda_ggplot/qplots/box_violin_dot.rb +1 -0
- data/examples/sthda_ggplot/qplots/scatter_plots.rb +1 -0
- data/examples/sthda_ggplot/scatter_gg.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_bin2d.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_density2d.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_hex.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/geom_point.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/geom_smooth.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/misc.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_function/geom_area.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_bar.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_boxplot.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_dotplot.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_jitter.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_line.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_violin.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_disc/geom_jitter.rb +1 -0
- data/examples/sthda_ggplot/two_variables_error/geom_crossbar.rb +1 -0
- data/lib/R_interface/r_arrow.rb +37 -0
- data/lib/R_interface/rvector.rb +15 -0
- data/lib/galaaz/arrow_ipc/java_arrow_backend.rb +250 -0
- data/lib/galaaz/arrow_ipc/red_arrow_backend.rb +126 -0
- data/lib/galaaz/arrow_ipc.rb +167 -0
- data/lib/galaaz/cli.rb +531 -0
- data/lib/galaaz.rb +6 -0
- data/lib/galaaz_jruby.rb +17 -2
- data/lib/galaaz_ruby.rb +34 -0
- data/lib/new_bridge/envelope.rb +2 -2
- data/new_bridge_specs/arrow_ipc_async_spec.rb +90 -0
- data/new_bridge_specs/arrow_ipc_export_async_spec.rb +68 -0
- data/r_requires/arrow.txt +3 -0
- data/r_requires/knit-extras.txt +4 -0
- data/r_requires/knit.txt +4 -0
- data/specs/arrow_ipc_export_spec.rb +62 -0
- data/specs/arrow_ipc_handoff_spec.rb +97 -0
- data/specs/gknit_include_engine_spec.rb +2 -2
- data/specs/gknit_install_timeout_report_spec.rb +2 -2
- data/specs/gknit_internal_error_report_spec.rb +2 -2
- data/specs/gknit_vector_map_output_spec.rb +2 -2
- data/version.rb +1 -1
- metadata +50 -13
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 671c2fe1fb15dbd12a7f491e52c318d01ebe9625e78d9acdaaa3dffbe3efbd3d
|
|
4
|
+
data.tar.gz: 2d4d63f915f0b8a09cff53df98802f7bcaff0298f9247ef38b583b2a0e902730
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1dc8f434731a05f495909d5f8bc70949f7ec7ae8034fbe8dd6a026f6b9592d426d573dffc0f13326ddfb7c830112bd6922fcdb8ca717e091d8722e219ea43de6
|
|
7
|
+
data.tar.gz: d594128365977ac52a3a2d15f6e3dd5f1b97ee312c0c6232883f6fb8838fb7c1440f2272c3302e53853026befd7e625f6e4124a8d3ddd32959f7bf665cb8cda7
|
data/CHANGELOG.md
CHANGED
|
@@ -1,26 +1,64 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.1.1
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
- CLI on `bin/galaaz`: `setup`, `blogs init`, `doctor`, `add` (knit / arrow / tex / bio /
|
|
8
|
+
examples / ledger / demo). Legacy unknown args still forward to rake.
|
|
9
|
+
- Omarchy dogfood helpers under `script/omarchy/` (install/remove scripts + menu overlay).
|
|
10
|
+
- `galaaz doctor` reports both CRuby and JRuby when available (mise-aware).
|
|
11
|
+
- Package lists `r_requires/knit.txt`, `knit-extras.txt`, `arrow.txt` for add-on installs.
|
|
12
|
+
- Cold-install CRuby path uses `galaaz setup` + `galaaz blogs init`.
|
|
13
|
+
|
|
14
|
+
### Notes
|
|
15
|
+
|
|
16
|
+
- Omarchy / stranger install remains `gem install galaaz` then `galaaz setup` (no
|
|
17
|
+
`bundle install` for core use). See `Documentation/PLAN_OMARCHY_INTEGRATION.md`.
|
|
18
|
+
|
|
19
|
+
## 2.1.0
|
|
20
|
+
|
|
21
|
+
Galaaz **2.1** keeps the **galaaz2_0** integration line as the main development branch and marks
|
|
22
|
+
**CRuby + JRuby** NewBridge support as a released milestone (R-on-Rails positioning).
|
|
23
|
+
|
|
24
|
+
### Added
|
|
25
|
+
|
|
26
|
+
- First-class **CRuby** NewBridge path alongside JRuby (engine-aware launchers, cold-install /
|
|
27
|
+
CI coverage for both).
|
|
28
|
+
- Docs: treat JRuby and CRuby as equal supported runtimes for the bridge.
|
|
29
|
+
- [Documentation/ROADMAP_ARROW_RUBY_R.md](Documentation/ROADMAP_ARROW_RUBY_R.md) — stages A (copy),
|
|
30
|
+
B1/B2 (IPC/mmap file; shipped), C (shared-memory bus; future).
|
|
31
|
+
- Stage **B1/B2** APIs: `Galaaz::ArrowIpc`, `R::Arrow.open_ipc`, `R::Arrow.write_ipc` (CRuby
|
|
32
|
+
red-arrow + system Arrow GLib; JRuby Arrow Java + `JAVA_OPTS` nio opens).
|
|
33
|
+
|
|
34
|
+
### Notes
|
|
35
|
+
|
|
36
|
+
- Git branch **`galaaz2_0`** remains the integration branch name; the gem version is **2.1.0**.
|
|
37
|
+
- Apache Arrow **zero-copy shared RAM** (Stage C) is still future work. Stage A copies into R;
|
|
38
|
+
Stage B uses an IPC file and only the path crosses NewBridge. See the roadmap.
|
|
39
|
+
|
|
3
40
|
## 2.0.0
|
|
4
41
|
|
|
5
|
-
Galaaz 2.0 drives **GNU R** from **JRuby** over a process bridge. It is not the older GraalVM / TruffleRuby / FastR stack, and it does not embed Renjin.
|
|
42
|
+
Galaaz 2.0 drives **GNU R** from **JRuby** or **CRuby** over a process bridge. It is not the older GraalVM / TruffleRuby / FastR stack, and it does not embed Renjin. Both Rubies are supported for NewBridge (tested on **JRuby 10.1.1.0** with **Java 21**, and **CRuby 3.3.12**).
|
|
6
43
|
|
|
7
44
|
### Added
|
|
8
45
|
|
|
9
|
-
- NewBridge:
|
|
46
|
+
- NewBridge: Ruby talks to standard GNU R (CRAN / Bioconductor, including compiled packages).
|
|
10
47
|
- Native gatekeeper under `ext/new_bridge` (Rcpp / C++); compile after install with `make -C ext/new_bridge all`.
|
|
11
48
|
- Optional Apache Arrow helpers for large tables on the R side.
|
|
12
49
|
- Rendered manuals, blogs, and examples on [GitHub Pages](https://rbotafogo.github.io/galaaz/).
|
|
13
|
-
- Ubuntu cold-install proof (`docker/cold-install`) and GitHub Actions CI for the fast suite plus
|
|
50
|
+
- Ubuntu cold-install proof (`docker/cold-install` for JRuby, `docker/cold-install-cruby` for CRuby) and GitHub Actions CI for the fast suite on both engines plus those images.
|
|
51
|
+
- Engine-aware launchers (`GALAAZ_RUBY`, `bin/galaaz-ruby`) so the same NewBridge runs under JRuby or CRuby.
|
|
14
52
|
|
|
15
53
|
### Changed
|
|
16
54
|
|
|
17
55
|
- The gem ships sources, specs, and examples; it does not pack prebuilt `.so` objects or knitted HTML/PDF.
|
|
18
|
-
- The supported install is `
|
|
56
|
+
- The supported install is `gem install galaaz` (under JRuby or CRuby), then compile the gatekeeper (see the README). A git clone is for contributors.
|
|
19
57
|
|
|
20
58
|
### Known limits
|
|
21
59
|
|
|
22
60
|
- Primary targets are Linux (x86_64) and macOS; on Windows use WSL2. Native Windows is not supported.
|
|
23
|
-
- `grun`, `gknit_Rscript`, and the polyglot path in `gknit-draft` are leftover from the Graal era; use `galaaz-jruby` / `gknit
|
|
61
|
+
- `grun`, `gknit_Rscript`, and the polyglot path in `gknit-draft` are leftover from the Graal era; use `galaaz-jruby` / `gknit` (or `GALAAZ_RUBY=ruby` for CRuby).
|
|
24
62
|
- The gatekeeper must be compiled on the install machine (Rcpp + C++ toolchain); it is not a prebuilt binary in the gem.
|
|
25
63
|
- Multi-runtime Docker R (Phase 4.5) is a slow integration spec, not part of cold-install CI.
|
|
26
64
|
- The gknit installation-timeout spec is skipped pending a dedicated pass; SimpleCov is a development gem and is not installed in the smoke image.
|
data/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "Galaaz Manual"
|
|
3
|
-
subtitle: "
|
|
3
|
+
subtitle: "R-on-Rails: GNU R meets Ruby for the web"
|
|
4
4
|
author: "Rodrigo Botafogo"
|
|
5
|
-
tags: [Galaaz, Ruby, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
|
|
5
|
+
tags: [Galaaz, "R-on-Rails", Ruby, Rails, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
|
|
6
6
|
date: "2026"
|
|
7
7
|
bibliography: "../../examples/Bibliography/stats.bib"
|
|
8
8
|
output:
|
|
@@ -28,64 +28,66 @@ fontsize: 11pt
|
|
|
28
28
|
|
|
29
29
|
# Introduction
|
|
30
30
|
|
|
31
|
-
Galaaz is
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
not
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
Ruby
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
**
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
Galaaz
|
|
87
|
-
|
|
88
|
-
|
|
31
|
+
**Galaaz is R-on-Rails:** keep **GNU R** for statistics, graphics, and the CRAN /
|
|
32
|
+
Bioconductor ecosystem, and use **Ruby** (with **Rails** when you need a web app) for
|
|
33
|
+
everything R was never meant to own—HTTP, auth, databases, background jobs, HTML, APIs.
|
|
34
|
+
|
|
35
|
+
The primary audience is the **R scientist** who already has models, dplyr/ggplot pipelines,
|
|
36
|
+
and domain knowledge, and wants a **one-person path to production on the web**. Learning
|
|
37
|
+
enough Ruby and Rails to ship an app is usually easier than hiring (or becoming) a dedicated
|
|
38
|
+
web-and-stats dual specialist. Ruby developers gain full access to R libraries as a
|
|
39
|
+
secondary benefit; the design priority is **R workflow first, web second**.
|
|
40
|
+
|
|
41
|
+
Galaaz does **not** re-implement scientific libraries in Ruby. Ruby and R run in **separate
|
|
42
|
+
processes**; the **Galaaz bridge** sends requests to standard **GNU R** and returns typed
|
|
43
|
+
results. You keep calling familiar ideas from Ruby (`R.c(...)`, `R.library('ggplot2')`,
|
|
44
|
+
`~R[:mtcars]`, dplyr-style chains) while Rails (or plain Ruby) owns the application shell.
|
|
45
|
+
|
|
46
|
+
**Python** remains a strong competitor for end-to-end data science stacks (NumPy, pandas,
|
|
47
|
+
SciPy, scikit-learn, Django/Flask). Galaaz’s bet is different: if your science already lives
|
|
48
|
+
in **R**, do not rewrite it in Python to get a website—**put R on Rails**.
|
|
49
|
+
|
|
50
|
+
Ruby is a dynamic, object-oriented language that became widely known through **Ruby on Rails**
|
|
51
|
+
(RoR). It remains a practical choice for web applications and general-purpose scripting.
|
|
52
|
+
Until Galaaz, Ruby lacked a tight, ecosystem-complete bridge to GNU R. Library wrapping
|
|
53
|
+
(one gem per package) does not scale; Galaaz wraps **the R language**, so thousands of R
|
|
54
|
+
packages are available without a new handcrafted wrapper for each one.
|
|
55
|
+
|
|
56
|
+
**Galaaz 2.0** supports **JRuby** and **CRuby** equally for the same NewBridge protocol.
|
|
57
|
+
Pick the Ruby that fits your app: JRuby when you want real multithreading for web and I/O;
|
|
58
|
+
CRuby when you prefer MRI. R remains the same **GNU R** you use interactively—including
|
|
59
|
+
compiled extensions and Bioconductor. Earlier GraalVM / TruffleRuby / FastR experiments
|
|
60
|
+
are no longer the focus.
|
|
61
|
+
|
|
62
|
+
The bridge handles **communication and typing** between the two worlds. Large tables can use
|
|
63
|
+
**Apache Arrow** in two shipped modes (described later): **Stage A** copies Ruby batches into an
|
|
64
|
+
R-side Arrow table (`R::Arrow.from_ruby_batches`); **Stage B** writes an Arrow IPC file and only
|
|
65
|
+
the **path** crosses NewBridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). Shared-heap
|
|
66
|
+
zero-copy is **Stage C** and is not shipped.
|
|
67
|
+
|
|
68
|
+
## R-on-Rails: the one-person app for R scientists
|
|
69
|
+
|
|
70
|
+
If you already think in R, the usual web options are painful: Shiny for some apps, or a full
|
|
71
|
+
rewrite in another stack. **R-on-Rails** means:
|
|
72
|
+
|
|
73
|
+
1. **Keep your science in R** — packages, formulas, plots, Bioconductor, the same engine as RStudio.
|
|
74
|
+
2. **Learn enough Ruby/Rails** — routes, controllers, views, jobs, auth—not a second statistics career.
|
|
75
|
+
3. **Call R from the app** — Galaaz loads R behind the scenes; long jobs can complete asynchronously
|
|
76
|
+
while Rails stays responsive (see later sections on the bridge and `R::Async`).
|
|
77
|
+
4. **Ship alone when you need to** — one developer can own both the analysis and the product UI,
|
|
78
|
+
without waiting for a separate “stats engineer” and “Rails engineer.”
|
|
79
|
+
|
|
80
|
+
Typical shape:
|
|
81
|
+
|
|
82
|
+
- Interactive exploration and reports: **gstudio**, **gknit** (R Markdown with Ruby chunks).
|
|
83
|
+
- Product: a **Rails** (or Sinatra) app that calls `R.*` for the heavy statistical steps.
|
|
84
|
+
- Scale for many users: more **R worker processes/containers** behind the app (R is single-threaded
|
|
85
|
+
per process); the Ruby web tier scales separately (JRuby threads or a multi-process CRuby
|
|
86
|
+
setup). Galaaz’s instance manager is the starting point for that pattern—not a rewrite of
|
|
87
|
+
every algorithm à la enterprise ScaleR.
|
|
88
|
+
|
|
89
|
+
Rubyists are welcome: the same bridge exposes CRAN to application code. The **intended** on-ramp,
|
|
90
|
+
though, is **R scientist → small Rails app**, not “hire a stats team to teach Rails developers R.”
|
|
89
91
|
|
|
90
92
|
## What does Galaaz mean
|
|
91
93
|
|
|
@@ -108,10 +110,11 @@ The Galaaz repository ships many helpers under **`bin/`**. When working from a *
|
|
|
108
110
|
them as **`bin/<name>`** from the project root (or `./bin/<name>`). If you install the **gem**,
|
|
109
111
|
only a subset is guaranteed on your `PATH` (see the gemspec: **`galaaz`**, **`gstudio`**, **`gknit`**, **`grun`**, **`gknit-draft`**); for development and CI, prefer the **`bin/`** copies so JVM flags and paths stay correct.
|
|
110
112
|
|
|
111
|
-
Below, **current (Galaaz 2.0 + JRuby + GNU R)** means the tool
|
|
112
|
-
**`
|
|
113
|
-
|
|
114
|
-
|
|
113
|
+
Below, **current (Galaaz 2.0 + JRuby or CRuby + GNU R)** means the tool uses **`bin/galaaz-ruby`**
|
|
114
|
+
/ **`GALAAZ_RUBY`** (default: `ruby` on `PATH`) and applies JVM flags only when the interpreter is
|
|
115
|
+
JRuby (`bin/galaaz_jruby_env.inc.sh` / `lib/galaaz_jruby.rb`). **`bin/galaaz-jruby`** forces JRuby.
|
|
116
|
+
**Legacy** means the script still targets **GraalVM** polyglot Ruby / FastR-era invocation and is
|
|
117
|
+
**not** expected to work on a typical JRuby or CRuby NewBridge setup.
|
|
115
118
|
|
|
116
119
|
**Table layout:** names in the first column are **`bin/`** filenames (run as `bin/<name>` from the repo root). Long options and examples sit **outside** the tables so PDF columns stay readable.
|
|
117
120
|
|
|
@@ -131,14 +134,24 @@ expected to work on a typical JRuby-only setup.
|
|
|
131
134
|
<td style="text-align:left;"> WSL2 helper: Docker checks; optional TinyTeX or poppler for gKnit PDF. </td>
|
|
132
135
|
<td style="text-align:left;"> Yes* </td>
|
|
133
136
|
</tr>
|
|
137
|
+
<tr>
|
|
138
|
+
<td style="text-align:left;"> galaaz-ruby </td>
|
|
139
|
+
<td style="text-align:left;"> Selected Ruby (default: ruby on PATH) with repo lib/ on LOAD_PATH; JVM flags only on JRuby. </td>
|
|
140
|
+
<td style="text-align:left;"> Yes </td>
|
|
141
|
+
</tr>
|
|
134
142
|
<tr>
|
|
135
143
|
<td style="text-align:left;"> galaaz-jruby </td>
|
|
136
|
-
<td style="text-align:left;">
|
|
144
|
+
<td style="text-align:left;"> Thin wrapper that forces JRuby (same flags as galaaz-ruby under JRuby). </td>
|
|
137
145
|
<td style="text-align:left;"> Yes </td>
|
|
138
146
|
</tr>
|
|
147
|
+
<tr>
|
|
148
|
+
<td style="text-align:left;"> galaaz_ruby_env.inc.sh </td>
|
|
149
|
+
<td style="text-align:left;"> Sourced by bash wrappers; sets GALAAZ_RUBY_BIN and optional JVM args. </td>
|
|
150
|
+
<td style="text-align:left;"> Yes† </td>
|
|
151
|
+
</tr>
|
|
139
152
|
<tr>
|
|
140
153
|
<td style="text-align:left;"> galaaz_jruby_env.inc.sh </td>
|
|
141
|
-
<td style="text-align:left;"> Sourced
|
|
154
|
+
<td style="text-align:left;"> Sourced when the interpreter is JRuby; sets GALAAZ_REQUIRED_JRUBY_J_ARGS. </td>
|
|
142
155
|
<td style="text-align:left;"> Yes† </td>
|
|
143
156
|
</tr>
|
|
144
157
|
<tr>
|
|
@@ -153,11 +166,13 @@ expected to work on a typical JRuby-only setup.
|
|
|
153
166
|
|
|
154
167
|
† Not run directly.
|
|
155
168
|
|
|
156
|
-
**`galaaz-jruby` examples** (from repo root):
|
|
169
|
+
**`galaaz-ruby` / `galaaz-jruby` examples** (from repo root):
|
|
157
170
|
|
|
158
171
|
```text
|
|
172
|
+
bin/galaaz-ruby my_script.rb
|
|
173
|
+
GALAAZ_RUBY=jruby bin/galaaz-ruby my_script.rb
|
|
159
174
|
bin/galaaz-jruby my_script.rb
|
|
160
|
-
bin/galaaz-
|
|
175
|
+
bin/galaaz-ruby -S rspec
|
|
161
176
|
```
|
|
162
177
|
|
|
163
178
|
## Interactive use, examples, and Rake
|
|
@@ -173,17 +188,17 @@ bin/galaaz-jruby -S rspec
|
|
|
173
188
|
<tbody>
|
|
174
189
|
<tr>
|
|
175
190
|
<td style="text-align:left;"> gstudio </td>
|
|
176
|
-
<td style="text-align:left;"> IRB or Pry with Galaaz preloaded (
|
|
191
|
+
<td style="text-align:left;"> IRB or Pry with Galaaz preloaded (ruby on PATH; JVM flags on JRuby). </td>
|
|
177
192
|
<td style="text-align:left;"> Yes </td>
|
|
178
193
|
</tr>
|
|
179
194
|
<tr>
|
|
180
195
|
<td style="text-align:left;"> run_example </td>
|
|
181
|
-
<td style="text-align:left;"> Run one Ruby file using the same
|
|
196
|
+
<td style="text-align:left;"> Run one Ruby file using the same Ruby/JVM setup as tests. </td>
|
|
182
197
|
<td style="text-align:left;"> Yes </td>
|
|
183
198
|
</tr>
|
|
184
199
|
<tr>
|
|
185
200
|
<td style="text-align:left;"> galaaz </td>
|
|
186
|
-
<td style="text-align:left;"> Forward arguments to rake (needs rake;
|
|
201
|
+
<td style="text-align:left;"> Forward arguments to rake (needs rake; same GALAAZ_RUBY as other launchers). </td>
|
|
187
202
|
<td style="text-align:left;"> Yes </td>
|
|
188
203
|
</tr>
|
|
189
204
|
</tbody>
|
|
@@ -202,7 +217,7 @@ bin/galaaz-jruby -S rspec
|
|
|
202
217
|
<tbody>
|
|
203
218
|
<tr>
|
|
204
219
|
<td style="text-align:left;"> gknit </td>
|
|
205
|
-
<td style="text-align:left;"> Knit .Rmd via JRuby and R Markdown render. </td>
|
|
220
|
+
<td style="text-align:left;"> Knit .Rmd via ruby on PATH (JRuby or CRuby) and R Markdown render. </td>
|
|
206
221
|
<td style="text-align:left;"> Yes </td>
|
|
207
222
|
</tr>
|
|
208
223
|
<tr>
|
|
@@ -212,8 +227,8 @@ bin/galaaz-jruby -S rspec
|
|
|
212
227
|
</tr>
|
|
213
228
|
<tr>
|
|
214
229
|
<td style="text-align:left;"> gknit-draft.rb </td>
|
|
215
|
-
<td style="text-align:left;"> Ruby entry: GKnit.draft (use with
|
|
216
|
-
<td style="text-align:left;">
|
|
230
|
+
<td style="text-align:left;"> Ruby entry: GKnit.draft (use with galaaz-ruby + LOAD_PATH). </td>
|
|
231
|
+
<td style="text-align:left;"> Yes </td>
|
|
217
232
|
</tr>
|
|
218
233
|
<tr>
|
|
219
234
|
<td style="text-align:left;"> gknit_Rscript </td>
|
|
@@ -290,21 +305,23 @@ Prefer **`galaaz-jruby`** for **`gknit-draft`** workflows until that wrapper mat
|
|
|
290
305
|
</tbody>
|
|
291
306
|
</table>
|
|
292
307
|
|
|
293
|
-
For day-to-day **2.0** use, rely on **`bin/galaaz-jruby
|
|
308
|
+
For day-to-day **2.0** use, rely on **`bin/galaaz-ruby`** (or **`bin/galaaz-jruby`** when you want to force JRuby), **`bin/gstudio`**, **`bin/gknit`**, **`bin/run_example`**, **`bin/run_rspec`** / **`bin/run_all_rspec`**, and **`bin/galaaz-bootstrap`** on WSL when using Dockerized R. Treat **`grun`**, **`gknit_Rscript`**, and the polyglot **`ruby`** invocation in **`gknit-draft`** as **legacy** until they are ported to the same launcher path as **`gknit`**.
|
|
294
309
|
|
|
295
310
|
# System Compatibility
|
|
296
311
|
|
|
297
312
|
Typical development and CI targets:
|
|
298
313
|
|
|
299
314
|
* **Linux** — recent Ubuntu LTS or comparable distributions (x86_64).
|
|
300
|
-
* **macOS** — recent releases with JRuby and GNU R available.
|
|
315
|
+
* **macOS** — recent releases with JRuby or CRuby and GNU R available.
|
|
301
316
|
* **Windows** — use **WSL2** (same Linux stack as above); native Windows is not the primary target.
|
|
302
317
|
|
|
303
318
|
The native **gatekeeper** component under `ext/new_bridge` is built with `make` and a C++ toolchain; see the project `README` if compilation fails on your platform.
|
|
304
319
|
|
|
305
320
|
# Dependencies
|
|
306
321
|
|
|
307
|
-
* **
|
|
322
|
+
* **Ruby** — **JRuby** or **CRuby** (both supported for NewBridge). Tested with **JRuby 10.1.1.0**
|
|
323
|
+
(+ **JDK 21**) and **CRuby 3.3.12**. Use `bin/galaaz-ruby` (honors `GALAAZ_RUBY`) or plain
|
|
324
|
+
`gem install galaaz` under the Ruby you choose.
|
|
308
325
|
* **GNU R** — `R` and `Rscript` on your `PATH` (tested with **4.3.3**), plus a C++ toolchain (`g++`, `make`) and the **Rcpp** package to compile the gatekeeper.
|
|
309
326
|
* **galaaz gem** — runtime dependency `msgpack` is pulled in by `gem install`.
|
|
310
327
|
* Optional: **Docker** — if you run R in a container (common on WSL2); see bootstrap below.
|
|
@@ -314,29 +331,34 @@ The native **gatekeeper** component under `ext/new_bridge` is built with `make`
|
|
|
314
331
|
|
|
315
332
|
The supported install is **`gem install` + compile the gatekeeper**. You do not need a git clone.
|
|
316
333
|
|
|
317
|
-
1. Install **JRuby
|
|
334
|
+
1. Install **JRuby** (and a compatible **JDK**) **or CRuby 3.3+**, plus **GNU R** (with `Rscript` and a C++ compiler).
|
|
318
335
|
2. In R, install **Rcpp**: `install.packages("Rcpp")`.
|
|
319
|
-
3. Install the gem: `jruby -S gem install galaaz`
|
|
336
|
+
3. Install the gem: `jruby -S gem install galaaz` (or `gem install galaaz` under CRuby).
|
|
320
337
|
4. Compile the native gatekeeper from the installed gem:
|
|
321
338
|
|
|
322
339
|
```
|
|
323
|
-
gem_dir="$(
|
|
340
|
+
gem_dir="$(ruby -e "puts Gem::Specification.find_by_name('galaaz').full_gem_path")"
|
|
341
|
+
# under JRuby: gem_dir="$(jruby -e "puts Gem::Specification.find_by_name('galaaz').full_gem_path")"
|
|
324
342
|
make -C "${gem_dir}/ext/new_bridge" all
|
|
325
343
|
```
|
|
326
344
|
|
|
327
|
-
5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on Java 9
|
|
345
|
+
5. Ensure **`R`** starts GNU R and can install packages (network access to CRAN when you first call `R.install_and_loads`). For **Apache Arrow** on **JRuby** (Java 9+), the child JVM needs `--add-opens=java.base/java.nio=ALL-UNNAMED` via **`JAVA_OPTS`** (from a checkout, `bin/galaaz-jruby` and `mise.toml` set this; a leading `jruby -J... -S bundle exec` does **not** pass `-J` to rspec). On **CRuby**, install Apache Arrow GLib (`libarrow-glib-dev` from the [Apache Arrow APT](https://arrow.apache.org/install/) repo) and `gem install red-arrow` matching `pkg-config --modversion arrow-glib`. Do not install the unrelated Rubygems package named `arrow`.
|
|
328
346
|
|
|
329
347
|
For **gKnit**, **knitr**, **rmarkdown**, and LaTeX (PDF output), install the corresponding R packages, **Pandoc**, and a TeX distribution if you need PDF; the repository includes helpers such as **`bin/install-tinytex`** where appropriate.
|
|
330
348
|
|
|
331
|
-
A **table of all `bin/` scripts** (bootstrap,
|
|
349
|
+
A **table of all `bin/` scripts** (bootstrap, Ruby launcher, gstudio, gknit, test runners, and which ones are legacy) is in the section **Command-line tools (`bin/`)** earlier in this manual.
|
|
332
350
|
|
|
333
351
|
### From a repository checkout (contributors)
|
|
334
352
|
|
|
335
|
-
1. Install **bundler** if needed, then run **`
|
|
353
|
+
1. Install **bundler** if needed, then run **`bundle install`** with your chosen Ruby
|
|
354
|
+
(`jruby -S bundle install` or CRuby `bundle install`) in the repository root.
|
|
336
355
|
2. Build the bridge native code: **`make -C ext/new_bridge all`** (or **`rake compile_gatekeeper`**).
|
|
337
|
-
3. Run scripts with **`bin/galaaz-
|
|
356
|
+
3. Run scripts with **`bin/galaaz-ruby`** (uses `ruby` on `PATH`; set **`GALAAZ_RUBY=jruby`** or
|
|
357
|
+
**`GALAAZ_RUBY=ruby`** to force an engine). Spec runners: **`bin/run_rspec`** /
|
|
358
|
+
**`bin/run_all_rspec`** (same `GALAAZ_RUBY` rule). **`bin/galaaz-jruby`** remains a thin
|
|
359
|
+
wrapper that forces JRuby.
|
|
338
360
|
|
|
339
|
-
Maintainers can prove a
|
|
361
|
+
A **gstudio** try image (JRuby + R + Galaaz already installed) is **`docker run --rm -it ghcr.io/rbotafogo/galaaz-try:gstudio`** (or **`./docker/try-gstudio/run.sh`** from a checkout). Maintainers can prove a RubyGems install on a throwaway Ubuntu machine (no repo inside the container) with **`./docker/cold-install/run.sh published-specs`** (JRuby) or **`./docker/cold-install-cruby/run.sh published-specs`** (CRuby).
|
|
340
362
|
|
|
341
363
|
## Windows + WSL2 (optional: Docker / R in a container)
|
|
342
364
|
|
|
@@ -404,11 +426,13 @@ WSL integration is enabled for the distro where Galaaz is installed.
|
|
|
404
426
|
|
|
405
427
|
> galaaz master_list:scatter_plot
|
|
406
428
|
|
|
407
|
-
# JRuby, multithreading, and the R bridge
|
|
429
|
+
# JRuby, CRuby, multithreading, and the R bridge
|
|
408
430
|
|
|
409
|
-
Galaaz 2.0
|
|
410
|
-
I/O-bound work (HTTP clients, database
|
|
411
|
-
|
|
431
|
+
Galaaz 2.0 supports **JRuby** and **CRuby** equally for NewBridge. On **JRuby**, your
|
|
432
|
+
application can use **real parallel threads** for I/O-bound work (HTTP clients, database
|
|
433
|
+
connections, message consumers, and so on). On **CRuby**, prefer multi-process scaling for
|
|
434
|
+
CPU-bound concurrency. R itself is still executed in a **single GNU R process** behind the
|
|
435
|
+
Galaaz bridge (use multiple R workers when you need more R throughput).
|
|
412
436
|
|
|
413
437
|
When several Ruby threads call into R at the same time, the bridge **serializes** those calls:
|
|
414
438
|
each request is matched to a reply using an internal per-call **queue**, so you do not need to
|
|
@@ -420,13 +444,15 @@ A practical pattern is:
|
|
|
420
444
|
|
|
421
445
|
1. Use threads (or a connection pool) to read from **multiple databases or shards** in parallel.
|
|
422
446
|
2. Merge the rows in Ruby under a `Mutex` if you collect into one structure.
|
|
423
|
-
3. Hand the merged table to R **once
|
|
424
|
-
|
|
447
|
+
3. Hand the merged table to R **once**: **`R::Arrow.from_ruby_batches`** (Stage A: copy into R) or
|
|
448
|
+
**`Galaaz::ArrowIpc.write` / `write_batches`** then **`R::Arrow.open_ipc`** (Stage B: IPC file;
|
|
449
|
+
only the path crosses the bridge). Then run dplyr in R.
|
|
425
450
|
|
|
426
451
|
A runnable sketch lives in
|
|
427
452
|
`examples/multithread_shards_to_r/shards_to_r.rb` (simulated shard queries; swap in your DB
|
|
428
453
|
driver). For concurrency tests on the bridge itself, see `specs/bridge_concurrent_spec.rb` and
|
|
429
|
-
`specs/arrow_from_ruby_batches_spec.rb`.
|
|
454
|
+
`specs/arrow_from_ruby_batches_spec.rb`. Stage B IPC tests: `specs/arrow_ipc_handoff_spec.rb`,
|
|
455
|
+
`specs/arrow_ipc_export_spec.rb`.
|
|
430
456
|
|
|
431
457
|
## Long-running R calls and a completion block
|
|
432
458
|
|
|
@@ -488,19 +514,22 @@ the outcome to storage, and notify the client (poll, WebSocket, Turbo Stream, et
|
|
|
488
514
|
Ruby pattern above is only to show **when** the result exists (inside the block, or after data
|
|
489
515
|
written there is observed elsewhere). Runnable specs live in **`new_bridge_specs/eval_r_async_spec.rb`**.
|
|
490
516
|
|
|
491
|
-
## Galaaz + Rails (
|
|
517
|
+
## Galaaz + Rails (R-on-Rails) integration baseline
|
|
492
518
|
|
|
493
|
-
This
|
|
494
|
-
The
|
|
519
|
+
This is the practical **R-on-Rails** starter: an R scientist’s analysis behind a small Rails
|
|
520
|
+
app. The baseline we used in WSL aimed at:
|
|
495
521
|
|
|
496
|
-
1. Rails boots under **JRuby
|
|
522
|
+
1. Rails boots under **JRuby or CRuby** (same bridge; see Installation).
|
|
497
523
|
2. Galaaz is loaded from a local checkout (before publishing to RubyGems).
|
|
498
|
-
3. A request path can execute **`R.eval(...)`** and return a result.
|
|
524
|
+
3. A request path can execute **`R.eval(...)`** (or `R.*`) and return a result.
|
|
499
525
|
|
|
500
|
-
|
|
526
|
+
You keep statistical work in R; Rails owns HTTP, sessions, and HTML. See **R-on-Rails: the
|
|
527
|
+
one-person app for R scientists** in the Introduction for the product framing.
|
|
501
528
|
|
|
502
|
-
|
|
503
|
-
|
|
529
|
+
### 1) Create the app with Ruby-friendly options
|
|
530
|
+
|
|
531
|
+
Rails defaults can pull gems that are awkward on some setups (for example sqlite native
|
|
532
|
+
extension paths on JRuby, or deployment extras you do not need). A minimal app avoids early friction:
|
|
504
533
|
|
|
505
534
|
```bash
|
|
506
535
|
cd /home/rbotafogo/desenv_linux
|
|
@@ -788,7 +817,7 @@ puts vec.map { |x| x + 2 }
|
|
|
788
817
|
|
|
789
818
|
This manual has been formatted using gKnit. gKnit uses knitr and R Markdown to knit
|
|
790
819
|
a document in Ruby or R and output it in any of the available formats for R Markdown.
|
|
791
|
-
gKnit runs with **JRuby**, **GNU R**, and Galaaz. In gKnit, Ruby variables are persisted between
|
|
820
|
+
gKnit runs with **JRuby or CRuby**, **GNU R**, and Galaaz. In gKnit, Ruby variables are persisted between
|
|
792
821
|
chunks, making it an ideal solution for literate programming. Also, since it is based
|
|
793
822
|
on Galaaz, Ruby chunks can have access to R variables and combining Ruby with R in one
|
|
794
823
|
document is natural.
|
|
@@ -4000,35 +4029,77 @@ ans = flights[:all, E.list(R[:arr_delay], R[:dep_delay])]
|
|
|
4000
4029
|
# Apache Arrow
|
|
4001
4030
|
|
|
4002
4031
|
[Apache Arrow](https://arrow.apache.org/) is a **columnar** in-memory format used heavily in R
|
|
4003
|
-
and Python for analytics.
|
|
4004
|
-
|
|
4005
|
-
|
|
4006
|
-
|
|
4007
|
-
|
|
4008
|
-
|
|
4009
|
-
|
|
4010
|
-
|
|
4011
|
-
|
|
4032
|
+
and Python for analytics. GNU R still runs in a **separate process**. Ruby does **not** hold a
|
|
4033
|
+
shared Arrow C++ table with R. Stages:
|
|
4034
|
+
|
|
4035
|
+
1. **Stage A (copy over the bridge):** Ruby row hashes → **`R::Arrow.from_ruby_batches`** builds
|
|
4036
|
+
an Arrow `Table` **inside GNU R**. You get a **proxy**.
|
|
4037
|
+
2. **Stage B1 (Ruby → R IPC file):** **`Galaaz::ArrowIpc.write`** / **`write_batches`** writes an
|
|
4038
|
+
Arrow IPC file (prefer **`/dev/shm`**); **`R::Arrow.open_ipc(path)`** opens it in R. Only the
|
|
4039
|
+
**path** crosses NewBridge. This is **mmap/IPC file handoff**, not a shared heap.
|
|
4040
|
+
3. **Stage B2 (R → Ruby IPC file):** **`R::Arrow.write_ipc(obj)`** writes uncompressed IPC; Ruby
|
|
4041
|
+
reads with **`Galaaz::ArrowIpc.read`** (column hash) or **`read_batches`** (row hashes). Call
|
|
4042
|
+
**`Galaaz::ArrowIpc.release(path)`** when finished.
|
|
4043
|
+
4. **Stage C (not shipped):** named shared-memory bus. Do not claim 0 ms shared RAM until then.
|
|
4044
|
+
See **`Documentation/ROADMAP_ARROW_RUBY_R.md`**.
|
|
4045
|
+
|
|
4046
|
+
After ingest, use R’s **`arrow`** / **`dplyr`** on the proxy (`group_by`, `summarise`, `collect`)
|
|
4047
|
+
and unbox only KPIs you need in Ruby.
|
|
4048
|
+
|
|
4049
|
+
**Optional Ruby backends for Stage B**
|
|
4050
|
+
|
|
4051
|
+
* **CRuby:** Apache **red-arrow** — `gem install red-arrow` pinned to the same major as
|
|
4052
|
+
`pkg-config --modversion arrow-glib`, plus system **Arrow GLib** (`libarrow-glib-dev` from the
|
|
4053
|
+
[Apache Arrow APT](https://arrow.apache.org/install/) repo). Do **not** install the unrelated
|
|
4054
|
+
legacy Rubygems package named `arrow`. `bundle exec` still sees a user-installed `red-arrow`
|
|
4055
|
+
via Galaaz’s load-path helper.
|
|
4056
|
+
* **JRuby:** Apache Arrow **Java** JARs — **`GALAAZ_ARROW_JARS`**, `~/arrow_jars`, or
|
|
4057
|
+
`jar-dependencies`. Export **`JAVA_OPTS=--add-opens=java.base/java.nio=ALL-UNNAMED`** on the
|
|
4058
|
+
**child** JVM (`bin/galaaz-jruby`, `mise.toml`). `jruby -J... -S bundle exec rspec` does **not**
|
|
4059
|
+
pass `-J` to rspec.
|
|
4060
|
+
|
|
4061
|
+
**R packages:** **`arrow`** and **`dplyr`**. B2 writes IPC with **`compression: 'uncompressed'`**
|
|
4062
|
+
so JRuby Arrow Java can read without extra compression JARs.
|
|
4063
|
+
|
|
4064
|
+
**Tests:** `specs/arrow_from_ruby_batches_spec.rb` (A);
|
|
4065
|
+
`specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb` (B, sync);
|
|
4066
|
+
`new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb` (B, async).
|
|
4067
|
+
|
|
4068
|
+
## `R::Arrow` and `Galaaz::ArrowIpc`
|
|
4069
|
+
|
|
4070
|
+
* **`R::Arrow.from_ruby_batches`** — Stage A ingest.
|
|
4071
|
+
* **`R::Arrow.open_ipc(path)`** — Stage B1: IPC file → R Table proxy.
|
|
4072
|
+
* **`R::Arrow.write_ipc(obj, path = nil)`** — Stage B2: R Table/tibble → IPC path (scratch if omitted).
|
|
4073
|
+
* **`Galaaz::ArrowIpc.write` / `write_batches` / `read` / `read_batches` / `allocate_path` / `release` / `available?`**
|
|
4074
|
+
* **`R::Arrow.table_from(df)`** — wrap an R `data.frame` / tibble as an Arrow table.
|
|
4075
|
+
* **`R::Arrow.read_feather` / `write_feather`**, **`read_parquet`**, **`dataset(path)`** — file and
|
|
4076
|
+
dataset IO on paths visible to R.
|
|
4012
4077
|
|
|
4013
|
-
|
|
4014
|
-
**`bin/galaaz-jruby`** (or the same JVM flags as in **`docs/testing.md`**) so the Arrow JNI stack is
|
|
4015
|
-
available.
|
|
4078
|
+
## Example: Stage B round-trip (IPC file)
|
|
4016
4079
|
|
|
4017
|
-
|
|
4080
|
+
Requires `Galaaz::ArrowIpc.available?` (red-arrow or Arrow JARs) and R **`arrow`**. Not knitted
|
|
4081
|
+
below so a machine without the optional backend still builds this manual.
|
|
4018
4082
|
|
|
4019
|
-
|
|
4083
|
+
```ruby
|
|
4084
|
+
path = Galaaz::ArrowIpc.write(id: [1, 2, 3], grp: %w[a a b], value: [1.0, 2.0, 3.5])
|
|
4085
|
+
tbl = R::Arrow.open_ipc(path)
|
|
4086
|
+
Galaaz::ArrowIpc.release(path)
|
|
4020
4087
|
|
|
4021
|
-
|
|
4022
|
-
|
|
4023
|
-
|
|
4088
|
+
summed = R.dplyr___summarise(R.dplyr___group_by(tbl, :grp), total: E.sum(:value))
|
|
4089
|
+
out_path = R::Arrow.write_ipc(summed)
|
|
4090
|
+
rows = Galaaz::ArrowIpc.read_batches(out_path)
|
|
4091
|
+
Galaaz::ArrowIpc.release(out_path)
|
|
4092
|
+
# rows => [{:grp=>"a", :total=>3.0}, {:grp=>"b", :total=>3.5}] (illustrative)
|
|
4093
|
+
```
|
|
4024
4094
|
|
|
4025
4095
|
## Example: many Ruby rows → Arrow in R → grouped statistics
|
|
4026
4096
|
|
|
4027
4097
|
The repository test **`slow-specs/arrow_large_pipeline_spec.rb`** builds **200k rows** in parallel
|
|
4028
|
-
(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches
|
|
4029
|
-
**dplyr** group summaries match a Ruby reference calculation. The same logic appears
|
|
4030
|
-
**smaller scale** so this manual can knit quickly; increase `thread_count` and
|
|
4031
|
-
when experimenting locally.
|
|
4098
|
+
(eight threads × 25,000 rows), pushes them through **`R::Arrow.from_ruby_batches`** (Stage A), then
|
|
4099
|
+
checks that **dplyr** group summaries match a Ruby reference calculation. The same logic appears
|
|
4100
|
+
below at a **smaller scale** so this manual can knit quickly; increase `thread_count` and
|
|
4101
|
+
`rows_per_thread` when experimenting locally. For the same ingest **without** copying every cell
|
|
4102
|
+
over NewBridge, use Stage B (`write_batches` + `open_ipc`) instead of `from_ruby_batches`.
|
|
4032
4103
|
|
|
4033
4104
|
|
|
4034
4105
|
``` ruby
|
|
@@ -4094,10 +4165,10 @@ end
|
|
|
4094
4165
|
```
|
|
4095
4166
|
|
|
4096
4167
|
**What to notice:** (1) Ruby only sees **`Hash`** rows and Ruby **`Thread`** objects; (2) a single
|
|
4097
|
-
**`from_ruby_batches`** call
|
|
4168
|
+
**`from_ruby_batches`** call **copies** those columns into an Arrow table in R; (3) **`dplyr___group_by`** /
|
|
4098
4169
|
**`dplyr___summarise`** / **`dplyr___collect`** mirror **`dplyr::group_by`** /
|
|
4099
4170
|
**`dplyr::summarise`** / **`dplyr::collect`** on an Arrow-backed table. For a lighter test, see
|
|
4100
|
-
**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size benchmark, run
|
|
4171
|
+
**`specs/arrow_from_ruby_batches_spec.rb`**; for the full-size Stage A benchmark, run
|
|
4101
4172
|
**`bin/run_slow_rspec slow-specs/arrow_large_pipeline_spec.rb`**.
|
|
4102
4173
|
|
|
4103
4174
|
# Bioconductor and DESeq2
|
|
@@ -4235,8 +4306,9 @@ Practical tips:
|
|
|
4235
4306
|
glue.
|
|
4236
4307
|
* **Reuse one process**: running many short scripts cold-starts Ruby, the JVM, and R each time;
|
|
4237
4308
|
a long-lived process or repeated calls in one run amortize setup (see benchmarks below).
|
|
4238
|
-
* **Batch data**: merge shards in Ruby, then
|
|
4239
|
-
|
|
4309
|
+
* **Batch data**: merge shards in Ruby, then **`R::Arrow.from_ruby_batches`** (Stage A) or
|
|
4310
|
+
**`Galaaz::ArrowIpc`** + **`R::Arrow.open_ipc`** (Stage B) instead of millions of tiny R calls.
|
|
4311
|
+
When Ruby needs a bulky result table back, **`R::Arrow.write_ipc`** + **`Galaaz::ArrowIpc.read_batches`**.
|
|
4240
4312
|
|
|
4241
4313
|
For measured discussion (including DESeq2-style workloads and warm comparisons), see
|
|
4242
4314
|
**`docs/performance.md`** and **`docs/deseq2_airway_benchmark.md`** in the Galaaz repository.
|
|
@@ -4923,8 +4995,8 @@ arguments.
|
|
|
4923
4995
|
|
|
4924
4996
|
* Fork it
|
|
4925
4997
|
* Create your feature branch (`git checkout -b my-new-feature`)
|
|
4926
|
-
* Write tests — use **`bin/run_rspec`** or **`bin/run_all_rspec`**
|
|
4927
|
-
the load path match **`docs/testing.md`**
|
|
4998
|
+
* Write tests — use **`bin/run_rspec`** or **`bin/run_all_rspec`** (JRuby or CRuby via
|
|
4999
|
+
**`GALAAZ_RUBY`**) so JVM flags and the load path match **`docs/testing.md`**
|
|
4928
5000
|
* Commit your changes (`git commit -am 'Add some feature'`)
|
|
4929
5001
|
* Push to the branch (`git push origin my-new-feature`)
|
|
4930
5002
|
* Open a pull request
|