galaaz 2.0.0 → 2.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +43 -5
- data/README.md +200 -128
- data/Rakefile +17 -10
- data/bin/galaaz +10 -2
- data/bin/galaaz-jruby +3 -6
- data/bin/galaaz-ruby +16 -0
- data/bin/galaaz_jruby_env.inc.sh +19 -3
- data/bin/galaaz_ruby_env.inc.sh +38 -0
- data/bin/gbookdown +2 -3
- data/bin/gknit +2 -3
- data/bin/gstudio +4 -4
- data/bin/run_all_rspec +6 -5
- data/bin/run_example +10 -7
- data/bin/run_old_rspec +6 -5
- data/bin/run_rspec +9 -8
- data/bin/run_slow_rspec +6 -5
- data/blogs/R-on-Rails-Planning-Document.md +79 -112
- data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/midwest_rb.png +0 -0
- data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/scatter_plot_rb.png +0 -0
- data/blogs/gknit/gknit_files/figure-gfm/bubble-1.png +0 -0
- data/blogs/gknit/gknit_files/figure-gfm/diverging_bar.png +0 -0
- data/blogs/manual/manual.Rmd +22 -19
- data/blogs/manual/manual.md +200 -128
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/dose_len.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_delivery.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_dose.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color2.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_decorations.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_jitter.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_points.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_box_plot.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_violin_plot.png +0 -0
- data/blogs/ruby_plot/ruby_plot_files/figure-gfm/violin_with_jitter.png +0 -0
- data/blogs/test/test.md +10 -0
- data/examples/50Plots_MasterList/ScatterPlot.rb +2 -1
- data/examples/50Plots_MasterList/scatter_plot.rb +1 -0
- data/examples/bioconductor_deseq2_airway/bench_galaaz_three_same_process.rb +1 -0
- data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz.rb +1 -0
- data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz_optimized.rb +1 -0
- data/examples/islr/all.rb +1 -0
- data/examples/islr/ch2.spec.rb +1 -0
- data/examples/islr/ch3.spec.rb +1 -0
- data/examples/islr/ch3_boston.rb +1 -0
- data/examples/islr/ch3_multiple_regression.rb +1 -0
- data/examples/islr/ch6.spec.rb +1 -0
- data/examples/islr/x_y_rnorm.jpg +0 -0
- data/examples/misc/ggplot.rb +2 -1
- data/examples/misc/moneyball.rb +1 -0
- data/examples/misc/subsetting.rb +1 -0
- data/examples/multithread_shards_to_r/shards_to_r.rb +1 -0
- data/examples/sthda_ggplot/all.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/density_gg.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_area.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_density.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_dotplot.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_freqpoly.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/geom_histogram.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/histogram_density.rb +1 -0
- data/examples/sthda_ggplot/one_variable_continuous/stat.rb +1 -0
- data/examples/sthda_ggplot/one_variable_discrete/bar.rb +1 -0
- data/examples/sthda_ggplot/qplots/box_violin_dot.rb +1 -0
- data/examples/sthda_ggplot/qplots/scatter_plots.rb +1 -0
- data/examples/sthda_ggplot/scatter_gg.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_bin2d.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_density2d.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_hex.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/geom_point.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/geom_smooth.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_cont/misc.rb +1 -0
- data/examples/sthda_ggplot/two_variables_cont_function/geom_area.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_bar.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_boxplot.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_dotplot.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_jitter.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_line.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_cont/geom_violin.rb +1 -0
- data/examples/sthda_ggplot/two_variables_disc_disc/geom_jitter.rb +1 -0
- data/examples/sthda_ggplot/two_variables_error/geom_crossbar.rb +1 -0
- data/lib/R_interface/r_arrow.rb +37 -0
- data/lib/R_interface/rvector.rb +15 -0
- data/lib/galaaz/arrow_ipc/java_arrow_backend.rb +250 -0
- data/lib/galaaz/arrow_ipc/red_arrow_backend.rb +126 -0
- data/lib/galaaz/arrow_ipc.rb +167 -0
- data/lib/galaaz/cli.rb +531 -0
- data/lib/galaaz.rb +6 -0
- data/lib/galaaz_jruby.rb +17 -2
- data/lib/galaaz_ruby.rb +34 -0
- data/lib/new_bridge/envelope.rb +2 -2
- data/new_bridge_specs/arrow_ipc_async_spec.rb +90 -0
- data/new_bridge_specs/arrow_ipc_export_async_spec.rb +68 -0
- data/r_requires/arrow.txt +3 -0
- data/r_requires/knit-extras.txt +4 -0
- data/r_requires/knit.txt +4 -0
- data/specs/arrow_ipc_export_spec.rb +62 -0
- data/specs/arrow_ipc_handoff_spec.rb +97 -0
- data/specs/gknit_include_engine_spec.rb +2 -2
- data/specs/gknit_install_timeout_report_spec.rb +2 -2
- data/specs/gknit_internal_error_report_spec.rb +2 -2
- data/specs/gknit_vector_map_output_spec.rb +2 -2
- data/version.rb +1 -1
- metadata +50 -13
|
@@ -68,9 +68,9 @@ Rails never went away—it matured. Several factors make Rails compelling again:
|
|
|
68
68
|
|
|
69
69
|
1. **Rails (Ruby)**: Handles web requests, authentication, authorization, database operations, background jobs, email, APIs, caching, sessions, and orchestration
|
|
70
70
|
2. **R**: Handles statistical modeling, machine learning, visualization, and data analysis
|
|
71
|
-
3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—
|
|
71
|
+
3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—proxies for analytics in R, Apache Arrow for bulky tables (Stage A copy, Stage B IPC/mmap file; shared-heap zero-copy is Stage C and not shipped)
|
|
72
72
|
|
|
73
|
-
**The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications.
|
|
73
|
+
**The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications. For large tables, prefer Arrow **IPC files** (path across the bridge, bytes on disk or `/dev/shm`) over CSV/JSON serialization. Do **not** claim Ruby and R share one physical Arrow heap until Stage C ships.
|
|
74
74
|
|
|
75
75
|
---
|
|
76
76
|
|
|
@@ -86,25 +86,24 @@ Earlier versions of Galaaz explored GraalVM's polyglot capabilities with FastR (
|
|
|
86
86
|
├─────────────────────────────────────────────────────────────────┤
|
|
87
87
|
│ │
|
|
88
88
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
|
|
89
|
-
│ │
|
|
90
|
-
│ │
|
|
91
|
-
│ └──────────────┘
|
|
92
|
-
│ │ │ │
|
|
89
|
+
│ │ JRuby/CRuby │ ←─────→ │ Galaaz Bridge│ ←─────→ │ GNU R │ │
|
|
90
|
+
│ │ │ path + │ (NewBridge) │ C/R │ Process │ │
|
|
91
|
+
│ └──────────────┘ cmds └──────────────┘ └──────────┘ │
|
|
93
92
|
│ │ │ │
|
|
94
93
|
│ ▼ ▼ │
|
|
95
94
|
│ ┌──────────────┐ ┌──────────┐│
|
|
96
95
|
│ │ Rails App │ │ Arrow ││
|
|
97
|
-
│ │
|
|
98
|
-
│ │
|
|
96
|
+
│ │ (threads / │ bulky tables: IPC file │ Table in ││
|
|
97
|
+
│ │ processes) │ ──── mmap / /dev/shm (Stage B) ──│ R (proxy)││
|
|
99
98
|
│ └──────────────┘ └──────────┘│
|
|
100
99
|
│ │
|
|
101
100
|
└─────────────────────────────────────────────────────────────────┘
|
|
102
101
|
```
|
|
103
102
|
|
|
104
103
|
**How it works:**
|
|
105
|
-
1. **JRuby** runs
|
|
106
|
-
2. **Galaaz Bridge** communicates with a GNU R process
|
|
107
|
-
3. **Apache Arrow** (optional)
|
|
104
|
+
1. **JRuby or CRuby** runs Rails and application code (JRuby for true JVM threads; CRuby for MRI). Both talk the same NewBridge protocol.
|
|
105
|
+
2. **Galaaz Bridge** communicates with a GNU R process (NewBridge / gatekeeper)—commands and small values, not bulk bytes when you use Stage B.
|
|
106
|
+
3. **Apache Arrow** (optional): **Stage A** copies Ruby batches into an R-side Arrow Table (`R::Arrow.from_ruby_batches`). **Stage B** writes an Arrow IPC file; only the **path** crosses the bridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). **Stage C** (shared-heap zero-copy) is not shipped—see `Documentation/ROADMAP_ARROW_RUBY_R.md`.
|
|
108
107
|
4. **GNU R** is the actual R interpreter—full compatibility with all R packages (ggplot2, dplyr, Bioconductor, etc.)
|
|
109
108
|
|
|
110
109
|
**Why this is better than FastR/GraalVM:**
|
|
@@ -448,13 +447,22 @@ end
|
|
|
448
447
|
|
|
449
448
|
---
|
|
450
449
|
|
|
451
|
-
### Section 6: Apache Arrow
|
|
450
|
+
### Section 6: Apache Arrow — bulky tables without CSV/JSON
|
|
451
|
+
|
|
452
|
+
**Key Message**: Traditional bridges often serialize through text (CSV, JSON). Galaaz uses Apache Arrow in **two shipped modes**. Neither is a shared Ruby+R heap. Maintainer detail: `Documentation/ROADMAP_ARROW_RUBY_R.md`. User manual: `# Apache Arrow` in `README.md` / `blogs/manual/manual.md`.
|
|
452
453
|
|
|
453
|
-
|
|
454
|
+
| Stage | API | What actually happens |
|
|
455
|
+
|-------|-----|------------------------|
|
|
456
|
+
| **A** (shipped) | `R::Arrow.from_ruby_batches` / `table_from` | Columns are **copied** into GNU R; Ruby holds a **proxy**. dplyr then runs in R. |
|
|
457
|
+
| **B1** (shipped) | `Galaaz::ArrowIpc.write` → `R::Arrow.open_ipc(path)` | Ruby writes an Arrow **IPC file** (prefer `/dev/shm`); NewBridge carries only the **path**. R memory-maps / reads the file into an Arrow Table. |
|
|
458
|
+
| **B2** (shipped) | `R::Arrow.write_ipc` → `Galaaz::ArrowIpc.read` / `read_batches` | R writes IPC (uncompressed so JRuby Arrow Java can read); Ruby reads columns or row hashes for DB/API. |
|
|
459
|
+
| **C** (future) | not shipped | Named shared segment; do **not** claim 0 ms shared RAM until this exists and is measured. |
|
|
460
|
+
|
|
461
|
+
**Writers/readers (Stage B):** CRuby uses **red-arrow** (`gem install red-arrow` matching `pkg-config --modversion arrow-glib`, plus Apache Arrow APT / `libarrow-glib-dev`). JRuby uses **Apache Arrow Java** JARs (`GALAAZ_ARROW_JARS` or `~/arrow_jars`) and `JAVA_OPTS` `--add-opens=java.base/java.nio=ALL-UNNAMED` on the **child** JVM (`bin/galaaz-jruby` / `mise.toml`). Do not install the unrelated Rubygems gem named `arrow`.
|
|
454
462
|
|
|
455
463
|
**The Problem: Data Transfer Overhead**
|
|
456
464
|
|
|
457
|
-
In traditional polyglot systems (rpy2, reticulate), passing data between languages
|
|
465
|
+
In traditional polyglot systems (rpy2, reticulate), passing data between languages often means a text round-trip:
|
|
458
466
|
|
|
459
467
|
```python
|
|
460
468
|
# Python → R via JSON/CSV serialization (SLOW)
|
|
@@ -470,36 +478,31 @@ ro.globalenv['r_df'] = pandas2ri.py2rpy(df)
|
|
|
470
478
|
This approach:
|
|
471
479
|
- Copies all data through a text format
|
|
472
480
|
- Loses type information (factors become strings, dates become strings)
|
|
473
|
-
- Consumes
|
|
474
|
-
- Becomes a bottleneck at
|
|
481
|
+
- Consumes extra memory during transfer
|
|
482
|
+
- Becomes a bottleneck at large row counts
|
|
475
483
|
|
|
476
|
-
**
|
|
484
|
+
**Stage A: copy into R, then Remote Control**
|
|
477
485
|
|
|
478
|
-
|
|
486
|
+
Assemble row hashes in Ruby (threads OK on JRuby), then one ingest. Analytics stay on the R proxy; unbox KPIs only.
|
|
479
487
|
|
|
480
488
|
```ruby
|
|
481
489
|
require 'galaaz'
|
|
482
490
|
|
|
483
|
-
# 1. Build data in Ruby—using parallel threads for multi-source ingestion
|
|
484
491
|
batches = []
|
|
485
492
|
mutex = Mutex.new
|
|
486
493
|
threads = []
|
|
487
494
|
|
|
488
|
-
# Simulate parallel data ingestion from multiple sources
|
|
489
495
|
[0, 1, 2, 3].each do |tid|
|
|
490
496
|
threads << Thread.new do
|
|
491
|
-
|
|
492
|
-
local_data = fetch_from_source(tid) # Returns array of hashes
|
|
497
|
+
local_data = fetch_from_source(tid) # array of hashes
|
|
493
498
|
mutex.synchronize { batches << local_data }
|
|
494
499
|
end
|
|
495
500
|
end
|
|
496
501
|
threads.each(&:join)
|
|
497
502
|
|
|
498
|
-
#
|
|
499
|
-
# This is the key: data moves to R without serialization!
|
|
503
|
+
# Copy into an Arrow Table *inside GNU R* (not shared heap with Ruby)
|
|
500
504
|
table = R::Arrow.from_ruby_batches(batches)
|
|
501
505
|
|
|
502
|
-
# 3. Use R's dplyr directly on the Arrow table
|
|
503
506
|
R.install_and_loads('dplyr', 'arrow')
|
|
504
507
|
|
|
505
508
|
grouped = R.dplyr___group_by(table, :region)
|
|
@@ -510,57 +513,59 @@ summarised = R.dplyr___summarise(
|
|
|
510
513
|
total: E.sum(:value)
|
|
511
514
|
)
|
|
512
515
|
results = R.dplyr___collect(summarised)
|
|
513
|
-
|
|
514
516
|
puts results
|
|
515
|
-
#> # A tibble: 4 × 4
|
|
516
|
-
#> region count avg_value total
|
|
517
|
-
#> <chr> <int> <dbl> <dbl>
|
|
518
|
-
#> 1 North 25000 5.5 137500
|
|
519
|
-
#> 2 South 25000 5.5 137500
|
|
520
|
-
#> 3 East 25000 5.5 137500
|
|
521
|
-
#> 4 West 25000 5.5 137500
|
|
522
517
|
```
|
|
523
518
|
|
|
524
|
-
**
|
|
519
|
+
**Stage B: IPC file; path on the bridge**
|
|
520
|
+
|
|
521
|
+
Use this when the table is large enough that copying every cell over NewBridge/MsgPack is the wrong tax. Scratch files prefer `/dev/shm`.
|
|
522
|
+
|
|
523
|
+
```ruby
|
|
524
|
+
require 'galaaz'
|
|
525
|
+
|
|
526
|
+
# B1: Ruby → file → R
|
|
527
|
+
path = Galaaz::ArrowIpc.write_batches(batches) # or .write(col => array, ...)
|
|
528
|
+
tbl = R::Arrow.open_ipc(path)
|
|
529
|
+
Galaaz::ArrowIpc.release(path) # unlink after R has opened (default B1 lifetime)
|
|
530
|
+
|
|
531
|
+
grouped = R.dplyr___group_by(tbl, :region)
|
|
532
|
+
summarised = R.dplyr___summarise(grouped, total: E.sum(:value))
|
|
533
|
+
|
|
534
|
+
# B2: R → file → Ruby (e.g. rows for ActiveRecord / JSON)
|
|
535
|
+
out_path = R::Arrow.write_ipc(summarised)
|
|
536
|
+
rows = Galaaz::ArrowIpc.read_batches(out_path)
|
|
537
|
+
Galaaz::ArrowIpc.release(out_path)
|
|
538
|
+
|
|
539
|
+
render json: { analytics: rows }
|
|
540
|
+
```
|
|
541
|
+
|
|
542
|
+
**Why this matters (honest):**
|
|
525
543
|
|
|
526
|
-
- **
|
|
527
|
-
- **Type
|
|
528
|
-
- **
|
|
529
|
-
- **
|
|
530
|
-
- **
|
|
544
|
+
- **Columnar Arrow**, not CSV, for the bulky payload
|
|
545
|
+
- **Type-preserving** numeric/string columns on the B1/B2 contract (int32/int64, float64, utf8; nulls kept)
|
|
546
|
+
- **Stage A** still copies; **Stage B** still materializes a file R (or Ruby) then reads—R may mmap the IPC file
|
|
547
|
+
- **Not** “Ruby and R map the same live buffer” until Stage C
|
|
548
|
+
- **Scale**: Stage A pipeline in `slow-specs/arrow_large_pipeline_spec.rb` (200K+ rows); Stage B specs in `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, and the matching `new_bridge_specs/arrow_ipc_*_async_spec.rb`
|
|
549
|
+
- **Parquet/Feather/dataset** stay R-side file APIs (`R::Arrow.write_parquet`, `dataset`)
|
|
531
550
|
|
|
532
|
-
**Real-
|
|
551
|
+
**Real-world sketch: shards in Ruby, analytics in R**
|
|
533
552
|
|
|
534
553
|
```ruby
|
|
535
|
-
#
|
|
536
|
-
# then analyze in R with zero-copy transfer
|
|
554
|
+
# Parallel ingest in Ruby, then one Arrow handoff (A or B), then dplyr in R.
|
|
537
555
|
|
|
538
556
|
require 'galaaz'
|
|
539
557
|
require 'active_record'
|
|
540
558
|
|
|
541
|
-
# Connect to multiple database shards
|
|
542
|
-
SHARDS = ['shard_1', 'shard_2', 'shard_3', 'shard_4'].map do |shard_name|
|
|
543
|
-
ActiveRecord::Base.establish_connection(
|
|
544
|
-
adapter: 'postgresql',
|
|
545
|
-
host: "#{shard_name}.db.internal",
|
|
546
|
-
database: 'analytics'
|
|
547
|
-
)
|
|
548
|
-
end
|
|
549
|
-
|
|
550
|
-
# Parallel data collection from all shards
|
|
551
559
|
batches = []
|
|
552
560
|
mutex = Mutex.new
|
|
553
561
|
|
|
554
562
|
SHARDS.each do |shard|
|
|
555
563
|
Thread.new do
|
|
556
|
-
# Query this shard
|
|
557
564
|
records = shard.connection.select_all(<<-SQL).cast_values
|
|
558
565
|
SELECT user_id, region, event_type, value, created_at
|
|
559
566
|
FROM events
|
|
560
567
|
WHERE created_at > NOW() - INTERVAL '7 days'
|
|
561
568
|
SQL
|
|
562
|
-
|
|
563
|
-
# Convert to array of hashes for Arrow
|
|
564
569
|
batch = records.map do |row|
|
|
565
570
|
{
|
|
566
571
|
user_id: row[0],
|
|
@@ -570,60 +575,37 @@ SHARDS.each do |shard|
|
|
|
570
575
|
created_at: row[4]
|
|
571
576
|
}
|
|
572
577
|
end
|
|
573
|
-
|
|
574
578
|
mutex.synchronize { batches << batch }
|
|
575
579
|
end
|
|
576
580
|
end
|
|
577
581
|
|
|
578
|
-
|
|
579
|
-
table = R::Arrow.from_ruby_batches(batches)
|
|
582
|
+
R.install_and_loads('dplyr', 'arrow')
|
|
580
583
|
|
|
581
|
-
#
|
|
582
|
-
|
|
584
|
+
# Prefer B1 when the merged table is large:
|
|
585
|
+
path = Galaaz::ArrowIpc.write_batches(batches)
|
|
586
|
+
table = R::Arrow.open_ipc(path)
|
|
587
|
+
Galaaz::ArrowIpc.release(path)
|
|
583
588
|
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
.dplyr___group_by(:region, :event_type) \
|
|
588
|
-
.dplyr___summarise(
|
|
589
|
+
analysis = R.dplyr___collect(
|
|
590
|
+
R.dplyr___summarise(
|
|
591
|
+
R.dplyr___group_by(table, :region, :event_type),
|
|
589
592
|
count: E.n(),
|
|
590
|
-
total_value: E.sum(:value)
|
|
591
|
-
avg_value: E.mean(:value),
|
|
592
|
-
unique_users: E.n_distinct(:user_id)
|
|
593
|
+
total_value: E.sum(:value)
|
|
593
594
|
)
|
|
595
|
+
)
|
|
594
596
|
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
results_hash = results.to_ruby
|
|
600
|
-
render json: { analytics: results_hash }
|
|
597
|
+
out_path = R::Arrow.write_ipc(analysis)
|
|
598
|
+
rows = Galaaz::ArrowIpc.read_batches(out_path)
|
|
599
|
+
Galaaz::ArrowIpc.release(out_path)
|
|
600
|
+
render json: { analytics: rows }
|
|
601
601
|
```
|
|
602
602
|
|
|
603
|
-
**Parquet and Feather
|
|
603
|
+
**Parquet and Feather (R-visible paths):**
|
|
604
604
|
|
|
605
605
|
```ruby
|
|
606
|
-
# Save Ruby data as Parquet (columnar, compressed)
|
|
607
|
-
# Then read directly into R without parsing
|
|
608
|
-
|
|
609
|
-
# 1. Build large dataset in Ruby
|
|
610
|
-
data = (1..1_000_000).map do |i|
|
|
611
|
-
{
|
|
612
|
-
id: i,
|
|
613
|
-
category: ["A", "B", "C", "D"][i % 4],
|
|
614
|
-
value: rand * 100,
|
|
615
|
-
timestamp: Time.now - (i % 86400)
|
|
616
|
-
}
|
|
617
|
-
end
|
|
618
|
-
|
|
619
|
-
# 2. Convert to R data.frame, then write as Parquet
|
|
620
606
|
df = R.data__frame(data)
|
|
621
607
|
R::Arrow.write_parquet(df, '/data/events.parquet')
|
|
622
|
-
|
|
623
|
-
# 3. Later, read directly into R as a dataset (lazy, memory-mapped)
|
|
624
608
|
dataset = R::Arrow.dataset('/data/events.parquet')
|
|
625
|
-
|
|
626
|
-
# Query without loading entire file
|
|
627
609
|
summary = R.dplyr___collect(
|
|
628
610
|
dataset \
|
|
629
611
|
.dplyr___filter(R[:value] > 50) \
|
|
@@ -632,28 +614,13 @@ summary = R.dplyr___collect(
|
|
|
632
614
|
)
|
|
633
615
|
```
|
|
634
616
|
|
|
635
|
-
**
|
|
636
|
-
|
|
637
|
-
```ruby
|
|
638
|
-
# Test: 8 threads × 25,000 rows = 200,000 rows
|
|
639
|
-
# With weighted aggregations in R
|
|
640
|
-
thread_count = 8
|
|
641
|
-
rows_per_thread = 25_000
|
|
642
|
-
group_count = 10
|
|
643
|
-
|
|
644
|
-
# Ruby parallel batch construction... 8 threads
|
|
645
|
-
# Arrow table creation from batches → instant
|
|
646
|
-
# R dplyr group_by + summarise → native speed
|
|
647
|
-
# Results verified accurate against Ruby reference implementation
|
|
648
|
-
```
|
|
649
|
-
|
|
650
|
-
**Key Insight**: With Arrow, the boundary between Ruby and R disappears for data. You can build data pipelines in Ruby (with its superior concurrency and database libraries) and analyze in R (with its statistical ecosystem)—with **zero overhead** at the language boundary.
|
|
617
|
+
**Key Insight**: Build pipelines in Ruby (concurrency, ActiveRecord); run statistics in R. Pay **one** bulky handoff (copy or IPC file), then **move commands** on proxies. That is the production story today—not a shared Arrow heap.
|
|
651
618
|
|
|
652
619
|
---
|
|
653
620
|
|
|
654
621
|
### Section 7: Parallelism and Process Orchestration
|
|
655
622
|
|
|
656
|
-
**Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby
|
|
623
|
+
**Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby—Arrow for bulky ingest/export, proxies for analytics per worker.
|
|
657
624
|
|
|
658
625
|
**The Architecture**:
|
|
659
626
|
```
|
|
@@ -892,7 +859,7 @@ Galaaz 2.0 represents the maturation of the Ruby-R bridge:
|
|
|
892
859
|
- **JRuby + GNU R architecture**: Moved from experimental GraalVM/FastR to battle-tested JRuby and standard GNU R for full package compatibility and production stability
|
|
893
860
|
- **New bridge architecture**: More robust, faster, better error handling
|
|
894
861
|
- **Process management**: Built-in support for R process pools
|
|
895
|
-
- **Apache Arrow
|
|
862
|
+
- **Apache Arrow**: Stage A copy (`from_ruby_batches`) and Stage B IPC/mmap (`Galaaz::ArrowIpc`, `open_ipc` / `write_ipc`). Shared-heap zero-copy is Stage C. Build pipelines in Ruby, analyze in R, unbox KPIs.
|
|
896
863
|
- **gKnit improvements**: Better R Markdown integration
|
|
897
864
|
- **Rails integration**: First-class support for Rails patterns
|
|
898
865
|
|
|
@@ -907,7 +874,7 @@ For the full blog post, include references to:
|
|
|
907
874
|
- **NSE/dplyr**: `blogs/nse_dplyr/nse_dplyr.md`
|
|
908
875
|
- **Object-Oriented**: `blogs/oh_my/oh_my.md`
|
|
909
876
|
- **Plotting Tutorial**: `blogs/ruby_plot/ruby_plot.md`
|
|
910
|
-
- **Apache Arrow**: `specs/
|
|
877
|
+
- **Apache Arrow**: `Documentation/ROADMAP_ARROW_RUBY_R.md`; `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, `new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb`; Stage A: `specs/arrow_from_ruby_batches_spec.rb`, `slow-specs/arrow_large_pipeline_spec.rb`
|
|
911
878
|
- **Specs**: `specs/r_nse.spec.rb`, `specs/r_vector_functions.spec.rb`
|
|
912
879
|
|
|
913
880
|
---
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
data/blogs/manual/manual.Rmd
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: "Galaaz Manual"
|
|
3
|
-
subtitle: "
|
|
3
|
+
subtitle: "R-on-Rails: GNU R meets Ruby for the web"
|
|
4
4
|
author: "Rodrigo Botafogo"
|
|
5
|
-
tags: [Galaaz, Ruby, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
|
|
5
|
+
tags: [Galaaz, "R-on-Rails", Ruby, Rails, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
|
|
6
6
|
date: "2026"
|
|
7
7
|
bibliography: "../../examples/Bibliography/stats.bib"
|
|
8
8
|
output:
|
|
@@ -114,10 +114,11 @@ The Galaaz repository ships many helpers under **`bin/`**. When working from a *
|
|
|
114
114
|
them as **`bin/<name>`** from the project root (or `./bin/<name>`). If you install the **gem**,
|
|
115
115
|
only a subset is guaranteed on your `PATH` (see the gemspec: **`galaaz`**, **`gstudio`**, **`gknit`**, **`grun`**, **`gknit-draft`**); for development and CI, prefer the **`bin/`** copies so JVM flags and paths stay correct.
|
|
116
116
|
|
|
117
|
-
Below, **current (Galaaz 2.0 + JRuby + GNU R)** means the tool
|
|
118
|
-
**`
|
|
119
|
-
|
|
120
|
-
|
|
117
|
+
Below, **current (Galaaz 2.0 + JRuby or CRuby + GNU R)** means the tool uses **`bin/galaaz-ruby`**
|
|
118
|
+
/ **`GALAAZ_RUBY`** (default: `ruby` on `PATH`) and applies JVM flags only when the interpreter is
|
|
119
|
+
JRuby (`bin/galaaz_jruby_env.inc.sh` / `lib/galaaz_jruby.rb`). **`bin/galaaz-jruby`** forces JRuby.
|
|
120
|
+
**Legacy** means the script still targets **GraalVM** polyglot Ruby / FastR-era invocation and is
|
|
121
|
+
**not** expected to work on a typical JRuby or CRuby NewBridge setup.
|
|
121
122
|
|
|
122
123
|
**Table layout:** names in the first column are **`bin/`** filenames (run as `bin/<name>` from the repo root). Long options and examples sit **outside** the tables so PDF columns stay readable.
|
|
123
124
|
|
|
@@ -233,7 +234,7 @@ df_ot <- data.frame(
|
|
|
233
234
|
bin_tbl(df_ot)
|
|
234
235
|
```
|
|
235
236
|
|
|
236
|
-
For day-to-day **2.0** use, rely on **`bin/galaaz-jruby
|
|
237
|
+
For day-to-day **2.0** use, rely on **`bin/galaaz-ruby`** (or **`bin/galaaz-jruby`** when you want to force JRuby), **`bin/gstudio`**, **`bin/gknit`**, **`bin/run_example`**, **`bin/run_rspec`** / **`bin/run_all_rspec`**, and **`bin/galaaz-bootstrap`** on WSL when using Dockerized R. Treat **`grun`**, **`gknit_Rscript`**, and the polyglot **`ruby`** invocation in **`gknit-draft`** as **legacy** until they are ported to the same launcher path as **`gknit`**.
|
|
237
238
|
|
|
238
239
|
# System Compatibility
|
|
239
240
|
|
|
@@ -279,7 +280,7 @@ A **table of all `bin/` scripts** (bootstrap, JRuby wrapper, gstudio, gknit, tes
|
|
|
279
280
|
2. Build the bridge native code: **`make -C ext/new_bridge all`** (or **`rake compile_gatekeeper`**).
|
|
280
281
|
3. Run scripts with **`bin/galaaz-jruby`** (sources **`bin/galaaz_jruby_env.inc.sh`** and adds **`-I lib`**).
|
|
281
282
|
|
|
282
|
-
Maintainers can prove a
|
|
283
|
+
A **gstudio** try image (JRuby + R + Galaaz already installed) is **`docker run --rm -it ghcr.io/rbotafogo/galaaz-try:gstudio`** (or **`./docker/try-gstudio/run.sh`** from a checkout). Maintainers can prove a RubyGems install on a throwaway Ubuntu machine (no repo inside the container) with **`./docker/cold-install/run.sh published-specs`**.
|
|
283
284
|
|
|
284
285
|
## Windows + WSL2 (optional: Docker / R in a container)
|
|
285
286
|
|
|
@@ -342,11 +343,13 @@ WSL integration is enabled for the distro where Galaaz is installed.
|
|
|
342
343
|
|
|
343
344
|
> galaaz master_list:scatter_plot
|
|
344
345
|
|
|
345
|
-
# JRuby, multithreading, and the R bridge
|
|
346
|
+
# JRuby, CRuby, multithreading, and the R bridge
|
|
346
347
|
|
|
347
|
-
Galaaz 2.0
|
|
348
|
-
I/O-bound work (HTTP clients, database
|
|
349
|
-
|
|
348
|
+
Galaaz 2.0 supports **JRuby** and **CRuby** equally for NewBridge. On **JRuby**, your
|
|
349
|
+
application can use **real parallel threads** for I/O-bound work (HTTP clients, database
|
|
350
|
+
connections, message consumers, and so on). On **CRuby**, prefer multi-process scaling for
|
|
351
|
+
CPU-bound concurrency. R itself is still executed in a **single GNU R process** behind the
|
|
352
|
+
Galaaz bridge (use multiple R workers when you need more R throughput).
|
|
350
353
|
|
|
351
354
|
When several Ruby threads call into R at the same time, the bridge **serializes** those calls:
|
|
352
355
|
each request is matched to a reply using an internal per-call **queue**, so you do not need to
|
|
@@ -417,19 +420,19 @@ the outcome to storage, and notify the client (poll, WebSocket, Turbo Stream, et
|
|
|
417
420
|
Ruby pattern above is only to show **when** the result exists (inside the block, or after data
|
|
418
421
|
written there is observed elsewhere). Runnable specs live in **`new_bridge_specs/eval_r_async_spec.rb`**.
|
|
419
422
|
|
|
420
|
-
## Galaaz + Rails (
|
|
423
|
+
## Galaaz + Rails (R-on-Rails) integration baseline
|
|
421
424
|
|
|
422
|
-
This
|
|
423
|
-
The
|
|
425
|
+
This is the practical **R-on-Rails** starter: an R scientist’s analysis behind a small Rails
|
|
426
|
+
app. The baseline we used in WSL aimed at:
|
|
424
427
|
|
|
425
|
-
1. Rails boots under **JRuby
|
|
428
|
+
1. Rails boots under **JRuby or CRuby** (same bridge; see Installation).
|
|
426
429
|
2. Galaaz is loaded from a local checkout (before publishing to RubyGems).
|
|
427
430
|
3. A request path can execute **`R.eval(...)`** and return a result.
|
|
428
431
|
|
|
429
|
-
### 1) Create the app with
|
|
432
|
+
### 1) Create the app with Ruby-friendly options
|
|
430
433
|
|
|
431
|
-
Rails defaults can pull gems that are
|
|
432
|
-
extension paths
|
|
434
|
+
Rails defaults can pull gems that are awkward on some setups (for example sqlite native
|
|
435
|
+
extension paths on JRuby, or deployment extras you do not need). A minimal app avoids early friction:
|
|
433
436
|
|
|
434
437
|
```bash
|
|
435
438
|
cd /home/rbotafogo/desenv_linux
|