galaaz 2.0.0 → 2.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (103) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +43 -5
  3. data/README.md +200 -128
  4. data/Rakefile +17 -10
  5. data/bin/galaaz +10 -2
  6. data/bin/galaaz-jruby +3 -6
  7. data/bin/galaaz-ruby +16 -0
  8. data/bin/galaaz_jruby_env.inc.sh +19 -3
  9. data/bin/galaaz_ruby_env.inc.sh +38 -0
  10. data/bin/gbookdown +2 -3
  11. data/bin/gknit +2 -3
  12. data/bin/gstudio +4 -4
  13. data/bin/run_all_rspec +6 -5
  14. data/bin/run_example +10 -7
  15. data/bin/run_old_rspec +6 -5
  16. data/bin/run_rspec +9 -8
  17. data/bin/run_slow_rspec +6 -5
  18. data/blogs/R-on-Rails-Planning-Document.md +79 -112
  19. data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/midwest_rb.png +0 -0
  20. data/blogs/galaaz_ggplot/galaaz_ggplot_files/figure-gfm/scatter_plot_rb.png +0 -0
  21. data/blogs/gknit/gknit_files/figure-gfm/bubble-1.png +0 -0
  22. data/blogs/gknit/gknit_files/figure-gfm/diverging_bar.png +0 -0
  23. data/blogs/manual/manual.Rmd +22 -19
  24. data/blogs/manual/manual.md +200 -128
  25. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/dose_len.png +0 -0
  26. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_delivery.png +0 -0
  27. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facet_by_dose.png +0 -0
  28. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color.png +0 -0
  29. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_by_delivery_color2.png +0 -0
  30. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_decorations.png +0 -0
  31. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_jitter.png +0 -0
  32. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/facets_with_points.png +0 -0
  33. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_box_plot.png +0 -0
  34. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/final_violin_plot.png +0 -0
  35. data/blogs/ruby_plot/ruby_plot_files/figure-gfm/violin_with_jitter.png +0 -0
  36. data/blogs/test/test.md +10 -0
  37. data/examples/50Plots_MasterList/ScatterPlot.rb +2 -1
  38. data/examples/50Plots_MasterList/scatter_plot.rb +1 -0
  39. data/examples/bioconductor_deseq2_airway/bench_galaaz_three_same_process.rb +1 -0
  40. data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz.rb +1 -0
  41. data/examples/bioconductor_deseq2_airway/deseq2_airway_galaaz_optimized.rb +1 -0
  42. data/examples/islr/all.rb +1 -0
  43. data/examples/islr/ch2.spec.rb +1 -0
  44. data/examples/islr/ch3.spec.rb +1 -0
  45. data/examples/islr/ch3_boston.rb +1 -0
  46. data/examples/islr/ch3_multiple_regression.rb +1 -0
  47. data/examples/islr/ch6.spec.rb +1 -0
  48. data/examples/islr/x_y_rnorm.jpg +0 -0
  49. data/examples/misc/ggplot.rb +2 -1
  50. data/examples/misc/moneyball.rb +1 -0
  51. data/examples/misc/subsetting.rb +1 -0
  52. data/examples/multithread_shards_to_r/shards_to_r.rb +1 -0
  53. data/examples/sthda_ggplot/all.rb +1 -0
  54. data/examples/sthda_ggplot/one_variable_continuous/density_gg.rb +1 -0
  55. data/examples/sthda_ggplot/one_variable_continuous/geom_area.rb +1 -0
  56. data/examples/sthda_ggplot/one_variable_continuous/geom_density.rb +1 -0
  57. data/examples/sthda_ggplot/one_variable_continuous/geom_dotplot.rb +1 -0
  58. data/examples/sthda_ggplot/one_variable_continuous/geom_freqpoly.rb +1 -0
  59. data/examples/sthda_ggplot/one_variable_continuous/geom_histogram.rb +1 -0
  60. data/examples/sthda_ggplot/one_variable_continuous/histogram_density.rb +1 -0
  61. data/examples/sthda_ggplot/one_variable_continuous/stat.rb +1 -0
  62. data/examples/sthda_ggplot/one_variable_discrete/bar.rb +1 -0
  63. data/examples/sthda_ggplot/qplots/box_violin_dot.rb +1 -0
  64. data/examples/sthda_ggplot/qplots/scatter_plots.rb +1 -0
  65. data/examples/sthda_ggplot/scatter_gg.rb +1 -0
  66. data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_bin2d.rb +1 -0
  67. data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_density2d.rb +1 -0
  68. data/examples/sthda_ggplot/two_variables_cont_bivariate/geom_hex.rb +1 -0
  69. data/examples/sthda_ggplot/two_variables_cont_cont/geom_point.rb +1 -0
  70. data/examples/sthda_ggplot/two_variables_cont_cont/geom_smooth.rb +1 -0
  71. data/examples/sthda_ggplot/two_variables_cont_cont/misc.rb +1 -0
  72. data/examples/sthda_ggplot/two_variables_cont_function/geom_area.rb +1 -0
  73. data/examples/sthda_ggplot/two_variables_disc_cont/geom_bar.rb +1 -0
  74. data/examples/sthda_ggplot/two_variables_disc_cont/geom_boxplot.rb +1 -0
  75. data/examples/sthda_ggplot/two_variables_disc_cont/geom_dotplot.rb +1 -0
  76. data/examples/sthda_ggplot/two_variables_disc_cont/geom_jitter.rb +1 -0
  77. data/examples/sthda_ggplot/two_variables_disc_cont/geom_line.rb +1 -0
  78. data/examples/sthda_ggplot/two_variables_disc_cont/geom_violin.rb +1 -0
  79. data/examples/sthda_ggplot/two_variables_disc_disc/geom_jitter.rb +1 -0
  80. data/examples/sthda_ggplot/two_variables_error/geom_crossbar.rb +1 -0
  81. data/lib/R_interface/r_arrow.rb +37 -0
  82. data/lib/R_interface/rvector.rb +15 -0
  83. data/lib/galaaz/arrow_ipc/java_arrow_backend.rb +250 -0
  84. data/lib/galaaz/arrow_ipc/red_arrow_backend.rb +126 -0
  85. data/lib/galaaz/arrow_ipc.rb +167 -0
  86. data/lib/galaaz/cli.rb +531 -0
  87. data/lib/galaaz.rb +6 -0
  88. data/lib/galaaz_jruby.rb +17 -2
  89. data/lib/galaaz_ruby.rb +34 -0
  90. data/lib/new_bridge/envelope.rb +2 -2
  91. data/new_bridge_specs/arrow_ipc_async_spec.rb +90 -0
  92. data/new_bridge_specs/arrow_ipc_export_async_spec.rb +68 -0
  93. data/r_requires/arrow.txt +3 -0
  94. data/r_requires/knit-extras.txt +4 -0
  95. data/r_requires/knit.txt +4 -0
  96. data/specs/arrow_ipc_export_spec.rb +62 -0
  97. data/specs/arrow_ipc_handoff_spec.rb +97 -0
  98. data/specs/gknit_include_engine_spec.rb +2 -2
  99. data/specs/gknit_install_timeout_report_spec.rb +2 -2
  100. data/specs/gknit_internal_error_report_spec.rb +2 -2
  101. data/specs/gknit_vector_map_output_spec.rb +2 -2
  102. data/version.rb +1 -1
  103. metadata +50 -13
@@ -68,9 +68,9 @@ Rails never went away—it matured. Several factors make Rails compelling again:
68
68
 
69
69
  1. **Rails (Ruby)**: Handles web requests, authentication, authorization, database operations, background jobs, email, APIs, caching, sessions, and orchestration
70
70
  2. **R**: Handles statistical modeling, machine learning, visualization, and data analysis
71
- 3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—using Apache Arrow for zero-copy data transfer at scale
71
+ 3. **Galaaz**: The seamless bridge that makes this integration feel like a single system—proxies for analytics in R, Apache Arrow for bulky tables (Stage A copy, Stage B IPC/mmap file; shared-heap zero-copy is Stage C and not shipped)
72
72
 
73
- **The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications. Move data between languages at memory speed, not serialization speed.
73
+ **The Value Proposition**: Keep everything you love about R—the packages, the syntax, the statistical rigor—while gaining everything Rails provides for production applications. For large tables, prefer Arrow **IPC files** (path across the bridge, bytes on disk or `/dev/shm`) over CSV/JSON serialization. Do **not** claim Ruby and R share one physical Arrow heap until Stage C ships.
74
74
 
75
75
  ---
76
76
 
@@ -86,25 +86,24 @@ Earlier versions of Galaaz explored GraalVM's polyglot capabilities with FastR (
86
86
  ├─────────────────────────────────────────────────────────────────┤
87
87
  │ │
88
88
  │ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
89
- │ │ JRuby │ ←─────→ │ Galaaz Bridge│ ←─────→ │ GNU R │ │
90
- │ │ (JVM) (Java) │ (JNI/R API) │ (C/R) │ Process │ │
91
- │ └──────────────┘ └──────────────┘ └──────────┘ │
92
- │ │ │ │
89
+ │ │ JRuby/CRuby │ ←─────→ │ Galaaz Bridge│ ←─────→ │ GNU R │ │
90
+ │ │ path + │ (NewBridge) │ C/R │ Process │ │
91
+ │ └──────────────┘ cmds └──────────────┘ └──────────┘ │
93
92
  │ │ │ │
94
93
  │ ▼ ▼ │
95
94
  │ ┌──────────────┐ ┌──────────┐│
96
95
  │ │ Rails App │ │ Arrow ││
97
- │ │ Multi-thread Shared ││
98
- │ │ ActiveRecord │ │ Memory ││
96
+ │ │ (threads / bulky tables: IPC file Table in ││
97
+ │ │ processes) │ ──── mmap / /dev/shm (Stage B) ──│ R (proxy)││
99
98
  │ └──────────────┘ └──────────┘│
100
99
  │ │
101
100
  └─────────────────────────────────────────────────────────────────┘
102
101
  ```
103
102
 
104
103
  **How it works:**
105
- 1. **JRuby** runs on the JVM with true multi-threading, running Rails and application code
106
- 2. **Galaaz Bridge** communicates with a GNU R process via R's C API (or alternative mechanisms)
107
- 3. **Apache Arrow** (optional) provides efficient, shared-memory data transfer for large datasets
104
+ 1. **JRuby or CRuby** runs Rails and application code (JRuby for true JVM threads; CRuby for MRI). Both talk the same NewBridge protocol.
105
+ 2. **Galaaz Bridge** communicates with a GNU R process (NewBridge / gatekeeper)—commands and small values, not bulk bytes when you use Stage B.
106
+ 3. **Apache Arrow** (optional): **Stage A** copies Ruby batches into an R-side Arrow Table (`R::Arrow.from_ruby_batches`). **Stage B** writes an Arrow IPC file; only the **path** crosses the bridge (`Galaaz::ArrowIpc` + `R::Arrow.open_ipc` / `write_ipc`). **Stage C** (shared-heap zero-copy) is not shipped—see `Documentation/ROADMAP_ARROW_RUBY_R.md`.
108
107
  4. **GNU R** is the actual R interpreter—full compatibility with all R packages (ggplot2, dplyr, Bioconductor, etc.)
109
108
 
110
109
  **Why this is better than FastR/GraalVM:**
@@ -448,13 +447,22 @@ end
448
447
 
449
448
  ---
450
449
 
451
- ### Section 6: Apache Arrow - Zero-Copy Data at Scale
450
+ ### Section 6: Apache Arrow bulky tables without CSV/JSON
451
+
452
+ **Key Message**: Traditional bridges often serialize through text (CSV, JSON). Galaaz uses Apache Arrow in **two shipped modes**. Neither is a shared Ruby+R heap. Maintainer detail: `Documentation/ROADMAP_ARROW_RUBY_R.md`. User manual: `# Apache Arrow` in `README.md` / `blogs/manual/manual.md`.
452
453
 
453
- **Key Message**: Traditional bridges serialize data (JSON, CSV) to pass between languages. Galaaz uses Apache Arrow for zero-copy, memory-mapped data transfer—enabling huge datasets to move from Ruby to R instantly.
454
+ | Stage | API | What actually happens |
455
+ |-------|-----|------------------------|
456
+ | **A** (shipped) | `R::Arrow.from_ruby_batches` / `table_from` | Columns are **copied** into GNU R; Ruby holds a **proxy**. dplyr then runs in R. |
457
+ | **B1** (shipped) | `Galaaz::ArrowIpc.write` → `R::Arrow.open_ipc(path)` | Ruby writes an Arrow **IPC file** (prefer `/dev/shm`); NewBridge carries only the **path**. R memory-maps / reads the file into an Arrow Table. |
458
+ | **B2** (shipped) | `R::Arrow.write_ipc` → `Galaaz::ArrowIpc.read` / `read_batches` | R writes IPC (uncompressed so JRuby Arrow Java can read); Ruby reads columns or row hashes for DB/API. |
459
+ | **C** (future) | not shipped | Named shared segment; do **not** claim 0 ms shared RAM until this exists and is measured. |
460
+
461
+ **Writers/readers (Stage B):** CRuby uses **red-arrow** (`gem install red-arrow` matching `pkg-config --modversion arrow-glib`, plus Apache Arrow APT / `libarrow-glib-dev`). JRuby uses **Apache Arrow Java** JARs (`GALAAZ_ARROW_JARS` or `~/arrow_jars`) and `JAVA_OPTS` `--add-opens=java.base/java.nio=ALL-UNNAMED` on the **child** JVM (`bin/galaaz-jruby` / `mise.toml`). Do not install the unrelated Rubygems gem named `arrow`.
454
462
 
455
463
  **The Problem: Data Transfer Overhead**
456
464
 
457
- In traditional polyglot systems (rpy2, reticulate), passing data between languages requires serialization:
465
+ In traditional polyglot systems (rpy2, reticulate), passing data between languages often means a text round-trip:
458
466
 
459
467
  ```python
460
468
  # Python → R via JSON/CSV serialization (SLOW)
@@ -470,36 +478,31 @@ ro.globalenv['r_df'] = pandas2ri.py2rpy(df)
470
478
  This approach:
471
479
  - Copies all data through a text format
472
480
  - Loses type information (factors become strings, dates become strings)
473
- - Consumes 2-3x memory during transfer
474
- - Becomes a bottleneck at ~100K+ rows
481
+ - Consumes extra memory during transfer
482
+ - Becomes a bottleneck at large row counts
475
483
 
476
- **The Galaaz Solution: Apache Arrow**
484
+ **Stage A: copy into R, then Remote Control**
477
485
 
478
- Apache Arrow is a columnar, in-memory format designed for zero-copy data sharing across languages. With Galaaz:
486
+ Assemble row hashes in Ruby (threads OK on JRuby), then one ingest. Analytics stay on the R proxy; unbox KPIs only.
479
487
 
480
488
  ```ruby
481
489
  require 'galaaz'
482
490
 
483
- # 1. Build data in Ruby—using parallel threads for multi-source ingestion
484
491
  batches = []
485
492
  mutex = Mutex.new
486
493
  threads = []
487
494
 
488
- # Simulate parallel data ingestion from multiple sources
489
495
  [0, 1, 2, 3].each do |tid|
490
496
  threads << Thread.new do
491
- # Each thread queries its own database/shard/API
492
- local_data = fetch_from_source(tid) # Returns array of hashes
497
+ local_data = fetch_from_source(tid) # array of hashes
493
498
  mutex.synchronize { batches << local_data }
494
499
  end
495
500
  end
496
501
  threads.each(&:join)
497
502
 
498
- # 2. Convert to Arrow Table—zero-copy transfer to R
499
- # This is the key: data moves to R without serialization!
503
+ # Copy into an Arrow Table *inside GNU R* (not shared heap with Ruby)
500
504
  table = R::Arrow.from_ruby_batches(batches)
501
505
 
502
- # 3. Use R's dplyr directly on the Arrow table
503
506
  R.install_and_loads('dplyr', 'arrow')
504
507
 
505
508
  grouped = R.dplyr___group_by(table, :region)
@@ -510,57 +513,59 @@ summarised = R.dplyr___summarise(
510
513
  total: E.sum(:value)
511
514
  )
512
515
  results = R.dplyr___collect(summarised)
513
-
514
516
  puts results
515
- #> # A tibble: 4 × 4
516
- #> region count avg_value total
517
- #> <chr> <int> <dbl> <dbl>
518
- #> 1 North 25000 5.5 137500
519
- #> 2 South 25000 5.5 137500
520
- #> 3 East 25000 5.5 137500
521
- #> 4 West 25000 5.5 137500
522
517
  ```
523
518
 
524
- **Why This Matters:**
519
+ **Stage B: IPC file; path on the bridge**
520
+
521
+ Use this when the table is large enough that copying every cell over NewBridge/MsgPack is the wrong tax. Scratch files prefer `/dev/shm`.
522
+
523
+ ```ruby
524
+ require 'galaaz'
525
+
526
+ # B1: Ruby → file → R
527
+ path = Galaaz::ArrowIpc.write_batches(batches) # or .write(col => array, ...)
528
+ tbl = R::Arrow.open_ipc(path)
529
+ Galaaz::ArrowIpc.release(path) # unlink after R has opened (default B1 lifetime)
530
+
531
+ grouped = R.dplyr___group_by(tbl, :region)
532
+ summarised = R.dplyr___summarise(grouped, total: E.sum(:value))
533
+
534
+ # B2: R → file → Ruby (e.g. rows for ActiveRecord / JSON)
535
+ out_path = R::Arrow.write_ipc(summarised)
536
+ rows = Galaaz::ArrowIpc.read_batches(out_path)
537
+ Galaaz::ArrowIpc.release(out_path)
538
+
539
+ render json: { analytics: rows }
540
+ ```
541
+
542
+ **Why this matters (honest):**
525
543
 
526
- - **No serialization cost**: Data stays in Arrow's columnar format throughout
527
- - **Type preservation**: Factors, dates, timestamps remain intact
528
- - **Memory efficiency**: No intermediate copies; Ruby and R share the same memory
529
- - **Scale**: Tested with millions of rows (see slow-specs/arrow_large_pipeline_spec.rb with 200K+ rows)
530
- - **Lazy evaluation**: Arrow datasets can be filtered/aggregated before materializing
544
+ - **Columnar Arrow**, not CSV, for the bulky payload
545
+ - **Type-preserving** numeric/string columns on the B1/B2 contract (int32/int64, float64, utf8; nulls kept)
546
+ - **Stage A** still copies; **Stage B** still materializes a file R (or Ruby) then reads—R may mmap the IPC file
547
+ - **Not** “Ruby and R map the same live buffer” until Stage C
548
+ - **Scale**: Stage A pipeline in `slow-specs/arrow_large_pipeline_spec.rb` (200K+ rows); Stage B specs in `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, and the matching `new_bridge_specs/arrow_ipc_*_async_spec.rb`
549
+ - **Parquet/Feather/dataset** stay R-side file APIs (`R::Arrow.write_parquet`, `dataset`)
531
550
 
532
- **Real-World Example: Multi-DB Aggregation**:
551
+ **Real-world sketch: shards in Ruby, analytics in R**
533
552
 
534
553
  ```ruby
535
- # Aggregate data from multiple PostgreSQL shards in Ruby,
536
- # then analyze in R with zero-copy transfer
554
+ # Parallel ingest in Ruby, then one Arrow handoff (A or B), then dplyr in R.
537
555
 
538
556
  require 'galaaz'
539
557
  require 'active_record'
540
558
 
541
- # Connect to multiple database shards
542
- SHARDS = ['shard_1', 'shard_2', 'shard_3', 'shard_4'].map do |shard_name|
543
- ActiveRecord::Base.establish_connection(
544
- adapter: 'postgresql',
545
- host: "#{shard_name}.db.internal",
546
- database: 'analytics'
547
- )
548
- end
549
-
550
- # Parallel data collection from all shards
551
559
  batches = []
552
560
  mutex = Mutex.new
553
561
 
554
562
  SHARDS.each do |shard|
555
563
  Thread.new do
556
- # Query this shard
557
564
  records = shard.connection.select_all(<<-SQL).cast_values
558
565
  SELECT user_id, region, event_type, value, created_at
559
566
  FROM events
560
567
  WHERE created_at > NOW() - INTERVAL '7 days'
561
568
  SQL
562
-
563
- # Convert to array of hashes for Arrow
564
569
  batch = records.map do |row|
565
570
  {
566
571
  user_id: row[0],
@@ -570,60 +575,37 @@ SHARDS.each do |shard|
570
575
  created_at: row[4]
571
576
  }
572
577
  end
573
-
574
578
  mutex.synchronize { batches << batch }
575
579
  end
576
580
  end
577
581
 
578
- # Zero-copy transfer to R as Arrow Table
579
- table = R::Arrow.from_ruby_batches(batches)
582
+ R.install_and_loads('dplyr', 'arrow')
580
583
 
581
- # Now use R's full power on the complete dataset
582
- R.install_and_loads('dplyr', 'arrow', 'lubridate')
584
+ # Prefer B1 when the merged table is large:
585
+ path = Galaaz::ArrowIpc.write_batches(batches)
586
+ table = R::Arrow.open_ipc(path)
587
+ Galaaz::ArrowIpc.release(path)
583
588
 
584
- # R's dplyr works directly on Arrow tables (lazy evaluation)
585
- analysis = table \
586
- .dplyr___filter(E.created_at > (E.now() - E.days(7))) \
587
- .dplyr___group_by(:region, :event_type) \
588
- .dplyr___summarise(
589
+ analysis = R.dplyr___collect(
590
+ R.dplyr___summarise(
591
+ R.dplyr___group_by(table, :region, :event_type),
589
592
  count: E.n(),
590
- total_value: E.sum(:value),
591
- avg_value: E.mean(:value),
592
- unique_users: E.n_distinct(:user_id)
593
+ total_value: E.sum(:value)
593
594
  )
595
+ )
594
596
 
595
- # Materialize results when ready
596
- results = R.dplyr___collect(analysis)
597
-
598
- # Convert back to Ruby objects for API response
599
- results_hash = results.to_ruby
600
- render json: { analytics: results_hash }
597
+ out_path = R::Arrow.write_ipc(analysis)
598
+ rows = Galaaz::ArrowIpc.read_batches(out_path)
599
+ Galaaz::ArrowIpc.release(out_path)
600
+ render json: { analytics: rows }
601
601
  ```
602
602
 
603
- **Parquet and Feather: Persistence Without Conversion**:
603
+ **Parquet and Feather (R-visible paths):**
604
604
 
605
605
  ```ruby
606
- # Save Ruby data as Parquet (columnar, compressed)
607
- # Then read directly into R without parsing
608
-
609
- # 1. Build large dataset in Ruby
610
- data = (1..1_000_000).map do |i|
611
- {
612
- id: i,
613
- category: ["A", "B", "C", "D"][i % 4],
614
- value: rand * 100,
615
- timestamp: Time.now - (i % 86400)
616
- }
617
- end
618
-
619
- # 2. Convert to R data.frame, then write as Parquet
620
606
  df = R.data__frame(data)
621
607
  R::Arrow.write_parquet(df, '/data/events.parquet')
622
-
623
- # 3. Later, read directly into R as a dataset (lazy, memory-mapped)
624
608
  dataset = R::Arrow.dataset('/data/events.parquet')
625
-
626
- # Query without loading entire file
627
609
  summary = R.dplyr___collect(
628
610
  dataset \
629
611
  .dplyr___filter(R[:value] > 50) \
@@ -632,28 +614,13 @@ summary = R.dplyr___collect(
632
614
  )
633
615
  ```
634
616
 
635
- **Large-Scale Test Results** (from `slow-specs/arrow_large_pipeline_spec.rb`):
636
-
637
- ```ruby
638
- # Test: 8 threads × 25,000 rows = 200,000 rows
639
- # With weighted aggregations in R
640
- thread_count = 8
641
- rows_per_thread = 25_000
642
- group_count = 10
643
-
644
- # Ruby parallel batch construction... 8 threads
645
- # Arrow table creation from batches → instant
646
- # R dplyr group_by + summarise → native speed
647
- # Results verified accurate against Ruby reference implementation
648
- ```
649
-
650
- **Key Insight**: With Arrow, the boundary between Ruby and R disappears for data. You can build data pipelines in Ruby (with its superior concurrency and database libraries) and analyze in R (with its statistical ecosystem)—with **zero overhead** at the language boundary.
617
+ **Key Insight**: Build pipelines in Ruby (concurrency, ActiveRecord); run statistics in R. Pay **one** bulky handoff (copy or IPC file), then **move commands** on proxies. That is the production story today—not a shared Arrow heap.
651
618
 
652
619
  ---
653
620
 
654
621
  ### Section 7: Parallelism and Process Orchestration
655
622
 
656
- **Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby's multi-threaded environment—now with Arrow for efficient data distribution.
623
+ **Key Message**: R is single-threaded. Galaaz solves this by orchestrating multiple R processes from Ruby—Arrow for bulky ingest/export, proxies for analytics per worker.
657
624
 
658
625
  **The Architecture**:
659
626
  ```
@@ -892,7 +859,7 @@ Galaaz 2.0 represents the maturation of the Ruby-R bridge:
892
859
  - **JRuby + GNU R architecture**: Moved from experimental GraalVM/FastR to battle-tested JRuby and standard GNU R for full package compatibility and production stability
893
860
  - **New bridge architecture**: More robust, faster, better error handling
894
861
  - **Process management**: Built-in support for R process pools
895
- - **Apache Arrow integration**: Zero-copy data transfer for large datasets—build data pipelines in Ruby's multi-threaded environment, analyze in R with zero serialization overhead
862
+ - **Apache Arrow**: Stage A copy (`from_ruby_batches`) and Stage B IPC/mmap (`Galaaz::ArrowIpc`, `open_ipc` / `write_ipc`). Shared-heap zero-copy is Stage C. Build pipelines in Ruby, analyze in R, unbox KPIs.
896
863
  - **gKnit improvements**: Better R Markdown integration
897
864
  - **Rails integration**: First-class support for Rails patterns
898
865
 
@@ -907,7 +874,7 @@ For the full blog post, include references to:
907
874
  - **NSE/dplyr**: `blogs/nse_dplyr/nse_dplyr.md`
908
875
  - **Object-Oriented**: `blogs/oh_my/oh_my.md`
909
876
  - **Plotting Tutorial**: `blogs/ruby_plot/ruby_plot.md`
910
- - **Apache Arrow**: `specs/arrow_semantics_spec.rb`, `specs/arrow_from_ruby_batches_spec.rb`, `slow-specs/arrow_large_pipeline_spec.rb`
877
+ - **Apache Arrow**: `Documentation/ROADMAP_ARROW_RUBY_R.md`; `specs/arrow_ipc_handoff_spec.rb`, `specs/arrow_ipc_export_spec.rb`, `new_bridge_specs/arrow_ipc_async_spec.rb`, `new_bridge_specs/arrow_ipc_export_async_spec.rb`; Stage A: `specs/arrow_from_ruby_batches_spec.rb`, `slow-specs/arrow_large_pipeline_spec.rb`
911
878
  - **Specs**: `specs/r_nse.spec.rb`, `specs/r_vector_functions.spec.rb`
912
879
 
913
880
  ---
@@ -1,8 +1,8 @@
1
1
  ---
2
2
  title: "Galaaz Manual"
3
- subtitle: "Coupling Ruby (JRuby) and GNU R for data science"
3
+ subtitle: "R-on-Rails: GNU R meets Ruby for the web"
4
4
  author: "Rodrigo Botafogo"
5
- tags: [Galaaz, Ruby, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
5
+ tags: [Galaaz, "R-on-Rails", Ruby, Rails, JRuby, R, "GNU R", ggplot2, knitr, dplyr, Bioconductor, Arrow]
6
6
  date: "2026"
7
7
  bibliography: "../../examples/Bibliography/stats.bib"
8
8
  output:
@@ -114,10 +114,11 @@ The Galaaz repository ships many helpers under **`bin/`**. When working from a *
114
114
  them as **`bin/<name>`** from the project root (or `./bin/<name>`). If you install the **gem**,
115
115
  only a subset is guaranteed on your `PATH` (see the gemspec: **`galaaz`**, **`gstudio`**, **`gknit`**, **`grun`**, **`gknit-draft`**); for development and CI, prefer the **`bin/`** copies so JVM flags and paths stay correct.
116
116
 
117
- Below, **current (Galaaz 2.0 + JRuby + GNU R)** means the tool is wired to **`jruby`** and
118
- **`bin/galaaz_jruby_env.inc.sh`** (or equivalent logic in Ruby via `lib/galaaz_jruby.rb`). **Legacy**
119
- means the script still targets **GraalVM** polyglot Ruby / FastR-era invocation and is **not**
120
- expected to work on a typical JRuby-only setup.
117
+ Below, **current (Galaaz 2.0 + JRuby or CRuby + GNU R)** means the tool uses **`bin/galaaz-ruby`**
118
+ / **`GALAAZ_RUBY`** (default: `ruby` on `PATH`) and applies JVM flags only when the interpreter is
119
+ JRuby (`bin/galaaz_jruby_env.inc.sh` / `lib/galaaz_jruby.rb`). **`bin/galaaz-jruby`** forces JRuby.
120
+ **Legacy** means the script still targets **GraalVM** polyglot Ruby / FastR-era invocation and is
121
+ **not** expected to work on a typical JRuby or CRuby NewBridge setup.
121
122
 
122
123
  **Table layout:** names in the first column are **`bin/`** filenames (run as `bin/<name>` from the repo root). Long options and examples sit **outside** the tables so PDF columns stay readable.
123
124
 
@@ -233,7 +234,7 @@ df_ot <- data.frame(
233
234
  bin_tbl(df_ot)
234
235
  ```
235
236
 
236
- For day-to-day **2.0** use, rely on **`bin/galaaz-jruby`**, **`bin/gstudio`**, **`bin/gknit`**, **`bin/run_example`**, **`bin/run_rspec`** / **`bin/run_all_rspec`**, and **`bin/galaaz-bootstrap`** on WSL when using Dockerized R. Treat **`grun`**, **`gknit_Rscript`**, and the polyglot **`ruby`** invocation in **`gknit-draft`** as **legacy** until they are ported to the same JRuby path as **`gknit`**.
237
+ For day-to-day **2.0** use, rely on **`bin/galaaz-ruby`** (or **`bin/galaaz-jruby`** when you want to force JRuby), **`bin/gstudio`**, **`bin/gknit`**, **`bin/run_example`**, **`bin/run_rspec`** / **`bin/run_all_rspec`**, and **`bin/galaaz-bootstrap`** on WSL when using Dockerized R. Treat **`grun`**, **`gknit_Rscript`**, and the polyglot **`ruby`** invocation in **`gknit-draft`** as **legacy** until they are ported to the same launcher path as **`gknit`**.
237
238
 
238
239
  # System Compatibility
239
240
 
@@ -279,7 +280,7 @@ A **table of all `bin/` scripts** (bootstrap, JRuby wrapper, gstudio, gknit, tes
279
280
  2. Build the bridge native code: **`make -C ext/new_bridge all`** (or **`rake compile_gatekeeper`**).
280
281
  3. Run scripts with **`bin/galaaz-jruby`** (sources **`bin/galaaz_jruby_env.inc.sh`** and adds **`-I lib`**).
281
282
 
282
- Maintainers can prove a built `.gem` on a throwaway Ubuntu machine (no repo inside the container) with **`./docker/cold-install/run.sh`**.
283
+ A **gstudio** try image (JRuby + R + Galaaz already installed) is **`docker run --rm -it ghcr.io/rbotafogo/galaaz-try:gstudio`** (or **`./docker/try-gstudio/run.sh`** from a checkout). Maintainers can prove a RubyGems install on a throwaway Ubuntu machine (no repo inside the container) with **`./docker/cold-install/run.sh published-specs`**.
283
284
 
284
285
  ## Windows + WSL2 (optional: Docker / R in a container)
285
286
 
@@ -342,11 +343,13 @@ WSL integration is enabled for the distro where Galaaz is installed.
342
343
 
343
344
  > galaaz master_list:scatter_plot
344
345
 
345
- # JRuby, multithreading, and the R bridge
346
+ # JRuby, CRuby, multithreading, and the R bridge
346
347
 
347
- Galaaz 2.0 runs Ruby on **JRuby**, so your application can use **real parallel threads** for
348
- I/O-bound work (HTTP clients, database connections, message consumers, and so on). R itself is
349
- still executed in a **single GNU R process** behind the Galaaz bridge.
348
+ Galaaz 2.0 supports **JRuby** and **CRuby** equally for NewBridge. On **JRuby**, your
349
+ application can use **real parallel threads** for I/O-bound work (HTTP clients, database
350
+ connections, message consumers, and so on). On **CRuby**, prefer multi-process scaling for
351
+ CPU-bound concurrency. R itself is still executed in a **single GNU R process** behind the
352
+ Galaaz bridge (use multiple R workers when you need more R throughput).
350
353
 
351
354
  When several Ruby threads call into R at the same time, the bridge **serializes** those calls:
352
355
  each request is matched to a reply using an internal per-call **queue**, so you do not need to
@@ -417,19 +420,19 @@ the outcome to storage, and notify the client (poll, WebSocket, Turbo Stream, et
417
420
  Ruby pattern above is only to show **when** the result exists (inside the block, or after data
418
421
  written there is observed elsewhere). Runnable specs live in **`new_bridge_specs/eval_r_async_spec.rb`**.
419
422
 
420
- ## Galaaz + Rails (JRuby) integration baseline
423
+ ## Galaaz + Rails (R-on-Rails) integration baseline
421
424
 
422
- This section documents the baseline we used to create a working Rails app with Galaaz in WSL.
423
- The goals were:
425
+ This is the practical **R-on-Rails** starter: an R scientist’s analysis behind a small Rails
426
+ app. The baseline we used in WSL aimed at:
424
427
 
425
- 1. Rails boots under **JRuby**.
428
+ 1. Rails boots under **JRuby or CRuby** (same bridge; see Installation).
426
429
  2. Galaaz is loaded from a local checkout (before publishing to RubyGems).
427
430
  3. A request path can execute **`R.eval(...)`** and return a result.
428
431
 
429
- ### 1) Create the app with JRuby-friendly options
432
+ ### 1) Create the app with Ruby-friendly options
430
433
 
431
- Rails defaults can pull gems that are not ideal on JRuby-first setups (for example sqlite native
432
- extension paths and deployment extras). A minimal app avoids early friction:
434
+ Rails defaults can pull gems that are awkward on some setups (for example sqlite native
435
+ extension paths on JRuby, or deployment extras you do not need). A minimal app avoids early friction:
433
436
 
434
437
  ```bash
435
438
  cd /home/rbotafogo/desenv_linux