arrow-datafusion 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f2a3634503431cd79f0d59c12e8d4ed1d8a899f38d5510835ee7cccb5404b4d
-  data.tar.gz: 02bba867e5c3da26395cc1d5f38457ce8769179f92551f80be898e5b43685284
+  metadata.gz: 6e8ea4613597694b0e11d65afafd8d31a0230fada4d84c9ecec842afe09ef046
+  data.tar.gz: 9397ee0001f51084c1651c70fcae972b680b614e2548ab286e90c645187e388f
 SHA512:
-  metadata.gz: ce438e24815777aac6bbd9fffbb8b12f2d64031179c9b3111ee84970ce44cd2833ef48adaad39ae9faff25d326af48fa7e6120341d96ce6bad3f45216f96c610
-  data.tar.gz: 59317958d2fe722bed2e50b885a40d49583d5b44117142991debb1f78e77f59adea9e19b578cf6d271c91c6c866730f68702b10ec85f60d38ef9d3c576be1efe
+  metadata.gz: 2f2c972c13edee286ce6aa71538b910794c3a1950e2f17be037ca2d3a6e03704b7e1135f11860b08ea8422c0cad1500ec027159548d31926ade5c78116c0b5cf
+  data.tar.gz: 5f2c0406f9519b192b270b86fb70bd432e4235059e46cc2e311d003b393c56e1f5e7c7f48601954ccb650dc1ec38cef36743bfa948a3335f73985b1fd96fa5cb

data/README.md

@@ -1,8 +1,8 @@
 # DataFusion in Ruby
 
-This is a Ruby library that binds to [Apache Arrow](https://arrow.apache.org/) in-memory query engine [DataFusion](https://github.com/apache/arrow-datafusion).
+This is yet another Ruby library that binds to [Apache Arrow](https://arrow.apache.org/) in-memory query engine [DataFusion](https://github.com/apache/arrow-datafusion).
 
-It allows you to build a plan through SQL or a DataFrame API against in-memory data, parquet or CSV files, run it in a multi-threaded environment, and obtain the result back in Ruby.
+This is an alternative to [datafuion-contrib/datafusion-ruby](https://github.com/datafusion-contrib/datafusion-ruby). Please refer to FAQ below.
 
 ## Quick Start
 
@@ -15,7 +15,7 @@ App
 ```ruby
 require "datafusion"
 
-ctx = Datafusion.SessionContext.new
+ctx = Datafusion::SessionContext.new
 ctx.register_csv("csv", "test.csv")
 ctx.sql("SELECT * FROM csv").collect
 ```
@@ -24,15 +24,15 @@ ctx.sql("SELECT * FROM csv").collect
 
 SessionContext
 - [x] new
-- [ ] register_csv
-- [ ] sql
+- [x] register_csv
+- [x] sql
 - [ ] register_parquet
 - [ ] register_record_batches
 - [ ] register_udf
 
 Dataframe
-- [ ] new
-- [ ] collect
+- [x] new
+- [x] collect
 - [ ] schema
 - [ ] select_columns
 - [ ] select
@@ -46,4 +46,30 @@ Dataframe
 
 ## Contribution Guide
 
-Please see [Contribution Guide](CONTRIBUTING.md) for information about contributing to DataFusion in Ruby.
+Please see [Contribution Guide](CONTRIBUTING.md).
+
+## FAQ
+
+### Why another Ruby bindings for Arrow Datafusion?
+
+[datafuion-contrib/datafusion-python](https://github.com/datafusion-contrib/datafusion-python) is a `Rust -> Python` bindings using [pyo3](https://github.com/PyO3/pyo3) and I want to use Arrow Datafusion in Ruby. So I create a `Rust -> Ruby` bindings using [Magnus](https://github.com/matsadler/magnus).
+
+Other than Python, Datafusion Community also want to have Java and other language bindings. In order to share development resource, [datafuion-contrib/datafusion-c](https://github.com/datafusion-contrib/datafusion-c) is created and will be used for [datafuion-contrib/datafusion-ruby](https://github.com/datafusion-contrib/datafusion-ruby) and other languages. E.g. `Rust -> C -> Ruby/Python/Java/etc`.
+
+So I just keep this `Rust -> Python` implementation as my side project.
+
+### Why Magnus?
+
+As of 2022-07, there are a few popular Ruby bindings for Rust, [Rutie](https://github.com/danielpclark/rutie), [Magnus](https://github.com/matsadler/magnus) and [other alternatives](https://github.com/matsadler/magnus#alternatives). Magnus is picked because its API seems cleaner and it seems more clear about safe vs unsafe. The author of Magnus have a "maybe bias" comparison in this [reddit thread](https://www.reddit.com/r/ruby/comments/uskibb/comment/i98rds4/?utm_source=share&utm_medium=web2x&context=3). It is totally subjective and it should not be large effort if we decides to switch to different Ruby bindings fr Rust in future.
+
+### Why the module name and gem name are different?
+
+The module name `Datafusion` follows the [datafusion](https://github.com/apache/arrow-datafusion) and [datafusion-python](https://github.com/datafusion-contrib/datafusion-python). The gem name `datafusion` [is occupied in rubygems.org at 2016](https://rubygems.org/gems/datafusion), so our gem is called `arrow-datafusion`.
+
+Similarly to the Ruby bindings of Arrow, its gem name is called [red-arrow](https://github.com/apache/arrow/tree/master/ruby/red-arrow) and the module is called `arrow`.
+
+### What is the relationship between gem "arrow-datafusion" and "red-arrow"?
+
+"arrow-datafusion" is the Ruby bindings of Arrow Datafusion (Rust). "red-arrow" is the Ruby bindings of Arrow (C++). To keep Datafusion Ruby simpler, I try not to couple with Red Arrow in core features at the moment. If need, we can add additional gems to support "red-arrow" in "arrow-datafusion", similar to how [red-parquet](https://github.com/apache/arrow/blob/2c7c12fd408339817f0322f137d25e9f60a87a26/ruby/red-parquet/red-parquet.gemspec#L44) use red-arrow.
+
+ps: Datafusion Python was coupled with PyArrow. There is a proposal to separate them in medium to long term. For detail, please refer to [Can datafusion-python be used without pyarrow?](https://github.com/datafusion-contrib/datafusion-python/issues/22).

data/arrow-datafusion.gemspec

@@ -3,12 +3,12 @@ require_relative "lib/datafusion/version"
 Gem::Specification.new do |spec|
   spec.name = "arrow-datafusion"
   spec.version = Datafusion::VERSION
-  spec.authors = ["Datafusion Contrib Developers"]
-  spec.homepage = "https://github.com/datafusion-contrib/datafusion-ruby"
+  spec.authors = ["jychen7"]
+  spec.homepage = "https://github.com/jychen7/arrow-datafusion-ruby"
 
-  spec.summary = "Ruby bindings of Apache Arrow Datafusion"
+  spec.summary = "yet another Ruby bindings of Apache Arrow Datafusion"
   spec.description =
-    "DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format."
+    "yet another Ruby bindings of Apache Arrow Datafusion"
   spec.license = "Apache-2.0"
 
   spec.files = ["README.md", "#{spec.name}.gemspec", "LICENSE"]
@@ -19,4 +19,5 @@ Gem::Specification.new do |spec|
 
   # actually a build time dependency, but that's not an option.
   spec.add_runtime_dependency "rake", "> 1"
+  spec.required_ruby_version = ">= 2.6.0"
 end

data/ext/datafusion_ruby/Cargo.lock

@@ -503,6 +503,7 @@ version = "0.0.1"
 dependencies = [
  "datafusion",
  "magnus",
+ "tokio",
 ]
 
 [[package]]
@@ -918,8 +919,7 @@ dependencies = [
 [[package]]
 name = "magnus"
 version = "0.3.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "983e15338a2e9644f804de8b5e52fb930bcd53b6859de4f4feb85753532b69d3"
+source = "git+https://github.com/matsadler/magnus#3466dbb87c7f4ec589e07b343ad6843c65555928"
 dependencies = [
  "bindgen",
  "magnus-macros",
@@ -928,8 +928,7 @@ dependencies = [
 [[package]]
 name = "magnus-macros"
 version = "0.1.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "27968fcabe2ef7e35b8331d71a62882282996f6222c133c0255cf7f33b8d9b48"
+source = "git+https://github.com/matsadler/magnus#3466dbb87c7f4ec589e07b343ad6843c65555928"
 dependencies = [
  "darling",
  "proc-macro2",

data/ext/datafusion_ruby/Cargo.toml

@@ -8,5 +8,7 @@ edition = "2018"
 crate-type = ["cdylib"]
 
 [dependencies]
-magnus = "0.3"
+# as of 2022-07, magnus v0.3.2 does NOT include "define_error" in RModule
+magnus = { git = "https://github.com/matsadler/magnus" }
 datafusion = { version = "^8.0.0" }
+tokio = { version = "1.0", features = ["macros", "rt", "rt-multi-thread", "sync"] }

data/ext/datafusion_ruby/src/context.rs

@@ -1,4 +1,10 @@
 use datafusion::execution::context::SessionContext;
+use datafusion::prelude::CsvReadOptions;
+use magnus::Error;
+
+use crate::dataframe::RbDataFrame;
+use crate::errors::DataFusionError;
+use crate::utils::wait_for_future;
 
 #[magnus::wrap(class = "Datafusion::SessionContext")]
 pub(crate) struct RbSessionContext {
@@ -11,4 +17,18 @@ impl RbSessionContext {
             ctx: SessionContext::new(),
         }
     }
+
+    pub(crate) fn register_csv(&self, name: String, table_path: String) -> Result<(), Error> {
+        let result =
+            self.ctx
+                .register_csv(name.as_ref(), table_path.as_ref(), CsvReadOptions::new());
+        wait_for_future(result).map_err(DataFusionError::from)?;
+        Ok(())
+    }
+
+    pub(crate) fn sql(&self, query: String) -> Result<RbDataFrame, Error> {
+        let result = self.ctx.sql(query.as_ref());
+        let df = wait_for_future(result).map_err(DataFusionError::from)?;
+        Ok(RbDataFrame::new(df))
+    }
 }

data/ext/datafusion_ruby/src/dataframe.rs

@@ -0,0 +1,27 @@
+use datafusion::dataframe::DataFrame;
+use magnus::Error;
+use std::sync::Arc;
+
+use crate::errors::DataFusionError;
+use crate::record_batch::RbRecordBatch;
+use crate::utils::wait_for_future;
+
+#[magnus::wrap(class = "Datafusion::DataFrame")]
+pub(crate) struct RbDataFrame {
+    df: Arc<DataFrame>,
+}
+
+impl RbDataFrame {
+    pub(crate) fn new(df: Arc<DataFrame>) -> Self {
+        Self { df }
+    }
+
+    pub(crate) fn collect(&self) -> Result<Vec<RbRecordBatch>, Error> {
+        let result = self.df.collect();
+        let batches = wait_for_future(result).map_err(DataFusionError::from)?;
+        Ok(batches
+            .into_iter()
+            .map(|batch| RbRecordBatch::new(batch))
+            .collect())
+    }
+}

data/ext/datafusion_ruby/src/errors.rs

@@ -0,0 +1,42 @@
+use core::fmt;
+
+use datafusion::arrow::error::ArrowError;
+use datafusion::error::DataFusionError as InnerDataFusionError;
+use magnus::Error as MagnusError;
+
+use crate::datafusion_error;
+
+#[derive(Debug)]
+pub enum DataFusionError {
+    ExecutionError(InnerDataFusionError),
+    ArrowError(ArrowError),
+    CommonError(String),
+}
+
+impl fmt::Display for DataFusionError {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        match self {
+            DataFusionError::ExecutionError(e) => write!(f, "Rust DataFusion error: {:?}", e),
+            DataFusionError::ArrowError(e) => write!(f, "Rust Arrow error: {:?}", e),
+            DataFusionError::CommonError(e) => write!(f, "Ruby DataFusion error: {:?}", e),
+        }
+    }
+}
+
+impl From<ArrowError> for DataFusionError {
+    fn from(err: ArrowError) -> DataFusionError {
+        DataFusionError::ArrowError(err)
+    }
+}
+
+impl From<InnerDataFusionError> for DataFusionError {
+    fn from(err: InnerDataFusionError) -> DataFusionError {
+        DataFusionError::ExecutionError(err)
+    }
+}
+
+impl From<DataFusionError> for MagnusError {
+    fn from(err: DataFusionError) -> MagnusError {
+        MagnusError::new(datafusion_error(), err.to_string())
+    }
+}

data/ext/datafusion_ruby/src/lib.rs

@@ -1,11 +1,39 @@
-use magnus::{define_module, function, prelude::*, Error};
+use magnus::{
+    define_module, exception::ExceptionClass, function, memoize, method, prelude::*, Error, RModule,
+};
 
 mod context;
+mod dataframe;
+mod errors;
+mod record_batch;
+mod utils;
+
+fn datafusion() -> RModule {
+    *memoize!(RModule: define_module("Datafusion").unwrap())
+}
+
+fn datafusion_error() -> ExceptionClass {
+    *memoize!(ExceptionClass: datafusion().define_error("Error", Default::default()).unwrap())
+}
 
 #[magnus::init]
 fn init() -> Result<(), Error> {
-    let module = define_module("Datafusion")?;
-    let class = module.define_class("SessionContext", Default::default())?;
-    class.define_singleton_method("new", function!(context::RbSessionContext::new, 0))?;
+    // ensure error is defined on load
+    datafusion_error();
+
+    let ctx_class = datafusion().define_class("SessionContext", Default::default())?;
+    ctx_class.define_singleton_method("new", function!(context::RbSessionContext::new, 0))?;
+    ctx_class.define_method(
+        "register_csv",
+        method!(context::RbSessionContext::register_csv, 2),
+    )?;
+    ctx_class.define_method("sql", method!(context::RbSessionContext::sql, 1))?;
+
+    let df_class = datafusion().define_class("DataFrame", Default::default())?;
+    df_class.define_method("collect", method!(dataframe::RbDataFrame::collect, 0))?;
+
+    let rb_class = datafusion().define_class("RecordBatch", Default::default())?;
+    rb_class.define_method("to_h", method!(record_batch::RbRecordBatch::to_hash, 0))?;
+
     Ok(())
 }

data/ext/datafusion_ruby/src/record_batch.rs

@@ -0,0 +1,56 @@
+use datafusion::arrow::{
+    array::{Float64Array, Int64Array, StringArray},
+    datatypes::DataType,
+    record_batch::RecordBatch,
+};
+use magnus::{Error, Value};
+
+use crate::errors::DataFusionError;
+use std::collections::HashMap;
+
+#[magnus::wrap(class = "Datafusion::RecordBatch")]
+pub(crate) struct RbRecordBatch {
+    rb: RecordBatch,
+}
+
+impl RbRecordBatch {
+    pub(crate) fn new(rb: RecordBatch) -> Self {
+        Self { rb }
+    }
+
+    pub(crate) fn to_hash(&self) -> Result<HashMap<String, Vec<Value>>, Error> {
+        let mut columns_by_name: HashMap<String, Vec<Value>> = HashMap::new();
+        for (i, field) in self.rb.schema().fields().iter().enumerate() {
+            let column = self.rb.column(i);
+            columns_by_name.insert(
+                field.name().clone(),
+                match column.data_type() {
+                    DataType::Int64 => {
+                        let array = column.as_any().downcast_ref::<Int64Array>().unwrap();
+                        array.values().iter().map(|v| (*v as i64).into()).collect()
+                    }
+                    DataType::Float64 => {
+                        let array = column.as_any().downcast_ref::<Float64Array>().unwrap();
+                        array.values().iter().map(|v| (*v as f64).into()).collect()
+                    }
+                    DataType::Utf8 => {
+                        let array = column.as_any().downcast_ref::<StringArray>().unwrap();
+                        let mut values: Vec<Value> = vec![];
+                        for i in 0..(column.len()) {
+                            values.push(std::string::String::from(array.value(i)).into())
+                        }
+                        values
+                    }
+                    unknown => {
+                        return Err(DataFusionError::CommonError(format!(
+                            "unhandle data type: {}",
+                            unknown
+                        ))
+                        .into())
+                    }
+                },
+            );
+        }
+        Ok(columns_by_name)
+    }
+}

data/ext/datafusion_ruby/src/utils.rs

@@ -0,0 +1,11 @@
+use std::future::Future;
+use tokio::runtime::Runtime;
+
+pub fn wait_for_future<F: Future>(f: F) -> F::Output
+where
+    F: Send,
+    F::Output: Send,
+{
+    let rt = Runtime::new().unwrap();
+    rt.block_on(f)
+}

data/lib/datafusion/version.rb

@@ -1,3 +1,3 @@
 module Datafusion
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: arrow-datafusion
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
-- Datafusion Contrib Developers
+- jychen7
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2022-07-03 00:00:00.000000000 Z
+date: 2022-07-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -24,8 +24,7 @@ dependencies:
     - - ">"
       - !ruby/object:Gem::Version
         version: '1'
-description: DataFusion is an extensible query execution framework, written in Rust,
-  that uses Apache Arrow as its in-memory format.
+description: yet another Ruby bindings of Apache Arrow Datafusion
 email: 
 executables: []
 extensions:
@@ -39,10 +38,14 @@ files:
 - ext/datafusion_ruby/Cargo.toml
 - ext/datafusion_ruby/Rakefile
 - ext/datafusion_ruby/src/context.rs
+- ext/datafusion_ruby/src/dataframe.rs
+- ext/datafusion_ruby/src/errors.rs
 - ext/datafusion_ruby/src/lib.rs
+- ext/datafusion_ruby/src/record_batch.rs
+- ext/datafusion_ruby/src/utils.rs
 - lib/datafusion.rb
 - lib/datafusion/version.rb
-homepage: https://github.com/datafusion-contrib/datafusion-ruby
+homepage: https://github.com/jychen7/arrow-datafusion-ruby
 licenses:
 - Apache-2.0
 metadata: {}
@@ -54,7 +57,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 2.6.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
@@ -64,5 +67,5 @@ requirements: []
 rubygems_version: 3.1.6
 signing_key: 
 specification_version: 4
-summary: Ruby bindings of Apache Arrow Datafusion
+summary: yet another Ruby bindings of Apache Arrow Datafusion
 test_files: []