galaaz 0.4.7 → 0.4.8

data/blogs/nse_dplyr/nse_dplyr.md

@@ -19,6 +19,79 @@ output:
 
 # Introduction
 
+In this post we will see how to program with dplyr in Galaaz.  
+
+### But first, what is Galaaz??
+
+Galaaz is a system for tightly coupling Ruby and R.  Ruby is a powerful language, with
+a large community, a very large set of libraries and great for web development.  However,
+it lacks libraries for data science, statistics, scientific plotting and machine learning.
+On the other hand, R is considered one of the most powerful languages for solving all of the
+above problems.  Maybe the strongest competitor to R is Python with libraries such as NumPy,
+Panda, SciPy, SciKit-Learn and a couple more.
+
+With Galaaz we do not intend to re-implement any of the scientific libraries in R. However, we 
+allow for very tight coupling between the two languages to the point that the Ruby 
+developer does not need to know that there is an R engine running.  For this to happen we 
+use new technologies provided by Oracle: GraalVM, TruffleRuby and FastR:
+
+     GraalVM is a universal virtual machine for running applications
+     written in JavaScript, Python 3, Ruby, R, JVM-based languages like Java,
+     Scala, Kotlin, and LLVM-based languages such as C and C++.
+
+     GraalVM removes the isolation between programming languages and enables
+     interoperability in a shared runtime. It can run either standalone or in
+     the context of OpenJDK, Node.js, Oracle Database, or MySQL.
+
+     GraalVM allows you to write polyglot applications with a seamless way to
+     pass values from one language to another. With GraalVM there is no copying
+     or marshaling necessary as it is with other polyglot systems. This lets
+     you achieve high performance when language boundaries are crossed. Most
+     of the time there is no additional cost for crossing a language boundary
+     at all.
+
+     Often developers have to make uncomfortable compromises that require them
+     to rewrite their software in other languages. For example:
+
+      * “That library is not available in my language. I need to rewrite it.” 
+      * “That language would be the perfect fit for my problem, but we cannot
+        run it in our environment.” 
+      * “That problem is already solved in my language, but the language is
+        too slow.”
+    
+    With GraalVM we aim to allow developers to freely choose the right language
+    for the task at hand without making compromises.
+
+Interested readers should also check out the following sites:
+
+* [GraalVM Home](https://www.graalvm.org/)
+* [TruffleRuby](https://github.com/oracle/truffleruby)
+* [FastR](https://github.com/oracle/fastr)
+* [Faster R with FastR](https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb)
+
+### Now to programming with dplyr
+
+According to Hardley (https://dplyr.tidyverse.org/articles/programming.html)
+
+> Most dplyr functions use non-standard evaluation (NSE). This is a catch-all term that 
+> means they don’t follow the usual R rules of evaluation. Instead, they capture the 
+> expression that you typed and evaluate it in a custom way. This has two main 
+> benefits for dplyr code:
+
+> Operations on data frames can be expressed succinctly because you don’t need to repeat 
+> the name of the data frame. For example, you can write filter(df, x == 1, y == 2, z == 3) 
+> instead of df[df$x == 1 & df$y ==2 & df$z == 3, ].
+
+> dplyr can choose to compute results in a different way to base R. This is important for 
+> database backends because dplyr itself doesn’t do any work, but instead generates the SQL 
+> that tells the database what to do.
+
+> Unfortunately these benefits do not come for free. There are two main drawbacks:
+
+> Most dplyr arguments are not referentially transparent. That means you can’t replace a value 
+> with a seemingly equivalent object that you’ve defined elsewhere. In other words, this code:
+
+
 
 ```r
 df <- data.frame(x = 1:3, y = 3:1)
@@ -47,6 +120,7 @@ print(filter(df, x == 1))
 #>   <int> <int>
 #> 1     1     3
 ```
+> Is not equivalent to this code:
 
 
 ```r
@@ -55,13 +129,149 @@ my_var <- x
 filter(df, my_var == 1)
 #> Error: object 'my_var' not found
 ```
+> This makes it hard to create functions with arguments that change how dplyr verbs are computed.
+
+# Writing Expressions in Galaaz
+
+Galaaz extends Ruby to work with complex expressions, similar to R's expressions build with 'quote' 
+(base R) or 'quo' (tidyverse).  Let's take a look at some of those expressions.  
+
+## Expressions from operators
+
+The code bellow 
+creates an expression summing two symbols
+
+
+```ruby
+exp1 = :a + :b
+puts exp1
+```
+
+```
+## a + b
+```
+We can build any complex mathematical expression
+
+
+```ruby
+exp2 = (:a + :b) * 2.0 + :c ** 2 / :z
+puts exp2
+```
+
+```
+## (a + b) * 2 + c^2L/z
+```
+
+It is also possible to use inequality operators in building expressions
+
+
+```ruby
+exp3 = (:a + :b) >= :z
+puts exp3
+```
+
+```
+## a + b >= z
+```
+
+Galaaz provides both symbolic representations for operators, such as (>, <, !=) as functional 
+notation for those operators such as (.gt, .ge, etc.).  So the same expression written 
+above can also be written as
+
+
+```ruby
+exp4 = (:a + :b).ge :z
+puts exp4
+```
+
+```
+## a + b >= z
+```
+
+Two type of expression can only be created with the functional representation of the operators, 
+those are expressions involving '==', and '='.  In order to write an expression involving '==' we
+need to use the method '.eq' and for '=' we need the function '.assign'
+
+
+```ruby
+exp5 = (:a + :b).eq :z
+puts exp5
+```
+
+```
+## a + b == z
+```
+
+
+```ruby
+exp6 = :y.assign :a + :b
+puts exp6
+```
+
+```
+## y <- a + b
+```
+In general we think that using the functional notation is preferable to using the 
+symbolic notation as otherwise, we end up writing invalid expressions such as
+
+
+```ruby
+exp_wrong = (:a + :b) == :z
+puts exp_wrong
+```
+
+```
+## Message:
+##  Error in function (x, y, num.eq = TRUE, single.NA = TRUE, attrib.as.set = TRUE,  :
+##   object 'a' not found (RError)
+## Translated to internal error
+```
+and it might be difficult to understand what is going on here.  The problem lies with the fact that
+when using '==' we are comparing expression (:a + :b) to expression :z with '=='.  When the 
+comparison is executed, the system tries to evaluate :a, :b and :z, and those symbols, at 
+this time are not bound to anything and we get a "object 'a' not found" message.  
+If we only use functional notation, this type of error will never occur.
+
+## Expressions with R methods
+
+It is often necessary to create an expression that uses a method or function.  For instance, in
+mathematics, it's quite natural to write an expressin such as $y = sin(x)$. In this case, the
+'sin' function is part of the expression and should not immediately executed. Now, let's say
+that 'x' is an angle of 45$^\circ$ and we acttually want our expression to be $y = 0.850...$.
+When we want the function to be part of the expression, we call the function preceeding it
+by the letter E, such as 'E.sin(x)'
+
+
+```ruby
+exp7 = :y.assign E.sin(:x)
+puts exp7
+```
+
+```
+## y <- sin(x)
+```
+However, if we want the function to be evaluated, then
+we use the normal call to function with R as 'R.sin(x)'.
+
+
+```ruby
+x = 45
+exp8 = :y.assign R.sin(x)
+puts exp8
+```
+
+```
+## y <- 0.850903524534118
+```
+# Filtering using expressions
+
+Now that we now how to write expression, we can use then to filter a data frame by expressions.  
+Let's first start by creating a simple data frame with two columns named 'x' and 'y'
 
 
 ```ruby
 @df = R.data__frame(x: (1..3), y: (3..1))
 puts @df
-
-puts @df.filter(:x.eq 1)
 ```
 
 ```
@@ -69,10 +279,37 @@ puts @df.filter(:x.eq 1)
 ## 1 1 3
 ## 2 2 2
 ## 3 3 1
+```
+In the code bellow we want to filter the data frame by rows in which the value of 'x' is
+equal to 1.
+
+
+```ruby
+puts @df.filter(:x.eq 1)
+```
+
+```
 ##   x y
 ## 1 1 3
 ```
 
+In R, and when coding with 'tidyverse', arguments to a function are usually not 
+*referencially transparent*. That is, ou can’t replace a value with a seemingly equivalent 
+object that you’ve defined elsewhere. In other words, this code
+
+
+```r
+my_var <- x
+filter(df, my_var == 1)
+```
+Generates the following error: "object 'x' not found.
+
+However, in Ruby and Galaaz, arguments are referencially transparent as can be seen by the 
+code bellow.  Note, initally that 'my_var = :x' will not give the error "object 'x' not found" 
+since ':x' is treated as an expression and assigned to my\_var. Then when doing (my\_var.eq 1), 
+my\_var is a variable that resolves to ':x' and it becomes equivalent to (:x.eq 1) which is
+what we want.
+
 
 ```ruby
 my_var = :x
@@ -83,9 +320,10 @@ puts @df.filter(my_var.eq 1)
 ##   x y
 ## 1 1 3
 ```
+As stated by Hardley
 
->> dplyr code is ambiguous. Depending on what variables are defined where, 
->> filter(df, x == y) could be equivalent to any of:
+> dplyr code is ambiguous. Depending on what variables are defined where, 
+> filter(df, x == y) could be equivalent to any of:
 
 ```
 df[df$x == df$y, ]
@@ -93,44 +331,419 @@ df[df$x == y, ]
 df[x == df$y, ]
 df[x == y, ]
 ```
+In galaaz this ambiguity does not exist, filter(df, x.eq y) is not a valid expression as 
+expressions are build with symbols.  In doing filter(df, :x.eq y) we are looking for elements
+of the 'x' column that are equal to a previously defined y variable.  Finally, 
+filter(df, :x.eq :y) we are looking for elements in which the 'x' column value is equal to
+the 'y' column value. This can be seen in the following two chunks of code:
+
+
+```ruby
+@y = 1
+@x = 2
+
+# looking for values where the 'x' column is equal to the 'y' column
+puts @df.filter(:x.eq :y)
+```
+
+```
+##   x y
+## 1 2 2
+```
+
+
+```ruby
+# looking for values where the 'x' column is equal to the 'y' variable
+# in this case, the number 1
+puts @df.filter(:x.eq @y)
+```
+
+```
+##   x y
+## 1 1 3
+```
+# Writing a function that applies to different data sets
+
+
+
+```
+mutate(df1, y = a + x)
+mutate(df2, y = a + x)
+mutate(df3, y = a + x)
+mutate(df4, y = a + x)
+```
 
-In galaaz this ambiguity does not exist
+Here we create a mutate_y Ruby method.
 
 
 ```ruby
-y = 2
-x = 2
+def mutate_y(df)
+  df.mutate(:y.assign :a + :x)
+end
+```
+
+Note that contrary to what happens in R, method mutate_y will fail independetly from the fact
+that variable 'a' is defined or not.
 
-@df[:x.eq :y, :all]
-@df[:x.eq y, :all]
-# @df[x.eq :y, :all]
-# @df[x == y, :all]
+
+```ruby
+df1 = R.data__frame(x: (1..3))
+puts df1
+a = 10
+mutate_y(df1)
 ```
 
 ```
 ## Message:
-##  wrong number of arguments (given 2, expected 1)
+##  Error in mutate_impl(.data, dots) :
+##   Evaluation error: object 'a' not found.
+## In addition: Warning message:
+## In mutate_impl(.data, dots) :
+##   mismatched protect/unprotect (unprotect with empty protect stack) (RError)
+## Translated to internal error
 ```
 
+# Different expressions
+
+
+```r
+df <- data.frame(
+  g1 = c(1, 1, 2, 2, 2),
+  g2 = c(1, 2, 1, 2, 1),
+  a = sample(5),
+  b = sample(5)
+)
+
+d2 <- df %>%
+  group_by(g1) %>%
+  summarise(a = mean(a))
+	       
+as.data.frame(d2)	       
 ```
-## Message:
-##  /home/rbotafogo/desenv/galaaz/lib/R_interface/rbinary_operators.rb:134:in `eq'
-## (eval):4:in `exec_ruby'
-## /home/rbotafogo/desenv/galaaz/lib/util/exec_ruby.rb:138:in `instance_eval'
-## /home/rbotafogo/desenv/galaaz/lib/util/exec_ruby.rb:138:in `exec_ruby'
-## /home/rbotafogo/desenv/galaaz/lib/gknit/knitr_engine.rb:650:in `block in initialize'
-## /home/rbotafogo/desenv/galaaz/lib/R_interface/ruby_callback.rb:77:in `call'
-## /home/rbotafogo/desenv/galaaz/lib/R_interface/ruby_callback.rb:77:in `callback'
-## (eval):3:in `function(...) {\n          rb_method(...)'
-## unknown.r:1:in `in_dir'
-## unknown.r:1:in `block_exec'
-## /home/rbotafogo/lib/graalvm-ce-1.0.0-rc12/jre/languages/R/library/knitr/R/block.R:91:in `call_block'
-## /home/rbotafogo/lib/graalvm-ce-1.0.0-rc12/jre/languages/R/library/knitr/R/block.R:6:in `process_group.block'
-## /home/rbotafogo/lib/graalvm-ce-1.0.0-rc12/jre/languages/R/library/knitr/R/block.R:3:in `<no source>'
-## unknown.r:1:in `withCallingHandlers'
-## unknown.r:1:in `process_file'
-## unknown.r:1:in `<no source>'
-## unknown.r:1:in `<no source>'
-## <REPL>:5:in `<repl wrapper>'
-## <REPL>:1
+
+```
+##   g1 a
+## 1  1 3
+## 2  2 3
+```
+
+```r
+d2 <- df %>%
+  group_by(g2) %>%
+  summarise(a = mean(a))
+	       
+as.data.frame(d2)	       
+```
+
+```
+##   g2        a
+## 1  1 3.666667
+## 2  2 2.000000
+```
+
+Trying to write a function in R that will receive two argumens, the first a variable and
+the second an expression is not trivia. As shown by Hardley, one might expect this function 
+to do the trick:
+
+
+```r
+my_summarise <- function(df, group_var) {
+  df %>%
+    group_by(group_var) %>%
+    summarise(a = mean(a))
+}
+
+# my_summarise(df, g1)
+#> Error: Column `group_var` is unknown
+```
+
+In order to solve this problem, coding with dplyr requires the introduction of many new concepts
+and functions such as 'quo', 'quos', 'enquo', 'enquos', '!!' (bang bang), '!!!' (triple bang).
+
+Now, let's try to implement the same function in galaaz.  The next code block first prints the
+'df' data frame define previously in R, then creates the my_summarize function and calls it 
+passing the R data frame and the group by variable ':g1'
+
+
+```ruby
+puts ~:df
+print "\n"
+
+def my_summarize(df, group_var)
+  df.group_by(group_var).
+    summarize(a: E.mean(:a))
+end
+
+puts my_summarize((~:df), :g1).as__data__frame
+```
+
+```
+##   g1 g2 a b
+## 1  1  1 5 2
+## 2  1  2 1 5
+## 3  2  1 2 4
+## 4  2  2 3 1
+## 5  2  1 4 3
+## 
+##   g1 a
+## 1  1 3
+## 2  2 3
+```
+It works!!! Well let's make sure this was not just some coincidence
+
+
+```ruby
+puts my_summarize((~:df), :g2).as__data__frame
+```
+
+```
+##   g2        a
+## 1  1 3.666667
+## 2  2 2.000000
+```
+
+Great, everything is fine! No magic, no new functions, no complexities, just normal, standard Ruby
+code.  If you've ever done NSE in R, this certainly feels much safer and easy to implement.
+
+# Different input variables
+
+In the previous section we've managed to get rid of all NSE formulation for a simple example, but
+does this remain true for more complex examples, or will the Ruby way prove inpractical for
+more complex code?
+
+In the next example Hardley proposes us to write a function that given an expression such as 'a'
+or 'a * b', calculates three summaries.  What we want a function that does the same as these R
+statements:
+
+```
+summarise(df, mean = mean(a), sum = sum(a), n = n())
+#> # A tibble: 1 x 3
+#>    mean   sum     n
+#>   <dbl> <int> <int>
+#> 1     3    15     5
+
+summarise(df, mean = mean(a * b), sum = sum(a * b), n = n())
+#> # A tibble: 1 x 3
+#>    mean   sum     n
+#>   <dbl> <int> <int>
+#> 1   9.6    48     5
+```
+
+Let's try it in galaaz:
+
+
+```ruby
+def my_summarise2(df, expr)
+  df.summarize(
+    mean: E.mean(expr),
+    sum: E.sum(expr),
+    n: E.n
+  )
+end
+
+puts my_summarise2((~:df), :a)
+puts my_summarise2((~:df), :a * :b)
+```
+
+```
+##   mean sum n
+## 1    3  15 5
+##   mean sum n
+## 1  7.6  38 5
+```
+
+Once again, there is no need to use any special theory or functions.  The only point to be 
+careful about is the use of 'E' to build an expression that uses the mean, sum and n.
+
+# Different input and output variable
+
+Now the next challenge presented by Hardley is to vary the name of the output variables based on 
+the received expression.  So, if the input expression is 'a', we want our data frame columns to
+be named 'mean\_a' and 'sum\_a'.  Now, if the input expression is 'b', columns
+should be named 'mean\_b' and 'sum\_b'.
+
+```
+mutate(df, mean_a = mean(a), sum_a = sum(a))
+#> # A tibble: 5 x 6
+#>      g1    g2     a     b mean_a sum_a
+#>   <dbl> <dbl> <int> <int>  <dbl> <int>
+#> 1     1     1     1     3      3    15
+#> 2     1     2     4     2      3    15
+#> 3     2     1     2     1      3    15
+#> 4     2     2     5     4      3    15
+#> # … with 1 more row
+
+mutate(df, mean_b = mean(b), sum_b = sum(b))
+#> # A tibble: 5 x 6
+#>      g1    g2     a     b mean_b sum_b
+#>   <dbl> <dbl> <int> <int>  <dbl> <int>
+#> 1     1     1     1     3      3    15
+#> 2     1     2     4     2      3    15
+#> 3     2     1     2     1      3    15
+#> 4     2     2     5     4      3    15
+#> # … with 1 more row
+```
+
+Here is our Ruby code
+
+
+```ruby
+def my_mutate(df, expr)
+  mean_name = "mean_#{expr.to_s}"
+  sum_name = "sum_#{expr.to_s}"
+
+  df.mutate(mean_name => E.mean(expr),
+            sum_name => E.sum(expr))
+end
+
+puts my_mutate((~:df), :a)
+puts my_mutate((~:df), :b)
+```
+
+```
+##   g1 g2 a b mean_a sum_a
+## 1  1  1 5 2      3    15
+## 2  1  2 1 5      3    15
+## 3  2  1 2 4      3    15
+## 4  2  2 3 1      3    15
+## 5  2  1 4 3      3    15
+##   g1 g2 a b mean_b sum_b
+## 1  1  1 5 2      3    15
+## 2  1  2 1 5      3    15
+## 3  2  1 2 4      3    15
+## 4  2  2 3 1      3    15
+## 5  2  1 4 3      3    15
+```
+It really seems that "Non Standard Evaluation" is actually quite standard in Galaaz! But, you 
+might have noticed a small change in the way the arguments to the mutate method were called.
+In a previous example we used df.summarise(mean: E.mean(:a), ...) where the column name was
+followed by a ':' colom.  In this example, we have df.mutate(mean_name => E.mean(expr), ...)
+and variable mean\_name is not followed by ':' but by '=>'.  This is standard Ruby notation.
+
+[explain....]
+
+# Capturing multiple variables
+
+
+```ruby
+def my_summarise3(df, *group_vars)
+  df.group_by(*group_vars).
+    summarise(a: E.mean(:a))
+end
+
+puts my_summarise3((~:df), :g1, :g2).as__data__frame
+```
+
+```
+##   g1 g2 a
+## 1  1  1 5
+## 2  1  2 1
+## 3  2  1 3
+## 4  2  2 3
+```
+
+# Advanced dplyr features
+https://www.r-bloggers.com/programming-with-dplyr-by-using-dplyr/
+
+
+```ruby
+puts (~:starwars).head.as__data__frame
+```
+
+```
+##             name height mass  hair_color  skin_color eye_color birth_year
+## 1 Luke Skywalker    172   77       blond        fair      blue       19.0
+## 2          C-3PO    167   75        <NA>        gold    yellow      112.0
+## 3          R2-D2     96   32        <NA> white, blue       red       33.0
+## 4    Darth Vader    202  136        none       white    yellow       41.9
+## 5    Leia Organa    150   49       brown       light     brown       19.0
+## 6      Owen Lars    178  120 brown, grey       light      blue       52.0
+##   gender homeworld species
+## 1   male  Tatooine   Human
+## 2   <NA>  Tatooine   Droid
+## 3   <NA>     Naboo   Droid
+## 4   male  Tatooine   Human
+## 5 female  Alderaan   Human
+## 6   male  Tatooine   Human
+##                                                                                                                                       films
+## 1                                           Revenge of the Sith, Return of the Jedi, The Empire Strikes Back, A New Hope, The Force Awakens
+## 2                    Attack of the Clones, The Phantom Menace, Revenge of the Sith, Return of the Jedi, The Empire Strikes Back, A New Hope
+## 3 Attack of the Clones, The Phantom Menace, Revenge of the Sith, Return of the Jedi, The Empire Strikes Back, A New Hope, The Force Awakens
+## 4                                                              Revenge of the Sith, Return of the Jedi, The Empire Strikes Back, A New Hope
+## 5                                           Revenge of the Sith, Return of the Jedi, The Empire Strikes Back, A New Hope, The Force Awakens
+## 6                                                                                     Attack of the Clones, Revenge of the Sith, A New Hope
+##                             vehicles                starships
+## 1 Snowspeeder, Imperial Speeder Bike X-wing, Imperial shuttle
+## 2                                                            
+## 3                                                            
+## 4                                             TIE Advanced x1
+## 5              Imperial Speeder Bike                         
+## 6
+```
+
+
+```r
+grouped_mean <- function(data, grouping_variables, value_variables) {
+  data %>%
+    group_by_at(grouping_variables) %>%
+    mutate(count = n()) %>%
+    summarise_at(c(value_variables, "count"), mean, na.rm = TRUE) %>%
+    rename_at(value_variables, funs(paste0("mean_", .)))
+    }
+
+gm = starwars %>% 
+   grouped_mean("eye_color", c("mass", "birth_year"))
+
+as.data.frame(gm)   
+```
+
+```
+##        eye_color mean_mass mean_birth_year count
+## 1          black  76.28571        33.00000    10
+## 2           blue  86.51667        67.06923    19
+## 3      blue-gray  77.00000        57.00000     1
+## 4          brown  66.09231       108.96429    21
+## 5           dark       NaN             NaN     1
+## 6           gold       NaN             NaN     1
+## 7  green, yellow 159.00000             NaN     1
+## 8          hazel  66.00000        34.50000     3
+## 9         orange 282.33333       231.00000     8
+## 10          pink       NaN             NaN     1
+## 11           red  81.40000        33.66667     5
+## 12     red, blue       NaN             NaN     1
+## 13       unknown  31.50000             NaN     3
+## 14         white  48.00000             NaN     1
+## 15        yellow  81.11111        76.38000    11
+```
+
+
+```ruby
+def grouped_mean(data, grouping_variables, value_variables)
+  data.
+    group_by_at(grouping_variables).
+    mutate(count: E.n).
+    summarise_at(E.c(value_variables, "count"), ~:mean, na__rm: true).
+    rename_at(value_variables, R.funs(E.paste0("mean_", value_variables)))
+end
+
+puts grouped_mean((~:starwars), "eye_color", R.c("mass", "birth_year")).as__data__frame
+```
+
+```
+##        eye_color mean_mass mean_birth_year count
+## 1          black  76.28571        33.00000    10
+## 2           blue  86.51667        67.06923    19
+## 3      blue-gray  77.00000        57.00000     1
+## 4          brown  66.09231       108.96429    21
+## 5           dark       NaN             NaN     1
+## 6           gold       NaN             NaN     1
+## 7  green, yellow 159.00000             NaN     1
+## 8          hazel  66.00000        34.50000     3
+## 9         orange 282.33333       231.00000     8
+## 10          pink       NaN             NaN     1
+## 11           red  81.40000        33.66667     5
+## 12     red, blue       NaN             NaN     1
+## 13       unknown  31.50000             NaN     3
+## 14         white  48.00000             NaN     1
+## 15        yellow  81.11111        76.38000    11
 ```