statsailr 0.7.1 → 0.7.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 51e84bd76b9ed92c47e11f3baac54ade7ef338b4ab8a1935e5aaac106f559c65
-  data.tar.gz: 2f68acb064cdb8297052ee9246a49da73db1a70750d2290b83eebe2e21c15db2
+  metadata.gz: 8701d54a846ae0d81c7f91080586c7dd81b05c081743ba84606c46ecc53f25a8
+  data.tar.gz: 7f2636ea0b00008f160e303b389d977b94e7698a1eb800603850cff38901b14e
 SHA512:
-  metadata.gz: c1c37162684765074a27dd2a2ec2373c939ab5b125df6ae5a7ba1bd86dfd66fe9ece1e5f39a525db403a296daae8cf594687b8ba4d405cff78202472324b1389
-  data.tar.gz: 4f0908c784e59eefec18664ba79ea33cc24148ac3c5de2a005b64553a2c9fd5a48d60d9821140f3c3b2065ea757c4f8700ef1d8e2ff99171a175ed137c747cdc
+  metadata.gz: 4007addec1d2a15f05c62bef40850a31648b4e2099ac7cf79890f0cb480854561990b478420bf3ab750089c88b1013b7fe4261c15738cb608b893257c2ef5134
+  data.tar.gz: e38f0f64d035ab90bfdd8c052f63c1b4f0cd605cac841ddaefa18de6af127658f2c6453b4fbd74b7dfcf31e29920ba26c9b6db8b8211ee452ad983dce8c0ae3e

data/README.md

@@ -1,16 +1,16 @@
 # StatSailr
 
-StatSailr provides a platform for users to focus on statistics. The backend statistics engine is [R](https://www.r-project.org/), so the results are reliable. The SataSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. Each block has its way of writing instructions, which works as an intuitive interface for R.
+StatSailr provides a platform for users to focus on statistics. The backend statistics engine is [R](https://www.r-project.org/), so the results are reliable. The StatSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. Each block has its way of writing instructions, which works as an intuitive interface for R.
 
 
 ## Overview
 
-StatSailr is a Ruby program that enables users to manipulate data and to apply statistical procedures in an intuiitive way. StatSailr converts StatSailr script into R's internal representation, and executes it. The SataSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. TOPLEVEL loads and saves datasets. DATA blocks utilize DataSailr package as its backend, which enables wrting data manipulation insturctions in a rowwise way. PROC blocks have a series of PROC instructions, which are converted to R functions and are executed sequentially.
+StatSailr is a Ruby program that enables users to manipulate data and to apply statistical procedures in an intuitive way. StatSailr converts StatSailr script into R's internal representation, and executes it. The SataSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. TOPLEVEL loads and saves datasets. DATA blocks utilize DataSailr package as its backend, which enables wrting data manipulation instructions in a rowwise way. PROC blocks have a series of PROC instructions, which are converted to R functions and are executed sequentially.
 
 
 ### Quick Introduction
 
-The following is an example of StatSailr script. It consists of TOPLEVEL instruction, DATA block and PROC blocks.
+The following is an example of StatSailr script. It consists of TOPLEVEL, DATA and PROC blocks.
 
 ```
 READ builtin="mtcars"
@@ -30,6 +30,8 @@ END
 PROC REG data=new_mtcars
   lm hp ~ powerful
 END
+
+SAVE new_mtcars file="./new_mtcars.rda" type="rdata"
 ```
 
 Save this script as, say, create_new_mtcars.slr and run.
@@ -124,7 +126,7 @@ $ gem install statsailr
 * Then, 'sailr' and 'sailrREPL' become available.
 
 
-## Grammar of StatSalr
+## Grammar of StatSailr
 
 StatSailr script consists of three parts, TOPLEVEL, DATA block and PROC block.
 
@@ -140,7 +142,7 @@ TOPLEVEL statements import and save datasets, and also StatSailr's current worki
 
 Datasets can come from built-in datasets and files. In R, built-in datasets can be used by data() function, and StatSailr READ with 'builtin=' option does the same job.
 
-When importing datasets from files, currently there are three types of files availble, RDS, RDATA and CSV. RDS contains a single R object, and when you import it, you can neme the object using 'as=' option. If you omit 'as=' option, the name is created based on the filename. RDATA can contain multiple R objects, but their names cannot be changed when importing that are decided when saveing. CSV is a comma separated values file.
+When importing datasets from files, currently there are three types of files available, RDS, RDATA and CSV. RDS contains a single R object, and when you import it, you can name the object using 'as=' option. If you omit 'as=' option, the name is created based on the filename. RDATA can contain multiple R objects, but their names cannot be changed when importing that are decided when saving. CSV is a comma separated values file.
 
 These dataset types are decided as follows. If you specify 'type' option, its type is used. If you do not specify it, it is inferred from the file extension. 
 
@@ -162,7 +164,7 @@ SAVE new_mtcars file="./new_mtcars.csv" type="csv"
 
 * show and change working directory
 
-The concept of working directory is really important. If you run your SailrScript in an unintentional place and ouput some data, those data might overwrite your importan data.
+The concept of working directory is really important. If you run your StatSailr script in an unintentional place and output some data, those data might overwrite your important data.
 
 The default working directory should be the directory where StatSailr script file exists. If you do not specify script file, such as when you run REPL, the default working directory should be the directory where you start your command (such as REPL).
 
@@ -176,7 +178,7 @@ SETWD "~/sailr_workspace"
 
 ### DATA block
 
-DATA block starts with the line of DATA, new dataset name and DATA options. For DATA options, 'set=' option is required which specify the input dataset. (Note that unlinke PROC options where 'data=' usually speifies input dataset, "set=" does the same job in DATA block. This difference comes from just an aesthetic reason.) Lines that follws the first DATA line represent how to manipulate input dataset. The lines are writtein in DataSailr script. END keyword specifies the end of DATA block.
+DATA block starts with the line of DATA, new dataset name and DATA options. For DATA options, 'set=' option is required which specify the input dataset. (Note that unlink PROC options where 'data=' usually specifies input dataset, "set=" does the same job in DATA block. This difference comes from just an aesthetic reason.) Lines that follws the first DATA line represent how to manipulate input dataset. The lines are written in DataSailr script. END keyword specifies the end of DATA block.
 
 ```
 DATA new_dataset set=ori_dataset
@@ -194,24 +196,23 @@ The DataSailr script is described in detail at [its official website](https://da
 
 Briefly speaking, 
 
-1. Rowwise dataset manipulation
-    + Varables correspond to column names.
+1. Row by row dataset processing
+    + Variable names correspond to column names.
 2. Simplified available types
-    + Int, Double and String(=Characters) are basic types, that can be used in DataSailr script and also those values can be assigned to column value (of dataset).
-    + Regular expression and boolean are not assigned to dataset. They can be held by variables, but do not modify dataset.
+    + Int, Double and String are basic types, that can be used in DataSailr script and also those values can be assigned to data sets.
+    + Regular expression and Boolean do not affect dataset.
         + Regular expression is used for if condition and extracting substrings.
         + Boolean is internal type that is used for if condition.
-3. Assignment operator (=) creates new column with the column name same as the variable left-hand-side(LHS) of assignment operator.
+3. Assignment operator (=) creates new column or updates the existing column with the same column name as the variable left-hand-side of assignment operator.
     + If the variable already exits, the column is updated.
-    + Exceptions are assigning regular expressions and boolen, which do not modify dataset. Variables pointing to those objects are only used in the script.
-4. Control flow can be done using if-(else if)-(else) statement.
-    + Condition part needs parentheses (), and statement part require curly braces.
+    + Exceptions are assigning Regular Expressions, which do not modify dataset. This is used to reuse them at different lines of code.
+4. If-else statement is the only control flow statement.
 5. Arithmetic operators
 6. Built-in functions
     + Mainly used to manipulate strings.
-7. Regular expression
+7. Regular Expression
 8. UTF-8
-    + Use UTF-8 for script and dataset. It is highly recommended that dataset should be saved using UTF-8 beforehand.
+    + It is highly recommended t use UTF-8 for script and dataset. Data set needs to be saved in UTF-8 beforehand.
 9. push!() and discard!() built-in functions
     + push!() can create multiple rows from current row.
     + discard!() can filter out specific rows by being used with if statements.
@@ -228,7 +229,7 @@ The PROCs gem holds basic PROC settings, such as PRINT and PLOT, and its main cl
 
 #### Format
 
-A typical PROC block looks like the follwing. The first line start with PROC, followed by PROC command name and PROC options. The PROC first line is followed by a list of instuctions with their main and optional arguments. The PROC block ends with END keyword.
+A typical PROC block looks like the following. The first line start with PROC, followed by PROC command name and PROC options. The PROC first line is followed by a list of instructions with their main and optional arguments. The PROC block ends with END keyword.
 
 ```
 PROC COMMAND proc_opts
@@ -241,7 +242,7 @@ END
 * COMMAND
     + PROC command name
 * proc_opts
-    + This parameter can be refered from any instructions in this block. In other words, this can be seen as global settings of this PROC block.
+    + This parameter can be referred from any instructions in this block. In other words, this can be seen as global settings of this PROC block.
     + Internally, this parameter is managed by RBridge::ParamManager.
 * proc_statement line
     + Each line consists of instruction, main argument and optional arguments. Main argument and optional arguments are separated by slash(/).
@@ -253,7 +254,7 @@ END
             + How main argument part is parsed varies on each instruction. This is defined in main_arg_and_how_to_treat variable in setting.
         + opt_args
             + Optional arguments. Values specified in opt_args are also passed to R function's argument.
-            + opt_args part consists of (a) key-value(s) or key(s).
+            + opt_args part consists of (a) key-value(s).
             + Internally, this argument parsing is conducted by methods in STSBlockParseProcOpts.
 
 

data/lib/statsailr/block_builder/sts_block.rb

@@ -1,8 +1,82 @@
 require "r_bridge"
 require_relative "./sts_block_parse_proc_opts.rb"
 
+module QuotedStringSupport
+  def interpret_escape_sequences(str)
+    # This deals with escape sequences in double quoted string literals
+    # The behavior should be same as libsailr (or datasailr)
+    new_str = ""
+    str_array = str.split(//)
+    idx = 0
+    while( idx < str_array.size) do
+      c = str_array[idx]
+      if(c == "\\")
+        idx = idx + 1
+        c = str_array[idx]
+        raise "Tokenizer error: double quoted string literal should never end with \\" if idx >= str_array.size
+        case c
+        when 't'
+          new_str << "\t"
+        when 'n'
+          new_str << "\n"
+        when 'r'
+          new_str << "\r"
+        when "\\"
+          new_str << "\\"
+        when "\'"
+          new_str << "\'"
+        when "\""
+          new_str << "\""
+        when '?'
+          new_str << '?'
+        else
+          new_str << c
+        end
+      else
+        new_str << c
+      end
+      idx = idx + 1
+    end
+    return new_str
+  end
+
+  def escape_backslashes(str)
+    str.gsub("\\", "\\\\")
+  end
+end
+
 module BlockSupport
-  def type_adjust(obj , type)
+  include QuotedStringSupport
+  class QuotedStringR
+    include QuotedStringSupport
+    def initialize(str, quote_type)
+      raise ":dq or :sq should be specified for quote_type" unless [:dq, :sq].include? quote_type
+      @ori_str = str
+      @quote_type = quote_type
+    end
+
+    def to_s
+      to_s_for_r_bridge
+    end
+
+    def to_s_for_r_bridge
+      if @quote_type == :dq
+        interpret_escape_sequences( @ori_str )
+      elsif
+        @ori_str
+      end
+    end
+
+    def to_s_for_r_parsing
+      if @quote_type == :dq
+        %q{"} + @ori_str + %q{"}
+      elsif @quote_type == :sq
+        %q{'} + escape_backslashes( @ori_str ) + %q{'}
+      end
+    end
+  end
+
+  def type_adjust(obj , type , *opts)
     case type
     when :ident
       if obj.is_a?(String)
@@ -22,6 +96,28 @@ module BlockSupport
       else
         raise "GramNode with inconsistent type(#{type.to_s}) and object(#{obj.class})"
       end
+    when :sq_string
+      if obj.is_a?(String)
+        unless opts.include?( :retain_input_string )
+          # default behavior
+          result = obj
+        else
+          result = QuotedStringR.new( obj, :sq )
+        end
+      else
+        raise "GramNode with inconsistent type(#{type.to_s}) and object(#{obj.class})"
+      end
+    when :dq_string
+      if obj.is_a?(String)
+        unless opts.include?( :retain_input_string )
+          # default behavior
+          result = interpret_escape_sequences( obj )
+        else
+          result = QuotedStringR.new( obj, :dq )
+        end
+      else
+        raise "GramNode with inconsistent type(#{type.to_s}) and object(#{obj.class})"
+      end
     when :sign
       if obj.is_a?(String)
         result = RBridge::SignR.new(obj)
@@ -137,10 +233,16 @@ class ProcBlock
           idx = 0
           while idx < proc_stmt_arg_ori.size() do
             elem = proc_stmt_arg_ori[idx]
-            if( elem.type == :sign && elem.e1 == "/" )
-              break
+            next_elem = proc_stmt_arg_ori[idx + 1]
+            next_next_elem = proc_stmt_arg_ori[idx + 2]
+            if(elem.type == :sign && elem.e1 == "/" ) then
+               if( ! next_elem.nil? && next_elem.type == :ident &&   # After /, optional arguments start
+                   ! next_next_elem.nil? && next_next_elem.type == :sign && next_next_elem.e1 == "=") ||
+                   next_elem.nil? then  # After /, there is nothing
+                 break
+               end
             else
-              x = type_adjust( elem.e1, elem.type )
+              x = type_adjust( elem.e1, elem.type, :retain_input_string )
               proc_stmt_arg << x
               idx = idx + 1
             end

data/lib/statsailr/block_builder/sts_block_parse_proc_opts.rb

@@ -50,7 +50,6 @@ class STSBlockParseProcOpts
 # arg_opt  : IDENT = primary
 #          | IDENT = array
 #          | IDENT = func 
-#          | IDENT
 #
 # parimary : STRING
 #          | NUM
@@ -87,15 +86,15 @@ class STSBlockParseProcOpts
         else
           opt_value = ident() # According to BNF, this should be parimary(). However, ident() is more direct and makes sense here.
         end
-      elsif [:string, :num].include? peek.type
+      elsif [:dq_string, :sq_string, :num].include? peek.type
          next_token()
          opt_value = primary()
       else
         p current_token()
-        raise "the token should be :ident or primaries such as :ident, :num and :string after = . Current token: " + current_token().type.to_s
+        raise "the token should be :ident or primaries such as :ident, :num and :string after = . Next token: " + peek.type.to_s
       end
     else
-      opt_value = true
+      raise "proc instruction optional argumeents should be in the form of a sequence of 'key = value'"
     end
     @result_hash[opt_key.to_s] = opt_value
   end
@@ -127,12 +126,14 @@ class STSBlockParseProcOpts
     case current_token.type
     when :ident
       return ident()
-    when :string
+    when :dq_string
+      return string()
+    when :sq_string
       return string()
     when :num
       return num()
     else
-      raise "the current token should be :ident, :string or :num."
+      raise "the current token should be :ident, :dq_string, :sq_string or :num."
     end
   end
 
@@ -155,8 +156,13 @@ class STSBlockParseProcOpts
   end
 
   def string()
-    raise "the current token should be string" if current_token.type != :string 
-    return type_adjust( current_token.e1, :string)
+    if (current_token.type != :dq_string)
+      return type_adjust( current_token.e1, :dq_string)
+    elsif (current_token.type != :sq_string)
+      return type_adjust( current_token.e1, :sq_string)
+    else
+      raise "the current token should be string (:dq_string or :sq_string)"
+    end
   end
 
   def num()

data/lib/statsailr/block_to_r/sts_lazy_func_gen.rb

@@ -40,6 +40,50 @@ module LazyFuncGeneratorSettingUtility
     return RBridge.create_strvec( ary.map(){|elem| elem.to_s } )
   end
 
+  def read_symbols_or_functions_as_strvec(ary)
+    deapth_ary = Array.new( ary.size )
+    idx = 0
+    last_idx = ary.size - 1
+    deapth = 0
+    while( idx <= last_idx )
+      if( ary[idx].is_a?(RBridge::SymbolR) && (ary[idx + 1].is_a?(RBridge::SignR) && ary[idx + 1].to_s == "("))
+        # function starts
+        deapth_ary[idx] = deapth
+        deapth = deapth + 1
+        idx = idx + 1
+        deapth_ary[idx] = deapth
+      elsif( ary[idx].is_a?(RBridge::SignR) && ary[idx].to_s == "(")
+        # parenthesis starts
+        deapth = deapth + 1
+        deapth_ary[idx] = deapth
+      elsif( ary[idx].is_a?(RBridge::SignR) && ary[idx].to_s == ")")
+        # parenthesis ends or function ends
+        deapth_ary[idx] = deapth
+        deapth = deapth - 1
+      else
+        deapth_ary[idx] = deapth
+      end
+      idx = idx + 1
+    end
+
+    result_ary = []
+    ary.zip( deapth_ary).each(){|elem, deapth|
+      if elem.respond_to? :to_s_for_r_parsing
+        elem_str = elem.to_s_for_r_parsing
+      else
+        elem_str = elem.to_s
+      end
+
+      if( deapth == 0)
+        result_ary.push( elem_str )
+      else
+        result_ary.last << " " << elem_str
+      end
+    }
+
+    return RBridge.create_strvec( result_ary )
+  end
+
   def result( name , *addl )
     if addl.empty?
       return RBridge::RResultName.new(name)