statsailr 0.7.1

data/README.md

@@ -0,0 +1,287 @@
+# StatSailr
+
+StatSailr provides a platform for users to focus on statistics. The backend statistics engine is [R](https://www.r-project.org/), so the results are reliable. The SataSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. Each block has its way of writing instructions, which works as an intuitive interface for R.
+
+
+## Overview
+
+StatSailr is a Ruby program that enables users to manipulate data and to apply statistical procedures in an intuiitive way. StatSailr converts StatSailr script into R's internal representation, and executes it. The SataSailr script consists of three major blocks, TOPLEVEL, DATA, and PROC. TOPLEVEL loads and saves datasets. DATA blocks utilize DataSailr package as its backend, which enables wrting data manipulation insturctions in a rowwise way. PROC blocks have a series of PROC instructions, which are converted to R functions and are executed sequentially.
+
+
+### Quick Introduction
+
+The following is an example of StatSailr script. It consists of TOPLEVEL instruction, DATA block and PROC blocks.
+
+```
+READ builtin="mtcars"
+
+DATA new_mtcars set=mtcars
+  if(hp > 100){
+    powerful = 1
+  }else{
+    powerful = 0
+  }
+END
+
+PROC PRINT data=new_mtcars
+  head 10
+END
+
+PROC REG data=new_mtcars
+  lm hp ~ powerful
+END
+```
+
+Save this script as, say, create_new_mtcars.slr and run.
+
+```
+# From command line
+# By installing statsailr gem, sailr command should become available.
+
+sailr create_new_mtcars.slr
+```
+
+
+### Commands and options
+
+#### sailr
+
+sailr command executes StatSailr script.
+
+* sailr [filename]
+    + --procs-gem <gemX,gemY,...>
+        + allow users to specify PROCs gems when they need to use other than default PROCs gem, statsailr_procs_base.
+        + To specify multiple gems, separate them with comma(,)
+
+
+#### sailrREPL
+
+Not only sailr command, but sailrREPL command is also provided. It enables interactive executaion of StatSailr script. (For non-UNIX system, sailrREPL --thread can be used, which is thread based.)
+
+* sailrREPL
+    + --procs-gem <gemX,gemY,...>
+        + allow users to specify PROCs gems
+    + --thread
+        + Thread based implementation of REPL. (Windows requires this option.)
+
+
+sailrREPL requires the explicit commands. Lines are stored as input script until the following commands are executed. Commands within REPL start from !.
+
+
++ !! or !exec : Executes StatSailr script input. (and clears the input stored.)
++ !clear : Clears input script.
++ !exit : Finishes StatSailr REPL.
+
+
+The following example shows how sailrREPL works.
+
+```
+cli > sailrREPL  # start REPL
+
+(^^)v: READ builtin=mtcars
+(^^)v: !!  # execute READ
+  ... output ...
+(^^)v: DATA new_mtcars set=mtcars
+(^^)v:   if( hp > 100){
+(^^)v:     powerful = 1
+(^^)v:   }else{
+(^^)v:     powerful = 0
+(^^)v:   }
+(^^)v: END
+(^^)v: !!  # execute DATA block
+  ... output ...
+(^^)v: PROC PRINT data=new_mtcars
+(^^)v:   head 5
+(^^)v: END
+(^^)v: !!  # execute PROC block
+  ... output ...
+(^^)v: !exit  # exit REPL
+
+cli > 
+
+```
+
+## Installation
+
+### Requirements
+
+* R (>= 4.0 is preferable)
+    + datasailr package
+        + From R's interpreter, execute 'install.packages("datasailr")'.
+* Ruby (>= 2.7)
+    + r_bridge gem
+        + Configure PATH, LD_LIBRARY_PATH or RUBY_DLL_PATH to access R's shared library (libR.so or R.dll).
+            + Details are mentioned in r_bridge gem's README.
+        + After configuring these settings, install r_bridge gem via 'gem install r_bridge'
+
+
+### Install StatSailr
+
+```
+$ gem install statsailr
+```
+
+* Then, 'sailr' and 'sailrREPL' become available.
+
+
+## Grammar of StatSalr
+
+StatSailr script consists of three parts, TOPLEVEL, DATA block and PROC block.
+
+The following shows structures of these blocks. Details about TOPLEVEL, DATA and PROC blocks are documented in the [StatSailr official site](https://statsailr.io)
+
+
+### TOPLEVEL statement
+
+TOPLEVEL statements import and save datasets, and also StatSailr's current working directory.
+
+
+* import and save datasets.
+
+Datasets can come from built-in datasets and files. In R, built-in datasets can be used by data() function, and StatSailr READ with 'builtin=' option does the same job.
+
+When importing datasets from files, currently there are three types of files availble, RDS, RDATA and CSV. RDS contains a single R object, and when you import it, you can neme the object using 'as=' option. If you omit 'as=' option, the name is created based on the filename. RDATA can contain multiple R objects, but their names cannot be changed when importing that are decided when saveing. CSV is a comma separated values file.
+
+These dataset types are decided as follows. If you specify 'type' option, its type is used. If you do not specify it, it is inferred from the file extension. 
+
+
+```
+READ builtin="mtcars"
+READ file="./mtcars.rds" type="rds" as="mtcars"
+READ file="./mtcars.rda" type="rdata"
+READ file="./mtcars.csv" type="csv" # 'as=' can be used optionally.
+```
+
+For StatSailr SAVE, specifying dataset name(s) followed by 'file=' option is required. "type=" option can be omitted, and in such a case, file extension is used to infer the file type.
+
+```
+SAVE new_mtcars file="./new_mtcars.rds" type="rds"
+SAVE mtcars new_mtcars file="./new_mtcars.rda" type="rdata"
+SAVE new_mtcars file="./new_mtcars.csv" type="csv"
+```
+
+* show and change working directory
+
+The concept of working directory is really important. If you run your SailrScript in an unintentional place and ouput some data, those data might overwrite your importan data.
+
+The default working directory should be the directory where StatSailr script file exists. If you do not specify script file, such as when you run REPL, the default working directory should be the directory where you start your command (such as REPL).
+
+The following commands show the current working directory and change it to new directory.
+
+```
+GETWD
+SETWD "~/sailr_workspace"
+```
+
+
+### DATA block
+
+DATA block starts with the line of DATA, new dataset name and DATA options. For DATA options, 'set=' option is required which specify the input dataset. (Note that unlinke PROC options where 'data=' usually speifies input dataset, "set=" does the same job in DATA block. This difference comes from just an aesthetic reason.) Lines that follws the first DATA line represent how to manipulate input dataset. The lines are writtein in DataSailr script. END keyword specifies the end of DATA block.
+
+```
+DATA new_dataset set=ori_dataset
+  // This part is a plain DataSailr script
+  // (e.g.)
+  if(hp > 100){
+    powerful = 1
+  }else{
+    powerful = 0
+  }
+END
+```
+
+The DataSailr script is described in detail at [its official website](https://datasailr.io).
+
+Briefly speaking, 
+
+1. Rowwise dataset manipulation
+    + Varables correspond to column names.
+2. Simplified available types
+    + Int, Double and String(=Characters) are basic types, that can be used in DataSailr script and also those values can be assigned to column value (of dataset).
+    + Regular expression and boolean are not assigned to dataset. They can be held by variables, but do not modify dataset.
+        + Regular expression is used for if condition and extracting substrings.
+        + Boolean is internal type that is used for if condition.
+3. Assignment operator (=) creates new column with the column name same as the variable left-hand-side(LHS) of assignment operator.
+    + If the variable already exits, the column is updated.
+    + Exceptions are assigning regular expressions and boolen, which do not modify dataset. Variables pointing to those objects are only used in the script.
+4. Control flow can be done using if-(else if)-(else) statement.
+    + Condition part needs parentheses (), and statement part require curly braces.
+5. Arithmetic operators
+6. Built-in functions
+    + Mainly used to manipulate strings.
+7. Regular expression
+8. UTF-8
+    + Use UTF-8 for script and dataset. It is highly recommended that dataset should be saved using UTF-8 beforehand.
+9. push!() and discard!() built-in functions
+    + push!() can create multiple rows from current row.
+    + discard!() can filter out specific rows by being used with if statements.
+
+
+### PROC block
+
+#### statsailr_procs_base gem
+
+PROC commands and instructions are now managed in a separate PROCs gem. The gem name is 'statsailr_procs_base' (from ver. 0.71). You need to have the gem installed to use PROCs ,though it is usually installed as a dependency of 'statsailr' gem. If StatSailr complains that PROC PRINT command is not found, please make sure that the PROCs gem is installed.
+
+The PROCs gem holds basic PROC settings, such as PRINT and PLOT, and its main class (StatSailr::ProcsBase) just returns the base path for those PROC settings. Please read the README of the PROCs gem if you are interested in how PROCs are implemented.
+
+
+#### Format
+
+A typical PROC block looks like the follwing. The first line start with PROC, followed by PROC command name and PROC options. The PROC first line is followed by a list of instuctions with their main and optional arguments. The PROC block ends with END keyword.
+
+```
+PROC COMMAND proc_opts
+  instX main_arg / opt_args  # proc_statement X
+  instY main_arg / opt_args
+  instZ main_arg / opt_args
+END
+```
+
+* COMMAND
+    + PROC command name
+* proc_opts
+    + This parameter can be refered from any instructions in this block. In other words, this can be seen as global settings of this PROC block.
+    + Internally, this parameter is managed by RBridge::ParamManager.
+* proc_statement line
+    + Each line consists of instruction, main argument and optional arguments. Main argument and optional arguments are separated by slash(/).
+    + Internally, each line is converted into an R function, and its return value is managed by RBridge::ResultManager. (Whether the result is stored or not can be changed in PROC setting.) 
+        + inst
+            + PROC instruction name. Instruction names are associated with R's function names.
+        + main_arg
+            + Main argument. Value specified in main_arg is passed to R function's argument.
+            + How main argument part is parsed varies on each instruction. This is defined in main_arg_and_how_to_treat variable in setting.
+        + opt_args
+            + Optional arguments. Values specified in opt_args are also passed to R function's argument.
+            + opt_args part consists of (a) key-value(s) or key(s).
+            + Internally, this argument parsing is conducted by methods in STSBlockParseProcOpts.
+
+
+## About plotting device
+
+R outputs graphics to various programs called devices. When using R interactively, graphics are usually output to a window by default. StatSailr uses such a default device, and the device or the plotting window closes as the script execution finishes. 
+
+To keep the plotting result, it is currently recommended to copy the device graphics to some file. dev.copy() does this work. For such PROCs that have plot related instructions, dev.copy instruction are provided. dev.copy can save the graphic on the current graphics device to a file.
+
+```
+// Example
+
+PROC PLOT data=mtcars
+  scatter cyl carb   // This instruction creates scatter plot and displays it on a device (window).
+  dev.copy png / file="./myplot.png"  // This instruction copies graphics on the device to myplot.png file.
+END
+```
+
+
+## License
+
+The gem is available as open source under the terms of the [GPL v3 License](https://www.gnu.org/licenses/gpl-3.0.en.html).
+
+
+## Contact
+
+Your feedback is welcome.
+
+Maintainer: Toshi Umehara toshi@niceume.com
+
+

data/Rakefile

@@ -0,0 +1,10 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task :default => :test

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "statsailr"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/example/blank.slr

@@ -0,0 +1,3 @@
+
+
+

data/example/category.slr

@@ -0,0 +1,5 @@
+READ builtin="mtcars"
+
+PROC CAT data=mtcars
+  crosstable cyl carb /  missing.include = T
+END

data/example/example_read.slr

@@ -0,0 +1,10 @@
+// comment
+/*
+ multipe line comments 
+*/
+
+Read builtin = "mtcars"
+
+READ file = "./iris.csv" as = iris_csv  sep = "," 
+
+

data/example/iris.csv

@@ -0,0 +1,151 @@
+"","Sepal.Length","Sepal.Width","Petal.Length","Petal.Width","Species"
+"1",5.1,3.5,1.4,0.2,"setosa"
+"2",4.9,3,1.4,0.2,"setosa"
+"3",4.7,3.2,1.3,0.2,"setosa"
+"4",4.6,3.1,1.5,0.2,"setosa"
+"5",5,3.6,1.4,0.2,"setosa"
+"6",5.4,3.9,1.7,0.4,"setosa"
+"7",4.6,3.4,1.4,0.3,"setosa"
+"8",5,3.4,1.5,0.2,"setosa"
+"9",4.4,2.9,1.4,0.2,"setosa"
+"10",4.9,3.1,1.5,0.1,"setosa"
+"11",5.4,3.7,1.5,0.2,"setosa"
+"12",4.8,3.4,1.6,0.2,"setosa"
+"13",4.8,3,1.4,0.1,"setosa"
+"14",4.3,3,1.1,0.1,"setosa"
+"15",5.8,4,1.2,0.2,"setosa"
+"16",5.7,4.4,1.5,0.4,"setosa"
+"17",5.4,3.9,1.3,0.4,"setosa"
+"18",5.1,3.5,1.4,0.3,"setosa"
+"19",5.7,3.8,1.7,0.3,"setosa"
+"20",5.1,3.8,1.5,0.3,"setosa"
+"21",5.4,3.4,1.7,0.2,"setosa"
+"22",5.1,3.7,1.5,0.4,"setosa"
+"23",4.6,3.6,1,0.2,"setosa"
+"24",5.1,3.3,1.7,0.5,"setosa"
+"25",4.8,3.4,1.9,0.2,"setosa"
+"26",5,3,1.6,0.2,"setosa"
+"27",5,3.4,1.6,0.4,"setosa"
+"28",5.2,3.5,1.5,0.2,"setosa"
+"29",5.2,3.4,1.4,0.2,"setosa"
+"30",4.7,3.2,1.6,0.2,"setosa"
+"31",4.8,3.1,1.6,0.2,"setosa"
+"32",5.4,3.4,1.5,0.4,"setosa"
+"33",5.2,4.1,1.5,0.1,"setosa"
+"34",5.5,4.2,1.4,0.2,"setosa"
+"35",4.9,3.1,1.5,0.2,"setosa"
+"36",5,3.2,1.2,0.2,"setosa"
+"37",5.5,3.5,1.3,0.2,"setosa"
+"38",4.9,3.6,1.4,0.1,"setosa"
+"39",4.4,3,1.3,0.2,"setosa"
+"40",5.1,3.4,1.5,0.2,"setosa"
+"41",5,3.5,1.3,0.3,"setosa"
+"42",4.5,2.3,1.3,0.3,"setosa"
+"43",4.4,3.2,1.3,0.2,"setosa"
+"44",5,3.5,1.6,0.6,"setosa"
+"45",5.1,3.8,1.9,0.4,"setosa"
+"46",4.8,3,1.4,0.3,"setosa"
+"47",5.1,3.8,1.6,0.2,"setosa"
+"48",4.6,3.2,1.4,0.2,"setosa"
+"49",5.3,3.7,1.5,0.2,"setosa"
+"50",5,3.3,1.4,0.2,"setosa"
+"51",7,3.2,4.7,1.4,"versicolor"
+"52",6.4,3.2,4.5,1.5,"versicolor"
+"53",6.9,3.1,4.9,1.5,"versicolor"
+"54",5.5,2.3,4,1.3,"versicolor"
+"55",6.5,2.8,4.6,1.5,"versicolor"
+"56",5.7,2.8,4.5,1.3,"versicolor"
+"57",6.3,3.3,4.7,1.6,"versicolor"
+"58",4.9,2.4,3.3,1,"versicolor"
+"59",6.6,2.9,4.6,1.3,"versicolor"
+"60",5.2,2.7,3.9,1.4,"versicolor"
+"61",5,2,3.5,1,"versicolor"
+"62",5.9,3,4.2,1.5,"versicolor"
+"63",6,2.2,4,1,"versicolor"
+"64",6.1,2.9,4.7,1.4,"versicolor"
+"65",5.6,2.9,3.6,1.3,"versicolor"
+"66",6.7,3.1,4.4,1.4,"versicolor"
+"67",5.6,3,4.5,1.5,"versicolor"
+"68",5.8,2.7,4.1,1,"versicolor"
+"69",6.2,2.2,4.5,1.5,"versicolor"
+"70",5.6,2.5,3.9,1.1,"versicolor"
+"71",5.9,3.2,4.8,1.8,"versicolor"
+"72",6.1,2.8,4,1.3,"versicolor"
+"73",6.3,2.5,4.9,1.5,"versicolor"
+"74",6.1,2.8,4.7,1.2,"versicolor"
+"75",6.4,2.9,4.3,1.3,"versicolor"
+"76",6.6,3,4.4,1.4,"versicolor"
+"77",6.8,2.8,4.8,1.4,"versicolor"
+"78",6.7,3,5,1.7,"versicolor"
+"79",6,2.9,4.5,1.5,"versicolor"
+"80",5.7,2.6,3.5,1,"versicolor"
+"81",5.5,2.4,3.8,1.1,"versicolor"
+"82",5.5,2.4,3.7,1,"versicolor"
+"83",5.8,2.7,3.9,1.2,"versicolor"
+"84",6,2.7,5.1,1.6,"versicolor"
+"85",5.4,3,4.5,1.5,"versicolor"
+"86",6,3.4,4.5,1.6,"versicolor"
+"87",6.7,3.1,4.7,1.5,"versicolor"
+"88",6.3,2.3,4.4,1.3,"versicolor"
+"89",5.6,3,4.1,1.3,"versicolor"
+"90",5.5,2.5,4,1.3,"versicolor"
+"91",5.5,2.6,4.4,1.2,"versicolor"
+"92",6.1,3,4.6,1.4,"versicolor"
+"93",5.8,2.6,4,1.2,"versicolor"
+"94",5,2.3,3.3,1,"versicolor"
+"95",5.6,2.7,4.2,1.3,"versicolor"
+"96",5.7,3,4.2,1.2,"versicolor"
+"97",5.7,2.9,4.2,1.3,"versicolor"
+"98",6.2,2.9,4.3,1.3,"versicolor"
+"99",5.1,2.5,3,1.1,"versicolor"
+"100",5.7,2.8,4.1,1.3,"versicolor"
+"101",6.3,3.3,6,2.5,"virginica"
+"102",5.8,2.7,5.1,1.9,"virginica"
+"103",7.1,3,5.9,2.1,"virginica"
+"104",6.3,2.9,5.6,1.8,"virginica"
+"105",6.5,3,5.8,2.2,"virginica"
+"106",7.6,3,6.6,2.1,"virginica"
+"107",4.9,2.5,4.5,1.7,"virginica"
+"108",7.3,2.9,6.3,1.8,"virginica"
+"109",6.7,2.5,5.8,1.8,"virginica"
+"110",7.2,3.6,6.1,2.5,"virginica"
+"111",6.5,3.2,5.1,2,"virginica"
+"112",6.4,2.7,5.3,1.9,"virginica"
+"113",6.8,3,5.5,2.1,"virginica"
+"114",5.7,2.5,5,2,"virginica"
+"115",5.8,2.8,5.1,2.4,"virginica"
+"116",6.4,3.2,5.3,2.3,"virginica"
+"117",6.5,3,5.5,1.8,"virginica"
+"118",7.7,3.8,6.7,2.2,"virginica"
+"119",7.7,2.6,6.9,2.3,"virginica"
+"120",6,2.2,5,1.5,"virginica"
+"121",6.9,3.2,5.7,2.3,"virginica"
+"122",5.6,2.8,4.9,2,"virginica"
+"123",7.7,2.8,6.7,2,"virginica"
+"124",6.3,2.7,4.9,1.8,"virginica"
+"125",6.7,3.3,5.7,2.1,"virginica"
+"126",7.2,3.2,6,1.8,"virginica"
+"127",6.2,2.8,4.8,1.8,"virginica"
+"128",6.1,3,4.9,1.8,"virginica"
+"129",6.4,2.8,5.6,2.1,"virginica"
+"130",7.2,3,5.8,1.6,"virginica"
+"131",7.4,2.8,6.1,1.9,"virginica"
+"132",7.9,3.8,6.4,2,"virginica"
+"133",6.4,2.8,5.6,2.2,"virginica"
+"134",6.3,2.8,5.1,1.5,"virginica"
+"135",6.1,2.6,5.6,1.4,"virginica"
+"136",7.7,3,6.1,2.3,"virginica"
+"137",6.3,3.4,5.6,2.4,"virginica"
+"138",6.4,3.1,5.5,1.8,"virginica"
+"139",6,3,4.8,1.8,"virginica"
+"140",6.9,3.1,5.4,2.1,"virginica"
+"141",6.7,3.1,5.6,2.4,"virginica"
+"142",6.9,3.1,5.1,2.3,"virginica"
+"143",5.8,2.7,5.1,1.9,"virginica"
+"144",6.8,3.2,5.9,2.3,"virginica"
+"145",6.7,3.3,5.7,2.5,"virginica"
+"146",6.7,3,5.2,2.3,"virginica"
+"147",6.3,2.5,5,1.9,"virginica"
+"148",6.5,3,5.2,2,"virginica"
+"149",6.2,3.4,5.4,2.3,"virginica"
+"150",5.9,3,5.1,1.8,"virginica"