datafarming 2.3.2 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/LICENSE.md +1 -1
- data/README.md +13 -9
- data/datafarming.gemspec +2 -2
- data/exe/datafarming.rb +18 -14
- data/exe/generate_design.rb +195 -70
- data/exe/mser_nolbm.rb +8 -6
- data/exe/mser_olbm.rb +2 -0
- data/exe/run_yaml_design.rb +186 -0
- data/exe/rundesign_general.rb +132 -73
- data/lib/datafarming/freq_sets.rb +9 -9
- data/lib/datafarming/scale_design.rb +13 -5
- data/lib/datafarming/version.rb +1 -1
- metadata +143 -132
- data/exe/mser.rb +0 -71
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 67ecc038ff0d8fd2e83ef1d0db8715bbec3395e3b234c96e54ed22fb4bba595d
|
|
4
|
+
data.tar.gz: 083b7b5911b44da84796f8b3f8005d762f091799ab7978fc95e536f324fb2cac
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 981413887afe289ae19c3ce2f11b4c9bd59326bce95019c0441b0090912250c34527c96ba013a1a3e38b858cc1f8ee05636c41cda436ed436b46afd5575b7a4e
|
|
7
|
+
data.tar.gz: 1d796a86fed0ef6666130ef388bd0a8aa5099e6a21d527326b7218d70c1f7ee60a5b80788980fdb56ea543d18b95b616b89e1772d1cf250a34890b7873a7daf2
|
data/LICENSE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
|
|
2
2
|
The MIT License (MIT)
|
|
3
3
|
|
|
4
|
-
Copyright (c) 2017-
|
|
4
|
+
Copyright (c) 2017-2026 Paul J. Sanchez
|
|
5
5
|
|
|
6
6
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
7
7
|
of this software and associated documentation files (the "Software"), to deal
|
data/README.md
CHANGED
|
@@ -1,25 +1,29 @@
|
|
|
1
1
|
### SETUP
|
|
2
2
|
|
|
3
|
-
This gem requires Ruby
|
|
3
|
+
This gem requires Ruby v2.6 or later to be installed on your system. Ruby v4 or higher is strongly recommended, offering security updates along with better speed and feature improvements.
|
|
4
4
|
|
|
5
|
-
- On MacOS, Ruby comes pre-installed. Note: You can use
|
|
5
|
+
- On MacOS, Ruby comes pre-installed, but is many years out-of-date at v2.6.10. Note: You can use package managers such as [mise](https://mise.jdx.dev) or [homebrew](https://brew.sh) to install the latest version of Ruby.
|
|
6
6
|
|
|
7
7
|
- Windows users should use one of the installer packages available from [rubyinstaller.org](http://rubyinstaller.org/). You will need to determine whether your have a 32 or 64 bit processor, and download the appropriate installer. When running the installer, check the checkboxes to add the Ruby installation to your PATH and to associate `.rb` and `.rbw` files with Ruby. A terminal window will pop up during the installation—press "enter" to automatically install all of the required tools, and when prompted again (several minutes later) press "enter" again to end the installation.
|
|
8
8
|
|
|
9
9
|
You can check that Ruby is properly installed by opening `Terminal.app` (MacOS) or `CMD.EXE` (Windows) and typing `ruby --version` (note: two dashes). You should see a response telling you which version of Ruby you are running.
|
|
10
10
|
|
|
11
|
-
After your Ruby installation is confirmed you can install the data farming Ruby scripts by running the following command.
|
|
11
|
+
After your Ruby installation is confirmed, if you have an active network connection you can install the data farming Ruby scripts by running the following command.
|
|
12
12
|
|
|
13
13
|
gem install datafarming
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
Additional dependencies will be installed automatically. On MacOS or Linux systems, you may be required to use the `sudo` command to validate and complete a system-wide installation. In that case, type `sudo gem install datafarming` and enter your password when prompted.
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
We don't generally recommend doing the following, but if all else fails you can perform a user-owned installation with the command
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
gem install datafarming --user-install
|
|
20
|
+
|
|
21
|
+
The specific location of the installation will depend on your OS. You will need to read documentation to identify the destination folder and manually add it to your runtime environment's search PATH.
|
|
20
22
|
|
|
21
|
-
|
|
23
|
+
Gem installation only needs to be done once. If you explicitly downloaded the `.gem` file rather than using a network installation, you can delete it after performing the installation. When a newer version of the gem is released, you can upgrade to it by running `gem update`.
|
|
24
|
+
|
|
25
|
+
### USAGE
|
|
22
26
|
|
|
23
|
-
|
|
27
|
+
Ruby is a powerful and concise object-oriented scripting language. Ruby scripts are normally run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments. However, scripts installed as gems do not need the `ruby` command to be typed explicitly. For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory. Additionally, on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
|
|
24
28
|
|
|
25
|
-
After installing the gem and making sure that gems are on your PATH, run the command `datafarming.rb`
|
|
29
|
+
All scripts in this distribution are self documenting if invoked with a `--help`, or `-h` option. After installing the gem and making sure that gems are on your PATH, run the command `datafarming.rb` at any time to see short overview descriptions of the various data farming tools provided in this distribution.
|
data/datafarming.gemspec
CHANGED
|
@@ -7,9 +7,9 @@ Gem::Specification.new do |s|
|
|
|
7
7
|
s.name = "datafarming"
|
|
8
8
|
s.version = DataFarming::VERSION
|
|
9
9
|
s.date = "#{Date.today}"
|
|
10
|
-
s.summary = "Useful
|
|
10
|
+
s.summary = "Useful tools for data farming."
|
|
11
11
|
s.homepage = "https://bitbucket.org/paul_j_sanchez/datafarmingrubyscripts"
|
|
12
|
-
s.email = "pjs@
|
|
12
|
+
s.email = "prof.pjs@icloud.com"
|
|
13
13
|
s.description = "Ruby scripts for data farming, including pre- and" \
|
|
14
14
|
"post-processing, design generation and scaling, and run control."
|
|
15
15
|
s.author = "Paul J Sanchez"
|
data/exe/datafarming.rb
CHANGED
|
@@ -22,25 +22,25 @@ lines = [
|
|
|
22
22
|
' • blank2csv.rb —',
|
|
23
23
|
' convert whitespace-delimited files to Comma Separated Values (CSV) files',
|
|
24
24
|
' • cat.rb —',
|
|
25
|
-
' Unix
|
|
26
|
-
' or concatenating multiple files, based on stdio',
|
|
25
|
+
' Unix \'cat\' program work-alike for Windows—for viewing or creating text files',
|
|
26
|
+
' or concatenating multiple files, based on stdio.',
|
|
27
27
|
' • convert_line_endings.rb —',
|
|
28
28
|
' convert text files (including CSV files) to your current operating system',
|
|
29
|
-
' environment (e.g., DOS to Unix or Unix to DOS)',
|
|
29
|
+
' environment (e.g., DOS to Unix or Unix to DOS).',
|
|
30
30
|
' • csv2blank.rb —',
|
|
31
|
-
' convert CSV files to whitespace-delimited files',
|
|
31
|
+
' convert CSV files to whitespace-delimited files.',
|
|
32
32
|
' • pool_files.rb —',
|
|
33
33
|
' pool columns from separate files into a new file, useful for combining',
|
|
34
|
-
' design file
|
|
34
|
+
' design file with output files or multiple output files.',
|
|
35
35
|
' • stripheaders.rb , stripheaderdups.rb —',
|
|
36
36
|
' remove column headers from a file, or duplicate headers within a file,',
|
|
37
|
-
' respectively',
|
|
37
|
+
' respectively.',
|
|
38
38
|
'',
|
|
39
39
|
'#### DESIGN GENERATORS',
|
|
40
40
|
'',
|
|
41
41
|
' • cross.rb —',
|
|
42
42
|
' create a combinatorial design by crossing all combinations of any # of',
|
|
43
|
-
' individual smaller designs',
|
|
43
|
+
' individual smaller designs.',
|
|
44
44
|
' • generate_design.rb —',
|
|
45
45
|
' allowable design types are',
|
|
46
46
|
' - resv : Resolution V fractional factorial at 2 levels per factor;',
|
|
@@ -57,21 +57,25 @@ lines = [
|
|
|
57
57
|
' allow partial correlation amongst the interactions which',
|
|
58
58
|
' significantly reduces the number of design points;',
|
|
59
59
|
' all of which generate designs with stacking, replication, and individualized',
|
|
60
|
-
' factor scaling',
|
|
60
|
+
' factor scaling. For continuous-factor designs (fbd, fbs, nolh) individual',
|
|
61
|
+
' factors can be converted to discrete with some potential loss of',
|
|
62
|
+
' orthogonality, or nearly-balanced categoricals where category counts',
|
|
63
|
+
' differ by no more than 1.',
|
|
64
|
+
' Note - use caution with multiple categoricals, since not all possible',
|
|
65
|
+
' combinations may be realized.',
|
|
61
66
|
'',
|
|
62
67
|
'#### RUN CONTROL',
|
|
63
68
|
'',
|
|
64
69
|
' • batchrunner.rb —',
|
|
65
|
-
' run a model interactively with replication',
|
|
70
|
+
' run a model interactively with replication.',
|
|
66
71
|
' • rundesign_general.rb —',
|
|
67
|
-
' run control to replicate a designed experiment applied to a model',
|
|
72
|
+
' run control to replicate a designed experiment applied to a model.',
|
|
73
|
+
' • run_yaml_design.rb —',
|
|
74
|
+
' run control to replicate a designed experiment applied to a model that',
|
|
75
|
+
' uses YAML for inputs.',
|
|
68
76
|
'',
|
|
69
77
|
'#### CONFIDENCE INTERVALS FOR TIME SERIES',
|
|
70
78
|
'',
|
|
71
|
-
' • mser.rb —',
|
|
72
|
-
' use MSER truncation to remove initial transient effects for time-series',
|
|
73
|
-
' output, reporting truncated average and number of observations for each run',
|
|
74
|
-
' to facilitate construction of a properly weighted confidence interval.',
|
|
75
79
|
' • mser_nolbm.rb —',
|
|
76
80
|
' use MSER truncation to remove initial transient effects for time-series',
|
|
77
81
|
' output, then calculate a 95% confidence interval on the remaining data using',
|
data/exe/generate_design.rb
CHANGED
|
@@ -1,9 +1,5 @@
|
|
|
1
|
-
#!/usr/bin/env ruby
|
|
2
|
-
# frozen_string_literal: true
|
|
1
|
+
#!/usr/bin/env ruby
|
|
3
2
|
|
|
4
|
-
# require 'colorize'
|
|
5
|
-
# String.disable_colorization false
|
|
6
|
-
# require 'datafarming/error_handling'
|
|
7
3
|
RubyVM::YJIT.enable if defined?(RubyVM::YJIT) && RubyVM::YJIT.respond_to?(:enable)
|
|
8
4
|
|
|
9
5
|
require 'optparse'
|
|
@@ -11,12 +7,24 @@ require 'yaml'
|
|
|
11
7
|
|
|
12
8
|
require 'datafarming/scale_design'
|
|
13
9
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
10
|
+
CLASSICAL_DESIGNS = %i[resv ccd rotatable nolh].freeze
|
|
11
|
+
FREQ_DESIGNS = %i[fbd fbs].freeze
|
|
12
|
+
DESIGN_TYPES = (CLASSICAL_DESIGNS + FREQ_DESIGNS).freeze
|
|
13
|
+
MIN_MAX = %i[min max].freeze
|
|
14
|
+
ELECTIVE_SPECS = %i[id decimals cat disc].freeze
|
|
15
|
+
NOLH_LEVELS = [17, 33, 65, 129, 257, 512].freeze
|
|
18
16
|
TWO_PI = 2.0 * Math::PI
|
|
19
17
|
|
|
18
|
+
DEFAULT_OPTIONS = {
|
|
19
|
+
design: nil,
|
|
20
|
+
levels: nil,
|
|
21
|
+
headers: true,
|
|
22
|
+
replicate: 1,
|
|
23
|
+
stack: 1,
|
|
24
|
+
centerdup: false,
|
|
25
|
+
specs: []
|
|
26
|
+
}
|
|
27
|
+
|
|
20
28
|
HELP_MSG = [
|
|
21
29
|
'Generate, scale, stack, and replicate a variety of statistical designs',
|
|
22
30
|
"of experiments. You are required to specify the desired '--design TYPE'.",
|
|
@@ -25,7 +33,7 @@ HELP_MSG = [
|
|
|
25
33
|
].freeze
|
|
26
34
|
|
|
27
35
|
EXTENDED_HELP = [
|
|
28
|
-
"====================\n",
|
|
36
|
+
# "====================\n",
|
|
29
37
|
"Using the '-k' option will specify the number of factors for a standardized",
|
|
30
38
|
'design, in which all factors are normalized to the range of [-1, 1].',
|
|
31
39
|
'',
|
|
@@ -35,19 +43,47 @@ EXTENDED_HELP = [
|
|
|
35
43
|
"specifications provided (thus overriding the '-k' option). Each factor's",
|
|
36
44
|
'specification is separated from the others by whitespace.',
|
|
37
45
|
'',
|
|
38
|
-
'
|
|
39
|
-
'cannot include any whitespace. The format is:',
|
|
46
|
+
'Continuous factor specifications contain two to four comma-separated',
|
|
47
|
+
'components, and cannot include any whitespace. The format is:',
|
|
40
48
|
"\n id:STRING,min:NUMBER,max:NUMBER,decimals:INTEGER\n",
|
|
41
|
-
"Components can be
|
|
49
|
+
"Components can be provided in any order. The components 'min' and 'max'",
|
|
42
50
|
'are required, and define the range of experimentation for this factor.',
|
|
43
|
-
"Components 'id' and 'decimals' are optional. If 'id' is omitted, id",
|
|
44
|
-
"
|
|
45
|
-
'
|
|
46
|
-
'
|
|
51
|
+
"Components 'id' and 'decimals' are optional. If 'id' is omitted, id values",
|
|
52
|
+
"will be generated for use with the '--headers' option. If 'decimals' is",
|
|
53
|
+
'omitted, the values will be displayed with default (full) precision.',
|
|
54
|
+
'',
|
|
55
|
+
'Discrete and categorical factors are also options. The core design',
|
|
56
|
+
'underpinning mixtures of continuous/discrete/categorical should be one',
|
|
57
|
+
"of the continuous designs: fbd, fbs, or nolh",
|
|
58
|
+
'',
|
|
59
|
+
"Discrete factors are generated using a specification of the form:",
|
|
60
|
+
"\n disc:INTEGER,id:STRING,min:NUMBER,max:NUMBER,decimals:INTEGER\n",
|
|
61
|
+
'The first INTEGER specifies the number of discrete levels. As with',
|
|
62
|
+
"continuous factors, 'min' and 'max' are required, 'id' and 'decimals'",
|
|
63
|
+
"are optional. The results can be non-integer if 'min' and 'max' are",
|
|
64
|
+
'specified with decimal values.',
|
|
65
|
+
'',
|
|
66
|
+
"Categorical factors are generated using a specification of the form:",
|
|
67
|
+
"\n cat:RANGE,id:STRING\n",
|
|
68
|
+
'or',
|
|
69
|
+
"\n cat:'[ARRAY]',id:STRING\n",
|
|
70
|
+
"If the RANGE option is used, it is a Ruby range of the form 'FIRST..LAST'",
|
|
71
|
+
"or 'FIRST...LAST'. The two-dot version generates the sequence of values",
|
|
72
|
+
'between FIRST and LAST, inclusive, while the three-dot version excludes the',
|
|
73
|
+
'LAST value. Ruby allows ranges of integers or lexicographic strings. If the',
|
|
74
|
+
'ARRAY specification is used, a comma-separated set of specific values is',
|
|
75
|
+
'enumerated. Note that the square brackets surrounding the array values MUST',
|
|
76
|
+
'be contained between single quotation marks.',
|
|
77
|
+
'',
|
|
78
|
+
"For large designs with many factors, using the '-k' and '-o' options",
|
|
79
|
+
'together is highly recommended. The resulting YAML output can then be',
|
|
80
|
+
'edited at your leisure, while a long multi-factor set of specification',
|
|
81
|
+
'strings is prone to human error on entry.',
|
|
82
|
+
'',
|
|
47
83
|
].freeze
|
|
48
84
|
|
|
49
85
|
EXAMPLES = [
|
|
50
|
-
"\n====================\n",
|
|
86
|
+
# "\n====================\n",
|
|
51
87
|
'EXAMPLE 1:',
|
|
52
88
|
"\n #{File.basename($PROGRAM_NAME)} --design fbd -k 3 --no-headers",
|
|
53
89
|
"\n Generate a standardized frequency-based design with three factors,",
|
|
@@ -55,7 +91,7 @@ EXAMPLES = [
|
|
|
55
91
|
'',
|
|
56
92
|
'EXAMPLE 2:',
|
|
57
93
|
"\n #{File.basename($PROGRAM_NAME)} -d nolh id:Fred,min:-10,max:10 decimals:5,max:100,min:0",
|
|
58
|
-
"\n Generate
|
|
94
|
+
"\n Generate an NOLH for two factors. The first factor will have a column",
|
|
59
95
|
" header of 'Fred', and values will be scaled in the range -10..10. The",
|
|
60
96
|
" second column will have an automatically generated header 'X_002', and",
|
|
61
97
|
' values will be in the range 0..100 with five decimal places of precision.',
|
|
@@ -63,15 +99,23 @@ EXAMPLES = [
|
|
|
63
99
|
" by whitespace.",
|
|
64
100
|
'',
|
|
65
101
|
'EXAMPLE 3:',
|
|
66
|
-
"\n #{File.basename($PROGRAM_NAME)} -d
|
|
67
|
-
"\n Generate a
|
|
102
|
+
"\n #{File.basename($PROGRAM_NAME)} -d fbd disc:5,min:-1.0,max:1.0 cat:a..e",
|
|
103
|
+
"\n Generate a mixed discrete/categorical design based on an underlying",
|
|
104
|
+
' frequency-based design. The first factor will have values stepped between',
|
|
105
|
+
' -1 and 1 in steps of 0.5. The second column will consist of a balanced mix',
|
|
106
|
+
' of the letters a, b, c, d, and e. Both columns will have automatically',
|
|
107
|
+
' generated headers.',
|
|
108
|
+
'',
|
|
109
|
+
'EXAMPLE 4:',
|
|
110
|
+
"\n #{File.basename($PROGRAM_NAME)} -d nolh -k 5 -o my_template.yml",
|
|
111
|
+
"\n Generate a YAML template for a 5-factor NOLH design, and send it",
|
|
68
112
|
" to the file 'my_template.yml'. (The output file can be given any name",
|
|
69
113
|
' of your choosing.) The resulting YAML file has an intuitive human',
|
|
70
114
|
" readable format. It can be edited using your favorite programmer's",
|
|
71
115
|
' text editor (NOTE: word processors will not work!). After saving your',
|
|
72
116
|
' revisions, you can read it back in to generate the actual design using',
|
|
73
117
|
" the '--input_yaml' option:\n",
|
|
74
|
-
" #{File.basename($PROGRAM_NAME)}
|
|
118
|
+
" #{File.basename($PROGRAM_NAME)} -i my_template.yml",
|
|
75
119
|
' '
|
|
76
120
|
].freeze
|
|
77
121
|
|
|
@@ -81,13 +125,13 @@ def prog_help(opts)
|
|
|
81
125
|
end
|
|
82
126
|
|
|
83
127
|
def verbose(opts)
|
|
84
|
-
puts opts
|
|
128
|
+
# puts opts
|
|
85
129
|
puts EXTENDED_HELP.join("\n")
|
|
86
130
|
exit
|
|
87
131
|
end
|
|
88
132
|
|
|
89
133
|
def examples(opts)
|
|
90
|
-
puts opts
|
|
134
|
+
# puts opts
|
|
91
135
|
puts EXAMPLES.join("\n")
|
|
92
136
|
exit
|
|
93
137
|
end
|
|
@@ -97,6 +141,53 @@ def fatal_err(msg)
|
|
|
97
141
|
exit 1
|
|
98
142
|
end
|
|
99
143
|
|
|
144
|
+
def thresholds(k)
|
|
145
|
+
const = Math::PI / (2 * k)
|
|
146
|
+
Array.new(k) { |idx| Math.sin(const * (2 * (idx+1) - k)) }
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
class Array
|
|
150
|
+
@@toggle = true
|
|
151
|
+
def toggle_cmp(x, y)
|
|
152
|
+
if x == 0.0
|
|
153
|
+
(@@toggle ^= true) ? x > y : x >= y
|
|
154
|
+
else
|
|
155
|
+
x >= y
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def discretize(spec)
|
|
160
|
+
stepsize = (spec[:max] - spec[:min]) / (spec[:disc] - 1)
|
|
161
|
+
ary = Array.new(spec[:disc]) { |i| spec[:min] + i * stepsize }
|
|
162
|
+
if FREQ_DESIGNS.include? spec[:design]
|
|
163
|
+
t = thresholds(spec[:disc])
|
|
164
|
+
else
|
|
165
|
+
stepsize = 2.0 / spec[:disc]
|
|
166
|
+
t = Array.new(spec[:disc] - 1) { |i| (i + 1) * stepsize - 1 } + [1]
|
|
167
|
+
end
|
|
168
|
+
decimals = spec[:decimals] || 15
|
|
169
|
+
self.map { |value| ary[t.bsearch_index { |x| x >= value }].round(decimals) }
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
def categorize(spec)
|
|
173
|
+
categories = if spec[:cat].include? '..'
|
|
174
|
+
Range.new(*(spec[:cat].split('..'))).to_a
|
|
175
|
+
elsif spec[:cat].is_a? Array
|
|
176
|
+
spec[:cat]
|
|
177
|
+
else
|
|
178
|
+
fail "Invalid category specification: #{spec[:cat]}"
|
|
179
|
+
end
|
|
180
|
+
# t = thresholds(categories.size)
|
|
181
|
+
if FREQ_DESIGNS.include? spec[:design]
|
|
182
|
+
t = thresholds(categories.size)
|
|
183
|
+
else
|
|
184
|
+
stepsize = 2.0 / categories.size
|
|
185
|
+
t = Array.new(categories.size - 1) { |i| (i + 1) * stepsize - 1 } + [1]
|
|
186
|
+
end
|
|
187
|
+
self.map { |x| categories[t.bsearch_index { |entry| entry >= x }] }
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
|
|
100
191
|
def resv(options)
|
|
101
192
|
require 'datafarming/res_v_seqs'
|
|
102
193
|
num_factors = options[:specs].size
|
|
@@ -155,12 +246,10 @@ def freqs2cols(options, design_set)
|
|
|
155
246
|
freqs = ds.freqs[design_num]
|
|
156
247
|
Array.new(freqs.size) do |row|
|
|
157
248
|
omega = freqs[row]
|
|
158
|
-
|
|
249
|
+
Array.new(ds.nyq) do |i|
|
|
159
250
|
f = ((i * omega) % ds.nyq).to_f / ds.nyq
|
|
160
251
|
Math.sin(f * TWO_PI)
|
|
161
252
|
end
|
|
162
|
-
scale_factor = v.max
|
|
163
|
-
v.map { |x| (x / scale_factor).round(15) }
|
|
164
253
|
end.transpose
|
|
165
254
|
end
|
|
166
255
|
|
|
@@ -175,6 +264,7 @@ def fbs(options)
|
|
|
175
264
|
end
|
|
176
265
|
|
|
177
266
|
def validate_opts(options)
|
|
267
|
+
options.delete_if { |k, v| v.nil? }
|
|
178
268
|
required_options = [:design]
|
|
179
269
|
missing = required_options - options.keys
|
|
180
270
|
fatal_err("Missing required options: --#{missing.join(', --')}") unless missing.empty?
|
|
@@ -182,18 +272,18 @@ def validate_opts(options)
|
|
|
182
272
|
end
|
|
183
273
|
|
|
184
274
|
def validate(spec)
|
|
185
|
-
unknown = spec.keys -
|
|
275
|
+
unknown = spec.keys - MIN_MAX - ELECTIVE_SPECS
|
|
186
276
|
fatal_err("Unknown factor spec(s): #{unknown}") unless unknown.empty?
|
|
187
|
-
missing =
|
|
277
|
+
missing = (has_cat = spec.keys.include?(:cat)) ? [] : MIN_MAX - spec.keys
|
|
188
278
|
fatal_err("Factor spec #{spec} missing #{missing}") unless missing.empty?
|
|
189
279
|
end
|
|
190
280
|
|
|
191
281
|
def optional_specs(factor_spec, i)
|
|
192
|
-
YAML.safe_load("{#{factor_spec}}".gsub(':', ': '),
|
|
193
|
-
permitted_classes: [Symbol]).tap do |spec|
|
|
282
|
+
YAML.safe_load("{#{factor_spec}}".gsub(':', ': '), permitted_classes: [Symbol]).tap do |spec|
|
|
194
283
|
spec.transform_keys!(&:to_sym)
|
|
195
|
-
spec[:decimals] ||= nil
|
|
196
284
|
spec[:id] ||= "X_#{format('%03d', (i + 1))}"
|
|
285
|
+
spec.delete_if { |k, v| v.nil? }
|
|
286
|
+
spec[:decimals] = nil unless (spec.has_key?(:decimals) || spec.has_key?(:cat))
|
|
197
287
|
validate(spec)
|
|
198
288
|
end
|
|
199
289
|
end
|
|
@@ -207,20 +297,12 @@ def parse_specs(options, remainder)
|
|
|
207
297
|
end
|
|
208
298
|
|
|
209
299
|
def parse_args
|
|
210
|
-
default_options = {
|
|
211
|
-
headers: true,
|
|
212
|
-
replicate: 1,
|
|
213
|
-
stack: 1,
|
|
214
|
-
center: false,
|
|
215
|
-
specs: []
|
|
216
|
-
}
|
|
217
|
-
|
|
218
300
|
yaml_input = false
|
|
219
301
|
if ARGV.size > 2 && ARGV.any? { |elt| elt.eql?("-i") || elt.eql?("--input_yaml") }
|
|
220
302
|
STDERR.puts 'NOTE: All other arguments ignored when using YAML input'
|
|
221
303
|
end
|
|
222
304
|
|
|
223
|
-
options =
|
|
305
|
+
options = DEFAULT_OPTIONS
|
|
224
306
|
|
|
225
307
|
begin
|
|
226
308
|
remainder = OptionParser.new do |opts|
|
|
@@ -243,25 +325,48 @@ def parse_args
|
|
|
243
325
|
'Replicate QTY times', '(default: 1)')
|
|
244
326
|
opts.on('-s', '--stack QTY', /[1-9][0-9]*/, Integer,
|
|
245
327
|
'Stack QTY times', '(default: 1)')
|
|
246
|
-
opts.on('--[no-]
|
|
247
|
-
'Stacking
|
|
328
|
+
opts.on('-c', '--[no-]centerdup', 'NOTE: Applies only to stacked designs',
|
|
329
|
+
'Stacking can duplicate the center point','(default: no-centerdup)')
|
|
248
330
|
opts.on('--[no-]headers', 'Output has column headers', '(default: headers)')
|
|
249
|
-
opts.on('-l', '--levels QTY', Regexp.new("(#{
|
|
331
|
+
opts.on('-l', '--levels QTY', Regexp.new("(#{NOLH_LEVELS.join(')|(')})"),
|
|
250
332
|
Integer, 'NOTE: Applies only to NOLH designs',
|
|
251
333
|
'Number of levels in the base NOLH:',
|
|
252
|
-
" #{
|
|
253
|
-
opts.on('-o', '--output_yaml',
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
334
|
+
" #{NOLH_LEVELS.join ' '}")
|
|
335
|
+
opts.on('-o', '--output_yaml [FILENAME]',
|
|
336
|
+
'Write design YAML to FILENAME (or STDOUT)'
|
|
337
|
+
) do |fname|
|
|
338
|
+
if fname.nil? || '-' == fname
|
|
339
|
+
STDOUT
|
|
340
|
+
else
|
|
341
|
+
fname
|
|
342
|
+
end
|
|
343
|
+
end
|
|
344
|
+
opts.on(
|
|
345
|
+
"-i", "--input_yaml [FILENAME]",
|
|
346
|
+
'Read design YAML from FILENAME (or STDIN)',
|
|
347
|
+
'(Mutually exclusive with all other options,',
|
|
348
|
+
'prints warning to STDERR if other options',
|
|
349
|
+
'are passed)'
|
|
350
|
+
) do |fname|
|
|
257
351
|
ARGV.clear
|
|
258
|
-
options = {specs: []}
|
|
352
|
+
# options = {specs: []}
|
|
353
|
+
options = DEFAULT_OPTIONS
|
|
259
354
|
yaml_input = true
|
|
260
|
-
|
|
261
|
-
|
|
355
|
+
if fname.nil? || '-' == fname
|
|
356
|
+
YAML::safe_load(
|
|
357
|
+
STDIN.read, permitted_classes: [Symbol]
|
|
358
|
+
).each do |k, v|
|
|
359
|
+
options[k.to_sym] = v
|
|
360
|
+
end
|
|
361
|
+
else
|
|
362
|
+
YAML::safe_load_file(
|
|
363
|
+
fname, permitted_classes: [Symbol]
|
|
364
|
+
).each do |k, v|
|
|
365
|
+
options[k.to_sym] = v
|
|
366
|
+
end
|
|
262
367
|
end
|
|
263
368
|
end
|
|
264
|
-
|
|
369
|
+
end.parse!(into: options)
|
|
265
370
|
rescue OptionParser::ParseError => error
|
|
266
371
|
fatal_err("#{error}\n(-h or --help will show valid options)")
|
|
267
372
|
end
|
|
@@ -280,23 +385,23 @@ def parse_args
|
|
|
280
385
|
options
|
|
281
386
|
end
|
|
282
387
|
|
|
283
|
-
def stack(design, num_stacks,
|
|
388
|
+
def stack(design, num_stacks, center_pt_dup)
|
|
284
389
|
return design if num_stacks < 2
|
|
285
|
-
fatal_err("Stacking #{num_stacks} times exceeds number of design columns") if num_stacks > design
|
|
390
|
+
fatal_err("Stacking #{num_stacks} times exceeds number of design columns") if num_stacks > design.size
|
|
286
391
|
num_stacks -= 1
|
|
287
|
-
result = design
|
|
392
|
+
result = design
|
|
288
393
|
if num_stacks > 0
|
|
289
|
-
design_t = if
|
|
290
|
-
design.
|
|
394
|
+
design_t = if center_pt_dup
|
|
395
|
+
design.clone
|
|
291
396
|
else
|
|
292
|
-
design.delete_if {|row| row.all? &:zero? }.transpose
|
|
397
|
+
design.transpose.delete_if {|row| row.all? &:zero? }.transpose
|
|
293
398
|
end
|
|
294
399
|
num_stacks.times do
|
|
295
400
|
design_t.rotate!
|
|
296
|
-
result.
|
|
401
|
+
result.size.times { |i| result[i] += design_t[i] }
|
|
297
402
|
end
|
|
298
403
|
end
|
|
299
|
-
result
|
|
404
|
+
result
|
|
300
405
|
end
|
|
301
406
|
|
|
302
407
|
ARGV << '--help' if ARGV.empty?
|
|
@@ -304,26 +409,46 @@ ARGV << '--help' if ARGV.empty?
|
|
|
304
409
|
options = parse_args
|
|
305
410
|
|
|
306
411
|
if options[:output_yaml]
|
|
307
|
-
options
|
|
308
|
-
|
|
412
|
+
o_file = (STDOUT == options[:output_yaml]) ? STDOUT : File.open(options[:output_yaml], 'w')
|
|
413
|
+
options.delete(:output_yaml)
|
|
414
|
+
begin
|
|
415
|
+
o_file.puts options.to_yaml
|
|
416
|
+
ensure
|
|
417
|
+
o_file.close unless STDIN == o_file
|
|
418
|
+
end
|
|
309
419
|
else
|
|
310
|
-
base_design = method(options[:design]).call(options)
|
|
420
|
+
base_design = method(options[:design]).call(options).transpose.first(options[:specs].size)
|
|
311
421
|
separator = ','
|
|
312
422
|
puts Array.new(options[:specs].size) { |i| options[:specs][i][:id] }.join(separator) if options[:headers]
|
|
313
423
|
|
|
314
424
|
# stack
|
|
315
|
-
design = stack(base_design, options[:stack], options[:
|
|
316
|
-
.transpose.first(options[:specs].size).transpose
|
|
425
|
+
design = stack(base_design, options[:stack], options[:centerdup])
|
|
317
426
|
|
|
318
427
|
# scale
|
|
319
|
-
|
|
320
|
-
|
|
428
|
+
scale_factor = design.map &:max
|
|
429
|
+
d_type = options[:design]
|
|
430
|
+
design.map!.with_index do |v, i|
|
|
431
|
+
break if i >= options[:specs].size
|
|
432
|
+
spec = options[:specs][i]
|
|
433
|
+
if spec[:cat]
|
|
434
|
+
spec[:design] = d_type
|
|
435
|
+
v.categorize(spec)
|
|
436
|
+
elsif spec[:disc]
|
|
437
|
+
spec[:design] = d_type
|
|
438
|
+
v.discretize(spec)
|
|
439
|
+
else
|
|
440
|
+
scaler = Scaler.new(spec)
|
|
441
|
+
v.map do |x|
|
|
442
|
+
scaler.scale((x / scale_factor[i]).round(15))
|
|
443
|
+
end
|
|
444
|
+
end
|
|
321
445
|
end
|
|
322
446
|
|
|
323
447
|
# replicate
|
|
448
|
+
result = design.transpose
|
|
324
449
|
options[:replicate].times do
|
|
325
|
-
|
|
326
|
-
puts line.
|
|
450
|
+
result.each do |line|
|
|
451
|
+
puts line.join(separator)
|
|
327
452
|
end
|
|
328
453
|
end
|
|
329
454
|
end
|
data/exe/mser_nolbm.rb
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env ruby -w
|
|
2
2
|
|
|
3
|
+
RubyVM::YJIT.enable if defined?(RubyVM::YJIT) && RubyVM::YJIT.respond_to?(:enable)
|
|
4
|
+
|
|
3
5
|
require 'rubygems' if RUBY_VERSION =~ /^1\.8/
|
|
4
6
|
require 'colorize'
|
|
5
7
|
|
|
@@ -84,19 +86,19 @@ def nolbm(data)
|
|
|
84
86
|
avg = best[1]
|
|
85
87
|
index = best[2]
|
|
86
88
|
length = data.length - index
|
|
87
|
-
b = [length / 30, length / index].min
|
|
89
|
+
b = (index > 0) ? [length / 30, length / index].min : length / 30
|
|
88
90
|
m, excess = length.divmod b
|
|
89
|
-
incr, discard = excess.divmod
|
|
90
|
-
|
|
91
|
+
incr, discard = excess.divmod m
|
|
92
|
+
b += incr
|
|
91
93
|
index += discard
|
|
92
|
-
usable = data.drop(index).each_slice(
|
|
93
|
-
batch_means = Array.new(
|
|
94
|
+
usable = data.drop(index).each_slice(b).to_a
|
|
95
|
+
batch_means = Array.new(m) do |i|
|
|
94
96
|
QuickStats.new.add_set(usable[i]).avg
|
|
95
97
|
end
|
|
96
98
|
sum_squared_deviations = batch_means
|
|
97
99
|
.map { |y| square(y - avg)}
|
|
98
100
|
.inject(&:+)
|
|
99
|
-
df = (
|
|
101
|
+
df = (m - 1)
|
|
100
102
|
var_hat = sum_squared_deviations / df
|
|
101
103
|
std_dev = Math.sqrt(var_hat)
|
|
102
104
|
std_err = Math.sqrt(var_hat / b)
|