datafarming 2.3.1 → 3.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/LICENSE.md +1 -1
- data/README.md +13 -9
- data/datafarming.gemspec +2 -2
- data/exe/datafarming.rb +18 -14
- data/exe/generate_design.rb +196 -70
- data/exe/mser_nolbm.rb +20 -22
- data/exe/mser_olbm.rb +9 -5
- data/exe/run_yaml_design.rb +186 -0
- data/exe/rundesign_general.rb +132 -73
- data/lib/datafarming/freq_sets.rb +412 -34
- data/lib/datafarming/scale_design.rb +13 -5
- data/lib/datafarming/screen_freq_sets.rb +451 -89
- data/lib/datafarming/version.rb +1 -1
- metadata +143 -132
- data/exe/mser.rb +0 -71
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 67ecc038ff0d8fd2e83ef1d0db8715bbec3395e3b234c96e54ed22fb4bba595d
|
|
4
|
+
data.tar.gz: 083b7b5911b44da84796f8b3f8005d762f091799ab7978fc95e536f324fb2cac
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 981413887afe289ae19c3ce2f11b4c9bd59326bce95019c0441b0090912250c34527c96ba013a1a3e38b858cc1f8ee05636c41cda436ed436b46afd5575b7a4e
|
|
7
|
+
data.tar.gz: 1d796a86fed0ef6666130ef388bd0a8aa5099e6a21d527326b7218d70c1f7ee60a5b80788980fdb56ea543d18b95b616b89e1772d1cf250a34890b7873a7daf2
|
data/LICENSE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
|
|
2
2
|
The MIT License (MIT)
|
|
3
3
|
|
|
4
|
-
Copyright (c) 2017-
|
|
4
|
+
Copyright (c) 2017-2026 Paul J. Sanchez
|
|
5
5
|
|
|
6
6
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
7
7
|
of this software and associated documentation files (the "Software"), to deal
|
data/README.md
CHANGED
|
@@ -1,25 +1,29 @@
|
|
|
1
1
|
### SETUP
|
|
2
2
|
|
|
3
|
-
This gem requires Ruby
|
|
3
|
+
This gem requires Ruby v2.6 or later to be installed on your system. Ruby v4 or higher is strongly recommended, offering security updates along with better speed and feature improvements.
|
|
4
4
|
|
|
5
|
-
- On MacOS, Ruby comes pre-installed. Note: You can use
|
|
5
|
+
- On MacOS, Ruby comes pre-installed, but is many years out-of-date at v2.6.10. Note: You can use package managers such as [mise](https://mise.jdx.dev) or [homebrew](https://brew.sh) to install the latest version of Ruby.
|
|
6
6
|
|
|
7
7
|
- Windows users should use one of the installer packages available from [rubyinstaller.org](http://rubyinstaller.org/). You will need to determine whether your have a 32 or 64 bit processor, and download the appropriate installer. When running the installer, check the checkboxes to add the Ruby installation to your PATH and to associate `.rb` and `.rbw` files with Ruby. A terminal window will pop up during the installation—press "enter" to automatically install all of the required tools, and when prompted again (several minutes later) press "enter" again to end the installation.
|
|
8
8
|
|
|
9
9
|
You can check that Ruby is properly installed by opening `Terminal.app` (MacOS) or `CMD.EXE` (Windows) and typing `ruby --version` (note: two dashes). You should see a response telling you which version of Ruby you are running.
|
|
10
10
|
|
|
11
|
-
After your Ruby installation is confirmed you can install the data farming Ruby scripts by running the following command.
|
|
11
|
+
After your Ruby installation is confirmed, if you have an active network connection you can install the data farming Ruby scripts by running the following command.
|
|
12
12
|
|
|
13
13
|
gem install datafarming
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
Additional dependencies will be installed automatically. On MacOS or Linux systems, you may be required to use the `sudo` command to validate and complete a system-wide installation. In that case, type `sudo gem install datafarming` and enter your password when prompted.
|
|
16
16
|
|
|
17
|
-
|
|
17
|
+
We don't generally recommend doing the following, but if all else fails you can perform a user-owned installation with the command
|
|
18
18
|
|
|
19
|
-
|
|
19
|
+
gem install datafarming --user-install
|
|
20
|
+
|
|
21
|
+
The specific location of the installation will depend on your OS. You will need to read documentation to identify the destination folder and manually add it to your runtime environment's search PATH.
|
|
20
22
|
|
|
21
|
-
|
|
23
|
+
Gem installation only needs to be done once. If you explicitly downloaded the `.gem` file rather than using a network installation, you can delete it after performing the installation. When a newer version of the gem is released, you can upgrade to it by running `gem update`.
|
|
24
|
+
|
|
25
|
+
### USAGE
|
|
22
26
|
|
|
23
|
-
|
|
27
|
+
Ruby is a powerful and concise object-oriented scripting language. Ruby scripts are normally run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments. However, scripts installed as gems do not need the `ruby` command to be typed explicitly. For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory. Additionally, on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
|
|
24
28
|
|
|
25
|
-
After installing the gem and making sure that gems are on your PATH, run the command `datafarming.rb`
|
|
29
|
+
All scripts in this distribution are self documenting if invoked with a `--help`, or `-h` option. After installing the gem and making sure that gems are on your PATH, run the command `datafarming.rb` at any time to see short overview descriptions of the various data farming tools provided in this distribution.
|
data/datafarming.gemspec
CHANGED
|
@@ -7,9 +7,9 @@ Gem::Specification.new do |s|
|
|
|
7
7
|
s.name = "datafarming"
|
|
8
8
|
s.version = DataFarming::VERSION
|
|
9
9
|
s.date = "#{Date.today}"
|
|
10
|
-
s.summary = "Useful
|
|
10
|
+
s.summary = "Useful tools for data farming."
|
|
11
11
|
s.homepage = "https://bitbucket.org/paul_j_sanchez/datafarmingrubyscripts"
|
|
12
|
-
s.email = "pjs@
|
|
12
|
+
s.email = "prof.pjs@icloud.com"
|
|
13
13
|
s.description = "Ruby scripts for data farming, including pre- and" \
|
|
14
14
|
"post-processing, design generation and scaling, and run control."
|
|
15
15
|
s.author = "Paul J Sanchez"
|
data/exe/datafarming.rb
CHANGED
|
@@ -22,25 +22,25 @@ lines = [
|
|
|
22
22
|
' • blank2csv.rb —',
|
|
23
23
|
' convert whitespace-delimited files to Comma Separated Values (CSV) files',
|
|
24
24
|
' • cat.rb —',
|
|
25
|
-
' Unix
|
|
26
|
-
' or concatenating multiple files, based on stdio',
|
|
25
|
+
' Unix \'cat\' program work-alike for Windows—for viewing or creating text files',
|
|
26
|
+
' or concatenating multiple files, based on stdio.',
|
|
27
27
|
' • convert_line_endings.rb —',
|
|
28
28
|
' convert text files (including CSV files) to your current operating system',
|
|
29
|
-
' environment (e.g., DOS to Unix or Unix to DOS)',
|
|
29
|
+
' environment (e.g., DOS to Unix or Unix to DOS).',
|
|
30
30
|
' • csv2blank.rb —',
|
|
31
|
-
' convert CSV files to whitespace-delimited files',
|
|
31
|
+
' convert CSV files to whitespace-delimited files.',
|
|
32
32
|
' • pool_files.rb —',
|
|
33
33
|
' pool columns from separate files into a new file, useful for combining',
|
|
34
|
-
' design file
|
|
34
|
+
' design file with output files or multiple output files.',
|
|
35
35
|
' • stripheaders.rb , stripheaderdups.rb —',
|
|
36
36
|
' remove column headers from a file, or duplicate headers within a file,',
|
|
37
|
-
' respectively',
|
|
37
|
+
' respectively.',
|
|
38
38
|
'',
|
|
39
39
|
'#### DESIGN GENERATORS',
|
|
40
40
|
'',
|
|
41
41
|
' • cross.rb —',
|
|
42
42
|
' create a combinatorial design by crossing all combinations of any # of',
|
|
43
|
-
' individual smaller designs',
|
|
43
|
+
' individual smaller designs.',
|
|
44
44
|
' • generate_design.rb —',
|
|
45
45
|
' allowable design types are',
|
|
46
46
|
' - resv : Resolution V fractional factorial at 2 levels per factor;',
|
|
@@ -57,21 +57,25 @@ lines = [
|
|
|
57
57
|
' allow partial correlation amongst the interactions which',
|
|
58
58
|
' significantly reduces the number of design points;',
|
|
59
59
|
' all of which generate designs with stacking, replication, and individualized',
|
|
60
|
-
' factor scaling',
|
|
60
|
+
' factor scaling. For continuous-factor designs (fbd, fbs, nolh) individual',
|
|
61
|
+
' factors can be converted to discrete with some potential loss of',
|
|
62
|
+
' orthogonality, or nearly-balanced categoricals where category counts',
|
|
63
|
+
' differ by no more than 1.',
|
|
64
|
+
' Note - use caution with multiple categoricals, since not all possible',
|
|
65
|
+
' combinations may be realized.',
|
|
61
66
|
'',
|
|
62
67
|
'#### RUN CONTROL',
|
|
63
68
|
'',
|
|
64
69
|
' • batchrunner.rb —',
|
|
65
|
-
' run a model interactively with replication',
|
|
70
|
+
' run a model interactively with replication.',
|
|
66
71
|
' • rundesign_general.rb —',
|
|
67
|
-
' run control to replicate a designed experiment applied to a model',
|
|
72
|
+
' run control to replicate a designed experiment applied to a model.',
|
|
73
|
+
' • run_yaml_design.rb —',
|
|
74
|
+
' run control to replicate a designed experiment applied to a model that',
|
|
75
|
+
' uses YAML for inputs.',
|
|
68
76
|
'',
|
|
69
77
|
'#### CONFIDENCE INTERVALS FOR TIME SERIES',
|
|
70
78
|
'',
|
|
71
|
-
' • mser.rb —',
|
|
72
|
-
' use MSER truncation to remove initial transient effects for time-series',
|
|
73
|
-
' output, reporting truncated average and number of observations for each run',
|
|
74
|
-
' to facilitate construction of a properly weighted confidence interval.',
|
|
75
79
|
' • mser_nolbm.rb —',
|
|
76
80
|
' use MSER truncation to remove initial transient effects for time-series',
|
|
77
81
|
' output, then calculate a 95% confidence interval on the remaining data using',
|
data/exe/generate_design.rb
CHANGED
|
@@ -1,21 +1,30 @@
|
|
|
1
|
-
#!/usr/bin/env ruby
|
|
2
|
-
# frozen_string_literal: true
|
|
1
|
+
#!/usr/bin/env ruby
|
|
3
2
|
|
|
4
|
-
|
|
5
|
-
# String.disable_colorization false
|
|
6
|
-
# require 'datafarming/error_handling'
|
|
3
|
+
RubyVM::YJIT.enable if defined?(RubyVM::YJIT) && RubyVM::YJIT.respond_to?(:enable)
|
|
7
4
|
|
|
8
5
|
require 'optparse'
|
|
9
6
|
require 'yaml'
|
|
10
7
|
|
|
11
8
|
require 'datafarming/scale_design'
|
|
12
9
|
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
10
|
+
CLASSICAL_DESIGNS = %i[resv ccd rotatable nolh].freeze
|
|
11
|
+
FREQ_DESIGNS = %i[fbd fbs].freeze
|
|
12
|
+
DESIGN_TYPES = (CLASSICAL_DESIGNS + FREQ_DESIGNS).freeze
|
|
13
|
+
MIN_MAX = %i[min max].freeze
|
|
14
|
+
ELECTIVE_SPECS = %i[id decimals cat disc].freeze
|
|
15
|
+
NOLH_LEVELS = [17, 33, 65, 129, 257, 512].freeze
|
|
17
16
|
TWO_PI = 2.0 * Math::PI
|
|
18
17
|
|
|
18
|
+
DEFAULT_OPTIONS = {
|
|
19
|
+
design: nil,
|
|
20
|
+
levels: nil,
|
|
21
|
+
headers: true,
|
|
22
|
+
replicate: 1,
|
|
23
|
+
stack: 1,
|
|
24
|
+
centerdup: false,
|
|
25
|
+
specs: []
|
|
26
|
+
}
|
|
27
|
+
|
|
19
28
|
HELP_MSG = [
|
|
20
29
|
'Generate, scale, stack, and replicate a variety of statistical designs',
|
|
21
30
|
"of experiments. You are required to specify the desired '--design TYPE'.",
|
|
@@ -24,7 +33,7 @@ HELP_MSG = [
|
|
|
24
33
|
].freeze
|
|
25
34
|
|
|
26
35
|
EXTENDED_HELP = [
|
|
27
|
-
"====================\n",
|
|
36
|
+
# "====================\n",
|
|
28
37
|
"Using the '-k' option will specify the number of factors for a standardized",
|
|
29
38
|
'design, in which all factors are normalized to the range of [-1, 1].',
|
|
30
39
|
'',
|
|
@@ -34,19 +43,47 @@ EXTENDED_HELP = [
|
|
|
34
43
|
"specifications provided (thus overriding the '-k' option). Each factor's",
|
|
35
44
|
'specification is separated from the others by whitespace.',
|
|
36
45
|
'',
|
|
37
|
-
'
|
|
38
|
-
'cannot include any whitespace. The format is:',
|
|
46
|
+
'Continuous factor specifications contain two to four comma-separated',
|
|
47
|
+
'components, and cannot include any whitespace. The format is:',
|
|
39
48
|
"\n id:STRING,min:NUMBER,max:NUMBER,decimals:INTEGER\n",
|
|
40
|
-
"Components can be
|
|
49
|
+
"Components can be provided in any order. The components 'min' and 'max'",
|
|
41
50
|
'are required, and define the range of experimentation for this factor.',
|
|
42
|
-
"Components 'id' and 'decimals' are optional. If 'id' is omitted, id",
|
|
43
|
-
"
|
|
44
|
-
'
|
|
45
|
-
'
|
|
51
|
+
"Components 'id' and 'decimals' are optional. If 'id' is omitted, id values",
|
|
52
|
+
"will be generated for use with the '--headers' option. If 'decimals' is",
|
|
53
|
+
'omitted, the values will be displayed with default (full) precision.',
|
|
54
|
+
'',
|
|
55
|
+
'Discrete and categorical factors are also options. The core design',
|
|
56
|
+
'underpinning mixtures of continuous/discrete/categorical should be one',
|
|
57
|
+
"of the continuous designs: fbd, fbs, or nolh",
|
|
58
|
+
'',
|
|
59
|
+
"Discrete factors are generated using a specification of the form:",
|
|
60
|
+
"\n disc:INTEGER,id:STRING,min:NUMBER,max:NUMBER,decimals:INTEGER\n",
|
|
61
|
+
'The first INTEGER specifies the number of discrete levels. As with',
|
|
62
|
+
"continuous factors, 'min' and 'max' are required, 'id' and 'decimals'",
|
|
63
|
+
"are optional. The results can be non-integer if 'min' and 'max' are",
|
|
64
|
+
'specified with decimal values.',
|
|
65
|
+
'',
|
|
66
|
+
"Categorical factors are generated using a specification of the form:",
|
|
67
|
+
"\n cat:RANGE,id:STRING\n",
|
|
68
|
+
'or',
|
|
69
|
+
"\n cat:'[ARRAY]',id:STRING\n",
|
|
70
|
+
"If the RANGE option is used, it is a Ruby range of the form 'FIRST..LAST'",
|
|
71
|
+
"or 'FIRST...LAST'. The two-dot version generates the sequence of values",
|
|
72
|
+
'between FIRST and LAST, inclusive, while the three-dot version excludes the',
|
|
73
|
+
'LAST value. Ruby allows ranges of integers or lexicographic strings. If the',
|
|
74
|
+
'ARRAY specification is used, a comma-separated set of specific values is',
|
|
75
|
+
'enumerated. Note that the square brackets surrounding the array values MUST',
|
|
76
|
+
'be contained between single quotation marks.',
|
|
77
|
+
'',
|
|
78
|
+
"For large designs with many factors, using the '-k' and '-o' options",
|
|
79
|
+
'together is highly recommended. The resulting YAML output can then be',
|
|
80
|
+
'edited at your leisure, while a long multi-factor set of specification',
|
|
81
|
+
'strings is prone to human error on entry.',
|
|
82
|
+
'',
|
|
46
83
|
].freeze
|
|
47
84
|
|
|
48
85
|
EXAMPLES = [
|
|
49
|
-
"\n====================\n",
|
|
86
|
+
# "\n====================\n",
|
|
50
87
|
'EXAMPLE 1:',
|
|
51
88
|
"\n #{File.basename($PROGRAM_NAME)} --design fbd -k 3 --no-headers",
|
|
52
89
|
"\n Generate a standardized frequency-based design with three factors,",
|
|
@@ -54,7 +91,7 @@ EXAMPLES = [
|
|
|
54
91
|
'',
|
|
55
92
|
'EXAMPLE 2:',
|
|
56
93
|
"\n #{File.basename($PROGRAM_NAME)} -d nolh id:Fred,min:-10,max:10 decimals:5,max:100,min:0",
|
|
57
|
-
"\n Generate
|
|
94
|
+
"\n Generate an NOLH for two factors. The first factor will have a column",
|
|
58
95
|
" header of 'Fred', and values will be scaled in the range -10..10. The",
|
|
59
96
|
" second column will have an automatically generated header 'X_002', and",
|
|
60
97
|
' values will be in the range 0..100 with five decimal places of precision.',
|
|
@@ -62,15 +99,23 @@ EXAMPLES = [
|
|
|
62
99
|
" by whitespace.",
|
|
63
100
|
'',
|
|
64
101
|
'EXAMPLE 3:',
|
|
65
|
-
"\n #{File.basename($PROGRAM_NAME)} -d
|
|
66
|
-
"\n Generate a
|
|
102
|
+
"\n #{File.basename($PROGRAM_NAME)} -d fbd disc:5,min:-1.0,max:1.0 cat:a..e",
|
|
103
|
+
"\n Generate a mixed discrete/categorical design based on an underlying",
|
|
104
|
+
' frequency-based design. The first factor will have values stepped between',
|
|
105
|
+
' -1 and 1 in steps of 0.5. The second column will consist of a balanced mix',
|
|
106
|
+
' of the letters a, b, c, d, and e. Both columns will have automatically',
|
|
107
|
+
' generated headers.',
|
|
108
|
+
'',
|
|
109
|
+
'EXAMPLE 4:',
|
|
110
|
+
"\n #{File.basename($PROGRAM_NAME)} -d nolh -k 5 -o my_template.yml",
|
|
111
|
+
"\n Generate a YAML template for a 5-factor NOLH design, and send it",
|
|
67
112
|
" to the file 'my_template.yml'. (The output file can be given any name",
|
|
68
113
|
' of your choosing.) The resulting YAML file has an intuitive human',
|
|
69
114
|
" readable format. It can be edited using your favorite programmer's",
|
|
70
115
|
' text editor (NOTE: word processors will not work!). After saving your',
|
|
71
116
|
' revisions, you can read it back in to generate the actual design using',
|
|
72
117
|
" the '--input_yaml' option:\n",
|
|
73
|
-
" #{File.basename($PROGRAM_NAME)}
|
|
118
|
+
" #{File.basename($PROGRAM_NAME)} -i my_template.yml",
|
|
74
119
|
' '
|
|
75
120
|
].freeze
|
|
76
121
|
|
|
@@ -80,13 +125,13 @@ def prog_help(opts)
|
|
|
80
125
|
end
|
|
81
126
|
|
|
82
127
|
def verbose(opts)
|
|
83
|
-
puts opts
|
|
128
|
+
# puts opts
|
|
84
129
|
puts EXTENDED_HELP.join("\n")
|
|
85
130
|
exit
|
|
86
131
|
end
|
|
87
132
|
|
|
88
133
|
def examples(opts)
|
|
89
|
-
puts opts
|
|
134
|
+
# puts opts
|
|
90
135
|
puts EXAMPLES.join("\n")
|
|
91
136
|
exit
|
|
92
137
|
end
|
|
@@ -96,6 +141,53 @@ def fatal_err(msg)
|
|
|
96
141
|
exit 1
|
|
97
142
|
end
|
|
98
143
|
|
|
144
|
+
def thresholds(k)
|
|
145
|
+
const = Math::PI / (2 * k)
|
|
146
|
+
Array.new(k) { |idx| Math.sin(const * (2 * (idx+1) - k)) }
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
class Array
|
|
150
|
+
@@toggle = true
|
|
151
|
+
def toggle_cmp(x, y)
|
|
152
|
+
if x == 0.0
|
|
153
|
+
(@@toggle ^= true) ? x > y : x >= y
|
|
154
|
+
else
|
|
155
|
+
x >= y
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def discretize(spec)
|
|
160
|
+
stepsize = (spec[:max] - spec[:min]) / (spec[:disc] - 1)
|
|
161
|
+
ary = Array.new(spec[:disc]) { |i| spec[:min] + i * stepsize }
|
|
162
|
+
if FREQ_DESIGNS.include? spec[:design]
|
|
163
|
+
t = thresholds(spec[:disc])
|
|
164
|
+
else
|
|
165
|
+
stepsize = 2.0 / spec[:disc]
|
|
166
|
+
t = Array.new(spec[:disc] - 1) { |i| (i + 1) * stepsize - 1 } + [1]
|
|
167
|
+
end
|
|
168
|
+
decimals = spec[:decimals] || 15
|
|
169
|
+
self.map { |value| ary[t.bsearch_index { |x| x >= value }].round(decimals) }
|
|
170
|
+
end
|
|
171
|
+
|
|
172
|
+
def categorize(spec)
|
|
173
|
+
categories = if spec[:cat].include? '..'
|
|
174
|
+
Range.new(*(spec[:cat].split('..'))).to_a
|
|
175
|
+
elsif spec[:cat].is_a? Array
|
|
176
|
+
spec[:cat]
|
|
177
|
+
else
|
|
178
|
+
fail "Invalid category specification: #{spec[:cat]}"
|
|
179
|
+
end
|
|
180
|
+
# t = thresholds(categories.size)
|
|
181
|
+
if FREQ_DESIGNS.include? spec[:design]
|
|
182
|
+
t = thresholds(categories.size)
|
|
183
|
+
else
|
|
184
|
+
stepsize = 2.0 / categories.size
|
|
185
|
+
t = Array.new(categories.size - 1) { |i| (i + 1) * stepsize - 1 } + [1]
|
|
186
|
+
end
|
|
187
|
+
self.map { |x| categories[t.bsearch_index { |entry| entry >= x }] }
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
|
|
99
191
|
def resv(options)
|
|
100
192
|
require 'datafarming/res_v_seqs'
|
|
101
193
|
num_factors = options[:specs].size
|
|
@@ -154,12 +246,10 @@ def freqs2cols(options, design_set)
|
|
|
154
246
|
freqs = ds.freqs[design_num]
|
|
155
247
|
Array.new(freqs.size) do |row|
|
|
156
248
|
omega = freqs[row]
|
|
157
|
-
|
|
249
|
+
Array.new(ds.nyq) do |i|
|
|
158
250
|
f = ((i * omega) % ds.nyq).to_f / ds.nyq
|
|
159
251
|
Math.sin(f * TWO_PI)
|
|
160
252
|
end
|
|
161
|
-
scale_factor = v.max
|
|
162
|
-
v.map { |x| (x / scale_factor).round(15) }
|
|
163
253
|
end.transpose
|
|
164
254
|
end
|
|
165
255
|
|
|
@@ -174,6 +264,7 @@ def fbs(options)
|
|
|
174
264
|
end
|
|
175
265
|
|
|
176
266
|
def validate_opts(options)
|
|
267
|
+
options.delete_if { |k, v| v.nil? }
|
|
177
268
|
required_options = [:design]
|
|
178
269
|
missing = required_options - options.keys
|
|
179
270
|
fatal_err("Missing required options: --#{missing.join(', --')}") unless missing.empty?
|
|
@@ -181,18 +272,18 @@ def validate_opts(options)
|
|
|
181
272
|
end
|
|
182
273
|
|
|
183
274
|
def validate(spec)
|
|
184
|
-
unknown = spec.keys -
|
|
275
|
+
unknown = spec.keys - MIN_MAX - ELECTIVE_SPECS
|
|
185
276
|
fatal_err("Unknown factor spec(s): #{unknown}") unless unknown.empty?
|
|
186
|
-
missing =
|
|
277
|
+
missing = (has_cat = spec.keys.include?(:cat)) ? [] : MIN_MAX - spec.keys
|
|
187
278
|
fatal_err("Factor spec #{spec} missing #{missing}") unless missing.empty?
|
|
188
279
|
end
|
|
189
280
|
|
|
190
281
|
def optional_specs(factor_spec, i)
|
|
191
|
-
YAML.safe_load("{#{factor_spec}}".gsub(':', ': '),
|
|
192
|
-
permitted_classes: [Symbol]).tap do |spec|
|
|
282
|
+
YAML.safe_load("{#{factor_spec}}".gsub(':', ': '), permitted_classes: [Symbol]).tap do |spec|
|
|
193
283
|
spec.transform_keys!(&:to_sym)
|
|
194
|
-
spec[:decimals] ||= nil
|
|
195
284
|
spec[:id] ||= "X_#{format('%03d', (i + 1))}"
|
|
285
|
+
spec.delete_if { |k, v| v.nil? }
|
|
286
|
+
spec[:decimals] = nil unless (spec.has_key?(:decimals) || spec.has_key?(:cat))
|
|
196
287
|
validate(spec)
|
|
197
288
|
end
|
|
198
289
|
end
|
|
@@ -206,20 +297,12 @@ def parse_specs(options, remainder)
|
|
|
206
297
|
end
|
|
207
298
|
|
|
208
299
|
def parse_args
|
|
209
|
-
default_options = {
|
|
210
|
-
headers: true,
|
|
211
|
-
replicate: 1,
|
|
212
|
-
stack: 1,
|
|
213
|
-
center: false,
|
|
214
|
-
specs: []
|
|
215
|
-
}
|
|
216
|
-
|
|
217
300
|
yaml_input = false
|
|
218
301
|
if ARGV.size > 2 && ARGV.any? { |elt| elt.eql?("-i") || elt.eql?("--input_yaml") }
|
|
219
302
|
STDERR.puts 'NOTE: All other arguments ignored when using YAML input'
|
|
220
303
|
end
|
|
221
304
|
|
|
222
|
-
options =
|
|
305
|
+
options = DEFAULT_OPTIONS
|
|
223
306
|
|
|
224
307
|
begin
|
|
225
308
|
remainder = OptionParser.new do |opts|
|
|
@@ -242,25 +325,48 @@ def parse_args
|
|
|
242
325
|
'Replicate QTY times', '(default: 1)')
|
|
243
326
|
opts.on('-s', '--stack QTY', /[1-9][0-9]*/, Integer,
|
|
244
327
|
'Stack QTY times', '(default: 1)')
|
|
245
|
-
opts.on('--[no-]
|
|
246
|
-
'Stacking
|
|
328
|
+
opts.on('-c', '--[no-]centerdup', 'NOTE: Applies only to stacked designs',
|
|
329
|
+
'Stacking can duplicate the center point','(default: no-centerdup)')
|
|
247
330
|
opts.on('--[no-]headers', 'Output has column headers', '(default: headers)')
|
|
248
|
-
opts.on('-l', '--levels QTY', Regexp.new("(#{
|
|
331
|
+
opts.on('-l', '--levels QTY', Regexp.new("(#{NOLH_LEVELS.join(')|(')})"),
|
|
249
332
|
Integer, 'NOTE: Applies only to NOLH designs',
|
|
250
333
|
'Number of levels in the base NOLH:',
|
|
251
|
-
" #{
|
|
252
|
-
opts.on('-o', '--output_yaml',
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
334
|
+
" #{NOLH_LEVELS.join ' '}")
|
|
335
|
+
opts.on('-o', '--output_yaml [FILENAME]',
|
|
336
|
+
'Write design YAML to FILENAME (or STDOUT)'
|
|
337
|
+
) do |fname|
|
|
338
|
+
if fname.nil? || '-' == fname
|
|
339
|
+
STDOUT
|
|
340
|
+
else
|
|
341
|
+
fname
|
|
342
|
+
end
|
|
343
|
+
end
|
|
344
|
+
opts.on(
|
|
345
|
+
"-i", "--input_yaml [FILENAME]",
|
|
346
|
+
'Read design YAML from FILENAME (or STDIN)',
|
|
347
|
+
'(Mutually exclusive with all other options,',
|
|
348
|
+
'prints warning to STDERR if other options',
|
|
349
|
+
'are passed)'
|
|
350
|
+
) do |fname|
|
|
256
351
|
ARGV.clear
|
|
257
|
-
options = {specs: []}
|
|
352
|
+
# options = {specs: []}
|
|
353
|
+
options = DEFAULT_OPTIONS
|
|
258
354
|
yaml_input = true
|
|
259
|
-
|
|
260
|
-
|
|
355
|
+
if fname.nil? || '-' == fname
|
|
356
|
+
YAML::safe_load(
|
|
357
|
+
STDIN.read, permitted_classes: [Symbol]
|
|
358
|
+
).each do |k, v|
|
|
359
|
+
options[k.to_sym] = v
|
|
360
|
+
end
|
|
361
|
+
else
|
|
362
|
+
YAML::safe_load_file(
|
|
363
|
+
fname, permitted_classes: [Symbol]
|
|
364
|
+
).each do |k, v|
|
|
365
|
+
options[k.to_sym] = v
|
|
366
|
+
end
|
|
261
367
|
end
|
|
262
368
|
end
|
|
263
|
-
|
|
369
|
+
end.parse!(into: options)
|
|
264
370
|
rescue OptionParser::ParseError => error
|
|
265
371
|
fatal_err("#{error}\n(-h or --help will show valid options)")
|
|
266
372
|
end
|
|
@@ -279,23 +385,23 @@ def parse_args
|
|
|
279
385
|
options
|
|
280
386
|
end
|
|
281
387
|
|
|
282
|
-
def stack(design, num_stacks,
|
|
388
|
+
def stack(design, num_stacks, center_pt_dup)
|
|
283
389
|
return design if num_stacks < 2
|
|
284
|
-
fatal_err("Stacking #{num_stacks} times exceeds number of design columns") if num_stacks > design
|
|
390
|
+
fatal_err("Stacking #{num_stacks} times exceeds number of design columns") if num_stacks > design.size
|
|
285
391
|
num_stacks -= 1
|
|
286
|
-
result = design
|
|
392
|
+
result = design
|
|
287
393
|
if num_stacks > 0
|
|
288
|
-
design_t = if
|
|
289
|
-
design.
|
|
394
|
+
design_t = if center_pt_dup
|
|
395
|
+
design.clone
|
|
290
396
|
else
|
|
291
|
-
design.delete_if {|row| row.all? &:zero? }.transpose
|
|
397
|
+
design.transpose.delete_if {|row| row.all? &:zero? }.transpose
|
|
292
398
|
end
|
|
293
399
|
num_stacks.times do
|
|
294
400
|
design_t.rotate!
|
|
295
|
-
result.
|
|
401
|
+
result.size.times { |i| result[i] += design_t[i] }
|
|
296
402
|
end
|
|
297
403
|
end
|
|
298
|
-
result
|
|
404
|
+
result
|
|
299
405
|
end
|
|
300
406
|
|
|
301
407
|
ARGV << '--help' if ARGV.empty?
|
|
@@ -303,26 +409,46 @@ ARGV << '--help' if ARGV.empty?
|
|
|
303
409
|
options = parse_args
|
|
304
410
|
|
|
305
411
|
if options[:output_yaml]
|
|
306
|
-
options
|
|
307
|
-
|
|
412
|
+
o_file = (STDOUT == options[:output_yaml]) ? STDOUT : File.open(options[:output_yaml], 'w')
|
|
413
|
+
options.delete(:output_yaml)
|
|
414
|
+
begin
|
|
415
|
+
o_file.puts options.to_yaml
|
|
416
|
+
ensure
|
|
417
|
+
o_file.close unless STDIN == o_file
|
|
418
|
+
end
|
|
308
419
|
else
|
|
309
|
-
base_design = method(options[:design]).call(options)
|
|
420
|
+
base_design = method(options[:design]).call(options).transpose.first(options[:specs].size)
|
|
310
421
|
separator = ','
|
|
311
422
|
puts Array.new(options[:specs].size) { |i| options[:specs][i][:id] }.join(separator) if options[:headers]
|
|
312
423
|
|
|
313
424
|
# stack
|
|
314
|
-
design = stack(base_design, options[:stack], options[:
|
|
315
|
-
.transpose.first(options[:specs].size).transpose
|
|
425
|
+
design = stack(base_design, options[:stack], options[:centerdup])
|
|
316
426
|
|
|
317
427
|
# scale
|
|
318
|
-
|
|
319
|
-
|
|
428
|
+
scale_factor = design.map &:max
|
|
429
|
+
d_type = options[:design]
|
|
430
|
+
design.map!.with_index do |v, i|
|
|
431
|
+
break if i >= options[:specs].size
|
|
432
|
+
spec = options[:specs][i]
|
|
433
|
+
if spec[:cat]
|
|
434
|
+
spec[:design] = d_type
|
|
435
|
+
v.categorize(spec)
|
|
436
|
+
elsif spec[:disc]
|
|
437
|
+
spec[:design] = d_type
|
|
438
|
+
v.discretize(spec)
|
|
439
|
+
else
|
|
440
|
+
scaler = Scaler.new(spec)
|
|
441
|
+
v.map do |x|
|
|
442
|
+
scaler.scale((x / scale_factor[i]).round(15))
|
|
443
|
+
end
|
|
444
|
+
end
|
|
320
445
|
end
|
|
321
446
|
|
|
322
447
|
# replicate
|
|
448
|
+
result = design.transpose
|
|
323
449
|
options[:replicate].times do
|
|
324
|
-
|
|
325
|
-
puts line.
|
|
450
|
+
result.each do |line|
|
|
451
|
+
puts line.join(separator)
|
|
326
452
|
end
|
|
327
453
|
end
|
|
328
454
|
end
|
data/exe/mser_nolbm.rb
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env ruby -w
|
|
2
2
|
|
|
3
|
+
RubyVM::YJIT.enable if defined?(RubyVM::YJIT) && RubyVM::YJIT.respond_to?(:enable)
|
|
4
|
+
|
|
3
5
|
require 'rubygems' if RUBY_VERSION =~ /^1\.8/
|
|
4
6
|
require 'colorize'
|
|
5
7
|
|
|
@@ -84,35 +86,31 @@ def nolbm(data)
|
|
|
84
86
|
avg = best[1]
|
|
85
87
|
index = best[2]
|
|
86
88
|
length = data.length - index
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
batch_means = Array.new(
|
|
94
|
-
|
|
95
|
-
m.times do
|
|
96
|
-
mean_stats.new_obs(data[index])
|
|
97
|
-
index += 1
|
|
98
|
-
end
|
|
99
|
-
mean_stats.avg
|
|
89
|
+
b = (index > 0) ? [length / 30, length / index].min : length / 30
|
|
90
|
+
m, excess = length.divmod b
|
|
91
|
+
incr, discard = excess.divmod m
|
|
92
|
+
b += incr
|
|
93
|
+
index += discard
|
|
94
|
+
usable = data.drop(index).each_slice(b).to_a
|
|
95
|
+
batch_means = Array.new(m) do |i|
|
|
96
|
+
QuickStats.new.add_set(usable[i]).avg
|
|
100
97
|
end
|
|
101
|
-
sum_squared_deviations = batch_means
|
|
102
|
-
map { |y| square(y - avg)}
|
|
103
|
-
inject(&:+)
|
|
104
|
-
|
|
105
|
-
|
|
98
|
+
sum_squared_deviations = batch_means
|
|
99
|
+
.map { |y| square(y - avg)}
|
|
100
|
+
.inject(&:+)
|
|
101
|
+
df = (m - 1)
|
|
102
|
+
var_hat = sum_squared_deviations / df
|
|
103
|
+
std_dev = Math.sqrt(var_hat)
|
|
106
104
|
std_err = Math.sqrt(var_hat / b)
|
|
107
105
|
half_width = T_VALUE[df] * std_err
|
|
108
106
|
lower = avg - half_width
|
|
109
107
|
upper = avg + half_width
|
|
110
|
-
printf "%f,%f,%d,%f,%f\n", avg, std_err, df, lower, upper
|
|
108
|
+
printf "%f,%f,%f,%d,%f,%f\n", avg, std_dev, std_err, df, lower, upper
|
|
111
109
|
end
|
|
112
110
|
|
|
113
|
-
puts "sample_mean,std_err,df,lower95_bound,upper95_bound"
|
|
114
|
-
if
|
|
115
|
-
data =
|
|
111
|
+
puts "sample_mean,std_dev,std_err,df,lower95_bound,upper95_bound"
|
|
112
|
+
if ARGV.empty?
|
|
113
|
+
data = ARGF.readlines
|
|
116
114
|
nolbm(data)
|
|
117
115
|
else
|
|
118
116
|
ARGV.each do |fname|
|