datafarming 1.4.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 760e46472df38776a825b576376f9ce407398647fc3bd5fd484c8787caa505bc
-  data.tar.gz: 57a92791102a99a4d8780c4f2584a4cb21ca6a4ae60545a06f1a1675d5633e82
+  metadata.gz: 420d0eef5a4e09793bbbcaea21a57047eb96ffa44aa28a96455d6d695002f556
+  data.tar.gz: 9e6dba22e9c9734c6da2555d01d4d2ed4f7fa5f0876badacb155c6a8a8fd542f
 SHA512:
-  metadata.gz: 76cd410243bdc003a488c8dae6cc5d15fd6723d5525ad28398be5febf2a34109e50e40359e2bc1782d369c2dabd89bb5a33b66769a6872c841e5e2c32b9830b9
-  data.tar.gz: 70d896019b4b1e6d439ec1977ac7fd1ed6969bdb2d6b368f14a993d37dc187e88d1537a0d2c48b77086808b57b6445b558ee243174b619381a82c94d7e0a3e68
+  metadata.gz: 1849ba07886a55ed337fd699b62075de26a2050756d2f5a6a00394efb1e4ca4098a488336a8e0071a766793ece844f518adc777e9783f061da760f949f6b58f7
+  data.tar.gz: 60eaa33d6d9d66f2fa12e4d7bce18f8c67ccffdce26964064ec414a2f68e11e9fff76c649b446436f25c29768848f8b53b11ea79d516727fd31469009d893253

data/LICENSE.md

@@ -1,7 +1,7 @@
 
 The MIT License (MIT)
 
-Copyright (c) 2017-2021 Paul J. Sanchez
+Copyright (c) 2017-2023 Paul J. Sanchez
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,8 +1,10 @@
 ### SETUP
 
-This gem requires Ruby v2.5 or later to be installed on your system.
-  - On MacOS, Ruby comes pre-installed.  Note: You can use the homebrew package manager from [brew.sh](https://brew.sh) if you want to install a newer version of Ruby which is noticeably faster than the one that currently ships with MacOS.
-  - Windows users should use one of the installer packages available from [rubyinstaller.org](http://rubyinstaller.org/).  You will need to determine whether your have a 32 or 64 bit processor, and download the appropriate installer.  We recommend Ruby v2.5.x or later, as it is substantially faster than prior versions.  When running the installer, check the checkboxes to add the Ruby installation to your PATH and to associate `.rb` and `.rbw` files with Ruby. A terminal window will pop up during the installation—press "enter" to automatically install all of the required tools, and when prompted again (several minutes later) press "enter" again to end the installation.
+This gem requires Ruby v3 or later to be installed on your system.
+
+  - On MacOS, Ruby comes pre-installed.  Note: You can use the homebrew package manager from [brew.sh](https://brew.sh) if you want to install a newer version of Ruby which has numerous advantages over the one that currently ships with MacOS.
+  
+  - Windows users should use one of the installer packages available from [rubyinstaller.org](http://rubyinstaller.org/).  You will need to determine whether your have a 32 or 64 bit processor, and download the appropriate installer.  When running the installer, check the checkboxes to add the Ruby installation to your PATH and to associate `.rb` and `.rbw` files with Ruby. A terminal window will pop up during the installation—press "enter" to automatically install all of the required tools, and when prompted again (several minutes later) press "enter" again to end the installation.
 
 You can check that Ruby is properly installed by opening `Terminal.app` (MacOS) or `CMD.EXE` (Windows) and typing `ruby --version` (note: two dashes).  You should see a response telling you which version of Ruby you are running.
 
@@ -10,12 +12,12 @@ After your Ruby installation is confirmed you can install the data farming Ruby
 
     gem install datafarming
 
-This assumes that you have a network connection. Additional dependencies will be installed automatically.  On MacOS or Linux systems, you may be required to use the `sudo` command to to authenticate the installation.  In that case, type `sudo gem install datafarming` and enter your password when prompted.
+This assumes that you have a network connection. Additional dependencies will be installed automatically.  On MacOS or Linux systems, you may be required to use the `sudo` command to authenticate the installation.  In that case, type `sudo gem install datafarming` and enter your password when prompted.
 
-Gem installation only needs to be done once.  If you explicitly downloaded the `.gem` file rather than using a network installation, you can delete it after performing the installation.
+Gem installation only needs to be done once.  If you explicitly downloaded the `.gem` file rather than using a network installation, you can delete it after performing the installation. If a newer version of the gem is released, you can upgrade to it with `gem update`.
 
 ### USAGE
 
 Ruby is a powerful and concise object-oriented scripting language.  Normally, Ruby scripts are run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments.  However, scripts installed as gems do not need the `ruby` command to be typed explicitly.  For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory.  Note that on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
 
-You can run the command `datafarming.rb` at any point after installing to see descriptions of the various data farming utilities included in this distribution.
+All scripts in this distribution are self documenting if run with a `--help`, or `-h` option. You can run the command `datafarming.rb` at any point after installing this gem to see descriptions of the various data farming utilities included in thie distribution.

data/datafarming.gemspec

@@ -1,14 +1,17 @@
 # -*- ruby -*-
-_VERSION = "1.4.0"
+
+require 'date'
+require_relative "lib/datafarming/version"
 
 Gem::Specification.new do |s|
   s.name = "datafarming"
-  s.version = _VERSION
-  s.date = "2021-01-19"
+  s.version = DataFarming::VERSION
+  s.date = "#{Date.today}"
   s.summary = "Useful scripts for data farming."
   s.homepage = "https://bitbucket.org/paul_j_sanchez/datafarmingrubyscripts"
   s.email = "pjs@alum.mit.edu"
-  s.description = "Ruby scripts for data farming, including pre- and post-processing, design generation and scaling, and run control."
+  s.description = "Ruby scripts for data farming, including pre- and" \
+    "post-processing, design generation and scaling, and run control."
   s.author = "Paul J Sanchez"
   s.files = `git ls-files -z`.split("\x0").reject do |f|
     f.match(%r{^(test/|features/|.gitignore)})
@@ -17,8 +20,10 @@ Gem::Specification.new do |s|
   s.executables = s.files.grep(%r{^exe/}) { |f| File.basename(f) }
   s.require_paths = ["lib"]
   s.add_runtime_dependency 'fwt', '~> 1.2'
-  s.add_runtime_dependency 'colorize', '~> 0.8'
+  s.add_runtime_dependency 'colorize', '~> 1'
   s.add_runtime_dependency 'quickstats', '~> 2'
-  s.required_ruby_version = '>= 2.5'
+  s.add_runtime_dependency 'optparse', '~> 0.4'
+  s.add_runtime_dependency 'yaml', '~> 0.3'
+  s.required_ruby_version = '>= 2.6'
   s.license = 'MIT'
 end

data/exe/datafarming.rb

@@ -2,22 +2,22 @@
 
 lines = [
 '',
-'  #### COMMAND-LINE TOOLS',
+'#### COMMAND-LINE TOOLS OVERVIEW',
 '',
-'  Ruby is a powerful and concise  object-oriented scripting language.  Scripts',
-'  are normally run from a command-line or terminal environment by typing the',
-'  ruby  command followed by a script name, often followed by one or more',
-'  command-line arguments.  However, scripts installed as gems do not need the',
-'  ruby  command to be typed explicitly.  For example, typing',
-'  stripheaderdups.rb my_file.txt  will invoke the  stripheaderdups.rb  script',
-'  and apply it to file  my_file.txt  in your current working directory.  Note',
-'  that on Windows the  .rb  suffix is not needed to run your scripts, e.g.,',
-'  stripheaderdups my_file.txt  will suffice.',
+'  Ruby is a powerful and concise object-oriented scripting language.',
+'  Scripts are normally run from a command-line or terminal environment by',
+'  typing the ruby command followed by a script name, often followed by one',
+'  or more command-line arguments.  However, scripts installed as gems do',
+'  not need the ruby  command to be typed explicitly.  For example, typing',
+'  \'stripheaderdups.rb my_file.txt\' will invoke the \'stripheaderdups.rb\'',
+'  script and apply it to file \'my_file.txt\' in your current working directory.',
+'  Note that on Windows the \'.rb\' suffix is not needed to run your scripts,',
+'  e.g.,\'stripheaderdups my_file.txt\' will suffice.',
 '',
-'  All scripts in this distribution are self documenting if run with a --help,',
-'  -h, or -? option.  Quick descriptions follow.',
+'  All scripts in this distribution are self documenting if run with a \'--help\'',
+'  or \'-h\' option.  Quick descriptions follow.',
 '',
-'  #### DATA MANIPULATION',
+'#### DATA MANIPULATION',
 '',
 '  •  blank2csv.rb  —',
 '  convert whitespace-delimited files to Comma Separated Values (CSV) files',
@@ -36,31 +36,37 @@ lines = [
 '  remove column headers from a file, or duplicate headers within a file,',
 '  respectively',
 '',
-'  #### DESIGN GENERATORS',
+'#### DESIGN GENERATORS',
 '',
-'  •  augment_design.rb  —',
-'  generate star points to augment an already existing factorial design',
 '  •  cross.rb  —',
 '  create a combinatorial design by crossing all combinations of any # of',
 '  individual smaller designs',
-'  •  scaled_fde.rb  —',
-'  generate fully orthogonal quadratic designs based on discrete Fourier',
-'  frequencies, with factor scaling',
-'  •  scaled_rf_cubed.rb  —',
-'  generate 2-level Resolution V fractional factorial designs and Central',
-'  Composite Designs (CCDs), with factor scaling',
-'  •  stack_nolhs.rb  —',
-'  generate designs by reassigning columns from pre-tabled NOLHs, with factor',
-'  scaling, for up to 100 factors',
+'  •  generate_design.rb  —',
+'  allowable design types are',
+'     - resv      : Resolution V fractional factorial at 2 levels per factor;',
+'     - ccd       : Central Composite Designs are resv augmented with "star',
+'                   points", yielding 3 levels per factor;',
+'     - rotatable : CCDs with corner points scaled so that all non-central',
+'                   design points are equidistant from the center;',
+'     - nolh      : Nearly Orthogonal Latin Hypercubes are highly efficient',
+'                   space filling designs for main effects models;',
+'     - fbd       : Frequency-Based Designs are space filling and orthogonal',
+'                   for full 2nd order models, offering zero correlation',
+'                   between main effects, quadratics, and two-way interactions;',
+'     - fbs       : Frequency-Based Screening Designs are similar to FBDs, but',
+'                   allow partial correlation amongst the interactions which',
+'                   significantly reduces the number of design points;',
+'  all of which generate designs with stacking, replication, and individualized',
+'  factor scaling',
 '',
-'  #### RUN CONTROL',
+'#### RUN CONTROL',
 '',
 '  •  batchrunner.rb  —',
 '  run a model interactively with replication',
 '  •  rundesign_general.rb  —',
 '  run control to replicate a designed experiment applied to a model',
 '',
-'  #### CONFIDENCE INTERVALS FOR TIME SERIES',
+'#### CONFIDENCE INTERVALS FOR TIME SERIES',
 '',
 '  •  mser.rb  —',
 '  use MSER truncation to remove initial transient effects for time-series',

data/exe/generate_design.rb

@@ -0,0 +1,304 @@
+#!/usr/bin/env ruby -w
+# frozen_string_literal: true
+
+# require 'colorize'
+# String.disable_colorization false
+# require 'datafarming/error_handling'
+
+require 'optparse'
+require 'yaml'
+
+require 'datafarming/scale_design'
+
+DESIGN_TYPES = %i[resv ccd rotatable nolh fbd fbs].freeze
+REQUIRED_SPECS = %i[min max].freeze
+ELECTIVE_SPECS = %i[id decimals].freeze
+LEVELS = [17, 33, 65, 129, 257, 512].freeze
+TWO_PI = 2.0 * Math::PI
+
+HELP_MSG = [
+  'Generate, scale, stack, and replicate a variety of statistical designs',
+  "of experiments. You are required to specify the desired '--design TYPE'.",
+  '',
+  "Usage: #{File.basename($PROGRAM_NAME)} [options] [optional factor specs]"
+].freeze
+
+EXTENDED_HELP = [
+  "====================\n",
+  "Using the '-k' option will specify the number of factors for a standardized",
+  'design, in which all factors are normalized to the range of [-1, 1].',
+  '',
+  'Optional factor specifications can be used to create a design which scales',
+  'the factors to individualized ranges. Specifications are then required for',
+  'all factors, and the number of factors will be derived from the number of',
+  "specifications provided (thus overriding the '-k' option). Each factor's",
+  'specification is separated from the others by whitespace.',
+  '',
+  'Factor specifications contain two to four comma-separated components, and',
+  'cannot include any whitespace. The format is:',
+  "\n   id:STRING,min:NUMBER,max:NUMBER,decimals:INTEGER\n",
+  "Components can be arranged in any order.  The components 'min' and 'max'",
+  'are required, and define the range of experimentation for this factor.',
+  "Components 'id' and 'decimals' are optional. If 'id' is omitted, id",
+  "values will be generated for use with the '--headers' option. If 'decimals'",
+  'is omitted, the factors values will be displayed with default precision.',
+  ' '
+].freeze
+
+EXAMPLES = [
+  "\n====================\n",
+  'EXAMPLE 1:',
+  "\n  #{File.basename($PROGRAM_NAME)} --design fbd -k 3 --no-headers",
+  "\n  Generate a standardized frequency-based design with three factors,",
+  '  without headers. All ranges are -1..1 and output has default precision.',
+  '',
+  'EXAMPLE 2:',
+  "\n  #{File.basename($PROGRAM_NAME)} -d nolh id:Fred,min:-10,max:10 decimals:5,max:100,min:0",
+  "\n  Generate a NOLH for two factors. The first factor will have a column",
+  "  header of 'Fred', and values will be scaled in the range -10..10. The",
+  "  second column will have an automatically generated header 'X_002', and",
+  '  values will be in the range 0..100 with five decimal places of precision.',
+  "  As described in '--verbose-help', factor specifications must be separated",
+  "  by whitespace.",
+  '',
+  'EXAMPLE 3:',
+  "\n  #{File.basename($PROGRAM_NAME)} -d nolh -k 5 --output_yaml > my_template.yaml",
+  "\n  Generate a YAML template for a 5-factor NOLH design, and redirect it",
+  "  to the file 'my_template.yaml'. (The output file can be given any name",
+  '  of your choosing.) The resulting YAML file has an intuitive human',
+  "  readable format. It can be edited using your favorite programmer's",
+  '  text editor (NOTE: word processors will not work!). After saving your',
+  '  revisions, you can read it back in to generate the actual design using',
+  "  the '--input_yaml' option:\n",
+  "  #{File.basename($PROGRAM_NAME)} --input_yaml < my_template.yaml",
+  ' '
+].freeze
+
+def prog_help(opts)
+  puts opts
+  exit
+end
+
+def verbose(opts)
+  puts opts
+  puts EXTENDED_HELP.join("\n")
+  exit
+end
+
+def examples(opts)
+  puts opts
+  puts EXAMPLES.join("\n")
+  exit
+end
+
+def resv(options)
+  require 'datafarming/res_v_seqs'
+  raise 'Too many factors for ResV designs' if options[:num_factors] > RESV::INDEX.size
+
+  RESV.make_design options[:num_factors]
+end
+
+def ccd(options)
+  require 'datafarming/res_v_seqs'
+  raise 'Too many factors for CCD designs' if options[:num_factors] > RESV::INDEX.size
+
+  RESV.make_design(options[:num_factors]) + RESV.star_pts(options[:num_factors])
+end
+
+def rotatable(options)
+  require 'datafarming/res_v_seqs'
+  raise 'Too many factors for Rotatable designs' if options[:num_factors] > RESV::INDEX.size
+
+  inv_len = 1.0 / Math.sqrt(options[:num_factors])
+  RESV.make_design(options[:num_factors]).each do |line|
+    line.map! { |value| value * inv_len }
+  end + RESV.star_pts(options[:num_factors])
+end
+
+def nolh(options)
+  require 'datafarming/nolh_designs'
+  minimal_size =
+    case options[:num_factors]
+    when 1..7
+      17
+    when 8..11
+      33
+    when 12..16
+      65
+    when 17..22
+      129
+    when 23..29
+      257
+    when 30..100
+      512
+    else
+      raise "invalid number of factors: #{options[:num_factors]}"
+    end
+
+  options[:levels] ||= minimal_size
+  if options[:levels] < minimal_size
+    raise "Latin hypercube with #{options[:levels]} levels is" \
+          "too small for #{options[:num_factors]} factors."
+  end
+
+  NOLH::DESIGN_TABLE[options[:levels]]
+end
+
+def freqs2cols(options, design_set)
+  raise "No design (yet) for #{options[:num_factors]} factors" unless
+    design_set.key? options[:num_factors]
+
+  design_num = -1 # the last one has largest min frequency
+
+  ds = design_set[options[:num_factors]]
+  freqs = ds.freqs[design_num]
+  Array.new(freqs.size) do |row|
+    omega = freqs[row]
+    Array.new(ds.nyq) do |i|
+      f = ((i * omega) % ds.nyq).to_f / ds.nyq
+      Math.sin(f * TWO_PI)
+    end
+  end.transpose
+end
+
+def fbd(options)
+  require 'datafarming/freq_sets'
+  freqs2cols(options, FBD::DESIGN_SETS)
+end
+
+def fbs(options)
+  require 'datafarming/screen_freq_sets'
+  freqs2cols(options, FBS::DESIGN_SETS)
+end
+
+def validate_opts(options)
+  required_options = [:design]
+  missing = required_options - options.keys
+  raise "Missing required options: #{missing}" unless missing.empty?
+
+  options[:design] = options[:design].to_sym
+end
+
+def validate(spec)
+  unknown = spec.keys - REQUIRED_SPECS - ELECTIVE_SPECS
+  raise "Unknown factor spec(s): #{unknown}" unless unknown.empty?
+
+  missing = REQUIRED_SPECS - spec.keys
+  raise "Factor spec #{spec} missing #{missing}" unless missing.empty?
+end
+
+def optional_specs(factor_spec, i)
+  YAML.safe_load("{#{factor_spec}}".gsub(':', ': '),
+    permitted_classes: [Symbol]).tap do |spec|
+    spec.transform_keys!(&:to_sym)
+    spec[:decimals] ||= nil
+    spec[:id] ||= "X_#{format('%03d', (i + 1))}"
+    validate(spec)
+  end
+end
+
+def parse_specs(options, remainder)
+  if remainder.size.positive?
+    options[:num_factors] = remainder.size
+    remainder.map.with_index { |factor_spec, i| optional_specs(factor_spec, i) }
+  elsif options.key? :num_factors
+    Array.new(options[:num_factors]) do |i|
+      { "id": "X_#{format('%03d', (i + 1))}", "min": -1, "max": 1, "decimals": nil }
+    end
+  else
+    y_options, factor_specs = YAML.safe_load($stdin, permitted_classes: [Symbol])
+    y_options.each_key { |key| options[key] ||= y_options[key] }
+    factor_specs.each { |spec| validate(spec) }
+  end
+end
+
+def parse_args
+  options = {
+    headers: true,
+    replicate: 1,
+    stack: 1,
+    center: false
+  }
+
+  remainder = OptionParser.new do |opts|
+    opts.banner = HELP_MSG.join("\n")
+    opts.on('-h', '--help', 'Prints help') { prog_help(opts) }
+    opts.on('-v', '--verbose-help', 'Print more verbose help') { verbose(opts) }
+    opts.on('-x', '--examples', 'Show some examples') { examples(opts) }
+    opts.on('-d', '--design=TYPE', DESIGN_TYPES,
+            'REQUIRED: Type of design, one of:',
+            "  #{DESIGN_TYPES.join ' '}")
+    opts.on('-k', '--num_factors=QTY', /[1-9][0-9]*/, Integer,
+            'Number of factors',
+            'Ignored if optional factor specs provided')
+    opts.on('-r', '--replicate=QTY', /[1-9][0-9]*/, Integer,
+            'Replicate QTY times', '(default: 1)')
+    opts.on('-s', '--stack=QTY', /[1-9][0-9]*/, Integer,
+            'Stack QTY times', '(default: 1)')
+    opts.on('--[no-]center', 'NOTE: Applies only to stacked designs',
+            'Stacking includes center pt','(default: no-center)')
+    opts.on('--[no-]headers', 'Output has column headers', '(default: headers)')
+    opts.on('-l', '--levels=QTY', Regexp.new("(#{LEVELS.join(')|(')})"),
+            Integer, 'NOTE: Applies only to NOLH designs',
+            'Number of levels in the base NOLH:',
+            "  #{LEVELS.join ' '}")
+    opts.on('-o', '--output_yaml', 'Write design YAML to STDOUT')
+    opts.on('-i', '--input_yaml', 'Read design YAML from STDIN')
+  end.parse!(into: options)
+
+  raise 'Optional factor specs ignored when using YAML input' if options[:input_yaml] && remainder.size.positive?
+
+  specs = parse_specs(options, remainder)
+  validate_opts options
+  [options, specs]
+end
+
+def stack(design, num_stacks, centered)
+  return design if num_stacks < 2
+  raise "Stacking #{num_stacks} times exceeds number of design columns" if num_stacks > design[0].length
+
+  num_stacks -= 1
+  result = design.transpose
+  if num_stacks > 0
+    design_t = if centered
+      design.transpose
+    else
+      design.delete_if {|row| row.all? &:zero? }.transpose
+    end
+    num_stacks.times do
+      design_t.rotate!
+      result.map!.with_index { |line, i| line + design_t[i] }
+    end
+  end
+  result.transpose
+end
+
+
+ARGV << '--help' if ARGV.empty?
+
+options, specs = parse_args
+
+if options[:output_yaml]
+  options.delete :output_yaml
+  puts [options, specs].to_yaml
+else
+  base_design = method(options[:design]).call(options)
+  separator = ','
+  puts Array.new(options[:num_factors]) { |i| specs[i][:id] }.join(separator) if options[:headers]
+
+  # stack
+  design = stack(base_design, options[:stack], options[:center])
+           .transpose.first(options[:num_factors]).transpose
+
+  # scale
+  scaler = specs.map do |spec|
+    Scaler.new(min: spec[:min], max: spec[:max], decimals: spec[:decimals])
+  end
+
+  # replicate
+  options[:replicate].times do
+    design.each do |line|
+      puts line.map.with_index { |v, i| scaler[i].scale(v) }.join(separator)
+    end
+  end
+end