datafarming 1.3.0 → 1.4.0

data/DESCRIPTIONS.md

@@ -0,0 +1,45 @@
+#### COMMAND-LINE TOOLS
+
+Ruby is a powerful and concise  object-oriented scripting language.  Scripts are normally run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments.  However, scripts installed as gems do not need the `ruby` command to be typed explicitly.  For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory.  Note that on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
+
+All scripts in this distribution are self documenting if run with a `--help`, `-h`, or `-?` option.  Quick descriptions follow.
+
+#### DATA MANIPULATION
+- `blank2csv.rb` —
+convert whitespace-delimited files to Comma Separated Values (CSV) files
+- `cat.rb` —
+Unix `cat` program work-alike for Windows—for viewing or creating text files or concatenating multiple files, based on stdio
+- `convert_line_endings.rb` —
+convert text files (including CSV files) to your current operating system environment (e.g., DOS to Unix or Unix to DOS)
+- `csv2blank.rb` —
+convert CSV files to whitespace-delimited files
+- `pool_files.rb` —
+pool columns from separate files into a new file, useful for combining design file & output files or multiple output files
+- `stripheaders.rb`, `stripheaderdups.rb` —
+remove column headers from a file, or duplicate headers within a file, respectively
+
+#### DESIGN GENERATORS
+- `augment_design.rb` —
+generate star points to augment an already existing factorial design
+- `cross.rb` —
+create a combinatorial design by crossing all combinations of any # of individual smaller designs
+- `scaled_fde.rb` —
+generate fully orthogonal quadratic designs based on discrete Fourier frequencies, with factor scaling
+- `scaled_rf_cubed.rb` —
+generate 2-level Resolution V fractional factorial designs and Central Composite Designs (CCDs), with factor scaling
+- `stack_nolhs.rb` —
+generate designs by reassigning columns from pre-tabled NOLHs, with factor scaling, for up to 100 factors
+
+#### RUN CONTROL
+  - `batchrunner.rb` —
+run a model interactively with replication
+  - `rundesign_general.rb` —
+run control to replicate a designed experiment applied to a model
+
+#### CONFIDENCE INTERVALS FOR TIME SERIES
+  - `mser.rb` —
+use MSER truncation to remove initial transient effects for time-series output, reporting truncated average and number of observations for each run to facilitate construction of a properly weighted confidence interval.
+  - `mser_nolbm.rb` —
+use MSER truncation to remove initial transient effects for time-series output, then calculate a 95% confidence interval on the remaining data using non-overlapping batch means.
+  - `mser_olbm.rb` —
+use MSER truncation to remove initial transient effects for time-series output, then calculate a 95% confidence interval on the remaining data using overlapping batch means.

data/LICENSE.md

@@ -1,7 +1,7 @@
 
 The MIT License (MIT)
 
-Copyright (c) 2017-2018 Paul J. Sanchez
+Copyright (c) 2017-2021 Paul J. Sanchez
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,6 +1,6 @@
 ### SETUP
 
-This gem requires Ruby v2.3 or later to be installed on your system.
+This gem requires Ruby v2.5 or later to be installed on your system.
   - On MacOS, Ruby comes pre-installed.  Note: You can use the homebrew package manager from [brew.sh](https://brew.sh) if you want to install a newer version of Ruby which is noticeably faster than the one that currently ships with MacOS.
   - Windows users should use one of the installer packages available from [rubyinstaller.org](http://rubyinstaller.org/).  You will need to determine whether your have a 32 or 64 bit processor, and download the appropriate installer.  We recommend Ruby v2.5.x or later, as it is substantially faster than prior versions.  When running the installer, check the checkboxes to add the Ruby installation to your PATH and to associate `.rb` and `.rbw` files with Ruby. A terminal window will pop up during the installation—press "enter" to automatically install all of the required tools, and when prompted again (several minutes later) press "enter" again to end the installation.
 
@@ -16,42 +16,6 @@ Gem installation only needs to be done once.  If you explicitly downloaded the `
 
 ### USAGE
 
-Ruby is a powerful and concise  object-oriented scripting language.  Scripts are normally run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments.  However, scripts installed as gems do not need the `ruby` command to be typed explicitly.  For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory.  Note that on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
-
-All scripts in this distribution are self documenting if run with a `--help`, `-h`, or `-?` option.  Quick descriptions follow.
-
-### GENERAL FILE MANIPULATION SCRIPTS:
-  - `blank2csv.rb` —
-converts whitespace-delimited files to Comma Separated Values (CSV) files
-  - `cat.rb` —
-Unix “cat” program work-alike for Windows—for viewing or creating text files or concatenating multiple files, based on stdio
-  - `convert_line_endings.rb` —
-converts text files (including CSV files) to your current operating system environment (e.g., DOS to Unix or Unix to DOS)
-  - `csv2blank.rb` —
-converts CSV files to whitespace-delimited files
-
-### DATA FARMING RUBY SCRIPTS:
-  - `batchrunner.rb` —
-run a model interactively with replication
-  - `rundesign_general.rb` —
-run control to replicate a designed experiment with a model
-  - `pool_files.rb` —
-pool columns from separate files into a new file, useful for combining design file & output files or multiple output files
-  - `scaled_fde.rb` —
-generate fully orthogonal quadratic designs based on discrete Fourier frequencies, with factor scaling
-  - `scaled_rf_cubed.rb` —
-generate 2-level Resolution V fractional factorial designs and Central Composite Designs (CCDs), with factor scaling
-  - `stack_nolhs.rb` —
-generate designs by reassigning columns from built-in NOLHs, with factor scaling, for up to 100 factors
-  - `stripheaders.rb`, `stripheaderdups.rb` —
-used to remove headers from a file, or duplicate headers within a file, respectively
-  - `augment_design.rb` —
-generates star points to augment a fractional factorial
-  - `cross.rb` —
-creates a combinatorial design by crossing all combinations of any # of individual smaller designs
-  - `mser.rb` —
-uses MSER truncation to remove initial transient effects for time-series output, reports truncated average and number of observations for each run to facilitate construction of a properly weighted confidence interval.
-  - `mser_nolbm.rb` —
-uses MSER truncation to remove initial transient effects for time-series output, then calculates a 95% confidence interval on the remaining data using non-overlapping batch means.
-  - `mser_olbm.rb` —
-uses MSER truncation to remove initial transient effects for time-series output, then calculates a 95% confidence interval on the remaining data using overlapping batch means.
+Ruby is a powerful and concise object-oriented scripting language.  Normally, Ruby scripts are run from a command-line or terminal environment by typing the `ruby` command followed by a script name, often followed by one or more command-line arguments.  However, scripts installed as gems do not need the `ruby` command to be typed explicitly.  For example, typing `stripheaderdups.rb my_file.txt` will invoke the `stripheaderdups.rb` script and apply it to file `my_file.txt` in your current working directory.  Note that on Windows the `.rb` suffix is not needed to run your scripts, e.g., `stripheaderdups my_file.txt` will suffice.
+
+You can run the command `datafarming.rb` at any point after installing to see descriptions of the various data farming utilities included in this distribution.

data/datafarming.gemspec

@@ -1,12 +1,12 @@
 # -*- ruby -*-
-_VERSION = "1.3.0"
+_VERSION = "1.4.0"
 
 Gem::Specification.new do |s|
   s.name = "datafarming"
   s.version = _VERSION
-  s.date = "2018-08-17"
+  s.date = "2021-01-19"
   s.summary = "Useful scripts for data farming."
-  s.homepage = "https://gitlab.nps.edu/pjsanche/datafarmingrubyscripts.git"
+  s.homepage = "https://bitbucket.org/paul_j_sanchez/datafarmingrubyscripts"
   s.email = "pjs@alum.mit.edu"
   s.description = "Ruby scripts for data farming, including pre- and post-processing, design generation and scaling, and run control."
   s.author = "Paul J Sanchez"
@@ -19,6 +19,6 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'fwt', '~> 1.2'
   s.add_runtime_dependency 'colorize', '~> 0.8'
   s.add_runtime_dependency 'quickstats', '~> 2'
-  s.required_ruby_version = '>= 2.3'
+  s.required_ruby_version = '>= 2.5'
   s.license = 'MIT'
 end

data/exe/datafarming.rb

@@ -0,0 +1,80 @@
+#!/usr/bin/env ruby
+
+lines = [
+'',
+'  #### COMMAND-LINE TOOLS',
+'',
+'  Ruby is a powerful and concise  object-oriented scripting language.  Scripts',
+'  are normally run from a command-line or terminal environment by typing the',
+'  ruby  command followed by a script name, often followed by one or more',
+'  command-line arguments.  However, scripts installed as gems do not need the',
+'  ruby  command to be typed explicitly.  For example, typing',
+'  stripheaderdups.rb my_file.txt  will invoke the  stripheaderdups.rb  script',
+'  and apply it to file  my_file.txt  in your current working directory.  Note',
+'  that on Windows the  .rb  suffix is not needed to run your scripts, e.g.,',
+'  stripheaderdups my_file.txt  will suffice.',
+'',
+'  All scripts in this distribution are self documenting if run with a --help,',
+'  -h, or -? option.  Quick descriptions follow.',
+'',
+'  #### DATA MANIPULATION',
+'',
+'  •  blank2csv.rb  —',
+'  convert whitespace-delimited files to Comma Separated Values (CSV) files',
+'  •  cat.rb  —',
+'  Unix  cat  program work-alike for Windows—for viewing or creating text files',
+'  or concatenating multiple files, based on stdio',
+'  •  convert_line_endings.rb  —',
+'  convert text files (including CSV files) to your current operating system',
+'  environment (e.g., DOS to Unix or Unix to DOS)',
+'  •  csv2blank.rb  —',
+'  convert CSV files to whitespace-delimited files',
+'  •  pool_files.rb  —',
+'  pool columns from separate files into a new file, useful for combining',
+'  design file & output files or multiple output files',
+'  •  stripheaders.rb ,  stripheaderdups.rb  —',
+'  remove column headers from a file, or duplicate headers within a file,',
+'  respectively',
+'',
+'  #### DESIGN GENERATORS',
+'',
+'  •  augment_design.rb  —',
+'  generate star points to augment an already existing factorial design',
+'  •  cross.rb  —',
+'  create a combinatorial design by crossing all combinations of any # of',
+'  individual smaller designs',
+'  •  scaled_fde.rb  —',
+'  generate fully orthogonal quadratic designs based on discrete Fourier',
+'  frequencies, with factor scaling',
+'  •  scaled_rf_cubed.rb  —',
+'  generate 2-level Resolution V fractional factorial designs and Central',
+'  Composite Designs (CCDs), with factor scaling',
+'  •  stack_nolhs.rb  —',
+'  generate designs by reassigning columns from pre-tabled NOLHs, with factor',
+'  scaling, for up to 100 factors',
+'',
+'  #### RUN CONTROL',
+'',
+'  •  batchrunner.rb  —',
+'  run a model interactively with replication',
+'  •  rundesign_general.rb  —',
+'  run control to replicate a designed experiment applied to a model',
+'',
+'  #### CONFIDENCE INTERVALS FOR TIME SERIES',
+'',
+'  •  mser.rb  —',
+'  use MSER truncation to remove initial transient effects for time-series',
+'  output, reporting truncated average and number of observations for each run',
+'  to facilitate construction of a properly weighted confidence interval.',
+'  •  mser_nolbm.rb  —',
+'  use MSER truncation to remove initial transient effects for time-series',
+'  output, then calculate a 95% confidence interval on the remaining data using',
+'  non-overlapping batch means.',
+'  •  mser_olbm.rb  —',
+'  use MSER truncation to remove initial transient effects for time-series',
+'  output, then calculate a 95% confidence interval on the remaining data using',
+'  overlapping batch means.',
+''
+]
+
+lines.each { |line| puts line }

data/exe/scaled_rf_cubed.rb

@@ -3,8 +3,8 @@
 require 'colorize'
 String.disable_colorization false
 
-require 'datafarming/error_handling'
-require 'datafarming/factorial_generator'
+require_relative '../lib/datafarming/error_handling'
+require_relative '../lib/datafarming/factorial_generator'
 
 def print_descriptor
   description = [

data/lib/datafarming/factorial_generator.rb

@@ -13,43 +13,37 @@ require 'fwt'
 #     corresponds to a factor and each row is a design point.
 #
 # Author:: Paul J Sanchez (mailto:pjs@alum.mit.edu)
-# Copyright:: Copyright (c) 2018 Paul J Sanchez
+# Copyright:: Copyright (c) 2020 Paul J Sanchez
 # License:: MIT
 #
-def make_design(number_of_factors)
-  index = [1, 2, 4, 8, 15, 16, 32, 51, 64, 85, 106, 128, 150, 171,
-           219, 237, 247, 256, 279, 297, 455, 512, 537, 557, 594, 643, 803,
-           863, 998, 1024, 1051, 1070, 1112, 1169, 1333, 1345, 1620, 1866,
-           2048, 2076, 2085, 2185, 2372, 2456, 2618, 2800, 2873, 3127, 3284,
-           3483, 3557, 3763, 4096, 4125, 4135, 4174, 4435, 4459, 4469, 4497,
-           4752, 5255, 5732, 5804, 5915, 6100, 6369, 6907, 7069, 8192, 8263,
-           8351, 8422, 8458, 8571, 8750, 8858, 9124, 9314, 9500, 10_026,
-           10_455, 10_556, 11_778, 11_885, 11_984, 13_548, 14_007, 14_514,
-           14_965, 15_125, 15_554, 16_384, 16_457, 16_517, 16_609, 16_771,
-           16_853, 17_022, 17_453, 17_891, 18_073, 18_562, 18_980, 19_030,
-           19_932, 20_075, 20_745, 21_544, 22_633, 23_200, 24_167, 25_700,
-           26_360, 26_591, 26_776, 28_443, 28_905, 29_577, 32_705
-         ]
-  power = [1, 2, 4, 8, 16, 16, 32, 64, 64, 128, 128, 128, 256, 256,
-           256, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024,
-           1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048,
-           2048, 2048, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096,
-           4096, 4096, 4096, 4096, 4096, 8192, 8192, 8192, 8192, 8192, 8192,
-           8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192,
-           16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384,
-           16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384,
-           16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 32_768,
-           32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
-           32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
-           32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
-           32_768, 32_768, 32_768
-         ]
+INDEX = [1, 2, 4, 8, 15, 16, 32, 51, 64, 85, 106, 128, 150, 171,
+         219, 237, 247, 256, 279, 297, 455, 512, 537, 557, 594, 643, 803,
+         863, 998, 1024, 1051, 1070, 1112, 1169, 1333, 1345, 1620, 1866,
+         2048, 2076, 2085, 2185, 2372, 2456, 2618, 2800, 2873, 3127, 3284,
+         3483, 3557, 3763, 4096, 4125, 4135, 4174, 4435, 4459, 4469, 4497,
+         4752, 5255, 5732, 5804, 5915, 6100, 6369, 6907, 7069, 8192, 8263,
+         8351, 8422, 8458, 8571, 8750, 8858, 9124, 9314, 9500, 10_026,
+         10_455, 10_556, 11_778, 11_885, 11_984, 13_548, 14_007, 14_514,
+         14_965, 15_125, 15_554, 16_384, 16_457, 16_517, 16_609, 16_771,
+         16_853, 17_022, 17_453, 17_891, 18_073, 18_562, 18_980, 19_030,
+         19_932, 20_075, 20_745, 21_544, 22_633, 23_200, 24_167, 25_700,
+         26_360, 26_591, 26_776, 28_443, 28_905, 29_577, 32_705]
+POWER = [1, 2, 4, 8, 16, 16, 32, 64, 64, 128, 128, 128, 256, 256,
+         256, 256, 256, 256, 512, 512, 512, 512, 1024, 1024, 1024, 1024,
+         1024, 1024, 1024, 1024, 2048, 2048, 2048, 2048, 2048, 2048, 2048,
+         2048, 2048, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096, 4096,
+         4096, 4096, 4096, 4096, 4096, 8192, 8192, 8192, 8192, 8192, 8192,
+         8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192, 8192,
+         16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384,
+         16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384,
+         16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 16_384, 32_768,
+         32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
+         32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
+         32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768, 32_768,
+         32_768, 32_768, 32_768]
 
-  design = Array.new(number_of_factors)
-  design.each_index do |i|
-    design[i] = Array.new(power[number_of_factors], 0)
-    design[i][index[i]] = 1
-    design[i].hadamard
-  end
-  design.transpose
+def make_design(number_of_factors)
+  Array.new(number_of_factors) do |i|
+    Array.new(POWER[number_of_factors], 0).tap { |a| a[INDEX[i]] = 1 }.hadamard
+  end.transpose
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: datafarming
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.0
 platform: ruby
 authors:
 - Paul J Sanchez
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2018-08-17 00:00:00.000000000 Z
+date: 2021-01-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fwt
@@ -63,6 +63,7 @@ executables:
 - convert_line_endings.rb
 - cross.rb
 - csv2blank.rb
+- datafarming.rb
 - moving_average.rb
 - mser.rb
 - mser_nolbm.rb
@@ -77,6 +78,8 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- DESCRIPTIONS.html
+- DESCRIPTIONS.md
 - LICENSE.md
 - README.md
 - datafarming.gemspec
@@ -87,6 +90,7 @@ files:
 - exe/convert_line_endings.rb
 - exe/cross.rb
 - exe/csv2blank.rb
+- exe/datafarming.rb
 - exe/moving_average.rb
 - exe/mser.rb
 - exe/mser_nolbm.rb
@@ -104,11 +108,11 @@ files:
 - lib/datafarming/freq_sets.rb
 - lib/datafarming/moving_average.rb
 - lib/datafarming/nolh_designs.rb
-homepage: https://gitlab.nps.edu/pjsanche/datafarmingrubyscripts.git
+homepage: https://bitbucket.org/paul_j_sanchez/datafarmingrubyscripts
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -116,16 +120,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.3'
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.7
-signing_key: 
+rubygems_version: 3.2.3
+signing_key:
 specification_version: 4
 summary: Useful scripts for data farming.
 test_files: []