advance 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b554bb49d7d5003e2fc5031754afe1a4c08f7741
-  data.tar.gz: 5231a634d47aca8a8d15a578b0b4e0d2f7b17fd9
+  metadata.gz: 6f3f8b730fb4d9ecaeabd457c7157b2b8d44e5bb
+  data.tar.gz: 59bb0f09290486935b6ac303d7a8bdbd93d3e985
 SHA512:
-  metadata.gz: 02c498e7694e15b3b59cb2ac69582b0ddefab93de3b2863cc838e4b700dcc08ec873801268cbcb0fc47f308d705ce31f4d7db03a6fc11c9fa5d4d8791f590757
-  data.tar.gz: 3526bb8ed0671d1cc43a1aae89b91ecaaa541d5143526acec587cdcdef9a1e605483f068449f7d279080197dee6c5ae9bf5e32d87dfb40d9f184430d31044e87
+  metadata.gz: 8eeaa8b0c713aa37d2041789a7a3f360ccee41ed6e7ae001063539d9d00e9da6b1fbcd050a16d0aad8a8b51a6f7d3f22c6a2938d2f3eee4b5bbf1a898a233f72
+  data.tar.gz: 5a95d341914d9df76f0b108e6264ff6e21a93f1b7fcb39cc5472074779a56c23c673db36ba15f3732ccfc3f0b0156fba536b15130678224f6eb89ce07cbe6ad7

data/.gitignore

@@ -7,3 +7,5 @@
 /spec/reports/
 /tmp/
 .idea
+.ruby-gemset
+.ruby-version

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    advance (0.1.1)
+    advance (0.1.3)
       team_effort
 
 GEM

data/README.md

@@ -16,11 +16,19 @@ are preserved in directories prefixed with "tmp_". This isolates incomplete
 step data and ensures that the step is re-processed when the problem is 
 resolved.
 
-Advance scripts are easy to understand. They are ruby scripts, 
-composed of a series of function calls that invoke your scripts
-or commands to transform your data. Each step is composed of a
-step processing type function, followed by a 
-slug for the step, followed by the command or script. For example:
+Your project utilizing Advance contains a primary ruby script
+that imports Advance and includes your data transformation steps,
+which we will call "your Advance script."
+Each step describes a command to be run on your data. These commands can be
+one of the prepackaged Advance scripts, unix commands (like split, cut,
+etc), or scripts/commands that you create in whatever language is 
+convenient for you. Advance invokes these scripts one by one much like
+you would at the command line. Advance logs the exact command that is invoked
+so that you can run it yourself to check the output manually and to 
+debug failures.
+
+Advance steps are composed of a step processing type function, followed 
+by a slug for the step, followed by the command or script. For example:
 
 ```ruby
 single :unzip_7z_raw_data_file, "7z x {previous_file}"
@@ -31,7 +39,7 @@ multi :add_local_time, "cat {file_path} | add_local_time.rb timestamp local_time
 
 The step processing functions are `single` and `multi`. `Single` applies the command
 to the last output, which should be a single file. `Multi` speeds processing of multiple
-files by doing working in parallel (via the [TeamEffort gem][1]).
+files by doing work in parallel (via the [TeamEffort gem][1]).
 
 [1]: https://rubygems.org/gems/team_effort
 
@@ -49,18 +57,15 @@ to your script:
 
     $ gem install advance
     
- * install [bundler][3], and add this ruby snippet to the beginning of your script:
+ * install [bundler][3], and add Advance to your `Gemfile`:
  
 [3]: https://rubygems.org/gems/bundler
  
 ```ruby
-    #!/usr/bin/env ruby
-    require "bundler/inline"
-    
-    gemfile do
-      source "https://rubygems.org"
-      gem "advance"
-    end
+source "https://rubygems.org"
+
+gem "advance"
+# other gems...
 ```
 
 ## Usage
@@ -86,10 +91,10 @@ Steps have 3 components:
 
 Advance adds the bin dir of the Advance gem to PATH, so that you can invoke the 
 supporting advance scripts in your pipeline without specifying the full path
-of the script. Advance also adds the path of your script to PATH so that you can 
-invoke scripts in the same directory as your main script without specifying 
-the full path of the script. Of course, you can invoke any script if the path
-to the script is fully specified or the path is already on PATH.
+of the script. Advance also adds the path of _your Advance script_ to PATH so 
+that you can invoke scripts in the same directory as your main script without 
+specifying the full path of the script. Of course, you can invoke any script 
+if the path to the script is fully specified or the path is already on PATH.
 
 **Specifying Script Input and Output**
 
@@ -97,29 +102,26 @@ Since your command is transforming data, you need a way to specify the input
 file or directory and the output file name. Advance provides a few tokens 
 that can be inserted in the command string for this purpose:
 
- * **{previous_file}** indicates the output file from the previous step when
+ * **`{previous_file}`** indicates the output file from the previous step when
    the output of the previous step was a single output file. It is also used
    to indicate the first file to be used and it finds that file in the current 
    working dir.
- * **{file_path}** indicates an output file from the previous step when the
+ * **`{file_path}`** indicates an output file from the previous step when the
    previous step generated multiple output files and the current step is a 
    `multi` step.
- * **{file}** indicates an output file name, which is the basename from 
-   {file_path}. Commands often process multiple files from previous steps, 
+ * **`{file}`** indicates an output file name, which is the basename from 
+   `{file_path}`. Commands often process multiple files from previous steps, 
    generating multiple output files. Those output files are placed in the
    step directory.
- * **{previous_dir}** indicates the directory a previous step.
+ * **`{previous_dir}`** indicates the directory of the previous step.
  
 **Example Script**
 
 ```ruby
 #!/usr/bin/env ruby
-require "bundler/inline"
 
-gemfile do
-  source "https://rubygems.org"
-  gem "advance"
-end
+require "advance"
+include Advance
 
 ensure_bin_on_path # ensures the directory for this script is on
                    # the path so that related scripts can be referenced
@@ -137,14 +139,28 @@ When running your pipeline, it is helpful to have a directory with the single, i
 1. Move to your data directory with your single initial file.
 2. invoke your script from there.
 
-## Development
+## Questions / Answers
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+ * Q: **My script fails with `undefined local variable or method 'label' for main:Object`**  
 
+   A: This indicates a ruby script is running that does not have access to a ruby gem.
+      First, make sure your script is using the expected ruby by adding to the beginning of your
+      script `puts RUBY_VERSION`. Make sure the gem is installed by listing installed gems with 
+      `$ gem list`. Finally, check that the script requires the library with `require 'my-library'` 
+   
 ## Contributing
 
+We ♥️ contributions!
+
+Found a bug? Ideally submit a pull request. And if that's not possible, make a bug report.
+
+Did you create a data transformation script? Please consider adding it to the 
+script collection in Advance by submitting a pull request.
+
+Do you find the Advance documentation lacking? Please help us improve it. 
+
+Can you translate the Advance documentation to your language? 
+
 Bug reports and pull requests are welcome on GitHub at https://github.com/doctorjane/advance.
 
 ## License

data/bin/concat_csv_nh.rb

@@ -0,0 +1,18 @@
+#!/usr/bin/env ruby
+
+require "find"
+
+def do_cmd(cmd)
+  system cmd
+  status = $?
+  raise "'#{cmd}' failed with #{status}" if !status.success?
+end
+
+files_dir_path = ARGV[0]
+output_file = ARGV[1]
+files = Find.find(files_dir_path).reject { |p| FileTest.directory?(p) || File.basename(p) == "log" }
+
+files.each_slice(20) do |files_to_concat|
+  file_list = files_to_concat.join(' ')
+  do_cmd "gcat #{file_list} >> #{output_file}"
+end

data/bin/csv_select_nh.rb

@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+require 'csv'
+# $stderr.puts "#{__FILE__}:#{__LINE__}"
+
+test_proc = eval "lambda {|row| #{ARGV.shift}}"
+
+input = CSV.new(ARGF, :headers => true, :return_headers => true, :converters => :numeric)
+output = CSV.new($stdout, :headers => true, :write_headers => true)
+
+input.each.with_index do |row, index|
+  # $stderr.puts "#{index}: >>#{row.to_s.chomp}<<"
+  if row.header_row?
+    output << row
+    next
+  end
+
+  if test_proc.call(row)
+    output << row
+    next
+  end
+end

data/lib/advance.rb

@@ -1,6 +1,6 @@
 require "advance/version"
-require 'open3'
-
+require "find"
+require "open3"
 require "team_effort"
 
 module Advance
@@ -48,9 +48,7 @@ module Advance
   end
 
   def previous_file_path
-    dir_entries = Dir.glob(File.join(previous_dir_path, "*"))
-    dir_entries_clean = dir_entries.reject { |f| File.directory?(f) || f =~ %r{^\.\.?|log} }
-    dir_entries_clean.first
+    Find.find(previous_dir_path).reject { |p| FileTest.directory?(p) || File.basename(p) == "log" }.first
   end
 
   def single(label, command)
@@ -61,6 +59,9 @@ module Advance
       if command =~ /\{previous_dir\}/
         command.gsub!("{previous_dir}", previous_dir_path)
       end
+      if command =~ /\{file\}/
+        command.gsub!("{file}", File.basename(previous_file_path))
+      end
       do_command command
     end
   end

data/lib/advance/version.rb

@@ -1,3 +1,3 @@
 module Advance
-  VERSION = "0.1.1"
+  VERSION = "0.1.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: advance
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - janemacfarlane
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-01-10 00:00:00.000000000 Z
+date: 2019-01-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: team_effort
@@ -76,8 +76,10 @@ email:
 - jfmacfarlane@lbl.gov
 executables:
 - concat_csv.rb
+- concat_csv_nh.rb
 - console
 - csv_select.rb
+- csv_select_nh.rb
 - csv_split_on_change.rb
 - setup
 - split_csv.rb
@@ -93,8 +95,10 @@ files:
 - Rakefile
 - advance.gemspec
 - bin/concat_csv.rb
+- bin/concat_csv_nh.rb
 - bin/console
 - bin/csv_select.rb
+- bin/csv_select_nh.rb
 - bin/csv_split_on_change.rb
 - bin/setup
 - bin/split_csv.rb