batch_experiment 1.0.0 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e6db331d43fbfee1847a2385bc1c90616433451b
-  data.tar.gz: 0b5b00e526fe8ff5d78d32991ee79e2655d24c5c
+  metadata.gz: b01a82b2fe96315d3ce7f39ee99e40719a56ee7e
+  data.tar.gz: 94f0f3f697a48958353f66f521e6b462806ee044
 SHA512:
-  metadata.gz: 41b4f4c3866c68c86677630466e88eb574ca0ccd417a55c94eaa378d55c368673f07c5bd6dfb4018ac69d1519c730e8d6b41cc8d0c9e6ac58e51e12684435322
-  data.tar.gz: babb03ca7415332843d231e21e30a9ecea32f3da5792fc48ccc5023c4c31eaf374c48091869e9a42c45e83b01f8d6647ea70e384d14fdecd511cdac9fc580be3
+  metadata.gz: 33806f836956132aa78b8efe7a2cb5972e353159ad848e156f7bde542d44d6d5e8d6fce85cdbf23e70308f367dc6bf81ba44c1645d22f65437cc4cedba2b2af7
+  data.tar.gz: b7b17a54beca0742a6679dccb3db6e45a49aeb1389b99421c95e5414f222d85645d985f59a42e888833e3957b4c426e99a552093caf3be795d7806f5fb76390b

data/README.md

@@ -0,0 +1,51 @@
+# batch_experiment
+
+Things you could want to do, and this tool does for you:
+* You want to run a batch of sh commands, but only one of those per core/cpu.
+* You want to maximize the core use, the moment a core/cpu becomes free from one of your commands, you want the next command to take its place.
+* You want the output of those commands to be saved by command/file (you want to have a file with what exactly the command outputted).
+* You want to specify timeouts for those commands to be killed.
+* You want the power to resume the batch from an interruption (i.e. system crashes, or energy goes down) without losing any work.
+
+What conditions you need to use this tool:
+* You use linux.
+* You have installed: sh (the shell); ruby (obvioulsy); time (NOT the bash/sh internal command, but the [package one](https://www.archlinux.org/packages/extra/x86_64/time/)); timeout (from coreutils); taskset (from [util-linux](https://www.archlinux.org/packages/core/x86_64/util-linux/)).
+
+What is not needed:
+* To know how to program in ruby. Only taking less than 5 minutes to learn some basic syntax will suffice to run commands on multiple cores and save the results to files. However, there's an exception, if you want not only to execute the commands but want to extract and group some information from their output to a CVS, you will need to tell ruby how to do the extracting part.
+
+## How to use it
+
+You will need to create a ruby script (copy, past and adapt one of the provided examples), give it execution permissions ("chmod +x script.rb"), and execute it ("./script.rb"). It's good to remember that this will probably flood the folder from were you called the command with files containing the output of the commands. Also good to remember that the commands will be called from the folder where you called the script, so they probably will expect that any relative paths/filenames given to them to be relative to the current folder.
+
+## Examples
+
+After installing the gem, you will have a examples folder (/home/YOUR_USER/.gem/ruby/RUBY_VERSION/gems/batch_experiment-GEM_VERSION/examples). The sample_batch.rb gives you a good ideia of how to use #batch (no csv creation). The example_batch.rb together with the lib/batch_experiment/sample_extractors.rb gives a good ideia of how to use #experiment with multiple commands and extractors (csv creation).
+
+```ruby
+#!/bin/ruby
+
+require 'batch_experiment'
+
+commands = [
+  'sleep 2 && echo orange',
+  'sleep 4 && echo banana',
+  'sleep 100 && echo "never gonna happen"',
+]
+
+conf = {
+  # IDs of the CPU cores that can be used for executing tests.
+  cpus_available: [0, 1],
+  # Maximum number of seconds that a command can run. After this a kill command
+  # (TERM signal) will be issued.
+  timeout: 5,
+  # Maximum number of seconds that a command can run after a kill command was
+  # issued. After this a kill -9 command (KILL signal) will be issued.
+  post_timeout: 1,
+}
+
+BatchExperiment::batch(commands, conf)
+```
+
+This code was born in [this repository](https://github.com/henriquebecker91/masters/tree/master/codes/rb/batch_experiment).
+

data/examples/debug_batch.rb

@@ -0,0 +1,15 @@
+#!/bin/ruby
+
+require_relative '../lib/batch_experiment.rb'
+
+commands = []
+10000.times { | n | commands << "sleep 1 && echo #{n}" }
+
+conf = {
+  cpus_available: [1, 2, 3],
+  timeout: 5,
+  post_timeout: 1,
+}
+
+BatchExperiment::batch(commands, conf)
+

data/lib/batch_experiment.rb

@@ -23,12 +23,16 @@ module BatchExperiment
   # terminated commands on comms_executed.
   def self.update_finished(free_cpus, comms_running, comms_executed)
     comms_running.delete_if do | job |
-      if job[:proc].exited?
+      # Don't call '#exited?' twice, store value at variable. If you call
+      # it twice it's possible to remove it from the list of running commands
+      # without freeing a cpu, what will end locking all cpus forever.
+      exited = job[:proc].exited?
+      if exited
         free_cpus.push(job[:cpu])
         File.delete(job[:lockfname])
         comms_executed << job[:command]
       end
-      job[:proc].exited? # bool returned to delete_if
+      exited # bool returned to delete_if
     end
   end
 
@@ -42,7 +46,8 @@ module BatchExperiment
   # After the command ends its execution this file is removed. If the command
   # ends its execution by means of a timeout the file is also removed. The file
   # only remains if the batch procedure is interrupted (script was killed,
-  # or system crashed).
+  # or system crashed). This '.unfinished' file will contain the process pid,
+  # if the corresponding process started with success.
   #
   # @param commands [Array<String>] The shell commands.
   # @param conf [Hash] The configurations, as follows:
@@ -53,8 +58,8 @@ module BatchExperiment
   #   parameter. Is the same for all the commands.
   #   -- time_fmt [String] A string in the time (external command) format. See
   #   http://linux.die.net/man/1/time. Default: 'ext_time: %e\next_mem: %M\n'.
-  #   busy_loop_sleep [Number] How many seconds to wait before checking if a
-  #   command ended execution. This is max time a cpu will be vacant between
+  #   -- busy_loop_sleep [Number] How many seconds to wait before checking if
+  #   a command ended execution. This is max time a cpu will be vacant between
   #   two commands. Default: 0.1.
   #   -- post_timeout [Number] A command isn't guaranteed to end after
   #   receiving a TERM signal. If the command hasn't stopped, waits
@@ -168,6 +173,9 @@ module BatchExperiment
         command: command
       }
 
+      # The lock file now stores the process pid for debug reasons.
+      File.open(lockfname, 'w') { | f | f.write cproc.pid }
+
       puts "command assigned to cpu#{cpu}"
       STDOUT.flush
     end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: batch_experiment
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
 platform: ruby
 authors:
 - Henrique Becker
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-03-23 00:00:00.000000000 Z
+date: 2016-03-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: childprocess
@@ -30,7 +30,9 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- README.md
 - examples/bible.txt
+- examples/debug_batch.rb
 - examples/example_batch.rb
 - examples/sample_batch.rb
 - examples/taoteching.txt