once-only 0.0.1 → 0.2.1

data/README.md

@@ -7,13 +7,24 @@ Relax with PBS!
 * Computations only happen once
 * A completed job does not get submitted again to PBS
 * A job already in the queue does not get submitted again to PBS
+* A completed job in the PBS queue does not run again
+* Guarantee independently executed jobs
+* Do not worry about submitting serial jobs
+
+and coming
+
+* Automatically use a scratch disk (nyi)
+* Garbage collect jobs (nyi)
 
 Once-only makes a program or script run only *once*, provided the inputs don't
 change (in a functional style!). This is very useful when running a range of
 jobs on a compute cluster or GRID. It may even be useful in the context of
 webservices.  Once-only makes it relaxed to run many jobs on compute clusters!
 A mistake, interruption, or even a parameter tweak, does not mean everything
-has to be run again.
+has to be run again. When running jobs serially you can just batch
+submit them after getting the first results. Any missed jobs can be
+run later again. This way you can get better utilisation of the
+cluster.
 
 Instead of running a tool or script directly, such as
 
@@ -27,17 +38,18 @@ Prepend once-only
   once-only bowtie -t e_coli reads/e_coli_1000.fq e_coli.map
 ```
 
-and once-only will parse the command line for existing files and run a checksum
-on them (here the binary executable 'bowtie' and data files
+and once-only will parse the command line for existing files and run a
+checksum on them (here the binary executable 'bowtie' and data files
 reads/e_coli_1000.fq and e_coli.map).  This checksum, in fact an MD5
-cryptographic hash, or optionally [pfff](https://github.com/pfff/pfff) for
-large files, is a unique identifier (aka fingerprint) and saved in a file in the running
-directory.  When the checksum file does not exist in the directory the command
-'bowtie -t e_coli reads/e_coli_1000.fq e_coli.map' is executed.
+cryptographic hash, or optionally [pfff](https://github.com/pfff/pfff)
+for large files, is a unique identifier (aka fingerprint) and saved in
+a file in the running directory.  When the checksum file does not
+exist in the directory the command 'bowtie -t e_coli
+reads/e_coli_1000.fq e_coli.map' is executed.
 
-When the file already exists execution is skipped. In other words, the checksum
-file guarantees the program is only run once with the same inputs. Really
-simple! 
+When the file already exists execution is skipped. In other words, the
+checksum file guarantees the program is only run once with the same
+inputs. Really simple! 
 
 In combination with PBS this could be
 
@@ -94,13 +106,18 @@ With Ruby 1.9 or later on your system you can run
 gem install once-only
 ```
 
+It is also easy to check out the git repository, as once-only has no
+library dependencies.
+
 ### Dependencies
 
 'md5sum' is used for calculating MD5 hash values.
 
-'pfff' is optional and used for calculating pfff hash values on very large files.
+'pfff' is optional and used for calculating pfff hash values on very
+large files (nyi).
 
-When you are using PBS, once-only requires the 'qsub' and 'qstat' commands.
+When you are using the --pbs option, once-only will use the 'qsub' and
+'qstat' commands.
 
 ## Usage (command line)
 
@@ -112,7 +129,7 @@ once-only --help
 
 Useful switches can be -v (verbose) and -q (quiet).
 
-If you want to skip scanning the executable file (useful in heterogenous environments, 
+To skip scanning the executable file (useful in heterogenous environments, 
 such as the GRID) use the --skip-exe switch:
 
 ```sh
@@ -141,8 +158,11 @@ once-only --skip-exe --skip-glob 'out*' --skip-glob '*.ph' muscle -in aa.fa -out
 
 For a full range of glob patterns, see this [page](http://ruby.about.com/od/beginningruby/a/dir2.htm).
 
-Sometimes you want to include input files that are not on the command line for generating the hash. Maybe some default input file name is being picked up, or it is defined in a 
-configuration file. In that case use the --include/--in options.
+Sometimes you want to include input files that are not on the command
+line for generating the hash. Maybe some default input file name is
+being picked up, or it is defined in a configuration file. In that
+case use the --include/--in options. Another feature is that if an -in
+file does not exist once-only does not run.
 
 Another once-only command line option is to change directory before executing the script
 
@@ -152,9 +172,30 @@ once-only -d run001 --skip-regex 'out|\.ph$' muscle -in aa.fa -out out-alignment
 
 which is useful with PBS and in scripted environments.
 
+### Pipes and redirection
+
+Once-only supports pipes and redirection by stringifying a command on
+STDIN:
+
+```sh
+echo "/bin/cat README.md > tmp.out" | once-only --skip tmp.out
+```
+
+With PBS the tricky thing here is using more quotes for spacing. At this point it is
+recommended to escape internal quotes, and avoid using single quotes, e.g.
+
+```sh
+echo "/bin/cat \\\"README.md Version 2\\\" > tmp.out" | once-only --pbs --skip tmp.out
+```
+
 ### PBS
 
-Once-only has PBS support built-in. It only uses the 'qsub' and 'qstat' commands.
+Once-only has PBS support built-in. When a job is in the queue, it
+won't get submitted again. When a job has completed, it won't run
+again. Thiss achieved by using once-only before submitting the job
+to the queue, and right before running the job.
+
+Once-only with PBS only uses the 'qsub' and 'qstat' commands. 
 
 Basically use the --pbs option:
 
@@ -176,6 +217,31 @@ once-only --pbs --skip-exe /bin/cat ~/.bashrc
 
 so once-only won't check the file /bin/cat.
 
+### Guarantee independent jobs with --copy
+
+Because once-only 'knows' the input files we can copy them to a unique
+place before execution. By using the --copy switch a new directory is
+created in the run directory using the hash value of the process.
+Input files are copied and the job is run inside that directory. When
+the job is finished the output file(s) are copied back to the working
+directory. Example
+
+```sh
+once-only --copy /bin/cat ~/.bashrc
+```
+
+Note that files that come with a path will be stripped of their path
+before execution. When files are very large you may want to consider
+the --scratch option.
+
+### Use the scratch disk with --scratch (nyi)
+
+watch this page
+
+### Garbage collect jobs (nyi)
+
+watch this page
+
 ## Project home page
 
 Information on the source tree, documentation, examples, issues and

data/VERSION

@@ -1 +1 @@
-0.0.1
+0.2.1

data/bin/once-only

@@ -11,20 +11,35 @@ once-only runs a command once only when inputs don't change!
 Usage:
 
    -d path             change to directory before executing
+   --copy              copy files to hash dir first
    --pbs [opts]        convert to PBS command with optional options
    --skip|--out file   skip making a checksum of the named file (multiple allowed)
    --skip-exe          skip making a checksum of the executable command/script
    --skip-cli          skip making a checksum of full command line
    --skip-regex regex  skip making checksumes of filenames that match the regex (multiple allowed)
    --skip-glob regex   skip making checksumes of filenames that match the glob (multiple allowed)
-   --include|--in file include input filename for making the checksumes
+   --include|--in file include input filename for making the checksums (file should exist)
    -v                  increase verbosity
    -q                  run quietly
    --debug             give debug information
    --dry-run           do not execute command
    --force             force execute command
 
-See the README for examples
+Examples:
+
+   Basic use
+
+     once-only /bin/cat README.md
+
+   With PBS
+
+     once-only --pbs /bin/cat README.md
+
+   Using redirects
+
+     echo "/bin/cat README.md > tmp.out" | ./bin/once-only --skip tmp.out
+
+See the README for more examples!
 
 EOB
 
@@ -45,8 +60,9 @@ if ARGV.size == 0
   exit 1
 end
 
-def exit_error errval = 1
-  $stderr.print "\nonce-only returned error #{errval}\n"
+def exit_error errval = 1, msg = nil
+  $stderr.print msg if msg
+  $stderr.print "\n**ERROR** once-only returned error #{errval}\n"
   exit errval
 end
 
@@ -54,20 +70,28 @@ def parse_args(args)
   options = { :skip => [], :skip_regex => [], :skip_glob => [], :include => [] }
 
   consume = lambda { |args|
+    if not args[0]
+      # check stdin
+      cmd = $stdin.gets
+      exit_error(1,"Empty command on STDIN") if cmd == nil
+      $stderr.print "Command (STDIN): ",cmd,"\n"
+      options[:stdin] = true
+      return cmd.split(/\s/)
+    end
     return args if File.exist?(args[0]) # reached the executable command
     case args[0]
       when '-d'
         options[:dir] = File.expand_path(args[1])
         consume.call(args[2..-1])
       when '--pbs'
-        if args[1] =~ /\s+/ # optional argument
+        if args[1] and args[1] =~ /\s+/ # optional PBS argument with spacing
           options[:pbs] = args[1]
           consume.call(args[2..-1])
         else
           options[:pbs] = "''"
           consume.call(args[1..-1])
         end
-      when '--skip','--out'
+      when '--skip', '--out'
         options[:skip] << args[1]
         consume.call(args[2..-1])
       when '--skip-exe'
@@ -82,9 +106,12 @@ def parse_args(args)
       when '--skip-glob'
         options[:skip_glob] << args[1]
         consume.call(args[2..-1])
-      when '--include','--in'
+      when '--include', '--in', '-in'
         options[:include] << args[1]
         consume.call(args[2..-1])
+      when '--copy' 
+        options[:copy] = true
+        consume.call(args[1..-1])
       when '-h', '--help'
         print USAGE
         exit 1
@@ -104,7 +131,7 @@ def parse_args(args)
         options[:force] = true
         consume.call(args[1..-1])
       else
-        $stderr.print "Can not parse arguments",args
+        $stderr.print "**ERROR** Can not parse arguments",args
         exit_error(1)
       end
   }
@@ -126,6 +153,7 @@ if options[:debug]
   p options   
 end
 
+# --- Rewrite once-only args for PBS
 once_only_args = OnceOnly::Check.drop_pbs_option(once_only_args)
 once_only_args = OnceOnly::Check.drop_dir_option(once_only_args)
 once_only_command = once_only_args.join(' ')
@@ -140,6 +168,8 @@ if options[:dir]
   Dir.chdir options[:dir] 
 end
 
+base_dir = Dir.pwd
+
 executable = args[0]
 args = args[1..-1] if options[:skip_exe]
 
@@ -154,34 +184,53 @@ file_list -= options[:skip]
 
 OnceOnly::Check::check_files_exist(options[:include])
 file_list += options[:include]
+file_list = file_list.uniq
 
 checksums = OnceOnly::Check::calc_file_checksums(file_list)
 checksums.push ['SHA1',command_sha1,command_sorted] if not options[:skip_cli]
 
+# ---- Create filenames
 once_only_filename = OnceOnly::Check::make_once_filename(checksums,File.basename(executable))
 $stderr.print "Check file name ",once_only_filename,"\n" if options[:verbose]
 error_filename = once_only_filename + '.err'
-$stderr.print "Job file exists ",once_only_filename,"!\n" if options[:debug] and File.exist?(once_only_filename)
+tag_filename = once_only_filename + '.run'
+$stderr.print "**STATUS** Job file exists ",once_only_filename,"!\n" if options[:debug] and File.exist?(once_only_filename)
 
+# ---- Create job name
 dirname = File.basename(Dir.pwd).rjust(8,"-") # make sure it is long enough
 
 job_name = (dirname[-5..-1] + once_only_filename.split(/-/).map{|s|s[0..5]}.join).gsub(/[_-]/,'')[0..15]
 $stderr.print "Job name ",job_name,"\n" if options[:verbose]
 
+# ---- Create copy destination
+if options[:copy]
+  copy_dir = base_dir + '/' + File.basename(once_only_filename,".txt") 
+end
+
 if options[:force] or not File.exist?(once_only_filename)
   $stderr.print "Running #{command}\n" if not options[:quiet]
+  OnceOnly::Check::write_file(tag_filename,checksums)
   if options[:pbs]
+    # --- Submit PBS job
+    pbs_command = 'echo \'' +
+      if options[:stdin]
+        'echo "' + command + '"|'+ once_only_command 
+      else
+        once_only_command + ' ' + command 
+      end
+    # --- Add PBS part
+    pbs_command +=  "'|qsub -N #{job_name} "+options[:pbs]
+    pbs_command += ' -d ' + (options[:dir] ? options[:dir] : Dir.pwd) 
+ 
+    $stderr.print("PBS command: ",pbs_command,"\n") if options[:verbose]
+
     # --- Check if job is already queued in PBS
     qstat = `/usr/bin/qstat`
     if qstat =~ /#{job_name}/
-      $stderr.print "Job #{job_name} already in queue!\n"
+      $stderr.print "**STATUS** Job #{job_name} already in queue!\n"
       exit 0
     end
-    # --- Submit PBS job
-    pbs_command = 'echo "' + once_only_command + ' ' + command + "\"|qsub -N #{job_name} "+options[:pbs]+' '
-    pbs_command += '-d ' + (options[:dir] ? options[:dir] : Dir.pwd) 
- 
-    $stderr.print(pbs_command,"\n") if options[:verbose]
+
     if !options[:dry_run]
       if not system(pbs_command) 
         OnceOnly::Check::write_file(error_filename,checksums)
@@ -191,18 +240,54 @@ if options[:force] or not File.exist?(once_only_filename)
   else
     # --- Run on command line
     if !options[:dry_run]
-      if not system(command)
+      success = 
+        if options[:copy]
+          exit_error(1,"Directory #{copy_dir} already exists!") if File.directory?(copy_dir)
+          $stderr.print "Running in directory ",copy_dir
+          Dir.mkdir(copy_dir)
+          # --- copy files
+          # p args
+          # p file_list
+          clist = args.dup
+          file_list.each { | fn | 
+            # copy file
+            res = `cp #{fn} #{copy_dir}` 
+            print res if options[:verbose]
+            # replace command
+            clist = clist.map { |arg| ( fn == arg ? File.basename(arg) : arg ) }
+          }
+          p clist if options[:debug]
+          command_stripped = clist.join(' ')
+          Dir.chdir(copy_dir)
+          system_result = system(command_stripped)
+          if system_result
+            # Copy results back
+            Dir.glob(copy_dir+'/*').each { |outfn|
+              if clist.index(File.basename(outfn)) > 0
+                res = `cp #{outfn} #{base_dir}`
+                print res if options[:verbose]
+              end
+            }
+          end
+          system_result
+        else
+          system(command)
+        end
+      Dir.chdir(base_dir) if options[:copy]
+      if not success
         OnceOnly::Check::write_file(error_filename,checksums)
+        File.unlink(tag_filename) if File.exist?(tag_filename)
         exit_error($?.exitstatus)
       else
         # --- Success!
         File.unlink(error_filename) if File.exist?(error_filename)
         OnceOnly::Check::write_file(once_only_filename,checksums)
+        File.unlink(tag_filename) if File.exist?(tag_filename)
      end
     end
   end
 else
-  $stderr.print "Inputs unchanged. No need to rerun '#{original_commands}'!\n" if not options[:quiet]
+  $stderr.print "**STATUS** Inputs unchanged. No need to rerun '#{original_commands}'!\n" if not options[:quiet]
 end
 
 exit 0  # success!

data/lib/once-only/check.rb

@@ -16,7 +16,7 @@ module OnceOnly
 
     def Check::check_files_exist list
       list.each { |fn| 
-        raise "File #{fn} does not exist!" if not File.exist?(fn)
+        Check::exit_error("File #{fn} does not exist!") if not File.exist?(fn)
       }
     end
 
@@ -95,6 +95,12 @@ protected
       return filename if filename and File.exist?(filename)
       nil
     end
+
+    def Check::exit_error msg, errval=1
+      $stderr.print "\nERROR: ",msg
+      $stderr.print " (once-only returned error #{errval})!\n"
+      exit errval
+    end
   end
 
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: once-only
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.2.1
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-12 00:00:00.000000000Z
+date: 2013-08-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &17265660 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,10 +21,15 @@ dependencies:
         version: 2.8.0
   type: :development
   prerelease: false
-  version_requirements: *17265660
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.8.0
 - !ruby/object:Gem::Dependency
   name: cucumber
-  requirement: &17264920 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +37,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *17264920
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: jeweler
-  requirement: &17264040 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +53,15 @@ dependencies:
         version: 1.8.4
   type: :development
   prerelease: false
-  version_requirements: *17264040
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 1.8.4
 - !ruby/object:Gem::Dependency
   name: bundler
-  requirement: &17263420 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,7 +69,12 @@ dependencies:
         version: 1.0.21
   type: :development
   prerelease: false
-  version_requirements: *17263420
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.0.21
 description: ! "Run programs and scripts once only. Especially\n  useful for PBS and
   GRID computing"
 email: pjotr.public01@thebird.nl
@@ -98,7 +118,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -995396904619724027
+      hash: -4434987383675774210
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -107,7 +127,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.10
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Run commands once only if inputs do not change