once-only 0.2.1 → 0.2.2

data/README.md

@@ -2,14 +2,46 @@
 
 [![Build Status](https://secure.travis-ci.org/pjotrp/once-only.png)](http://travis-ci.org/pjotrp/once-only)
 
-Relax with PBS!
+Relax with PBS! 
+
+No worries about running jobs concurrently from the command line (also
+on multi-core).  Once-only is inspired by the Lisp once-only function,
+which wraps another function and calculates a result only once, based
+on the same inputs. Simply prepend your command with once-only:
+
+When running
+
+```bash
+  once-only -d cluster00073 --pbs --in output.best.dnd ~/opt/paml/bin/codeml ~/paml7-8.ctl
+```
+
+This is what you want to see when same the job was executed before
+
+```bash
+  **STATUS** Job 00073codemla4817 already completed!
+```
+
+This is what you see when a job is running
+
+```bash
+  **STATUS** Job 00073codemla4817 is locked!
+```
+
+With PBS, this is what you want to see when a job is already in the queue
+
+```bash
+  **STATUS** Job 00073codemla4817 already in queue!
+```
+
+Features
 
 * Computations only happen once
-* A completed job does not get submitted again to PBS
+* A completed job does not get submitted again (to PBS)
 * A job already in the queue does not get submitted again to PBS
 * A completed job in the PBS queue does not run again
+* A running job is locked
 * Guarantee independently executed jobs
-* Do not worry about submitting serial jobs
+* Do not worry about submitting serial jobs multiple times
 
 and coming
 
@@ -19,12 +51,18 @@ and coming
 Once-only makes a program or script run only *once*, provided the inputs don't
 change (in a functional style!). This is very useful when running a range of
 jobs on a compute cluster or GRID. It may even be useful in the context of
-webservices.  Once-only makes it relaxed to run many jobs on compute clusters!
-A mistake, interruption, or even a parameter tweak, does not mean everything
-has to be run again. When running jobs serially you can just batch
-submit them after getting the first results. Any missed jobs can be
-run later again. This way you can get better utilisation of the
-cluster.
+webservices.  
+
+Once-only makes it relaxed to run many jobs on compute clusters!  A
+mistake, interruption, or even a parameter tweak, does not mean
+everything has to be run again. When running jobs serially you can
+just batch submit them after getting the first results. Any missed
+jobs can be run later again. This way you can get better utilisation
+of your cores or a cluster. You can even use it as a poor-mans PBS on
+your multi-core machine, or over NFS by firing up scripts
+concurrently.
+
+Examples:
 
 Instead of running a tool or script directly, such as
 
@@ -89,12 +127,6 @@ md5sum on the one-only has file, for example
 grep MD5 bio-table-ce4ceee0d2ee08ef235662c35b8238ad47fed030.txt |awk 'BEGIN { FS = "[ \t\n]+" }{ print $2,"",$3 }'|md5sum -c
 ```
 
-Once-only is inspired by the Lisp once-only function, which wraps another
-function and calculates a result only once, based on the same inputs. It is
-also inspired by the NixOS software deployment system, which guarantees
-packages are uniquely deployed, based on the source code inputs and the
-configuration at compile time.
-
 ## Installation
 
 Note: once-only is written in Ruby, but you don't need to understand
@@ -234,6 +266,18 @@ Note that files that come with a path will be stripped of their path
 before execution. When files are very large you may want to consider
 the --scratch option.
 
+### Precalculated hashes
+
+The --precalc option allows for using precalculated hash values. The
+extension says what hash to use. Example:
+
+```sh
+once-only --precalc hash.md5 /bin/cat ~/.bashrc
+```
+
+Once-only will pick up the values from 'hash.md5' and use those after
+making sure the time stamp of the hash file is most recent.
+
 ### Use the scratch disk with --scratch (nyi)
 
 watch this page

data/VERSION

@@ -1 +1 @@
-0.2.1
+0.2.2

data/bin/once-only

@@ -19,10 +19,12 @@ Usage:
    --skip-regex regex  skip making checksumes of filenames that match the regex (multiple allowed)
    --skip-glob regex   skip making checksumes of filenames that match the glob (multiple allowed)
    --include|--in file include input filename for making the checksums (file should exist)
+   --precalc file      use precalculated Hash values (extension .md5)
    -v                  increase verbosity
    -q                  run quietly
    --debug             give debug information
    --dry-run           do not execute command
+   --ignore-lock       ignore locked files (they expire normally after 5 hours)
    --force             force execute command
 
 Examples:
@@ -67,7 +69,7 @@ def exit_error errval = 1, msg = nil
 end
 
 def parse_args(args)
-  options = { :skip => [], :skip_regex => [], :skip_glob => [], :include => [] }
+  options = { :precalc => [], :skip => [], :skip_regex => [], :skip_glob => [], :include => [] }
 
   consume = lambda { |args|
     if not args[0]
@@ -112,6 +114,10 @@ def parse_args(args)
       when '--copy' 
         options[:copy] = true
         consume.call(args[1..-1])
+      when '--precalc'
+        p args
+        options[:precalc] << args[1]
+        consume.call(args[2..-1])
       when '-h', '--help'
         print USAGE
         exit 1
@@ -127,6 +133,9 @@ def parse_args(args)
       when '--dry-run' 
         options[:dry_run] = true
         consume.call(args[1..-1])
+      when '--ignore-lock' 
+        options[:ignore_lock] = true
+        consume.call(args[1..-1])
       when '--force' 
         options[:force] = true
         consume.call(args[1..-1])
@@ -158,6 +167,10 @@ once_only_args = OnceOnly::Check.drop_pbs_option(once_only_args)
 once_only_args = OnceOnly::Check.drop_dir_option(once_only_args)
 once_only_command = once_only_args.join(' ')
 
+# --- Fetch the pre-calculated checksums
+precalc = OnceOnly::Check.precalculated_checksums(options[:precalc])
+
+# --- Calculate the checksums for the items in the list
 command = args.join(' ')
 command_sorted = args.sort.join(' ')
 command_sha1 = OnceOnly::Check::calc_checksum(command_sorted)
@@ -173,6 +186,7 @@ base_dir = Dir.pwd
 executable = args[0]
 args = args[1..-1] if options[:skip_exe]
 
+# Handle the file list
 file_list = OnceOnly::Check::get_file_list(args)
 options[:skip_regex].each { |regex|
   file_list = OnceOnly::Check::filter_file_list(file_list,regex)
@@ -186,16 +200,29 @@ OnceOnly::Check::check_files_exist(options[:include])
 file_list += options[:include]
 file_list = file_list.uniq
 
-checksums = OnceOnly::Check::calc_file_checksums(file_list)
+checksums = OnceOnly::Check::calc_file_checksums(file_list,precalc)
 checksums.push ['SHA1',command_sha1,command_sorted] if not options[:skip_cli]
 
 # ---- Create filenames
 once_only_filename = OnceOnly::Check::make_once_filename(checksums,File.basename(executable))
 $stderr.print "Check file name ",once_only_filename,"\n" if options[:verbose]
 error_filename = once_only_filename + '.err'
-tag_filename = once_only_filename + '.run'
 $stderr.print "**STATUS** Job file exists ",once_only_filename,"!\n" if options[:debug] and File.exist?(once_only_filename)
 
+# ---- The 'run' file is used to prepare for a job
+tag_filename = once_only_filename + '.run'
+
+# ---- The 'lock' file is used when the job is running
+lock_filename = once_only_filename + '.lock'
+if File.exist?(lock_filename) and not options[:force] and not options[:ignore_lock]
+  $stderr.print "**STATUS** Job is locked with #{lock_filename} '#{original_commands}'!\n" if not options[:quiet]
+  if File.mtime(lock_filename) < Time.now - 18000
+    $stderr.print "**STATUS ** Lock is stale, retrying now\n"
+  else
+    exit 0
+  end
+end
+
 # ---- Create job name
 dirname = File.basename(Dir.pwd).rjust(8,"-") # make sure it is long enough
 
@@ -207,7 +234,7 @@ if options[:copy]
   copy_dir = base_dir + '/' + File.basename(once_only_filename,".txt") 
 end
 
-if options[:force] or not File.exist?(once_only_filename)
+if options[:force] or not File.exist?(once_only_filename) 
   $stderr.print "Running #{command}\n" if not options[:quiet]
   OnceOnly::Check::write_file(tag_filename,checksums)
   if options[:pbs]
@@ -240,6 +267,7 @@ if options[:force] or not File.exist?(once_only_filename)
   else
     # --- Run on command line
     if !options[:dry_run]
+      File.open(lock_filename, "w") {}
       success = 
         if options[:copy]
           exit_error(1,"Directory #{copy_dir} already exists!") if File.directory?(copy_dir)
@@ -274,6 +302,7 @@ if options[:force] or not File.exist?(once_only_filename)
           system(command)
         end
       Dir.chdir(base_dir) if options[:copy]
+      File.unlink(lock_filename)
       if not success
         OnceOnly::Check::write_file(error_filename,checksums)
         File.unlink(tag_filename) if File.exist?(tag_filename)
@@ -283,7 +312,7 @@ if options[:force] or not File.exist?(once_only_filename)
         File.unlink(error_filename) if File.exist?(error_filename)
         OnceOnly::Check::write_file(once_only_filename,checksums)
         File.unlink(tag_filename) if File.exist?(tag_filename)
-     end
+      end
     end
   end
 else

data/lib/once-only/check.rb

@@ -31,10 +31,36 @@ module OnceOnly
       list.map { |name| ( Dir.glob(glob).index(name) ? nil : name ) }.compact
     end
 
-    # Calculate the checksums for each file in the list
-    def Check::calc_file_checksums list
+    # Return a hash of files with their hash type, hash value and check time
+    def Check::precalculated_checksums(files)
+      precalc = {}
+      files.each do | fn |
+        dir = File.dirname(fn)
+        raise "Precalculated hash file should have .md5 extension!" if fn !~ /\.md5$/
+        t = File.mtime(fn)
+        File.open(fn).each { |s|
+          a = s.split
+          checkfn = File.expand_path(a[1],dir)
+          precalc[checkfn] = { type: 'MD5', hash: a[0], time: t }
+        }
+      end
+      precalc
+    end
+
+    # Calculate the checksums for each file in the list and return a list
+    # of array - each row containing the Hash type (MD5), the value and the (relative)
+    # file path.
+    def Check::calc_file_checksums list, precalc
       list.map { |fn|
-        ['MD5'] + `/usr/bin/md5sum #{fn}`.split
+        # First see if fn is in the precalculated list
+        fqn = File.expand_path(fn)
+        if precalc[fqn] and File.mtime(fqn) < precalc[fqn][:time]
+          $stderr.print "Precalculated ",fn,"\n"
+          rec = precalc[fqn]
+          [rec[:type],rec[:hash],fqn]
+        else
+          ['MD5'] + `/usr/bin/md5sum #{fqn}`.split
+        end
       }
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: once-only
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.2.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-27 00:00:00.000000000 Z
+date: 2013-11-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -118,7 +118,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -4434987383675774210
+      hash: -2270452427765269751
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements: