bio-gemma-wrapper 0.97.1 → 0.99.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 740af7561d0fb801b810713cd623f3d25e78971a
-  data.tar.gz: ce3aaae3418073dc1365bc8755ac73745ef4b01a
+SHA256:
+  metadata.gz: e27a8a3abb00b758095df5956b3854674faf5ff681a93bc028df273c40125c0d
+  data.tar.gz: e9675dbb0ea0f087dd21774635d38f3cda11b46a88b36c77dd308086fd0ec5f2
 SHA512:
-  metadata.gz: 30043473cf8b09ecf8a6fdcfa97468ba5d169bc7a0b6b6ab2102f7f978b8cb2c63fb5c5cefd7267cfe526e4748b6481aa9b6dbe7309dceff62f0351f79b16b0e
-  data.tar.gz: 07e50ab7b2b2d87bed1ce21b5b2d123693898163ff9a175841db190fcc22b43ee37ecc1df7cbc697e89b3dcaa193c489c299d3c50c1aa557d9512c4d494adec6
+  metadata.gz: 81cf5440fa531d5a831efa787800c8bea230d47cddc666a31fff066551ff347708a41ddf1368c0d3946c7ba9faef8e5882e398ad340850253c53961cce96f662
+  data.tar.gz: 582ae78c48a1eb8eeca01172eaeaba9d5ca23e69601967e334f8c218e3a4dd74b297861b01ce49b1357798b49a96c12e737100dcacec7fc34b70da1fc9c75f0d

data/README.md

@@ -1,10 +1,19 @@
-# GEMMA wrapper caches K between runs with LOCO support
+[![gemma-wrapper gem version](https://badge.fury.io/rb/bio-gemma-wrapper.svg)](https://badge.fury.io/rb/bio-gemma-wrapper)
+
+# GEMMA with LOCO, permutations and slurm support (and caching)
 
 ![Genetic associations identified in CFW mice using GEMMA (Parker et al,
 Nat. Genet., 2016)](cfw.gif)
 
 ## Introduction
 
+Gemma-wrapper allows running GEMMA with LOCO, GEMMA with caching,
+GEMMA in parallel (now the default), and GEMMA on PBS. Gemma-wrapper
+is used to run GEMMA as part of the https://genenetwork.org/
+environment.
+
+Note that gemma-wrapper is projected to be integrated into gemma2/lib.
+
 GEMMA is a software toolkit for fast application of linear mixed
 models (LMMs) and related models to genome-wide association studies
 (GWAS) and other large-scale data sets.
@@ -12,16 +21,14 @@ models (LMMs) and related models to genome-wide association studies
 This repository contains gemma-wrapper, essentially a wrapper of
 GEMMA that provides support for caching the kinship or relatedness
 matrix (K) and caching LM and LMM computations with the option of full
-leave-one-chromosome-out genome scans (LOCO).
+leave-one-chromosome-out genome scans (LOCO). Jobs can also be
+submitted to HPC PBS, i.e., slurm.
 
 gemma-wrapper requires a recent version of GEMMA and essentially
 does a pass-through of all standard GEMMA invocation switches. On
 return gemma-wrapper can return a JSON object (--json) which is
 useful for web-services.
 
-Note that this a work in progress (WIP). What is described below
-should work.
-
 ## Installation
 
 Prerequisites are
@@ -30,8 +37,9 @@ Prerequisites are
 * Standard [Ruby >2.0 ](https://www.ruby-lang.org/en/) which comes on
   almost all Linux systems
 
-gemma-wrapper comes as a Ruby [gem](https://rubygems.org/gems/bio-gemma-wrapper) and
-can be installed with
+gemma-wrapper comes as a Ruby
+[gem](https://rubygems.org/gems/bio-gemma-wrapper) and can be
+installed with
 
     gem install bio-gemma-wrapper
 
@@ -39,15 +47,18 @@ Invoke the tool with
 
     gemma-wrapper --help
 
-and it will render
+and it will render something like
 
 ```
 Usage: gemma-wrapper [options] -- [gemma-options]
+        --permutate n                Permutate # times by shuffling phenotypes
+        --permute-phenotypes filen   Phenotypes to be shuffled in permutations
         --loco [x,y,1,2,3...]        Run full LOCO
         --input filen                JSON input variables (used for LOCO)
         --cache-dir path             Use a cache directory
         --json                       Create output file in JSON format
         --force                      Force computation
+        --slurm [options]            Submit to slurm PBS
         --q, --quiet                 Run quietly
     -v, --verbose                    Run verbosely
         --debug                      Show debug messages and keep intermediate output
@@ -65,6 +76,8 @@ Unpack it and run the tool as
 
     ./bin/gemma-wrapper --help
 
+See below for using a GNU Guix environment.
+
 ## Usage
 
 gemma-wrapper picks up GEMMA from the PATH. To override that behaviour
@@ -91,11 +104,12 @@ the data files are found):
 
 Run it twice to see
 
-    /tmp/3079151e14b219c3b243b673d88001c1675168b4.log.txt gemma-wrapper CACHE HIT!
+    /tmp/0bdd7add5e8f7d9af36b283d0341c115124273e0.log.txt CACHE HIT!
 
 gemma-wrapper computes the unique HASH value over the command
 line switches passed into GEMMA as well as the contents of the files
-passed in (here the genotype and phenotype files).
+passed in (here the genotype and phenotype files - actually it ignores the phenotype with K because
+GEMMA always computes the same K).
 
 You can also get JSON output on STDOUT by providing the --json switch
 
@@ -103,9 +117,10 @@ You can also get JSON output on STDOUT by providing the --json switch
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
         -gk \
-        -debug
+        -debug > K.json
 
-prints out something that can be parsed with a calling program
+K.json is something that can be parsed with a calling program, and is
+also below as input for the GWA step. Example:
 
 ```json
 {"warnings":[],"errno":0,"debug":[],"type":"K","files":[["/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.log.txt","/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.cXX.txt"]],"cache_hit":true,"gemma_command":"../gemma/bin/gemma -g test/data/input/BXD_geno.txt.gz -p test/data/input/BXD_pheno.txt -gk -debug -outdir /tmp -o 18ce786ab92064a7ee38a7422e7838abf91f5eb0"}
@@ -123,6 +138,23 @@ default. If you want something else provide a --cache-dir, e.g.
 
 will store K in ~/.gemma-cache.
 
+### GWA
+
+Run the LMM using the K's captured earlier in K.json using the --input
+switch
+
+    gemma-wrapper --json --loco --input K.json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+Running it twice should show that GWA is not recomputed.
+
+    /tmp/9e411810ad341de6456ce0c6efd4f973356d0bad.log.txt CACHE HIT!
+
 ### LOCO
 
 Recent versions of GEMMA have LOCO support for a single chromosome
@@ -158,6 +190,45 @@ GWA.json contains the file names of every chromosome
 The -k switch is injected automatically. Again output switches are not
 allowed (-o, -outdir)
 
+### Permutations
+
+Permutations can be run with and without LOCO. First create K
+
+    gemma-wrapper --json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -gk \
+        -debug > K.json
+
+Next, using K.json, permute the phenotypes with something like
+
+    gemma-wrapper --json --loco --input K.json \
+        --permutate 100 --permute-phenotype test/data/input/BXD_pheno.txt -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+This should get the estimated 95% (significant) and 67% (suggestive) thresholds:
+
+    ["95 percentile (significant) ", 1.92081e-05, 4.7]
+    ["67 percentile (suggestive)  ", 5.227785e-05, 4.3]
+
+### Slurm PBS
+
+To run gemma-wrapper on HPC use the '--slurm' switch.
+
+## Development
+
+We use GNU Guix for development and deployment. Use the [.guix-deploy](.guix-deploy) script in the checked out git repo:
+
+```
+source .guix-deploy
+ruby bin/gemma-wrapper --help
+```
+
 ## Copyright
 
-Copyright (c) 2017 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.
+Copyright (c) 2017-2021 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.

data/VERSION

@@ -1 +1 @@
-0.97.1
+0.99.2

data/bin/gemma-wrapper

@@ -4,7 +4,7 @@
 # Author:: Pjotr Prins
 # License:: GPL3
 #
-# Copyright (C) 2017,2018 Pjotr Prins <pjotr.prins@thebird.nl>
+# Copyright (C) 2017-2021 Pjotr Prins <pjotr.prins@thebird.nl>
 
 USAGE = "
 GEMMA wrapper example:
@@ -35,10 +35,13 @@ GEMMA wrapper example:
         -lmm 2 -maf 0.1 \\
         -debug > GWA.json
 
+Gemma gets used from the path. You can override by setting
+
+      env GEMMA_COMMAND=path/bin/gemma gemma-wrapper ...
 "
-# These are used for testing compatibility
-GEMMA_V_MAJOR = 97
-GEMMA_V_MINOR = 0
+# These are used for testing compatibility with the gemma tool
+GEMMA_V_MAJOR = 98
+GEMMA_V_MINOR = 4
 
 basepath = File.dirname(File.dirname(__FILE__))
 $: << File.join(basepath,'lib')
@@ -61,32 +64,34 @@ if not gemma_command
   end
 end
 
+
+require 'digest/sha1'
 require 'fileutils'
 require 'optparse'
-require 'tmpdir'
 require 'tempfile'
+require 'tmpdir'
 
 split_at = ARGV.index('--')
 if split_at
   gemma_args = ARGV[split_at+1..-1]
 end
 
-options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir() }
+options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir(), quiet: false, parallel: true }
 
 opts = OptionParser.new do |o|
   o.banner = "\nUsage: #{File.basename($0)} [options] -- [gemma-options]"
 
-  o.on('--permutate n', Integer, 'Permutate by shuffling phenotypes') do |lst|
+  o.on('--permutate n', Integer, 'Permutate # times by shuffling phenotypes') do |lst|
     options[:permutate] = lst
     options[:force] = true
   end
 
-  o.on('--phenotypes filen',String, 'Phenotypes to be shuffled in permutations') do |phenotypes|
-    options[:phenotypes] = phenotypes
+  o.on('--permute-phenotypes filen',String, 'Phenotypes to be shuffled in permutations') do |phenotypes|
+    options[:permute_phenotypes] = phenotypes
     raise "Phenotype input file #{phenotypes} does not exist" if !File.exist?(phenotypes)
   end
 
-  o.on('--loco [x,y,1,2,3...]', Array, 'Run full LOCO') do |lst|
+  o.on('--loco [x,y,1,2,3...]', Array, 'Run full leave-one-chromosome-out (LOCO)') do |lst|
     options[:loco] = lst
   end
 
@@ -107,6 +112,18 @@ opts = OptionParser.new do |o|
     options[:force] = true
   end
 
+  o.on("--no-parallel", "Do not run jobs in parallel") do |b|
+    options[:parallel] = false
+  end
+
+  o.on("--slurm[=opts]",String,"Use slurm PBS for submitting jobs") do |slurm|
+    options[:slurm_opts] = ""
+    options[:slurm] = true
+    if slurm
+      options[:slurm_opts] = slurm
+    end
+  end
+
   o.on("--q", "--quiet", "Run quietly") do |q|
     options[:quiet] = true
   end
@@ -115,15 +132,20 @@ opts = OptionParser.new do |o|
     options[:verbose] = true
   end
 
-  o.on("--debug", "Show debug messages and keep intermediate output") do |v|
+  o.on("-d", "--debug", "Show debug messages and keep intermediate output") do |v|
     options[:debug] = true
   end
 
+  o.on("--dry-run", "Show commands, but don't execute") do |b|
+    options[:dry_run] = b
+  end
+
   o.on('--','Anything after gets passed to GEMMA') do
     o.terminate()
   end
 
   o.separator ""
+
   o.on_tail('-h', '--help', 'display this help and exit') do
     options[:show_help] = true
   end
@@ -171,17 +193,28 @@ end
 # ---- Start banner
 
 GEMMA_K_VERSION=version
-GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017,2018\n"
+GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017-2021\n"
 info.call GEMMA_K_BANNER
 
 # Check gemma version
 GEMMA_COMMAND=options[:gemma_command]
-gemma_version_header = `#{GEMMA_COMMAND}`.split("\n").grep(/GEMMA|Version/)[0].strip
+info.call "NOTE: gemma-wrapper is soon to be replaced by gemma2/lib"
+
+begin
+  GEMMA_INFO = `#{GEMMA_COMMAND}`
+rescue Errno::ENOENT
+  GEMMA_COMMAND = "gemma" if not GEMMA_COMMAND
+  error.call "<#{GEMMA_COMMAND}> command not found"
+end
+
+gemma_version_header = GEMMA_INFO.split("\n").grep(/GEMMA|Version/)[0].strip
 info.call "Using ",gemma_version_header,"\n"
 gemma_version = gemma_version_header.split(/[,\s]+/)[1]
 v_version, v_major, v_minor = gemma_version.split(".")
 info.call "Found #{gemma_version}, comparing against expected v0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}"
 
+info.call gemma_version_header
+
 warning.call "GEMMA version is out of date. Update GEMMA to 0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}!" if v_major.to_i < GEMMA_V_MAJOR or (v_major.to_i == GEMMA_V_MAJOR and (v_minor != nil and v_minor.to_i < GEMMA_V_MINOR))
 
 options[:gemma_version_header] = gemma_version_header
@@ -197,60 +230,143 @@ if RUBY_VERSION =~ /^1/
   warning "runs on Ruby 2.x only\n"
 end
 
+debug.call(options) # some debug output
+debug.call(record)
+
+DO_COMPUTE_KINSHIP = gemma_args.include?("-gk")
+DO_COMPUTE_GWA = !DO_COMPUTE_KINSHIP
+
+# ---- Set up parallel
+if options[:parallel]
+  begin
+    skip_cite = `echo "will cite" |parallel --citation`
+    debug.call(skip_cite)
+    PARALLEL_INFO = `parallel --help`
+  rescue Errno::ENOENT
+    error.call "<parallel> command not found"
+  end
+  parallel_cmds = []
+end
+
 # ---- Compute HASH on inputs
 hashme = []
 geno_idx = gemma_args.index '-g'
-raise "Expected GEMMA -g switch" if geno_idx == nil
-hashme = gemma_args
-hashme += ['-p', options[:phenotypes]] if options[:phenotypes]
+raise "Expected GEMMA -g genotype file switch" if geno_idx == nil
+pheno_idx = gemma_args.index '-p'
 
-require 'digest/sha1'
-debug.call "Hashing on ",hashme,"\n"
-hashes = []
-hashme.each do | item |
-  if File.exist?(item)
-    hashes << Digest::SHA1.hexdigest(File.read(item))
-    debug.call [item,hashes.last]
+if DO_COMPUTE_GWA and options[:permute_phenotypes]
+  raise "Did not expect GEMMA -p phenotype whith permutations (only use --permutate-phenotypes)" if pheno_idx
+end
+
+execute = lambda { |cmd|
+  info.call("Executing: #{cmd}")
+  err = 0
+  if not options[:debug]
+    # send output to stderr line by line
+    IO.popen("#{cmd}") do |io|
+      while s = io.gets
+        $stderr.print s
+      end
+      io.close
+      err = $?.to_i
+    end
   else
-    hashes << item
+    $stderr.print `#{cmd}`
+    err = $?.to_i
+  end
+  err
+}
+
+compute_hash = lambda do | phenofn = nil |
+  # Compute a HASH on the inputs
+  debug.call "Hashing on ",hashme,"\n"
+  hashes = []
+  hm = if phenofn
+         hashme + ["-p", phenofn]
+       else
+         hashme
+       end
+  debug.call(hm)
+  hm.each do | item |
+    if File.file?(item)
+      hashes << Digest::SHA1.hexdigest(File.read(item))
+      debug.call [item,hashes.last]
+    else
+      hashes << item
+    end
   end
+  debug.call(hashes)
+  Digest::SHA1.hexdigest hashes.join(' ')
 end
-HASH = Digest::SHA1.hexdigest hashes.join(' ')
 
+HASH = compute_hash.call()
 options[:hash] = HASH
 
 # Create cache dir
 FileUtils::mkdir_p options[:cache_dir]
 
+Dir.mktmpdir do |tmpdir| # tmpdir for GEMMA output
+
 error.call "Do not use the GEMMA -o switch!" if gemma_args.include? '-o'
 error.call "Do not use the GEMMA -outdir switch!" if gemma_args.include? '-outdir'
+GEMMA_ARGS_HASH = gemma_args.dup # do not include outdir
 gemma_args << '-outdir'
-gemma_args << options[:cache_dir]
+gemma_args << tmpdir
 GEMMA_ARGS = gemma_args
 
+hashme =
+  if DO_COMPUTE_KINSHIP and pheno_idx != nil
+    # Remove the phenotype file from the hash for GRM computation
+    GEMMA_ARGS_HASH[0..pheno_idx-1] + GEMMA_ARGS_HASH[pheno_idx+2..-1]
+  else
+    GEMMA_ARGS_HASH
+  end
+
 debug.call "Options: ",options,"\n" if !options[:quiet]
 
-invoke_gemma = lambda do |extra_args, cache_hit = false|
-  cmd="#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
+invoke_gemma = lambda do |extra_args, cache_hit = false, chr = "full", permutation = 1|
+  cmd = "#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
   record[:gemma_command] = cmd
   return if cache_hit
-  # debug.call cmd
+  if options[:slurm]
+    info.call cmd
+    hashi = HASH
+    prefix = tmpdir+'/'+hashi
+    scriptfn  = prefix+".#{chr}.#{permutation}-pbs.sh"
+    script = "#!/bin/bash
+#SBATCH --job-name=gemma-#{scriptfn}
+#SBATCH --ntasks=1
+#SBATCH --time=20:00
+srun #{cmd}
+"
+    debug.call(script)
+    File.open(scriptfn,"w") { |f|
+      f.write(script)
+    }
+    cmd = "sbatch "+options[:slurm_opts] + scriptfn
+  end
   errno =
     if options[:json]
       # capture output
       err = 0
-      IO.popen(cmd) do |io|
-        while s = io.gets
-          $stderr.print s
-        end
-        io.close
-        err = $?.to_i
+      if options[:dry_run]
+        info.call("Would have invoked: ",cmd)
+      elsif options[:parallel]
+        info.call("Add parallel job: ",cmd)
+        parallel_cmds << cmd
+      else
+        err = execute.call(cmd)
       end
       err
     else
-      debug.call("Invoking ",cmd) if options[:debug]
-      system(cmd)
-      $?.exitstatus
+      if options[:dry_run]
+        info.call("Would have invoked ",cmd)
+        0
+      else
+        debug.call("Invoking ",cmd) if options[:debug]
+        system(cmd)
+        $?.exitstatus
+      end
     end
   if errno != 0
     debug.call "Gemma exit ",errno
@@ -260,11 +376,14 @@ invoke_gemma = lambda do |extra_args, cache_hit = false|
   end
 end
 
+# Takes the hash value and checks whether the (output) file exists
 # returns datafn, logfn, cache_hit
-cache = lambda do | chr, ext |
+cache = lambda do | chr, ext, h=HASH, permutation=0 |
   inject = (chr==nil ? "" : ".#{chr}" )+ext
-  hashi = (chr==nil ? HASH : HASH+inject)
-  prefix = options[:cache_dir]+'/'+hashi
+  hashi = (chr==nil ? h : h+inject)
+  prefix = options[:cache_dir]+'/'+hashi+(permutation!=0 ? "."+permutation.to_s : "")
+  # for chr 3 and permutation 1 forms something like
+  #   /tmp/1b700-a996f.3.cXX.txt.1.log.txt
   logfn  = prefix+".log.txt"
   datafn = prefix+ext
   record[:files] ||= []
@@ -300,20 +419,22 @@ kinship = lambda do | chr = nil |
 end
 
 # ---- Run GWA
-gwas = lambda do | chr, kfn, pfn |
+gwas = lambda do | chr, kfn, pfn, permutation=0 |
   record[:type] = "GWA"
-  error.call "Do not use the GEMMA -k switch with gemma-wrapper!" if GEMMA_ARGS.include? '-k' # K is automatic
-  hashi, cache_hit = cache.call chr,".assoc.txt"
+  error.call "Do not use the GEMMA -k switch with gemma-wrapper - it is automatic!" if GEMMA_ARGS.include? '-k' # K is automatic
+  # Update hash for each permutation
+  hash = compute_hash.call(pfn)
+  hashi, cache_hit = cache.call(chr,".assoc.txt",hash,permutation)
   if not cache_hit
     args = [ '-k', kfn, '-o', hashi ]
     args << [ '-loco', chr ] if chr != nil
     args << [ '-p', pfn ] if pfn
-    invoke_gemma.call args
+    invoke_gemma.call args,false,chr,permutation
   end
 end
 
 LOCO = options[:loco]
-if GEMMA_ARGS.include? '-gk'
+if DO_COMPUTE_KINSHIP
   # compute K
   info.call LOCO
   if LOCO != nil
@@ -325,11 +446,11 @@ if GEMMA_ARGS.include? '-gk'
     kinship.call # no LOCO
   end
 else
-  # GWAS
+  # DO_COMPUTE_GWA
   json_in = JSON.parse(File.read(options[:input]))
   raise "JSON problem, file #{options[:input]} is not -gk derived" if json_in["type"] != "K"
 
-  pfn = options[:phenotypes] # can be nil
+  pfn = options[:permute_phenotypes] # can be nil
   k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
   k_files.each do | chr, kfn | # call a GWA for each chromosome
     gwas.call(chr,kfn,pfn)
@@ -337,16 +458,16 @@ else
   # Permute
   if options[:permutate]
     ps = []
-    raise "You should supply --phenotype with gemma-wrapper --permutate" if not pfn
+    raise "You should supply --permute-phenotypes with gemma-wrapper --permutate" if not pfn
     File.foreach(pfn).with_index do |line, line_num|
       ps << line
     end
     score_list = []
     debug.call(options[:permutate],"x permutations")
-    (1..options[:permutate]).each do |i|
-      $stderr.print "Iteration ",i,"\n"
+    (1..options[:permutate]).each do |permutation|
+      $stderr.print "Iteration ",permutation,"\n"
       # Create a shuffled phenotype file
-      file = File.open("phenotypes-#{i}","w")
+      file = File.open("phenotypes-#{permutation}","w")
       tmp_pfn = file.path
       p tmp_pfn
       ps.shuffle.each do | l |
@@ -354,20 +475,23 @@ else
       end
       file.close
       k_files.each do | chr, kfn | # call a GWA for each chromosome
-        gwas.call(chr,kfn,tmp_pfn)
+        gwas.call(chr,kfn,tmp_pfn,permutation)
       end
-      # p [:HEY,record[:files].last]
-      assocfn = record[:files].last[2]
-      debug.call("Reading ",assocfn)
       score_min = 1000.0
-      File.foreach(assocfn).with_index do |assoc, assoc_line_num|
-        if assoc_line_num > 0
-          value = assoc.strip.split(/\t/).last.to_f
-          score_min = value if value < score_min
+      if false and not options[:slurm]
+        # p [:HEY,record[:files].last]
+        assocfn = record[:files].last[2]
+        debug.call("Reading ",assocfn)
+        File.foreach(assocfn).with_index do |assoc, assoc_line_num|
+          if assoc_line_num > 0
+            value = assoc.strip.split(/\t/).last.to_f
+            score_min = value if value < score_min
+          end
         end
       end
       score_list << score_min
     end
+    exit 0 if options[:slurm]
     ls = score_list.sort
     p ls
     significant = ls[(ls.size - ls.size*0.95).floor]
@@ -378,5 +502,38 @@ else
   end
 end
 
+# ---- Invoke parallel
+if options[:parallel]
+  # parallel_cmds = ["echo 1","sleep 1 && echo 2", "false", "echo 3"]
+  cmd = parallel_cmds.join("\\n")
+
+  cmd = "echo -e \"#{cmd}\""
+  err = execute.call(cmd+"|parallel") # all jobs in parallel
+  if err != 0
+    [16,8,4,1].each do |jobs|
+      info.call("Failed to complete parallel run -- retrying with smaller RAM footprint!")
+      err = execute.call(cmd+"|parallel -j #{jobs}")
+      break if err == 0
+    end
+    if err != 0
+      info.call("Run failed!")
+      exit err
+    end
+  end
+  info.call("Run successful!")
+end
 json_out.call
-exit 0
+
+# copy all output files to the cache_dir. If a file exists only emit a warning
+Dir.glob("*.txt", base: tmpdir) do | fn |
+  source = tmpdir + "/" + fn
+  dest = options[:cache_dir] + "/" + fn
+  if not File.exist?(dest) or options[:force]
+    info.call "Move #{source} to #{dest}"
+    FileUtils.mv source, dest, verbose: false
+  else
+    warning.call "File #{dest} already exists. Not overwriting"
+  end
+end
+
+end # tmpdir

data/gemma-wrapper.gemspec

@@ -2,7 +2,7 @@ Gem::Specification.new do |s|
   s.name        = 'bio-gemma-wrapper'
   s.version     = File.read('VERSION')
   s.summary     = "GEMMA with LOCO and permutations"
-  s.description = "GEMMA wrapper adds LOCO and permutation support. Also caches K between runs with LOCO support"
+  s.description = "GEMMA wrapper adds LOCO and permutation support. Also runs in parallel and caches K between runs with LOCO support"
   s.authors     = ["Pjotr Prins"]
   s.email       = 'pjotr.public01@thebird.nl'
   s.files       = ["bin/gemma-wrapper",

metadata

@@ -1,17 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: bio-gemma-wrapper
 version: !ruby/object:Gem::Version
-  version: 0.97.1
+  version: 0.99.2
 platform: ruby
 authors:
 - Pjotr Prins
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-07-19 00:00:00.000000000 Z
+date: 2021-08-08 00:00:00.000000000 Z
 dependencies: []
-description: GEMMA wrapper adds LOCO and permutation support. Also caches K between
-  runs with LOCO support
+description: GEMMA wrapper adds LOCO and permutation support. Also runs in parallel
+  and caches K between runs with LOCO support
 email: pjotr.public01@thebird.nl
 executables:
 - gemma-wrapper
@@ -43,8 +43,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.8
+rubygems_version: 3.2.5
 signing_key: 
 specification_version: 4
 summary: GEMMA with LOCO and permutations