bio-gemma-wrapper 0.98.1 → 0.99.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: b7a8bfe787236f397dba6e05aef202486ba53389
-  data.tar.gz: 9fe2398ed3fcd053e8258f73a64b1610bfddf9e2
+SHA256:
+  metadata.gz: da5f26b8acd9c3782c2b3f5f2a39af965fc7e1785cc820b49faca82924d74e51
+  data.tar.gz: 17035ee5fada269ae88dd0ed91d84075b2af88b400de1d0e9829cbdb60d5d0cb
 SHA512:
-  metadata.gz: 0a691906f13da3469597517d160874315f5962822b0273757d33c37283c2a9a98da9a41d08c51e4ed7854b3d3593f9d96ac81d59212690edd628a2677499d501
-  data.tar.gz: 5d4b898ff3566f52652cbb7db6597dc245e9a5da6b0d9da107de81deec0d51c02c822762e0ab921a2309c1dbc3455b55c1adc5fe94c891e31a9c32cb6d343f2e
+  metadata.gz: eaec3c7dad4fc1bda713765e056bfe11dd69d4ca850333fed5a1a27e344724365a705ddf7845ce63b5af6b35ab6140da10f4bc7067aaa4539e47f6c6f94de1f0
+  data.tar.gz: c26b282c0fd7c70a702467e58c3f6ea22f820d91a8a364b335cdab7e807add9cf1079faa25c46a87b89c78e4293990cdea2210427c5f9c3565bd5040fdbef496

data/README.md

@@ -1,12 +1,20 @@
 [![gemma-wrapper gem version](https://badge.fury.io/rb/bio-gemma-wrapper.svg)](https://badge.fury.io/rb/bio-gemma-wrapper)
 
-# GEMMA wrapper caches K between runs with LOCO support
+# GEMMA with LOCO, permutations and slurm support (and caching)
 
 ![Genetic associations identified in CFW mice using GEMMA (Parker et al,
 Nat. Genet., 2016)](cfw.gif)
 
 ## Introduction
 
+Gemma-wrapper allows running GEMMA with LOCO, GEMMA with caching,
+GEMMA in parallel (now the default with LOCO), and GEMMA on
+PBS. Gemma-wrapper is used to run GEMMA as part of the
+https://genenetwork.org/ environment.
+
+Note that a version of gemma-wrapper is projected to be integrated
+into gemma itself.
+
 GEMMA is a software toolkit for fast application of linear mixed
 models (LMMs) and related models to genome-wide association studies
 (GWAS) and other large-scale data sets.
@@ -14,15 +22,21 @@ models (LMMs) and related models to genome-wide association studies
 This repository contains gemma-wrapper, essentially a wrapper of
 GEMMA that provides support for caching the kinship or relatedness
 matrix (K) and caching LM and LMM computations with the option of full
-leave-one-chromosome-out genome scans (LOCO).
+leave-one-chromosome-out genome scans (LOCO). Jobs can also be
+submitted to HPC PBS, i.e., slurm.
 
 gemma-wrapper requires a recent version of GEMMA and essentially
 does a pass-through of all standard GEMMA invocation switches. On
 return gemma-wrapper can return a JSON object (--json) which is
 useful for web-services.
 
-Note that this a work in progress (WIP). What is described below
-should work.
+## Performance
+
+LOCO runs in parallel by default which is at least a 5x performance
+improvement on a machine with enough cores. GEMMA without LOCO,
+however, does not run in parallel by default.  Performance
+improvements with the parallel implementation for LOCO and non-LOCO
+can be viewed [here](./test/performance/releases.gmi).
 
 ## Installation
 
@@ -32,8 +46,9 @@ Prerequisites are
 * Standard [Ruby >2.0 ](https://www.ruby-lang.org/en/) which comes on
   almost all Linux systems
 
-gemma-wrapper comes as a Ruby [gem](https://rubygems.org/gems/bio-gemma-wrapper) and
-can be installed with
+gemma-wrapper comes as a Ruby
+[gem](https://rubygems.org/gems/bio-gemma-wrapper) and can be
+installed with
 
     gem install bio-gemma-wrapper
 
@@ -47,14 +62,19 @@ and it will render something like
 Usage: gemma-wrapper [options] -- [gemma-options]
         --permutate n                Permutate # times by shuffling phenotypes
         --permute-phenotypes filen   Phenotypes to be shuffled in permutations
-        --loco [x,y,1,2,3...]        Run full LOCO
+        --loco                       Run full leave-one-chromosome-out (LOCO)
+        --chromosomes [1,2,3]        Run specific chromosomes
         --input filen                JSON input variables (used for LOCO)
         --cache-dir path             Use a cache directory
         --json                       Create output file in JSON format
-        --force                      Force computation
+        --force                      Force computation (override cache)
+        --parallel                   Run jobs in parallel
+        --no-parallel                Do not run jobs in parallel
+        --slurm[=opts]               Use slurm PBS for submitting jobs
         --q, --quiet                 Run quietly
     -v, --verbose                    Run verbosely
-        --debug                      Show debug messages and keep intermediate output
+    -d, --debug                      Show debug messages and keep intermediate output
+        --dry-run                    Show commands, but don't execute
         --                           Anything after gets passed to GEMMA
 
     -h, --help                       display this help and exit
@@ -69,6 +89,8 @@ Unpack it and run the tool as
 
     ./bin/gemma-wrapper --help
 
+See below for using a GNU Guix environment.
+
 ## Usage
 
 gemma-wrapper picks up GEMMA from the PATH. To override that behaviour
@@ -90,12 +112,13 @@ the data files are found):
     gemma-wrapper -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
         -debug
 
 Run it twice to see
 
-    /tmp/3079151e14b219c3b243b673d88001c1675168b4.log.txt gemma-wrapper CACHE HIT!
+    /tmp/0bdd7add5e8f7d9af36b283d0341c115124273e0.log.txt CACHE HIT!
 
 gemma-wrapper computes the unique HASH value over the command
 line switches passed into GEMMA as well as the contents of the files
@@ -107,10 +130,12 @@ You can also get JSON output on STDOUT by providing the --json switch
     gemma-wrapper --json -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
-        -debug
+        -debug > K.json
 
-prints out something that can be parsed with a calling program
+K.json is something that can be parsed with a calling program, and is
+also below as input for the GWA step. Example:
 
 ```json
 {"warnings":[],"errno":0,"debug":[],"type":"K","files":[["/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.log.txt","/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.cXX.txt"]],"cache_hit":true,"gemma_command":"../gemma/bin/gemma -g test/data/input/BXD_geno.txt.gz -p test/data/input/BXD_pheno.txt -gk -debug -outdir /tmp -o 18ce786ab92064a7ee38a7422e7838abf91f5eb0"}
@@ -123,6 +148,7 @@ default. If you want something else provide a --cache-dir, e.g.
     gemma-wrapper --cache-dir ~/.gemma-cache -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
         -debug
 
@@ -130,10 +156,10 @@ will store K in ~/.gemma-cache.
 
 ### GWA
 
-Run the LMM using the K's captured in K.json using the --input
+Run the LMM using the K's captured earlier in K.json using the --input
 switch
 
-    gemma-wrapper --json --loco --input K.json -- \
+    gemma-wrapper --json --input K.json -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
         -c test/data/input/BXD_covariates2.txt \
@@ -153,7 +179,7 @@ https://github.com/genetics-statistics/GEMMA/issues/46). To loop all
 chromosomes first create all K's with
 
     gemma-wrapper --json \
-        --loco 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,X -- \
+        --loco -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
         -a test/data/input/BXD_snps.txt \
@@ -201,12 +227,24 @@ Next, using K.json, permute the phenotypes with something like
         -lmm 2 -maf 0.1 \
         -debug > GWA.json
 
-This should get the 95% significant and 67% suggestive thresholds:
+This should get the estimated 95% (significant) and 67% (suggestive) thresholds:
+
+    ["95 percentile (significant) ", 1.92081e-05, 4.7]
+    ["67 percentile (suggestive)  ", 5.227785e-05, 4.3]
+
+### Slurm PBS
 
-    ["95 percentile (significant) ", 2.015475e-05, 4.7]
-    ["67 percentile (suggestive)  ", 2.015475e-05, 4.7]
+To run gemma-wrapper on HPC use the '--slurm' switch.
 
+## Development
+
+We use GNU Guix for development and deployment. Use the [.guix-deploy](.guix-deploy) script in the checked out git repo:
+
+```
+source .guix-deploy
+ruby bin/gemma-wrapper --help
+```
 
 ## Copyright
 
-Copyright (c) 2017,2018 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.
+Copyright (c) 2017-2021 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.

data/VERSION

@@ -1 +1 @@
-0.98.1
+0.99.4

data/bin/gemma-wrapper

@@ -4,7 +4,7 @@
 # Author:: Pjotr Prins
 # License:: GPL3
 #
-# Copyright (C) 2017,2018 Pjotr Prins <pjotr.prins@thebird.nl>
+# Copyright (C) 2017-2021 Pjotr Prins <pjotr.prins@thebird.nl>
 
 USAGE = "
 GEMMA wrapper example:
@@ -14,12 +14,12 @@ GEMMA wrapper example:
       gemma-wrapper -- \\
         -g test/data/input/BXD_geno.txt.gz \\
         -p test/data/input/BXD_pheno.txt \\
+        -a test/data/input/BXD_snps.txt \
         -gk
 
     LOCO K computation with caching and JSON output
 
-      gemma-wrapper --json \\
-        --loco 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,X -- \\
+      gemma-wrapper --json --loco -- \\
         -g test/data/input/BXD_geno.txt.gz \\
         -p test/data/input/BXD_pheno.txt \\
         -a test/data/input/BXD_snps.txt \\
@@ -38,11 +38,10 @@ GEMMA wrapper example:
 Gemma gets used from the path. You can override by setting
 
       env GEMMA_COMMAND=path/bin/gemma gemma-wrapper ...
-
 "
 # These are used for testing compatibility with the gemma tool
 GEMMA_V_MAJOR = 98
-GEMMA_V_MINOR = 0
+GEMMA_V_MINOR = 4
 
 basepath = File.dirname(File.dirname(__FILE__))
 $: << File.join(basepath,'lib')
@@ -66,17 +65,21 @@ if not gemma_command
 end
 
 
+require 'digest/sha1'
 require 'fileutils'
 require 'optparse'
-require 'tmpdir'
 require 'tempfile'
+require 'tmpdir'
+
+require 'lock'
 
 split_at = ARGV.index('--')
+
 if split_at
   gemma_args = ARGV[split_at+1..-1]
 end
 
-options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir() }
+options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir(), quiet: false, permute_phenotypes: false, parallel: nil }
 
 opts = OptionParser.new do |o|
   o.banner = "\nUsage: #{File.basename($0)} [options] -- [gemma-options]"
@@ -91,8 +94,12 @@ opts = OptionParser.new do |o|
     raise "Phenotype input file #{phenotypes} does not exist" if !File.exist?(phenotypes)
   end
 
-  o.on('--loco [x,y,1,2,3...]', Array, 'Run full LOCO') do |lst|
-    options[:loco] = lst
+  o.on('--loco', 'Run full leave-one-chromosome-out (LOCO)') do |b|
+    options[:loco] = b
+  end
+
+  o.on('--chromosomes [1,2,3]',Array,'Run specific chromosomes') do |lst|
+    options[:chromosomes] = lst
   end
 
   o.on('--input filen',String, 'JSON input variables (used for LOCO)') do |filen|
@@ -112,6 +119,22 @@ opts = OptionParser.new do |o|
     options[:force] = true
   end
 
+  o.on("--parallel", "Run jobs in parallel") do |b|
+    options[:parallel] = true
+  end
+
+  o.on("--no-parallel", "Do not run jobs in parallel") do |b|
+    options[:parallel] = false
+  end
+
+  o.on("--slurm[=opts]",String,"Use slurm PBS for submitting jobs") do |slurm|
+    options[:slurm_opts] = ""
+    options[:slurm] = true
+    if slurm
+      options[:slurm_opts] = slurm
+    end
+  end
+
   o.on("--q", "--quiet", "Run quietly") do |q|
     options[:quiet] = true
   end
@@ -120,15 +143,20 @@ opts = OptionParser.new do |o|
     options[:verbose] = true
   end
 
-  o.on("--debug", "Show debug messages and keep intermediate output") do |v|
+  o.on("-d", "--debug", "Show debug messages and keep intermediate output") do |v|
     options[:debug] = true
   end
 
+  o.on("--dry-run", "Show commands, but don't execute") do |b|
+    options[:dry_run] = b
+  end
+
   o.on('--','Anything after gets passed to GEMMA') do
     o.terminate()
   end
 
   o.separator ""
+
   o.on_tail('-h', '--help', 'display this help and exit') do
     options[:show_help] = true
   end
@@ -168,26 +196,46 @@ warning = lambda do |*msg|
   record[:warnings].push *msg.join("")
   OUTPUT.print "WARNING: ",*msg,"\n"
 end
+
 info = lambda do |*msg|
   record[:debug].push *msg.join("") if options[:json] and options[:debug]
   OUTPUT.print *msg,"\n" if !options[:quiet]
 end
 
+# Fetch chromosomes
+def get_chromosomes annofn
+  h = {}
+  File.open(annofn,"r").each_line do | line |
+    chr = line.split(/\s+/)[2]
+    h[chr] = true
+  end
+  h.map { |k,v| k }
+end
 # ---- Start banner
 
 GEMMA_K_VERSION=version
-GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017,2018\n"
+GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017-2021\n"
 info.call GEMMA_K_BANNER
 
 # Check gemma version
-GEMMA_COMMAND=options[:gemma_command]
+begin
+  gemma_command2 = options[:gemma_command]
+  info.call "NOTE: gemma-wrapper is soon to be replaced by gemma2/lib"
+
+  GEMMA_INFO = `#{gemma_command2}`
+rescue Errno::ENOENT
+  gemma_command2 = "gemma"
+  error.call "<#{gemma_command2}> command not found"
+end
 
-gemma_version_header = `#{GEMMA_COMMAND}`.split("\n").grep(/GEMMA|Version/)[0].strip
+gemma_version_header = GEMMA_INFO.split("\n").grep(/GEMMA|Version/)[0].strip
 info.call "Using ",gemma_version_header,"\n"
 gemma_version = gemma_version_header.split(/[,\s]+/)[1]
 v_version, v_major, v_minor = gemma_version.split(".")
 info.call "Found #{gemma_version}, comparing against expected v0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}"
 
+info.call gemma_version_header
+
 warning.call "GEMMA version is out of date. Update GEMMA to 0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}!" if v_major.to_i < GEMMA_V_MAJOR or (v_major.to_i == GEMMA_V_MAJOR and (v_minor != nil and v_minor.to_i < GEMMA_V_MINOR))
 
 options[:gemma_version_header] = gemma_version_header
@@ -203,74 +251,160 @@ if RUBY_VERSION =~ /^1/
   warning "runs on Ruby 2.x only\n"
 end
 
+# ---- LOCO defaults to parallel
+if options[:parallel] == nil
+  options[:parallel] = true if options[:loco]
+end
+
+debug.call(options) # some debug output
+debug.call(record)
+
 DO_COMPUTE_KINSHIP = gemma_args.include?("-gk")
 DO_COMPUTE_GWA = !DO_COMPUTE_KINSHIP
 
+if options[:parallel]
+  begin
+    skip_cite = `echo "will cite" |parallel --citation`
+    debug.call(skip_cite)
+    PARALLEL_INFO = `parallel --help`
+  rescue Errno::ENOENT
+    error.call "<parallel> command not found"
+  end
+  parallel_cmds = []
+end
+
+# ---- Fetch chromosomes from SNP annotation file
+anno_idx = gemma_args.index '-a'
+raise "Expected GEMMA -a genotype file switch" if anno_idx == nil
+CHROMOSOMES = get_chromosomes(gemma_args[anno_idx+1])
+
 # ---- Compute HASH on inputs
 hashme = []
 geno_idx = gemma_args.index '-g'
 raise "Expected GEMMA -g genotype file switch" if geno_idx == nil
 pheno_idx = gemma_args.index '-p'
-hashme =
-  if DO_COMPUTE_KINSHIP and pheno_idx != nil
-    # Remove the phenotype file from the hash
-    gemma_args[0..pheno_idx-1] + gemma_args[pheno_idx+2..-1]
-  else
-    gemma_args
-  end
 
 if DO_COMPUTE_GWA and options[:permute_phenotypes]
   raise "Did not expect GEMMA -p phenotype whith permutations (only use --permutate-phenotypes)" if pheno_idx
-  hashme += ['-p', options[:permute_phenotypes]]
 end
 
-require 'digest/sha1'
-debug.call "Hashing on ",hashme,"\n"
-hashes = []
-hashme.each do | item |
-  if File.exist?(item)
-    hashes << Digest::SHA1.hexdigest(File.read(item))
-    debug.call [item,hashes.last]
+execute = lambda { |cmd|
+  info.call("Executing: #{cmd}")
+  err = 0
+  if not options[:debug]
+    # send output to stderr line by line
+    IO.popen("#{cmd}") do |io|
+      while s = io.gets
+        $stderr.print s
+      end
+      io.close
+      err = $?.to_i
+    end
   else
-    hashes << item
+    $stderr.print `#{cmd}`
+    err = $?.to_i
+  end
+  err
+}
+
+compute_hash = lambda do | phenofn = nil |
+  # Compute a HASH on the inputs
+  debug.call "Hashing on ",hashme,"\n"
+  hashes = []
+  hm = if phenofn
+         hashme + ["-p", phenofn]
+       else
+         hashme
+       end
+  debug.call(hm)
+  hm.each do | item |
+    if File.file?(item)
+      hashes << Digest::SHA1.hexdigest(File.read(item))
+      debug.call [item,hashes.last]
+    else
+      hashes << item
+    end
   end
+  debug.call(hashes)
+  Digest::SHA1.hexdigest hashes.join(' ')
 end
-HASH = Digest::SHA1.hexdigest hashes.join(' ')
 
+HASH = compute_hash.call()
 options[:hash] = HASH
 
+at_exit do
+  Lock.release(HASH)
+end
+
+Lock.create(HASH) # this will wait for a lock to expire
+
+joblog = options[:cache_dir]+"/"+HASH+"-parallel.log"
+
 # Create cache dir
 FileUtils::mkdir_p options[:cache_dir]
 
+Dir.mktmpdir do |tmpdir| # tmpdir for GEMMA output
+
 error.call "Do not use the GEMMA -o switch!" if gemma_args.include? '-o'
 error.call "Do not use the GEMMA -outdir switch!" if gemma_args.include? '-outdir'
+GEMMA_ARGS_HASH = gemma_args.dup # do not include outdir
 gemma_args << '-outdir'
-gemma_args << options[:cache_dir]
+gemma_args << tmpdir
 GEMMA_ARGS = gemma_args
 
+hashme =
+  if DO_COMPUTE_KINSHIP and pheno_idx != nil
+    # Remove the phenotype file from the hash for GRM computation
+    GEMMA_ARGS_HASH[0..pheno_idx-1] + GEMMA_ARGS_HASH[pheno_idx+2..-1]
+  else
+    GEMMA_ARGS_HASH
+  end
+
 debug.call "Options: ",options,"\n" if !options[:quiet]
 
-invoke_gemma = lambda do |extra_args, cache_hit = false|
-  cmd="#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
+invoke_gemma = lambda do |extra_args, cache_hit = false, chr = "full", permutation = 1|
+  cmd = "#{gemma_command2} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
   record[:gemma_command] = cmd
   return if cache_hit
-  debug.call cmd
+  if options[:slurm]
+    info.call cmd
+    hashi = HASH
+    prefix = tmpdir+'/'+hashi
+    scriptfn  = prefix+".#{chr}.#{permutation}-pbs.sh"
+    script = "#!/bin/bash
+#SBATCH --job-name=gemma-#{scriptfn}
+#SBATCH --ntasks=1
+#SBATCH --time=20:00
+srun #{cmd}
+"
+    debug.call(script)
+    File.open(scriptfn,"w") { |f|
+      f.write(script)
+    }
+    cmd = "sbatch "+options[:slurm_opts] + scriptfn
+  end
   errno =
     if options[:json]
       # capture output
       err = 0
-      IO.popen(cmd) do |io|
-        while s = io.gets
-          $stderr.print s
-        end
-        io.close
-        err = $?.to_i
+      if options[:dry_run]
+        info.call("Would have invoked: ",cmd)
+      elsif options[:parallel]
+        info.call("Add parallel job: ",cmd)
+        parallel_cmds << cmd
+      else
+        err = execute.call(cmd)
       end
       err
     else
-      debug.call("Invoking ",cmd) if options[:debug]
-      system(cmd)
-      $?.exitstatus
+      if options[:dry_run]
+        info.call("Would have invoked ",cmd)
+        0
+      else
+        debug.call("Invoking ",cmd) if options[:debug]
+        system(cmd)
+        $?.exitstatus
+      end
     end
   if errno != 0
     debug.call "Gemma exit ",errno
@@ -280,11 +414,14 @@ invoke_gemma = lambda do |extra_args, cache_hit = false|
   end
 end
 
+# Takes the hash value and checks whether the (output) file exists
 # returns datafn, logfn, cache_hit
-cache = lambda do | chr, ext |
+cache = lambda do | chr, ext, h=HASH, permutation=0 |
   inject = (chr==nil ? "" : ".#{chr}" )+ext
-  hashi = (chr==nil ? HASH : HASH+inject)
-  prefix = options[:cache_dir]+'/'+hashi
+  hashi = (chr==nil ? h : h+inject)
+  prefix = options[:cache_dir]+'/'+hashi+(permutation!=0 ? "."+permutation.to_s : "")
+  # for chr 3 and permutation 1 forms something like
+  #   /tmp/1b700-a996f.3.cXX.txt.1.log.txt
   logfn  = prefix+".log.txt"
   datafn = prefix+ext
   record[:files] ||= []
@@ -320,25 +457,32 @@ kinship = lambda do | chr = nil |
 end
 
 # ---- Run GWA
-gwas = lambda do | chr, kfn, pfn |
+gwas = lambda do | chr, kfn, pfn, permutation=0 |
   record[:type] = "GWA"
   error.call "Do not use the GEMMA -k switch with gemma-wrapper - it is automatic!" if GEMMA_ARGS.include? '-k' # K is automatic
-  hashi, cache_hit = cache.call chr,".assoc.txt"
+  # Update hash for each permutation
+  hash = compute_hash.call(pfn)
+  hashi, cache_hit = cache.call(chr,".assoc.txt",hash,permutation)
   if not cache_hit
     args = [ '-k', kfn, '-o', hashi ]
     args << [ '-loco', chr ] if chr != nil
     args << [ '-p', pfn ] if pfn
-    invoke_gemma.call args
+    invoke_gemma.call args,false,chr,permutation
   end
 end
 
 LOCO = options[:loco]
-# if GEMMA_ARGS.include? '-gk'
+if LOCO
+  if options[:chromosomes]
+    CHROMOSOMES = options[:chromosomes]
+  end
+end
+
 if DO_COMPUTE_KINSHIP
   # compute K
-  info.call LOCO
-  if LOCO != nil
-    LOCO.each do |chr|
+  info.call CHROMOSOMES
+  if LOCO
+    CHROMOSOMES.each do |chr|
       info.call "LOCO for ",chr
       kinship.call(chr)
     end
@@ -347,13 +491,24 @@ if DO_COMPUTE_KINSHIP
   end
 else
   # DO_COMPUTE_GWA
-  json_in = JSON.parse(File.read(options[:input]))
+  begin
+    json_in = JSON.parse(File.read(options[:input]))
+  rescue TypeError
+    raise "Missing JSON input file?"
+  end
   raise "JSON problem, file #{options[:input]} is not -gk derived" if json_in["type"] != "K"
 
   pfn = options[:permute_phenotypes] # can be nil
-  k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
-  k_files.each do | chr, kfn | # call a GWA for each chromosome
-    gwas.call(chr,kfn,pfn)
+  if LOCO
+    k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
+    k_files.each do | chr, kfn | # call a GWA for each chromosome
+      gwas.call(chr,kfn,pfn)
+    end
+  else
+    kfn = json_in["files"][0][2]
+    CHROMOSOMES.each do | chr |
+      gwas.call(chr,kfn,pfn)
+    end
   end
   # Permute
   if options[:permutate]
@@ -364,10 +519,10 @@ else
     end
     score_list = []
     debug.call(options[:permutate],"x permutations")
-    (1..options[:permutate]).each do |i|
-      $stderr.print "Iteration ",i,"\n"
+    (1..options[:permutate]).each do |permutation|
+      $stderr.print "Iteration ",permutation,"\n"
       # Create a shuffled phenotype file
-      file = File.open("phenotypes-#{i}","w")
+      file = File.open("phenotypes-#{permutation}","w")
       tmp_pfn = file.path
       p tmp_pfn
       ps.shuffle.each do | l |
@@ -375,20 +530,23 @@ else
       end
       file.close
       k_files.each do | chr, kfn | # call a GWA for each chromosome
-        gwas.call(chr,kfn,tmp_pfn)
+        gwas.call(chr,kfn,tmp_pfn,permutation)
       end
-      # p [:HEY,record[:files].last]
-      assocfn = record[:files].last[2]
-      debug.call("Reading ",assocfn)
       score_min = 1000.0
-      File.foreach(assocfn).with_index do |assoc, assoc_line_num|
-        if assoc_line_num > 0
-          value = assoc.strip.split(/\t/).last.to_f
-          score_min = value if value < score_min
+      if false and not options[:slurm]
+        # p [:HEY,record[:files].last]
+        assocfn = record[:files].last[2]
+        debug.call("Reading ",assocfn)
+        File.foreach(assocfn).with_index do |assoc, assoc_line_num|
+          if assoc_line_num > 0
+            value = assoc.strip.split(/\t/).last.to_f
+            score_min = value if value < score_min
+          end
         end
       end
       score_list << score_min
     end
+    exit 0 if options[:slurm]
     ls = score_list.sort
     p ls
     significant = ls[(ls.size - ls.size*0.95).floor]
@@ -399,5 +557,40 @@ else
   end
 end
 
+# ---- Invoke parallel
+if options[:parallel]
+  # parallel_cmds = ["echo 1","sleep 1 && echo 2", "false", "echo 3"]
+  cmd = parallel_cmds.join("\\n")
+
+  cmd = "echo -e \"#{cmd}\""
+  err = execute.call(cmd+"|parallel --joblog #{joblog}") # first try optimistically to run all jobs in parallel
+  if err != 0
+    [16,8,4,1].each do |jobs|
+      info.call("Failed to complete parallel run -- retrying with smaller RAM footprint!")
+      err = execute.call(cmd+"|parallel -j #{jobs} --resume --joblog #{joblog}")
+      break if err == 0
+    end
+    if err != 0
+      info.call("Run failed!")
+      # Remove remaining files
+      FileUtils.rm_rf("#{tmpdir}/*", secure: true)
+      exit err
+    end
+  end
+  info.call("Run successful!")
+end
 json_out.call
-exit 0
+
+# copy all output files to the cache_dir. If a file exists only emit a warning
+Dir.glob("*.txt", base: tmpdir) do | fn |
+  source = tmpdir + "/" + fn
+  dest = options[:cache_dir] + "/" + fn
+  if not File.exist?(dest) or options[:force]
+    info.call "Move #{source} to #{dest}"
+    FileUtils.mv source, dest, verbose: false
+  else
+    warning.call "File #{dest} already exists. Not overwriting"
+  end
+end
+
+end # tmpdir

data/gemma-wrapper.gemspec

@@ -2,10 +2,11 @@ Gem::Specification.new do |s|
   s.name        = 'bio-gemma-wrapper'
   s.version     = File.read('VERSION')
   s.summary     = "GEMMA with LOCO and permutations"
-  s.description = "GEMMA wrapper adds LOCO and permutation support. Also caches K between runs with LOCO support"
+  s.description = "GEMMA wrapper adds LOCO and permutation support. Also runs in parallel and caches K between runs with LOCO support"
   s.authors     = ["Pjotr Prins"]
   s.email       = 'pjotr.public01@thebird.nl'
   s.files       = ["bin/gemma-wrapper",
+                   "lib/lock.rb",
                    "Gemfile",
                    "LICENSE.txt",
                    "README.md",

data/lib/lock.rb

@@ -0,0 +1,95 @@
+# Locking module for gemma (wrapper)
+#
+
+=begin
+
+The logic is as follows:
+
+1. a program creates a named lock file (based on a hash of its inputs) with its PID
+2. on exit it destroys the file
+3. a new program checks for the lock file
+4. if it exists and the PID is still in the ps table - wait
+5. when the pid disappears or the lock file - continue
+6. a timeout will return an error in 3 minutes
+
+Note that there is a theoretical chance the lock file existed, but disappeared. I think I have it covered by ignoring the unlink errors. Also the use of /proc/PID is Linux specific.
+
+=end
+
+
+require 'timeout'
+
+module Lock
+
+  def self.local name
+    ENV['HOME']+"/."+name.gsub("/","-")+".lck"
+  end
+
+  def self.lock_pid name
+    lockfn = local(name)
+    if File.exist?(lockfn)
+      File.read(lockfn).to_i
+    else
+      0
+    end
+  end
+
+  def self.locked? name
+    lockfn = local(name)
+    pid = lock_pid(name)
+    if File.exist?("/proc/#{pid}")
+      true
+    else
+      # the program went away - remove any 'stale' lock
+      begin
+        File.unlink(lockfn)
+      rescue Errno::ENOENT
+        # ignore error when the lock file went missing
+      end
+      false # --> no longer locked
+    end
+  end
+
+  def Lock::create name
+    wait_for(name)
+    lockfn = local(name)
+    if File.exist?(lockfn)
+      $stderr.print "\nERROR: Can not steal #{lockfn}"
+      exit 1
+    end
+    File.open(lockfn, File::RDWR|File::CREAT, 0644) do |f|
+      f.flock(File::LOCK_EX)
+      f.print(Process.pid)
+    end
+  end
+
+  def Lock::wait_for name
+    lockfn = local(name)
+    begin
+      status = Timeout::timeout(180) { # 3 minutes
+        while locked?(name)
+          $stderr.print("\nWaiting for lock #{lockfn}...")
+          sleep 2
+        end
+      }
+    rescue Timeout::Error
+      $stderr.print "\nERROR: Timed out, but I can not steal #{lockfn}"
+      exit 1
+    end
+    # yah! lock is released
+  end
+
+  def Lock::release name
+    lockfn = local(name)
+    if Process.pid == lock_pid(name)
+      begin
+        File.unlink(lockfn) # PID expired
+      rescue Errno::ENOENT
+        # ignore error when the lock file went missing
+      end
+    else
+      $stderr.print "\nERROR: can not release #{lockfn} because it is not owned by me"
+    end
+  end
+
+end

metadata

@@ -1,17 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: bio-gemma-wrapper
 version: !ruby/object:Gem::Version
-  version: 0.98.1
+  version: 0.99.4
 platform: ruby
 authors:
 - Pjotr Prins
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-11-20 00:00:00.000000000 Z
+date: 2021-11-25 00:00:00.000000000 Z
 dependencies: []
-description: GEMMA wrapper adds LOCO and permutation support. Also caches K between
-  runs with LOCO support
+description: GEMMA wrapper adds LOCO and permutation support. Also runs in parallel
+  and caches K between runs with LOCO support
 email: pjotr.public01@thebird.nl
 executables:
 - gemma-wrapper
@@ -24,6 +24,7 @@ files:
 - VERSION
 - bin/gemma-wrapper
 - gemma-wrapper.gemspec
+- lib/lock.rb
 homepage: https://github.com/genetics-statistics/gemma-wrapper
 licenses:
 - GPL3
@@ -43,8 +44,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.8
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: GEMMA with LOCO and permutations