bio-gemma-wrapper 0.98 → 0.99.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: d8f36b92e82dda9e1e592724204521f9f4f1a950
-  data.tar.gz: 077abf1c1a704ab93a27627d511aeba0dbf6b4c5
+SHA256:
+  metadata.gz: 0bd37b153e121de9c1758af736cd6904744da2de3540f2a7c547cc423382d8d1
+  data.tar.gz: 84298a943e7cfe6126653895d9714babc83a7be2bf903c7b61ff9072f1d4e4a8
 SHA512:
-  metadata.gz: b49bb2c9362eb7babd2cf20f640fc7806719a8c82e31f93a97b4517d5c2777bd4ae55b9e5a1fd37df5f22e63e2673fa01ca4afbc4e07de6955391d77bd7f90fb
-  data.tar.gz: 863ef847e4ddff6b544a733e001bcd2291c73ab394573a299d54a8f2581066fb423b40be052ce6a440c1daffbd6ddb7500c5f01b7ce2de0e38e4277c8ae9e53e
+  metadata.gz: f32d48ec2f194a513e0cf8f15463b05662e1b269fdd22a110e4cdfb9f6bc541238bac7e766cbc31a8b782379891636e4f79fb3e3667d567ab3c610298d4f11c2
+  data.tar.gz: abc9b3faf8ef2f63d566caa14312bde2b2f438c722a83e90f7da775c55e2415171efbfab385273c82e45398c31e5306fb99437c38e287570652a3b04a5432883

data/README.md

@@ -1,12 +1,20 @@
-[![Gem Version](https://badge.fury.io/rb/bio-gemma-wrapper.svg)](https://badge.fury.io/rb/bio-gemma-wrapper)
+[![gemma-wrapper gem version](https://badge.fury.io/rb/bio-gemma-wrapper.svg)](https://badge.fury.io/rb/bio-gemma-wrapper)
 
-# GEMMA wrapper caches K between runs with LOCO support
+# GEMMA with LOCO, permutations and slurm support (and caching)
 
 ![Genetic associations identified in CFW mice using GEMMA (Parker et al,
 Nat. Genet., 2016)](cfw.gif)
 
 ## Introduction
 
+Gemma-wrapper allows running GEMMA with LOCO, GEMMA with caching,
+GEMMA in parallel (now the default with LOCO), and GEMMA on
+PBS. Gemma-wrapper is used to run GEMMA as part of the
+https://genenetwork.org/ environment.
+
+Note that a version of gemma-wrapper is projected to be integrated
+into gemma itself.
+
 GEMMA is a software toolkit for fast application of linear mixed
 models (LMMs) and related models to genome-wide association studies
 (GWAS) and other large-scale data sets.
@@ -14,15 +22,21 @@ models (LMMs) and related models to genome-wide association studies
 This repository contains gemma-wrapper, essentially a wrapper of
 GEMMA that provides support for caching the kinship or relatedness
 matrix (K) and caching LM and LMM computations with the option of full
-leave-one-chromosome-out genome scans (LOCO).
+leave-one-chromosome-out genome scans (LOCO). Jobs can also be
+submitted to HPC PBS, i.e., slurm.
 
 gemma-wrapper requires a recent version of GEMMA and essentially
 does a pass-through of all standard GEMMA invocation switches. On
 return gemma-wrapper can return a JSON object (--json) which is
 useful for web-services.
 
-Note that this a work in progress (WIP). What is described below
-should work.
+## Performance
+
+LOCO runs in parallel by default which is at least a 5x performance
+improvement on a machine with enough cores. GEMMA without LOCO,
+however, does not run in parallel by default.  Performance
+improvements with the parallel implementation for LOCO and non-LOCO
+can be viewed [here](./test/performance/releases.gmi).
 
 ## Installation
 
@@ -32,8 +46,9 @@ Prerequisites are
 * Standard [Ruby >2.0 ](https://www.ruby-lang.org/en/) which comes on
   almost all Linux systems
 
-gemma-wrapper comes as a Ruby [gem](https://rubygems.org/gems/bio-gemma-wrapper) and
-can be installed with
+gemma-wrapper comes as a Ruby
+[gem](https://rubygems.org/gems/bio-gemma-wrapper) and can be
+installed with
 
     gem install bio-gemma-wrapper
 
@@ -47,14 +62,19 @@ and it will render something like
 Usage: gemma-wrapper [options] -- [gemma-options]
         --permutate n                Permutate # times by shuffling phenotypes
         --permute-phenotypes filen   Phenotypes to be shuffled in permutations
-        --loco [x,y,1,2,3...]        Run full LOCO
+        --loco                       Run full leave-one-chromosome-out (LOCO)
+        --chromosomes [1,2,3]        Run specific chromosomes
         --input filen                JSON input variables (used for LOCO)
         --cache-dir path             Use a cache directory
         --json                       Create output file in JSON format
-        --force                      Force computation
+        --force                      Force computation (override cache)
+        --parallel                   Run jobs in parallel
+        --no-parallel                Do not run jobs in parallel
+        --slurm[=opts]               Use slurm PBS for submitting jobs
         --q, --quiet                 Run quietly
     -v, --verbose                    Run verbosely
-        --debug                      Show debug messages and keep intermediate output
+    -d, --debug                      Show debug messages and keep intermediate output
+        --dry-run                    Show commands, but don't execute
         --                           Anything after gets passed to GEMMA
 
     -h, --help                       display this help and exit
@@ -69,6 +89,8 @@ Unpack it and run the tool as
 
     ./bin/gemma-wrapper --help
 
+See below for using a GNU Guix environment.
+
 ## Usage
 
 gemma-wrapper picks up GEMMA from the PATH. To override that behaviour
@@ -90,12 +112,13 @@ the data files are found):
     gemma-wrapper -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
         -debug
 
 Run it twice to see
 
-    /tmp/3079151e14b219c3b243b673d88001c1675168b4.log.txt gemma-wrapper CACHE HIT!
+    /tmp/0bdd7add5e8f7d9af36b283d0341c115124273e0.log.txt CACHE HIT!
 
 gemma-wrapper computes the unique HASH value over the command
 line switches passed into GEMMA as well as the contents of the files
@@ -107,10 +130,12 @@ You can also get JSON output on STDOUT by providing the --json switch
     gemma-wrapper --json -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
-        -debug
+        -debug > K.json
 
-prints out something that can be parsed with a calling program
+K.json is something that can be parsed with a calling program, and is
+also below as input for the GWA step. Example:
 
 ```json
 {"warnings":[],"errno":0,"debug":[],"type":"K","files":[["/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.log.txt","/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.cXX.txt"]],"cache_hit":true,"gemma_command":"../gemma/bin/gemma -g test/data/input/BXD_geno.txt.gz -p test/data/input/BXD_pheno.txt -gk -debug -outdir /tmp -o 18ce786ab92064a7ee38a7422e7838abf91f5eb0"}
@@ -123,11 +148,29 @@ default. If you want something else provide a --cache-dir, e.g.
     gemma-wrapper --cache-dir ~/.gemma-cache -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
+        -a test/data/input/BXD_snps.txt \
         -gk \
         -debug
 
 will store K in ~/.gemma-cache.
 
+### GWA
+
+Run the LMM using the K's captured earlier in K.json using the --input
+switch
+
+    gemma-wrapper --json --input K.json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+Running it twice should show that GWA is not recomputed.
+
+    /tmp/9e411810ad341de6456ce0c6efd4f973356d0bad.log.txt CACHE HIT!
+
 ### LOCO
 
 Recent versions of GEMMA have LOCO support for a single chromosome
@@ -136,7 +179,7 @@ https://github.com/genetics-statistics/GEMMA/issues/46). To loop all
 chromosomes first create all K's with
 
     gemma-wrapper --json \
-        --loco 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,X -- \
+        --loco -- \
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
         -a test/data/input/BXD_snps.txt \
@@ -163,6 +206,45 @@ GWA.json contains the file names of every chromosome
 The -k switch is injected automatically. Again output switches are not
 allowed (-o, -outdir)
 
+### Permutations
+
+Permutations can be run with and without LOCO. First create K
+
+    gemma-wrapper --json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -gk \
+        -debug > K.json
+
+Next, using K.json, permute the phenotypes with something like
+
+    gemma-wrapper --json --loco --input K.json \
+        --permutate 100 --permute-phenotype test/data/input/BXD_pheno.txt -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+This should get the estimated 95% (significant) and 67% (suggestive) thresholds:
+
+    ["95 percentile (significant) ", 1.92081e-05, 4.7]
+    ["67 percentile (suggestive)  ", 5.227785e-05, 4.3]
+
+### Slurm PBS
+
+To run gemma-wrapper on HPC use the '--slurm' switch.
+
+## Development
+
+We use GNU Guix for development and deployment. Use the [.guix-deploy](.guix-deploy) script in the checked out git repo:
+
+```
+source .guix-deploy
+ruby bin/gemma-wrapper --help
+```
+
 ## Copyright
 
-Copyright (c) 2017 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.
+Copyright (c) 2017-2021 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.

data/VERSION

@@ -1 +1 @@
-0.98
+0.99.3

data/bin/gemma-wrapper

@@ -4,7 +4,7 @@
 # Author:: Pjotr Prins
 # License:: GPL3
 #
-# Copyright (C) 2017,2018 Pjotr Prins <pjotr.prins@thebird.nl>
+# Copyright (C) 2017-2021 Pjotr Prins <pjotr.prins@thebird.nl>
 
 USAGE = "
 GEMMA wrapper example:
@@ -14,12 +14,12 @@ GEMMA wrapper example:
       gemma-wrapper -- \\
         -g test/data/input/BXD_geno.txt.gz \\
         -p test/data/input/BXD_pheno.txt \\
+        -a test/data/input/BXD_snps.txt \
         -gk
 
     LOCO K computation with caching and JSON output
 
-      gemma-wrapper --json \\
-        --loco 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,X -- \\
+      gemma-wrapper --json --loco -- \\
         -g test/data/input/BXD_geno.txt.gz \\
         -p test/data/input/BXD_pheno.txt \\
         -a test/data/input/BXD_snps.txt \\
@@ -38,11 +38,10 @@ GEMMA wrapper example:
 Gemma gets used from the path. You can override by setting
 
       env GEMMA_COMMAND=path/bin/gemma gemma-wrapper ...
-
 "
-# These are used for testing compatibility
+# These are used for testing compatibility with the gemma tool
 GEMMA_V_MAJOR = 98
-GEMMA_V_MINOR = 0
+GEMMA_V_MINOR = 4
 
 basepath = File.dirname(File.dirname(__FILE__))
 $: << File.join(basepath,'lib')
@@ -66,17 +65,19 @@ if not gemma_command
 end
 
 
+require 'digest/sha1'
 require 'fileutils'
 require 'optparse'
-require 'tmpdir'
 require 'tempfile'
+require 'tmpdir'
 
 split_at = ARGV.index('--')
+
 if split_at
   gemma_args = ARGV[split_at+1..-1]
 end
 
-options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir() }
+options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir(), quiet: false, permute_phenotypes: false, parallel: nil }
 
 opts = OptionParser.new do |o|
   o.banner = "\nUsage: #{File.basename($0)} [options] -- [gemma-options]"
@@ -91,8 +92,12 @@ opts = OptionParser.new do |o|
     raise "Phenotype input file #{phenotypes} does not exist" if !File.exist?(phenotypes)
   end
 
-  o.on('--loco [x,y,1,2,3...]', Array, 'Run full LOCO') do |lst|
-    options[:loco] = lst
+  o.on('--loco', 'Run full leave-one-chromosome-out (LOCO)') do |b|
+    options[:loco] = b
+  end
+
+  o.on('--chromosomes [1,2,3]',Array,'Run specific chromosomes') do |lst|
+    options[:chromosomes] = lst
   end
 
   o.on('--input filen',String, 'JSON input variables (used for LOCO)') do |filen|
@@ -112,6 +117,22 @@ opts = OptionParser.new do |o|
     options[:force] = true
   end
 
+  o.on("--parallel", "Run jobs in parallel") do |b|
+    options[:parallel] = true
+  end
+
+  o.on("--no-parallel", "Do not run jobs in parallel") do |b|
+    options[:parallel] = false
+  end
+
+  o.on("--slurm[=opts]",String,"Use slurm PBS for submitting jobs") do |slurm|
+    options[:slurm_opts] = ""
+    options[:slurm] = true
+    if slurm
+      options[:slurm_opts] = slurm
+    end
+  end
+
   o.on("--q", "--quiet", "Run quietly") do |q|
     options[:quiet] = true
   end
@@ -120,15 +141,20 @@ opts = OptionParser.new do |o|
     options[:verbose] = true
   end
 
-  o.on("--debug", "Show debug messages and keep intermediate output") do |v|
+  o.on("-d", "--debug", "Show debug messages and keep intermediate output") do |v|
     options[:debug] = true
   end
 
+  o.on("--dry-run", "Show commands, but don't execute") do |b|
+    options[:dry_run] = b
+  end
+
   o.on('--','Anything after gets passed to GEMMA') do
     o.terminate()
   end
 
   o.separator ""
+
   o.on_tail('-h', '--help', 'display this help and exit') do
     options[:show_help] = true
   end
@@ -173,21 +199,40 @@ info = lambda do |*msg|
   OUTPUT.print *msg,"\n" if !options[:quiet]
 end
 
+# Fetch chromosomes
+def get_chromosomes annofn
+  h = {}
+  File.open(annofn,"r").each_line do | line |
+    chr = line.split(/\s+/)[2]
+    h[chr] = true
+  end
+  h.map { |k,v| k }
+end
 # ---- Start banner
 
 GEMMA_K_VERSION=version
-GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017,2018\n"
+GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017-2021\n"
 info.call GEMMA_K_BANNER
 
 # Check gemma version
 GEMMA_COMMAND=options[:gemma_command]
+info.call "NOTE: gemma-wrapper is soon to be replaced by gemma2/lib"
+
+begin
+  GEMMA_INFO = `#{GEMMA_COMMAND}`
+rescue Errno::ENOENT
+  GEMMA_COMMAND = "gemma" if not GEMMA_COMMAND
+  error.call "<#{GEMMA_COMMAND}> command not found"
+end
 
-gemma_version_header = `#{GEMMA_COMMAND}`.split("\n").grep(/GEMMA|Version/)[0].strip
+gemma_version_header = GEMMA_INFO.split("\n").grep(/GEMMA|Version/)[0].strip
 info.call "Using ",gemma_version_header,"\n"
 gemma_version = gemma_version_header.split(/[,\s]+/)[1]
 v_version, v_major, v_minor = gemma_version.split(".")
 info.call "Found #{gemma_version}, comparing against expected v0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}"
 
+info.call gemma_version_header
+
 warning.call "GEMMA version is out of date. Update GEMMA to 0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}!" if v_major.to_i < GEMMA_V_MAJOR or (v_major.to_i == GEMMA_V_MAJOR and (v_minor != nil and v_minor.to_i < GEMMA_V_MINOR))
 
 options[:gemma_version_header] = gemma_version_header
@@ -203,74 +248,152 @@ if RUBY_VERSION =~ /^1/
   warning "runs on Ruby 2.x only\n"
 end
 
+# ---- LOCO defaults to parallel
+if options[:parallel] == nil
+  options[:parallel] = true if options[:loco]
+end
+
+debug.call(options) # some debug output
+debug.call(record)
+
 DO_COMPUTE_KINSHIP = gemma_args.include?("-gk")
 DO_COMPUTE_GWA = !DO_COMPUTE_KINSHIP
 
+if options[:parallel]
+  begin
+    skip_cite = `echo "will cite" |parallel --citation`
+    debug.call(skip_cite)
+    PARALLEL_INFO = `parallel --help`
+  rescue Errno::ENOENT
+    error.call "<parallel> command not found"
+  end
+  parallel_cmds = []
+end
+
+# ---- Fetch chromosomes from SNP annotation file
+anno_idx = gemma_args.index '-a'
+raise "Expected GEMMA -a genotype file switch" if anno_idx == nil
+CHROMOSOMES = get_chromosomes(gemma_args[anno_idx+1])
+
 # ---- Compute HASH on inputs
 hashme = []
 geno_idx = gemma_args.index '-g'
 raise "Expected GEMMA -g genotype file switch" if geno_idx == nil
 pheno_idx = gemma_args.index '-p'
-hashme =
-  if DO_COMPUTE_KINSHIP and pheno_idx != nil
-    p [pheno_idx,gemma_args[pheno_idx+2..-1]]
-    gemma_args[0..pheno_idx-1] + gemma_args[pheno_idx+2..-1]
-  else
-    gemma_args
-  end
 
-if DO_COMPUTE_GWA
-  raise "Did not expect GEMMA -p phenotype file switch" if pheno_idx
-  hashme += ['-p', options[:permute_phenotypes]] if options[:permute_phenotypes]
+if DO_COMPUTE_GWA and options[:permute_phenotypes]
+  raise "Did not expect GEMMA -p phenotype whith permutations (only use --permutate-phenotypes)" if pheno_idx
 end
 
-require 'digest/sha1'
-debug.call "Hashing on ",hashme,"\n"
-hashes = []
-hashme.each do | item |
-  if File.exist?(item)
-    hashes << Digest::SHA1.hexdigest(File.read(item))
-    debug.call [item,hashes.last]
+execute = lambda { |cmd|
+  info.call("Executing: #{cmd}")
+  err = 0
+  if not options[:debug]
+    # send output to stderr line by line
+    IO.popen("#{cmd}") do |io|
+      while s = io.gets
+        $stderr.print s
+      end
+      io.close
+      err = $?.to_i
+    end
   else
-    hashes << item
+    $stderr.print `#{cmd}`
+    err = $?.to_i
+  end
+  err
+}
+
+compute_hash = lambda do | phenofn = nil |
+  # Compute a HASH on the inputs
+  debug.call "Hashing on ",hashme,"\n"
+  hashes = []
+  hm = if phenofn
+         hashme + ["-p", phenofn]
+       else
+         hashme
+       end
+  debug.call(hm)
+  hm.each do | item |
+    if File.file?(item)
+      hashes << Digest::SHA1.hexdigest(File.read(item))
+      debug.call [item,hashes.last]
+    else
+      hashes << item
+    end
   end
+  debug.call(hashes)
+  Digest::SHA1.hexdigest hashes.join(' ')
 end
-HASH = Digest::SHA1.hexdigest hashes.join(' ')
 
+HASH = compute_hash.call()
 options[:hash] = HASH
 
 # Create cache dir
 FileUtils::mkdir_p options[:cache_dir]
 
+Dir.mktmpdir do |tmpdir| # tmpdir for GEMMA output
+
 error.call "Do not use the GEMMA -o switch!" if gemma_args.include? '-o'
 error.call "Do not use the GEMMA -outdir switch!" if gemma_args.include? '-outdir'
+GEMMA_ARGS_HASH = gemma_args.dup # do not include outdir
 gemma_args << '-outdir'
-gemma_args << options[:cache_dir]
+gemma_args << tmpdir
 GEMMA_ARGS = gemma_args
 
+hashme =
+  if DO_COMPUTE_KINSHIP and pheno_idx != nil
+    # Remove the phenotype file from the hash for GRM computation
+    GEMMA_ARGS_HASH[0..pheno_idx-1] + GEMMA_ARGS_HASH[pheno_idx+2..-1]
+  else
+    GEMMA_ARGS_HASH
+  end
+
 debug.call "Options: ",options,"\n" if !options[:quiet]
 
-invoke_gemma = lambda do |extra_args, cache_hit = false|
-  cmd="#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
+invoke_gemma = lambda do |extra_args, cache_hit = false, chr = "full", permutation = 1|
+  cmd = "#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
   record[:gemma_command] = cmd
   return if cache_hit
-  # debug.call cmd
+  if options[:slurm]
+    info.call cmd
+    hashi = HASH
+    prefix = tmpdir+'/'+hashi
+    scriptfn  = prefix+".#{chr}.#{permutation}-pbs.sh"
+    script = "#!/bin/bash
+#SBATCH --job-name=gemma-#{scriptfn}
+#SBATCH --ntasks=1
+#SBATCH --time=20:00
+srun #{cmd}
+"
+    debug.call(script)
+    File.open(scriptfn,"w") { |f|
+      f.write(script)
+    }
+    cmd = "sbatch "+options[:slurm_opts] + scriptfn
+  end
   errno =
     if options[:json]
       # capture output
       err = 0
-      IO.popen(cmd) do |io|
-        while s = io.gets
-          $stderr.print s
-        end
-        io.close
-        err = $?.to_i
+      if options[:dry_run]
+        info.call("Would have invoked: ",cmd)
+      elsif options[:parallel]
+        info.call("Add parallel job: ",cmd)
+        parallel_cmds << cmd
+      else
+        err = execute.call(cmd)
       end
       err
     else
-      debug.call("Invoking ",cmd) if options[:debug]
-      system(cmd)
-      $?.exitstatus
+      if options[:dry_run]
+        info.call("Would have invoked ",cmd)
+        0
+      else
+        debug.call("Invoking ",cmd) if options[:debug]
+        system(cmd)
+        $?.exitstatus
+      end
     end
   if errno != 0
     debug.call "Gemma exit ",errno
@@ -280,11 +403,14 @@ invoke_gemma = lambda do |extra_args, cache_hit = false|
   end
 end
 
+# Takes the hash value and checks whether the (output) file exists
 # returns datafn, logfn, cache_hit
-cache = lambda do | chr, ext |
+cache = lambda do | chr, ext, h=HASH, permutation=0 |
   inject = (chr==nil ? "" : ".#{chr}" )+ext
-  hashi = (chr==nil ? HASH : HASH+inject)
-  prefix = options[:cache_dir]+'/'+hashi
+  hashi = (chr==nil ? h : h+inject)
+  prefix = options[:cache_dir]+'/'+hashi+(permutation!=0 ? "."+permutation.to_s : "")
+  # for chr 3 and permutation 1 forms something like
+  #   /tmp/1b700-a996f.3.cXX.txt.1.log.txt
   logfn  = prefix+".log.txt"
   datafn = prefix+ext
   record[:files] ||= []
@@ -320,25 +446,32 @@ kinship = lambda do | chr = nil |
 end
 
 # ---- Run GWA
-gwas = lambda do | chr, kfn, pfn |
+gwas = lambda do | chr, kfn, pfn, permutation=0 |
   record[:type] = "GWA"
   error.call "Do not use the GEMMA -k switch with gemma-wrapper - it is automatic!" if GEMMA_ARGS.include? '-k' # K is automatic
-  hashi, cache_hit = cache.call chr,".assoc.txt"
+  # Update hash for each permutation
+  hash = compute_hash.call(pfn)
+  hashi, cache_hit = cache.call(chr,".assoc.txt",hash,permutation)
   if not cache_hit
     args = [ '-k', kfn, '-o', hashi ]
     args << [ '-loco', chr ] if chr != nil
     args << [ '-p', pfn ] if pfn
-    invoke_gemma.call args
+    invoke_gemma.call args,false,chr,permutation
   end
 end
 
 LOCO = options[:loco]
-# if GEMMA_ARGS.include? '-gk'
+if LOCO
+  if options[:chromosomes]
+    CHROMOSOMES = options[:chromosomes]
+  end
+end
+
 if DO_COMPUTE_KINSHIP
   # compute K
-  info.call LOCO
-  if LOCO != nil
-    LOCO.each do |chr|
+  info.call CHROMOSOMES
+  if LOCO
+    CHROMOSOMES.each do |chr|
       info.call "LOCO for ",chr
       kinship.call(chr)
     end
@@ -347,27 +480,38 @@ if DO_COMPUTE_KINSHIP
   end
 else
   # DO_COMPUTE_GWA
-  json_in = JSON.parse(File.read(options[:input]))
+  begin
+    json_in = JSON.parse(File.read(options[:input]))
+  rescue TypeError
+    raise "Missing JSON input file?"
+  end
   raise "JSON problem, file #{options[:input]} is not -gk derived" if json_in["type"] != "K"
 
-  pfn = options[:phenotypes] # can be nil
-  k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
-  k_files.each do | chr, kfn | # call a GWA for each chromosome
-    gwas.call(chr,kfn,pfn)
+  pfn = options[:permute_phenotypes] # can be nil
+  if LOCO
+    k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
+    k_files.each do | chr, kfn | # call a GWA for each chromosome
+      gwas.call(chr,kfn,pfn)
+    end
+  else
+    kfn = json_in["files"][0][2]
+    CHROMOSOMES.each do | chr |
+      gwas.call(chr,kfn,pfn)
+    end
   end
   # Permute
   if options[:permutate]
     ps = []
-    raise "You should supply --phenotype with gemma-wrapper --permutate" if not pfn
+    raise "You should supply --permute-phenotypes with gemma-wrapper --permutate" if not pfn
     File.foreach(pfn).with_index do |line, line_num|
       ps << line
     end
     score_list = []
     debug.call(options[:permutate],"x permutations")
-    (1..options[:permutate]).each do |i|
-      $stderr.print "Iteration ",i,"\n"
+    (1..options[:permutate]).each do |permutation|
+      $stderr.print "Iteration ",permutation,"\n"
       # Create a shuffled phenotype file
-      file = File.open("phenotypes-#{i}","w")
+      file = File.open("phenotypes-#{permutation}","w")
       tmp_pfn = file.path
       p tmp_pfn
       ps.shuffle.each do | l |
@@ -375,20 +519,23 @@ else
       end
       file.close
       k_files.each do | chr, kfn | # call a GWA for each chromosome
-        gwas.call(chr,kfn,tmp_pfn)
+        gwas.call(chr,kfn,tmp_pfn,permutation)
       end
-      # p [:HEY,record[:files].last]
-      assocfn = record[:files].last[2]
-      debug.call("Reading ",assocfn)
       score_min = 1000.0
-      File.foreach(assocfn).with_index do |assoc, assoc_line_num|
-        if assoc_line_num > 0
-          value = assoc.strip.split(/\t/).last.to_f
-          score_min = value if value < score_min
+      if false and not options[:slurm]
+        # p [:HEY,record[:files].last]
+        assocfn = record[:files].last[2]
+        debug.call("Reading ",assocfn)
+        File.foreach(assocfn).with_index do |assoc, assoc_line_num|
+          if assoc_line_num > 0
+            value = assoc.strip.split(/\t/).last.to_f
+            score_min = value if value < score_min
+          end
         end
       end
       score_list << score_min
     end
+    exit 0 if options[:slurm]
     ls = score_list.sort
     p ls
     significant = ls[(ls.size - ls.size*0.95).floor]
@@ -399,5 +546,38 @@ else
   end
 end
 
+# ---- Invoke parallel
+if options[:parallel]
+  # parallel_cmds = ["echo 1","sleep 1 && echo 2", "false", "echo 3"]
+  cmd = parallel_cmds.join("\\n")
+
+  cmd = "echo -e \"#{cmd}\""
+  err = execute.call(cmd+"|parallel") # all jobs in parallel
+  if err != 0
+    [16,8,4,1].each do |jobs|
+      info.call("Failed to complete parallel run -- retrying with smaller RAM footprint!")
+      err = execute.call(cmd+"|parallel -j #{jobs}")
+      break if err == 0
+    end
+    if err != 0
+      info.call("Run failed!")
+      exit err
+    end
+  end
+  info.call("Run successful!")
+end
 json_out.call
-exit 0
+
+# copy all output files to the cache_dir. If a file exists only emit a warning
+Dir.glob("*.txt", base: tmpdir) do | fn |
+  source = tmpdir + "/" + fn
+  dest = options[:cache_dir] + "/" + fn
+  if not File.exist?(dest) or options[:force]
+    info.call "Move #{source} to #{dest}"
+    FileUtils.mv source, dest, verbose: false
+  else
+    warning.call "File #{dest} already exists. Not overwriting"
+  end
+end
+
+end # tmpdir

data/gemma-wrapper.gemspec

@@ -2,7 +2,7 @@ Gem::Specification.new do |s|
   s.name        = 'bio-gemma-wrapper'
   s.version     = File.read('VERSION')
   s.summary     = "GEMMA with LOCO and permutations"
-  s.description = "GEMMA wrapper adds LOCO and permutation support. Also caches K between runs with LOCO support"
+  s.description = "GEMMA wrapper adds LOCO and permutation support. Also runs in parallel and caches K between runs with LOCO support"
   s.authors     = ["Pjotr Prins"]
   s.email       = 'pjotr.public01@thebird.nl'
   s.files       = ["bin/gemma-wrapper",

metadata

@@ -1,17 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: bio-gemma-wrapper
 version: !ruby/object:Gem::Version
-  version: '0.98'
+  version: 0.99.3
 platform: ruby
 authors:
 - Pjotr Prins
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-11-19 00:00:00.000000000 Z
+date: 2021-08-22 00:00:00.000000000 Z
 dependencies: []
-description: GEMMA wrapper adds LOCO and permutation support. Also caches K between
-  runs with LOCO support
+description: GEMMA wrapper adds LOCO and permutation support. Also runs in parallel
+  and caches K between runs with LOCO support
 email: pjotr.public01@thebird.nl
 executables:
 - gemma-wrapper
@@ -43,8 +43,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.8
+rubygems_version: 3.2.5
 signing_key: 
 specification_version: 4
 summary: GEMMA with LOCO and permutations