bio-gemma-wrapper 0.92.2 → 0.99.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: a5750dc833b64764d55eaec57f9ce981302659b9
-  data.tar.gz: f8ebc55f80ca39d84f03f4bae4385212aa5e35f7
+SHA256:
+  metadata.gz: 9ddfd904e74beebe0de1b97732d872fce171732965a835b101b9cc9be815bb05
+  data.tar.gz: 2dae1c019da23f2f87216694d641fc1eb852aa7800557bd10cfb08cb3425e844
 SHA512:
-  metadata.gz: 634361cf98042f4653d3ea4bd19d883ec08f7a1edff5f7c44935b4ef58d0b17d007451a23f7304ae29dd08c477549459b3d0cb2bcbda3060a6641d9ad665d012
-  data.tar.gz: 55f75d9a5208595e5f3ff1dffd530f17f4fdb3040c2311ea993d69b1c21001673d3825015d5e4ec327d450225470d078b46f2db84a364d27bdbcbd5852741404
+  metadata.gz: 38454a3f12dab85bef711051e73e20a015fe6b6d9c71bafada2197b9aef1aa0eabe3f3709cb0dc9d0c39f4cc454c15bc4d3aea5d06140ccde72fa13aa6285f51
+  data.tar.gz: 28e77a6995893245c501e602d488b5e0c504549fa91d8c94f902591b87b4454fe9b7923667dfacae2ab1dac7f6f7d814df1ec036b2b4f616dfd4b84c549d35d1

data/README.md

@@ -1,10 +1,19 @@
-# GEMMA wrapper caches K between runs with LOCO support
+[![gemma-wrapper gem version](https://badge.fury.io/rb/bio-gemma-wrapper.svg)](https://badge.fury.io/rb/bio-gemma-wrapper)
+
+# GEMMA with LOCO, permutations and slurm support (and caching)
 
 ![Genetic associations identified in CFW mice using GEMMA (Parker et al,
 Nat. Genet., 2016)](cfw.gif)
 
 ## Introduction
 
+Gemma-wrapper allows running GEMMA with LOCO, GEMMA with caching,
+GEMMA in parallel (now the default), and GEMMA on PBS. Gemma-wrapper
+is used to run GEMMA as part of the https://genenetwork.org/
+environment.
+
+Note that gemma-wrapper is projected to be integrated into gemma2/lib.
+
 GEMMA is a software toolkit for fast application of linear mixed
 models (LMMs) and related models to genome-wide association studies
 (GWAS) and other large-scale data sets.
@@ -12,16 +21,14 @@ models (LMMs) and related models to genome-wide association studies
 This repository contains gemma-wrapper, essentially a wrapper of
 GEMMA that provides support for caching the kinship or relatedness
 matrix (K) and caching LM and LMM computations with the option of full
-leave-one-chromosome-out genome scans (LOCO).
+leave-one-chromosome-out genome scans (LOCO). Jobs can also be
+submitted to HPC PBS, i.e., slurm.
 
 gemma-wrapper requires a recent version of GEMMA and essentially
 does a pass-through of all standard GEMMA invocation switches. On
 return gemma-wrapper can return a JSON object (--json) which is
 useful for web-services.
 
-Note that this a work in progress (WIP). What is described below
-should work.
-
 ## Installation
 
 Prerequisites are
@@ -30,8 +37,9 @@ Prerequisites are
 * Standard [Ruby >2.0 ](https://www.ruby-lang.org/en/) which comes on
   almost all Linux systems
 
-gemma-wrapper comes as a Ruby [gem](https://rubygems.org/gems/bio-gemma-wrapper) and
-can be installed with
+gemma-wrapper comes as a Ruby
+[gem](https://rubygems.org/gems/bio-gemma-wrapper) and can be
+installed with
 
     gem install bio-gemma-wrapper
 
@@ -39,15 +47,18 @@ Invoke the tool with
 
     gemma-wrapper --help
 
-and it will render
+and it will render something like
 
 ```
 Usage: gemma-wrapper [options] -- [gemma-options]
+        --permutate n                Permutate # times by shuffling phenotypes
+        --permute-phenotypes filen   Phenotypes to be shuffled in permutations
         --loco [x,y,1,2,3...]        Run full LOCO
         --input filen                JSON input variables (used for LOCO)
         --cache-dir path             Use a cache directory
         --json                       Create output file in JSON format
         --force                      Force computation
+        --slurm [options]            Submit to slurm PBS
         --q, --quiet                 Run quietly
     -v, --verbose                    Run verbosely
         --debug                      Show debug messages and keep intermediate output
@@ -65,6 +76,8 @@ Unpack it and run the tool as
 
     ./bin/gemma-wrapper --help
 
+See below for using a GNU Guix environment.
+
 ## Usage
 
 gemma-wrapper picks up GEMMA from the PATH. To override that behaviour
@@ -91,11 +104,12 @@ the data files are found):
 
 Run it twice to see
 
-    /tmp/3079151e14b219c3b243b673d88001c1675168b4.log.txt gemma-wrapper CACHE HIT!
+    /tmp/0bdd7add5e8f7d9af36b283d0341c115124273e0.log.txt CACHE HIT!
 
 gemma-wrapper computes the unique HASH value over the command
 line switches passed into GEMMA as well as the contents of the files
-passed in (here the genotype and phenotype files).
+passed in (here the genotype and phenotype files - actually it ignores the phenotype with K because
+GEMMA always computes the same K).
 
 You can also get JSON output on STDOUT by providing the --json switch
 
@@ -103,9 +117,10 @@ You can also get JSON output on STDOUT by providing the --json switch
         -g test/data/input/BXD_geno.txt.gz \
         -p test/data/input/BXD_pheno.txt \
         -gk \
-        -debug
+        -debug > K.json
 
-prints out something that can be parsed with a calling program
+K.json is something that can be parsed with a calling program, and is
+also below as input for the GWA step. Example:
 
 ```json
 {"warnings":[],"errno":0,"debug":[],"type":"K","files":[["/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.log.txt","/tmp/18ce786ab92064a7ee38a7422e7838abf91f5eb0.cXX.txt"]],"cache_hit":true,"gemma_command":"../gemma/bin/gemma -g test/data/input/BXD_geno.txt.gz -p test/data/input/BXD_pheno.txt -gk -debug -outdir /tmp -o 18ce786ab92064a7ee38a7422e7838abf91f5eb0"}
@@ -123,6 +138,23 @@ default. If you want something else provide a --cache-dir, e.g.
 
 will store K in ~/.gemma-cache.
 
+### GWA
+
+Run the LMM using the K's captured earlier in K.json using the --input
+switch
+
+    gemma-wrapper --json --loco --input K.json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+Running it twice should show that GWA is not recomputed.
+
+    /tmp/9e411810ad341de6456ce0c6efd4f973356d0bad.log.txt CACHE HIT!
+
 ### LOCO
 
 Recent versions of GEMMA have LOCO support for a single chromosome
@@ -158,6 +190,45 @@ GWA.json contains the file names of every chromosome
 The -k switch is injected automatically. Again output switches are not
 allowed (-o, -outdir)
 
+### Permutations
+
+Permutations can be run with and without LOCO. First create K
+
+    gemma-wrapper --json -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -gk \
+        -debug > K.json
+
+Next, using K.json, permute the phenotypes with something like
+
+    gemma-wrapper --json --loco --input K.json \
+        --permutate 100 --permute-phenotype test/data/input/BXD_pheno.txt -- \
+        -g test/data/input/BXD_geno.txt.gz \
+        -p test/data/input/BXD_pheno.txt \
+        -c test/data/input/BXD_covariates2.txt \
+        -a test/data/input/BXD_snps.txt \
+        -lmm 2 -maf 0.1 \
+        -debug > GWA.json
+
+This should get the estimated 95% (significant) and 67% (suggestive) thresholds:
+
+    ["95 percentile (significant) ", 1.92081e-05, 4.7]
+    ["67 percentile (suggestive)  ", 5.227785e-05, 4.3]
+
+### Slurm PBS
+
+To run gemma-wrapper on HPC use the '--slurm' switch.
+
+## Development
+
+We use GNU Guix for development and deployment. Use the [.guix-deploy](.guix-deploy) script in the checked out git repo:
+
+```
+source .guix-deploy
+ruby bin/gemma-wrapper --help
+```
+
 ## Copyright
 
-Copyright (c) 2017 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.
+Copyright (c) 2017-2021 Pjotr Prins. See [LICENSE.txt](LICENSE.txt) for further details.

data/VERSION

@@ -1 +1 @@
-0.92.2
+0.99.1

data/bin/gemma-wrapper

@@ -4,9 +4,10 @@
 # Author:: Pjotr Prins
 # License:: GPL3
 #
-# Copyright (C) 2017 Pjotr Prins <pjotr.prins@thebird.nl>
+# Copyright (C) 2017-2021 Pjotr Prins <pjotr.prins@thebird.nl>
 
-USAGE = "GEMMA wrapper example:
+USAGE = "
+GEMMA wrapper example:
 
     Simple caching of K computation with
 
@@ -34,9 +35,13 @@ USAGE = "GEMMA wrapper example:
         -lmm 2 -maf 0.1 \\
         -debug > GWA.json
 
+Gemma gets used from the path. You can override by setting
+
+      env GEMMA_COMMAND=path/bin/gemma gemma-wrapper ...
 "
-GEMMA_V_MAJOR = 97
-GEMMA_V_MINOR = 2
+# These are used for testing compatibility with the gemma tool
+GEMMA_V_MAJOR = 98
+GEMMA_V_MINOR = 1
 
 basepath = File.dirname(File.dirname(__FILE__))
 $: << File.join(basepath,'lib')
@@ -59,8 +64,11 @@ if not gemma_command
   end
 end
 
+
+require 'digest/sha1'
 require 'fileutils'
 require 'optparse'
+require 'tempfile'
 require 'tmpdir'
 
 split_at = ARGV.index('--')
@@ -68,12 +76,22 @@ if split_at
   gemma_args = ARGV[split_at+1..-1]
 end
 
-options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir() }
+options = { show_help: false, source: 'https://github.com/genetics-statistics/gemma-wrapper', version: version+' (Pjotr Prins)', date: Time.now.to_s, gemma_command: gemma_command, cache_dir: Dir.tmpdir(), quiet: false, parallel: true }
 
 opts = OptionParser.new do |o|
-  o.banner = "Usage: #{File.basename($0)} [options] -- [gemma-options]"
+  o.banner = "\nUsage: #{File.basename($0)} [options] -- [gemma-options]"
+
+  o.on('--permutate n', Integer, 'Permutate # times by shuffling phenotypes') do |lst|
+    options[:permutate] = lst
+    options[:force] = true
+  end
+
+  o.on('--permute-phenotypes filen',String, 'Phenotypes to be shuffled in permutations') do |phenotypes|
+    options[:permute_phenotypes] = phenotypes
+    raise "Phenotype input file #{phenotypes} does not exist" if !File.exist?(phenotypes)
+  end
 
-  o.on('--loco [x,y,1,2,3...]', Array, 'Run full LOCO') do |lst|
+  o.on('--loco [x,y,1,2,3...]', Array, 'Run full leave-one-chromosome-out (LOCO)') do |lst|
     options[:loco] = lst
   end
 
@@ -90,10 +108,22 @@ opts = OptionParser.new do |o|
     options[:json] = b
   end
 
-  o.on("--force", "Force computation") do |q|
+  o.on("--force", "Force computation (override cache)") do |q|
     options[:force] = true
   end
 
+  o.on("--no-parallel", "Do not run jobs in parallel") do |b|
+    options[:parallel] = false
+  end
+
+  o.on("--slurm[=opts]",String,"Use slurm PBS for submitting jobs") do |slurm|
+    options[:slurm_opts] = ""
+    options[:slurm] = true
+    if slurm
+      options[:slurm_opts] = slurm
+    end
+  end
+
   o.on("--q", "--quiet", "Run quietly") do |q|
     options[:quiet] = true
   end
@@ -102,15 +132,20 @@ opts = OptionParser.new do |o|
     options[:verbose] = true
   end
 
-  o.on("--debug", "Show debug messages and keep intermediate output") do |v|
+  o.on("-d", "--debug", "Show debug messages and keep intermediate output") do |v|
     options[:debug] = true
   end
 
+  o.on("--dry-run", "Show commands, but don't execute") do |b|
+    options[:dry_run] = b
+  end
+
   o.on('--','Anything after gets passed to GEMMA') do
     o.terminate()
   end
 
   o.separator ""
+
   o.on_tail('-h', '--help', 'display this help and exit') do
     options[:show_help] = true
   end
@@ -129,6 +164,7 @@ json_out = lambda do
   print record.to_json if options[:json]
 end
 
+# ---- Some error handlers
 error = lambda do |*msg|
   if options[:json]
     record[:error] = *msg.join(" ")
@@ -137,12 +173,14 @@ error = lambda do |*msg|
   end
   raise *msg
 end
+
 debug = lambda do |*msg|
   if options[:debug]
     record[:debug].push *msg.join("") if options[:json]
     OUTPUT.print "DEBUG: ",*msg,"\n"
   end
 end
+
 warning = lambda do |*msg|
   record[:warnings].push *msg.join("")
   OUTPUT.print "WARNING: ",*msg,"\n"
@@ -152,18 +190,32 @@ info = lambda do |*msg|
   OUTPUT.print *msg,"\n" if !options[:quiet]
 end
 
+# ---- Start banner
+
 GEMMA_K_VERSION=version
-GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017\n"
+GEMMA_K_BANNER = "gemma-wrapper #{version} (Ruby #{RUBY_VERSION}) by Pjotr Prins 2017-2021\n"
 info.call GEMMA_K_BANNER
 
 # Check gemma version
 GEMMA_COMMAND=options[:gemma_command]
-gemma_version_header = `#{GEMMA_COMMAND}`.split("\n").grep(/Version/)[0].strip
-info.call "Using GEMMA ",gemma_version_header,"\n"
+info.call "NOTE: gemma-wrapper is soon to be replaced by gemma2/lib"
+
+begin
+  GEMMA_INFO = `#{GEMMA_COMMAND}`
+rescue Errno::ENOENT
+  GEMMA_COMMAND = "gemma" if not GEMMA_COMMAND
+  error.call "<#{GEMMA_COMMAND}> command not found"
+end
+
+gemma_version_header = GEMMA_INFO.split("\n").grep(/GEMMA|Version/)[0].strip
+info.call "Using ",gemma_version_header,"\n"
 gemma_version = gemma_version_header.split(/[,\s]+/)[1]
 v_version, v_major, v_minor = gemma_version.split(".")
+info.call "Found #{gemma_version}, comparing against expected v0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}"
+
+info.call gemma_version_header
 
-error.call "GEMMA version is out of date. Update GEMMA to 0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}!" if v_major.to_i < GEMMA_V_MAJOR or (v_major.to_i == GEMMA_V_MAJOR and (v_minor == nil or v_minor.to_i < GEMMA_V_MINOR))
+warning.call "GEMMA version is out of date. Update GEMMA to 0.#{GEMMA_V_MAJOR}.#{GEMMA_V_MINOR}!" if v_major.to_i < GEMMA_V_MAJOR or (v_major.to_i == GEMMA_V_MAJOR and (v_minor != nil and v_minor.to_i < GEMMA_V_MINOR))
 
 options[:gemma_version_header] = gemma_version_header
 options[:gemma_version] = gemma_version
@@ -178,25 +230,82 @@ if RUBY_VERSION =~ /^1/
   warning "runs on Ruby 2.x only\n"
 end
 
+debug.call(options) # some debug output
+debug.call(record)
+
+DO_COMPUTE_KINSHIP = gemma_args.include?("-gk")
+DO_COMPUTE_GWA = !DO_COMPUTE_KINSHIP
+
+# ---- Set up parallel
+if options[:parallel]
+  begin
+    PARALLEL_INFO = `parallel --help`
+  rescue Errno::ENOENT
+    error.call "<parallel> command not found"
+  end
+  parallel_cmds = []
+end
+
 # ---- Compute HASH on inputs
 hashme = []
 geno_idx = gemma_args.index '-g'
-raise "Expected GEMMA -g switch" if geno_idx == nil
-hashme = gemma_args
+raise "Expected GEMMA -g genotype file switch" if geno_idx == nil
+pheno_idx = gemma_args.index '-p'
 
-require 'digest/sha1'
-debug.call "Hashing on ",hashme,"\n"
-hashes = []
-hashme.each do | item |
-  if File.exist?(item)
-    hashes << Digest::SHA1.hexdigest(File.read(item))
-    debug.call [item,hashes.last]
+if DO_COMPUTE_GWA and options[:permute_phenotypes]
+  raise "Did not expect GEMMA -p phenotype whith permutations (only use --permutate-phenotypes)" if pheno_idx
+end
+
+
+execute = lambda { |cmd|
+  info.call("Executing: #{cmd}")
+  err = 0
+  if not options[:debug]
+    # send output to stderr line by line
+    IO.popen("#{cmd}") do |io|
+      while s = io.gets
+        $stderr.print s
+      end
+      io.close
+      err = $?.to_i
+    end
   else
-    hashes << item
+    $stderr.print `#{cmd}`
+    err = $?.to_i
   end
+  err
+}
+
+hashme =
+  if DO_COMPUTE_KINSHIP and pheno_idx != nil
+    # Remove the phenotype file from the hash for GRM computation
+    gemma_args[0..pheno_idx-1] + gemma_args[pheno_idx+2..-1]
+  else
+    gemma_args
+  end
+
+compute_hash = lambda do | phenofn = nil |
+  # Compute a HASH on the inputs
+  debug.call "Hashing on ",hashme,"\n"
+  hashes = []
+  hm = if phenofn
+         hashme + ["-p", phenofn]
+       else
+         hashme
+       end
+  debug.call(hm)
+  hm.each do | item |
+    if File.file?(item)
+      hashes << Digest::SHA1.hexdigest(File.read(item))
+      debug.call [item,hashes.last]
+    else
+      hashes << item
+    end
+  end
+  Digest::SHA1.hexdigest hashes.join(' ')
 end
-HASH = Digest::SHA1.hexdigest hashes.join(' ')
 
+HASH = compute_hash.call()
 options[:hash] = HASH
 
 # Create cache dir
@@ -210,26 +319,49 @@ GEMMA_ARGS = gemma_args
 
 debug.call "Options: ",options,"\n" if !options[:quiet]
 
-invoke_gemma = lambda do |extra_args, cache_hit = false|
-  cmd="#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
+invoke_gemma = lambda do |extra_args, cache_hit = false, chr = "full", permutation = 1|
+  cmd = "#{GEMMA_COMMAND} #{GEMMA_ARGS.join(' ')} #{extra_args.join(' ')}"
   record[:gemma_command] = cmd
   return if cache_hit
-  # debug.call cmd
+  if options[:slurm]
+    info.call cmd
+    hashi = HASH
+    prefix = options[:cache_dir]+'/'+hashi
+    scriptfn  = prefix+".#{chr}.#{permutation}-pbs.sh"
+    script = "#!/bin/bash
+#SBATCH --job-name=gemma-#{scriptfn}
+#SBATCH --ntasks=1
+#SBATCH --time=20:00
+srun #{cmd}
+"
+    debug.call(script)
+    File.open(scriptfn,"w") { |f|
+      f.write(script)
+    }
+    cmd = "sbatch "+options[:slurm_opts] + scriptfn
+  end
   errno =
     if options[:json]
       # capture output
       err = 0
-      IO.popen(cmd) do |io|
-        while s = io.gets
-          $stderr.print s
-        end
-        io.close
-        err = $?.to_i
+      if options[:dry_run]
+        info.call("Would have invoked: ",cmd)
+      elsif options[:parallel]
+        info.call("Add parallel job: ",cmd)
+        parallel_cmds << cmd
+      else
+        err = execute.call(cmd)
       end
       err
     else
-      system(cmd)
-      $?.exitstatus
+      if options[:dry_run]
+        info.call("Would have invoked ",cmd)
+        0
+      else
+        debug.call("Invoking ",cmd) if options[:debug]
+        system(cmd)
+        $?.exitstatus
+      end
     end
   if errno != 0
     debug.call "Gemma exit ",errno
@@ -240,10 +372,12 @@ invoke_gemma = lambda do |extra_args, cache_hit = false|
 end
 
 # returns datafn, logfn, cache_hit
-cache = lambda do | chr, ext |
+cache = lambda do | chr, ext, h=HASH, permutation=0 |
   inject = (chr==nil ? "" : ".#{chr}" )+ext
-  hashi = HASH+inject
-  prefix = options[:cache_dir]+'/'+hashi
+  hashi = (chr==nil ? h : h+inject)
+  prefix = options[:cache_dir]+'/'+hashi+(permutation!=0 ? "."+permutation.to_s : "")
+  # for chr 3 and permutation 1 forms something like
+  #   /tmp/1b700-a996f.3.cXX.txt.1.log.txt
   logfn  = prefix+".log.txt"
   datafn = prefix+ext
   record[:files] ||= []
@@ -260,6 +394,7 @@ cache = lambda do | chr, ext |
   return hashi,false
 end
 
+# ---- Compute K
 kinship = lambda do | chr = nil |
   record[:type] = "K"
   ext = case (GEMMA_ARGS[GEMMA_ARGS.index('-gk')+1]).to_i
@@ -277,21 +412,23 @@ kinship = lambda do | chr = nil |
   end
 end
 
-gwas = lambda do | chr, kfn |
+# ---- Run GWA
+gwas = lambda do | chr, kfn, pfn, permutation=0 |
   record[:type] = "GWA"
-  error.call "Do not use the GEMMA -k switch!" if GEMMA_ARGS.include? '-k'
-  hashi, cache_hit = cache.call chr,".assoc.txt"
+  error.call "Do not use the GEMMA -k switch with gemma-wrapper - it is automatic!" if GEMMA_ARGS.include? '-k' # K is automatic
+  # Update hash for each permutation
+  hash = compute_hash.call(pfn)
+  hashi, cache_hit = cache.call(chr,".assoc.txt",hash,permutation)
   if not cache_hit
-    if chr != nil
-      invoke_gemma.call [ '-loco', chr, '-k', kfn, '-o', hashi ]
-    else
-      error.call "Not supported"
-    end
+    args = [ '-k', kfn, '-o', hashi ]
+    args << [ '-loco', chr ] if chr != nil
+    args << [ '-p', pfn ] if pfn
+    invoke_gemma.call args,false,chr,permutation
   end
 end
 
 LOCO = options[:loco]
-if GEMMA_ARGS.include? '-gk'
+if DO_COMPUTE_KINSHIP
   # compute K
   info.call LOCO
   if LOCO != nil
@@ -303,14 +440,80 @@ if GEMMA_ARGS.include? '-gk'
     kinship.call # no LOCO
   end
 else
-  # GWAS
+  # DO_COMPUTE_GWA
   json_in = JSON.parse(File.read(options[:input]))
   raise "JSON problem, file #{options[:input]} is not -gk derived" if json_in["type"] != "K"
+
+  pfn = options[:permute_phenotypes] # can be nil
   k_files = json_in["files"].map { |rec| [rec[0],rec[2]] }
-  k_files.each do | chr, kfn |
-    gwas.call(chr,kfn)
+  k_files.each do | chr, kfn | # call a GWA for each chromosome
+    gwas.call(chr,kfn,pfn)
+  end
+  # Permute
+  if options[:permutate]
+    ps = []
+    raise "You should supply --permute-phenotypes with gemma-wrapper --permutate" if not pfn
+    File.foreach(pfn).with_index do |line, line_num|
+      ps << line
+    end
+    score_list = []
+    debug.call(options[:permutate],"x permutations")
+    (1..options[:permutate]).each do |permutation|
+      $stderr.print "Iteration ",permutation,"\n"
+      # Create a shuffled phenotype file
+      file = File.open("phenotypes-#{permutation}","w")
+      tmp_pfn = file.path
+      p tmp_pfn
+      ps.shuffle.each do | l |
+        file.print(l)
+      end
+      file.close
+      k_files.each do | chr, kfn | # call a GWA for each chromosome
+        gwas.call(chr,kfn,tmp_pfn,permutation)
+      end
+      score_min = 1000.0
+      if false and not options[:slurm]
+        # p [:HEY,record[:files].last]
+        assocfn = record[:files].last[2]
+        debug.call("Reading ",assocfn)
+        File.foreach(assocfn).with_index do |assoc, assoc_line_num|
+          if assoc_line_num > 0
+            value = assoc.strip.split(/\t/).last.to_f
+            score_min = value if value < score_min
+          end
+        end
+      end
+      score_list << score_min
+    end
+    exit 0 if options[:slurm]
+    ls = score_list.sort
+    p ls
+    significant = ls[(ls.size - ls.size*0.95).floor]
+    suggestive = ls[(ls.size - ls.size*0.67).floor]
+    p ["95 percentile (significant) ",significant,(-Math.log10(significant)).round(1)]
+    p ["67 percentile (suggestive)  ",suggestive,(-Math.log10(suggestive)).round(1)]
+    exit 0
   end
 end
 
+# ---- Invoke parallel
+if options[:parallel]
+  # parallel_cmds = ["echo 1","sleep 1 && echo 2", "false", "echo 3"]
+  cmd = parallel_cmds.join("\\n")
+
+  cmd = "echo -e \"#{cmd}\""
+  err = execute.call(cmd+"|parallel") # all jobs in parallel
+  if err != 0
+    [16,8,4,1].each do |jobs|
+      info.call("Failed to complete parallel run -- retrying with smaller RAM footprint!")
+      err = execute.call(cmd+"|parallel -j #{jobs}")
+      break if err == 0
+    end
+    if err != 0
+      info.call("Run failed!")
+      exit err
+    end
+  end
+  info.call("Run successful!")
+end
 json_out.call
-exit 0

data/gemma-wrapper.gemspec

@@ -1,8 +1,8 @@
 Gem::Specification.new do |s|
   s.name        = 'bio-gemma-wrapper'
   s.version     = File.read('VERSION')
-  s.summary     = "Cache GEMMA with LOCO"
-  s.description = "GEMMA wrapper caches K between runs with LOCO support"
+  s.summary     = "GEMMA with LOCO and permutations"
+  s.description = "GEMMA wrapper adds LOCO and permutation support. Also caches K between runs with LOCO support"
   s.authors     = ["Pjotr Prins"]
   s.email       = 'pjotr.public01@thebird.nl'
   s.files       = ["bin/gemma-wrapper",

metadata

@@ -1,16 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: bio-gemma-wrapper
 version: !ruby/object:Gem::Version
-  version: 0.92.2
+  version: 0.99.1
 platform: ruby
 authors:
 - Pjotr Prins
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-03 00:00:00.000000000 Z
+date: 2021-07-11 00:00:00.000000000 Z
 dependencies: []
-description: GEMMA wrapper caches K between runs with LOCO support
+description: GEMMA wrapper adds LOCO and permutation support. Also caches K between
+  runs with LOCO support
 email: pjotr.public01@thebird.nl
 executables:
 - gemma-wrapper
@@ -43,8 +44,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.7.6.2
 signing_key: 
 specification_version: 4
-summary: Cache GEMMA with LOCO
+summary: GEMMA with LOCO and permutations
 test_files: []