bio-vcf 0.9.4 → 0.9.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f5d7a81871906abfffc93455b4d664d5755fe8d79312134eae94e84659506198
-  data.tar.gz: 8029269859aedd53c613ea9bbb17f951972b062060b5a40c22bdbe65c6c3dfa7
+  metadata.gz: 814f6cb6c8bc237fd08ab4f22f2bfea514525fc6c2fd1b9081cd314bfa3c2fd2
+  data.tar.gz: 194aa006ac5c46c157360e37e98e42dbf88d121f9c64983792e8a7cfe43d0142
 SHA512:
-  metadata.gz: ed231c3a918e5f9ab9cd8a618f3f25f0c39613ac934b496af334d77dabe64831ff08cfc722a467fc51ab8c583358ca21be769ba1d9654437d54e7d21b811ee2c
-  data.tar.gz: df49786c4f4aa5e3a3659c678fb66aeb4b7dd4bb575aacf34cc468663c18fa893502699d238b5034c308ff51a4dc05e0fadf929b923c1d646f61c3f07fef26c7
+  metadata.gz: 4cad19fa108652d42aaaff95296176a21781813848a1dacae879cd44fcb324b9f38f763fc1e516ce41cb492433457c17d27e4aca0f64308758594fb68c0abadf
+  data.tar.gz: 5290a23fe85fe063b6fb8606c77cca3abd3df3af624ac9da2ac3348336bbfafa7e8b3d70d2c5c3ba57a55da75a052f1fa2c1d20212ba7b4e656ac7263f5f1af0

data/Gemfile

@@ -1,13 +1,9 @@
 source "http://rubygems.org"
 
-# Add dependencies to develop your gem here.
-# Include everything needed to run rake, tests, features, etc.
 group :development do
-  # gem "minitest"
   gem "rake"
   gem "rspec"
   gem "cucumber"
-  gem "regressiontest", ">= 0.0.3"
 end
 
 

data/README.md

@@ -1,10 +1,35 @@
 # bio-vcf
 
-[![Build Status](https://secure.travis-ci.org/vcflib/bio-vcf.png)](http://travis-ci.org/vcflib/bio-vcf)
+[![Build Status](https://secure.travis-ci.org/vcflib/bio-vcf.png)](http://travis-ci.org/vcflib/bio-vcf) [![rubygem](https://img.shields.io/gem/v/bio-vcf.svg?style=flat)](http://rubygems.org/gems/bio-vcf "Install with Rubygems") [![AnacondaBadge](https://anaconda.org/bioconda/bio-vcf/badges/installer/conda.svg)](https://anaconda.org/bioconda/bio-vcf) [![DL](https://anaconda.org/bioconda/bio-vcf/badges/downloads.svg)](https://anaconda.org/bioconda/bio-vcf)
+[![DebianBadge](https://badges.debian.net/badges/debian/testing/bio-vcf/version.svg)](https://packages.debian.org/testing/bio-vcf)
+
+Quick index:
+
+- [INSTALL](#Install)
+- [Command line interface (CLI)](#command-line-interface-cli)
+  + [Set analysis](#set-analysis)
+  + [Genotype processing](#genotype-processing)
+  + [Sample counting](#sample-counting)
+  + [Filter with lambda](#reorder-filter-with-lambda)
+  + [Modify VCF files](#modify-vcf-files)
+  + [RDF output](#rdf-output)
+- [Templates](#templates)
+- [Metadata](#metadata)
+- [Statistics](#statistics)
+- [API](#api)
+- [Cite](#cite)
 
 
 ## Bio-vcf
 
+Bio-vcf provides a domain specific language (DSL) for processing the
+VCF format. Record named fields can be queried with regular
+expressions, e.g.
+
+```ruby
+sample.dp>20 and rec.filter !~ /LowQD/ and rec.tumor.bcount[rec.alt]>4
+```
+
 Bio-vcf is a new generation VCF parser, filter and converter. Bio-vcf
 is not only very fast for genome-wide (WGS) data, it also comes with a
 really nice filtering, evaluation and rewrite language and it can
@@ -26,9 +51,62 @@ So, why would you use bio-vcf over other parsers? Because
 11. Bio-vcf can convert *any* VCF to *any* output, including tabular data, BED, HTML, LaTeX, RDF, JSON and JSON-LD and even other VCFs by using (erb) templates
 12. Bio-vcf has soft filters
 
+Some examples are documented for [reducing GTeX](doc/GTEx_reduce.md),
+[comparing GATK](doc/GATK_comparison.md), [comparing
+VCFs](doc/Compare_VCFs.md), JSON [loading Mongo
+database](doc/Using_Mongo.md), and [generating RDF](doc/Using_RDF.md).
+
+## Options
+
+In true Unix fashion files can be piped in or passed on the command
+line:
+
+    bio-vcf --help
+
+```
+bio-vcf (biogem with pcows) by Pjotr Prins 2015-2020
+
+Usage: bio-vcf [options] filename
+e.g.  bio-vcf < test/data/input/somaticsniper.vcf
+    -i, --ignore-missing             Ignore missing data
+        --filter cmd                 Evaluate filter on each record
+        --sfilter cmd                Evaluate filter on each sample
+        --sfilter-samples list       Filter on selected samples (e.g., 0,1
+        --ifilter, --if cmd          Include filter
+        --ifilter-samples list       Include set - implicitely defines exclude set
+        --efilter, --ef cmd          Exclude filter
+        --efilter-samples list       Exclude set - overrides exclude set
+        --add-filter name            Set/add filter field to name
+        --bed bedfile                Filter on BED elements
+    -e, --eval cmd                   Evaluate command on each record
+        --eval-once cmd              Evaluate command once (usually for header info)
+        --seval cmd                  Evaluate command on each sample
+        --rewrite eval               Rewrite INFO
+        --samples list               Output selected samples
+        --rdf                        Generate Turtle RDF (also check out --template!)
+        --num-threads [num]          Multi-core version (default ALL)
+        --thread-lines num           Fork thread on num lines (default 40000)
+        --skip-header                Do not output VCF header info
+        --set-header list            Set a special tab delimited output header (#samples expands to sample names)
+    -t, --template erb               Use ERB template for output
+        --add-header-tag             Add bio-vcf status tag to header output
+        --timeout [num]              Timeout waiting for thread to complete (default 180)
+        --names                      Output sample names
+        --statistics                 Output statistics
+    -q, --quiet                      Run quietly
+    -v, --verbose                    Run verbosely
+        --debug                      Show debug messages and keep intermediate output
+
+        --id name                    Identifier
+        --tags list                  Add tags
+    -h, --help                       display this help and exit
+```
+
+## Performance
+
 Bio-vcf has better performance than other tools because of lazy
 parsing, multi-threading, and useful combinations of (fancy) command
-line filtering (who says Ruby is slow?). Adding cores, bio-vcf just
+line filtering. Adding cores, bio-vcf just
 does better. The more complicated the filters, the larger the
 gain. First a base line test to show IO performance
 
@@ -86,8 +164,7 @@ gzipped (vcf.gz) files into bio-vcf, e.g.
     --eval '[r.chrom,r.pos,r.pos+1]' > test.bed
 ```
 
-bio-vcf comes with a sensible parser definition language
-(interestingly it is 100% Ruby), an embedded Ragel parser for INFO and
+bio-vcf comes with a sensible parser definition language, an embedded Ragel parser for INFO and
 FORMAT header definitions, as well as primitives for set analysis. Few
 assumptions are made about the actual contents of the VCF file (field
 names are resolved on the fly), so bio-vcf should work with all VCF
@@ -251,17 +328,58 @@ If something is not working, check out the feature descriptions and
 the source code. It is not hard to add features. Otherwise, send a short
 example of a VCF statement you need to work on.
 
-## Installation
+## Install
+
+Requirements:
+
+* ruby
 
-The bio-vcf has no other dependencies but Ruby.
+To install bio-vcf with Ruby gems, install Ruby first, e.g. on Debian
+(as root)
 
-To install bio-vcf with Ruby gems:
+```sh
+apt-get install ruby
+```
+
+Installing ruby includes the `gem` command to install bio-vcf:
 
 ```sh
 gem install bio-vcf
+export PATH=/usr/local/bin:$PATH
 bio-vcf -h
 ```
 
+displays the help
+
+```
+bio-vcf x.x (biogem Ruby with pcows) by Pjotr Prins 2015-2020
+Usage: bio-vcf [options] filename
+e.g.  bio-vcf < test/data/input/somaticsniper.vcf
+    -i, --ignore-missing             Ignore missing data
+        --filter cmd                 Evaluate filter on each record
+(etc.)
+```
+
+To install without root you may install a gem locally with
+
+```sh
+gem install --install-dir ~/bio-vcf bio-vcf
+```
+
+and run it with something like
+
+```sh
+~/bio-vcf/gems/bio-vcf-0.9.4/bin/bio-vcf -h
+```
+
+Finally, it is possible to checkout the git repository and simply
+run the tool with
+
+```sh
+git clone https://github.com/vcflib/bio-vcf.git
+cd bio-vcf
+ruby ./bin/bio-vcf -h
+```
 
 ## Command line interface (CLI)
 
@@ -1091,12 +1209,19 @@ bundle install --path vendor/bundle
 bundle exec rake
 ```
 
+Note: we develop in a GNU Guix environment, see the header of
+[guix.scm](guix.scm) which does not use bundler.
+
 ### Debugging
 
 To debug output use '-v --num-threads=1' for generating useful
 output. Also do not use the -i switch (ignore errors) when there
 are problems.
 
+### Could not find rake-10.4.2 in any of the sources
+
+Remove Gemfile.lock before running other tools.
+
 ### Tmpdir contains (old) bio-vcf directories
 
 Multi-threaded bio-vcf writes into a temporary directory during

data/RELEASE_NOTES.md

@@ -1,8 +1,15 @@
-## ChangeLog v0.9.4 (2020????)
+## ChangeLog v0.9.5 (20210118)
+
++ Improved README and installation instructions
++ Added guix.scm build and instructions (no need for bundler)
++ Moved regressiontest into tree
+
+## ChangeLog v0.9.4 (20201222)
 
 This is an important maintenance release of bio-vcf:
 
-+ Rename bioruby-vcf to bio-vcf and migrate project to [vcflib](https://github.com/vcflib/bio-vcf).
++ Rename bioruby-vcf to bio-vcf and migrate project to [vcflib](https://github.com/vcflib/bio-vcf)
++ Fixed tests to match recent Ruby updates
 
 ## Older release notes
 

data/Rakefile

@@ -1,16 +1,21 @@
 # encoding: utf-8
 
-require 'rubygems'
+# require 'rubygems'
 require 'rake'
+# require 'cucumber/rake/task'
 
-require 'cucumber/rake/task'
-Cucumber::Rake::Task.new(:features) do |t|
+# Cucumber::Rake::Task.new(:features) do |t|
   # t.cucumber_opts = "--bundler false"
+# end
+
+desc 'Run cucumber' # without bundler
+task :features do
+  sh 'cucumber features'
 end
 
 task :default => :features
 
-task :test => [ :features ] 
+task :test => [ :features ]
 
 require 'rdoc/task'
 Rake::RDocTask.new do |rdoc|

data/VERSION

@@ -1 +1 @@
-0.9.4
+0.9.5

data/bio-vcf.gemspec

@@ -7,7 +7,6 @@ Gem::Specification.new do |s|
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Pjotr Prins"]
-  # s.date = "2015-12-28"
   s.description = "Smart lazy multi-threaded parser for VCF format with useful filtering and output rewriting (JSON, RDF etc.)"
   s.email = "pjotr.public01@thebird.nl"
   s.executables = ["bio-vcf"]

data/features/support/env.rb

@@ -1,4 +1,3 @@
-# require 'mini/test'
 
 $LOAD_PATH.unshift(File.dirname(__FILE__) + '/../../lib')
 require 'bio-vcf'
@@ -7,7 +6,7 @@ require 'rspec/expectations'
 
 # Add the regression module if in the path (it can also be a gem)
 rootdir = File.dirname(__FILE__) + '/../..'
-$LOAD_PATH.unshift(rootdir+'/lib',rootdir+'/../regressiontest/lib')
+$LOAD_PATH.unshift(rootdir+'/lib/regressiontest',rootdir+'/../regressiontest/lib')
 require 'regressiontest'
 
 include BioVcf

data/lib/regressiontest.rb

@@ -0,0 +1,11 @@
+# Please require your code below, respecting the naming conventions in the
+# bioruby directory tree.
+#
+# For example, say you have a plugin named bio-plugin, the only uncommented
+# line in this file would be 
+#
+#   require 'bio/bio-plugin/plugin'
+#
+# In this file only require other files. Avoid other source code.
+
+require 'regressiontest/cli_exec'

data/lib/regressiontest/cli_exec.rb

@@ -0,0 +1,101 @@
+require 'fileutils'
+
+module RegressionTest
+
+  DEFAULT_TESTDIR = "test/data/regression"
+
+  # Regression test runner compares output in ./test/data/regression
+  # (by default).  The convention is to have a file with names .ref
+  # (reference) and create .new
+  #
+  # You can add an :ignore regex option which ignores lines in the
+  # comparson files matching a regex
+  #
+  # :timeout sets the time out for calling a system command
+  #
+  # :should_fail expects the system command to return a non-zero
+  module CliExec
+    FilePair = Struct.new(:outfn,:reffn)
+
+    def CliExec::exec command, testname, options = {}
+      # ---- Find .ref file
+      fullname = DEFAULT_TESTDIR + "/" + testname 
+      basefn = if File.exist?(testname+".ref") || File.exist?(testname+"-stderr.ref")
+                testname 
+              elsif File.exist?(fullname + ".ref") || File.exist?(fullname+"-stderr.ref")
+                FileUtils.mkdir_p DEFAULT_TESTDIR
+                fullname
+              else
+                raise "Can not find reference file for #{testname} - expected #{fullname}.ref"
+              end
+      std_out = FilePair.new(basefn + ".new", basefn + ".ref")
+      std_err = FilePair.new(basefn + "-stderr.new", basefn + "-stderr.ref")
+      files = [std_out,std_err]
+      # ---- Create .new file
+      cmd = command + " > #{std_out.outfn} 2>#{std_err.outfn}"
+      $stderr.print cmd,"\n"
+      exec_ret = nil
+      if options[:timeout] && options[:timeout] > 0
+        Timeout.timeout(options[:timeout]) do
+          begin
+            exec_ret = Kernel.system(cmd)
+          rescue Timeout::Error
+            $stderr.print cmd, " failed to finish in under #{options[:timeout]}\n"
+            return false
+          end
+        end
+      else
+        exec_ret = Kernel.system(cmd)
+      end
+      expect_fail = (options[:should_fail] != nil)
+      if !expect_fail and exec_ret==0
+        $stderr.print cmd," returned an error\n"
+        return false 
+      end
+      if expect_fail and exec_ret
+        $stderr.print cmd," did not return an error\n"
+        return false 
+      end
+      if options[:ignore]
+        regex = options[:ignore]
+        files.each do |f|
+          outfn = f.outfn
+          outfn1 = outfn + ".1"
+          FileUtils.mv(outfn,outfn1)
+          f1 = File.open(outfn1)
+          f2 = File.open(outfn,"w")
+          f1.each_line do | line |
+            f2.print(line) if line !~ /#{regex}/
+          end
+          f1.close
+          f2.close
+          FileUtils::rm(outfn1)
+        end
+      end
+      # ---- Compare the two files
+      files.each do |f|
+        next unless File.exist?(f.reffn)
+        return false unless compare_files(f.outfn,f.reffn,options[:ignore])
+      end
+      return true
+    end
+
+    def CliExec::compare_files fn1, fn2, ignore = nil
+      if not File.exist?(fn2)
+        FileUtils::cp(fn1,fn2)
+        true
+      else
+        cmd = "diff #{fn2} #{fn1}"
+        $stderr.print cmd+"\n"
+        return true if Kernel.system(cmd) == true
+        # Hmmm. We have a different result. We are going to try again
+        # because sometimes threads have not completed
+        sleep 0.25 
+        return true if Kernel.system(cmd) == true
+        $stderr.print "If it is correct, execute \"cp #{fn1} #{fn2}\", and run again"
+        false
+      end
+    end
+  end
+
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bio-vcf
 version: !ruby/object:Gem::Version
-  version: 0.9.4
+  version: 0.9.5
 platform: ruby
 authors:
 - Pjotr Prins
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-12-22 00:00:00.000000000 Z
+date: 2021-01-18 00:00:00.000000000 Z
 dependencies: []
 description: Smart lazy multi-threaded parser for VCF format with useful filtering
   and output rewriting (JSON, RDF etc.)
@@ -22,7 +22,6 @@ extra_rdoc_files:
 files:
 - ".travis.yml"
 - Gemfile
-- Gemfile.lock
 - LICENSE.txt
 - README.md
 - RELEASE_NOTES.md
@@ -63,6 +62,8 @@ files:
 - lib/bio-vcf/vcfrecord.rb
 - lib/bio-vcf/vcfsample.rb
 - lib/bio-vcf/vcfstatistics.rb
+- lib/regressiontest.rb
+- lib/regressiontest/cli_exec.rb
 - ragel/gen_vcfheaderline_parser.rl
 - ragel/generate.sh
 - template/gatk_vcf2rdf.erb

data/Gemfile.lock

@@ -1,44 +0,0 @@
-GEM
-  remote: http://rubygems.org/
-  specs:
-    builder (3.2.2)
-    cucumber (2.1.0)
-      builder (>= 2.1.2)
-      cucumber-core (~> 1.3.0)
-      diff-lcs (>= 1.1.3)
-      gherkin3 (~> 3.1.0)
-      multi_json (>= 1.7.5, < 2.0)
-      multi_test (>= 0.1.2)
-    cucumber-core (1.3.0)
-      gherkin3 (~> 3.1.0)
-    diff-lcs (1.2.5)
-    gherkin3 (3.1.1)
-    multi_json (1.11.2)
-    multi_test (0.1.2)
-    rake (10.4.2)
-    regressiontest (0.0.3)
-    rspec (3.3.0)
-      rspec-core (~> 3.3.0)
-      rspec-expectations (~> 3.3.0)
-      rspec-mocks (~> 3.3.0)
-    rspec-core (3.3.2)
-      rspec-support (~> 3.3.0)
-    rspec-expectations (3.3.1)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-mocks (3.3.2)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.3.0)
-    rspec-support (3.3.0)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  cucumber
-  rake
-  regressiontest (>= 0.0.3)
-  rspec
-
-BUNDLED WITH
-   1.10.6