bio-vcf 0.8.2 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 515319faec0710075f13a0265a4027130ec5f10a
-  data.tar.gz: aed2ff09861568291363ca21944567ad36987813
+  metadata.gz: 1364a1b5bd401632c6c2ae2497a358c206d7a9e6
+  data.tar.gz: 2a8cad615012dcd175cc3bb614888b33db16483b
 SHA512:
-  metadata.gz: 94ff3bfda4357fc187a89c9a55116ceefe15fc2b8fa28af45e92afcad452c8d2bd65e5eae17dd2c40b046f89288c640ba5d4b40b8efb711781caed766e48f518
-  data.tar.gz: 3d810db35d1ad862aad6f4ec81d695c6d7d74d46336d4e5563e925da267d04521387994d794ff7d8384cf10d8c94701e0e2af9380ddc0b4505e00edbbedb7c3e
+  metadata.gz: d8420c9926835ff632cd8dd93c057c08f37dc557c86d76fd1766a044dcc5a8b7c573553c2ef6efe8e348d868666c7fa6d62b34779e214b090cf3287e087e61c8
+  data.tar.gz: b666a3d97a63ac18ef7ff88e002b88dee8a81e703cadaacaffc968fe7f634591bcf5df40cd930b9493e293530047d8cc6a31df446ebf1e5cd7bfb2f6164e3e0a

data/.travis.yml

@@ -1,8 +1,14 @@
+sudo: false  # required for the new containers
+
 language: ruby
 rvm:
 #  - 1.9.3 <- No longer working
-  - 2.0.0
-  - 2.1.0
+# - 2.0.0
+# - 2.1.0
+  - 2.2.2
+
+# install: 
+#   - gem install cucumber rspec regressiontest
 
 branches:
   only: 

data/Gemfile

@@ -1,15 +1,13 @@
 source "http://rubygems.org"
-# Add dependencies required to use your gem here.
-# Example:
-#   gem "activesupport", ">= 2.3.5"
 
 # Add dependencies to develop your gem here.
 # Include everything needed to run rake, tests, features, etc.
 group :development do
   # gem "minitest"
-  gem "rspec", ">= 2.14.0"
-  gem "cucumber", ">= 1.3.11"
-  gem "jeweler", ">= 2.0.1" # , "~> 1.8.4", :git => "https://github.com/technicalpickles/jeweler.git"
+  gem "rake"
+  gem "rspec"
+  gem "cucumber"
   gem "regressiontest", ">= 0.0.3"
 end
 
+

data/README.md

@@ -2,10 +2,19 @@
 
 [![Build Status](https://secure.travis-ci.org/pjotrp/bioruby-vcf.png)](http://travis-ci.org/pjotrp/bioruby-vcf) 
 
-A new generation VCF parser. Bio-vcf is not only fast for genome-wide
-(WGS) data, it also comes with a really nice filtering, evaluation and
-rewrite language and it can output any type of textual data, including
-VCF header and contents in RDF and JSON.
+## Updates
+
+* The outputter now writes (properly) in parallel with the parser
+* bio-vcf turns any VCF into JSON with header information, and
+  allows you to pipe that JSON directly into any JSON supporting
+  language, including Python and Javascript!
+
+## Bio-vcf
+
+Bio-vcf is a new generation VCF parser, filter and converter. Bio-vcf is not only
+very fast for genome-wide (WGS) data, it also comes with a really nice
+filtering, evaluation and rewrite language and it can output any type
+of textual data, including VCF header and contents in RDF and JSON.
 
 So, why would you use bio-vcf over other parsers? Because
 
@@ -14,36 +23,47 @@ So, why would you use bio-vcf over other parsers? Because
 3. Bio-vcf has great multi-sample support
 4. Bio-vcf has multiple global filters and sample filters
 5. Bio-vcf can access any VCF format
-6. Bio-vcf can do calculations on fields
-7. Bio-vcf allows for genotype processing
-8. Bio-vcf has support for set analysis
-9. Bio-vcf has sane error handling
-10. Bio-vcf can convert *any* VCF to *any* output, including tabular data, HTML, LaTeX, RDF, JSON and JSON-LD and even other VCFs by using (erb) templates
+6. Bio-vcf can parse and query the VCF header (META)
+7. Bio-vcf can do calculations on fields
+8. Bio-vcf allows for genotype processing
+9. Bio-vcf has support for set analysis
+10. Bio-vcf has sane error handling
+11. Bio-vcf can convert *any* VCF to *any* output, including tabular data, BED, HTML, LaTeX, RDF, JSON and JSON-LD and even other VCFs by using (erb) templates
 
 Bio-vcf has better performance than other tools
 because of lazy parsing, multi-threading, and useful combinations of
-(fancy) command line filtering. For example on an 2 core machine
-bio-vcf is typically 50% faster than JVM based SnpSift. Adding
+(fancy) command line filtering (who says Ruby is slow?). Adding
 cores, bio-vcf just does better. The more complicated the filters,
-the larger the gain.
+the larger the gain. First the base line test to show IO performance
+
+```sh
+  time cat ESP6500SI-V2-SSA137.GRCh38-liftover.*.vcf|wc
+  1987143 15897724 1003214613
+  real    0m7.823s
+  user    0m7.002s
+  sys     0m2.972s
+```
+
+Next run the 1Gb data with bio-vcf effectively using 5 cores on AMD Opteron(tm) Processor 6174 using Linux
 
 ```sh
-  time ./bin/bio-vcf -iv --num-threads 8 --filter 'r.info.cp>0.3' < ESP6500SI_V2_SSA137.vcf > test1.vcf
-  real    0m21.095s
-  user    1m41.101s
-  sys     0m7.852s
+  time cat ESP6500SI-V2-SSA137.GRCh38-liftover.*.vcf|./bin/bio-vcf -iv --num-threads 8 --filter 'r.info.cp.to_f>0.3' > /dev/null
+  real    0m32.491s
+  user    2m34.767s
+  sys     0m12.733s
 ```
 
-while parsing with SnpSift takes
+The same with SnpSift v4.0 takes
 
 ```sh
-  time cat ESP6500SI_V2_SSA137.vcf |java -jar snpEff/SnpSift.jar filter "( CP>0.3 )" > test.vcf
-  real    1m4.913s
-  user    0m58.071s
-  sys     0m7.982s
+time cat ESP6500SI-V2-SSA137.GRCh38-liftover.*.vcf|java -jar snpEff/SnpSift.jar filter "( CP>0.3 )" > /dev/null
+real    12m36.121s
+user    12m53.273s
+sys     0m9.913s
 ```
 
-Bio-vcf is perfect for parsing large data files. Parsing a 650 Mb GATK
+This means that on this machine bio-vcf is 24x faster than SnpSift even for a simple filter.
+In fact, bio-vcf is perfect for complex filters and parsing large data files on powerful machines. Parsing a 650 Mb GATK
 Illumina Hiseq VCF file and evaluating the results into a BED format on
 a 16 core machine takes
 
@@ -72,11 +92,17 @@ assumptions are made about the actual contents of the VCF file (field
 names are resolved on the fly), so bio-vcf should work with
 all VCF files.
 
-To fetch all entries where all samples have depth larger than 20 use
-a sample filter
+To fetch all entries where all samples have depth larger than 20 and
+filter set to PASS use a sample filter
 
 ```ruby
-  bio-vcf --sfilter 'sample.dp>20' < file.vcf
+  bio-vcf --sfilter 'sample.dp>20 and rec.filter=="PASS"' < file.vcf
+```
+
+or with a regex
+
+```ruby
+  bio-vcf --sfilter 'sample.dp>20 and rec.filter !~ /LowQD/' < file.vcf
 ```
 
 To only filter on some samples number 0 and 3:
@@ -265,6 +291,12 @@ Get the sample names
   NORMAL,TUMOR
 ```
 
+Get information from the header (META)
+
+```ruby
+  bio-vcf -q --skip-header --eval-once 'header.meta["GATKCommandLine"]' < gatk_exome.vcf
+```
+
 The 'fields' array contains unprocessed data (strings).  Print first
 five raw fields
 
@@ -304,7 +336,10 @@ Info fields are referenced by
   bio-vcf --filter 'rec.info.dp>100 and rec.info.readposranksum<=0.815' < file.vcf 
 ```
 
-With subfields defined by rec.format
+(alternatively you can use the indexed rec.info['DP'] and list INFO fields with
+rec.info.fields).
+
+Subfields defined by rec.format:
 
 ```ruby
   bio-vcf --filter 'rec.tumor.ss != 2' < file.vcf 
@@ -695,7 +730,7 @@ template could be
   "seq:ref": "<%= rec.ref %>" ,
   "seq:alt": "<%= rec.alt[0] %>" ,
   "seq:maf": <%= rec.info.maf[0] %> ,
-  "dp":      <%= rec.info.dp %> ,
+  "dp":      <%= rec.info.dp %>
 };
 ```
 
@@ -715,7 +750,7 @@ which renders
   "seq:ref": "C" ,
   "seq:alt": "T" ,
   "seq:maf": 0.0151 ,
-  "dp":      86 ,
+  "dp":      86
 };
 ```
 
@@ -767,24 +802,23 @@ can be
 ```Javascript
 =HEADER
 <% require 'json' %>
-[
-  { "HEADER": {
+{ "HEADER": {
     "options":  <%= options.to_h.to_json %>,
     "files":    <%= ARGV %>,
     "version":  "<%= BIOVCF_VERSION %>"
   },
-
+  "BODY":[
 =BODY
-
-{
-  "seq:chr": "<%= rec.chrom %>" ,
-  "seq:pos": <%= rec.pos %> ,
-  "seq:ref": "<%= rec.ref %>" ,
-  "seq:alt": "<%= rec.alt[0] %>" ,
-  "dp":      <%= rec.info.dp %> ,
-},
+    {
+      "seq:chr": "<%= rec.chrom %>" ,
+      "seq:pos": <%= rec.pos %> ,
+      "seq:ref": "<%= rec.ref %>" ,
+      "seq:alt": "<%= rec.alt[0] %>" ,
+      "dp":      <%= rec.info.dp %>
+    },
 =FOOTER
-]
+  ]
+}
 ```
 
 with
@@ -796,27 +830,28 @@ with
 may generate something like
 
 ```Javascript
-[
-  { "HEADER": {
+{ "HEADER": {
     "options":  {"show_help":false,"source":"https://github.com/CuppenResearch/bioruby-vcf","version":"0.8.1-pre3 (Pjotr Prins)","date":"2014-11-26 12:51:36 +0000","thread_lines":40000,"template":"template/vcf2json.erb","skip_header":true},
     "files":    [],
     "version":  "0.8.1-pre3"
   },
-{
-  "seq:chr": "1" ,
-  "seq:pos": 883516 ,
-  "seq:ref": "G" ,
-  "seq:alt": "A" ,
-  "dp":       ,
-},
-{
-  "seq:chr": "1" ,
-  "seq:pos": 891344 ,
-  "seq:ref": "G" ,
-  "seq:alt": "A" ,
-  "dp":       ,
-},
-]
+  "BODY":[
+    {
+      "seq:chr": "1" ,
+      "seq:pos": 883516 ,
+      "seq:ref": "G" ,
+      "seq:alt": "A" ,
+      "dp":
+    },
+    {
+      "seq:chr": "1" ,
+      "seq:pos": 891344 ,
+      "seq:ref": "G" ,
+      "seq:alt": "A" ,
+      "dp": ,
+    },
+  ]
+}
 ```
 
 Note that the template is not smart enough to remove the final comma

data/Rakefile

@@ -1,50 +1,12 @@
 # encoding: utf-8
 
 require 'rubygems'
-require 'bundler'
-begin
-  Bundler.setup(:default, :development)
-rescue Bundler::BundlerError => e
-  $stderr.puts e.message
-  $stderr.puts "Run `bundle install` to install missing gems"
-  exit e.status_code
-end
 require 'rake'
 
-require 'jeweler'
-Jeweler::Tasks.new do |gem|
-  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
-  gem.name = "bio-vcf"
-  gem.homepage = "http://github.com/pjotrp/bioruby-vcf"
-  gem.license = "MIT"
-  gem.summary = %Q{Fast multi-threaded VCF parser}
-  gem.description = %Q{Smart lazy multi-threaded parser for VCF format with useful filtering and output rewriting}
-  gem.email = "pjotr.public01@thebird.nl"
-  gem.authors = ["Pjotr Prins"]
-  gem.required_ruby_version = '>=2.0.0'
-  # dependencies defined in Gemfile
-end
-Jeweler::RubygemsDotOrgTasks.new
-
-# require 'rspec/core'
-# require 'rspec/core/rake_task'
-# RSpec::Core::RakeTask.new(:spec) do |spec|
-#   spec.pattern = FileList['spec/**/*_spec.rb']
-# end
-
-# RSpec::Core::RakeTask.new(:rcov) do |spec|
-#   spec.pattern = 'spec/**/*_spec.rb'
-#   spec.rcov = true
-# end
-
-# require 'rake/testtask'
-
-# Rake::TestTask.new do |t|
-#   t.pattern = "spec/*_spec.rb"
-# end
-
 require 'cucumber/rake/task'
-Cucumber::Rake::Task.new(:features)
+Cucumber::Rake::Task.new(:features) do |t|
+  # t.cucumber_opts = "--bundler false"
+end
 
 task :default => :features
 

data/TAGS

@@ -0,0 +1,115 @@
+
+./bin/bio-vcf,0
+
+./lib/bio-vcf.rb,0
+
+./lib/bio-vcf/vcfgenotypefield.rb,1553
+module BioVcf::BioVcf
1,0
+  class VcfNucleotides::BioVcf::VcfNucleotides
7,167
+    def initialize::BioVcf::VcfNucleotides#BioVcf::VcfNucleotides.new
8,193
+    def []::BioVcf::VcfNucleotides#[]
13,284
+    def to_ary::BioVcf::VcfNucleotides#to_ary
27,628
+    def max::BioVcf::VcfNucleotides#max
32,742
+    def min::BioVcf::VcfNucleotides#min
37,856
+    def sum::BioVcf::VcfNucleotides#sum
42,975
+  class VcfAltInfo::BioVcf::VcfAltInfo
50,1082
+    def initialize::BioVcf::VcfAltInfo#BioVcf::VcfAltInfo.new
51,1103
+    def []::BioVcf::VcfAltInfo#[]
56,1194
+    def to_ary::BioVcf::VcfAltInfo#to_ary
70,1512
+    def max::BioVcf::VcfAltInfo#max
75,1626
+    def min::BioVcf::VcfAltInfo#min
79,1702
+    def sum::BioVcf::VcfAltInfo#sum
83,1783
+  class VcfGenotypeField::BioVcf::VcfGenotypeField
88,1850
+    def initialize::BioVcf::VcfGenotypeField#BioVcf::VcfGenotypeField.new
89,1877
+    def dp4::BioVcf::VcfGenotypeField#dp4
96,2020
+    def ad::BioVcf::VcfGenotypeField#ad
100,2098
+    def pl::BioVcf::VcfGenotypeField#pl
104,2174
+    def bcount::BioVcf::VcfGenotypeField#bcount
108,2250
+    def bq::BioVcf::VcfGenotypeField#bq
112,2343
+    def amq::BioVcf::VcfGenotypeField#amq
116,2424
+    def method_missing::BioVcf::VcfGenotypeField#method_missing
120,2507
+  class VcfGenotypeFields::BioVcf::VcfGenotypeFields
130,2709
+    def initialize::BioVcf::VcfGenotypeFields#BioVcf::VcfGenotypeFields.new
131,2737
+    def []::BioVcf::VcfGenotypeFields#[]
141,3021
+    def method_missing::BioVcf::VcfGenotypeFields#method_missing
145,3136
+
+./lib/bio-vcf/vcfrdf.rb,156
+module BioVcf::BioVcf
1,0
+  module VcfRdf::BioVcf::VcfRdf
5,93
+    def VcfRdf::BioVcf::VcfRdf#VcfRdf
7,112
+    def VcfRdf::BioVcf::VcfRdf#VcfRdf
18,463
+
+./lib/bio-vcf/vcf.rb,27
+module BioVcf::BioVcf
2,1
+
+./lib/bio-vcf/vcfline.rb,118
+module BioVcf::BioVcf
1,0
+  module VcfLine::BioVcf::VcfLine
2,16
+    def VcfLine.parse::BioVcf::VcfLine.parse
5,82
+
+./lib/bio-vcf/vcfrecord.rb,1831
+module BioVcf::BioVcf
1,0
+  class VcfRecordInfo::BioVcf::VcfRecordInfo
3,17
+    def initialize::BioVcf::VcfRecordInfo#BioVcf::VcfRecordInfo.new
4,41
+    def method_missing::BioVcf::VcfRecordInfo#method_missing
9,163
+  module VcfRecordParser::BioVcf::VcfRecordParser
18,329
+    def VcfRecordParser.get_format::BioVcf::VcfRecordParser.get_format
20,397
+    def VcfRecordParser.get_info::BioVcf::VcfRecordParser.get_info
25,517
+  module VcfRecordCall::BioVcf::VcfRecordCall
30,592
+    def call_diff::BioVcf::VcfRecordCall#call_diff
31,617
+    def call_nuc::BioVcf::VcfRecordCall#call_nuc
35,705
+    def call_tumor_count::BioVcf::VcfRecordCall#call_tumor_count
39,764
+    def call_tumor_relative_count::BioVcf::VcfRecordCall#call_tumor_relative_count
43,833
+    def call_normal_count::BioVcf::VcfRecordCall#call_normal_count
47,955
+    def index::BioVcf::VcfRecordCall#index
51,1026
+  class VcfRecord::BioVcf::VcfRecord
56,1125
+    attr_reader :header::BioVcf::VcfRecord#header
60,1173
+    def initialize::BioVcf::VcfRecord#BioVcf::VcfRecord.new
62,1198
+    def chrom::BioVcf::VcfRecord#chrom
67,1292
+    def pos::BioVcf::VcfRecord#pos
71,1332
+    def ids::BioVcf::VcfRecord#ids
75,1384
+    def id::BioVcf::VcfRecord#id
79,1443
+    def ref::BioVcf::VcfRecord#ref
83,1476
+    def alt::BioVcf::VcfRecord#alt
87,1524
+    def qual::BioVcf::VcfRecord#qual
91,1582
+    def info::BioVcf::VcfRecord#info
95,1636
+    def format::BioVcf::VcfRecord#format
99,1711
+    def normal::BioVcf::VcfRecord#normal
104,1848
+    def tumor::BioVcf::VcfRecord#tumor
109,1997
+    def sample::BioVcf::VcfRecord#sample
114,2134
+    def sample_by_name::BioVcf::VcfRecord#sample_by_name
118,2227
+    def missing_samples?::BioVcf::VcfRecord#missing_samples?
122,2283
+    def method_missing::BioVcf::VcfRecord#method_missing
126,2341
+
+./lib/bio-vcf/variant.rb,470
+module BioVcf::BioVcf
1,0
+  module Variant::BioVcf::Variant
3,17
+    def Variant.diff::BioVcf::Variant.diff
5,37
+    def Variant.threshold_diff::BioVcf::Variant.threshold_diff
9,132
+    def Variant.relative_diff::BioVcf::Variant.relative_diff
14,269
+    def Variant.relative_threshold_diff::BioVcf::Variant.relative_threshold_diff
20,497
+    def Variant.index::BioVcf::Variant.index
25,652
+    def Variant.apply_threshold::BioVcf::Variant.apply_threshold
31,809
+
+./lib/bio-vcf/vcfheader.rb,598
+module BioVcf::BioVcf
2,1
+  module VcfHeaderParser::BioVcf::VcfHeaderParser
4,18
+    def VcfHeaderParser.get_column_names::BioVcf::VcfHeaderParser.get_column_names
5,45
+  class VcfHeader::BioVcf::VcfHeader
18,339
+    attr_reader :lines::BioVcf::VcfHeader#lines
20,360
+    def initialize::BioVcf::VcfHeader#BioVcf::VcfHeader.new
22,384
+    def add::BioVcf::VcfHeader#add
26,430
+    def version::BioVcf::VcfHeader#version
30,483
+    def column_names::BioVcf::VcfHeader#column_names
34,578
+    def columns::BioVcf::VcfHeader#columns
38,674
+    def samples::BioVcf::VcfHeader#samples
42,735
+
+./features/step_definitions/diff_count.rb,0
+
+./features/step_definitions/bio-vcf_steps.rb,0
+
+./features/step_definitions/somaticsniper.rb,0
+
+./features/step_definitions/multisample.rb,0
+
+./features/support/env.rb,0