bio-vcf 0.9.2 → 0.9.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: a09729e3548751923f4b3c5ef81c8c9d7402b6b2
-  data.tar.gz: 4c525ad745c5486075e9a0f14fe5372a21c8f056
+SHA256:
+  metadata.gz: f5d7a81871906abfffc93455b4d664d5755fe8d79312134eae94e84659506198
+  data.tar.gz: 8029269859aedd53c613ea9bbb17f951972b062060b5a40c22bdbe65c6c3dfa7
 SHA512:
-  metadata.gz: 343083ee8c055f534a840c8f668cb35a0c33fbccabe2b580edf859747aff8c8069266168ac66631bc3bbd2c8f58691847796eb00cef7784c7ebf966ec85e1d4f
-  data.tar.gz: f55292d0d744a496a5b39123285904a120f4c4ffae066dc33f244e09ae021618e94071cf8e587f56debfaf0c54233c3a5688887e21b21f5640bc8b271a3a00bb
+  metadata.gz: ed231c3a918e5f9ab9cd8a618f3f25f0c39613ac934b496af334d77dabe64831ff08cfc722a467fc51ab8c583358ca21be769ba1d9654437d54e7d21b811ee2c
+  data.tar.gz: df49786c4f4aa5e3a3659c678fb66aeb4b7dd4bb575aacf34cc468663c18fa893502699d238b5034c308ff51a4dc05e0fadf929b923c1d646f61c3f07fef26c7

data/.travis.yml

@@ -1,23 +1,3 @@
-sudo: false  # required for the new containers
-
 language: ruby
-rvm:
-#  - 1.9.3 <- No longer working
-  - 2.1.0
-  - 2.2.3
-
-# install: 
-#   - gem install cucumber rspec regressiontest
-
-branches:
-  only: 
-    - master
-
-#  - jruby-head
-#  - jruby-19mode # JRuby in 1.9 mode
-#  - 1.8.7
-#  - jruby-18mode # JRuby in 1.8 mode
-#  - rbx-18mode
 
-# uncomment this line if your project needs to run something other than `rake`:
-# script: bundle exec rspec spec
+arch: arm64

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2013 Pjotr Prins
+Copyright (c) 2013-2020 Pjotr Prins <pjotr.public68@thebird.nl>
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,23 +1,15 @@
 # bio-vcf
 
-[![Build Status](https://secure.travis-ci.org/pjotrp/bioruby-vcf.png)](http://travis-ci.org/pjotrp/bioruby-vcf) 
+[![Build Status](https://secure.travis-ci.org/vcflib/bio-vcf.png)](http://travis-ci.org/vcflib/bio-vcf)
 
-## Updates
-
-* Getting ready for a 1.0 release
-* 0.9.1 removed a rare threading bug and cleanup on error
-* Added support for soft filters (request by Brad Chapman)
-* The outputter now writes (properly) in parallel with the parser
-* bio-vcf turns any VCF into JSON with header information, and
-  allows you to pipe that JSON directly into any JSON supporting
-  language, including Python and Javascript!
 
 ## Bio-vcf
 
-Bio-vcf is a new generation VCF parser, filter and converter. Bio-vcf is not only
-very fast for genome-wide (WGS) data, it also comes with a really nice
-filtering, evaluation and rewrite language and it can output any type
-of textual data, including VCF header and contents in RDF and JSON.
+Bio-vcf is a new generation VCF parser, filter and converter. Bio-vcf
+is not only very fast for genome-wide (WGS) data, it also comes with a
+really nice filtering, evaluation and rewrite language and it can
+output any type of textual data, including VCF header and contents in
+RDF and JSON.
 
 So, why would you use bio-vcf over other parsers? Because
 
@@ -79,18 +71,18 @@ BED format on a 16 core machine takes
   sys     0m5.039s
 ```
 
-which shows decent core utilisation (10x). Running 
+which shows decent core utilisation (10x). Running
 gzip compressed VCF files of 30+ Gb has similar performance gains.
 
 To view some complex filters on an 80Gb SNP file check out a
-[GTEx exercise](https://github.com/pjotrp/bioruby-vcf/blob/master/doc/GTEx_reduce.md).
+[GTEx exercise](https://github.com/vcflib/bio-vcf/blob/master/doc/GTEx_reduce.md).
 
 Use zcat (or even better pigz which is multi-core itself) to pipe such
 gzipped (vcf.gz) files into bio-vcf, e.g.
 
 ```sh
   zcat huge_file.vcf.gz| bio-vcf --num-threads 36 --filter 'r.chrom.to_i>0 and r.chrom.to_i<21 and r.qual>50'
-    --sfilter '!s.empty? and s.dp>20' 
+    --sfilter '!s.empty? and s.dp>20'
     --eval '[r.chrom,r.pos,r.pos+1]' > test.bed
 ```
 
@@ -124,7 +116,7 @@ Where 's.dp' is the shorter name for 'sample.dp'.
 
 It is also possible to specify sample names, or info fields:
 
-For example, to filter somatic data 
+For example, to filter somatic data
 
 ```ruby
   bio-vcf --filter 'rec.info.dp>5 and rec.alt.size==1 and rec.tumor.bq[rec.alt]>30 and rec.tumor.mq>20' < file.vcf
@@ -252,7 +244,7 @@ The VCF format is commonly used for variant calling between NGS
 samples. The fast parser needs to carry some state, recorded for each
 file in VcfHeader, which contains the VCF file header. Individual
 lines (variant calls) first go through a raw parser returning an array
-of fields. Further (lazy) parsing is handled through VcfRecord. 
+of fields. Further (lazy) parsing is handled through VcfRecord.
 
 At this point the filter is pretty generic with multi-sample support.
 If something is not working, check out the feature descriptions and
@@ -261,17 +253,16 @@ example of a VCF statement you need to work on.
 
 ## Installation
 
-Note that you need Ruby 2.x or later. The 2.x Ruby series also give
-a performance improvement. Bio-vcf will show the Ruby version when
-typing the command 'bio-vcf -h'.
+The bio-vcf has no other dependencies but Ruby.
 
-To intall bio-vcf with gem:
+To install bio-vcf with Ruby gems:
 
 ```sh
 gem install bio-vcf
 bio-vcf -h
 ```
 
+
 ## Command line interface (CLI)
 
 Get the version of the VCF file
@@ -295,6 +286,13 @@ Get the sample names
   NORMAL,TUMOR
 ```
 
+Alternatively use the command line switch for --names, e.g.
+
+```ruby
+  bio-vcf --names < file.vcf
+  NORMAL,TUMOR
+```
+
 Get information from the header (META)
 
 ```ruby
@@ -305,39 +303,39 @@ The 'fields' array contains unprocessed data (strings).  Print first
 five raw fields
 
 ```ruby
-  bio-vcf --eval 'fields[0..4]' < file.vcf 
+  bio-vcf --eval 'fields[0..4]' < file.vcf
 ```
 
 Add a filter to display the fields on chromosome 12
 
 ```ruby
-  bio-vcf --filter 'fields[0]=="12"' --eval 'fields[0..4]' < file.vcf 
+  bio-vcf --filter 'fields[0]=="12"' --eval 'fields[0..4]' < file.vcf
 ```
 
 It gets better when we start using processed data, represented by an
 object named 'rec'. Position is a value, so we can filter a range
 
 ```ruby
-  bio-vcf --filter 'rec.chrom=="12" and rec.pos>96_641_270 and rec.pos<96_641_276' < file.vcf 
+  bio-vcf --filter 'rec.chrom=="12" and rec.pos>96_641_270 and rec.pos<96_641_276' < file.vcf
 ```
 
 The shorter name for 'rec.chrom' is 'r.chrom', so you may write
 
 ```ruby
-  bio-vcf --filter 'r.chrom=="12" and r.pos>96_641_270 and r.pos<96_641_276' < file.vcf 
+  bio-vcf --filter 'r.chrom=="12" and r.pos>96_641_270 and r.pos<96_641_276' < file.vcf
 ```
 
 To ignore and continue parsing on missing data use the
 --ignore-missing (-i) and or --quiet (-q) switches
 
 ```ruby
-  bio-vcf -i --filter 'r.chrom=="12" and r.pos>96_641_270 and r.pos<96_641_276' < file.vcf 
+  bio-vcf -i --filter 'r.chrom=="12" and r.pos>96_641_270 and r.pos<96_641_276' < file.vcf
 ```
 
 Info fields are referenced by
 
 ```ruby
-  bio-vcf --filter 'rec.info.dp>100 and rec.info.readposranksum<=0.815' < file.vcf 
+  bio-vcf --filter 'rec.info.dp>100 and rec.info.readposranksum<=0.815' < file.vcf
 ```
 
 (alternatively you can use the indexed rec.info['DP'] and list INFO fields with
@@ -346,14 +344,14 @@ rec.info.fields).
 Subfields defined by rec.format:
 
 ```ruby
-  bio-vcf --filter 'rec.tumor.ss != 2' < file.vcf 
+  bio-vcf --filter 'rec.tumor.ss != 2' < file.vcf
 ```
 
 Output
 
 ```ruby
-  bio-vcf --filter 'rec.tumor.gq>30' 
-    --eval '[rec.ref,rec.alt,rec.tumor.bcount,rec.tumor.gq,rec.normal.gq]' 
+  bio-vcf --filter 'rec.tumor.gq>30'
+    --eval '[rec.ref,rec.alt,rec.tumor.bcount,rec.tumor.gq,rec.normal.gq]'
     < file.vcf
 ```
 
@@ -367,26 +365,26 @@ Show the count of the bases that were scored as somatic
 Actually, we have a convenience implementation for bcount, so this is the same
 
 ```ruby
-  bio-vcf --eval 'rec.alt+"\t"+rec.tumor.bcount[rec.alt].to_s+"\t"+rec.tumor.gq.to_s' 
+  bio-vcf --eval 'rec.alt+"\t"+rec.tumor.bcount[rec.alt].to_s+"\t"+rec.tumor.gq.to_s'
     < file.vcf
 ```
 
 Filter on the somatic results that were scored at least 4 times
- 
+
 ```ruby
-  bio-vcf --filter 'rec.alt.size==1 and rec.tumor.bcount[rec.alt]>4' < test.vcf 
+  bio-vcf --filter 'rec.alt.size==1 and rec.tumor.bcount[rec.alt]>4' < test.vcf
 ```
 
 Similar for base quality scores
 
 ```ruby
-  bio-vcf --filter 'rec.alt.size==1 and rec.tumor.amq[rec.alt]>30' < test.vcf 
+  bio-vcf --filter 'rec.alt.size==1 and rec.tumor.amq[rec.alt]>30' < test.vcf
 ```
 
 Filter out on sample values
 
 ```ruby
-  bio-vcf --sfilter 's.dp>20' < test.vcf 
+  bio-vcf --sfilter 's.dp>20' < test.vcf
 ```
 
 To filter missing on samples:
@@ -468,17 +466,17 @@ Even shorter r is an alias for rec
 Note: special functions are not yet implemented! Look below
 for genotype processing which has indexing in 'gti'.
 
-Sometime you want to use a special function in a filter. For 
-example percentage variant reads can be defined as [a,c,g,t] 
-with frequencies against sample read depth (dp) as 
-[0,0.03,0.47,0.50]. Filtering would with a special function, 
+Sometime you want to use a special function in a filter. For
+example percentage variant reads can be defined as [a,c,g,t]
+with frequencies against sample read depth (dp) as
+[0,0.03,0.47,0.50]. Filtering would with a special function,
 which we named freq
 
 ```sh
   bio-vcf --sfilter "s.freq(2)>0.30" < file.vcf
 ```
 
-which is equal to 
+which is equal to
 
 ```sh
   bio-vcf --sfilter "s.freq.g>0.30" < file.vcf
@@ -498,7 +496,7 @@ ref should always be identical across samples.
 
 ## DbSNP
 
-One clinical variant DbSNP example 
+One clinical variant DbSNP example
 
 ```sh
     bio-vcf --eval '[rec.id,rec.chr,rec.pos,rec.alt,rec.info.sao,rec.info.CLNDBN]' < clinvar_20140303.vcf
@@ -523,16 +521,16 @@ renders
 
 bio-vcf allows for set analysis. With the complement filter, for
 example, samples are selected that evaluate to true, all others should
-evaluate to false. For this we create three filters, one for all 
+evaluate to false. For this we create three filters, one for all
 samples that are included (the --ifilter or -if), for all samples that
 are excluded (the --efilter or -ef) and for any sample (the --sfilter
 or -sf). So i=include (OR filter), e=exclude and s=any sample (AND
-filter). 
+filter).
 
 The equivalent of the union filter is by using the --sfilter, so
 
 ```sh
-  bio-vcf --sfilter 's.dp>20' 
+  bio-vcf --sfilter 's.dp>20'
 ```
 
 Filters DP on all samples and is true if all samples match the
@@ -540,7 +538,7 @@ criterium (AND). To filter on a subset you can add a
 selector
 
 ```sh
-  bio-vcf --sfilter-samples 0,1,4 --sfilter 's.dp>20' 
+  bio-vcf --sfilter-samples 0,1,4 --sfilter 's.dp>20'
 ```
 
 For set analysis there are the additional ifilter (include) and
@@ -560,7 +558,7 @@ values
 
 The equivalent of the complement filter is by specifying what samples
 to include, here with a regex and define filters on the included
- and excluded samples (the ones not in ifilter-samples) and the 
+ and excluded samples (the ones not in ifilter-samples) and the
 
 ```sh
   ./bin/bio-vcf -i --sfilter 's.dp>20' --ifilter-samples 2,4 --ifilter 's.gt==r.s1t1.gt'
@@ -581,7 +579,7 @@ To print out the GT's add --seval
 To set an additional filter on the excluded samples:
 
 ```sh
-  bio-vcf -i --ifilter-samples 0,1,4 --ifilter 's.gt==rec.s1t1.gt and s.gq>10' --seval s.gq --efilter 's.gq==99' 
+  bio-vcf -i --ifilter-samples 0,1,4 --ifilter 's.gt==rec.s1t1.gt and s.gq>10' --seval s.gq --efilter 's.gq==99'
 ```
 
 Etc. etc. Any combination of sfilter, ifilter and efilter is possible.
@@ -594,15 +592,15 @@ In the near future it is also possible to select samples on a regex (here
 select all samples where the name starts with s3)
 
 ```sh
-  bio-vcf --isample-regex '/^s3/' --ifilter 's.dp>20' 
+  bio-vcf --isample-regex '/^s3/' --ifilter 's.dp>20'
 ```
 
 ```sh
-  bio-vcf --include /s3.+/ --sfilter 'dp>20'  --ifilter 'gt==s3t1.gt' --efilter 'gt!=s3t1.gt' 
+  bio-vcf --include /s3.+/ --sfilter 'dp>20'  --ifilter 'gt==s3t1.gt' --efilter 'gt!=s3t1.gt'
 --set-intersect  include=true
-  bio-vcf --include /s3.+/ --sample-regex /^t2/ --sfilter 'dp>20'  --ifilter 'gt==s3t1.gt'  
+  bio-vcf --include /s3.+/ --sample-regex /^t2/ --sfilter 'dp>20'  --ifilter 'gt==s3t1.gt'
 --set-catesian   one in include=true, rest=false
-  bio-vcf --unique-sample (any) --include /s3.+/ --sfilter 'dp>20' --ifilter 'gt!="0/0"'  
+  bio-vcf --unique-sample (any) --include /s3.+/ --sfilter 'dp>20' --ifilter 'gt!="0/0"'
 ```
 
 With the filter commands you can use --ignore-missing to skip errors.
@@ -625,7 +623,7 @@ results in a string value
 to access components of the genotype field we can use standard Ruby
 
 ```ruby
-  bio-vcf --seval 's.gt.split(/\//)[0]' 
+  bio-vcf --seval 's.gt.split(/\//)[0]'
     1       10665   .     .     0     0     .     0     0
     1       10694   .     .     1     1     .     .     .
     1       12783   0     0     0     0     0     0     0
@@ -636,7 +634,7 @@ or special functions, such as 'gti' which gives the genotype as an
 indexed value array
 
 ```ruby
-  bio-vcf --seval 's.gti[0]' 
+  bio-vcf --seval 's.gti[0]'
     1       10665                   0       0               0       0
     1       10694                   1       1
     1       12783   0       0       0       0       0       0       0
@@ -646,7 +644,7 @@ indexed value array
 and 'gts' as a nucleotide string array
 
 ```ruby
-  bio-vcf --seval 's.gts' 
+  bio-vcf --seval 's.gts'
     1       10665                   C       C               C       C
     1       10694                   G       G
     1       12783   G       G       G       G       G       G       G
@@ -670,9 +668,9 @@ example signficance, use
 Now you can index other fields, e.g. GL
 
 ```ruby
-    ./bin/bio-vcf --seval '[(!s.empty? ? s.gl[s.gtindex]:-1)]' 
+    ./bin/bio-vcf --seval '[(!s.empty? ? s.gl[s.gtindex]:-1)]'
     1       900057  1.0     1.0     0.994   1.0     1.0     -1      0.999   1.0     0.997   -1  0.994    0.989   -1      0.991   -1      0.972   0.992   1.0
-    ```
+```
 
 shows a number of SNPs have been scored with high significance and a
 number are missing, here marked as -1.
@@ -741,6 +739,17 @@ To remove/select 3 samples:
   bio-vcf --samples 0,1,3 < mytest.vcf
 ```
 
+You can also select samples by name (as long as they do not contain
+spaces)
+
+
+```sh
+  bio-vcf --names < mytest.vcf
+    Original        s1t1    s2t1    s3t1    s1t2    s2t2    s3t2
+  bio-vcf --samples "Original,s1t1,s3t1" < mytest.vcf
+```
+
+
 Filter on a BED file and annotate the gene name in the resulting VCF
 
 ```sh
@@ -791,7 +800,7 @@ To have more output options bio-vcf can use an [ERB
 template](http://www.stuartellis.eu/articles/erb/) for every match. This is a
 very flexible option that can output textual formats such as JSON, YAML, HTML
 and RDF. Examples are provided in
-[./templates](https://github.com/pjotrp/bioruby-vcf/templates/). A JSON
+[./templates](https://github.com/vcflib/bio-vcf/templates/). A JSON
 template could be
 
 ```Javascript
@@ -805,7 +814,7 @@ template could be
 };
 ```
 
-To get JSON, run with something like (combining 
+To get JSON, run with something like (combining
 with a filter)
 
 ```sh
@@ -831,11 +840,11 @@ Likewise for RDF output:
   bio-vcf --template template/vcf2rdf.erb --filter 'r.info.sao==1' < dbsnp.vcf
 ```
 
-renders the ERB template 
+renders the ERB template
 
 ```ruby
 <%
-  id = Turtle::mangle_identifier(['ch'+rec.chrom,rec.pos,rec.alt.join('')].join('_')) 
+  id = Turtle::mangle_identifier(['ch'+rec.chrom,rec.pos,rec.alt.join('')].join('_'))
 %>
 :<%= id %>
   :query_id    "<%= id %>",
@@ -848,7 +857,7 @@ renders the ERB template
   db:vcf       true .
 ```
 
-into 
+into
 
 ```
 :ch13_33703698_A
@@ -936,9 +945,9 @@ To get and put the full information from the header, simple use
 vcf.meta.to_json.  See ./template/vcf2json_full_header.erb for an
 example. This meta information can also be used to output info fields
 and sample values on the fly! For an example, see the template at
-[./template/vcf2json_use_meta.erb](https://github.com/pjotrp/bioruby-vcf/tree/master/template/vcf2json_use_meta.erb)
+[./template/vcf2json_use_meta.erb](https://github.com/vcflib/bio-vcf/tree/master/template/vcf2json_use_meta.erb)
 and the generated output at
-[./test/data/regression/vcf2json_use_meta.ref](https://github.com/pjotrp/bioruby-vcf/tree/master/test/data/regression/vcf2json_use_meta.ref).
+[./test/data/regression/vcf2json_use_meta.ref](https://github.com/vcflib/bio-vcf/tree/master/test/data/regression/vcf2json_use_meta.ref).
 
 This way, it is possible to write templates that can convert the content of
 *any* VCF file without prior knowledge to JSON, RDF, etc.
@@ -955,7 +964,7 @@ Simple statistics are available for REF>ALT changes:
       G>A             59      45%
       C>T             30      23%
       A>G              5       4%
-      C>G              5       4% 
+      C>G              5       4%
       C>A              5       4%
       G>T              4       3%
       T>C              4       3%
@@ -976,9 +985,9 @@ Simple statistics are available for REF>ALT changes:
 ## Other examples
 
 For more exercises and examples see
-[doc](https://github.com/pjotrp/bioruby-vcf/tree/master/doc) directory
+[doc](https://github.com/vcflib/bio-vcf/tree/master/doc) directory
 and the the feature
-[section](https://github.com/pjotrp/bioruby-vcf/tree/master/features).
+[section](https://github.com/vcflib/bio-vcf/tree/master/features).
 
 ## API
 
@@ -1009,6 +1018,23 @@ what the command line interface uses (see ./bin/bio-vcf)
   end
 ```
 
+### VCFFile
+
+The class ```BioVcf::VCFfile``` wraps a file and provides an ```enum``` with the
+method each, that can be used as in iterator.
+
+```ruby
+vcf_file = "dbsnp.vcf"
+vcf  = BioVcf::VCFfile.new(file:file, is_gz: false )
+it vcf.each
+puts it.peek
+
+vcf_file = "dbsnp.vcf.gz"
+vcf  = BioVcf::VCFfile.new(file:file, is_gz: true )
+it vcf.each
+puts it.peek
+```
+
 ## Trouble shooting
 
 ### MRI supports threading
@@ -1037,7 +1063,7 @@ For more complex filters use lambda inside a conditional
 ```ruby
     ( fast_check ? lambda { slow_check }.call : false )
 ```
-    
+
 where slow_check is the slow section of your query. As is shown
 earlier in this document. Don't forget the .call!
 
@@ -1056,6 +1082,15 @@ For larger files set the timeout to 600, or so. --timeout 600.
 
 Different values may show different core use on a machine.
 
+### Development
+
+To run the tests from source
+
+```sh
+bundle install --path vendor/bundle
+bundle exec rake
+```
+
 ### Debugging
 
 To debug output use '-v --num-threads=1' for generating useful
@@ -1073,12 +1108,12 @@ temporary directory may remain.
 Information on the source tree, documentation, examples, issues and
 how to contribute, see
 
-  http://github.com/pjotrp/bioruby-vcf
+  http://github.com/vcflib/bio-vcf
 
 ## Cite
 
 If you use this software, please cite one of
-  
+
 * [BioRuby: bioinformatics software for the Ruby programming language](http://dx.doi.org/10.1093/bioinformatics/btq475)
 * [Biogem: an effective tool-based approach for scaling up open source software development in bioinformatics](http://dx.doi.org/10.1093/bioinformatics/bts080)
 
@@ -1088,5 +1123,4 @@ This Biogem is published at (http://biogems.info/index.html#bio-vcf)
 
 ## Copyright
 
-Copyright (c) 2014 Pjotr Prins. See LICENSE.txt for further details.
-
+Copyright (c) 2014-2020 Pjotr Prins. See LICENSE.txt for further details.