htslib 0.2.0 → 0.2.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 87b06f64c496150db76d689f2cd06ba61e8ff424511be781bf17abbdf5d92d6c
-  data.tar.gz: 10126e01757aafb8fbb18c4f5d2f58541511226ddf922ffb253d29ccaf6c2027
+  metadata.gz: ea47a473d63b90fd2b1db4ecd65aa1ef38d47bbe5624e24ac583b5329301e5cb
+  data.tar.gz: aab96376972d7b0a587978484b22e41ee049ea742cba937cd57c66a9c0c42f2a
 SHA512:
-  metadata.gz: 2bb5806ae23d58192a74c567767de193ed21f2b8f8f87cf2bef4ef9beccc62da684ce5a0b8a6a2b96afbdbd84fb13c14b27ea2738a4d70829b7d3c1a57396430
-  data.tar.gz: 9e49efbb3bdce645013e0aaa33dec335204a0c62f48cd0ab4b5128314b97f6237b5faec6a0acf0059d8a276e26323d93ad53c4b5e7e2c36b273592b970b7dfe4
+  metadata.gz: 76d9aa699d0176557d8e44171018b132fc4df2c0e1f47a9784388693103799cf920c2ae437fb97f8429422a1edb5ebd43eae7b28a1f18452a5eeeac38244a981
+  data.tar.gz: f2f1d50099a66e71574c4019e66452dd100371cbdc7d08b144856bab8f59f04aac4e207585534f26e29a06342a7fafe5c3ea0fb0d0e2e8dd9ed1822ddbcbdfe8

data/README.md

@@ -6,11 +6,11 @@
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
 [![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/) in the Ruby language.
+Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/), in the Ruby language.
 
 :apple: Feel free to fork it out if you can develop it! 
 
-:bowtie: alpha stage.
+:bowtie: Alpha stage.
 
 ## Requirements
 
@@ -83,6 +83,19 @@ end
 bcf.close
 ```
 
+<details>
+<summary><b>Faidx</b></summary>
+
+```ruby
+fa = HTS::Faidx.open("c.fa")
+
+fa.fetch("chr1:1-10")
+
+fa.close
+```
+
+</details>
+
 ### Low level API
 
 `HTS::LibHTS` provides native C functions. 
@@ -96,11 +109,11 @@ p b[:category]
 p b[:format]
 ```
 
-Note: htslib makes extensive use of macro functions for speed. you cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
+Note: htslib makes extensive use of macro functions for speed. You cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
 
 ### Need more speed?
 
-Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crsytal language is not as flexible as Ruby language. You can not use `eval` methods, and you must always be careful with the data types. Writing one-time scripts in Crystal or playing with REPL may not be as much fun. However, if you have a clear idea of what you want to do in your mind, have already written code in Ruby, and need to run them over and over, try creating a command line tool in Crystal.  The Crystal language is fast, as fast as the Rust and C languages. It will give you great power to create tools.
+Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crystal language is not as flexible as Ruby language. You can not use `eval` methods, and you must always be careful with the data types. Writing one-time scripts in Crystal or playing with REPL may not be as much fun. However, if you have a clear idea of what you want to do in your mind, have already written code in Ruby, and need to run them over and over, try creating a command line tool in Crystal.  The Crystal language is fast, as fast as the Rust and C languages. It will give you great power to create tools.
 
 ## Documentation
 
@@ -125,8 +138,8 @@ Many macro functions are used in HTSlib. Since these macro functions cannot be c
 
 * Use the new version of Ruby to take full advantage of Ruby's potential.
   * This is possible because we have a small number of users. What a deal!
-* Remain compatibile with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
-  * The most difficult part is the return value. In the Crystal language, methods are expected to return only one type. On the other hand, in the Ruby language, methods that return multiple classes are very common. For example, in the Crystal language, the compiler gets confused if the return value is one of six types: Int32, Int64, Float32, Float64, Nil, or String. In fact Crystal can do this. But the code gets a little messy. In Ruby, this is very common and doesn't cause any problems. 
+* Remain compatible with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
+  * The most challenging part is the return value. In the Crystal language, methods are expected to return only one type. On the other hand, in the Ruby language, methods that return multiple classes are very common. For example, in the Crystal language, the compiler gets confused if the return value is one of six types: Int32, Int64, Float32, Float64, Nil, or String. In fact Crystal can do this. But the code gets a little messy. In Ruby, this is very common and doesn't cause any problems. 
 
 In the script directory, there are several tools to help implement ruby-htslib. These tools may be forked into independent repository in the future.
 

data/lib/hts/bam/{aux.rb → auxi.rb}

@@ -1,5 +1,11 @@
 # frozen_string_literal: true
 
+# Q. Why is the file name auxi.rb and not aux.rb?
+#
+# A. This is for compatibility with Windows.
+# In Windows, aux is a reserved word
+# You cannot create a file named aux.
+
 module HTS
   class Bam < Hts
     # Auxiliary record data

data/lib/hts/bam/record.rb

@@ -2,7 +2,7 @@
 
 require_relative "flag"
 require_relative "cigar"
-require_relative "aux"
+require_relative "auxi"
 
 module HTS
   class Bam < Hts

data/lib/hts/bam.rb

@@ -13,7 +13,7 @@ module HTS
   class Bam
     include Enumerable
 
-    attr_reader :file_name, :index_name, :mode, :header
+    attr_reader :file_name, :index_name, :mode, :header, :nthreads
 
     def self.open(*args, **kw)
       file = new(*args, **kw) # do not yield
@@ -28,7 +28,7 @@ module HTS
     end
 
     def initialize(file_name, mode = "r", index: nil, fai: nil, threads: nil,
-                   create_index: false)
+                   build_index: false)
       if block_given?
         message = "HTS::Bam.new() dose not take block; Please use HTS::Bam.open() instead"
         raise message
@@ -39,6 +39,7 @@ module HTS
       @file_name  = file_name
       @index_name = index
       @mode       = mode
+      @nthreads   = threads
       @hts_file   = LibHTS.hts_open(@file_name, mode)
 
       raise Errno::ENOENT, "Failed to open #{@file_name}" if @hts_file.null?
@@ -53,21 +54,24 @@ module HTS
       return if @mode[0] == "w"
 
       @header = Bam::Header.new(@hts_file)
-      create_index(index) if create_index
+      build_index(index) if build_index
       @idx = load_index(index)
       @start_position = tell
       super # do nothing
     end
 
-    def create_index(index_name = nil)
+    def build_index(index_name = nil, min_shift: 0)
       check_closed
 
-      warn "Create index for #{@file_name} to #{index_name}"
-      if index
-        LibHTS.sam_index_build2(@file_name, index_name, -1)
+      if index_name
+        warn "Create index for #{@file_name} to #{index_name}"
       else
-        LibHTS.sam_index_build(@file_name, -1)
+        warn "Create index for #{@file_name}"
       end
+      r = LibHTS.sam_index_build3(@file_name, index_name, min_shift, @nthreads)
+      raise "Failed to build index for #{@file_name}" if r < 0
+
+      self
     end
 
     def load_index(index_name = nil)
@@ -139,23 +143,51 @@ module HTS
       self
     end
 
-    # query [WIP]
-    def query(region)
+    def query(region, copy: false, &block)
+      if copy
+        query_copy(region, &block)
+      else
+        query_reuse(region, &block)
+      end
+    end
+
+    private def query_copy(region)
       check_closed
       raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
 
       qiter = LibHTS.sam_itr_querys(@idx, header, region)
+
       begin
-        bam1 = LibHTS.bam_init1
-        slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
-        while slen > 0
-          yield Record.new(bam1, header)
+        loop do
           bam1 = LibHTS.bam_init1
           slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
+          break if slen == -1
+          raise if slen < -1
+
+          yield Record.new(bam1, header)
         end
       ensure
         LibHTS.hts_itr_destroy(qiter)
       end
+      self
+    end
+
+    private def query_reuse(region)
+      check_closed
+      raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
+
+      qiter = LibHTS.sam_itr_querys(@idx, header, region)
+
+      bam1 = LibHTS.bam_init1
+      record = Record.new(bam1, header)
+      begin
+        yield record while LibHTS.sam_itr_next(@hts_file, qiter, bam1) > 0
+      ensure
+        LibHTS.hts_itr_destroy(qiter)
+      end
+      self
     end
 
     # @!macro [attach] define_getter

data/lib/hts/bcf/header_record.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module HTS
+  class Bcf < Hts
+    class HeaderRecord
+      def initialize
+        @bcf_hrec
+      end
+    end
+  end
+end

data/lib/hts/bcf/record.rb

@@ -137,7 +137,7 @@ module HTS
 
       private
 
-      def initialize_copy(orig)\
+      def initialize_copy(orig)
         @header = orig.header
         @bcf1 = LibHTS.bcf_dup(orig.struct)
       end

data/lib/hts/bcf.rb

@@ -13,7 +13,7 @@ module HTS
   class Bcf < Hts
     include Enumerable
 
-    attr_reader :file_name, :index_name, :mode, :header
+    attr_reader :file_name, :index_name, :mode, :header, :nthreads
 
     def self.open(*args, **kw)
       file = new(*args, **kw) # do not yield
@@ -28,7 +28,7 @@ module HTS
     end
 
     def initialize(file_name, mode = "r", index: nil, threads: nil,
-                   create_index: false)
+                   build_index: false)
       if block_given?
         message = "HTS::Bcf.new() dose not take block; Please use HTS::Bcf.open() instead"
         raise message
@@ -39,6 +39,7 @@ module HTS
       @file_name  = file_name
       @index_name = index
       @mode       = mode
+      @nthreads   = threads
       @hts_file   = LibHTS.hts_open(@file_name, mode)
 
       raise Errno::ENOENT, "Failed to open #{@file_name}" if @hts_file.null?
@@ -48,21 +49,24 @@ module HTS
       return if @mode[0] == "w"
 
       @header = Bcf::Header.new(@hts_file)
-      create_index(index) if create_index
+      build_index(index) if build_index
       @idx = load_index(index)
       @start_position = tell
       super # do nothing
     end
 
-    def create_index(index_name = nil)
+    def build_index(index_name = nil, min_shift: 14)
       check_closed
 
-      warn "Create index for #{@file_name} to #{index_name}"
       if index_name
-        LibHTS.bcf_index_build2(@file_name, index_name, -1)
+        warn "Create index for #{@file_name} to #{index_name}"
       else
-        LibHTS.bcf_index_build(@file_name, -1)
+        warn "Create index for #{@file_name}"
       end
+      r = LibHTS.bcf_index_build3(@file_name, index_name, min_shift, @nthreads)
+      raise "Failed to build index for #{@file_name}" if r < 0
+
+      self
     end
 
     def load_index(index_name = nil)
@@ -132,8 +136,7 @@ module HTS
 
     private def each_record_reuse
       check_closed
-      # Each does not always start at the beginning of the file.
-      # This is the common behavior of IO objects in Ruby.
+
       return to_enum(__method__) unless block_given?
 
       bcf1 = LibHTS.bcf_init
@@ -142,6 +145,76 @@ module HTS
       self
     end
 
+    def query(...)
+      querys(...) # Fixme
+    end
+
+    # def queryi
+    # end
+
+    def querys(region, copy: false, &block)
+      if copy
+        querys_copy(region, &block)
+      else
+        querys_reuse(region, &block)
+      end
+    end
+
+    # private def queryi_copy
+    # end
+
+    # private def queryi_reuse
+    # end
+
+    private def querys_copy(region)
+      check_closed
+
+      raise "query is only available for BCF files" unless file_format == "bcf"
+      raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
+
+      qitr = LibHTS.bcf_itr_querys(@idx, header, region)
+
+      begin
+        loop do
+          bcf1 = LibHTS.bcf_init
+          slen = LibHTS.hts_itr_next(@hts_file[:fp][:bgzf], qitr, bcf1, ::FFI::Pointer::NULL)
+          break if slen == -1
+          raise if slen < -1
+
+          yield Record.new(bcf1, header)
+        end
+      ensure
+        LibHTS.bcf_itr_destroy(qitr)
+      end
+      self
+    end
+
+    private def querys_reuse(region)
+      check_closed
+
+      raise "query is only available for BCF files" unless file_format == "bcf"
+      raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
+
+      qitr = LibHTS.bcf_itr_querys(@idx, header, region)
+
+      bcf1 = LibHTS.bcf_init
+      record = Record.new(bcf1, header)
+      begin
+        loop do
+          slen = LibHTS.hts_itr_next(@hts_file[:fp][:bgzf], qitr, bcf1, ::FFI::Pointer::NULL)
+          break if slen == -1
+          raise if slen < -1
+
+          yield record
+        end
+      ensure
+        LibHTS.bcf_itr_destroy(qitr)
+      end
+      self
+    end
+
     # @!macro [attach] define_getter
     #   @method $1
     #   Get $1 array
@@ -197,7 +270,7 @@ module HTS
 
     def each_info(key)
       check_closed
-      return to_enum(__method__) unless block
+      return to_enum(__method__, key) unless block
 
       each do |r|
         yield r.info(key)
@@ -206,7 +279,7 @@ module HTS
 
     def each_format(key)
       check_closed
-      return to_enum(__method__) unless block
+      return to_enum(__method__, key) unless block
 
       each do |r|
         yield r.format(key)

data/lib/hts/faidx.rb

@@ -38,6 +38,11 @@ module HTS
       LibHTS.fai_destroy(@fai)
     end
 
+    # FIXME: This doesn't seem to work as expected
+    # def closed?
+    #   @fai.null?
+    # end
+
     # the number of sequences in the index.
     def length
       LibHTS.faidx_nseq(@fai)
@@ -46,10 +51,7 @@ module HTS
 
     # return the length of the requested chromosome.
     def chrom_size(chrom)
-      unless chrom.is_a?(String) || chrom.is_a?(Symbol)
-        # FIXME
-        raise ArgumentError, "Expect chrom to be String or Symbol"
-      end
+      raise ArgumentError, "Expect chrom to be String or Symbol" unless chrom.is_a?(String) || chrom.is_a?(Symbol)
 
       chrom = chrom.to_s
       result = LibHTS.faidx_seq_len(@fai, chrom)
@@ -57,12 +59,41 @@ module HTS
     end
     alias chrom_length chrom_size
 
-    # FIXME: naming and syntax
-    # def cget; end
+    # return the length of the requested chromosome.
+    def chrom_names
+      Array.new(length) { |i| LibHTS.faidx_iseq(@fai, i) }
+    end
+
+    # @overload fetch(name)
+    #   Fetch the sequence as a String.
+    #   @param name [String] chr1:0-10
+    # @overload fetch(name, start, stop)
+    #   Fetch the sequence as a String.
+    #   @param name [String] the name of the chromosome
+    #   @param start [Integer] the start position of the sequence (0-based)
+    #   @param stop [Integer] the end position of the sequence (0-based)
+    #   @return [String] the sequence
+
+    def seq(name, start = nil, stop = nil)
+      name = name.to_s
+      rlen = FFI::MemoryPointer.new(:int)
 
-    # FIXME: naming and syntax
-    # def get; end
+      if start.nil? && stop.nil?
+        result = LibHTS.fai_fetch(@fai, name, rlen)
+      else
+        start < 0    && raise(ArgumentError, "Expect start to be >= 0")
+        stop  < 0    && raise(ArgumentError, "Expect stop to be >= 0")
+        start > stop && raise(ArgumentError, "Expect start to be <= stop")
 
-    # __iter__
+        result = LibHTS.faidx_fetch_seq(@fai, name, start, stop, rlen)
+      end
+
+      case rlen.read_int
+      when -2 then raise "Invalid chromosome name: #{name}"
+      when -1 then raise "Error fetching sequence: #{name}:#{start}-#{stop}"
+      end
+
+      result
+    end
   end
 end

data/lib/hts/hts.rb

@@ -69,13 +69,18 @@ module HTS
       @hts_file.nil? || @hts_file.null?
     end
 
-    def set_threads(n)
-      raise TypeError unless n.is_a(Integer)
-
-      if n > 0
-        r = LibHTS.hts_set_threads(@hts_file, n)
-        raise "Failed to set number of threads: #{threads}" if r < 0
+    def set_threads(n = nil)
+      if n.nil?
+        require "etc"
+        n = [Etc.nprocessors - 1, 1].max
       end
+      raise TypeError unless n.is_a?(Integer)
+      raise ArgumentError, "Number of threads must be positive" if n < 1
+
+      r = LibHTS.hts_set_threads(@hts_file, n)
+      raise "Failed to set number of threads: #{threads}" if r < 0
+
+      @nthreads = n
       self
     end
 

data/lib/hts/libhts/constants.rb

@@ -195,7 +195,7 @@ module HTS
         :reg,            :string,
         :intervals,      :pointer, # hts_pair_pos_t
         :tid,            :int,
-        :count,          :uint32_t,
+        :count,          :uint32,
         :min_beg,        :hts_pos_t,
         :max_end,        :hts_pos_t
     end
@@ -352,7 +352,7 @@ module HTS
         :qpos,           :int32_t,
         :indel,          :int,
         :level,          :int,
-        :_flags,         :uint32_t, # bit_fields
+        :_flags,         :uint32, # bit_fields
         :cd,             BamPileupCd,
         :cigar_ind,      :int
 
@@ -403,6 +403,7 @@ module HTS
         :n,              :int
     end
 
+    # Complete textual representation of a header line
     class BcfHrec < FFI::Struct
       layout \
         :type,           :int,
@@ -413,21 +414,6 @@ module HTS
         :vals,           :pointer
     end
 
-    class BcfFmt < FFI::BitStruct
-      layout \
-        :id,             :int,
-        :n,              :int,
-        :size,           :int,
-        :type,           :int,
-        :p,              :pointer, # uint8_t
-        :p_len,          :uint32,
-        :_p_off_free,    :uint32 # bit_fields
-
-      bit_fields :_p_off_free,
-                 :p_off,  31,
-                 :p_free, 1
-    end
-
     class BcfInfo < FFI::BitStruct
       layout \
         :key,            :int,
@@ -477,6 +463,21 @@ module HTS
         :m,              [:int, 3]
     end
 
+    class BcfFmt < FFI::BitStruct
+      layout \
+        :id,             :int,
+        :n,              :int,
+        :size,           :int,
+        :type,           :int,
+        :p,              :pointer, # uint8_t
+        :p_len,          :uint32,
+        :_p_off_free,    :uint32 # bit_fields
+
+      bit_fields :_p_off_free,
+                 :p_off,  31,
+                 :p_free, 1
+    end
+
     class BcfDec < FFI::Struct
       layout \
         :m_fmt,          :int,
@@ -505,8 +506,8 @@ module HTS
         :rlen,           :hts_pos_t,
         :rid,            :int32_t,
         :qual,           :float,
-        :_n_info_allele, :uint32_t,
-        :_n_fmt_sample,  :uint32_t,
+        :_n_info_allele, :uint32,
+        :_n_fmt_sample,  :uint32,
         :shared,         KString,
         :indiv,          KString,
         :d,              BcfDec,