htslib 0.1.0 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c90062954caa8dc2155f77ff3d03534d933ba81148f9bdb7c8dc173f73efca64
-  data.tar.gz: fb24ced637a8b9b897ecd9c5afed1c6f75902995e48c25daa849baa06611517b
+  metadata.gz: 93ee916dafed0ba04e486a59c81d2bd9befbceda3d679ef820e32d8817597afe
+  data.tar.gz: 7dda13879d0c3707354ab69fe9967f60f0300c1971011218a5cf79e5e211ee25
 SHA512:
-  metadata.gz: fed4e6689d48493416ebd409401df284fedaddb567d3558196b3ba3c253376dd1c4bea8db536fdb6ff8bbebe85512850d7d5c17605544ce26a09e814e7172eb5
-  data.tar.gz: b9d596e6445671254568b4bd3cc5b694a1011914ae67b1afc11aaa66a195f49b7371d001f9e10cc204a4c20e501cd8eb63092d63b9b392c701571a12843b5c91
+  metadata.gz: 8cf68e9276f353bdea4389f5584066870a4887f07891387cf143408cf215c43232f2c390c08b80c57225a8d8b3722a9dcc5ba5cc3cc716079cf004bd34df0da0
+  data.tar.gz: f30abf9c0d0dd07201e150dea5ec6976e7d968bc2f6bf2c347c1dec18cdfaaa5c141793208587f1629fa3cefeaaadd268083a105ab42dd450ce2ac8cfa0e2d7e

data/README.md

@@ -6,15 +6,15 @@
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
 [![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/) in the Ruby language.
+Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/), in the Ruby language.
 
 :apple: Feel free to fork it out if you can develop it! 
 
-:bowtie: alpha stage.
+:bowtie: Alpha stage.
 
 ## Requirements
 
-* [Ruby](https://github.com/ruby/ruby) 2.7 or above.
+* [Ruby](https://github.com/ruby/ruby) 3.1 or above.
 * [HTSlib](https://github.com/samtools/htslib)
   * Ubuntu : `apt install libhts-dev`
   * macOS : `brew install htslib`
@@ -27,7 +27,7 @@ gem install htslib
 ```
 
 If you have installed htslib with apt on Ubuntu or homebrew on Mac, [pkg-config](https://github.com/ruby-gnome/pkg-config) 
-will automatically detect the location of the shared library. 
+will automatically detect the location of the shared library. If pkg-config does not work well, set `PKG_CONFIG_PATH`.
 Alternatively, you can specify the directory of the shared library by setting the environment variable `HTSLIBDIR`.
 
 ```sh
@@ -83,9 +83,22 @@ end
 bcf.close
 ```
 
+<details>
+<summary><b>Faidx</b></summary>
+
+```ruby
+fa = HTS::Faidx.open("c.fa")
+
+fa.fetch("chr1:1-10")
+
+fa.close
+```
+
+</details>
+
 ### Low level API
 
-`HTS::LibHTS` provides native　C functions. 
+`HTS::LibHTS` provides native C functions. 
 
 ```ruby
 require 'htslib'
@@ -96,11 +109,11 @@ p b[:category]
 p b[:format]
 ```
 
-Note: htslib makes extensive use of macro functions for speed. you cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
+Note: htslib makes extensive use of macro functions for speed. You cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
 
 ### Need more speed?
 
-Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crsytal language is not as flexible as Ruby language. You can not use eval methods, and you must always be aware of the types. It is not very suitable for writing one-time scripts or experimenting with different code. However, If you have already written code in ruby-htslib, have a clear idea of the manipulations you want to do, and need to execute them many times, then by all means try to implement the command line tool using htslib.cr. The Crystal language is very fast and can perform almost as well as the Rust and C languages.
+Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crystal language is not as flexible as Ruby language. You can not use `eval` methods, and you must always be careful with the data types. Writing one-time scripts in Crystal or playing with REPL may not be as much fun. However, if you have a clear idea of what you want to do in your mind, have already written code in Ruby, and need to run them over and over, try creating a command line tool in Crystal.  The Crystal language is fast, as fast as the Rust and C languages. It will give you great power to create tools.
 
 ## Documentation
 
@@ -123,9 +136,12 @@ bundle exec rake test
 
 Many macro functions are used in HTSlib. Since these macro functions cannot be called using FFI, they must be reimplemented in Ruby.
 
-* Actively use the advanced features of Ruby.
-* Remain compatibile with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
-  * The most difficult part is the return value. In the Crystal language, it is convenient for a method to return only one type. In the Ruby language, on the other hand, it is more convenient to return multiple classes. For example, in the Crystal language, it is confusing that a return value can take four types: Int32, Float32, Nil, and String. In Ruby, on the other hand, it is very common and does not cause any problems.
+* Use the new version of Ruby to take full advantage of Ruby's potential.
+  * This is possible because we have a small number of users. What a deal!
+* Remain compatible with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
+  * The most challenging part is the return value. In the Crystal language, methods are expected to return only one type. On the other hand, in the Ruby language, methods that return multiple classes are very common. For example, in the Crystal language, the compiler gets confused if the return value is one of six types: Int32, Int64, Float32, Float64, Nil, or String. In fact Crystal can do this. But the code gets a little messy. In Ruby, this is very common and doesn't cause any problems. 
+
+In the script directory, there are several tools to help implement ruby-htslib. These tools may be forked into independent repository in the future.
 
 #### FFI Extensions
 
@@ -145,7 +161,10 @@ Ruby-htslib is a library under development, so even small improvements like typo
 * Suggest or add new features
 * [financial contributions](https://github.com/sponsors/kojix2)
 
+
 ```
+# Ownership and Commitment Rights
+
 Do you need commit rights to ruby-htslib repository?
 Do you want to get admin rights and take over the project?
 If so, please feel free to contact us @kojix2.
@@ -153,7 +172,7 @@ If so, please feel free to contact us @kojix2.
 
 #### Why do you implement htslib in a language like Ruby, which is not widely used in the bioinformatics?
 
-One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding myself to better understand how the pioneers of bioinformatics felt when establishing the file format and how they created their tools. And that effort is still going on today...
+One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding myself to better understand how the pioneers of bioinformatics felt when establishing the file format and how they created their tools. I hope one day we can work on bioinformatics using Ruby and Crystal languages, not to replace other languages such as Python and R, but to add new power and value to this advancing field.
 
 ## Links
 

data/lib/hts/bam/aux.rb

@@ -2,6 +2,7 @@
 
 module HTS
   class Bam < Hts
+    # Auxiliary record data
     class Aux
       def initialize(record)
         @record = record

data/lib/hts/bam/cigar.rb

@@ -2,16 +2,15 @@
 
 module HTS
   class Bam < Hts
+    # CIGAR string
     class Cigar
       include Enumerable
 
       def initialize(pointer, n_cigar)
-        @pointer = pointer
         @n_cigar = n_cigar
-      end
-
-      def to_ptr
-        @pointer
+        # Read the pointer before the memory is changed.
+        # Especially when called from a block of `each` iterator.
+        @c = pointer.read_array_of_uint32(n_cigar)
       end
 
       def to_s
@@ -21,8 +20,7 @@ module HTS
       def each
         return to_enum(__method__) unless block_given?
 
-        @n_cigar.times do |i|
-          c = @pointer[i].read_uint32
+        @c.each do |c|
           op =  LibHTS.bam_cigar_opchr(c)
           len = LibHTS.bam_cigar_oplen(c)
           yield [op, len]

data/lib/hts/bam/flag.rb

@@ -5,6 +5,7 @@
 
 module HTS
   class Bam < Hts
+    # SAM flags
     class Flag
       def initialize(flag_value)
         raise TypeError unless flag_value.is_a? Integer
@@ -52,6 +53,10 @@ module HTS
         (@value & f) != 0
       end
 
+      def to_i
+        @value
+      end
+
       def to_s
         LibHTS.bam_flag2str(@value)
         # "0x#{format('%x', @value)}\t#{@value}\t#{LibHTS.bam_flag2str(@value)}"

data/lib/hts/bam/header.rb

@@ -2,6 +2,7 @@
 
 module HTS
   class Bam < Hts
+    # A class for working with alignment header.
     class Header
       def initialize(hts_file)
         @sam_hdr = LibHTS.sam_hdr_read(hts_file)

data/lib/hts/bam/record.rb

@@ -6,6 +6,7 @@ require_relative "aux"
 
 module HTS
   class Bam < Hts
+    # A class for working with alignment records.
     class Record
       SEQ_NT16_STR = "=ACMGRSVTWYHKDBN"
 
@@ -62,14 +63,17 @@ module HTS
       end
 
       # returns 0-based mate position
-      def mpos
+      def mate_pos
         @bam1[:core][:mpos]
       end
 
-      def mpos=(mpos)
+      def mate_pos=(mpos)
         @bam1[:core][:mpos] = mpos
       end
 
+      alias mpos mate_pos
+      alias mpos= mate_pos=
+
       def bin
         @bam1[:core][:bin]
       end
@@ -111,12 +115,11 @@ module HTS
         @bam1[:core][:isize]
       end
 
-      alias isize insert_size
-
       def insert_size=(isize)
         @bam1[:core][:isize] = isize
       end
 
+      alias isize insert_size
       alias isize= insert_size=
 
       # mapping quality

data/lib/hts/bam.rb

@@ -9,10 +9,11 @@ require_relative "bam/flag"
 require_relative "bam/record"
 
 module HTS
+  # A class for working with SAM, BAM, CRAM files.
   class Bam
     include Enumerable
 
-    attr_reader :file_name, :index_name, :mode, :header
+    attr_reader :file_name, :index_name, :mode, :header, :nthreads
 
     def self.open(*args, **kw)
       file = new(*args, **kw) # do not yield
@@ -27,7 +28,7 @@ module HTS
     end
 
     def initialize(file_name, mode = "r", index: nil, fai: nil, threads: nil,
-                   create_index: false)
+                   build_index: false)
       if block_given?
         message = "HTS::Bam.new() dose not take block; Please use HTS::Bam.open() instead"
         raise message
@@ -38,6 +39,7 @@ module HTS
       @file_name  = file_name
       @index_name = index
       @mode       = mode
+      @nthreads   = threads
       @hts_file   = LibHTS.hts_open(@file_name, mode)
 
       raise Errno::ENOENT, "Failed to open #{@file_name}" if @hts_file.null?
@@ -47,32 +49,34 @@ module HTS
         raise "Failed to load fasta index: #{fai}" if r < 0
       end
 
-      if threads&.> 0
-        r = LibHTS.hts_set_threads(@hts_file, threads)
-        raise "Failed to set number of threads: #{threads}" if r < 0
-      end
+      set_threads(threads) if threads
 
       return if @mode[0] == "w"
 
       @header = Bam::Header.new(@hts_file)
-
-      create_index(index) if create_index
-
+      build_index(index) if build_index
       @idx = load_index(index)
-
       @start_position = tell
+      super # do nothing
     end
 
-    def create_index(index_name = nil)
-      warn "Create index for #{@file_name} to #{index_name}"
-      if index
-        LibHTS.sam_index_build2(@file_name, index_name, -1)
+    def build_index(index_name = nil, min_shift: 0)
+      check_closed
+
+      if index_name
+        warn "Create index for #{@file_name} to #{index_name}"
       else
-        LibHTS.sam_index_build(@file_name, -1)
+        warn "Create index for #{@file_name}"
       end
+      r = LibHTS.sam_index_build3(@file_name, index_name, min_shift, @nthreads)
+      raise "Failed to build index for #{@file_name}" if r < 0
+
+      self
     end
 
     def load_index(index_name = nil)
+      check_closed
+
       if index_name
         LibHTS.sam_index_load2(@hts_file, @file_name, index_name)
       else
@@ -81,6 +85,8 @@ module HTS
     end
 
     def index_loaded?
+      check_closed
+
       !@idx.null?
     end
 
@@ -92,7 +98,7 @@ module HTS
     end
 
     def write_header(header)
-      raise IOError, "closed stream" if closed?
+      check_closed
 
       @header = header.dup
       LibHTS.hts_set_fai_filename(@hts_file, @file_name)
@@ -100,17 +106,22 @@ module HTS
     end
 
     def write(aln)
-      raise IOError, "closed stream" if closed?
+      check_closed
 
       aln_dup = aln.dup
       LibHTS.sam_write1(@hts_file, header, aln_dup) > 0 || raise
     end
 
-    # Iterate over each record.
-    # Generate a new Record object each time.
-    # Slower than each.
-    def each_copy
-      raise IOError, "closed stream" if closed?
+    def each(copy: false, &block)
+      if copy
+        each_record_copy(&block)
+      else
+        each_record_reuse(&block)
+      end
+    end
+
+    private def each_record_copy
+      check_closed
       return to_enum(__method__) unless block_given?
 
       while LibHTS.sam_read1(@hts_file, header, bam1 = LibHTS.bam_init1) != -1
@@ -120,14 +131,10 @@ module HTS
       self
     end
 
-    # Iterate over each record.
-    # Record object is reused.
-    # Faster than each_copy.
-    def each
-      raise IOError, "closed stream" if closed?
+    private def each_record_reuse
+      check_closed
       # Each does not always start at the beginning of the file.
       # This is the common behavior of IO objects in Ruby.
-      # This may change in the future.
       return to_enum(__method__) unless block_given?
 
       bam1 = LibHTS.bam_init1
@@ -136,23 +143,109 @@ module HTS
       self
     end
 
-    # query [WIP]
-    def query(region)
-      raise IOError, "closed stream" if closed?
+    def query(region, copy: false, &block)
+      if copy
+        query_copy(region, &block)
+      else
+        query_reuse(region, &block)
+      end
+    end
+
+    private def query_copy(region)
+      check_closed
       raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
 
       qiter = LibHTS.sam_itr_querys(@idx, header, region)
+
       begin
-        bam1 = LibHTS.bam_init1
-        slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
-        while slen > 0
-          yield Record.new(bam1, header)
+        loop do
           bam1 = LibHTS.bam_init1
           slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
+          break if slen == -1
+          raise if slen < -1
+
+          yield Record.new(bam1, header)
         end
       ensure
         LibHTS.hts_itr_destroy(qiter)
       end
+      self
+    end
+
+    private def query_reuse(region)
+      check_closed
+      raise "Index file is required to call the query method." unless index_loaded?
+      return to_enum(__method__, region) unless block_given?
+
+      qiter = LibHTS.sam_itr_querys(@idx, header, region)
+
+      bam1 = LibHTS.bam_init1
+      record = Record.new(bam1, header)
+      begin
+        yield record while LibHTS.sam_itr_next(@hts_file, qiter, bam1) > 0
+      ensure
+        LibHTS.hts_itr_destroy(qiter)
+      end
+      self
+    end
+
+    # @!macro [attach] define_getter
+    #   @method $1
+    #   Get $1 array
+    #   @return [Array] the $1 array
+    define_getter :qname
+    define_getter :flag
+    define_getter :chrom
+    define_getter :pos
+    define_getter :mapq
+    define_getter :cigar
+    define_getter :mate_chrom
+    define_getter :mate_pos
+    define_getter :insert_size
+    define_getter :seq
+    define_getter :qual
+
+    alias isize insert_size
+    alias mpos mate_pos
+
+    def aux(tag)
+      warn "experimental"
+      check_closed
+      position = tell
+      ary = map { |r| r.aux(tag) }
+      seek(position)
+      ary
+    end
+
+    # @!macro [attach] define_iterator
+    #   @method each_$1
+    #   Get $1 iterator
+    define_iterator :qname
+    define_iterator :flag
+    define_iterator :chrom
+    define_iterator :pos
+    define_iterator :mapq
+    define_iterator :cigar
+    define_iterator :mate_chrom
+    define_iterator :mate_pos
+    define_iterator :insert_size
+    define_iterator :seq
+    define_iterator :qual
+
+    alias each_isize each_insert_size
+    alias each_mpos each_mate_pos
+
+    def each_aux(tag)
+      warn "experimental"
+      check_closed
+      return to_enum(__method__, tag) unless block_given?
+
+      each do |record|
+        yield record.aux(tag)
+      end
+
+      self
     end
   end
 end

data/lib/hts/bcf/format.rb

@@ -38,8 +38,8 @@ module HTS
         h = @record.header.struct
         r = @record.struct
 
-        format_values = proc do |type|
-          ret = LibHTS.bcf_get_format_values(h, r, key, p1, n, type)
+        format_values = proc do |typ|
+          ret = LibHTS.bcf_get_format_values(h, r, key, p1, n, typ)
           return nil if ret < 0 # return from method.
 
           p1.read_pointer
@@ -79,10 +79,10 @@ module HTS
           num  = LibHTS.bcf_hdr_id2number(@record.header.struct, LibHTS::BCF_HL_FMT, id)
           type = LibHTS.bcf_hdr_id2type(@record.header.struct, LibHTS::BCF_HL_FMT, id)
           {
-            name: name,
+            name:,
             n: num,
             type: ht_type_to_sym(type),
-            id: id
+            id:
           }
         end
       end

data/lib/hts/bcf/header.rb

@@ -2,6 +2,7 @@
 
 module HTS
   class Bcf < Hts
+    # A class for working with VCF records.
     class Header
       def initialize(hts_file)
         @bcf_hdr = LibHTS.bcf_hdr_read(hts_file)

data/lib/hts/bcf/header_record.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+module HTS
+  class Bcf < Hts
+    class HeaderRecord
+      def initialize
+        @bcf_hrec
+      end
+    end
+  end
+end

data/lib/hts/bcf/info.rb

@@ -2,6 +2,7 @@
 
 module HTS
   class Bcf < Hts
+    # Info field
     class Info
       def initialize(record)
         @record = record
@@ -76,10 +77,10 @@ module HTS
           num  = LibHTS.bcf_hdr_id2number(@record.header.struct, LibHTS::BCF_HL_INFO, key)
           type = LibHTS.bcf_hdr_id2type(@record.header.struct, LibHTS::BCF_HL_INFO, key)
           {
-            name: name,
+            name:,
             n: num,
             type: ht_type_to_sym(type),
-            key: key
+            key:
           }
         end
       end

data/lib/hts/bcf/record.rb

@@ -2,6 +2,7 @@
 
 module HTS
   class Bcf < Hts
+    # A class for working with VCF records.
     class Record
       def initialize(bcf_t, header)
         @bcf1 = bcf_t
@@ -60,35 +61,6 @@ module HTS
         LibHTS.bcf_update_id(@header, @bcf1, ".")
       end
 
-      def filter
-        LibHTS.bcf_unpack(@bcf1, LibHTS::BCF_UN_FLT)
-        d = @bcf1[:d]
-        n_flt = d[:n_flt]
-
-        case n_flt
-        when 0
-          "PASS"
-        when 1
-          i = d[:flt].read_int
-          LibHTS.bcf_hdr_int2id(@header.struct, LibHTS::BCF_DT_ID, i)
-        when 2..nil
-          d[:flt].get_array_of_int(0, n_flt).map do |i|
-            LibHTS.bcf_hdr_int2id(@header.struct, LibHTS::BCF_DT_ID, i)
-          end
-        else
-          raise "Unexpected number of filters. n_flt: #{n_flt}"
-        end
-      end
-
-      # Get variant quality.
-      def qual
-        @bcf1[:qual]
-      end
-
-      def qual=(qual)
-        @bcf1[:qual] = qual
-      end
-
       def ref
         LibHTS.bcf_unpack(@bcf1, LibHTS::BCF_UN_STR)
         @bcf1[:d][:allele].get_pointer(0).read_string
@@ -108,6 +80,35 @@ module HTS
         ).map(&:read_string)
       end
 
+      # Get variant quality.
+      def qual
+        @bcf1[:qual]
+      end
+
+      def qual=(qual)
+        @bcf1[:qual] = qual
+      end
+
+      def filter
+        LibHTS.bcf_unpack(@bcf1, LibHTS::BCF_UN_FLT)
+        d = @bcf1[:d]
+        n_flt = d[:n_flt]
+
+        case n_flt
+        when 0
+          "PASS"
+        when 1
+          id = d[:flt].read_int
+          LibHTS.bcf_hdr_int2id(@header.struct, LibHTS::BCF_DT_ID, id)
+        when 2..nil
+          d[:flt].get_array_of_int(0, n_flt).map do |i|
+            LibHTS.bcf_hdr_int2id(@header.struct, LibHTS::BCF_DT_ID, i)
+          end
+        else
+          raise "Unexpected number of filters. n_flt: #{n_flt}"
+        end
+      end
+
       def info(key = nil)
         LibHTS.bcf_unpack(@bcf1, LibHTS::BCF_UN_SHR)
         info = Info.new(self)
@@ -136,7 +137,7 @@ module HTS
 
       private
 
-      def initialize_copy(orig)\
+      def initialize_copy(orig)
         @header = orig.header
         @bcf1 = LibHTS.bcf_dup(orig.struct)
       end