htslib 0.0.8 → 0.0.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dd81ebe87741d1019acd341189ee0f8fb6ef09d3a10690686b3d40bdfea73076
-  data.tar.gz: 39a7fa296cc1797c7cd6317923b3336c67f0cf16e8c9f7eb8ae8dbef9e6c7f56
+  metadata.gz: 2ecf4c183a1720313371e493fbf7876a08d17639ad853b1d26faeff3addb7c6e
+  data.tar.gz: cd1ed908a5dd8be184c8d9195c6fb5fabda5487d3c35e11476d9bd6d15c7dae8
 SHA512:
-  metadata.gz: c349f647e798efc920492877807db88d03dfe88edf1434e733d38221ab15f1bea179ac524b3e26feb83c7063c878138918be3dcb30bcbbc583e0cc5bbbe1b8df
-  data.tar.gz: 120052721db361d285968331d2091f4bfe17b45a4204443a591141bc12aef3b153866077dc85ddff63451873800f78db377a151ecbec7084b1f5f409fe5507b0
+  metadata.gz: 4980914c7b92528055d2d0cf62fad6ca5edcde8ba1b9d426740538aee42fab53c7ef96ebe53adcaa9663368f024546c048b71760b68bd88e7cbb7647e200847c
+  data.tar.gz: a94b5d01702cc9873fdf6d1498658cff15c44ba8c90b08cafe514a883560c6aaa90cfbd83cede8a4ae1a5095c6f99710e76a8aacb0e0851e628c607c37719bd5

data/README.md

@@ -6,10 +6,7 @@
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
 [![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-:dna: [HTSlib](https://github.com/samtools/htslib) - for Ruby
-
-Ruby-htslib is the Ruby bindings to HTSlib, a C library for processing high throughput sequencing (HTS) data. 
-It will provide APIs to read and write file formats such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/).
+Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/) in the Ruby language.
 
 :apple: Feel free to fork it out if you can develop it! 
 
@@ -41,34 +38,35 @@ export HTSLIBDIR="/your/path/to/htslib" # libhts.so
 
 ### High level API
 
-A high-level API is under development. 
-Classes such as `Cram` `Bam` `Bcf` `Faidx` `Tabix` are partially implemented.
-
-Read SAM / BAM - Sequence Alignment Map file
+Read SAM / BAM / CRAM - Sequence Alignment Map file
 
 ```ruby
 require 'htslib'
 
-bam = HTS::Bam.new("a.bam")
+bam = HTS::Bam.open("a.bam")
 
 bam.each do |r|
   p name:  r.qname,
     flag:  r.flag,
+    chr:   r.chrom,
     pos:   r.start + 1,
-    mpos:  r.mate_start + 1,
     mqual: r.mapping_quality,
-    seq:   r.sequence,
     cigar: r.cigar.to_s,
-    qual:  r.base_qualities.map { |i| (i + 33).chr }.join
+    mchr:  r.mate_chrom,
+    mpos:  r.mate_start + 1,
+    isize: r.insert_size,
+    seq:   r.sequence,
+    qual:  r.base_qualities.map { |i| (i + 33).chr }.join,
+    tagMC: r.tag("MC")
 end
 
 bam.close
 ```
 
-Read VCF / BCF - Variant Call Format
+Read VCF / BCF - Variant Call Format file
 
 ```ruby
-bcf = HTS::Bcf.new("b.bcf")
+bcf = HTS::Bcf.open("b.bcf")
 
 bcf.each do |r|
   p chrom:  r.chrom,
@@ -78,16 +76,16 @@ bcf.each do |r|
     ref:    r.ref,
     alt:    r.alt,
     filter: r.filter
+  # info:   r.info
+  # format: r.format
 end
 
 bcf.close
 ```
 
-The methods for reading are implemented first. Methods for writing will be implemented in the coming days.
-
 ### Low level API
 
-`HTS::LibHTS` provides native functions. 
+`HTS::LibHTS` provides native　C functions. 
 
 ```ruby
 require 'htslib'
@@ -98,19 +96,20 @@ p b[:category]
 p b[:format]
 ```
 
-Note: Only some C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually.
+Note: htslib makes extensive use of macro functions for speed. you cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
 
 ### Need more speed?
 
-Try [htslib.cr](https://github.com/bio-crystal/htslib.cr). htslib.cr is implemented in Crystal language and provides an API compatible with ruby-htslib. crsytal language is not as flexible as Ruby language. If you have a clear idea of the manipulation you want to do and need to perform it many times, then by all means try to implement a command line tool using htslib.cr. The Crystal language is very fast and can perform almost as well as the Rust and C languages.
+Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crsytal language is not as flexible as Ruby language. You can not use eval methods, and you must always be aware of the types. It is not very suitable for writing one-time scripts or experimenting with different code. However, If you have already written code in ruby-htslib, have a clear idea of the manipulations you want to do, and need to execute them many times, then by all means try to implement the command line tool using htslib.cr. The Crystal language is very fast and can perform almost as well as the Rust and C languages.
 
 ## Documentation
 
+* [API Documentation (develop branch)](https://kojix2.github.io/ruby-htslib/)
 * [RubyDoc.info - HTSlib](https://rdoc.info/gems/htslib)
 
 ## Development
 
-To get started with development
+To get started with development:
 
 ```sh
 git clone --recursive https://github.com/kojix2/ruby-htslib
@@ -120,10 +119,13 @@ bundle exec rake htslib:build
 bundle exec rake test
 ```
 
+[GNU Autotools](https://en.wikipedia.org/wiki/GNU_Autotools) is required to compile htslib.
+
 Many macro functions are used in HTSlib. Since these macro functions cannot be called using FFI, they must be reimplemented in Ruby.
 
 * Actively use the advanced features of Ruby.
-* Consider compatibility with [htslib.cr](https://github.com/bio-crystal/htslib.cr) to some extent.
+* Remain compatibile with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
+  * The most difficult part is the return value. In the Crystal language, it is convenient for a method to return only one type. In the Ruby language, on the other hand, it is more convenient to return multiple classes. For example, in the Crystal language, it is confusing that a return value can take four types: Int32, Float32, Nil, and String. In Ruby, on the other hand, it is very common and does not cause any problems.
 
 #### FFI Extensions
 
@@ -131,7 +133,6 @@ Many macro functions are used in HTSlib. Since these macro functions cannot be c
 
 #### Automatic generation or automatic validation (Future plan)
 
-
 + [c2ffi](https://github.com/rpav/c2ffi) is a tool to create JSON format metadata from C header files. It is planned to use c2ffi to automatically generate bindings or tests.
 
 ## Contributing
@@ -145,14 +146,14 @@ Ruby-htslib is a library under development, so even small improvements like typo
 * [financial contributions](https://github.com/sponsors/kojix2)
 
 ```
-Do you need commit rights to my repository?
+Do you need commit rights to ruby-htslib repository?
 Do you want to get admin rights and take over the project?
 If so, please feel free to contact us @kojix2.
 ```
 
 #### Why do you implement htslib in a language like Ruby, which is not widely used in the bioinformatics?
 
-One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding to better understand how to use the file formats and tools. And that effort is still going on today...
+One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding myself to better understand how the pioneers of bioinformatics felt when establishing the file format and how they created their tools. And that effort is still going on today...
 
 ## Links
 

data/lib/hts/bam/cigar.rb

@@ -4,7 +4,7 @@
 # https://github.com/quinlan-lab/hts-python
 
 module HTS
-  class Bam
+  class Bam < Hts
     class Cigar
       include Enumerable
 
@@ -18,7 +18,7 @@ module HTS
       end
 
       def to_s
-        to_a.flatten.join
+        map { |op, len| "#{len}#{op}" }.join
       end
 
       def each
@@ -26,8 +26,9 @@ module HTS
 
         @n_cigar.times do |i|
           c = @pointer[i].read_uint32
-          yield [LibHTS.bam_cigar_oplen(c),
-                 LibHTS.bam_cigar_opchr(c)]
+          op =  LibHTS.bam_cigar_opchr(c)
+          len = LibHTS.bam_cigar_oplen(c)
+          yield [op, len]
         end
       end
     end

data/lib/hts/bam/flag.rb

@@ -4,7 +4,7 @@
 # https://github.com/brentp/hts-nim/blob/master/src/hts/bam/flag.nim
 
 module HTS
-  class Bam
+  class Bam < Hts
     class Flag
       def initialize(flag_value)
         raise TypeError unless flag_value.is_a? Integer

data/lib/hts/bam/header.rb

@@ -4,7 +4,7 @@
 # https://github.com/quinlan-lab/hts-python
 
 module HTS
-  class Bam
+  class Bam < Hts
     class Header
       def initialize(hts_file)
         @sam_hdr = LibHTS.sam_hdr_read(hts_file)

data/lib/hts/bam/record.rb

@@ -4,10 +4,12 @@
 # https://github.com/quinlan-lab/hts-python
 
 module HTS
-  class Bam
+  class Bam < Hts
     class Record
       SEQ_NT16_STR = "=ACMGRSVTWYHKDBN"
 
+      attr_reader :header
+
       def initialize(bam1_t, header)
         @bam1 = bam1_t
         @header = header
@@ -21,16 +23,6 @@ module HTS
         @bam1.to_ptr
       end
 
-      attr_reader :header
-
-      # def initialize_copy
-      #   super
-      # end
-
-      def self.rom_sam_str; end
-
-      def tags; end
-
       # returns the query name.
       def qname
         LibHTS.bam_get_qname(@bam1).read_string
@@ -189,6 +181,8 @@ module HTS
         end
       end
 
+      # def tags; end
+
       def to_s
         kstr = LibHTS::KString.new
         raise "Failed to format bam record" if LibHTS.sam_format1(@header.struct, @bam1, kstr) == -1

data/lib/hts/bam.rb

@@ -3,6 +3,9 @@
 # Based on hts-python
 # https://github.com/quinlan-lab/hts-python
 
+require_relative "../htslib"
+
+require_relative "hts"
 require_relative "bam/header"
 require_relative "bam/cigar"
 require_relative "bam/flag"
@@ -12,10 +15,10 @@ module HTS
   class Bam
     include Enumerable
 
-    attr_reader :file_path, :mode, :header
+    attr_reader :file_name, :index_path, :mode, :header
 
-    def self.open(...)
-      file = new(...)
+    def self.open(*args, **kw)
+      file = new(*args, **kw) # do not yield
       return file unless block_given?
 
       begin
@@ -26,22 +29,23 @@ module HTS
       file
     end
 
-    def initialize(filename, mode = "r", fai: nil, threads: nil, index: nil)
-      raise "HTS::Bam.new() dose not take block; Please use HTS::Bam.open() instead" if block_given?
-
-      @file_path = filename == "-" ? "-" : File.expand_path(filename)
-
-      if mode[0] == "r" && !File.exist?(file_path)
-        message = "No such SAM/BAM file - #{file_path}"
+    def initialize(file_name, mode = "r", index: nil, fai: nil, threads: nil,
+                   create_index: false)
+      if block_given?
+        message = "HTS::Bam.new() dose not take block; Please use HTS::Bam.open() instead"
         raise message
       end
 
+      # NOTE: Do not check for the existence of local files, since file_names may be remote URIs.
+
+      @file_name = file_name
       @mode      = mode
-      @hts_file  = LibHTS.hts_open(file_path, mode)
+      @hts_file  = LibHTS.hts_open(@file_name, mode)
+
+      raise Errno::ENOENT, "Failed to open #{@file_name}" if @hts_file.null?
 
       if fai
-        fai_path = File.expand_path(fai)
-        r = LibHTS.hts_set_fai_filename(@hts_file, fai_path)
+        r = LibHTS.hts_set_fai_filename(@hts_file, fai)
         raise "Failed to load fasta index: #{fai}" if r < 0
       end
 
@@ -50,46 +54,54 @@ module HTS
         raise "Failed to set number of threads: #{threads}" if r < 0
       end
 
-      return if mode[0] == "w"
+      return if @mode[0] == "w"
 
       @header = Bam::Header.new(@hts_file)
 
-      create_index if index
+      create_index(index) if create_index
+
+      @idx = load_index(index)
 
-      # load index
-      @idx = LibHTS.sam_index_load(@hts_file, file_path)
+      @start_position = tell
     end
 
-    def create_index
-      warn "Create index for #{file_path}"
-      LibHTS.sam_index_build(file_path, -1)
-      idx = LibHTS.sam_index_load(@hts_file, file_path)
-      raise "Failed to load index: #{file_path}" if idx.null?
+    def create_index(index_name = nil)
+      if index
+        warn "Create index for #{@file_name} to #{index_name}"
+        LibHTS.sam_index_build2(@file_name, index_name, -1)
+      else
+        warn "Create index for #{@file_name} to #{index_name}"
+        LibHTS.sam_index_build(@file_name, -1)
+      end
     end
 
-    def struct
-      @hts_file
+    def load_index(index_name = nil)
+      if index_name
+        LibHTS.sam_index_load2(@hts_file, @file_name, index_name)
+      else
+        LibHTS.sam_index_load3(@hts_file, @file_name, nil, 2) # should be 3 ? (copy remote file to local?)
+      end
     end
 
-    def to_ptr
-      @hts_file.to_ptr
+    def index_loaded?
+      !@idx.null?
     end
 
     # Close the current file.
     def close
-      LibHTS.hts_idx_destroy(@idx) if @idx
+      LibHTS.hts_idx_destroy(@idx) if @idx&.null?
       @idx = nil
       LibHTS.hts_close(@hts_file)
       @hts_file = nil
     end
 
     def closed?
-      @hts_file.nil?
+      @hts_file.nil? || @hts_file.null?
     end
 
     def write_header(header)
       @header = header.dup
-      LibHTS.hts_set_fai_filename(@hts_file, @file_path)
+      LibHTS.hts_set_fai_filename(@hts_file, @file_name)
       LibHTS.sam_hdr_write(@hts_file, header)
     end
 
@@ -98,9 +110,17 @@ module HTS
       LibHTS.sam_write1(@hts_file, header, aln_dup) > 0 || raise
     end
 
-    # Flush the current file.
-    def flush
-      # LibHTS.bgzf_flush(@@hts_file.fp.bgzf)
+    # Iterate over each record.
+    # Generate a new Record object each time.
+    # Slower than each.
+    def each_copy
+      return to_enum(__method__) unless block_given?
+
+      while LibHTS.sam_read1(@hts_file, header, bam1 = LibHTS.bam_init1) != -1
+        record = Record.new(bam1, header)
+        yield record
+      end
+      self
     end
 
     # Iterate over each record.
@@ -114,24 +134,14 @@ module HTS
 
       bam1 = LibHTS.bam_init1
       record = Record.new(bam1, header)
-      yield record while LibHTS.sam_read1(@hts_file, header, bam1) > 0
-    end
-
-    # Iterate over each record.
-    # Generate a new Record object each time.
-    # Slower than each.
-    def each_copy
-      return to_enum(__method__) unless block_given?
-
-      while LibHTS.sam_read1(@hts_file, header, bam1 = LibHTS.bam_init1) > 0
-        record = Record.new(bam1, header)
-        yield record
-      end
+      yield record while LibHTS.sam_read1(@hts_file, header, bam1) != -1
+      self
     end
 
     # query [WIP]
     def query(region)
-      # FIXME: when @idx is nil
+      raise "Index file is required to call the query method." unless index_loaded?
+
       qiter = LibHTS.sam_itr_querys(@idx, header, region)
       begin
         bam1 = LibHTS.bam_init1

data/lib/hts/bcf/format.rb

@@ -5,7 +5,7 @@
 # TODO: Make it more like Ruby.
 
 module HTS
-  class Bcf
+  class Bcf < Hts
     class Format
       def initialize(record)
         @record = record

data/lib/hts/bcf/header.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module HTS
-  class Bcf
+  class Bcf < Hts
     class Header
       def initialize(hts_file)
         @bcf_hdr = LibHTS.bcf_hdr_read(hts_file)
@@ -19,14 +19,14 @@ module HTS
         LibHTS.bcf_hdr_get_version(@bcf_hdr)
       end
 
-      def sample_count
+      def nsamples
         LibHTS.bcf_hdr_nsamples(@bcf_hdr)
       end
 
-      def sample_names
+      def samples
         # bcf_hdr_id2name is macro function
         @bcf_hdr[:samples]
-          .read_array_of_pointer(sample_count)
+          .read_array_of_pointer(nsamples)
           .map(&:read_string)
       end
 

data/lib/hts/bcf/info.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module HTS
-  class Bcf
+  class Bcf < Hts
     class Info
       def initialize(record)
         @record = record

data/lib/hts/bcf/record.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module HTS
-  class Bcf
+  class Bcf < Hts
     class Record
       def initialize(bcf_t, header)
         @bcf1 = bcf_t
@@ -109,8 +109,9 @@ module HTS
 
       private
 
-      def initialize_copy
-        raise "Not implemented"
+      def initialize_copy(orig)\
+        @header = orig.header
+        @bcf1 = LibHTS.bcf_dup(orig.struct)
       end
     end
   end

data/lib/hts/bcf.rb

@@ -3,19 +3,22 @@
 # Based on hts-python
 # https://github.com/quinlan-lab/hts-python
 
+require_relative "../htslib"
+
+require_relative "hts"
 require_relative "bcf/header"
 require_relative "bcf/info"
 require_relative "bcf/format"
 require_relative "bcf/record"
 
 module HTS
-  class Bcf
+  class Bcf < Hts
     include Enumerable
 
-    attr_reader :file_path, :mode, :header
+    attr_reader :file_name, :index_path, :mode, :header
 
-    def self.open(...)
-      file = new(...)
+    def self.open(*args, **kw)
+      file = new(*args, **kw) # do not yield
       return file unless block_given?
 
       begin
@@ -26,40 +29,34 @@ module HTS
       file
     end
 
-    def initialize(filename, mode = "r", threads: nil)
-      raise "HTS::Bcf.new() dose not take block; Please use HTS::Bcf.open() instead" if block_given?
-
-      @file_path = filename == "-" ? "-" : File.expand_path(filename)
-
-      if mode[0] == "r" && !File.exist?(file_path)
-        message = "No such VCF/BCF file - #{file_path}"
+    def initialize(file_name, mode = "r", index: nil, fai: nil, threads: nil,
+                   create_index: false)
+      if block_given?
+        message = "HTS::Bcf.new() dose not take block; Please use HTS::Bcf.open() instead"
         raise message
       end
 
+      # NOTE: Do not check for the existence of local files, since file_names may be remote URIs.
+
+      @file_name = file_name
       @mode      = mode
-      @hts_file  = LibHTS.hts_open(file_path, mode)
+      @hts_file  = LibHTS.hts_open(@file_name, mode)
+
+      raise Errno::ENOENT, "Failed to open #{@file_name}" if @hts_file.null?
 
       if threads&.> 0
         r = LibHTS.hts_set_threads(@hts_file, threads)
         raise "Failed to set number of threads: #{threads}" if r < 0
       end
 
-      return if mode[0] == "w"
+      return if @mode[0] == "w"
 
       @header = Bcf::Header.new(@hts_file)
     end
 
-    def struct
-      @hts_file
-    end
-
-    def to_ptr
-      @hts_file.to_ptr
-    end
-
     def write_header
       @header = header.dup
-      LibHTS.hts_set_fai_filename(header, @file_path)
+      LibHTS.hts_set_fai_filename(header, @file_name)
       LibHTS.bcf_hdr_write(@hts_file, header.struct)
     end
 
@@ -78,15 +75,18 @@ module HTS
       @hts_file.nil?
     end
 
-    def sample_count
-      header.sample_count
+    def nsamples
+      header.nsamples
     end
 
-    def sample_names
-      header.sample_names
+    def samples
+      header.samples
     end
 
-    def each
+    # Iterate over each record.
+    # Generate a new Record object each time.
+    # Slower than each.
+    def each_copy
       return to_enum(__method__) unless block_given?
 
       while LibHTS.bcf_read(@hts_file, header, bcf1 = LibHTS.bcf_init) != -1
@@ -95,5 +95,17 @@ module HTS
       end
       self
     end
+
+    # Iterate over each record.
+    # Record object is reused.
+    # Faster than each_copy.
+    def each
+      return to_enum(__method__) unless block_given?
+
+      bcf1 = LibHTS.bcf_init
+      record = Record.new(bcf1, header)
+      yield record while LibHTS.bcf_read(@hts_file, header, bcf1) != -1
+      self
+    end
   end
 end

data/lib/hts/faidx.rb

@@ -3,17 +3,19 @@
 # Based on hts-python
 # https://github.com/quinlan-lab/hts-python
 
+require_relative "../htslib"
+
 module HTS
   class Faidx
-    attr_reader :file_path
+    attr_reader :file_name
 
     class << self
       alias open new
     end
 
-    def initialize(file_path)
-      @file_path = File.expand_path(file_path)
-      @fai = LibHTS.fai_load(file_path)
+    def initialize(file_name)
+      @file_name = file_name
+      @fai = LibHTS.fai_load(@file_name)
 
       # IO like API
       if block_given?