htslib 0.0.4 → 0.0.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c8048806df4c335ea698c8d3ad9e51e12644f0d32667d3db5fac59a004db75bf
-  data.tar.gz: 3d18183921f70b7b42dc5657195607c519c7d83c1e0e94b9b7a8f647d96f5504
+  metadata.gz: dd81ebe87741d1019acd341189ee0f8fb6ef09d3a10690686b3d40bdfea73076
+  data.tar.gz: 39a7fa296cc1797c7cd6317923b3336c67f0cf16e8c9f7eb8ae8dbef9e6c7f56
 SHA512:
-  metadata.gz: f50e96a23cb75130c5aa463329cce3c2d5784ed75c608d4685d2662ee22fa93e55208c356bce0fee319028c6ae8e33b5b20d270a17fb63ca6e39a736f668a2e4
-  data.tar.gz: c223f9b61979ace926fa53b408e85fa6f2e127cdde6ffa5bcfa1ac0ad7ce6c9c1ac5570522021c61e82d27498473299b8875f7d633b24367ae99dd2494758556
+  metadata.gz: c349f647e798efc920492877807db88d03dfe88edf1434e733d38221ab15f1bea179ac524b3e26feb83c7063c878138918be3dcb30bcbbc583e0cc5bbbe1b8df
+  data.tar.gz: 120052721db361d285968331d2091f4bfe17b45a4204443a591141bc12aef3b153866077dc85ddff63451873800f78db377a151ecbec7084b1f5f409fe5507b0

data/README.md

@@ -9,11 +9,12 @@
 :dna: [HTSlib](https://github.com/samtools/htslib) - for Ruby
 
 Ruby-htslib is the Ruby bindings to HTSlib, a C library for processing high throughput sequencing (HTS) data. 
-It will provide APIs to read and write file formats such as SAM, BAM, VCF, and BCF.
+It will provide APIs to read and write file formats such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/).
 
 :apple: Feel free to fork it out if you can develop it! 
 
 :bowtie: alpha stage.
+
 ## Requirements
 
 * [Ruby](https://github.com/ruby/ruby) 2.7 or above.
@@ -38,26 +39,12 @@ export HTSLIBDIR="/your/path/to/htslib" # libhts.so
 
 ## Overview
 
-### Low level API
-
-`HTS::LibHTS` provides native functions. 
-
-```ruby
-require 'htslib'
+### High level API
 
-a = HTS::LibHTS.hts_open("a.bam", "r")
-b = HTS::LibHTS.hts_get_format(a)
-p b[:category]
-p b[:format]
-```
+A high-level API is under development. 
+Classes such as `Cram` `Bam` `Bcf` `Faidx` `Tabix` are partially implemented.
 
-Note: Managed struct is not used in ruby-htslib. You may need to free the memory by yourself.
-
-### High level API (Plan)
-
-`Cram` `Bam` `Bcf` `Faidx` `Tabix`
-
-A high-level API is under development. We will change and improve the API to make it better.
+Read SAM / BAM - Sequence Alignment Map file
 
 ```ruby
 require 'htslib'
@@ -67,15 +54,56 @@ bam = HTS::Bam.new("a.bam")
 bam.each do |r|
   p name:  r.qname,
     flag:  r.flag,
-    start: r.start + 1,
-    mpos:  r.mate_pos + 1,
+    pos:   r.start + 1,
+    mpos:  r.mate_start + 1,
     mqual: r.mapping_quality,
     seq:   r.sequence,
     cigar: r.cigar.to_s,
     qual:  r.base_qualities.map { |i| (i + 33).chr }.join
 end
+
+bam.close
+```
+
+Read VCF / BCF - Variant Call Format
+
+```ruby
+bcf = HTS::Bcf.new("b.bcf")
+
+bcf.each do |r|
+  p chrom:  r.chrom,
+    pos:    r.pos,
+    id:     r.id,
+    qual:   r.qual.round(2),
+    ref:    r.ref,
+    alt:    r.alt,
+    filter: r.filter
+end
+
+bcf.close
+```
+
+The methods for reading are implemented first. Methods for writing will be implemented in the coming days.
+
+### Low level API
+
+`HTS::LibHTS` provides native functions. 
+
+```ruby
+require 'htslib'
+
+a = HTS::LibHTS.hts_open("a.bam", "r")
+b = HTS::LibHTS.hts_get_format(a)
+p b[:category]
+p b[:format]
 ```
 
+Note: Only some C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually.
+
+### Need more speed?
+
+Try [htslib.cr](https://github.com/bio-crystal/htslib.cr). htslib.cr is implemented in Crystal language and provides an API compatible with ruby-htslib. crsytal language is not as flexible as Ruby language. If you have a clear idea of the manipulation you want to do and need to perform it many times, then by all means try to implement a command line tool using htslib.cr. The Crystal language is very fast and can perform almost as well as the Rust and C languages.
+
 ## Documentation
 
 * [RubyDoc.info - HTSlib](https://rdoc.info/gems/htslib)
@@ -92,8 +120,10 @@ bundle exec rake htslib:build
 bundle exec rake test
 ```
 
-We plan to actively use the new features of Ruby. Since the number of users is small, backward compatibility is not important.
-On the other hand, we will consider compatibility with [Crystal](https://github.com/bio-crystal/htslib.cr) to some extent.
+Many macro functions are used in HTSlib. Since these macro functions cannot be called using FFI, they must be reimplemented in Ruby.
+
+* Actively use the advanced features of Ruby.
+* Consider compatibility with [htslib.cr](https://github.com/bio-crystal/htslib.cr) to some extent.
 
 #### FFI Extensions
 
@@ -101,6 +131,7 @@ On the other hand, we will consider compatibility with [Crystal](https://github.
 
 #### Automatic generation or automatic validation (Future plan)
 
+
 + [c2ffi](https://github.com/rpav/c2ffi) is a tool to create JSON format metadata from C header files. It is planned to use c2ffi to automatically generate bindings or tests.
 
 ## Contributing
@@ -113,6 +144,16 @@ Ruby-htslib is a library under development, so even small improvements like typo
 * Suggest or add new features
 * [financial contributions](https://github.com/sponsors/kojix2)
 
+```
+Do you need commit rights to my repository?
+Do you want to get admin rights and take over the project?
+If so, please feel free to contact us @kojix2.
+```
+
+#### Why do you implement htslib in a language like Ruby, which is not widely used in the bioinformatics?
+
+One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding to better understand how to use the file formats and tools. And that effort is still going on today...
+
 ## Links
 
 * [samtools/hts-spec](https://github.com/samtools/hts-specs)

data/lib/hts/bam/cigar.rb

@@ -22,6 +22,8 @@ module HTS
       end
 
       def each
+        return to_enum(__method__) unless block_given?
+
         @n_cigar.times do |i|
           c = @pointer[i].read_uint32
           yield [LibHTS.bam_cigar_oplen(c),

data/lib/hts/bam/flag.rb

@@ -85,8 +85,12 @@ module HTS
         has_flag? LibHTS::BAM_FSUPPLEMENTARY
       end
 
-      def has_flag?(o)
-        @value[o] != 0
+      def has_flag?(m)
+        (@value & m) != 0
+      end
+
+      def to_s
+        "0x#{format('%x', @value)}\t#{@value}\t#{LibHTS.bam_flag2str(@value)}"
       end
     end
   end

data/lib/hts/bam/header.rb

@@ -6,8 +6,8 @@
 module HTS
   class Bam
     class Header
-      def initialize(pointer)
-        @sam_hdr = pointer
+      def initialize(hts_file)
+        @sam_hdr = LibHTS.sam_hdr_read(hts_file)
       end
 
       def struct
@@ -18,16 +18,31 @@ module HTS
         @sam_hdr.to_ptr
       end
 
-      # FIXME: better name?
-      def seqs
-        Array.new(@sam_hdr[:n_targets]) do |i|
+      def target_count
+        @sam_hdr[:n_targets]
+      end
+
+      def target_names
+        Array.new(target_count) do |i|
           LibHTS.sam_hdr_tid2name(@sam_hdr, i)
         end
       end
 
-      def text
+      def target_lengths
+        Array.new(target_count) do |i|
+          LibHTS.sam_hdr_tid2len(@sam_hdr, i)
+        end
+      end
+
+      def to_s
         LibHTS.sam_hdr_str(@sam_hdr)
       end
+
+      private
+
+      def initialize_copy(orig)
+        @sam_hdr = LibHTS.sam_hdr_dup(orig.struct)
+      end
     end
   end
 end

data/lib/hts/bam/record.rb

@@ -21,6 +21,8 @@ module HTS
         @bam1.to_ptr
       end
 
+      attr_reader :header
+
       # def initialize_copy
       #   super
       # end
@@ -67,18 +69,22 @@ module HTS
 
       # returns the chromosome or '' if not mapped.
       def chrom
-        tid = @bam1[:core][:tid]
         return "" if tid == -1
 
         LibHTS.sam_hdr_tid2name(@header, tid)
       end
 
+      # returns the chromosome or '' if not mapped.
+      def contig
+        chrom
+      end
+
       # returns the chromosome of the mate or '' if not mapped.
       def mate_chrom
-        tid = @bam1[:core][:mtid]
-        return "" if tid == -1
+        mtid = mate_tid
+        return "" if mtid == -1
 
-        LibHTS.sam_hdr_tid2name(@header, tid)
+        LibHTS.sam_hdr_tid2name(@header, mtid)
       end
 
       def strand
@@ -90,7 +96,7 @@ module HTS
       # end
 
       # insert size
-      def isize
+      def insert_size
         @bam1[:core][:isize]
       end
 
@@ -161,6 +167,28 @@ module HTS
         Flag.new(@bam1[:core][:flag])
       end
 
+      def tag(str)
+        aux = LibHTS.bam_aux_get(@bam1, str)
+        return nil if aux.null?
+
+        t = aux.read_string(1)
+
+        # A (character), B (general array),
+        # f (real number), H (hexadecimal array),
+        # i (integer), or Z (string).
+
+        case t
+        when "i", "I", "c", "C", "s", "S"
+          LibHTS.bam_aux2i(aux)
+        when "f", "d"
+          LibHTS.bam_aux2f(aux)
+        when "Z", "H"
+          LibHTS.bam_aux2Z(aux)
+        when "A" # char
+          LibHTS.bam_aux2A(aux).chr
+        end
+      end
+
       def to_s
         kstr = LibHTS::KString.new
         raise "Failed to format bam record" if LibHTS.sam_format1(@header.struct, @bam1, kstr) == -1
@@ -168,9 +196,12 @@ module HTS
         kstr[:s]
       end
 
-      # TODO:
-      # def eql?
-      # def hash
+      private
+
+      def initialize_copy(orig)
+        @header = orig.header
+        @bam = LibHTS.bam_dup1(orig.struct)
+      end
     end
   end
 end

data/lib/hts/bam.rb

@@ -7,104 +7,139 @@ require_relative "bam/header"
 require_relative "bam/cigar"
 require_relative "bam/flag"
 require_relative "bam/record"
-require_relative "utils/open_method"
 
 module HTS
   class Bam
     include Enumerable
-    extend Utils::OpenMethod
 
     attr_reader :file_path, :mode, :header
-    # HtfFile is FFI::BitStruct
-    attr_reader :htf_file
 
-    class << self
-      alias open new
+    def self.open(...)
+      file = new(...)
+      return file unless block_given?
+
+      begin
+        yield file
+      ensure
+        file.close
+      end
+      file
     end
 
-    def initialize(file_path, mode = "r", create_index: nil)
-      file_path = File.expand_path(file_path)
+    def initialize(filename, mode = "r", fai: nil, threads: nil, index: nil)
+      raise "HTS::Bam.new() dose not take block; Please use HTS::Bam.open() instead" if block_given?
 
-      unless File.exist?(file_path)
+      @file_path = filename == "-" ? "-" : File.expand_path(filename)
+
+      if mode[0] == "r" && !File.exist?(file_path)
         message = "No such SAM/BAM file - #{file_path}"
         raise message
       end
 
-      @file_path = file_path
       @mode      = mode
-      @htf_file  = LibHTS.hts_open(file_path, mode)
-      @header    = Bam::Header.new(LibHTS.sam_hdr_read(htf_file))
-
-      # FIXME: should be defined here?
-      @bam1      = LibHTS.bam_init1
-
-      # read
-      if mode[0] == "r"
-        # load index
-        @idx = LibHTS.sam_index_load(htf_file, file_path)
-        # create index
-        if create_index || (@idx.null? && create_index.nil?)
-          warn "Create index for #{file_path}"
-          LibHTS.sam_index_build(file_path, -1)
-          @idx = LibHTS.sam_index_load(htf_file, file_path)
-        end
-      else
-        # FIXME: implement
-        raise "not implemented yet."
+      @hts_file  = LibHTS.hts_open(file_path, mode)
+
+      if fai
+        fai_path = File.expand_path(fai)
+        r = LibHTS.hts_set_fai_filename(@hts_file, fai_path)
+        raise "Failed to load fasta index: #{fai}" if r < 0
       end
 
-      # IO like API
-      if block_given?
-        begin
-          yield self
-        ensure
-          close
-        end
+      if threads&.> 0
+        r = LibHTS.hts_set_threads(@hts_file, threads)
+        raise "Failed to set number of threads: #{threads}" if r < 0
       end
+
+      return if mode[0] == "w"
+
+      @header = Bam::Header.new(@hts_file)
+
+      create_index if index
+
+      # load index
+      @idx = LibHTS.sam_index_load(@hts_file, file_path)
     end
 
-    def struct
-      htf_file
+    def create_index
+      warn "Create index for #{file_path}"
+      LibHTS.sam_index_build(file_path, -1)
+      idx = LibHTS.sam_index_load(@hts_file, file_path)
+      raise "Failed to load index: #{file_path}" if idx.null?
     end
 
-    def to_ptr
-      htf_file.to_ptr
+    def struct
+      @hts_file
     end
 
-    def write(alns)
-      alns.each do
-        LibHTS.sam_write1(htf_file, header, alns.b) > 0 || raise
-      end
+    def to_ptr
+      @hts_file.to_ptr
     end
 
     # Close the current file.
     def close
-      LibHTS.hts_close(htf_file)
+      LibHTS.hts_idx_destroy(@idx) if @idx
+      @idx = nil
+      LibHTS.hts_close(@hts_file)
+      @hts_file = nil
+    end
+
+    def closed?
+      @hts_file.nil?
+    end
+
+    def write_header(header)
+      @header = header.dup
+      LibHTS.hts_set_fai_filename(@hts_file, @file_path)
+      LibHTS.sam_hdr_write(@hts_file, header)
+    end
+
+    def write(aln)
+      aln_dup = aln.dup
+      LibHTS.sam_write1(@hts_file, header, aln_dup) > 0 || raise
     end
 
     # Flush the current file.
     def flush
-      # LibHTS.bgzf_flush(@htf_file.fp.bgzf)
+      # LibHTS.bgzf_flush(@@hts_file.fp.bgzf)
     end
 
-    def each(&block)
+    # Iterate over each record.
+    # Record object is reused.
+    # Faster than each_copy.
+    def each
       # Each does not always start at the beginning of the file.
       # This is the common behavior of IO objects in Ruby.
       # This may change in the future.
-      while LibHTS.sam_read1(htf_file, header, @bam1) > 0
-        record = Record.new(@bam1, header)
-        block.call(record)
+      return to_enum(__method__) unless block_given?
+
+      bam1 = LibHTS.bam_init1
+      record = Record.new(bam1, header)
+      yield record while LibHTS.sam_read1(@hts_file, header, bam1) > 0
+    end
+
+    # Iterate over each record.
+    # Generate a new Record object each time.
+    # Slower than each.
+    def each_copy
+      return to_enum(__method__) unless block_given?
+
+      while LibHTS.sam_read1(@hts_file, header, bam1 = LibHTS.bam_init1) > 0
+        record = Record.new(bam1, header)
+        yield record
       end
     end
 
     # query [WIP]
     def query(region)
+      # FIXME: when @idx is nil
       qiter = LibHTS.sam_itr_querys(@idx, header, region)
       begin
-        slen = LibHTS.sam_itr_next(htf_file, qiter, @bam1)
+        bam1 = LibHTS.bam_init1
+        slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
         while slen > 0
-          yield Record.new(@bam1, header)
-          slen = LibHTS.sam_itr_next(htf_file, qiter, @bam1)
+          yield Record.new(bam1, header)
+          bam1 = LibHTS.bam_init1
+          slen = LibHTS.sam_itr_next(@hts_file, qiter, bam1)
         end
       ensure
         LibHTS.hts_itr_destroy(qiter)

data/lib/hts/bcf/format.rb

@@ -12,10 +12,30 @@ module HTS
         @p1 = FFI::MemoryPointer.new(:pointer) # FIXME: naming
       end
 
+      # For compatibility with htslib.cr.
+      def get_int(key)
+        get(key, :int)
+      end
+
+      # For compatibility with htslib.cr.
+      def get_float(key)
+        get(key, :float)
+      end
+
+      # For compatibility with htslib.cr.
+      def get_flag(key)
+        get(key, :flag)
+      end
+
+      # For compatibility with htslib.cr.
+      def get_string(key)
+        get(key, :string)
+      end
+
       def get(key, type = nil)
         n = FFI::MemoryPointer.new(:int)
         p1 = @p1
-        h = @record.bcf.header.struct
+        h = @record.header.struct
         r = @record.struct
 
         format_values = proc do |type|
@@ -33,11 +53,13 @@ module HTS
           format_values.call(LibHTS::BCF_HT_REAL)
                        .read_array_of_float(n.read_int)
         when :flag
-          format_values.call(LibHTS::BCF_HT_FLAG)
-                       .read_int == 1
+          raise NotImplementedError, "Flag type not implemented yet."
+          # format_values.call(LibHTS::BCF_HT_FLAG)
+          #              .read_int == 1
         when :string, :str
-          format_values.call(LibHTS::BCF_HT_STR)
-                       .read_pointer.read_string
+          raise NotImplementedError, "String type not implemented yet."
+          # format_values.call(LibHTS::BCF_HT_STR)
+          #              .read_string
         end
       end
 

data/lib/hts/bcf/header.rb

@@ -3,16 +3,44 @@
 module HTS
   class Bcf
     class Header
-      def initialize(h)
-        @h = h
+      def initialize(hts_file)
+        @bcf_hdr = LibHTS.bcf_hdr_read(hts_file)
       end
 
       def struct
-        @h
+        @bcf_hdr
       end
 
       def to_ptr
-        @h.to_ptr
+        @bcf_hdr.to_ptr
+      end
+
+      def get_version
+        LibHTS.bcf_hdr_get_version(@bcf_hdr)
+      end
+
+      def sample_count
+        LibHTS.bcf_hdr_nsamples(@bcf_hdr)
+      end
+
+      def sample_names
+        # bcf_hdr_id2name is macro function
+        @bcf_hdr[:samples]
+          .read_array_of_pointer(sample_count)
+          .map(&:read_string)
+      end
+
+      def to_s
+        kstr = LibHTS::KString.new
+        raise "Failed to get header string" unless LibHTS.bcf_hdr_format(@bcf_hdr, 0, kstr)
+
+        kstr[:s]
+      end
+
+      private
+
+      def initialize_copy(orig)
+        @bcf_hdr = LibHTS.bcf_hdr_dup(orig.struct)
       end
     end
   end