htslib 0.0.2 → 0.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d4fe8c4a8f710ee3b35793e997f394ba8f80e0fe5b507768de2af2c5ab1297a0
-  data.tar.gz: 1bb06ced745342f4de8665f046304364c9d419ae05eee355c3cd852d6b2b454a
+  metadata.gz: f05bc93a621f3d5fb8d06e44c20ac4a3a95c5f6e243df2aebc97e10125fcb779
+  data.tar.gz: 73e747aa0999e54b3c8f01b981b91c4e293167265276a24eac85ea0fd6b85c13
 SHA512:
-  metadata.gz: 8c7fb677d2462a1ddf4cd146d9f9962b570f39762e98d893d77f89bbbf61c92329ed8391030f772bf4f98fb3dc874edc3c365dd2d85266e1e966b24effc1715c
-  data.tar.gz: fee9ec647e3ef51e83a1aec1c62ac02c796dcfefe4d5fd6c901e4b04e4253619e7337cee651df1ece9ca58c74a7a50620befd5cd8ab823696d0f893a437bd562
+  metadata.gz: 8d0f3f8bb9f7a9c063222fd2dc6463f58bdfd784aff39757921dda1c8d417023b024ca9f6c92d7f0eecd30b8064adb1b94966cf962cf9e0e56d67303de0a903b
+  data.tar.gz: 8ef35b7aef04bf2f51ae0ba110d9ffc81c26d7a0c97a821aa1274c04b31ec29515b33c278391494f6a0d8efc5e2d965060ccd190ee0f92a2ec3562fb59c49169

data/README.md

@@ -1,4 +1,4 @@
-# HTSlib
+# ruby-htslib
 
 [![Gem Version](https://badge.fury.io/rb/htslib.svg)](https://badge.fury.io/rb/htslib)
 ![CI](https://github.com/kojix2/ruby-htslib/workflows/CI/badge.svg)
@@ -6,17 +6,22 @@
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
 [![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-:dna: [HTSlib](https://github.com/samtools/htslib) - high-throughput sequencing data manipulation - for Ruby
+:dna: [HTSlib](https://github.com/samtools/htslib) - for Ruby
+
+Ruby-htslib is the Ruby bindings to HTSlib, a C library for processing high throughput sequencing (HTS) data. 
+It will provide APIs to read and write file formats such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/).
 
 :apple: Feel free to fork it out if you can develop it! 
 
-:bowtie: Just a prototype. Pre-alpha stage.
+:bowtie: alpha stage.
 
 ## Requirements
 
-* [htslib](https://github.com/samtools/htslib)
+* [Ruby](https://github.com/ruby/ruby) 2.7 or above.
+* [HTSlib](https://github.com/samtools/htslib)
   * Ubuntu : `apt install libhts-dev`
   * macOS : `brew install htslib`
+  * Build from source code (see Development section)
 
 ## Installation
 
@@ -24,43 +29,51 @@
 gem install htslib
 ```
 
-If you installed htslib with Ubuntu/apt or Mac/homebrew, pkg-config will automatically detect the location of the shared library.
-
-Or you can set the environment variable `HTSLIBDIR`.
+If you have installed htslib with apt on Ubuntu or homebrew on Mac, [pkg-config](https://github.com/ruby-gnome/pkg-config) 
+will automatically detect the location of the shared library. 
+Alternatively, you can specify the directory of the shared library by setting the environment variable `HTSLIBDIR`.
 
 ```sh
 export HTSLIBDIR="/your/path/to/htslib" # libhts.so
 ```
 
-## Usage
+## Overview
+
+### Low level API
 
-HTS::FFI - Low-level API 
+`HTS::LibHTS` provides native functions. 
 
 ```ruby
 require 'htslib'
 
-a = HTS::FFI.hts_open("a.bam", "r")
-b = HTS::FFI.hts_get_format(a)
+a = HTS::LibHTS.hts_open("a.bam", "r")
+b = HTS::LibHTS.hts_get_format(a)
 p b[:category]
 p b[:format]
 ```
 
-A high-level API based on [hts-python](https://github.com/quinlan-lab/hts-python) is under development.
+Note: Managed struct is not used in ruby-htslib. You may need to free the memory by yourself.
+
+### High level API (Plan)
+
+`Cram` `Bam` `Bcf` `Faidx` `Tabix`
+
+A high-level API is under development. We will change and improve the API to make it better.
 
 ```ruby
 require 'htslib'
 
 bam = HTS::Bam.new("a.bam")
 
-bam.each do |aln|
-  p name:  aln.qname,
-    flag:  aln.flag,
-    start: aln.start + 1,
-    mpos:  aln.mate_pos + 1,
-    mqual: aln.mapping_quality,
-    seq:   aln.sequence,
-    cigar: aln.cigar.to_s,
-    qual:  aln.base_qualities.map { |i| (i + 33).chr }.join
+bam.each do |r|
+  p name:  r.qname,
+    flag:  r.flag,
+    start: r.start + 1,
+    mpos:  r.mate_pos + 1,
+    mqual: r.mapping_quality,
+    seq:   r.sequence,
+    cigar: r.cigar.to_s,
+    qual:  r.base_qualities.map { |i| (i + 33).chr }.join
 end
 ```
 
@@ -80,6 +93,17 @@ bundle exec rake htslib:build
 bundle exec rake test
 ```
 
+* Actively use the advanced features of Ruby.
+* Consider compatibility with [htslib.cr](https://github.com/bio-crystal/htslib.cr) to some extent.
+
+#### FFI Extensions
+
+* [ffi-bitfield](https://github.com/kojix2/ffi-bitfield) : Extension of Ruby-FFI to support bitfields.
+
+#### Automatic generation or automatic validation (Future plan)
+
++ [c2ffi](https://github.com/rpav/c2ffi) is a tool to create JSON format metadata from C header files. It is planned to use c2ffi to automatically generate bindings or tests.
+
 ## Contributing
 
 Ruby-htslib is a library under development, so even small improvements like typofix are welcome! Please feel free to send us your pull requests.
@@ -88,12 +112,16 @@ Ruby-htslib is a library under development, so even small improvements like typo
 * Fix bugs and [submit pull requests](https://github.com/kojix2/ruby-htslib/pulls)
 * Write, clarify, or fix documentation
 * Suggest or add new features
+* [financial contributions](https://github.com/sponsors/kojix2)
 
 ## Links
 
 * [samtools/hts-spec](https://github.com/samtools/hts-specs)
-* [c2ffi](https://github.com/rpav/c2ffi)
+* [bioruby](https://github.com/bioruby/bioruby)
+
+## Funding support
 
+This work was supported partially by [Ruby Association Grant 2020](https://www.ruby.or.jp/en/news/20201022).
 ## License
 
 [MIT License](https://opensource.org/licenses/MIT).

data/lib/hts/bam/cigar.rb

@@ -7,22 +7,27 @@ module HTS
   class Bam
     class Cigar
       include Enumerable
-      OPS = 'MIDNSHP=XB'
 
-      def initialize(cigar, n_cigar)
-        @c = cigar
+      def initialize(pointer, n_cigar)
+        @pointer = pointer
         @n_cigar = n_cigar
       end
 
+      def to_ptr
+        @pointer
+      end
+
       def to_s
         to_a.flatten.join
       end
 
       def each
+        return to_enum(__method__) unless block_given?
+
         @n_cigar.times do |i|
-          c = @c[i].read_uint32
-          yield [FFI.bam_cigar_oplen(c),
-                 FFI.bam_cigar_opchr(c)]
+          c = @pointer[i].read_uint32
+          yield [LibHTS.bam_cigar_oplen(c),
+                 LibHTS.bam_cigar_opchr(c)]
         end
       end
     end

data/lib/hts/bam/flag.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+# Based on hts-nim
+# https://github.com/brentp/hts-nim/blob/master/src/hts/bam/flag.nim
+
+module HTS
+  class Bam
+    class Flag
+      def initialize(flag_value)
+        raise TypeError unless flag_value.is_a? Integer
+
+        @value = flag_value
+      end
+
+      attr_accessor :value
+
+      # BAM_FPAIRED        =    1
+      # BAM_FPROPER_PAIR   =    2
+      # BAM_FUNMAP         =    4
+      # BAM_FMUNMAP        =    8
+      # BAM_FREVERSE       =   16
+      # BAM_FMREVERSE      =   32
+      # BAM_FREAD1         =   64
+      # BAM_FREAD2         =  128
+      # BAM_FSECONDARY     =  256
+      # BAM_FQCFAIL        =  512
+      # BAM_FDUP           = 1024
+      # BAM_FSUPPLEMENTARY = 2048
+
+      # TODO: Enabling bitwise operations
+      # hts-nim
+      # proc `and`*(f: Flag, o: uint16): uint16 {. borrow, inline .}
+      # proc `and`*(f: Flag, o: Flag): uint16 {. borrow, inline .}
+      # proc `or`*(f: Flag, o: uint16): uint16 {. borrow .}
+      # proc `or`*(o: uint16, f: Flag): uint16 {. borrow .}
+      # proc `==`*(f: Flag, o: Flag): bool {. borrow, inline .}
+      # proc `==`*(f: Flag, o: uint16): bool {. borrow, inline .}
+      # proc `==`*(o: uint16, f: Flag): bool {. borrow, inline .}
+
+      def paired?
+        has_flag? LibHTS::BAM_FPAIRED
+      end
+
+      def proper_pair?
+        has_flag? LibHTS::BAM_FPROPER_PAIR
+      end
+
+      def unmapped?
+        has_flag? LibHTS::BAM_FUNMAP
+      end
+
+      def mate_unmapped?
+        has_flag? LibHTS::BAM_FMUNMAP
+      end
+
+      def reverse?
+        has_flag? LibHTS::BAM_FREVERSE
+      end
+
+      def mate_reverse?
+        has_flag? LibHTS::BAM_FMREVERSE
+      end
+
+      def read1?
+        has_flag? LibHTS::BAM_FREAD1
+      end
+
+      def read2?
+        has_flag? LibHTS::BAM_FREAD2
+      end
+
+      def secondary?
+        has_flag? LibHTS::BAM_FSECONDARY
+      end
+
+      def qcfail?
+        has_flag? LibHTS::BAM_FQCFAIL
+      end
+
+      def dup?
+        has_flag? LibHTS::BAM_FDUP
+      end
+
+      def supplementary?
+        has_flag? LibHTS::BAM_FSUPPLEMENTARY
+      end
+
+      def has_flag?(m)
+        (@value & m) != 0
+      end
+
+      def to_s
+        "0x#{format('%x', @value)}\t#{@value}\t#{LibHTS.bam_flag2str(@value)}"
+      end
+    end
+  end
+end

data/lib/hts/bam/header.rb

@@ -6,21 +6,31 @@
 module HTS
   class Bam
     class Header
-      attr_reader :h
+      def initialize(pointer)
+        @sam_hdr = pointer
+      end
+
+      def struct
+        @sam_hdr
+      end
+
+      def to_ptr
+        @sam_hdr.to_ptr
+      end
 
-      def initialize(h)
-        @h = h
+      def target_count
+        @sam_hdr[:n_targets]
       end
 
       # FIXME: better name?
       def seqs
-        Array.new(@h[:n_targets]) do |i|
-          FFI.sam_hdr_tid2name(@h, i)
+        Array.new(@sam_hdr[:n_targets]) do |i|
+          LibHTS.sam_hdr_tid2name(@sam_hdr, i)
         end
       end
 
-      def text
-        FFI.sam_hdr_str(@h)
+      def to_s
+        LibHTS.sam_hdr_str(@sam_hdr)
       end
     end
   end

data/lib/hts/bam/record.rb

@@ -0,0 +1,199 @@
+# frozen_string_literal: true
+
+# Based on hts-python
+# https://github.com/quinlan-lab/hts-python
+
+module HTS
+  class Bam
+    class Record
+      SEQ_NT16_STR = "=ACMGRSVTWYHKDBN"
+
+      def initialize(bam1_t, header)
+        @bam1 = bam1_t
+        @header = header
+      end
+
+      def struct
+        @bam1
+      end
+
+      def to_ptr
+        @bam1.to_ptr
+      end
+
+      attr_reader :header
+
+      # def initialize_copy
+      #   super
+      # end
+
+      def self.rom_sam_str; end
+
+      def tags; end
+
+      # returns the query name.
+      def qname
+        LibHTS.bam_get_qname(@bam1).read_string
+      end
+
+      # Set (query) name.
+      # def qname=(name)
+      #   raise 'Not Implemented'
+      # end
+
+      # returns the tid of the record or -1 if not mapped.
+      def tid
+        @bam1[:core][:tid]
+      end
+
+      # returns the tid of the mate or -1 if not mapped.
+      def mate_tid
+        @bam1[:core][:mtid]
+      end
+
+      # returns 0-based start position.
+      def start
+        @bam1[:core][:pos]
+      end
+
+      # returns end position of the read.
+      def stop
+        LibHTS.bam_endpos @bam1
+      end
+
+      # returns 0-based mate position
+      def mate_start
+        @bam1[:core][:mpos]
+      end
+      alias mate_pos mate_start
+
+      # returns the chromosome or '' if not mapped.
+      def chrom
+        return "" if tid == -1
+
+        LibHTS.sam_hdr_tid2name(@header, tid)
+      end
+
+      # returns the chromosome of the mate or '' if not mapped.
+      def mate_chrom
+        mtid = mate_tid
+        return "" if mtid == -1
+
+        LibHTS.sam_hdr_tid2name(@header, mtid)
+      end
+
+      def strand
+        LibHTS.bam_is_rev(@bam1) ? "-" : "+"
+      end
+
+      # def start=(v)
+      #   raise 'Not Implemented'
+      # end
+
+      # insert size
+      def isize
+        @bam1[:core][:isize]
+      end
+
+      # mapping quality
+      def mapping_quality
+        @bam1[:core][:qual]
+      end
+
+      # returns a `Cigar` object.
+      def cigar
+        Cigar.new(LibHTS.bam_get_cigar(@bam1), @bam1[:core][:n_cigar])
+      end
+
+      def qlen
+        LibHTS.bam_cigar2qlen(
+          @bam1[:core][:n_cigar],
+          LibHTS.bam_get_cigar(@bam1)
+        )
+      end
+
+      def rlen
+        LibHTS.bam_cigar2rlen(
+          @bam1[:core][:n_cigar],
+          LibHTS.bam_get_cigar(@bam1)
+        )
+      end
+
+      # return the read sequence
+      def sequence
+        r = LibHTS.bam_get_seq(@bam1)
+        seq = String.new
+        (@bam1[:core][:l_qseq]).times do |i|
+          seq << SEQ_NT16_STR[LibHTS.bam_seqi(r, i)]
+        end
+        seq
+      end
+
+      # return only the base of the requested index "i" of the query sequence.
+      def base_at(n)
+        n += @bam1[:core][:l_qseq] if n < 0
+        return "." if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_at(-1000)
+
+        r = LibHTS.bam_get_seq(@bam1)
+        SEQ_NT16_STR[LibHTS.bam_seqi(r, n)]
+      end
+
+      # return the base qualities
+      def base_qualities
+        q_ptr = LibHTS.bam_get_qual(@bam1)
+        q_ptr.read_array_of_uint8(@bam1[:core][:l_qseq])
+      end
+
+      # return only the base quality of the requested index "i" of the query sequence.
+      def base_quality_at(n)
+        n += @bam1[:core][:l_qseq] if n < 0
+        return 0 if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_quality_at(-1000)
+
+        q_ptr = LibHTS.bam_get_qual(@bam1)
+        q_ptr.get_uint8(n)
+      end
+
+      def flag_str
+        LibHTS.bam_flag2str(@bam1[:core][:flag])
+      end
+
+      # returns a `Flag` object.
+      def flag
+        Flag.new(@bam1[:core][:flag])
+      end
+
+      def tag(str)
+        aux = LibHTS.bam_aux_get(@bam1, str)
+        return nil if aux.null?
+
+        t = aux.read_string(1)
+
+        # A (character), B (general array),
+        # f (real number), H (hexadecimal array),
+        # i (integer), or Z (string).
+
+        case t
+        when "i", "I", "c", "C", "s", "S"
+          LibHTS.bam_aux2i(aux)
+        when "f", "d"
+          LibHTS.bam_aux2f(aux)
+        when "Z", "H"
+          LibHTS.bam_aux2Z(aux)
+        when "A" # char
+          LibHTS.bam_aux2A(aux).chr
+        end
+      end
+
+      def to_s
+        kstr = LibHTS::KString.new
+        raise "Failed to format bam record" if LibHTS.sam_format1(@header.struct, @bam1, kstr) == -1
+
+        kstr[:s]
+      end
+
+      # TODO:
+      # def eql?
+      # def hash
+    end
+  end
+end

data/lib/hts/bam.rb

@@ -3,76 +3,111 @@
 # Based on hts-python
 # https://github.com/quinlan-lab/hts-python
 
-require_relative 'bam/header'
-require_relative 'bam/cigar'
-require_relative 'bam/alignment'
+require_relative "bam/header"
+require_relative "bam/cigar"
+require_relative "bam/flag"
+require_relative "bam/record"
 
 module HTS
   class Bam
     include Enumerable
-    attr_reader :file_path, :mode, :header, :htf
 
-    def initialize(file_path, mode = 'r', create_index: nil, header: nil, fasta: nil)
-      @file_path = File.expand_path(file_path)
-      File.exist?(@file_path) || raise("No such SAM/BAM file - #{@file_path}")
+    attr_reader :file_path, :mode, :header
+    # HtfFile is FFI::BitStruct
+    attr_reader :htf_file
 
-      @mode = mode
-      @htf = FFI.hts_open(@file_path, mode)
+    class << self
+      alias open new
+    end
 
-      if mode[0] == 'r'
-        @idx = FFI.sam_index_load(@htf, @file_path)
-        if (@idx.null? && create_index.nil?) || create_index
-          FFI.sam_index_build(file_path, -1)
-          @idx = FFI.sam_index_load(@htf, @file_path)
-          warn 'NO querying'
-        end
-        @header = Bam::Header.new(FFI.sam_hdr_read(@htf))
-        @b = FFI.bam_init1
+    def initialize(file_path, mode = "r", create_index: nil)
+      file_path = File.expand_path(file_path)
+
+      unless File.exist?(file_path)
+        message = "No such SAM/BAM file - #{file_path}"
+        raise message
+      end
 
+      @file_path = file_path
+      @mode      = mode
+      @htf_file  = LibHTS.hts_open(file_path, mode)
+      @header    = Bam::Header.new(LibHTS.sam_hdr_read(htf_file))
+
+      # read
+      if mode[0] == "r"
+        # load index
+        @idx = LibHTS.sam_index_load(htf_file, file_path)
+        # create index
+        if create_index || (@idx.null? && create_index.nil?)
+          warn "Create index for #{file_path}"
+          LibHTS.sam_index_build(file_path, -1)
+          @idx = LibHTS.sam_index_load(htf_file, file_path)
+        end
       else
-        # FIXME
-        raise 'not implemented yet.'
+        # FIXME: implement
+        raise "not implemented yet."
+      end
 
+      # IO like API
+      if block_given?
+        begin
+          yield self
+        ensure
+          close
+        end
       end
     end
 
-    def self.header_from_fasta; end
+    def struct
+      htf_file
+    end
+
+    def to_ptr
+      htf_file.to_ptr
+    end
 
     def write(alns)
       alns.each do
-        FFI.sam_write1(@htf, @header, alns.b) > 0 || raise
+        LibHTS.sam_write1(htf_file, header, alns.b) > 0 || raise
       end
     end
 
     # Close the current file.
     def close
-      FFI.hts_close(@htf)
+      LibHTS.hts_close(htf_file)
     end
 
     # Flush the current file.
     def flush
-      raise
-      # FFI.bgzf_flush(@htf.fp.bgzf)
+      # LibHTS.bgzf_flush(@htf_file.fp.bgzf)
     end
 
-    def each(&block)
+    def each
       # Each does not always start at the beginning of the file.
       # This is the common behavior of IO objects in Ruby.
       # This may change in the future.
-      block.call(Alignment.new(@b, @header.h)) while FFI.sam_read1(@htf, @header.h, @b) > 0
+      return to_enum(__method__) unless block_given?
+
+      while LibHTS.sam_read1(htf_file, header, bam1 = LibHTS.bam_init1) > 0
+        record = Record.new(bam1, header)
+        yield record
+      end
+      self
     end
 
     # query [WIP]
     def query(region)
-      qiter = FFI.sam_itr_querys(@idx, @header.h, region)
+      qiter = LibHTS.sam_itr_querys(@idx, header, region)
       begin
-        slen = FFI.sam_itr_next(@htf, qiter, @b)
+        bam1 = LibHTS.bam_init1
+        slen = LibHTS.sam_itr_next(htf_file, qiter, bam1)
         while slen > 0
-          yield Alignment.new(@b, @header.h)
-          slen = FFI.sam_itr_next(@htf, qiter, @b)
+          yield Record.new(bam1, header)
+          bam1 = LibHTS.bam_init1
+          slen = LibHTS.sam_itr_next(htf_file, qiter, bam1)
         end
       ensure
-        FFI.hts_itr_destroy(qiter)
+        LibHTS.hts_itr_destroy(qiter)
       end
     end
   end