htslib 0.0.1 → 0.0.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5c0c00377d4485fa702cc429c3005e98d1d095842d51fa6d27346fad9891c280
-  data.tar.gz: 56e680d3f890ef1df37f2d99644c13bde250a130ca46e5d6b756ca16366fe81e
+  metadata.gz: 3eb6fbb228a9fa0642c55cdfa39d5189b311df191725954ae46b8dda135dc8c7
+  data.tar.gz: c90ad39aa4919cefa56e534a706bf9659bba9c6ad69eb374491092ce16ae93f6
 SHA512:
-  metadata.gz: 1a38ecbb819edb2f81e5b76250c8ef070bf5d252925fecb859a4083462a7934feefd8ac50b2628f9f3912915982354bd61d424224302b45e9e2911321469bba6
-  data.tar.gz: 01d5ac7826b5db6b6838986d05801b69c76320bbb4a89029cf21a6f01031c9bcc23c287526fbda02c8a2c0a32453e53ed6d689373754680c584f969214c58f6f
+  metadata.gz: 392ea7c6acb5b86e9e1c1a29f066049c54954edfd277be30541aee433a7349f9d74df09e663ec0269f30d6b87ff47d66a4a668470d3aa536e25a2058cf963546
+  data.tar.gz: 9cc235f8876b8fd4339593eb84a0f2cb8b605b9c82b624cd0f97691b4109a672f55ae90447bba17fb5efcb8257279ad8c2d87234bca390b1624c7aa8f9a7014e

data/README.md

@@ -1,15 +1,26 @@
-# HTSlib
+# ruby-htslib
 
 [![Gem Version](https://badge.fury.io/rb/htslib.svg)](https://badge.fury.io/rb/htslib)
 ![CI](https://github.com/kojix2/ruby-htslib/workflows/CI/badge.svg)
 [![The MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE.txt)
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
+[![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-:dna: [HTSlib](https://github.com/samtools/htslib) - high-throughput sequencing data manipulation - for Ruby
+:dna: [HTSlib](https://github.com/samtools/htslib) - for Ruby
+
+Ruby-htslib is the Ruby bindings to HTSlib, a C library for processing high throughput sequencing (HTS) data. 
+It will provide APIs to read and write file formats such as SAM, BAM, VCF, and BCF.
 
 :apple: Feel free to fork it out if you can develop it! 
 
-:bowtie: Just a prototype. 
+:bowtie: alpha stage.
+## Requirements
+
+* [Ruby](https://github.com/ruby/ruby) 2.7 or above.
+* [HTSlib](https://github.com/samtools/htslib)
+  * Ubuntu : `apt install libhts-dev`
+  * macOS : `brew install htslib`
+  * Build from source code (see Development section)
 
 ## Installation
 
@@ -17,30 +28,53 @@
 gem install htslib
 ```
 
-Set environment variable HTSLIBDIR. 
+If you have installed htslib with apt on Ubuntu or homebrew on Mac, [pkg-config](https://github.com/ruby-gnome/pkg-config) 
+will automatically detect the location of the shared library. 
+Alternatively, you can specify the directory of the shared library by setting the environment variable `HTSLIBDIR`.
 
 ```sh
-export HTSLIBDIR="/your/path/to/htslib"
+export HTSLIBDIR="/your/path/to/htslib" # libhts.so
 ```
 
-## Requirements
-
-* [htslib](https://github.com/samtools/htslib)
+## Overview
 
-## Usage
+### Low level API
 
-HTS::FFI - Low-level API 
+`HTS::LibHTS` provides native functions. 
 
 ```ruby
 require 'htslib'
 
-a = HTS::FFI.hts_open("a.bam", "r")
-b = HTS::FFI.hts_get_format(a)
+a = HTS::LibHTS.hts_open("a.bam", "r")
+b = HTS::LibHTS.hts_get_format(a)
 p b[:category]
 p b[:format]
 ```
 
-A high-level API based on [hts-python](https://github.com/quinlan-lab/hts-python) is under development.
+Note: Managed struct is not used in ruby-htslib. You may need to free the memory by yourself.
+
+### High level API (Plan)
+
+`Cram` `Bam` `Bcf` `Faidx` `Tabix`
+
+A high-level API is under development. We will change and improve the API to make it better.
+
+```ruby
+require 'htslib'
+
+bam = HTS::Bam.new("a.bam")
+
+bam.each do |r|
+  p name:  r.qname,
+    flag:  r.flag,
+    start: r.start + 1,
+    mpos:  r.mate_pos + 1,
+    mqual: r.mapping_quality,
+    seq:   r.sequence,
+    cigar: r.cigar.to_s,
+    qual:  r.base_qualities.map { |i| (i + 33).chr }.join
+end
+```
 
 ## Documentation
 
@@ -51,25 +85,42 @@ A high-level API based on [hts-python](https://github.com/quinlan-lab/hts-python
 To get started with development
 
 ```sh
-git clone --recurse-submodules https://github.com/kojix2/ruby-htslib
+git clone --recursive https://github.com/kojix2/ruby-htslib
 cd ruby-htslib
 bundle install
-bundle exec rake htslib:compile
-bundle exec rake spec
+bundle exec rake htslib:build
+bundle exec rake test
 ```
 
+We plan to actively use the new features of Ruby. Since the number of users is small, backward compatibility is not important.
+On the other hand, we will consider compatibility with [Crystal](https://github.com/bio-crystal/htslib.cr) to some extent.
+
+#### FFI Extensions
+
+* [ffi-bitfield](https://github.com/kojix2/ffi-bitfield) : Extension of Ruby-FFI to support bitfields.
+
+#### Automatic generation or automatic validation (Future plan)
+
++ [c2ffi](https://github.com/rpav/c2ffi) is a tool to create JSON format metadata from C header files. It is planned to use c2ffi to automatically generate bindings or tests.
+
 ## Contributing
 
+Ruby-htslib is a library under development, so even small improvements like typofix are welcome! Please feel free to send us your pull requests.
+
 * [Report bugs](https://github.com/kojix2/ruby-htslib/issues)
 * Fix bugs and [submit pull requests](https://github.com/kojix2/ruby-htslib/pulls)
 * Write, clarify, or fix documentation
 * Suggest or add new features
+* [financial contributions](https://github.com/sponsors/kojix2)
 
 ## Links
 
 * [samtools/hts-spec](https://github.com/samtools/hts-specs)
-* [c2ffi](https://github.com/rpav/c2ffi)
+* [bioruby](https://github.com/bioruby/bioruby)
+
+## Funding support
 
+This work was supported partially by [Ruby Association Grant 2020](https://www.ruby.or.jp/en/news/20201022).
 ## License
 
 [MIT License](https://opensource.org/licenses/MIT).

data/lib/hts/bam/cigar.rb

@@ -7,22 +7,25 @@ module HTS
   class Bam
     class Cigar
       include Enumerable
-      OPS = 'MIDNSHP=XB'
 
-      def initialize(cigar, n_cigar)
-        @c = cigar
+      def initialize(pointer, n_cigar)
+        @pointer = pointer
         @n_cigar = n_cigar
       end
 
+      def to_ptr
+        @pointer
+      end
+
       def to_s
         to_a.flatten.join
       end
 
       def each
         @n_cigar.times do |i|
-          c = @c[i].read_uint32
-          yield [FFI.bam_cigar_oplen(c),
-                 FFI.bam_cigar_opchr(c)]
+          c = @pointer[i].read_uint32
+          yield [LibHTS.bam_cigar_oplen(c),
+                 LibHTS.bam_cigar_opchr(c)]
         end
       end
     end

data/lib/hts/bam/flag.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+# Based on hts-nim
+# https://github.com/brentp/hts-nim/blob/master/src/hts/bam/flag.nim
+
+module HTS
+  class Bam
+    class Flag
+      def initialize(flag_value)
+        raise TypeError unless flag_value.is_a? Integer
+
+        @value = flag_value
+      end
+
+      attr_accessor :value
+
+      # BAM_FPAIRED        =    1
+      # BAM_FPROPER_PAIR   =    2
+      # BAM_FUNMAP         =    4
+      # BAM_FMUNMAP        =    8
+      # BAM_FREVERSE       =   16
+      # BAM_FMREVERSE      =   32
+      # BAM_FREAD1         =   64
+      # BAM_FREAD2         =  128
+      # BAM_FSECONDARY     =  256
+      # BAM_FQCFAIL        =  512
+      # BAM_FDUP           = 1024
+      # BAM_FSUPPLEMENTARY = 2048
+
+      # TODO: Enabling bitwise operations
+      # hts-nim
+      # proc `and`*(f: Flag, o: uint16): uint16 {. borrow, inline .}
+      # proc `and`*(f: Flag, o: Flag): uint16 {. borrow, inline .}
+      # proc `or`*(f: Flag, o: uint16): uint16 {. borrow .}
+      # proc `or`*(o: uint16, f: Flag): uint16 {. borrow .}
+      # proc `==`*(f: Flag, o: Flag): bool {. borrow, inline .}
+      # proc `==`*(f: Flag, o: uint16): bool {. borrow, inline .}
+      # proc `==`*(o: uint16, f: Flag): bool {. borrow, inline .}
+
+      def paired?
+        has_flag? LibHTS::BAM_FPAIRED
+      end
+
+      def proper_pair?
+        has_flag? LibHTS::BAM_FPROPER_PAIR
+      end
+
+      def unmapped?
+        has_flag? LibHTS::BAM_FUNMAP
+      end
+
+      def mate_unmapped?
+        has_flag? LibHTS::BAM_FMUNMAP
+      end
+
+      def reverse?
+        has_flag? LibHTS::BAM_FREVERSE
+      end
+
+      def mate_reverse?
+        has_flag? LibHTS::BAM_FMREVERSE
+      end
+
+      def read1?
+        has_flag? LibHTS::BAM_FREAD1
+      end
+
+      def read2?
+        has_flag? LibHTS::BAM_FREAD2
+      end
+
+      def secondary?
+        has_flag? LibHTS::BAM_FSECONDARY
+      end
+
+      def qcfail?
+        has_flag? LibHTS::BAM_FQCFAIL
+      end
+
+      def dup?
+        has_flag? LibHTS::BAM_FDUP
+      end
+
+      def supplementary?
+        has_flag? LibHTS::BAM_FSUPPLEMENTARY
+      end
+
+      def has_flag?(o)
+        @value[o] != 0
+      end
+    end
+  end
+end

data/lib/hts/bam/header.rb

@@ -6,21 +6,27 @@
 module HTS
   class Bam
     class Header
-      attr_reader :h
+      def initialize(pointer)
+        @sam_hdr = pointer
+      end
+
+      def struct
+        @sam_hdr
+      end
 
-      def initialize(h)
-        @h = h
+      def to_ptr
+        @sam_hdr.to_ptr
       end
 
       # FIXME: better name?
       def seqs
-        Array.new(@h[:n_targets]) do |i|
-          FFI.sam_hdr_tid2name(@h, i)
+        Array.new(@sam_hdr[:n_targets]) do |i|
+          LibHTS.sam_hdr_tid2name(@sam_hdr, i)
         end
       end
 
       def text
-        FFI.sam_hdr_str(@h)
+        LibHTS.sam_hdr_str(@sam_hdr)
       end
     end
   end

data/lib/hts/bam/record.rb

@@ -0,0 +1,195 @@
+# frozen_string_literal: true
+
+# Based on hts-python
+# https://github.com/quinlan-lab/hts-python
+
+module HTS
+  class Bam
+    class Record
+      SEQ_NT16_STR = "=ACMGRSVTWYHKDBN"
+
+      def initialize(bam1_t, header)
+        @bam1 = bam1_t
+        @header = header
+      end
+
+      def struct
+        @bam1
+      end
+
+      def to_ptr
+        @bam1.to_ptr
+      end
+
+      attr_reader :header
+
+      # def initialize_copy
+      #   super
+      # end
+
+      def self.rom_sam_str; end
+
+      def tags; end
+
+      # returns the query name.
+      def qname
+        LibHTS.bam_get_qname(@bam1).read_string
+      end
+
+      # Set (query) name.
+      # def qname=(name)
+      #   raise 'Not Implemented'
+      # end
+
+      # returns the tid of the record or -1 if not mapped.
+      def tid
+        @bam1[:core][:tid]
+      end
+
+      # returns the tid of the mate or -1 if not mapped.
+      def mate_tid
+        @bam1[:core][:mtid]
+      end
+
+      # returns 0-based start position.
+      def start
+        @bam1[:core][:pos]
+      end
+
+      # returns end position of the read.
+      def stop
+        LibHTS.bam_endpos @bam1
+      end
+
+      # returns 0-based mate position
+      def mate_start
+        @bam1[:core][:mpos]
+      end
+      alias mate_pos mate_start
+
+      # returns the chromosome or '' if not mapped.
+      def chrom
+        tid = @bam1[:core][:tid]
+        return "" if tid == -1
+
+        LibHTS.sam_hdr_tid2name(@header, tid)
+      end
+
+      # returns the chromosome of the mate or '' if not mapped.
+      def mate_chrom
+        tid = @bam1[:core][:mtid]
+        return "" if tid == -1
+
+        LibHTS.sam_hdr_tid2name(@header, tid)
+      end
+
+      def strand
+        LibHTS.bam_is_rev(@bam1) ? "-" : "+"
+      end
+
+      # def start=(v)
+      #   raise 'Not Implemented'
+      # end
+
+      # insert size
+      def isize
+        @bam1[:core][:isize]
+      end
+
+      # mapping quality
+      def mapping_quality
+        @bam1[:core][:qual]
+      end
+
+      # returns a `Cigar` object.
+      def cigar
+        Cigar.new(LibHTS.bam_get_cigar(@bam1), @bam1[:core][:n_cigar])
+      end
+
+      def qlen
+        LibHTS.bam_cigar2qlen(
+          @bam1[:core][:n_cigar],
+          LibHTS.bam_get_cigar(@bam1)
+        )
+      end
+
+      def rlen
+        LibHTS.bam_cigar2rlen(
+          @bam1[:core][:n_cigar],
+          LibHTS.bam_get_cigar(@bam1)
+        )
+      end
+
+      # return the read sequence
+      def sequence
+        r = LibHTS.bam_get_seq(@bam1)
+        seq = String.new
+        (@bam1[:core][:l_qseq]).times do |i|
+          seq << SEQ_NT16_STR[LibHTS.bam_seqi(r, i)]
+        end
+        seq
+      end
+
+      # return only the base of the requested index "i" of the query sequence.
+      def base_at(n)
+        n += @bam1[:core][:l_qseq] if n < 0
+        return "." if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_at(-1000)
+
+        r = LibHTS.bam_get_seq(@bam1)
+        SEQ_NT16_STR[LibHTS.bam_seqi(r, n)]
+      end
+
+      # return the base qualities
+      def base_qualities
+        q_ptr = LibHTS.bam_get_qual(@bam1)
+        q_ptr.read_array_of_uint8(@bam1[:core][:l_qseq])
+      end
+
+      # return only the base quality of the requested index "i" of the query sequence.
+      def base_quality_at(n)
+        n += @bam1[:core][:l_qseq] if n < 0
+        return 0 if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_quality_at(-1000)
+
+        q_ptr = LibHTS.bam_get_qual(@bam1)
+        q_ptr.get_uint8(n)
+      end
+
+      def flag_str
+        LibHTS.bam_flag2str(@bam1[:core][:flag])
+      end
+
+      # returns a `Flag` object.
+      def flag
+        Flag.new(@bam1[:core][:flag])
+      end
+
+      def tag(str)
+        aux = LibHTS.bam_aux_get(@bam1, str)
+        return nil if aux.null?
+
+        t = aux.read_string(1)
+        case t
+        when "i", "I", "c", "C", "s", "S"
+          LibHTS.bam_aux2i(aux)
+        when "f", "d"
+          LibHTS.bam_aux2f(aux)
+        when "Z", "H"
+          LibHTS.bam_aux2Z(aux)
+        when "A"
+          LibHTS.bam_aux2A(aux)
+        end
+      end
+
+      def to_s
+        kstr = LibHTS::KString.new
+        raise "Failed to format bam record" if LibHTS.sam_format1(@header.struct, @bam1, kstr) == -1
+
+        kstr[:s]
+      end
+
+      # TODO:
+      # def eql?
+      # def hash
+    end
+  end
+end

data/lib/hts/bam.rb

@@ -3,76 +3,111 @@
 # Based on hts-python
 # https://github.com/quinlan-lab/hts-python
 
-require_relative 'bam/header'
-require_relative 'bam/cigar'
-require_relative 'bam/alignment'
+require_relative "bam/header"
+require_relative "bam/cigar"
+require_relative "bam/flag"
+require_relative "bam/record"
 
 module HTS
   class Bam
     include Enumerable
-    attr_reader :fname, :mode, :header, :htf
 
-    def initialize(fname, mode = 'r', create_index: nil, header: nil, fasta: nil)
-      @fname = File.expand_path(fname)
-      File.exist?(@fname) || raise("No such SAM/BAM file - #{@fname}")
+    attr_reader :file_path, :mode, :header
+    # HtfFile is FFI::BitStruct
+    attr_reader :htf_file
 
-      @mode = mode
-      @htf = FFI.hts_open(@fname, mode)
+    class << self
+      alias open new
+    end
 
-      if mode[0] == 'r'
-        @idx = FFI.sam_index_load(@htf, @fname)
-        if (@idx.null? && create_index.nil?) || create_index
-          FFI.sam_index_build(fname, -1)
-          @idx = FFI.sam_index_load(@htf, @fname)
-          warn 'NO querying'
-        end
-        @header = Bam::Header.new(FFI.sam_hdr_read(@htf))
-        @b = FFI.bam_init1
+    def initialize(file_path, mode = "r", create_index: nil)
+      file_path = File.expand_path(file_path)
+
+      unless File.exist?(file_path)
+        message = "No such SAM/BAM file - #{file_path}"
+        raise message
+      end
 
+      @file_path = file_path
+      @mode      = mode
+      @htf_file  = LibHTS.hts_open(file_path, mode)
+      @header    = Bam::Header.new(LibHTS.sam_hdr_read(htf_file))
+
+      # read
+      if mode[0] == "r"
+        # load index
+        @idx = LibHTS.sam_index_load(htf_file, file_path)
+        # create index
+        if create_index || (@idx.null? && create_index.nil?)
+          warn "Create index for #{file_path}"
+          LibHTS.sam_index_build(file_path, -1)
+          @idx = LibHTS.sam_index_load(htf_file, file_path)
+        end
       else
-        # FIXME
-        raise 'not implemented yet.'
+        # FIXME: implement
+        raise "not implemented yet."
+      end
 
+      # IO like API
+      if block_given?
+        begin
+          yield self
+        ensure
+          close
+        end
       end
     end
 
-    def self.header_from_fasta; end
+    def struct
+      htf_file
+    end
+
+    def to_ptr
+      htf_file.to_ptr
+    end
 
     def write(alns)
       alns.each do
-        FFI.sam_write1(@htf, @header, alns.b) > 0 || raise
+        LibHTS.sam_write1(htf_file, header, alns.b) > 0 || raise
       end
     end
 
     # Close the current file.
     def close
-      FFI.hts_close(@htf)
+      LibHTS.hts_close(htf_file)
     end
 
     # Flush the current file.
     def flush
-      raise
-      # FFI.bgzf_flush(@htf.fp.bgzf)
+      # LibHTS.bgzf_flush(@htf_file.fp.bgzf)
     end
 
-    def each(&block)
+    def each
       # Each does not always start at the beginning of the file.
       # This is the common behavior of IO objects in Ruby.
       # This may change in the future.
-      block.call(Alignment.new(@b, @header.h)) while FFI.sam_read1(@htf, @header.h, @b) > 0
+      return to_enum(__method__) unless block_given?
+
+      while LibHTS.sam_read1(htf_file, header, bam1 = LibHTS.bam_init1) > 0
+        record = Record.new(bam1, header)
+        yield record
+      end
+      self
     end
 
     # query [WIP]
     def query(region)
-      qiter = FFI.sam_itr_querys(@idx, @header.h, region)
+      qiter = LibHTS.sam_itr_querys(@idx, header, region)
       begin
-        slen = FFI.sam_itr_next(@htf, qiter, @b)
+        bam1 = LibHTS.bam_init1
+        slen = LibHTS.sam_itr_next(htf_file, qiter, bam1)
         while slen > 0
-          yield Alignment.new(@b, @header.h)
-          slen = FFI.sam_itr_next(@htf, qiter, @b)
+          yield Record.new(bam1, header)
+          bam1 = LibHTS.bam_init1
+          slen = LibHTS.sam_itr_next(htf_file, qiter, bam1)
         end
       ensure
-        FFI.hts_itr_destroy(qiter)
+        LibHTS.hts_itr_destroy(qiter)
       end
     end
   end