htslib 0.0.8 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dd81ebe87741d1019acd341189ee0f8fb6ef09d3a10690686b3d40bdfea73076
-  data.tar.gz: 39a7fa296cc1797c7cd6317923b3336c67f0cf16e8c9f7eb8ae8dbef9e6c7f56
+  metadata.gz: 87b06f64c496150db76d689f2cd06ba61e8ff424511be781bf17abbdf5d92d6c
+  data.tar.gz: 10126e01757aafb8fbb18c4f5d2f58541511226ddf922ffb253d29ccaf6c2027
 SHA512:
-  metadata.gz: c349f647e798efc920492877807db88d03dfe88edf1434e733d38221ab15f1bea179ac524b3e26feb83c7063c878138918be3dcb30bcbbc583e0cc5bbbe1b8df
-  data.tar.gz: 120052721db361d285968331d2091f4bfe17b45a4204443a591141bc12aef3b153866077dc85ddff63451873800f78db377a151ecbec7084b1f5f409fe5507b0
+  metadata.gz: 2bb5806ae23d58192a74c567767de193ed21f2b8f8f87cf2bef4ef9beccc62da684ce5a0b8a6a2b96afbdbd84fb13c14b27ea2738a4d70829b7d3c1a57396430
+  data.tar.gz: 9e49efbb3bdce645013e0aaa33dec335204a0c62f48cd0ab4b5128314b97f6237b5faec6a0acf0059d8a276e26323d93ad53c4b5e7e2c36b273592b970b7dfe4

data/README.md

@@ -6,10 +6,7 @@
 [![DOI](https://zenodo.org/badge/247078205.svg)](https://zenodo.org/badge/latestdoi/247078205)
 [![Docs Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://rubydoc.info/gems/htslib)
 
-:dna: [HTSlib](https://github.com/samtools/htslib) - for Ruby
-
-Ruby-htslib is the Ruby bindings to HTSlib, a C library for processing high throughput sequencing (HTS) data. 
-It will provide APIs to read and write file formats such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/).
+Ruby-htslib is the [Ruby](https://www.ruby-lang.org) bindings to [HTSlib](https://github.com/samtools/htslib), a C library for high-throughput sequencing data formats. It allows you to read and write file formats commonly used in genomics, such as [SAM, BAM, VCF, and BCF](http://samtools.github.io/hts-specs/) in the Ruby language.
 
 :apple: Feel free to fork it out if you can develop it! 
 
@@ -17,7 +14,7 @@ It will provide APIs to read and write file formats such as [SAM, BAM, VCF, and
 
 ## Requirements
 
-* [Ruby](https://github.com/ruby/ruby) 2.7 or above.
+* [Ruby](https://github.com/ruby/ruby) 3.1 or above.
 * [HTSlib](https://github.com/samtools/htslib)
   * Ubuntu : `apt install libhts-dev`
   * macOS : `brew install htslib`
@@ -30,7 +27,7 @@ gem install htslib
 ```
 
 If you have installed htslib with apt on Ubuntu or homebrew on Mac, [pkg-config](https://github.com/ruby-gnome/pkg-config) 
-will automatically detect the location of the shared library. 
+will automatically detect the location of the shared library. If pkg-config does not work well, set `PKG_CONFIG_PATH`.
 Alternatively, you can specify the directory of the shared library by setting the environment variable `HTSLIBDIR`.
 
 ```sh
@@ -41,34 +38,35 @@ export HTSLIBDIR="/your/path/to/htslib" # libhts.so
 
 ### High level API
 
-A high-level API is under development. 
-Classes such as `Cram` `Bam` `Bcf` `Faidx` `Tabix` are partially implemented.
-
-Read SAM / BAM - Sequence Alignment Map file
+Read SAM / BAM / CRAM - Sequence Alignment Map file
 
 ```ruby
 require 'htslib'
 
-bam = HTS::Bam.new("a.bam")
+bam = HTS::Bam.open("test/fixtures/moo.bam")
 
 bam.each do |r|
-  p name:  r.qname,
-    flag:  r.flag,
-    pos:   r.start + 1,
-    mpos:  r.mate_start + 1,
-    mqual: r.mapping_quality,
-    seq:   r.sequence,
-    cigar: r.cigar.to_s,
-    qual:  r.base_qualities.map { |i| (i + 33).chr }.join
+  pp name: r.qname,
+     flag: r.flag,
+     chrm: r.chrom,
+     strt: r.pos + 1,
+     mapq: r.mapq,
+     cigr: r.cigar.to_s,
+     mchr: r.mate_chrom,
+     mpos: r.mpos + 1,
+     isiz: r.isize,
+     seqs: r.seq,
+     qual: r.qual.map { |i| (i + 33).chr }.join,
+     MC:   r.aux("MC")
 end
 
 bam.close
 ```
 
-Read VCF / BCF - Variant Call Format
+Read VCF / BCF - Variant Call Format file
 
 ```ruby
-bcf = HTS::Bcf.new("b.bcf")
+bcf = HTS::Bcf.open("b.bcf")
 
 bcf.each do |r|
   p chrom:  r.chrom,
@@ -77,17 +75,17 @@ bcf.each do |r|
     qual:   r.qual.round(2),
     ref:    r.ref,
     alt:    r.alt,
-    filter: r.filter
+    filter: r.filter,
+    info:   r.info.to_h,
+    format: r.format.to_h
 end
 
 bcf.close
 ```
 
-The methods for reading are implemented first. Methods for writing will be implemented in the coming days.
-
 ### Low level API
 
-`HTS::LibHTS` provides native functions. 
+`HTS::LibHTS` provides native C functions. 
 
 ```ruby
 require 'htslib'
@@ -98,19 +96,20 @@ p b[:category]
 p b[:format]
 ```
 
-Note: Only some C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually.
+Note: htslib makes extensive use of macro functions for speed. you cannot use C macro functions in Ruby if they are not reimplemented in ruby-htslib. Only small number of C structs are implemented with FFI's ManagedStruct, which frees memory when Ruby's garbage collection fires. Other structs will need to be freed manually. 
 
 ### Need more speed?
 
-Try [htslib.cr](https://github.com/bio-crystal/htslib.cr). htslib.cr is implemented in Crystal language and provides an API compatible with ruby-htslib. crsytal language is not as flexible as Ruby language. If you have a clear idea of the manipulation you want to do and need to perform it many times, then by all means try to implement a command line tool using htslib.cr. The Crystal language is very fast and can perform almost as well as the Rust and C languages.
+Try Crystal. [htslib.cr](https://github.com/bio-crystal/htslib.cr) is implemented in Crystal language and provides an API compatible with ruby-htslib. Crsytal language is not as flexible as Ruby language. You can not use `eval` methods, and you must always be careful with the data types. Writing one-time scripts in Crystal or playing with REPL may not be as much fun. However, if you have a clear idea of what you want to do in your mind, have already written code in Ruby, and need to run them over and over, try creating a command line tool in Crystal.  The Crystal language is fast, as fast as the Rust and C languages. It will give you great power to create tools.
 
 ## Documentation
 
+* [API Documentation (develop branch)](https://kojix2.github.io/ruby-htslib/)
 * [RubyDoc.info - HTSlib](https://rdoc.info/gems/htslib)
 
 ## Development
 
-To get started with development
+To get started with development:
 
 ```sh
 git clone --recursive https://github.com/kojix2/ruby-htslib
@@ -120,10 +119,16 @@ bundle exec rake htslib:build
 bundle exec rake test
 ```
 
+[GNU Autotools](https://en.wikipedia.org/wiki/GNU_Autotools) is required to compile htslib.
+
 Many macro functions are used in HTSlib. Since these macro functions cannot be called using FFI, they must be reimplemented in Ruby.
 
-* Actively use the advanced features of Ruby.
-* Consider compatibility with [htslib.cr](https://github.com/bio-crystal/htslib.cr) to some extent.
+* Use the new version of Ruby to take full advantage of Ruby's potential.
+  * This is possible because we have a small number of users. What a deal!
+* Remain compatibile with [htslib.cr](https://github.com/bio-crystal/htslib.cr).
+  * The most difficult part is the return value. In the Crystal language, methods are expected to return only one type. On the other hand, in the Ruby language, methods that return multiple classes are very common. For example, in the Crystal language, the compiler gets confused if the return value is one of six types: Int32, Int64, Float32, Float64, Nil, or String. In fact Crystal can do this. But the code gets a little messy. In Ruby, this is very common and doesn't cause any problems. 
+
+In the script directory, there are several tools to help implement ruby-htslib. These tools may be forked into independent repository in the future.
 
 #### FFI Extensions
 
@@ -131,7 +136,6 @@ Many macro functions are used in HTSlib. Since these macro functions cannot be c
 
 #### Automatic generation or automatic validation (Future plan)
 
-
 + [c2ffi](https://github.com/rpav/c2ffi) is a tool to create JSON format metadata from C header files. It is planned to use c2ffi to automatically generate bindings or tests.
 
 ## Contributing
@@ -144,15 +148,18 @@ Ruby-htslib is a library under development, so even small improvements like typo
 * Suggest or add new features
 * [financial contributions](https://github.com/sponsors/kojix2)
 
+
 ```
-Do you need commit rights to my repository?
+# Ownership and Commitment Rights
+
+Do you need commit rights to ruby-htslib repository?
 Do you want to get admin rights and take over the project?
 If so, please feel free to contact us @kojix2.
 ```
 
 #### Why do you implement htslib in a language like Ruby, which is not widely used in the bioinformatics?
 
-One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding to better understand how to use the file formats and tools. And that effort is still going on today...
+One of the greatest joys of using a minor language like Ruby in bioinformatics is that there is nothing stopping you from reinventing the wheel. Reinventing the wheel can be fun. But with languages like Python and R, where many bioinformatics masters work, there is no chance left for beginners to create htslib bindings. Bioinformatics file formats, libraries and tools are very complex and I don't know how to understand them. So I wanted to implement the HTSLib binding myself to better understand how the pioneers of bioinformatics felt when establishing the file format and how they created their tools. I hope one day we can work on bioinformatics using Ruby and Crystal languages, not to replace other languages such as Python and R, but to add new power and value to this advancing field.
 
 ## Links
 

data/lib/hts/bam/aux.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module HTS
+  class Bam < Hts
+    # Auxiliary record data
+    class Aux
+      def initialize(record)
+        @record = record
+      end
+
+      def get(key, type = nil)
+        aux = LibHTS.bam_aux_get(@record.struct, key)
+        return nil if aux.null?
+
+        type ||= aux.read_string(1)
+
+        # A (character), B (general array),
+        # f (real number), H (hexadecimal array),
+        # i (integer), or Z (string).
+
+        case type
+        when "i", "I", "c", "C", "s", "S"
+          LibHTS.bam_aux2i(aux)
+        when "f", "d"
+          LibHTS.bam_aux2f(aux)
+        when "Z", "H"
+          LibHTS.bam_aux2Z(aux)
+        when "A" # char
+          LibHTS.bam_aux2A(aux).chr
+        else
+          raise NotImplementedError, "type: #{t}"
+        end
+      end
+
+      def [](key)
+        get(key)
+      end
+    end
+  end
+end

data/lib/hts/bam/cigar.rb

@@ -1,33 +1,29 @@
 # frozen_string_literal: true
 
-# Based on hts-python
-# https://github.com/quinlan-lab/hts-python
-
 module HTS
-  class Bam
+  class Bam < Hts
+    # CIGAR string
     class Cigar
       include Enumerable
 
       def initialize(pointer, n_cigar)
-        @pointer = pointer
         @n_cigar = n_cigar
-      end
-
-      def to_ptr
-        @pointer
+        # Read the pointer before the memory is changed.
+        # Especially when called from a block of `each` iterator.
+        @c = pointer.read_array_of_uint32(n_cigar)
       end
 
       def to_s
-        to_a.flatten.join
+        map { |op, len| "#{len}#{op}" }.join
       end
 
       def each
         return to_enum(__method__) unless block_given?
 
-        @n_cigar.times do |i|
-          c = @pointer[i].read_uint32
-          yield [LibHTS.bam_cigar_oplen(c),
-                 LibHTS.bam_cigar_opchr(c)]
+        @c.each do |c|
+          op =  LibHTS.bam_cigar_opchr(c)
+          len = LibHTS.bam_cigar_oplen(c)
+          yield [op, len]
         end
       end
     end

data/lib/hts/bam/flag.rb

@@ -4,7 +4,8 @@
 # https://github.com/brentp/hts-nim/blob/master/src/hts/bam/flag.nim
 
 module HTS
-  class Bam
+  class Bam < Hts
+    # SAM flags
     class Flag
       def initialize(flag_value)
         raise TypeError unless flag_value.is_a? Integer
@@ -27,70 +28,38 @@ module HTS
       # BAM_FDUP           = 1024
       # BAM_FSUPPLEMENTARY = 2048
 
-      # TODO: Enabling bitwise operations
-      # hts-nim
-      # proc `and`*(f: Flag, o: uint16): uint16 {. borrow, inline .}
-      # proc `and`*(f: Flag, o: Flag): uint16 {. borrow, inline .}
-      # proc `or`*(f: Flag, o: uint16): uint16 {. borrow .}
-      # proc `or`*(o: uint16, f: Flag): uint16 {. borrow .}
-      # proc `==`*(f: Flag, o: Flag): bool {. borrow, inline .}
-      # proc `==`*(f: Flag, o: uint16): bool {. borrow, inline .}
-      # proc `==`*(o: uint16, f: Flag): bool {. borrow, inline .}
-
-      def paired?
-        has_flag? LibHTS::BAM_FPAIRED
-      end
-
-      def proper_pair?
-        has_flag? LibHTS::BAM_FPROPER_PAIR
-      end
-
-      def unmapped?
-        has_flag? LibHTS::BAM_FUNMAP
-      end
-
-      def mate_unmapped?
-        has_flag? LibHTS::BAM_FMUNMAP
-      end
-
-      def reverse?
-        has_flag? LibHTS::BAM_FREVERSE
-      end
-
-      def mate_reverse?
-        has_flag? LibHTS::BAM_FMREVERSE
-      end
-
-      def read1?
-        has_flag? LibHTS::BAM_FREAD1
-      end
-
-      def read2?
-        has_flag? LibHTS::BAM_FREAD2
-      end
-
-      def secondary?
-        has_flag? LibHTS::BAM_FSECONDARY
-      end
-
-      def qcfail?
-        has_flag? LibHTS::BAM_FQCFAIL
-      end
-
-      def dup?
-        has_flag? LibHTS::BAM_FDUP
+      # TODO: Enabling bitwise operations?
+
+      TABLE = { paired?: LibHTS::BAM_FPAIRED,
+                proper_pair?: LibHTS::BAM_FPROPER_PAIR,
+                unmapped?: LibHTS::BAM_FUNMAP,
+                mate_unmapped?: LibHTS::BAM_FMUNMAP,
+                reverse?: LibHTS::BAM_FREVERSE,
+                mate_reverse?: LibHTS::BAM_FMREVERSE,
+                read1?: LibHTS::BAM_FREAD1,
+                read2?: LibHTS::BAM_FREAD2,
+                secondary?: LibHTS::BAM_FSECONDARY,
+                qcfail?: LibHTS::BAM_FQCFAIL,
+                duplicate?: LibHTS::BAM_FDUP,
+                supplementary?: LibHTS::BAM_FSUPPLEMENTARY }.freeze
+
+      TABLE.each do |name, v|
+        define_method(name) do
+          has_flag?(v)
+        end
       end
 
-      def supplementary?
-        has_flag? LibHTS::BAM_FSUPPLEMENTARY
+      def has_flag?(f)
+        (@value & f) != 0
       end
 
-      def has_flag?(m)
-        (@value & m) != 0
+      def to_i
+        @value
       end
 
       def to_s
-        "0x#{format('%x', @value)}\t#{@value}\t#{LibHTS.bam_flag2str(@value)}"
+        LibHTS.bam_flag2str(@value)
+        # "0x#{format('%x', @value)}\t#{@value}\t#{LibHTS.bam_flag2str(@value)}"
       end
     end
   end

data/lib/hts/bam/header.rb

@@ -1,10 +1,8 @@
 # frozen_string_literal: true
 
-# Based on hts-python
-# https://github.com/quinlan-lab/hts-python
-
 module HTS
-  class Bam
+  class Bam < Hts
+    # A class for working with alignment header.
     class Header
       def initialize(hts_file)
         @sam_hdr = LibHTS.sam_hdr_read(hts_file)
@@ -28,7 +26,7 @@ module HTS
         end
       end
 
-      def target_lengths
+      def target_len
         Array.new(target_count) do |i|
           LibHTS.sam_hdr_tid2len(@sam_hdr, i)
         end

data/lib/hts/bam/record.rb

@@ -1,13 +1,17 @@
 # frozen_string_literal: true
 
-# Based on hts-python
-# https://github.com/quinlan-lab/hts-python
+require_relative "flag"
+require_relative "cigar"
+require_relative "aux"
 
 module HTS
-  class Bam
+  class Bam < Hts
+    # A class for working with alignment records.
     class Record
       SEQ_NT16_STR = "=ACMGRSVTWYHKDBN"
 
+      attr_reader :header
+
       def initialize(bam1_t, header)
         @bam1 = bam1_t
         @header = header
@@ -21,16 +25,6 @@ module HTS
         @bam1.to_ptr
       end
 
-      attr_reader :header
-
-      # def initialize_copy
-      #   super
-      # end
-
-      def self.rom_sam_str; end
-
-      def tags; end
-
       # returns the query name.
       def qname
         LibHTS.bam_get_qname(@bam1).read_string
@@ -46,26 +40,52 @@ module HTS
         @bam1[:core][:tid]
       end
 
+      def tid=(tid)
+        @bam1[:core][:tid] = tid
+      end
+
       # returns the tid of the mate or -1 if not mapped.
-      def mate_tid
+      def mtid
         @bam1[:core][:mtid]
       end
 
+      def mtid=(mtid)
+        @bam1[:core][:mtid] = mtid
+      end
+
       # returns 0-based start position.
-      def start
+      def pos
         @bam1[:core][:pos]
       end
 
-      # returns end position of the read.
-      def stop
-        LibHTS.bam_endpos @bam1
+      def pos=(pos)
+        @bam1[:core][:pos] = pos
       end
 
       # returns 0-based mate position
-      def mate_start
+      def mate_pos
         @bam1[:core][:mpos]
       end
-      alias mate_pos mate_start
+
+      def mate_pos=(mpos)
+        @bam1[:core][:mpos] = mpos
+      end
+
+      alias mpos mate_pos
+      alias mpos= mate_pos=
+
+      def bin
+        @bam1[:core][:bin]
+      end
+
+      def bin=(bin)
+        @bam1[:core][:bin] = bin
+      end
+
+      # returns end position of the read.
+      def endpos
+        LibHTS.bam_endpos @bam1
+      end
 
       # returns the chromosome or '' if not mapped.
       def chrom
@@ -74,37 +94,43 @@ module HTS
         LibHTS.sam_hdr_tid2name(@header, tid)
       end
 
-      # returns the chromosome or '' if not mapped.
-      def contig
-        chrom
-      end
+      alias contig chrom
 
       # returns the chromosome of the mate or '' if not mapped.
       def mate_chrom
-        mtid = mate_tid
         return "" if mtid == -1
 
         LibHTS.sam_hdr_tid2name(@header, mtid)
       end
 
+      alias mate_contig mate_chrom
+
+      # Get strand information.
       def strand
         LibHTS.bam_is_rev(@bam1) ? "-" : "+"
       end
 
-      # def start=(v)
-      #   raise 'Not Implemented'
-      # end
-
       # insert size
       def insert_size
         @bam1[:core][:isize]
       end
 
+      def insert_size=(isize)
+        @bam1[:core][:isize] = isize
+      end
+
+      alias isize insert_size
+      alias isize= insert_size=
+
       # mapping quality
-      def mapping_quality
+      def mapq
         @bam1[:core][:qual]
       end
 
+      def mapq=(mapq)
+        @bam1[:core][:qual] = mapq
+      end
+
       # returns a `Cigar` object.
       def cigar
         Cigar.new(LibHTS.bam_get_cigar(@bam1), @bam1[:core][:n_cigar])
@@ -125,7 +151,7 @@ module HTS
       end
 
       # return the read sequence
-      def sequence
+      def seq
         r = LibHTS.bam_get_seq(@bam1)
         seq = String.new
         (@bam1[:core][:l_qseq]).times do |i|
@@ -133,59 +159,74 @@ module HTS
         end
         seq
       end
+      alias sequence seq
+
+      def len
+        @bam1[:core][:l_qseq]
+      end
 
       # return only the base of the requested index "i" of the query sequence.
-      def base_at(n)
+      def base(n)
         n += @bam1[:core][:l_qseq] if n < 0
-        return "." if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_at(-1000)
+        return "." if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base(-1000)
 
         r = LibHTS.bam_get_seq(@bam1)
         SEQ_NT16_STR[LibHTS.bam_seqi(r, n)]
       end
 
       # return the base qualities
-      def base_qualities
+      def qual
         q_ptr = LibHTS.bam_get_qual(@bam1)
         q_ptr.read_array_of_uint8(@bam1[:core][:l_qseq])
       end
 
       # return only the base quality of the requested index "i" of the query sequence.
-      def base_quality_at(n)
+      def base_qual(n)
         n += @bam1[:core][:l_qseq] if n < 0
-        return 0 if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_quality_at(-1000)
+        return 0 if (n >= @bam1[:core][:l_qseq]) || (n < 0) # eg. base_qual(-1000)
 
         q_ptr = LibHTS.bam_get_qual(@bam1)
         q_ptr.get_uint8(n)
       end
 
-      def flag_str
-        LibHTS.bam_flag2str(@bam1[:core][:flag])
-      end
-
       # returns a `Flag` object.
       def flag
         Flag.new(@bam1[:core][:flag])
       end
 
-      def tag(str)
-        aux = LibHTS.bam_aux_get(@bam1, str)
-        return nil if aux.null?
+      def flag=(flag)
+        case flag
+        when Integer
+          @bam1[:core][:flag] = flag
+        when Flag
+          @bam1[:core][:flag] = flag.value
+        else
+          raise "Invalid flag type: #{flag.class}"
+        end
+      end
+
+      # retruns the auxillary fields.
+      def aux(key = nil)
+        aux = Aux.new(self)
+        if key
+          aux.get(key)
+        else
+          aux
+        end
+      end
+
+      # TODO: add a method to get the auxillary fields as a hash.
+
+      # TODO: add a method to set the auxillary fields.
 
-        t = aux.read_string(1)
+      # TODO: add a method to remove the auxillary fields.
 
-        # A (character), B (general array),
-        # f (real number), H (hexadecimal array),
-        # i (integer), or Z (string).
+      # TODO: add a method to set variable length data (qname, cigar, seq, qual).
 
-        case t
-        when "i", "I", "c", "C", "s", "S"
-          LibHTS.bam_aux2i(aux)
-        when "f", "d"
-          LibHTS.bam_aux2f(aux)
-        when "Z", "H"
-          LibHTS.bam_aux2Z(aux)
-        when "A" # char
-          LibHTS.bam_aux2A(aux).chr
+      # Calling flag is delegated to the Flag object.
+      Flag::TABLE.each_key do |m|
+        define_method(m) do
+          flag.send(m)
         end
       end