iop 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2631f9835b20e87513aed68b469ca251815ccd9517c1b6879555ab851bdae02a
+  data.tar.gz: b1a7d1794c962323692117181c1d5485343ed38d3fec65738af5c21076801fd4
+SHA512:
+  metadata.gz: d1cbb150db80d8de4aae4caa144618c143c84572949488508ee149b7fdcb8040385c966022a242fea08caf7533a96035fa0ee7e466cc8a583faec70b3e3c48b4
+  data.tar.gz: 42f468cd335d5087d4471b0f17fd190d0b7346611fe6f00da8a3a565c85042ce8a41f0a351cbe8bd40ef316311eaeae5456ec8fa79af71db3461621313435f6c

data/CHANGES.md

@@ -0,0 +1,3 @@
+# 0.1.0
+
+Initial release.

data/README.md

@@ -0,0 +1,90 @@
+# [IOP](https://bitbucket.org/fougas/iop) - the data processing pipeline construction framework for Ruby
+
+## Synopsis
+
+IOP is intended for construction of the data processing pipelines in a manner of UNIX shell pipes.
+
+Instead of the standard Ruby way of handling such I/O tasks in form of nested blocks the IOP offers a simpler flat chaining scheme.
+
+Consider the example:
+
+```ruby
+# One-liner example
+(FileReader.new('input.dat') | GzipCompressor.new | DigestComputer.new(MD5.new) | FileWriter.new('output.dat.gz')).process!
+```
+
+The above snippet reads input file and compresses it into the GZip-compatible output file simultaneously computing the MD5 hash of compressed data being written.
+
+The next snippet presents the incremental pipeline construction capability - a feature not easily implementable with the standard Ruby I/O blocks nesting.
+
+```ruby
+# Incremental pipeline construction example
+pipe  = FileReader.new('input')
+pipe |= GzipCompressor.new if need_compression?
+pipe |= FileWriter.new('output')
+pipe.process!
+```
+
+Here the GZip compression is made optional and is thrown in depending on external condition.
+
+## Features
+
+The following capabilities are currently implemented:
+
+- String splitting/merging
+- IO or local file reading/writing
+- FTP file reading/writing
+- Digest computing
+- GZip/Zlib (de)compression
+- Zstd (de)compression
+- Symmetric cipher (de,en)cryption
+- Random data generation
+
+## Basic usage
+
+- IOP is split into a set of files which should be required separately depending on which components are needed.
+
+```ruby
+require 'iop/file'
+require 'iop/zlib'
+require 'iop/digest'
+require 'iop/string'
+```
+
+- The `IOP` module can be included into current namespace to conserve some writing.
+
+```ruby
+include IOP
+```
+
+- A chain of processing objects is created either in-line or incrementally.
+
+```ruby
+pipe  = StringSplitter.new('Greetings from IOP', 10)
+pipe |= GzipCompressor.new | (digest = DigestComputer.new(MD5.new))
+pipe |= FileWriter.new('output.gz')
+```
+
+It is convenient to set local variables to the created instances which are expected to have some kind of valuable state.
+
+- The actual processing is initiated with the `process!` method.
+
+```ruby
+pipe.process!
+```
+
+The IOP instances do normally perform self-cleanup operations, such as closing file handles, network connections etc., even during exception handling.
+
+- The variable-bound instances can be then examined.
+
+```ruby
+puts digest.hexdigest
+```
+
+For further information refer to IOP documentation.
+
+# The end
+
+Cheers,
+
+Oleg A. Khlybov <[fougas@mail.ru](mailto:fougas@mail.ru)>

data/lib/iop/digest.rb

@@ -0,0 +1,45 @@
+require 'iop'
+require 'digest'
+
+
+module IOP
+
+
+  #
+  # Filter class to compute digest of the data being passed through.
+  # It can be used with digest computing classes from +digest+ and +openssl+ standard Ruby modules.
+  #
+  # ### Use case: generate 1024 bytes of random data and compute and print MD5 hash sum of it.
+  #
+  #     require 'iop/digest'
+  #     require 'iop/securerandom'
+  #     ( IOP::SecureRandomGenerator.new(1024) | ( d = IOP::DigestComputer.new(Digest::MD5.new)) ).process!
+  #     puts d.digest.hexdigest
+  #
+  # @since 0.1
+  #
+  class DigestComputer
+
+    include Feed
+    include Sink
+
+    # Returns digest object passed to constructor.
+    attr_reader :digest
+
+
+    # Creates class instance.
+    #
+    # @param digest computer instance to be fed with data
+    def initialize(digest)
+      @digest = digest
+    end
+
+    def process(data = nil)
+      digest.update(data) unless data.nil?
+      super
+    end
+
+  end
+
+
+end

data/lib/iop/file.rb

@@ -0,0 +1,209 @@
+require 'iop'
+
+
+module IOP
+
+
+  #
+  # Feed class to read data from external +IO+ stream and send it in blocks downstream.
+  #
+  # Contrary to {FileReader}, this class does not manage attached +IO+ instance, e.g.
+  # it makes no attempt to close it after processing.
+  #
+  # ### Use case: sequential read of two 1024-byte blocks from the same +IO+ stream.
+  #
+  #     require 'iop/file'
+  #     require 'iop/string'
+  #     io = File.new('input.dat', 'rb')
+  #     begin
+  #       ( IOP::IOReader.new(io, size: 1024) | (first = IOP::StringMerger.new) ).process!
+  #       ( IOP::IOReader.new(io, size: 1024, offset: 1024) | (second = IOP::StringMerger.new) ).process!
+  #     ensure
+  #       io.close
+  #     end
+  #     puts first.to_s
+  #     puts second.to_s
+  #
+  # @since 0.1
+  #
+  class IOReader
+
+    include Feed
+
+    # Creates class instance.
+    #
+    # @param io [IO] +IO+ instance to read data from
+    #
+    # @param size [Integer] total number of bytes to read; +nil+ value instructs to read until end-of-data is reached
+    #
+    # @param offset [Integer] offset in bytes from the stream start to seek to; +nil+ value means no seeking is performed
+    #
+    # @param block_size [Integer] size of blocks to read data with
+    def initialize(io, size: nil, offset: nil, block_size: DEFAULT_BLOCK_SIZE)
+      @block_size = size.nil? ? block_size : IOP.min(size, block_size)
+      @left = @size = size
+      @offset = offset
+      @io = io
+    end
+
+    def process!
+      @io.seek(@offset) unless @offset.nil?
+      data = IOP.allocate_string(@block_size)
+      loop do
+        read_size = @size.nil? ? @block_size : IOP.min(@left, @block_size)
+        break if read_size.zero?
+        if @io.read(read_size, data).nil?
+          if @size.nil?
+            break
+          else
+            raise EOFError, INSUFFICIENT_DATA
+          end
+        else
+          unless @left.nil?
+            @left -= data.size
+            raise IOError, EXTRA_DATA if @left < 0
+          end
+        end
+        process(data) unless data.size.zero?
+      end
+      process
+    end
+
+  end
+
+
+  #
+  # Feed class to read data from local file and send it in blocks downstream.
+  #
+  # Contrary to {IOReader}, this class manages underlying +IO+ instance in order to close it when the process is finished
+  # even if exception is risen.
+  #
+  # ### Use case: compute MD5 hash sum of the first 1024 bytes of a local file.
+  #     require 'iop/file'
+  #     require 'iop/digest'
+  #     ( IOP::FileReader.new('input.dat', size: 1024) | (d = IOP::DigestComputer.new(Digest::MD5.new)) ).process!
+  #     puts d.digest.hexdigest
+  #
+  # @since 0.1
+  #
+  class FileReader < IOReader
+
+    # Creates class instance.
+    #
+    # @param file [String] name of file to read from
+    #
+    # @param mode [String] open mode for the file; refer to {File} for details
+    #
+    # @param options [Hash] extra keyword parameters passed to {IOReader} constructor
+    def initialize(file, mode: 'rb', **options)
+      super(nil, **options)
+      @file = file
+      @mode = mode
+    end
+
+    def process!
+      @io = File.new(@file, @mode)
+      begin
+        super
+      ensure
+        @io.close
+      end
+    end
+
+  end
+
+
+  #
+  # Sink class to write received upstream data to external +IO+ stream.
+  #
+  # Contrary to {FileWriter}, this class does not manage attached +IO+ instance, e.g.
+  # it makes no attempt to close it after processing.
+  #
+  # ### Use case: concatenate two files.
+  #
+  #     require 'iop/file'
+  #     io = File.new('output.dat', 'wb')
+  #     begin
+  #       ( IOP::FileReader.new('file1.dat') | IOP::IOWriter.new(io) ).process!
+  #       ( IOP::FileReader.new('file2.dat') | IOP::IOWriter.new(io) ).process!
+  #     ensure
+  #       io.close
+  #     end
+  #
+  # @since 0.1
+  #
+  class IOWriter
+
+    include Sink
+
+    # Creates class instance.
+    #
+    # @param io [IO] +IO+ instance to write data to
+    def initialize(io)
+      @io = io
+    end
+
+    def process(data = nil)
+      @io.write(data)
+    end
+
+  end
+
+
+  #
+  # Sink class to write received upstream data to a local file.
+  #
+  # Contrary to {IOWriter}, this class manages underlying +IO+ instance in order to close it when the process is finished
+  # even if exception is risen.
+  #
+  # ### Use case: generate 1024 bytes of random data and write it to file.
+  #
+  #     require 'iop/file'
+  #     require 'iop/securerandom'
+  #     ( IOP::SecureRandomGenerator.new(1024) | IOP::FileWriter.new('random.dat') ).process!
+  #
+  # @since 0.1
+  #
+  class FileWriter < IOWriter
+
+    # Creates class instance.
+    #
+    # @param file [String] name of file to write to
+    #
+    # @param mode [String] open mode for the file; refer to {File} for details
+    def initialize(file, mode: 'wb')
+      super(nil)
+      @file = file
+      @mode = mode
+    end
+
+    def process!
+      @io = File.new(@file, @mode)
+      begin
+        super
+      ensure
+        @io.close
+      end
+    end
+
+  end
+
+
+  # @private
+  class IOSegmentReader
+
+    include BufferingFeed
+
+    def initialize(io, block_size: DEFAULT_BLOCK_SIZE)
+      @io = io
+      @block_size = block_size
+    end
+
+    private def next_data
+      @io.read(@block_size)
+    end
+
+  end
+
+
+end

data/lib/iop/net/ftp.rb

@@ -0,0 +1,206 @@
+require 'iop'
+require 'net/ftp'
+
+
+# @private
+class Net::FTP
+  public :transfercmd, :voidresp
+end
+
+
+module IOP
+
+
+  # @private
+  module FTPFile
+
+    private
+
+    def setup
+      if @ftp.is_a?(String)
+        @ftp = Net::FTP.open(@ftp, @options)
+        @managed = true
+        @ftp.login
+      end
+      unless @offset.nil?
+        # Override resume status when offset is specified remembering current value
+        @resume = @ftp.resume
+        @ftp.resume = true
+      end
+    end
+
+    def cleanup
+      # Revert resume status if previously overridden
+      @ftp.resume = @resume unless @resume.nil?
+      @ftp.close if @managed
+    end
+
+    def transfercmd(cmd, offset = nil)
+      @ftp.transfercmd(cmd, offset)
+    end
+
+    def voidresp
+      @ftp.voidresp
+    end
+
+  end
+
+  #
+  # Feed class to read file from FTP server.
+  #
+  # This class an adapter for the standard Ruby +Net::FTP+ class.
+  #
+  # ### Use case: retrieve file from FTP server and store it locally.
+  #
+  #     require 'iop/net/ftp'
+  #     ( IOP::FTPFileReader.new('ftp.gnu.org', '/pub/README') | IOP::FileWriter.new('README') ).process!
+  #
+  # @since 0.1
+  #
+  class FTPFileReader
+
+    include Feed
+    include FTPFile
+
+    # Creates class instance.
+    #
+    # @param ftp [String, Net::FTP] FTP server to connect to
+    #
+    # @param file [String] file name to process
+    #
+    # @param size [Integer] total number of bytes to read; +nil+ value instructs to read until end-of-data is reached
+    #
+    # @param offset [Integer] offset in bytes from the stream start to seek to; +nil+ value means no seeking is performed
+    #
+    # @param block_size [Integer] size of blocks to process data with
+    #
+    # @param options [Hash] extra keyword parameters passed to +Net::FTP+ constructor
+    #
+    # _ftp_ can be either a +String+ of +Net::FTP+ instance.
+    # If it is a string a corresponding +Net::FTP+ instance will be created with _options_ passed to its constructor.
+    #
+    # If _ftp_ is a string, a created FTP connection is managed, e.g. it is closed after the process is complete,
+    # otherwise supplied object is left as is and no closing is performed.
+    # This allows to reuse FTP connection for a sequence of operations.
+    #
+    # Refer to +Net::FTP+ documentation for available options.
+    def initialize(ftp, file, size: nil, offset: nil, block_size: DEFAULT_BLOCK_SIZE, **options)
+      @block_size = size.nil? ? block_size : IOP.min(size, block_size)
+      @left = @size = size
+      @options = options
+      @offset = offset
+      @file = file
+      @ftp = ftp
+    end
+
+    def process!
+      setup
+      begin
+        # FTP logic taken from Net::FTP#retrbinary
+        @io = transfercmd('RETR ' << @file, @offset)
+        begin
+          loop do
+            read_size = @size.nil? ? @block_size : IOP.min(@left, @block_size)
+            break if read_size.zero?
+            data = @io.read(read_size)
+            if data.nil?
+              if @size.nil?
+                break
+              else
+                raise EOFError, INSUFFICIENT_DATA
+              end
+            else
+              unless @left.nil?
+                @left -= data.size
+                raise IOError, EXTRA_DATA if @left < 0
+              end
+              process(data) unless data.size.zero?
+            end
+          end
+          process
+          @io.shutdown(Socket::SHUT_WR)
+          @io.read_timeout = 1
+          @io.read
+        ensure
+          @io.close
+        end
+        voidresp
+      ensure
+        cleanup
+      end
+    end
+
+  end
+
+
+
+  #
+  # Sink class to write file to FTP server.
+  #
+  # This class an adapter for the standard Ruby +Net::FTP+ class.
+  #
+  # ### Use case: store a number of files filled with random data to an FTP server reusing connection.
+  #
+  #     require 'iop/net/ftp'
+  #     require 'iop/securerandom'
+  #     ftp = Net::FTP.open('ftp.server', username: 'user')
+  #     begin
+  #       ftp.login
+  #       (1..3).each do |i|
+  #         ( IOP::SecureRandomGenerator.new(1024) | IOP::FTPFileWriter.new(ftp, "random#{i}.dat") ).process!
+  #       end
+  #     ensure
+  #       ftp.close
+  #     end
+  #
+  # @since 0.1
+  #
+  class FTPFileWriter
+
+    include Sink
+    include FTPFile
+
+    # Creates class instance.
+    #
+    # @param ftp [String, Net::FTP] FTP server to connect to
+    #
+    # @param file [String] file name to process
+    #
+    # @param options [Hash] extra keyword parameters passed to +Net::FTP+ constructor
+    #
+    # _ftp_ can be either a +String+ of +Net::FTP+ instance.
+    # If it is a string a corresponding +Net::FTP+ instance will be created with _options_ passed to its constructor.
+    #
+    # If _ftp_ is a string, a created FTP connection is managed, e.g. it is closed after the process is complete,
+    # otherwise supplied object is left as is and no closing is performed.
+    # This allows to reuse FTP connection for a sequence of operations.
+    def initialize(ftp, file, **options)
+      @options = options
+      @file = file
+      @ftp = ftp
+    end
+
+    def process!
+      setup
+      begin
+        # FTP logic taken from Net::FTP#storbinary
+        @io = transfercmd('STOR ' << @file)
+        begin
+          super
+        ensure
+          @io.close
+        end
+        voidresp
+      ensure
+        cleanup
+      end
+    end
+
+    def process(data = nil)
+      @io.write(data) unless data.nil?
+    end
+
+  end
+
+
+end

data/lib/iop/openssl.rb

@@ -0,0 +1,147 @@
+require 'iop'
+require 'openssl'
+
+
+module IOP
+
+
+  # Default cipher ID for OpenSSL adapters.
+  DEFAULT_OPENSSL_CIPHER = 'AES-256-CBC'.freeze
+
+
+  #
+  # Filter class to perform encryption with a symmetric key algorithm (ciphering) of the data passed through.
+  #
+  # The class is an adapter for +OpenSSL::Cipher+ & compatible classes.
+  #
+  # ### Use case: generate 1024 bytes of random data encrypt is with default cipher algorithm and generated key & initial vector.
+  #
+  #     require 'iop/openssl'
+  #     require 'iop/securerandom'
+  #     ( IOP::SecureRandomGenerator.new(1024) | (c = IOP::CipherEncryptor.new) ).process!
+  #     puts c.key
+  #
+  # @since 0.1
+  #
+  class CipherEncryptor
+
+    include Feed
+    include Sink
+
+    # Returns initial vector (IV) for encryption session.
+    attr_reader :iv
+
+    # Returns encryption key.
+    attr_reader :key
+
+    # Creates class instance.
+    #
+    # @param cipher [String, OpenSSL::Cipher] cipher used for encryption
+    #
+    # @param key [String] string representing an encryption key or +nil+
+    #
+    # @param iv [String] string representing an initial vector or +nil+
+    #
+    # _cipher_ can be either a +String+ or +OpenSSL::Cipher+ instance.
+    # If it is a string, a corresponding +OpenSSL::Cipher+ instance will be created.
+    #
+    # If _key_ is +nil+, a new key will be generated in secure manner which can be accessed later with {#key} method.
+    #
+    # If _iv_ is +nil+, a new initial vector will be generated in secure manner which can be accessed later with {#iv} method.
+    # If _iv_ is +nil+ the generated initial vector will be injected into the downstream data preceding the encrypted data itself.
+    #
+    # Note that key and initial vector are both cipher-dependent. Refer to +OpenSSL::Cipher+ documentation for more information.
+    def initialize(cipher = DEFAULT_OPENSSL_CIPHER, key: nil, iv: nil)
+      @cipher = cipher.is_a?(String) ? OpenSSL::Cipher.new(cipher) : cipher
+      @cipher.encrypt
+      @key = key.nil? ? @cipher.random_key : @cipher.key = key
+      @iv = if iv.nil?
+              @embed_iv = true
+              @cipher.random_iv
+            else
+              @cipher.iv = iv
+            end
+    end
+
+    def process(data = nil)
+      unless @continue
+        @continue = true
+        super(iv) if @embed_iv
+        @buffer = IOP.allocate_string(data.size)
+      end
+      if data.nil?
+        super(@cipher.final)
+        super
+      else
+        super(@cipher.update(data, @buffer)) unless data.size.zero?
+      end
+    end
+
+  end
+
+
+  #
+  # Filter class to perform decryption with a symmetric key algorithm (ciphering) of the data passed through.
+  #
+  # The class is an adapter for +OpenSSL::Cipher+ & compatible classes.
+  #
+  # ### Use case: decrypt a file with default algorithm and embedded initial vector.
+  #
+  #     require 'iop/file'
+  #     require 'iop/openssl'
+  #     ( IOP::FileReader.new('input.aes') | IOP::CipherDecryptor.new(key: my_secret_key) | (s = IOP::StringMerger.new) ).process!
+  #     puts s.to_s
+  #
+  # @since 0.1
+  #
+  class CipherDecryptor
+
+    include Feed
+    include Sink
+
+    # Returns initial vector (IV) for decryption session.
+    attr_reader :iv
+
+    # Returns decryption key.
+    attr_reader :key
+
+    # Creates class instance.
+    #
+    # @param cipher [String, OpenSSL::Cipher] cipher used for decryption
+    #
+    # @param key [String] string representing an encryption key
+    #
+    # @param iv [String] string representing an initial vector or +nil+
+    #
+    # _cipher_ can be either a +String+ or +OpenSSL::Cipher+ instance.
+    # If it is a string, a corresponding +OpenSSL::Cipher+ instance will be created.
+    #
+    # If _iv_ is +nil+, the initial vector will be obtained from the upstream data. Refer to {CipherEncryptor#initialize} for details.
+    def initialize(cipher = DEFAULT_OPENSSL_CIPHER, key:, iv: nil)
+      @cipher = cipher.is_a?(String) ? OpenSSL::Cipher.new(cipher) : cipher
+      @cipher.decrypt
+      @cipher.key = @key = key
+      @cipher.iv = @iv = iv unless iv.nil?
+    end
+
+    def process(data = nil)
+      unless @continue
+        @continue = true
+        @buffer = IOP.allocate_string(data.size)
+        if iv.nil?
+          @cipher.iv = @iv = data[0, @cipher.iv_len]
+          data = data[@cipher.iv_len..-1]
+        end
+      end
+      if data.nil?
+        super(@cipher.final)
+        super
+      else
+        super(@cipher.update(data, @buffer)) unless data.size.zero?
+      end
+    end
+
+  end
+
+
+end