adsp 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 7bd22e8463be2343be9f61ce80d54261f32aefed31b3f6b6550f38e41a5c0401
+  data.tar.gz: 690e5d1c232c781f2bd48b099b01f8ee2d60a7b79aadec7255775110acfedf1f
+SHA512:
+  metadata.gz: 03e5c419cebc46df3e64aa5f71905358c0f7dc5d16a417b2239d22c9e93eca734258da71a819037d6c754b4c4a077a9d177e13f9ec8310ee87187d875606f120
+  data.tar.gz: 5759268ce78057265a84f667966e0869973da161035498438ef88e9c41fc06bdc4984ac8fd73f3b58a0010b2971eabc932ff874d152487c5f8ab6d60d1c92e3d

data/AUTHORS

@@ -0,0 +1 @@
+Andrew Aladjev

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2021 AUTHORS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,300 @@
+# Abstract data stream processor
+
+| AppVeyor | Jenkins | Github actions | Codecov | Gem  |
+| :------: | :-----: | :------------: | :-----: | :--: |
+| [![AppVeyor test status](https://ci.appveyor.com/api/projects/status/github/andrew-aladev/adsp?branch=master&svg=true)](https://ci.appveyor.com/project/andrew-aladev/adsp/branch/master) | [![Jenkins test status](http://37.187.122.190:58182/buildStatus/icon?job=adsp)](http://37.187.122.190:58182/job/adsp) | [![Github Actions test status](https://github.com/andrew-aladev/adsp/workflows/test/badge.svg?branch=master)](https://github.com/andrew-aladev/adsp/actions) | [![Codecov](https://codecov.io/gh/andrew-aladev/adsp/branch/master/graph/badge.svg)](https://codecov.io/gh/andrew-aladev/adsp) | [![Gem](https://img.shields.io/gem/v/adsp.svg)](https://rubygems.org/gems/adsp) |
+
+## Installation
+
+Operating systems: GNU/Linux, FreeBSD, OSX.
+
+## Usage
+
+There are simple APIs: `String` and `File`. Also you can use generic streaming API: `Stream::Writer` and `Stream::Reader`.
+
+```ruby
+require "adsp"
+
+data = ADSP::String.compress "sample string"
+puts ADSP::String.decompress(data)
+
+ADSP::File.compress "file.txt", "file.txt.archive"
+ADSP::File.decompress "file.txt.archive", "file.txt"
+
+ADSP::Stream::Writer.open("file.txt.archive") { |writer| writer << "sample string" }
+puts ADSP::Stream::Reader.open("file.txt.archive") { |reader| reader.read }
+
+writer = ADSP::Stream::Writer.new output_socket
+begin
+  bytes_written = writer.write_nonblock "sample string"
+  # handle "bytes_written"
+rescue IO::WaitWritable
+  # handle wait
+ensure
+  writer.close
+end
+
+reader = ADSP::Stream::Reader.new input_socket
+begin
+  puts reader.read_nonblock(512)
+rescue IO::WaitReadable
+  # handle wait
+rescue ::EOFError
+  # handle eof
+ensure
+  reader.close
+end
+```
+
+You can create and read `tar.archive` archives with [minitar](https://github.com/halostatue/minitar).
+
+```ruby
+require "adsp"
+require "minitar"
+
+ADSP::Stream::Writer.open "file.tar.archive" do |writer|
+  Minitar::Writer.open writer do |tar|
+    tar.add_file_simple "file", :data => "sample string"
+  end
+end
+
+ADSP::Stream::Reader.open "file.tar.archive" do |reader|
+  Minitar::Reader.open reader do |tar|
+    tar.each_entry do |entry|
+      puts entry.name
+      puts entry.read
+    end
+  end
+end
+```
+
+All functionality (including streaming) can be used inside multiple threads with [parallel](https://github.com/grosser/parallel).
+This code will provide heavy load for your CPU.
+
+```ruby
+require "adsp"
+require "parallel"
+
+Parallel.each large_datas do |large_data|
+  ADSP::String.compress large_data
+end
+```
+
+## Options
+
+| Option                          | Values         | Default    | Description |
+|---------------------------------|----------------|------------|-------------|
+| `source_buffer_length`          | 0 - inf        | 0 (auto)   | internal buffer length for source data |
+| `destination_buffer_length`     | 0 - inf        | 0 (auto)   | internal buffer length for description data |
+
+There are internal buffers for compressed and decompressed data.
+For example you want to use 1 KB as `source_buffer_length` for compressor - please use 256 B as `destination_buffer_length`.
+You want to use 256 B as `source_buffer_length` for decompressor - please use 1 KB as `destination_buffer_length`.
+
+Possible compressor options:
+```
+:source_buffer_length
+:destination_buffer_length
+```
+
+Possible decompressor options:
+```
+:source_buffer_length
+:destination_buffer_length
+```
+
+Example:
+
+```ruby
+require "adsp"
+
+data = ADSP::String.compress "sample string", :source_buffer_length => 512
+puts ADSP::String.decompress(data, :source_buffer_length => 512)
+```
+
+## String
+
+String maintains destination buffer only, so it accepts `destination_buffer_length` option only.
+
+```
+::compress(source, options = {})
+::decompress(source, options = {})
+```
+
+`source` is a source string.
+
+## File
+
+File maintains both source and destination buffers, it accepts both `source_buffer_length` and `destination_buffer_length` options.
+
+```
+::compress(source, destination, options = {})
+::decompress(source, destination, options = {})
+```
+
+`source` and `destination` are file pathes.
+
+## Stream::Writer
+
+Its behaviour is similar to builtin [`Zlib::GzipWriter`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipWriter.html).
+
+Writer maintains destination buffer only, so it accepts `destination_buffer_length` option only.
+
+```
+::open(file_path, options = {}, :external_encoding => nil, :transcode_options => {}, &block)
+```
+
+Open file path and create stream writer associated with opened file.
+Data will be transcoded to `:external_encoding` using `:transcode_options` before compressing.
+
+It may be tricky to use both `:pledged_size` and `:transcode_options`. You have to provide size of transcoded input.
+
+```
+::new(destination_io, options = {}, :external_encoding => nil, :transcode_options => {})
+```
+
+Create stream writer associated with destination io.
+Data will be transcoded to `:external_encoding` using `:transcode_options` before compressing.
+
+It may be tricky to use both `:pledged_size` and `:transcode_options`. You have to provide size of transcoded input.
+
+```
+#set_encoding(external_encoding, nil, transcode_options)
+```
+
+Set another encodings, `nil` is just for compatibility with `IO`.
+
+```
+#io
+#to_io
+#stat
+#external_encoding
+#transcode_options
+#pos
+#tell
+```
+
+See [`IO`](https://ruby-doc.org/core/IO.html) docs.
+
+```
+#write(*objects)
+#flush
+#rewind
+#close
+#closed?
+```
+
+See [`Zlib::GzipWriter`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipWriter.html) docs.
+
+```
+#write_nonblock(object, *options)
+#flush_nonblock(*options)
+#rewind_nonblock(*options)
+#close_nonblock(*options)
+```
+
+Special asynchronous methods missing in `Zlib::GzipWriter`.
+`rewind` wants to `close`, `close` wants to `write` something and `flush`, `flush` want to `write` something.
+So it is possible to have asynchronous variants for these synchronous methods.
+Behaviour is the same as `IO#write_nonblock` method.
+
+All nonblock operations for file will raise `EBADF` error on Windows.
+Setting file into nonblocking mode is [not available on Windows](https://github.com/ruby/ruby/blob/master/win32/win32.c#L4388).
+
+```
+#<<(object)
+#print(*objects)
+#printf(*args)
+#putc(object, encoding: ::Encoding::BINARY)
+#puts(*objects)
+```
+
+Typical helpers, see [`Zlib::GzipWriter`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipWriter.html) docs.
+
+## Stream::Reader
+
+Its behaviour is similar to builtin [`Zlib::GzipReader`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipReader.html).
+
+Reader maintains both source and destination buffers, it accepts both `source_buffer_length` and `destination_buffer_length` options.
+
+```
+::open(file_path, options = {}, :external_encoding => nil, :internal_encoding => nil, :transcode_options => {}, &block)
+```
+
+Open file path and create stream reader associated with opened file.
+Data will be force encoded to `:external_encoding` and transcoded to `:internal_encoding` using `:transcode_options` after decompressing.
+
+```
+::new(source_io, options = {}, :external_encoding => nil, :internal_encoding => nil, :transcode_options => {})
+```
+
+Create stream reader associated with source io.
+Data will be force encoded to `:external_encoding` and transcoded to `:internal_encoding` using `:transcode_options` after decompressing.
+
+```
+#set_encoding(external_encoding, internal_encoding, transcode_options)
+```
+
+Set another encodings.
+
+```
+#io
+#to_io
+#stat
+#external_encoding
+#internal_encoding
+#transcode_options
+#pos
+#tell
+```
+
+See [`IO`](https://ruby-doc.org/core/IO.html) docs.
+
+```
+#read(bytes_to_read = nil, out_buffer = nil)
+#eof?
+#rewind
+#close
+#closed?
+```
+
+See [`Zlib::GzipReader`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipReader.html) docs.
+
+```
+#readpartial(bytes_to_read = nil, out_buffer = nil)
+#read_nonblock(bytes_to_read, out_buffer = nil, *options)
+```
+
+See [`IO`](https://ruby-doc.org/core/IO.html) docs.
+
+```
+#getbyte
+#each_byte(&block)
+#readbyte
+#ungetbyte(byte)
+
+#getc
+#readchar
+#each_char(&block)
+#ungetc(char)
+
+#lineno
+#lineno=
+#gets(separator = $OUTPUT_RECORD_SEPARATOR, limit = nil)
+#readline
+#readlines
+#each(&block)
+#each_line(&block)
+#ungetline(line)
+```
+
+Typical helpers, see [`Zlib::GzipReader`](https://ruby-doc.org/stdlib/libdoc/zlib/rdoc/Zlib/GzipReader.html) docs.
+
+## CI
+
+Please visit [scripts/test-images](scripts/test-images).
+See universal test script [scripts/ci_test.sh](scripts/ci_test.sh) for CI.
+
+## License
+
+MIT license, see [LICENSE](LICENSE) and [AUTHORS](AUTHORS).

data/lib/adsp/error.rb

@@ -0,0 +1,14 @@
+# Abstract data stream processor.
+# Copyright (c) 2021 AUTHORS, MIT License.
+
+module ADSP
+  class BaseError < ::StandardError; end
+
+  class ValidateError < BaseError; end
+
+  class NotEnoughDestinationError < BaseError; end
+  class UsedAfterCloseError       < BaseError; end
+
+  class NotImplementedError < BaseError; end
+  class UnexpectedError     < BaseError; end
+end

data/lib/adsp/file.rb

@@ -0,0 +1,76 @@
+# Abstract data stream processor.
+# Copyright (c) 2021 AUTHORS, MIT License.
+
+require_relative "error"
+require_relative "option"
+require_relative "validation"
+
+module ADSP
+  # ADSP::File class.
+  class File
+    # Current option class.
+    Option = ADSP::Option
+
+    # Current buffer length names.
+    # It is a part of compressor and decompressor options.
+    BUFFER_LENGTH_NAMES = %i[source_buffer_length destination_buffer_length].freeze
+
+    # Compresses data from +source+ file path to +destination+ file path.
+    # Option: +:source_buffer_length+ source buffer length.
+    # Option: +:destination_buffer_length+ destination buffer length.
+    def self.compress(source, destination, options = {})
+      Validation.validate_string source
+      Validation.validate_string destination
+
+      options = self::Option.get_compressor_options options, BUFFER_LENGTH_NAMES
+
+      open_files source, destination do |source_io, destination_io|
+        native_compress_io source_io, destination_io, options
+      end
+
+      nil
+    end
+
+    # :nocov:
+
+    # Internal method for compressing data from +source_io+ file to +destination_io+ file.
+    def self.native_compress_io(source_io, destination_io, options)
+      raise NotImplementedError
+    end
+
+    # :nocov:
+
+    # Decompresses data from +source+ file path to +destination+ file path.
+    # Option: +:source_buffer_length+ source buffer length.
+    # Option: +:destination_buffer_length+ destination buffer length.
+    def self.decompress(source, destination, options = {})
+      Validation.validate_string source
+      Validation.validate_string destination
+
+      options = self::Option.get_decompressor_options options, BUFFER_LENGTH_NAMES
+
+      open_files source, destination do |source_io, destination_io|
+        native_decompress_io source_io, destination_io, options
+      end
+
+      nil
+    end
+
+    # :nocov:
+
+    # Internal method for decompressing data from +source_io+ file to +destination_io+ file.
+    def self.native_decompress_io(source_io, destination_io, options)
+      raise NotImplementedError
+    end
+
+    # :nocov:
+
+    private_class_method def self.open_files(source, destination, &_block)
+      ::File.open source, "rb" do |source_io|
+        ::File.open destination, "wb" do |destination_io|
+          yield source_io, destination_io
+        end
+      end
+    end
+  end
+end

data/lib/adsp/option.rb

@@ -0,0 +1,51 @@
+# Abstract data stream processor.
+# Copyright (c) 2021 AUTHORS, MIT License.
+
+require_relative "validation"
+
+module ADSP
+  # ADSP::Option class.
+  class Option
+    # Current default buffer length.
+    # It will be used when buffer length option is not defined.
+    DEFAULT_BUFFER_LENGTH = 0
+
+    # Validates and processes default values for compressor +options+.
+    # +buffer_length_names+ is an array of buffer length names (option names).
+    def self.get_compressor_options(options, buffer_length_names = [])
+      Validation.validate_hash options
+      Validation.validate_array buffer_length_names
+
+      buffer_length_names.each { |name| Validation.validate_symbol name }
+
+      buffer_length_defaults = buffer_length_names.each_with_object({}) do |name, defaults|
+        defaults[name] = DEFAULT_BUFFER_LENGTH
+      end
+
+      options = buffer_length_defaults.merge options
+
+      buffer_length_names.each { |name| Validation.validate_not_negative_integer options[name] }
+
+      options
+    end
+
+    # Validates and processes default values for decompressor +options+.
+    # +buffer_length_names+ is an array of buffer length names (option names).
+    def self.get_decompressor_options(options, buffer_length_names = [])
+      Validation.validate_hash options
+      Validation.validate_array buffer_length_names
+
+      buffer_length_names.each { |name| Validation.validate_symbol name }
+
+      buffer_length_defaults = buffer_length_names.each_with_object({}) do |name, defaults|
+        defaults[name] = DEFAULT_BUFFER_LENGTH
+      end
+
+      options = buffer_length_defaults.merge options
+
+      buffer_length_names.each { |name| Validation.validate_not_negative_integer options[name] }
+
+      options
+    end
+  end
+end

data/lib/adsp/stream/abstract.rb

@@ -0,0 +1,206 @@
+# Abstract data stream processor.
+# Copyright (c) 2021 AUTHORS, MIT License.
+
+require_relative "delegates"
+require_relative "stat"
+require_relative "../error"
+require_relative "../validation"
+
+module ADSP
+  module Stream
+    # ADSP::Stream::Abstract class.
+    class Abstract
+      # Native stream is typically not seekable.
+      # We don't need to implement methods like "seek" and "pos=".
+
+      # Typically we may not maintain correspondance between bytes
+      #   consumed from source and bytes written to destination.
+      # So we are going to consume all source bytes and
+      #   maintain buffer with remaining destination data.
+
+      include Delegates
+
+      # Native stream.
+      attr_reader :io
+
+      # Native stream status info.
+      attr_reader :stat
+
+      # Encoding name for destination data.
+      attr_reader :external_encoding
+
+      # Encoding name for source data.
+      attr_reader :internal_encoding
+
+      # Transcode options for native stream.
+      attr_reader :transcode_options
+
+      # Current offset for source data.
+      attr_reader :pos
+      alias tell pos
+
+      # Initializes stream using +io+ native stream and +options+.
+      # Option: +:external_encoding+ encoding name for destination data.
+      # Option: +:internal_encoding+ encoding name for source data.
+      # Option: +:transcode_options+ transcode options for data.
+      def initialize(io, options = {})
+        @raw_stream = create_raw_stream
+        @io         = io
+
+        @stat = Stat.new @io.stat if @io.respond_to? :stat
+
+        set_encoding options[:external_encoding], options[:internal_encoding], options[:transcode_options]
+        reset_buffer
+        reset_io_advise
+
+        @pos = 0
+      end
+
+      # :nocov:
+
+      # Creates raw stream.
+      protected def create_raw_stream
+        raise NotImplementedError
+      end
+
+      # :nocov:
+
+      # -- buffer --
+
+      # Resets internal source buffer.
+      protected def reset_buffer
+        @buffer = ::String.new :encoding => ::Encoding::BINARY
+      end
+
+      # -- advise --
+
+      # Resets native stream advise.
+      protected def reset_io_advise
+        # Both compressor and decompressor need sequential io access.
+        @io.advise :sequential if @io.respond_to? :advise
+      rescue ::Errno::ESPIPE
+        # ok
+      end
+
+      # Sets access mode for native stream, noop.
+      def advise
+        # Noop
+        nil
+      end
+
+      # -- encoding --
+
+      # Sets encoding for source and destination data.
+      # First argument: +:external_encoding+ encoding name for destination data.
+      # Second argument: +:internal_encoding+ encoding name for source data.
+      # Third argument: +:transcode_options+ transcode options for data.
+      def set_encoding(*args)
+        external_encoding, internal_encoding, transcode_options = process_set_encoding_arguments(*args)
+
+        set_target_encoding :@external_encoding, external_encoding
+        set_target_encoding :@internal_encoding, internal_encoding
+        @transcode_options = transcode_options
+
+        self
+      end
+
+      # Processes encoding for source and destination data.
+      # First argument: +:external_encoding+ encoding name for destination data.
+      # Second argument: +:internal_encoding+ encoding name for source data.
+      # Third argument: +:transcode_options+ transcode options for data.
+      protected def process_set_encoding_arguments(*args)
+        external_encoding = args[0]
+
+        unless external_encoding.nil? || external_encoding.is_a?(::Encoding)
+          Validation.validate_string external_encoding
+
+          # First argument can be "external_encoding:internal_encoding".
+          match = %r{(.+?):(.+)}.match external_encoding
+
+          unless match.nil?
+            external_encoding = match[0]
+            internal_encoding = match[1]
+
+            transcode_options = args[1]
+            Validation.validate_hash transcode_options unless transcode_options.nil?
+
+            return [external_encoding, internal_encoding, transcode_options]
+          end
+        end
+
+        internal_encoding = args[1]
+        unless internal_encoding.nil? || internal_encoding.is_a?(::Encoding)
+          Validation.validate_string internal_encoding
+        end
+
+        transcode_options = args[2]
+        Validation.validate_hash transcode_options unless transcode_options.nil?
+
+        [external_encoding, internal_encoding, transcode_options]
+      end
+
+      # Sets +value+ for encoding +name+.
+      protected def set_target_encoding(name, value)
+        unless value.nil? || value.is_a?(::Encoding)
+          begin
+            value = ::Encoding.find value
+          rescue ::ArgumentError
+            raise ValidateError, "invalid #{name} encoding"
+          end
+        end
+
+        instance_variable_set name, value
+      end
+
+      # Returns encoding for source data if defined.
+      # Returns encoding for destination data if encoding for source data is not defined.
+      # Returns binary encoding if encodings for source and destination dara are not defined.
+      protected def target_encoding
+        return @internal_encoding unless @internal_encoding.nil?
+        return @external_encoding unless @external_encoding.nil?
+
+        ::Encoding::BINARY
+      end
+
+      # -- etc --
+
+      # Resets stream and source position.
+      # Returns zero (offset for source data).
+      def rewind
+        @raw_stream = create_raw_stream
+
+        @io.rewind if @io.respond_to? :rewind
+
+        reset_buffer
+        reset_io_advise
+
+        @pos = 0
+
+        0
+      end
+
+      # Closes stream.
+      def close
+        @io.close if @io.respond_to? :close
+
+        nil
+      end
+
+      # Returns whether stream is closed.
+      def closed?
+        return false unless @raw_stream.closed?
+
+        if @io.respond_to? :closed
+          @io.closed?
+        else
+          true
+        end
+      end
+
+      # Returns self object.
+      def to_io
+        self
+      end
+    end
+  end
+end