piperator 0.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: e41e83c8480b4089277077c8c9e067a973113f2a
-  data.tar.gz: '0780f80b1baf8104a16abd923df4664f909b9756'
+SHA256:
+  metadata.gz: 1475b2b74289ce3c32f638a67156b943101bff0b548283e2e1ea635d5048a6e5
+  data.tar.gz: 3d06228a633975fde9409387471bb505eb192971c8e50e7ce0df34bbc74e2ae1
 SHA512:
-  metadata.gz: 3b433fd2b72437ac5b02a0905c39fb94d1a4b62a084f4907f427c7103100347d2fdd318588d93ef081bcf1faca37174deca31018e0d720f954950750e14e7018
-  data.tar.gz: 59dcd7ce0c99342b7e02fa14d2705cc05a0abe6c5d8a65511fc632872659e03ca44347589b14e361424b9b50735062d12a4ddc7ebf3cd3eb25e45e7b212c7657
+  metadata.gz: 9a11be79d16a0a54c21cb62518ce6426a184c9c9d6609dc6424533413d6cee875f57d74411d92b39dc57fbf9d25d5e712b112149d7bc16d3f42a6f972e52284a
+  data.tar.gz: e2cec664b6832086b53cdaa200b03852f6dcd62cb4f3c5b18532ea636054b0c599471d84a181857e36919cea400a68f1e6fdd93bdeb03e9f612107acdcd4768f

data/.travis.yml

@@ -1,5 +1,11 @@
 sudo: false
+cache: bundler
 language: ruby
 rvm:
-  - 2.4.0
-before_install: gem install bundler -v 1.14.6
+  - 2.3
+  - 2.4
+  - 2.5
+  - 2.6
+  - 2.7
+  - jruby
+  - truffleruby

data/CHANGELOG.md

@@ -0,0 +1,12 @@
+## 1.1.0 (6 December 2019)
+
+- add `#pos` to Piperator::IO to get current position
+
+## 1.0.0. (13 October 2019)
+
+- add `Piperator.build` to build pipelines with DSL
+
+## 0.3.0 (13 July 2017)
+
+- remove implicit wrapping to callable from `Pipeline.pipe`
+- add `Pipeline.wrap` to wrap a value as callable

data/README.md

@@ -6,6 +6,20 @@ The library is heavily inspired by [Elixir pipe operator](https://elixirschool.c
 
 [![Build Status](https://travis-ci.org/lautis/piperator.svg?branch=master)](https://travis-ci.org/lautis/piperator)
 
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**
+
+- [Installation](#installation)
+- [Usage](#usage)
+  - [Pipelines](#pipelines)
+  - [Enumerators as IO objects](#enumerators-as-io-objects)
+- [Development](#development)
+- [Related Projects](#related-projects)
+- [Contributing](#contributing)
+- [License](#license)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
 ## Installation
 
@@ -23,7 +37,9 @@ Start by requiring the gem
 require 'piperator'
 ```
 
-As an appetiser, here's a pipeline that triples all input values and then sums the values.
+### Pipelines
+
+As an appetizer, here's a pipeline that triples all input values and then sums the values.
 
 ```ruby
 Piperator.
@@ -33,17 +49,38 @@ Piperator.
 # => 18
 ```
 
-If desired, the input enumerable can also be given as the first pipe.
+The same could also be achieved using DSL instead of method chaining:
+
+```ruby
+Piperator.build do
+  pipe(->(values) { values.lazy.map { |i| i * 3 } })
+  pipe(->(values) { values.sum })
+end.call([1, 2, 3])
+```
+
+If desired, the input enumerable can also be given as the first element of the pipeline using `Piperator.wrap`.
 
 ```ruby
 Piperator.
-  pipe([1, 2, 3]).
+  wrap([1, 2, 3]).
   pipe(->(values) { values.lazy.map { |i| i * 3 } }).
   pipe(->(values) { values.sum }).
   call
 # => 18
 ```
 
+Have reasons to defer constructing a pipe? Evaluate it lazily:
+
+```ruby
+summing = ->(values) { values.sum }
+Piperator.build
+  pipe(->(values) { values.lazy.map { |i| i * 3 } })
+  lazy do
+    summing
+  end
+end.call([1, 2, 3])
+```
+
 There is, of course, a much more idiomatic alternative in Ruby:
 
 ```ruby
@@ -124,12 +161,42 @@ Piperator.pipe(double).pipe(prepend_append).call([1, 2, 3]).to_a
 # => ['start', 2, 4, 6, 'end']
 ```
 
+### Enumerators as IO objects
+
+Piperator also provides a helper class that allows `Enumerator`s to be used as
+IO objects. This is useful to provide integration with libraries that work only
+with IO objects such as [Nokogiri](http://www.nokogiri.org) or
+[Oj](https://github.com/ohler55/oj).
+
+An example pipe that would yield all XML node in a document read in streams:
+
+```ruby
+
+require 'nokogiri'
+streaming_xml = lambda do |enumerable|
+  Enumerator.new do |yielder|
+    io = Piperator::IO.new(enumerable.each)
+    reader = Nokogiri::XML::Reader(io)
+    reader.each { |node| yielder << node }
+  end
+end
+```
+
+In real-world scenarios, the pipe would need to filter the nodes. Passing every
+single XML node forward is not that useful.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+## Related Projects
+
+* [D★Stream](https://github.com/ddfreyne/d-stream) - Set of extensions for writing stream-processing code.
+* [ddbuffer](https://github.com/ddfreyne/ddbuffer) - Buffer enumerables/enumerators.
+* [Down::ChunkedIO](https://github.com/janko-m/down/blob/master/lib/down/chunked_io.rb) - A similar IO class as Piperator::IO.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/lautis/piperator.

data/lib/piperator.rb

@@ -1,14 +1,54 @@
 require 'piperator/version'
 require 'piperator/pipeline'
+require 'piperator/io'
+require 'piperator/builder'
 
 # Top-level shortcuts
 module Piperator
+  # Build a new pipeline using DSL. This enables easy control of the pipeline
+  # stucture.
+  #
+  #   Piperator.build do
+  #     wrap [1, 2, 3]
+  #     pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
+  #   end
+  #   # => Pipeline that returns [2, 3, 4] called
+  #
+  #   # Alternatively, the Builder is also given as argument to the block
+  #   Piperator.build do |p|
+  #     p.wrap [1, 2, 3]
+  #     p.pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
+  #   end
+  #   # This is similar, but allows access to instance variables.
+  #
+  # @return [Pipeline] Pipeline containing defined steps
+  def self.build(&block)
+    return Pipeline.new unless block_given?
+
+    Builder.new(block&.binding).tap do |builder|
+      if block.arity.positive?
+        yield builder
+      else
+        builder.instance_eval(&block)
+      end
+    end.to_pipeline
+  end
+
   # Build a new pipeline from a callable or an enumerable object
   #
   # @see Piperator::Pipeline.pipe
-  # @param callable An object responding to call(enumerable)
+  # @param enumerable An object responding to call(enumerable)
   # @return [Pipeline] A pipeline containing only the callable
   def self.pipe(enumerable)
     Pipeline.pipe(enumerable)
   end
+
+  # Build a new pipeline from a from a non-callable, i.e. string, array, etc.
+  #
+  # @see Piperator::Pipeline.wrap
+  # @param value A raw value which will be passed through the pipeline
+  # @return [Pipeline] A pipeline containing only the callable
+  def self.wrap(value)
+    Pipeline.wrap(value)
+  end
 end

data/lib/piperator/builder.rb

@@ -0,0 +1,54 @@
+module Piperator
+  # Builder is used to provide DSL-based Pipeline building. Using Builder,
+  # Pipelines can be built without pipe chaining, which might be easier if
+  # some steps need to be included only on specific conditions.
+  #
+  # @see Piperator.build
+  class Builder
+    # Expose a chained method in Pipeline in DSL
+    #
+    # @param method_name Name of method in Pipeline
+    # @see Pipeline
+    #
+    # @!macro [attach] dsl_method
+    #   @method $1
+    #   Call Pipeline#$1 given arguments and use the return value as builder state.
+    #
+    #   @see Pipeline.$1
+    def self.dsl_method(method_name)
+      define_method(method_name) do |*arguments, &block|
+        @pipeline = @pipeline.send(method_name, *arguments, &block)
+      end
+    end
+
+    dsl_method :lazy
+    dsl_method :pipe
+    dsl_method :wrap
+
+    def initialize(saved_binding, pipeline = Pipeline.new)
+      @pipeline = pipeline
+      @saved_binding = saved_binding
+    end
+
+    # Return build pipeline
+    #
+    # @return [Pipeline]
+    def to_pipeline
+      @pipeline
+    end
+
+    private
+
+    def method_missing(method_name, *arguments, &block)
+      if @saved_binding.receiver.respond_to?(method_name, true)
+        @saved_binding.receiver.send(method_name, *arguments, &block)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(method_name, include_private = false)
+      @saved_binding.receiver.respond_to?(method_name, include_private) || super
+    end
+  end
+end

data/lib/piperator/io.rb

@@ -0,0 +1,121 @@
+require 'English'
+
+module Piperator
+  # Pseudo I/O on Enumerators
+  class IO
+    FLUSH_THRESHOLD = 128 * 1028 # 128KiB
+
+    attr_reader :eof
+    attr_reader :pos
+
+    def initialize(enumerator, flush_threshold: FLUSH_THRESHOLD)
+      @enumerator = enumerator
+      @flush_threshold = flush_threshold
+      @buffer = StringIO.new
+      @pos = 0
+      @buffer_read_pos = 0
+      @eof = false
+    end
+
+    alias eof? eof
+    alias tell pos
+
+    # Return the first bytes of the buffer without marking the buffer as read.
+    def peek(bytes)
+      while @eof == false && readable_bytes < (bytes || 1)
+        @buffer.write(@enumerator.next)
+      end
+      peek_buffer(bytes)
+    rescue StopIteration
+      @eof = true
+      peek_buffer(bytes)
+    end
+
+    # Reads the next "line" from the I/O stream; lines are separated by
+    # separator.
+    #
+    # @param separator [String] separator to split input
+    # @param _limit Unused parameter for compatiblity
+    # @return [String]
+    def gets(separator = $INPUT_RECORD_SEPARATOR, _limit = nil)
+      while !@eof && !contains_line?(separator)
+        begin
+          @buffer.write(@enumerator.next)
+        rescue StopIteration
+          @eof = true
+          nil
+        end
+      end
+      read_with { @buffer.gets(separator) }
+    end
+
+    # Flush internal buffer until the last unread byte
+    def flush
+      if @buffer.pos == @buffer_read_pos
+        initialize_buffer
+      else
+        @buffer.pos = @buffer_read_pos
+        initialize_buffer(@buffer.read)
+      end
+    end
+
+    # Reads length bytes from the I/O stream.
+    #
+    # @param length [Integer] number of bytes to read
+    # @return String
+    def read(length = nil)
+      return @enumerator.next.tap { |e| @pos += e.bytesize } if length.nil? && readable_bytes.zero?
+      @buffer.write(@enumerator.next) while !@eof && readable_bytes < (length || 1)
+      read_with { @buffer.read(length) }
+    rescue StopIteration
+      @eof = true
+      read_with { @buffer.read(length) } if readable_bytes > 0
+    end
+
+    # Current buffer size - including non-freed read content
+    #
+    # @return [Integer] number of bytes stored in buffer
+    def used
+      @buffer.size
+    end
+
+    private
+
+    def readable_bytes
+      @buffer.pos - @buffer_read_pos
+    end
+
+    def read_with
+      pos = @buffer.pos
+      @buffer.pos = @buffer_read_pos
+
+      yield.tap do |data|
+        @buffer_read_pos += data.bytesize if data
+        @buffer.pos = pos
+        flush if flush?
+      end
+    end
+
+    def peek_buffer(bytes)
+      @buffer.string.byteslice(@buffer_read_pos...@buffer_read_pos + bytes)
+    end
+
+    def flush?
+      @buffer.pos == @buffer_read_pos || @buffer.pos > @flush_threshold
+    end
+
+    def initialize_buffer(data = nil)
+      @pos += @buffer_read_pos
+      @buffer_read_pos = 0
+      @buffer = StringIO.new
+      @buffer.write(data) if data
+    end
+
+    def contains_line?(separator = $INPUT_RECORD_SEPARATOR)
+      return true if @eof
+      @buffer.string.byteslice(@buffer_read_pos..-1).include?(separator)
+    rescue ArgumentError # Invalid UTF-8
+      false
+    end
+  end
+end

data/lib/piperator/pipeline.rb

@@ -6,16 +6,42 @@ module Piperator
   # For streaming purposes, it usually is desirable to have pipes that takes
   # a lazy Enumerator as an argument a return a (modified) lazy Enumerator.
   class Pipeline
+    # Build a new pipeline from a lazily evaluated callable or an enumerable
+    # object.
+    #
+    # @param block A block returning a callable(enumerable)
+    # @return [Pipeline] A pipeline containing only the lazily evaluated
+    # callable.
+    def self.lazy(&block)
+      callable = nil
+      Pipeline.new([lambda do |e|
+        callable ||= block.call
+        callable.call(e)
+      end])
+    end
+
     # Build a new pipeline from a callable or an enumerable object
     #
     # @param callable An object responding to call(enumerable)
     # @return [Pipeline] A pipeline containing only the callable
     def self.pipe(callable)
-      if callable.respond_to?(:call)
-        Pipeline.new([callable])
-      else
-        Pipeline.new([->(_) { callable }])
-      end
+      Pipeline.new([callable])
+    end
+
+    # Build a new pipeline from a from a non-callable, i.e. string, array, etc.
+    # This method will wrap the value in a proc, thus making it callable.
+    #
+    #   Piperator::Pipeline.wrap([1, 2, 3]).pipe(add_one)
+    #   # => [2, 3, 4]
+    #
+    #   # Wrap is syntactic sugar for wrapping a value in a proc
+    #   Piperator::Pipeline.pipe(->(_) { [1, 2, 3] }).pipe(add_one)
+    #   # => [2, 3, 4]
+    #
+    # @param value A raw value which will be passed through the pipeline
+    # @return [Pipeline] A pipeline containing only the callable
+    def self.wrap(value)
+      Pipeline.new([->(_) { value }])
     end
 
     # Returns enumerable given as an argument without modifications. Usable when
@@ -29,6 +55,7 @@ module Piperator
 
     def initialize(pipes = [])
       @pipes = pipes
+      freeze
     end
 
     # Compute the pipeline and return a lazy enumerable with all the pipes.
@@ -46,6 +73,19 @@ module Piperator
       call(enumerable).to_a
     end
 
+    # Add a new lazily evaluated part to the pipeline.
+    #
+    # @param block A block returning a callable(enumerable) to append in
+    # pipeline.
+    # @return [Pipeline] A new pipeline instance
+    def lazy(&block)
+      callable = nil
+      Pipeline.new(@pipes + [lambda do |e|
+        callable ||= block.call
+        callable.call(e)
+      end])
+    end
+
     # Add a new part to the pipeline
     #
     # @param other A pipe to append in pipeline. Responds to #call.
@@ -53,5 +93,14 @@ module Piperator
     def pipe(other)
       Pipeline.new(@pipes + [other])
     end
+
+    # Add a new value to the pipeline
+    #
+    # @param other A value which is wrapped into a pipe, then appended to the
+    # pipeline.
+    # @return [Pipeline] A new pipeline instance
+    def wrap(other)
+      Pipeline.new(@pipes + [->(_) { other }])
+    end
   end
 end

data/lib/piperator/version.rb

@@ -1,3 +1,4 @@
 module Piperator
-  VERSION = '0.1.0'.freeze
+  # Piperator version
+  VERSION = '1.2.0'.freeze
 end

data/piperator.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
   spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
-  spec.add_development_dependency 'bundler', '~> 1.14'
+  spec.add_development_dependency 'bundler', '>= 1.14'
   spec.add_development_dependency 'rake', '~> 12.0'
   spec.add_development_dependency 'rspec', '~> 3.0'
 end

metadata

@@ -1,27 +1,27 @@
 --- !ruby/object:Gem::Specification
 name: piperator
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Ville Lautanala
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-05-23 00:00:00.000000000 Z
+date: 2020-06-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '1.14'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '1.14'
 - !ruby/object:Gem::Dependency
@@ -63,6 +63,7 @@ files:
 - ".gitignore"
 - ".rspec"
 - ".travis.yml"
+- CHANGELOG.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -70,6 +71,8 @@ files:
 - bin/console
 - bin/setup
 - lib/piperator.rb
+- lib/piperator/builder.rb
+- lib/piperator/io.rb
 - lib/piperator/pipeline.rb
 - lib/piperator/version.rb
 - piperator.gemspec
@@ -92,8 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.11
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Composable pipelines for streaming large collections