piperator 0.1.0 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: e41e83c8480b4089277077c8c9e067a973113f2a
4
- data.tar.gz: '0780f80b1baf8104a16abd923df4664f909b9756'
2
+ SHA256:
3
+ metadata.gz: 1475b2b74289ce3c32f638a67156b943101bff0b548283e2e1ea635d5048a6e5
4
+ data.tar.gz: 3d06228a633975fde9409387471bb505eb192971c8e50e7ce0df34bbc74e2ae1
5
5
  SHA512:
6
- metadata.gz: 3b433fd2b72437ac5b02a0905c39fb94d1a4b62a084f4907f427c7103100347d2fdd318588d93ef081bcf1faca37174deca31018e0d720f954950750e14e7018
7
- data.tar.gz: 59dcd7ce0c99342b7e02fa14d2705cc05a0abe6c5d8a65511fc632872659e03ca44347589b14e361424b9b50735062d12a4ddc7ebf3cd3eb25e45e7b212c7657
6
+ metadata.gz: 9a11be79d16a0a54c21cb62518ce6426a184c9c9d6609dc6424533413d6cee875f57d74411d92b39dc57fbf9d25d5e712b112149d7bc16d3f42a6f972e52284a
7
+ data.tar.gz: e2cec664b6832086b53cdaa200b03852f6dcd62cb4f3c5b18532ea636054b0c599471d84a181857e36919cea400a68f1e6fdd93bdeb03e9f612107acdcd4768f
@@ -1,5 +1,11 @@
1
1
  sudo: false
2
+ cache: bundler
2
3
  language: ruby
3
4
  rvm:
4
- - 2.4.0
5
- before_install: gem install bundler -v 1.14.6
5
+ - 2.3
6
+ - 2.4
7
+ - 2.5
8
+ - 2.6
9
+ - 2.7
10
+ - jruby
11
+ - truffleruby
@@ -0,0 +1,12 @@
1
+ ## 1.1.0 (6 December 2019)
2
+
3
+ - add `#pos` to Piperator::IO to get current position
4
+
5
+ ## 1.0.0. (13 October 2019)
6
+
7
+ - add `Piperator.build` to build pipelines with DSL
8
+
9
+ ## 0.3.0 (13 July 2017)
10
+
11
+ - remove implicit wrapping to callable from `Pipeline.pipe`
12
+ - add `Pipeline.wrap` to wrap a value as callable
data/README.md CHANGED
@@ -6,6 +6,20 @@ The library is heavily inspired by [Elixir pipe operator](https://elixirschool.c
6
6
 
7
7
  [![Build Status](https://travis-ci.org/lautis/piperator.svg?branch=master)](https://travis-ci.org/lautis/piperator)
8
8
 
9
+ <!-- START doctoc generated TOC please keep comment here to allow auto update -->
10
+ <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
11
+ **Table of Contents**
12
+
13
+ - [Installation](#installation)
14
+ - [Usage](#usage)
15
+ - [Pipelines](#pipelines)
16
+ - [Enumerators as IO objects](#enumerators-as-io-objects)
17
+ - [Development](#development)
18
+ - [Related Projects](#related-projects)
19
+ - [Contributing](#contributing)
20
+ - [License](#license)
21
+
22
+ <!-- END doctoc generated TOC please keep comment here to allow auto update -->
9
23
 
10
24
  ## Installation
11
25
 
@@ -23,7 +37,9 @@ Start by requiring the gem
23
37
  require 'piperator'
24
38
  ```
25
39
 
26
- As an appetiser, here's a pipeline that triples all input values and then sums the values.
40
+ ### Pipelines
41
+
42
+ As an appetizer, here's a pipeline that triples all input values and then sums the values.
27
43
 
28
44
  ```ruby
29
45
  Piperator.
@@ -33,17 +49,38 @@ Piperator.
33
49
  # => 18
34
50
  ```
35
51
 
36
- If desired, the input enumerable can also be given as the first pipe.
52
+ The same could also be achieved using DSL instead of method chaining:
53
+
54
+ ```ruby
55
+ Piperator.build do
56
+ pipe(->(values) { values.lazy.map { |i| i * 3 } })
57
+ pipe(->(values) { values.sum })
58
+ end.call([1, 2, 3])
59
+ ```
60
+
61
+ If desired, the input enumerable can also be given as the first element of the pipeline using `Piperator.wrap`.
37
62
 
38
63
  ```ruby
39
64
  Piperator.
40
- pipe([1, 2, 3]).
65
+ wrap([1, 2, 3]).
41
66
  pipe(->(values) { values.lazy.map { |i| i * 3 } }).
42
67
  pipe(->(values) { values.sum }).
43
68
  call
44
69
  # => 18
45
70
  ```
46
71
 
72
+ Have reasons to defer constructing a pipe? Evaluate it lazily:
73
+
74
+ ```ruby
75
+ summing = ->(values) { values.sum }
76
+ Piperator.build
77
+ pipe(->(values) { values.lazy.map { |i| i * 3 } })
78
+ lazy do
79
+ summing
80
+ end
81
+ end.call([1, 2, 3])
82
+ ```
83
+
47
84
  There is, of course, a much more idiomatic alternative in Ruby:
48
85
 
49
86
  ```ruby
@@ -124,12 +161,42 @@ Piperator.pipe(double).pipe(prepend_append).call([1, 2, 3]).to_a
124
161
  # => ['start', 2, 4, 6, 'end']
125
162
  ```
126
163
 
164
+ ### Enumerators as IO objects
165
+
166
+ Piperator also provides a helper class that allows `Enumerator`s to be used as
167
+ IO objects. This is useful to provide integration with libraries that work only
168
+ with IO objects such as [Nokogiri](http://www.nokogiri.org) or
169
+ [Oj](https://github.com/ohler55/oj).
170
+
171
+ An example pipe that would yield all XML node in a document read in streams:
172
+
173
+ ```ruby
174
+
175
+ require 'nokogiri'
176
+ streaming_xml = lambda do |enumerable|
177
+ Enumerator.new do |yielder|
178
+ io = Piperator::IO.new(enumerable.each)
179
+ reader = Nokogiri::XML::Reader(io)
180
+ reader.each { |node| yielder << node }
181
+ end
182
+ end
183
+ ```
184
+
185
+ In real-world scenarios, the pipe would need to filter the nodes. Passing every
186
+ single XML node forward is not that useful.
187
+
127
188
  ## Development
128
189
 
129
190
  After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
130
191
 
131
192
  To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
132
193
 
194
+ ## Related Projects
195
+
196
+ * [D★Stream](https://github.com/ddfreyne/d-stream) - Set of extensions for writing stream-processing code.
197
+ * [ddbuffer](https://github.com/ddfreyne/ddbuffer) - Buffer enumerables/enumerators.
198
+ * [Down::ChunkedIO](https://github.com/janko-m/down/blob/master/lib/down/chunked_io.rb) - A similar IO class as Piperator::IO.
199
+
133
200
  ## Contributing
134
201
 
135
202
  Bug reports and pull requests are welcome on GitHub at https://github.com/lautis/piperator.
@@ -1,14 +1,54 @@
1
1
  require 'piperator/version'
2
2
  require 'piperator/pipeline'
3
+ require 'piperator/io'
4
+ require 'piperator/builder'
3
5
 
4
6
  # Top-level shortcuts
5
7
  module Piperator
8
+ # Build a new pipeline using DSL. This enables easy control of the pipeline
9
+ # stucture.
10
+ #
11
+ # Piperator.build do
12
+ # wrap [1, 2, 3]
13
+ # pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
14
+ # end
15
+ # # => Pipeline that returns [2, 3, 4] called
16
+ #
17
+ # # Alternatively, the Builder is also given as argument to the block
18
+ # Piperator.build do |p|
19
+ # p.wrap [1, 2, 3]
20
+ # p.pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
21
+ # end
22
+ # # This is similar, but allows access to instance variables.
23
+ #
24
+ # @return [Pipeline] Pipeline containing defined steps
25
+ def self.build(&block)
26
+ return Pipeline.new unless block_given?
27
+
28
+ Builder.new(block&.binding).tap do |builder|
29
+ if block.arity.positive?
30
+ yield builder
31
+ else
32
+ builder.instance_eval(&block)
33
+ end
34
+ end.to_pipeline
35
+ end
36
+
6
37
  # Build a new pipeline from a callable or an enumerable object
7
38
  #
8
39
  # @see Piperator::Pipeline.pipe
9
- # @param callable An object responding to call(enumerable)
40
+ # @param enumerable An object responding to call(enumerable)
10
41
  # @return [Pipeline] A pipeline containing only the callable
11
42
  def self.pipe(enumerable)
12
43
  Pipeline.pipe(enumerable)
13
44
  end
45
+
46
+ # Build a new pipeline from a from a non-callable, i.e. string, array, etc.
47
+ #
48
+ # @see Piperator::Pipeline.wrap
49
+ # @param value A raw value which will be passed through the pipeline
50
+ # @return [Pipeline] A pipeline containing only the callable
51
+ def self.wrap(value)
52
+ Pipeline.wrap(value)
53
+ end
14
54
  end
@@ -0,0 +1,54 @@
1
+ module Piperator
2
+ # Builder is used to provide DSL-based Pipeline building. Using Builder,
3
+ # Pipelines can be built without pipe chaining, which might be easier if
4
+ # some steps need to be included only on specific conditions.
5
+ #
6
+ # @see Piperator.build
7
+ class Builder
8
+ # Expose a chained method in Pipeline in DSL
9
+ #
10
+ # @param method_name Name of method in Pipeline
11
+ # @see Pipeline
12
+ #
13
+ # @!macro [attach] dsl_method
14
+ # @method $1
15
+ # Call Pipeline#$1 given arguments and use the return value as builder state.
16
+ #
17
+ # @see Pipeline.$1
18
+ def self.dsl_method(method_name)
19
+ define_method(method_name) do |*arguments, &block|
20
+ @pipeline = @pipeline.send(method_name, *arguments, &block)
21
+ end
22
+ end
23
+
24
+ dsl_method :lazy
25
+ dsl_method :pipe
26
+ dsl_method :wrap
27
+
28
+ def initialize(saved_binding, pipeline = Pipeline.new)
29
+ @pipeline = pipeline
30
+ @saved_binding = saved_binding
31
+ end
32
+
33
+ # Return build pipeline
34
+ #
35
+ # @return [Pipeline]
36
+ def to_pipeline
37
+ @pipeline
38
+ end
39
+
40
+ private
41
+
42
+ def method_missing(method_name, *arguments, &block)
43
+ if @saved_binding.receiver.respond_to?(method_name, true)
44
+ @saved_binding.receiver.send(method_name, *arguments, &block)
45
+ else
46
+ super
47
+ end
48
+ end
49
+
50
+ def respond_to_missing?(method_name, include_private = false)
51
+ @saved_binding.receiver.respond_to?(method_name, include_private) || super
52
+ end
53
+ end
54
+ end
@@ -0,0 +1,121 @@
1
+ require 'English'
2
+
3
+ module Piperator
4
+ # Pseudo I/O on Enumerators
5
+ class IO
6
+ FLUSH_THRESHOLD = 128 * 1028 # 128KiB
7
+
8
+ attr_reader :eof
9
+ attr_reader :pos
10
+
11
+ def initialize(enumerator, flush_threshold: FLUSH_THRESHOLD)
12
+ @enumerator = enumerator
13
+ @flush_threshold = flush_threshold
14
+ @buffer = StringIO.new
15
+ @pos = 0
16
+ @buffer_read_pos = 0
17
+ @eof = false
18
+ end
19
+
20
+ alias eof? eof
21
+ alias tell pos
22
+
23
+ # Return the first bytes of the buffer without marking the buffer as read.
24
+ def peek(bytes)
25
+ while @eof == false && readable_bytes < (bytes || 1)
26
+ @buffer.write(@enumerator.next)
27
+ end
28
+ peek_buffer(bytes)
29
+ rescue StopIteration
30
+ @eof = true
31
+ peek_buffer(bytes)
32
+ end
33
+
34
+ # Reads the next "line" from the I/O stream; lines are separated by
35
+ # separator.
36
+ #
37
+ # @param separator [String] separator to split input
38
+ # @param _limit Unused parameter for compatiblity
39
+ # @return [String]
40
+ def gets(separator = $INPUT_RECORD_SEPARATOR, _limit = nil)
41
+ while !@eof && !contains_line?(separator)
42
+ begin
43
+ @buffer.write(@enumerator.next)
44
+ rescue StopIteration
45
+ @eof = true
46
+ nil
47
+ end
48
+ end
49
+ read_with { @buffer.gets(separator) }
50
+ end
51
+
52
+ # Flush internal buffer until the last unread byte
53
+ def flush
54
+ if @buffer.pos == @buffer_read_pos
55
+ initialize_buffer
56
+ else
57
+ @buffer.pos = @buffer_read_pos
58
+ initialize_buffer(@buffer.read)
59
+ end
60
+ end
61
+
62
+ # Reads length bytes from the I/O stream.
63
+ #
64
+ # @param length [Integer] number of bytes to read
65
+ # @return String
66
+ def read(length = nil)
67
+ return @enumerator.next.tap { |e| @pos += e.bytesize } if length.nil? && readable_bytes.zero?
68
+ @buffer.write(@enumerator.next) while !@eof && readable_bytes < (length || 1)
69
+ read_with { @buffer.read(length) }
70
+ rescue StopIteration
71
+ @eof = true
72
+ read_with { @buffer.read(length) } if readable_bytes > 0
73
+ end
74
+
75
+ # Current buffer size - including non-freed read content
76
+ #
77
+ # @return [Integer] number of bytes stored in buffer
78
+ def used
79
+ @buffer.size
80
+ end
81
+
82
+ private
83
+
84
+ def readable_bytes
85
+ @buffer.pos - @buffer_read_pos
86
+ end
87
+
88
+ def read_with
89
+ pos = @buffer.pos
90
+ @buffer.pos = @buffer_read_pos
91
+
92
+ yield.tap do |data|
93
+ @buffer_read_pos += data.bytesize if data
94
+ @buffer.pos = pos
95
+ flush if flush?
96
+ end
97
+ end
98
+
99
+ def peek_buffer(bytes)
100
+ @buffer.string.byteslice(@buffer_read_pos...@buffer_read_pos + bytes)
101
+ end
102
+
103
+ def flush?
104
+ @buffer.pos == @buffer_read_pos || @buffer.pos > @flush_threshold
105
+ end
106
+
107
+ def initialize_buffer(data = nil)
108
+ @pos += @buffer_read_pos
109
+ @buffer_read_pos = 0
110
+ @buffer = StringIO.new
111
+ @buffer.write(data) if data
112
+ end
113
+
114
+ def contains_line?(separator = $INPUT_RECORD_SEPARATOR)
115
+ return true if @eof
116
+ @buffer.string.byteslice(@buffer_read_pos..-1).include?(separator)
117
+ rescue ArgumentError # Invalid UTF-8
118
+ false
119
+ end
120
+ end
121
+ end
@@ -6,16 +6,42 @@ module Piperator
6
6
  # For streaming purposes, it usually is desirable to have pipes that takes
7
7
  # a lazy Enumerator as an argument a return a (modified) lazy Enumerator.
8
8
  class Pipeline
9
+ # Build a new pipeline from a lazily evaluated callable or an enumerable
10
+ # object.
11
+ #
12
+ # @param block A block returning a callable(enumerable)
13
+ # @return [Pipeline] A pipeline containing only the lazily evaluated
14
+ # callable.
15
+ def self.lazy(&block)
16
+ callable = nil
17
+ Pipeline.new([lambda do |e|
18
+ callable ||= block.call
19
+ callable.call(e)
20
+ end])
21
+ end
22
+
9
23
  # Build a new pipeline from a callable or an enumerable object
10
24
  #
11
25
  # @param callable An object responding to call(enumerable)
12
26
  # @return [Pipeline] A pipeline containing only the callable
13
27
  def self.pipe(callable)
14
- if callable.respond_to?(:call)
15
- Pipeline.new([callable])
16
- else
17
- Pipeline.new([->(_) { callable }])
18
- end
28
+ Pipeline.new([callable])
29
+ end
30
+
31
+ # Build a new pipeline from a from a non-callable, i.e. string, array, etc.
32
+ # This method will wrap the value in a proc, thus making it callable.
33
+ #
34
+ # Piperator::Pipeline.wrap([1, 2, 3]).pipe(add_one)
35
+ # # => [2, 3, 4]
36
+ #
37
+ # # Wrap is syntactic sugar for wrapping a value in a proc
38
+ # Piperator::Pipeline.pipe(->(_) { [1, 2, 3] }).pipe(add_one)
39
+ # # => [2, 3, 4]
40
+ #
41
+ # @param value A raw value which will be passed through the pipeline
42
+ # @return [Pipeline] A pipeline containing only the callable
43
+ def self.wrap(value)
44
+ Pipeline.new([->(_) { value }])
19
45
  end
20
46
 
21
47
  # Returns enumerable given as an argument without modifications. Usable when
@@ -29,6 +55,7 @@ module Piperator
29
55
 
30
56
  def initialize(pipes = [])
31
57
  @pipes = pipes
58
+ freeze
32
59
  end
33
60
 
34
61
  # Compute the pipeline and return a lazy enumerable with all the pipes.
@@ -46,6 +73,19 @@ module Piperator
46
73
  call(enumerable).to_a
47
74
  end
48
75
 
76
+ # Add a new lazily evaluated part to the pipeline.
77
+ #
78
+ # @param block A block returning a callable(enumerable) to append in
79
+ # pipeline.
80
+ # @return [Pipeline] A new pipeline instance
81
+ def lazy(&block)
82
+ callable = nil
83
+ Pipeline.new(@pipes + [lambda do |e|
84
+ callable ||= block.call
85
+ callable.call(e)
86
+ end])
87
+ end
88
+
49
89
  # Add a new part to the pipeline
50
90
  #
51
91
  # @param other A pipe to append in pipeline. Responds to #call.
@@ -53,5 +93,14 @@ module Piperator
53
93
  def pipe(other)
54
94
  Pipeline.new(@pipes + [other])
55
95
  end
96
+
97
+ # Add a new value to the pipeline
98
+ #
99
+ # @param other A value which is wrapped into a pipe, then appended to the
100
+ # pipeline.
101
+ # @return [Pipeline] A new pipeline instance
102
+ def wrap(other)
103
+ Pipeline.new(@pipes + [->(_) { other }])
104
+ end
56
105
  end
57
106
  end
@@ -1,3 +1,4 @@
1
1
  module Piperator
2
- VERSION = '0.1.0'.freeze
2
+ # Piperator version
3
+ VERSION = '1.2.0'.freeze
3
4
  end
@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
23
23
  spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
24
24
  spec.require_paths = ['lib']
25
25
 
26
- spec.add_development_dependency 'bundler', '~> 1.14'
26
+ spec.add_development_dependency 'bundler', '>= 1.14'
27
27
  spec.add_development_dependency 'rake', '~> 12.0'
28
28
  spec.add_development_dependency 'rspec', '~> 3.0'
29
29
  end
metadata CHANGED
@@ -1,27 +1,27 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: piperator
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 1.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ville Lautanala
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2017-05-23 00:00:00.000000000 Z
11
+ date: 2020-06-02 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - "~>"
17
+ - - ">="
18
18
  - !ruby/object:Gem::Version
19
19
  version: '1.14'
20
20
  type: :development
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - "~>"
24
+ - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: '1.14'
27
27
  - !ruby/object:Gem::Dependency
@@ -63,6 +63,7 @@ files:
63
63
  - ".gitignore"
64
64
  - ".rspec"
65
65
  - ".travis.yml"
66
+ - CHANGELOG.md
66
67
  - Gemfile
67
68
  - LICENSE.txt
68
69
  - README.md
@@ -70,6 +71,8 @@ files:
70
71
  - bin/console
71
72
  - bin/setup
72
73
  - lib/piperator.rb
74
+ - lib/piperator/builder.rb
75
+ - lib/piperator/io.rb
73
76
  - lib/piperator/pipeline.rb
74
77
  - lib/piperator/version.rb
75
78
  - piperator.gemspec
@@ -92,8 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
92
95
  - !ruby/object:Gem::Version
93
96
  version: '0'
94
97
  requirements: []
95
- rubyforge_project:
96
- rubygems_version: 2.6.11
98
+ rubygems_version: 3.1.2
97
99
  signing_key:
98
100
  specification_version: 4
99
101
  summary: Composable pipelines for streaming large collections