piperator 0.3.0 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.travis.yml +8 -2
- data/CHANGELOG.md +8 -0
- data/README.md +42 -1
- data/lib/piperator/builder.rb +54 -0
- data/lib/piperator/io.rb +28 -26
- data/lib/piperator/pipeline.rb +27 -0
- data/lib/piperator/version.rb +2 -1
- data/lib/piperator.rb +31 -1
- data/piperator.gemspec +1 -1
- metadata +6 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 1475b2b74289ce3c32f638a67156b943101bff0b548283e2e1ea635d5048a6e5
|
4
|
+
data.tar.gz: 3d06228a633975fde9409387471bb505eb192971c8e50e7ce0df34bbc74e2ae1
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 9a11be79d16a0a54c21cb62518ce6426a184c9c9d6609dc6424533413d6cee875f57d74411d92b39dc57fbf9d25d5e712b112149d7bc16d3f42a6f972e52284a
|
7
|
+
data.tar.gz: e2cec664b6832086b53cdaa200b03852f6dcd62cb4f3c5b18532ea636054b0c599471d84a181857e36919cea400a68f1e6fdd93bdeb03e9f612107acdcd4768f
|
data/.travis.yml
CHANGED
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,11 @@
|
|
1
|
+
## 1.1.0 (6 December 2019)
|
2
|
+
|
3
|
+
- add `#pos` to Piperator::IO to get current position
|
4
|
+
|
5
|
+
## 1.0.0. (13 October 2019)
|
6
|
+
|
7
|
+
- add `Piperator.build` to build pipelines with DSL
|
8
|
+
|
1
9
|
## 0.3.0 (13 July 2017)
|
2
10
|
|
3
11
|
- remove implicit wrapping to callable from `Pipeline.pipe`
|
data/README.md
CHANGED
@@ -6,6 +6,20 @@ The library is heavily inspired by [Elixir pipe operator](https://elixirschool.c
|
|
6
6
|
|
7
7
|
[![Build Status](https://travis-ci.org/lautis/piperator.svg?branch=master)](https://travis-ci.org/lautis/piperator)
|
8
8
|
|
9
|
+
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
|
10
|
+
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
|
11
|
+
**Table of Contents**
|
12
|
+
|
13
|
+
- [Installation](#installation)
|
14
|
+
- [Usage](#usage)
|
15
|
+
- [Pipelines](#pipelines)
|
16
|
+
- [Enumerators as IO objects](#enumerators-as-io-objects)
|
17
|
+
- [Development](#development)
|
18
|
+
- [Related Projects](#related-projects)
|
19
|
+
- [Contributing](#contributing)
|
20
|
+
- [License](#license)
|
21
|
+
|
22
|
+
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
|
9
23
|
|
10
24
|
## Installation
|
11
25
|
|
@@ -35,7 +49,16 @@ Piperator.
|
|
35
49
|
# => 18
|
36
50
|
```
|
37
51
|
|
38
|
-
|
52
|
+
The same could also be achieved using DSL instead of method chaining:
|
53
|
+
|
54
|
+
```ruby
|
55
|
+
Piperator.build do
|
56
|
+
pipe(->(values) { values.lazy.map { |i| i * 3 } })
|
57
|
+
pipe(->(values) { values.sum })
|
58
|
+
end.call([1, 2, 3])
|
59
|
+
```
|
60
|
+
|
61
|
+
If desired, the input enumerable can also be given as the first element of the pipeline using `Piperator.wrap`.
|
39
62
|
|
40
63
|
```ruby
|
41
64
|
Piperator.
|
@@ -46,6 +69,18 @@ Piperator.
|
|
46
69
|
# => 18
|
47
70
|
```
|
48
71
|
|
72
|
+
Have reasons to defer constructing a pipe? Evaluate it lazily:
|
73
|
+
|
74
|
+
```ruby
|
75
|
+
summing = ->(values) { values.sum }
|
76
|
+
Piperator.build
|
77
|
+
pipe(->(values) { values.lazy.map { |i| i * 3 } })
|
78
|
+
lazy do
|
79
|
+
summing
|
80
|
+
end
|
81
|
+
end.call([1, 2, 3])
|
82
|
+
```
|
83
|
+
|
49
84
|
There is, of course, a much more idiomatic alternative in Ruby:
|
50
85
|
|
51
86
|
```ruby
|
@@ -156,6 +191,12 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
|
|
156
191
|
|
157
192
|
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
158
193
|
|
194
|
+
## Related Projects
|
195
|
+
|
196
|
+
* [D★Stream](https://github.com/ddfreyne/d-stream) - Set of extensions for writing stream-processing code.
|
197
|
+
* [ddbuffer](https://github.com/ddfreyne/ddbuffer) - Buffer enumerables/enumerators.
|
198
|
+
* [Down::ChunkedIO](https://github.com/janko-m/down/blob/master/lib/down/chunked_io.rb) - A similar IO class as Piperator::IO.
|
199
|
+
|
159
200
|
## Contributing
|
160
201
|
|
161
202
|
Bug reports and pull requests are welcome on GitHub at https://github.com/lautis/piperator.
|
@@ -0,0 +1,54 @@
|
|
1
|
+
module Piperator
|
2
|
+
# Builder is used to provide DSL-based Pipeline building. Using Builder,
|
3
|
+
# Pipelines can be built without pipe chaining, which might be easier if
|
4
|
+
# some steps need to be included only on specific conditions.
|
5
|
+
#
|
6
|
+
# @see Piperator.build
|
7
|
+
class Builder
|
8
|
+
# Expose a chained method in Pipeline in DSL
|
9
|
+
#
|
10
|
+
# @param method_name Name of method in Pipeline
|
11
|
+
# @see Pipeline
|
12
|
+
#
|
13
|
+
# @!macro [attach] dsl_method
|
14
|
+
# @method $1
|
15
|
+
# Call Pipeline#$1 given arguments and use the return value as builder state.
|
16
|
+
#
|
17
|
+
# @see Pipeline.$1
|
18
|
+
def self.dsl_method(method_name)
|
19
|
+
define_method(method_name) do |*arguments, &block|
|
20
|
+
@pipeline = @pipeline.send(method_name, *arguments, &block)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
dsl_method :lazy
|
25
|
+
dsl_method :pipe
|
26
|
+
dsl_method :wrap
|
27
|
+
|
28
|
+
def initialize(saved_binding, pipeline = Pipeline.new)
|
29
|
+
@pipeline = pipeline
|
30
|
+
@saved_binding = saved_binding
|
31
|
+
end
|
32
|
+
|
33
|
+
# Return build pipeline
|
34
|
+
#
|
35
|
+
# @return [Pipeline]
|
36
|
+
def to_pipeline
|
37
|
+
@pipeline
|
38
|
+
end
|
39
|
+
|
40
|
+
private
|
41
|
+
|
42
|
+
def method_missing(method_name, *arguments, &block)
|
43
|
+
if @saved_binding.receiver.respond_to?(method_name, true)
|
44
|
+
@saved_binding.receiver.send(method_name, *arguments, &block)
|
45
|
+
else
|
46
|
+
super
|
47
|
+
end
|
48
|
+
end
|
49
|
+
|
50
|
+
def respond_to_missing?(method_name, include_private = false)
|
51
|
+
@saved_binding.receiver.respond_to?(method_name, include_private) || super
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
data/lib/piperator/io.rb
CHANGED
@@ -6,22 +6,24 @@ module Piperator
|
|
6
6
|
FLUSH_THRESHOLD = 128 * 1028 # 128KiB
|
7
7
|
|
8
8
|
attr_reader :eof
|
9
|
+
attr_reader :pos
|
9
10
|
|
10
11
|
def initialize(enumerator, flush_threshold: FLUSH_THRESHOLD)
|
11
12
|
@enumerator = enumerator
|
12
13
|
@flush_threshold = flush_threshold
|
13
|
-
@
|
14
|
-
@
|
15
|
-
@
|
14
|
+
@buffer = StringIO.new
|
15
|
+
@pos = 0
|
16
|
+
@buffer_read_pos = 0
|
16
17
|
@eof = false
|
17
18
|
end
|
18
19
|
|
19
20
|
alias eof? eof
|
21
|
+
alias tell pos
|
20
22
|
|
21
23
|
# Return the first bytes of the buffer without marking the buffer as read.
|
22
24
|
def peek(bytes)
|
23
25
|
while @eof == false && readable_bytes < (bytes || 1)
|
24
|
-
@
|
26
|
+
@buffer.write(@enumerator.next)
|
25
27
|
end
|
26
28
|
peek_buffer(bytes)
|
27
29
|
rescue StopIteration
|
@@ -38,22 +40,22 @@ module Piperator
|
|
38
40
|
def gets(separator = $INPUT_RECORD_SEPARATOR, _limit = nil)
|
39
41
|
while !@eof && !contains_line?(separator)
|
40
42
|
begin
|
41
|
-
@
|
43
|
+
@buffer.write(@enumerator.next)
|
42
44
|
rescue StopIteration
|
43
45
|
@eof = true
|
44
46
|
nil
|
45
47
|
end
|
46
48
|
end
|
47
|
-
read_with { @
|
49
|
+
read_with { @buffer.gets(separator) }
|
48
50
|
end
|
49
51
|
|
50
52
|
# Flush internal buffer until the last unread byte
|
51
53
|
def flush
|
52
|
-
if @
|
54
|
+
if @buffer.pos == @buffer_read_pos
|
53
55
|
initialize_buffer
|
54
56
|
else
|
55
|
-
@
|
56
|
-
initialize_buffer(@
|
57
|
+
@buffer.pos = @buffer_read_pos
|
58
|
+
initialize_buffer(@buffer.read)
|
57
59
|
end
|
58
60
|
end
|
59
61
|
|
@@ -62,56 +64,56 @@ module Piperator
|
|
62
64
|
# @param length [Integer] number of bytes to read
|
63
65
|
# @return String
|
64
66
|
def read(length = nil)
|
65
|
-
return @enumerator.next if length.nil? && readable_bytes.zero?
|
66
|
-
@
|
67
|
-
read_with { @
|
67
|
+
return @enumerator.next.tap { |e| @pos += e.bytesize } if length.nil? && readable_bytes.zero?
|
68
|
+
@buffer.write(@enumerator.next) while !@eof && readable_bytes < (length || 1)
|
69
|
+
read_with { @buffer.read(length) }
|
68
70
|
rescue StopIteration
|
69
71
|
@eof = true
|
70
|
-
read_with { @
|
72
|
+
read_with { @buffer.read(length) } if readable_bytes > 0
|
71
73
|
end
|
72
74
|
|
73
75
|
# Current buffer size - including non-freed read content
|
74
76
|
#
|
75
77
|
# @return [Integer] number of bytes stored in buffer
|
76
78
|
def used
|
77
|
-
@
|
79
|
+
@buffer.size
|
78
80
|
end
|
79
81
|
|
80
82
|
private
|
81
83
|
|
82
84
|
def readable_bytes
|
83
|
-
@
|
85
|
+
@buffer.pos - @buffer_read_pos
|
84
86
|
end
|
85
87
|
|
86
88
|
def read_with
|
87
|
-
pos = @
|
88
|
-
@
|
89
|
+
pos = @buffer.pos
|
90
|
+
@buffer.pos = @buffer_read_pos
|
89
91
|
|
90
92
|
yield.tap do |data|
|
91
|
-
@
|
92
|
-
@
|
93
|
+
@buffer_read_pos += data.bytesize if data
|
94
|
+
@buffer.pos = pos
|
93
95
|
flush if flush?
|
94
96
|
end
|
95
97
|
end
|
96
98
|
|
97
99
|
def peek_buffer(bytes)
|
98
|
-
@
|
100
|
+
@buffer.string.byteslice(@buffer_read_pos...@buffer_read_pos + bytes)
|
99
101
|
end
|
100
102
|
|
101
103
|
def flush?
|
102
|
-
@
|
104
|
+
@buffer.pos == @buffer_read_pos || @buffer.pos > @flush_threshold
|
103
105
|
end
|
104
106
|
|
105
107
|
def initialize_buffer(data = nil)
|
106
|
-
@
|
107
|
-
@
|
108
|
-
@
|
109
|
-
@
|
108
|
+
@pos += @buffer_read_pos
|
109
|
+
@buffer_read_pos = 0
|
110
|
+
@buffer = StringIO.new
|
111
|
+
@buffer.write(data) if data
|
110
112
|
end
|
111
113
|
|
112
114
|
def contains_line?(separator = $INPUT_RECORD_SEPARATOR)
|
113
115
|
return true if @eof
|
114
|
-
@
|
116
|
+
@buffer.string.byteslice(@buffer_read_pos..-1).include?(separator)
|
115
117
|
rescue ArgumentError # Invalid UTF-8
|
116
118
|
false
|
117
119
|
end
|
data/lib/piperator/pipeline.rb
CHANGED
@@ -6,6 +6,20 @@ module Piperator
|
|
6
6
|
# For streaming purposes, it usually is desirable to have pipes that takes
|
7
7
|
# a lazy Enumerator as an argument a return a (modified) lazy Enumerator.
|
8
8
|
class Pipeline
|
9
|
+
# Build a new pipeline from a lazily evaluated callable or an enumerable
|
10
|
+
# object.
|
11
|
+
#
|
12
|
+
# @param block A block returning a callable(enumerable)
|
13
|
+
# @return [Pipeline] A pipeline containing only the lazily evaluated
|
14
|
+
# callable.
|
15
|
+
def self.lazy(&block)
|
16
|
+
callable = nil
|
17
|
+
Pipeline.new([lambda do |e|
|
18
|
+
callable ||= block.call
|
19
|
+
callable.call(e)
|
20
|
+
end])
|
21
|
+
end
|
22
|
+
|
9
23
|
# Build a new pipeline from a callable or an enumerable object
|
10
24
|
#
|
11
25
|
# @param callable An object responding to call(enumerable)
|
@@ -59,6 +73,19 @@ module Piperator
|
|
59
73
|
call(enumerable).to_a
|
60
74
|
end
|
61
75
|
|
76
|
+
# Add a new lazily evaluated part to the pipeline.
|
77
|
+
#
|
78
|
+
# @param block A block returning a callable(enumerable) to append in
|
79
|
+
# pipeline.
|
80
|
+
# @return [Pipeline] A new pipeline instance
|
81
|
+
def lazy(&block)
|
82
|
+
callable = nil
|
83
|
+
Pipeline.new(@pipes + [lambda do |e|
|
84
|
+
callable ||= block.call
|
85
|
+
callable.call(e)
|
86
|
+
end])
|
87
|
+
end
|
88
|
+
|
62
89
|
# Add a new part to the pipeline
|
63
90
|
#
|
64
91
|
# @param other A pipe to append in pipeline. Responds to #call.
|
data/lib/piperator/version.rb
CHANGED
data/lib/piperator.rb
CHANGED
@@ -1,13 +1,43 @@
|
|
1
1
|
require 'piperator/version'
|
2
2
|
require 'piperator/pipeline'
|
3
3
|
require 'piperator/io'
|
4
|
+
require 'piperator/builder'
|
4
5
|
|
5
6
|
# Top-level shortcuts
|
6
7
|
module Piperator
|
8
|
+
# Build a new pipeline using DSL. This enables easy control of the pipeline
|
9
|
+
# stucture.
|
10
|
+
#
|
11
|
+
# Piperator.build do
|
12
|
+
# wrap [1, 2, 3]
|
13
|
+
# pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
|
14
|
+
# end
|
15
|
+
# # => Pipeline that returns [2, 3, 4] called
|
16
|
+
#
|
17
|
+
# # Alternatively, the Builder is also given as argument to the block
|
18
|
+
# Piperator.build do |p|
|
19
|
+
# p.wrap [1, 2, 3]
|
20
|
+
# p.pipe(-> (enumerable) { enumerable.map { |i| i + 1 } })
|
21
|
+
# end
|
22
|
+
# # This is similar, but allows access to instance variables.
|
23
|
+
#
|
24
|
+
# @return [Pipeline] Pipeline containing defined steps
|
25
|
+
def self.build(&block)
|
26
|
+
return Pipeline.new unless block_given?
|
27
|
+
|
28
|
+
Builder.new(block&.binding).tap do |builder|
|
29
|
+
if block.arity.positive?
|
30
|
+
yield builder
|
31
|
+
else
|
32
|
+
builder.instance_eval(&block)
|
33
|
+
end
|
34
|
+
end.to_pipeline
|
35
|
+
end
|
36
|
+
|
7
37
|
# Build a new pipeline from a callable or an enumerable object
|
8
38
|
#
|
9
39
|
# @see Piperator::Pipeline.pipe
|
10
|
-
# @param
|
40
|
+
# @param enumerable An object responding to call(enumerable)
|
11
41
|
# @return [Pipeline] A pipeline containing only the callable
|
12
42
|
def self.pipe(enumerable)
|
13
43
|
Pipeline.pipe(enumerable)
|
data/piperator.gemspec
CHANGED
@@ -23,7 +23,7 @@ Gem::Specification.new do |spec|
|
|
23
23
|
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
|
24
24
|
spec.require_paths = ['lib']
|
25
25
|
|
26
|
-
spec.add_development_dependency 'bundler', '
|
26
|
+
spec.add_development_dependency 'bundler', '>= 1.14'
|
27
27
|
spec.add_development_dependency 'rake', '~> 12.0'
|
28
28
|
spec.add_development_dependency 'rspec', '~> 3.0'
|
29
29
|
end
|
metadata
CHANGED
@@ -1,27 +1,27 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: piperator
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 1.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ville Lautanala
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-06-02 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "
|
17
|
+
- - ">="
|
18
18
|
- !ruby/object:Gem::Version
|
19
19
|
version: '1.14'
|
20
20
|
type: :development
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - "
|
24
|
+
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: '1.14'
|
27
27
|
- !ruby/object:Gem::Dependency
|
@@ -71,6 +71,7 @@ files:
|
|
71
71
|
- bin/console
|
72
72
|
- bin/setup
|
73
73
|
- lib/piperator.rb
|
74
|
+
- lib/piperator/builder.rb
|
74
75
|
- lib/piperator/io.rb
|
75
76
|
- lib/piperator/pipeline.rb
|
76
77
|
- lib/piperator/version.rb
|
@@ -94,8 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
94
95
|
- !ruby/object:Gem::Version
|
95
96
|
version: '0'
|
96
97
|
requirements: []
|
97
|
-
|
98
|
-
rubygems_version: 2.6.11
|
98
|
+
rubygems_version: 3.1.2
|
99
99
|
signing_key:
|
100
100
|
specification_version: 4
|
101
101
|
summary: Composable pipelines for streaming large collections
|