enumerating 1.0.0

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*

data/Gemfile

@@ -0,0 +1,6 @@
+source :rubygems
+
+gemspec
+
+gem "rake"
+gem "rspec", ">= 2"

data/README.markdown

@@ -0,0 +1,101 @@
+Enumerating
+===========
+
+Lazy "filtering" and transforming
+---------------------------------
+
+Enumerating extends Enumerable with "lazy" versions of various common operations:
+
+* `#selecting` selects elements that pass a test block (like `Enumerable#select`)
+* `#rejecting` selects elements that fail a test block (like `Enumerable#reject`)
+* `#collecting` applies a transforming block to each element (like `Enumerable#collect`)
+* `#uniqing` discards duplicates (like `Enumerable#uniq`)
+
+We say the "...ing" variants are "lazy", because they defer per-element processing until the result is used.  They return Enumerable "result proxy" objects, rather than Arrays, and only perform the actual filtering (or transformation) as the result proxy is enumerated.
+
+Perhaps an example would help.  Consider the following snippet:
+
+    >> (1..10).collect { |x| puts "#{x}^2 = #{x*x}"; x*x }.take_while { |x| x < 20 }
+    1^2 = 1
+    2^2 = 4
+    3^2 = 9
+    4^2 = 16
+    5^2 = 25
+    6^2 = 36
+    7^2 = 49
+    8^2 = 64
+    9^2 = 81
+    10^2 = 100
+    => [1, 4, 9, 16]
+    
+Here we use plain old `#collect` to square a bunch of numbers, and then grab the ones less than 20. We can do the same thing using `#collecting`, rather than `#collect`:
+
+    >> (1..10).collecting { |x| puts "#{x}^2 = #{x*x}"; x*x }.take_while { |x| x < 20 }
+    1^2 = 1
+    2^2 = 4
+    3^2 = 9
+    4^2 = 16
+    5^2 = 25
+    => [1, 4, 9, 16]
+
+Same result, but notice how only the first five inputs were ever squared; just enough to find the first result above 20.
+
+Lazy pipelines
+--------------
+
+By combining two or more of the lazy operations provided by Enumerating, you can create an efficient "pipeline", e.g.
+
+    # enumerate all users
+    users = User.to_enum(:find_each)
+
+    # where first and last names start with the same letter
+    users = users.selecting { |u| u.first_name[0] == u.last_name[0] }
+
+    # grab their company (weeding out duplicates)
+    companies = users.collecting(&:company).uniqing
+
+    # resolve
+    companies.to_a              #=> ["Disney"]
+
+Because each processing step proceeds in parallel, without creation of intermediate collections (Arrays), you can efficiently operate on large (or even infinite) Enumerable collections.
+
+Lazy combination of Enumerables
+-------------------------------
+
+Enumerating also provides some interesting ways to combine several Enumerable collections to create a new collection.  Again, these operate "lazily".
+
+`Enumerating.zipping` pulls elements from a number of collections in parallel, yielding each group.
+
+    array1 = [1,3,6]
+    array2 = [2,4,7]
+    Enumerating.zipping(array1, array2)    # generates: [1,2], [3,4], [6,7]
+
+`Enumerating.merging` merges multiple collections, preserving sort-order.  The inputs are assumed to be sorted already.
+
+    array1 = [1,4,5]
+    array2 = [2,3,6]
+    Enumerating.merging(array1, array2)    # generates: 1, 2, 3, 4, 5, 6
+
+Variant `Enumerating.merging_by` uses a block to determine sort-order.
+
+    array1 = %w(a dd cccc)
+    array2 = %w(eee bbbbb)
+    Enumerating.merging_by(array1, array2) { |x| x.length }
+                                            # generates: %w(a dd eee cccc bbbbb)
+
+Same but different
+------------------
+
+There are numerous similar implementations of lazy operations on Enumerables.  A nod, in particular, to:
+
+* Greg Spurrier's gem "`lazing`" (from which Enumerating borrows the convention of using "..ing" to name lazy methods)
+* `Enumerable#defer` from the Ruby Facets library
+
+In the end, though, I felt the world deserved another.  Enumerating's selling point is that it's basic (filtering/transforming) operations work on any Ruby, whereas most of the other implementations depend on the availablity of Ruby 1.9's "`Enumerator`".  Enumerating has been tested on:
+
+* MRI 1.8.6
+* MRI 1.8.7
+* MRI 1.9.2
+* JRuby 1.5.3
+* JRuby 1.6.0
+* Rubinius 1.2.3

data/Rakefile

@@ -0,0 +1,13 @@
+require "rake"
+
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new do |t|
+  t.pattern = 'spec/**/*_spec.rb'
+  t.rspec_opts = ["--colour", "--format", "nested"]
+end
+
+task "default" => "spec"

data/benchmarks/pipeline_bench.rb

@@ -0,0 +1,71 @@
+require 'benchmark'
+
+$: << File.expand_path("../../lib", __FILE__)
+
+RUBY19 = RUBY_VERSION =~ /^1\.9/
+
+if RUBY19
+  require "lazing"
+  module Enumerable
+    alias :lazing_select :selecting
+    alias :lazing_collect :collecting
+  end
+end
+
+require "enumerating"
+
+require 'facets'
+
+array = (1..100000).to_a
+
+# Test scenario:
+#  - filter out even numbers
+#  - square them
+#  - grab the first thousand
+
+printf "%-30s", "IMPLEMENTATION"
+printf "%12s", "take(10)"
+printf "%12s", "take(100)"
+printf "%12s", "take(1000)"
+printf "%12s", "to_a"
+puts ""
+
+def measure(&block)
+  begin
+    printf "%12.5f", Benchmark.realtime(&block)
+  rescue
+    printf "%12s", "n/a"
+  end
+end
+
+def benchmark(description, control_result = nil)
+  result = nil
+  printf "%-30s", description
+  measure { yield.take(10).to_a }
+  measure { yield.take(100).to_a }
+  measure { result = yield.take(1000).to_a }
+  measure { yield.to_a }
+  puts ""
+  unless control_result.nil? || result == control_result
+    raise "unexpected result from '#{description}'"
+  end
+  result
+end
+
+@control = benchmark "conventional" do
+  array.select { |x| x.even? }.collect { |x| x*x }
+end
+
+benchmark "enumerating", @control do 
+  array.selecting { |x| x.even? }.collecting { |x| x*x }
+end
+
+if RUBY19
+  benchmark "lazing", @control do
+    array.lazing_select { |x| x.even? }.lazing_collect { |x| x*x }
+  end
+end
+
+benchmark "facets Enumerable#defer", @control do
+  array.defer.select { |x| x.even? }.collect { |x| x*x }
+end

data/enumerating.gemspec

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "enumerating/version"
+
+Gem::Specification.new do |s|
+
+  s.name          = "enumerating"
+  s.version       = Enumerating::VERSION.dup
+  s.platform      = Gem::Platform::RUBY
+  s.authors       = ["Mike Williams"]
+  s.email         = "mdub@dogbiscuit.org"
+  s.homepage      = "http://github.com/mdub/enumerating"
+
+  s.summary       = %{Lazy filtering/transforming of Enumerable collections}
+  s.description   = %{Enumerating extends Enumerable with "lazy" versions of various operations, allowing streamed processing of large (or even infinite) collections.  Even in Ruby 1.8.x.}
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.require_paths = ["lib"]
+
+end

data/lib/enumerating.rb

@@ -0,0 +1,2 @@
+require 'enumerating/filtering'
+require 'enumerating/mixing'

data/lib/enumerating/filtering.rb

@@ -0,0 +1,68 @@
+require 'set'
+
+module Enumerating
+
+  class Filter
+
+    include Enumerable
+
+    def initialize(&generator)
+      @generator = generator
+    end
+
+    def each
+      return to_enum unless block_given?
+      yielder = proc { |x| yield x }
+      @generator.call(yielder)
+    end
+
+  end
+
+end
+
+module Enumerable
+  
+  def collecting
+    Enumerating::Filter.new do |output|
+      each do |element|
+        output.call yield(element)
+      end
+    end
+  end
+  
+  def selecting
+    Enumerating::Filter.new do |output|
+      each do |element|
+        output.call(element) if yield(element)
+      end
+    end
+  end
+  
+  def rejecting
+    Enumerating::Filter.new do |output|
+      each do |element|
+        output.call(element) unless yield(element)
+      end
+    end
+  end
+  
+  def uniqing
+    Enumerating::Filter.new do |output|
+      seen = Set.new
+      each do |element|
+        output.call(element) if seen.add?(element)
+      end
+    end
+  end
+
+  def uniqing_by
+    Enumerating::Filter.new do |output|
+      seen = Set.new
+      each do |element|
+        output.call(element) if seen.add?(yield element)
+      end
+    end
+  end
+  
+end
+

data/lib/enumerating/merging.rb

@@ -0,0 +1,70 @@
+module Enumerating
+
+  class Merger
+
+    include Enumerable
+
+    def initialize(enumerables, &transformer)
+      @enumerables = enumerables
+      @transformer = transformer
+    end
+
+    def each(&block)
+      return to_enum unless block_given?
+      Generator.new(@enumerables.map(&:to_enum), @transformer).each(&block)
+    end
+
+    class Generator
+
+      def initialize(enumerators, transformer)
+        @enumerators = enumerators
+        @transformer = transformer
+      end
+
+      def each
+        while true do
+          discard_empty_enumerators
+          break if @enumerators.empty?
+          yield next_enumerator.next
+        end
+      end
+
+      private
+
+      def discard_empty_enumerators
+        @enumerators.delete_if do |e|
+          begin
+            e.peek
+            false
+          rescue StopIteration
+            true
+          end
+        end
+      end
+
+      def next_enumerator
+        @enumerators.min_by { |enumerator| transform(enumerator.peek) }
+      end
+
+      def transform(item)
+        return item unless @transformer
+        @transformer.call(item)
+      end
+
+    end
+
+  end
+
+end
+
+class << Enumerating
+  
+  def merging(*enumerables)
+    Enumerating::Merger.new(enumerables)
+  end
+
+  def merging_by(*enumerables, &block)
+    Enumerating::Merger.new(enumerables, &block)
+  end
+  
+end

data/lib/enumerating/mixing.rb

@@ -0,0 +1,2 @@
+require 'enumerating/merging'
+require 'enumerating/zipping'

data/lib/enumerating/version.rb

@@ -0,0 +1,3 @@
+module Enumerating
+  VERSION = "1.0.0".freeze
+end

data/lib/enumerating/zipping.rb

@@ -0,0 +1,36 @@
+module Enumerating
+
+  class Zipper
+
+    include Enumerable
+
+    def initialize(enumerables)
+      @enumerables = enumerables
+    end
+
+    def each
+      enumerators = @enumerables.map(&:to_enum)
+      while true
+        chunk = enumerators.map do |enumerator|
+          begin 
+            enumerator.next
+          rescue StopIteration
+            nil
+          end
+        end
+        break if chunk.all?(&:nil?)
+        yield chunk
+      end
+    end
+
+  end
+
+end
+
+class << Enumerating
+
+  def zipping(*enumerables)
+    Enumerating::Zipper.new(enumerables)
+  end
+
+end

data/spec/enumerating/filtering_spec.rb

@@ -0,0 +1,74 @@
+require "spec_helper"
+
+module Enumerable
+  
+  unless method_defined?(:first)
+    def first
+      each do |first_item|
+        return first_item
+      end
+    end
+  end
+  
+end
+
+describe Enumerable do
+
+  describe "#collecting" do
+
+    it "transforms items" do
+      [1,2,3].collecting { |x| x * 2 }.to_a.should == [2,4,6]
+    end
+
+    it "is lazy" do
+      [1,2,3].with_time_bomb.collecting { |x| x * 2 }.first.should == 2
+    end
+
+  end
+
+  describe "#selecting" do
+
+    it "excludes items that don't pass the predicate" do
+      (1..6).selecting { |x| x%2 == 0 }.to_a.should == [2,4,6]
+    end
+
+    it "is lazy" do
+      (1..6).with_time_bomb.selecting { |x| x%2 == 0 }.first == 2
+    end
+
+  end
+
+  describe "#rejecting" do
+
+    it "excludes items that do pass the predicate" do
+      (1..6).rejecting { |x| x%2 == 0 }.to_a.should == [1,3,5]
+    end
+
+    it "is lazy" do
+      (1..6).with_time_bomb.rejecting { |x| x%2 == 0 }.first == 1
+    end
+
+  end
+
+  describe "#uniqing" do
+
+    it "removes duplicates" do
+      [1,3,2,4,3,5,4,6].uniqing.to_a.should == [1,3,2,4,5,6]
+    end
+
+    it "is lazy" do
+      [1,2,3].with_time_bomb.uniqing.first.should == 1
+    end
+
+  end
+
+  describe "#uniqing_by" do
+
+    it "uses the block to derive identity" do
+      @array = %w(A1 A2 B1 A3 C1 B2 C2)
+      @array.uniqing_by { |s| s[0,1] }.to_a.should == %w(A1 B1 C1)
+    end
+
+  end
+
+end

data/spec/enumerating/merging_spec.rb

@@ -0,0 +1,35 @@
+require "spec_helper"
+
+describe Enumerating, :needs_enumerators => true do
+
+  describe ".merging" do
+
+    it "merges multiple Enumerators" do
+      @array1 = [1,3,6]
+      @array2 = [2,4,7]
+      @array3 = [5,8]
+      @merge = Enumerating.merging(@array1, @array2, @array3)
+      @merge.to_a.should == [1,2,3,4,5,6,7,8]
+    end
+
+    it "is lazy" do
+      @enum1 = [1,3]
+      @enum2 = [2,4].with_time_bomb
+      @merge = Enumerating.merging(@enum1, @enum2)
+      @merge.take(4).should == [1,2,3,4]
+    end
+
+  end
+
+  describe ".merging_by" do
+
+    it "uses the block to determine order" do
+      @array1 = %w(cccc dd a)
+      @array2 = %w(eeeee bbb)
+      @merge = Enumerating.merging_by(@array1, @array2) { |s| -s.length }
+      @merge.to_a.should == %w(eeeee cccc bbb dd a)
+    end
+
+  end
+
+end

data/spec/enumerating/zipping_spec.rb

@@ -0,0 +1,22 @@
+require "spec_helper"
+
+describe Enumerating, :needs_enumerators => true do
+
+  describe "#zipping" do
+
+    it "zips together multiple Enumerables" do
+      @array1 = [1,3,6]
+      @array2 = [2,4,7]
+      @array3 = [5,8]
+      @zip = Enumerating.zipping(@array1, @array2, @array3)
+      @zip.to_a.should == [[1,2,5], [3,4,8], [6,7,nil]]
+    end
+
+    it "is lazy" do
+      @zip = Enumerating.zipping(%w(a b c), [1,2].with_time_bomb)
+      @zip.take(2).should == [["a", 1], ["b", 2]]
+    end
+
+  end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,37 @@
+require "rubygems"
+
+require 'rspec'
+
+RSpec.configure do |config|
+  unless defined?(::Enumerator)
+    config.filter_run_excluding :needs_enumerators => true
+  end
+end
+
+class Boom < StandardError; end
+
+class WithTimeBomb
+
+  include Enumerable
+
+  def initialize(source)
+    @source = source
+  end
+
+  def each(&block)
+    @source.each(&block)
+    raise Boom
+  end
+
+end
+
+module Enumerable
+  
+  # extend an Enumerable to throw an exception after last element
+  def with_time_bomb
+    WithTimeBomb.new(self)
+  end
+  
+end
+
+require "enumerating"

metadata

@@ -0,0 +1,72 @@
+--- !ruby/object:Gem::Specification 
+name: enumerating
+version: !ruby/object:Gem::Version 
+  prerelease: 
+  version: 1.0.0
+platform: ruby
+authors: 
+- Mike Williams
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-05-05 00:00:00 Z
+dependencies: []
+
+description: Enumerating extends Enumerable with "lazy" versions of various operations, allowing streamed processing of large (or even infinite) collections.  Even in Ruby 1.8.x.
+email: mdub@dogbiscuit.org
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- .gitignore
+- Gemfile
+- README.markdown
+- Rakefile
+- benchmarks/pipeline_bench.rb
+- enumerating.gemspec
+- lib/enumerating.rb
+- lib/enumerating/filtering.rb
+- lib/enumerating/merging.rb
+- lib/enumerating/mixing.rb
+- lib/enumerating/version.rb
+- lib/enumerating/zipping.rb
+- spec/enumerating/filtering_spec.rb
+- spec/enumerating/merging_spec.rb
+- spec/enumerating/zipping_spec.rb
+- spec/spec_helper.rb
+homepage: http://github.com/mdub/enumerating
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.7.2
+signing_key: 
+specification_version: 3
+summary: Lazy filtering/transforming of Enumerable collections
+test_files: 
+- spec/enumerating/filtering_spec.rb
+- spec/enumerating/merging_spec.rb
+- spec/enumerating/zipping_spec.rb
+- spec/spec_helper.rb