pacer-bloomfilter 1.0.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.rspec

@@ -0,0 +1 @@
+--color

data/Gemfile

@@ -0,0 +1,9 @@
+source "http://rubygems.org"
+gem 'pacer', '>= 0.6.1'
+
+group :development do
+  gem "bundler", "~> 1.0.0"
+  gem "jeweler", "~> 1.5.2"
+  gem "rspec", "~> 2.3.0"
+  gem "rcov", ">= 0"
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2011 Darrick Wiebe
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,121 @@
+# Pacer BloomFilter plugin (pacer-bloomfilter)
+
+This plugin adds set filtering using [bloom filters](http://en.wikipedia.org/wiki/Bloom_filter) to the [Pacer](https://github.com/pangloss/pacer) graph and
+streaming data processing library.
+
+This plugin is also meant to serve as an example of how easy it is to
+build a plugin for Pacer.
+
+## Usage
+
+The bloomfilter method is added to the core route object in Pacer,
+which means that it will be available to all routes. The method takes 2
+arguments and a block:
+
+- false_positive_probability: between 0 and 1 with a lower number
+  indicating a lower chance of different keys being considered equal
+- expected_count: the maximum number of elements you think will be added
+  to the bloom filter. The more accurate this number is the more
+  accurate your false_positive_probability will be.
+- block: The block should map the elements that will be iterated over to
+  a value that will be used by the filter. You should return a string.
+  This block does not affect the actual output of the route.
+  If no block is given, to_s on the element itself will used for the
+  filter.
+
+### Example
+
+Map the vertices to names and then filter by name:
+
+    graph.v[:name].bloomfilter(0.001, 10).except(['sam', 'bob']) 
+
+    "steve" "gary"
+    Total: 2
+    => #<GraphV -> Obj(name) -> Obj-Bloom>
+
+Wrong! There is no way to map the vertices to the name, all vertices
+pass through:
+
+    graph.v.bloomfilter(0.001, 10).except(['sam', 'bob']) 
+
+    #<V[0]> #<V[1]> #<V[2]> #<V[3]>
+    Total: 4
+    => #<GraphV -> V-Bloom>
+
+That's better. Here we tell the bloomfilter how to map the vertices to
+the name field (we switched it to #only though just to get more mileage
+out of the example):
+
+    graph.v.bloomfilter(0.001, 10) { |v| v[:name] }.only(['sam', 'bob']) 
+
+    #<V[0]> #<V[3]>
+    Total: 2
+    => #<GraphV -> V-Bloom>
+
+And for completeness, the uniq method is pretty self explanitory I hope:
+
+    graph.v[:type].bloomfilter(0.001, 10).uniq
+
+    "band member"
+    Total: 1
+    => #<GraphV -> V-Bloom>
+
+## Is it Fast?
+
+I don't know. This plugin is currently only a proof of concept and has
+not been optimized, profiled or benchmarked! If you want to spend a
+couple of hours to make it blazing fast, I'll be quite impressed!
+
+## How Pacer Plugins Work
+
+The plugin architecture of Pacer is very simple. You can define a module
+in one of 3 namespaces which correspond to the categories of functions
+that I've identified thus far:
+
+- Pacer::Filter
+- Pacer::Transform
+- Pacer::SideEffect
+
+Rather than try to explain those clearly, it is probably best to
+dig into the code and documenation of Pacer, Pipes and Gremlin.
+
+The module that you define will be mixed in to a Pacer::Route instance
+at runtime when chain_route is called pointing to your module. Nearly
+every method that is called when a user is building a route is actually
+just a friendlier version of chain_route. The chain_route method takes a
+hash of arguments, a few of which are reserved, and the rest of which
+will be applied to your module via property setters. Note that in this
+plugin, :false_pos_prob, :expected_count, :bloomfilter, and :block all
+have corresponding attr_accessors in the BloomFilter module. In that
+way, settings are carried over from the route definition to the route
+instance.
+
+Once the route is defined, it may be executed multiple times. Whenever
+it is executed, a new pipeline is built, of which your module is just
+one part. In almost all cases, all you will need to do is define the
+protected attach_pipe method. Inside that method you just need to create
+your pipe, call setStarts on it with the pipe that was given as a
+parameter, and return the pipe you created. Pacer will look after the
+rest of the pipe building process.
+
+## Contributing to pacer-bloomfilter
+ 
+* Check out the latest master to make sure the feature hasn't been
+  implemented or the bug hasn't been fixed yet
+* Check out the issue tracker to make sure someone already hasn't
+  requested it and/or contributed it
+* Fork the project
+* Start a feature/bugfix branch
+* Commit and push until you are happy with your contribution
+* Make sure to add tests for it. This is important so I don't break it
+  in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you
+  want to have your own version, or is otherwise necessary, that is
+  fine, but please isolate to its own commit so I can cherry-pick around
+  it.
+
+## Copyright
+
+Copyright (c) 2011 Darrick Wiebe. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,50 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "pacer-bloomfilter"
+  gem.homepage = "http://github.com/pangloss/pacer-bloomfilter"
+  gem.license = "MIT"
+  gem.summary = %Q{Filter object streams in Pacer by using a bloom filter}
+  gem.description = %Q{Bloom filters are fast, compact, probabalistic data structures that allow set filtering with a configurable rate of false positives. This plugin adds .bloom_filter.uniq, .bloom_filter.only([collection]), and .bloom_filter.except([collection]) to the available routes methods in Pacer.}
+  gem.email = "darrick@innatesoftware.com"
+  gem.authors = ["Darrick Wiebe"]
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  gem.add_runtime_dependency 'pacer', '>= 0.6.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+RSpec::Core::RakeTask.new(:rcov) do |spec|
+  spec.pattern = 'spec/**/*_spec.rb'
+  spec.rcov = true
+end
+
+task :default => :spec
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "pacer-bloomfilter #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/lib/pacer-bloomfilter.rb

@@ -0,0 +1,21 @@
+module PacerBloomFilter
+  unless const_defined? :VERSION
+    PATH = File.expand_path(File.join(File.dirname(__FILE__), '..'))
+    VERSION = File.read(PATH + '/VERSION').chomp
+
+    $: << File.dirname(__FILE__)
+    require File.dirname(__FILE__) + '/../vendor/java-bloomfilter.jar'
+  end
+
+  def self.reload!
+    Dir[File.dirname(__FILE__) + '/**/*.rb'].each do |file|
+      puts file
+      load file
+    end
+    nil
+  end
+end
+
+require 'pacer/pipe/bloomfilter_reject'
+require 'pacer/filter/bloomfilter'
+

data/lib/pacer/filter/bloomfilter.rb

@@ -0,0 +1,77 @@
+module Pacer
+  module Core
+    module Route
+      def bloomfilter(false_pos_prob, expected_count, opts = {}, &block)
+        chain_route :filter => :bloom,
+          :false_pos_prob => false_pos_prob,
+          :expected_count => expected_count,
+          :bloomfilter => opts[:bloomfilter],
+          :block => block
+      end
+    end
+  end
+
+  module Filter
+    module BloomFilter
+      attr_accessor :false_pos_prob, :expected_count, :block, :bloomfilter
+
+      def uniq
+        @except ||= []
+        @uniq = true
+        self
+      end
+
+      def except(others)
+        @except ||= []
+        @except << others
+        self
+      end
+
+      def only(others)
+        @only ||= []
+        @only << others
+        self
+      end
+
+    protected
+
+      def attach_pipe(pipe)
+        pipe = except_pipe(pipe) if @except
+        pipe = only_pipe(pipe) if @only
+        pipe
+      end
+
+    private
+
+      def except_pipe(pipe)
+        bfp = Pacer::Pipes::BloomFilter::RejectPipe.new false_pos_prob, expected_count, sideline_pipe
+        bfp.accumulate if @uniq
+        prepare_pipe(bfp, @except, pipe)
+      end
+
+      def only_pipe(pipe)
+        bfp = Pacer::Pipes::BloomFilter::SelectPipe.new false_pos_prob, expected_count, sideline_pipe
+        prepare_pipe(bfp, @except, pipe)
+      end
+
+      def prepare_pipe(bfp, all_items, pipe)
+        bfp.bloomfilter = bloomfilter if bloomfilter
+        all_items.each do |items|
+          if items.is_a? Enumerable
+            bfp.addAll items
+          else
+            bfp.addAll [items]
+          end
+        end
+        bfp.setStarts pipe if pipe
+        bfp
+      end
+
+      def sideline_pipe
+        if block
+          Pacer::Route.pipeline Pacer::Route.empty(self).map(&block)
+        end
+      end
+    end
+  end
+end

data/lib/pacer/pipe/bloomfilter_reject.rb

@@ -0,0 +1,103 @@
+module Pacer
+  module Pipes
+    module BloomFilter
+      class SideliningPipe < AbstractPipe
+        def initialize(pipe)
+          super()
+          if pipe
+            @sideline = pipe
+            @sidelineExpando = ExpandableIterator.new(java.util.ArrayList.new.iterator);
+            @sideline.setStarts(@sidelineExpando);
+          end
+        end
+
+      protected
+
+        def sidelineValue(value)
+          if @sideline
+            @sideline.reset
+            @sidelineExpando.add value
+            @sideline.next
+          else
+            value
+          end
+        rescue NativeException => e
+          if e.cause.getClass == Pacer::NoSuchElementException.getClass
+            nil
+          else
+            raise e
+          end
+        end
+      end
+
+      class RejectPipe < SideliningPipe
+        import com.skjegstad.utils.BloomFilter
+        field_accessor :starts
+        attr_accessor :filter
+
+        def initialize(false_pos_prob, expected_count, sideline_pipe = nil)
+          super(sideline_pipe)
+          @filter = BloomFilter.new(false_pos_prob, expected_count)
+        end
+
+        def addAll(elements)
+          @filter.addAll(elements)
+        end
+
+        def accumulate
+          @accumulate = true
+        end
+
+      protected
+
+        def processNextStart()
+          while raw_element = starts.next
+            value = sidelineValue(raw_element)
+            unless @filter.contains? value.to_s
+              @filter.add(value.to_s) if @accumulate and value
+              return raw_element
+            end
+          end
+        rescue NativeException => e
+          if e.cause.getClass == Pacer::NoSuchElementException.getClass
+            raise e.cause
+          else
+            raise e
+          end
+        end
+      end
+
+      class SelectPipe < SideliningPipe
+        import com.skjegstad.utils.BloomFilter
+        field_accessor :starts
+        attr_accessor :filter
+
+        def initialize(false_pos_prob, expected_count, sideline_pipe = nil)
+          super(sideline_pipe)
+          @filter = BloomFilter.new(false_pos_prob, expected_count)
+        end
+
+        def addAll(elements)
+          @filter.addAll(elements)
+        end
+
+      protected
+
+        def processNextStart()
+          while raw_element = starts.next
+            value = sidelineValue(raw_element)
+            if @filter.contains? value.to_s
+              return raw_element
+            end
+          end
+        rescue NativeException => e
+          if e.cause.getClass == Pacer::NoSuchElementException.getClass
+            raise e.cause
+          else
+            raise e
+          end
+        end
+      end
+    end
+  end
+end

data/spec/pacer-bloomfilter_spec.rb

@@ -0,0 +1,7 @@
+require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
+
+describe "PacerBloomfilter" do
+  it "fails" do
+    fail "hey buddy, you should probably rename this file and start specing for real"
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,12 @@
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'rspec'
+require 'pacer-bloomfilter'
+
+# Requires supporting files with custom matchers and macros, etc,
+# in ./support/ and its subdirectories.
+Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each {|f| require f}
+
+RSpec.configure do |config|
+  
+end

metadata

@@ -0,0 +1,138 @@
+--- !ruby/object:Gem::Specification 
+name: pacer-bloomfilter
+version: !ruby/object:Gem::Version 
+  prerelease: 
+  version: 1.0.0
+platform: ruby
+authors: 
+  - Darrick Wiebe
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-03-29 00:00:00 -04:00
+default_executable: 
+dependencies: 
+  - !ruby/object:Gem::Dependency 
+    name: pacer
+    version_requirements: &id001 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ">="
+          - !ruby/object:Gem::Version 
+            version: 0.6.1
+    requirement: *id001
+    prerelease: false
+    type: :runtime
+  - !ruby/object:Gem::Dependency 
+    name: bundler
+    version_requirements: &id002 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ~>
+          - !ruby/object:Gem::Version 
+            version: 1.0.0
+    requirement: *id002
+    prerelease: false
+    type: :development
+  - !ruby/object:Gem::Dependency 
+    name: jeweler
+    version_requirements: &id003 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ~>
+          - !ruby/object:Gem::Version 
+            version: 1.5.2
+    requirement: *id003
+    prerelease: false
+    type: :development
+  - !ruby/object:Gem::Dependency 
+    name: rspec
+    version_requirements: &id004 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ~>
+          - !ruby/object:Gem::Version 
+            version: 2.3.0
+    requirement: *id004
+    prerelease: false
+    type: :development
+  - !ruby/object:Gem::Dependency 
+    name: rcov
+    version_requirements: &id005 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ">="
+          - !ruby/object:Gem::Version 
+            version: "0"
+    requirement: *id005
+    prerelease: false
+    type: :development
+  - !ruby/object:Gem::Dependency 
+    name: pacer
+    version_requirements: &id006 !ruby/object:Gem::Requirement 
+      none: false
+      requirements: 
+        - - ">="
+          - !ruby/object:Gem::Version 
+            version: 0.6.1
+    requirement: *id006
+    prerelease: false
+    type: :runtime
+description: Bloom filters are fast, compact, probabalistic data structures that allow set filtering with a configurable rate of false positives. This plugin adds .bloom_filter.uniq, .bloom_filter.only([collection]), and .bloom_filter.except([collection]) to the available routes methods in Pacer.
+email: darrick@innatesoftware.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+  - LICENSE.txt
+  - README.md
+files: 
+  - .document
+  - .rspec
+  - Gemfile
+  - LICENSE.txt
+  - README.md
+  - Rakefile
+  - VERSION
+  - lib/pacer-bloomfilter.rb
+  - lib/pacer/filter/bloomfilter.rb
+  - lib/pacer/pipe/bloomfilter_reject.rb
+  - spec/pacer-bloomfilter_spec.rb
+  - spec/spec_helper.rb
+  - vendor/java-bloomfilter.jar
+has_rdoc: true
+homepage: http://github.com/pangloss/pacer-bloomfilter
+licenses: 
+  - MIT
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+  - lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 2
+        segments: 
+          - 0
+        version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.5.1
+signing_key: 
+specification_version: 3
+summary: Filter object streams in Pacer by using a bloom filter
+test_files: 
+  - spec/pacer-bloomfilter_spec.rb
+  - spec/spec_helper.rb