bloombroom 1.0.0

data/CHANGELOG.md

@@ -0,0 +1 @@
+tbd

data/LICENSE.md

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,242 @@
+# Bloombroom v1.0.0
+
+- Standard **Bloomfilter** class for bounded key space
+- **ContinuousBloomfilter** class for unbounded keys (**stream**)
+- Bitfield class
+- BitBucketField class (multi bits)
+- native, C & FFI extensions FNV hash classes
+
+The Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is a member of a set. False positives are possible, but false negatives are not. See [wikipedia](http://en.wikipedia.org/wiki/Bloom_filter).
+
+Bloom filters are normally used in the context of a bounded set since the filter size must be known in advance. for a given filter capacity, its total bit size will affect the false positive error rate. The total number of bits required for a given filter can be computed from the required filter capacity and target error rate. See the [references](#references) section for more info.
+
+### ContinuousBloomfilter
+The **ContinuousBloomfilter** provides a bloom filter implementation which support unbounded stream of elements. Elements are expired after a chosen TTL. At initialization the filter capacity must be estimated for the numbers of elements expected over the given TTL period. 
+
+For example to do dedupping on a stream with a rate of **5000 items/sec** over a period of **60 minutes** would require a filter capacity of **18M** elements. For a required error rate of **0.1%** the filter would need **246mb** of memory (which include all Ruby objects overhead).
+
+The ContinuousBloomfilter uses 4 bits for each filter *position* or *bucket* (instead of 1 bit in a normal bloom filter) for keeping track of the keys TTL. 
+The internal timer resolution is set to half of the required TTL (resolution divisor of 2). using 4 bits gives us
+15 usable time slots (slot 0 is for the unset state). Basically the internal time bookeeping is similar to a
+ring buffer using the current timer tick modulo 15. The timer ticks will be time slot=1, 2, ... 15, 1, 2 and so on. The total 
+time of our internal clock will thus be 15 * (TTL / 2). We keep track of TTL by writing the current time slot 
+in the key k buckets when inserted in the filter. For a key lookup if the interval betweem the current time slot and any of the k buckets value 
+is greater than 2 (resolution divisor) we know this key is expired. See [continuous_bloom_filter.rb](https://github.com/colinsurprenant/bloombroom/blob/master/lib/bloombroom/filter/continuous_bloom_filter.rb)
+
+This means that an element is garanteed to not be expired before the given TTL but in the worst case could survive until 3 * (TTL / 2). 
+
+### Hashing
+Bloom filters require the use of multiple (k) hash functions for each inserted element. We actually simulate multiple hash functions by having just two hash functions which are actually the upper and lower 32 bits of our FFI FNV1a 64 bits hash function. Double hashing with one hash function. Very very fast. See [bloom_helper.rb](https://github.com/colinsurprenant/bloombroom/blob/master/lib/bloombroom/filter/bloom_helper.rb) and the [references](#references) section for more info on this technique.
+
+
+## Installation
+tested in both MRI Ruby 1.9.2, 1.9.3 and JRuby 1.6.7 in 1.9 mode.
+
+``` sh
+$ gem install bloombroom
+```
+
+## Examples
+
+### Standard Bloom filter
+``` ruby
+require 'bloombroom'
+
+bf = Bloombroom::BloomFilter.new(1000, 3) # 1000 bits and 3 hash functions
+
+bf.add("key1")
+bf.add("key2")
+
+bf.include?("key1") # => true
+bf.include?("key3") # => false
+```
+
+``` ruby
+require 'bloombroom'
+
+# compute optimal m,k for a filter capacity of 1000 elements and 0.1% error rate
+m, k = Bloombroom::BloomHelper.find_m_k(1000, 0.001)
+
+bf = Bloombroom::BloomFilter.new(m, k)
+
+bf << "key1"
+bf << "key2"
+
+bf["key1"] # => true
+bf["key3"] # => false
+```
+### Continuous Bloom filter
+
+``` ruby
+require 'bloombroom'
+
+# 1000 buckets, 3 hash functions and a TTL of 2 seconds
+bf = Bloombroom::ContinuousBloomFilter.new(1000, 3, 2)
+bf.start_timer
+
+bf << "key1"
+bf << "key2"
+
+bf["key1"] # => true
+bf["key2"] # => true
+bf["key3"] # => false
+
+sleep(3)
+
+bf["key1"] # => false
+bf["key2"] # => false
+bf["key3"] # => false
+```
+
+## Memory footprint
+The calculated memory footprints **includes all Ruby objects overhead**. In fact the footprint is calculated by querying the process size (rss) before and after the bloom filter object initialization.
+
+### Bloomfilter
+``` sh
+ruby benchmark/bloom_filter_memory.rb auto 1000000 0.01
+ruby benchmark/bloom_filter_memory.rb auto 100000000 0.01
+ruby benchmark/bloom_filter_memory.rb auto 100000000 0.001
+```
+
+- **1.0%** error rate for **1M** keys: **2.3mb**
+- **1.0%** error rate for **100M** keys: **228mb**
+- **0.1%** error rate for **100M** keys: **342mb**
+
+### ContinuousBloomfilter
+``` sh
+ruby benchmark/continuous_bloom_filter_memory.rb auto 1000000 0.01
+ruby benchmark/continuous_bloom_filter_memory.rb auto 100000000 0.01
+ruby benchmark/continuous_bloom_filter_memory.rb auto 100000000 0.001
+```
+
+- **1.0%** error rate for **1M** keys: **9.1mb**
+- **1.0%** error rate for **100M** keys: **914mb**
+- **0.1%** error rate for **100M** keys: **1371mb**
+
+
+## Benchmarks
+All benchmarks have been run on a MacbookPro with a 2.66GHz i7 with 8GB RAM on OSX 10.6.8 with MRI Ruby 1.9.3p194
+
+### Hashing
+The Hashing benchmark compares the performance of SHA1, MD5, two native Ruby FNV (A & B) implementations, a C implementation as a C extension and FFI extension for 32 and 64 bits hashes. 
+
+``` sh
+ruby benchmark/fnv.rb
+```
+
+```
+benchmarking for 1000000 iterations
+                         user     system      total        real
+MD5:                 1.900000   0.010000   1.910000 (  1.912995)
+SHA-1:               2.110000   0.000000   2.110000 (  2.109739)
+native FNV A 32:    32.470000   0.110000  32.580000 ( 32.596759)
+native FNV A 64:    38.330000   0.570000  38.900000 ( 38.923384)
+native FNV B 32:     4.870000   0.020000   4.890000 (  4.882862)
+native FNV B 64:    37.700000   0.110000  37.810000 ( 37.842873)
+ffi FNV 32:          0.760000   0.010000   0.770000 (  0.754941)
+ffi FNV 64:          0.890000   0.000000   0.890000 (  0.901954)
+c-ext FNV 32:        0.310000   0.000000   0.310000 (  0.307131)
+c-ext FNV 64:        0.480000   0.000000   0.480000 (  0.485310)
+
+MD5:                   522740 ops/s
+SHA-1:                 473992 ops/s
+native FNV A 32:        30678 ops/s
+native FNV A 64:        25691 ops/s
+native FNV B 32:       204798 ops/s
+native FNV B 64:        26425 ops/s
+ffi FNV 32:           1324607 ops/s
+ffi FNV 64:           1108704 ops/s
+c-ext FNV 32:         3255939 ops/s
+c-ext FNV 64:         2060538 ops/s
+```
+
+### Bloomfilter
+The Bloomfilter class is using the FFI FNV hashing by default, for speed and compatibility.
+
+``` sh
+ruby benchmark/bloom_filter.rb 
+```
+
+```
+benchmarking for 150000 keys with 1.0%, 0.1%, 0.01% error rates
+                                               user     system      total        real
+BloomFilter m=1437759, k=07 add            0.940000   0.000000   0.940000 (  0.948075)
+BloomFilter m=1437759, k=07 include?       0.830000   0.010000   0.840000 (  0.834414)
+BloomFilter m=2156639, k=10 add            1.220000   0.000000   1.220000 (  1.227294)
+BloomFilter m=2156639, k=10 include?       1.050000   0.010000   1.060000 (  1.052358)
+BloomFilter m=2875518, k=13 add            1.500000   0.010000   1.510000 (  1.516086)
+BloomFilter m=2875518, k=13 include?       1.260000   0.010000   1.270000 (  1.258877)
+
+BloomFilter m=1437759, k=07 add            158215 ops/s
+BloomFilter m=1437759, k=07 include?       179767 ops/s
+BloomFilter m=2156639, k=10 add            122220 ops/s
+BloomFilter m=2156639, k=10 include?       142537 ops/s
+BloomFilter m=2875518, k=13 add             98939 ops/s
+BloomFilter m=2875518, k=13 include?       119154 ops/s
+```
+
+### ContinuousBloomfilter
+The ContinuousBloomfilter class is using the FFI FNV hashing by default, for speed and compatibility.
+
+``` sh
+ruby benchmark/continuous_bloom_filter.rb 
+```
+
+```
+benchmarking WITHOUT expiration for 150000 keys with 1.0%, 0.1%, 0.01% error rates
+                                                            user     system      total        real
+ContinuousBloomFilter m=1437759, k=07 add               1.720000   0.000000   1.720000 (  1.733903)
+ContinuousBloomFilter m=1437759, k=07 include?          1.630000   0.010000   1.640000 (  1.630668)
+ContinuousBloomFilter m=2156639, k=10 add               2.130000   0.010000   2.140000 (  2.142091)
+ContinuousBloomFilter m=2156639, k=10 include?          2.160000   0.000000   2.160000 (  2.159395)
+ContinuousBloomFilter m=2875518, k=13 add               2.650000   0.010000   2.660000 (  2.655585)
+ContinuousBloomFilter m=2875518, k=13 include?          2.570000   0.010000   2.580000 (  2.586032)
+
+ContinuousBloomFilter m=1437759, k=07 add               86510 ops/s
+ContinuousBloomFilter m=1437759, k=07 include?          91987 ops/s
+ContinuousBloomFilter m=2156639, k=10 add               70025 ops/s
+ContinuousBloomFilter m=2156639, k=10 include?          69464 ops/s
+ContinuousBloomFilter m=2875518, k=13 add               56485 ops/s
+ContinuousBloomFilter m=2875518, k=13 include?          58004 ops/s
+
+benchmarking WITH expiration for 500000 keys with 1.0%, 0.1%, 0.01% error rates
+                                                            user     system      total        real
+ContinuousBloomFilter m=1437759, k=07 add+include      11.110000   0.040000  11.150000 ( 11.146869)
+ContinuousBloomFilter m=2156639, k=10 add+include      14.220000   0.040000  14.260000 ( 14.269583)
+ContinuousBloomFilter m=2875518, k=13 add+include      17.600000   0.060000  17.660000 ( 17.665917)
+
+ContinuousBloomFilter m=1437759, k=07 add+include      89711 ops/s
+ContinuousBloomFilter m=2156639, k=10 add+include      70079 ops/s
+ContinuousBloomFilter m=2875518, k=13 add+include      56606 ops/s
+```
+
+## JRuby
+- to run specs use
+
+``` sh
+jruby --1.9 -S rake spec
+```
+- to run benchmarks use 
+
+``` sh
+jruby --1.9 benchmark/some_benchmark.rb
+```
+
+<a id="reference" />
+## References ##
+- [Bloom filter on wikipedia](http://en.wikipedia.org/wiki/Bloom_filter)
+- [Scalable Datasets: Bloom Filters in Ruby](http://www.igvita.com/2008/12/27/scalable-datasets-bloom-filters-in-ruby/)
+- [Flow Analysis & Time-based Bloom Filters](http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/)
+- [Stable Bloom filters](http://webdocs.cs.ualberta.ca/~drafiei/papers/DupDet06Sigmod.pdf)
+- [The maths to compute optimal m and k ](http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/BloomFilter.rdoc)
+- [Producing n hash functions by hashing only once](http://willwhim.wordpress.com/2011/09/03/producing-n-hash-functions-by-hashing-only-once/)
+- [Less Hashing, Same Performance: Building a Better Bloom Filter](http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.152.579&rep=rep1&type=pdf)
+
+## Author
+Colin Surprenant, [@colinsurprenant][twitter], [http://github.com/colinsurprenant][github], colin.surprenant@needium.com, colin.surprenant@gmail.com
+
+## License
+Bloombroom is distributed under the Apache License, Version 2.0. 
+
+[twitter]: http://twitter.com/colinsurprenant
+[github]: http://github.com/colinsurprenant

data/ext/bloombroom/hash/cext/cext_fnv.c

@@ -0,0 +1,91 @@
+/*
+ * based on https://github.com/robey/rbfnv with various fixes from forks
+ */
+
+#include <stdint.h>
+#include "ruby.h"
+
+#define PRIME32 16777619
+#define PRIME64 1099511628211ULL
+
+/**
+ * FNV fast hashing algorithm in 32 bits.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint32_t fnv1_32(const char *data, uint64_t len) {
+  uint32_t rv = 0x811c9dc5U;
+  uint64_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv * PRIME32) ^ (unsigned char)(data[i]);
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 32 bits, variant with operations reversed.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint32_t fnv1a_32(const char *data, uint64_t len) {
+  uint32_t rv = 0x811c9dc5U;
+  uint64_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv ^ (unsigned char)data[i]) * PRIME32;
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 64 bits.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint64_t fnv1_64(const char *data, uint64_t len) {
+  uint64_t rv = 0xcbf29ce484222325ULL;
+  uint64_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv * PRIME64) ^ (unsigned char)data[i];
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 64 bits, variant with operations reversed.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint64_t fnv1a_64(const char *data, uint64_t len) {
+  uint64_t rv = 0xcbf29ce484222325ULL;
+  uint64_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv ^ (unsigned char)data[i]) * PRIME64;
+  }
+  return rv;
+}
+
+/* ----- ruby bindings ----- */
+
+VALUE rb_fnv1_32(VALUE self, VALUE data) {
+  return UINT2NUM(fnv1_32(RSTRING_PTR(data), RSTRING_LEN(data)));
+}
+
+VALUE rb_fnv1a_32(VALUE self, VALUE data) {
+  return UINT2NUM(fnv1a_32(RSTRING_PTR(data), RSTRING_LEN(data)));
+}
+
+VALUE rb_fnv1_64(VALUE self, VALUE data) {
+  return ULL2NUM(fnv1_64(RSTRING_PTR(data), RSTRING_LEN(data)));
+}
+
+VALUE rb_fnv1a_64(VALUE self, VALUE data) {
+  return ULL2NUM(fnv1a_64(RSTRING_PTR(data), RSTRING_LEN(data)));
+}
+
+VALUE rb_class;
+VALUE rb_module;
+
+void Init_cext_fnv() {
+  rb_module = rb_define_module("Bloombroom");
+  rb_class = rb_define_class_under(rb_module, "FNVEXT", rb_cObject);
+  rb_define_singleton_method(rb_class, "fnv1_32", rb_fnv1_32, 1);
+  rb_define_singleton_method(rb_class, "fnv1a_32", rb_fnv1a_32, 1);
+  rb_define_singleton_method(rb_class, "fnv1_64", rb_fnv1_64, 1);
+  rb_define_singleton_method(rb_class, "fnv1a_64", rb_fnv1a_64, 1);
+}

data/ext/bloombroom/hash/cext/extconf.rb

@@ -0,0 +1,3 @@
+require 'mkmf'
+
+create_makefile 'bloombroom/hash/cext_fnv'

data/ext/bloombroom/hash/ffi/extconf.rb

@@ -0,0 +1,3 @@
+require 'mkmf'
+
+create_makefile 'bloombroom/hash/ffi_fnv'

data/ext/bloombroom/hash/ffi/ffi_fnv.c

@@ -0,0 +1,60 @@
+/*
+ * based on https://github.com/robey/rbfnv with various fixes from forks
+ */
+
+#include <stdint.h>
+
+#define PRIME32 16777619
+#define PRIME64 1099511628211ULL
+
+/**
+ * FNV fast hashing algorithm in 32 bits.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint32_t fnv1_32(const char *data, uint32_t len) {
+  uint32_t rv = 0x811c9dc5U;
+  uint32_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv * PRIME32) ^ (unsigned char)(data[i]);
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 32 bits, variant with operations reversed.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint32_t fnv1a_32(const char *data, uint32_t len) {
+  uint32_t rv = 0x811c9dc5U;
+  uint32_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv ^ (unsigned char)data[i]) * PRIME32;
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 64 bits.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint64_t fnv1_64(const char *data, uint32_t len) {
+  uint64_t rv = 0xcbf29ce484222325ULL;
+  uint32_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv * PRIME64) ^ (unsigned char)data[i];
+  }
+  return rv;
+}
+
+/**
+ * FNV fast hashing algorithm in 64 bits, variant with operations reversed.
+ * @see http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash
+ */
+uint64_t fnv1a_64(const char *data, uint32_t len) {
+  uint64_t rv = 0xcbf29ce484222325ULL;
+  uint32_t i;
+  for (i = 0; i < len; i++) {
+    rv = (rv ^ (unsigned char)data[i]) * PRIME64;
+  }
+  return rv;
+}

data/lib/bloombroom.rb

@@ -0,0 +1,13 @@
+require "bloombroom/version"
+require "bloombroom/bits/bit_field"
+require "bloombroom/bits/bit_bucket_field"
+require "bloombroom/filter/bloom_helper"
+require "bloombroom/filter/bloom_filter"
+require "bloombroom/filter/continuous_bloom_filter"
+require "bloombroom/hash/fnv_a"
+require "bloombroom/hash/fnv_b"
+require "bloombroom/hash/cext_fnv"
+require "bloombroom/hash/ffi_fnv"
+
+module Bloombroom
+end

data/lib/bloombroom/bits/bit_bucket_field.rb

@@ -0,0 +1,90 @@
+# create a bit bucket field of 100 buckets of 4 bits
+#   bf = BitBucketField.new(4, 100)
+#
+#   bf[10] = 5 or bf.set(10, 5)
+#   bf[10] => 5 or bf.get(10) => 5
+#   bf[10] = 0
+#   bf.zero?(10) => true
+#
+#   bf.to_s = "10101000101010101"
+#   bf.to_s(2) = "10101000101010101"
+#   bf.to_s(10) = "5 23 7"
+
+module Bloombroom
+  class BitBucketField
+    attr_reader :size
+    include Enumerable
+    
+    ELEMENT_WIDTH = 32
+    
+    # new BitBucketField
+    # @param bits [Fixnum] number of bits per bucket
+    # @param size [Fixnum] number of buckets in field
+    def initialize(bits, size)
+      @size = size
+      @bits = bits
+      @buckets_per_element = ELEMENT_WIDTH / bits
+      @field = Array.new(((size - 1) / @buckets_per_element) + 1, 0)
+      @bucket_mask = (2 ** @bits) - 1
+    end
+    
+    # set a bucket
+    # @param position [Fixnum] bucket position
+    # @param value [Fixnum] bucket value
+    def []=(position, value)
+      element, offset = position.divmod(@buckets_per_element)
+      shift_bits = offset * @bits
+      if value == 0
+        @field[element] &= ~(@bucket_mask << shift_bits)
+      else
+        @field[element] = (@field[element] & ~(@bucket_mask << shift_bits)) | value << shift_bits
+      end
+    end
+    alias_method :set, :[]=
+
+    # read a bucket
+    # @param position [Fixnum] bucket position
+    # @return [Fixnum] bucket value
+    def [](position)
+      element, offset = position.divmod(@buckets_per_element)
+      shift_bits = (position % @buckets_per_element) * @bits
+      (@field[element] & (@bucket_mask << shift_bits)) >> shift_bits
+    end
+    alias_method :get, :[]
+
+    def zero?(position)
+      element, offset = position.divmod(@buckets_per_element)
+      shift_bits = (position % @buckets_per_element) * @bits
+      (@field[element] & (@bucket_mask << shift_bits)) == 0
+    end
+
+    def inc(position)
+    end
+
+    def dec(position)
+    end
+    
+    # iterate over each bucket
+    def each(&block)
+      @size.times { |position| yield self[position] }
+    end
+    
+    # returns the field as a string like "0101010100111100," etc.
+    def to_s(base = 2)
+      case base 
+      when 2
+        inject("") { |a, b| a + "%0#{@bits}b " % b }.strip
+      when 10
+        self.inject("") { |a, b| a + "%1d " % b }.strip
+      else
+        raise(ArgumentError, "unsupported base")
+      end
+    end
+
+    # returns the total number of non zero buckets
+    def total_set
+      self.inject(0) { |a, bucket| a += bucket.zero? ? 0 : 1; a }
+    end
+
+  end
+end

data/lib/bloombroom/bits/bit_field.rb

@@ -0,0 +1,90 @@
+# inspired by Peter Cooper's http://snippets.dzone.com/posts/show/4234
+# 
+# create a bit field 1000 bits wide
+#   bf = BitField.new(1000)
+#
+#   bf[100] = 1 or bf.set(100)
+#   bf[100] => 1 or bg.get(100) => 1
+#   bf[100] = 0 or bf.unset(100)
+#   bf.zero?(100) => true
+#
+#   bf.to_s = "10101000101010101"
+#   bf.total_set => 10  (example - 10 bits are set to "1")
+
+module Bloombroom
+  class BitField
+    attr_reader :size
+    include Enumerable
+    
+    ELEMENT_WIDTH = 32
+    
+    def initialize(size)
+      @size = size
+      @field = Array.new(((size - 1) / ELEMENT_WIDTH) + 1, 0)
+    end
+    
+    # set a bit
+    # @param position [Fixnum] bit position
+    # @param value [Fixnum] bit value 0/1
+    def []=(position, value)
+      if value == 0
+        @field[position / ELEMENT_WIDTH] &= ~(1 << (position % ELEMENT_WIDTH))
+      else
+        @field[position / ELEMENT_WIDTH] |= 1 << (position % ELEMENT_WIDTH)
+      end
+    end
+    
+    # read a bit
+    # @param position [Fixnum] bit position
+    # @return [Fixnum] bit value 0/1
+    def [](position)
+      @field[position / ELEMENT_WIDTH] & 1 << (position % ELEMENT_WIDTH) > 0 ? 1 : 0
+    end
+    alias_method :get, :[]
+
+    # set a bit to 1
+    # @param position [Fixnum] bit position
+    def set(position)
+      # duplicated code to avoid a method call
+      @field[position / ELEMENT_WIDTH] |= 1 << (position % ELEMENT_WIDTH)
+    end
+
+    # set a bit to 0
+    # @param position [Fixnum] bit position
+    def unset(position)
+      # duplicated code to avoid a method call
+      @field[position / ELEMENT_WIDTH] &= ~(1 << (position % ELEMENT_WIDTH))
+    end
+
+    # check if bit is set
+    # @param position [Fixnum] bit position
+    # @return [Boolean] true if bit is set
+    def include?(position)
+      @field[position / ELEMENT_WIDTH] & 1 << (position % ELEMENT_WIDTH) > 0
+    end
+    
+    # check if bit is not set
+    # @param position [Fixnum] bit position
+    # @return [Boolean] true if bit is not set
+    def zero?(position)
+      # duplicated code to avoid a method call
+      @field[position / ELEMENT_WIDTH] & 1 << (position % ELEMENT_WIDTH) == 0
+    end
+
+    # iterate over each bit
+    def each(&block)
+      @size.times { |position| yield self[position] }
+    end
+    
+    # returns the field as a string like "0101010100111100," etc.
+    def to_s
+      inject("") { |a, b| a + b.to_s }
+    end
+    
+    # returns the total number of bits that are set
+    # (the technique used here is about 6 times faster than using each or inject direct on the bitfield)
+    def total_set
+      @field.inject(0) { |a, byte| a += byte & 1 and byte >>= 1 until byte == 0; a }
+    end
+  end
+end

data/lib/bloombroom/filter/bloom_filter.rb

@@ -0,0 +1,45 @@
+require 'bloombroom/hash/ffi_fnv'
+require 'bloombroom/bits/bit_field'
+require 'bloombroom/filter/bloom_helper'
+
+module Bloombroom
+
+  # BloomFilter false positive probability rule of thumb: see http://www.igvita.com/2008/12/27/scalable-datasets-bloom-filters-in-ruby/
+  # a Bloom filter with a 1% error rate and an optimal value for k only needs 9.6 bits per key, and each time we add 4.8 bits 
+  # per element we decrease the error rate by ten times. 
+  #
+  # 10000 elements, 1% error rate: m = 10000 * 10 bits -> 12k of memory, k = 0.7 * (10000 * 10 bits / 10000) = 7 hash functions
+  # 10000 elements, 0.1% error rate: m = 10000 * 15 bits -> 18k of memory, k = 0.7 * (10000 * 15 bits / 10000) = 11 hash functions
+  #
+  # Bloombroom::BloomHelper.find_m_k can be used to compute optimal m & k values for a required capacity and error rate.
+  class BloomFilter
+
+    attr_reader :m, :k, :bits, :size
+
+    # @param m [Fixnum] filter size in bits
+    # @param k [Fixnum] number of hashing functions
+    def initialize(m, k)
+      @bits = BitField.new(m)
+      @m = m
+      @k = k
+      @size = 0
+    end
+    
+    # @param key [String] the key to add in the filter
+    # @return [Fixnum] the total number of keys in the filter
+    def add(key)
+      BloomHelper.multi_hash(key, @k).each{|position| @bits.set(position % @m)}
+      @size += 1
+    end
+    alias_method :<<, :add
+    
+    # @param key [String] test for the inclusion if key in the filter
+    # @return [Boolean] true if given key is present in the filter. false positive are possible and dependant on the m and k filter parameters.
+    def include?(key)
+      BloomHelper.multi_hash(key, @k).each{|position| return false unless @bits.include?(position % @m)}
+      true
+    end
+    alias_method :[], :include?
+    
+  end
+end

data/lib/bloombroom/filter/bloom_helper.rb

@@ -0,0 +1,35 @@
+require 'bloombroom/hash/ffi_fnv'
+
+module Bloombroom
+
+  class BloomHelper
+
+    # compute optimal m and k for a given capacity and error rate
+    # @param capacity [Fixnum] number of expected keys
+    # @param error [Float] error rate (0.0 < error < 1.0). Ex: 1% == 0.01, 0.1% == 0.001, ...
+    def self.find_m_k(capacity, error)
+      # thanks to http://www.siaris.net/index.cgi/Programming/LanguageBits/Ruby/BloomFilter.rdoc
+      m = (capacity * Math.log(error) / Math.log(1.0 / 2 ** Math.log(2))).ceil
+      k = (Math.log(2) * m / capacity).round
+      [m, k]
+    end
+    
+    # produce k hash values for key
+    # @param key [String] key to hash
+    # @param k [Fixnum] number of hash functions
+    def self.multi_hash(key, k)
+      # simulate n hash functions by having just two hash functions
+      # see http://citeseer.ist.psu.edu/viewdoc/download?doi=10.1.1.152.579&rep=rep1&type=pdf
+      # see http://willwhim.wordpress.com/2011/09/03/producing-n-hash-functions-by-hashing-only-once/
+      #
+      # fake two hash functions by using the upper/lower 32 bits of a 64 bits FNV1a hash
+
+      h = Bloombroom::FNVFFI.fnv1a_64(key)
+      a = (h & 0xFFFFFFFF00000000) >> 32
+      b = h & 0xFFFFFFFF
+
+      Array.new(k) {|i| (a + b * (i + 1))}
+    end
+
+  end
+end

data/lib/bloombroom/filter/continuous_bloom_filter.rb

@@ -0,0 +1,108 @@
+require 'bloombroom/hash/ffi_fnv'
+require 'bloombroom/bits/bit_bucket_field'
+require 'bloombroom/filter/bloom_helper'
+require 'thread'
+
+module Bloombroom
+
+  # ContinuousBloomFilter is a bloom filter for unbounded stream of keys where keys are expired over a given period 
+  # of time. The expected capacity of the bloom filter for the desired validity period must be known or estimated. 
+  # For a given capacity and error rate, BloomHelper.find_m_k can be used to compute optimal m & k values. 
+  #
+  # 4 bits per key (instead of 1 bit in a normal bloom filter) are used for keeping track of the keys ttl. 
+  # the internal timer resolution is set to half of the ttl (resolution divisor of 2). using 4 bits gives us
+  # 15 usable time slots (slot 0 is for the unset state). basically the internal time bookeeping is similar to a
+  # ring buffer where the first timer tick will be time slot=1, slot=2, .. slot=15, slot=1 and so on. The total 
+  # time of our internal clock will thus be 15 * (ttl / 2). We keep track of ttl by writing the current time slot 
+  # in the key k buckets when first inserted in the filter. when doing a key lookup if any of the bucket contain 
+  # the 0 value the key is not found. if the interval betweem the current time slot and any of the k buckets value 
+  # is greater than 2 (resolution divisor) we know this key is expired and we reset the expired buckets to 0.
+  class ContinuousBloomFilter
+
+    attr_reader :m, :k, :ttl, :buckets
+
+    RESOLUTION_DIVISOR = 2
+    BITS_PER_BUCKET = 4
+
+    # @param m [Fixnum] total filter size in number of buckets. optimal m can be computed using BloomHelper.find_m_k
+    # @param k [Fixnum] number of hashing functions. optimal k can be computed using BloomHelper.find_m_k
+    # @param ttl [Fixnum] key time to live in seconds (validity period)
+    def initialize(m, k, ttl)
+      @m = m
+      @k = k
+      @ttl = ttl
+      @buckets = BitBucketField.new(BITS_PER_BUCKET, m)
+
+      # time management
+      @increment_period = @ttl / RESOLUTION_DIVISOR
+      @current_slot = 1
+      @max_slot = (2 ** BITS_PER_BUCKET) - 1 # ex. with 4 bits -> we want range 1..15
+      @lock = Mutex.new
+    end
+    
+    # @param key [String] the key to add in the filter
+    # @return [ContinuousBloomFilter] self
+    def add(key)
+      current_slot = @lock.synchronize{@current_slot}
+      BloomHelper.multi_hash(key, @k).each{|position| @buckets[position % @m] = current_slot}
+      self
+    end
+    alias_method :<<, :add
+    
+    # @param key [String] test for the inclusion if key in the filter
+    # @return [Boolean] true if given key is present in the filter. false positive are possible and dependant on the m and k filter parameters.
+    def include?(key)
+      current_slot = @lock.synchronize{@current_slot}
+      expired = false
+
+      BloomHelper.multi_hash(key, @k).each do |position| 
+        start_slot = @buckets[position % @m]
+        if start_slot == 0
+          expired = true
+        elsif elapsed(start_slot, current_slot) > RESOLUTION_DIVISOR
+          expired = true
+          @buckets[position % @m] = 0
+        end
+      end
+      !expired
+    end
+    alias_method :[], :include?
+
+    # start the internal timer thread for managing ttls. must be explicitely called 
+    def start_timer
+      @timer ||= detach_timer
+    end
+
+    # advance internal time slot. this is exposed primarily for spec'ing purposes.
+    # normally this is automatically called by the internal timer thread but if not 
+    # using the internal timer thread it can be called explicitly when doing your
+    # own time management.
+    def inc_time_slot
+      # ex. with 4 bits -> we want range 1..15, 
+      @lock.synchronize{@current_slot = (@current_slot % @max_slot) + 1}
+    end
+
+    private
+
+    def current_slot
+      @lock.synchronize{@current_slot}
+    end
+
+    def elapsed(start_slot, current_slot)
+      # ring buffer style
+      current_slot >= start_slot ? current_slot - start_slot : (current_slot + @max_slot) - start_slot
+    end
+
+    def detach_timer
+      Thread.new do
+        Thread.current.abort_on_exception = true
+
+        loop do
+          sleep(@increment_period)
+          inc_time_slot
+        end 
+      end
+    end
+
+  end
+end

data/lib/bloombroom/hash/ffi_fnv.rb

@@ -0,0 +1,30 @@
+require 'ffi'
+
+module Bloombroom
+  class FNVFFI
+    extend FFI::Library
+
+    ffi_lib File.dirname(__FILE__) + "/" + (FFI::Platform.mac? ? "ffi_fnv.bundle" : FFI.map_library_name("ffi_fnv"))
+
+    attach_function :c_fnv1_32, :fnv1_32, [:string, :uint32], :uint32
+    attach_function :c_fnv1a_32, :fnv1a_32, [:string, :uint32], :uint32
+    attach_function :c_fnv1_64, :fnv1_64, [:string, :uint32], :uint64
+    attach_function :c_fnv1a_64, :fnv1a_64, [:string, :uint32], :uint64
+
+    def self.fnv1_32(data)
+      c_fnv1_32(data, data.size)
+    end
+
+    def self.fnv1_64(data)
+      c_fnv1_64(data, data.size)
+    end
+
+    def self.fnv1a_32(data)
+      c_fnv1a_32(data, data.size)
+    end
+
+    def self.fnv1a_64(data)
+      c_fnv1a_64(data, data.size)
+    end
+  end
+end

data/lib/bloombroom/hash/fnv_a.rb

@@ -0,0 +1,100 @@
+# based on https://github.com/andyjeffries/digestfnv
+
+module Bloombroom 
+  class FNVA
+
+    OFFSET32   = 2166136261
+    OFFSET64   = 14695981039346656037
+    OFFSET128  = 144066263297769815596495629667062367629
+    OFFSET256  = 100029257958052580907070968620625704837092796014241193945225284501741471925557
+    OFFSET512  = 9659303129496669498009435400716310466090418745672637896108374329434462657994582932197716438449813051892206539805784495328239340083876191928701583869517785
+    OFFSET1024 = 14197795064947621068722070641403218320880622795441933960878474914617582723252296732303717722150864096521202355549365628174669108571814760471015076148029755969804077320157692458563003215304957150157403644460363550505412711285966361610267868082893823963790439336411086884584107735010676915
+
+    PRIME32   = 16777619
+    PRIME64   = 1099511628211
+    PRIME128  = 309485009821345068724781371
+    PRIME256  = 374144419156711147060143317175368453031918731002211
+    PRIME512  = 35835915874844867368919076489095108449946327955754392558399825615420669938882575126094039892345713852759
+    PRIME1024 = 5016456510113118655434598811035278955030765345404790744303017523831112055108147451509157692220295382716162651878526895249385292291816524375083746691371804094271873160484737966720260389217684476157468082573
+
+    MASK32   = (2 ** 32) - 1
+    MASK64   = (2 ** 64) - 1
+    MASK128  = (2 ** 128) - 1
+    MASK256  = (2 ** 256) - 1
+    MASK512  = (2 ** 512) - 1
+    MASK1024 = (2 ** 1024) - 1
+
+    def self.fnv1_32(input)
+      hash = OFFSET32
+      input.each_byte { |b| hash = (hash * PRIME32) ^ b }
+      hash & MASK32
+    end
+    
+    def self.fnv1_64(input)
+      hash = OFFSET64
+      input.each_byte { |b| hash = (hash * PRIME64) ^ b }
+      hash & MASK64
+    end
+
+    def self.fnv1_128(input)
+      hash = OFFSET128
+      input.each_byte { |b| hash = (hash * PRIME128) ^ b }
+      hash & MASK128
+    end
+
+    def self.fnv1_256(input)
+      hash = OFFSET256
+      input.each_byte { |b| hash = (hash * PRIME256) ^ b }
+      hash & MASK256
+    end
+
+    def self.fnv1_512(input)
+      hash = OFFSET512
+      input.each_byte { |b| hash = (hash * PRIME512) ^ b }
+      hash & MASK512
+    end
+
+    def self.fnv1_1024(input)
+      hash = OFFSET1024
+      input.each_byte { |b| hash = (hash * PRIME1024) ^ b }
+      hash & MASK1024
+    end
+
+    def self.fnv1a_32(input)
+      hash = OFFSET32
+      input.each_byte { |b| hash = (hash ^ b) * PRIME32 }
+      hash & MASK32
+    end
+    
+    def self.fnv1a_64(input)
+      hash = OFFSET64
+      input.each_byte { |b| hash = (hash ^ b) * PRIME64 }
+      hash & MASK64
+    end
+
+    def self.fnv1a_128(input)
+      hash = OFFSET128
+      input.each_byte { |b| hash = (hash ^ b) * PRIME128 }
+      hash & MASK128
+    end
+
+    def self.fnv1a_256(input)
+      hash = OFFSET256
+      input.each_byte { |b| hash = (hash ^ b) * PRIME256 }
+      hash & MASK256
+    end
+
+    def self.fnv1a_512(input)
+      hash = OFFSET512
+      input.each_byte { |b| hash = (hash ^ b) * PRIME512 }
+      hash & MASK512
+    end
+
+    def self.fnv1a_1024(input)
+      hash = OFFSET1024
+      input.each_byte { |b| hash = (hash ^ b) * PRIME1024 }
+      hash & MASK1024
+    end
+
+  end
+end

data/lib/bloombroom/hash/fnv_b.rb

@@ -0,0 +1,56 @@
+# based on https://github.com/jakedouglas/fnv-ruby
+
+module Bloombroom
+  class FNVB
+    INIT32  = 0x811c9dc5
+    INIT64  = 0xcbf29ce484222325
+    PRIME32 = 0x01000193
+    PRIME64 = 0x100000001b3
+    MOD32   = 2 ** 32
+    MOD64   = 2 ** 64
+
+    def self.fnv1_32(data)
+      hash = INIT32
+
+      data.each_byte do |byte|
+        hash = (hash * PRIME32) % MOD32
+        hash = hash ^ byte
+      end
+
+      hash
+    end
+
+    def self.fnv1_64(data)
+      hash = INIT64
+
+      data.each_byte do |byte|
+        hash = (hash * PRIME64) % MOD64
+        hash = hash ^ byte
+      end
+
+      hash
+    end
+
+    def self.fnv1a_32(data)
+      hash = INIT32
+
+      data.each_byte do |byte|
+        hash = hash ^ byte
+        hash = (hash * PRIME32) % MOD32
+      end
+
+      hash
+    end
+
+    def self.fnv1a_64(data)
+      hash = INIT64
+
+      data.each_byte do |byte|
+        hash = hash ^ byte
+        hash = (hash * PRIME64) % MOD64
+      end
+
+      hash
+    end
+  end
+end

data/lib/bloombroom/version.rb

@@ -0,0 +1,3 @@
+module Bloombroom
+  VERSION = "1.0.0"
+end

metadata

@@ -0,0 +1,100 @@
+--- !ruby/object:Gem::Specification
+name: bloombroom
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+  prerelease: 
+platform: ruby
+authors:
+- Colin Surprenant
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-05-09 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.8.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.8.0
+- !ruby/object:Gem::Dependency
+  name: ffi
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: bloombroom has two bloom filter implementations, a standard filter for
+  bounded key space                    and a continuous filter for unbounded keys
+  (stream). also contains fast bit field and                    bit bucket field (multi
+  bits), native/C-ext/FFI FNV hashing and benchmarks for all these.
+email:
+- colin.surprenant@gmail.com
+executables: []
+extensions:
+- ext/bloombroom/hash/cext/extconf.rb
+- ext/bloombroom/hash/ffi/extconf.rb
+extra_rdoc_files: []
+files:
+- lib/bloombroom/bits/bit_bucket_field.rb
+- lib/bloombroom/bits/bit_field.rb
+- lib/bloombroom/filter/bloom_filter.rb
+- lib/bloombroom/filter/bloom_helper.rb
+- lib/bloombroom/filter/continuous_bloom_filter.rb
+- lib/bloombroom/hash/ffi_fnv.rb
+- lib/bloombroom/hash/fnv_a.rb
+- lib/bloombroom/hash/fnv_b.rb
+- lib/bloombroom/version.rb
+- lib/bloombroom.rb
+- ext/bloombroom/hash/cext/extconf.rb
+- ext/bloombroom/hash/ffi/extconf.rb
+- ext/bloombroom/hash/cext/cext_fnv.c
+- ext/bloombroom/hash/ffi/ffi_fnv.c
+- README.md
+- CHANGELOG.md
+- LICENSE.md
+homepage: https://github.com/colinsurprenant/bloombroom
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: bloombroom
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: bloom filters for bounded and unbounded (streaming) data, FNV hashing and
+  bit fields
+test_files: []