campchair 0.0.4

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+tmp/ccdb

data/.rspec

@@ -0,0 +1,2 @@
+--colour
+--order random

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - 1.8.7
+  - 1.9.3
+

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in campchair.gemspec
+gemspec

data/README.md

@@ -0,0 +1,273 @@
+# ⑁ Campchair
+### Lightweight, ghetto-tech CouchDB-like views in Ruby
+
+[![Build Status](https://secure.travis-ci.org/captainpete/campchair.png?branch=master)](http://travis-ci.org/captainpete/campchair)
+
+___This README is a wishlist. The thing doesn't entirely work yet!___
+
+Campchair provides some helpers for lazily materialized views and a
+persistence mapping for different Ruby types.
+
+Views are described a bit like those found in [CouchDB](http://couchdb.apache.org/).
+Campchair is not a service and has no client-server stuff.
+Parallel writes are not supported, but that's okay – you can use something
+else for concurrency and use this in your persistence fiber/thread/process.
+Persistence happens through [LevelDB](http://code.google.com/p/leveldb/).
+This is not append-only and replication is not a feature.
+For these reasons it's called campchair, not couch :)
+
+This project was born at [Railscamp 11](http://railscamps.com/) for the heavy-metrics project.
+
+### How does it look?
+
+For these examples we'll work with two documents.
+
+```ruby
+fred = {
+  :name       => 'Fred', :height => 1.748,
+  :location   => 'Flinders St Station, Melbourne, Australia',
+  :first_seen => Time.new(2012, 3, 6, 12, 10),
+  :last_seen  => Time.new(2012, 3, 6, 12, 40)
+}
+jane = {
+  :name       => 'Jane', :height => 1.634,
+  :location   => 'Flinders St Station, Melbourne, Australia',
+  :first_seen => Time.new(2012, 3, 6, 12, 20),
+  :last_seen  => Time.new(2012, 3, 6, 12, 50)
+}
+```
+
+Mixing in Campchair DB behavior to a class is as simple as...
+
+```ruby
+class People
+  include Campchair::LevelDB
+end
+```
+
+Now, use it like a `Hash`.
+
+```ruby
+# Add a document to the db
+id = People << fred
+
+# Add another person
+People << jane
+
+# Rerieve a document from the db
+fred = People[id]
+
+# Delete the document from the db
+People.delete(id)
+
+# Update/create the document in the db using an id
+fred[:last_seen] = Time.new(2012, 3, 6, 12, 40, 25)
+doc[id] = fred
+```
+
+### Basic views _not yet implemented_
+
+For views to be available on classes we need to include `Campchair::Views`.
+
+```ruby
+class People # add some views to the people class
+  include Campchair::Views
+
+  # Map to document by id
+  view :all do
+    map { |id, person| emit id, person }
+  end
+
+  # Map to name by doc id
+  view :names do
+    map { |id, person| emit id, person.name }
+  end
+end
+
+People.names
+# ['Fred', 'Jane']
+```
+
+### Using reduce _not yet implemented_
+
+Reduce gets called on groups of map results.
+
+```ruby
+class People
+  # Map to count by doc id
+  # Reduce to sum of counts
+  view :count do
+    map { |id, person| emit id, 1 }
+    reduce do |keys, values|
+      values.inject(0) { |memo, values| memo + values }
+    end
+  end
+end
+
+People.count
+# 2
+```
+
+Reduce happens in stages. Supply a rereduce step to handle reducing reduce
+results. The advantage of rereduce is results can be cached in a b-tree index.
+
+If `rereduce` is supplied then `reduce` is called with a portion of the `map`
+results and those reductions are passed to `rereduce`. If `rereduce` is
+omitted then `reduce` results will not be cached.
+
+```ruby
+class People
+  # Map to location by doc id
+  # Reduce by unique location
+  view :unique_locations do
+    map { |id, person| emit id, person.location }
+    reduce do |keys, values|
+      results = values.uniq
+    end
+    rereduce do |results|
+      results.inject(Set.new) { |memo, values| memo.merge values }
+    end
+  end
+end
+
+People.unique_locations
+# <Set: {"Flinders St Station, Melbourne, Australia"}>
+```
+
+Keep in mind that if `rereduce` is not supplied, `reduce` will always get
+called with the entire keyspace. If the dataset is large this might consume a
+lot of memory.
+
+### Using keys with reduce _not yet implemented_
+
+```ruby
+class People
+  # Map to count by name
+  # Reduce by sum of counts
+  view :count_by_name do
+    map { |id, person| emit name, 1 }
+    reduce do |keys, values|
+      values.inject(0) { |memo, value| memo + value }
+    end
+    rereduce do |results|
+      results.inject(0) { |memo, value| memo + value }
+    end
+  end
+end
+
+People.count_by_name['Jane']
+# 1
+People.count_by_name
+# 2
+```
+
+### Controlling the index _not yet relevant_
+
+It's possible to write reduce methods that return results larger than the
+input. If you're doing this, you're gonna have a bad time. Unless you're sure
+the dataset will remain small enough that the index for each `rereduce` doesn't
+blow out your disk store.
+
+```ruby
+class People
+  # DANGER: this view will blow out the index
+
+  # Map to count by name
+  # Reduce by sum of counts by name
+  view :count_all_by_name do
+    map { |id, person| emit person.name, 1 }
+    reduce do |keys, values|
+      result = Hash.new
+      keys.each_with_index do |key, index|
+        result[key] ||= 0
+        result[key] += values[index]
+      end
+      result # scary, nearly as large as the input
+    end
+    rereduce(:cache => false) do |results|
+      result = {}
+      results.each do |name, count|
+        result[name] ||= 0
+        result[name] += count
+      end
+      result # likewise, this will explode the index
+    end
+  end
+end
+
+People.count_all_by_name['Jane']
+# { 'Jane' => 1 }
+People.count_all_by_name
+# { 'Fred' => 1, 'Jane' => 1 }
+```
+
+### Re-re-reducing _not yet implemented_
+
+Sometimes it's useful to post-process reduce results. Add another rereduce to
+process rereduced values. Only the first rereduce gets called on values it
+produces itself. Subsequent rereduces are chained to the initial result.
+
+```ruby
+clas People
+  # Map to height by location
+  # Reduce by count and sum of heights
+  # Rereduce by the calculating average
+  view :average_height_by_location do
+    map { |id, person| emit location, height }
+    reduce do |keys, values|
+      result = Hash.new
+      keys.each_with_index do |key, index|
+        result[key] ||= { :count => 0, :sum => 0 }
+        result[key][:count] += 1
+        result[key][:sum] += height
+      end
+      result
+    end
+    rereduce do |results|
+      result = { :count => 0, :sum => 0 }
+      results.each do |result|
+        result[:count] += part[:count]
+        result[:sum] += part[:sum]
+      end
+      result
+    end
+    rereduce do |results|
+      count = results.inject(0) { |memo, result| result[:count] }
+      sum = results.inject(0) { |memo, result| result[:sum] }
+      count == 0 ? nil : sum / count
+    end
+  end
+end
+
+People.average_height_by_location
+# 1.690999999
+People.average_height_by_location['Flinders St Station, Melbourne, Australia']
+# 1.690999999
+People.average_height_by_location['Docklands, Melbourne, Australia']
+# nil
+```
+
+### Custom database path
+
+By default, the folder for db files is 'cddb'.  
+You can change this with:
+```ruby
+Campchair.db_path = 'db/heavy_metrics'
+```
+
+You can also change the db on a per-class basis.
+```ruby
+class Person
+  include Campchair::LevelDB
+  self.db_path = 'db/heavy_metrics/Person'
+end
+```
+
+### TODO
+
+- Make the README examples work
+- Views spanning views
+- Caching view results
+- Caching reduce results
+- method_missing for `count_by_*, sum_of_*s, min_*, max_*, unique_*s`
+- cache priming

data/Rakefile

@@ -0,0 +1,9 @@
+require 'bundler/gem_tasks'
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |spec|
+  spec.pattern = FileList['spec/**/*_spec.rb']
+end
+
+task :default => :spec

data/campchair.gemspec

@@ -0,0 +1,24 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+require "campchair/version"
+
+Gem::Specification.new do |s|
+  s.name        = "campchair"
+  s.version     = Campchair::VERSION
+  s.authors     = ["Peter Hollows"]
+  s.email       = ["pete@dojo7.com"]
+  s.homepage    = "https://github.com/captainpete/campchair"
+  s.summary     = %q{Lightweight, ghetto-tech CouchDB-like views in Ruby}
+  s.description = %q{Campchair is a map-reduce framework using levelDB for persistence, and Ruby for everything else.}
+
+  s.rubyforge_project = "campchair"
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  s.add_dependency(%q<leveldb-ruby>, ["~> 0.14"])
+  s.add_development_dependency(%q<rspec>)
+  s.add_development_dependency(%q<rake>)
+end

data/lib/campchair.rb

@@ -0,0 +1,14 @@
+require "campchair/version"
+
+module Campchair
+  autoload :Views,    'campchair/views'
+  autoload :LevelDB,  'campchair/leveldb'
+
+  class << self
+    attr_writer :db_path
+
+    def db_path
+      @db_path || 'ccdb'
+    end
+  end
+end

data/lib/campchair/leveldb.rb

@@ -0,0 +1,70 @@
+require 'leveldb'
+require 'digest/sha1'
+
+module Campchair
+  module LevelDB
+    class << self
+      def generate_key
+        @alpha ||= [('a'..'f'), ('0'..'9')].map(&:to_a).flatten
+        (0...40).map { @alpha[rand(@alpha.size)] }.join
+      end
+
+      def encode(value)
+        Marshal.dump(value)
+      end
+
+      def decode(value)
+        Marshal.load(value)
+      end
+
+      def included(base)
+        base.class_eval do
+          class << self
+            attr_writer :db_path, :db_name, :db
+
+            def db_path
+              @db_path || File.join(Campchair.db_path, db_name)
+            end
+
+            def db_name
+              @db_name || name
+            end
+
+            def db
+              @db ||= create_db
+            end
+
+            def [](key)
+              val = db[key]
+              val && Campchair::LevelDB.decode(db[key])
+            end
+
+            def []=(key, value)
+              db[key] = Campchair::LevelDB.encode(value)
+            end
+
+            def <<(value)
+              id = Campchair::LevelDB.generate_key
+              self[id] = value
+              return id
+            end
+
+            def delete(key)
+              db.delete(key)
+            end
+
+          protected
+
+            def create_db
+              FileUtils.mkdir_p(db_path)
+              ::LevelDB::DB.new(db_path)
+            end
+
+          end # class << self
+
+        end # base.class_eval
+      end # included
+    end # class << self
+
+  end # module LevelDB
+end # module Campchair

data/lib/campchair/version.rb

@@ -0,0 +1,3 @@
+module Campchair
+  VERSION = "0.0.4"
+end

data/lib/campchair/views.rb

@@ -0,0 +1,4 @@
+module Campchair
+  module Views
+  end
+end

data/spec/campchair/leveldb_spec.rb

@@ -0,0 +1,139 @@
+require 'spec_helper'
+
+describe Campchair::LevelDB do
+  let(:test_module) do
+    Campchair::LevelDB.dup
+  end
+
+  describe ".generate_key" do
+    it "generates a hex key 40 characters long" do
+      test_module.generate_key.should =~ /\A[a-f0-9]{40}\Z/
+    end
+  end
+
+  context "included" do
+    let(:test_class) do
+      Object.send(:remove_const, :TestEntity) if defined? TestEntity
+      TestEntity = Class.new do
+        include Campchair::LevelDB
+      end
+    end
+
+    after do
+      `rm -fr tmp/ccdb`
+    end
+
+    describe "setting the db_name" do
+      describe ".db_name" do
+        it "is the class name by default" do
+          test_class.stub! :name => 'TestEntity'
+          test_class.db_name.should == 'TestEntity'
+        end
+      end
+
+      describe ".db_name=" do
+        it "is the custom db_name specified" do
+          test_class.db_name = 'TestEntityCustomName'
+          test_class.db_name.should == 'TestEntityCustomName'
+        end
+      end
+    end
+
+    describe "setting the db_path" do
+      describe ".db_path" do
+        it "is the Campchair.db_path, and the db_name by default" do
+          Campchair.should_receive(:db_path).and_return('some/path')
+          test_class.db_path.should == 'some/path/TestEntity'
+        end
+      end
+
+      describe ".db_path=" do
+        it "is the custom db_path specified" do
+          test_class.db_path = 'some/other/path'
+          test_class.db_path.should == 'some/other/path'
+        end
+      end
+    end
+
+    context "with test db_path set" do
+      before do
+        test_class.db_path = 'tmp/ccdb/test'
+      end
+
+      describe ".db" do
+        it "is a LevelDB, at the db_path" do
+          test_class.db.should be_a(LevelDB::DB)
+          test_class.db.pathname.should == 'tmp/ccdb/test'
+        end
+      end
+
+      context "no data serialization" do
+        before do
+          # Passthrough {en,de}coding for these tests
+          Campchair::LevelDB.stub(:encode).and_return {|v| v}
+          Campchair::LevelDB.stub(:decode).and_return {|v| v}
+
+          FileUtils.mkdir_p('tmp/ccdb/TestEntity')
+          @db = ::LevelDB::DB.new('tmp/ccdb/TestEntity')
+
+          test_class.stub! :db => @db
+        end
+
+        describe ".[]" do
+          it "is nil when there's no matching key in the database" do
+            test_class['nonexistent'].should be_nil
+          end
+
+          it "is the value of the matching key in the database" do
+            @db['something'] = '22'
+            test_class['something'].should == '22'
+          end
+        end
+
+        describe ".[]=" do
+          it "sets the value of the key in the database" do
+            test_class['something'] = '22'
+            @db['something'].should == '22'
+          end
+        end
+
+        describe ".<<" do
+          it "adds the value to the database, generating a key" do
+            key = 'b78aa2f6b718651743fac2682003f2c63340ae34b'
+            Campchair::LevelDB.stub! :generate_key => key
+
+            id = TestEntity << '22'
+            id.should == key
+            test_class[id].should == '22'
+          end
+        end
+
+        describe ".delete" do
+          it "deletes the key its values from the db" do
+            test_class['something'] = '22'
+            test_class.delete('something')
+            
+            test_class['something'].should be_nil
+          end
+        end
+      end
+
+      context "Campchair::LevelDB serialization" do
+        describe ".[]{,=}" do
+          it "persists non-string data structures" do
+            doc = { :sym => :sym, 'string' => 'string', 'types' => [1234.12341234, (55..44), /regexen/, Time.now, { 'a' => 22 }] }
+            test_class['something'] = doc
+            test_class['something'].should == doc
+          end
+        end
+
+        describe ".[]" do
+          it "is nil when the document does not exist" do
+            test_class['something'].should be_nil
+          end
+        end
+      end
+    end
+
+  end # context "included"
+end # describe Campchair::LevelDB

data/spec/campchair/views_spec.rb

@@ -0,0 +1,4 @@
+require 'spec_helper'
+
+describe Campchair::Views do
+end

data/spec/campchair_spec.rb

@@ -0,0 +1,21 @@
+require 'spec_helper'
+
+describe Campchair do
+  let(:test_module) do
+    Campchair::dup
+  end
+
+  describe ".db_path" do
+    it "is 'ccdb' by default" do
+      test_module.db_path.should == 'ccdb'
+    end
+  end
+
+  describe ".db_path=" do
+    it "sets the default LevelDB for Campchair classes" do
+      test_module.db_path = 'tmp/ccdb'
+      test_module.db_path.should == 'tmp/ccdb'
+    end
+  end
+  
+end

data/spec/integration/readme_examples_spec.rb

@@ -0,0 +1,60 @@
+require 'spec_helper'
+
+describe "DB behavior" do
+  before do
+    @fred = {
+      :name       => 'Fred', :height => 1.748,
+      :location   => 'Flinders St Station, Melbourne, Australia',
+      :first_seen => Time.local(2012, 3, 6, 12, 10),
+      :last_seen  => Time.local(2012, 3, 6, 12, 40)
+    }
+    @jane = {
+      :name       => 'Jane', :height => 1.634,
+      :location   => 'Flinders St Station, Melbourne, Australia',
+      :first_seen => Time.local(2012, 3, 6, 12, 20),
+      :last_seen  => Time.local(2012, 3, 6, 12, 50)
+    }
+
+    class TestPeople
+      include Campchair::LevelDB
+      self.db_path = 'tmp/ccdb'
+    end
+  end
+
+  after do
+    `rm -fr tmp/ccdb`
+  end
+
+  it "persists documents without an id" do
+    id = TestPeople << @fred
+    TestPeople  << @jane
+  end
+
+  it "retrieves documents" do
+    id = TestPeople << @fred
+    TestPeople[id].should == @fred
+  end
+
+  it "deletes documents" do
+    id = TestPeople << @fred
+    TestPeople.delete(id)
+    TestPeople[id].should be_nil
+  end
+
+  it "updates docuemnts with an id" do
+    id = TestPeople << @fred
+    new_last_seen = Time.local(2012, 3, 6, 12, 50)
+    @fred[:last_seen] = new_last_seen
+    TestPeople[id] = @fred
+
+    TestPeople[id][:last_seen].should == new_last_seen
+  end
+
+  it "creates documents with an id" do
+    id = 'something'
+    TestPeople[id] = @fred
+
+    TestPeople[id].should == @fred
+  end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,2 @@
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+require 'campchair'

metadata

@@ -0,0 +1,102 @@
+--- !ruby/object:Gem::Specification
+name: campchair
+version: !ruby/object:Gem::Version
+  version: 0.0.4
+  prerelease: 
+platform: ruby
+authors:
+- Peter Hollows
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2012-07-18 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: leveldb-ruby
+  requirement: &70242209955760 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '0.14'
+  type: :runtime
+  prerelease: false
+  version_requirements: *70242209955760
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: &70242209955340 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70242209955340
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &70242209954880 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70242209954880
+description: Campchair is a map-reduce framework using levelDB for persistence, and
+  Ruby for everything else.
+email:
+- pete@dojo7.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .rspec
+- .travis.yml
+- Gemfile
+- README.md
+- Rakefile
+- campchair.gemspec
+- lib/campchair.rb
+- lib/campchair/leveldb.rb
+- lib/campchair/version.rb
+- lib/campchair/views.rb
+- spec/.gitkeep
+- spec/campchair/leveldb_spec.rb
+- spec/campchair/views_spec.rb
+- spec/campchair_spec.rb
+- spec/integration/readme_examples_spec.rb
+- spec/spec_helper.rb
+- tmp/.gitkeep
+homepage: https://github.com/captainpete/campchair
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: campchair
+rubygems_version: 1.8.17
+signing_key: 
+specification_version: 3
+summary: Lightweight, ghetto-tech CouchDB-like views in Ruby
+test_files:
+- spec/campchair/leveldb_spec.rb
+- spec/campchair/views_spec.rb
+- spec/campchair_spec.rb
+- spec/integration/readme_examples_spec.rb
+- spec/spec_helper.rb