hyper_record 0.2.8

data/.gitignore

@@ -0,0 +1,3 @@
+*.sw?
+.DS_Store
+coverage

data/CHANGELOG

@@ -0,0 +1,83 @@
+0.2.8 (2010/03/01)
+- support friendly migration-syntax declared using blocks
+- new assemble_row_key_from_attributes that generates row keys in the same 
+  order declared in row_key_attributes
+- cache row_key_attributes results so that regex is not constantly
+  reevaluated for same value
+- new :instantiate_only_requested_columns finder option that does not
+  instantiate object columns if they were not included in the :select
+  - makes large queries against large tables memory-friendly
+- :select option allows column families to be identified using symbols
+- tests and groundwork for mutator flush_interval (asynchronous writes)
+  - requires Hypertable 0.9.2.7
+- fix bug in with_thrift_client invocation
+- update tests to be compatible with Rails 2.3.4
+- update tests to be compatible with RSpec 1.2.8
+
+0.2.7 2009/10/01
+- new row_key_attributes method: allows parts of the row key to be extracted
+  into attributes on the instantiated object
+- retry block of write_cells was occasionally attemption to write to a
+  closed mutator (causing an exception)
+- support mutators with auto-flush
+  - update Thrift-generated ruby code to code supplied in Hypertable 0.9.2.6
+- some exceptions in write_cells method were being masked by exceptions in 
+  ensure block, making it difficult to identify root cause of problem
+- fix potential infinite loop in each_row
+- handle deleted columns in schema operations
+  - hypertable deletes columns lazily, so they will still show up in the output
+    from schema commands.
+
+0.2.6 2009/04/20
+- fix bug interacting with new open_scanner API
+
+0.2.5 2009/04/20
+- added methods to raw ruby thrift client for iterating over large results
+  - each_cell_as_arrays
+  - each_row (still has bug in underlying code)
+  - each_row_as_arrays (still has bug in underlying code)
+- new HyperRecord methods for iterating over results in small batches, doesn't 
+  exhaust memory on million-row queries
+  - find_each_row (returns each row as HyperRecord object)
+  - find_each_row_as_arrays (returns each row in native array format - fast)
+  - find_to_scan_spec methods (converts finder options to scan spec)
+  - find_with_scanner (converts finder options to a scanner)
+- new finder option (:row_intervals) that retrieves cells from multiple 
+  row intervals in a single query
+- now depends on Thrift SVN revision 765279 or later.
+
+0.2.4 2009/04/06
+- find_by_hql support
+- support for :select option in calls to ActiveRecord.find
+- expose open_mutator, flush_mutator and close_mutator methods to HyperRecord
+  to help speed up write-heavy applications
+- don't duplicate cells unnecessarily while assembling rows
+
+0.2.3 on 2009/03/18
+- optimize writes by using set_cells_as_arrays to write data in cell native
+  array form
+- monkey patch borrow method of Thrift::FramedTransport needed to return
+  substrings as well
+
+0.2.2 on 2009/03/17
+- monkey patch Thrift::FramedTransport to fix performance issue in thrift
+  transport ruby code (1000x faster than default implementation)
+- import and utilize new thrift client moethods that implement native
+  array interface for cell retrieval (100x faster than creating
+  Hypertable::ThriftGen::Cell objects)
+
+0.2.1 on 2009/03/06 
+- add rudimentary support for timestamps [kball]
+
+0.2.0 on 2009/03/05
+- bundle hypertable_adapter and hypertable_thrift_client code to
+  simplify installation
+- support for simple migrations in raw HQL or ActiveRecord::Migration format
+- attempt to reconnect to ThirftBroker on connection errors
+- intercept Thrift exceptions to make them compatible with Rails
+
+0.1.1 on 2009/03/03
+- don't use quoted table name in find_by_options
+
+0.1.0 on 2009/02/01
+- initial release with basic ActiveRecord and association features

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 tylerkovacs
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,49 @@
+hyper_record
+============
+
+Hypertable is a high performance distributed data storage system designed
+to support applications requiring maximum performance, scalability,
+and reliability. Modeled after Google's well known Bigtable project,
+Hypertable is designed to manage the storage and processing of information
+on a large cluster of commodity servers, providing resilience to machine
+and component failures.
+
+HyperRecord exists to integrate Hypertable with ActiveRecord (the Ruby on
+Rails ORM) allowing objects to be stored in Hypertable while maintaining
+as many of the built-in ORM features as possible.
+
+See:
+- http://www.hypertable.org
+- http://code.google.com/p/hypertable/wiki/HyperRecord
+
+INSTALLATION
+============
+
+1. Add github as a gem source (if you haven't already)
+
+  gem sources -a http://gems.github.com
+
+2. Install HyperRecord gem
+
+  gem install tylerkovacs-hyper_record
+
+3. Configure Hypertable Adapter by adding an entry to config/database.yml
+   that points to an instance of the Hypertable Thrift Broker (uses port
+   38080 by default).
+
+   hypertable:
+     adapter: hypertable
+     host: localhost
+     port: 38080
+
+   NOTE: HyperRecord requires Thrift SVN revision 765279 or later.
+
+4. Add the following to the bottom of config/environment.rb:
+
+  require 'hyper_record'
+
+
+COPYRIGHT
+=========
+
+Copyright (c) 2008 tylerkovacs. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,43 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rcov/rcovtask'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "hyper_record"
+    s.summary = %Q{Fully integrates ActiveRecord with Hypertable.}
+    s.email = "tyler.kovacs@gmail.com"
+    s.homepage = "http://github.com/tylerkovacs/hyper_record"
+    s.description = "See README"
+    s.authors = ["tylerkovacs"]
+  end
+
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+Rake::TestTask.new do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'hyper_record'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+Rcov::RcovTask.new do |t|
+  t.libs << 'spec'
+  t.test_files = FileList['spec/**/*_spec.rb']
+  t.verbose = true
+end
+
+task :default => :rcov

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:major: 0
+:minor: 2
+:patch: 8

data/benchmark/save.rb

@@ -0,0 +1,58 @@
+require File.join(File.dirname(__FILE__), '../spec/spec_helper.rb')
+
+# Results with flush_interval=10
+# 1000 save              requests in 14.879574 sec: 67.2062251244559 r/s
+#      hypertable write latency was 13.632525
+# 1000 save_with_mutator requests in 1.624659 sec: 615.513778583691 r/s
+#      hypertable write latency was 0.557777
+#
+# Results with flush_interval=100
+# 1000 save              requests in 13.127422 sec: 76.1764191019379 r/s
+#      hypertable write latency was 11.971095
+# 1000 save_with_mutator requests in 0.886474 sec: 1128.06466969138 r/s
+#      hypertable write latency was 0.454088000000001
+
+class Bench < ActiveRecord::HyperBase
+  def self.create_table
+    hql = "CREATE TABLE #{table_name} (
+      'value' MAX_VERSIONS=1
+    )"
+    connection.execute(hql)
+  end
+end
+
+Bench.drop_table if Bench.table_exists?
+Bench.create_table
+
+n = 1000
+
+t1 = Time.now
+for i in 1..n
+  record = Bench.new({:ROW => i, :value => i})
+  record.save!
+end
+ts1 = Time.now - t1
+
+read_latency, write_latency, cells_read = ActiveRecord::ConnectionAdapters::HypertableAdapter.get_timing
+ActiveRecord::ConnectionAdapters::HypertableAdapter.reset_timing
+puts "#{n} save              requests in #{ts1} sec: #{n/ts1} r/s"
+puts "     hypertable write latency was #{write_latency}"
+
+Bench.drop_table if Bench.table_exists?
+Bench.create_table
+
+flush_interval = 100
+t2 = Time.now
+m = Bench.open_mutator 
+for i in 1..n
+  record = Bench.new({:ROW => i, :value => i})
+  record.save_with_mutator!(m)
+  Bench.flush_mutator(m) if i % flush_interval == 0
+end
+Bench.close_mutator(m)
+ts2 = Time.now - t2
+
+read_latency, write_latency, cells_read = ActiveRecord::ConnectionAdapters::HypertableAdapter.get_timing
+ActiveRecord::ConnectionAdapters::HypertableAdapter.reset_timing
+puts "#{n} save_with_mutator requests in #{ts2} sec: #{n/ts2} r/s"
+puts "     hypertable write latency was #{write_latency}"

data/hyper_record.gemspec

@@ -0,0 +1,76 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE
+# Instead, edit Jeweler::Tasks in Rakefile, and run `rake gemspec`
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{hyper_record}
+  s.version = "0.2.8"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["tylerkovacs"]
+  s.date = %q{2010-03-01}
+  s.description = %q{See README}
+  s.email = %q{tyler.kovacs@gmail.com}
+  s.extra_rdoc_files = [
+    "LICENSE",
+     "README"
+  ]
+  s.files = [
+    ".gitignore",
+     "CHANGELOG",
+     "LICENSE",
+     "README",
+     "Rakefile",
+     "VERSION.yml",
+     "benchmark/save.rb",
+     "hyper_record.gemspec",
+     "init.rb",
+     "lib/active_record/connection_adapters/hyper_table_definition.rb",
+     "lib/active_record/connection_adapters/hypertable_adapter.rb",
+     "lib/active_record/connection_adapters/qualified_column.rb",
+     "lib/associations/hyper_has_and_belongs_to_many_association_extension.rb",
+     "lib/associations/hyper_has_many_association_extension.rb",
+     "lib/hyper_record.rb",
+     "lib/hypertable/gen-rb/client_constants.rb",
+     "lib/hypertable/gen-rb/client_service.rb",
+     "lib/hypertable/gen-rb/client_types.rb",
+     "lib/hypertable/gen-rb/hql_constants.rb",
+     "lib/hypertable/gen-rb/hql_service.rb",
+     "lib/hypertable/gen-rb/hql_types.rb",
+     "lib/hypertable/thrift_client.rb",
+     "lib/hypertable/thrift_transport_monkey_patch.rb",
+     "pkg/hyper_record-0.2.8.gem",
+     "spec/fixtures/pages.yml",
+     "spec/fixtures/qualified_pages.yml",
+     "spec/lib/associations_spec.rb",
+     "spec/lib/hyper_record_spec.rb",
+     "spec/lib/hypertable_adapter_spec.rb",
+     "spec/spec_helper.rb",
+     "test/test_helper.rb",
+     "test/thrift_client_test.rb"
+  ]
+  s.homepage = %q{http://github.com/tylerkovacs/hyper_record}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.6}
+  s.summary = %q{Fully integrates ActiveRecord with Hypertable.}
+  s.test_files = [
+    "spec/lib/hypertable_adapter_spec.rb",
+     "spec/lib/hyper_record_spec.rb",
+     "spec/lib/associations_spec.rb",
+     "spec/spec_helper.rb",
+     "test/thrift_client_test.rb",
+     "test/test_helper.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end

data/init.rb

@@ -0,0 +1 @@
+require 'hyper_record'

data/lib/active_record/connection_adapters/hyper_table_definition.rb

@@ -0,0 +1,26 @@
+# HyperTableDefinition is a subclass of TableDefinition that provides 
+# support for Hypertable-specific features (e.g, max_versions) to
+# Rails migrations.
+
+module ActiveRecord
+  module ConnectionAdapters #:nodoc:
+    class HyperColumnDefinition < Struct.new(:base, :name, :type, :limit, :max_versions, :options) #:nodoc:
+    end
+
+    class HyperTableDefinition < TableDefinition
+      def column(name, type, options = {})
+        column = self[name] || HyperColumnDefinition.new(@base, name, type)
+        if options[:limit]
+          column.limit = options[:limit]
+        elsif native[type.to_sym].is_a?(Hash)
+          column.limit = native[type.to_sym][:limit]
+        end
+        column.max_versions = options[:max_versions]
+        column.options = options
+        @columns << column unless @columns.include? column
+        self
+      end
+    end
+  end
+end
+