bigrecord 0.0.5

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 openplaces
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,44 @@
+= Big Record
+
+A Ruby Object/Data Mapper for distributed column-oriented data stores (inspired by BigTable) such as HBase. Intended to work as a drop-in for Rails applications. More information soon...
+
+== Features
+* Dynamic schemas (due to the schema-less design of BigTable).
+* Support for column-oriented data stores.
+* Similar usage to Active Record.
+* Embedded records that store within a single table row.
+* Automatic versioning.
+* Scalability (depending on the data store used).
+
+== Why would I use it?
+
+BigTable, and by extension, Big Record isn't right for everyone. A great introductory article discussing this topic can be found at http://blog.rapleaf.com/dev/?p=26 explaining why you would or wouldn't use BigTable. The rule of thumb, however, is that if your data model is simple or can fit into a standard RDBMS, then you probably don't need it.
+
+Beyond this though, there are two basic motivations that almost immediately demand a BigTable model database:
+1. Your data is highly dynamic in nature and would not fit in a schema bound model, or you cannot define a schema ahead of time.
+2. You know that your database will grow to tens or hundreds of gigabytes, and can't afford big iron servers. Instead, you'd like to scale horizontally across many commodity servers.
+
+== Requirements
+
+* Big Record: Ruby Object/Data Mapper. Inspired and architected similarly to Active Record.
+* Big Record Driver: JRuby application that bridges Ruby and Java (through JRuby's Drb protocol) to interact with Java-based data stores and their native APIs. Required for HBase and Cassandra. This application can be run from a separate server than your Rails application.
+  * JRuby 1.1.6+ is needed to run Big Record Driver.
+* Any other requirements needed to run Hadoop, HBase or your data store of choice.
+
+== Optional Requirements
+
+* Big Index (highly recommended): Due to the nature of Big Table data stores, some limitations occur while using Big Record standalone when compared to Active Record. Some major limitations include the inability to query for data other than with the row ID, indexing, searching, and dynamic finders (find_by_attribute_name). Since these data access patterns are vital for most Rails applications to function, Big Index was created to address these issues, and bring the feature set more up to par with Active Record. Please refer to the <tt>Big Index</tt> package for more information and its requirements.
+
+== Getting Started
+
+Check out the doc/getting_started.rdoc file for more information.
+
+== License
+
+Big Record is released under the MIT license.
+
+== Links
+
+* Contact Us
+  * Website - http://www.bigrecord.org
+  * IRC Channel - <tt>#bigrecord</tt> on irc.freenode.net

data/Rakefile

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'rake'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+require 'pathname'
+require 'spec/rake/spectask'
+
+
+DATA_STORES = ["hbase", "cassandra"]
+ROOT = Pathname(__FILE__).dirname.expand_path
+
+require ROOT + 'tasks/gem'
+require ROOT + 'tasks/rdoc'
+require ROOT + 'tasks/spec'
+require ROOT + 'tasks/data_store'

data/VERSION

@@ -0,0 +1 @@
+0.0.5

data/doc/bigrecord_specs.rdoc

@@ -0,0 +1,36 @@
+= Bigrecord Specs
+
+== Data store information
+
+The default settings for the Bigrecord specs can be found at spec/connections/bigrecord.yml with each environment broken down by the data store type (Hbase and Cassandra at the time of writing). These are the minimal settings required to connect to each data store, and should be modified freely to reflect your own system configurations.
+
+== Data store migration
+
+There are also migrations to create the necessary tables for the specs to run. To ensure migrations are functioning properly before actually running the migrations, you can run: spec spec/unit/migration_spec.rb. Alternatively, you can manually create the tables according to the migration files under: spec/lib/migrations
+
+Migrations have their own log file for debugging purposes. It's created under: bigrecord/migrate.log
+
+== Running the specs
+
+After setting it up, you can run the Bigrecord specs with:
+
+  rake spec
+
+or alternatively, with either:
+
+  rake spec:hbase
+  rake spec:cassandra
+
+if you want to run the spec suite against a specific data store.
+
+== Running specific specs
+
+To run a specific spec, you can run the following command from the bigrecord root folder:
+
+  spec -O spec/spec.opts spec/<spec_file>_spec.rb
+
+== Debugging
+
+If any problems or failures arise during the unit tests, please refer to the log files before submitting it as an issue. Often, it's a simple matter of forgetting to turn on BigrecordDriver, the tables weren't created, or configurations weren't set properly.
+
+The log file for specs is created under: <bigrecord root>/spec/debug.log

data/doc/getting_started.rdoc

@@ -0,0 +1,157 @@
+= Getting Started
+
+== Requirements
+
+* Database set up (please refer to the database's own documentation) with the required information known such as host, port, username, password, etc. in order for Bigrecord to connect to it.
+* Bigrecord Driver (if your database requires it for connecting)
+  * JRuby 1.1.6+ is needed to run Bigrecord Driver.
+
+== *NOTE* At the time of this writing, Hbase is the only supported database for Bigrecord, and the following instructions are for specific to Hbase.
+
+== Installation
+
+(1) Install the Bigrecord Driver gem and its dependencies, then start up a DRb server. Please see the Bigrecord Driver documentation for more detailed instructions. (http://github.com/openplaces/bigrecord/blob/master/bigrecord-driver/README.rdoc)
+
+=== Method 1 (recommended): Install as a gem
+
+(2) Add the following line into the Rails::Initializer.run do |config| block:
+
+  config.gem "bigrecord", :source => "http://gemcutter.org"
+
+(3) Bootstrap Bigrecord into your project:
+
+  script/generate bigrecord
+
+(4) Edit the config/bigrecord.yml[.sample] file in your Rails root to the information corresponding to your database install and Bigrecord Driver DRb server.
+
+=== Method 2: Install as a plugin
+
+(2) Run the following command to install it as a plugin:
+
+  script/plugin install git://github.com/openplaces/bigrecord.git
+
+(3) Edit the config/bigrecord.yml[.sample] file in your Rails root to the information corresponding to your database install and Bigrecord Driver DRb server.
+
+== Usage
+
+=== Generators
+
+Once Bigrecord is working in your Rails project, you can use the following generators:
+
+  script/generate bigrecord_model ModelName
+
+This will add a model in app/models and a migration file in db/bigrecord_migrate. Note: This generator does not accept attributes.
+
+  script/generate bigrecord_migration MigrationName
+
+Creates a Bigrecord specific migration and adds it into db/bigrecord_migrate
+
+=== Migration File
+
+Although column-oriented databases are generally schema-less, certain ones (like Hbase) require the creation of tables and column families ahead of time. The individual columns, however, are defined in the model itself and can be modified dynamically without the need for migrations.
+
+Unless you're familiar with column families, the majority of use cases work perfectly fine within one column family. When you generate a bigrecord_model, it will default to creating the :attribute column family.
+
+The following is a standard migration file that creates a table called "Books" with the default column family :attribute that has the following option of 100 versions and uses the 'lzo' compression scheme. Leave any options blank for the default value.
+
+  class CreateBooks < BigRecord::Migration
+    def self.up
+      create_table :books, :force => true do |t|
+        t.family :attribute, :versions => 100, :compression => 'lzo'
+      end
+    end
+
+    def self.down
+      drop_table :books
+    end
+  end
+
+=== HBase column family options (HBase specific)
+
+* versions: integer. By default, Hbase will store 3 versions of changes for any column family. Changing this value on the creation will change this behavior.
+* compression: 'none', 'gz', 'lzo'. Defaults to 'none'. Since Hbase 0.20, column families can be stored using compression. The compression scheme you define here must be installed on the Hbase servers!
+
+=== Migrating
+
+Run the following rake task to migrate your tables and column families up to the latest version:
+
+  rake bigrecord:migrate
+
+=== Column and Attribute Definition
+
+Now that you have your tables and column families all set up, you can begin adding columns to your model. The following is an example of a model named book.rb
+
+  class Book < BigRecord::Base
+    column 'attribute:title',   :string
+    column :author,             :string
+    column :description,        :string
+    column :links,              :string,  :collection => true
+  end
+
+This simple model defines 4 columns of type string. An important thing to notice here is that the first column 'attribute:title' had the column family prepended to it. This is identical to just passing the symbol :title to the column method, and the default behaviour is to prepend the column family (attribute) automatically if one is not defined. Furthermore, in Hbase, there's the option of storing collections for a given column. This will return an array for the links attribute on a Book record.
+
+=== Associations
+
+There are also associations available in Bigrecord, as well as the ability to associate to Activerecord models. The following are a few models demonstrating this:
+
+animal.rb
+  class Animal < BigRecord::Base
+    # Typical attributes
+    column :name,         :string
+    column :type,         :integer
+    column :description,  :string
+
+    # Association attributes
+    column :zoo_id,       :string
+    column :trainer_id,   :integer
+
+    # Associations
+    belongs_to_big_record :zoo, :foreign_key => :zoo_id
+    belongs_to :trainer,        :foreign_key => :trainer_id
+  end
+
+In this example, an Animal is related to Zoo and Trainer. Both Animal and Zoo are Bigrecord models, and Trainer is an Activerecord model. Notice here that we need to define both the association field for storing the information and the association itself. It's also important to remember that Bigrecord models have their IDs stored as string, and Activerecord models use integers.
+
+Once the association columns are defined, you define the associations themselves with either belongs_to_bigrecord or belongs_to_many and defining the :foreign_key (this is required for all associations).
+
+=== Specifying return columns
+
+There are two ways to define specific columns to be returned with your models: 1) at the model level and 2) during the query.
+
+(1) At the model level, a collection of columns are called named views, and are defined like the following:
+
+  class Book < BigRecord::Base
+    column :title,   :string
+    column :author,             :string
+    column :description,        :string
+    column :links,              :string,  :collection => true
+
+    view :front_page, :title, :author
+    view :summary_page, :title, :author, :description
+
+    # Override default if you don't want all columns returend
+    view :default, :title, :author, :description
+  end
+
+Now, whenever you work with a Book record, it will only returned the columns you specify according to the view option you pass. i.e.
+
+  >> Book.find(:first, :view => :front_page)
+  => #<Book id: "2e13f182-1085-495e-9841-fe5c84ae9992", attribute:title: "Hello Thar", attribute:author: "Greg">
+
+  >> Book.find(:first, :view => :summary_page)
+  => #<Book id: "2e13f182-1085-495e-9841-fe5c84ae9992", attribute:description: "Masterpiece!", attribute:title: "Hello Thar", attribute:author: "Greg">
+
+  >> Book.find(:first, :view => :default)
+  => #<Book id: "2e13f182-1085-495e-9841-fe5c84ae9992", attribute:description: "Masterpiece!", attribute:title: "Hello Thar", attribute:links: ["link1", "link2", "link3", "link4"], attribute:author: "Greg">
+
+Note: A Bigrecord model will return all the columns within the default column family (when :view option is left blank, for example). You can override the :default name view to change this behaviour.
+
+
+(2) If you don't want to define named views ahead of time, you can just pass an array of columns to the :columns option and it will return only those attributes:
+
+  >> Book.find(:first, :columns => [:author, :description])
+  => #<Book id: "2e13f182-1085-495e-9841-fe5c84ae9992", attribute:description: "Masterpiece!", attribute:author: "Greg">
+
+As you may have noticed, this functionality is synonymous with the :select option in Activerecord.
+
+=== At this point, usage patterns for a Bigrecord model are similar to that of an Activerecord model, and much of that documentation applies as well. Please refer to those and see if they work!

data/examples/bigrecord.yml

@@ -0,0 +1,25 @@
+development:
+  adapter: hbase
+  zookeeper_quorum: localhost
+  zookeeper_client_port: 2181
+  drb_host: localhost
+  drb_port: 40000
+test:
+  adapter: hbase
+  zookeeper_quorum: localhost
+  zookeeper_client_port: 2182
+  drb_host: localhost
+  drb_port: 40001
+production:
+  adapter: hbase
+  zookeeper_quorum: localhost
+  zookeeper_client_port: 2183
+  drb_host: localhost
+  drb_port: 40002
+
+cassandra:
+  adapter: cassandra
+  address: localhost
+  port: 9160
+  drb_host: localhost
+  drb_port: 40003

data/generators/bigrecord/bigrecord_generator.rb

@@ -0,0 +1,17 @@
+# This generator bootstraps a Rails project for use with Bigrecord
+class BigrecordGenerator < Rails::Generator::Base
+ 
+  def initialize(runtime_args, runtime_options = {})
+    require File.join(File.dirname(__FILE__), "..", "..", "install.rb")
+    Dir.mkdir('lib/tasks') unless File.directory?('lib/tasks')
+    super
+  end
+ 
+  def manifest
+    record do |m|
+      m.directory 'lib/tasks'
+      m.file 'bigrecord.rake', 'lib/tasks/bigrecord.rake'
+    end
+  end
+ 
+end

data/generators/bigrecord/templates/bigrecord.rake

@@ -0,0 +1,47 @@
+namespace :bigrecord do
+
+  desc "Migrate the Bigrecord database through scripts in db/bigrecord_migrate. Target specific version with VERSION=x. Turn off output with VERBOSE=false."
+  task :migrate => :environment do
+    BigRecord::Migrator.migrate("db/bigrecord_migrate/", ENV["VERSION"] ? ENV["VERSION"].to_i : nil)
+  end
+
+  namespace :migrate do
+    desc 'Rollbacks the database one migration and re migrate up. If you want to rollback more than one step, define STEP=x. Target specific version with VERSION=x.'
+    task :redo => :environment do
+      if ENV["VERSION"]
+        Rake::Task["bigrecord:migrate:down"].invoke
+        Rake::Task["bigrecord:migrate:up"].invoke
+      else
+        Rake::Task["bigrecord:rollback"].invoke
+        Rake::Task["bigrecord:migrate"].invoke
+      end
+    end
+
+    desc 'Runs the "up" for a given migration VERSION.'
+    task :up => :environment do
+      version = ENV["VERSION"] ? ENV["VERSION"].to_i : nil
+      raise "VERSION is required" unless version
+      BigRecord::Migrator.run(:up, "db/bigrecord_migrate/", version)
+    end
+
+    desc 'Runs the "down" for a given migration VERSION.'
+    task :down => :environment do
+      version = ENV["VERSION"] ? ENV["VERSION"].to_i : nil
+      raise "VERSION is required" unless version
+      BigRecord::Migrator.run(:up, "db/bigrecord_migrate/", version)
+    end
+  end
+
+  desc 'Rolls the schema back to the previous version. Specify the number of steps with STEP=n'
+  task :rollback => :environment do
+    step = ENV['STEP'] ? ENV['STEP'].to_i : 1
+    BigRecord::Migrator.rollback('db/bigrecord_migrate/', step)
+  end
+
+  desc 'Pushes the schema to the next version. Specify the number of steps with STEP=n'
+  task :forward => :environment do
+    step = ENV['STEP'] ? ENV['STEP'].to_i : 1
+    BigRecord::Migrator.forward('db/bigrecord_migrate/', step)
+  end
+
+end

data/generators/bigrecord_migration/bigrecord_migration_generator.rb

@@ -0,0 +1,13 @@
+require 'rails_generator/generators/components/migration/migration_generator'
+
+class BigrecordMigrationGenerator < MigrationGenerator
+
+  def manifest
+
+    record do |m|
+      m.migration_template 'migration.rb', 'db/bigrecord_migrate', :assigns => get_local_assigns
+    end
+
+  end
+
+end

data/generators/bigrecord_migration/templates/migration.rb

@@ -0,0 +1,9 @@
+class <%= class_name.underscore.camelize %> < BigRecord::Migration
+
+  def self.up
+  end
+
+  def self.down
+  end
+
+end

data/generators/bigrecord_model/bigrecord_model_generator.rb

@@ -0,0 +1,28 @@
+require 'rails_generator/generators/components/model/model_generator'
+
+class BigrecordModelGenerator < ModelGenerator
+
+  def manifest
+
+    record do |m|
+      # Check for class naming collisions.
+      m.class_collisions class_path, class_name
+
+      # Model, spec, and fixture directories.
+      m.directory File.join('app/models', class_path)
+      m.directory File.join('spec/models', class_path)
+
+      # Model class, spec and fixtures.
+      m.template 'model.rb',      File.join('app/models', class_path, "#{file_name}.rb")
+      m.template 'model_spec.rb', File.join('spec/models', class_path, "#{file_name}_spec.rb")
+
+      unless options[:skip_migration]
+        m.migration_template 'migration.rb', 'db/bigrecord_migrate', :assigns => {
+          :migration_name => "Create#{class_name.pluralize.gsub(/::/, '')}"
+        }, :migration_file_name => "create_#{file_path.gsub(/\//, '_').pluralize}"
+      end
+    end
+
+  end
+
+end

data/generators/bigrecord_model/templates/migration.rb

@@ -0,0 +1,13 @@
+class <%= migration_name %> < BigRecord::Migration
+
+  def self.up
+    create_table :<%= table_name %>, :force => true do |t|
+      t.family :attribute
+    end
+  end
+
+  def self.down
+    drop_table :<%= table_name %>
+  end
+
+end

data/generators/bigrecord_model/templates/model.rb

@@ -0,0 +1,7 @@
+class <%= class_name %> < BigRecord::Base
+
+<% attributes.each do |attribute| -%>
+  column :<%= attribute.name %>,<%= " " * (20 - attribute.name.length) %>:<%= attribute.type %>
+<% end -%>
+
+end

data/generators/bigrecord_model/templates/model_spec.rb

@@ -0,0 +1,12 @@
+require File.expand_path(File.dirname(__FILE__) + '<%= '/..' * class_nesting_depth %>/../spec_helper')
+
+describe <%= class_name %> do
+  before(:each) do
+    @valid_attributes = {
+    }
+  end
+
+  it "should create a new instance given valid attributes" do
+    <%= class_name %>.create!(@valid_attributes)
+  end
+end

data/init.rb

@@ -0,0 +1,9 @@
+require 'big_record'
+
+# Use the same logger as ActiveRecord to make sure that the access to the log file is properly handled
+BigRecord::Base.logger = ActiveRecord::Base.logger
+BigRecord::Embedded.logger = ActiveRecord::Base.logger
+
+# Establish the connection with the database
+BigRecord::Base.configurations = YAML::load(File.open("#{RAILS_ROOT}/config/bigrecord.yml"))
+BigRecord::Base.establish_connection

data/install.rb

@@ -0,0 +1,22 @@
+require 'fileutils'
+
+puts "[Bigrecord] Copying example config file to your RAILS_ROOT...\n"
+
+config_dir = File.join(RAILS_ROOT, "config")
+source = File.join(File.dirname(__FILE__), "examples", "bigrecord.yml")
+target = File.join(config_dir, "bigrecord.yml")
+alternate_target = File.join(config_dir, "bigrecord.yml.sample")
+
+migration_dir = File.join(RAILS_ROOT, "db", "bigrecord_migrate")
+
+if !File.exist?(target)
+  FileUtils.cp(source, target)
+else
+  puts "[Bigrecord] RAILS_ROOT/config/bigrecord.yml file already exists. Copying it as bigrecord.yml.sample for reference."
+  FileUtils.cp(source, alternate_target)
+end
+
+unless File.exist?(migration_dir)
+  puts "[Bigrecord] Migration folder not found at \"#{migration_dir}\" Creating now..."
+  FileUtils.mkdir_p(migration_dir)
+end