databasion 0.0.1

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+ 
+Copyright (c) 2010 Brian Jones (Istpika)
+ 
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+ 
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,144 @@
+# Databasion
+
+## Google Spreadsheet/Excel -> YAML -> Ruby Migration -> Database Management Tool
+
+If we were all part of a hive mind, we wouldn't need management anything.  Databases would get built, the correct columns would get used, programmers would align their ORMs, and keeping it all together wouldn't be some kind of management nightmare.
+
+Fortunately we aren't a hive mind (and if we were I would be just as confused as I am any other day, our office's main language is Japanese).  Still, even though my coworkers can't read my mind, we've come up with a tool that allows top end planners to describe system data in a spreadsheet, programmers to fudge in the column types, sizes, and database relationships, and everyone to export it out into YAML and Ruby migration scripts which update your infrastructure.  Ideally the fully automated suite is used in your test environment so your planner can quickly test changes, and a step by step process used if you are updating production machines.
+
+## Requirements
+
+### Ruby
+
+* Ruby >= 1.8.7
+
+### Gems  
+
+* ActiveRecord >= 2.3.5
+* ActiveSupport >= 2.3.5
+* Google Spreadsheet >= 0.1.1
+* Spreadsheet >= 0.6.4.1
+
+## Installation
+
+### Install
+
+    sudo gem install databasion
+    
+## Spreadsheet Conventions
+
+None of this would really work if there weren't some conventions in place.  The following explains how the worksheet needs to be formatted, how the data spreadsheets themselves needs to be formatted, what fields are required, and what fields can be ignored.
+
+At the highest level there needs to be a worksheet named _Database_.  This is simply a master list of related spreadsheets, and what database they correspond to (for split table designs).  The column names are required.
+
+### Database
+
+| spreadsheet | dbname| database| username| password| adapter| host     | port| options
+|:------------|:------|:--------|:--------|:--------|:-------|:---------|:----|:-------
+| superheroes | db1   | db_test | dbuser  | dbuser  | mysql  | 127.0.0.1|     |        
+
+The options column currently support's _force_, which tells the database to drop and recreate the table.
+
+Next we define the actual table spreadsheets.
+
+### Superheroes
+
+| column0  |             |              |                    
+|:---------|:------------|:-------------|:-------------------
+| ignore   |             |              |                    
+| comment  |             |              |                    
+| table    | superheroes |              |                    
+| field    | id          | name         | power              
+| type     | integer     | string, 20   | string, 20, Wimp   
+|          | 1           | Brian Jones  | Ruby Hacker        
+|          | 2           | Superman     | Invincible         
+|          | 3           | Batman       | Rich               
+
+### Keywords
+
+* ignore - Anything written in this column will cause this column and it's data to be ignored.
+* table - The name of the table, and an optional comma delimited 'false' if the table name should not be auto-pluralized.
+* field - The name of the table column.
+* type  - A comma delimited list giving the type of the column (using Ruby migration terms), optional size, and optional default value.
+
+Note: If an 'id' column is specified, then it is assumed the id's are supplied by hand.  Auto-incrementation is disabled, and 'id' is the primary key.
+
+__Ruby Migration Types__
+
+* binary
+* boolean
+* date
+* datetime
+* decimal
+* float
+* integer
+* string
+* text
+* time
+* timestamp
+
+### Columns
+
+Currently column0 is reserved for keywords and comments.
+
+If something besides a keyword is written in column0, that row is ignored and will not be used.  This is useful if you need to edit out some data.
+
+### Rows
+
+Row0 isn't technically reserved, but should ideally be saved for use with the _ignore_ flag.  If any text is written in a column (with the exception of column0), that column will be ignored.  This is useful for editing out columns that one doesn't currently want in the database.
+
+## Usage
+    
+Setup the project space.    
+    
+    databasion --create project
+    cd project
+
+Edit _config/google.yml_.  Then run the scripts.
+
+    databasion --system google
+    databasion --migrate
+    
+Or run them both in order.
+
+    databasion --system google --migrate
+    
+You can supply a different config path as well.
+
+    databasion -s google -m --config config/my.other.config.yml
+    
+### YAML Configuration
+
+#### Google
+
+* _login_: A valid Google username and password.
+* _sheets_: A list of the keys gleaned from the Google Docs URL, and a human readable name.
+* _output_: Where to output the relevant data.
+
+## Roadmap
+
+__0.0.2__
+
+* Add ability to read existing tables, and make relative alter table migration scripts.
+
+__0.0.1__
+
+* <del>Write this documentation.</del>
+* <del>Add table name specification.</del>
+* <del>Add database level table relationships.</del>
+* <del>Spit out ActiveRecord Models for migrations.</del>
+* <del>Create Ruby migration script.</del> <del>Still needs polished.</del>
+* <del>Add ability to read and update existing data.</del>
+* <del>Build a command line tool.</del>
+* <del>Add logging.</del>
+
+## Testing
+
+Currently Databasion uses the cucumber test suite.  Any patches or pull requests must have a corresponding Feature, and all tests must pass.  Feature branches get bonus points.
+
+## Author
+
+__Brian Jones__ - Server Engineer, [Istpika](http://www.istpika.com)
+
+* Work: <brian.jones@istpika.com>
+* Personal: <mojobojo@gmail.com>

data/Rakefile

@@ -0,0 +1,23 @@
+require 'rubygems'
+require 'rake'
+require 'jeweler'
+
+Jeweler::Tasks.new do |gem|
+  gem.name        = "databasion"
+  gem.summary     = %Q{A Google Spreadsheet/Excel -> YAML -> Ruby Migration Tool}
+  gem.email       = "mojobojo@gmail.com"
+  gem.homepage    = "http://github.com/boj/databasion"
+  gem.authors     = ["Brian Jones", "Istpika"]
+  gem.version     = "0.0.1"
+
+  gem.add_dependency('activerecord', '>= 2.3.5')
+  gem.add_dependency('activesupport', '>= 2.3.5')
+  gem.add_dependency('google-spreadsheet-ruby', '>= 0.1.1')
+  gem.add_dependency('spreadsheet', '>= 0.6.4.1')
+end
+
+Jeweler::GemcutterTasks.new
+
+Dir['lib/tasks/**/*.rake'].each { |rake| load rake }
+
+task :default  => :test

data/bin/databasion

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__) + '/../lib') unless $:.include?(File.dirname(__FILE__) + '/../lib')
+
+require 'databasion'
+
+Databasion::Applcize.run

data/config/example.google.yml

@@ -0,0 +1,13 @@
+login:
+  username: fakeuser
+  password: fakepass
+  
+sheets:  
+  - name: non_existant_spreadsheet
+    key: key_google_doc_key
+    
+output:
+  yaml_path: 'data'
+  migrations:
+    path: 'migrations'
+    models: 'models'

data/databasion.gemspec

@@ -0,0 +1,78 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{databasion}
+  s.version = "0.0.1"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Brian Jones", "Istpika"]
+  s.date = %q{2010-05-19}
+  s.default_executable = %q{databasion}
+  s.email = %q{mojobojo@gmail.com}
+  s.executables = ["databasion"]
+  s.extra_rdoc_files = [
+    "LICENSE",
+     "README.md"
+  ]
+  s.files = [
+    "LICENSE",
+     "README.md",
+     "Rakefile",
+     "bin/databasion",
+     "config/example.google.yml",
+     "databasion.gemspec",
+     "features/databasion.feature",
+     "features/databasion/googlize.feature",
+     "features/databasion/migitize.feature",
+     "features/databasion/step_definitions/googlize_steps.rb",
+     "features/databasion/step_definitions/migitize_steps.rb",
+     "features/databasion/step_definitions/yamalize_steps.rb",
+     "features/databasion/yamalize.feature",
+     "features/step_definitions/databasion_steps.rb",
+     "lib/databasion.rb",
+     "lib/databasion/applcize.rb",
+     "lib/databasion/csvilize.rb",
+     "lib/databasion/excelize.rb",
+     "lib/databasion/googlize.rb",
+     "lib/databasion/migitize.rb",
+     "lib/databasion/yamalize.rb",
+     "lib/migration_helpers/MIT-LICENSE",
+     "lib/migration_helpers/README.markdown",
+     "lib/migration_helpers/init.rb",
+     "lib/migration_helpers/lib/migration_helper.rb",
+     "lib/tasks/databasion.rake",
+     "lib/tasks/test.rake",
+     "lib/trollop.rb"
+  ]
+  s.homepage = %q{http://github.com/boj/databasion}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{A Google Spreadsheet/Excel -> YAML -> Ruby Migration Tool}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<activerecord>, [">= 2.3.5"])
+      s.add_runtime_dependency(%q<activesupport>, [">= 2.3.5"])
+      s.add_runtime_dependency(%q<google-spreadsheet-ruby>, [">= 0.1.1"])
+      s.add_runtime_dependency(%q<spreadsheet>, [">= 0.6.4.1"])
+    else
+      s.add_dependency(%q<activerecord>, [">= 2.3.5"])
+      s.add_dependency(%q<activesupport>, [">= 2.3.5"])
+      s.add_dependency(%q<google-spreadsheet-ruby>, [">= 0.1.1"])
+      s.add_dependency(%q<spreadsheet>, [">= 0.6.4.1"])
+    end
+  else
+    s.add_dependency(%q<activerecord>, [">= 2.3.5"])
+    s.add_dependency(%q<activesupport>, [">= 2.3.5"])
+    s.add_dependency(%q<google-spreadsheet-ruby>, [">= 0.1.1"])
+    s.add_dependency(%q<spreadsheet>, [">= 0.6.4.1"])
+  end
+end
+

data/features/databasion/googlize.feature

@@ -0,0 +1,13 @@
+Feature: Use Googlize
+  In order for a user to access google
+  They need to connect first
+    
+  Scenario: Log into Google
+    Given a google account
+    When Googlize logs in
+    Then we should have a GoogleSpreadsheet session
+    
+  Scenario: Run system with config
+    Given a google yaml config
+    When Googlize starts up
+    Then Googlize should not fail

data/features/databasion/migitize.feature

@@ -0,0 +1,8 @@
+Feature: Parse YAML into Ruby migrations
+  In order to run Ruby migrations
+  We must build the migrations from YAML
+  
+  Scenario: Build Ruby migrations
+    Given a complete set of YAML definitions
+    When the YAML files are parsed
+    Then the result is Ruby migration files

data/features/databasion/step_definitions/googlize_steps.rb

@@ -0,0 +1,32 @@
+require 'lib/databasion'
+
+Given /a google account/ do
+  @config = YAML.load(File.open('config/google.yml'))
+end
+
+When /Googlize logs in/ do
+  Databasion::Googlize.config = @config
+  Databasion::Googlize.login
+end
+
+Then /we should have a GoogleSpreadsheet session/ do
+  Databasion::Googlize.session.should be_kind_of(GoogleSpreadsheet::Session)
+end
+
+Given /a google yaml config/ do
+  @config = YAML.load(File.open('config/google.yml'))
+end
+
+When /Googlize starts up/ do
+  Databasion::Googlize.config = @config
+  begin
+    Databasion::Googlize.googlebate
+    @error = false
+  rescue
+    @error = true
+  end
+end
+
+Then /Googlize should not fail/ do
+  @error.should == false
+end

data/features/databasion/step_definitions/migitize_steps.rb

@@ -0,0 +1,26 @@
+require 'lib/databasion'
+
+Given /a complete set of YAML definitions/ do
+  @config = YAML.load(File.open('config/google.yml'))
+
+  @parse_data = {
+    'name'        => 'mock',
+    'plural'      => true,
+    'fields'      => ["id", "name", "power"],
+    'types'       => ["integer", "string, 20", "string, 40"],
+    'data'        => [[1, "Brian Jones", "Super Intelligence"], [2, "Superman", "Invincible"], [3, "Batman", "Strength"]],
+    'ignore_cols' => [2],
+    'connection'  => [{'database' => 'moon'}]
+  }
+  Databasion::Yamalize.yamlbate(@parse_data, @config['output']['yaml_path'])
+
+  @files = Dir["%s/%s.yml" % [@config['output']['yaml_path'], @parse_data['name']]]
+end
+
+When /the YAML files are parsed/ do
+  Databasion::Migitize.migrabate(@files, @config)
+end
+
+Then /the result is Ruby migration files/ do
+  File.exist?("%s/100_%s.rb" % [@config['output']['migrations']['path'], @parse_data['name']]).should == true
+end

data/features/databasion/step_definitions/yamalize_steps.rb

@@ -0,0 +1,29 @@
+require 'lib/databasion'
+
+Given /a chunk of (.*) data/ do |name|
+  @config = YAML.load(File.open('config/google.yml'))
+  @parse_data = {
+    'name'        => 'mock',
+    'plural'      => true,
+    'fields'      => ["id", "name", "power"],
+    'types'       => ["integer", "string, 20", "string, 40"],
+    'data'        => [[1, "Brian Jones", "Super Intelligence"], [2, "Superman", "Invincible"], [3, "Batman", "Strength"]],
+    'ignore_cols' => [2],
+    'connection'  => [{'database' => 'moon'}]
+  }
+end
+
+When /we parse it/ do
+  Databasion::Yamalize.yamlbate(@parse_data, @config['output']['yaml_path'])
+end
+
+Then /it should create a relevant YAML file/ do
+  File.exist?("%s/%s.yml" % [@config['output']['yaml_path'], @parse_data['name']]).should == true
+end
+
+And /should contain the correct data/ do
+  data = YAML.load(File.open("%s/%s.yml" % [@config['output']['yaml_path'], @parse_data['name']]))
+  data.should include('meta')
+  data.should include('data')
+  data['meta'].should include('connection')
+end

data/features/databasion/yamalize.feature

@@ -0,0 +1,9 @@
+Feature: Convert parsed hash into YAML
+  In order to convert the higher level into YAML
+  We need to process it and write it out to files
+  
+  Scenario: Google passes data
+    Given a chunk of google data
+    When we parse it
+    Then it should create a relevant YAML file
+    And should contain the correct data

data/features/databasion.feature

@@ -0,0 +1,23 @@
+Feature: Run System
+  In order for the user to choose what they want to do
+  The user should be able to pass in configuration settings
+  
+  Scenario: User calls google databate with config
+    Given there is an actual yaml config file for google
+    When the user runs databate for google
+    Then Databasion should not fail for google
+    
+  Scenario: User calls google databate without config
+    Given there is no config file for google
+    When the user calls databate without config for google
+    Then Databasion should fail for google
+    
+  Scenario: User calls excel databate with config
+    Given there is an actual yaml config file for excel
+    When the user runs databate for excel
+    Then Databasion should not fail for excel
+
+  Scenario: User calls excel databate without config
+    Given there is no config file for excel
+    When the user calls databate without config for excel
+    Then Databasion should fail for excel

data/features/step_definitions/databasion_steps.rb

@@ -0,0 +1,35 @@
+require 'lib/databasion'
+
+Given /there is an actual yaml config file for (.*)/ do |name|
+  @config = 'config/google.yml'
+end
+
+When /the user runs databate for (.*)/ do |name|
+  begin
+    Databasion.databate(name, @config)
+    @error = false
+  rescue
+    @error = true
+  end
+end
+
+Then /Databasion should not fail for (.*)/ do |name|
+  @error.should == false
+end
+
+Given /there is no config file for (.*)/ do |name|
+  @config = nil
+end
+
+When /the user calls databate without config for (.*)/ do |name|
+  begin
+    Databasion.databate(name, @config)
+    @error = false
+  rescue
+    @error = true
+  end
+end
+
+Then /Databasion should fail for (.*)/ do |name|
+  @error.should == true
+end

data/lib/databasion/applcize.rb

@@ -0,0 +1,86 @@
+require 'trollop'
+
+module Databasion
+  
+  class Applcize
+    
+    def self.run
+      opts = Trollop::options do
+        banner <<-EOS
+      Databasion - A Google Spreadsheet/Excel -> YAML -> Ruby Migration Database Tool
+
+      Usage:
+          databasion [options]
+      where [options] are:
+        EOS
+        opt :create, "Create a base deploy directory", :type => String
+        opt :config, "Path to YAML config.  Looks for config/google.yml by default", :type => String
+        opt :system, "google, excel, migrate, update", :type => String
+        opt :migrate, "Migrate after Googlizing or Excelizing"
+        opt :update, "Load parsed YAML into migrated database"
+      end
+      if opts[:config].nil? and opts[:create].nil?
+        config = "config/google.yml"
+        puts Dir.pwd
+        if File.exist?(Dir.pwd + "/" + config)
+          opts[:config] = config
+        else  
+          Trollop::die :config, "A YAML config must be specified"
+        end
+      end
+      Trollop::die :system, "System requires a parameter" if opts[:system].nil? and opts[:create].nil?
+
+      if opts[:create]
+        create_project(opts)
+      else
+        execute_databasion(opts)
+      end
+    end
+    
+    def self.execute_databasion(opts)
+      if opts[:system] and opts[:config]
+        Databasion.databate(opts[:system], opts[:config])
+        if opts[:migrate] and opts[:system] != 'migrate'
+          Databasion.databate('migrate', opts[:config])
+        end
+        if opts[:update] and opts[:system] != 'update'
+          Databasion.databate('update', opts[:config])
+        end
+      end
+    end
+    
+    def self.create_project(opts)
+      dir = Dir.pwd
+      if File.exist?(dir + "/" + opts[:create])
+        Databasion::LOGGER.info "A directory with the name %s already exists" % opts[:create]
+      else
+        Databasion::LOGGER.info "Creating new project directory..."
+        create_base(dir, opts[:create])
+        create_config(dir, opts[:create])
+        copy_config(dir, opts[:create])
+        Databasion::LOGGER.info "Done."
+      end
+    end
+    
+    def self.create_base(dir, name)
+      path = dir + "/" + name
+      FileUtils.mkdir path
+      Databasion::LOGGER.info "created: %s" % path
+    end
+    
+    def self.create_config(dir, name)
+      path = dir + "/" + "%s/config" % name
+      FileUtils.mkdir path
+      Databasion::LOGGER.info "created: %s" % path
+    end
+    
+    def self.copy_config(dir, name)
+      base = File.dirname(File.expand_path(__FILE__)) + "/../../" + "config/example.google.yml"
+      path = dir + "/" + "%s/config/google.yml" % name
+      FileUtils.cp base, path
+      Databasion::LOGGER.info "copied: %s" % path
+    end
+    
+  end
+  
+end

data/lib/databasion/csvilize.rb

@@ -0,0 +1,11 @@
+module Databasion
+  
+  class Csvilize
+    
+    def self.csvbate
+      
+    end
+    
+  end
+  
+end

data/lib/databasion/excelize.rb

@@ -0,0 +1,11 @@
+module Databasion
+  
+  class Excelize
+    
+    def self.excelbate
+      
+    end
+    
+  end
+  
+end