mlins-godwit 1.0.0

data/CHANGELOG

@@ -0,0 +1,3 @@
+*1.0.0 (November 23, 2008)*
+
+* Initial Release

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Matt Lins
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,110 @@
+= Godwit
+
+Github[http://github.com/mlins/godwit]
+
+Godwit is a framework for the ActiveMigration[http://github.com/mlins/active_migration] library.  It provides
+a directory structure and scripts to bring ease to a migration.  Godwit is a console application.
+
+This should not be confused with Rails migrations.  Godwit and
+ActiveMigration[http://github.com/mlins/active_migration] are used for migrating data from a legacy dataset.
+As of now it relys on ActiveRecord.  You must define models for both your legacy data and your active data.
+However, Godwit can load a Rails environment so you don't have to redefine or sync your models.
+
+Godwit and ActiveMigration[http://github.com/mlins/active_migration] were written to maintain data integrity
+during a migration.  Speed was of little concern.  So, this is not an ideal solution for people who are just
+looking for raw dumps and/or imports.  I designed this to use all of the validations, callbacks and plugins
+from my new(active) application.  See ActiveMigration[http://github.com/mlins/active_migration] for more details.
+
+*Features*
+
+- Console access if a migration fails (helpful for debugging).
+- Extendable via plugins (similar to Rails).  You can use some Rails/ActiveRecord plugins in Godwit.
+- Loads Rails environment to prevent having to write models twice.
+
+Godwit was written by: Matt Lins.
+
+== Installation
+
+  $ gem sources -a http://gems.github.com
+
+  $ sudo gem install mlins-godwit
+
+== Usage
+
+You'll first want to create a base application with the Godwit binary:
+
+  $ godwit myapp
+
+If you plan on migrating to a database that is used by a Rails application, you can load the Rails environment
+so you don't have to redefine your models.  Inside of *config/environment.rb* set the path to the root of the
+Rails app:
+
+  config[:rails_root] = '/my/new/rails/app'
+
+You can than test that the environment was loaded properly by starting a console session within Godwit:
+
+  $ ./script/console
+
+If you're not using a Rails application then you can define your models in *app/models*.  There is a generator
+to help with this:
+
+  $ ./script/generate active_model my_model
+
+Next, you'll need to define some models (ActiveRecord) for your legacy tables.  Godwit namespaces legacy models
+with the Legacy module.  They live in the *app/models/legacy* directory.  There is a generator for this:
+
+  $ ./script/generate legacy_model my_model
+
+When you have all your models defined and tested, you can start creating migrations.  They live in *app/migrations*.
+See ActiveMigration[http://github.com/mlins/active_migration] for more details.  There is a generator to create
+migrations:
+
+  $ ./script/generate active_migration my_migration
+
+When you're actually ready to run your migrations, you can use the *migrate* script.  This script has a few
+options, so run this for help:
+
+  $ ./script/migrate -h
+
+If you run *migrate* without any options, it'll run all of your migrations.  It will respect any dependencies
+defined in your migrations.
+
+If you need to debug a migration, Godwit provides a handy IRB session if #save fails during the migration
+of a specific model.  You can inspect the entire binding of the migration.  Use the -D option to run in debug mode.
+
+  $ ./script/migrate -D
+
+Once you drop to a IRB session you can inspect @active_record or @legacy_record.  Ideally you want to get
+@active_record #valid? before you're done.  When you're done, you can just type 'exit' and the migration will continue.
+If you failed to resolve the issue, Godwit will continue to drop back to the IRB session.  You can type 'skip' and
+then 'exit' to skip this particular record.
+
+== Requirements
+
+- ActiveSupport
+- ActiveMigration
+
+== License
+
+(The MIT License)
+
+Copyright (c) 2008 Matt Lins
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,37 @@
+require 'rake'
+require 'rake/rdoctask'
+require 'spec/rake/spectask'
+
+desc 'Run the specs'
+Spec::Rake::SpecTask.new(:spec) do |t|
+  t.spec_opts = ['--colour --format progress --loadby mtime --reverse']
+  t.spec_files = FileList['spec/**/*_spec.rb']
+end
+
+Rake::RDocTask.new do |t|
+  t.rdoc_dir = 'doc'
+  t.rdoc_files.include('README')
+  t.rdoc_files.include('lib/**/*.rb')
+  t.options << '--inline-source'
+  t.options << '--all'
+  t.options << '--line-numbers'
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "godwit"
+    s.summary = "A framework to assist in the migration of data from legacy databases."
+    s.email = "mattlins@gmail.com"
+    s.homepage = "http://github.com/mlins/godwit"
+    s.description = "A framework to assist in the migration of data from legacy databases."
+    s.authors = ["Matt Lins"]
+    s.files = FileList["[A-Z]*", "{lib,spec,app_generators,godwit_generators,tmp}/**/*"]
+    s.executables = ['godwit']
+    s.add_dependency 'activesupport', '>= 2.1.0'
+    s.add_dependency 'mlins-active_migration', '>= 1.0.4'
+    s.add_dependency 'rubigen', '>= 1.3.4'
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end

data/VERSION.yml

@@ -0,0 +1,4 @@
+---
+major: 1
+patch: 0
+minor: 0

data/app_generators/godwit/godwit_generator.rb

@@ -0,0 +1,91 @@
+class GodwitGenerator < RubiGen::Base
+  
+  DEFAULT_SHEBANG = File.join(Config::CONFIG['bindir'],
+                              Config::CONFIG['ruby_install_name'])
+  
+  default_options :author => nil
+  
+  attr_reader :name
+  
+  def initialize(runtime_args, runtime_options = {})
+    super
+    usage if args.empty?
+    @destination_root = File.expand_path(args.shift)
+    @name = base_name
+    extract_options
+  end
+
+  def manifest
+    record do |m|
+      # Ensure appropriate folder(s) exists
+      m.directory ''
+      BASEDIRS.each { |path| m.directory path }
+
+      script_options     = { :chmod => 0755, :shebang => options[:shebang] == DEFAULT_SHEBANG ? nil : options[:shebang] }
+
+      # Create stubs
+      # m.template "template.rb",  "some_file_after_erb.rb"
+      # m.file     "file",         "some_file_copied"
+      
+      # Config
+      %w(environment.rb database.yml).each do |file|
+        m.file "config/#{file}", "config/#{file}"
+      end
+      
+      # Scripts
+      %w(console migrate).each do |file|
+        m.file "script/#{file}", "script/#{file}", script_options
+      end
+      
+      # Script Helper
+      m.file "script/lib/console.rb", "script/lib/console.rb"
+      
+      # Blank Log
+      m.file "log/active_migration.log", "log/active_migration.log"
+      m.file "log/active_record.log", "log/active_record.log"
+      
+      m.file "Rakefile", "Rakefile"
+      
+      m.dependency "install_rubigen_scripts", [destination_root, 'godwit'], 
+        :shebang => options[:shebang], :collision => :force
+    end
+  end
+
+  protected
+    def banner
+      'USAGE: godwit [application_name]'
+    end
+
+    def add_options!(opts)
+      opts.separator ''
+      opts.separator 'Options:'
+      # For each option below, place the default
+      # at the top of the file next to "default_options"
+      # opts.on("-a", "--author=\"Your Name\"", String,
+      #         "Some comment about this option",
+      #         "Default: none") { |options[:author]| }
+      opts.on("-v", "--version", "Show the #{File.basename($0)} version number and quit.")
+    end
+    
+    def extract_options
+      # for each option, extract it into a local variable (and create an "attr_reader :author" at the top)
+      # Templates can access these value via the attr_reader-generated methods, but not the
+      # raw instance variable value.
+      # @author = options[:author]
+    end
+
+    # Installation skeleton.  Intermediate directories are automatically
+    # created so don't sweat their absence here.
+    BASEDIRS = %w(
+      lib
+      vendor
+      vendor/plugins
+      data
+      log
+      script/lib
+      tmp
+      config
+      app/models/legacy
+      app/migrations
+    )
+end

data/app_generators/godwit/templates/Rakefile

@@ -0,0 +1,8 @@
+require 'godwit'
+
+Godwit::Config.setup
+Godwit::Config[:silence] = true
+Godwit::Base.new
+
+Dir["#{Godwit::Config[:godwit_root]}/vendor/plugins/*/**/tasks/**/*.rake"].sort.each { |ext| load ext }
+Dir["#{Godwit::Config[:godwit_root]}/lib/tasks/**/*.rake"].sort.each { |ext| load ext }

data/app_generators/godwit/templates/config/database.yml

@@ -0,0 +1,13 @@
+legacy:
+  adapter: sqlserver
+  mode: odbc
+  dsn: 
+  username: 
+  password: 
+
+active:
+  adapter: mysql
+  database: 
+  username: 
+  password:
+  host: 

data/app_generators/godwit/templates/config/environment.rb

@@ -0,0 +1,22 @@
+Godwit::Config.use do |config|
+
+  # Change this if you would like to store key maps somewhere else.
+  #
+  #config[:key_mapper_path] = File.join(Godwit::Config[:godwit_root], 'tmp')
+
+  # This will load a Rails instance to use for your active dataset.
+  #
+  #config[:rails_root] = '/my/new/rails/app'
+  
+  config[:active_record_log] = Logger.new(File.open(File.join(Godwit::Config[:godwit_root], 'log', 'active_record.log'), File::WRONLY | File::APPEND))
+  
+  config[:active_record_log_level] = Logger::INFO
+  
+  config[:active_migration_log] = Logger.new(File.open(File.join(Godwit::Config[:godwit_root], 'log', 'active_migration.log'), File::WRONLY | File::APPEND))
+  
+  config[:active_migration_log_level] = Logger::INFO
+
+end
+
+# This is usually just a dep warning, but we want it to fail.
+Object.send :undef_method, :id

data/app_generators/godwit/templates/script/console

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+irb = RUBY_PLATFORM =~ /(:?mswin|mingw)/ ? 'irb.bat' : 'irb'
+ 
+libs =  " -r irb/completion"
+libs =  " -r #{File.dirname(__FILE__)}/lib/console.rb"
+
+puts "Loading Godwit Environment..."
+exec "#{irb} #{libs} --simple-prompt"

data/app_generators/godwit/templates/script/lib/console.rb

@@ -0,0 +1,3 @@
+require 'rubygems'
+require 'godwit'
+Godwit::Base.new

data/app_generators/godwit/templates/script/migrate

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'godwit'
+
+Godwit::Base.new.run

data/bin/godwit

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'rubigen'
+
+if %w(-v --version).include? ARGV.first
+  require 'godwit/version'
+  puts "#{File.basename($0)} #{Godwit::VERSION::STRING}"
+  exit(0)
+end
+
+require 'rubigen/scripts/generate'
+source = RubiGen::PathSource.new(:application, 
+  File.join(File.dirname(__FILE__), "../app_generators"))
+RubiGen::Base.reset_sources
+RubiGen::Base.append_sources source
+RubiGen::Scripts::Generate.new.run(ARGV, :generator => 'godwit')

data/godwit_generators/active_migration/active_migration_generator.rb

@@ -0,0 +1,55 @@
+class ActiveMigrationGenerator < RubiGen::Base
+
+  default_options :author => nil
+
+  attr_reader :name, :active_model, :legacy_model
+
+  def initialize(runtime_args, runtime_options = {})
+    super
+    usage if args.empty?
+    @name = args.shift
+    extract_options
+    @active_model = args.shift
+    @legacy_model = args.shift
+  end
+
+  def manifest
+    record do |m|
+      # Ensure appropriate folder(s) exists
+      m.directory 'app/migrations/'
+
+      # Create stubs
+      m.template "active_migration.rb",  "app/migrations/#{@name}_migration.rb"
+      # m.template_copy_each ["template.rb", "template2.rb"]
+      # m.file     "file",         "some_file_copied"
+      # m.file_copy_each ["path/to/file", "path/to/file2"]
+    end
+  end
+
+  protected
+    def banner
+      <<-EOS
+Creates an active migration
+
+USAGE: #{$0} #{spec.name} name
+EOS
+    end
+
+    def add_options!(opts)
+      # opts.separator ''
+      # opts.separator 'Options:'
+      # For each option below, place the default
+      # at the top of the file next to "default_options"
+      # opts.on("-a", "--author=\"Your Name\"", String,
+      #         "Some comment about this option",
+      #         "Default: none") { |options[:author]| }
+      # opts.on("-v", "--version", "Show the #{File.basename($0)} version number and quit.")
+    end
+
+    def extract_options
+      # for each option, extract it into a local variable (and create an "attr_reader :author" at the top)
+      # Templates can access these value via the attr_reader-generated methods, but not the
+      # raw instance variable value.
+      # @author = options[:author]
+    end
+end

data/godwit_generators/active_migration/templates/active_migration.rb

@@ -0,0 +1,9 @@
+class <%= name.camelize %>Migration < ActiveMigration::Base
+  
+  set_active_model '<%= active_model %>'
+  
+  set_legacy_model '<%= legacy_model %>'
+  
+  map              []
+  
+end

data/godwit_generators/active_model/active_model_generator.rb

@@ -0,0 +1,53 @@
+class ActiveModelGenerator < RubiGen::Base
+
+  default_options :author => nil
+
+  attr_reader :name
+
+  def initialize(runtime_args, runtime_options = {})
+    super
+    usage if args.empty?
+    @name = args.shift
+    extract_options
+  end
+
+  def manifest
+    record do |m|
+      # Ensure appropriate folder(s) exists
+      m.directory 'app/models/'
+
+      # Create stubs
+      m.template "active_model.rb",  "app/models/#{@name}.rb"
+      # m.template_copy_each ["template.rb", "template2.rb"]
+      # m.file     "file",         "some_file_copied"
+      # m.file_copy_each ["path/to/file", "path/to/file2"]
+    end
+  end
+
+  protected
+    def banner
+      <<-EOS
+Creates a active model
+
+USAGE: #{$0} #{spec.name} name
+EOS
+    end
+
+    def add_options!(opts)
+      # opts.separator ''
+      # opts.separator 'Options:'
+      # For each option below, place the default
+      # at the top of the file next to "default_options"
+      # opts.on("-a", "--author=\"Your Name\"", String,
+      #         "Some comment about this option",
+      #         "Default: none") { |options[:author]| }
+      # opts.on("-v", "--version", "Show the #{File.basename($0)} version number and quit.")
+    end
+
+    def extract_options
+      # for each option, extract it into a local variable (and create an "attr_reader :author" at the top)
+      # Templates can access these value via the attr_reader-generated methods, but not the
+      # raw instance variable value.
+      # @author = options[:author]
+    end
+end

data/godwit_generators/active_model/templates/active_model.rb

@@ -0,0 +1,3 @@
+class <%= name.camelize %> < ActiveRecord::Base
+  
+end

data/godwit_generators/legacy_model/legacy_model_generator.rb

@@ -0,0 +1,53 @@
+class LegacyModelGenerator < RubiGen::Base
+
+  default_options :author => nil
+
+  attr_reader :name
+
+  def initialize(runtime_args, runtime_options = {})
+    super
+    usage if args.empty?
+    @name = args.shift
+    extract_options
+  end
+
+  def manifest
+    record do |m|
+      # Ensure appropriate folder(s) exists
+      m.directory 'app/models/legacy'
+
+      # Create stubs
+      m.template "legacy_model.rb",  "app/models/legacy/#{@name}.rb"
+      # m.template_copy_each ["template.rb", "template2.rb"]
+      # m.file     "file",         "some_file_copied"
+      # m.file_copy_each ["path/to/file", "path/to/file2"]
+    end
+  end
+
+  protected
+    def banner
+      <<-EOS
+Creates a legacy model
+
+USAGE: #{$0} #{spec.name} name
+EOS
+    end
+
+    def add_options!(opts)
+      # opts.separator ''
+      # opts.separator 'Options:'
+      # For each option below, place the default
+      # at the top of the file next to "default_options"
+      # opts.on("-a", "--author=\"Your Name\"", String,
+      #         "Some comment about this option",
+      #         "Default: none") { |options[:author]| }
+      # opts.on("-v", "--version", "Show the #{File.basename($0)} version number and quit.")
+    end
+
+    def extract_options
+      # for each option, extract it into a local variable (and create an "attr_reader :author" at the top)
+      # Templates can access these value via the attr_reader-generated methods, but not the
+      # raw instance variable value.
+      # @author = options[:author]
+    end
+end

data/godwit_generators/legacy_model/templates/legacy_model.rb

@@ -0,0 +1,7 @@
+class Legacy::<%= name.camelize %> < LegacyRecord::Base
+  
+  #set_table_name 'some_legacy_table'
+  
+  #set_primary_key 'some_legacy_pk'
+  
+end

data/lib/godwit/active_migration.rb

@@ -0,0 +1,97 @@
+module ActiveMigration #:nodoc:
+  class Base #:nodoc:
+
+    def initialize #:nodoc:
+      @error = false
+      @count = 0
+      @backspaces = -1
+      @percentage = 0
+      @printed_class = false
+    end
+
+    def handle_error #:nodoc:
+      if Godwit::Config[:silence] || !Godwit::Config[:debug]
+        puts "\n\n\n"
+        raise ActiveMigrationError, 'Failed to save the active record. You should check the logs or run migrate with the -D option to debug.'
+      end
+      unless @error || @skip
+        debugger
+        @error = true
+      else
+        no_resolve
+        @error = false
+        handle_error
+      end
+    end
+
+    def handle_success() #:nodoc:
+      return if Godwit::Config[:silence]
+      unless @error
+        success
+      else
+        resolve
+        success
+        @error = false
+      end
+    end
+
+    def resolve #:nodoc:
+      system('clear')
+      puts "The error has been resolved."
+      puts "\nGodwit will continue the migration in 5 seconds..."
+      sleep(5)
+      system('clear')
+      print Godwit::Buffer.buffer
+    end
+
+    def no_resolve #:nodoc:
+      system('clear')
+      puts "You have not resolved the error!"
+      puts "Please make sure you check that @active_record.valid? is true before you type 'exit'."
+      puts "\nYou will return to the debug console in 5 seconds..."
+      sleep(5)
+    end
+
+    def debugger #:nodoc:
+      system('clear')
+      puts "Godwit " + Godwit::VERSION::STRING
+      puts "\nCurrent Migration: " + self.class.to_s
+      puts "\nYou encountered an invalid record while trying to save.  You can attempt to reconcile this in the debug console."
+      puts "When you are done type 'exit' and godwit will try to save the record again.  Do not try to save the record with #save yourself!"
+      puts "Use methods like #valid? and #errors to determine the problem."
+      puts "\nYou have the following instance variables available to manipulate:"
+      puts "\n@active_record\t\tThis is the record that failed to save.  You should fix this record."
+      puts "@legacy_record\t\tThis should only be used for information.  It is unwise to manipulate this object."
+      puts "@mapping\t\tThis should only be used for information.  It is unwise to manipulate this object."
+      puts "\nLoading Debug Console..."
+      puts "\n"
+      IRB.start_session(binding)
+    end
+
+    def success #:nodoc:
+      if @active_record.is_a?(Array)
+        if @active_array
+          if @active_array != @active_record.object_id
+            @count += 1
+            @active_array = @active_record.object_id
+          end
+        else
+          @count += 1
+          @active_array = @active_record.object_id
+        end
+      else
+        @count += 1
+      end
+      percent = ((@count.to_f / @num_of_records.to_f) * 100).to_i.to_s
+      return if (@percentage == percent) && @count > 1
+      @percentage = percent
+      if @count == 1 && !@printed_class
+        Godwit::Buffer.print("\n" + self.class.to_s + "\t\t\t")
+        @printed_class = true
+      end
+      Godwit::Buffer.print(("\b" * (@backspaces + 1)) + @percentage + '%')
+      @backspaces = @percentage.length
+    end
+
+  end
+end