wineskins 0.2.2

data/.gitignore

@@ -0,0 +1,9 @@
+scratch
+*.sqlite3
+*.log
+*.mdw
+*.mdb
+*.ldb
+
+*.gem
+test/old

data/LICENSE.txt

@@ -0,0 +1,18 @@
+Copyright (c) 2012 Eric Gjertsen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to
+deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+  
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+   
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,235 @@
+# Wineskins
+
+A Ruby database transfer utility built on [Sequel](http://sequel.rubyforge.org/).
+
+Sometimes your old wine needs to be poured into new skins too.
+
+## Basic usage
+
+From the command line, the utility runs __transfer instructions__ for a specified
+__source__ and __destination__ database. By default, it looks for these instructions
+in the file `./transfer.rb`. (The syntax of these instructions will be 
+described in a moment.) So the easiest way to execute the utility is:
+
+    wskins some-db://source/url some-other-db://dest/url
+    
+This will run the instructions in ./transfer.rb to transfer the schema and/or 
+data from the specified source database (as a URL recognized by 
+`Sequel.connect`), to the specified destination database.
+
+You can specify another transfer instructions file via `--config`:
+
+    wskins --config path/to/transfer.rb some-db://source/url some-other-db://dest/url
+    
+If your databases can't be opened easily via a URL, instead you can manually set
+the constants `SOURCE_DB` and `DEST_DB` to your Sequel databases within a ruby 
+script, and require that script from the command line like this:
+
+    wskins --require path/to/db/setup.rb
+
+(This is necessary for instance if you are using ADO adapters.)
+
+Run `wskins --help` to see complete usage options.
+
+## How to install
+
+[Get Ruby](http://www.ruby-lang.org/en/downloads/) if you don't have it.
+
+Then
+  
+    gem install wineskins
+    
+This will also install the sequel gem. If Sequel needs native drivers for your 
+database(s), install them separately. Consult the 
+[Sequel docs](http://sequel.rubyforge.org/) for more info.
+
+## Transfer instructions syntax
+    
+### A very simple case:
+
+    tables :students, :classes, :enrollments
+    
+This will copy the tables, indexes, and foreign key constraints, and then insert 
+all the records, for the three listed tables.
+
+### To rename tables:
+
+    tables :students, [:classes, :courses], :enrollments
+    
+The `classes` table in the source will be renamed `courses` in the destination. 
+All foreign keys referencing `classes` (in e.g. the `enrollments` table) will be 
+changed accordingly.
+
+If you are copying records, be careful in the order you list the tables -- this
+is the order that the records will be inserted. If you have foreign key
+constraints between two tables, you must list the target (primary key) table 
+first to ensure that the keys exist before inserting the foreign key table
+records. 
+
+So in this example, the `:students` and `:classes` tables are transferred _before_
+the `:enrollments` table, which presumably has foreign keys to `:students` and
+`:classes`.
+
+### To rename fields:
+
+    table :classes, :rename => {:class_id => :id}
+    
+Primary keys, indexes, foreign keys will be changed accordingly in the 
+destination database.
+
+### Want to copy the schema, but not import data yet?
+
+    tables :students, :classes, :enrollments, :schema_only => true
+    
+### Have the schema in place, just need to import data?
+
+    tables :students, :classes, :enrollments, :records_only => true
+    
+You have finer-grained control as well:
+
+    table :students, :create_tables => true, 
+                     :create_indexes => false,
+                     :create_fk_constraints => false,
+                     :insert_records => true
+
+### Adjusting column definitions
+
+Sometimes you need to manually adjust column types or other options in the
+destination: for example due to different conventions between databases. You can 
+pass through column definitions to Sequel's schema generator, and they will be 
+used _instead of_ the source database table:
+
+    table :classes do
+      column :slots, :integer, :null => false, :default => 25
+    end
+    
+Note that in this example, all of the column definitions _except `slots`_ will 
+be copied from the source table, while `slots` will be defined as specified.
+
+### Excluding and including columns
+
+(_Note: not yet implemented._)
+You can also exclude specific columns entirely, or include only specified columns:
+
+    table :enrollments, :exclude => [:final_grade, :status]
+    table :students, :include => [:id, :name, :grad_year]
+
+Although it's nearly as easy to do this manually in a hook (see below).
+
+### Limiting the imported data
+
+(_Note: not yet implemented._)
+It's also possible to specify a filter on the source records that get imported.
+
+    table :students do
+      insert_records :grad_year => (2010..2012)
+    end
+
+Filters can be anything that Sequel accepts as arguments to `Dataset#filter`.
+
+### Generating a transcript
+    
+If you just want a script for generating the schema later, and don't actually
+want to make database changes, do something like this:
+
+    transcript 'path/to/transfer.sql'
+    tables :schema_only => true
+
+and include the `--dry-run` option on the command line.
+
+### Hooks for manual futzing
+
+Wineskins executes a given transfer in four stages:
+
+  1. All the tables are created (`:create_table`)
+  2. All the indexes are created via `alter_table` (`:create_indexes`)
+  3. All the foreign key constraints are created via `alter_table` (`:create_fk_constraints`)
+  4. The records are inserted into each table from the source database (`:insert_records`)
+  
+Each of these stages has a `before_*` and `after_*` hook where you can stick
+whatever custom steps you need using Sequel's incredibly wide toolset, and there
+are also general `before` and `after` hooks that run before and after the entire
+transfer. You can define as many of these as you want at each hook. For 
+instance (to turn off foreign key constraints before inserting records):
+
+    before_insert_records do
+      dest.pragma_set 'foreign_keys', 'off'
+    end
+    
+Or to take the example above of excluding columns, you could do this manually
+in a callback like:
+
+    after_create_tables do
+      dest[:enrollments].alter_table do
+        drop_column :final_grade
+        drop_column :status
+      end
+    end
+  
+Within callbacks, the source database is referenced via `source`, the 
+destination database via `dest`.
+
+### A note on the syntax
+
+In the examples above I've used both a 'hash-options' style and a block syntax.
+Either can be used interchangably and even in combination if you want (although
+it's ugly looking). The options set in the block always override the options
+hash. Also, note that custom `column` definitions must be done within a block.
+
+## Motivations
+
+This tool aims to simplify transferring data between databases, and is designed 
+around the canonical case where the destination database is completely empty, 
+and you want to set up everything the way it is in the source and then import 
+the data. Of course, many other scenarios are possible, but the point is that 
+the only things you should need to specify are either (1) differences from this
+scenario, or (2) differences between database adapters that Sequel cannot 
+handle automatically. 
+
+Note that accordingly, if schema or records already exist in your destination,
+you are responsible for dealing with this in whatever way makes sense for your
+scenario. No tables, indexes, constraints, or records are automatically deleted
+in the destination.
+
+So, you might want to wipe out and replace what exists (via `drop_table`,
+`dest[:table].delete`, etc. in callbacks); or you might want to keep what 
+exists (omitting changes via `:schema_only`, `:records_only` options, etc.); or
+you might want to alter what exists (via custom `alter_table`, 
+`dest[:table].filter(some_filter).delete`, etc. in callbacks).
+
+The principle is that _as much as possible, the source database should determine
+the schema of the destination database_, thus minimizing manually-entered (and 
+possibly incorrect) schema definition code. Also it helps avoid, for simple but 
+typical cases, the great pain and knashing of the teeth involved in massaging 
+the source data into the right format for for importing.
+
+## Alternatives / Similar projects
+
+- Sequel's [schema dumper extension](http://sequel.rubyforge.org/rdoc-plugins/files/lib/sequel/extensions/schema_dumper_rb.html) lets you dump and load schema using Sequel's migration DSL.
+- [DbCopier](https://github.com/santosh79/db-copier), apparently unmaintained?
+- [Linkage](https://github.com/coupler/linkage) mimics joins between tables in
+different databases.
+
+## Please help
+
+This is a young young project, don't expect it will work out of the box without
+some futzing. It's only been formally tested on Sqlite to Sqlite transfers, and
+ad-hoc tested on a 'real' MS Access to Sqlite transfer.
+
+If you start using it and run into weird shit, at the very least let me know 
+about it. Better still if you send some informed guesses as to what's going on. 
+Pull requests are awesome and going the extra mile and all that... but before 
+you go to the trouble, unless it's a really minor fix, let me know about the 
+issue, I might be able to save you some time and we can have a conversation 
+about it you know?
+
+There's a TODO list in the project root if you want to see where I'm thinking 
+of heading, comments welcome.
+
+
+## Requirements
+
+  - ruby >= 1.8.7
+  - sequel ~> 3.0 (note >= 3.39 needed for MS Access source databases)
+  - progressbar (optional)
+  

data/Rakefile

@@ -0,0 +1,7 @@
+require 'rake'
+
+task(:test) do
+  require './test/suite.rb'
+end
+
+task :default => :test

data/bin/wskins

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require File.expand_path('../lib/wineskins_cli', File.dirname(__FILE__))
+
+Wineskins::Runner.new(ARGV).run

data/lib/wineskins.rb

@@ -0,0 +1,269 @@
+require 'sequel'
+
+# Modification of ADO/Access adapter to get schema info
+# incorporated into Sequel itself v3.39.0
+# require File.expand_path('sequel_ext/adapters/shared/access', File.dirname(__FILE__))
+
+require File.expand_path('wineskins/version', File.dirname(__FILE__))
+require File.expand_path('wineskins/utils', File.dirname(__FILE__))
+require File.expand_path('wineskins/transcript', File.dirname(__FILE__))
+require File.expand_path('wineskins/schema_methods', File.dirname(__FILE__))
+require File.expand_path('wineskins/record_methods', File.dirname(__FILE__))
+
+module Wineskins
+
+  def self.transfer(source, dest, opts={}, &block)
+    Transfer.new(source, dest, &block).run(opts)
+  end
+    
+  class Transfer
+    include SchemaMethods
+    include RecordMethods
+    
+    attr_accessor :source, :dest
+    attr_reader :table_defs, :progressbar
+    attr_reader :before_hooks, :after_hooks
+    
+    def initialize(source, dest, &block)
+      self.source = source
+      self.dest = dest
+      @table_defs = []
+      @before_hooks = Hash.new{|h,k|h[k]=[]}
+      @after_hooks = Hash.new{|h,k|h[k]=[]}
+      self.define(&block) if block_given?
+    end
+
+    def define(&block)
+      instance_eval(&block)
+      self
+    end
+    
+    def run(opts={})
+      rollback = (opts[:dryrun] ? :always : nil)
+      dest.transaction(:rollback => rollback) do
+        trigger_before_hooks
+        trigger_before_hooks :create_tables
+        create_tables!
+        trigger_after_hooks :create_tables
+        trigger_before_hooks :create_indexes
+        create_indexes!
+        trigger_after_hooks :create_indexes
+        trigger_before_hooks :create_fk_constraints
+        create_fk_constraints!
+        trigger_after_hooks :create_fk_constraints
+        trigger_before_hooks :insert_records
+        insert_records!
+        trigger_after_hooks :insert_records
+        trigger_after_hooks
+      end
+    end
+    
+    def transcript(file=nil)
+      self.dest.loggers << Transcript.new(file)
+    end
+    
+    def tables(*args)
+      opts = (Hash === args.last ? args.pop : {})
+      tbls = (args.empty? ? self.source.tables : args)
+      tbls.each do |tbl| table(tbl, opts) end
+    end
+    
+    def table(name, opts={}, &block)
+      @table_defs << Table.new(name, opts, &block)
+    end
+    
+    def before(event=nil, &cb)
+      before_hooks[event] << cb
+    end
+    
+    def after(event=nil, &cb)
+      after_hooks[event] << cb
+    end
+    
+    [:before, 
+     :after
+    ].product([
+     :create_tables, 
+     :create_indexes, 
+     :create_fk_constraints, 
+     :insert_records
+    ]).each do |(hook, event)|
+      define_method("#{hook}_#{event}") do |&block|
+        send hook, event, &block
+      end
+    end
+      
+    def set_progressbar(title, total)
+      require 'progressbar'
+      @progressbar = ProgressBar.new(title, total)
+    rescue LoadError
+      @progressbar = nil
+    end
+    
+    private
+    
+    def create_tables!
+      @table_defs.select {|t| t.create_table?}.each do |table|
+        transfer_table table
+      end
+    end
+    
+    def create_indexes!
+      @table_defs.select {|t| t.create_indexes?}.each do |table|
+        transfer_indexes table
+      end
+    end
+    
+    def create_fk_constraints!
+      @table_defs.select {|t| t.create_fk_constraints?}.each do |table|
+        transfer_fk_constraints table, table_rename
+      end
+    end
+    
+    def insert_records!
+      @table_defs.select {|t| t.insert_records?}.each do |table|
+        transfer_records table
+      end
+    end
+    
+    # used in create_fk_constraints
+    def table_rename
+      @table_defs.inject({}) do |memo, table|
+        memo[table.source_name] = table.dest_name
+        memo
+      end
+    end
+    
+    def trigger_before_hooks(event=nil)
+      before_hooks[event].each do |cb|
+        cb.call
+      end
+    end
+
+    def trigger_after_hooks(event=nil)
+      after_hooks[event].each do |cb|
+        cb.call
+      end
+    end
+    
+  end
+  
+  # data structure for table transfer definition
+  class Table
+  
+    attr_accessor :source_name, :dest_name, :dest_columns
+    attr_accessor :include, 
+                  :exclude, 
+                  :rename, 
+                  :create_table, 
+                  :create_indexes, 
+                  :create_fk_constraints, 
+                  :insert_records
+       
+    def initialize(name, opts={}, &block)
+      self.source_name, self.dest_name = Array(name)
+      self.dest_name ||= self.source_name
+      self.dest_columns = {}
+      Builder.new(self, default_opts.merge(opts), &block)
+    end
+    
+    def create_table?
+      !!create_table
+    end
+    
+    def create_indexes?
+      !!create_indexes
+    end
+
+    def create_fk_constraints?
+      !!create_fk_constraints
+    end
+    
+    def insert_records?
+      !!insert_records
+    end
+    
+    # todo: handle Proc or Regex === rename
+    def rename_map(cols)
+      col_map = cols.inject({}) {|m,c| m[c]=c;m}
+      col_map.merge(rename)
+    end
+    
+    def default_opts
+      { include:                nil,
+        exclude:                [],
+        rename:                 {},
+        create_table:           true,
+        create_indexes:         true,
+        create_fk_constraints:  true,
+        insert_records:         true
+      }
+    end
+    
+    class Builder
+                              
+      def initialize(target, opts={}, &block)
+        @target = target
+        set_options opts
+        instance_eval(&block) if block_given?
+      end        
+      
+      def include(flds)
+        @target.include = flds
+      end
+      
+      def exclude(flds)
+        @target.exclude = flds
+      end
+      
+      def rename(fldmap=nil, &block)
+        @target.rename = fldmap || block
+      end
+         
+      def column(name, *args)
+        @target.dest_columns[name] = args
+      end      
+      
+      def create_table(bool=true)
+        @target.create_table = bool
+      end
+      
+      def create_indexes(bool=true)
+        @target.create_indexes = bool
+      end
+
+      def create_fk_constraints(bool=true)
+        @target.create_fk_constraints = bool
+      end
+      
+      def insert_records(bool=true)
+        @target.insert_records = bool
+      end
+      
+      def schema_only
+        create_table; create_indexes; create_fk_constraints
+        insert_records false
+      end
+      
+      def records_only
+        create_table(false); create_indexes(false); create_fk_constraints(false)
+        insert_records
+      end
+      
+      def set_options(opts)
+        [:include, :exclude, :rename, 
+         :create_table, :create_indexes, :create_fk_constraints, :insert_records,
+        ].each do |opt|
+          self.send(opt, opts[opt]) if opts[opt]
+        end 
+        [:schema_only,  :records_only].each do |opt|
+          self.send(opt) if opts[opt]
+        end
+      end
+          
+    end
+    
+  end
+  
+end
+