rigrate 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/README.md +160 -0
- data/Rakefile +9 -0
- data/bin/rigrate +69 -0
- data/lib/rigrate.rb +97 -0
- data/lib/rigrate/data_source.rb +43 -0
- data/lib/rigrate/error.rb +11 -0
- data/lib/rigrate/interface.rb +20 -0
- data/lib/rigrate/interface/driver.rb +41 -0
- data/lib/rigrate/interface/mysql.rb +148 -0
- data/lib/rigrate/interface/oracle.rb +172 -0
- data/lib/rigrate/interface/result_set.rb +389 -0
- data/lib/rigrate/interface/row.rb +69 -0
- data/lib/rigrate/interface/sqlite.rb +106 -0
- data/lib/rigrate/migration.rb +89 -0
- data/lib/rigrate/parser.rb +270 -0
- data/rigrate.gemspec +21 -0
- data/test/test_datasource.rb +66 -0
- data/test/test_driver.rb +45 -0
- data/test/test_driver_mysql.rb +87 -0
- data/test/test_driver_oracle.rb +109 -0
- data/test/test_driver_sqlite.rb +85 -0
- data/test/test_helper.rb +9 -0
- data/test/test_migration.rb +108 -0
- data/test/test_parser.rb +262 -0
- data/test/test_resultset.rb +300 -0
- data/test/test_rigrate.rb +6 -0
- data/test/test_rigrate_row.rb +51 -0
- metadata +124 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 187fa5716dd1625fe30f8a5d1fcab945cf1b8293
|
4
|
+
data.tar.gz: 0bdffca0c4724b9d1a36b36c3a8f46dc54fd0b52
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 5a5af45d6cade3df5147184238d8ad3ae41b0afdd0c399d4789fe4b6820a345205c742ba9f8ba4b9b4e1d94573d9844f0c058d1d9647c70322a420de539a8c48
|
7
|
+
data.tar.gz: e281b403611bef6f7fa738ad3cf64a153b2a57a91c1e35027582c47435cfb9d6003a6e801e6edc79362a33446a485f91a779d6d4f4f8a4fd7fd42efb8805169f
|
data/README.md
ADDED
@@ -0,0 +1,160 @@
|
|
1
|
+
# Rigrate
|
2
|
+
|
3
|
+
## Overview
|
4
|
+
This gem is used to migrate data between different data source in an easy way. SQLite3、MySQL、Oracle supported in current.
|
5
|
+
|
6
|
+
## Installation
|
7
|
+
```ruby
|
8
|
+
gem install rigrate
|
9
|
+
```
|
10
|
+
|
11
|
+
if you are using oracle, you need install instant oracle clent at least.
|
12
|
+
|
13
|
+
## Basic Usage
|
14
|
+
|
15
|
+
An easy example, migrate entries hrdb users table to accounts table in oadb.
|
16
|
+
|
17
|
+
Firstly, Write the migrate script
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
# file : hr_users_to_oa_accounts.rigrate
|
21
|
+
|
22
|
+
# define the data sources, use URI pattern like DB://USR:PWD@HOST/DBNAME
|
23
|
+
ds :hr, "oracle://scott:tiger@hrdb"
|
24
|
+
ds :oa, "mysql://root:testpwd@127.0.0.1:oadb"
|
25
|
+
|
26
|
+
# migrate snippet code
|
27
|
+
from hr.users to oa.accounts
|
28
|
+
```
|
29
|
+
|
30
|
+
Then, you can run as following:
|
31
|
+
|
32
|
+
```shell
|
33
|
+
$ rigrate execute -f hr_users_to_oa_accounts.rigrate
|
34
|
+
```
|
35
|
+
|
36
|
+
## More Examples
|
37
|
+
|
38
|
+
1. Using native SQL. Just migrate DepNO is D00001 users to target data source.
|
39
|
+
|
40
|
+
```ruby
|
41
|
+
from
|
42
|
+
hr.sql("select * from users where deptno = 'D00001'")
|
43
|
+
to
|
44
|
+
oa.accounts
|
45
|
+
```
|
46
|
+
|
47
|
+
2. Specify the sync condition columns. When the migration has condition columns, it will update target data instead of delete the records directly.
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
from
|
51
|
+
hr.sql("select * from users")
|
52
|
+
to
|
53
|
+
oa.accounts
|
54
|
+
on :user_code => :job_code
|
55
|
+
```
|
56
|
+
|
57
|
+
3. Just migrate a part of entry source columns. In this situation migration with one condition is advised.
|
58
|
+
|
59
|
+
```ruby
|
60
|
+
from
|
61
|
+
hr.users(:user_code, :name, :age, :passwd)
|
62
|
+
to
|
63
|
+
oa.account(:job_code, :name, :age, :password)
|
64
|
+
on :user_code => :job_code
|
65
|
+
```
|
66
|
+
4. Union two data source.
|
67
|
+
|
68
|
+
```ruby
|
69
|
+
from
|
70
|
+
hr.users union oa.accounts
|
71
|
+
to
|
72
|
+
erp.users
|
73
|
+
```
|
74
|
+
5. Minus two data source
|
75
|
+
|
76
|
+
```ruby
|
77
|
+
from
|
78
|
+
hr.users minus oa.accounts
|
79
|
+
to
|
80
|
+
erp.users
|
81
|
+
```
|
82
|
+
|
83
|
+
6. Join two data source. JOIN must with condition.
|
84
|
+
|
85
|
+
```ruby
|
86
|
+
from
|
87
|
+
hr.users join oa.account on :user_code => :job_code
|
88
|
+
to
|
89
|
+
erp.users
|
90
|
+
```
|
91
|
+
|
92
|
+
7. Data source select with after hooks
|
93
|
+
|
94
|
+
```ruby
|
95
|
+
from
|
96
|
+
hr.users(:user_code, :user_name, :password, :age) do |row|
|
97
|
+
row[2] = "useless_password" # make password useless
|
98
|
+
row[3] = 16 # make everyone 16 forever :-)
|
99
|
+
end
|
100
|
+
to
|
101
|
+
oa.accounts(:job_code, :name, :pwd, :age)
|
102
|
+
on :user_code => :job_code
|
103
|
+
```
|
104
|
+
|
105
|
+
## Advance Tips
|
106
|
+
|
107
|
+
1. Sync Mode
|
108
|
+
Rigrate support support three mode, inspired by `SyncToy` a file sync tool by Microsoft.
|
109
|
+
|
110
|
+
**:echo** is the default mode, will delete all records of target ds which not exists in source ds
|
111
|
+
**:contrbiute** is same as :echo, but keep the records even it deleted in source ds
|
112
|
+
**:sync** mode will make two side of migration the same
|
113
|
+
|
114
|
+
use :contribute mode in migration:
|
115
|
+
```ruby
|
116
|
+
from hr.users to oa.users mode :contribute
|
117
|
+
```
|
118
|
+
|
119
|
+
2. Using Transaction
|
120
|
+
Rigrate will do not use transaction as default. Switch it on, you can do it througt command line `--strict`.
|
121
|
+
|
122
|
+
```shell
|
123
|
+
$ rigrate -f script.rigrate --strict
|
124
|
+
```
|
125
|
+
|
126
|
+
3. Using Rigrate in other ruby script
|
127
|
+
|
128
|
+
```ruby
|
129
|
+
require 'rigrate'
|
130
|
+
|
131
|
+
str =<<EOS
|
132
|
+
ds :hr, "oracle://scott:tiger@hrdb"
|
133
|
+
ds :oa, "mysql://root:testpwd@127.0.0.1:oadb"
|
134
|
+
|
135
|
+
from hr.users to oa.accounts
|
136
|
+
EOS
|
137
|
+
|
138
|
+
parser = Rigrate::Parser.new
|
139
|
+
parser.lex(str).parsing
|
140
|
+
```
|
141
|
+
4. QuickScript. a fast way to operate data, Rigrate will load a predefined data sources file in {home}/.rigrate/ds.
|
142
|
+
you can define some frequently used data source, and run in migration throught command line. like blow:
|
143
|
+
|
144
|
+
```shell
|
145
|
+
$ rigrate execute -c "from hr.users to oa.accounts"
|
146
|
+
```
|
147
|
+
|
148
|
+
for more usage to check `Rigrate help`
|
149
|
+
|
150
|
+
## TODO Lists
|
151
|
+
|
152
|
+
1. Integrate with schedule framework
|
153
|
+
2. Add pg/sql sever/access/csv support
|
154
|
+
3. Row data type and length checking
|
155
|
+
4. Rigrate file checking
|
156
|
+
5. More logger details
|
157
|
+
|
158
|
+
## Others
|
159
|
+
|
160
|
+
None :-)
|
data/Rakefile
ADDED
data/bin/rigrate
ADDED
@@ -0,0 +1,69 @@
|
|
1
|
+
#! ruby
|
2
|
+
|
3
|
+
lib = File.expand_path(File.dirname(__FILE__) + '/../lib')
|
4
|
+
$LOAD_PATH.unshift(lib) if File.directory?(lib) && ! $LOAD_PATH.include?(lib)
|
5
|
+
|
6
|
+
require 'thor'
|
7
|
+
require 'rigrate'
|
8
|
+
|
9
|
+
class RigrateApp < Thor
|
10
|
+
desc 'execute', 'execute a task.'
|
11
|
+
method_option :file, :aliases => '-f',
|
12
|
+
:type => :string,
|
13
|
+
:desc => 'script file path.'
|
14
|
+
method_option :mode, :aliases => '-m',
|
15
|
+
:type => :string,
|
16
|
+
:default => :echo,
|
17
|
+
:desc => "migration mode, :echo and :contribute support in current"
|
18
|
+
method_option :loglevel, :aliases => '-l',
|
19
|
+
:type => :string,
|
20
|
+
:default => "999",
|
21
|
+
:desc => "Do you want to log the actions? using number 1~5 or info/warn/debug... to this option"
|
22
|
+
method_option :stdout, :aliases => '-o',
|
23
|
+
:type => :boolean,
|
24
|
+
:default => false,
|
25
|
+
:desc => "switch on to using STDOUT in prompt."
|
26
|
+
method_option :strict, :aliases => '-s',
|
27
|
+
:type => :boolean,
|
28
|
+
:default => false,
|
29
|
+
:desc => "switch the option on will using db transaction to commit"
|
30
|
+
method_option :script, :aliases => '-c',
|
31
|
+
:type => :string,
|
32
|
+
:desc => "eval migration script in command line. using predefined DataSource in default ds file"
|
33
|
+
method_option :dir, :aliases => '-d',
|
34
|
+
:type => :string,
|
35
|
+
:desc => "directory of rigrate script"
|
36
|
+
def execute
|
37
|
+
# format all key from symbol to string
|
38
|
+
opts = options.inject({}) do |hash, (k, v)|
|
39
|
+
hash[k.to_sym] = v
|
40
|
+
hash
|
41
|
+
end
|
42
|
+
|
43
|
+
opts[:loglevel] = format_loglevel(opts[:loglevel])
|
44
|
+
Rigrate.run(opts)
|
45
|
+
end
|
46
|
+
|
47
|
+
desc "verify", "verify a file is right or not"
|
48
|
+
method_option :file, :aliases => '-f',
|
49
|
+
:desc => "the file to be verified"
|
50
|
+
def verify
|
51
|
+
|
52
|
+
end
|
53
|
+
|
54
|
+
no_commands {
|
55
|
+
def format_loglevel(level)
|
56
|
+
level = level.to_s.downcase
|
57
|
+
return level.to_i if level =~ /[0-5]+/
|
58
|
+
|
59
|
+
['debug', 'info', 'warn', 'error', 'fatal', 'unknown'].each_with_index do |ele, idx|
|
60
|
+
return idx if ele == level.to_s.downcase
|
61
|
+
end
|
62
|
+
|
63
|
+
999
|
64
|
+
end
|
65
|
+
}
|
66
|
+
|
67
|
+
end
|
68
|
+
|
69
|
+
RigrateApp.start
|
data/lib/rigrate.rb
ADDED
@@ -0,0 +1,97 @@
|
|
1
|
+
# encoding : utf-8
|
2
|
+
|
3
|
+
require 'logger'
|
4
|
+
|
5
|
+
require 'rigrate/data_source'
|
6
|
+
require 'rigrate/error'
|
7
|
+
require 'rigrate/interface'
|
8
|
+
require 'rigrate/migration'
|
9
|
+
require 'rigrate/parser'
|
10
|
+
|
11
|
+
module Rigrate
|
12
|
+
|
13
|
+
##
|
14
|
+
# used to output logger to mutli place
|
15
|
+
#
|
16
|
+
class MutiOutput
|
17
|
+
def initialize(file, prompt)
|
18
|
+
@target = []
|
19
|
+
@target << file
|
20
|
+
@target << STDOUT if prompt
|
21
|
+
end
|
22
|
+
|
23
|
+
def write(*args)
|
24
|
+
@target.each { |dest| dest.write(*args) }
|
25
|
+
end
|
26
|
+
|
27
|
+
def close
|
28
|
+
@target.each(&:close)
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
# create rigrate main path
|
33
|
+
path = File.join(Dir.home, ".rigrate")
|
34
|
+
Dir.mkdir path unless (Dir.exist? path)
|
35
|
+
|
36
|
+
@config = {
|
37
|
+
:loglevel => 999, # default logger not show
|
38
|
+
:logpath => path,
|
39
|
+
:mode => :echo,
|
40
|
+
:strict => false,
|
41
|
+
:stdout => false,
|
42
|
+
:ds => File.join(Dir.home, ".rigrate/ds"),
|
43
|
+
:script => nil,
|
44
|
+
:file => nil,
|
45
|
+
}
|
46
|
+
|
47
|
+
def self.run(opts = {})
|
48
|
+
scripts = []
|
49
|
+
configure(opts)
|
50
|
+
|
51
|
+
# read all scripts <- from command line , file script , rigrate dir
|
52
|
+
scripts << opts[:script] if opts[:script]
|
53
|
+
scripts << File.read(opts[:file]) if opts[:file]
|
54
|
+
if Dir.exist? opts[:dir].to_s
|
55
|
+
Dir["#{File.join(opts[:dir], '*')}"].each do |item|
|
56
|
+
scripts << File.read(item)
|
57
|
+
end
|
58
|
+
end
|
59
|
+
|
60
|
+
raise RigrateError.new("your should specify one script at least.") if scripts.size <= 0
|
61
|
+
|
62
|
+
parser = Parser.new
|
63
|
+
# loading data source
|
64
|
+
if File.exist? config[:ds]
|
65
|
+
parser.lex(File.read(config[:ds])).parsing
|
66
|
+
end
|
67
|
+
|
68
|
+
scripts.each do |script|
|
69
|
+
parser.lex(script).parsing
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
def self.logger
|
74
|
+
logger_file = File.open(File.join(config[:logpath],logger_name), 'a')
|
75
|
+
|
76
|
+
@log = Logger.new MutiOutput.new(logger_file, config[:stdout])
|
77
|
+
@log.level = config[:loglevel]
|
78
|
+
|
79
|
+
@log
|
80
|
+
end
|
81
|
+
|
82
|
+
def self.configure(opts)
|
83
|
+
opts.each do |k, v|
|
84
|
+
raise ParserError.new("arguments #{k} not valid.") unless config.keys.include? k.to_sym
|
85
|
+
end
|
86
|
+
|
87
|
+
config.merge! opts
|
88
|
+
end
|
89
|
+
|
90
|
+
def self.config
|
91
|
+
@config
|
92
|
+
end
|
93
|
+
|
94
|
+
def self.logger_name
|
95
|
+
"#{Time.now.strftime('%Y-%m-%d')}.log"
|
96
|
+
end
|
97
|
+
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
# encoding : utf-8
|
2
|
+
|
3
|
+
require 'uri'
|
4
|
+
|
5
|
+
module Rigrate
|
6
|
+
class DataSource
|
7
|
+
attr_accessor :dbh
|
8
|
+
|
9
|
+
def initialize(conn_uri)
|
10
|
+
uri = URI.parse(conn_uri)
|
11
|
+
@dbh = eval(uri.scheme.capitalize).new conn_uri
|
12
|
+
end
|
13
|
+
|
14
|
+
def sql(str, *args)
|
15
|
+
begin
|
16
|
+
@dbh.select(str, *args)
|
17
|
+
rescue SQLite3::SQLException => e
|
18
|
+
puts "DB: #{@dbh.inspect} SQL: #{str}"
|
19
|
+
raise e
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
def method_missing(mth, *args, &block)
|
24
|
+
begin
|
25
|
+
table_name = mth
|
26
|
+
columns = args
|
27
|
+
|
28
|
+
str_sql = build_sql(table_name, *columns)
|
29
|
+
@dbh.select(str_sql, &block)
|
30
|
+
rescue Exception => e
|
31
|
+
raise e
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
private
|
36
|
+
|
37
|
+
def build_sql(table, *columns)
|
38
|
+
columns = ['*'] if columns.size == 0
|
39
|
+
|
40
|
+
"select #{columns.map(&:to_s).join(',')} from #{table}"
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,11 @@
|
|
1
|
+
# encoding : utf-8
|
2
|
+
|
3
|
+
module Rigrate
|
4
|
+
class BasicError < ::StandardError; end
|
5
|
+
|
6
|
+
class RigrateError < BasicError; end
|
7
|
+
class ParserError < BasicError; end
|
8
|
+
class InterfaceError < BasicError; end
|
9
|
+
class ResultSetError < BasicError; end
|
10
|
+
class DirverError < BasicError; end
|
11
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
# encoding : utf-8
|
2
|
+
|
3
|
+
require 'uri'
|
4
|
+
require 'rigrate/interface/driver'
|
5
|
+
require 'rigrate/interface/result_set'
|
6
|
+
require 'rigrate/interface/row'
|
7
|
+
require 'rigrate/interface/sqlite'
|
8
|
+
require 'rigrate/interface/mysql'
|
9
|
+
require 'rigrate/interface/oracle'
|
10
|
+
|
11
|
+
module Rigrate
|
12
|
+
def self.lazy_load_driver(driver_name)
|
13
|
+
driver_path = "./interface/#{driver_name}"
|
14
|
+
if File.exists? driver_path
|
15
|
+
raise InterfaceError.new('Driver not found.')
|
16
|
+
end
|
17
|
+
|
18
|
+
require driver_path
|
19
|
+
end
|
20
|
+
end
|
@@ -0,0 +1,41 @@
|
|
1
|
+
# encoding : utf-8
|
2
|
+
|
3
|
+
module Rigrate
|
4
|
+
# defination for column include name and type
|
5
|
+
# column name is a string
|
6
|
+
Column = Struct.new(:name, :type)
|
7
|
+
|
8
|
+
class Driver
|
9
|
+
attr_accessor :db
|
10
|
+
|
11
|
+
def connect(db)
|
12
|
+
self.db = db
|
13
|
+
end
|
14
|
+
|
15
|
+
def method_missing(mth, *args, &block)
|
16
|
+
@db.send(mth, *args, &block)
|
17
|
+
end
|
18
|
+
|
19
|
+
def extract_conn_param(uri)
|
20
|
+
opts = {}
|
21
|
+
opts['db_type'] = uri.scheme if uri.scheme
|
22
|
+
opts['host'] = uri.host if uri.host
|
23
|
+
opts['username'] = uri.user if uri.user
|
24
|
+
opts['password'] = uri.password if uri.password
|
25
|
+
opts['port'] = uri.port if uri.port
|
26
|
+
opts['db_name'] = uri.path.tr('/','') if uri.path.tr('/','').size > 0
|
27
|
+
|
28
|
+
opts
|
29
|
+
end
|
30
|
+
|
31
|
+
def to_native_row(row, column_info)
|
32
|
+
row
|
33
|
+
end
|
34
|
+
|
35
|
+
def extract_tbl_from_sql(sql_str)
|
36
|
+
return $1 if sql_str =~ /from\s+(\w*)\s*/
|
37
|
+
|
38
|
+
raise Exception.new('a lastest one table name must specify.')
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|