mysql2psql 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,4 @@
1
+ *.yml
2
+ configs
3
+ test/fixtures/test*.sql
4
+ pkg
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2009-2010 name <email>
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,135 @@
1
+ = MYSQL2PSQL - MySQL to PostgreSQL Migration Tool
2
+
3
+ Tool for converting mysql database to postgresql.
4
+ It can create postgresql dump from mysql database or directly load data from mysql to
5
+ postgresql (at about 100 000 records per minute).
6
+
7
+ It can translate now most data types and indexes, but if you experience some problems, feel free
8
+ to contact me, I'll help you.
9
+
10
+ == Installation
11
+ Mysql2psql is packaged as a gem. Install as usual (use sudo if required).
12
+
13
+ $ gem install mysql2psql
14
+ NB: the gem hasn't been formally released on http://rubygems.org/ yet. For now you need to clone the git repo at http://github.com/tardate/mysql2postgres and 'rake install'
15
+
16
+ After installation, the "mysql2psql" command will be available.
17
+ When run, it will generate a default configuration file in the current directory if a config
18
+ file is not already found. The default configuration filename is mysql2psql.yml.
19
+
20
+ $ mysql2psql
21
+
22
+ You can use an alternate config file by passing the filename/path as a parameter:
23
+
24
+ $ mysql2psql ../another/world.yml
25
+
26
+ == Running Migrations
27
+ See the configuration file for documentation about settings. Key choices include:
28
+ * whether to dump to a file, or migrate direct to PostgreSQL
29
+ * migrate only the schema definition, only the data, or both schema and data
30
+
31
+
32
+ After editing the configuration file and launching tool, you will see something like..
33
+
34
+ Creating table friendships...
35
+ Created table friendships
36
+ Loading friendships...
37
+ 620000 of 638779 rows loaded. [ETA: 2010/01/21 21:32 (00h:00m:01s)]
38
+ 638779 rows loaded in 1min 3s
39
+ Indexing table friendships...
40
+ Indexed table friendships
41
+ Table creation 0 min, loading 1 min, indexing 0 min, total 1 min
42
+
43
+
44
+ == Database driver dependencies
45
+ Mysql2psql uses the 'mysql' and 'pg' gems for database connectivity.
46
+ Mysql2psql gem installation should automatically install these too.
47
+
48
+ If you encounter any issues with db connectivity, verify that the 'mysql' and 'pg' gems are installed and working correctly first.
49
+ The 'mysql' gem in particular may need a hint on where to find the mysql headers and libraries:
50
+
51
+ $ [sudo] gem install mysql -- --with-mysql-dir=/usr/local/mysql
52
+
53
+ NB: With Ruby 1.8.x, the gem install will usually complain about "No definition for ..." errors. These are non-fatal and just affect the documentation install. This doesn't happen with Ruby 1.9.2.
54
+
55
+ == Getting the source
56
+ Mysql2psql was first produced by Max Lapshin <max@maxidoors.ru> who maintains the master
57
+ repository at http://github.com/maxlapshin/mysql2postgres
58
+
59
+ The gem packaged version you are currently looking at has yet to be merged into the master repo.
60
+ The gem bundling and refactoring for testing was done by Paul Gallagher <gallagher.paul@gmail.com> and
61
+ the repository is at http://github.com/tardate/mysql2postgres
62
+
63
+
64
+ == Running tests
65
+ If you fork/clone the project, you will want to run the test suite (and add to it if you are developing new features or fixing bugs).
66
+
67
+ A suite of tests are provided and are run using rake (rake -T for more info)
68
+ rake test:units
69
+ rake test:integration
70
+ rake test
71
+
72
+ Rake without parameters (or "rake test") will run both the unit and integration tests.
73
+
74
+ Unit tests are small standalone tests of the mysql2psql codebase that have no external dependencies (like a database)
75
+
76
+ Integration tests require suitable mysql and postgres databases to be setup in advance.
77
+ Running the integration tests *will* rewrite data and schema for the "mysql2psql_test" databases in mysql and postgres. Don't use this database for anything else!
78
+
79
+ mysql on localhost:3306
80
+ - database created called "mysql2psql_test"
81
+ - user setup for "mysql2psql" with no password
82
+ - e.g.
83
+ mysqladmin -uroot -p create mysql2psql_test
84
+ mysql -uroot -p -e "grant all on mysql2psql_test.* to 'mysql2psql'@'localhost';"
85
+ # verify connecction:
86
+ mysql -umysql2psql -e "select database(),'yes' as connected" mysql2psql_test
87
+
88
+ postgres on localhost:5432
89
+ - database created called "mysql2psql_test"
90
+ - role (user) access setup for "mysql2psql" with no password
91
+ - e.g.
92
+ psql postgres -c "create role mysql2psql with login"
93
+ psql postgres -c "create database mysql2psql_test with owner mysql2psql encoding='UTF8'"
94
+ # verify connection:
95
+ psql mysql2psql_test -U mysql2psql -c "\c"
96
+
97
+ == Notes, Limitations, Outstanding Issues..
98
+
99
+ Todo:
100
+ - more tests
101
+ - release gem
102
+ - a windows cmd shim
103
+
104
+ === note from mgkimsal
105
+ I'm still having trouble with bit(1)/boolean fields
106
+ workaround I've found is to put output in file
107
+ then in VIM on file, search/replace the true/false binary fields with t/f
108
+ specifically
109
+
110
+ (reversed on 3/23 - should be ^A gets f)
111
+ :%s/^@/t/g
112
+ :%s/^A/f/g
113
+ keystrokes are ctrl-v ctrl-shift-@ to get the 'true' binary field
114
+ keystrokes are ctrl-v ctrl-shift-A to get the 'false' binary field
115
+
116
+ == Contributors
117
+ Project founded by Max Lapshin <max@maxidoors.ru>
118
+
119
+ Other contributors (in git log order):
120
+ - Anton Ageev <anton@ageev.name>
121
+ - Samuel Tribehou <cracoucax@gmail.com>
122
+ - Marco Nenciarini <marco.nenciarini@devise.it>
123
+ - James Nobis <jnobis@jnobis.controldocs.com>
124
+ - quel <github@quelrod.net>
125
+ - Holger Amann <keeney@fehu.org>
126
+ - Maxim Dobriakov <closer.main@gmail.com>
127
+ - Michael Kimsal <mgkimsal@gmail.com>
128
+ - Jacob Coby <jcoby@portallabs.com>
129
+ - Neszt Tibor <neszt@tvnetwork.hu>
130
+ - Miroslav Kratochvil <exa.exa@gmail.com>
131
+ - Paul Gallagher <gallagher.paul@gmail.com>
132
+
133
+
134
+
135
+
@@ -0,0 +1,87 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+
4
+ $LOAD_PATH.unshift('lib')
5
+ require 'mysql2psql/version'
6
+
7
+ begin
8
+ require 'jeweler'
9
+ Jeweler::Tasks.new do |gem|
10
+ gem.name = "mysql2psql"
11
+ gem.version = Mysql2psql::Version::STRING
12
+ gem.summary = %Q{Tool for converting mysql database to postgresql}
13
+ gem.description = %Q{It can create postgresql dump from mysql database or directly load data from mysql to
14
+ postgresql (at about 100 000 records per minute). Translates most data types and indexes.}
15
+ gem.email = "gallagher.paul@gmail.com"
16
+ gem.homepage = "http://github.com/tardate/mysql2postgresql"
17
+ gem.authors = [
18
+ "Max Lapshin <max@maxidoors.ru>",
19
+ "Anton Ageev <anton@ageev.name>",
20
+ "Samuel Tribehou <cracoucax@gmail.com>",
21
+ "Marco Nenciarini <marco.nenciarini@devise.it>",
22
+ "James Nobis <jnobis@jnobis.controldocs.com>",
23
+ "quel <github@quelrod.net>",
24
+ "Holger Amann <keeney@fehu.org>",
25
+ "Maxim Dobriakov <closer.main@gmail.com>",
26
+ "Michael Kimsal <mgkimsal@gmail.com>",
27
+ "Jacob Coby <jcoby@portallabs.com>",
28
+ "Neszt Tibor <neszt@tvnetwork.hu>",
29
+ "Miroslav Kratochvil <exa.exa@gmail.com>",
30
+ "Paul Gallagher <gallagher.paul@gmail.com>"
31
+ ]
32
+ gem.add_dependency "mysql", "= 2.8.1"
33
+ gem.add_dependency "pg", "= 0.9.0"
34
+ gem.add_development_dependency "test-unit", ">= 2.1.1"
35
+ # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
36
+ end
37
+ Jeweler::GemcutterTasks.new
38
+ rescue LoadError
39
+ puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
40
+ end
41
+
42
+ require 'rake/testtask'
43
+ namespace :test do
44
+ Rake::TestTask.new(:units) do |test|
45
+ test.libs << 'lib' << 'test/lib'
46
+ test.pattern = 'test/units/*test.rb'
47
+ test.verbose = true
48
+ end
49
+
50
+ Rake::TestTask.new(:integration) do |test|
51
+ test.libs << 'lib' << 'test/lib'
52
+ test.pattern = 'test/integration/*test.rb'
53
+ test.verbose = true
54
+ end
55
+ end
56
+
57
+ desc "Run all tests"
58
+ task :test do
59
+ Rake::Task['test:units'].invoke
60
+ Rake::Task['test:integration'].invoke
61
+ end
62
+
63
+ begin
64
+ require 'rcov/rcovtask'
65
+ Rcov::RcovTask.new do |test|
66
+ test.libs << 'test'
67
+ test.pattern = 'test/**/*test.rb'
68
+ test.verbose = true
69
+ end
70
+ rescue LoadError
71
+ task :rcov do
72
+ abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
73
+ end
74
+ end
75
+
76
+
77
+ task :default => :test
78
+
79
+ require 'rake/rdoctask'
80
+ Rake::RDocTask.new do |rdoc|
81
+ version = Mysql2psql::Version::STRING
82
+
83
+ rdoc.rdoc_dir = 'rdoc'
84
+ rdoc.title = "mysql2psql #{version}"
85
+ rdoc.rdoc_files.include('README*')
86
+ rdoc.rdoc_files.include('lib/**/*.rb')
87
+ end
@@ -0,0 +1,7 @@
1
+ #!/usr/bin/env ruby
2
+ $LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
3
+
4
+ require 'rubygems'
5
+ require 'mysql2psql'
6
+
7
+ exit Mysql2psql.new(ARGV).convert
@@ -0,0 +1,41 @@
1
+ require 'mysql2psql/errors'
2
+ require 'mysql2psql/version'
3
+ require 'mysql2psql/config'
4
+ require 'mysql2psql/converter'
5
+ require 'mysql2psql/mysql_reader'
6
+ require 'mysql2psql/writer'
7
+ require 'mysql2psql/postgres_writer'
8
+ require 'mysql2psql/postgres_db_writer.rb'
9
+ require 'mysql2psql/postgres_file_writer.rb'
10
+
11
+
12
+ class Mysql2psql
13
+
14
+ attr_reader :options, :reader, :writer
15
+
16
+ def initialize(args)
17
+ help if args.length==1 && args[0] =~ /^-.?|^-*he?l?p?$/i
18
+ configfile = args[0] || File.expand_path('mysql2psql.yml')
19
+ @options = Config.new( configfile, true )
20
+ end
21
+
22
+ def convert
23
+ @reader = MysqlReader.new( options )
24
+
25
+ if options.destfile(nil)
26
+ @writer = PostgresFileWriter.new(options.destfile)
27
+ else
28
+ @writer = PostgresDbWriter.new(options)
29
+ end
30
+
31
+ Converter.new(reader, writer, options).convert
32
+ end
33
+
34
+ def help
35
+ puts <<EOS
36
+ MySQL to PostgreSQL Conversion
37
+
38
+ EOS
39
+ exit -2
40
+ end
41
+ end
@@ -0,0 +1,100 @@
1
+ require 'mysql2psql/config_base'
2
+
3
+ class Mysql2psql
4
+
5
+ class Config < ConfigBase
6
+
7
+ def initialize(configfilepath, generate_default_if_not_found = true)
8
+ unless File.exists?(configfilepath)
9
+ reset_configfile(configfilepath) if generate_default_if_not_found
10
+ if File.exists?(configfilepath)
11
+ raise Mysql2psql::ConfigurationFileInitialized.new("\n
12
+ No configuration file found.
13
+ A new file has been initialized at: #{configfilepath}
14
+ Please review the configuration and retry..\n\n\n")
15
+ else
16
+ raise Mysql2psql::ConfigurationFileNotFound.new("cannot load config file #{configfilepath}")
17
+ end
18
+ end
19
+ super(configfilepath)
20
+ end
21
+
22
+ def reset_configfile(filepath)
23
+ file = File.new(filepath,'w')
24
+ self.class.template.each_line do | line|
25
+ file.puts line
26
+ end
27
+ file.close
28
+ end
29
+
30
+ def self.template(to_filename = nil, include_tables = [], exclude_tables = [], supress_data = false, supress_ddl = false, force_truncate = false)
31
+ configtext = <<EOS
32
+ mysql:
33
+ hostname: localhost
34
+ port: 3306
35
+ socket: /tmp/mysql.sock
36
+ username: mysql2psql
37
+ password:
38
+ database: mysql2psql_test
39
+
40
+ destination:
41
+ # if file is given, output goes to file, else postgres
42
+ file: #{ to_filename ? to_filename : ''}
43
+ postgres:
44
+ hostname: localhost
45
+ port: 5432
46
+ username: mysql2psql
47
+ password:
48
+ database: mysql2psql_test
49
+
50
+ # if tables is given, only the listed tables will be converted. leave empty to convert all tables.
51
+ #tables:
52
+ #- table1
53
+ #- table2
54
+ EOS
55
+ if include_tables.length>0
56
+ configtext += "\ntables:\n"
57
+ include_tables.each do |t|
58
+ configtext += "- #{t}\n"
59
+ end
60
+ end
61
+ configtext += <<EOS
62
+ # if exclude_tables is given, exclude the listed tables from the conversion.
63
+ #exclude_tables:
64
+ #- table3
65
+ #- table4
66
+
67
+ EOS
68
+ if exclude_tables.length>0
69
+ configtext += "\nexclude_tables:\n"
70
+ exclude_tables.each do |t|
71
+ configtext += "- #{t}\n"
72
+ end
73
+ end
74
+ if !supress_data.nil?
75
+ configtext += <<EOS
76
+
77
+ # if supress_data is true, only the schema definition will be exported/migrated, and not the data
78
+ supress_data: #{supress_data}
79
+ EOS
80
+ end
81
+ if !supress_ddl.nil?
82
+ configtext += <<EOS
83
+
84
+ # if supress_ddl is true, only the data will be exported/imported, and not the schema
85
+ supress_ddl: #{supress_ddl}
86
+ EOS
87
+ end
88
+ if !force_truncate.nil?
89
+ configtext += <<EOS
90
+
91
+ # if force_truncate is true, forces a table truncate before table loading
92
+ force_truncate: #{force_truncate}
93
+ EOS
94
+ end
95
+ configtext
96
+ end
97
+
98
+ end
99
+
100
+ end
@@ -0,0 +1,39 @@
1
+ require 'yaml'
2
+ require 'mysql2psql/errors'
3
+
4
+ class Mysql2psql
5
+
6
+ class ConfigBase
7
+ attr_reader :config, :filepath
8
+
9
+ def initialize(configfilepath)
10
+ @filepath=configfilepath
11
+ @config = YAML::load(File.read(filepath))
12
+ end
13
+ def [](key)
14
+ self.send( key )
15
+ end
16
+ def method_missing(name, *args)
17
+ token=name.to_s
18
+ default = args.length>0 ? args[0] : ''
19
+ must_be_defined = default == :none
20
+ case token
21
+ when /mysql/i
22
+ key=token.sub( /^mysql/, '' )
23
+ value=config["mysql"][key]
24
+ when /pg/i
25
+ key=token.sub( /^pg/, '' )
26
+ value=config["destination"]["postgres"][key]
27
+ when /dest/i
28
+ key=token.sub( /^dest/, '' )
29
+ value=config["destination"][key]
30
+ when /only_tables/i
31
+ value=config["tables"]
32
+ else
33
+ value=config[token]
34
+ end
35
+ value.nil? ? ( must_be_defined ? (raise Mysql2psql::UninitializedValueError.new("no value and no default for #{name}")) : default ) : value
36
+ end
37
+ end
38
+
39
+ end
@@ -0,0 +1,55 @@
1
+ class Mysql2psql
2
+
3
+ class Converter
4
+ attr_reader :reader, :writer, :options
5
+ attr_reader :exclude_tables, :only_tables, :supress_data, :supress_ddl, :force_truncate
6
+
7
+ def initialize(reader, writer, options)
8
+ @reader = reader
9
+ @writer = writer
10
+ @options = options
11
+ @exclude_tables = options.exclude_tables([])
12
+ @only_tables = options.only_tables(nil)
13
+ @supress_data = options.supress_data(false)
14
+ @supress_ddl = options.supress_ddl(false)
15
+ @force_truncate = options.force_truncate(false)
16
+ end
17
+
18
+ def convert
19
+ _time1 = Time.now
20
+
21
+ tables = reader.tables.
22
+ reject {|table| @exclude_tables.include?(table.name)}.
23
+ select {|table| @only_tables ? @only_tables.include?(table.name) : true}
24
+
25
+
26
+ tables.each do |table|
27
+ writer.write_table(table)
28
+ end unless @supress_ddl
29
+
30
+ _time2 = Time.now
31
+ tables.each do |table|
32
+ writer.truncate(table) if force_truncate && supress_ddl
33
+ writer.write_contents(table, reader)
34
+ end unless @supress_data
35
+
36
+ _time3 = Time.now
37
+ tables.each do |table|
38
+ writer.write_indexes(table)
39
+ end unless @supress_ddl
40
+ tables.each do |table|
41
+ writer.write_constraints(table)
42
+ end unless @supress_ddl
43
+
44
+
45
+ writer.close
46
+ _time4 = Time.now
47
+ puts "Table creation #{((_time2 - _time1) / 60).round} min, loading #{((_time3 - _time2) / 60).round} min, indexing #{((_time4 - _time3) / 60).round} min, total #{((_time4 - _time1) / 60).round} min"
48
+ return 0
49
+ rescue => e
50
+ $stderr.puts "Mysql2psql: conversion failed: #{e.to_s}"
51
+ return -1
52
+ end
53
+ end
54
+
55
+ end