tapsoob 0.2.7-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 5e733ad5dfac7d36dda9f7d5447a437454d0224d6743041545e3a75f5b65b7d4
4
+ data.tar.gz: f073d871bf3e188c97ea50411c58da41b45709913a72582443f3ffc0c06a34ec
5
+ SHA512:
6
+ metadata.gz: df1e0f59bfe3cce43db3be4f716b922e1f11e56e9fbd6bc2a669340478d19eec1936a6d26eab6c67adaa582e030a0a3010d0ed1cd295e767c4b1124f5e6f3a81
7
+ data.tar.gz: b6f09e52587bc528b3d9bc8b8292e3f63927b51cb6c9fb58b3128b7e266cfbb6215349a9bc237bdbb6b1ea34e142bcfbaa896d09199512fab1f8f198d46f099e
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ # Ignore dev stuffs
2
+ config/
3
+ coverage/
4
+ db/
5
+ dist/
6
+ lib/java/
7
+ *.dat
8
+ *.gem
9
+ *.lock
10
+ .ruby-version
data/.rspec ADDED
@@ -0,0 +1,2 @@
1
+ --color
2
+ --require spec_helper
data/Gemfile ADDED
@@ -0,0 +1,44 @@
1
+ source "http://rubygems.org"
2
+
3
+ # monkey patching to support dual booting
4
+ module Bundler::SharedHelpers
5
+ def default_lockfile=(path)
6
+ @default_lockfile = path
7
+ end
8
+ def default_lockfile
9
+ @default_lockfile ||= Pathname.new("#{default_gemfile}.lock")
10
+ end
11
+ end
12
+
13
+ module ::Kernel
14
+ def jruby?
15
+ !(RUBY_PLATFORM =~ /java/).nil?
16
+ end
17
+ end
18
+
19
+ if jruby?
20
+ Bundler::SharedHelpers.default_lockfile = Pathname.new("#{Bundler::SharedHelpers.default_gemfile}_jruby.lock")
21
+
22
+ # Bundler::Dsl.evaluate already called with an incorrect lockfile ... fix it
23
+ class Bundler::Dsl
24
+ # A bit messy, this can be called multiple times by bundler, avoid blowing the stack
25
+ unless self.method_defined? :to_definition_unpatched
26
+ alias_method :to_definition_unpatched, :to_definition
27
+ end
28
+ def to_definition(bad_lockfile, unlock)
29
+ to_definition_unpatched(Bundler::SharedHelpers.default_lockfile, unlock)
30
+ end
31
+ end
32
+ end
33
+
34
+ # gemspec
35
+ gemspec
36
+
37
+ group :development do
38
+ gem 'warbler', '~> 2.0.4', platform: :jruby, require: false
39
+ end
40
+
41
+ group :test do
42
+ gem 'rspec', '~> 3.2.0'
43
+ gem 'simplecov', '~> 0.9.2'
44
+ end
data/README.md ADDED
@@ -0,0 +1,73 @@
1
+ # Tapsoob
2
+
3
+ Tapsoob is a simple tool to export and import databases. It is inspired by <https://github.com/ricardochimal/taps> but instead of having to rely on a server and a client, databases are exported to the filesystem or imported from a previous export, hence the OOB (out-of-band). This was done in order to avoid exposing a whole database using the HyperText Protocol for security reasons.
4
+
5
+ Although the code is not quite perfect yet and some improvement will probably be made in the future it's usable. However if you notice an issue don't hesitate to report it.
6
+
7
+
8
+ ## Database support
9
+
10
+ Tapsoob currently rely on the Sequel ORM (<http://sequel.rubyforge.org/>) so we currently support every database that Sequel supports. If additionnal support is required in the future (NoSQL databases) I'll make my best to figure out a way to bring that in my ToDo list.
11
+
12
+ ### Oracle
13
+
14
+ If you're using either Oracle or Oracle XE you will need some extra requirements. If you're using Ruby you'll need to have your ORACLE_HOME environnement variable set properly and the `ruby-oci8` gem installed. However if you're using jRuby you'll need to have the official Oracle JDBC driver (see here for more informations: <http://www.oracle.com/technetwork/articles/dsl/jruby-oracle11g-330825.html>) and it should be loaded prior to using Tapsoob otherwise you won't be able to connect the database.
15
+
16
+
17
+ ## Exporting your data
18
+
19
+ tapsoob pull [OPTIONS] <dump_path> <database_url>
20
+
21
+ The `dump_path` is the location on the filesystem where you want your database exported and the `database_url` is an URL looking like this `postgres://user:password@localhost/blog`, you can find out more about how to connect to your RDBMS by refering yourself to this page: <http://sequel.rubyforge.org/rdoc/files/doc/opening_databases_rdoc.html> on the Sequel documentation.
22
+
23
+ Regarding options a complete list of options can be displayed using the following command:
24
+
25
+ tapsoob pull -h
26
+
27
+
28
+ ## Importing your data
29
+
30
+ tapsoob push [OPTIONS] <dump_path> <database_url>
31
+
32
+ As for exporting the `dump_path` is the path you previously exported a database you now wish to import and the `database_url` should conform to the same format described before.
33
+
34
+ You can list all available options using the command:
35
+
36
+ tapsoob push -h
37
+
38
+
39
+ ## Integration with Rails
40
+
41
+ If you're using Rails, there's also two Rake tasks provided:
42
+
43
+ * `tapsoob:pull` which dumps the database into a new folder under the `db` folder
44
+ * `tapsoob:push` which reads the last dump you made from `tapsoob:pull` from the `db` folder
45
+
46
+
47
+ ## Notes
48
+
49
+ Your exports can be moved from one machine to another for backups or replication, you can also use Tapsoob to switch your RDBMS from one of the supported system to another.
50
+
51
+
52
+ ## ToDo
53
+
54
+ * Add a compression layer
55
+ * Tests (in progress)
56
+
57
+
58
+ ## Contributors
59
+
60
+ * Félix Bellanger <felix.bellanger@gmail.com>
61
+ * Michael Chrisco <michaelachrisco@gmail.com>
62
+
63
+
64
+ ## License
65
+
66
+ The MIT License (MIT)
67
+ Copyright © 2015 Félix Bellanger <felix.bellanger@gmail.com>
68
+
69
+ Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
70
+
71
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
72
+
73
+ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/Rakefile ADDED
@@ -0,0 +1,4 @@
1
+ unless (RUBY_PLATFORM =~ /java/).nil?
2
+ require 'warbler'
3
+ Warbler::Task.new
4
+ end
data/bin/tapsoob ADDED
@@ -0,0 +1,6 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $:.unshift File.dirname(__FILE__) + '/../lib'
4
+ require 'tapsoob/cli'
5
+
6
+ Tapsoob::CLI::Root.start(ARGV)
@@ -0,0 +1,53 @@
1
+ # -*- encoding : utf-8 -*-
2
+ require 'tapsoob/errors'
3
+
4
+ class Tapsoob::Chunksize
5
+ attr_accessor :idle_secs, :time_in_db, :start_time, :end_time, :retries
6
+ attr_reader :chunksize
7
+
8
+ def initialize(chunksize)
9
+ @chunksize = chunksize
10
+ @idle_secs = 0.0
11
+ @retries = 0
12
+ end
13
+
14
+ def to_i
15
+ chunksize
16
+ end
17
+
18
+ def reset_chunksize
19
+ @chunksize = (retries <= 1) ? 10 : 1
20
+ end
21
+
22
+ def diff
23
+ end_time - start_time - time_in_db - idle_secs
24
+ end
25
+
26
+ def time_in_db=(t)
27
+ @time_in_db = t
28
+ @time_in_db = @time_in_db.to_f rescue 0.0
29
+ end
30
+
31
+ def time_delta
32
+ t1 = Time.now
33
+ yield if block_given?
34
+ t2 = Time.now
35
+ t2 - t1
36
+ end
37
+
38
+ def calc_new_chunksize
39
+ new_chunksize = if retries > 0
40
+ chunksize
41
+ elsif diff > 3.0
42
+ (chunksize / 3).ceil
43
+ elsif diff > 1.1
44
+ chunksize - 100
45
+ elsif diff < 0.8
46
+ chunksize * 2
47
+ else
48
+ chunksize + 100
49
+ end
50
+ new_chunksize = 1 if new_chunksize < 1
51
+ new_chunksize
52
+ end
53
+ end
@@ -0,0 +1,192 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'rubygems'
4
+ require 'fileutils'
5
+ require 'sequel'
6
+ require 'tempfile'
7
+ require 'thor'
8
+
9
+ # tapsoob deps
10
+ require 'tapsoob/config'
11
+ require 'tapsoob/log'
12
+ require 'tapsoob/operation'
13
+ require 'tapsoob/schema'
14
+ require 'tapsoob/version'
15
+
16
+ Tapsoob::Config.tapsoob_database_url = ENV['TAPSOOB_DATABASE_URL'] || begin
17
+ # this is dirty but it solves a weird problem where the tempfile disappears mid-process
18
+ #require ((RUBY_PLATFORM =~ /java/).nil? ? 'sqlite3' : 'jdbc-sqlite3')
19
+ $__taps_database = Tempfile.new('tapsoob.db')
20
+ $__taps_database.open()
21
+ "sqlite://#{$__taps_database.path}"
22
+ end
23
+
24
+ module Tapsoob
25
+ module CLI
26
+ class Schema < Thor
27
+ desc "console DATABASE_URL", "Create an IRB REPL connected to a database"
28
+ def console(database_url)
29
+ $db = Sequel.connect(database_url)
30
+ require 'irb'
31
+ require 'irb/completion'
32
+ IRB.start
33
+ end
34
+
35
+ desc "dump DATABASE_URL", "Dump a database using a database URL"
36
+ def dump(database_url)
37
+ puts Tapsoob::Schema.dump(database_url)
38
+ end
39
+
40
+ desc "dump_table DATABASE_URL TABLE", "Dump a table from a database using a database URL"
41
+ def dump_table(database_url, table)
42
+ puts Tapsoob::Schema.dump_table(database_url, table)
43
+ end
44
+
45
+ desc "indexes DATABASE_URL", "Dump indexes from a database using a database URL"
46
+ def indexes(database_url)
47
+ puts Tapsoob::Schema.indexes(database_url)
48
+ end
49
+
50
+ desc "indexes_individual DATABASE_URL", "Dump indexes per table individually using a database URL"
51
+ def indexes_individual(database_url)
52
+ puts Tapsoob::Schema.indexes_individual(database_url)
53
+ end
54
+
55
+ desc "reset_db_sequences DATABASE_URL", "Reset database sequences using a database URL"
56
+ def reset_db_sequences(database_url)
57
+ Tapsoob::Schema.reset_db_sequences(database_url)
58
+ end
59
+
60
+ desc "load DATABASE_URL FILENAME", "Load a database schema from a file to a database using a database URL"
61
+ def load(database_url, filename)
62
+ schema = File.read(filename) rescue help
63
+ Tapsoob::Schema.load(database_url, schema)
64
+ end
65
+
66
+ desc "load_indexes DATABASE_URL FILENAME", "Load indexes from a file to a database using a database URL"
67
+ def load_indexes(database_url, filename)
68
+ indexes = File.read(filename) rescue help
69
+ Tapsoob::Schema.load_indexes(database_url, indexes)
70
+ end
71
+ end
72
+
73
+ class Root < Thor
74
+ desc "pull DUMP_PATH DATABASE_URL", "Pull a dump from a database to a folder"
75
+ option :"skip-schema", desc: "Don't transfer the schema just data", default: false, type: :boolean, aliases: "-s"
76
+ option :"indexes-first", desc: "Transfer indexes first before data", default: false, type: :boolean, aliases: "-i"
77
+ option :resume, desc: "Resume a Tapsoob Session from a stored file", type: :string, aliases: "-r"
78
+ option :chunksize, desc: "Initial chunksize", default: 1000, type: :numeric, aliases: "-c"
79
+ option :"disable-compression", desc: "Disable Compression", default: false, type: :boolean, aliases: "-g"
80
+ option :filter, desc: "Regex Filter for tables", type: :string, aliases: "-f"
81
+ option :tables, desc: "Shortcut to filter on a list of tables", type: :array, aliases: "-t"
82
+ option :"exclude-tables", desc: "Shortcut to exclude a list of tables", type: :array, aliases: "-e"
83
+ option :debug, desc: "Enable debug messages", default: false, type: :boolean, aliases: "-d"
84
+ def pull(dump_path, database_url)
85
+ opts = parse_opts(options)
86
+ Tapsoob.log.level = Logger::DEBUG if opts[:debug]
87
+ if opts[:resume_filename]
88
+ clientresumexfer(:pull, dump_path, database_url, opts)
89
+ else
90
+ clientxfer(:pull, dump_path, database_url, opts)
91
+ end
92
+ end
93
+
94
+ desc "push DUMP_PATH DATABASE_URL", "Push a previously tapsoob dump to a database"
95
+ option :"skip-schema", desc: "Don't transfer the schema just data", default: false, type: :boolean, aliases: "-s"
96
+ option :"indexes-first", desc: "Transfer indexes first before data", default: false, type: :boolean, aliases: "-i"
97
+ option :resume, desc: "Resume a Tapsoob Session from a stored file", type: :string, aliases: "-r"
98
+ option :chunksize, desc: "Initial chunksize", default: 1000, type: :numeric, aliases: "-c"
99
+ option :"disable-compression", desc: "Disable Compression", default: false, type: :boolean, aliases: "-g"
100
+ option :filter, desc: "Regex Filter for tables", type: :string, aliases: "-f"
101
+ option :tables, desc: "Shortcut to filter on a list of tables", type: :array, aliases: "-t"
102
+ option :"exclude-tables", desc: "Shortcut to exclude a list of tables", type: :array, aliases: "-e"
103
+ option :debug, desc: "Enable debug messages", default: false, type: :boolean, aliases: "-d"
104
+ def push(dump_path, database_url)
105
+ opts = parse_opts(options)
106
+ Tapsoob.log.level = Logger::DEBUG if opts[:debug]
107
+ if opts[:resume_filename]
108
+ clientresumexfer(:push, dump_path, database_url, opts)
109
+ else
110
+ clientxfer(:push, dump_path, database_url, opts)
111
+ end
112
+ end
113
+
114
+ desc "version", "Show tapsoob version"
115
+ def version
116
+ puts Tapsoob::VERSION.dup
117
+ end
118
+
119
+ desc "schema SUBCOMMAND ...ARGS", "Direct access to Tapsoob::Schema class methods"
120
+ subcommand "schema", Schema
121
+
122
+ private
123
+ def parse_opts(options)
124
+ # Default options
125
+ opts = {
126
+ skip_schema: options[:"skip-schema"],
127
+ indexes_first: options[:"indexes_first"],
128
+ disable_compression: options[:"disable-compression"],
129
+ debug: options[:debug]
130
+ }
131
+
132
+ # Resume
133
+ if options[:resume]
134
+ if File.exists?(options[:resume])
135
+ opts[:resume_file] = options[:resume]
136
+ else
137
+ raise "Unable to find resume file."
138
+ end
139
+ end
140
+
141
+ # Default chunksize
142
+ if options[:chunksize]
143
+ opts[:default_chunksize] = (options[:chunksize] < 10 ? 10 : options[:chunksize])
144
+ end
145
+
146
+ # Regex filter
147
+ opts[:table_filter] = options[:filter] if options[:filter]
148
+
149
+ # Table filter
150
+ if options[:tables]
151
+ r_tables = options[:tables].collect { |t| "^#{t}" }.join("|")
152
+ opts[:table_filter] = "#{r_tables}"
153
+ end
154
+
155
+ # Exclude tables
156
+ opts[:exclude_tables] = options[:"exclude-tables"] if options[:"exclude-tables"]
157
+
158
+ opts
159
+ end
160
+
161
+ def clientxfer(method, dump_path, database_url, opts)
162
+ Tapsoob::Config.verify_database_url(database_url)
163
+
164
+ FileUtils.mkpath "#{dump_path}/schemas"
165
+ FileUtils.mkpath "#{dump_path}/data"
166
+ FileUtils.mkpath "#{dump_path}/indexes"
167
+
168
+ require 'tapsoob/operation'
169
+
170
+ Tapsoob::Operation.factory(method, database_url, dump_path, opts).run
171
+ end
172
+
173
+ def clientresumexfer(method, dump_path, database_url, opts)
174
+ session = JSON.parse(File.read(opts.delete(:resume_filename)))
175
+ session.symbolize_recursively!
176
+
177
+ dump_path = dump_path || session.delete(:dump_path)
178
+
179
+ require 'taps/operation'
180
+
181
+ newsession = session.merge({
182
+ :default_chunksize => opts[:default_chunksize],
183
+ :disable_compression => opts[:disable_compression],
184
+ :resume => true
185
+ })
186
+
187
+ Tapsoob::Operation.factory(method, database_url, dump_path, newsession).run
188
+ end
189
+ end
190
+ end
191
+ end
192
+
@@ -0,0 +1,33 @@
1
+ # -*- encoding : utf-8 -*-
2
+ require 'sequel'
3
+ require 'tapsoob/version'
4
+
5
+ Sequel.datetime_class = DateTime
6
+
7
+ module Tapsoob
8
+ def self.exiting=(val)
9
+ @@exiting = val
10
+ end
11
+
12
+ def exiting?
13
+ (@@exiting ||= false) == true
14
+ end
15
+
16
+ class Config
17
+ class << self
18
+ attr_accessor :tapsoob_database_url
19
+ attr_accessor :login, :password, :database_url, :dump_path
20
+ attr_accessor :chunksize
21
+
22
+ def verify_database_url(db_url=nil)
23
+ db_url ||= self.database_url
24
+ db = Sequel.connect(db_url)
25
+ db.tables
26
+ db.disconnect
27
+ rescue Object => e
28
+ puts "Failed to connect to database:\n #{e.class} -> #{e}"
29
+ exit 1
30
+ end
31
+ end
32
+ end
33
+ end