gitlab_yaml_db 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,40 @@
1
+ # GitLab fork | YamlDb [![build status](https://secure.travis-ci.org/gitlabhq/yaml_db.png)](https://secure.travis-ci.org/gitlabhq/yaml_db)
2
+
3
+ YamlDb is a database-independent format for dumping and restoring data. It complements the the database-independent schema format found in db/schema.rb. The data is saved into db/data.yml.
4
+
5
+ This can be used as a replacement for mysqldump or pg_dump, but only for the databases typically used by Rails apps. Users, permissions, schemas, triggers, and other advanced database features are not supported - by design.
6
+
7
+ Any database that has an ActiveRecord adapter should work. This gem is now Rails 3 only. For Rails 2, clone and checkout the Rails2 branch.
8
+
9
+ ## Usage
10
+
11
+ rake db:data:dump -> Dump contents of Rails database to db/data.yml
12
+ rake db:data:load -> Load contents of db/data.yml into the database
13
+
14
+ Further, there are tasks db:dump and db:load which do the entire database (the equivalent of running db:schema:dump followed by db:data:load). Also, there are other tasks recently added that allow the export of the database contents to/from multiple files (each one named after the table being dumped or loaded).
15
+
16
+ rake db:data:dump_dir -> Dump contents of database to curr_dir_name/tablename.extension (defaults to yaml)
17
+ rake db:data:load_dir -> Load contents of db/data_dir into database
18
+
19
+ In addition, we have plugins whereby you can export your database to/from various formats. We only deal with yaml and csv right now, but you can easily write tools for your own formats (such as Excel or XML). To use another format, just load setting the "class" parameter to the class you are using. This defaults to "YamlDb::Helper" which is a refactoring of the old yaml_db code. We'll shorten this to use class nicknames in a little bit.
20
+
21
+ ## Examples
22
+
23
+ One common use would be to switch your data from one database backend to another. For example, let's say you wanted to switch from SQLite to MySQL. You might execute the following steps:
24
+
25
+ 1. rake db:dump
26
+
27
+ 2. Edit config/database.yml and change your adapter to mysql, set up database params
28
+
29
+ 3. mysqladmin create [database name]
30
+
31
+ 4. rake db:load
32
+
33
+ ## Credits
34
+
35
+ Created by Orion Henry and Adam Wiggins. Major updates by Ricardo Chimal, Jr.
36
+
37
+ Patches contributed by Michael Irwin, Tom Locke, and Tim Galeckas.
38
+
39
+ Send questions, feedback, or patches to the Heroku mailing list: http://groups.google.com/group/heroku
40
+
@@ -0,0 +1,30 @@
1
+ require 'rake'
2
+ require "rspec/core/rake_task"
3
+
4
+ begin
5
+ require 'jeweler'
6
+ Jeweler::Tasks.new do |gem|
7
+ gem.name = "gitlab_yaml_db"
8
+ gem.summary = %Q{yaml_db allows export/import of database into/from yaml files}
9
+ gem.description = %Q{
10
+ GitLab fork. YamlDb is a database-independent format for dumping and restoring data. It complements the the database-independent schema format found in db/schema.rb. The data is saved into db/data.yml.
11
+ This can be used as a replacement for mysqldump or pg_dump, but only for the databases typically used by Rails apps. Users, permissions, schemas, triggers, and other advanced database features are not supported - by design.
12
+ Any database that has an ActiveRecord adapter should work
13
+ }
14
+ gem.email = "nate@ludicast.com"
15
+ gem.homepage = "http://github.com/ludicast/yaml_db"
16
+ gem.authors = ["Adam Wiggins","Orion Henry"]
17
+ end
18
+ Jeweler::GemcutterTasks.new
19
+ rescue LoadError
20
+ puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
21
+ end
22
+
23
+
24
+ RSpec::Core::RakeTask.new(:spec) do |spec|
25
+ spec.pattern = 'spec/*_spec.rb'
26
+ spec.rspec_opts = ['--backtrace']
27
+ end
28
+
29
+ task :default => :spec
30
+
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 1.0.0
@@ -0,0 +1,5 @@
1
+ author: Orion Henry and Adam Wiggins of Heroku
2
+ summary: Dumps and loads a database-independent data dump format in db/data.yml.
3
+ homepage: http://opensource.heroku.com/
4
+ license: MIT
5
+ rails_version: 1.2+
data/init.rb ADDED
@@ -0,0 +1 @@
1
+ require 'yaml_db'
@@ -0,0 +1,78 @@
1
+ #require 'FasterCSV'
2
+ module CsvDb
3
+ module Helper
4
+ def self.loader
5
+ Load
6
+ end
7
+
8
+ def self.dumper
9
+ Dump
10
+ end
11
+
12
+ def self.extension
13
+ "csv"
14
+ end
15
+ end
16
+
17
+ class Load < SerializationHelper::Load
18
+ def self.load_documents(io, truncate = true)
19
+ tables = {}
20
+ curr_table = nil
21
+ io.each do |line|
22
+ if /BEGIN_CSV_TABLE_DECLARATION(.+)END_CSV_TABLE_DECLARATION/ =~ line
23
+ curr_table = $1
24
+ tables[curr_table] = {}
25
+ else
26
+ if tables[curr_table]["columns"]
27
+ tables[curr_table]["records"] << FasterCSV.parse(line)[0]
28
+ else
29
+ tables[curr_table]["columns"] = FasterCSV.parse(line)[0]
30
+ tables[curr_table]["records"] = []
31
+ end
32
+ end
33
+ end
34
+
35
+ tables.each_pair do |table_name, contents|
36
+ load_table(table_name, contents, truncate)
37
+ end
38
+ end
39
+ end
40
+
41
+ class Dump < SerializationHelper::Dump
42
+
43
+ def self.before_table(io,table)
44
+ io.write "BEGIN_CSV_TABLE_DECLARATION#{table}END_CSV_TABLE_DECLARATION\n"
45
+ end
46
+
47
+ def self.dump(io)
48
+ tables.each do |table|
49
+ before_table(io, table)
50
+ dump_table(io, table)
51
+ after_table(io, table)
52
+ end
53
+ end
54
+
55
+ def self.after_table(io,table)
56
+ io.write ""
57
+ end
58
+
59
+ def self.dump_table_columns(io, table)
60
+ io.write(table_column_names(table).to_csv)
61
+ end
62
+
63
+ def self.dump_table_records(io, table)
64
+
65
+ column_names = table_column_names(table)
66
+
67
+ each_table_page(table) do |records|
68
+ rows = SerializationHelper::Utils.unhash_records(records, column_names)
69
+ records.each do |record|
70
+ io.write(record.to_csv)
71
+ end
72
+ end
73
+ end
74
+
75
+ end
76
+
77
+
78
+ end
@@ -0,0 +1,197 @@
1
+ module SerializationHelper
2
+
3
+ class Base
4
+ attr_reader :extension
5
+
6
+ def initialize(helper)
7
+ @dumper = helper.dumper
8
+ @loader = helper.loader
9
+ @extension = helper.extension
10
+ end
11
+
12
+ def dump(filename)
13
+ disable_logger
14
+ @dumper.dump(File.new(filename, "w"))
15
+ reenable_logger
16
+ end
17
+
18
+ def dump_to_dir(dirname)
19
+ Dir.mkdir(dirname)
20
+ tables = @dumper.tables
21
+ tables.each do |table|
22
+ io = File.new "#{dirname}/#{table}.#{@extension}", "w"
23
+ @dumper.before_table(io, table)
24
+ @dumper.dump_table io, table
25
+ @dumper.after_table(io, table)
26
+ end
27
+ end
28
+
29
+ def load(filename, truncate = true)
30
+ disable_logger
31
+ @loader.load(File.new(filename, "r"), truncate)
32
+ reenable_logger
33
+ end
34
+
35
+ def load_from_dir(dirname, truncate = true)
36
+ Dir.entries(dirname).each do |filename|
37
+ if filename =~ /^[.]/
38
+ next
39
+ end
40
+ @loader.load(File.new("#{dirname}/#{filename}", "r"), truncate)
41
+ end
42
+ end
43
+
44
+ def disable_logger
45
+ @@old_logger = ActiveRecord::Base.logger
46
+ ActiveRecord::Base.logger = nil
47
+ end
48
+
49
+ def reenable_logger
50
+ ActiveRecord::Base.logger = @@old_logger
51
+ end
52
+ end
53
+
54
+ class Load
55
+ def self.load(io, truncate = true)
56
+ ActiveRecord::Base.connection.transaction do
57
+ load_documents(io, truncate)
58
+ end
59
+ end
60
+
61
+ def self.truncate_table(table)
62
+ begin
63
+ ActiveRecord::Base.connection.execute("TRUNCATE #{SerializationHelper::Utils.quote_table(table)}")
64
+ rescue Exception
65
+ ActiveRecord::Base.connection.execute("DELETE FROM #{SerializationHelper::Utils.quote_table(table)}")
66
+ end
67
+ end
68
+
69
+ def self.load_table(table, data, truncate = true)
70
+ column_names = data['columns']
71
+ if truncate
72
+ truncate_table(table)
73
+ end
74
+ load_records(table, column_names, data['records'])
75
+ reset_pk_sequence!(table)
76
+ end
77
+
78
+ def self.load_records(table, column_names, records)
79
+ if column_names.nil?
80
+ return
81
+ end
82
+ columns = column_names.map{|cn| ActiveRecord::Base.connection.columns(table).detect{|c| c.name == cn}}
83
+ quoted_column_names = column_names.map { |column| ActiveRecord::Base.connection.quote_column_name(column) }.join(',')
84
+ quoted_table_name = SerializationHelper::Utils.quote_table(table)
85
+ records.each do |record|
86
+ quoted_values = record.zip(columns).map{|c| ActiveRecord::Base.connection.quote(c.first, c.last)}.join(',')
87
+ ActiveRecord::Base.connection.execute("INSERT INTO #{quoted_table_name} (#{quoted_column_names}) VALUES (#{quoted_values})")
88
+ end
89
+ end
90
+
91
+ def self.reset_pk_sequence!(table_name)
92
+ if ActiveRecord::Base.connection.respond_to?(:reset_pk_sequence!)
93
+ ActiveRecord::Base.connection.reset_pk_sequence!(table_name)
94
+ end
95
+ end
96
+
97
+
98
+ end
99
+
100
+ module Utils
101
+
102
+ def self.unhash(hash, keys)
103
+ keys.map { |key| hash[key] }
104
+ end
105
+
106
+ def self.unhash_records(records, keys)
107
+ records.each_with_index do |record, index|
108
+ records[index] = unhash(record, keys)
109
+ end
110
+
111
+ records
112
+ end
113
+
114
+ def self.convert_booleans(records, columns)
115
+ records.each do |record|
116
+ columns.each do |column|
117
+ next if is_boolean(record[column])
118
+ record[column] = convert_boolean(record[column])
119
+ end
120
+ end
121
+ records
122
+ end
123
+
124
+ def self.convert_boolean(value)
125
+ ['t', '1', true, 1].include?(value)
126
+ end
127
+
128
+ def self.boolean_columns(table)
129
+ columns = ActiveRecord::Base.connection.columns(table).reject { |c| silence_warnings { c.type != :boolean } }
130
+ columns.map { |c| c.name }
131
+ end
132
+
133
+ def self.is_boolean(value)
134
+ value.kind_of?(TrueClass) or value.kind_of?(FalseClass)
135
+ end
136
+
137
+ def self.quote_table(table)
138
+ ActiveRecord::Base.connection.quote_table_name(table)
139
+ end
140
+
141
+ end
142
+
143
+ class Dump
144
+ def self.before_table(io, table)
145
+
146
+ end
147
+
148
+ def self.dump(io)
149
+ tables.each do |table|
150
+ before_table(io, table)
151
+ dump_table(io, table)
152
+ after_table(io, table)
153
+ end
154
+ end
155
+
156
+ def self.after_table(io, table)
157
+
158
+ end
159
+
160
+ def self.tables
161
+ ActiveRecord::Base.connection.tables.reject { |table| ['schema_info', 'schema_migrations'].include?(table) }
162
+ end
163
+
164
+ def self.dump_table(io, table)
165
+ return if table_record_count(table).zero?
166
+
167
+ dump_table_columns(io, table)
168
+ dump_table_records(io, table)
169
+ end
170
+
171
+ def self.table_column_names(table)
172
+ ActiveRecord::Base.connection.columns(table).map { |c| c.name }
173
+ end
174
+
175
+
176
+ def self.each_table_page(table, records_per_page=1000)
177
+ total_count = table_record_count(table)
178
+ pages = (total_count.to_f / records_per_page).ceil - 1
179
+ id = table_column_names(table).first
180
+ boolean_columns = SerializationHelper::Utils.boolean_columns(table)
181
+ quoted_table_name = SerializationHelper::Utils.quote_table(table)
182
+
183
+ (0..pages).to_a.each do |page|
184
+ query = Arel::Table.new(table).order(id).skip(records_per_page*page).take(records_per_page).project(Arel.sql('*'))
185
+ records = ActiveRecord::Base.connection.select_all(query)
186
+ records = SerializationHelper::Utils.convert_booleans(records, boolean_columns)
187
+ yield records
188
+ end
189
+ end
190
+
191
+ def self.table_record_count(table)
192
+ ActiveRecord::Base.connection.select_one("SELECT COUNT(*) FROM #{SerializationHelper::Utils.quote_table(table)}").values.first.to_i
193
+ end
194
+
195
+ end
196
+
197
+ end
@@ -0,0 +1,45 @@
1
+ namespace :db do
2
+ desc "Dump schema and data to db/schema.rb and db/data.yml"
3
+ task(:dump => [ "db:schema:dump", "db:data:dump" ])
4
+
5
+ desc "Load schema and data from db/schema.rb and db/data.yml"
6
+ task(:load => [ "db:schema:load", "db:data:load" ])
7
+
8
+ namespace :data do
9
+ def db_dump_data_file (extension = "yml")
10
+ "#{dump_dir}/data.#{extension}"
11
+ end
12
+
13
+ def dump_dir(dir = "")
14
+ "#{Rails.root}/db#{dir}"
15
+ end
16
+
17
+ desc "Dump contents of database to db/data.extension (defaults to yaml)"
18
+ task :dump => :environment do
19
+ format_class = ENV['class'] || "YamlDb::Helper"
20
+ helper = format_class.constantize
21
+ SerializationHelper::Base.new(helper).dump db_dump_data_file helper.extension
22
+ end
23
+
24
+ desc "Dump contents of database to curr_dir_name/tablename.extension (defaults to yaml)"
25
+ task :dump_dir => :environment do
26
+ format_class = ENV['class'] || "YamlDb::Helper"
27
+ dir = ENV['dir'] || "#{Time.now.to_s.gsub(/ /, '_')}"
28
+ SerializationHelper::Base.new(format_class.constantize).dump_to_dir dump_dir("/#{dir}")
29
+ end
30
+
31
+ desc "Load contents of db/data.extension (defaults to yaml) into database"
32
+ task :load => :environment do
33
+ format_class = ENV['class'] || "YamlDb::Helper"
34
+ helper = format_class.constantize
35
+ SerializationHelper::Base.new(helper).load(db_dump_data_file helper.extension)
36
+ end
37
+
38
+ desc "Load contents of db/data_dir into database"
39
+ task :load_dir => :environment do
40
+ dir = ENV['dir'] || "base"
41
+ format_class = ENV['class'] || "YamlDb::Helper"
42
+ SerializationHelper::Base.new(format_class.constantize).load_from_dir dump_dir("/#{dir}")
43
+ end
44
+ end
45
+ end
@@ -0,0 +1,76 @@
1
+ require 'rubygems'
2
+ require 'yaml'
3
+ require 'active_record'
4
+ require 'serialization_helper'
5
+ require 'active_support/core_ext/kernel/reporting'
6
+ require 'rails/railtie'
7
+
8
+ module YamlDb
9
+ module Helper
10
+ def self.loader
11
+ YamlDb::Load
12
+ end
13
+
14
+ def self.dumper
15
+ YamlDb::Dump
16
+ end
17
+
18
+ def self.extension
19
+ "yml"
20
+ end
21
+ end
22
+
23
+
24
+ module Utils
25
+ def self.chunk_records(records)
26
+ yaml = [ records ].to_yaml
27
+ yaml.sub!(/---\s\n|---\n/, '')
28
+ yaml.sub!('- - -', ' - -')
29
+ yaml
30
+ end
31
+
32
+ end
33
+
34
+ class Dump < SerializationHelper::Dump
35
+
36
+ def self.dump_table_columns(io, table)
37
+ io.write("\n")
38
+ io.write({ table => { 'columns' => table_column_names(table) } }.to_yaml)
39
+ end
40
+
41
+ def self.dump_table_records(io, table)
42
+ table_record_header(io)
43
+
44
+ column_names = table_column_names(table)
45
+
46
+ each_table_page(table) do |records|
47
+ rows = SerializationHelper::Utils.unhash_records(records, column_names)
48
+ io.write(YamlDb::Utils.chunk_records(records))
49
+ end
50
+ end
51
+
52
+ def self.table_record_header(io)
53
+ io.write(" records: \n")
54
+ end
55
+
56
+ end
57
+
58
+ class Load < SerializationHelper::Load
59
+ def self.load_documents(io, truncate = true)
60
+ YAML.load_documents(io) do |ydoc|
61
+ ydoc.keys.each do |table_name|
62
+ next if ydoc[table_name].nil?
63
+ load_table(table_name, ydoc[table_name], truncate)
64
+ end
65
+ end
66
+ end
67
+ end
68
+
69
+ class Railtie < Rails::Railtie
70
+ rake_tasks do
71
+ load File.expand_path('../tasks/yaml_db_tasks.rake',
72
+ __FILE__)
73
+ end
74
+ end
75
+
76
+ end
@@ -0,0 +1,4 @@
1
+ $LOAD_PATH.unshift(File.dirname(__FILE__))
2
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
3
+ require 'rspec'
4
+ require 'yaml_db'
@@ -0,0 +1,56 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe SerializationHelper::Base do
4
+ def prestub_active_record
5
+
6
+ end
7
+
8
+ before do
9
+ @io = StringIO.new
10
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
11
+ ActiveRecord::Base.stub(:connection).and_return(mock('connection'))
12
+ ActiveRecord::Base.connection.stub!(:tables).and_return([ 'mytable', 'schema_info', 'schema_migrations' ])
13
+ end
14
+
15
+ def stub_helper!
16
+ @helper = mock("MyHelper")
17
+ @dumper = mock("MyDumper");
18
+ @loader = mock("MyLoader");
19
+ @helper.stub!(:dumper).and_return(@dumper)
20
+ @helper.stub!(:loader).and_return(@loader)
21
+ @helper.stub!(:extension).and_return("yml")
22
+ @dumper.stub!(:tables).and_return([ActiveRecord::Base.connection.tables[0]])
23
+ @dumper.stub!(:before_table).and_return(nil)
24
+ @dumper.stub!(:after_table).and_return(nil)
25
+ end
26
+
27
+ context "for multi-file dumps" do
28
+ before do
29
+ File.should_receive(:new).once.with("dir_name/mytable.yml", "w").and_return(@io)
30
+ Dir.should_receive(:mkdir).once.with("dir_name")
31
+ stub_helper!
32
+ @dumper.should_receive(:dump_table).once.with(@io, "mytable")
33
+ end
34
+
35
+ it "should create the number of files that there are tables" do
36
+ SerializationHelper::Base.new(@helper).dump_to_dir "dir_name"
37
+ end
38
+
39
+ end
40
+
41
+ context "for multi-file loads" do
42
+
43
+ before do
44
+ stub_helper!
45
+ @loader.should_receive(:load).once.with(@io, true)
46
+ File.should_receive(:new).once.with("dir_name/mytable.yml", "r").and_return(@io)
47
+ Dir.stub!(:entries).and_return(["mytable.yml"])
48
+ end
49
+
50
+ it "should insert into then umber of tables that there are files" do
51
+ SerializationHelper::Base.new(@helper).load_from_dir "dir_name"
52
+ end
53
+
54
+ end
55
+
56
+ end
@@ -0,0 +1,65 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe SerializationHelper::Dump do
4
+
5
+ before do
6
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
7
+ ActiveRecord::Base.stub(:connection).and_return(stub('connection').as_null_object)
8
+ ActiveRecord::Base.connection.stub!(:tables).and_return([ 'mytable', 'schema_info', 'schema_migrations' ])
9
+ ActiveRecord::Base.connection.stub!(:columns).with('mytable').and_return([ mock('a', :name => 'a', :type => :string), mock('b', :name => 'b', :type => :string) ])
10
+ ActiveRecord::Base.connection.stub!(:select_one).and_return({"count"=>"2"})
11
+ ActiveRecord::Base.connection.stub!(:select_all).and_return([ { 'a' => 1, 'b' => 2 }, { 'a' => 3, 'b' => 4 } ])
12
+ SerializationHelper::Utils.stub!(:quote_table).with('mytable').and_return('mytable')
13
+ end
14
+
15
+ before(:each) do
16
+ File.stub!(:new).with('dump.yml', 'w').and_return(StringIO.new)
17
+ @io = StringIO.new
18
+ end
19
+
20
+ it "should return a list of column names" do
21
+ SerializationHelper::Dump.table_column_names('mytable').should == [ 'a', 'b' ]
22
+ end
23
+
24
+ it "should return a list of tables without the rails schema table" do
25
+ SerializationHelper::Dump.tables.should == ['mytable']
26
+ end
27
+
28
+ it "should return the total number of records in a table" do
29
+ SerializationHelper::Dump.table_record_count('mytable').should == 2
30
+ end
31
+
32
+ it "should return all records from the database and return them when there is only 1 page" do
33
+ SerializationHelper::Dump.each_table_page('mytable') do |records|
34
+ records.should == [ { 'a' => 1, 'b' => 2 }, { 'a' => 3, 'b' => 4 } ]
35
+ end
36
+ end
37
+
38
+ it "should paginate records from the database and return them" do
39
+ ActiveRecord::Base.connection.stub!(:select_all).and_return([ { 'a' => 1, 'b' => 2 } ], [ { 'a' => 3, 'b' => 4 } ])
40
+
41
+ records = [ ]
42
+ SerializationHelper::Dump.each_table_page('mytable', 1) do |page|
43
+ page.size.should == 1
44
+ records.concat(page)
45
+ end
46
+
47
+ records.should == [ { 'a' => 1, 'b' => 2 }, { 'a' => 3, 'b' => 4 } ]
48
+ end
49
+
50
+ it "should dump a table's contents to yaml" do
51
+ SerializationHelper::Dump.should_receive(:dump_table_columns)
52
+ SerializationHelper::Dump.should_receive(:dump_table_records)
53
+ SerializationHelper::Dump.dump_table(@io, 'mytable')
54
+ end
55
+
56
+ it "should not dump a table's contents when the record count is zero" do
57
+ SerializationHelper::Dump.stub!(:table_record_count).with('mytable').and_return(0)
58
+ SerializationHelper::Dump.should_not_receive(:dump_table_columns)
59
+ SerializationHelper::Dump.should_not_receive(:dump_table_records)
60
+ SerializationHelper::Dump.dump_table(@io, 'mytable')
61
+ end
62
+
63
+
64
+
65
+ end
@@ -0,0 +1,77 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe SerializationHelper::Load do
4
+ before do
5
+ SerializationHelper::Utils.stub!(:quote_table).with('mytable').and_return('mytable')
6
+
7
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
8
+ ActiveRecord::Base.stub(:connection).and_return(stub('connection').as_null_object)
9
+ ActiveRecord::Base.connection.stub!(:transaction).and_yield
10
+ @io = StringIO.new
11
+ end
12
+
13
+ it "should truncate the table" do
14
+ ActiveRecord::Base.connection.stub!(:execute).with("TRUNCATE mytable").and_return(true)
15
+ ActiveRecord::Base.connection.should_not_receive(:execute).with("DELETE FROM mytable")
16
+ SerializationHelper::Load.truncate_table('mytable')
17
+ end
18
+
19
+ it "should delete the table if truncate throws an exception" do
20
+ ActiveRecord::Base.connection.should_receive(:execute).with("TRUNCATE mytable").and_raise()
21
+ ActiveRecord::Base.connection.should_receive(:execute).with("DELETE FROM mytable").and_return(true)
22
+ SerializationHelper::Load.truncate_table('mytable')
23
+ end
24
+
25
+
26
+ it "should call reset pk sequence if the connection adapter is postgres" do
27
+ ActiveRecord::Base.connection.should_receive(:respond_to?).with(:reset_pk_sequence!).and_return(true)
28
+ ActiveRecord::Base.connection.should_receive(:reset_pk_sequence!).with('mytable')
29
+ SerializationHelper::Load.reset_pk_sequence!('mytable')
30
+ end
31
+
32
+ it "should not call reset pk sequence for other adapters" do
33
+ ActiveRecord::Base.connection.should_receive(:respond_to?).with(:reset_pk_sequence!).and_return(false)
34
+ ActiveRecord::Base.connection.should_not_receive(:reset_pk_sequence!)
35
+ SerializationHelper::Load.reset_pk_sequence!('mytable')
36
+ end
37
+
38
+ it "should insert records into a table" do
39
+ mca = mock('a',:name => 'a')
40
+ mcb = mock('b', :name => 'b')
41
+ ActiveRecord::Base.connection.stub!(:columns).with('mytable').and_return([mca , mcb ])
42
+ ActiveRecord::Base.connection.stub!(:quote_column_name).with('a').and_return('a')
43
+ ActiveRecord::Base.connection.stub!(:quote_column_name).with('b').and_return('b')
44
+ ActiveRecord::Base.connection.stub!(:quote).with(1, mca).and_return("'1'")
45
+ ActiveRecord::Base.connection.stub!(:quote).with(2, mcb).and_return("'2'")
46
+ ActiveRecord::Base.connection.stub!(:quote).with(3, mca).and_return("'3'")
47
+ ActiveRecord::Base.connection.stub!(:quote).with(4, mcb).and_return("'4'")
48
+ ActiveRecord::Base.connection.should_receive(:execute).with("INSERT INTO mytable (a,b) VALUES ('1','2')")
49
+ ActiveRecord::Base.connection.should_receive(:execute).with("INSERT INTO mytable (a,b) VALUES ('3','4')")
50
+
51
+ SerializationHelper::Load.load_records('mytable', ['a', 'b'], [[1, 2], [3, 4]])
52
+ end
53
+
54
+ it "should quote column names that correspond to sql keywords" do
55
+ mca = mock('a',:name => 'a')
56
+ mccount = mock('count', :name => 'count')
57
+ ActiveRecord::Base.connection.stub!(:columns).with('mytable').and_return([mca , mccount ])
58
+ ActiveRecord::Base.connection.stub!(:quote_column_name).with('a').and_return('a')
59
+ ActiveRecord::Base.connection.stub!(:quote_column_name).with('count').and_return('"count"')
60
+ ActiveRecord::Base.connection.stub!(:quote).with(1, mca).and_return("'1'")
61
+ ActiveRecord::Base.connection.stub!(:quote).with(2, mccount).and_return("'2'")
62
+ ActiveRecord::Base.connection.stub!(:quote).with(3, mca).and_return("'3'")
63
+ ActiveRecord::Base.connection.stub!(:quote).with(4, mccount).and_return("'4'")
64
+ ActiveRecord::Base.connection.should_receive(:execute).with("INSERT INTO mytable (a,\"count\") VALUES ('1','2')")
65
+ ActiveRecord::Base.connection.should_receive(:execute).with("INSERT INTO mytable (a,\"count\") VALUES ('3','4')")
66
+
67
+ SerializationHelper::Load.load_records('mytable', ['a', 'count'], [[1, 2], [3, 4]])
68
+ end
69
+
70
+ it "should truncate the table and then load the records into the table" do
71
+ SerializationHelper::Load.should_receive(:truncate_table).with('mytable')
72
+ SerializationHelper::Load.should_receive(:load_records).with('mytable', ['a', 'b'], [[1, 2], [3, 4]])
73
+ SerializationHelper::Load.should_receive(:reset_pk_sequence!).with('mytable')
74
+
75
+ SerializationHelper::Load.load_table('mytable', { 'columns' => [ 'a', 'b' ], 'records' => [[1, 2], [3, 4]] })
76
+ end
77
+ end
@@ -0,0 +1,51 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe SerializationHelper::Utils, " convert records utility method" do
4
+ before do
5
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
6
+ ActiveRecord::Base.stub(:connection).and_return(stub('connection').as_null_object)
7
+ end
8
+
9
+ it "returns an array of hash values using an array of ordered keys" do
10
+ SerializationHelper::Utils.unhash({ 'a' => 1, 'b' => 2 }, [ 'b', 'a' ]).should == [ 2, 1 ]
11
+ end
12
+
13
+ it "should unhash each hash an array using an array of ordered keys" do
14
+ SerializationHelper::Utils.unhash_records([ { 'a' => 1, 'b' => 2 }, { 'a' => 3, 'b' => 4 } ], [ 'b', 'a' ]).should == [ [ 2, 1 ], [ 4, 3 ] ]
15
+ end
16
+
17
+ it "should return true if it is a boolean type" do
18
+ SerializationHelper::Utils.is_boolean(true).should == true
19
+ SerializationHelper::Utils.is_boolean('true').should_not == true
20
+ end
21
+
22
+ it "should return an array of boolean columns" do
23
+ ActiveRecord::Base.connection.stub!(:columns).with('mytable').and_return([ mock('a',:name => 'a',:type => :string), mock('b', :name => 'b',:type => :boolean) ])
24
+ SerializationHelper::Utils.boolean_columns('mytable').should == ['b']
25
+ end
26
+
27
+ it "should quote the table name" do
28
+ ActiveRecord::Base.connection.should_receive(:quote_table_name).with('values').and_return('`values`')
29
+ SerializationHelper::Utils.quote_table('values').should == '`values`'
30
+ end
31
+
32
+ it "should convert ruby booleans to true and false" do
33
+ SerializationHelper::Utils.convert_boolean(true).should == true
34
+ SerializationHelper::Utils.convert_boolean(false).should == false
35
+ end
36
+
37
+ it "should convert ruby string t and f to true and false" do
38
+ SerializationHelper::Utils.convert_boolean('t').should == true
39
+ SerializationHelper::Utils.convert_boolean('f').should == false
40
+ end
41
+
42
+ it "should convert ruby string 1 and 0 to true and false" do
43
+ SerializationHelper::Utils.convert_boolean('1').should == true
44
+ SerializationHelper::Utils.convert_boolean('0').should == false
45
+ end
46
+
47
+ it "should convert ruby integer 1 and 0 to true and false" do
48
+ SerializationHelper::Utils.convert_boolean(1).should == true
49
+ SerializationHelper::Utils.convert_boolean(0).should == false
50
+ end
51
+ end
@@ -0,0 +1,50 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe YamlDb::Dump do
4
+ before do
5
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
6
+ ActiveRecord::Base.stub(:connection).and_return(stub('connection').as_null_object)
7
+ ActiveRecord::Base.connection.stub!(:tables).and_return([ 'mytable', 'schema_info', 'schema_migrations' ])
8
+ ActiveRecord::Base.connection.stub!(:columns).with('mytable').and_return([ mock('a',:name => 'a', :type => :string), mock('b', :name => 'b', :type => :string) ])
9
+ ActiveRecord::Base.connection.stub!(:select_one).and_return({"count"=>"2"})
10
+ ActiveRecord::Base.connection.stub!(:select_all).and_return([ { 'a' => 1, 'b' => 2 }, { 'a' => 3, 'b' => 4 } ])
11
+ YamlDb::Utils.stub!(:quote_table).with('mytable').and_return('mytable')
12
+ end
13
+
14
+ before(:each) do
15
+ File.stub!(:new).with('dump.yml', 'w').and_return(StringIO.new)
16
+ @io = StringIO.new
17
+ end
18
+
19
+ it "should return a formatted string" do
20
+ YamlDb::Dump.table_record_header(@io)
21
+ @io.rewind
22
+ @io.read.should == " records: \n"
23
+ end
24
+
25
+ it "should return a yaml string that contains a table header and column names" do
26
+ YamlDb::Dump.stub!(:table_column_names).with('mytable').and_return([ 'a', 'b' ])
27
+ YamlDb::Dump.dump_table_columns(@io, 'mytable')
28
+ @io.rewind
29
+ @io.read.should == <<EOYAML
30
+
31
+ ---
32
+ mytable:
33
+ columns:
34
+ - a
35
+ - b
36
+ EOYAML
37
+ end
38
+
39
+ it "should return dump the records for a table in yaml to a given io stream" do
40
+ YamlDb::Dump.dump_table_records(@io, 'mytable')
41
+ @io.rewind
42
+ @io.read.should == <<EOYAML
43
+ records:
44
+ - - 1
45
+ - 2
46
+ - - 3
47
+ - 4
48
+ EOYAML
49
+ end
50
+ end
@@ -0,0 +1,33 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+ require 'active_support/core_ext/kernel/debugger'
3
+
4
+ describe YamlDb::Load do
5
+ before do
6
+ SerializationHelper::Utils.stub!(:quote_table).with('mytable').and_return('mytable')
7
+
8
+ silence_warnings { ActiveRecord::Base = mock('ActiveRecord::Base', :null_object => true) }
9
+ ActiveRecord::Base.stub(:connection).and_return(stub('connection').as_null_object)
10
+ ActiveRecord::Base.connection.stub!(:transaction).and_yield
11
+ end
12
+
13
+ before(:each) do
14
+ @io = StringIO.new
15
+ end
16
+
17
+
18
+ it "should call load structure for each document in the file" do
19
+ YAML.should_receive(:load_documents).with(@io).and_yield({ 'mytable' => {
20
+ 'columns' => [ 'a', 'b' ],
21
+ 'records' => [[1, 2], [3, 4]]
22
+ } } )
23
+ YamlDb::Load.should_receive(:load_table).with('mytable', { 'columns' => [ 'a', 'b' ], 'records' => [[1, 2], [3, 4]] },true)
24
+ YamlDb::Load.load(@io)
25
+ end
26
+
27
+ it "should not call load structure when the document in the file contains no records" do
28
+ YAML.should_receive(:load_documents).with(@io).and_yield({ 'mytable' => nil })
29
+ YamlDb::Load.should_not_receive(:load_table)
30
+ YamlDb::Load.load(@io)
31
+ end
32
+
33
+ end
@@ -0,0 +1,21 @@
1
+ require File.dirname(__FILE__) + '/base'
2
+
3
+ describe YamlDb::Utils, " convert records utility method" do
4
+
5
+ it "turns an array with one record into a yaml chunk" do
6
+ YamlDb::Utils.chunk_records([ %w(a b) ]).should == <<EOYAML
7
+ - - a
8
+ - b
9
+ EOYAML
10
+ end
11
+
12
+ it "turns an array with two records into a yaml chunk" do
13
+ YamlDb::Utils.chunk_records([ %w(a b), %w(x y) ]).should == <<EOYAML
14
+ - - a
15
+ - b
16
+ - - x
17
+ - y
18
+ EOYAML
19
+ end
20
+
21
+ end
metadata ADDED
@@ -0,0 +1,139 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: gitlab_yaml_db
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ prerelease:
6
+ platform: ruby
7
+ authors:
8
+ - Adam Wiggins
9
+ - Orion Henry
10
+ autorequire:
11
+ bindir: bin
12
+ cert_chain: []
13
+ date: 2012-12-28 00:00:00.000000000 Z
14
+ dependencies:
15
+ - !ruby/object:Gem::Dependency
16
+ name: jeweler
17
+ requirement: !ruby/object:Gem::Requirement
18
+ none: false
19
+ requirements:
20
+ - - ! '>='
21
+ - !ruby/object:Gem::Version
22
+ version: '0'
23
+ type: :development
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ none: false
27
+ requirements:
28
+ - - ! '>='
29
+ - !ruby/object:Gem::Version
30
+ version: '0'
31
+ - !ruby/object:Gem::Dependency
32
+ name: rspec
33
+ requirement: !ruby/object:Gem::Requirement
34
+ none: false
35
+ requirements:
36
+ - - ! '>='
37
+ - !ruby/object:Gem::Version
38
+ version: '0'
39
+ type: :development
40
+ prerelease: false
41
+ version_requirements: !ruby/object:Gem::Requirement
42
+ none: false
43
+ requirements:
44
+ - - ! '>='
45
+ - !ruby/object:Gem::Version
46
+ version: '0'
47
+ - !ruby/object:Gem::Dependency
48
+ name: rake
49
+ requirement: !ruby/object:Gem::Requirement
50
+ none: false
51
+ requirements:
52
+ - - ! '>='
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ type: :development
56
+ prerelease: false
57
+ version_requirements: !ruby/object:Gem::Requirement
58
+ none: false
59
+ requirements:
60
+ - - ! '>='
61
+ - !ruby/object:Gem::Version
62
+ version: '0'
63
+ - !ruby/object:Gem::Dependency
64
+ name: rails
65
+ requirement: !ruby/object:Gem::Requirement
66
+ none: false
67
+ requirements:
68
+ - - ! '>='
69
+ - !ruby/object:Gem::Version
70
+ version: '0'
71
+ type: :development
72
+ prerelease: false
73
+ version_requirements: !ruby/object:Gem::Requirement
74
+ none: false
75
+ requirements:
76
+ - - ! '>='
77
+ - !ruby/object:Gem::Version
78
+ version: '0'
79
+ description: ! '
80
+
81
+ GitLab fork. YamlDb is a database-independent format for dumping and restoring data. It
82
+ complements the the database-independent schema format found in db/schema.rb. The
83
+ data is saved into db/data.yml.
84
+
85
+ This can be used as a replacement for mysqldump or pg_dump, but only for the databases
86
+ typically used by Rails apps. Users, permissions, schemas, triggers, and other
87
+ advanced database features are not supported - by design.
88
+
89
+ Any database that has an ActiveRecord adapter should work
90
+
91
+ '
92
+ email: nate@ludicast.com
93
+ executables: []
94
+ extensions: []
95
+ extra_rdoc_files:
96
+ - README.markdown
97
+ files:
98
+ - README.markdown
99
+ - Rakefile
100
+ - VERSION
101
+ - about.yml
102
+ - init.rb
103
+ - lib/csv_db.rb
104
+ - lib/serialization_helper.rb
105
+ - lib/tasks/yaml_db_tasks.rake
106
+ - lib/yaml_db.rb
107
+ - spec/base.rb
108
+ - spec/serialization_helper_base_spec.rb
109
+ - spec/serialization_helper_dump_spec.rb
110
+ - spec/serialization_helper_load_spec.rb
111
+ - spec/serialization_utils_spec.rb
112
+ - spec/yaml_dump_spec.rb
113
+ - spec/yaml_load_spec.rb
114
+ - spec/yaml_utils_spec.rb
115
+ homepage: http://github.com/ludicast/yaml_db
116
+ licenses: []
117
+ post_install_message:
118
+ rdoc_options: []
119
+ require_paths:
120
+ - lib
121
+ required_ruby_version: !ruby/object:Gem::Requirement
122
+ none: false
123
+ requirements:
124
+ - - ! '>='
125
+ - !ruby/object:Gem::Version
126
+ version: '0'
127
+ required_rubygems_version: !ruby/object:Gem::Requirement
128
+ none: false
129
+ requirements:
130
+ - - ! '>='
131
+ - !ruby/object:Gem::Version
132
+ version: '0'
133
+ requirements: []
134
+ rubyforge_project:
135
+ rubygems_version: 1.8.23
136
+ signing_key:
137
+ specification_version: 3
138
+ summary: yaml_db allows export/import of database into/from yaml files
139
+ test_files: []