squealer 1.0.2 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +1 -0
- data/.rvmrc +3 -0
- data/README.md +31 -14
- data/Rakefile +7 -3
- data/VERSION +1 -1
- data/lib/example_squeal.rb +64 -18
- data/lib/squealer/object.rb +1 -1
- data/lib/squealer/target.rb +54 -6
- data/lib/squealer/time.rb +1 -1
- data/spec/spec_helper.rb +0 -1
- data/spec/squealer/object_spec.rb +6 -4
- data/spec/squealer/target_spec.rb +203 -48
- data/squealer.gemspec +5 -4
- metadata +14 -4
data/.gitignore
CHANGED
data/.rvmrc
ADDED
data/README.md
CHANGED
@@ -1,30 +1,47 @@
|
|
1
1
|
# Squealer
|
2
2
|
|
3
|
-
##
|
4
|
-
|
5
|
-
|
6
|
-
* `Hash#method_missing` - You prefer dot notation. JSON uses dot notation. You are importing from a data store which represents collections as arrays of hashmaps. Dot notation for navigating those collections is convenient. If you use a field name that happens to be a method on Hash you will have to use index notation. (e.g. `kitten.toys` is good, however `kitten.freeze` is not good. Use `kitten['freeze']` instead.)
|
7
|
-
* `NilClass#each` - As you are importing from schemaless repositories and you may be trying to iterate on fields that contain embedded collections, if a specific parent does not contain one of those child collections, the driver will be returning "nil" as the value for that field. Having `NilClass#each` return a `[]` for a nil is convenient, semantically correct in this context, and removes the need for many nil checks in the block you provide to `Object#assign`
|
8
|
-
* `Object` - `#import`, `#export`, `#target`, and `#assign` "keywords" are provided for convenience
|
9
|
-
* `Time#to_s` - As you are exporting to a SQL database, we represent your timestamp in a format that it will parse unequivocally (mongodb stores all temporal data as a timestamp)
|
3
|
+
## Usage
|
4
|
+
See lib/example_squeal.rb for the example squeal.
|
10
5
|
|
11
6
|
To run standalone, simply make your data squeal thusly:
|
12
7
|
|
13
8
|
`ruby example_squeal.rb`
|
14
9
|
|
15
|
-
where the squeal script
|
10
|
+
where the squeal script includes a `require 'squealer'`.
|
11
|
+
|
12
|
+
## Release Notes
|
13
|
+
### v1.2
|
14
|
+
* `Object#target` verifies there is a variable in scope with the same name as the `table_name` being targetted, it must be a `Hash` and must have an `_id` key
|
15
|
+
* Block to `Object#assign` not required, infers value from source scope
|
16
|
+
* A block returning `nil` now uses `nil` as the value to `Object#assign`, rather than inferring value from source scope
|
17
|
+
|
18
|
+
## Warning
|
19
|
+
Squealer is for _standalone_ operation. DO NOT use it directly from within your Ruby application. To make the DSL easy to use, we alter some core types:
|
20
|
+
|
21
|
+
* `FalseClass#to_i` - You'll be storing booleans as a `tinyint(1)`, or similar. `false` is `0`.
|
22
|
+
* `Hash#method_missing` - You prefer dot notation. JSON uses dot notation. You are importing from a data store which represents collections as arrays of hashmaps. Dot notation for navigating those collections is convenient. If you use a field name that happens to be a method on Hash you will have to use index notation. (e.g. `kitten.toys` is good, however `kitten.freeze` is not good. Use `kitten['freeze']` instead.)
|
23
|
+
* `NilClass#each` - As you are importing from schemaless repositories and you may be trying to iterate on fields that contain embedded collections, if a specific parent does not contain one of those child collections, the driver will be returning `nil` as the value for that field. Having `NilClass#each` return a `[]` for a nil is convenient, semantically correct in this context, and removes the need for many `nil` checks in the block you provide to `Object#assign`
|
24
|
+
* `Object` - `#import`, `#export`, `#target`, and `#assign` "keywords" are provided for convenience
|
25
|
+
* `Time#to_s` - As you are exporting to a SQL database, we represent your timestamp in a format that it will parse unequivocally (MongoDB stores all temporal data as a timestamp)
|
26
|
+
* `TrueClass#to_i` - You'll be storing booleans as a `tinyint(1)`, or similar. `true` is `1`.
|
16
27
|
|
17
|
-
|
28
|
+
## It is a data mapper, it doesn't use one.
|
29
|
+
Squealer doesn't use your application classes. It doesn't use your ActiveRecord models. It doesn't use mongoid (as awesome as that is), or mongomapper. It's an ETL tool. It could even be called a HRM (Hashmap-Relational-Mapper), but only in hushed tones in the corner boothes of dark pubs. It directly uses the Ruby driver for MongoDB and the Ruby driver for mySQL.
|
18
30
|
|
19
31
|
## Databases supported
|
20
|
-
For now, this is specifically for _MongoDB_ exporting to _mySQL_
|
32
|
+
For now, this is specifically for _MongoDB_ exporting to _mySQL_.
|
33
|
+
|
34
|
+
## Deprecation Warning
|
35
|
+
Since version 1.1, the primary key value is inferred from the source document `_id` field based on the `Object#target` `table_name` argument matching the name of a variable holding the source document, `row_id` is no longer a parameter on `Object#target`. It will be invalid in version 1.3 and above.
|
21
36
|
|
22
37
|
## Notes
|
23
|
-
The target SQL database
|
38
|
+
The target SQL database _must_ have no foreign keys (because it can't rely on the primary key values and referential integrity is the responsibility of the source data store or the application that uses it).
|
39
|
+
|
40
|
+
The target SQL database must use a primary key of `char(24)`. For now, we've assumed that column name is `id`. Each record's `id` value will get the source document `_id` value.
|
24
41
|
|
25
|
-
|
42
|
+
It is assumed the target data will be quite denormalized - particularly that the hierarchy keys for embedded documents are flattened. This means that a document from `office.room.box` will be exported to a record containing the `id` for `office`, the `id` for `room` and the `id` for `box`.
|
26
43
|
|
27
|
-
It is assumed no indexes are present in the target database table (performance drag). You may want to create indexes for pulling data out of the database Squealer exports to.
|
44
|
+
It is assumed no indexes are present in the target database table (performance drag). You may want to create indexes for pulling data out of the database Squealer exports to. Run a SQL DDL script on your mySQL database after squealing to add the indexes. You should drop the indexes before squealing again.
|
28
45
|
|
29
|
-
The target row is inserted, or updated if present. We are using MySQL `INSERT ... UPDATE ON DUPLICATE KEY` extended syntax to achieve this for now. This allows an event-driven update of exported data as well as a bulk batch process.
|
46
|
+
The target row is inserted, or updated if present. We are using MySQL `INSERT ... UPDATE ON DUPLICATE KEY` extended syntax to achieve this for now. This allows an event-driven update of exported data (e.g. through redis queues) as well as a bulk batch process.
|
30
47
|
|
data/Rakefile
CHANGED
@@ -5,9 +5,6 @@ require 'rake'
|
|
5
5
|
require 'spec/rake/spectask'
|
6
6
|
require 'rake/rdoctask'
|
7
7
|
|
8
|
-
task :default => [:spec]
|
9
|
-
Rake::Task[:default].clear
|
10
|
-
|
11
8
|
begin
|
12
9
|
require 'jeweler'
|
13
10
|
Jeweler::Tasks.new do |gemspec|
|
@@ -25,3 +22,10 @@ rescue LoadError
|
|
25
22
|
puts "Jeweler not available. Install it with: gem install jeweler"
|
26
23
|
end
|
27
24
|
|
25
|
+
desc "Run all specs in spec directory (excluding plugin specs)"
|
26
|
+
Spec::Rake::SpecTask.new(:spec) do |t|
|
27
|
+
t.spec_opts = ['--options', "\"#{File.dirname(__FILE__)}/spec/spec.opts\""]
|
28
|
+
t.spec_files = FileList['spec/**/*/*_spec.rb']
|
29
|
+
end
|
30
|
+
|
31
|
+
task :default => [:spec]
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
1.0
|
1
|
+
1.2.0
|
data/lib/example_squeal.rb
CHANGED
@@ -1,38 +1,84 @@
|
|
1
1
|
require 'squealer'
|
2
2
|
|
3
|
+
# connect to the source mongodb database
|
3
4
|
import('localhost', 27017, 'development')
|
5
|
+
|
6
|
+
# connect to the target mysql database
|
4
7
|
export('localhost', 'root', '', 'reporting_export')
|
5
8
|
|
9
|
+
# Here we extract, transform and load all documents in a collection...
|
6
10
|
import.collection("users").find({}).each do |user|
|
7
|
-
|
8
|
-
|
9
|
-
|
11
|
+
# Insert or Update on table 'user' where 'id' is the column name of the primary key.
|
12
|
+
#
|
13
|
+
# The primary key value is taken from the '_id' field of the source document,
|
14
|
+
# referenced using a variable with the same name as the table name passed to target().
|
15
|
+
#
|
16
|
+
# The block parameter |user| above, matches the target() parameter :user below...
|
17
|
+
target(:user) do
|
18
|
+
#
|
19
|
+
# assign() takes a column name and a block to set the value.
|
20
|
+
#
|
21
|
+
# You can use an valid arbitrary expression...
|
22
|
+
assign(:name) { "#{user.last_name.upcase}, #{user.first_name}" }
|
23
|
+
#
|
24
|
+
# You can use a simple access on the source document...
|
25
|
+
assign(:dob) { user.date_of_birth }
|
26
|
+
#
|
27
|
+
# You can use an empty block to infer the value from a field of the same
|
28
|
+
# name on the source document...
|
29
|
+
assign(:gender) #or# assign(:gender) { user.gender }
|
30
|
+
|
31
|
+
#
|
32
|
+
# You can normalize the export...
|
33
|
+
# home_address and work_address are a formatted string like: "661 W Lake St, Suite 3NE, Chicago IL, 60611, USA"
|
34
|
+
addresses = []
|
35
|
+
addresses << atomize_address(user.home_address)
|
36
|
+
addresses << atomize_address(user.work_address)
|
37
|
+
addresses.each do |address|
|
38
|
+
target(:address) do
|
39
|
+
assign(:street)
|
40
|
+
assign(:city)
|
41
|
+
assign(:state)
|
42
|
+
assign(:zip)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
#
|
47
|
+
# You can denormalize the export...
|
48
|
+
# user.home_address = { street: '661 W Lake St', city: 'Chicago', state: 'IL' }
|
49
|
+
assign(:home_address) { flatten_address(user.home_address) }
|
50
|
+
assign(:work_address) { flatten_address(user.work_address) }
|
10
51
|
|
11
52
|
user.activities.each do |activity|
|
12
|
-
target(:activity
|
13
|
-
|
14
|
-
|
53
|
+
target(:activity) do
|
54
|
+
#
|
55
|
+
# You can use an empty block to infer the value from the '_id' field
|
56
|
+
# of a parent document where the name of the parent collection matches
|
57
|
+
# a variable that is in scope.
|
58
|
+
#
|
59
|
+
assign(:user_id) #or# assign(:user_id) { user._id }
|
60
|
+
assign(:name) #or# assign(:name) { activity.name }
|
61
|
+
assign(:due_date) #or# assign(:due_date) { activity.due_date }
|
15
62
|
end
|
16
63
|
|
17
64
|
activity.tasks.each do |task|
|
18
|
-
target(:task
|
19
|
-
assign(:user_id) { user._id }
|
20
|
-
assign(:activity_id) { activity._id }
|
21
|
-
assign(:
|
65
|
+
target(:task) do
|
66
|
+
assign(:user_id) #or# assign(:user_id) { user._id }
|
67
|
+
assign(:activity_id) #or# assign(:activity_id) { activity._id }
|
68
|
+
assign(:due_date) #or# assign(:due_date) { task.due_date }
|
22
69
|
end
|
23
70
|
end #activity.tasks
|
24
71
|
end #user.activities
|
25
72
|
end
|
26
73
|
end #collection("users")
|
27
74
|
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
75
|
+
# Here we use a procedural "join" on related collections to update a target...
|
76
|
+
import.collection("organization").find({'disabled_date' : { 'exists' : 'true' }}).each do |organization|
|
77
|
+
import.collection("users").find({ :organization_id => organization.id }) do |user|
|
78
|
+
target(:user) do
|
79
|
+
#
|
80
|
+
# Source boolean values are converted to integer (0 or 1)...
|
81
|
+
assign(:disabled) { true }
|
34
82
|
end
|
35
|
-
else
|
36
|
-
# something else
|
37
83
|
end
|
38
84
|
end
|
data/lib/squealer/object.rb
CHANGED
data/lib/squealer/target.rb
CHANGED
@@ -1,6 +1,9 @@
|
|
1
1
|
require 'delegate'
|
2
2
|
require 'singleton'
|
3
3
|
|
4
|
+
#TODO: Use logger and log throughout
|
5
|
+
#TODO: Counters and timers
|
6
|
+
|
4
7
|
module Squealer
|
5
8
|
class Target
|
6
9
|
|
@@ -8,11 +11,16 @@ module Squealer
|
|
8
11
|
Queue.instance.current
|
9
12
|
end
|
10
13
|
|
11
|
-
def initialize(database_connection, table_name, row_id, &block)
|
12
|
-
|
14
|
+
def initialize(database_connection, table_name, row_id=nil, &block)
|
15
|
+
raise BlockRequired, "Block must be given to target (otherwise, there's no work to do)" unless block_given?
|
16
|
+
raise ArgumentError, "Table name must be supplied" if table_name.to_s.strip.empty?
|
13
17
|
|
14
18
|
@table_name = table_name.to_s
|
15
|
-
@
|
19
|
+
@binding = block.binding
|
20
|
+
|
21
|
+
verify_table_name_in_scope
|
22
|
+
|
23
|
+
@row_id = obtain_row_id(row_id)
|
16
24
|
@column_names = []
|
17
25
|
@column_values = []
|
18
26
|
@sql = ''
|
@@ -26,13 +34,52 @@ module Squealer
|
|
26
34
|
|
27
35
|
def assign(column_name, &block)
|
28
36
|
@column_names << column_name
|
29
|
-
|
37
|
+
if block_given?
|
38
|
+
@column_values << yield
|
39
|
+
else
|
40
|
+
@column_values << infer_value(column_name, @binding)
|
41
|
+
end
|
30
42
|
end
|
31
43
|
|
32
44
|
|
33
45
|
private
|
34
46
|
|
35
|
-
def
|
47
|
+
def obtain_row_id(row_id)
|
48
|
+
#TODO: Remove in version 1.3 - just call infer_row_id in initialize
|
49
|
+
if row_id != nil
|
50
|
+
puts "\033[33mWARNING - squealer:\033[0m the 'target' row_id parameter is deprecated and will be invalid in version 1.3 and above. Remove it, and ensure the table_name matches a variable containing a hashmap with an _id key"
|
51
|
+
row_id
|
52
|
+
else
|
53
|
+
infer_row_id
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def infer_row_id
|
58
|
+
@binding.eval "#{@table_name}._id"
|
59
|
+
end
|
60
|
+
|
61
|
+
def verify_table_name_in_scope
|
62
|
+
table = @binding.eval "#{@table_name}"
|
63
|
+
raise ArgumentError, "The variable '#{@table_name}' is not a hashmap" unless table.is_a? Hash
|
64
|
+
raise ArgumentError, "The hashmap '#{@table_name}' must have an '_id' key" unless table.has_key? '_id'
|
65
|
+
rescue NameError
|
66
|
+
raise NameError, "A variable named '#{@table_name}' must be in scope, and reference a hashmap with at least an '_id' key."
|
67
|
+
end
|
68
|
+
|
69
|
+
|
70
|
+
def infer_value(column_name, binding)
|
71
|
+
value = binding.eval "#{@table_name}.#{column_name}"
|
72
|
+
unless value
|
73
|
+
name = column_name.to_s
|
74
|
+
if name.end_with?("_id")
|
75
|
+
related = name[0..-4] #strip "_id"
|
76
|
+
value = binding.eval "#{related}._id"
|
77
|
+
end
|
78
|
+
end
|
79
|
+
value
|
80
|
+
end
|
81
|
+
|
82
|
+
def target
|
36
83
|
Queue.instance.push(self)
|
37
84
|
|
38
85
|
yield self
|
@@ -99,8 +146,9 @@ module Squealer
|
|
99
146
|
def initialize
|
100
147
|
super([])
|
101
148
|
end
|
102
|
-
|
103
149
|
end
|
104
150
|
|
151
|
+
class BlockRequired < ArgumentError; end
|
152
|
+
|
105
153
|
end
|
106
154
|
end
|
data/lib/squealer/time.rb
CHANGED
data/spec/spec_helper.rb
CHANGED
@@ -10,6 +10,7 @@ describe NilClass do
|
|
10
10
|
end
|
11
11
|
|
12
12
|
describe Object do
|
13
|
+
let(:test_table) { {'_id' => 1} }
|
13
14
|
|
14
15
|
describe "#target" do
|
15
16
|
|
@@ -19,12 +20,12 @@ describe Object do
|
|
19
20
|
|
20
21
|
it "invokes Squealer::Target.new" do
|
21
22
|
Squealer::Target.should_receive(:new)
|
22
|
-
target(:test_table
|
23
|
+
target(:test_table) { nil }
|
23
24
|
end
|
24
25
|
|
25
26
|
it "uses the export database connection" do
|
26
27
|
mock_mysql
|
27
|
-
target(:test_table
|
28
|
+
target(:test_table) { nil }
|
28
29
|
end
|
29
30
|
|
30
31
|
end
|
@@ -37,11 +38,12 @@ describe Object do
|
|
37
38
|
|
38
39
|
it "invokes assign on the target it is immediately nested within" do
|
39
40
|
mock_mysql
|
40
|
-
target(:test_table
|
41
|
+
target(:test_table) do |target1|
|
41
42
|
target1.should_receive(:assign)
|
42
43
|
assign(:colA) { 42 }
|
43
44
|
|
44
|
-
|
45
|
+
test_table_2 = test_table
|
46
|
+
target(:test_table_2) do |target2|
|
45
47
|
target2.should_receive(:assign)
|
46
48
|
assign(:colspeak) { 1984 }
|
47
49
|
end
|
@@ -1,29 +1,82 @@
|
|
1
1
|
require 'spec_helper'
|
2
2
|
|
3
3
|
describe Squealer::Target do
|
4
|
-
let(:export_dbc) { mock(Mysql) }
|
5
4
|
let(:table_name) { :test_table }
|
6
|
-
let(:
|
5
|
+
let(:test_table) { {'_id' => 0} }
|
7
6
|
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
7
|
+
let(:export_dbc) { mock(Mysql) }
|
8
|
+
|
9
|
+
after(:each) { Squealer::Target::Queue.instance.clear }
|
10
|
+
|
11
|
+
|
12
|
+
context "targeting" do
|
13
|
+
describe "initialize" do
|
14
|
+
let(:faqs) { [{'_id' => 123}] }
|
15
|
+
|
16
|
+
context "without a target row id" do
|
17
|
+
context "with the inferred variable in scope" do
|
18
|
+
it "infers the value from the _id field in the hashmap referenced by the variable" do
|
19
|
+
mock_mysql
|
20
|
+
|
21
|
+
faqs.each do |faq|
|
22
|
+
Squealer::Target.new(nil, :faq) do |target|
|
23
|
+
target.send(:instance_variable_get, '@row_id').should == faq['_id']
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
context "but it doesn't have an _id key" do
|
29
|
+
it "throws an argument error" do
|
30
|
+
hash_with_no_id = {}
|
31
|
+
lambda do
|
32
|
+
Squealer::Target.new(nil, :hash_with_no_id) {}
|
33
|
+
end.should raise_error(ArgumentError)
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
context "but it isn't a hashmap" do
|
38
|
+
it "throws an argument error" do
|
39
|
+
not_a_hash = nil
|
40
|
+
lambda do
|
41
|
+
Squealer::Target.new(nil, :not_a_hash) {}
|
42
|
+
end.should raise_error(ArgumentError)
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
|
47
|
+
context "without the inferred variable in scope" do
|
48
|
+
it "throws a name error" do
|
49
|
+
lambda do
|
50
|
+
Squealer::Target.new(nil, :missing_variable) {}
|
51
|
+
end.should raise_error(NameError)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
14
55
|
|
15
|
-
|
16
|
-
|
56
|
+
context "with a target row id" do
|
57
|
+
it "uses the passed value" do
|
58
|
+
mock_mysql
|
59
|
+
|
60
|
+
faqs.each do |faq|
|
61
|
+
Squealer::Target.new(nil, :faq, 1) do |target|
|
62
|
+
target.send(:instance_variable_get, '@row_id').should == 1
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
17
68
|
end
|
18
69
|
|
19
|
-
describe "#target" do
|
20
70
|
|
71
|
+
context "nesting" do
|
21
72
|
it "pushes itself onto the targets stack when starting" do
|
22
|
-
|
23
|
-
@target2 = nil
|
24
|
-
|
73
|
+
mock_mysql
|
74
|
+
@target1 = @target2 = nil
|
75
|
+
|
76
|
+
target1 = Squealer::Target.new(export_dbc, table_name) do
|
25
77
|
@target1 = Squealer::Target.current
|
26
|
-
|
78
|
+
test_table_2 = test_table
|
79
|
+
Squealer::Target.new(export_dbc, "#{table_name}_2") do
|
27
80
|
@target2 = Squealer::Target.current
|
28
81
|
@target2.should_not == @target1
|
29
82
|
end
|
@@ -32,57 +85,157 @@ describe Squealer::Target do
|
|
32
85
|
end
|
33
86
|
|
34
87
|
it "pops itself off the targets stack when finished" do
|
35
|
-
|
88
|
+
mock_mysql
|
89
|
+
|
90
|
+
Squealer::Target.new(export_dbc, table_name) { nil }
|
36
91
|
Squealer::Target.current.should be_nil
|
37
92
|
end
|
93
|
+
end
|
38
94
|
|
39
|
-
context "generates SQL command strings" do
|
40
95
|
|
41
|
-
|
96
|
+
context "yielding" do
|
97
|
+
it "yields" do
|
98
|
+
mock_mysql
|
99
|
+
|
100
|
+
block_done = false
|
101
|
+
target = Squealer::Target.new(export_dbc, table_name) { block_done = true }
|
102
|
+
block_done.should be_true
|
103
|
+
end
|
42
104
|
|
43
|
-
|
44
|
-
|
105
|
+
it "yields inner blocks before executing its own SQL" do
|
106
|
+
mock_mysql
|
107
|
+
|
108
|
+
blocks_done = []
|
109
|
+
Squealer::Target.new(export_dbc, table_name) do |target_1|
|
110
|
+
blocks_done << target_1
|
111
|
+
blocks_done.first.sql.should be_empty
|
112
|
+
Squealer::Target.new(export_dbc, table_name) do |target_2|
|
113
|
+
blocks_done << target_2
|
114
|
+
blocks_done.first.sql.should be_empty
|
115
|
+
blocks_done.last.sql.should be_empty
|
116
|
+
end
|
117
|
+
blocks_done.first.sql.should be_empty
|
118
|
+
blocks_done.last.sql.should_not be_empty
|
45
119
|
end
|
120
|
+
blocks_done.first.sql.should_not be_empty
|
121
|
+
blocks_done.last.sql.should_not be_empty
|
122
|
+
end
|
123
|
+
end
|
124
|
+
|
46
125
|
|
47
|
-
|
48
|
-
|
126
|
+
context "assigning" do
|
127
|
+
describe "#assign" do
|
128
|
+
let(:col1) { :meaning }
|
129
|
+
let(:value1) { 42 }
|
130
|
+
let(:faqs) { [{'_id' => nil, col1.to_s => value1}] }
|
131
|
+
let(:askers) { [{'_id' => 2001, 'name' => 'Zarathustra'}] }
|
132
|
+
|
133
|
+
context "with a block" do
|
134
|
+
it "uses the value from the block" do
|
135
|
+
mock_mysql
|
136
|
+
|
137
|
+
faqs.each do |faq|
|
138
|
+
Squealer::Target.new(nil, :faq) do
|
139
|
+
assign(col1) { value1 }
|
140
|
+
Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
|
141
|
+
Squealer::Target.current.instance_variable_get('@column_values').should == [value1]
|
142
|
+
end
|
143
|
+
end
|
144
|
+
end
|
145
|
+
it "uses the value from the block even if it is nil" do
|
146
|
+
mock_mysql
|
147
|
+
|
148
|
+
faqs.each do |faq|
|
149
|
+
Squealer::Target.new(nil, :faq) do
|
150
|
+
assign(col1) { nil }
|
151
|
+
Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
|
152
|
+
Squealer::Target.current.instance_variable_get('@column_values').should == [nil]
|
153
|
+
end
|
154
|
+
end
|
155
|
+
end
|
49
156
|
end
|
50
157
|
|
51
|
-
|
52
|
-
|
158
|
+
context "without a block" do
|
159
|
+
context "with the inferred variable in scope" do
|
160
|
+
it "infers source from target name" do
|
161
|
+
mock_mysql
|
162
|
+
|
163
|
+
faqs.each do |faq|
|
164
|
+
Squealer::Target.new(nil, :faq) do
|
165
|
+
assign(col1)
|
166
|
+
Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
|
167
|
+
Squealer::Target.current.instance_variable_get('@column_values').should == [value1]
|
168
|
+
end
|
169
|
+
end
|
170
|
+
end
|
171
|
+
it "infers related source from target name" do
|
172
|
+
mock_mysql
|
173
|
+
|
174
|
+
askers.each do |asker|
|
175
|
+
faqs.each do |faq|
|
176
|
+
Squealer::Target.new(nil, :faq) do
|
177
|
+
assign(:asker_id)
|
178
|
+
Squealer::Target.current.instance_variable_get('@column_names').should == [:asker_id]
|
179
|
+
Squealer::Target.current.instance_variable_get('@column_values').should == [2001]
|
180
|
+
end
|
181
|
+
end
|
182
|
+
end
|
183
|
+
end
|
184
|
+
end
|
53
185
|
end
|
54
186
|
|
55
|
-
|
56
|
-
|
57
|
-
|
187
|
+
context "with an empty block" do
|
188
|
+
it "assumes nil" do
|
189
|
+
mock_mysql
|
190
|
+
|
191
|
+
faqs.each do |faq|
|
192
|
+
Squealer::Target.new(nil, :faq) do
|
193
|
+
assign(col1) {}
|
194
|
+
Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
|
195
|
+
Squealer::Target.current.instance_variable_get('@column_values').should == [nil]
|
196
|
+
end
|
197
|
+
end
|
198
|
+
end
|
199
|
+
|
58
200
|
end
|
201
|
+
end
|
202
|
+
end
|
203
|
+
|
59
204
|
|
205
|
+
context "exporting" do
|
206
|
+
before(:each) { mock_mysql }
|
207
|
+
|
208
|
+
it "sends the sql to the export database" do
|
209
|
+
Squealer::Target.new(export_dbc, table_name) { nil }
|
60
210
|
end
|
61
211
|
|
62
|
-
|
212
|
+
describe "#target" do
|
213
|
+
context "generates SQL command strings" do
|
214
|
+
let(:target) { Squealer::Target.new(export_dbc, table_name) { nil } }
|
63
215
|
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
block_done.should be_true
|
68
|
-
end
|
216
|
+
it "targets the table" do
|
217
|
+
target.sql.should =~ /^INSERT #{table_name} /
|
218
|
+
end
|
69
219
|
|
70
|
-
|
71
|
-
|
72
|
-
|
220
|
+
it "uses an INSERT ... ON DUPLICATE KEY UPDATE statement" do
|
221
|
+
target.sql.should =~ /^INSERT .* ON DUPLICATE KEY UPDATE /
|
222
|
+
end
|
223
|
+
|
224
|
+
it "includes the primary key name in the INSERT" do
|
225
|
+
target.sql.should =~ / \(id\) VALUES/
|
226
|
+
end
|
227
|
+
|
228
|
+
it "includes the primary key value in the INSERT" do
|
229
|
+
# target.sql.should =~ / VALUES \('#{row_id}'\) /
|
230
|
+
target.sql.should =~ / VALUES \(\?\) /
|
231
|
+
end
|
73
232
|
|
74
|
-
it "yields inner blocks first and they can assign to this target" do
|
75
|
-
target = Squealer::Target.new(export_dbc, table_name, row_id) { Squealer::Target.current.assign(:colA) { 42 } }
|
76
|
-
target.sql.should =~ /colA/
|
77
|
-
# target.sql.should =~ /42/
|
78
|
-
target.sql.should =~ /\?/
|
79
233
|
end
|
80
234
|
|
81
235
|
context "with 2 columns" do
|
82
|
-
|
83
236
|
let(:value_1) { 42 }
|
84
237
|
let(:target) do
|
85
|
-
Squealer::Target.new(export_dbc, table_name
|
238
|
+
Squealer::Target.new(export_dbc, table_name) { Squealer::Target.current.assign(:colA) { value_1 } }
|
86
239
|
end
|
87
240
|
|
88
241
|
it "includes the column name in the INSERT" do
|
@@ -102,11 +255,10 @@ describe Squealer::Target do
|
|
102
255
|
end
|
103
256
|
|
104
257
|
context "with 3 columns" do
|
105
|
-
|
106
258
|
let(:value_1) { 42 }
|
107
259
|
let(:value_2) { 'foobar' }
|
108
260
|
let(:target) do
|
109
|
-
Squealer::Target.new(export_dbc, table_name
|
261
|
+
Squealer::Target.new(export_dbc, table_name) do
|
110
262
|
Squealer::Target.current.assign(:colA) { value_1 }
|
111
263
|
Squealer::Target.current.assign(:colB) { value_2 }
|
112
264
|
end
|
@@ -125,11 +277,14 @@ describe Squealer::Target do
|
|
125
277
|
# target.sql.should =~ /UPDATE colA='#{value_1}',colB='#{value_2}'/
|
126
278
|
target.sql.should =~ /UPDATE colA=\?,colB=\?/
|
127
279
|
end
|
128
|
-
|
129
280
|
end
|
130
|
-
|
131
281
|
end
|
132
|
-
|
133
282
|
end
|
283
|
+
end
|
134
284
|
|
285
|
+
def mock_mysql
|
286
|
+
Squealer::Database.instance.should_receive(:export).at_least(:once).and_return(export_dbc)
|
287
|
+
st = mock(Mysql::Stmt)
|
288
|
+
export_dbc.should_receive(:prepare).at_least(:once).and_return(st)
|
289
|
+
st.should_receive(:execute).at_least(:once)
|
135
290
|
end
|
data/squealer.gemspec
CHANGED
@@ -5,11 +5,11 @@
|
|
5
5
|
|
6
6
|
Gem::Specification.new do |s|
|
7
7
|
s.name = %q{squealer}
|
8
|
-
s.version = "1.0
|
8
|
+
s.version = "1.2.0"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
11
|
s.authors = ["Josh Graham", "Durran Jordan"]
|
12
|
-
s.date = %q{2010-
|
12
|
+
s.date = %q{2010-05-17}
|
13
13
|
s.description = %q{Exports mongodb to mysql. More later.}
|
14
14
|
s.email = %q{joshua.graham@grahamis.com}
|
15
15
|
s.extra_rdoc_files = [
|
@@ -17,6 +17,7 @@ Gem::Specification.new do |s|
|
|
17
17
|
]
|
18
18
|
s.files = [
|
19
19
|
".gitignore",
|
20
|
+
".rvmrc",
|
20
21
|
".watchr",
|
21
22
|
"README.md",
|
22
23
|
"Rakefile",
|
@@ -43,7 +44,7 @@ Gem::Specification.new do |s|
|
|
43
44
|
s.homepage = %q{http://github.com/delitescere/squealer/}
|
44
45
|
s.rdoc_options = ["--charset=UTF-8"]
|
45
46
|
s.require_paths = ["lib"]
|
46
|
-
s.rubygems_version = %q{1.3.
|
47
|
+
s.rubygems_version = %q{1.3.7}
|
47
48
|
s.summary = %q{Document-oriented to Relational database exporter}
|
48
49
|
s.test_files = [
|
49
50
|
"spec/spec_helper.rb",
|
@@ -59,7 +60,7 @@ Gem::Specification.new do |s|
|
|
59
60
|
current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
|
60
61
|
s.specification_version = 3
|
61
62
|
|
62
|
-
if Gem::Version.new(Gem::
|
63
|
+
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
63
64
|
s.add_runtime_dependency(%q<mysql>, [">= 2.8.1"])
|
64
65
|
s.add_runtime_dependency(%q<mongo>, [">= 0.18.3"])
|
65
66
|
else
|
metadata
CHANGED
@@ -1,12 +1,13 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: squealer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
+
hash: 31
|
4
5
|
prerelease: false
|
5
6
|
segments:
|
6
7
|
- 1
|
7
|
-
- 0
|
8
8
|
- 2
|
9
|
-
|
9
|
+
- 0
|
10
|
+
version: 1.2.0
|
10
11
|
platform: ruby
|
11
12
|
authors:
|
12
13
|
- Josh Graham
|
@@ -15,16 +16,18 @@ autorequire:
|
|
15
16
|
bindir: bin
|
16
17
|
cert_chain: []
|
17
18
|
|
18
|
-
date: 2010-
|
19
|
+
date: 2010-05-17 00:00:00 -05:00
|
19
20
|
default_executable:
|
20
21
|
dependencies:
|
21
22
|
- !ruby/object:Gem::Dependency
|
22
23
|
name: mysql
|
23
24
|
prerelease: false
|
24
25
|
requirement: &id001 !ruby/object:Gem::Requirement
|
26
|
+
none: false
|
25
27
|
requirements:
|
26
28
|
- - ">="
|
27
29
|
- !ruby/object:Gem::Version
|
30
|
+
hash: 45
|
28
31
|
segments:
|
29
32
|
- 2
|
30
33
|
- 8
|
@@ -36,9 +39,11 @@ dependencies:
|
|
36
39
|
name: mongo
|
37
40
|
prerelease: false
|
38
41
|
requirement: &id002 !ruby/object:Gem::Requirement
|
42
|
+
none: false
|
39
43
|
requirements:
|
40
44
|
- - ">="
|
41
45
|
- !ruby/object:Gem::Version
|
46
|
+
hash: 81
|
42
47
|
segments:
|
43
48
|
- 0
|
44
49
|
- 18
|
@@ -56,6 +61,7 @@ extra_rdoc_files:
|
|
56
61
|
- README.md
|
57
62
|
files:
|
58
63
|
- .gitignore
|
64
|
+
- .rvmrc
|
59
65
|
- .watchr
|
60
66
|
- README.md
|
61
67
|
- Rakefile
|
@@ -88,23 +94,27 @@ rdoc_options:
|
|
88
94
|
require_paths:
|
89
95
|
- lib
|
90
96
|
required_ruby_version: !ruby/object:Gem::Requirement
|
97
|
+
none: false
|
91
98
|
requirements:
|
92
99
|
- - ">="
|
93
100
|
- !ruby/object:Gem::Version
|
101
|
+
hash: 3
|
94
102
|
segments:
|
95
103
|
- 0
|
96
104
|
version: "0"
|
97
105
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
106
|
+
none: false
|
98
107
|
requirements:
|
99
108
|
- - ">="
|
100
109
|
- !ruby/object:Gem::Version
|
110
|
+
hash: 3
|
101
111
|
segments:
|
102
112
|
- 0
|
103
113
|
version: "0"
|
104
114
|
requirements: []
|
105
115
|
|
106
116
|
rubyforge_project:
|
107
|
-
rubygems_version: 1.3.
|
117
|
+
rubygems_version: 1.3.7
|
108
118
|
signing_key:
|
109
119
|
specification_version: 3
|
110
120
|
summary: Document-oriented to Relational database exporter
|