squealer 1.0.2 → 1.2.0

Sign up to get free protection for your applications and to get access to all the features.
data/.gitignore CHANGED
@@ -1 +1,2 @@
1
1
  pkg/
2
+ tags
data/.rvmrc ADDED
@@ -0,0 +1,3 @@
1
+ # These are the rubies that squealer has been tested on. Uncomment the one you want to use right now.
2
+ rvm use 1.9.1@squealer
3
+ # rvm use 1.8.7@squealer
data/README.md CHANGED
@@ -1,30 +1,47 @@
1
1
  # Squealer
2
2
 
3
- ## Warning
4
- Squealer is for standalone operation. Do not use it from within your application. To make the DSL easy to use, we alter `Hash`, `NilClass`, `Object`, and `Time`.
5
-
6
- * `Hash#method_missing` - You prefer dot notation. JSON uses dot notation. You are importing from a data store which represents collections as arrays of hashmaps. Dot notation for navigating those collections is convenient. If you use a field name that happens to be a method on Hash you will have to use index notation. (e.g. `kitten.toys` is good, however `kitten.freeze` is not good. Use `kitten['freeze']` instead.)
7
- * `NilClass#each` - As you are importing from schemaless repositories and you may be trying to iterate on fields that contain embedded collections, if a specific parent does not contain one of those child collections, the driver will be returning "nil" as the value for that field. Having `NilClass#each` return a `[]` for a nil is convenient, semantically correct in this context, and removes the need for many nil checks in the block you provide to `Object#assign`
8
- * `Object` - `#import`, `#export`, `#target`, and `#assign` "keywords" are provided for convenience
9
- * `Time#to_s` - As you are exporting to a SQL database, we represent your timestamp in a format that it will parse unequivocally (mongodb stores all temporal data as a timestamp)
3
+ ## Usage
4
+ See lib/example_squeal.rb for the example squeal.
10
5
 
11
6
  To run standalone, simply make your data squeal thusly:
12
7
 
13
8
  `ruby example_squeal.rb`
14
9
 
15
- where the squeal script requires 'squealer'.
10
+ where the squeal script includes a `require 'squealer'`.
11
+
12
+ ## Release Notes
13
+ ### v1.2
14
+ * `Object#target` verifies there is a variable in scope with the same name as the `table_name` being targetted, it must be a `Hash` and must have an `_id` key
15
+ * Block to `Object#assign` not required, infers value from source scope
16
+ * A block returning `nil` now uses `nil` as the value to `Object#assign`, rather than inferring value from source scope
17
+
18
+ ## Warning
19
+ Squealer is for _standalone_ operation. DO NOT use it directly from within your Ruby application. To make the DSL easy to use, we alter some core types:
20
+
21
+ * `FalseClass#to_i` - You'll be storing booleans as a `tinyint(1)`, or similar. `false` is `0`.
22
+ * `Hash#method_missing` - You prefer dot notation. JSON uses dot notation. You are importing from a data store which represents collections as arrays of hashmaps. Dot notation for navigating those collections is convenient. If you use a field name that happens to be a method on Hash you will have to use index notation. (e.g. `kitten.toys` is good, however `kitten.freeze` is not good. Use `kitten['freeze']` instead.)
23
+ * `NilClass#each` - As you are importing from schemaless repositories and you may be trying to iterate on fields that contain embedded collections, if a specific parent does not contain one of those child collections, the driver will be returning `nil` as the value for that field. Having `NilClass#each` return a `[]` for a nil is convenient, semantically correct in this context, and removes the need for many `nil` checks in the block you provide to `Object#assign`
24
+ * `Object` - `#import`, `#export`, `#target`, and `#assign` "keywords" are provided for convenience
25
+ * `Time#to_s` - As you are exporting to a SQL database, we represent your timestamp in a format that it will parse unequivocally (MongoDB stores all temporal data as a timestamp)
26
+ * `TrueClass#to_i` - You'll be storing booleans as a `tinyint(1)`, or similar. `true` is `1`.
16
27
 
17
- Squealer doesn't use your application classes. It doesn't use your ActiveRecord models. It's an ETL tool. It could even be called a HRM (Hashmap-Relational-Mapper), but only in hushed tones in the corner boothes of dark pubs.
28
+ ## It is a data mapper, it doesn't use one.
29
+ Squealer doesn't use your application classes. It doesn't use your ActiveRecord models. It doesn't use mongoid (as awesome as that is), or mongomapper. It's an ETL tool. It could even be called a HRM (Hashmap-Relational-Mapper), but only in hushed tones in the corner boothes of dark pubs. It directly uses the Ruby driver for MongoDB and the Ruby driver for mySQL.
18
30
 
19
31
  ## Databases supported
20
- For now, this is specifically for _MongoDB_ exporting to _mySQL_ with the assumption that the data will be heavily denormalized - particularly that the hierarchy keys for embedded documents are flattened. This means that a document from `office.room.box` will be exported to a record containing the `id` for `office`, the `id` for `room` and the `id` for `box`.
32
+ For now, this is specifically for _MongoDB_ exporting to _mySQL_.
33
+
34
+ ## Deprecation Warning
35
+ Since version 1.1, the primary key value is inferred from the source document `_id` field based on the `Object#target` `table_name` argument matching the name of a variable holding the source document, `row_id` is no longer a parameter on `Object#target`. It will be invalid in version 1.3 and above.
21
36
 
22
37
  ## Notes
23
- The target SQL database must have no foreign keys (because it can't rely on the primary key values and referential integrity is the responsibility of the source data store or the application that uses it).
38
+ The target SQL database _must_ have no foreign keys (because it can't rely on the primary key values and referential integrity is the responsibility of the source data store or the application that uses it).
39
+
40
+ The target SQL database must use a primary key of `char(24)`. For now, we've assumed that column name is `id`. Each record's `id` value will get the source document `_id` value.
24
41
 
25
- The target SQL database must use a primary key of char(16) with value of the MongoDB id.
42
+ It is assumed the target data will be quite denormalized - particularly that the hierarchy keys for embedded documents are flattened. This means that a document from `office.room.box` will be exported to a record containing the `id` for `office`, the `id` for `room` and the `id` for `box`.
26
43
 
27
- It is assumed no indexes are present in the target database table (performance drag). You may want to create indexes for pulling data out of the database Squealer exports to. You should drop them again when you've done the pull.
44
+ It is assumed no indexes are present in the target database table (performance drag). You may want to create indexes for pulling data out of the database Squealer exports to. Run a SQL DDL script on your mySQL database after squealing to add the indexes. You should drop the indexes before squealing again.
28
45
 
29
- The target row is inserted, or updated if present. We are using MySQL `INSERT ... UPDATE ON DUPLICATE KEY` extended syntax to achieve this for now. This allows an event-driven update of exported data as well as a bulk batch process.
46
+ The target row is inserted, or updated if present. We are using MySQL `INSERT ... UPDATE ON DUPLICATE KEY` extended syntax to achieve this for now. This allows an event-driven update of exported data (e.g. through redis queues) as well as a bulk batch process.
30
47
 
data/Rakefile CHANGED
@@ -5,9 +5,6 @@ require 'rake'
5
5
  require 'spec/rake/spectask'
6
6
  require 'rake/rdoctask'
7
7
 
8
- task :default => [:spec]
9
- Rake::Task[:default].clear
10
-
11
8
  begin
12
9
  require 'jeweler'
13
10
  Jeweler::Tasks.new do |gemspec|
@@ -25,3 +22,10 @@ rescue LoadError
25
22
  puts "Jeweler not available. Install it with: gem install jeweler"
26
23
  end
27
24
 
25
+ desc "Run all specs in spec directory (excluding plugin specs)"
26
+ Spec::Rake::SpecTask.new(:spec) do |t|
27
+ t.spec_opts = ['--options', "\"#{File.dirname(__FILE__)}/spec/spec.opts\""]
28
+ t.spec_files = FileList['spec/**/*/*_spec.rb']
29
+ end
30
+
31
+ task :default => [:spec]
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.0.2
1
+ 1.2.0
@@ -1,38 +1,84 @@
1
1
  require 'squealer'
2
2
 
3
+ # connect to the source mongodb database
3
4
  import('localhost', 27017, 'development')
5
+
6
+ # connect to the target mysql database
4
7
  export('localhost', 'root', '', 'reporting_export')
5
8
 
9
+ # Here we extract, transform and load all documents in a collection...
6
10
  import.collection("users").find({}).each do |user|
7
- target(:user, user._id) do # insert or update on user where id is primary key column name
8
- assign(:name) { user.first_name + " " + user.last_name.upcase }
9
- assign(:dob) { user.dob }
11
+ # Insert or Update on table 'user' where 'id' is the column name of the primary key.
12
+ #
13
+ # The primary key value is taken from the '_id' field of the source document,
14
+ # referenced using a variable with the same name as the table name passed to target().
15
+ #
16
+ # The block parameter |user| above, matches the target() parameter :user below...
17
+ target(:user) do
18
+ #
19
+ # assign() takes a column name and a block to set the value.
20
+ #
21
+ # You can use an valid arbitrary expression...
22
+ assign(:name) { "#{user.last_name.upcase}, #{user.first_name}" }
23
+ #
24
+ # You can use a simple access on the source document...
25
+ assign(:dob) { user.date_of_birth }
26
+ #
27
+ # You can use an empty block to infer the value from a field of the same
28
+ # name on the source document...
29
+ assign(:gender) #or# assign(:gender) { user.gender }
30
+
31
+ #
32
+ # You can normalize the export...
33
+ # home_address and work_address are a formatted string like: "661 W Lake St, Suite 3NE, Chicago IL, 60611, USA"
34
+ addresses = []
35
+ addresses << atomize_address(user.home_address)
36
+ addresses << atomize_address(user.work_address)
37
+ addresses.each do |address|
38
+ target(:address) do
39
+ assign(:street)
40
+ assign(:city)
41
+ assign(:state)
42
+ assign(:zip)
43
+ end
44
+ end
45
+
46
+ #
47
+ # You can denormalize the export...
48
+ # user.home_address = { street: '661 W Lake St', city: 'Chicago', state: 'IL' }
49
+ assign(:home_address) { flatten_address(user.home_address) }
50
+ assign(:work_address) { flatten_address(user.work_address) }
10
51
 
11
52
  user.activities.each do |activity|
12
- target(:activity, activity._id) do
13
- assign(:user_id) { user._id }
14
- assign(:name) { activity.name }
53
+ target(:activity) do
54
+ #
55
+ # You can use an empty block to infer the value from the '_id' field
56
+ # of a parent document where the name of the parent collection matches
57
+ # a variable that is in scope.
58
+ #
59
+ assign(:user_id) #or# assign(:user_id) { user._id }
60
+ assign(:name) #or# assign(:name) { activity.name }
61
+ assign(:due_date) #or# assign(:due_date) { activity.due_date }
15
62
  end
16
63
 
17
64
  activity.tasks.each do |task|
18
- target(:task, task._id) do
19
- assign(:user_id) { user._id }
20
- assign(:activity_id) { activity._id }
21
- assign(:date) { task.date }
65
+ target(:task) do
66
+ assign(:user_id) #or# assign(:user_id) { user._id }
67
+ assign(:activity_id) #or# assign(:activity_id) { activity._id }
68
+ assign(:due_date) #or# assign(:due_date) { task.due_date }
22
69
  end
23
70
  end #activity.tasks
24
71
  end #user.activities
25
72
  end
26
73
  end #collection("users")
27
74
 
28
- import.collection("organization").find({}).each do |organization|
29
- if organization.disabled
30
- import.collection("users").find({ :organization_id => organization.id }) do |user|
31
- target(:user, user.id) do
32
- assign(:disabled) { true }
33
- end
75
+ # Here we use a procedural "join" on related collections to update a target...
76
+ import.collection("organization").find({'disabled_date' : { 'exists' : 'true' }}).each do |organization|
77
+ import.collection("users").find({ :organization_id => organization.id }) do |user|
78
+ target(:user) do
79
+ #
80
+ # Source boolean values are converted to integer (0 or 1)...
81
+ assign(:disabled) { true }
34
82
  end
35
- else
36
- # something else
37
83
  end
38
84
  end
@@ -1,6 +1,6 @@
1
1
  class Object
2
2
 
3
- def target(table_name, row_id, &block)
3
+ def target(table_name, row_id=nil, &block)
4
4
  Squealer::Target.new(Squealer::Database.instance.export, table_name, row_id, &block)
5
5
  end
6
6
 
@@ -1,6 +1,9 @@
1
1
  require 'delegate'
2
2
  require 'singleton'
3
3
 
4
+ #TODO: Use logger and log throughout
5
+ #TODO: Counters and timers
6
+
4
7
  module Squealer
5
8
  class Target
6
9
 
@@ -8,11 +11,16 @@ module Squealer
8
11
  Queue.instance.current
9
12
  end
10
13
 
11
- def initialize(database_connection, table_name, row_id, &block)
12
- throw "Block must be given to target (otherwise, there's no work to do)" unless block_given?
14
+ def initialize(database_connection, table_name, row_id=nil, &block)
15
+ raise BlockRequired, "Block must be given to target (otherwise, there's no work to do)" unless block_given?
16
+ raise ArgumentError, "Table name must be supplied" if table_name.to_s.strip.empty?
13
17
 
14
18
  @table_name = table_name.to_s
15
- @row_id = row_id
19
+ @binding = block.binding
20
+
21
+ verify_table_name_in_scope
22
+
23
+ @row_id = obtain_row_id(row_id)
16
24
  @column_names = []
17
25
  @column_values = []
18
26
  @sql = ''
@@ -26,13 +34,52 @@ module Squealer
26
34
 
27
35
  def assign(column_name, &block)
28
36
  @column_names << column_name
29
- @column_values << block.call
37
+ if block_given?
38
+ @column_values << yield
39
+ else
40
+ @column_values << infer_value(column_name, @binding)
41
+ end
30
42
  end
31
43
 
32
44
 
33
45
  private
34
46
 
35
- def target(&block)
47
+ def obtain_row_id(row_id)
48
+ #TODO: Remove in version 1.3 - just call infer_row_id in initialize
49
+ if row_id != nil
50
+ puts "\033[33mWARNING - squealer:\033[0m the 'target' row_id parameter is deprecated and will be invalid in version 1.3 and above. Remove it, and ensure the table_name matches a variable containing a hashmap with an _id key"
51
+ row_id
52
+ else
53
+ infer_row_id
54
+ end
55
+ end
56
+
57
+ def infer_row_id
58
+ @binding.eval "#{@table_name}._id"
59
+ end
60
+
61
+ def verify_table_name_in_scope
62
+ table = @binding.eval "#{@table_name}"
63
+ raise ArgumentError, "The variable '#{@table_name}' is not a hashmap" unless table.is_a? Hash
64
+ raise ArgumentError, "The hashmap '#{@table_name}' must have an '_id' key" unless table.has_key? '_id'
65
+ rescue NameError
66
+ raise NameError, "A variable named '#{@table_name}' must be in scope, and reference a hashmap with at least an '_id' key."
67
+ end
68
+
69
+
70
+ def infer_value(column_name, binding)
71
+ value = binding.eval "#{@table_name}.#{column_name}"
72
+ unless value
73
+ name = column_name.to_s
74
+ if name.end_with?("_id")
75
+ related = name[0..-4] #strip "_id"
76
+ value = binding.eval "#{related}._id"
77
+ end
78
+ end
79
+ value
80
+ end
81
+
82
+ def target
36
83
  Queue.instance.push(self)
37
84
 
38
85
  yield self
@@ -99,8 +146,9 @@ module Squealer
99
146
  def initialize
100
147
  super([])
101
148
  end
102
-
103
149
  end
104
150
 
151
+ class BlockRequired < ArgumentError; end
152
+
105
153
  end
106
154
  end
data/lib/squealer/time.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  class Time
2
2
  def to_s
3
- self.strftime("%Y-%m-%d %H:%M:%S %Z")
3
+ strftime("%Y-%m-%d %H:%M:%S %Z")
4
4
  end
5
5
  end
data/spec/spec_helper.rb CHANGED
@@ -1,5 +1,4 @@
1
1
  require 'rubygems'
2
- require 'spork'
3
2
 
4
3
  $LOAD_PATH.unshift(File.dirname(__FILE__))
5
4
  $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
@@ -10,6 +10,7 @@ describe NilClass do
10
10
  end
11
11
 
12
12
  describe Object do
13
+ let(:test_table) { {'_id' => 1} }
13
14
 
14
15
  describe "#target" do
15
16
 
@@ -19,12 +20,12 @@ describe Object do
19
20
 
20
21
  it "invokes Squealer::Target.new" do
21
22
  Squealer::Target.should_receive(:new)
22
- target(:test_table, 1) { nil }
23
+ target(:test_table) { nil }
23
24
  end
24
25
 
25
26
  it "uses the export database connection" do
26
27
  mock_mysql
27
- target(:test_table, 1) { nil }
28
+ target(:test_table) { nil }
28
29
  end
29
30
 
30
31
  end
@@ -37,11 +38,12 @@ describe Object do
37
38
 
38
39
  it "invokes assign on the target it is immediately nested within" do
39
40
  mock_mysql
40
- target(:test_table, 1) do |target1|
41
+ target(:test_table) do |target1|
41
42
  target1.should_receive(:assign)
42
43
  assign(:colA) { 42 }
43
44
 
44
- target(:test_table_2, 1) do |target2|
45
+ test_table_2 = test_table
46
+ target(:test_table_2) do |target2|
45
47
  target2.should_receive(:assign)
46
48
  assign(:colspeak) { 1984 }
47
49
  end
@@ -1,29 +1,82 @@
1
1
  require 'spec_helper'
2
2
 
3
3
  describe Squealer::Target do
4
- let(:export_dbc) { mock(Mysql) }
5
4
  let(:table_name) { :test_table }
6
- let(:row_id) { 0 }
5
+ let(:test_table) { {'_id' => 0} }
7
6
 
8
- before(:each) do
9
- Squealer::Database.instance.should_receive(:export).at_least(:once).and_return(export_dbc)
10
- st = mock(Mysql::Stmt)
11
- export_dbc.should_receive(:prepare).at_least(:once).and_return(st)
12
- st.should_receive(:execute).at_least(:once)
13
- end
7
+ let(:export_dbc) { mock(Mysql) }
8
+
9
+ after(:each) { Squealer::Target::Queue.instance.clear }
10
+
11
+
12
+ context "targeting" do
13
+ describe "initialize" do
14
+ let(:faqs) { [{'_id' => 123}] }
15
+
16
+ context "without a target row id" do
17
+ context "with the inferred variable in scope" do
18
+ it "infers the value from the _id field in the hashmap referenced by the variable" do
19
+ mock_mysql
20
+
21
+ faqs.each do |faq|
22
+ Squealer::Target.new(nil, :faq) do |target|
23
+ target.send(:instance_variable_get, '@row_id').should == faq['_id']
24
+ end
25
+ end
26
+ end
27
+
28
+ context "but it doesn't have an _id key" do
29
+ it "throws an argument error" do
30
+ hash_with_no_id = {}
31
+ lambda do
32
+ Squealer::Target.new(nil, :hash_with_no_id) {}
33
+ end.should raise_error(ArgumentError)
34
+ end
35
+ end
36
+
37
+ context "but it isn't a hashmap" do
38
+ it "throws an argument error" do
39
+ not_a_hash = nil
40
+ lambda do
41
+ Squealer::Target.new(nil, :not_a_hash) {}
42
+ end.should raise_error(ArgumentError)
43
+ end
44
+ end
45
+ end
46
+
47
+ context "without the inferred variable in scope" do
48
+ it "throws a name error" do
49
+ lambda do
50
+ Squealer::Target.new(nil, :missing_variable) {}
51
+ end.should raise_error(NameError)
52
+ end
53
+ end
54
+ end
14
55
 
15
- it "sends the sql to the export database" do
16
- Squealer::Target.new(export_dbc, table_name, row_id) { nil }
56
+ context "with a target row id" do
57
+ it "uses the passed value" do
58
+ mock_mysql
59
+
60
+ faqs.each do |faq|
61
+ Squealer::Target.new(nil, :faq, 1) do |target|
62
+ target.send(:instance_variable_get, '@row_id').should == 1
63
+ end
64
+ end
65
+ end
66
+ end
67
+ end
17
68
  end
18
69
 
19
- describe "#target" do
20
70
 
71
+ context "nesting" do
21
72
  it "pushes itself onto the targets stack when starting" do
22
- @target1 = nil
23
- @target2 = nil
24
- target1 = Squealer::Target.new(export_dbc, table_name, row_id) do
73
+ mock_mysql
74
+ @target1 = @target2 = nil
75
+
76
+ target1 = Squealer::Target.new(export_dbc, table_name) do
25
77
  @target1 = Squealer::Target.current
26
- Squealer::Target.new(export_dbc, "#{table_name}_2", row_id) do
78
+ test_table_2 = test_table
79
+ Squealer::Target.new(export_dbc, "#{table_name}_2") do
27
80
  @target2 = Squealer::Target.current
28
81
  @target2.should_not == @target1
29
82
  end
@@ -32,57 +85,157 @@ describe Squealer::Target do
32
85
  end
33
86
 
34
87
  it "pops itself off the targets stack when finished" do
35
- Squealer::Target.new(export_dbc, table_name, row_id) { nil }
88
+ mock_mysql
89
+
90
+ Squealer::Target.new(export_dbc, table_name) { nil }
36
91
  Squealer::Target.current.should be_nil
37
92
  end
93
+ end
38
94
 
39
- context "generates SQL command strings" do
40
95
 
41
- let(:target) { Squealer::Target.new(export_dbc, table_name, row_id) { nil } }
96
+ context "yielding" do
97
+ it "yields" do
98
+ mock_mysql
99
+
100
+ block_done = false
101
+ target = Squealer::Target.new(export_dbc, table_name) { block_done = true }
102
+ block_done.should be_true
103
+ end
42
104
 
43
- it "targets the table" do
44
- target.sql.should =~ /^INSERT #{table_name} /
105
+ it "yields inner blocks before executing its own SQL" do
106
+ mock_mysql
107
+
108
+ blocks_done = []
109
+ Squealer::Target.new(export_dbc, table_name) do |target_1|
110
+ blocks_done << target_1
111
+ blocks_done.first.sql.should be_empty
112
+ Squealer::Target.new(export_dbc, table_name) do |target_2|
113
+ blocks_done << target_2
114
+ blocks_done.first.sql.should be_empty
115
+ blocks_done.last.sql.should be_empty
116
+ end
117
+ blocks_done.first.sql.should be_empty
118
+ blocks_done.last.sql.should_not be_empty
45
119
  end
120
+ blocks_done.first.sql.should_not be_empty
121
+ blocks_done.last.sql.should_not be_empty
122
+ end
123
+ end
124
+
46
125
 
47
- it "uses an INSERT ... ON DUPLICATE KEY UPDATE statement" do
48
- target.sql.should =~ /^INSERT .* ON DUPLICATE KEY UPDATE /
126
+ context "assigning" do
127
+ describe "#assign" do
128
+ let(:col1) { :meaning }
129
+ let(:value1) { 42 }
130
+ let(:faqs) { [{'_id' => nil, col1.to_s => value1}] }
131
+ let(:askers) { [{'_id' => 2001, 'name' => 'Zarathustra'}] }
132
+
133
+ context "with a block" do
134
+ it "uses the value from the block" do
135
+ mock_mysql
136
+
137
+ faqs.each do |faq|
138
+ Squealer::Target.new(nil, :faq) do
139
+ assign(col1) { value1 }
140
+ Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
141
+ Squealer::Target.current.instance_variable_get('@column_values').should == [value1]
142
+ end
143
+ end
144
+ end
145
+ it "uses the value from the block even if it is nil" do
146
+ mock_mysql
147
+
148
+ faqs.each do |faq|
149
+ Squealer::Target.new(nil, :faq) do
150
+ assign(col1) { nil }
151
+ Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
152
+ Squealer::Target.current.instance_variable_get('@column_values').should == [nil]
153
+ end
154
+ end
155
+ end
49
156
  end
50
157
 
51
- it "includes the primary key name in the INSERT" do
52
- target.sql.should =~ / \(id\) VALUES/
158
+ context "without a block" do
159
+ context "with the inferred variable in scope" do
160
+ it "infers source from target name" do
161
+ mock_mysql
162
+
163
+ faqs.each do |faq|
164
+ Squealer::Target.new(nil, :faq) do
165
+ assign(col1)
166
+ Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
167
+ Squealer::Target.current.instance_variable_get('@column_values').should == [value1]
168
+ end
169
+ end
170
+ end
171
+ it "infers related source from target name" do
172
+ mock_mysql
173
+
174
+ askers.each do |asker|
175
+ faqs.each do |faq|
176
+ Squealer::Target.new(nil, :faq) do
177
+ assign(:asker_id)
178
+ Squealer::Target.current.instance_variable_get('@column_names').should == [:asker_id]
179
+ Squealer::Target.current.instance_variable_get('@column_values').should == [2001]
180
+ end
181
+ end
182
+ end
183
+ end
184
+ end
53
185
  end
54
186
 
55
- it "includes the primary key value in the INSERT" do
56
- # target.sql.should =~ / VALUES \('#{row_id}'\) /
57
- target.sql.should =~ / VALUES \(\?\) /
187
+ context "with an empty block" do
188
+ it "assumes nil" do
189
+ mock_mysql
190
+
191
+ faqs.each do |faq|
192
+ Squealer::Target.new(nil, :faq) do
193
+ assign(col1) {}
194
+ Squealer::Target.current.instance_variable_get('@column_names').should == [col1]
195
+ Squealer::Target.current.instance_variable_get('@column_values').should == [nil]
196
+ end
197
+ end
198
+ end
199
+
58
200
  end
201
+ end
202
+ end
203
+
59
204
 
205
+ context "exporting" do
206
+ before(:each) { mock_mysql }
207
+
208
+ it "sends the sql to the export database" do
209
+ Squealer::Target.new(export_dbc, table_name) { nil }
60
210
  end
61
211
 
62
- context "with inner block" do
212
+ describe "#target" do
213
+ context "generates SQL command strings" do
214
+ let(:target) { Squealer::Target.new(export_dbc, table_name) { nil } }
63
215
 
64
- it "yields inner blocks" do
65
- block_done = false
66
- target = Squealer::Target.new(export_dbc, table_name, row_id) { block_done = true }
67
- block_done.should be_true
68
- end
216
+ it "targets the table" do
217
+ target.sql.should =~ /^INSERT #{table_name} /
218
+ end
69
219
 
70
- it "yields inner blocks first" do
71
- Squealer::Target.new(export_dbc, table_name, row_id) { Squealer::Target.current.sql.should be_empty }
72
- end
220
+ it "uses an INSERT ... ON DUPLICATE KEY UPDATE statement" do
221
+ target.sql.should =~ /^INSERT .* ON DUPLICATE KEY UPDATE /
222
+ end
223
+
224
+ it "includes the primary key name in the INSERT" do
225
+ target.sql.should =~ / \(id\) VALUES/
226
+ end
227
+
228
+ it "includes the primary key value in the INSERT" do
229
+ # target.sql.should =~ / VALUES \('#{row_id}'\) /
230
+ target.sql.should =~ / VALUES \(\?\) /
231
+ end
73
232
 
74
- it "yields inner blocks first and they can assign to this target" do
75
- target = Squealer::Target.new(export_dbc, table_name, row_id) { Squealer::Target.current.assign(:colA) { 42 } }
76
- target.sql.should =~ /colA/
77
- # target.sql.should =~ /42/
78
- target.sql.should =~ /\?/
79
233
  end
80
234
 
81
235
  context "with 2 columns" do
82
-
83
236
  let(:value_1) { 42 }
84
237
  let(:target) do
85
- Squealer::Target.new(export_dbc, table_name, row_id) { Squealer::Target.current.assign(:colA) { value_1 } }
238
+ Squealer::Target.new(export_dbc, table_name) { Squealer::Target.current.assign(:colA) { value_1 } }
86
239
  end
87
240
 
88
241
  it "includes the column name in the INSERT" do
@@ -102,11 +255,10 @@ describe Squealer::Target do
102
255
  end
103
256
 
104
257
  context "with 3 columns" do
105
-
106
258
  let(:value_1) { 42 }
107
259
  let(:value_2) { 'foobar' }
108
260
  let(:target) do
109
- Squealer::Target.new(export_dbc, table_name, row_id) do
261
+ Squealer::Target.new(export_dbc, table_name) do
110
262
  Squealer::Target.current.assign(:colA) { value_1 }
111
263
  Squealer::Target.current.assign(:colB) { value_2 }
112
264
  end
@@ -125,11 +277,14 @@ describe Squealer::Target do
125
277
  # target.sql.should =~ /UPDATE colA='#{value_1}',colB='#{value_2}'/
126
278
  target.sql.should =~ /UPDATE colA=\?,colB=\?/
127
279
  end
128
-
129
280
  end
130
-
131
281
  end
132
-
133
282
  end
283
+ end
134
284
 
285
+ def mock_mysql
286
+ Squealer::Database.instance.should_receive(:export).at_least(:once).and_return(export_dbc)
287
+ st = mock(Mysql::Stmt)
288
+ export_dbc.should_receive(:prepare).at_least(:once).and_return(st)
289
+ st.should_receive(:execute).at_least(:once)
135
290
  end
data/squealer.gemspec CHANGED
@@ -5,11 +5,11 @@
5
5
 
6
6
  Gem::Specification.new do |s|
7
7
  s.name = %q{squealer}
8
- s.version = "1.0.2"
8
+ s.version = "1.2.0"
9
9
 
10
10
  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
11
11
  s.authors = ["Josh Graham", "Durran Jordan"]
12
- s.date = %q{2010-04-27}
12
+ s.date = %q{2010-05-17}
13
13
  s.description = %q{Exports mongodb to mysql. More later.}
14
14
  s.email = %q{joshua.graham@grahamis.com}
15
15
  s.extra_rdoc_files = [
@@ -17,6 +17,7 @@ Gem::Specification.new do |s|
17
17
  ]
18
18
  s.files = [
19
19
  ".gitignore",
20
+ ".rvmrc",
20
21
  ".watchr",
21
22
  "README.md",
22
23
  "Rakefile",
@@ -43,7 +44,7 @@ Gem::Specification.new do |s|
43
44
  s.homepage = %q{http://github.com/delitescere/squealer/}
44
45
  s.rdoc_options = ["--charset=UTF-8"]
45
46
  s.require_paths = ["lib"]
46
- s.rubygems_version = %q{1.3.6}
47
+ s.rubygems_version = %q{1.3.7}
47
48
  s.summary = %q{Document-oriented to Relational database exporter}
48
49
  s.test_files = [
49
50
  "spec/spec_helper.rb",
@@ -59,7 +60,7 @@ Gem::Specification.new do |s|
59
60
  current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
60
61
  s.specification_version = 3
61
62
 
62
- if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
63
+ if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
63
64
  s.add_runtime_dependency(%q<mysql>, [">= 2.8.1"])
64
65
  s.add_runtime_dependency(%q<mongo>, [">= 0.18.3"])
65
66
  else
metadata CHANGED
@@ -1,12 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: squealer
3
3
  version: !ruby/object:Gem::Version
4
+ hash: 31
4
5
  prerelease: false
5
6
  segments:
6
7
  - 1
7
- - 0
8
8
  - 2
9
- version: 1.0.2
9
+ - 0
10
+ version: 1.2.0
10
11
  platform: ruby
11
12
  authors:
12
13
  - Josh Graham
@@ -15,16 +16,18 @@ autorequire:
15
16
  bindir: bin
16
17
  cert_chain: []
17
18
 
18
- date: 2010-04-27 00:00:00 -05:00
19
+ date: 2010-05-17 00:00:00 -05:00
19
20
  default_executable:
20
21
  dependencies:
21
22
  - !ruby/object:Gem::Dependency
22
23
  name: mysql
23
24
  prerelease: false
24
25
  requirement: &id001 !ruby/object:Gem::Requirement
26
+ none: false
25
27
  requirements:
26
28
  - - ">="
27
29
  - !ruby/object:Gem::Version
30
+ hash: 45
28
31
  segments:
29
32
  - 2
30
33
  - 8
@@ -36,9 +39,11 @@ dependencies:
36
39
  name: mongo
37
40
  prerelease: false
38
41
  requirement: &id002 !ruby/object:Gem::Requirement
42
+ none: false
39
43
  requirements:
40
44
  - - ">="
41
45
  - !ruby/object:Gem::Version
46
+ hash: 81
42
47
  segments:
43
48
  - 0
44
49
  - 18
@@ -56,6 +61,7 @@ extra_rdoc_files:
56
61
  - README.md
57
62
  files:
58
63
  - .gitignore
64
+ - .rvmrc
59
65
  - .watchr
60
66
  - README.md
61
67
  - Rakefile
@@ -88,23 +94,27 @@ rdoc_options:
88
94
  require_paths:
89
95
  - lib
90
96
  required_ruby_version: !ruby/object:Gem::Requirement
97
+ none: false
91
98
  requirements:
92
99
  - - ">="
93
100
  - !ruby/object:Gem::Version
101
+ hash: 3
94
102
  segments:
95
103
  - 0
96
104
  version: "0"
97
105
  required_rubygems_version: !ruby/object:Gem::Requirement
106
+ none: false
98
107
  requirements:
99
108
  - - ">="
100
109
  - !ruby/object:Gem::Version
110
+ hash: 3
101
111
  segments:
102
112
  - 0
103
113
  version: "0"
104
114
  requirements: []
105
115
 
106
116
  rubyforge_project:
107
- rubygems_version: 1.3.6
117
+ rubygems_version: 1.3.7
108
118
  signing_key:
109
119
  specification_version: 3
110
120
  summary: Document-oriented to Relational database exporter