postgres-copy 0.7.0 → 0.8.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 1ada774ac0dab27f58b81246ede246219546bd6f
4
- data.tar.gz: 7a7c83893ebe10c82501611fb72850b2d345946c
3
+ metadata.gz: 4b5212784e1c11d62f3996812f369a8702bff96b
4
+ data.tar.gz: 7bedcee0f0de6847cb8b35c0cedcc5b6ba3c801a
5
5
  SHA512:
6
- metadata.gz: 4ce6dc87b7bd78813b40cc6c566c1d2a760a4d8dee7e4cf0dca707d826b9982010dfe32cc851b9c18f4329d16a16cba614feb3b6f49beb561e0cfdb8cb5a4a70
7
- data.tar.gz: 62e04ba3a0368057d4c53ab8b5f0b730e994c26c54454d21cedd996525309667a2844b84ded5dc12a0dac5cd80b415b5605bcc1e8d22b71ab42ecdf7f1ab1ab3
6
+ metadata.gz: 5a7ce457d28c10521ab7ba677eb71f20d29e640783f77499f40a8a192c532b4520a567deda0bb37155f5d135780cbed1fb13a4445b3eddac1ea8626a004df888
7
+ data.tar.gz: 1c0c02a36b775e637aff34802e6890dd74e1e191b72a2e8e07f28c58a3ce69b3e78f70c75281b4ae80035cab09a060da8ad6562750b5ef2f8381d520d6bc8ca5
data/Gemfile.lock CHANGED
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- postgres-copy (0.7.0)
4
+ postgres-copy (0.8.0)
5
5
  activerecord (~> 4.0)
6
6
  pg
7
7
  rails (~> 4.0)
@@ -45,7 +45,7 @@ GEM
45
45
  mail (2.5.4)
46
46
  mime-types (~> 1.16)
47
47
  treetop (~> 1.4.8)
48
- mime-types (1.23)
48
+ mime-types (1.25)
49
49
  minitest (4.7.5)
50
50
  multi_json (1.7.9)
51
51
  pg (0.16.0)
@@ -91,7 +91,7 @@ GEM
91
91
  thread_safe (0.1.2)
92
92
  atomic
93
93
  tilt (1.4.1)
94
- treetop (1.4.14)
94
+ treetop (1.4.15)
95
95
  polyglot
96
96
  polyglot (>= 0.3.1)
97
97
  tzinfo (0.3.37)
data/README.md CHANGED
@@ -18,22 +18,34 @@ Run the bundle command
18
18
 
19
19
  bundle
20
20
 
21
+ ## IMPORTANT note about recent versions
22
+
23
+ * Rails 4 users should use the version 0.7 and onward, while if you use Rails 3.2 stick with the 0.6 versions.
24
+ * Since version 0.8 all methods lost the prefix pg_ and they should be included in models thourgh acts_as_copy_target.
25
+
21
26
  ## Usage
22
27
 
23
- The gem will add two aditiontal class methods to ActiveRecord::Base:
28
+ To enable the copy commands in an ActiveRecord model called User you should use:
29
+ ```ruby
30
+ class User < ActiveRecord::Base
31
+ acts_as_copy_target
32
+ end
33
+ ```
34
+
35
+ This will add the aditiontal class methods to your model:
24
36
 
25
- * pg_copy_to
26
- * pg_copy_to_string
27
- * pg_copy_from
37
+ * copy_to
38
+ * copy_to_string
39
+ * copy_from
28
40
 
29
- ### Using pg_copy_to and pg_copy_to_string
41
+ ### Using copy_to and copy_to_string
30
42
 
31
43
  You can go to the rails console and try some cool things first.
32
44
  The first and most basic use case, let's copy the enteire content of a database table to a CSV file on the database server disk.
33
45
  Assuming we have a users table and a User AR model:
34
46
 
35
47
  ```ruby
36
- User.pg_copy_to '/tmp/users.csv'
48
+ User.copy_to '/tmp/users.csv'
37
49
  ```
38
50
 
39
51
  This will execute in the database the command:
@@ -48,24 +60,24 @@ In this case you can pass a block and retrieve the generated lines and then writ
48
60
 
49
61
  ```ruby
50
62
  File.open('/tmp/users.csv', 'w') do |f|
51
- User.pg_copy_to do |line|
63
+ User.copy_to do |line|
52
64
  f.write line
53
65
  end
54
66
  end
55
67
  ```
56
68
 
57
- Or, if you have enough memory, you can read all table contents to a string using .pg_copy_to_string
69
+ Or, if you have enough memory, you can read all table contents to a string using .copy_to_string
58
70
 
59
71
  ```ruby
60
- puts User.pg_copy_to_string
72
+ puts User.copy_to_string
61
73
  ```
62
74
 
63
- Another insteresting feature of pg_copy_to is that it uses the scoped relation, it means that you can use ARel
75
+ Another insteresting feature of copy_to is that it uses the scoped relation, it means that you can use ARel
64
76
  operations to generate different CSV files according to your needs.
65
77
  Assuming we want to generate a file only with the names of users 1, 2 and 3:
66
78
 
67
79
  ```ruby
68
- User.select("name").where(:id => [1,2,3]).pg_copy_to "/tmp/users.csv"
80
+ User.select("name").where(:id => [1,2,3]).copy_to "/tmp/users.csv"
69
81
  ```
70
82
 
71
83
  Which will generate the following SQL command:
@@ -77,7 +89,7 @@ COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.c
77
89
  The COPY command also supports exporting the data in binary format.
78
90
 
79
91
  ```ruby
80
- User.select("name").where(:id => [1,2,3]).pg_copy_to "/tmp/users.dat", :format => :binary
92
+ User.select("name").where(:id => [1,2,3]).copy_to "/tmp/users.dat", :format => :binary
81
93
  ```
82
94
 
83
95
  Which will generate the following SQL command:
@@ -89,34 +101,34 @@ COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.d
89
101
  The copy_to_string method also supports this
90
102
 
91
103
  ```ruby
92
- puts User.pg_copy_to_string(:format => :binary)
104
+ puts User.copy_to_string(:format => :binary)
93
105
  ```
94
106
 
95
107
 
96
108
 
97
- ### Using pg_copy_from
109
+ ### Using copy_from
98
110
 
99
- Now, if you want to copy data from a CSV file into the database, you can use the pg_copy_from method.
111
+ Now, if you want to copy data from a CSV file into the database, you can use the copy_from method.
100
112
  It will allow you to copy data from an arbritary IO object or from a file in the database server (when you pass the path as string).
101
113
  Let's first copy from a file in the database server, assuming again that we have a users table and
102
114
  that we are in the Rails console:
103
115
 
104
116
  ```ruby
105
- User.pg_copy_from "/tmp/users.csv"
117
+ User.copy_from "/tmp/users.csv"
106
118
  ```
107
119
 
108
120
  This command will use the headers in the CSV file as fields of the target table, so beware to always have a header in the files you want to import.
109
121
  If the column names in the CSV header do not match the field names of the target table, you can pass a map in the options parameter.
110
122
 
111
123
  ```ruby
112
- User.pg_copy_from "/tmp/users.csv", :map => {'name' => 'first_name'}
124
+ User.copy_from "/tmp/users.csv", :map => {'name' => 'first_name'}
113
125
  ```
114
126
 
115
127
  In the above example the header name in the CSV file will be mapped to the field called first_name in the users table.
116
128
  You can also manipulate and modify the values of the file being imported before they enter into the database using a block:
117
129
 
118
130
  ```ruby
119
- User.pg_copy_from "/tmp/users.csv" do |row|
131
+ User.copy_from "/tmp/users.csv" do |row|
120
132
  row[0] = "fixed string"
121
133
  end
122
134
  ```
@@ -128,7 +140,7 @@ For each iteration of the block row receives an array with the same order as the
128
140
  To copy a binary formatted data file or IO object you can specify the format as binary
129
141
 
130
142
  ```ruby
131
- User.pg_copy_from "/tmp/users.dat", :format => :binary
143
+ User.copy_from "/tmp/users.dat", :format => :binary
132
144
  ```
133
145
 
134
146
  NOTE: Columns must line up with the table unless you specify how they map to table columns.
@@ -136,7 +148,7 @@ NOTE: Columns must line up with the table unless you specify how they map to tab
136
148
  To specify how the columns will map to the table you can specify the :columns option
137
149
 
138
150
  ```ruby
139
- User.pg_copy_from "/tmp/users.dat", :format => :binary, :columns => [:id, :name]
151
+ User.copy_from "/tmp/users.dat", :format => :binary, :columns => [:id, :name]
140
152
  ```
141
153
 
142
154
  Which will generate the following SQL command:
data/lib/postgres-copy.rb CHANGED
@@ -1,16 +1,10 @@
1
1
  require 'rubygems'
2
- require 'active_record'
3
- require 'postgres-copy/active_record'
4
- require 'rails'
2
+ require 'active_support'
5
3
 
6
- class PostgresCopy < Rails::Railtie
4
+ ActiveSupport.on_load :active_record do
5
+ require "postgres-copy/acts_as_copy_target"
6
+ end
7
7
 
8
- initializer 'postgres-copy' do
9
- ActiveSupport.on_load :active_record do
10
- require "postgres-copy/active_record"
11
- end
12
- ActiveSupport.on_load :action_controller do
13
- require "postgres-copy/csv_responder"
14
- end
15
- end
8
+ ActiveSupport.on_load :action_controller do
9
+ require "postgres-copy/csv_responder"
16
10
  end
@@ -0,0 +1,103 @@
1
+ module PostgresCopy
2
+ module ActsAsCopyTarget
3
+ extend ActiveSupport::Concern
4
+
5
+ included do
6
+ end
7
+
8
+ module CopyMethods
9
+ # Copy data to a file passed as a string (the file path) or to lines that are passed to a block
10
+ def copy_to path = nil, options = {}
11
+ options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
12
+ options_string = if options[:format] == :binary
13
+ "BINARY"
14
+ else
15
+ "DELIMITER '#{options[:delimiter]}' CSV #{options[:header] ? 'HEADER' : ''}"
16
+ end
17
+
18
+ if path
19
+ raise "You have to choose between exporting to a file or receiving the lines inside a block" if block_given?
20
+ connection.execute "COPY (#{self.all.to_sql}) TO #{sanitize(path)} WITH #{options_string}"
21
+ else
22
+ connection.execute "COPY (#{self.all.to_sql}) TO STDOUT WITH #{options_string}"
23
+ while line = connection.raw_connection.get_copy_data do
24
+ yield(line) if block_given?
25
+ end
26
+ end
27
+ return self
28
+ end
29
+
30
+ # Copy all data to a single string
31
+ def copy_to_string options = {}
32
+ data = ''
33
+ self.copy_to(nil, options){|l| data << l }
34
+ if options[:format] == :binary
35
+ data.force_encoding("ASCII-8BIT")
36
+ end
37
+ data
38
+ end
39
+
40
+ # Copy data from a CSV that can be passed as a string (the file path) or as an IO object.
41
+ # * You can change the default delimiter passing delimiter: '' in the options hash
42
+ # * You can map fields from the file to different fields in the table using a map in the options hash
43
+ # * For further details on usage take a look at the README.md
44
+ def copy_from path_or_io, options = {}
45
+ options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
46
+ options_string = if options[:format] == :binary
47
+ "BINARY"
48
+ else
49
+ "DELIMITER '#{options[:delimiter]}' CSV"
50
+ end
51
+ io = path_or_io.instance_of?(String) ? File.open(path_or_io, 'r') : path_or_io
52
+
53
+ if options[:format] == :binary
54
+ columns_list = options[:columns] || []
55
+ elsif options[:header]
56
+ line = io.gets
57
+ columns_list = options[:columns] || line.strip.split(options[:delimiter])
58
+ else
59
+ columns_list = options[:columns]
60
+ end
61
+
62
+ table = if options[:table]
63
+ connection.quote_table_name(options[:table])
64
+ else
65
+ quoted_table_name
66
+ end
67
+
68
+ columns_list = columns_list.map{|c| options[:map][c.to_s] } if options[:map]
69
+ columns_string = columns_list.size > 0 ? "(\"#{columns_list.join('","')}\")" : ""
70
+ connection.execute %{COPY #{table} #{columns_string} FROM STDIN #{options_string}}
71
+ if options[:format] == :binary
72
+ bytes = 0
73
+ begin
74
+ while line = io.readpartial(10240)
75
+ connection.raw_connection.put_copy_data line
76
+ bytes += line.bytesize
77
+ end
78
+ rescue EOFError
79
+ end
80
+ else
81
+ while line = io.gets do
82
+ next if line.strip.size == 0
83
+ if block_given?
84
+ row = line.strip.split(options[:delimiter])
85
+ yield(row)
86
+ line = row.join(options[:delimiter]) + "\n"
87
+ end
88
+ connection.raw_connection.put_copy_data line
89
+ end
90
+ end
91
+ connection.raw_connection.put_copy_end
92
+ end
93
+ end
94
+
95
+ module ClassMethods
96
+ def acts_as_copy_target
97
+ extend PostgresCopy::ActsAsCopyTarget::CopyMethods
98
+ end
99
+ end
100
+ end
101
+ end
102
+
103
+ ActiveRecord::Base.send :include, PostgresCopy::ActsAsCopyTarget
@@ -2,7 +2,7 @@ module Responders
2
2
  module CsvResponder
3
3
  def to_csv
4
4
  controller.response_body = Enumerator.new do |y|
5
- controller.send(:end_of_association_chain).pg_copy_to do |line|
5
+ controller.send(:end_of_association_chain).copy_to do |line|
6
6
  y << line
7
7
  end
8
8
  end
@@ -5,7 +5,7 @@ $:.unshift lib unless $:.include?(lib)
5
5
 
6
6
  Gem::Specification.new do |s|
7
7
  s.name = "postgres-copy"
8
- s.version = "0.7.0"
8
+ s.version = "0.8.0"
9
9
 
10
10
  s.platform = Gem::Platform::RUBY
11
11
  s.required_ruby_version = ">= 1.9.3"
@@ -9,12 +9,12 @@ describe "COPY FROM BINARY" do
9
9
  end
10
10
 
11
11
  it "should import from file if path is passed without field_map" do
12
- TestModel.pg_copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary, columns: [:id, :data]
12
+ TestModel.copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary, columns: [:id, :data]
13
13
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'text'}]
14
14
  end
15
15
 
16
16
  it "should import from file if columns are not specified" do
17
- TestModel.pg_copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary
17
+ TestModel.copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary
18
18
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'text'}]
19
19
  end
20
20
 
@@ -9,34 +9,34 @@ describe "COPY FROM" do
9
9
  end
10
10
 
11
11
  it "should import from file if path is passed without field_map" do
12
- TestModel.pg_copy_from File.expand_path('spec/fixtures/comma_with_header.csv')
12
+ TestModel.copy_from File.expand_path('spec/fixtures/comma_with_header.csv')
13
13
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
14
14
  end
15
15
 
16
16
  it "should import from IO without field_map" do
17
- TestModel.pg_copy_from File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')
17
+ TestModel.copy_from File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')
18
18
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
19
19
  end
20
20
 
21
21
  it "should import with custom delimiter from path" do
22
- TestModel.pg_copy_from File.expand_path('spec/fixtures/semicolon_with_header.csv'), :delimiter => ';'
22
+ TestModel.copy_from File.expand_path('spec/fixtures/semicolon_with_header.csv'), :delimiter => ';'
23
23
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
24
24
  end
25
25
 
26
26
  it "should import with custom delimiter from IO" do
27
- TestModel.pg_copy_from File.open(File.expand_path('spec/fixtures/semicolon_with_header.csv'), 'r'), :delimiter => ';'
27
+ TestModel.copy_from File.open(File.expand_path('spec/fixtures/semicolon_with_header.csv'), 'r'), :delimiter => ';'
28
28
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
29
29
  end
30
30
 
31
31
  it "should import and allow changes in block" do
32
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')) do |row|
32
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')) do |row|
33
33
  row[1] = 'changed this data'
34
34
  end
35
35
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'changed this data'}]
36
36
  end
37
37
 
38
38
  it "should import 2 lines and allow changes in block" do
39
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/tab_with_two_lines.csv'), 'r'), :delimiter => "\t") do |row|
39
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_two_lines.csv'), 'r'), :delimiter => "\t") do |row|
40
40
  row[1] = 'changed this data'
41
41
  end
42
42
  TestModel.order(:id).first.attributes.should == {'id' => 1, 'data' => 'changed this data'}
@@ -44,50 +44,50 @@ describe "COPY FROM" do
44
44
  end
45
45
 
46
46
  it "should be able to copy from using custom set of columns" do
47
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/tab_only_data.csv'), 'r'), :delimiter => "\t", :columns => ["data"])
47
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_only_data.csv'), 'r'), :delimiter => "\t", :columns => ["data"])
48
48
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
49
49
  end
50
50
 
51
51
  it "default set of columns should be all table columns minus [id, created_at, updated_at]" do
52
- ExtraField.pg_copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r'))
52
+ ExtraField.copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r'))
53
53
  ExtraField.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1', 'created_at' => nil, 'updated_at' => nil}]
54
54
  end
55
55
 
56
56
  it "should not expect a header when :header is false" do
57
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data])
57
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data])
58
58
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
59
59
  end
60
60
 
61
61
  it "should use the table name given by :table" do
62
- ActiveRecord::Base.pg_copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data], :table => "test_models")
62
+ ExtraField.copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data], :table => "test_models")
63
63
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
64
64
  end
65
65
 
66
66
  it "should be able to map the header in the file to diferent column names" do
67
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/tab_with_different_header.csv'), 'r'), :delimiter => "\t", :map => {'cod' => 'id', 'info' => 'data'})
67
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_different_header.csv'), 'r'), :delimiter => "\t", :map => {'cod' => 'id', 'info' => 'data'})
68
68
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
69
69
  end
70
70
 
71
71
  it "should be able to map the header in the file to diferent column names with custom delimiter" do
72
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/semicolon_with_different_header.csv'), 'r'), :delimiter => ';', :map => {'cod' => 'id', 'info' => 'data'})
72
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/semicolon_with_different_header.csv'), 'r'), :delimiter => ';', :map => {'cod' => 'id', 'info' => 'data'})
73
73
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
74
74
  end
75
75
 
76
76
  it "should ignore empty lines" do
77
- TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/tab_with_extra_line.csv'), 'r'), :delimiter => "\t")
77
+ TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_extra_line.csv'), 'r'), :delimiter => "\t")
78
78
  TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
79
79
  end
80
80
 
81
81
  #we should implement this later
82
82
  #it "should raise error in malformed files" do
83
83
  #lambda do
84
- #TestModel.pg_copy_from(File.open(File.expand_path('spec/fixtures/tab_with_error.csv'), 'r'))
84
+ #TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_error.csv'), 'r'))
85
85
  #end.should raise_error
86
86
  #TestModel.order(:id).map{|r| r.attributes}.should == []
87
87
  #end
88
88
 
89
89
  it "should copy from even when table fields need identifier quoting" do
90
- ReservedWordModel.pg_copy_from File.expand_path('spec/fixtures/reserved_words.csv'), :delimiter => "\t"
90
+ ReservedWordModel.copy_from File.expand_path('spec/fixtures/reserved_words.csv'), :delimiter => "\t"
91
91
  ReservedWordModel.order(:id).map{|r| r.attributes}.should == [{"group"=>"group name", "id"=>1, "select"=>"test select"}]
92
92
  end
93
93
  end
@@ -11,18 +11,18 @@ describe "COPY TO BINARY" do
11
11
 
12
12
  describe "should allow binary output to string" do
13
13
  context "with only binary option" do
14
- subject{ TestModel.pg_copy_to_string(:format => :binary) }
14
+ subject{ TestModel.copy_to_string(:format => :binary) }
15
15
  it{ should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read }
16
16
  end
17
17
  context "with custom select" do
18
- subject{ TestModel.select("id, data").pg_copy_to_string(:format => :binary) }
18
+ subject{ TestModel.select("id, data").copy_to_string(:format => :binary) }
19
19
  it{ should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read }
20
20
  end
21
21
  end
22
22
 
23
23
  describe "should allow binary output to file" do
24
24
  it "should copy to disk if block is not given and a path is passed" do
25
- TestModel.pg_copy_to '/tmp/export.dat', :format => :binary
25
+ TestModel.copy_to '/tmp/export.dat', :format => :binary
26
26
  str = File.open('/tmp/export.dat', 'r:ASCII-8BIT').read
27
27
 
28
28
  str.should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read
@@ -9,29 +9,29 @@ describe "COPY TO" do
9
9
  TestModel.create :data => 'test data 1'
10
10
  end
11
11
 
12
- describe ".pg_copy_to_string" do
12
+ describe ".copy_to_string" do
13
13
  context "with no options" do
14
- subject{ TestModel.pg_copy_to_string }
14
+ subject{ TestModel.copy_to_string }
15
15
  it{ should == File.open('spec/fixtures/comma_with_header.csv', 'r').read }
16
16
  end
17
17
 
18
18
  context "with tab as delimiter" do
19
- subject{ TestModel.pg_copy_to_string :delimiter => "\t" }
19
+ subject{ TestModel.copy_to_string :delimiter => "\t" }
20
20
  it{ should == File.open('spec/fixtures/tab_with_header.csv', 'r').read }
21
21
  end
22
22
  end
23
23
 
24
- describe ".pg_copy_to" do
24
+ describe ".copy_to" do
25
25
  it "should copy and pass data to block if block is given and no path is passed" do
26
26
  File.open('spec/fixtures/comma_with_header.csv', 'r') do |f|
27
- TestModel.pg_copy_to do |row|
27
+ TestModel.copy_to do |row|
28
28
  row.should == f.readline
29
29
  end
30
30
  end
31
31
  end
32
32
 
33
33
  it "should copy to disk if block is not given and a path is passed" do
34
- TestModel.pg_copy_to '/tmp/export.csv'
34
+ TestModel.copy_to '/tmp/export.csv'
35
35
  File.open('spec/fixtures/comma_with_header.csv', 'r') do |fixture|
36
36
  File.open('/tmp/export.csv', 'r') do |result|
37
37
  result.read.should == fixture.read
@@ -41,7 +41,7 @@ describe "COPY TO" do
41
41
 
42
42
  it "should raise exception if I pass a path and a block simultaneously" do
43
43
  lambda do
44
- TestModel.pg_copy_to('/tmp/bogus_path') do |row|
44
+ TestModel.copy_to('/tmp/bogus_path') do |row|
45
45
  end
46
46
  end.should raise_error
47
47
  end
@@ -1,5 +1,6 @@
1
1
  require 'postgres-copy'
2
2
 
3
3
  class ExtraField < ActiveRecord::Base
4
+ acts_as_copy_target
4
5
  end
5
6
 
@@ -1,5 +1,6 @@
1
1
  require 'postgres-copy'
2
2
 
3
3
  class ReservedWordModel < ActiveRecord::Base
4
+ acts_as_copy_target
4
5
  end
5
6
 
@@ -1,4 +1,5 @@
1
1
  require 'postgres-copy'
2
2
 
3
3
  class TestModel < ActiveRecord::Base
4
+ acts_as_copy_target
4
5
  end
data/spec/spec_helper.rb CHANGED
@@ -1,5 +1,6 @@
1
1
  $LOAD_PATH.unshift(File.dirname(__FILE__))
2
2
  $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
3
+ require 'active_record'
3
4
  require 'fixtures/test_model'
4
5
  require 'fixtures/extra_field'
5
6
  require 'fixtures/reserved_word_model'
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: postgres-copy
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.7.0
4
+ version: 0.8.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Diogo Biazus
@@ -126,9 +126,13 @@ files:
126
126
  - Rakefile
127
127
  - VERSION
128
128
  - lib/postgres-copy.rb
129
- - lib/postgres-copy/active_record.rb
129
+ - lib/postgres-copy/acts_as_copy_target.rb
130
130
  - lib/postgres-copy/csv_responder.rb
131
131
  - postgres-copy.gemspec
132
+ - spec/copy_from_binary_spec.rb
133
+ - spec/copy_from_spec.rb
134
+ - spec/copy_to_binary_spec.rb
135
+ - spec/copy_to_spec.rb
132
136
  - spec/fixtures/2_col_binary_data.dat
133
137
  - spec/fixtures/comma_with_header.csv
134
138
  - spec/fixtures/comma_without_header.csv
@@ -144,10 +148,6 @@ files:
144
148
  - spec/fixtures/tab_with_header.csv
145
149
  - spec/fixtures/tab_with_two_lines.csv
146
150
  - spec/fixtures/test_model.rb
147
- - spec/pg_copy_from_binary_spec.rb
148
- - spec/pg_copy_from_spec.rb
149
- - spec/pg_copy_to_binary_spec.rb
150
- - spec/pg_copy_to_spec.rb
151
151
  - spec/spec.opts
152
152
  - spec/spec_helper.rb
153
153
  homepage: http://github.com/diogob/postgres-copy
@@ -174,6 +174,10 @@ signing_key:
174
174
  specification_version: 4
175
175
  summary: Put COPY command functionality in ActiveRecord's model class
176
176
  test_files:
177
+ - spec/copy_from_binary_spec.rb
178
+ - spec/copy_from_spec.rb
179
+ - spec/copy_to_binary_spec.rb
180
+ - spec/copy_to_spec.rb
177
181
  - spec/fixtures/2_col_binary_data.dat
178
182
  - spec/fixtures/comma_with_header.csv
179
183
  - spec/fixtures/comma_without_header.csv
@@ -189,9 +193,5 @@ test_files:
189
193
  - spec/fixtures/tab_with_header.csv
190
194
  - spec/fixtures/tab_with_two_lines.csv
191
195
  - spec/fixtures/test_model.rb
192
- - spec/pg_copy_from_binary_spec.rb
193
- - spec/pg_copy_from_spec.rb
194
- - spec/pg_copy_to_binary_spec.rb
195
- - spec/pg_copy_to_spec.rb
196
196
  - spec/spec.opts
197
197
  - spec/spec_helper.rb
@@ -1,88 +0,0 @@
1
- module ActiveRecord
2
- class Base
3
- # Copy data to a file passed as a string (the file path) or to lines that are passed to a block
4
- def self.pg_copy_to path = nil, options = {}
5
- options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
6
- options_string = if options[:format] == :binary
7
- "BINARY"
8
- else
9
- "DELIMITER '#{options[:delimiter]}' CSV #{options[:header] ? 'HEADER' : ''}"
10
- end
11
-
12
- if path
13
- raise "You have to choose between exporting to a file or receiving the lines inside a block" if block_given?
14
- connection.execute "COPY (#{self.all.to_sql}) TO #{sanitize(path)} WITH #{options_string}"
15
- else
16
- connection.execute "COPY (#{self.all.to_sql}) TO STDOUT WITH #{options_string}"
17
- while line = connection.raw_connection.get_copy_data do
18
- yield(line) if block_given?
19
- end
20
- end
21
- return self
22
- end
23
-
24
- # Copy all data to a single string
25
- def self.pg_copy_to_string options = {}
26
- data = ''
27
- self.pg_copy_to(nil, options){|l| data << l }
28
- if options[:format] == :binary
29
- data.force_encoding("ASCII-8BIT")
30
- end
31
- data
32
- end
33
-
34
- # Copy data from a CSV that can be passed as a string (the file path) or as an IO object.
35
- # * You can change the default delimiter passing delimiter: '' in the options hash
36
- # * You can map fields from the file to different fields in the table using a map in the options hash
37
- # * For further details on usage take a look at the README.md
38
- def self.pg_copy_from path_or_io, options = {}
39
- options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
40
- options_string = if options[:format] == :binary
41
- "BINARY"
42
- else
43
- "DELIMITER '#{options[:delimiter]}' CSV"
44
- end
45
- io = path_or_io.instance_of?(String) ? File.open(path_or_io, 'r') : path_or_io
46
-
47
- if options[:format] == :binary
48
- columns_list = options[:columns] || []
49
- elsif options[:header]
50
- line = io.gets
51
- columns_list = options[:columns] || line.strip.split(options[:delimiter])
52
- else
53
- columns_list = options[:columns]
54
- end
55
-
56
- table = if options[:table]
57
- connection.quote_table_name(options[:table])
58
- else
59
- quoted_table_name
60
- end
61
-
62
- columns_list = columns_list.map{|c| options[:map][c.to_s] } if options[:map]
63
- columns_string = columns_list.size > 0 ? "(\"#{columns_list.join('","')}\")" : ""
64
- connection.execute %{COPY #{table} #{columns_string} FROM STDIN #{options_string}}
65
- if options[:format] == :binary
66
- bytes = 0
67
- begin
68
- while line = io.readpartial(10240)
69
- connection.raw_connection.put_copy_data line
70
- bytes += line.bytesize
71
- end
72
- rescue EOFError
73
- end
74
- else
75
- while line = io.gets do
76
- next if line.strip.size == 0
77
- if block_given?
78
- row = line.strip.split(options[:delimiter])
79
- yield(row)
80
- line = row.join(options[:delimiter]) + "\n"
81
- end
82
- connection.raw_connection.put_copy_data line
83
- end
84
- end
85
- connection.raw_connection.put_copy_end
86
- end
87
- end
88
- end