postgres-copy 0.7.0 → 0.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile.lock +3 -3
- data/README.md +32 -20
- data/lib/postgres-copy.rb +6 -12
- data/lib/postgres-copy/acts_as_copy_target.rb +103 -0
- data/lib/postgres-copy/csv_responder.rb +1 -1
- data/postgres-copy.gemspec +1 -1
- data/spec/{pg_copy_from_binary_spec.rb → copy_from_binary_spec.rb} +2 -2
- data/spec/{pg_copy_from_spec.rb → copy_from_spec.rb} +15 -15
- data/spec/{pg_copy_to_binary_spec.rb → copy_to_binary_spec.rb} +3 -3
- data/spec/{pg_copy_to_spec.rb → copy_to_spec.rb} +7 -7
- data/spec/fixtures/extra_field.rb +1 -0
- data/spec/fixtures/reserved_word_model.rb +1 -0
- data/spec/fixtures/test_model.rb +1 -0
- data/spec/spec_helper.rb +1 -0
- metadata +10 -10
- data/lib/postgres-copy/active_record.rb +0 -88
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4b5212784e1c11d62f3996812f369a8702bff96b
|
4
|
+
data.tar.gz: 7bedcee0f0de6847cb8b35c0cedcc5b6ba3c801a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5a7ce457d28c10521ab7ba677eb71f20d29e640783f77499f40a8a192c532b4520a567deda0bb37155f5d135780cbed1fb13a4445b3eddac1ea8626a004df888
|
7
|
+
data.tar.gz: 1c0c02a36b775e637aff34802e6890dd74e1e191b72a2e8e07f28c58a3ce69b3e78f70c75281b4ae80035cab09a060da8ad6562750b5ef2f8381d520d6bc8ca5
|
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
postgres-copy (0.
|
4
|
+
postgres-copy (0.8.0)
|
5
5
|
activerecord (~> 4.0)
|
6
6
|
pg
|
7
7
|
rails (~> 4.0)
|
@@ -45,7 +45,7 @@ GEM
|
|
45
45
|
mail (2.5.4)
|
46
46
|
mime-types (~> 1.16)
|
47
47
|
treetop (~> 1.4.8)
|
48
|
-
mime-types (1.
|
48
|
+
mime-types (1.25)
|
49
49
|
minitest (4.7.5)
|
50
50
|
multi_json (1.7.9)
|
51
51
|
pg (0.16.0)
|
@@ -91,7 +91,7 @@ GEM
|
|
91
91
|
thread_safe (0.1.2)
|
92
92
|
atomic
|
93
93
|
tilt (1.4.1)
|
94
|
-
treetop (1.4.
|
94
|
+
treetop (1.4.15)
|
95
95
|
polyglot
|
96
96
|
polyglot (>= 0.3.1)
|
97
97
|
tzinfo (0.3.37)
|
data/README.md
CHANGED
@@ -18,22 +18,34 @@ Run the bundle command
|
|
18
18
|
|
19
19
|
bundle
|
20
20
|
|
21
|
+
## IMPORTANT note about recent versions
|
22
|
+
|
23
|
+
* Rails 4 users should use the version 0.7 and onward, while if you use Rails 3.2 stick with the 0.6 versions.
|
24
|
+
* Since version 0.8 all methods lost the prefix pg_ and they should be included in models thourgh acts_as_copy_target.
|
25
|
+
|
21
26
|
## Usage
|
22
27
|
|
23
|
-
|
28
|
+
To enable the copy commands in an ActiveRecord model called User you should use:
|
29
|
+
```ruby
|
30
|
+
class User < ActiveRecord::Base
|
31
|
+
acts_as_copy_target
|
32
|
+
end
|
33
|
+
```
|
34
|
+
|
35
|
+
This will add the aditiontal class methods to your model:
|
24
36
|
|
25
|
-
*
|
26
|
-
*
|
27
|
-
*
|
37
|
+
* copy_to
|
38
|
+
* copy_to_string
|
39
|
+
* copy_from
|
28
40
|
|
29
|
-
### Using
|
41
|
+
### Using copy_to and copy_to_string
|
30
42
|
|
31
43
|
You can go to the rails console and try some cool things first.
|
32
44
|
The first and most basic use case, let's copy the enteire content of a database table to a CSV file on the database server disk.
|
33
45
|
Assuming we have a users table and a User AR model:
|
34
46
|
|
35
47
|
```ruby
|
36
|
-
User.
|
48
|
+
User.copy_to '/tmp/users.csv'
|
37
49
|
```
|
38
50
|
|
39
51
|
This will execute in the database the command:
|
@@ -48,24 +60,24 @@ In this case you can pass a block and retrieve the generated lines and then writ
|
|
48
60
|
|
49
61
|
```ruby
|
50
62
|
File.open('/tmp/users.csv', 'w') do |f|
|
51
|
-
User.
|
63
|
+
User.copy_to do |line|
|
52
64
|
f.write line
|
53
65
|
end
|
54
66
|
end
|
55
67
|
```
|
56
68
|
|
57
|
-
Or, if you have enough memory, you can read all table contents to a string using .
|
69
|
+
Or, if you have enough memory, you can read all table contents to a string using .copy_to_string
|
58
70
|
|
59
71
|
```ruby
|
60
|
-
puts User.
|
72
|
+
puts User.copy_to_string
|
61
73
|
```
|
62
74
|
|
63
|
-
Another insteresting feature of
|
75
|
+
Another insteresting feature of copy_to is that it uses the scoped relation, it means that you can use ARel
|
64
76
|
operations to generate different CSV files according to your needs.
|
65
77
|
Assuming we want to generate a file only with the names of users 1, 2 and 3:
|
66
78
|
|
67
79
|
```ruby
|
68
|
-
User.select("name").where(:id => [1,2,3]).
|
80
|
+
User.select("name").where(:id => [1,2,3]).copy_to "/tmp/users.csv"
|
69
81
|
```
|
70
82
|
|
71
83
|
Which will generate the following SQL command:
|
@@ -77,7 +89,7 @@ COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.c
|
|
77
89
|
The COPY command also supports exporting the data in binary format.
|
78
90
|
|
79
91
|
```ruby
|
80
|
-
User.select("name").where(:id => [1,2,3]).
|
92
|
+
User.select("name").where(:id => [1,2,3]).copy_to "/tmp/users.dat", :format => :binary
|
81
93
|
```
|
82
94
|
|
83
95
|
Which will generate the following SQL command:
|
@@ -89,34 +101,34 @@ COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.d
|
|
89
101
|
The copy_to_string method also supports this
|
90
102
|
|
91
103
|
```ruby
|
92
|
-
puts User.
|
104
|
+
puts User.copy_to_string(:format => :binary)
|
93
105
|
```
|
94
106
|
|
95
107
|
|
96
108
|
|
97
|
-
### Using
|
109
|
+
### Using copy_from
|
98
110
|
|
99
|
-
Now, if you want to copy data from a CSV file into the database, you can use the
|
111
|
+
Now, if you want to copy data from a CSV file into the database, you can use the copy_from method.
|
100
112
|
It will allow you to copy data from an arbritary IO object or from a file in the database server (when you pass the path as string).
|
101
113
|
Let's first copy from a file in the database server, assuming again that we have a users table and
|
102
114
|
that we are in the Rails console:
|
103
115
|
|
104
116
|
```ruby
|
105
|
-
User.
|
117
|
+
User.copy_from "/tmp/users.csv"
|
106
118
|
```
|
107
119
|
|
108
120
|
This command will use the headers in the CSV file as fields of the target table, so beware to always have a header in the files you want to import.
|
109
121
|
If the column names in the CSV header do not match the field names of the target table, you can pass a map in the options parameter.
|
110
122
|
|
111
123
|
```ruby
|
112
|
-
User.
|
124
|
+
User.copy_from "/tmp/users.csv", :map => {'name' => 'first_name'}
|
113
125
|
```
|
114
126
|
|
115
127
|
In the above example the header name in the CSV file will be mapped to the field called first_name in the users table.
|
116
128
|
You can also manipulate and modify the values of the file being imported before they enter into the database using a block:
|
117
129
|
|
118
130
|
```ruby
|
119
|
-
User.
|
131
|
+
User.copy_from "/tmp/users.csv" do |row|
|
120
132
|
row[0] = "fixed string"
|
121
133
|
end
|
122
134
|
```
|
@@ -128,7 +140,7 @@ For each iteration of the block row receives an array with the same order as the
|
|
128
140
|
To copy a binary formatted data file or IO object you can specify the format as binary
|
129
141
|
|
130
142
|
```ruby
|
131
|
-
User.
|
143
|
+
User.copy_from "/tmp/users.dat", :format => :binary
|
132
144
|
```
|
133
145
|
|
134
146
|
NOTE: Columns must line up with the table unless you specify how they map to table columns.
|
@@ -136,7 +148,7 @@ NOTE: Columns must line up with the table unless you specify how they map to tab
|
|
136
148
|
To specify how the columns will map to the table you can specify the :columns option
|
137
149
|
|
138
150
|
```ruby
|
139
|
-
User.
|
151
|
+
User.copy_from "/tmp/users.dat", :format => :binary, :columns => [:id, :name]
|
140
152
|
```
|
141
153
|
|
142
154
|
Which will generate the following SQL command:
|
data/lib/postgres-copy.rb
CHANGED
@@ -1,16 +1,10 @@
|
|
1
1
|
require 'rubygems'
|
2
|
-
require '
|
3
|
-
require 'postgres-copy/active_record'
|
4
|
-
require 'rails'
|
2
|
+
require 'active_support'
|
5
3
|
|
6
|
-
|
4
|
+
ActiveSupport.on_load :active_record do
|
5
|
+
require "postgres-copy/acts_as_copy_target"
|
6
|
+
end
|
7
7
|
|
8
|
-
|
9
|
-
|
10
|
-
require "postgres-copy/active_record"
|
11
|
-
end
|
12
|
-
ActiveSupport.on_load :action_controller do
|
13
|
-
require "postgres-copy/csv_responder"
|
14
|
-
end
|
15
|
-
end
|
8
|
+
ActiveSupport.on_load :action_controller do
|
9
|
+
require "postgres-copy/csv_responder"
|
16
10
|
end
|
@@ -0,0 +1,103 @@
|
|
1
|
+
module PostgresCopy
|
2
|
+
module ActsAsCopyTarget
|
3
|
+
extend ActiveSupport::Concern
|
4
|
+
|
5
|
+
included do
|
6
|
+
end
|
7
|
+
|
8
|
+
module CopyMethods
|
9
|
+
# Copy data to a file passed as a string (the file path) or to lines that are passed to a block
|
10
|
+
def copy_to path = nil, options = {}
|
11
|
+
options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
|
12
|
+
options_string = if options[:format] == :binary
|
13
|
+
"BINARY"
|
14
|
+
else
|
15
|
+
"DELIMITER '#{options[:delimiter]}' CSV #{options[:header] ? 'HEADER' : ''}"
|
16
|
+
end
|
17
|
+
|
18
|
+
if path
|
19
|
+
raise "You have to choose between exporting to a file or receiving the lines inside a block" if block_given?
|
20
|
+
connection.execute "COPY (#{self.all.to_sql}) TO #{sanitize(path)} WITH #{options_string}"
|
21
|
+
else
|
22
|
+
connection.execute "COPY (#{self.all.to_sql}) TO STDOUT WITH #{options_string}"
|
23
|
+
while line = connection.raw_connection.get_copy_data do
|
24
|
+
yield(line) if block_given?
|
25
|
+
end
|
26
|
+
end
|
27
|
+
return self
|
28
|
+
end
|
29
|
+
|
30
|
+
# Copy all data to a single string
|
31
|
+
def copy_to_string options = {}
|
32
|
+
data = ''
|
33
|
+
self.copy_to(nil, options){|l| data << l }
|
34
|
+
if options[:format] == :binary
|
35
|
+
data.force_encoding("ASCII-8BIT")
|
36
|
+
end
|
37
|
+
data
|
38
|
+
end
|
39
|
+
|
40
|
+
# Copy data from a CSV that can be passed as a string (the file path) or as an IO object.
|
41
|
+
# * You can change the default delimiter passing delimiter: '' in the options hash
|
42
|
+
# * You can map fields from the file to different fields in the table using a map in the options hash
|
43
|
+
# * For further details on usage take a look at the README.md
|
44
|
+
def copy_from path_or_io, options = {}
|
45
|
+
options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
|
46
|
+
options_string = if options[:format] == :binary
|
47
|
+
"BINARY"
|
48
|
+
else
|
49
|
+
"DELIMITER '#{options[:delimiter]}' CSV"
|
50
|
+
end
|
51
|
+
io = path_or_io.instance_of?(String) ? File.open(path_or_io, 'r') : path_or_io
|
52
|
+
|
53
|
+
if options[:format] == :binary
|
54
|
+
columns_list = options[:columns] || []
|
55
|
+
elsif options[:header]
|
56
|
+
line = io.gets
|
57
|
+
columns_list = options[:columns] || line.strip.split(options[:delimiter])
|
58
|
+
else
|
59
|
+
columns_list = options[:columns]
|
60
|
+
end
|
61
|
+
|
62
|
+
table = if options[:table]
|
63
|
+
connection.quote_table_name(options[:table])
|
64
|
+
else
|
65
|
+
quoted_table_name
|
66
|
+
end
|
67
|
+
|
68
|
+
columns_list = columns_list.map{|c| options[:map][c.to_s] } if options[:map]
|
69
|
+
columns_string = columns_list.size > 0 ? "(\"#{columns_list.join('","')}\")" : ""
|
70
|
+
connection.execute %{COPY #{table} #{columns_string} FROM STDIN #{options_string}}
|
71
|
+
if options[:format] == :binary
|
72
|
+
bytes = 0
|
73
|
+
begin
|
74
|
+
while line = io.readpartial(10240)
|
75
|
+
connection.raw_connection.put_copy_data line
|
76
|
+
bytes += line.bytesize
|
77
|
+
end
|
78
|
+
rescue EOFError
|
79
|
+
end
|
80
|
+
else
|
81
|
+
while line = io.gets do
|
82
|
+
next if line.strip.size == 0
|
83
|
+
if block_given?
|
84
|
+
row = line.strip.split(options[:delimiter])
|
85
|
+
yield(row)
|
86
|
+
line = row.join(options[:delimiter]) + "\n"
|
87
|
+
end
|
88
|
+
connection.raw_connection.put_copy_data line
|
89
|
+
end
|
90
|
+
end
|
91
|
+
connection.raw_connection.put_copy_end
|
92
|
+
end
|
93
|
+
end
|
94
|
+
|
95
|
+
module ClassMethods
|
96
|
+
def acts_as_copy_target
|
97
|
+
extend PostgresCopy::ActsAsCopyTarget::CopyMethods
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
end
|
102
|
+
|
103
|
+
ActiveRecord::Base.send :include, PostgresCopy::ActsAsCopyTarget
|
data/postgres-copy.gemspec
CHANGED
@@ -9,12 +9,12 @@ describe "COPY FROM BINARY" do
|
|
9
9
|
end
|
10
10
|
|
11
11
|
it "should import from file if path is passed without field_map" do
|
12
|
-
TestModel.
|
12
|
+
TestModel.copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary, columns: [:id, :data]
|
13
13
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'text'}]
|
14
14
|
end
|
15
15
|
|
16
16
|
it "should import from file if columns are not specified" do
|
17
|
-
TestModel.
|
17
|
+
TestModel.copy_from File.expand_path('spec/fixtures/2_col_binary_data.dat'), :format => :binary
|
18
18
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'text'}]
|
19
19
|
end
|
20
20
|
|
@@ -9,34 +9,34 @@ describe "COPY FROM" do
|
|
9
9
|
end
|
10
10
|
|
11
11
|
it "should import from file if path is passed without field_map" do
|
12
|
-
TestModel.
|
12
|
+
TestModel.copy_from File.expand_path('spec/fixtures/comma_with_header.csv')
|
13
13
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
14
14
|
end
|
15
15
|
|
16
16
|
it "should import from IO without field_map" do
|
17
|
-
TestModel.
|
17
|
+
TestModel.copy_from File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')
|
18
18
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
19
19
|
end
|
20
20
|
|
21
21
|
it "should import with custom delimiter from path" do
|
22
|
-
TestModel.
|
22
|
+
TestModel.copy_from File.expand_path('spec/fixtures/semicolon_with_header.csv'), :delimiter => ';'
|
23
23
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
24
24
|
end
|
25
25
|
|
26
26
|
it "should import with custom delimiter from IO" do
|
27
|
-
TestModel.
|
27
|
+
TestModel.copy_from File.open(File.expand_path('spec/fixtures/semicolon_with_header.csv'), 'r'), :delimiter => ';'
|
28
28
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
29
29
|
end
|
30
30
|
|
31
31
|
it "should import and allow changes in block" do
|
32
|
-
TestModel.
|
32
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r')) do |row|
|
33
33
|
row[1] = 'changed this data'
|
34
34
|
end
|
35
35
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'changed this data'}]
|
36
36
|
end
|
37
37
|
|
38
38
|
it "should import 2 lines and allow changes in block" do
|
39
|
-
TestModel.
|
39
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_two_lines.csv'), 'r'), :delimiter => "\t") do |row|
|
40
40
|
row[1] = 'changed this data'
|
41
41
|
end
|
42
42
|
TestModel.order(:id).first.attributes.should == {'id' => 1, 'data' => 'changed this data'}
|
@@ -44,50 +44,50 @@ describe "COPY FROM" do
|
|
44
44
|
end
|
45
45
|
|
46
46
|
it "should be able to copy from using custom set of columns" do
|
47
|
-
TestModel.
|
47
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_only_data.csv'), 'r'), :delimiter => "\t", :columns => ["data"])
|
48
48
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
49
49
|
end
|
50
50
|
|
51
51
|
it "default set of columns should be all table columns minus [id, created_at, updated_at]" do
|
52
|
-
ExtraField.
|
52
|
+
ExtraField.copy_from(File.open(File.expand_path('spec/fixtures/comma_with_header.csv'), 'r'))
|
53
53
|
ExtraField.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1', 'created_at' => nil, 'updated_at' => nil}]
|
54
54
|
end
|
55
55
|
|
56
56
|
it "should not expect a header when :header is false" do
|
57
|
-
TestModel.
|
57
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data])
|
58
58
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
59
59
|
end
|
60
60
|
|
61
61
|
it "should use the table name given by :table" do
|
62
|
-
|
62
|
+
ExtraField.copy_from(File.open(File.expand_path('spec/fixtures/comma_without_header.csv'), 'r'), :header => false, :columns => [:id,:data], :table => "test_models")
|
63
63
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
64
64
|
end
|
65
65
|
|
66
66
|
it "should be able to map the header in the file to diferent column names" do
|
67
|
-
TestModel.
|
67
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_different_header.csv'), 'r'), :delimiter => "\t", :map => {'cod' => 'id', 'info' => 'data'})
|
68
68
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
69
69
|
end
|
70
70
|
|
71
71
|
it "should be able to map the header in the file to diferent column names with custom delimiter" do
|
72
|
-
TestModel.
|
72
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/semicolon_with_different_header.csv'), 'r'), :delimiter => ';', :map => {'cod' => 'id', 'info' => 'data'})
|
73
73
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
74
74
|
end
|
75
75
|
|
76
76
|
it "should ignore empty lines" do
|
77
|
-
TestModel.
|
77
|
+
TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_extra_line.csv'), 'r'), :delimiter => "\t")
|
78
78
|
TestModel.order(:id).map{|r| r.attributes}.should == [{'id' => 1, 'data' => 'test data 1'}]
|
79
79
|
end
|
80
80
|
|
81
81
|
#we should implement this later
|
82
82
|
#it "should raise error in malformed files" do
|
83
83
|
#lambda do
|
84
|
-
#TestModel.
|
84
|
+
#TestModel.copy_from(File.open(File.expand_path('spec/fixtures/tab_with_error.csv'), 'r'))
|
85
85
|
#end.should raise_error
|
86
86
|
#TestModel.order(:id).map{|r| r.attributes}.should == []
|
87
87
|
#end
|
88
88
|
|
89
89
|
it "should copy from even when table fields need identifier quoting" do
|
90
|
-
ReservedWordModel.
|
90
|
+
ReservedWordModel.copy_from File.expand_path('spec/fixtures/reserved_words.csv'), :delimiter => "\t"
|
91
91
|
ReservedWordModel.order(:id).map{|r| r.attributes}.should == [{"group"=>"group name", "id"=>1, "select"=>"test select"}]
|
92
92
|
end
|
93
93
|
end
|
@@ -11,18 +11,18 @@ describe "COPY TO BINARY" do
|
|
11
11
|
|
12
12
|
describe "should allow binary output to string" do
|
13
13
|
context "with only binary option" do
|
14
|
-
subject{ TestModel.
|
14
|
+
subject{ TestModel.copy_to_string(:format => :binary) }
|
15
15
|
it{ should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read }
|
16
16
|
end
|
17
17
|
context "with custom select" do
|
18
|
-
subject{ TestModel.select("id, data").
|
18
|
+
subject{ TestModel.select("id, data").copy_to_string(:format => :binary) }
|
19
19
|
it{ should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read }
|
20
20
|
end
|
21
21
|
end
|
22
22
|
|
23
23
|
describe "should allow binary output to file" do
|
24
24
|
it "should copy to disk if block is not given and a path is passed" do
|
25
|
-
TestModel.
|
25
|
+
TestModel.copy_to '/tmp/export.dat', :format => :binary
|
26
26
|
str = File.open('/tmp/export.dat', 'r:ASCII-8BIT').read
|
27
27
|
|
28
28
|
str.should == File.open('spec/fixtures/2_col_binary_data.dat', 'r:ASCII-8BIT').read
|
@@ -9,29 +9,29 @@ describe "COPY TO" do
|
|
9
9
|
TestModel.create :data => 'test data 1'
|
10
10
|
end
|
11
11
|
|
12
|
-
describe ".
|
12
|
+
describe ".copy_to_string" do
|
13
13
|
context "with no options" do
|
14
|
-
subject{ TestModel.
|
14
|
+
subject{ TestModel.copy_to_string }
|
15
15
|
it{ should == File.open('spec/fixtures/comma_with_header.csv', 'r').read }
|
16
16
|
end
|
17
17
|
|
18
18
|
context "with tab as delimiter" do
|
19
|
-
subject{ TestModel.
|
19
|
+
subject{ TestModel.copy_to_string :delimiter => "\t" }
|
20
20
|
it{ should == File.open('spec/fixtures/tab_with_header.csv', 'r').read }
|
21
21
|
end
|
22
22
|
end
|
23
23
|
|
24
|
-
describe ".
|
24
|
+
describe ".copy_to" do
|
25
25
|
it "should copy and pass data to block if block is given and no path is passed" do
|
26
26
|
File.open('spec/fixtures/comma_with_header.csv', 'r') do |f|
|
27
|
-
TestModel.
|
27
|
+
TestModel.copy_to do |row|
|
28
28
|
row.should == f.readline
|
29
29
|
end
|
30
30
|
end
|
31
31
|
end
|
32
32
|
|
33
33
|
it "should copy to disk if block is not given and a path is passed" do
|
34
|
-
TestModel.
|
34
|
+
TestModel.copy_to '/tmp/export.csv'
|
35
35
|
File.open('spec/fixtures/comma_with_header.csv', 'r') do |fixture|
|
36
36
|
File.open('/tmp/export.csv', 'r') do |result|
|
37
37
|
result.read.should == fixture.read
|
@@ -41,7 +41,7 @@ describe "COPY TO" do
|
|
41
41
|
|
42
42
|
it "should raise exception if I pass a path and a block simultaneously" do
|
43
43
|
lambda do
|
44
|
-
TestModel.
|
44
|
+
TestModel.copy_to('/tmp/bogus_path') do |row|
|
45
45
|
end
|
46
46
|
end.should raise_error
|
47
47
|
end
|
data/spec/fixtures/test_model.rb
CHANGED
data/spec/spec_helper.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: postgres-copy
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Diogo Biazus
|
@@ -126,9 +126,13 @@ files:
|
|
126
126
|
- Rakefile
|
127
127
|
- VERSION
|
128
128
|
- lib/postgres-copy.rb
|
129
|
-
- lib/postgres-copy/
|
129
|
+
- lib/postgres-copy/acts_as_copy_target.rb
|
130
130
|
- lib/postgres-copy/csv_responder.rb
|
131
131
|
- postgres-copy.gemspec
|
132
|
+
- spec/copy_from_binary_spec.rb
|
133
|
+
- spec/copy_from_spec.rb
|
134
|
+
- spec/copy_to_binary_spec.rb
|
135
|
+
- spec/copy_to_spec.rb
|
132
136
|
- spec/fixtures/2_col_binary_data.dat
|
133
137
|
- spec/fixtures/comma_with_header.csv
|
134
138
|
- spec/fixtures/comma_without_header.csv
|
@@ -144,10 +148,6 @@ files:
|
|
144
148
|
- spec/fixtures/tab_with_header.csv
|
145
149
|
- spec/fixtures/tab_with_two_lines.csv
|
146
150
|
- spec/fixtures/test_model.rb
|
147
|
-
- spec/pg_copy_from_binary_spec.rb
|
148
|
-
- spec/pg_copy_from_spec.rb
|
149
|
-
- spec/pg_copy_to_binary_spec.rb
|
150
|
-
- spec/pg_copy_to_spec.rb
|
151
151
|
- spec/spec.opts
|
152
152
|
- spec/spec_helper.rb
|
153
153
|
homepage: http://github.com/diogob/postgres-copy
|
@@ -174,6 +174,10 @@ signing_key:
|
|
174
174
|
specification_version: 4
|
175
175
|
summary: Put COPY command functionality in ActiveRecord's model class
|
176
176
|
test_files:
|
177
|
+
- spec/copy_from_binary_spec.rb
|
178
|
+
- spec/copy_from_spec.rb
|
179
|
+
- spec/copy_to_binary_spec.rb
|
180
|
+
- spec/copy_to_spec.rb
|
177
181
|
- spec/fixtures/2_col_binary_data.dat
|
178
182
|
- spec/fixtures/comma_with_header.csv
|
179
183
|
- spec/fixtures/comma_without_header.csv
|
@@ -189,9 +193,5 @@ test_files:
|
|
189
193
|
- spec/fixtures/tab_with_header.csv
|
190
194
|
- spec/fixtures/tab_with_two_lines.csv
|
191
195
|
- spec/fixtures/test_model.rb
|
192
|
-
- spec/pg_copy_from_binary_spec.rb
|
193
|
-
- spec/pg_copy_from_spec.rb
|
194
|
-
- spec/pg_copy_to_binary_spec.rb
|
195
|
-
- spec/pg_copy_to_spec.rb
|
196
196
|
- spec/spec.opts
|
197
197
|
- spec/spec_helper.rb
|
@@ -1,88 +0,0 @@
|
|
1
|
-
module ActiveRecord
|
2
|
-
class Base
|
3
|
-
# Copy data to a file passed as a string (the file path) or to lines that are passed to a block
|
4
|
-
def self.pg_copy_to path = nil, options = {}
|
5
|
-
options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
|
6
|
-
options_string = if options[:format] == :binary
|
7
|
-
"BINARY"
|
8
|
-
else
|
9
|
-
"DELIMITER '#{options[:delimiter]}' CSV #{options[:header] ? 'HEADER' : ''}"
|
10
|
-
end
|
11
|
-
|
12
|
-
if path
|
13
|
-
raise "You have to choose between exporting to a file or receiving the lines inside a block" if block_given?
|
14
|
-
connection.execute "COPY (#{self.all.to_sql}) TO #{sanitize(path)} WITH #{options_string}"
|
15
|
-
else
|
16
|
-
connection.execute "COPY (#{self.all.to_sql}) TO STDOUT WITH #{options_string}"
|
17
|
-
while line = connection.raw_connection.get_copy_data do
|
18
|
-
yield(line) if block_given?
|
19
|
-
end
|
20
|
-
end
|
21
|
-
return self
|
22
|
-
end
|
23
|
-
|
24
|
-
# Copy all data to a single string
|
25
|
-
def self.pg_copy_to_string options = {}
|
26
|
-
data = ''
|
27
|
-
self.pg_copy_to(nil, options){|l| data << l }
|
28
|
-
if options[:format] == :binary
|
29
|
-
data.force_encoding("ASCII-8BIT")
|
30
|
-
end
|
31
|
-
data
|
32
|
-
end
|
33
|
-
|
34
|
-
# Copy data from a CSV that can be passed as a string (the file path) or as an IO object.
|
35
|
-
# * You can change the default delimiter passing delimiter: '' in the options hash
|
36
|
-
# * You can map fields from the file to different fields in the table using a map in the options hash
|
37
|
-
# * For further details on usage take a look at the README.md
|
38
|
-
def self.pg_copy_from path_or_io, options = {}
|
39
|
-
options = {:delimiter => ",", :format => :csv, :header => true}.merge(options)
|
40
|
-
options_string = if options[:format] == :binary
|
41
|
-
"BINARY"
|
42
|
-
else
|
43
|
-
"DELIMITER '#{options[:delimiter]}' CSV"
|
44
|
-
end
|
45
|
-
io = path_or_io.instance_of?(String) ? File.open(path_or_io, 'r') : path_or_io
|
46
|
-
|
47
|
-
if options[:format] == :binary
|
48
|
-
columns_list = options[:columns] || []
|
49
|
-
elsif options[:header]
|
50
|
-
line = io.gets
|
51
|
-
columns_list = options[:columns] || line.strip.split(options[:delimiter])
|
52
|
-
else
|
53
|
-
columns_list = options[:columns]
|
54
|
-
end
|
55
|
-
|
56
|
-
table = if options[:table]
|
57
|
-
connection.quote_table_name(options[:table])
|
58
|
-
else
|
59
|
-
quoted_table_name
|
60
|
-
end
|
61
|
-
|
62
|
-
columns_list = columns_list.map{|c| options[:map][c.to_s] } if options[:map]
|
63
|
-
columns_string = columns_list.size > 0 ? "(\"#{columns_list.join('","')}\")" : ""
|
64
|
-
connection.execute %{COPY #{table} #{columns_string} FROM STDIN #{options_string}}
|
65
|
-
if options[:format] == :binary
|
66
|
-
bytes = 0
|
67
|
-
begin
|
68
|
-
while line = io.readpartial(10240)
|
69
|
-
connection.raw_connection.put_copy_data line
|
70
|
-
bytes += line.bytesize
|
71
|
-
end
|
72
|
-
rescue EOFError
|
73
|
-
end
|
74
|
-
else
|
75
|
-
while line = io.gets do
|
76
|
-
next if line.strip.size == 0
|
77
|
-
if block_given?
|
78
|
-
row = line.strip.split(options[:delimiter])
|
79
|
-
yield(row)
|
80
|
-
line = row.join(options[:delimiter]) + "\n"
|
81
|
-
end
|
82
|
-
connection.raw_connection.put_copy_data line
|
83
|
-
end
|
84
|
-
end
|
85
|
-
connection.raw_connection.put_copy_end
|
86
|
-
end
|
87
|
-
end
|
88
|
-
end
|