postgres-copy 0.5.7 → 0.5.8
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +1 -1
- data/Gemfile.lock +38 -38
- data/README.md +52 -22
- data/lib/postgres-copy/active_record.rb +1 -1
- data/postgres-copy.gemspec +1 -1
- metadata +3 -3
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,41 +1,41 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
postgres-copy (0.5.
|
4
|
+
postgres-copy (0.5.8)
|
5
5
|
activerecord (>= 3.0.0)
|
6
6
|
pg
|
7
7
|
rails (>= 3.0.0)
|
8
8
|
responders
|
9
9
|
|
10
10
|
GEM
|
11
|
-
remote:
|
11
|
+
remote: https://rubygems.org/
|
12
12
|
specs:
|
13
|
-
actionmailer (3.2.
|
14
|
-
actionpack (= 3.2.
|
15
|
-
mail (~> 2.
|
16
|
-
actionpack (3.2.
|
17
|
-
activemodel (= 3.2.
|
18
|
-
activesupport (= 3.2.
|
13
|
+
actionmailer (3.2.13)
|
14
|
+
actionpack (= 3.2.13)
|
15
|
+
mail (~> 2.5.3)
|
16
|
+
actionpack (3.2.13)
|
17
|
+
activemodel (= 3.2.13)
|
18
|
+
activesupport (= 3.2.13)
|
19
19
|
builder (~> 3.0.0)
|
20
20
|
erubis (~> 2.7.0)
|
21
21
|
journey (~> 1.0.4)
|
22
|
-
rack (~> 1.4.
|
22
|
+
rack (~> 1.4.5)
|
23
23
|
rack-cache (~> 1.2)
|
24
24
|
rack-test (~> 0.6.1)
|
25
25
|
sprockets (~> 2.2.1)
|
26
|
-
activemodel (3.2.
|
27
|
-
activesupport (= 3.2.
|
26
|
+
activemodel (3.2.13)
|
27
|
+
activesupport (= 3.2.13)
|
28
28
|
builder (~> 3.0.0)
|
29
|
-
activerecord (3.2.
|
30
|
-
activemodel (= 3.2.
|
31
|
-
activesupport (= 3.2.
|
29
|
+
activerecord (3.2.13)
|
30
|
+
activemodel (= 3.2.13)
|
31
|
+
activesupport (= 3.2.13)
|
32
32
|
arel (~> 3.0.2)
|
33
33
|
tzinfo (~> 0.3.29)
|
34
|
-
activeresource (3.2.
|
35
|
-
activemodel (= 3.2.
|
36
|
-
activesupport (= 3.2.
|
37
|
-
activesupport (3.2.
|
38
|
-
i18n (
|
34
|
+
activeresource (3.2.13)
|
35
|
+
activemodel (= 3.2.13)
|
36
|
+
activesupport (= 3.2.13)
|
37
|
+
activesupport (3.2.13)
|
38
|
+
i18n (= 0.6.1)
|
39
39
|
multi_json (~> 1.0)
|
40
40
|
arel (3.0.2)
|
41
41
|
builder (3.0.4)
|
@@ -45,37 +45,37 @@ GEM
|
|
45
45
|
i18n (0.6.1)
|
46
46
|
journey (1.0.4)
|
47
47
|
json (1.7.6)
|
48
|
-
mail (2.
|
48
|
+
mail (2.5.3)
|
49
49
|
i18n (>= 0.4.0)
|
50
50
|
mime-types (~> 1.16)
|
51
51
|
treetop (~> 1.4.8)
|
52
|
-
mime-types (1.
|
53
|
-
multi_json (1.
|
54
|
-
pg (0.
|
52
|
+
mime-types (1.22)
|
53
|
+
multi_json (1.7.2)
|
54
|
+
pg (0.15.0)
|
55
55
|
polyglot (0.3.3)
|
56
|
-
rack (1.4.
|
56
|
+
rack (1.4.5)
|
57
57
|
rack-cache (1.2)
|
58
58
|
rack (>= 0.4)
|
59
59
|
rack-ssl (1.3.3)
|
60
60
|
rack
|
61
61
|
rack-test (0.6.2)
|
62
62
|
rack (>= 1.0)
|
63
|
-
rails (3.2.
|
64
|
-
actionmailer (= 3.2.
|
65
|
-
actionpack (= 3.2.
|
66
|
-
activerecord (= 3.2.
|
67
|
-
activeresource (= 3.2.
|
68
|
-
activesupport (= 3.2.
|
63
|
+
rails (3.2.13)
|
64
|
+
actionmailer (= 3.2.13)
|
65
|
+
actionpack (= 3.2.13)
|
66
|
+
activerecord (= 3.2.13)
|
67
|
+
activeresource (= 3.2.13)
|
68
|
+
activesupport (= 3.2.13)
|
69
69
|
bundler (~> 1.0)
|
70
|
-
railties (= 3.2.
|
71
|
-
railties (3.2.
|
72
|
-
actionpack (= 3.2.
|
73
|
-
activesupport (= 3.2.
|
70
|
+
railties (= 3.2.13)
|
71
|
+
railties (3.2.13)
|
72
|
+
actionpack (= 3.2.13)
|
73
|
+
activesupport (= 3.2.13)
|
74
74
|
rack-ssl (~> 1.3.2)
|
75
75
|
rake (>= 0.8.7)
|
76
76
|
rdoc (~> 3.4)
|
77
77
|
thor (>= 0.14.6, < 2.0)
|
78
|
-
rake (10.0.
|
78
|
+
rake (10.0.4)
|
79
79
|
rdoc (3.12)
|
80
80
|
json (~> 1.4)
|
81
81
|
responders (0.9.3)
|
@@ -93,12 +93,12 @@ GEM
|
|
93
93
|
multi_json (~> 1.0)
|
94
94
|
rack (~> 1.0)
|
95
95
|
tilt (~> 1.1, != 1.3.0)
|
96
|
-
thor (0.
|
97
|
-
tilt (1.3.
|
96
|
+
thor (0.18.1)
|
97
|
+
tilt (1.3.6)
|
98
98
|
treetop (1.4.12)
|
99
99
|
polyglot
|
100
100
|
polyglot (>= 0.3.1)
|
101
|
-
tzinfo (0.3.
|
101
|
+
tzinfo (0.3.37)
|
102
102
|
|
103
103
|
PLATFORMS
|
104
104
|
ruby
|
data/README.md
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# postgres-copy
|
1
|
+
# postgres-copy [![Build Status](https://travis-ci.org/diogob/postgres-copy.png?branch=master)](https://travis-ci.org/diogob/postgres-copy)
|
2
2
|
|
3
3
|
This Gem will enable your AR models to use the PostgreSQL COPY command to import/export data in CSV format.
|
4
4
|
If you need to tranfer data between a PostgreSQL database and CSV files, the PostgreSQL native CSV parser
|
@@ -32,47 +32,65 @@ You can go to the rails console and try some cool things first.
|
|
32
32
|
The first and most basic use case, let's copy the enteire content of a database table to a CSV file on the database server disk.
|
33
33
|
Assuming we have a users table and a User AR model:
|
34
34
|
|
35
|
-
|
35
|
+
```ruby
|
36
|
+
User.pg_copy_to '/tmp/users.csv'
|
37
|
+
```
|
36
38
|
|
37
39
|
This will execute in the database the command:
|
38
40
|
|
39
|
-
|
41
|
+
```sql
|
42
|
+
COPY (SELECT "users".* FROM "users" ) TO '/tmp/users.csv' WITH DELIMITER ',' CSV HEADER
|
43
|
+
```
|
40
44
|
|
41
45
|
Remark that the file will be created in the database server disk.
|
42
46
|
But what if you want to write the lines in a file on the server that is running Rails, instead of the database?
|
43
47
|
In this case you can pass a block and retrieve the generated lines and then write them to a file:
|
44
48
|
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
49
|
+
```ruby
|
50
|
+
File.open('/tmp/users.csv', 'w') do |f|
|
51
|
+
User.pg_copy_to do |line|
|
52
|
+
f.write line
|
53
|
+
end
|
54
|
+
end
|
55
|
+
```
|
50
56
|
|
51
57
|
Or, if you have enough memory, you can read all table contents to a string using .pg_copy_to_string
|
52
58
|
|
53
|
-
|
59
|
+
```ruby
|
60
|
+
puts User.pg_copy_to_string
|
61
|
+
```
|
54
62
|
|
55
63
|
Another insteresting feature of pg_copy_to is that it uses the scoped relation, it means that you can use ARel
|
56
64
|
operations to generate different CSV files according to your needs.
|
57
65
|
Assuming we want to generate a file only with the names of users 1, 2 and 3:
|
58
66
|
|
59
|
-
|
67
|
+
```ruby
|
68
|
+
User.select("name").where(:id => [1,2,3]).pg_copy_to "/tmp/users.csv"
|
69
|
+
```
|
60
70
|
|
61
71
|
Which will generate the following SQL command:
|
62
72
|
|
63
|
-
|
73
|
+
```sql
|
74
|
+
COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.csv' WITH DELIMITER ',' CSV HEADER
|
75
|
+
```
|
64
76
|
|
65
77
|
The COPY command also supports exporting the data in binary format.
|
66
78
|
|
67
|
-
|
79
|
+
```ruby
|
80
|
+
User.select("name").where(:id => [1,2,3]).pg_copy_to "/tmp/users.dat", :format => :binary
|
81
|
+
```
|
68
82
|
|
69
83
|
Which will generate the following SQL command:
|
70
84
|
|
71
|
-
|
85
|
+
```sql
|
86
|
+
COPY (SELECT name FROM "users" WHERE "users"."id" IN (1, 2, 3)) TO '/tmp/users.dat' WITH BINARY
|
87
|
+
```
|
72
88
|
|
73
89
|
The copy_to_string method also supports this
|
74
90
|
|
75
|
-
|
91
|
+
```ruby
|
92
|
+
puts User.pg_copy_to_string(:format => :binary)
|
93
|
+
```
|
76
94
|
|
77
95
|
|
78
96
|
|
@@ -83,19 +101,25 @@ It will allow you to copy data from an arbritary IO object or from a file in the
|
|
83
101
|
Let's first copy from a file in the database server, assuming again that we have a users table and
|
84
102
|
that we are in the Rails console:
|
85
103
|
|
86
|
-
|
104
|
+
```ruby
|
105
|
+
User.pg_copy_from "/tmp/users.csv"
|
106
|
+
```
|
87
107
|
|
88
108
|
This command will use the headers in the CSV file as fields of the target table, so beware to always have a header in the files you want to import.
|
89
109
|
If the column names in the CSV header do not match the field names of the target table, you can pass a map in the options parameter.
|
90
110
|
|
91
|
-
|
111
|
+
```ruby
|
112
|
+
User.pg_copy_from "/tmp/users.csv", :map => {'name' => 'first_name'}
|
113
|
+
```
|
92
114
|
|
93
115
|
In the above example the header name in the CSV file will be mapped to the field called first_name in the users table.
|
94
116
|
You can also manipulate and modify the values of the file being imported before they enter into the database using a block:
|
95
117
|
|
96
|
-
|
97
|
-
|
98
|
-
|
118
|
+
```ruby
|
119
|
+
User.pg_copy_from "/tmp/users.csv" do |row|
|
120
|
+
row[0] = "fixed string"
|
121
|
+
end
|
122
|
+
```
|
99
123
|
|
100
124
|
The above extample will always change the value of the first column to "fixed string" before storing it into the database.
|
101
125
|
For each iteration of the block row receives an array with the same order as the columns in the CSV file.
|
@@ -103,17 +127,23 @@ For each iteration of the block row receives an array with the same order as the
|
|
103
127
|
|
104
128
|
To copy a binary formatted data file or IO object you can specify the format as binary
|
105
129
|
|
106
|
-
|
130
|
+
```ruby
|
131
|
+
User.pg_copy_from "/tmp/users.dat", :format => :binary
|
132
|
+
```
|
107
133
|
|
108
134
|
NOTE: Columns must line up with the table unless you specify how they map to table columns.
|
109
135
|
|
110
136
|
To specify how the columns will map to the table you can specify the :columns option
|
111
137
|
|
112
|
-
|
138
|
+
```ruby
|
139
|
+
User.pg_copy_from "/tmp/users.dat", :format => :binary, :columns => [:id, :name]
|
140
|
+
```
|
113
141
|
|
114
142
|
Which will generate the following SQL command:
|
115
143
|
|
116
|
-
|
144
|
+
```sql
|
145
|
+
COPY users (id, name) FROM '/tmp/users.dat' WITH BINARY
|
146
|
+
```
|
117
147
|
|
118
148
|
|
119
149
|
### Using the CSV Responder
|
@@ -24,7 +24,7 @@ module ActiveRecord
|
|
24
24
|
# Copy all data to a single string
|
25
25
|
def self.pg_copy_to_string options = {}
|
26
26
|
data = ''
|
27
|
-
self.pg_copy_to(nil, options){|l| data
|
27
|
+
self.pg_copy_to(nil, options){|l| data << l }
|
28
28
|
if options[:format] == :binary
|
29
29
|
data.force_encoding("ASCII-8BIT")
|
30
30
|
end
|
data/postgres-copy.gemspec
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: postgres-copy
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.5.
|
4
|
+
version: 0.5.8
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -184,10 +184,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
184
184
|
version: '0'
|
185
185
|
segments:
|
186
186
|
- 0
|
187
|
-
hash:
|
187
|
+
hash: 489106335970440028
|
188
188
|
requirements: []
|
189
189
|
rubyforge_project:
|
190
|
-
rubygems_version: 1.8.
|
190
|
+
rubygems_version: 1.8.25
|
191
191
|
signing_key:
|
192
192
|
specification_version: 3
|
193
193
|
summary: Put COPY command functionality in ActiveRecord's model class
|