simple_csv 0.2.2 → 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.gitignore +3 -3
- data/.travis.yml +7 -3
- data/README.md +316 -2
- data/bin/console +22 -1
- data/lib/simple_csv.rb +9 -3
- data/lib/simple_csv/base.rb +22 -2
- data/lib/simple_csv/reader.rb +6 -24
- data/lib/simple_csv/settings.rb +1 -1
- data/lib/simple_csv/transformer.rb +68 -0
- data/lib/simple_csv/version.rb +1 -1
- data/lib/simple_csv/writer.rb +14 -12
- data/simple_csv.gemspec +4 -4
- metadata +18 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 3991cdfe21c73188eea30108998f63db2223aacfb3887936f300118f786a4488
|
4
|
+
data.tar.gz: 5842b9b5d17f74951041e125405e622c30e21787b3ce5326640b849ad3850a21
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dbd27b6e8ad4e4021a51866829c6ae285b6e7622fa44370d18fff205c8b49daa379ebf599f40347b7a05337785e17d0737cdcf0ea7a64fc8b7b9290a2caffd99
|
7
|
+
data.tar.gz: 0f8135d7a03e7bdb27218b6aa35b96d34288ab729c449ce0bcda51cd2c6700ae207c485095ecc8411fc3886f6e5c0d43937c9df75b6823453a1e5e06a8dd6ac9
|
data/.gitignore
CHANGED
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,3 +1,317 @@
|
|
1
|
-
#
|
1
|
+
# SimpleCsv
|
2
|
+
SimpleCsv is a simple gem that allows you to interact with CSV's in a more friendly way.
|
3
|
+
See the examples given below for more details :)
|
2
4
|
|
3
|
-
|
5
|
+
## Status
|
6
|
+
|
7
|
+
![Licence](https://img.shields.io/badge/license-MIT-E9573F.svg)
|
8
|
+
[![Gem Version](https://img.shields.io/gem/v/simple_csv.svg?colorB=E9573F&style=square)](rubygems.org/gems/simple_csv)
|
9
|
+
[![Issues](https://img.shields.io/github/issues/SidOfc/simple_csv.svg)](https://github.com/SidOfc/simple_csv/issues)
|
10
|
+
[![Build Status](https://img.shields.io/travis/SidOfc/simple_csv.svg)](https://travis-ci.org/SidOfc/simple_csv)
|
11
|
+
[![Coverage Status](https://img.shields.io/coveralls/SidOfc/simple_csv.svg)](https://coveralls.io/github/SidOfc/simple_csv?branch=master)
|
12
|
+
|
13
|
+
---
|
14
|
+
|
15
|
+
## Installation
|
16
|
+
|
17
|
+
Add this line to your application's Gemfile:
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
gem 'simple_csv'
|
21
|
+
```
|
22
|
+
|
23
|
+
And then execute:
|
24
|
+
|
25
|
+
$ bundle
|
26
|
+
|
27
|
+
Or install it yourself as:
|
28
|
+
|
29
|
+
$ gem install simple_csv
|
30
|
+
|
31
|
+
## Usage
|
32
|
+
|
33
|
+
### General
|
34
|
+
|
35
|
+
By default, the settings used will be those of `CSV::DEFAULT_OPTIONS` generally.
|
36
|
+
`SimpleCsv` sets the `:headers` property to true by default, this is due to the nature of `SimpleCsv`.
|
37
|
+
|
38
|
+
Headers have to be defined either before reading or generating a CSV.
|
39
|
+
Since `:headers` is now `true` by default, `SimpleCsv` will allow `CSV` to parse the first line as headers.
|
40
|
+
These headers are then converted in method calls to use within an `SimpleCsv::Reader#each_row` loop.
|
41
|
+
|
42
|
+
If however, your file lacks headers, you have the ability to set `:has_headers` to false and supply headers manually before calling `SimpleCsv::Reader#each_row`.
|
43
|
+
The headers will be picked up and used instead of the first line.
|
44
|
+
|
45
|
+
#### SimpleCsv default settings
|
46
|
+
|
47
|
+
These are the settings that will be merged with settings passed through either `SimpleCsv#generate` or `SimpleCsv#read`
|
48
|
+
|
49
|
+
| setting | value |
|
50
|
+
| ---------------------- | --------------------------------------- |
|
51
|
+
| `:col_sep` | `","` |
|
52
|
+
| `:row_sep` | `:auto` |
|
53
|
+
| `:quote_char` | `"\"` |
|
54
|
+
| `:field_size_limit` | `nil` |
|
55
|
+
| `:converters` | `[:all, :blank_to_nil, :null_to_nil]` |
|
56
|
+
| `:unconverted_fields` | `nil` |
|
57
|
+
| `:headers` | `true` |
|
58
|
+
| `:return_headers` | `false` |
|
59
|
+
| `:header_converters` | `nil` |
|
60
|
+
| `:skip_blanks` | `false` |
|
61
|
+
| `:force_quotes` | `true` |
|
62
|
+
| `:skip_lines` | `nil` |
|
63
|
+
|
64
|
+
The following settings differ from the `CSV::DEFAULT_OPTIONS`
|
65
|
+
|
66
|
+
* `:converters` is set to `[:all, :blank_to_nil, :null_to_nil]`
|
67
|
+
* `:headers` is `true` by default
|
68
|
+
* `:force_quotes` is `true` by default
|
69
|
+
|
70
|
+
This essentially means that when reading a CSV file, headers are required otherwise a `SimpleCsv::HeadersNotSet` exception will be thrown.
|
71
|
+
Also, when reading a CSV, all values will be parsed to their respective types, so `"1"` would become `1` as end value.
|
72
|
+
|
73
|
+
#### SimpleCsv::Writer additional default settings
|
74
|
+
|
75
|
+
Additionally, `SimpleCsv::Writer` has one additional default setting that ensures an entire row is written before being able to write another one.
|
76
|
+
This setting enforces you to call each method once before calling one of them again, if this condition is not met a `SimpleCsv::RowNotComplete` exception will be thrown
|
77
|
+
|
78
|
+
| setting | value |
|
79
|
+
|------------------------|---------------------------------------|
|
80
|
+
|`:force_row_completion` | `true` |
|
81
|
+
|
82
|
+
#### Setting aliasses
|
83
|
+
|
84
|
+
An _alias_ can be used instead of it's respective _setting_.
|
85
|
+
|
86
|
+
| setting | alias |
|
87
|
+
|---------------|---------------|
|
88
|
+
| `:col_sep` | `:seperator` |
|
89
|
+
| `headers` | `has_headers` |
|
90
|
+
|
91
|
+
#### Converters
|
92
|
+
|
93
|
+
The above `:converters` option is set to include `:all` converters and additionally, `:blank_to_nil` and `:null_to_nil`
|
94
|
+
The code for these two can be summed up in two lines:
|
95
|
+
|
96
|
+
```ruby
|
97
|
+
CSV::Converters[:blank_to_nil] = ->(f) { f && f.empty? ? nil : f }
|
98
|
+
CSV::Converters[:null_to_nil] = ->(f) { f && f == 'NULL' ? nil : f }
|
99
|
+
```
|
100
|
+
|
101
|
+
What they do replace empty values or the string `'NULL'` by nil within a column when it's being parsed.
|
102
|
+
For now, these are the default two and they are always used unless the `:converters` option is set to `nil` within `SimpleCsv#generate` or `SimpleCsv#read`
|
103
|
+
|
104
|
+
### Generating a CSV file
|
105
|
+
|
106
|
+
```ruby
|
107
|
+
SimpleCsv.generate path, options = { ... }, &block
|
108
|
+
```
|
109
|
+
|
110
|
+
The `SimpleCsv#generate` method takes a (required) path, an (optional) hash of options and a (required) block to start building a CSV file.
|
111
|
+
To generate a CSV file we use `SimpleCsv#generate` (using the [faker](https://github.com/stympy/faker) gem to provide fake data)
|
112
|
+
|
113
|
+
While writing a row to a CSV, the value of a set property can be accessed by calling that property method again without arguments (See the "inspect a value" comment in the following example).
|
114
|
+
|
115
|
+
```ruby
|
116
|
+
require 'faker'
|
117
|
+
|
118
|
+
# use SimpleCsv.generate('output.csv', seperator: '|') to generate a CSV with a pipe character as seperator
|
119
|
+
SimpleCsv.generate('output.csv') do
|
120
|
+
# first define the headers
|
121
|
+
headers :first_name, :last_name, :birth_date, :employed_at
|
122
|
+
|
123
|
+
# loop something
|
124
|
+
100.times do
|
125
|
+
# insert data in each field defined in headers to insert a row.
|
126
|
+
first_name Faker::Name.first_name
|
127
|
+
# inspect a value
|
128
|
+
p first_name
|
129
|
+
last_name Faker::Name.last_name
|
130
|
+
birth_date Faker::Date.between(Date.today << 900, Date.today << 200)
|
131
|
+
employed_at [Faker::Company.name, nil].sample
|
132
|
+
end
|
133
|
+
end
|
134
|
+
```
|
135
|
+
|
136
|
+
This method passes any unknown method to its caller (`main Object` if none).
|
137
|
+
If you need a reference to the instance of the current writer from within the block, it takes an optional argument:
|
138
|
+
|
139
|
+
```ruby
|
140
|
+
SimpleCsv.generate ... do |writer|
|
141
|
+
# writer is a reference to the self of this block.
|
142
|
+
# the following two are equivelant (assuming 'name' column exists in the CSV):
|
143
|
+
|
144
|
+
writer.name 'SidOfc'
|
145
|
+
name 'SidOfc'
|
146
|
+
end
|
147
|
+
```
|
148
|
+
|
149
|
+
### Reading a CSV file
|
150
|
+
|
151
|
+
```ruby
|
152
|
+
SimpleCsv.read path, options = { ... }, &block
|
153
|
+
```
|
154
|
+
|
155
|
+
The `SimpleCsv#generate` method takes a (required) path, an (optional) hash of options and a (required) block to start reading a CSV file.
|
156
|
+
|
157
|
+
To read a CSV file we use `SimpleCsv#read`, we will pass it a file path and a block as arguments.
|
158
|
+
Within the block we define the headers present in the file, these will be transformed into methods you can call within `SimpleCsv::Reader#each_row` to get that property's current value
|
159
|
+
|
160
|
+
```ruby
|
161
|
+
SimpleCsv.read('input.csv') do
|
162
|
+
# assumes headers are set, they will be read and callable within each_row
|
163
|
+
|
164
|
+
each_row do
|
165
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
166
|
+
end
|
167
|
+
end
|
168
|
+
```
|
169
|
+
|
170
|
+
This method passes any unknown method to its caller (`main Object` if none).
|
171
|
+
If you need a reference to the instance of the current reader from within the block, it takes an optional argument:
|
172
|
+
|
173
|
+
```ruby
|
174
|
+
SimpleCsv.read ... do |reader|
|
175
|
+
# reader is a reference to the self of this block.
|
176
|
+
# all the following are equivelant:
|
177
|
+
|
178
|
+
# the 'each_row' and `in_groups_of` methods also get a reference to self.
|
179
|
+
each_row do |reader_too|
|
180
|
+
puts reader_too.name
|
181
|
+
puts reader.name
|
182
|
+
puts name
|
183
|
+
end
|
184
|
+
|
185
|
+
in_groups_of 100 do |other_reader|
|
186
|
+
puts other_reader.name
|
187
|
+
puts reader.name
|
188
|
+
puts name
|
189
|
+
end
|
190
|
+
end
|
191
|
+
```
|
192
|
+
|
193
|
+
### Reading a CSV file without headers
|
194
|
+
|
195
|
+
If we have a CSV file that does not contain headers we can use the following setup.
|
196
|
+
Setting `:has_headers` to `false` means we do not expect the first line to be headers.
|
197
|
+
Therefore we have to explicitly define the headers before looping the CSV.
|
198
|
+
|
199
|
+
```ruby
|
200
|
+
SimpleCsv.read('headerless.csv', has_headers: false) do
|
201
|
+
# first define the headers in the file manually if the file does not have them
|
202
|
+
headers :first_name, :last_name, :birth_date, :employed_at
|
203
|
+
|
204
|
+
each_row do
|
205
|
+
# print each field defined in headers (that is not nil)
|
206
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
207
|
+
end
|
208
|
+
end
|
209
|
+
```
|
210
|
+
|
211
|
+
### Transforming s CSV file
|
212
|
+
|
213
|
+
When you want to alter or reduce the output of a given CSV file, `SimpleCsv#transform` can be used.
|
214
|
+
This allows you to apply call a block for each value in a specified column, you can also control the output headers to remove clutter from the input file.
|
215
|
+
|
216
|
+
A transformation is defined by calling the header you wish to modify with a block that performs the modification.
|
217
|
+
In below example, a CSV with columns `:name`, `:username`, `:age` and `:interests` is assumed. The `:age` of every row
|
218
|
+
will be incremented because `age` was defined with the block. **Only** `headers` _and_ `output_headers` are supported within the transform block.
|
219
|
+
|
220
|
+
```ruby
|
221
|
+
SimpleCsv.transform('people.csv', output: 'people2.csv') do
|
222
|
+
# define specific output headers, other columns will not be added to output csv file
|
223
|
+
output_headers :name, :username, :age, :interests
|
224
|
+
|
225
|
+
# everyone got one year older, increment all ages.
|
226
|
+
age { |n| n + 1 }
|
227
|
+
|
228
|
+
# replace all names with "#{name}_old".
|
229
|
+
name { |s| "#{name}_old" }
|
230
|
+
end
|
231
|
+
```
|
232
|
+
|
233
|
+
The above example will create a file called `people2.csv` that contains the result data. The original file is **not** destroyed.
|
234
|
+
There is one additional option for `SimpleCsv#transform` which is the `:output` option.
|
235
|
+
When this option not set, the returned file will have the same name as the input CSV followed by a timestamp
|
236
|
+
formatted in the following format: `[input_csv]-[%d-%m-%Y-%S&7N].csv` (`[input_csv]` will have `.csv` extension stripped and reapplied).
|
237
|
+
See Ruby's [`Time#strftime`](https://ruby-doc.org/core-2.5.0/Time.html) documentation for more information on formatting flags used.
|
238
|
+
|
239
|
+
If you need a reference to the instance of the current reader from within the block, it takes an optional argument:
|
240
|
+
|
241
|
+
```ruby
|
242
|
+
SimpleCsv.transform ... do |transformer|
|
243
|
+
# transformer is a reference to the self of this block.
|
244
|
+
# all the following are equivelant (assuming "age" property exists):
|
245
|
+
|
246
|
+
transformer.age { |n| n * 2 }
|
247
|
+
age { |n| n * 2 }
|
248
|
+
end
|
249
|
+
```
|
250
|
+
|
251
|
+
### Batch operations
|
252
|
+
|
253
|
+
If we have a large CSV we might want to batch operations (say, if we are inserting this data into a database or through an API).
|
254
|
+
For this we can use `SimpleCsv::Reader#in_groups_of` and pass the size of the group.
|
255
|
+
Within that we call `SimpleCsv::Reader#each_row` as usual
|
256
|
+
|
257
|
+
```ruby
|
258
|
+
SimpleCsv.read('input.csv') do
|
259
|
+
# assumes headers are set, they will be read and callable within each_row
|
260
|
+
|
261
|
+
in_groups_of(100) do
|
262
|
+
each_row do
|
263
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
264
|
+
end
|
265
|
+
# execute after every 100 rows
|
266
|
+
sleep 2
|
267
|
+
end
|
268
|
+
end
|
269
|
+
```
|
270
|
+
|
271
|
+
### Aliassing existing headers
|
272
|
+
|
273
|
+
Should you want to map existing headers to different names, this is possible by passing a hash at the end with key value pairs.
|
274
|
+
When generating a CSV file, aliasses are ignored and therefore should not be passed.
|
275
|
+
|
276
|
+
When defining columns manually using `headers` for a file without headers, ALL columns must be named before defining aliasses.
|
277
|
+
This means that if your CSV exists of 3 columns, 3 headers must be defined before aliassing any of those to something shorter or more concise.
|
278
|
+
|
279
|
+
To create an alias `date_of_birth` of `birth_date` *(In a CSV file without headers)* we would write *(notice `:birth_date` is present twice, once as column entry, and once more as key for an alias)*:
|
280
|
+
|
281
|
+
```ruby
|
282
|
+
headers :first_name, :last_name, :employed_at, :birth_date, birth_date: :date_of_birth
|
283
|
+
```
|
284
|
+
|
285
|
+
This allows you to use a method `#date_of_birth` inside any `#each_row` in addition to `#birth_date`:
|
286
|
+
|
287
|
+
```ruby
|
288
|
+
SimpleCsv.read ... do
|
289
|
+
headers :name, :age, :employed_at, employed_at: :job
|
290
|
+
|
291
|
+
each_row do
|
292
|
+
puts "#{name} is #{age} old and works at #{job}"
|
293
|
+
end
|
294
|
+
end
|
295
|
+
```
|
296
|
+
|
297
|
+
## Development
|
298
|
+
|
299
|
+
After checking out the repo, run `bundle` to install dependencies. Then, run `rspec` to run the tests. You can also use the `bin/console` file to play around and debug.
|
300
|
+
|
301
|
+
To install this gem onto your local machine, run `rake install`.
|
302
|
+
|
303
|
+
To release a new version:
|
304
|
+
|
305
|
+
* Run CI in dev branch, if tests pass, merge into master
|
306
|
+
* Update version number in _lib/simple_csv/version.rb_ according to [symver](http://semver.org/)
|
307
|
+
* Update _README.md_ to reflect your changes
|
308
|
+
* run `rake release` to push commits, create a tag for the current commit and push the `.gem` file to RubyGems
|
309
|
+
|
310
|
+
## Contributing
|
311
|
+
|
312
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/sidofc/simple_csv. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
313
|
+
|
314
|
+
|
315
|
+
## License
|
316
|
+
|
317
|
+
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
data/bin/console
CHANGED
@@ -5,4 +5,25 @@ require 'pry'
|
|
5
5
|
require 'faker'
|
6
6
|
require 'simple_csv'
|
7
7
|
|
8
|
-
|
8
|
+
# SimpleCsv.generate('sample.csv') do
|
9
|
+
# headers :name, :age
|
10
|
+
|
11
|
+
# 10.times do
|
12
|
+
# name Faker::Name.name
|
13
|
+
# age Faker::Number.between(20, 120)
|
14
|
+
# end
|
15
|
+
# end
|
16
|
+
|
17
|
+
SimpleCsv.transform('spec/files/result.csv', output: 'result.csv') do
|
18
|
+
output_headers 'user name'
|
19
|
+
|
20
|
+
user_name { |n| n * 2 }
|
21
|
+
end
|
22
|
+
|
23
|
+
SimpleCsv.read('result.csv') do
|
24
|
+
p headers
|
25
|
+
|
26
|
+
each_row do
|
27
|
+
p headers.map { |h| send(h) }
|
28
|
+
end
|
29
|
+
end
|
data/lib/simple_csv.rb
CHANGED
@@ -4,6 +4,7 @@ require 'simple_csv/version'
|
|
4
4
|
require 'simple_csv/settings'
|
5
5
|
require 'simple_csv/base'
|
6
6
|
require 'simple_csv/reader'
|
7
|
+
require 'simple_csv/transformer'
|
7
8
|
require 'simple_csv/writer'
|
8
9
|
|
9
10
|
module SimpleCsv
|
@@ -30,22 +31,27 @@ module SimpleCsv
|
|
30
31
|
end
|
31
32
|
|
32
33
|
def self.read(path, **options, &block)
|
33
|
-
initialize_converters unless converters_initialized
|
34
|
+
initialize_converters unless converters_initialized?
|
34
35
|
Reader.new path, options, &block
|
35
36
|
end
|
36
37
|
|
37
38
|
def self.generate(path, **options, &block)
|
38
|
-
initialize_converters unless converters_initialized
|
39
|
+
initialize_converters unless converters_initialized?
|
39
40
|
Writer.new path, options, &block
|
40
41
|
end
|
41
42
|
|
43
|
+
def self.transform(path, **options, &block)
|
44
|
+
initialize_converters unless converters_initialized?
|
45
|
+
Transformer.new path, options, &block
|
46
|
+
end
|
47
|
+
|
42
48
|
def self.initialize_converters
|
43
49
|
CSV::Converters[:blank_to_nil] = ->(f) { f && f.empty? ? nil : f }
|
44
50
|
CSV::Converters[:null_to_nil] = ->(f) { f && f == 'NULL' ? nil : f }
|
45
51
|
@converters_initialized = true
|
46
52
|
end
|
47
53
|
|
48
|
-
def self.converters_initialized
|
54
|
+
def self.converters_initialized?
|
49
55
|
@converters_initialized
|
50
56
|
end
|
51
57
|
end
|
data/lib/simple_csv/base.rb
CHANGED
@@ -23,12 +23,32 @@ module SimpleCsv
|
|
23
23
|
@col_map.merge! stringify_col_map(col_map) if col_map.any?
|
24
24
|
|
25
25
|
@headers_set ||= @headers.any?
|
26
|
+
@headers.uniq!
|
26
27
|
@headers
|
27
28
|
end
|
28
29
|
|
29
|
-
def
|
30
|
+
def find_headers
|
31
|
+
first_line.split(detect_delimiter).map { |h| h.gsub(/^"*|"*$/, '') }
|
32
|
+
end
|
33
|
+
|
34
|
+
def detect_delimiter
|
35
|
+
line = first_line
|
36
|
+
@delimiters = COMMON_DELIMITERS.map { |sep| [sep, line.scan(sep).length] }
|
37
|
+
.sort { |a, b| b[1] <=> a[1] }
|
38
|
+
@delimiter ||= @delimiters[0][0]
|
39
|
+
end
|
40
|
+
|
41
|
+
def first_line
|
42
|
+
@first_line ||= File.open @csv_path, &:readline
|
43
|
+
end
|
44
|
+
|
45
|
+
def headers?
|
46
|
+
@headers_set
|
47
|
+
end
|
48
|
+
|
49
|
+
def alias_to_friendly_headers(names = @headers)
|
30
50
|
@col_map ||= {}
|
31
|
-
aliasses =
|
51
|
+
aliasses = names.each_with_object({}) do |hdr, h|
|
32
52
|
n = hdr.to_s.strip.gsub(/([a-z])([A-Z])/, '\1_\2').downcase
|
33
53
|
.gsub(/[^\w]|\s/, '_')
|
34
54
|
h[n] = hdr unless @col_map.key? n
|
data/lib/simple_csv/reader.rb
CHANGED
@@ -4,19 +4,20 @@ module SimpleCsv
|
|
4
4
|
|
5
5
|
def initialize(path, **opts, &block)
|
6
6
|
@csv_path = File.expand_path path
|
7
|
+
@caller_self = eval 'self', block.binding
|
7
8
|
|
8
9
|
opts[:seperator] ||= detect_delimiter
|
9
10
|
settings.apply opts
|
10
11
|
|
11
12
|
load_csv_with_auto_headers if settings.for_csv[:headers]
|
12
13
|
|
13
|
-
|
14
|
+
instance_exec(self, &block)
|
14
15
|
end
|
15
16
|
|
16
17
|
def in_groups_of(size, &block)
|
17
18
|
@original.each_slice(size) do |group|
|
18
19
|
@csv = group
|
19
|
-
|
20
|
+
instance_exec(self, &block)
|
20
21
|
end
|
21
22
|
@index = nil
|
22
23
|
@csv = @original
|
@@ -29,7 +30,7 @@ module SimpleCsv
|
|
29
30
|
|
30
31
|
@csv.each do |record|
|
31
32
|
@record = record
|
32
|
-
|
33
|
+
instance_exec(self, &block)
|
33
34
|
@index += 1 if @index
|
34
35
|
end
|
35
36
|
end
|
@@ -43,7 +44,7 @@ module SimpleCsv
|
|
43
44
|
end
|
44
45
|
|
45
46
|
def load_csv_with_manual_headers
|
46
|
-
SimpleCsv.csv_manually_set_headers! unless
|
47
|
+
SimpleCsv.csv_manually_set_headers! unless headers?
|
47
48
|
csv_arr = CSV.open(@csv_path).to_a
|
48
49
|
|
49
50
|
if csv_arr.first.size == headers.size
|
@@ -55,30 +56,11 @@ module SimpleCsv
|
|
55
56
|
end
|
56
57
|
end
|
57
58
|
|
58
|
-
def find_headers
|
59
|
-
first_line.split(detect_delimiter).map { |h| h.gsub(/^"*|"*$/, '') }
|
60
|
-
end
|
61
|
-
|
62
|
-
def detect_delimiter
|
63
|
-
line = first_line
|
64
|
-
@delimiters = COMMON_DELIMITERS.map { |sep| [sep, line.scan(sep).length] }
|
65
|
-
.sort { |a, b| b[1] <=> a[1] }
|
66
|
-
@delimiter ||= @delimiters[0][0]
|
67
|
-
end
|
68
|
-
|
69
|
-
def first_line
|
70
|
-
@first_line ||= File.open @csv_path, &:readline
|
71
|
-
end
|
72
|
-
|
73
|
-
def respond_to_missing?(mtd, include_private)
|
74
|
-
headers.include?(m) || @col_map.key?(m)
|
75
|
-
end
|
76
|
-
|
77
59
|
def method_missing(mtd, *args, &block)
|
78
60
|
m = mtd.to_s
|
79
61
|
return @record[m] if headers.include?(m)
|
80
62
|
return @record[@col_map[m]] if @col_map.key?(m)
|
81
|
-
|
63
|
+
@caller_self.send mtd, *args, &block
|
82
64
|
end
|
83
65
|
end
|
84
66
|
end
|
data/lib/simple_csv/settings.rb
CHANGED
@@ -4,7 +4,7 @@ module SimpleCsv
|
|
4
4
|
force_quotes: true,
|
5
5
|
converters: [:all, :blank_to_nil, :null_to_nil] }.freeze
|
6
6
|
ALIASSED = { seperator: :col_sep, has_headers: :headers }.freeze
|
7
|
-
INVERTED_ALIASSES = ALIASSED.to_a.map(&:reverse).to_h
|
7
|
+
INVERTED_ALIASSES = ALIASSED.to_a.map(&:reverse).to_h.freeze
|
8
8
|
|
9
9
|
def initialize(**opts)
|
10
10
|
@settings = DEFAULTS.dup.merge opts
|
@@ -0,0 +1,68 @@
|
|
1
|
+
module SimpleCsv
|
2
|
+
class Transformer < Base
|
3
|
+
DEFAULT_FILENAME = 'converted.csv'
|
4
|
+
|
5
|
+
def initialize(path, **opts, &block)
|
6
|
+
@transforms = {}
|
7
|
+
@output_headers = []
|
8
|
+
|
9
|
+
@caller_self = eval 'self', block.binding
|
10
|
+
|
11
|
+
if settings.for_csv[:headers]
|
12
|
+
@csv_path = File.expand_path path
|
13
|
+
headers(*find_headers) unless headers.any?
|
14
|
+
end
|
15
|
+
|
16
|
+
instance_exec(self, &block)
|
17
|
+
|
18
|
+
apply_transforms path, **opts
|
19
|
+
end
|
20
|
+
|
21
|
+
def output_headers(*out_headers)
|
22
|
+
return @output_headers if @output_headers.any?
|
23
|
+
|
24
|
+
@output_headers = out_headers.map(&:to_s)
|
25
|
+
alias_to_friendly_headers @output_headers
|
26
|
+
@output_headers
|
27
|
+
end
|
28
|
+
|
29
|
+
private
|
30
|
+
|
31
|
+
def apply_transforms(path, **opts)
|
32
|
+
received_headers = headers
|
33
|
+
timestamp = Time.new.strftime '%d-%m-%Y-%S%7N'
|
34
|
+
output_path = opts.delete(:output) || "#{path.split('.')[0..-2].join}-#{timestamp}.csv"
|
35
|
+
output_headers = @output_headers.any? ? @output_headers : received_headers
|
36
|
+
|
37
|
+
SimpleCsv.read path, opts do |reader|
|
38
|
+
SimpleCsv.generate output_path, opts do |writer|
|
39
|
+
writer.headers *output_headers
|
40
|
+
|
41
|
+
reader.each_row do
|
42
|
+
output_headers.each do |column|
|
43
|
+
transform = find_transform column
|
44
|
+
result = transform ? transform.call(reader.send(column))
|
45
|
+
: reader.send(column)
|
46
|
+
|
47
|
+
writer.send column, result
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
def find_transform(column)
|
55
|
+
@transforms[(@col_map.key(column.to_s) || column).to_sym]
|
56
|
+
end
|
57
|
+
|
58
|
+
def method_missing(mtd, *args, &block)
|
59
|
+
mstr = mtd.to_s
|
60
|
+
|
61
|
+
if headers.include?(mstr) || @output_headers.include?(mstr) || @col_map.key?(mstr)
|
62
|
+
@transforms[mtd] = block || args.first unless @transforms.key? mtd
|
63
|
+
else
|
64
|
+
@caller_self.send mtd, *args, &block
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
data/lib/simple_csv/version.rb
CHANGED
data/lib/simple_csv/writer.rb
CHANGED
@@ -1,28 +1,36 @@
|
|
1
1
|
module SimpleCsv
|
2
2
|
class Writer < Base
|
3
3
|
def initialize(path, **opts, &block)
|
4
|
+
@caller_self = eval 'self', block.binding
|
5
|
+
|
4
6
|
settings.apply({force_row_completion: true}, opts)
|
5
7
|
CSV.open(File.expand_path(path), 'w', settings.for_csv) do |csv|
|
6
8
|
@csv = csv
|
7
9
|
@last_row = {}
|
8
10
|
@current_row = {}
|
9
|
-
|
11
|
+
instance_exec(self, &block)
|
10
12
|
end
|
11
13
|
end
|
12
14
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
15
|
+
def headers(*args)
|
16
|
+
super
|
17
|
+
(@csv << @headers) && @headers_written = true if !@headers_written && @csv
|
18
|
+
@headers
|
17
19
|
end
|
18
20
|
|
21
|
+
private
|
22
|
+
|
19
23
|
def method_missing(mtd, *args, &block)
|
20
24
|
SimpleCsv.csv_manually_set_headers! unless @headers_written
|
25
|
+
|
21
26
|
current_val = @current_row[mtd] if @current_row.key?(mtd)
|
22
27
|
current_val = @last_row[mtd] if @last_row.key?(mtd)
|
23
28
|
|
24
29
|
return current_val if args.empty? && current_val
|
25
|
-
|
30
|
+
|
31
|
+
unless headers.include?(mtd.to_s) || @col_map.key?(mtd.to_s)
|
32
|
+
return @caller_self.send mtd, *args, &block
|
33
|
+
end
|
26
34
|
|
27
35
|
if settings.force_row_completion && @current_row.key?(mtd) && args.any?
|
28
36
|
SimpleCsv.row_not_complete!(mtd, args.first)
|
@@ -37,11 +45,5 @@ module SimpleCsv
|
|
37
45
|
@current_row = {}
|
38
46
|
current_val
|
39
47
|
end
|
40
|
-
|
41
|
-
def headers(*args)
|
42
|
-
super
|
43
|
-
(@csv << @headers) && @headers_written = true if !@headers_written && @csv
|
44
|
-
@headers
|
45
|
-
end
|
46
48
|
end
|
47
49
|
end
|
data/simple_csv.gemspec
CHANGED
@@ -11,7 +11,7 @@ Gem::Specification.new do |spec|
|
|
11
11
|
|
12
12
|
spec.summary = 'CSV DSL'
|
13
13
|
spec.description = 'A simple DSL for reading and generating CSV files'
|
14
|
-
spec.homepage = 'https://
|
14
|
+
spec.homepage = 'https://github.com/SidOfc/simple_csv'
|
15
15
|
spec.license = 'MIT'
|
16
16
|
|
17
17
|
spec.files = `git ls-files -z`.split("\x0").reject do |f|
|
@@ -24,9 +24,9 @@ Gem::Specification.new do |spec|
|
|
24
24
|
|
25
25
|
spec.required_ruby_version = '>= 2.1'
|
26
26
|
|
27
|
-
spec.add_development_dependency 'bundler'
|
28
|
-
spec.add_development_dependency 'rake'
|
29
|
-
spec.add_development_dependency 'rspec'
|
27
|
+
spec.add_development_dependency 'bundler'
|
28
|
+
spec.add_development_dependency 'rake'
|
29
|
+
spec.add_development_dependency 'rspec'
|
30
30
|
spec.add_development_dependency 'pry'
|
31
31
|
spec.add_development_dependency 'pry-byebug'
|
32
32
|
spec.add_development_dependency 'faker'
|
metadata
CHANGED
@@ -1,57 +1,57 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: simple_csv
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sidney Liebrand
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2018-06-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "
|
17
|
+
- - ">="
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: '
|
19
|
+
version: '0'
|
20
20
|
type: :development
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - "
|
24
|
+
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: '
|
26
|
+
version: '0'
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: rake
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
|
-
- - "
|
31
|
+
- - ">="
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '
|
33
|
+
version: '0'
|
34
34
|
type: :development
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
|
-
- - "
|
38
|
+
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '
|
40
|
+
version: '0'
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: rspec
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
|
-
- - "
|
45
|
+
- - ">="
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '
|
47
|
+
version: '0'
|
48
48
|
type: :development
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
|
-
- - "
|
52
|
+
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '
|
54
|
+
version: '0'
|
55
55
|
- !ruby/object:Gem::Dependency
|
56
56
|
name: pry
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -130,10 +130,12 @@ files:
|
|
130
130
|
- lib/simple_csv/base.rb
|
131
131
|
- lib/simple_csv/reader.rb
|
132
132
|
- lib/simple_csv/settings.rb
|
133
|
+
- lib/simple_csv/transformer.rb
|
133
134
|
- lib/simple_csv/version.rb
|
134
135
|
- lib/simple_csv/writer.rb
|
135
136
|
- simple_csv.gemspec
|
136
|
-
|
137
|
+
- tmp/.gitkeep
|
138
|
+
homepage: https://github.com/SidOfc/simple_csv
|
137
139
|
licenses:
|
138
140
|
- MIT
|
139
141
|
metadata: {}
|
@@ -153,7 +155,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
153
155
|
version: '0'
|
154
156
|
requirements: []
|
155
157
|
rubyforge_project:
|
156
|
-
rubygems_version: 2.6
|
158
|
+
rubygems_version: 2.7.6
|
157
159
|
signing_key:
|
158
160
|
specification_version: 4
|
159
161
|
summary: CSV DSL
|