simple_csv 0.2.2 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.gitignore +3 -3
- data/.travis.yml +7 -3
- data/README.md +316 -2
- data/bin/console +22 -1
- data/lib/simple_csv.rb +9 -3
- data/lib/simple_csv/base.rb +22 -2
- data/lib/simple_csv/reader.rb +6 -24
- data/lib/simple_csv/settings.rb +1 -1
- data/lib/simple_csv/transformer.rb +68 -0
- data/lib/simple_csv/version.rb +1 -1
- data/lib/simple_csv/writer.rb +14 -12
- data/simple_csv.gemspec +4 -4
- metadata +18 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 3991cdfe21c73188eea30108998f63db2223aacfb3887936f300118f786a4488
|
4
|
+
data.tar.gz: 5842b9b5d17f74951041e125405e622c30e21787b3ce5326640b849ad3850a21
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dbd27b6e8ad4e4021a51866829c6ae285b6e7622fa44370d18fff205c8b49daa379ebf599f40347b7a05337785e17d0737cdcf0ea7a64fc8b7b9290a2caffd99
|
7
|
+
data.tar.gz: 0f8135d7a03e7bdb27218b6aa35b96d34288ab729c449ce0bcda51cd2c6700ae207c485095ecc8411fc3886f6e5c0d43937c9df75b6823453a1e5e06a8dd6ac9
|
data/.gitignore
CHANGED
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,3 +1,317 @@
|
|
1
|
-
#
|
1
|
+
# SimpleCsv
|
2
|
+
SimpleCsv is a simple gem that allows you to interact with CSV's in a more friendly way.
|
3
|
+
See the examples given below for more details :)
|
2
4
|
|
3
|
-
|
5
|
+
## Status
|
6
|
+
|
7
|
+

|
8
|
+
[](rubygems.org/gems/simple_csv)
|
9
|
+
[](https://github.com/SidOfc/simple_csv/issues)
|
10
|
+
[](https://travis-ci.org/SidOfc/simple_csv)
|
11
|
+
[](https://coveralls.io/github/SidOfc/simple_csv?branch=master)
|
12
|
+
|
13
|
+
---
|
14
|
+
|
15
|
+
## Installation
|
16
|
+
|
17
|
+
Add this line to your application's Gemfile:
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
gem 'simple_csv'
|
21
|
+
```
|
22
|
+
|
23
|
+
And then execute:
|
24
|
+
|
25
|
+
$ bundle
|
26
|
+
|
27
|
+
Or install it yourself as:
|
28
|
+
|
29
|
+
$ gem install simple_csv
|
30
|
+
|
31
|
+
## Usage
|
32
|
+
|
33
|
+
### General
|
34
|
+
|
35
|
+
By default, the settings used will be those of `CSV::DEFAULT_OPTIONS` generally.
|
36
|
+
`SimpleCsv` sets the `:headers` property to true by default, this is due to the nature of `SimpleCsv`.
|
37
|
+
|
38
|
+
Headers have to be defined either before reading or generating a CSV.
|
39
|
+
Since `:headers` is now `true` by default, `SimpleCsv` will allow `CSV` to parse the first line as headers.
|
40
|
+
These headers are then converted in method calls to use within an `SimpleCsv::Reader#each_row` loop.
|
41
|
+
|
42
|
+
If however, your file lacks headers, you have the ability to set `:has_headers` to false and supply headers manually before calling `SimpleCsv::Reader#each_row`.
|
43
|
+
The headers will be picked up and used instead of the first line.
|
44
|
+
|
45
|
+
#### SimpleCsv default settings
|
46
|
+
|
47
|
+
These are the settings that will be merged with settings passed through either `SimpleCsv#generate` or `SimpleCsv#read`
|
48
|
+
|
49
|
+
| setting | value |
|
50
|
+
| ---------------------- | --------------------------------------- |
|
51
|
+
| `:col_sep` | `","` |
|
52
|
+
| `:row_sep` | `:auto` |
|
53
|
+
| `:quote_char` | `"\"` |
|
54
|
+
| `:field_size_limit` | `nil` |
|
55
|
+
| `:converters` | `[:all, :blank_to_nil, :null_to_nil]` |
|
56
|
+
| `:unconverted_fields` | `nil` |
|
57
|
+
| `:headers` | `true` |
|
58
|
+
| `:return_headers` | `false` |
|
59
|
+
| `:header_converters` | `nil` |
|
60
|
+
| `:skip_blanks` | `false` |
|
61
|
+
| `:force_quotes` | `true` |
|
62
|
+
| `:skip_lines` | `nil` |
|
63
|
+
|
64
|
+
The following settings differ from the `CSV::DEFAULT_OPTIONS`
|
65
|
+
|
66
|
+
* `:converters` is set to `[:all, :blank_to_nil, :null_to_nil]`
|
67
|
+
* `:headers` is `true` by default
|
68
|
+
* `:force_quotes` is `true` by default
|
69
|
+
|
70
|
+
This essentially means that when reading a CSV file, headers are required otherwise a `SimpleCsv::HeadersNotSet` exception will be thrown.
|
71
|
+
Also, when reading a CSV, all values will be parsed to their respective types, so `"1"` would become `1` as end value.
|
72
|
+
|
73
|
+
#### SimpleCsv::Writer additional default settings
|
74
|
+
|
75
|
+
Additionally, `SimpleCsv::Writer` has one additional default setting that ensures an entire row is written before being able to write another one.
|
76
|
+
This setting enforces you to call each method once before calling one of them again, if this condition is not met a `SimpleCsv::RowNotComplete` exception will be thrown
|
77
|
+
|
78
|
+
| setting | value |
|
79
|
+
|------------------------|---------------------------------------|
|
80
|
+
|`:force_row_completion` | `true` |
|
81
|
+
|
82
|
+
#### Setting aliasses
|
83
|
+
|
84
|
+
An _alias_ can be used instead of it's respective _setting_.
|
85
|
+
|
86
|
+
| setting | alias |
|
87
|
+
|---------------|---------------|
|
88
|
+
| `:col_sep` | `:seperator` |
|
89
|
+
| `headers` | `has_headers` |
|
90
|
+
|
91
|
+
#### Converters
|
92
|
+
|
93
|
+
The above `:converters` option is set to include `:all` converters and additionally, `:blank_to_nil` and `:null_to_nil`
|
94
|
+
The code for these two can be summed up in two lines:
|
95
|
+
|
96
|
+
```ruby
|
97
|
+
CSV::Converters[:blank_to_nil] = ->(f) { f && f.empty? ? nil : f }
|
98
|
+
CSV::Converters[:null_to_nil] = ->(f) { f && f == 'NULL' ? nil : f }
|
99
|
+
```
|
100
|
+
|
101
|
+
What they do replace empty values or the string `'NULL'` by nil within a column when it's being parsed.
|
102
|
+
For now, these are the default two and they are always used unless the `:converters` option is set to `nil` within `SimpleCsv#generate` or `SimpleCsv#read`
|
103
|
+
|
104
|
+
### Generating a CSV file
|
105
|
+
|
106
|
+
```ruby
|
107
|
+
SimpleCsv.generate path, options = { ... }, &block
|
108
|
+
```
|
109
|
+
|
110
|
+
The `SimpleCsv#generate` method takes a (required) path, an (optional) hash of options and a (required) block to start building a CSV file.
|
111
|
+
To generate a CSV file we use `SimpleCsv#generate` (using the [faker](https://github.com/stympy/faker) gem to provide fake data)
|
112
|
+
|
113
|
+
While writing a row to a CSV, the value of a set property can be accessed by calling that property method again without arguments (See the "inspect a value" comment in the following example).
|
114
|
+
|
115
|
+
```ruby
|
116
|
+
require 'faker'
|
117
|
+
|
118
|
+
# use SimpleCsv.generate('output.csv', seperator: '|') to generate a CSV with a pipe character as seperator
|
119
|
+
SimpleCsv.generate('output.csv') do
|
120
|
+
# first define the headers
|
121
|
+
headers :first_name, :last_name, :birth_date, :employed_at
|
122
|
+
|
123
|
+
# loop something
|
124
|
+
100.times do
|
125
|
+
# insert data in each field defined in headers to insert a row.
|
126
|
+
first_name Faker::Name.first_name
|
127
|
+
# inspect a value
|
128
|
+
p first_name
|
129
|
+
last_name Faker::Name.last_name
|
130
|
+
birth_date Faker::Date.between(Date.today << 900, Date.today << 200)
|
131
|
+
employed_at [Faker::Company.name, nil].sample
|
132
|
+
end
|
133
|
+
end
|
134
|
+
```
|
135
|
+
|
136
|
+
This method passes any unknown method to its caller (`main Object` if none).
|
137
|
+
If you need a reference to the instance of the current writer from within the block, it takes an optional argument:
|
138
|
+
|
139
|
+
```ruby
|
140
|
+
SimpleCsv.generate ... do |writer|
|
141
|
+
# writer is a reference to the self of this block.
|
142
|
+
# the following two are equivelant (assuming 'name' column exists in the CSV):
|
143
|
+
|
144
|
+
writer.name 'SidOfc'
|
145
|
+
name 'SidOfc'
|
146
|
+
end
|
147
|
+
```
|
148
|
+
|
149
|
+
### Reading a CSV file
|
150
|
+
|
151
|
+
```ruby
|
152
|
+
SimpleCsv.read path, options = { ... }, &block
|
153
|
+
```
|
154
|
+
|
155
|
+
The `SimpleCsv#generate` method takes a (required) path, an (optional) hash of options and a (required) block to start reading a CSV file.
|
156
|
+
|
157
|
+
To read a CSV file we use `SimpleCsv#read`, we will pass it a file path and a block as arguments.
|
158
|
+
Within the block we define the headers present in the file, these will be transformed into methods you can call within `SimpleCsv::Reader#each_row` to get that property's current value
|
159
|
+
|
160
|
+
```ruby
|
161
|
+
SimpleCsv.read('input.csv') do
|
162
|
+
# assumes headers are set, they will be read and callable within each_row
|
163
|
+
|
164
|
+
each_row do
|
165
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
166
|
+
end
|
167
|
+
end
|
168
|
+
```
|
169
|
+
|
170
|
+
This method passes any unknown method to its caller (`main Object` if none).
|
171
|
+
If you need a reference to the instance of the current reader from within the block, it takes an optional argument:
|
172
|
+
|
173
|
+
```ruby
|
174
|
+
SimpleCsv.read ... do |reader|
|
175
|
+
# reader is a reference to the self of this block.
|
176
|
+
# all the following are equivelant:
|
177
|
+
|
178
|
+
# the 'each_row' and `in_groups_of` methods also get a reference to self.
|
179
|
+
each_row do |reader_too|
|
180
|
+
puts reader_too.name
|
181
|
+
puts reader.name
|
182
|
+
puts name
|
183
|
+
end
|
184
|
+
|
185
|
+
in_groups_of 100 do |other_reader|
|
186
|
+
puts other_reader.name
|
187
|
+
puts reader.name
|
188
|
+
puts name
|
189
|
+
end
|
190
|
+
end
|
191
|
+
```
|
192
|
+
|
193
|
+
### Reading a CSV file without headers
|
194
|
+
|
195
|
+
If we have a CSV file that does not contain headers we can use the following setup.
|
196
|
+
Setting `:has_headers` to `false` means we do not expect the first line to be headers.
|
197
|
+
Therefore we have to explicitly define the headers before looping the CSV.
|
198
|
+
|
199
|
+
```ruby
|
200
|
+
SimpleCsv.read('headerless.csv', has_headers: false) do
|
201
|
+
# first define the headers in the file manually if the file does not have them
|
202
|
+
headers :first_name, :last_name, :birth_date, :employed_at
|
203
|
+
|
204
|
+
each_row do
|
205
|
+
# print each field defined in headers (that is not nil)
|
206
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
207
|
+
end
|
208
|
+
end
|
209
|
+
```
|
210
|
+
|
211
|
+
### Transforming s CSV file
|
212
|
+
|
213
|
+
When you want to alter or reduce the output of a given CSV file, `SimpleCsv#transform` can be used.
|
214
|
+
This allows you to apply call a block for each value in a specified column, you can also control the output headers to remove clutter from the input file.
|
215
|
+
|
216
|
+
A transformation is defined by calling the header you wish to modify with a block that performs the modification.
|
217
|
+
In below example, a CSV with columns `:name`, `:username`, `:age` and `:interests` is assumed. The `:age` of every row
|
218
|
+
will be incremented because `age` was defined with the block. **Only** `headers` _and_ `output_headers` are supported within the transform block.
|
219
|
+
|
220
|
+
```ruby
|
221
|
+
SimpleCsv.transform('people.csv', output: 'people2.csv') do
|
222
|
+
# define specific output headers, other columns will not be added to output csv file
|
223
|
+
output_headers :name, :username, :age, :interests
|
224
|
+
|
225
|
+
# everyone got one year older, increment all ages.
|
226
|
+
age { |n| n + 1 }
|
227
|
+
|
228
|
+
# replace all names with "#{name}_old".
|
229
|
+
name { |s| "#{name}_old" }
|
230
|
+
end
|
231
|
+
```
|
232
|
+
|
233
|
+
The above example will create a file called `people2.csv` that contains the result data. The original file is **not** destroyed.
|
234
|
+
There is one additional option for `SimpleCsv#transform` which is the `:output` option.
|
235
|
+
When this option not set, the returned file will have the same name as the input CSV followed by a timestamp
|
236
|
+
formatted in the following format: `[input_csv]-[%d-%m-%Y-%S&7N].csv` (`[input_csv]` will have `.csv` extension stripped and reapplied).
|
237
|
+
See Ruby's [`Time#strftime`](https://ruby-doc.org/core-2.5.0/Time.html) documentation for more information on formatting flags used.
|
238
|
+
|
239
|
+
If you need a reference to the instance of the current reader from within the block, it takes an optional argument:
|
240
|
+
|
241
|
+
```ruby
|
242
|
+
SimpleCsv.transform ... do |transformer|
|
243
|
+
# transformer is a reference to the self of this block.
|
244
|
+
# all the following are equivelant (assuming "age" property exists):
|
245
|
+
|
246
|
+
transformer.age { |n| n * 2 }
|
247
|
+
age { |n| n * 2 }
|
248
|
+
end
|
249
|
+
```
|
250
|
+
|
251
|
+
### Batch operations
|
252
|
+
|
253
|
+
If we have a large CSV we might want to batch operations (say, if we are inserting this data into a database or through an API).
|
254
|
+
For this we can use `SimpleCsv::Reader#in_groups_of` and pass the size of the group.
|
255
|
+
Within that we call `SimpleCsv::Reader#each_row` as usual
|
256
|
+
|
257
|
+
```ruby
|
258
|
+
SimpleCsv.read('input.csv') do
|
259
|
+
# assumes headers are set, they will be read and callable within each_row
|
260
|
+
|
261
|
+
in_groups_of(100) do
|
262
|
+
each_row do
|
263
|
+
puts [first_name, last_name, birth_date, employed_at].compact.join ', '
|
264
|
+
end
|
265
|
+
# execute after every 100 rows
|
266
|
+
sleep 2
|
267
|
+
end
|
268
|
+
end
|
269
|
+
```
|
270
|
+
|
271
|
+
### Aliassing existing headers
|
272
|
+
|
273
|
+
Should you want to map existing headers to different names, this is possible by passing a hash at the end with key value pairs.
|
274
|
+
When generating a CSV file, aliasses are ignored and therefore should not be passed.
|
275
|
+
|
276
|
+
When defining columns manually using `headers` for a file without headers, ALL columns must be named before defining aliasses.
|
277
|
+
This means that if your CSV exists of 3 columns, 3 headers must be defined before aliassing any of those to something shorter or more concise.
|
278
|
+
|
279
|
+
To create an alias `date_of_birth` of `birth_date` *(In a CSV file without headers)* we would write *(notice `:birth_date` is present twice, once as column entry, and once more as key for an alias)*:
|
280
|
+
|
281
|
+
```ruby
|
282
|
+
headers :first_name, :last_name, :employed_at, :birth_date, birth_date: :date_of_birth
|
283
|
+
```
|
284
|
+
|
285
|
+
This allows you to use a method `#date_of_birth` inside any `#each_row` in addition to `#birth_date`:
|
286
|
+
|
287
|
+
```ruby
|
288
|
+
SimpleCsv.read ... do
|
289
|
+
headers :name, :age, :employed_at, employed_at: :job
|
290
|
+
|
291
|
+
each_row do
|
292
|
+
puts "#{name} is #{age} old and works at #{job}"
|
293
|
+
end
|
294
|
+
end
|
295
|
+
```
|
296
|
+
|
297
|
+
## Development
|
298
|
+
|
299
|
+
After checking out the repo, run `bundle` to install dependencies. Then, run `rspec` to run the tests. You can also use the `bin/console` file to play around and debug.
|
300
|
+
|
301
|
+
To install this gem onto your local machine, run `rake install`.
|
302
|
+
|
303
|
+
To release a new version:
|
304
|
+
|
305
|
+
* Run CI in dev branch, if tests pass, merge into master
|
306
|
+
* Update version number in _lib/simple_csv/version.rb_ according to [symver](http://semver.org/)
|
307
|
+
* Update _README.md_ to reflect your changes
|
308
|
+
* run `rake release` to push commits, create a tag for the current commit and push the `.gem` file to RubyGems
|
309
|
+
|
310
|
+
## Contributing
|
311
|
+
|
312
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/sidofc/simple_csv. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
|
313
|
+
|
314
|
+
|
315
|
+
## License
|
316
|
+
|
317
|
+
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
|
data/bin/console
CHANGED
@@ -5,4 +5,25 @@ require 'pry'
|
|
5
5
|
require 'faker'
|
6
6
|
require 'simple_csv'
|
7
7
|
|
8
|
-
|
8
|
+
# SimpleCsv.generate('sample.csv') do
|
9
|
+
# headers :name, :age
|
10
|
+
|
11
|
+
# 10.times do
|
12
|
+
# name Faker::Name.name
|
13
|
+
# age Faker::Number.between(20, 120)
|
14
|
+
# end
|
15
|
+
# end
|
16
|
+
|
17
|
+
SimpleCsv.transform('spec/files/result.csv', output: 'result.csv') do
|
18
|
+
output_headers 'user name'
|
19
|
+
|
20
|
+
user_name { |n| n * 2 }
|
21
|
+
end
|
22
|
+
|
23
|
+
SimpleCsv.read('result.csv') do
|
24
|
+
p headers
|
25
|
+
|
26
|
+
each_row do
|
27
|
+
p headers.map { |h| send(h) }
|
28
|
+
end
|
29
|
+
end
|
data/lib/simple_csv.rb
CHANGED
@@ -4,6 +4,7 @@ require 'simple_csv/version'
|
|
4
4
|
require 'simple_csv/settings'
|
5
5
|
require 'simple_csv/base'
|
6
6
|
require 'simple_csv/reader'
|
7
|
+
require 'simple_csv/transformer'
|
7
8
|
require 'simple_csv/writer'
|
8
9
|
|
9
10
|
module SimpleCsv
|
@@ -30,22 +31,27 @@ module SimpleCsv
|
|
30
31
|
end
|
31
32
|
|
32
33
|
def self.read(path, **options, &block)
|
33
|
-
initialize_converters unless converters_initialized
|
34
|
+
initialize_converters unless converters_initialized?
|
34
35
|
Reader.new path, options, &block
|
35
36
|
end
|
36
37
|
|
37
38
|
def self.generate(path, **options, &block)
|
38
|
-
initialize_converters unless converters_initialized
|
39
|
+
initialize_converters unless converters_initialized?
|
39
40
|
Writer.new path, options, &block
|
40
41
|
end
|
41
42
|
|
43
|
+
def self.transform(path, **options, &block)
|
44
|
+
initialize_converters unless converters_initialized?
|
45
|
+
Transformer.new path, options, &block
|
46
|
+
end
|
47
|
+
|
42
48
|
def self.initialize_converters
|
43
49
|
CSV::Converters[:blank_to_nil] = ->(f) { f && f.empty? ? nil : f }
|
44
50
|
CSV::Converters[:null_to_nil] = ->(f) { f && f == 'NULL' ? nil : f }
|
45
51
|
@converters_initialized = true
|
46
52
|
end
|
47
53
|
|
48
|
-
def self.converters_initialized
|
54
|
+
def self.converters_initialized?
|
49
55
|
@converters_initialized
|
50
56
|
end
|
51
57
|
end
|
data/lib/simple_csv/base.rb
CHANGED
@@ -23,12 +23,32 @@ module SimpleCsv
|
|
23
23
|
@col_map.merge! stringify_col_map(col_map) if col_map.any?
|
24
24
|
|
25
25
|
@headers_set ||= @headers.any?
|
26
|
+
@headers.uniq!
|
26
27
|
@headers
|
27
28
|
end
|
28
29
|
|
29
|
-
def
|
30
|
+
def find_headers
|
31
|
+
first_line.split(detect_delimiter).map { |h| h.gsub(/^"*|"*$/, '') }
|
32
|
+
end
|
33
|
+
|
34
|
+
def detect_delimiter
|
35
|
+
line = first_line
|
36
|
+
@delimiters = COMMON_DELIMITERS.map { |sep| [sep, line.scan(sep).length] }
|
37
|
+
.sort { |a, b| b[1] <=> a[1] }
|
38
|
+
@delimiter ||= @delimiters[0][0]
|
39
|
+
end
|
40
|
+
|
41
|
+
def first_line
|
42
|
+
@first_line ||= File.open @csv_path, &:readline
|
43
|
+
end
|
44
|
+
|
45
|
+
def headers?
|
46
|
+
@headers_set
|
47
|
+
end
|
48
|
+
|
49
|
+
def alias_to_friendly_headers(names = @headers)
|
30
50
|
@col_map ||= {}
|
31
|
-
aliasses =
|
51
|
+
aliasses = names.each_with_object({}) do |hdr, h|
|
32
52
|
n = hdr.to_s.strip.gsub(/([a-z])([A-Z])/, '\1_\2').downcase
|
33
53
|
.gsub(/[^\w]|\s/, '_')
|
34
54
|
h[n] = hdr unless @col_map.key? n
|
data/lib/simple_csv/reader.rb
CHANGED
@@ -4,19 +4,20 @@ module SimpleCsv
|
|
4
4
|
|
5
5
|
def initialize(path, **opts, &block)
|
6
6
|
@csv_path = File.expand_path path
|
7
|
+
@caller_self = eval 'self', block.binding
|
7
8
|
|
8
9
|
opts[:seperator] ||= detect_delimiter
|
9
10
|
settings.apply opts
|
10
11
|
|
11
12
|
load_csv_with_auto_headers if settings.for_csv[:headers]
|
12
13
|
|
13
|
-
|
14
|
+
instance_exec(self, &block)
|
14
15
|
end
|
15
16
|
|
16
17
|
def in_groups_of(size, &block)
|
17
18
|
@original.each_slice(size) do |group|
|
18
19
|
@csv = group
|
19
|
-
|
20
|
+
instance_exec(self, &block)
|
20
21
|
end
|
21
22
|
@index = nil
|
22
23
|
@csv = @original
|
@@ -29,7 +30,7 @@ module SimpleCsv
|
|
29
30
|
|
30
31
|
@csv.each do |record|
|
31
32
|
@record = record
|
32
|
-
|
33
|
+
instance_exec(self, &block)
|
33
34
|
@index += 1 if @index
|
34
35
|
end
|
35
36
|
end
|
@@ -43,7 +44,7 @@ module SimpleCsv
|
|
43
44
|
end
|
44
45
|
|
45
46
|
def load_csv_with_manual_headers
|
46
|
-
SimpleCsv.csv_manually_set_headers! unless
|
47
|
+
SimpleCsv.csv_manually_set_headers! unless headers?
|
47
48
|
csv_arr = CSV.open(@csv_path).to_a
|
48
49
|
|
49
50
|
if csv_arr.first.size == headers.size
|
@@ -55,30 +56,11 @@ module SimpleCsv
|
|
55
56
|
end
|
56
57
|
end
|
57
58
|
|
58
|
-
def find_headers
|
59
|
-
first_line.split(detect_delimiter).map { |h| h.gsub(/^"*|"*$/, '') }
|
60
|
-
end
|
61
|
-
|
62
|
-
def detect_delimiter
|
63
|
-
line = first_line
|
64
|
-
@delimiters = COMMON_DELIMITERS.map { |sep| [sep, line.scan(sep).length] }
|
65
|
-
.sort { |a, b| b[1] <=> a[1] }
|
66
|
-
@delimiter ||= @delimiters[0][0]
|
67
|
-
end
|
68
|
-
|
69
|
-
def first_line
|
70
|
-
@first_line ||= File.open @csv_path, &:readline
|
71
|
-
end
|
72
|
-
|
73
|
-
def respond_to_missing?(mtd, include_private)
|
74
|
-
headers.include?(m) || @col_map.key?(m)
|
75
|
-
end
|
76
|
-
|
77
59
|
def method_missing(mtd, *args, &block)
|
78
60
|
m = mtd.to_s
|
79
61
|
return @record[m] if headers.include?(m)
|
80
62
|
return @record[@col_map[m]] if @col_map.key?(m)
|
81
|
-
|
63
|
+
@caller_self.send mtd, *args, &block
|
82
64
|
end
|
83
65
|
end
|
84
66
|
end
|
data/lib/simple_csv/settings.rb
CHANGED
@@ -4,7 +4,7 @@ module SimpleCsv
|
|
4
4
|
force_quotes: true,
|
5
5
|
converters: [:all, :blank_to_nil, :null_to_nil] }.freeze
|
6
6
|
ALIASSED = { seperator: :col_sep, has_headers: :headers }.freeze
|
7
|
-
INVERTED_ALIASSES = ALIASSED.to_a.map(&:reverse).to_h
|
7
|
+
INVERTED_ALIASSES = ALIASSED.to_a.map(&:reverse).to_h.freeze
|
8
8
|
|
9
9
|
def initialize(**opts)
|
10
10
|
@settings = DEFAULTS.dup.merge opts
|
@@ -0,0 +1,68 @@
|
|
1
|
+
module SimpleCsv
|
2
|
+
class Transformer < Base
|
3
|
+
DEFAULT_FILENAME = 'converted.csv'
|
4
|
+
|
5
|
+
def initialize(path, **opts, &block)
|
6
|
+
@transforms = {}
|
7
|
+
@output_headers = []
|
8
|
+
|
9
|
+
@caller_self = eval 'self', block.binding
|
10
|
+
|
11
|
+
if settings.for_csv[:headers]
|
12
|
+
@csv_path = File.expand_path path
|
13
|
+
headers(*find_headers) unless headers.any?
|
14
|
+
end
|
15
|
+
|
16
|
+
instance_exec(self, &block)
|
17
|
+
|
18
|
+
apply_transforms path, **opts
|
19
|
+
end
|
20
|
+
|
21
|
+
def output_headers(*out_headers)
|
22
|
+
return @output_headers if @output_headers.any?
|
23
|
+
|
24
|
+
@output_headers = out_headers.map(&:to_s)
|
25
|
+
alias_to_friendly_headers @output_headers
|
26
|
+
@output_headers
|
27
|
+
end
|
28
|
+
|
29
|
+
private
|
30
|
+
|
31
|
+
def apply_transforms(path, **opts)
|
32
|
+
received_headers = headers
|
33
|
+
timestamp = Time.new.strftime '%d-%m-%Y-%S%7N'
|
34
|
+
output_path = opts.delete(:output) || "#{path.split('.')[0..-2].join}-#{timestamp}.csv"
|
35
|
+
output_headers = @output_headers.any? ? @output_headers : received_headers
|
36
|
+
|
37
|
+
SimpleCsv.read path, opts do |reader|
|
38
|
+
SimpleCsv.generate output_path, opts do |writer|
|
39
|
+
writer.headers *output_headers
|
40
|
+
|
41
|
+
reader.each_row do
|
42
|
+
output_headers.each do |column|
|
43
|
+
transform = find_transform column
|
44
|
+
result = transform ? transform.call(reader.send(column))
|
45
|
+
: reader.send(column)
|
46
|
+
|
47
|
+
writer.send column, result
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
def find_transform(column)
|
55
|
+
@transforms[(@col_map.key(column.to_s) || column).to_sym]
|
56
|
+
end
|
57
|
+
|
58
|
+
def method_missing(mtd, *args, &block)
|
59
|
+
mstr = mtd.to_s
|
60
|
+
|
61
|
+
if headers.include?(mstr) || @output_headers.include?(mstr) || @col_map.key?(mstr)
|
62
|
+
@transforms[mtd] = block || args.first unless @transforms.key? mtd
|
63
|
+
else
|
64
|
+
@caller_self.send mtd, *args, &block
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
data/lib/simple_csv/version.rb
CHANGED
data/lib/simple_csv/writer.rb
CHANGED
@@ -1,28 +1,36 @@
|
|
1
1
|
module SimpleCsv
|
2
2
|
class Writer < Base
|
3
3
|
def initialize(path, **opts, &block)
|
4
|
+
@caller_self = eval 'self', block.binding
|
5
|
+
|
4
6
|
settings.apply({force_row_completion: true}, opts)
|
5
7
|
CSV.open(File.expand_path(path), 'w', settings.for_csv) do |csv|
|
6
8
|
@csv = csv
|
7
9
|
@last_row = {}
|
8
10
|
@current_row = {}
|
9
|
-
|
11
|
+
instance_exec(self, &block)
|
10
12
|
end
|
11
13
|
end
|
12
14
|
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
15
|
+
def headers(*args)
|
16
|
+
super
|
17
|
+
(@csv << @headers) && @headers_written = true if !@headers_written && @csv
|
18
|
+
@headers
|
17
19
|
end
|
18
20
|
|
21
|
+
private
|
22
|
+
|
19
23
|
def method_missing(mtd, *args, &block)
|
20
24
|
SimpleCsv.csv_manually_set_headers! unless @headers_written
|
25
|
+
|
21
26
|
current_val = @current_row[mtd] if @current_row.key?(mtd)
|
22
27
|
current_val = @last_row[mtd] if @last_row.key?(mtd)
|
23
28
|
|
24
29
|
return current_val if args.empty? && current_val
|
25
|
-
|
30
|
+
|
31
|
+
unless headers.include?(mtd.to_s) || @col_map.key?(mtd.to_s)
|
32
|
+
return @caller_self.send mtd, *args, &block
|
33
|
+
end
|
26
34
|
|
27
35
|
if settings.force_row_completion && @current_row.key?(mtd) && args.any?
|
28
36
|
SimpleCsv.row_not_complete!(mtd, args.first)
|
@@ -37,11 +45,5 @@ module SimpleCsv
|
|
37
45
|
@current_row = {}
|
38
46
|
current_val
|
39
47
|
end
|
40
|
-
|
41
|
-
def headers(*args)
|
42
|
-
super
|
43
|
-
(@csv << @headers) && @headers_written = true if !@headers_written && @csv
|
44
|
-
@headers
|
45
|
-
end
|
46
48
|
end
|
47
49
|
end
|
data/simple_csv.gemspec
CHANGED
@@ -11,7 +11,7 @@ Gem::Specification.new do |spec|
|
|
11
11
|
|
12
12
|
spec.summary = 'CSV DSL'
|
13
13
|
spec.description = 'A simple DSL for reading and generating CSV files'
|
14
|
-
spec.homepage = 'https://
|
14
|
+
spec.homepage = 'https://github.com/SidOfc/simple_csv'
|
15
15
|
spec.license = 'MIT'
|
16
16
|
|
17
17
|
spec.files = `git ls-files -z`.split("\x0").reject do |f|
|
@@ -24,9 +24,9 @@ Gem::Specification.new do |spec|
|
|
24
24
|
|
25
25
|
spec.required_ruby_version = '>= 2.1'
|
26
26
|
|
27
|
-
spec.add_development_dependency 'bundler'
|
28
|
-
spec.add_development_dependency 'rake'
|
29
|
-
spec.add_development_dependency 'rspec'
|
27
|
+
spec.add_development_dependency 'bundler'
|
28
|
+
spec.add_development_dependency 'rake'
|
29
|
+
spec.add_development_dependency 'rspec'
|
30
30
|
spec.add_development_dependency 'pry'
|
31
31
|
spec.add_development_dependency 'pry-byebug'
|
32
32
|
spec.add_development_dependency 'faker'
|
metadata
CHANGED
@@ -1,57 +1,57 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: simple_csv
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sidney Liebrand
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2018-06-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "
|
17
|
+
- - ">="
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: '
|
19
|
+
version: '0'
|
20
20
|
type: :development
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - "
|
24
|
+
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: '
|
26
|
+
version: '0'
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: rake
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
|
-
- - "
|
31
|
+
- - ">="
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version: '
|
33
|
+
version: '0'
|
34
34
|
type: :development
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
|
-
- - "
|
38
|
+
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version: '
|
40
|
+
version: '0'
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: rspec
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
|
-
- - "
|
45
|
+
- - ">="
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '
|
47
|
+
version: '0'
|
48
48
|
type: :development
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
|
-
- - "
|
52
|
+
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '
|
54
|
+
version: '0'
|
55
55
|
- !ruby/object:Gem::Dependency
|
56
56
|
name: pry
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -130,10 +130,12 @@ files:
|
|
130
130
|
- lib/simple_csv/base.rb
|
131
131
|
- lib/simple_csv/reader.rb
|
132
132
|
- lib/simple_csv/settings.rb
|
133
|
+
- lib/simple_csv/transformer.rb
|
133
134
|
- lib/simple_csv/version.rb
|
134
135
|
- lib/simple_csv/writer.rb
|
135
136
|
- simple_csv.gemspec
|
136
|
-
|
137
|
+
- tmp/.gitkeep
|
138
|
+
homepage: https://github.com/SidOfc/simple_csv
|
137
139
|
licenses:
|
138
140
|
- MIT
|
139
141
|
metadata: {}
|
@@ -153,7 +155,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
153
155
|
version: '0'
|
154
156
|
requirements: []
|
155
157
|
rubyforge_project:
|
156
|
-
rubygems_version: 2.6
|
158
|
+
rubygems_version: 2.7.6
|
157
159
|
signing_key:
|
158
160
|
specification_version: 4
|
159
161
|
summary: CSV DSL
|