csvpack 0.1.0 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/HISTORY.md +4 -4
- data/README.md +354 -354
- data/Rakefile +32 -32
- data/lib/csvpack.rb +52 -52
- data/lib/csvpack/downloader.rb +72 -62
- data/lib/csvpack/pack.rb +47 -7
- data/lib/csvpack/version.rb +22 -22
- data/test/helper.rb +7 -7
- data/test/test_companies.rb +62 -61
- data/test/test_countries.rb +41 -40
- data/test/test_downloader.rb +32 -32
- data/test/test_import.rb +22 -22
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0615ac2810a14ad606d281a410dc82954a88404d
|
|
4
|
+
data.tar.gz: fd48dba7204cb843f5f30937efc8529b3b9c33c4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c39d996ef1d6ca86c8ce70285a5259dbf5f255d4e0cc3bed7232d7de8ec0e8fb4e87e78b0d6ac31dfcf1bda8f1862f207227dd6f25c29f2ac96ddd2a44a445de
|
|
7
|
+
data.tar.gz: a66e0742a75d0aabf1a41d2ce54c9ef8c7780afde641aa244d1ab37016a11afa32db26c3f6708589d71788854311619565b6c4ffa49a5d51b0a45826fde1947d
|
data/HISTORY.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
### 0.0.1 / 2015-04-23
|
|
2
|
-
|
|
3
|
-
* Everything is new. First release
|
|
4
|
-
|
|
1
|
+
### 0.0.1 / 2015-04-23
|
|
2
|
+
|
|
3
|
+
* Everything is new. First release
|
|
4
|
+
|
data/README.md
CHANGED
|
@@ -1,354 +1,354 @@
|
|
|
1
|
-
# csvpack
|
|
2
|
-
|
|
3
|
-
work with tabular data packages using comma-separated values (CSV) datafiles in text with datapackage.json; download, read into and query comma-separated values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of choice and much more
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
* home :: [github.com/csv11/csvpack](https://github.com/csv11/csvpack)
|
|
7
|
-
* bugs :: [github.com/csv11/csvpack/issues](https://github.com/csv11/csvpack/issues)
|
|
8
|
-
* gem :: [rubygems.org/gems/csvpack](https://rubygems.org/gems/csvpack)
|
|
9
|
-
* rdoc :: [rubydoc.info/gems/csvpack](http://rubydoc.info/gems/csvpack)
|
|
10
|
-
* forum :: [ruby-talk@ruby-lang.org](http://www.ruby-lang.org/en/community/mailing-lists/)
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
## Usage
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
### What's a tabular data package?
|
|
18
|
-
|
|
19
|
-
> Tabular Data Package is a simple structure for publishing and sharing
|
|
20
|
-
> tabular data with the following key features:
|
|
21
|
-
>
|
|
22
|
-
> - Data is stored in CSV (comma separated values) files
|
|
23
|
-
> - Metadata about the dataset both general (e.g. title, author)
|
|
24
|
-
> and the specific data files (e.g. schema) is stored in a single JSON file
|
|
25
|
-
> named `datapackage.json` which follows the Data Package format
|
|
26
|
-
|
|
27
|
-
(Source: [Tabular Data Packages, Frictionless Data Initiative • Data Hub.io • Open Knowledge Foundation • Data Protocols.org](https://datahub.io/docs/data-packages/tabular))
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
Here's a minimal example of a tabular data package holding two files, that is, `data.csv` and `datapackage.json`:
|
|
32
|
-
|
|
33
|
-
`data.csv`:
|
|
34
|
-
|
|
35
|
-
```
|
|
36
|
-
Brewery,City,Name,Abv
|
|
37
|
-
Andechser Klosterbrauerei,Andechs,Doppelbock Dunkel,7%
|
|
38
|
-
Augustiner Bräu München,München,Edelstoff,5.6%
|
|
39
|
-
Bayerische Staatsbrauerei Weihenstephan,Freising,Hefe Weissbier,5.4%
|
|
40
|
-
Brauerei Spezial,Bamberg,Rauchbier Märzen,5.1%
|
|
41
|
-
Hacker-Pschorr Bräu,München,Münchner Dunkel,5.0%
|
|
42
|
-
Staatliches Hofbräuhaus München,München,Hofbräu Oktoberfestbier,6.3%
|
|
43
|
-
...
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
`datapackage.json`:
|
|
47
|
-
|
|
48
|
-
``` json
|
|
49
|
-
{
|
|
50
|
-
"name": "beer",
|
|
51
|
-
"resources": [
|
|
52
|
-
{
|
|
53
|
-
"path": "data.csv",
|
|
54
|
-
"schema": {
|
|
55
|
-
"fields": [{ "name": "Brewery", "type": "string" },
|
|
56
|
-
{ "name": "City", "type": "string" },
|
|
57
|
-
{ "name": "Name", "type": "string" },
|
|
58
|
-
{ "name": "Abv", "type": "number" }]
|
|
59
|
-
}
|
|
60
|
-
}
|
|
61
|
-
]
|
|
62
|
-
}
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
### Where to find data packages?
|
|
68
|
-
|
|
69
|
-
For some real world examples see the [Data Packages Listing](https://datahub.io/core) ([Sources](https://github.com/datasets)) at the Data Hub.io • Frictionless Data Initiative
|
|
70
|
-
website for a start. Tabular data packages include:
|
|
71
|
-
|
|
72
|
-
Name | Comments
|
|
73
|
-
------------------------ | -------------
|
|
74
|
-
`country-codes` | Comprehensive country codes: ISO 3166, ITU, ISO 4217 currency codes and many more
|
|
75
|
-
`language-codes` | ISO Language Codes (639-1 and 693-2)
|
|
76
|
-
`currency-codes` | ISO 4217 Currency Codes
|
|
77
|
-
`gdb` | Country, Regional and World GDP (Gross Domestic Product)
|
|
78
|
-
`s-and-p-500-companies` | S&P 500 Companies with Financial Information
|
|
79
|
-
`un-locode` | UN-LOCODE Codelist
|
|
80
|
-
`gold-prices` | Gold Prices (Monthly in USD)
|
|
81
|
-
`bond-yields-uk-10y` | 10 Year UK Government Bond Yields (Long-Term Interest Rate)
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
and many more
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
### Code, Code, Code - Script Your Data Workflow with Ruby
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
``` ruby
|
|
92
|
-
require 'csvpack'
|
|
93
|
-
|
|
94
|
-
CsvPack.import(
|
|
95
|
-
's-and-p-500-companies',
|
|
96
|
-
'gdb'
|
|
97
|
-
)
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
Using `CsvPack.import` will:
|
|
101
|
-
|
|
102
|
-
1) download all data packages to the `./pack` folder
|
|
103
|
-
|
|
104
|
-
2) (auto-)add all tables to an in-memory SQLite database using SQL `create_table`
|
|
105
|
-
commands via `ActiveRecord` migrations e.g.
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
``` ruby
|
|
109
|
-
create_table :constituents_financials do |t|
|
|
110
|
-
t.string :symbol # Symbol (string)
|
|
111
|
-
t.string :name # Name (string)
|
|
112
|
-
t.string :sector # Sector (string)
|
|
113
|
-
t.float :price # Price (number)
|
|
114
|
-
t.float :dividend_yield # Dividend Yield (number)
|
|
115
|
-
t.float :price_earnings # Price/Earnings (number)
|
|
116
|
-
t.float :earnings_share # Earnings/Share (number)
|
|
117
|
-
t.float :book_value # Book Value (number)
|
|
118
|
-
t.float :_52_week_low # 52 week low (number)
|
|
119
|
-
t.float :_52_week_high # 52 week high (number)
|
|
120
|
-
t.float :market_cap # Market Cap (number)
|
|
121
|
-
t.float :ebitda # EBITDA (number)
|
|
122
|
-
t.float :price_sales # Price/Sales (number)
|
|
123
|
-
t.float :price_book # Price/Book (number)
|
|
124
|
-
t.string :sec_filings # SEC Filings (string)
|
|
125
|
-
end
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
3) (auto-)import all datasets using SQL inserts e.g.
|
|
129
|
-
|
|
130
|
-
``` sql
|
|
131
|
-
INSERT INTO constituents_financials
|
|
132
|
-
(symbol,
|
|
133
|
-
name,
|
|
134
|
-
sector,
|
|
135
|
-
price,
|
|
136
|
-
dividend_yield,
|
|
137
|
-
price_earnings,
|
|
138
|
-
earnings_share,
|
|
139
|
-
book_value,
|
|
140
|
-
_52_week_low,
|
|
141
|
-
_52_week_high,
|
|
142
|
-
market_cap,
|
|
143
|
-
ebitda,
|
|
144
|
-
price_sales,
|
|
145
|
-
price_book,
|
|
146
|
-
sec_filings)
|
|
147
|
-
VALUES
|
|
148
|
-
('MMM',
|
|
149
|
-
'3M
|
|
150
|
-
'Industrials',
|
|
151
|
-
162.27,
|
|
152
|
-
2.11,
|
|
153
|
-
22.28,
|
|
154
|
-
7.284,
|
|
155
|
-
25.238,
|
|
156
|
-
123.61,
|
|
157
|
-
162.92,
|
|
158
|
-
104.0,
|
|
159
|
-
8.467,
|
|
160
|
-
3.28,
|
|
161
|
-
6.43,
|
|
162
|
-
'http://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=MMM')
|
|
163
|
-
```
|
|
164
|
-
|
|
165
|
-
4) (auto-)add ActiveRecord models for all tables.
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
So what? Now you can use all the "magic" of ActiveRecord to query
|
|
169
|
-
the datasets. Example:
|
|
170
|
-
|
|
171
|
-
``` ruby
|
|
172
|
-
pp Constituent.count
|
|
173
|
-
|
|
174
|
-
# SELECT COUNT(*) FROM "constituents"
|
|
175
|
-
# => 496
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
pp Constituent.first
|
|
179
|
-
|
|
180
|
-
# SELECT "constituents".* FROM "constituents" ORDER BY "constituents"."id" ASC LIMIT 1
|
|
181
|
-
# => #<Constituent:0x9f8cb78
|
|
182
|
-
# id: 1,
|
|
183
|
-
# symbol: "MMM",
|
|
184
|
-
# name: "3M
|
|
185
|
-
# sector: "Industrials">
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
pp Constituent.find_by!( symbol: 'MMM' )
|
|
189
|
-
|
|
190
|
-
# SELECT "constituents".*
|
|
191
|
-
# FROM "constituents"
|
|
192
|
-
# WHERE "constituents"."symbol" = "MMM"
|
|
193
|
-
# LIMIT 1
|
|
194
|
-
# => #<Constituent:0x9f8cb78
|
|
195
|
-
# id: 1,
|
|
196
|
-
# symbol: "MMM",
|
|
197
|
-
# name: "3M
|
|
198
|
-
# sector: "Industrials">
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
pp Constituent.find_by!( name: '3M
|
|
202
|
-
|
|
203
|
-
# SELECT "constituents".*
|
|
204
|
-
# FROM "constituents"
|
|
205
|
-
# WHERE "constituents"."name" = "3M
|
|
206
|
-
# LIMIT 1
|
|
207
|
-
# => #<Constituent:0x9f8cb78
|
|
208
|
-
# id: 1,
|
|
209
|
-
# symbol: "MMM",
|
|
210
|
-
# name: "3M
|
|
211
|
-
# sector: "Industrials">
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
pp Constituent.where( sector: 'Industrials' ).count
|
|
215
|
-
|
|
216
|
-
# SELECT COUNT(*) FROM "constituents"
|
|
217
|
-
# WHERE "constituents"."sector" = "Industrials"
|
|
218
|
-
# => 63
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
pp Constituent.where( sector: 'Industrials' ).all
|
|
222
|
-
|
|
223
|
-
# SELECT "constituents".*
|
|
224
|
-
# FROM "constituents"
|
|
225
|
-
# WHERE "constituents"."sector" = "Industrials"
|
|
226
|
-
# => [#<Constituent:0x9f8cb78
|
|
227
|
-
# id: 1,
|
|
228
|
-
# symbol: "MMM",
|
|
229
|
-
# name: "3M
|
|
230
|
-
# sector: "Industrials">,
|
|
231
|
-
# #<Constituent:0xa2a4180
|
|
232
|
-
# id: 8,
|
|
233
|
-
# symbol: "ADT",
|
|
234
|
-
# name: "ADT Corp (The)",
|
|
235
|
-
# sector: "Industrials">,...]
|
|
236
|
-
```
|
|
237
|
-
|
|
238
|
-
and so on
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
### Frequently Asked Questions (F.A.Qs) and Answers
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
#### Q: How to dowload a data package ("by hand")?
|
|
246
|
-
|
|
247
|
-
Use the `CsvPack::Downloader` class to download a data package
|
|
248
|
-
to your disk (by default data packages get stored in `./pack`).
|
|
249
|
-
|
|
250
|
-
``` ruby
|
|
251
|
-
dl = CsvPack::Downloader.new
|
|
252
|
-
dl.fetch( 'language-codes' )
|
|
253
|
-
dl.fetch( 's-and-p-500-companies' )
|
|
254
|
-
dl.fetch( 'un-locode')
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
Will result in:
|
|
258
|
-
|
|
259
|
-
```
|
|
260
|
-
-- pack
|
|
261
|
-
|-- language-codes
|
|
262
|
-
| |-- data
|
|
263
|
-
| | |-- language-
|
|
264
|
-
| | |-- language-codes.csv
|
|
265
|
-
| |
|
|
266
|
-
| `--
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
|
270
|
-
| | `-- constituents
|
|
271
|
-
| `-- datapackage.json
|
|
272
|
-
`-- un-locode
|
|
273
|
-
|-- data
|
|
274
|
-
| |-- code-list.csv
|
|
275
|
-
| |-- country-codes.csv
|
|
276
|
-
| |-- function-classifiers.csv
|
|
277
|
-
| |-- status-indicators.csv
|
|
278
|
-
| `-- subdivision-codes.csv
|
|
279
|
-
`-- datapackage.json
|
|
280
|
-
```
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
#### Q: How to add and import a data package ("by hand")?
|
|
284
|
-
|
|
285
|
-
Use the `CsvPack::Pack` class to read-in a data package
|
|
286
|
-
and add and import into an SQL database.
|
|
287
|
-
|
|
288
|
-
``` ruby
|
|
289
|
-
pack = CsvPack::Pack.new( './pack/un-locode/datapackage.json' )
|
|
290
|
-
pack.tables.each do |table|
|
|
291
|
-
table.up! # (auto-) add table using SQL create_table via ActiveRecord migration
|
|
292
|
-
table.import! # import all records using SQL inserts
|
|
293
|
-
end
|
|
294
|
-
```
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
#### Q: How to connect to a different SQL database?
|
|
298
|
-
|
|
299
|
-
You can connect to any database supported by ActiveRecord. If you do NOT
|
|
300
|
-
establish a connection in your script - the standard (default fallback)
|
|
301
|
-
is using an in-memory SQLite3 database.
|
|
302
|
-
|
|
303
|
-
##### SQLite
|
|
304
|
-
|
|
305
|
-
For example, to create an SQLite3 database on disk - lets say `mine.db` -
|
|
306
|
-
use in your script (before the `CsvPack.import` statement):
|
|
307
|
-
|
|
308
|
-
``` ruby
|
|
309
|
-
ActiveRecord::Base.establish_connection( adapter: 'sqlite3',
|
|
310
|
-
database: './mine.db' )
|
|
311
|
-
```
|
|
312
|
-
|
|
313
|
-
##### PostgreSQL
|
|
314
|
-
|
|
315
|
-
For example, to connect to a PostgreSQL database use in your script
|
|
316
|
-
(before the `CsvPack.import` statement):
|
|
317
|
-
|
|
318
|
-
``` ruby
|
|
319
|
-
require 'pg' ## pull-in PostgreSQL (pg) machinery
|
|
320
|
-
|
|
321
|
-
ActiveRecord::Base.establish_connection( adapter: 'postgresql'
|
|
322
|
-
username: 'ruby',
|
|
323
|
-
password: 'topsecret',
|
|
324
|
-
database: 'database' )
|
|
325
|
-
```
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
## Install
|
|
331
|
-
|
|
332
|
-
Just install the gem:
|
|
333
|
-
|
|
334
|
-
```
|
|
335
|
-
$ gem install csvpack
|
|
336
|
-
```
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
## Alternatives
|
|
341
|
-
|
|
342
|
-
See the "[Tools and Plugins for working with Data Packages](https://frictionlessdata.io/software)"
|
|
343
|
-
page at the Frictionless Data Initiative.
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
## License
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
The `csvpack` scripts are dedicated to the public domain.
|
|
350
|
-
Use it as you please with no restrictions whatsoever.
|
|
351
|
-
|
|
352
|
-
## Questions? Comments?
|
|
353
|
-
|
|
354
|
-
Send them along to the ruby-talk mailing list. Thanks!
|
|
1
|
+
# csvpack
|
|
2
|
+
|
|
3
|
+
work with tabular data packages using comma-separated values (CSV) datafiles in text with datapackage.json; download, read into and query comma-separated values (CSV) datafiles with your SQL database (e.g. SQLite, PostgreSQL, ...) of choice and much more
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
* home :: [github.com/csv11/csvpack](https://github.com/csv11/csvpack)
|
|
7
|
+
* bugs :: [github.com/csv11/csvpack/issues](https://github.com/csv11/csvpack/issues)
|
|
8
|
+
* gem :: [rubygems.org/gems/csvpack](https://rubygems.org/gems/csvpack)
|
|
9
|
+
* rdoc :: [rubydoc.info/gems/csvpack](http://rubydoc.info/gems/csvpack)
|
|
10
|
+
* forum :: [ruby-talk@ruby-lang.org](http://www.ruby-lang.org/en/community/mailing-lists/)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
## Usage
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
### What's a tabular data package?
|
|
18
|
+
|
|
19
|
+
> Tabular Data Package is a simple structure for publishing and sharing
|
|
20
|
+
> tabular data with the following key features:
|
|
21
|
+
>
|
|
22
|
+
> - Data is stored in CSV (comma separated values) files
|
|
23
|
+
> - Metadata about the dataset both general (e.g. title, author)
|
|
24
|
+
> and the specific data files (e.g. schema) is stored in a single JSON file
|
|
25
|
+
> named `datapackage.json` which follows the Data Package format
|
|
26
|
+
|
|
27
|
+
(Source: [Tabular Data Packages, Frictionless Data Initiative • Data Hub.io • Open Knowledge Foundation • Data Protocols.org](https://datahub.io/docs/data-packages/tabular))
|
|
28
|
+
|
|
29
|
+
|
|
30
|
+
|
|
31
|
+
Here's a minimal example of a tabular data package holding two files, that is, `data.csv` and `datapackage.json`:
|
|
32
|
+
|
|
33
|
+
`data.csv`:
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
Brewery,City,Name,Abv
|
|
37
|
+
Andechser Klosterbrauerei,Andechs,Doppelbock Dunkel,7%
|
|
38
|
+
Augustiner Bräu München,München,Edelstoff,5.6%
|
|
39
|
+
Bayerische Staatsbrauerei Weihenstephan,Freising,Hefe Weissbier,5.4%
|
|
40
|
+
Brauerei Spezial,Bamberg,Rauchbier Märzen,5.1%
|
|
41
|
+
Hacker-Pschorr Bräu,München,Münchner Dunkel,5.0%
|
|
42
|
+
Staatliches Hofbräuhaus München,München,Hofbräu Oktoberfestbier,6.3%
|
|
43
|
+
...
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
`datapackage.json`:
|
|
47
|
+
|
|
48
|
+
``` json
|
|
49
|
+
{
|
|
50
|
+
"name": "beer",
|
|
51
|
+
"resources": [
|
|
52
|
+
{
|
|
53
|
+
"path": "data.csv",
|
|
54
|
+
"schema": {
|
|
55
|
+
"fields": [{ "name": "Brewery", "type": "string" },
|
|
56
|
+
{ "name": "City", "type": "string" },
|
|
57
|
+
{ "name": "Name", "type": "string" },
|
|
58
|
+
{ "name": "Abv", "type": "number" }]
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
]
|
|
62
|
+
}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
### Where to find data packages?
|
|
68
|
+
|
|
69
|
+
For some real world examples see the [Data Packages Listing](https://datahub.io/core) ([Sources](https://github.com/datasets)) at the Data Hub.io • Frictionless Data Initiative
|
|
70
|
+
website for a start. Tabular data packages include:
|
|
71
|
+
|
|
72
|
+
Name | Comments
|
|
73
|
+
------------------------ | -------------
|
|
74
|
+
`country-codes` | Comprehensive country codes: ISO 3166, ITU, ISO 4217 currency codes and many more
|
|
75
|
+
`language-codes` | ISO Language Codes (639-1 and 693-2)
|
|
76
|
+
`currency-codes` | ISO 4217 Currency Codes
|
|
77
|
+
`gdb` | Country, Regional and World GDP (Gross Domestic Product)
|
|
78
|
+
`s-and-p-500-companies` | S&P 500 Companies with Financial Information
|
|
79
|
+
`un-locode` | UN-LOCODE Codelist
|
|
80
|
+
`gold-prices` | Gold Prices (Monthly in USD)
|
|
81
|
+
`bond-yields-uk-10y` | 10 Year UK Government Bond Yields (Long-Term Interest Rate)
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
and many more
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
### Code, Code, Code - Script Your Data Workflow with Ruby
|
|
89
|
+
|
|
90
|
+
|
|
91
|
+
``` ruby
|
|
92
|
+
require 'csvpack'
|
|
93
|
+
|
|
94
|
+
CsvPack.import(
|
|
95
|
+
's-and-p-500-companies',
|
|
96
|
+
'gdb'
|
|
97
|
+
)
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Using `CsvPack.import` will:
|
|
101
|
+
|
|
102
|
+
1) download all data packages to the `./pack` folder
|
|
103
|
+
|
|
104
|
+
2) (auto-)add all tables to an in-memory SQLite database using SQL `create_table`
|
|
105
|
+
commands via `ActiveRecord` migrations e.g.
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
``` ruby
|
|
109
|
+
create_table :constituents_financials do |t|
|
|
110
|
+
t.string :symbol # Symbol (string)
|
|
111
|
+
t.string :name # Name (string)
|
|
112
|
+
t.string :sector # Sector (string)
|
|
113
|
+
t.float :price # Price (number)
|
|
114
|
+
t.float :dividend_yield # Dividend Yield (number)
|
|
115
|
+
t.float :price_earnings # Price/Earnings (number)
|
|
116
|
+
t.float :earnings_share # Earnings/Share (number)
|
|
117
|
+
t.float :book_value # Book Value (number)
|
|
118
|
+
t.float :_52_week_low # 52 week low (number)
|
|
119
|
+
t.float :_52_week_high # 52 week high (number)
|
|
120
|
+
t.float :market_cap # Market Cap (number)
|
|
121
|
+
t.float :ebitda # EBITDA (number)
|
|
122
|
+
t.float :price_sales # Price/Sales (number)
|
|
123
|
+
t.float :price_book # Price/Book (number)
|
|
124
|
+
t.string :sec_filings # SEC Filings (string)
|
|
125
|
+
end
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
3) (auto-)import all datasets using SQL inserts e.g.
|
|
129
|
+
|
|
130
|
+
``` sql
|
|
131
|
+
INSERT INTO constituents_financials
|
|
132
|
+
(symbol,
|
|
133
|
+
name,
|
|
134
|
+
sector,
|
|
135
|
+
price,
|
|
136
|
+
dividend_yield,
|
|
137
|
+
price_earnings,
|
|
138
|
+
earnings_share,
|
|
139
|
+
book_value,
|
|
140
|
+
_52_week_low,
|
|
141
|
+
_52_week_high,
|
|
142
|
+
market_cap,
|
|
143
|
+
ebitda,
|
|
144
|
+
price_sales,
|
|
145
|
+
price_book,
|
|
146
|
+
sec_filings)
|
|
147
|
+
VALUES
|
|
148
|
+
('MMM',
|
|
149
|
+
'3M Company',
|
|
150
|
+
'Industrials',
|
|
151
|
+
162.27,
|
|
152
|
+
2.11,
|
|
153
|
+
22.28,
|
|
154
|
+
7.284,
|
|
155
|
+
25.238,
|
|
156
|
+
123.61,
|
|
157
|
+
162.92,
|
|
158
|
+
104.0,
|
|
159
|
+
8.467,
|
|
160
|
+
3.28,
|
|
161
|
+
6.43,
|
|
162
|
+
'http://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=MMM')
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
4) (auto-)add ActiveRecord models for all tables.
|
|
166
|
+
|
|
167
|
+
|
|
168
|
+
So what? Now you can use all the "magic" of ActiveRecord to query
|
|
169
|
+
the datasets. Example:
|
|
170
|
+
|
|
171
|
+
``` ruby
|
|
172
|
+
pp Constituent.count
|
|
173
|
+
|
|
174
|
+
# SELECT COUNT(*) FROM "constituents"
|
|
175
|
+
# => 496
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
pp Constituent.first
|
|
179
|
+
|
|
180
|
+
# SELECT "constituents".* FROM "constituents" ORDER BY "constituents"."id" ASC LIMIT 1
|
|
181
|
+
# => #<Constituent:0x9f8cb78
|
|
182
|
+
# id: 1,
|
|
183
|
+
# symbol: "MMM",
|
|
184
|
+
# name: "3M Company",
|
|
185
|
+
# sector: "Industrials">
|
|
186
|
+
|
|
187
|
+
|
|
188
|
+
pp Constituent.find_by!( symbol: 'MMM' )
|
|
189
|
+
|
|
190
|
+
# SELECT "constituents".*
|
|
191
|
+
# FROM "constituents"
|
|
192
|
+
# WHERE "constituents"."symbol" = "MMM"
|
|
193
|
+
# LIMIT 1
|
|
194
|
+
# => #<Constituent:0x9f8cb78
|
|
195
|
+
# id: 1,
|
|
196
|
+
# symbol: "MMM",
|
|
197
|
+
# name: "3M Company",
|
|
198
|
+
# sector: "Industrials">
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
pp Constituent.find_by!( name: '3M Company' )
|
|
202
|
+
|
|
203
|
+
# SELECT "constituents".*
|
|
204
|
+
# FROM "constituents"
|
|
205
|
+
# WHERE "constituents"."name" = "3M Company"
|
|
206
|
+
# LIMIT 1
|
|
207
|
+
# => #<Constituent:0x9f8cb78
|
|
208
|
+
# id: 1,
|
|
209
|
+
# symbol: "MMM",
|
|
210
|
+
# name: "3M Company",
|
|
211
|
+
# sector: "Industrials">
|
|
212
|
+
|
|
213
|
+
|
|
214
|
+
pp Constituent.where( sector: 'Industrials' ).count
|
|
215
|
+
|
|
216
|
+
# SELECT COUNT(*) FROM "constituents"
|
|
217
|
+
# WHERE "constituents"."sector" = "Industrials"
|
|
218
|
+
# => 63
|
|
219
|
+
|
|
220
|
+
|
|
221
|
+
pp Constituent.where( sector: 'Industrials' ).all
|
|
222
|
+
|
|
223
|
+
# SELECT "constituents".*
|
|
224
|
+
# FROM "constituents"
|
|
225
|
+
# WHERE "constituents"."sector" = "Industrials"
|
|
226
|
+
# => [#<Constituent:0x9f8cb78
|
|
227
|
+
# id: 1,
|
|
228
|
+
# symbol: "MMM",
|
|
229
|
+
# name: "3M Company",
|
|
230
|
+
# sector: "Industrials">,
|
|
231
|
+
# #<Constituent:0xa2a4180
|
|
232
|
+
# id: 8,
|
|
233
|
+
# symbol: "ADT",
|
|
234
|
+
# name: "ADT Corp (The)",
|
|
235
|
+
# sector: "Industrials">,...]
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
and so on
|
|
239
|
+
|
|
240
|
+
|
|
241
|
+
|
|
242
|
+
### Frequently Asked Questions (F.A.Qs) and Answers
|
|
243
|
+
|
|
244
|
+
|
|
245
|
+
#### Q: How to dowload a data package ("by hand")?
|
|
246
|
+
|
|
247
|
+
Use the `CsvPack::Downloader` class to download a data package
|
|
248
|
+
to your disk (by default data packages get stored in `./pack`).
|
|
249
|
+
|
|
250
|
+
``` ruby
|
|
251
|
+
dl = CsvPack::Downloader.new
|
|
252
|
+
dl.fetch( 'language-codes' )
|
|
253
|
+
dl.fetch( 's-and-p-500-companies' )
|
|
254
|
+
dl.fetch( 'un-locode')
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
Will result in:
|
|
258
|
+
|
|
259
|
+
```
|
|
260
|
+
-- pack
|
|
261
|
+
|-- language-codes
|
|
262
|
+
| |-- data
|
|
263
|
+
| | |-- ietf-language-tags.csv
|
|
264
|
+
| | |-- language-codes-3b2.csv
|
|
265
|
+
| | |-- language-codes-full.csv
|
|
266
|
+
| | `-- language-codes.csv
|
|
267
|
+
| `-- datapackage.json
|
|
268
|
+
|-- s-and-p-500-companies
|
|
269
|
+
| |-- data
|
|
270
|
+
| | `-- constituents.csv
|
|
271
|
+
| `-- datapackage.json
|
|
272
|
+
`-- un-locode
|
|
273
|
+
|-- data
|
|
274
|
+
| |-- code-list.csv
|
|
275
|
+
| |-- country-codes.csv
|
|
276
|
+
| |-- function-classifiers.csv
|
|
277
|
+
| |-- status-indicators.csv
|
|
278
|
+
| `-- subdivision-codes.csv
|
|
279
|
+
`-- datapackage.json
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
|
|
283
|
+
#### Q: How to add and import a data package ("by hand")?
|
|
284
|
+
|
|
285
|
+
Use the `CsvPack::Pack` class to read-in a data package
|
|
286
|
+
and add and import into an SQL database.
|
|
287
|
+
|
|
288
|
+
``` ruby
|
|
289
|
+
pack = CsvPack::Pack.new( './pack/un-locode/datapackage.json' )
|
|
290
|
+
pack.tables.each do |table|
|
|
291
|
+
table.up! # (auto-) add table using SQL create_table via ActiveRecord migration
|
|
292
|
+
table.import! # import all records using SQL inserts
|
|
293
|
+
end
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
|
|
297
|
+
#### Q: How to connect to a different SQL database?
|
|
298
|
+
|
|
299
|
+
You can connect to any database supported by ActiveRecord. If you do NOT
|
|
300
|
+
establish a connection in your script - the standard (default fallback)
|
|
301
|
+
is using an in-memory SQLite3 database.
|
|
302
|
+
|
|
303
|
+
##### SQLite
|
|
304
|
+
|
|
305
|
+
For example, to create an SQLite3 database on disk - lets say `mine.db` -
|
|
306
|
+
use in your script (before the `CsvPack.import` statement):
|
|
307
|
+
|
|
308
|
+
``` ruby
|
|
309
|
+
ActiveRecord::Base.establish_connection( adapter: 'sqlite3',
|
|
310
|
+
database: './mine.db' )
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
##### PostgreSQL
|
|
314
|
+
|
|
315
|
+
For example, to connect to a PostgreSQL database use in your script
|
|
316
|
+
(before the `CsvPack.import` statement):
|
|
317
|
+
|
|
318
|
+
``` ruby
|
|
319
|
+
require 'pg' ## pull-in PostgreSQL (pg) machinery
|
|
320
|
+
|
|
321
|
+
ActiveRecord::Base.establish_connection( adapter: 'postgresql'
|
|
322
|
+
username: 'ruby',
|
|
323
|
+
password: 'topsecret',
|
|
324
|
+
database: 'database' )
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
|
|
328
|
+
|
|
329
|
+
|
|
330
|
+
## Install
|
|
331
|
+
|
|
332
|
+
Just install the gem:
|
|
333
|
+
|
|
334
|
+
```
|
|
335
|
+
$ gem install csvpack
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
|
|
339
|
+
|
|
340
|
+
## Alternatives
|
|
341
|
+
|
|
342
|
+
See the "[Tools and Plugins for working with Data Packages](https://frictionlessdata.io/software)"
|
|
343
|
+
page at the Frictionless Data Initiative.
|
|
344
|
+
|
|
345
|
+
|
|
346
|
+
## License
|
|
347
|
+
|
|
348
|
+
|
|
349
|
+
The `csvpack` scripts are dedicated to the public domain.
|
|
350
|
+
Use it as you please with no restrictions whatsoever.
|
|
351
|
+
|
|
352
|
+
## Questions? Comments?
|
|
353
|
+
|
|
354
|
+
Send them along to the ruby-talk mailing list. Thanks!
|