datahunter 0.2.4 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +48 -50
- data/bin/hunter +71 -148
- data/datahunter.gemspec +5 -4
- data/lib/datahunter/base.rb +55 -31
- data/lib/datahunter/version.rb +1 -1
- metadata +22 -8
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: e9919f3ad7ae09cbb6e8c15d69ad969b6a7d540f
|
|
4
|
+
data.tar.gz: 3ab09545715106aa731338edb0bbb5803d8956e5
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 06a00c3e9c22884397dcb87563ffeb603e0dd2c382f0e3723a9825d9dbe449b0f1e14d93856a14b72269c1030e24fb2880c81af42691e7aea748752868108296
|
|
7
|
+
data.tar.gz: 367e8b39398dd88bff9d00d6859ca1ef5aeec44b87393670dff8e1363f62c9b3ad53b18febbcb7acf21d18c8a8cb312313425a776f54b126b008ce9bb7c15078
|
data/README.md
CHANGED
|
@@ -25,67 +25,56 @@ We believe that pre-processing data is a bad idea and that hosting them could cr
|
|
|
25
25
|
### $ hunter find
|
|
26
26
|
|
|
27
27
|
$ hunter find consumer
|
|
28
|
-
### Response in 0.
|
|
29
|
-
###
|
|
28
|
+
### Response in 0.894806 seconds
|
|
29
|
+
### We've found 34 datasets corresponding to your query consumer:
|
|
30
30
|
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
31
|
+
1. Consumer Complaint Database id: 54de229aa82653913d1142ee
|
|
32
|
+
["us", "usa", "america", "united states", "united-states"]
|
|
33
|
+
These are complaints we’ve received about financial products and services.
|
|
34
|
+
2. Consumer Expenditure Survey id: 54de229ba82653913d1142f8
|
|
35
|
+
["us", "usa", "america", "united states", "united-states"]
|
|
36
|
+
The Consumer Expenditure Survey (CE) program consists of two surveys, the quarterly Interview ...
|
|
37
|
+
3. Farmers Markets Geographic Data id: 54de229ba82653913d1142fe
|
|
38
|
+
["us", "usa", "america", "united states", "united-states"]
|
|
39
|
+
longitude and latitude, state, address, name, and zip code of Farmers Markets in the United ...
|
|
40
|
+
4. Consumer Price Indices id: 54e8c7dfa8268b5c235ac352
|
|
41
|
+
["england", "scotland", "wales", "northern_ireland"]
|
|
42
|
+
Contains price indices, percentage changes and weights for the Consumer Prices Index (CPI), Retail ...
|
|
43
|
+
5. Organogram and staff pay data for the Consumer Council for Water id: 54e8c872a8268b5c235ac5d9
|
|
44
|
+
["uk", "england", "scotland", "wales", "ireland"]
|
|
45
|
+
A list of most Senior Civil Service posts in the Consumer Council for Water including title, ...
|
|
39
46
|
|
|
40
|
-
###
|
|
41
|
-
|
|
47
|
+
### GET A DATASET ? (1/..5), show next 5 datasets? (RET) or abort? (abort)
|
|
48
|
+
1
|
|
42
49
|
0. Consumer Complaint Database - CSV
|
|
43
50
|
1. Consumer Complaint Database - JSON
|
|
44
51
|
2. Consumer Complaint Database - XML
|
|
52
|
+
3. Consumer Complaint Database - api
|
|
45
53
|
### which one? (0/1/...)
|
|
46
|
-
|
|
54
|
+
1
|
|
55
|
+
Create/overwrite /Users/Terpolilli/views.json?(y/rename/n)
|
|
56
|
+
y
|
|
47
57
|
Start downloading...
|
|
48
|
-
Your file has been downloaded
|
|
58
|
+
Your file has been downloaded ;)
|
|
49
59
|
If this is not the file you expected, it's maybe because publisher don't always keep the metadata up-to-date. We try to clean most of uri's and check the url. Anyway you may be able to download your file by hand here:
|
|
50
60
|
https://data.consumerfinance.gov/api/views/x94z-ydhh/rows.csv?accessType=DOWNLOAD
|
|
51
61
|
|
|
52
|
-
|
|
63
|
+
|
|
76
64
|
### $ hunter info
|
|
77
65
|
|
|
78
|
-
$ hunter info
|
|
66
|
+
$ hunter info 54de229aa82653913d1142ee
|
|
67
|
+
|
|
68
|
+
Consumer Complaint Database
|
|
69
|
+
These are complaints we’ve received about financial products and services.
|
|
79
70
|
|
|
80
|
-
title: Consumer Complaint Database
|
|
81
|
-
description: These are complaints we’ve received about financial products and services.
|
|
82
71
|
publisher: Consumer Financial Protection Bureau
|
|
83
72
|
temporal: ["2011", "2012", "2013"]
|
|
84
73
|
spatial: ["us", "usa", "america", "united states", "united-states", "united states of america", "united-states-of-america", "world", "countries", "all"]
|
|
85
|
-
created: 2014-02-25T18:48:25.
|
|
86
|
-
updated:
|
|
87
|
-
score:
|
|
88
|
-
|
|
74
|
+
created: 2014-02-25T18:48:25.000Z
|
|
75
|
+
updated: 2015-01-30T15:16:28.000Z
|
|
76
|
+
score: 26.683
|
|
77
|
+
|
|
89
78
|
### $ hunter get
|
|
90
79
|
|
|
91
80
|
$ hunter get 548c82a7a826dfe85070e5fa
|
|
@@ -95,11 +84,11 @@ We believe that pre-processing data is a bad idea and that hosting them could cr
|
|
|
95
84
|
2. Consumer Complaint Database - XML
|
|
96
85
|
### which one? (0/1/...)
|
|
97
86
|
1
|
|
98
|
-
Create/overwrite /Users/Terpolilli/views.json?(y/rename/n)
|
|
87
|
+
### Create/overwrite /Users/Terpolilli/views.json?(y/rename/n)
|
|
99
88
|
rename
|
|
100
89
|
Path/to/filename: /Users/Terpolilli/Downloads/consumer-data.json
|
|
101
|
-
Start downloading...
|
|
102
|
-
Your file has been downloaded ;)
|
|
90
|
+
### Start downloading...
|
|
91
|
+
### Your file has been downloaded ;)
|
|
103
92
|
If this is not the file you expected, it's maybe because publisher don't always keep the metadata up-to-date. We try to clean most of uri's and check the url. Anyway you may be able to download your file by hand here:
|
|
104
93
|
https://data.consumerfinance.gov/api/views/x94z-ydhh/rows.csv?accessType=DOWNLOAD
|
|
105
94
|
|
|
@@ -108,13 +97,22 @@ Don't hesitate to [give us any feedback about you experience with Hunter!](https
|
|
|
108
97
|
|
|
109
98
|
## Update
|
|
110
99
|
|
|
111
|
-
* datasets indexed:
|
|
112
|
-
* last datasets indexed: Canada open data, NETL's Energy Data eXchange, dati.gov.it
|
|
100
|
+
* datasets indexed: 8336
|
|
101
|
+
* last datasets indexed: Canada open data, NETL's Energy Data eXchange, dati.gov.it, complete french health DAMIR data.
|
|
102
|
+
|
|
103
|
+
## Change Log
|
|
104
|
+
|
|
105
|
+
* 0.3.x - Merge `$ hunter find <keyword>` and `$ hunter search <keyword>` commands.
|
|
106
|
+
The new `$ hunter find` command displays the datasets corresponding to the query, 5 by 5,
|
|
107
|
+
sorted by popularity
|
|
108
|
+
* 0.2.x - New feature: download datasets directly from the CLI
|
|
109
|
+
* 0.1.x
|
|
113
110
|
|
|
114
111
|
## Roadmap
|
|
115
112
|
|
|
116
113
|
* more datasets
|
|
117
114
|
* improve the download of datasets
|
|
115
|
+
* searched and downloaded data management
|
|
118
116
|
* more generic lib to make the API more easy to reuse
|
|
119
117
|
|
|
120
118
|
## Contributing
|
data/bin/hunter
CHANGED
|
@@ -9,74 +9,65 @@ require 'launchy'
|
|
|
9
9
|
require 'colorize'
|
|
10
10
|
require 'downloadr'
|
|
11
11
|
|
|
12
|
+
program :name, 'Hunter'
|
|
12
13
|
program :version, Datahunter::VERSION
|
|
13
14
|
program :description, 'Find an download Open Datasets directly from your terminal'
|
|
14
15
|
program :help_formatter, :compact
|
|
15
16
|
|
|
16
17
|
command :find do |c|
|
|
17
18
|
c.syntax = 'hunter find keyword [spatial-coverage] [temporal-coverage]'
|
|
18
|
-
c.summary = '
|
|
19
|
-
* Returns the 3 most popular (most reused, viewed, etc)
|
|
20
|
-
datasets corresponding to the triple keyword,
|
|
21
|
-
spatial-coverage and temporal-coverage
|
|
22
|
-
* Allow you to either download the dataset or to open
|
|
23
|
-
your favorite browser directly at the good page'.colorize(:blue)
|
|
19
|
+
c.summary = 'Displays the datasets corresponding to the query'.colorize(:blue)
|
|
24
20
|
c.description = c.summary
|
|
25
|
-
c.example 'Find open data for the U.S. population', 'hunter find population us'
|
|
26
21
|
c.action do |args, options|
|
|
27
|
-
|
|
22
|
+
|
|
28
23
|
start_time = Time.now
|
|
29
24
|
|
|
30
25
|
if args[0].nil?
|
|
31
26
|
puts "usage: $ hunter find keyword [geo] [temp]"
|
|
32
27
|
elsif
|
|
33
|
-
url = Datahunter.datasets_url(args[0], args[1], args[2])
|
|
34
28
|
|
|
29
|
+
url = Datahunter.datasets_url(args[0], args[1], args[2])
|
|
35
30
|
response = RestClient.get(url, :content_type => :json) do |response, request, result, &block|
|
|
36
31
|
elapsed_time = Time.now - start_time
|
|
37
32
|
|
|
38
33
|
case response.code
|
|
39
34
|
when 200
|
|
40
35
|
datasets = JSON.parse(response.body).reverse
|
|
36
|
+
number_of_datasets = datasets.length
|
|
37
|
+
|
|
41
38
|
puts "### Response in #{elapsed_time} seconds".colorize(:blue)
|
|
39
|
+
puts "### We've found #{datasets.size} datasets corresponding to your query #{args[0].downcase}:".colorize(:blue)
|
|
40
|
+
puts
|
|
42
41
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
"### And this one?".colorize(:blue),
|
|
46
|
-
"### Last try...".colorize(:blue)
|
|
47
|
-
]
|
|
42
|
+
for i in 1..(number_of_datasets/5 + 1)
|
|
43
|
+
sub_datasets = datasets[(5 * i - 5) .. (5 * i - 1)]
|
|
48
44
|
|
|
49
|
-
|
|
50
|
-
puts messages[index]
|
|
51
|
-
puts
|
|
52
|
-
Datahunter.print_dataset_info datasets[index]
|
|
45
|
+
Datahunter.print_coll_of_datasets_info_light sub_datasets
|
|
53
46
|
|
|
54
|
-
case ask "###
|
|
55
|
-
when '
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
47
|
+
case ask "### GET A DATASET ? (1/..5), show next 5 datasets? (RET) or abort? (abort)".colorize(:yellow)
|
|
48
|
+
when '1'
|
|
49
|
+
Datahunter.get_dataset sub_datasets[0]
|
|
50
|
+
break
|
|
51
|
+
when '2'
|
|
52
|
+
Datahunter.get_dataset sub_datasets[1]
|
|
53
|
+
break
|
|
54
|
+
when '3'
|
|
55
|
+
Datahunter.get_dataset sub_datasets[2]
|
|
56
|
+
break
|
|
57
|
+
when '4'
|
|
58
|
+
Datahunter.get_dataset sub_datasets[3]
|
|
59
|
+
break
|
|
60
|
+
when '5'
|
|
61
|
+
Datahunter.get_dataset sub_datasets[4]
|
|
61
62
|
break
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
end
|
|
67
|
-
if index > 2 or datasets[index+1].nil?
|
|
68
|
-
Datahunter.print_excuse_message
|
|
69
|
-
Datahunter.print_request_dataset_message
|
|
70
|
-
Datahunter.print_feedback_request
|
|
71
|
-
break
|
|
72
|
-
end
|
|
63
|
+
when 'abort'
|
|
64
|
+
break
|
|
65
|
+
else
|
|
66
|
+
next
|
|
73
67
|
end
|
|
74
68
|
end
|
|
75
69
|
when 204
|
|
76
|
-
|
|
77
|
-
"Remember, this is a first prototype, there will be a lot more "\
|
|
78
|
-
"datasets indexed soon. If you want us to find a dataset for you, or "\
|
|
79
|
-
"if you just want to give us a feedback, don't hesitate!".colorize(:red)
|
|
70
|
+
Datahunter.print_no_dataset_message
|
|
80
71
|
Datahunter.print_request_dataset_message
|
|
81
72
|
else
|
|
82
73
|
response.return!(request, result, &block)
|
|
@@ -86,55 +77,9 @@ command :find do |c|
|
|
|
86
77
|
end
|
|
87
78
|
end
|
|
88
79
|
|
|
89
|
-
command :search do |c|
|
|
90
|
-
c.syntax = 'hunter search keyword [spatial-coverage] [temporal-coverage]'
|
|
91
|
-
c.summary = '$ hunter search <keyword> <geo-coverage> <temporal-coverage>
|
|
92
|
-
* Returns all the datasets corresponding to
|
|
93
|
-
the query, sorted by popularity.
|
|
94
|
-
* WARNING: this command is going to be modified soon!'.colorize(:blue)
|
|
95
|
-
c.description = c.summary
|
|
96
|
-
c.example 'Search open data for population', 'hunter search population'
|
|
97
|
-
c.action do |args, options|
|
|
98
|
-
|
|
99
|
-
start_time = Time.now
|
|
100
|
-
|
|
101
|
-
url = Datahunter.datasets_url(args[0], args[1], args[2])
|
|
102
|
-
|
|
103
|
-
response = RestClient.get(url, :content_type => :json) do |response, request, result, &block|
|
|
104
|
-
|
|
105
|
-
elapsed_time = Time.now - start_time
|
|
106
|
-
|
|
107
|
-
case response.code
|
|
108
|
-
when 200
|
|
109
|
-
datasets = JSON.parse(response.body).reverse
|
|
110
|
-
puts "### Response in #{elapsed_time} seconds".colorize(:blue)
|
|
111
|
-
puts "### We've found #{datasets.size} datasets corresponding to your query #{args[0].downcase}:".colorize(:blue)
|
|
112
|
-
puts
|
|
113
|
-
datasets.each do |result|
|
|
114
|
-
puts ("- ".colorize(:yellow) +
|
|
115
|
-
"#{result["title"]}".colorize(:green) +
|
|
116
|
-
" id: ".colorize(:blue) +
|
|
117
|
-
"#{result["_id"]}")
|
|
118
|
-
end
|
|
119
|
-
puts
|
|
120
|
-
puts ("You can get more informations about a dataset with " +
|
|
121
|
-
"$ hunter info <id>".colorize(:blue))
|
|
122
|
-
puts ("You can download directly a dataset with " +
|
|
123
|
-
"$ hunter get <id>".colorize(:blue))
|
|
124
|
-
when 204
|
|
125
|
-
puts "We found nothing for your query: as every good hunter... we are still learning".colorize(:red)
|
|
126
|
-
Datahunter.print_excuse_message
|
|
127
|
-
Datahunter.print_feedback_request
|
|
128
|
-
else
|
|
129
|
-
response.return!(request, result, &block)
|
|
130
|
-
end
|
|
131
|
-
end
|
|
132
|
-
end
|
|
133
|
-
end
|
|
134
|
-
|
|
135
80
|
command :info do |c|
|
|
136
81
|
c.syntax = 'hunter info <id>'
|
|
137
|
-
c.summary = '
|
|
82
|
+
c.summary = 'Displays all the informations about a dataset given its id'.colorize(:blue)
|
|
138
83
|
c.description = c.summary
|
|
139
84
|
c.action do |args, options|
|
|
140
85
|
|
|
@@ -142,7 +87,6 @@ command :info do |c|
|
|
|
142
87
|
puts "usage: $ hunter info <id>"
|
|
143
88
|
else
|
|
144
89
|
url = Datahunter.ds_url(args[0])
|
|
145
|
-
|
|
146
90
|
response = RestClient.get(url, :content_type => :json) do |response, request, result, &block|
|
|
147
91
|
|
|
148
92
|
case response.code
|
|
@@ -160,8 +104,7 @@ end
|
|
|
160
104
|
|
|
161
105
|
command :get do |c|
|
|
162
106
|
c.syntax = 'hunter get <id>'
|
|
163
|
-
c.summary =
|
|
164
|
-
'Get a dataset (direct download if possible, web page if not) from its ID'.colorize(:blue)
|
|
107
|
+
c.summary = 'Downloads a dataset given its id'.colorize(:blue)
|
|
165
108
|
c.description = c.summary
|
|
166
109
|
c.action do |args, options|
|
|
167
110
|
|
|
@@ -169,18 +112,13 @@ command :get do |c|
|
|
|
169
112
|
puts "usage: $ hunter get id"
|
|
170
113
|
else
|
|
171
114
|
url = Datahunter.ds_url(args[0])
|
|
172
|
-
|
|
173
115
|
response = RestClient.get(url, :content_type => :json) do |response, request, result, &block|
|
|
174
|
-
|
|
116
|
+
|
|
175
117
|
case response.code
|
|
176
118
|
when 200
|
|
177
119
|
ds = JSON.parse(response.body)
|
|
178
120
|
puts
|
|
179
|
-
|
|
180
|
-
Datahunter.download_the_data ds
|
|
181
|
-
else
|
|
182
|
-
Datahunter.open_in_browser ds["uri"]
|
|
183
|
-
end
|
|
121
|
+
Datahunter.get_dataset ds
|
|
184
122
|
when 500
|
|
185
123
|
puts "Invalid ID".colorize(:red)
|
|
186
124
|
end
|
|
@@ -191,43 +129,27 @@ end
|
|
|
191
129
|
|
|
192
130
|
command :about do |c|
|
|
193
131
|
c.syntax = 'hunter about'
|
|
194
|
-
c.summary = '
|
|
195
|
-
* Vision
|
|
196
|
-
* Value Proposition
|
|
197
|
-
* Stats
|
|
198
|
-
* Last datasets indexed'.colorize(:blue)
|
|
132
|
+
c.summary = 'Gives Hunter\'s vision, the number of datasets indexed etc'.colorize(:blue)
|
|
199
133
|
c.description = c.summary
|
|
200
134
|
c.action do |args, options|
|
|
201
|
-
puts "
|
|
202
|
-
.....
|
|
203
|
-
|
|
204
|
-
...
|
|
205
|
-
..
|
|
206
|
-
.
|
|
207
|
-
|
|
208
|
-
8888888.......................888888888.
|
|
209
|
-
8888888
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
................
|
|
217
|
-
................8
|
|
218
|
-
|
|
219
|
-
.. ...........................8888......
|
|
220
|
-
... .........................888........
|
|
221
|
-
.... .....................:88..........
|
|
222
|
-
..... ...............8.............".colorize(:red)
|
|
223
|
-
puts
|
|
224
|
-
puts 'Find and download Open Datasets directly from the terminal!'.colorize(:red)
|
|
225
|
-
puts
|
|
226
|
-
puts 'We believe there will be more and more data available on the web and the main issue will be to find them and to manage each sources. So we try to do it for you: not by pre-processing them but by orienting you.'.colorize(:red)
|
|
227
|
-
puts
|
|
228
|
-
puts 'Number of datasets currently indexed: 8335'.colorize(:blue)
|
|
229
|
-
puts
|
|
230
|
-
puts 'Last dataset indexed: "Canada Open Data" "NETLs Energy Data eXchange", dati.gov.it'
|
|
135
|
+
puts "
|
|
136
|
+
Find and download Open Datasets ....8888888888888888888888888888888.....
|
|
137
|
+
directly from the terminal! ...888888888888888888888888888~88888?...
|
|
138
|
+
..88888...888888888888888888..+888888...
|
|
139
|
+
We believe there will be more .888888.=....................O78888888..
|
|
140
|
+
and more data available on the web ?888888.......................Z88888888.
|
|
141
|
+
and the main issue will be to find 8888888.......................888888888.
|
|
142
|
+
them and to manage each source. 8888888.......................888888888.
|
|
143
|
+
8888888:......8O.......88.....888888888.
|
|
144
|
+
88......ZO.....8=.....Z8.....8888888888.
|
|
145
|
+
$.........:8...............888888888888.
|
|
146
|
+
So we try to show you the way! ............8.............8888888888888.
|
|
147
|
+
..............O.........:88888888888888.
|
|
148
|
+
...............8........88888888888888..
|
|
149
|
+
................O......8.......888888...
|
|
150
|
+
................8.88.8.........=8887 ..
|
|
151
|
+
indexed datasets: 8336 . ................8888...........?8.....
|
|
152
|
+
last dataset: DAMIR french health .. ...........................8888......".colorize(:red)
|
|
231
153
|
puts
|
|
232
154
|
puts "### Hunter is still a prototype, please don't hesitate to help us make open datasets workflow better!".colorize(:blue)
|
|
233
155
|
Datahunter.print_feedback_request
|
|
@@ -235,21 +157,22 @@ $.........:8...............888888888888.
|
|
|
235
157
|
end
|
|
236
158
|
|
|
237
159
|
command :guide do |c|
|
|
238
|
-
c.action
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
puts ('$ hunter
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
'
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
160
|
+
c.action do
|
|
161
|
+
puts ('$ hunter find <keyword> <spatial coverage> <temporal coverage> '.colorize(:blue) + "\n" +
|
|
162
|
+
'Datasets corresponding to your query, sorted by popularity, '\
|
|
163
|
+
'displayed 5 by 5. Search ~> Find ~> Get the data!')
|
|
164
|
+
puts
|
|
165
|
+
puts ('$ hunter info <id>'.colorize(:blue) + "\n" +
|
|
166
|
+
'Display all the informations about a dataset given its id: '\
|
|
167
|
+
'complete description, publisher, last update, etc')
|
|
168
|
+
puts
|
|
169
|
+
puts ('$ hunter get <id>'.colorize(:blue) + "\n" +
|
|
170
|
+
'Get a dataset given its id: '\
|
|
171
|
+
'if possible download the data, if not, it opens the web page '\
|
|
172
|
+
'from where it will be possible')
|
|
173
|
+
puts
|
|
174
|
+
puts ('$ hunter about'.colorize(:blue) + "\n" +
|
|
175
|
+
'About Hunter, the vision, number of datasets indexed etc')
|
|
176
|
+
end
|
|
254
177
|
end
|
|
255
178
|
default_command :guide
|
data/datahunter.gemspec
CHANGED
|
@@ -24,11 +24,12 @@ Gem::Specification.new do |spec|
|
|
|
24
24
|
spec.add_development_dependency "rake", "~> 10.0"
|
|
25
25
|
spec.add_development_dependency "rspec"
|
|
26
26
|
spec.add_development_dependency "aruba-rspec"
|
|
27
|
-
|
|
27
|
+
|
|
28
28
|
spec.add_runtime_dependency "json", "~> 1.8.1"
|
|
29
|
-
spec.add_runtime_dependency "commander", "~> 4.
|
|
30
|
-
spec.add_runtime_dependency "rest-client", "~> 1.7.
|
|
29
|
+
spec.add_runtime_dependency "commander", "~> 4.3.0"
|
|
30
|
+
spec.add_runtime_dependency "rest-client", "~> 1.7.3"
|
|
31
31
|
spec.add_runtime_dependency "launchy", "~> 2.4.3"
|
|
32
|
-
spec.add_runtime_dependency "colorize", "~> 0.7.
|
|
32
|
+
spec.add_runtime_dependency "colorize", "~> 0.7.5"
|
|
33
33
|
spec.add_runtime_dependency "downloadr", "0.0.41"
|
|
34
|
+
spec.add_runtime_dependency "addressable", "~> 2.3.7"
|
|
34
35
|
end
|
data/lib/datahunter/base.rb
CHANGED
|
@@ -5,13 +5,12 @@ require 'addressable/uri'
|
|
|
5
5
|
|
|
6
6
|
module Datahunter
|
|
7
7
|
|
|
8
|
-
# DATASETS_URL = "http://localhost:3000/api/datasets/"
|
|
9
8
|
DATASETS_URL = "http://shrouded-harbor-5877.herokuapp.com/api/datasets/"
|
|
10
9
|
FEEDBACK_URL = "https://docs.google.com/forms/d/1yNzZjCCXvWHQCbWz4sx-nui3LafeeLcT7FF9T-vbKvw/viewform"
|
|
11
10
|
REQUEST_URL =
|
|
12
11
|
"https://docs.google.com/forms/d/1NRKWmb_mcpKJmrutXvZSZnysM_v0rfLhjD897H3Myrw/viewform?usp=send_form"
|
|
13
12
|
|
|
14
|
-
@extensions = ["json", "csv", "xml", "zip", "gz", "xls", "xlsx", "tsv"]
|
|
13
|
+
@extensions = ["json", "csv", "xml", "zip", "gz", "xls", "xlsx", "shp", "docx", "doc", "pdf", "txt", "tsv"]
|
|
15
14
|
|
|
16
15
|
def self.datasets_url tag, geo=nil, temp=nil
|
|
17
16
|
tag = tag.downcase.split.first if tag
|
|
@@ -33,17 +32,29 @@ module Datahunter
|
|
|
33
32
|
end
|
|
34
33
|
|
|
35
34
|
def self.print_dataset_info dataset
|
|
36
|
-
puts ("
|
|
37
|
-
puts ("
|
|
38
|
-
puts ("publisher: ".colorize(:green) + "#{dataset["publisher"]}")
|
|
39
|
-
puts ("temporal: ".colorize(:green) + "#{dataset["temporal"]}")
|
|
40
|
-
puts ("spatial: ".colorize(:green) + "#{dataset["spatial"]}")
|
|
41
|
-
puts ("created: ".colorize(:green) + "#{dataset["created"]}")
|
|
42
|
-
puts ("updated: ".colorize(:green) + "#{dataset["updated"]}")
|
|
43
|
-
puts ("score: ".colorize(:green) + "#{dataset["huntscore"]}")
|
|
35
|
+
puts ("#{dataset["title"]}".colorize(:green))
|
|
36
|
+
puts ("#{dataset["description"]}".colorize(:blue))
|
|
44
37
|
puts
|
|
38
|
+
puts ("publisher: ".colorize(:blue) + "#{dataset["publisher"]}")
|
|
39
|
+
puts ("temporal: ".colorize(:blue) + "#{dataset["temporal"]}")
|
|
40
|
+
puts ("spatial: ".colorize(:blue) + "#{dataset["spatial"]}")
|
|
41
|
+
puts ("created: ".colorize(:blue) + "#{dataset["created"]}")
|
|
42
|
+
puts ("updated: ".colorize(:blue) + "#{dataset["updated"]}")
|
|
43
|
+
puts ("score: ".colorize(:blue) + "#{dataset["huntscore"]}")
|
|
45
44
|
end
|
|
46
|
-
|
|
45
|
+
|
|
46
|
+
def self.print_coll_of_datasets_info_light coll_of_datasets
|
|
47
|
+
coll_of_datasets.each_with_index do |ds, index|
|
|
48
|
+
puts ("#{index+1}. ".colorize(:yellow) +
|
|
49
|
+
"#{ds["title"]}".colorize(:green) +
|
|
50
|
+
" id: ".colorize(:blue) +
|
|
51
|
+
"#{ds["_id"]}")
|
|
52
|
+
puts ("#{ds["spatial"].take(5)}")
|
|
53
|
+
puts ("#{ds["description"][0..100].gsub(/\w+\s*$/,'...')}".colorize(:blue))
|
|
54
|
+
end
|
|
55
|
+
puts
|
|
56
|
+
end
|
|
57
|
+
|
|
47
58
|
def self.print_downloadable_links resources
|
|
48
59
|
resources.each_with_index do |dl, i|
|
|
49
60
|
puts ("#{i}. ".colorize(:yellow) +
|
|
@@ -68,15 +79,15 @@ module Datahunter
|
|
|
68
79
|
file_name = uri.basename
|
|
69
80
|
loc = location + "/" + file_name
|
|
70
81
|
|
|
71
|
-
case ask ("Create/overwrite #{loc}?(y/rename/n)".colorize(:yellow))
|
|
82
|
+
case ask ("### Create/overwrite #{loc}?(y/rename/n)".colorize(:yellow))
|
|
72
83
|
when 'rename'
|
|
73
84
|
loc = ask "Path/to/filename: ".colorize(:yellow)
|
|
74
85
|
when 'n'
|
|
75
86
|
abort("Ok then")
|
|
76
87
|
end
|
|
77
|
-
puts "Start downloading..."
|
|
88
|
+
puts "### Start downloading..."
|
|
78
89
|
Downloadr::HTTP.download(url, loc)
|
|
79
|
-
puts "Your file has been downloaded ;)".colorize(:green)
|
|
90
|
+
puts "### Your file has been downloaded ;)".colorize(:green)
|
|
80
91
|
Datahunter.print_excuse_and_alternative_url_message alt_url
|
|
81
92
|
end
|
|
82
93
|
end
|
|
@@ -105,32 +116,27 @@ module Datahunter
|
|
|
105
116
|
end
|
|
106
117
|
end
|
|
107
118
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
case ask "### give feedback? (y/n)".colorize(:yellow)
|
|
112
|
-
when 'y'
|
|
113
|
-
Launchy.open(FEEDBACK_URL, options = {})
|
|
119
|
+
def self.get_dataset dataset
|
|
120
|
+
if dataset.has_key?("resources") and dataset["resources"].any?
|
|
121
|
+
Datahunter.download_the_data dataset
|
|
114
122
|
else
|
|
115
|
-
|
|
123
|
+
Datahunter.open_in_browser dataset["uri"]
|
|
116
124
|
end
|
|
117
125
|
end
|
|
118
126
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
end
|
|
124
|
-
end
|
|
125
|
-
|
|
126
|
-
def self.print_excuse_message
|
|
127
|
-
puts "Remember, this is a first prototype, there will surely be a lot more "\
|
|
127
|
+
## Messages
|
|
128
|
+
def self.print_no_dataset_message
|
|
129
|
+
puts "We've found nothing for your query. "\
|
|
130
|
+
"Remember, this is a first prototype, there will be a lot more "\
|
|
128
131
|
"datasets indexed soon. If you want us to find a dataset for you, or "\
|
|
129
132
|
"if you just want to give us a feedback, don't hesitate!".colorize(:red)
|
|
130
133
|
end
|
|
131
134
|
|
|
132
135
|
def self.print_excuse_and_alternative_url_message alt_url=""
|
|
133
|
-
puts "If this is not the file you expected, it's maybe because publisher
|
|
136
|
+
puts "If this is not the file you expected, it's maybe because publisher"\
|
|
137
|
+
" don't always keep the metadata up-to-date. We try to clean most of"\
|
|
138
|
+
" uri's and check the url. Anyway you may be able to download your"\
|
|
139
|
+
" file by hand here:"
|
|
134
140
|
puts "#{alt_url}".colorize(:blue)
|
|
135
141
|
end
|
|
136
142
|
|
|
@@ -138,4 +144,22 @@ module Datahunter
|
|
|
138
144
|
puts "The URL given by the publisher is not valid. We'll try to find out why "\
|
|
139
145
|
"as soon as we can!".colorize(:red)
|
|
140
146
|
end
|
|
147
|
+
|
|
148
|
+
|
|
149
|
+
## Feedback requests
|
|
150
|
+
def self.print_feedback_request
|
|
151
|
+
case ask "### give feedback? (y/n)".colorize(:yellow)
|
|
152
|
+
when 'y'
|
|
153
|
+
Launchy.open(FEEDBACK_URL, options = {})
|
|
154
|
+
else
|
|
155
|
+
puts "Bye for now!"
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def self.print_request_dataset_message
|
|
160
|
+
case ask "### request a dataset? (y/n)".colorize(:yellow)
|
|
161
|
+
when 'y'
|
|
162
|
+
Launchy.open(REQUEST_URL, options = {})
|
|
163
|
+
end
|
|
164
|
+
end
|
|
141
165
|
end
|
data/lib/datahunter/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: datahunter
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.3.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Terpo
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2015-
|
|
11
|
+
date: 2015-03-04 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: bundler
|
|
@@ -86,28 +86,28 @@ dependencies:
|
|
|
86
86
|
requirements:
|
|
87
87
|
- - "~>"
|
|
88
88
|
- !ruby/object:Gem::Version
|
|
89
|
-
version: 4.
|
|
89
|
+
version: 4.3.0
|
|
90
90
|
type: :runtime
|
|
91
91
|
prerelease: false
|
|
92
92
|
version_requirements: !ruby/object:Gem::Requirement
|
|
93
93
|
requirements:
|
|
94
94
|
- - "~>"
|
|
95
95
|
- !ruby/object:Gem::Version
|
|
96
|
-
version: 4.
|
|
96
|
+
version: 4.3.0
|
|
97
97
|
- !ruby/object:Gem::Dependency
|
|
98
98
|
name: rest-client
|
|
99
99
|
requirement: !ruby/object:Gem::Requirement
|
|
100
100
|
requirements:
|
|
101
101
|
- - "~>"
|
|
102
102
|
- !ruby/object:Gem::Version
|
|
103
|
-
version: 1.7.
|
|
103
|
+
version: 1.7.3
|
|
104
104
|
type: :runtime
|
|
105
105
|
prerelease: false
|
|
106
106
|
version_requirements: !ruby/object:Gem::Requirement
|
|
107
107
|
requirements:
|
|
108
108
|
- - "~>"
|
|
109
109
|
- !ruby/object:Gem::Version
|
|
110
|
-
version: 1.7.
|
|
110
|
+
version: 1.7.3
|
|
111
111
|
- !ruby/object:Gem::Dependency
|
|
112
112
|
name: launchy
|
|
113
113
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -128,14 +128,14 @@ dependencies:
|
|
|
128
128
|
requirements:
|
|
129
129
|
- - "~>"
|
|
130
130
|
- !ruby/object:Gem::Version
|
|
131
|
-
version: 0.7.
|
|
131
|
+
version: 0.7.5
|
|
132
132
|
type: :runtime
|
|
133
133
|
prerelease: false
|
|
134
134
|
version_requirements: !ruby/object:Gem::Requirement
|
|
135
135
|
requirements:
|
|
136
136
|
- - "~>"
|
|
137
137
|
- !ruby/object:Gem::Version
|
|
138
|
-
version: 0.7.
|
|
138
|
+
version: 0.7.5
|
|
139
139
|
- !ruby/object:Gem::Dependency
|
|
140
140
|
name: downloadr
|
|
141
141
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -150,6 +150,20 @@ dependencies:
|
|
|
150
150
|
- - '='
|
|
151
151
|
- !ruby/object:Gem::Version
|
|
152
152
|
version: 0.0.41
|
|
153
|
+
- !ruby/object:Gem::Dependency
|
|
154
|
+
name: addressable
|
|
155
|
+
requirement: !ruby/object:Gem::Requirement
|
|
156
|
+
requirements:
|
|
157
|
+
- - "~>"
|
|
158
|
+
- !ruby/object:Gem::Version
|
|
159
|
+
version: 2.3.7
|
|
160
|
+
type: :runtime
|
|
161
|
+
prerelease: false
|
|
162
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
163
|
+
requirements:
|
|
164
|
+
- - "~>"
|
|
165
|
+
- !ruby/object:Gem::Version
|
|
166
|
+
version: 2.3.7
|
|
153
167
|
description: Command line interface to find open datasets via the Hunter API
|
|
154
168
|
email:
|
|
155
169
|
- nicolas.terpolilli@gmail.com
|