buzzdata 0.0.1 → 0.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +47 -356
- data/lib/buzzdata.rb +8 -12
- data/lib/buzzdata/rest_helpers.rb +1 -0
- data/lib/buzzdata/version.rb +1 -1
- data/samples/upload_data.rb +6 -19
- data/spec/buzzdata_spec.rb +23 -7
- data/spec/fixtures/data.csv +2 -0
- data/spec/fixtures/invalid_yaml.yml +10 -1
- metadata +34 -40
- data/lib/buzzdata/upload.rb +0 -38
data/README.md
CHANGED
@@ -1,6 +1,37 @@
|
|
1
1
|
# BuzzData Ruby Client Library
|
2
2
|
|
3
|
-
|
3
|
+
The BuzzData Ruby Client Library is a simple wrapper for the BuzzData HTTP API that allows you to easily interact with datasets on BuzzData.
|
4
|
+
|
5
|
+
## Example
|
6
|
+
|
7
|
+
# Setup the API
|
8
|
+
require "rubygems"
|
9
|
+
require "buzzdata"
|
10
|
+
buzzdata = Buzzdata.new('YOUR_API_KEY')
|
11
|
+
|
12
|
+
# Create a Dataset
|
13
|
+
dataset = buzzdata.create_dataset(:username => 'eviltrout',
|
14
|
+
:name => "My Awesome Dataset!",
|
15
|
+
:public => false,
|
16
|
+
:readme => "This is my awesome dataset",
|
17
|
+
:license => 'cc0',
|
18
|
+
:topics => ['testing-buzzdata'])
|
19
|
+
|
20
|
+
# Upload some data
|
21
|
+
buzzdata.upload(dataset['id'], File.new('datasets/celebrities.csv'), 'My first dataset!')
|
22
|
+
|
23
|
+
## Installation
|
24
|
+
|
25
|
+
If you already have Ruby and RubyGems installed then installation is as simple as running the following command in a terminal:
|
26
|
+
|
27
|
+
>> gem install buzzdata
|
28
|
+
|
29
|
+
Mac OS X and most Unix/Linux distributions come with an installation of Ruby and RubyGems. If you do not have Ruby and RubyGems installed please [check the Ruby website for instructions](http://www.ruby-lang.org/).
|
30
|
+
|
31
|
+
|
32
|
+
## API Documentation
|
33
|
+
|
34
|
+
### Getting Started
|
4
35
|
|
5
36
|
Create an instance of the Buzzdata client:
|
6
37
|
|
@@ -11,13 +42,13 @@ To make it even simpler, if you create a file, `config/buzzdata.yml` with your `
|
|
11
42
|
>> buzzdata = Buzzdata.new
|
12
43
|
|
13
44
|
|
14
|
-
|
45
|
+
### Downloading Data
|
15
46
|
|
16
47
|
To download data from a dataset, just do this:
|
17
48
|
|
18
49
|
>> buzzdata.download_data 'eviltrout/b-list-celebrities'
|
19
50
|
|
20
|
-
|
51
|
+
### Dataset Information
|
21
52
|
|
22
53
|
Using `dataset_overview` you can get an overview of a Dataset's information. It returns a hash of attribute names and their values. See the *API Documentation* below for a list of the returned attributes.
|
23
54
|
|
@@ -25,7 +56,7 @@ Using `dataset_overview` you can get an overview of a Dataset's information. It
|
|
25
56
|
>> puts ds['name'] # outputs B-List Celebrities
|
26
57
|
|
27
58
|
|
28
|
-
|
59
|
+
### Listing Datasets
|
29
60
|
|
30
61
|
You can view a user's datasets by calling `datasets_list`. You'll get back an array with information on their datasets.
|
31
62
|
|
@@ -33,14 +64,14 @@ You can view a user's datasets by calling `datasets_list`. You'll get back an ar
|
|
33
64
|
>> datasets.each {|ds| puts ds['id'] }
|
34
65
|
|
35
66
|
|
36
|
-
|
67
|
+
### Dataset History
|
37
68
|
|
38
69
|
You can retrieve a list of uploaded versions of a dataset by calling `dataset_history`:
|
39
70
|
|
40
71
|
>> buzzdata.dataset_history.each {|v| puts "version #{v['version']}!" }
|
41
72
|
|
42
73
|
|
43
|
-
|
74
|
+
### Creating a Dataset
|
44
75
|
|
45
76
|
You can use the `create_dataset` method to create a new dataset. All fields are required:
|
46
77
|
|
@@ -53,389 +84,49 @@ You can use the `create_dataset` method to create a new dataset. All fields are
|
|
53
84
|
|
54
85
|
>> puts ds['id'] # outputs eviltrout/my-awesome-dataset
|
55
86
|
|
56
|
-
|
87
|
+
### Uploading Data
|
57
88
|
|
58
89
|
If your account has the ability to upload data to a dataset, you can do so like this:
|
59
90
|
|
60
|
-
>>
|
61
|
-
|
62
|
-
Uploads take some time to be processed. You can poll how the processing is going using `in_progress?`
|
63
|
-
|
64
|
-
>> upload.in_progress? # true - upload is going on
|
65
|
-
|
66
|
-
# (wait for some time to pass..)
|
91
|
+
>> buzzdata.upload('eviltrout/b-list-celebrities', File.new('datasets/celebrities.csv'), 'Release notes...')
|
67
92
|
|
68
|
-
>> upload.in_progress? # false - upload is done!
|
69
93
|
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
## Publish a dataset
|
94
|
+
### Publish a dataset
|
74
95
|
|
75
96
|
>> buzzdata.publish_dataset('eviltrout/b-list-celebrities')
|
76
97
|
|
77
98
|
|
78
|
-
|
99
|
+
### Clone another user's dataset
|
79
100
|
|
80
101
|
>> buzzdata.clone_dataset('pete/pete-forde-s-genome')
|
81
102
|
|
82
103
|
|
83
|
-
|
104
|
+
### Delete a dataset
|
84
105
|
|
85
106
|
>> buzzdata.delete_dataset('eviltrout/tasteless-dataset')
|
86
107
|
|
87
108
|
|
88
|
-
|
109
|
+
### Get a user's information
|
89
110
|
|
90
111
|
>> user = buzzdata.user_info('eviltrout')
|
91
112
|
>> puts user['name'] # Robin Ward
|
92
113
|
|
93
114
|
|
94
|
-
|
115
|
+
### Search BuzzData
|
95
116
|
|
96
117
|
>> buzzdata.search("pets").each do |r|
|
97
118
|
puts r['label'] # Outputs each search result label
|
98
119
|
end
|
99
120
|
|
100
121
|
|
101
|
-
|
122
|
+
### Get a list of usable Licenses
|
102
123
|
|
103
124
|
>> buzzdata.licenses
|
104
125
|
|
105
126
|
|
106
|
-
|
127
|
+
### Get a list of usable Topics
|
107
128
|
|
108
129
|
>> buzzdata.topics
|
109
130
|
|
110
131
|
|
111
|
-
|
112
|
-
|
113
|
-
The BuzzData API returns results in JSON.
|
114
|
-
|
115
|
-
You must attach your API key as either a query parameter or post variable in the form of `api_key=YOUR_KEY`.
|
116
|
-
|
117
|
-
For example, to test if your API key works, try this url:
|
118
|
-
|
119
|
-
https://buzzdata.com/api/test?api_key=YOUR_KEY
|
120
|
-
|
121
|
-
You should get back a JSON object with your username in it, confirming that the key is yours and has authenticated properly.
|
122
|
-
|
123
|
-
## Rate Limits
|
124
|
-
|
125
|
-
The API currently limits the requests you can make against it hourly. If you need more requests than that, please contact us and we'll let you know.
|
126
|
-
|
127
|
-
We have provided two response headers with each request to the API with Rate Limiting Information. They are returned from every API call.
|
128
|
-
|
129
|
-
X-RateLimit-Limit: 5000
|
130
|
-
X-RateLimit-Remaining: 4998
|
131
|
-
|
132
|
-
`X-RateLimit-Limit` is your current limit per hour. `X-RateLimit-Remaining` is how many requests you have left.
|
133
|
-
|
134
|
-
# Datasets
|
135
|
-
|
136
|
-
## Dataset Details (Overview)
|
137
|
-
|
138
|
-
To retrieve information about a dataset, simply make a GET:
|
139
|
-
|
140
|
-
**GET https://buzzdata.com/api/:username/:dataset**
|
141
|
-
|
142
|
-
**GET Parameters:**
|
143
|
-
|
144
|
-
* `api_key` = your API Key (optional - but necessary for viewing private or unpublished datasets.)
|
145
|
-
|
146
|
-
**Returns JSON:**
|
147
|
-
|
148
|
-
{"dataset":
|
149
|
-
{"id":"eviltrout/b-list-celebrities",
|
150
|
-
"username":"eviltrout",
|
151
|
-
"shortname":"b-list-celebrities",
|
152
|
-
"name":"B-List Celebrities",
|
153
|
-
"readme":"Here's a list of B-List Celebrities that I've curated.",
|
154
|
-
"public":true,
|
155
|
-
"license":"cc0",
|
156
|
-
"published":true,
|
157
|
-
"url":"http://buzzdata.com/eviltrout/b-list-celebrities",
|
158
|
-
"avatar":"/images/avatars/b9/e987d17045c649da4de2a580e8109d655e6a12?1312292315",
|
159
|
-
"followers_count":10,
|
160
|
-
"clones_count":2,
|
161
|
-
"articles_count":5,
|
162
|
-
"visualizations_count":10,
|
163
|
-
"attachments_count":2,
|
164
|
-
"created_at":"2011-07-12T14:31:21-04:00",
|
165
|
-
"data_updated_at":"2011-07-12T14:41:52-04:00"}}
|
166
|
-
|
167
|
-
|
168
|
-
## Listing Datasets
|
169
|
-
|
170
|
-
You can view a list of any user's datasets.
|
171
|
-
|
172
|
-
**GET https://buzzdata.com/api/:username/datasets/list**
|
173
|
-
|
174
|
-
**GET Parameters:**
|
175
|
-
|
176
|
-
* `api_key` = your API Key (optional - but necessary for viewing private or unpublished datasets.)
|
177
|
-
|
178
|
-
**Returns JSON:**
|
179
|
-
|
180
|
-
[
|
181
|
-
{"id":"eviltrout/b-list-celebrities",
|
182
|
-
"url":"http://buzzdata.com/eviltrout/b-list-celebrities",
|
183
|
-
"name":"B-List Celebrities",
|
184
|
-
"public":true,
|
185
|
-
"published":true,
|
186
|
-
"readme":"Here's a list of B-List Celebrities that I've curated."},
|
187
|
-
{"id":"eviltrout/pets",
|
188
|
-
"url":"http://buzzdata.com/eviltrout/pets",
|
189
|
-
"name":"Pets",
|
190
|
-
"public":true,
|
191
|
-
"published":true,
|
192
|
-
"readme":"A list of pets by owner."}
|
193
|
-
...
|
194
|
-
]
|
195
|
-
|
196
|
-
|
197
|
-
## Dataset History
|
198
|
-
|
199
|
-
To retrieve a list of uploads and versions to a dataset:
|
200
|
-
|
201
|
-
**GET https://buzzdata.com/api/:username/:dataset/history**
|
202
|
-
|
203
|
-
**GET Paramters: **
|
204
|
-
|
205
|
-
* `api_key` = your API Key (optional - but necessary for viewing private or unpublished datasets.)
|
206
|
-
|
207
|
-
**Returns JSON:**
|
208
|
-
|
209
|
-
[
|
210
|
-
{"version":1,"created_at":"2011-07-12T14:41:52-04:00","username":"eviltrout"},
|
211
|
-
{"version":2,"created_at":"2011-07-13T13:00:21-04:00","username":"eviltrout"}
|
212
|
-
]
|
213
|
-
|
214
|
-
|
215
|
-
## Downloading Data
|
216
|
-
|
217
|
-
Before you can download data, you need to create a `download_request`. If successful you will be given a
|
218
|
-
url to download the data from.
|
219
|
-
|
220
|
-
**POST https://buzzdata.com/api/:username/:dataset/download_request**
|
221
|
-
|
222
|
-
* `:username` is your username: ex: 'eviltrout'
|
223
|
-
* `:dataset` is the short name (url name) of the dataset you are uploading to. For example: 'b-list-celebrities'
|
224
|
-
|
225
|
-
**POST Parameters:**
|
226
|
-
|
227
|
-
* `api_key` = your API key
|
228
|
-
* `version` = the version of the dataset you wish to download
|
229
|
-
|
230
|
-
**Returns JSON:**
|
231
|
-
|
232
|
-
{download_request: {path:'PATH_TO_DOWNLOAD_YOUR_DATA'}}
|
233
|
-
|
234
|
-
You can then perform a GET to download the data from the path you are given.
|
235
|
-
|
236
|
-
|
237
|
-
## Creating a Dataset
|
238
|
-
|
239
|
-
Before you can upload data to a dataset, you need to create a dataset object in our system with meta-data about the dataset.
|
240
|
-
|
241
|
-
**POST https://buzzdata.com/api/:username/datasets**
|
242
|
-
|
243
|
-
* `:username` is your username: ex: 'eviltrout'
|
244
|
-
|
245
|
-
**POST Parameters:**
|
246
|
-
|
247
|
-
* `api_key` = your API Key
|
248
|
-
* `dataset[name]` = the name of the dataset
|
249
|
-
* `dataset[public]` = (true/false) whether the dataset is public or private
|
250
|
-
* `dataset[readme]` = the content for "About this Dataset"
|
251
|
-
* `dataset[license]` = the license the dataset is being offered with. See *Licenses* below
|
252
|
-
* `dataset[topics]` = the ids of the topics associated with this dataset. See *Topics* below
|
253
|
-
|
254
|
-
**Returns JSON:**
|
255
|
-
|
256
|
-
It returns the same output from the *Dataset Details (Overview)* above of the completed dataset, or an error message if the dataset couldn't be created.
|
257
|
-
|
258
|
-
|
259
|
-
## Upload Requests
|
260
|
-
|
261
|
-
To upload data to the system, you need to create an `upload_request`. An upload request tells you the HTTP end point your upload should be going to, as well as an `upload_code` you should pass along with your upload to verify it.
|
262
|
-
|
263
|
-
**POST https://buzzdata.com/api/:username/:dataset/upload_request**
|
264
|
-
|
265
|
-
* `:username` = your username: ex: 'eviltrout'
|
266
|
-
* `:dataset` = the short name (url name) of the dataset you are uploading to. For example: 'b-list-celebrities'
|
267
|
-
|
268
|
-
|
269
|
-
**POST Parameters:**
|
270
|
-
|
271
|
-
* `api_key` = your API key
|
272
|
-
|
273
|
-
**Returns JSON:**
|
274
|
-
|
275
|
-
{upload_request: {upload_code: 'SOME_CODE_HERE', url: 'URL_TO_UPLOAD_TO'} }
|
276
|
-
|
277
|
-
* `upload_code` = a unique token that authenticates your upload
|
278
|
-
* `url` = the endpoint for where the file should be uploaded
|
279
|
-
|
280
|
-
|
281
|
-
## Performing an Upload
|
282
|
-
|
283
|
-
After creating an `upload_request`, you can then POST your data file to be ingested. To do this, send a POST request to the `url` you received in your `upload_request` JSON.
|
284
|
-
|
285
|
-
*note: Make sure your POST is a multipart, otherwise the file upload will not work.*
|
286
|
-
|
287
|
-
**POST Parameters:**
|
288
|
-
|
289
|
-
* `api_key` = your API Key
|
290
|
-
* `upload_code` = the `upload_code` returned in the `upload_request`
|
291
|
-
* `file` = the file data you are uploading to be ingested
|
292
|
-
|
293
|
-
**Returns JSON:**
|
294
|
-
|
295
|
-
[{"name"=>"kittens_born.csv", "size"=>187, "job_status_token"=>"a24b2155-e2ec-48d4-8bc0-f77e3758966f"}]
|
296
|
-
|
297
|
-
* `name` = the filename of the upload.
|
298
|
-
* `size` = the size of the upload in bytes
|
299
|
-
* `job_status_token` = an important
|
300
|
-
|
301
|
-
|
302
|
-
## Checking your Upload Status
|
303
|
-
|
304
|
-
After a file has been uploaded, you can check out the upload's status by making a GET to:
|
305
|
-
|
306
|
-
**GET https://buzzdata.com/api/:username/:dataset/upload_request/status**
|
307
|
-
|
308
|
-
* `:username` = your username: ex: 'eviltrout'
|
309
|
-
* `:dataset` = the short name (url name) of the dataset you are uploading to. For example: 'b-list-celebrities'
|
310
|
-
|
311
|
-
**GET Parameters:**
|
312
|
-
|
313
|
-
* `api_key` = your API Key
|
314
|
-
* `job_status_token` = The job status token you received when you performed your upload.
|
315
|
-
|
316
|
-
**Returns JSON:**
|
317
|
-
|
318
|
-
{"message"=>"Ingest Job Created", "status_code"=>"created"}
|
319
|
-
|
320
|
-
* `message` is a textual description of the current status, or an error message in the event of an error
|
321
|
-
* `status_code` is the status of the current job. The job has finished when it is `complete` or `error`.
|
322
|
-
|
323
|
-
Important! You should wait a little while between polls to the job status. We recommend sleeping for one second in most cases.
|
324
|
-
|
325
|
-
Note: If you receive a status of 'Unknown' it means the file has not begun processing yet. If you continue to poll it will move to 'created'
|
326
|
-
|
327
|
-
|
328
|
-
## Publishing a Dataset
|
329
|
-
|
330
|
-
Once a dataset has an upload associated with it, you can publish it:
|
331
|
-
|
332
|
-
**POST https://buzzdata.com/api/:username/:dataset/publish**
|
333
|
-
|
334
|
-
**POST Parameters:**
|
335
|
-
|
336
|
-
* `api_key` = your API Key
|
337
|
-
|
338
|
-
**Returns JSON:**
|
339
|
-
|
340
|
-
It returns the same output from the *Dataset Details (Overview)* above of the completed dataset, or an error message if the dataset couldn't be published.
|
341
|
-
|
342
|
-
|
343
|
-
## Cloning a Dataset
|
344
|
-
|
345
|
-
You can clone another's dataset by making a post. Note the username in this case is the user whose dataset you want to clone, not your own:
|
346
|
-
|
347
|
-
**POST https://buzzdata.com/api/:username/:dataset/publish**
|
348
|
-
|
349
|
-
**POST Parameters:**
|
350
|
-
|
351
|
-
* `api_key` = your API Key
|
352
|
-
|
353
|
-
**Returns JSON:**
|
354
|
-
|
355
|
-
It returns the same output from the *Dataset Details (Overview)* above of the completed dataset, or an error message if the dataset couldn't be cloned.
|
356
|
-
|
357
|
-
|
358
|
-
## Delete a Dataset
|
359
|
-
|
360
|
-
To delete a dataset, make a DELETE call:
|
361
|
-
|
362
|
-
**DELETE https://buzzdata.com/api/:username/:dataset**
|
363
|
-
|
364
|
-
**POST Parameters:**
|
365
|
-
|
366
|
-
* `api_key` = your API Key
|
367
|
-
|
368
|
-
**Returns JSON:**
|
369
|
-
|
370
|
-
{"id"=>"eviltrout/tasteless-dataset", "deleted"=>true}
|
371
|
-
|
372
|
-
# Users
|
373
|
-
|
374
|
-
To retrieve information about a particular BuzzData user, perform the following GET:
|
375
|
-
|
376
|
-
**GET https://buzzdata.com/api/:username**
|
377
|
-
|
378
|
-
**Returns JSON:**
|
379
|
-
|
380
|
-
{"user":
|
381
|
-
{"id":"eviltrout",
|
382
|
-
"name":"Robin Ward",
|
383
|
-
"description":"The Evilest Trout of them all and BuzzData Developer",
|
384
|
-
"location":"Toronto, Canada",
|
385
|
-
"url":"http://buzzdata.com/eviltrout",
|
386
|
-
"avatar":"/images/avatars/b9/e987d17045c649da4de2a580e8109d655e6a12?1312292315",
|
387
|
-
"followers_count": 12}
|
388
|
-
}
|
389
|
-
|
390
|
-
|
391
|
-
# Searching BuzzData
|
392
|
-
|
393
|
-
To search BuzzData, make a GET call:
|
394
|
-
|
395
|
-
**GET https://buzzdata.com/api/search**
|
396
|
-
|
397
|
-
**GET Parameters:**
|
398
|
-
|
399
|
-
* `query` = the string you'd like to search for
|
400
|
-
|
401
|
-
**Returns JSON:**
|
402
|
-
|
403
|
-
[
|
404
|
-
{"label":"Pets","value":"Pets","id":"pets","url":"/eviltrout/pets","cloned":false,"type":"Dataset"},
|
405
|
-
{"label":"Business","value":"Business","id":"business","url":"/topics/business","type":"Topic"},
|
406
|
-
{"label":"Momoko Price","value":"Momoko Price","id":"momoko","url":"/momoko","type":"User","icon":"http://buzzdata.s3.amazonaws.com/avatars/fe/fe361ff01695aa4741840f4f8851a6da9e2ef64c"},
|
407
|
-
...
|
408
|
-
]
|
409
|
-
|
410
|
-
Note that while in the sample output above there is one of each Dataset, Topic and Users, a search can return many more. The type of result is based on the `type` attribute.
|
411
|
-
|
412
|
-
|
413
|
-
# Topics
|
414
|
-
|
415
|
-
When creating a dataset, it is necessary to supply at least once topic. To retrieve a list of topics and their ids:
|
416
|
-
|
417
|
-
**GET https://buzzdata.com/api/topics**
|
418
|
-
|
419
|
-
**Returns JSON:**
|
420
|
-
|
421
|
-
[
|
422
|
-
{"id":"agriculture","name":"Agriculture"},
|
423
|
-
{"id":"animals","name":"Animals"},
|
424
|
-
{"id":"anthropology","name":"Anthropology"},
|
425
|
-
...
|
426
|
-
]
|
427
|
-
|
428
|
-
# Licenses
|
429
|
-
|
430
|
-
When creating a dataset, it is necessary to supply a valid license for the data. You can query the available licenses by:
|
431
|
-
|
432
|
-
**GET https://buzzdata.com/api/licenses**
|
433
|
-
|
434
|
-
**Returns JSON:**
|
435
|
-
|
436
|
-
[
|
437
|
-
{"id":"cc0"},
|
438
|
-
{"id":"pdm"},
|
439
|
-
{"id":"cc_by"},
|
440
|
-
...
|
441
|
-
]
|
132
|
+
Copyright © 2011 [BuzzData](http://buzzdata.com/), released under the MIT license
|
data/lib/buzzdata.rb
CHANGED
@@ -4,9 +4,8 @@ require 'json'
|
|
4
4
|
require 'yaml'
|
5
5
|
|
6
6
|
# Our code
|
7
|
-
|
8
|
-
|
9
|
-
require_relative 'buzzdata/upload'
|
7
|
+
require './buzzdata/error'
|
8
|
+
require './buzzdata/rest_helpers'
|
10
9
|
|
11
10
|
class Buzzdata
|
12
11
|
YAML_ERRORS = [ArgumentError]
|
@@ -56,18 +55,15 @@ class Buzzdata
|
|
56
55
|
result['upload_request']
|
57
56
|
end
|
58
57
|
|
59
|
-
def
|
58
|
+
def upload(dataset, file, release_notes = "")
|
60
59
|
upload_request = new_upload_request(dataset)
|
61
60
|
|
62
61
|
# Prepare our request
|
63
62
|
post_url = upload_request.delete('url')
|
64
63
|
upload_request['file'] = file
|
64
|
+
upload_request['release_notes'] = release_notes
|
65
65
|
|
66
|
-
|
67
|
-
end
|
68
|
-
|
69
|
-
def upload_status(dataset, job_status_token)
|
70
|
-
get_json(url_for("#{dataset}/upload_request/status"), :job_status_token => job_status_token)
|
66
|
+
post_json(post_url, upload_request)
|
71
67
|
end
|
72
68
|
|
73
69
|
def download_path(dataset)
|
@@ -95,7 +91,7 @@ class Buzzdata
|
|
95
91
|
end
|
96
92
|
|
97
93
|
def delete_dataset(dataset)
|
98
|
-
delete_json(url_for(
|
94
|
+
delete_json(url_for(dataset))
|
99
95
|
end
|
100
96
|
|
101
97
|
def licenses
|
@@ -126,10 +122,10 @@ class Buzzdata
|
|
126
122
|
def create_dataset(attributes)
|
127
123
|
|
128
124
|
# Validate attributes
|
129
|
-
raise
|
125
|
+
raise Buzzdata::Error, "Missing attributes" if attributes.nil?
|
130
126
|
raise Buzzdata::Error, "Username is required" if param_blank?(attributes, :username)
|
131
127
|
raise Buzzdata::Error, "Dataset name is required" if param_blank?(attributes, :name)
|
132
|
-
raise Buzzdata::Error, "Dataset readme is required" if param_blank?(attributes, :
|
128
|
+
raise Buzzdata::Error, "Dataset readme is required" if param_blank?(attributes, :readme)
|
133
129
|
raise Buzzdata::Error, "Dataset license is required" if param_blank?(attributes, :license)
|
134
130
|
raise Buzzdata::Error, "Dataset topics are required" if param_blank?(attributes, :topics)
|
135
131
|
|
data/lib/buzzdata/version.rb
CHANGED
data/samples/upload_data.rb
CHANGED
@@ -2,30 +2,17 @@
|
|
2
2
|
|
3
3
|
require '../lib/buzzdata'
|
4
4
|
|
5
|
-
if ARGV.size !=
|
6
|
-
puts "Usage: ./upload_data.rb dataset filename"
|
7
|
-
puts "Example: ./upload_data.rb eviltrout/kittens-born-by-month kittens_born.csv"
|
5
|
+
if ARGV.size != 3
|
6
|
+
puts "Usage: ./upload_data.rb dataset filename 'Change notes...'"
|
7
|
+
puts "Example: ./upload_data.rb eviltrout/kittens-born-by-month kittens_born.csv 'Added more kittens'"
|
8
8
|
exit(0)
|
9
9
|
end
|
10
10
|
|
11
11
|
buzzdata = Buzzdata.new
|
12
12
|
|
13
|
-
dataset_name, filename = *ARGV
|
13
|
+
dataset_name, filename, release_notes = *ARGV
|
14
14
|
|
15
15
|
# Upload a file to a dataset
|
16
16
|
print "Uploading #{filename}..."
|
17
|
-
upload = buzzdata.
|
18
|
-
puts "Done!"
|
19
|
-
|
20
|
-
# Wait while it's being processed
|
21
|
-
print "Waiting for processing to finish..."
|
22
|
-
while upload.in_progress?
|
23
|
-
print "."
|
24
|
-
sleep(1) # Let's not poll too frequently
|
25
|
-
end
|
26
|
-
|
27
|
-
if upload.success?
|
28
|
-
puts "Done!"
|
29
|
-
else
|
30
|
-
puts "ERROR! #{upload.status_message}"
|
31
|
-
end
|
17
|
+
upload = buzzdata.upload(dataset_name, File.new(filename), release_notes)
|
18
|
+
puts "Done!"
|
data/spec/buzzdata_spec.rb
CHANGED
@@ -1,19 +1,36 @@
|
|
1
1
|
require File.expand_path(File.dirname(__FILE__) + '/spec_helper')
|
2
2
|
|
3
|
+
require 'time'
|
4
|
+
require 'securerandom'
|
5
|
+
|
3
6
|
class Buzzdata
|
4
7
|
describe Buzzdata do
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
8
|
+
GENERIC_USER = 'eviltrout'
|
9
|
+
NONEXISTENT_USER = 'missing'
|
10
|
+
USER_WITH_PRIVATE_DATASET = 'jpmckinney'
|
11
|
+
PUBLISH_SLEEP_INTERVAL = 5
|
12
|
+
|
13
|
+
CLONABLE_DATASET = 'eviltrout/pets'
|
14
|
+
NONEXISTENT_DATASET = 'missing/missing'
|
15
|
+
PRIVATE_DATASET_BELONGING_TO_ANOTHER_USER = 'jpmckinney/private'
|
16
|
+
UNPUBLISHED_DATASET_BELONGING_TO_ANOTHER_USER = 'jpmckinney/unpublished'
|
17
|
+
|
18
|
+
def fixture_path(basename)
|
19
|
+
File.expand_path File.dirname(__FILE__) + '/fixtures/' + basename
|
20
|
+
end
|
9
21
|
|
22
|
+
describe '#initialize' do
|
10
23
|
it 'should use custom configuration file' do
|
11
24
|
client = Buzzdata.new nil, :config_file => fixture_path('custom.yml')
|
12
25
|
client.instance_variable_get('@api_key').should == 'dummy'
|
13
26
|
end
|
14
27
|
|
15
|
-
it "should not raise an error if
|
16
|
-
expect{Buzzdata.new
|
28
|
+
it "should not raise an error if an API key is provided" do
|
29
|
+
expect{Buzzdata.new 'dummy'}.not_to raise_error(Buzzdata::Error)
|
30
|
+
end
|
31
|
+
|
32
|
+
it "should raise an error if the configuration file is default and missing" do
|
33
|
+
expect{Buzzdata.new}.to raise_error(Buzzdata::Error, /No API key provided/)
|
17
34
|
end
|
18
35
|
|
19
36
|
it "should raise an error if the configuration file is custom and missing" do
|
@@ -35,7 +52,6 @@ class Buzzdata
|
|
35
52
|
|
36
53
|
it "should raise an error if the API key is missing from the configuration file" do
|
37
54
|
expect{Buzzdata.new nil, :config_file => fixture_path('missing_api_key.yml')}.to raise_error(Buzzdata::Error, /API key missing/)
|
38
|
-
|
39
55
|
end
|
40
56
|
end
|
41
57
|
end
|
metadata
CHANGED
@@ -1,50 +1,46 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: buzzdata
|
3
|
-
version: !ruby/object:Gem::Version
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.2
|
4
5
|
prerelease:
|
5
|
-
version: 0.0.1
|
6
6
|
platform: ruby
|
7
|
-
authors:
|
7
|
+
authors:
|
8
8
|
- BuzzData
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
|
13
|
-
date: 2011-11-22 00:00:00 -05:00
|
12
|
+
date: 2011-12-05 00:00:00.000000000 -05:00
|
14
13
|
default_executable:
|
15
|
-
dependencies:
|
16
|
-
- !ruby/object:Gem::Dependency
|
14
|
+
dependencies:
|
15
|
+
- !ruby/object:Gem::Dependency
|
17
16
|
name: rest-client
|
18
|
-
|
19
|
-
requirement: &id001 !ruby/object:Gem::Requirement
|
17
|
+
requirement: &2169622940 !ruby/object:Gem::Requirement
|
20
18
|
none: false
|
21
|
-
requirements:
|
19
|
+
requirements:
|
22
20
|
- - ~>
|
23
|
-
- !ruby/object:Gem::Version
|
21
|
+
- !ruby/object:Gem::Version
|
24
22
|
version: 1.6.7
|
25
23
|
type: :runtime
|
26
|
-
version_requirements: *id001
|
27
|
-
- !ruby/object:Gem::Dependency
|
28
|
-
name: rspec
|
29
24
|
prerelease: false
|
30
|
-
|
25
|
+
version_requirements: *2169622940
|
26
|
+
- !ruby/object:Gem::Dependency
|
27
|
+
name: rspec
|
28
|
+
requirement: &2169622200 !ruby/object:Gem::Requirement
|
31
29
|
none: false
|
32
|
-
requirements:
|
30
|
+
requirements:
|
33
31
|
- - ~>
|
34
|
-
- !ruby/object:Gem::Version
|
32
|
+
- !ruby/object:Gem::Version
|
35
33
|
version: 2.6.0
|
36
34
|
type: :development
|
37
|
-
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: *2169622200
|
38
37
|
description:
|
39
|
-
email:
|
38
|
+
email:
|
40
39
|
- support@buzzdata.com
|
41
40
|
executables: []
|
42
|
-
|
43
41
|
extensions: []
|
44
|
-
|
45
42
|
extra_rdoc_files: []
|
46
|
-
|
47
|
-
files:
|
43
|
+
files:
|
48
44
|
- .gitignore
|
49
45
|
- Gemfile
|
50
46
|
- LICENCE
|
@@ -54,7 +50,6 @@ files:
|
|
54
50
|
- lib/buzzdata.rb
|
55
51
|
- lib/buzzdata/error.rb
|
56
52
|
- lib/buzzdata/rest_helpers.rb
|
57
|
-
- lib/buzzdata/upload.rb
|
58
53
|
- lib/buzzdata/version.rb
|
59
54
|
- samples/config/buzzdata.yml.sample
|
60
55
|
- samples/dataset_overview.rb
|
@@ -63,6 +58,7 @@ files:
|
|
63
58
|
- samples/upload_data.rb
|
64
59
|
- spec/buzzdata_spec.rb
|
65
60
|
- spec/fixtures/custom.yml
|
61
|
+
- spec/fixtures/data.csv
|
66
62
|
- spec/fixtures/invalid_yaml.yml
|
67
63
|
- spec/fixtures/missing_api_key.yml
|
68
64
|
- spec/fixtures/not_a_hash.yml
|
@@ -71,34 +67,32 @@ files:
|
|
71
67
|
has_rdoc: true
|
72
68
|
homepage: http://buzzdata.com/
|
73
69
|
licenses: []
|
74
|
-
|
75
70
|
post_install_message:
|
76
71
|
rdoc_options: []
|
77
|
-
|
78
|
-
require_paths:
|
72
|
+
require_paths:
|
79
73
|
- lib
|
80
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
74
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
81
75
|
none: false
|
82
|
-
requirements:
|
83
|
-
- -
|
84
|
-
- !ruby/object:Gem::Version
|
85
|
-
version:
|
86
|
-
required_rubygems_version: !ruby/object:Gem::Requirement
|
76
|
+
requirements:
|
77
|
+
- - ! '>='
|
78
|
+
- !ruby/object:Gem::Version
|
79
|
+
version: '0'
|
80
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
87
81
|
none: false
|
88
|
-
requirements:
|
89
|
-
- -
|
90
|
-
- !ruby/object:Gem::Version
|
91
|
-
version:
|
82
|
+
requirements:
|
83
|
+
- - ! '>='
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '0'
|
92
86
|
requirements: []
|
93
|
-
|
94
87
|
rubyforge_project: buzzdata
|
95
88
|
rubygems_version: 1.6.2
|
96
89
|
signing_key:
|
97
90
|
specification_version: 3
|
98
91
|
summary: Ruby client for the BuzzData API
|
99
|
-
test_files:
|
92
|
+
test_files:
|
100
93
|
- spec/buzzdata_spec.rb
|
101
94
|
- spec/fixtures/custom.yml
|
95
|
+
- spec/fixtures/data.csv
|
102
96
|
- spec/fixtures/invalid_yaml.yml
|
103
97
|
- spec/fixtures/missing_api_key.yml
|
104
98
|
- spec/fixtures/not_a_hash.yml
|
data/lib/buzzdata/upload.rb
DELETED
@@ -1,38 +0,0 @@
|
|
1
|
-
class Buzzdata
|
2
|
-
class Upload
|
3
|
-
|
4
|
-
attr_reader :filename, :size, :job_status_token, :dataset
|
5
|
-
|
6
|
-
def initialize(buzzdata_api, dataset, upload_response)
|
7
|
-
@api = buzzdata_api
|
8
|
-
@dataset = dataset
|
9
|
-
@filename = upload_response[0]['name']
|
10
|
-
@size = upload_response[0]['size']
|
11
|
-
@job_status_token = upload_response[0]['job_status_token']
|
12
|
-
end
|
13
|
-
|
14
|
-
def in_progress?
|
15
|
-
!is_complete?(current_status)
|
16
|
-
end
|
17
|
-
|
18
|
-
def success?
|
19
|
-
(current_status['status_code'] == "complete")
|
20
|
-
end
|
21
|
-
|
22
|
-
def status_message
|
23
|
-
current_status['message']
|
24
|
-
end
|
25
|
-
|
26
|
-
private
|
27
|
-
|
28
|
-
def is_complete?(status)
|
29
|
-
return false if status.nil?
|
30
|
-
['complete', 'error'].include?(status['status_code'])
|
31
|
-
end
|
32
|
-
|
33
|
-
def current_status
|
34
|
-
return @current_status if is_complete?(@current_status)
|
35
|
-
@current_status = @api.upload_status(@dataset, @job_status_token)
|
36
|
-
end
|
37
|
-
end
|
38
|
-
end
|