cid 0.2.1 → 0.2.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,15 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 5dd888724adb78e68f3a6760d4cb64317ca5e3e2
4
- data.tar.gz: 610b3445181b32b18b0007f709e77185abf297b5
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ ZmU0ZTUwYjBlNGY2M2M1YTMwYzllNjhlMWJhZjE3OWZjMmM4ZTVlMw==
5
+ data.tar.gz: !binary |-
6
+ ZGUxYzBmODUyMWQ1MjRkYjM1YzNhMzNjNzFiNjA1M2Y4YjFkMDkyZA==
5
7
  SHA512:
6
- metadata.gz: f599d64d955fb3cc132bfc618210041f94ab421e893364ffb7f87612c6ec280a3f1aa2b978c88f94a0ef87b129ae8a0cddd512df1d67d554934465d58029d886
7
- data.tar.gz: 40b378091289e6a7d048a1afd7a1b66d36e4ed6fd6d7b7e85ef42974bb4919c6d0d3b62b925cd5e7435cf1aee5e3db60a6b135e21fd9ee5c057ad7d6a332fa82
8
+ metadata.gz: !binary |-
9
+ YTg5MGExZDBjZDdiYzc2NTI5MDZkOWJiMTNjMDI3ZTFlZDAxNzI1MTc3MGVj
10
+ YjIwOTQwNWIwYTU0ZjRiYzk5N2NhMDFhNDMwNmRmYWJhODRiMDNlNjNiODEy
11
+ NGFlNmU2NzExMzM3MmEzNjJkNTU2NGZjMDYyNWJkYmRkYmFiMmY=
12
+ data.tar.gz: !binary |-
13
+ ZTJhZmEyYTFjN2M2MGQ5NWY3YzhhMDVlN2M0MDdhODQwOGVlOGY0NzFmZGU2
14
+ N2RmZDdlOTU5M2NiOTYyZmE5MWFhZGQ1MjEzNzc4ZGMzYzY0NjdiZGVmNGI5
15
+ YmUyZDk4ZDg5MDk2YWE3YjBhNmNhNzFkZjdiNzMzYTJhNmNlYWM=
data/.travis.yml CHANGED
@@ -11,3 +11,11 @@ notifications:
11
11
  env:
12
12
  global:
13
13
  - secure: KTQ83GtNJyXwTUzYKbDtX1FzA3hsBSHefOqYsZ2KzOnj1dPZpGoeb4gcjWHXvNHqmR0mToT8U7KNq8AYWwHQLYCKf2+jRayd9UAzezhcs5u0SVfE5gmZLho57TW3AQvHI/uVQi9yg9/N3uDICqBplHJ7EMeMusUBc6ACF8FRmtY=
14
+ deploy:
15
+ provider: rubygems
16
+ api_key:
17
+ secure: eMaF/ItvetCL32WBFqpKxFbOO6uUXDbwIPDFYvwmtIE4q3khk7uiCwMVgw56t/GNyFKdaNkKlsrqcKgM0W3AmzHcA/gWJM0A69g6MtHswWkiiUzv6rPbZIAG99WTthTfs6RuvqXNuY4qaitO6IfrtZq5lXrlFKJgd/G7i7uZuDU=
18
+ gem: cid
19
+ on:
20
+ tags: true
21
+ repo: theodi/cid
data/README.md CHANGED
@@ -1,6 +1,22 @@
1
+ [![Build Status](https://travis-ci.org/theodi/cid.svg)](https://travis-ci.org/theodi/cid)
2
+ [![Coverage Status](http://img.shields.io/coveralls/theodi/cid.svg)](https://coveralls.io/r/theodi/cid)
3
+ [![Code Climate](http://img.shields.io/codeclimate/github/theodi/cid.svg)](https://codeclimate.com/github/theodi/cid)
4
+ [![Gem Version](http://img.shields.io/gem/v/cid.svg)](https://rubygems.org/gems/cid)
5
+ [![License](http://img.shields.io/:license-mit-blue.svg)](http://theodi.mit-license.org)
6
+ [![Badges](http://img.shields.io/:badges-6/6-ff6799.svg)](https://github.com/badges/badgerbadgerbadger)
7
+
1
8
  # Cid
2
9
 
3
- TODO: Write a gem description
10
+ Cid is **C**ontinuous **I**ntegraton for **D**ata.
11
+
12
+ There are two facets to Cid:
13
+
14
+ The first takes a [Datapackage](http://dataprotocols.org/data-packages/) with CSVs
15
+ in a Github repo and validates each one against a schema (in [JSON table schema](http://dataprotocols.org/json-table-schema/) format) which lives in the same folder.
16
+
17
+ The second updates the datapackage.json and pushes the changes to Github.
18
+
19
+ With these two things in place, we can continuously integrate our data!
4
20
 
5
21
  ## Installation
6
22
 
@@ -18,7 +34,65 @@ Or install it yourself as:
18
34
 
19
35
  ## Usage
20
36
 
21
- TODO: Write usage instructions here
37
+ Currently, Cid assumes your Datapackage lives in a cloned Github repo and has a structure like this:
38
+
39
+ ```
40
+ datapackage.json
41
+ - folder
42
+ |
43
+ |_ schema.json
44
+ |_ file.csv
45
+ |_ ..
46
+ ```
47
+
48
+ The schema.json file must match the [JSON table schema](http://dataprotocols.org/json-table-schema/).
49
+
50
+ You can have any number of folders, and each schema must correspond to the files within it.
51
+
52
+ When you run `cid validate` on the command line, Cid loops through each folder and validates each csv against the schema. It also checks for any common errors like ragged rows and missing characters using [csvlint](https://github.com/theodi/csvlint.rb).
53
+
54
+ When you run `cid publish` on the command line, Cid (again), loops through each folder and adds each csv to the `datapackage.json`. It then pushes the changes to the GitHub repo. For this to happen sucessfully, you must have a Github API key, which you can specify either as an environment variable like so:
55
+
56
+ export GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE"
57
+
58
+ Or as a command line option like so:
59
+
60
+ cid publish --github-token=YOUR_TOKEN_HERE
61
+
62
+ If you just want to skip the GitHub push altogether, just run
63
+
64
+ cid publish --skip-github
65
+
66
+ ## Getting this in Travis
67
+
68
+ Obviously, Cid is at its most powerful when used in a CI build. To get Cid working in Travis, simply add a `.travis.yml` file to your repo that looks a bit like this:
69
+
70
+ before_script:
71
+ - gem install cid
72
+ script:
73
+ - cid validate
74
+ after_success:
75
+ - '[ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && cid publish'
76
+
77
+ This installs cid, runs the validation, and then (if the branch is master) generates a new `datapackage.json` and pushes it to github.
78
+
79
+ You'll also need to add your Github token to the `.travis.yml` file. To do this, just run:
80
+
81
+ gem install travis
82
+ travis encrypt GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE" --add
83
+
84
+ Then just add your repo to Travis, and push your changes.
85
+
86
+ Now whenever someone makes a pull request on your data, Cid will validate the data against the schema, and you'll get a nice build status telling you if it's good to go!
87
+
88
+ ## Examples
89
+
90
+ There's only two repos that use Cid so far (it is a new thing after all), and you can see it here:
91
+
92
+ * https://github.com/theodi/euro-elections
93
+ * https://github.com/DemocracyClub/ge2015-candidates
94
+
95
+ If you decide to start using Cid, and want it listing here, open a PR and add it to this list!
22
96
 
23
97
  ## Contributing
24
98
 
data/bin/cid CHANGED
@@ -19,10 +19,13 @@ class CidBin < Thor
19
19
 
20
20
  > $ cid validate /this/is/a/datapackage
21
21
  LONGDESC
22
+ option :ignore, :type => :array, :desc => "folder or folders to ignore (relative to the package root)"
22
23
  def validate(source = ".")
23
24
  check_path(source)
24
25
 
25
- validation = Cid::Validation.validate(source)
26
+ ignore = options[:ignore] || []
27
+
28
+ validation = Cid::Validation.validate(source, ignore)
26
29
 
27
30
  code = 0
28
31
 
@@ -33,3 +33,11 @@ Feature: Validate a datapackage
33
33
  When I run `cid validate /this/is/fake`
34
34
  Then the output should contain "Path doesn't exist!"
35
35
  And the exit status should be 1
36
+
37
+ Scenario: Ignore folders
38
+ When I cd to "spec/fixtures/multiple_folders"
39
+ And I run `cid validate --ignore votes`
40
+ Then the output should contain "seats/seats-1.csv is VALID"
41
+ And the output should contain "seats/seats-2.csv is VALID"
42
+ And the output should not contain "votes/votes-1.csv is VALID"
43
+ And the output should not contain "votes/votes-2.csv is VALID"
@@ -4,12 +4,14 @@ module Cid
4
4
 
5
5
  class Validation
6
6
 
7
- def self.validate(path)
7
+ def self.validate(path, ignore = [])
8
8
  result = {}
9
9
 
10
10
  paths = Dir.glob("#{path}/**")
11
11
 
12
12
  paths.each do |path|
13
+ next if ignore.include?(path.split("/").last)
14
+
13
15
  begin
14
16
  schema = Csvlint::Schema.load_from_json_table(File.new(Dir["#{path}/schema.json"][0]))
15
17
  rescue
data/lib/cid/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Cid
2
- VERSION = "0.2.1"
2
+ VERSION = "0.2.2"
3
3
  end
@@ -37,6 +37,19 @@ describe Cid::Validation do
37
37
  validation["seats/seats-2.csv"][:warnings].should == []
38
38
  end
39
39
 
40
+ it "ignores a folder if `ignore` is set" do
41
+ Csvlint::Validator.should_receive(:new).exactly(2).times.and_call_original
42
+
43
+ validation = Cid::Validation.validate(File.join(File.dirname(__FILE__), 'fixtures', 'multiple_folders'), ['seats'])
44
+
45
+ validation["votes/votes-1.csv"][:errors].should == []
46
+ validation["votes/votes-1.csv"][:warnings].should == []
47
+ validation["votes/votes-2.csv"][:errors].should == []
48
+ validation["votes/votes-2.csv"][:warnings].should == []
49
+ validation["seats/seats-1.csv"].should be_nil
50
+ validation["seats/seats-2.csv"].should be_nil
51
+ end
52
+
40
53
  it "returns errors for an invalid csv" do
41
54
  validation = Cid::Validation.validate(File.join(File.dirname(__FILE__), 'fixtures', 'invalid'))
42
55
 
metadata CHANGED
@@ -1,125 +1,125 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cid
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.2.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - pezholio
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-05-23 00:00:00.000000000 Z
11
+ date: 2014-06-24 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: csvlint
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - '>='
17
+ - - ! '>='
18
18
  - !ruby/object:Gem::Version
19
19
  version: '0'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - '>='
24
+ - - ! '>='
25
25
  - !ruby/object:Gem::Version
26
26
  version: '0'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: colorize
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - '>='
31
+ - - ! '>='
32
32
  - !ruby/object:Gem::Version
33
33
  version: '0'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - '>='
38
+ - - ! '>='
39
39
  - !ruby/object:Gem::Version
40
40
  version: '0'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: json
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
- - - '>='
45
+ - - ! '>='
46
46
  - !ruby/object:Gem::Version
47
47
  version: '0'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
- - - '>='
52
+ - - ! '>='
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0'
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: git
57
57
  requirement: !ruby/object:Gem::Requirement
58
58
  requirements:
59
- - - '>='
59
+ - - ! '>='
60
60
  - !ruby/object:Gem::Version
61
61
  version: '0'
62
62
  type: :runtime
63
63
  prerelease: false
64
64
  version_requirements: !ruby/object:Gem::Requirement
65
65
  requirements:
66
- - - '>='
66
+ - - ! '>='
67
67
  - !ruby/object:Gem::Version
68
68
  version: '0'
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: github_api
71
71
  requirement: !ruby/object:Gem::Requirement
72
72
  requirements:
73
- - - '>='
73
+ - - ! '>='
74
74
  - !ruby/object:Gem::Version
75
75
  version: '0'
76
76
  type: :runtime
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
- - - '>='
80
+ - - ! '>='
81
81
  - !ruby/object:Gem::Version
82
82
  version: '0'
83
83
  - !ruby/object:Gem::Dependency
84
84
  name: dotenv
85
85
  requirement: !ruby/object:Gem::Requirement
86
86
  requirements:
87
- - - '>='
87
+ - - ! '>='
88
88
  - !ruby/object:Gem::Version
89
89
  version: '0'
90
90
  type: :runtime
91
91
  prerelease: false
92
92
  version_requirements: !ruby/object:Gem::Requirement
93
93
  requirements:
94
- - - '>='
94
+ - - ! '>='
95
95
  - !ruby/object:Gem::Version
96
96
  version: '0'
97
97
  - !ruby/object:Gem::Dependency
98
98
  name: memoist
99
99
  requirement: !ruby/object:Gem::Requirement
100
100
  requirements:
101
- - - '>='
101
+ - - ! '>='
102
102
  - !ruby/object:Gem::Version
103
103
  version: '0'
104
104
  type: :runtime
105
105
  prerelease: false
106
106
  version_requirements: !ruby/object:Gem::Requirement
107
107
  requirements:
108
- - - '>='
108
+ - - ! '>='
109
109
  - !ruby/object:Gem::Version
110
110
  version: '0'
111
111
  - !ruby/object:Gem::Dependency
112
112
  name: thor
113
113
  requirement: !ruby/object:Gem::Requirement
114
114
  requirements:
115
- - - '>='
115
+ - - ! '>='
116
116
  - !ruby/object:Gem::Version
117
117
  version: '0'
118
118
  type: :runtime
119
119
  prerelease: false
120
120
  version_requirements: !ruby/object:Gem::Requirement
121
121
  requirements:
122
- - - '>='
122
+ - - ! '>='
123
123
  - !ruby/object:Gem::Version
124
124
  version: '0'
125
125
  - !ruby/object:Gem::Dependency
@@ -140,140 +140,140 @@ dependencies:
140
140
  name: rake
141
141
  requirement: !ruby/object:Gem::Requirement
142
142
  requirements:
143
- - - '>='
143
+ - - ! '>='
144
144
  - !ruby/object:Gem::Version
145
145
  version: '0'
146
146
  type: :development
147
147
  prerelease: false
148
148
  version_requirements: !ruby/object:Gem::Requirement
149
149
  requirements:
150
- - - '>='
150
+ - - ! '>='
151
151
  - !ruby/object:Gem::Version
152
152
  version: '0'
153
153
  - !ruby/object:Gem::Dependency
154
154
  name: rspec
155
155
  requirement: !ruby/object:Gem::Requirement
156
156
  requirements:
157
- - - '>='
157
+ - - ! '>='
158
158
  - !ruby/object:Gem::Version
159
159
  version: '0'
160
160
  type: :development
161
161
  prerelease: false
162
162
  version_requirements: !ruby/object:Gem::Requirement
163
163
  requirements:
164
- - - '>='
164
+ - - ! '>='
165
165
  - !ruby/object:Gem::Version
166
166
  version: '0'
167
167
  - !ruby/object:Gem::Dependency
168
168
  name: pry
169
169
  requirement: !ruby/object:Gem::Requirement
170
170
  requirements:
171
- - - '>='
171
+ - - ! '>='
172
172
  - !ruby/object:Gem::Version
173
173
  version: '0'
174
174
  type: :development
175
175
  prerelease: false
176
176
  version_requirements: !ruby/object:Gem::Requirement
177
177
  requirements:
178
- - - '>='
178
+ - - ! '>='
179
179
  - !ruby/object:Gem::Version
180
180
  version: '0'
181
181
  - !ruby/object:Gem::Dependency
182
182
  name: pry-remote
183
183
  requirement: !ruby/object:Gem::Requirement
184
184
  requirements:
185
- - - '>='
185
+ - - ! '>='
186
186
  - !ruby/object:Gem::Version
187
187
  version: '0'
188
188
  type: :development
189
189
  prerelease: false
190
190
  version_requirements: !ruby/object:Gem::Requirement
191
191
  requirements:
192
- - - '>='
192
+ - - ! '>='
193
193
  - !ruby/object:Gem::Version
194
194
  version: '0'
195
195
  - !ruby/object:Gem::Dependency
196
196
  name: coveralls
197
197
  requirement: !ruby/object:Gem::Requirement
198
198
  requirements:
199
- - - '>='
199
+ - - ! '>='
200
200
  - !ruby/object:Gem::Version
201
201
  version: '0'
202
202
  type: :development
203
203
  prerelease: false
204
204
  version_requirements: !ruby/object:Gem::Requirement
205
205
  requirements:
206
- - - '>='
206
+ - - ! '>='
207
207
  - !ruby/object:Gem::Version
208
208
  version: '0'
209
209
  - !ruby/object:Gem::Dependency
210
210
  name: cucumber
211
211
  requirement: !ruby/object:Gem::Requirement
212
212
  requirements:
213
- - - '>='
213
+ - - ! '>='
214
214
  - !ruby/object:Gem::Version
215
215
  version: '0'
216
216
  type: :development
217
217
  prerelease: false
218
218
  version_requirements: !ruby/object:Gem::Requirement
219
219
  requirements:
220
- - - '>='
220
+ - - ! '>='
221
221
  - !ruby/object:Gem::Version
222
222
  version: '0'
223
223
  - !ruby/object:Gem::Dependency
224
224
  name: aruba
225
225
  requirement: !ruby/object:Gem::Requirement
226
226
  requirements:
227
- - - '>='
227
+ - - ! '>='
228
228
  - !ruby/object:Gem::Version
229
229
  version: '0'
230
230
  type: :development
231
231
  prerelease: false
232
232
  version_requirements: !ruby/object:Gem::Requirement
233
233
  requirements:
234
- - - '>='
234
+ - - ! '>='
235
235
  - !ruby/object:Gem::Version
236
236
  version: '0'
237
237
  - !ruby/object:Gem::Dependency
238
238
  name: spork
239
239
  requirement: !ruby/object:Gem::Requirement
240
240
  requirements:
241
- - - '>='
241
+ - - ! '>='
242
242
  - !ruby/object:Gem::Version
243
243
  version: '0'
244
244
  type: :development
245
245
  prerelease: false
246
246
  version_requirements: !ruby/object:Gem::Requirement
247
247
  requirements:
248
- - - '>='
248
+ - - ! '>='
249
249
  - !ruby/object:Gem::Version
250
250
  version: '0'
251
251
  - !ruby/object:Gem::Dependency
252
252
  name: vcr
253
253
  requirement: !ruby/object:Gem::Requirement
254
254
  requirements:
255
- - - '>='
255
+ - - ! '>='
256
256
  - !ruby/object:Gem::Version
257
257
  version: '0'
258
258
  type: :development
259
259
  prerelease: false
260
260
  version_requirements: !ruby/object:Gem::Requirement
261
261
  requirements:
262
- - - '>='
262
+ - - ! '>='
263
263
  - !ruby/object:Gem::Version
264
264
  version: '0'
265
265
  - !ruby/object:Gem::Dependency
266
266
  name: webmock
267
267
  requirement: !ruby/object:Gem::Requirement
268
268
  requirements:
269
- - - '>='
269
+ - - ! '>='
270
270
  - !ruby/object:Gem::Version
271
271
  version: '0'
272
272
  type: :development
273
273
  prerelease: false
274
274
  version_requirements: !ruby/object:Gem::Requirement
275
275
  requirements:
276
- - - '>='
276
+ - - ! '>='
277
277
  - !ruby/object:Gem::Version
278
278
  version: '0'
279
279
  description:
@@ -339,12 +339,12 @@ require_paths:
339
339
  - lib
340
340
  required_ruby_version: !ruby/object:Gem::Requirement
341
341
  requirements:
342
- - - '>='
342
+ - - ! '>='
343
343
  - !ruby/object:Gem::Version
344
344
  version: '0'
345
345
  required_rubygems_version: !ruby/object:Gem::Requirement
346
346
  requirements:
347
- - - '>='
347
+ - - ! '>='
348
348
  - !ruby/object:Gem::Version
349
349
  version: '0'
350
350
  requirements: []