cid 0.2.1 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,15 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 5dd888724adb78e68f3a6760d4cb64317ca5e3e2
4
- data.tar.gz: 610b3445181b32b18b0007f709e77185abf297b5
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ ZmU0ZTUwYjBlNGY2M2M1YTMwYzllNjhlMWJhZjE3OWZjMmM4ZTVlMw==
5
+ data.tar.gz: !binary |-
6
+ ZGUxYzBmODUyMWQ1MjRkYjM1YzNhMzNjNzFiNjA1M2Y4YjFkMDkyZA==
5
7
  SHA512:
6
- metadata.gz: f599d64d955fb3cc132bfc618210041f94ab421e893364ffb7f87612c6ec280a3f1aa2b978c88f94a0ef87b129ae8a0cddd512df1d67d554934465d58029d886
7
- data.tar.gz: 40b378091289e6a7d048a1afd7a1b66d36e4ed6fd6d7b7e85ef42974bb4919c6d0d3b62b925cd5e7435cf1aee5e3db60a6b135e21fd9ee5c057ad7d6a332fa82
8
+ metadata.gz: !binary |-
9
+ YTg5MGExZDBjZDdiYzc2NTI5MDZkOWJiMTNjMDI3ZTFlZDAxNzI1MTc3MGVj
10
+ YjIwOTQwNWIwYTU0ZjRiYzk5N2NhMDFhNDMwNmRmYWJhODRiMDNlNjNiODEy
11
+ NGFlNmU2NzExMzM3MmEzNjJkNTU2NGZjMDYyNWJkYmRkYmFiMmY=
12
+ data.tar.gz: !binary |-
13
+ ZTJhZmEyYTFjN2M2MGQ5NWY3YzhhMDVlN2M0MDdhODQwOGVlOGY0NzFmZGU2
14
+ N2RmZDdlOTU5M2NiOTYyZmE5MWFhZGQ1MjEzNzc4ZGMzYzY0NjdiZGVmNGI5
15
+ YmUyZDk4ZDg5MDk2YWE3YjBhNmNhNzFkZjdiNzMzYTJhNmNlYWM=
data/.travis.yml CHANGED
@@ -11,3 +11,11 @@ notifications:
11
11
  env:
12
12
  global:
13
13
  - secure: KTQ83GtNJyXwTUzYKbDtX1FzA3hsBSHefOqYsZ2KzOnj1dPZpGoeb4gcjWHXvNHqmR0mToT8U7KNq8AYWwHQLYCKf2+jRayd9UAzezhcs5u0SVfE5gmZLho57TW3AQvHI/uVQi9yg9/N3uDICqBplHJ7EMeMusUBc6ACF8FRmtY=
14
+ deploy:
15
+ provider: rubygems
16
+ api_key:
17
+ secure: eMaF/ItvetCL32WBFqpKxFbOO6uUXDbwIPDFYvwmtIE4q3khk7uiCwMVgw56t/GNyFKdaNkKlsrqcKgM0W3AmzHcA/gWJM0A69g6MtHswWkiiUzv6rPbZIAG99WTthTfs6RuvqXNuY4qaitO6IfrtZq5lXrlFKJgd/G7i7uZuDU=
18
+ gem: cid
19
+ on:
20
+ tags: true
21
+ repo: theodi/cid
data/README.md CHANGED
@@ -1,6 +1,22 @@
1
+ [![Build Status](https://travis-ci.org/theodi/cid.svg)](https://travis-ci.org/theodi/cid)
2
+ [![Coverage Status](http://img.shields.io/coveralls/theodi/cid.svg)](https://coveralls.io/r/theodi/cid)
3
+ [![Code Climate](http://img.shields.io/codeclimate/github/theodi/cid.svg)](https://codeclimate.com/github/theodi/cid)
4
+ [![Gem Version](http://img.shields.io/gem/v/cid.svg)](https://rubygems.org/gems/cid)
5
+ [![License](http://img.shields.io/:license-mit-blue.svg)](http://theodi.mit-license.org)
6
+ [![Badges](http://img.shields.io/:badges-6/6-ff6799.svg)](https://github.com/badges/badgerbadgerbadger)
7
+
1
8
  # Cid
2
9
 
3
- TODO: Write a gem description
10
+ Cid is **C**ontinuous **I**ntegraton for **D**ata.
11
+
12
+ There are two facets to Cid:
13
+
14
+ The first takes a [Datapackage](http://dataprotocols.org/data-packages/) with CSVs
15
+ in a Github repo and validates each one against a schema (in [JSON table schema](http://dataprotocols.org/json-table-schema/) format) which lives in the same folder.
16
+
17
+ The second updates the datapackage.json and pushes the changes to Github.
18
+
19
+ With these two things in place, we can continuously integrate our data!
4
20
 
5
21
  ## Installation
6
22
 
@@ -18,7 +34,65 @@ Or install it yourself as:
18
34
 
19
35
  ## Usage
20
36
 
21
- TODO: Write usage instructions here
37
+ Currently, Cid assumes your Datapackage lives in a cloned Github repo and has a structure like this:
38
+
39
+ ```
40
+ datapackage.json
41
+ - folder
42
+ |
43
+ |_ schema.json
44
+ |_ file.csv
45
+ |_ ..
46
+ ```
47
+
48
+ The schema.json file must match the [JSON table schema](http://dataprotocols.org/json-table-schema/).
49
+
50
+ You can have any number of folders, and each schema must correspond to the files within it.
51
+
52
+ When you run `cid validate` on the command line, Cid loops through each folder and validates each csv against the schema. It also checks for any common errors like ragged rows and missing characters using [csvlint](https://github.com/theodi/csvlint.rb).
53
+
54
+ When you run `cid publish` on the command line, Cid (again), loops through each folder and adds each csv to the `datapackage.json`. It then pushes the changes to the GitHub repo. For this to happen sucessfully, you must have a Github API key, which you can specify either as an environment variable like so:
55
+
56
+ export GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE"
57
+
58
+ Or as a command line option like so:
59
+
60
+ cid publish --github-token=YOUR_TOKEN_HERE
61
+
62
+ If you just want to skip the GitHub push altogether, just run
63
+
64
+ cid publish --skip-github
65
+
66
+ ## Getting this in Travis
67
+
68
+ Obviously, Cid is at its most powerful when used in a CI build. To get Cid working in Travis, simply add a `.travis.yml` file to your repo that looks a bit like this:
69
+
70
+ before_script:
71
+ - gem install cid
72
+ script:
73
+ - cid validate
74
+ after_success:
75
+ - '[ "$TRAVIS_BRANCH" == "master" ] && [ "$TRAVIS_PULL_REQUEST" == "false" ] && cid publish'
76
+
77
+ This installs cid, runs the validation, and then (if the branch is master) generates a new `datapackage.json` and pushes it to github.
78
+
79
+ You'll also need to add your Github token to the `.travis.yml` file. To do this, just run:
80
+
81
+ gem install travis
82
+ travis encrypt GITHUB_OAUTH_TOKEN="YOUR_TOKEN_HERE" --add
83
+
84
+ Then just add your repo to Travis, and push your changes.
85
+
86
+ Now whenever someone makes a pull request on your data, Cid will validate the data against the schema, and you'll get a nice build status telling you if it's good to go!
87
+
88
+ ## Examples
89
+
90
+ There's only two repos that use Cid so far (it is a new thing after all), and you can see it here:
91
+
92
+ * https://github.com/theodi/euro-elections
93
+ * https://github.com/DemocracyClub/ge2015-candidates
94
+
95
+ If you decide to start using Cid, and want it listing here, open a PR and add it to this list!
22
96
 
23
97
  ## Contributing
24
98
 
data/bin/cid CHANGED
@@ -19,10 +19,13 @@ class CidBin < Thor
19
19
 
20
20
  > $ cid validate /this/is/a/datapackage
21
21
  LONGDESC
22
+ option :ignore, :type => :array, :desc => "folder or folders to ignore (relative to the package root)"
22
23
  def validate(source = ".")
23
24
  check_path(source)
24
25
 
25
- validation = Cid::Validation.validate(source)
26
+ ignore = options[:ignore] || []
27
+
28
+ validation = Cid::Validation.validate(source, ignore)
26
29
 
27
30
  code = 0
28
31
 
@@ -33,3 +33,11 @@ Feature: Validate a datapackage
33
33
  When I run `cid validate /this/is/fake`
34
34
  Then the output should contain "Path doesn't exist!"
35
35
  And the exit status should be 1
36
+
37
+ Scenario: Ignore folders
38
+ When I cd to "spec/fixtures/multiple_folders"
39
+ And I run `cid validate --ignore votes`
40
+ Then the output should contain "seats/seats-1.csv is VALID"
41
+ And the output should contain "seats/seats-2.csv is VALID"
42
+ And the output should not contain "votes/votes-1.csv is VALID"
43
+ And the output should not contain "votes/votes-2.csv is VALID"
@@ -4,12 +4,14 @@ module Cid
4
4
 
5
5
  class Validation
6
6
 
7
- def self.validate(path)
7
+ def self.validate(path, ignore = [])
8
8
  result = {}
9
9
 
10
10
  paths = Dir.glob("#{path}/**")
11
11
 
12
12
  paths.each do |path|
13
+ next if ignore.include?(path.split("/").last)
14
+
13
15
  begin
14
16
  schema = Csvlint::Schema.load_from_json_table(File.new(Dir["#{path}/schema.json"][0]))
15
17
  rescue
data/lib/cid/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Cid
2
- VERSION = "0.2.1"
2
+ VERSION = "0.2.2"
3
3
  end
@@ -37,6 +37,19 @@ describe Cid::Validation do
37
37
  validation["seats/seats-2.csv"][:warnings].should == []
38
38
  end
39
39
 
40
+ it "ignores a folder if `ignore` is set" do
41
+ Csvlint::Validator.should_receive(:new).exactly(2).times.and_call_original
42
+
43
+ validation = Cid::Validation.validate(File.join(File.dirname(__FILE__), 'fixtures', 'multiple_folders'), ['seats'])
44
+
45
+ validation["votes/votes-1.csv"][:errors].should == []
46
+ validation["votes/votes-1.csv"][:warnings].should == []
47
+ validation["votes/votes-2.csv"][:errors].should == []
48
+ validation["votes/votes-2.csv"][:warnings].should == []
49
+ validation["seats/seats-1.csv"].should be_nil
50
+ validation["seats/seats-2.csv"].should be_nil
51
+ end
52
+
40
53
  it "returns errors for an invalid csv" do
41
54
  validation = Cid::Validation.validate(File.join(File.dirname(__FILE__), 'fixtures', 'invalid'))
42
55
 
metadata CHANGED
@@ -1,125 +1,125 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: cid
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.2.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - pezholio
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-05-23 00:00:00.000000000 Z
11
+ date: 2014-06-24 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: csvlint
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - '>='
17
+ - - ! '>='
18
18
  - !ruby/object:Gem::Version
19
19
  version: '0'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - '>='
24
+ - - ! '>='
25
25
  - !ruby/object:Gem::Version
26
26
  version: '0'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: colorize
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - '>='
31
+ - - ! '>='
32
32
  - !ruby/object:Gem::Version
33
33
  version: '0'
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - '>='
38
+ - - ! '>='
39
39
  - !ruby/object:Gem::Version
40
40
  version: '0'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: json
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
- - - '>='
45
+ - - ! '>='
46
46
  - !ruby/object:Gem::Version
47
47
  version: '0'
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
- - - '>='
52
+ - - ! '>='
53
53
  - !ruby/object:Gem::Version
54
54
  version: '0'
55
55
  - !ruby/object:Gem::Dependency
56
56
  name: git
57
57
  requirement: !ruby/object:Gem::Requirement
58
58
  requirements:
59
- - - '>='
59
+ - - ! '>='
60
60
  - !ruby/object:Gem::Version
61
61
  version: '0'
62
62
  type: :runtime
63
63
  prerelease: false
64
64
  version_requirements: !ruby/object:Gem::Requirement
65
65
  requirements:
66
- - - '>='
66
+ - - ! '>='
67
67
  - !ruby/object:Gem::Version
68
68
  version: '0'
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: github_api
71
71
  requirement: !ruby/object:Gem::Requirement
72
72
  requirements:
73
- - - '>='
73
+ - - ! '>='
74
74
  - !ruby/object:Gem::Version
75
75
  version: '0'
76
76
  type: :runtime
77
77
  prerelease: false
78
78
  version_requirements: !ruby/object:Gem::Requirement
79
79
  requirements:
80
- - - '>='
80
+ - - ! '>='
81
81
  - !ruby/object:Gem::Version
82
82
  version: '0'
83
83
  - !ruby/object:Gem::Dependency
84
84
  name: dotenv
85
85
  requirement: !ruby/object:Gem::Requirement
86
86
  requirements:
87
- - - '>='
87
+ - - ! '>='
88
88
  - !ruby/object:Gem::Version
89
89
  version: '0'
90
90
  type: :runtime
91
91
  prerelease: false
92
92
  version_requirements: !ruby/object:Gem::Requirement
93
93
  requirements:
94
- - - '>='
94
+ - - ! '>='
95
95
  - !ruby/object:Gem::Version
96
96
  version: '0'
97
97
  - !ruby/object:Gem::Dependency
98
98
  name: memoist
99
99
  requirement: !ruby/object:Gem::Requirement
100
100
  requirements:
101
- - - '>='
101
+ - - ! '>='
102
102
  - !ruby/object:Gem::Version
103
103
  version: '0'
104
104
  type: :runtime
105
105
  prerelease: false
106
106
  version_requirements: !ruby/object:Gem::Requirement
107
107
  requirements:
108
- - - '>='
108
+ - - ! '>='
109
109
  - !ruby/object:Gem::Version
110
110
  version: '0'
111
111
  - !ruby/object:Gem::Dependency
112
112
  name: thor
113
113
  requirement: !ruby/object:Gem::Requirement
114
114
  requirements:
115
- - - '>='
115
+ - - ! '>='
116
116
  - !ruby/object:Gem::Version
117
117
  version: '0'
118
118
  type: :runtime
119
119
  prerelease: false
120
120
  version_requirements: !ruby/object:Gem::Requirement
121
121
  requirements:
122
- - - '>='
122
+ - - ! '>='
123
123
  - !ruby/object:Gem::Version
124
124
  version: '0'
125
125
  - !ruby/object:Gem::Dependency
@@ -140,140 +140,140 @@ dependencies:
140
140
  name: rake
141
141
  requirement: !ruby/object:Gem::Requirement
142
142
  requirements:
143
- - - '>='
143
+ - - ! '>='
144
144
  - !ruby/object:Gem::Version
145
145
  version: '0'
146
146
  type: :development
147
147
  prerelease: false
148
148
  version_requirements: !ruby/object:Gem::Requirement
149
149
  requirements:
150
- - - '>='
150
+ - - ! '>='
151
151
  - !ruby/object:Gem::Version
152
152
  version: '0'
153
153
  - !ruby/object:Gem::Dependency
154
154
  name: rspec
155
155
  requirement: !ruby/object:Gem::Requirement
156
156
  requirements:
157
- - - '>='
157
+ - - ! '>='
158
158
  - !ruby/object:Gem::Version
159
159
  version: '0'
160
160
  type: :development
161
161
  prerelease: false
162
162
  version_requirements: !ruby/object:Gem::Requirement
163
163
  requirements:
164
- - - '>='
164
+ - - ! '>='
165
165
  - !ruby/object:Gem::Version
166
166
  version: '0'
167
167
  - !ruby/object:Gem::Dependency
168
168
  name: pry
169
169
  requirement: !ruby/object:Gem::Requirement
170
170
  requirements:
171
- - - '>='
171
+ - - ! '>='
172
172
  - !ruby/object:Gem::Version
173
173
  version: '0'
174
174
  type: :development
175
175
  prerelease: false
176
176
  version_requirements: !ruby/object:Gem::Requirement
177
177
  requirements:
178
- - - '>='
178
+ - - ! '>='
179
179
  - !ruby/object:Gem::Version
180
180
  version: '0'
181
181
  - !ruby/object:Gem::Dependency
182
182
  name: pry-remote
183
183
  requirement: !ruby/object:Gem::Requirement
184
184
  requirements:
185
- - - '>='
185
+ - - ! '>='
186
186
  - !ruby/object:Gem::Version
187
187
  version: '0'
188
188
  type: :development
189
189
  prerelease: false
190
190
  version_requirements: !ruby/object:Gem::Requirement
191
191
  requirements:
192
- - - '>='
192
+ - - ! '>='
193
193
  - !ruby/object:Gem::Version
194
194
  version: '0'
195
195
  - !ruby/object:Gem::Dependency
196
196
  name: coveralls
197
197
  requirement: !ruby/object:Gem::Requirement
198
198
  requirements:
199
- - - '>='
199
+ - - ! '>='
200
200
  - !ruby/object:Gem::Version
201
201
  version: '0'
202
202
  type: :development
203
203
  prerelease: false
204
204
  version_requirements: !ruby/object:Gem::Requirement
205
205
  requirements:
206
- - - '>='
206
+ - - ! '>='
207
207
  - !ruby/object:Gem::Version
208
208
  version: '0'
209
209
  - !ruby/object:Gem::Dependency
210
210
  name: cucumber
211
211
  requirement: !ruby/object:Gem::Requirement
212
212
  requirements:
213
- - - '>='
213
+ - - ! '>='
214
214
  - !ruby/object:Gem::Version
215
215
  version: '0'
216
216
  type: :development
217
217
  prerelease: false
218
218
  version_requirements: !ruby/object:Gem::Requirement
219
219
  requirements:
220
- - - '>='
220
+ - - ! '>='
221
221
  - !ruby/object:Gem::Version
222
222
  version: '0'
223
223
  - !ruby/object:Gem::Dependency
224
224
  name: aruba
225
225
  requirement: !ruby/object:Gem::Requirement
226
226
  requirements:
227
- - - '>='
227
+ - - ! '>='
228
228
  - !ruby/object:Gem::Version
229
229
  version: '0'
230
230
  type: :development
231
231
  prerelease: false
232
232
  version_requirements: !ruby/object:Gem::Requirement
233
233
  requirements:
234
- - - '>='
234
+ - - ! '>='
235
235
  - !ruby/object:Gem::Version
236
236
  version: '0'
237
237
  - !ruby/object:Gem::Dependency
238
238
  name: spork
239
239
  requirement: !ruby/object:Gem::Requirement
240
240
  requirements:
241
- - - '>='
241
+ - - ! '>='
242
242
  - !ruby/object:Gem::Version
243
243
  version: '0'
244
244
  type: :development
245
245
  prerelease: false
246
246
  version_requirements: !ruby/object:Gem::Requirement
247
247
  requirements:
248
- - - '>='
248
+ - - ! '>='
249
249
  - !ruby/object:Gem::Version
250
250
  version: '0'
251
251
  - !ruby/object:Gem::Dependency
252
252
  name: vcr
253
253
  requirement: !ruby/object:Gem::Requirement
254
254
  requirements:
255
- - - '>='
255
+ - - ! '>='
256
256
  - !ruby/object:Gem::Version
257
257
  version: '0'
258
258
  type: :development
259
259
  prerelease: false
260
260
  version_requirements: !ruby/object:Gem::Requirement
261
261
  requirements:
262
- - - '>='
262
+ - - ! '>='
263
263
  - !ruby/object:Gem::Version
264
264
  version: '0'
265
265
  - !ruby/object:Gem::Dependency
266
266
  name: webmock
267
267
  requirement: !ruby/object:Gem::Requirement
268
268
  requirements:
269
- - - '>='
269
+ - - ! '>='
270
270
  - !ruby/object:Gem::Version
271
271
  version: '0'
272
272
  type: :development
273
273
  prerelease: false
274
274
  version_requirements: !ruby/object:Gem::Requirement
275
275
  requirements:
276
- - - '>='
276
+ - - ! '>='
277
277
  - !ruby/object:Gem::Version
278
278
  version: '0'
279
279
  description:
@@ -339,12 +339,12 @@ require_paths:
339
339
  - lib
340
340
  required_ruby_version: !ruby/object:Gem::Requirement
341
341
  requirements:
342
- - - '>='
342
+ - - ! '>='
343
343
  - !ruby/object:Gem::Version
344
344
  version: '0'
345
345
  required_rubygems_version: !ruby/object:Gem::Requirement
346
346
  requirements:
347
- - - '>='
347
+ - - ! '>='
348
348
  - !ruby/object:Gem::Version
349
349
  version: '0'
350
350
  requirements: []