ckan_cli 1.0.0 → 1.0.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +121 -16
  3. data/lib/ckancli/commands/upload.rb +448 -444
  4. data/lib/ckancli/version.rb +1 -1
  5. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 0d5686c7f8bd8d3fe87ac9fc4de600667ecd455fa773e383f8af10410d237dab
4
- data.tar.gz: 0203fd33391a22b27eefcabc8f663b75080df194673800383fe2e563f448096f
3
+ metadata.gz: e6e9e4fe9afdc7e11b96c179e7eefe16b7c19247d4d1f08b029965688528bdf6
4
+ data.tar.gz: bd4ed9291ff176a1303f7ba30283070dd732ba30342347e60e25b856fdb33f16
5
5
  SHA512:
6
- metadata.gz: 9f2b823b83f3a9c9570fe45db2b797012e9b32d77f2db22b52d63db1bc1f7dfc1d2ffd3aa89ca6d019995b07f25f8dfaf7f43f3ffdf0b814cc409cfb018b584c
7
- data.tar.gz: 9597fcf8f12824676a93c23b92496e8e282ff847ac3b41657915873e024b969f8b276d3aa17cfa2d5a4a91682e5a8eeb27f4e20064beb64b7a29db8500072005
6
+ metadata.gz: 5cb9bbe079a905983d59158537b87ccc49901efa3e63b8ef9ca66a910224eba9b9cc7d253b864af9bd14bf1e152f828cd78ab0351a8a63767c8a93c2216b2c0f
7
+ data.tar.gz: 29f6dd5cf5a07bb95284dffe88529815544bb7fb95b0dc32b9ccf27542913bec7bd32307492060ead203b084fe4e49c0990be68f4591d26b72a106636dd2f4f3
data/README.md CHANGED
@@ -1,20 +1,16 @@
1
1
  # [CKAN CLI](https://github.com/datagovlv/ckan_cli)
2
2
 
3
- > CKAN on your command line. Use your terminal to validate and publish resources to CKAN. Works with CKAN API.
3
+ [![Gem Version](https://badge.fury.io/rb/ckan_cli.svg)][gem]
4
4
 
5
- ## Installation
5
+ [gem]: http://badge.fury.io/rb/ckan_cli
6
6
 
7
- Add this line to your application's Gemfile:
7
+ > CKAN on your command line. Use your terminal to validate and publish resources to CKAN. Works with CKAN API. Developed and adjusted for Latvian open data portal (Comprehensive Knowledge Archive Network).
8
8
 
9
- ```ruby
10
- gem 'ckan_cli'
11
- ```
12
-
13
- And then execute:
9
+ ![Interface](https://github.com/datagovlv/ckan_cli/raw/master/assets/interface.png)
14
10
 
15
- $ bundle
11
+ ## Installation
16
12
 
17
- Or install it yourself as:
13
+ Install Ruby Gem yourself:
18
14
 
19
15
  $ gem install ckan_cli
20
16
 
@@ -23,26 +19,135 @@ Or install it yourself as:
23
19
  Run it:
24
20
 
25
21
  ```shell
26
- $ ckancli
22
+ $ ckancli.rb
27
23
  ```
28
24
 
29
25
  For instance, to publish to CKAN all CSV files in folder do
30
26
 
31
27
  ```shell
32
- $ ckancli upload -d C:\my_csv_collection\ -c config.json -r resource.json
28
+ $ ckancli.rb upload -d C:\my_csv_collection\ -c config.json -r resource.json
33
29
  ```
34
30
 
35
31
  To see help do
36
32
 
37
33
  ```shell
38
- $ ckancli help
34
+ $ ckancli.rb help
35
+ ```
36
+
37
+ ## Configuration
38
+
39
+ > Example files including schemas and configurations are located in 'example_files' directory.
40
+
41
+ ### Global configuration
42
+
43
+ Contains configuration for CKAN API and e-mail notifications (sections "email_server" and "notification_receiver" are optional).
44
+
45
+ ```javascript
46
+ {
47
+ "ckan_api":{
48
+ "api_key":"YOUR_CKAN_API_KEY",
49
+ "url":"https://data.gov.lv/api/3/"
50
+ },
51
+ "email_server": {
52
+ "address": "smtp.yourdomain.com",
53
+ "port": "25",
54
+ "ssl": false,
55
+ "smtp_user": null,
56
+ "smtp_password": null,
57
+ "sender": "ckancli@data.gov.lv",
58
+ "subject": "CKAN CLI task summary"
59
+ },
60
+ "notification_receiver": {
61
+ "error": "mail_one@yourdomain.com",
62
+ "success": "mail_one@yourdomain.com, mail_two@yourdomain.com"
63
+ }
64
+ }
65
+ ```
66
+
67
+ ### Resource configuration
68
+
69
+ Contains configuration for resource metadata (parameters as specified in CKAN API guidelines). If resource name is not specified, file name will be used. If resource identifier is not specified, new resource will be created.
70
+
71
+ ```javascript
72
+ {
73
+ "result": {
74
+ "name": "CKAN CLI file",
75
+ "package_name": "ta",
76
+ "package_id": "d1819200-121a-4452-8868-34f2c2a898c1",
77
+ "last_modified": "2019-05-14T05:12:21.257451",
78
+ "package_title": "TA",
79
+ "id": "15f950f0-1d50-467f-8db3-87366d30f7db"
80
+ },
81
+ "success": true
82
+ }
83
+ ```
84
+
85
+ ### Package configuration
86
+
87
+ Contains configuration for package metadata (parameters as specified in CKAN API guidelines). If package configuration is not specified, package metadata won't be affected.
88
+
89
+ ```javascript
90
+ {
91
+ "result": {
92
+ "frequency": "http://publications.europa.eu/mdr/authority/frequency/DAILY",
93
+ "id": "d1819200-121a-4452-8868-34f2c2a898c1",
94
+ "metadata_modified": "2019-08-06T06:39:18.422714",
95
+ "name": "cli",
96
+ "title": "CKAN CLI test"
97
+ },
98
+ "success": true
99
+ }
100
+ ```
101
+
102
+ ### CSV schemas
103
+
104
+ CSV files can be validated against a schema. The structure currently follows [JSON Table Schema](http://www.w3.org/TR/tabular-metadata/). Detailed information for validations can be found at [CSV Lint project](https://github.com/theodi/csvlint.rb).
105
+
106
+ ```javascript
107
+ {
108
+ "fields": [
109
+ {
110
+ "name": "id",
111
+ "constraints": {
112
+ "required": true,
113
+ "type": "http://www.w3.org/2001/XMLSchema#int"
114
+ }
115
+ },
116
+ {
117
+ "name": "price",
118
+ "constraints": {
119
+ "required": true,
120
+ "minLength": 1
121
+ }
122
+ },
123
+ {
124
+ "name": "postcode",
125
+ "constraints": {
126
+ "required": true,
127
+ "pattern": "[A-Z]{1,2}[0-9]{4}"
128
+ }
129
+ }
130
+ ]
131
+ }
39
132
  ```
40
133
 
41
- ## Development
134
+ ## Known issues for windows users
135
+
136
+ ### Error "Could not open library 'libcurl.dll': The specified module could not be found.".
137
+
138
+ If this error is ocurring, copy file '\ext\libcurl.dll' to Ruby executables directory (e.g. 'C:\Ruby26-x64\bin\').
139
+
140
+ ## For developers
141
+
142
+ Add this line to your application's Gemfile:
42
143
 
43
- After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
144
+ ```ruby
145
+ gem 'ckan_cli'
146
+ ```
44
147
 
45
- To install this gem onto your local machine, run `bundle exec rake install`.
148
+ And then execute:
149
+
150
+ $ bundle
46
151
 
47
152
  ## Contributing
48
153
 
data/lib/ckancli/commands/upload.rb CHANGED
@@ -1,445 +1,449 @@
1
- # frozen_string_literal: true
2
-
3
- require 'colorize'
4
- require 'pathname'
5
- require 'csvlint'
6
- require 'json'
7
- require 'tty-logger'
8
- require 'tty-spinner'
9
- require 'net/smtp'
10
- require 'mail'
11
- require 'open-uri'
12
- require 'uri'
13
-
14
- require_relative '../command'
15
- require_relative '../../ckanclient/ckanclient'
16
-
17
- module Ckancli
18
- module Commands
19
- class Upload < Ckancli::Command
20
- def initialize(options)
21
- @options = options
22
-
23
- @dir = nil
24
- @resources = []
25
- @http_resource = false
26
- @schema = nil
27
- @resource_config = nil
28
- @dataset_config = nil
29
- @api_config = nil
30
- @logger = nil
31
- @ckan_client = nil
32
- @update_modified = false
33
- @mail_server = nil
34
- end
35
-
36
- def execute(input: $stdin, output: $stdout)
37
- begin
38
- prepare_options(@options)
39
-
40
- if !@schema.nil?
41
- validate()
42
- else
43
- puts ""
44
- show_info "No schema - skipping validation"
45
- end
46
-
47
- upload()
48
-
49
- if !@dataset_config.nil?
50
- update_dataset()
51
- else
52
- puts ""
53
- show_info "No dataset configuration - skipping update"
54
- end
55
-
56
- @resources.each do |res|
57
- if !res[:log_file].nil?
58
- res[:log_file].close
59
- end
60
- end
61
-
62
- if !@mail_server.nil? && !@mail_receivers.nil?
63
- mail_notify()
64
- end
65
- rescue StandardError => e
66
- return_error "Unexpected error", e
67
- end
68
- end
69
-
70
- private
71
- def prepare_options(options)
72
- @http_resource = true if options[:dir] =~ /^http(s)?/
73
-
74
- if @http_resource
75
- @dir = Dir.pwd
76
- else
77
- @dir = File.directory?(options[:dir]) ? options[:dir] : (File.file?(options[:dir]) ? File.dirname(options[:dir]) : nil)
78
- end
79
-
80
- if @dir.nil?
81
- return_error "Invalid directory - #{options[:dir]}"
82
- end
83
-
84
- begin
85
- @logger = TTY::Logger.new do |config|
86
- config.metadata = [:date, :time]
87
- config.handlers = [
88
- [:stream, output: File.open(File.join(@dir, "output.log"), "a")]
89
- ]
90
- end
91
- rescue StandartError => e
92
- return_error "Could not initialize logger", e
93
- end
94
-
95
- begin
96
- if @http_resource
97
- file_path = File.join(@dir, File.basename(URI.parse(options[:dir]).path))
98
-
99
- File.open(file_path, "wb") do |file|
100
- file.write open(options[:dir]).read
101
- end
102
-
103
- @resources.push(file_path)
104
- else
105
- if File.directory?(options[:dir])
106
- @resources = Dir.entries(options[:dir]).select{ |f| File.file?(File.join(options[:dir], f)) && (options[:ignoreextension] || File.extname(f).downcase == ".csv") }.map { |f| File.join(options[:dir], f) }
107
- else
108
- @resources.push(options[:dir])
109
- end
110
- end
111
-
112
- @resources = @resources.map{ |r|
113
- {
114
- :path => r,
115
- :valid => true,
116
- :summary => nil,
117
- :log_file => File.open(File.join(@dir, "#{File.basename(r)}.log"), "w"),
118
- :log => nil
119
- }
120
- }
121
-
122
- @resources.each do |res|
123
- res[:log] = TTY::Logger.new do |config|
124
- config.metadata = [:date, :time]
125
- config.handlers = [
126
- [:stream, output: res[:log_file]]
127
- ]
128
- end
129
- end
130
- rescue Errno::ENOENT => e
131
- return_error "Could not load resources", e
132
- rescue StandardError => e
133
- return_error "Could not load resources", e
134
- end
135
-
136
- if options[:validationschema]
137
- @schema = (Pathname.new(options[:validationschema])).absolute? ? options[:validationschema] : File.join(@dir, options[:validationschema])
138
- end
139
-
140
- if options[:resourceconfig]
141
- begin
142
- @resource_config = JSON.parse(File.read((Pathname.new(options[:resourceconfig])).absolute? ? options[:resourceconfig] : File.join(@dir, options[:resourceconfig])))
143
- rescue Errno::ENOENT => e
144
- return_error "Could not load resource configuration", e
145
- end
146
- end
147
-
148
- if options[:datasetconfig]
149
- begin
150
- @dataset_config = JSON.parse(File.read((Pathname.new(options[:datasetconfig])).absolute? ? options[:datasetconfig] : File.join(@dir, options[:datasetconfig])))
151
- rescue Errno::ENOENT => e
152
- return_error "Could not load dataset configuration", e
153
- end
154
- end
155
-
156
- if options[:configuration]
157
- begin
158
- @api_config = JSON.parse(File.read((Pathname.new(options[:configuration])).absolute? ? options[:configuration] : File.join(@dir, options[:configuration])))
159
- rescue Errno::ENOENT => e
160
- return_error "Could not load configuration", e
161
- end
162
- end
163
-
164
- if !@api_config.nil?
165
- @ckan_client = CkanClient::Client.new(@api_config["ckan_api"]["url"], @api_config["ckan_api"]["api_key"])
166
- @mail_server = @api_config["email_server"]
167
- @mail_receivers = @api_config["notification_receiver"]
168
- end
169
-
170
- @update_modified = options[:updatemodified]
171
- @overwrite = options[:overwrite]
172
- end
173
-
174
- def validate()
175
- puts ""
176
- show_info "Starting validations..."
177
- @resources.each do |res|
178
- task("Validating resource: #{File.basename(res[:path])}") do
179
- ok = false
180
- msg = nil
181
- exception = nil
182
-
183
- begin
184
- res[:log].info "Validating resource..."
185
-
186
- res[:valid] = false
187
-
188
- source = read_resource(res[:path])
189
- schema = read_schema(@schema)
190
-
191
- if schema.class == Csvlint::Schema && schema.description == "malformed"
192
- msg = "Invalid schema - malformed JSON"
193
- else
194
- validator = Csvlint::Validator.new(source, {}, schema)
195
-
196
- if validator.errors.length > 0 || validator.warnings.length > 0
197
- msg = "#{validator.errors.length} errors, #{validator.warnings.length} warnings"
198
- res[:summary] = {
199
- :errors => validator.errors.map { |v| format_validation(v, schema) },
200
- :warnings => validator.warnings.map { |v| format_validation(v, schema) }
201
- }
202
-
203
- res[:log].error res[:summary].to_json
204
- end
205
-
206
- ok = validator.valid?
207
- res[:valid] = ok
208
- end
209
- rescue Errno::ENOENT, OpenURI::HTTPError => e
210
- msg = "Could not load file - #{e.message}"
211
- rescue Csvlint::Csvw::MetadataError => e
212
- msg = "Invalid schema metadata - #{e.message}"
213
- rescue StandardError => e
214
- msg = "Could not validate resource"
215
- exception = e
216
- end
217
-
218
- if !exception.nil?
219
- res[:log].error msg, exception
220
- elsif !msg.nil?
221
- res[:log].info msg
222
- else
223
- res[:log].info "OK - Validation successfull"
224
- end
225
- res[:log].info "End of validation"
226
-
227
- [ok, msg, exception]
228
- end
229
- end
230
-
231
- show_info "End of validations"
232
- end
233
-
234
- def mail_notify
235
- puts ""
236
- show_info "Sending e-mail notifications..."
237
-
238
- begin
239
- mail = Mail.new
240
- mail[:from] = @mail_server["sender"]
241
- mail[:to] = @mail_receivers["error"]
242
- mail[:subject] = @mail_server["subject"]
243
- mail.delivery_method :smtp, {
244
- :address => @mail_server["address"],
245
- :port => @mail_server["port"],
246
- :user_name => @mail_server["smtp_user"],
247
- :password => @mail_server["smtp_password"],
248
- :authentication => @mail_server["smtp_user"].nil? ? "none" : "plain",
249
- :ssl => @mail_server["ssl"],
250
- :openssl_verify_mode => OpenSSL::SSL::VERIFY_NONE
251
- }
252
-
253
- # message
254
- msg = "CKAN CLI task completed.\r\n\r\n#{@resources.length} files processed"
255
-
256
- # attachments
257
- @resources.each do |res|
258
- msg = "#{msg}\r\n- #{File.basename(res[:path])}"
259
- if !res[:summary].nil?
260
- msg = "#{msg} (#{res[:summary][:errors].length} errors, #{res[:summary][:warnings].length} warnings)"
261
- end
262
- mail.add_file :filename => File.basename(res[:path] + ".log"), :content => File.read(res[:path] + ".log")
263
- end
264
- msg = "#{msg}\r\n\r\nLog files attached."
265
- mail[:body] = msg
266
-
267
- mail.deliver!
268
- rescue StandardError => e
269
- show_error "Could not send e-mail", e
270
- end
271
-
272
- show_info "End of sending"
273
- end
274
-
275
- def upload()
276
- puts ""
277
- show_info "Starting upload..."
278
- @resources.each do |res|
279
- next unless res[:valid]
280
-
281
- task("Uploading resource: #{File.basename(res[:path])}") do
282
- res[:log].info "Uploading resource..."
283
-
284
- ok = true
285
- msg = nil
286
- exception = nil
287
-
288
- params = @resource_config["result"].clone
289
-
290
- if @update_modified
291
- params["last_modified"] = JSON.parse(Time.new.to_json)
292
- end
293
-
294
- # check if resource exists and if overwriting
295
- if !params["id"].nil? && !@overwrite
296
- resource = @ckan_client.get_resource(params["id"])
297
-
298
- if !resource.nil?
299
- ok = false
300
- msg = "Resource already exists - skipping. Use parameter 'overwrite' to overwrite changes."
301
- end
302
- end
303
-
304
- if ok
305
- @ckan_client.create_or_update_resource(params, File.new(res[:path], 'rb'), true){ |response, status_ok|
306
- ok = status_ok
307
-
308
- if !ok
309
- msg = "HTTP error #{response.code}"
310
-
311
- log_error response.body
312
- end
313
- }
314
- end
315
-
316
- if !ok
317
- res[:log].error msg
318
- else
319
- res[:log].info "OK - upload successfull"
320
- end
321
- res[:log].info "End of upload"
322
-
323
- [ok, msg, exception]
324
- end
325
- end
326
-
327
- show_info "End of upload"
328
- end
329
-
330
- def update_dataset()
331
- puts ""
332
- show_info "Starting dataset update..."
333
-
334
- task("Updating dataset") do
335
- ok = false
336
- msg = nil
337
- exception = nil
338
-
339
- @ckan_client.update_package(@dataset_config["result"]["id"], @dataset_config["result"].clone, true){ |response, status_ok|
340
- ok = status_ok
341
-
342
- if !ok
343
- msg = "HTTP error #{response.code}"
344
-
345
- log_error response.body
346
- end
347
- }
348
-
349
- [ok, msg, exception]
350
- end
351
- show_info "End of dataset update"
352
- end
353
-
354
- def return_error(message, exception = nil)
355
- show_error(message, exception)
356
-
357
- exit 1
358
- end
359
-
360
- def show_error(message, exception = nil)
361
- if !exception.nil?
362
- message = "#{message} - #{exception.message}"
363
- end
364
-
365
- puts " #{message} ".white.on_red
366
-
367
- log_fatal message, exception
368
- end
369
-
370
- def show_info(message)
371
- puts " #{message} "
372
-
373
- log_info message
374
- end
375
-
376
- def task(message)
377
- log_info message
378
-
379
- spinner = TTY::Spinner.new("[:spinner] #{message}...")
380
- spinner.auto_spin
381
- ok, message, exception = yield
382
- spinner.stop(ok ? "OK".green : "ERROR".red)
383
-
384
- if !message.nil?
385
- message = " #{message}"
386
-
387
- if !exception.nil?
388
- show_error message, exception
389
- else
390
- show_info message
391
- end
392
- end
393
- end
394
-
395
- def read_resource(source)
396
- # check if URL or file
397
- unless source =~ /^http(s)?/
398
- source = File.new( source )
399
- end
400
-
401
- source
402
- end
403
-
404
- def read_schema(path)
405
- schema = Csvlint::Schema.load_from_uri(path, false)
406
-
407
- schema
408
- end
409
-
410
- def format_validation(error, schema = nil)
411
- h = {
412
- type: error.type,
413
- category: error.category,
414
- row: error.row,
415
- col: error.column
416
- }
417
-
418
- if error.column && !schema.nil? && schema.class == Csvlint::Schema && schema.fields[error.column - 1] != nil
419
- field = schema.fields[error.column - 1]
420
- h[:header] = field.name
421
- end
422
-
423
- h
424
- end
425
-
426
- def log_fatal(message, exception = nil)
427
- if !@logger.nil?
428
- @logger.fatal message, exception
429
- end
430
- end
431
-
432
- def log_info(message)
433
- if !@logger.nil?
434
- @logger.info message
435
- end
436
- end
437
-
438
- def log_error(message)
439
- if !@logger.nil?
440
- @logger.error message
441
- end
442
- end
443
- end
444
- end
1
+ # frozen_string_literal: true
2
+
3
+ require 'colorize'
4
+ require 'pathname'
5
+ require 'csvlint'
6
+ require 'json'
7
+ require 'tty-logger'
8
+ require 'tty-spinner'
9
+ require 'net/smtp'
10
+ require 'mail'
11
+ require 'open-uri'
12
+ require 'uri'
13
+
14
+ require_relative '../command'
15
+ require_relative '../../ckanclient/ckanclient'
16
+
17
+ module Ckancli
18
+ module Commands
19
+ class Upload < Ckancli::Command
20
+ def initialize(options)
21
+ @options = options
22
+
23
+ @dir = nil
24
+ @resources = []
25
+ @http_resource = false
26
+ @schema = nil
27
+ @resource_config = nil
28
+ @dataset_config = nil
29
+ @api_config = nil
30
+ @logger = nil
31
+ @ckan_client = nil
32
+ @update_modified = false
33
+ @mail_server = nil
34
+ end
35
+
36
+ def execute(input: $stdin, output: $stdout)
37
+ begin
38
+ prepare_options(@options)
39
+
40
+ if !@schema.nil?
41
+ validate()
42
+ else
43
+ puts ""
44
+ show_info "No schema - skipping validation"
45
+ end
46
+
47
+ upload()
48
+
49
+ if !@dataset_config.nil?
50
+ update_dataset()
51
+ else
52
+ puts ""
53
+ show_info "No dataset configuration - skipping update"
54
+ end
55
+
56
+ @resources.each do |res|
57
+ if !res[:log_file].nil?
58
+ res[:log_file].close
59
+ end
60
+ end
61
+
62
+ if !@mail_server.nil? && !@mail_receivers.nil?
63
+ mail_notify()
64
+ end
65
+ rescue StandardError => e
66
+ return_error "Unexpected error", e
67
+ end
68
+ end
69
+
70
+ private
71
+ def prepare_options(options)
72
+ @http_resource = true if options[:dir] =~ /^http(s)?/
73
+
74
+ if @http_resource
75
+ @dir = Dir.pwd
76
+ else
77
+ @dir = File.directory?(options[:dir]) ? options[:dir] : (File.file?(options[:dir]) ? File.dirname(options[:dir]) : nil)
78
+ end
79
+
80
+ if @dir.nil?
81
+ return_error "Invalid directory - #{options[:dir]}"
82
+ end
83
+
84
+ begin
85
+ @logger = TTY::Logger.new do |config|
86
+ config.metadata = [:date, :time]
87
+ config.handlers = [
88
+ [:stream, output: File.open(File.join(@dir, "output.log"), "a")]
89
+ ]
90
+ end
91
+ rescue StandardError => e
92
+ return_error "Could not initialize logger", e
93
+ end
94
+
95
+ begin
96
+ if @http_resource
97
+ file_path = File.join(@dir, File.basename(URI.parse(options[:dir]).path))
98
+
99
+ File.open(file_path, "wb") do |file|
100
+ file.write open(options[:dir]).read
101
+ end
102
+
103
+ @resources.push(file_path)
104
+ else
105
+ if File.directory?(options[:dir])
106
+ @resources = Dir.entries(options[:dir]).select{ |f| File.file?(File.join(options[:dir], f)) && (options[:ignoreextension] || File.extname(f).downcase == ".csv") }.map { |f| File.join(options[:dir], f) }
107
+ else
108
+ @resources.push(options[:dir])
109
+ end
110
+ end
111
+
112
+ @resources = @resources.map{ |r|
113
+ {
114
+ :path => r,
115
+ :valid => true,
116
+ :summary => nil,
117
+ :log_file => File.open(File.join(@dir, "#{File.basename(r)}.log"), "w"),
118
+ :log => nil
119
+ }
120
+ }
121
+
122
+ @resources.each do |res|
123
+ res[:log] = TTY::Logger.new do |config|
124
+ config.metadata = [:date, :time]
125
+ config.handlers = [
126
+ [:stream, output: res[:log_file]]
127
+ ]
128
+ end
129
+ end
130
+ rescue Errno::ENOENT => e
131
+ return_error "Could not load resources", e
132
+ rescue StandardError => e
133
+ return_error "Could not load resources", e
134
+ end
135
+
136
+ if options[:validationschema]
137
+ @schema = (Pathname.new(options[:validationschema])).absolute? ? options[:validationschema] : File.join(@dir, options[:validationschema])
138
+ end
139
+
140
+ if options[:resourceconfig]
141
+ begin
142
+ @resource_config = JSON.parse(File.read((Pathname.new(options[:resourceconfig])).absolute? ? options[:resourceconfig] : File.join(@dir, options[:resourceconfig])))
143
+ rescue Errno::ENOENT => e
144
+ return_error "Could not load resource configuration", e
145
+ end
146
+ end
147
+
148
+ if options[:datasetconfig]
149
+ begin
150
+ @dataset_config = JSON.parse(File.read((Pathname.new(options[:datasetconfig])).absolute? ? options[:datasetconfig] : File.join(@dir, options[:datasetconfig])))
151
+ rescue Errno::ENOENT => e
152
+ return_error "Could not load dataset configuration", e
153
+ end
154
+ end
155
+
156
+ if options[:configuration]
157
+ begin
158
+ @api_config = JSON.parse(File.read((Pathname.new(options[:configuration])).absolute? ? options[:configuration] : File.join(@dir, options[:configuration])))
159
+ rescue Errno::ENOENT => e
160
+ return_error "Could not load configuration", e
161
+ end
162
+ end
163
+
164
+ if !@api_config.nil?
165
+ @ckan_client = CkanClient::Client.new(@api_config["ckan_api"]["url"], @api_config["ckan_api"]["api_key"])
166
+ @mail_server = @api_config["email_server"]
167
+ @mail_receivers = @api_config["notification_receiver"]
168
+ end
169
+
170
+ @update_modified = options[:updatemodified]
171
+ @overwrite = options[:overwrite]
172
+ end
173
+
174
+ def validate()
175
+ puts ""
176
+ show_info "Starting validations..."
177
+ @resources.each do |res|
178
+ task("Validating resource: #{File.basename(res[:path])}") do
179
+ ok = false
180
+ msg = nil
181
+ exception = nil
182
+
183
+ begin
184
+ res[:log].info "Validating resource..."
185
+
186
+ res[:valid] = false
187
+
188
+ source = read_resource(res[:path])
189
+ schema = read_schema(@schema)
190
+
191
+ if schema.class == Csvlint::Schema && schema.description == "malformed"
192
+ msg = "Invalid schema - malformed JSON"
193
+ else
194
+ validator = Csvlint::Validator.new(source, {}, schema)
195
+
196
+ if validator.errors.length > 0 || validator.warnings.length > 0
197
+ msg = "#{validator.errors.length} errors, #{validator.warnings.length} warnings"
198
+ res[:summary] = {
199
+ :errors => validator.errors.map { |v| format_validation(v, schema) },
200
+ :warnings => validator.warnings.map { |v| format_validation(v, schema) }
201
+ }
202
+
203
+ res[:log].error res[:summary].to_json
204
+ end
205
+
206
+ ok = validator.valid?
207
+ res[:valid] = ok
208
+ end
209
+ rescue Errno::ENOENT, OpenURI::HTTPError => e
210
+ msg = "Could not load file - #{e.message}"
211
+ rescue Csvlint::Csvw::MetadataError => e
212
+ msg = "Invalid schema metadata - #{e.message}"
213
+ rescue StandardError => e
214
+ msg = "Could not validate resource"
215
+ exception = e
216
+ end
217
+
218
+ if !exception.nil?
219
+ res[:log].error msg, exception
220
+ elsif !msg.nil?
221
+ res[:log].info msg
222
+ else
223
+ res[:log].info "OK - Validation successfull"
224
+ end
225
+ res[:log].info "End of validation"
226
+
227
+ [ok, msg, exception]
228
+ end
229
+ end
230
+
231
+ show_info "End of validations"
232
+ end
233
+
234
+ def mail_notify
235
+ puts ""
236
+ show_info "Sending e-mail notifications..."
237
+
238
+ begin
239
+ mail = Mail.new
240
+ mail[:from] = @mail_server["sender"]
241
+ mail[:subject] = @mail_server["subject"]
242
+ mail.delivery_method :smtp, {
243
+ :address => @mail_server["address"],
244
+ :port => @mail_server["port"],
245
+ :user_name => @mail_server["smtp_user"],
246
+ :password => @mail_server["smtp_password"],
247
+ :authentication => @mail_server["smtp_user"].nil? ? "none" : "plain",
248
+ :ssl => @mail_server["ssl"],
249
+ :openssl_verify_mode => OpenSSL::SSL::VERIFY_NONE
250
+ }
251
+
252
+ # message
253
+ msg = "CKAN CLI task completed.\r\n\r\n#{@resources.length} files processed"
254
+ has_errors = false
255
+
256
+ # attachments
257
+ @resources.each do |res|
258
+ msg = "#{msg}\r\n- #{File.basename(res[:path])}"
259
+ if !res[:summary].nil?
260
+ msg = "#{msg} (#{res[:summary][:errors].length} errors, #{res[:summary][:warnings].length} warnings)"
261
+ if res[:summary][:errors].length > 0
262
+ has_errors = true
263
+ end
264
+ end
265
+ mail.add_file :filename => File.basename(res[:path] + ".log"), :content => File.read(res[:path] + ".log")
266
+ end
267
+ msg = "#{msg}\r\n\r\nLog files attached."
268
+ mail[:body] = msg
269
+ mail[:to] = has_errors ? @mail_receivers["error"] : @mail_receivers["success"]
270
+
271
+ mail.deliver!
272
+ rescue StandardError => e
273
+ show_error "Could not send e-mail", e
274
+ end
275
+
276
+ show_info "End of sending"
277
+ end
278
+
279
+ def upload()
280
+ puts ""
281
+ show_info "Starting upload..."
282
+ @resources.each do |res|
283
+ next unless res[:valid]
284
+
285
+ task("Uploading resource: #{File.basename(res[:path])}") do
286
+ res[:log].info "Uploading resource..."
287
+
288
+ ok = true
289
+ msg = nil
290
+ exception = nil
291
+
292
+ params = @resource_config["result"].clone
293
+
294
+ if @update_modified
295
+ params["last_modified"] = JSON.parse(Time.new.to_json)
296
+ end
297
+
298
+ # check if resource exists and if overwriting
299
+ if !params["id"].nil? && !@overwrite
300
+ resource = @ckan_client.get_resource(params["id"])
301
+
302
+ if !resource.nil?
303
+ ok = false
304
+ msg = "Resource already exists - skipping. Use parameter 'overwrite' to overwrite changes."
305
+ end
306
+ end
307
+
308
+ if ok
309
+ @ckan_client.create_or_update_resource(params, File.new(res[:path], 'rb'), true){ |response, status_ok|
310
+ ok = status_ok
311
+
312
+ if !ok
313
+ msg = "HTTP error #{response.code}"
314
+
315
+ log_error response.body
316
+ end
317
+ }
318
+ end
319
+
320
+ if !ok
321
+ res[:log].error msg
322
+ else
323
+ res[:log].info "OK - upload successfull"
324
+ end
325
+ res[:log].info "End of upload"
326
+
327
+ [ok, msg, exception]
328
+ end
329
+ end
330
+
331
+ show_info "End of upload"
332
+ end
333
+
334
+ def update_dataset()
335
+ puts ""
336
+ show_info "Starting dataset update..."
337
+
338
+ task("Updating dataset") do
339
+ ok = false
340
+ msg = nil
341
+ exception = nil
342
+
343
+ @ckan_client.update_package(@dataset_config["result"]["id"], @dataset_config["result"].clone, true){ |response, status_ok|
344
+ ok = status_ok
345
+
346
+ if !ok
347
+ msg = "HTTP error #{response.code}"
348
+
349
+ log_error response.body
350
+ end
351
+ }
352
+
353
+ [ok, msg, exception]
354
+ end
355
+ show_info "End of dataset update"
356
+ end
357
+
358
+ def return_error(message, exception = nil)
359
+ show_error(message, exception)
360
+
361
+ exit 1
362
+ end
363
+
364
+ def show_error(message, exception = nil)
365
+ if !exception.nil?
366
+ message = "#{message} - #{exception.message}"
367
+ end
368
+
369
+ puts " #{message} ".white.on_red
370
+
371
+ log_fatal message, exception
372
+ end
373
+
374
+ def show_info(message)
375
+ puts " #{message} "
376
+
377
+ log_info message
378
+ end
379
+
380
+ def task(message)
381
+ log_info message
382
+
383
+ spinner = TTY::Spinner.new("[:spinner] #{message}...")
384
+ spinner.auto_spin
385
+ ok, message, exception = yield
386
+ spinner.stop(ok ? "OK".green : "ERROR".red)
387
+
388
+ if !message.nil?
389
+ message = " #{message}"
390
+
391
+ if !exception.nil?
392
+ show_error message, exception
393
+ else
394
+ show_info message
395
+ end
396
+ end
397
+ end
398
+
399
+ def read_resource(source)
400
+ # check if URL or file
401
+ unless source =~ /^http(s)?/
402
+ source = File.new( source )
403
+ end
404
+
405
+ source
406
+ end
407
+
408
+ def read_schema(path)
409
+ schema = Csvlint::Schema.load_from_uri(path, false)
410
+
411
+ schema
412
+ end
413
+
414
+ def format_validation(error, schema = nil)
415
+ h = {
416
+ type: error.type,
417
+ category: error.category,
418
+ row: error.row,
419
+ col: error.column
420
+ }
421
+
422
+ if error.column && !schema.nil? && schema.class == Csvlint::Schema && schema.fields[error.column - 1] != nil
423
+ field = schema.fields[error.column - 1]
424
+ h[:header] = field.name
425
+ end
426
+
427
+ h
428
+ end
429
+
430
+ def log_fatal(message, exception = nil)
431
+ if !@logger.nil?
432
+ @logger.fatal message, exception
433
+ end
434
+ end
435
+
436
+ def log_info(message)
437
+ if !@logger.nil?
438
+ @logger.info message
439
+ end
440
+ end
441
+
442
+ def log_error(message)
443
+ if !@logger.nil?
444
+ @logger.error message
445
+ end
446
+ end
447
+ end
448
+ end
445
449
  end
data/lib/ckancli/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Ckancli
2
- VERSION = "1.0.0"
2
+ VERSION = "1.0.1"
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ckan_cli
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.0.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - datagovlv
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2019-08-07 00:00:00.000000000 Z
11
+ date: 2019-08-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: tty-color