spout 0.10.0.beta4 → 0.10.0.beta6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -1
- data/README.md +53 -0
- data/lib/spout/commands/deploy.rb +207 -0
- data/lib/spout/commands/graphs.rb +22 -3
- data/lib/spout/commands/images.rb +20 -3
- data/lib/spout/helpers/config_reader.rb +8 -2
- data/lib/spout/helpers/quietly.rb +27 -0
- data/lib/spout/helpers/send_file.rb +102 -0
- data/lib/spout/helpers/subject_loader.rb +3 -3
- data/lib/spout/version.rb +1 -1
- data/lib/spout.rb +9 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d2c5a33171e6f9d0ed9ea65595d6fa2cc846c522
|
4
|
+
data.tar.gz: b83e43ccf34a70eac9d4485861a40f4a0f2269dc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8c6f37201645cd745c38d14ad5599ee702d313e41dc6f27cea28883d9a21c7cdf616287f7340df1e2fd1588c621fdb43f3a0c52b584e5744d60a8fcf277d3ade
|
7
|
+
data.tar.gz: e5d504aa8ab9bf0acdb71e0d0ad5a17fbecf4889b5d577fb1f8ecbca10d21446af6dd0e17ad925db8950af1ea5783a8b8c83b15955f85aeb41c9f19469b4be3a
|
data/CHANGELOG.md
CHANGED
@@ -1,11 +1,14 @@
|
|
1
1
|
## 0.10.0
|
2
2
|
|
3
3
|
### Enhancements
|
4
|
+
- **Deploy Command**
|
5
|
+
- `spout deploy` command added that allows a data dictionary to be deployed to a web URL
|
6
|
+
- The webserver that is specified must respond to the web requests in a similar manner to nsrr/www.sleepdata.org
|
4
7
|
- **Import Command**
|
5
8
|
- `spout import` now changes variables with all caps `display_name` to use title case instead
|
6
9
|
- Display names that use mixed case are unaffected
|
7
10
|
- `spout import --domains` now marks options as `"missing": true` if the option value starts with a dot `.` or a dash `-`
|
8
|
-
- Display names for
|
11
|
+
- Display names for domain options are also changed to title case if they are all caps
|
9
12
|
- **Testing Changes**
|
10
13
|
- Tests now include checks to assure that variable display_name fields don't exceed 255 length requirement
|
11
14
|
- `include Spout::Tests::VariableDisplayNameLength`
|
data/README.md
CHANGED
@@ -327,3 +327,56 @@ This will generate charts and tables for each variable in the dataset plotted ag
|
|
327
327
|
}
|
328
328
|
```
|
329
329
|
|
330
|
+
### Deploy your data dictionary to a staging or production webserver
|
331
|
+
|
332
|
+
```
|
333
|
+
spout deploy NAME
|
334
|
+
```
|
335
|
+
|
336
|
+
This command pushes a tagged version of the data dictionary to a webserver specified in the `.spout.yml` file.
|
337
|
+
|
338
|
+
```
|
339
|
+
webservers:
|
340
|
+
- name: production
|
341
|
+
url: https://sleepdata.org
|
342
|
+
- name: staging
|
343
|
+
url: https://staging.sleepdata.org
|
344
|
+
```
|
345
|
+
|
346
|
+
Shorthand
|
347
|
+
|
348
|
+
**Deploy to Production**
|
349
|
+
```
|
350
|
+
spout d p
|
351
|
+
```
|
352
|
+
|
353
|
+
**Deploy to Staging**
|
354
|
+
```
|
355
|
+
spout d s
|
356
|
+
```
|
357
|
+
|
358
|
+
The following steps are run:
|
359
|
+
|
360
|
+
- **User Authorization**
|
361
|
+
- User authenticates via token, the user must be a dataset editor
|
362
|
+
- **Version Check**
|
363
|
+
- "v#{VERSION}" matches HEAD git tag annotation
|
364
|
+
- `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
365
|
+
- Git Repo should have zero uncommitted changes
|
366
|
+
- **Tests Pass**
|
367
|
+
- `spout t` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
368
|
+
- `spout c` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
369
|
+
- **Graph Generation**
|
370
|
+
- `spout g` is run
|
371
|
+
- Graphs are pushed to server
|
372
|
+
- **Image Generation**
|
373
|
+
- `spout p` is run
|
374
|
+
- `optipng` is run on image then uploaded to server
|
375
|
+
- Images are pushed to server
|
376
|
+
- **Dataset Uploads**
|
377
|
+
- Dataset CSV data dictionary is generated (variables, domains, forms)
|
378
|
+
- Dataset and data dictionary CSVs uploaded to files section of dataset
|
379
|
+
- **Server-Side Updates**
|
380
|
+
- Server checks out branch of specified tag
|
381
|
+
- Server runs `load_data_dictionary!` for specified dataset slug
|
382
|
+
- Server refreshes dataset folder to reflect new dataset and data dictionaries
|
@@ -0,0 +1,207 @@
|
|
1
|
+
require 'colorize'
|
2
|
+
require 'net/http'
|
3
|
+
|
4
|
+
require 'spout/helpers/config_reader'
|
5
|
+
require 'spout/helpers/quietly'
|
6
|
+
|
7
|
+
# - **User Authorization**
|
8
|
+
# - User authenticates via token, the user must be a dataset editor
|
9
|
+
# - **Version Check**
|
10
|
+
# - "v#{VERSION}" matches HEAD git tag annotation
|
11
|
+
# - `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
12
|
+
# - Git Repo should have zero uncommitted changes
|
13
|
+
# - **Tests Pass**
|
14
|
+
# - `spout t` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
15
|
+
# - `spout c` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
16
|
+
# - **Graph Generation**
|
17
|
+
# - `spout g` is run
|
18
|
+
# - Graphs are pushed to server
|
19
|
+
# - **Image Generation**
|
20
|
+
# - `spout p` is run
|
21
|
+
# - `optipng` is run on image then uploaded to server
|
22
|
+
# - Images are pushed to server
|
23
|
+
# - **Dataset Uploads**
|
24
|
+
# - Dataset CSV data dictionary is generated (variables, domains, forms)
|
25
|
+
# - Dataset and data dictionary CSVs uploaded to files section of dataset
|
26
|
+
# - **Server-Side Updates**
|
27
|
+
# - Server checks out branch of specified tag
|
28
|
+
# - Server runs `load_data_dictionary!` for specified dataset slug
|
29
|
+
# - Server refreshes dataset folder to reflect new dataset and data dictionaries
|
30
|
+
|
31
|
+
class DeployError < StandardError
|
32
|
+
end
|
33
|
+
|
34
|
+
module Spout
|
35
|
+
module Commands
|
36
|
+
class Deploy
|
37
|
+
|
38
|
+
include Spout::Helpers::Quietly
|
39
|
+
|
40
|
+
INDENT_LENGTH = 23
|
41
|
+
INDENT = " "*INDENT_LENGTH
|
42
|
+
|
43
|
+
attr_accessor :token, :version, :slug, :url, :config, :environment
|
44
|
+
|
45
|
+
def initialize(argv, version)
|
46
|
+
# puts "CODE GREEN INITIALIZED...".colorize(:green)
|
47
|
+
# puts "Deploying to server...".colorize(:red)
|
48
|
+
@environment = argv[1].to_s
|
49
|
+
@version = version
|
50
|
+
@skip_checks = false
|
51
|
+
run_all
|
52
|
+
end
|
53
|
+
|
54
|
+
def run_all
|
55
|
+
begin
|
56
|
+
config_file_load
|
57
|
+
version_check unless @skip_checks
|
58
|
+
test_check unless @skip_checks
|
59
|
+
user_authorization
|
60
|
+
graph_generation
|
61
|
+
image_generation
|
62
|
+
dataset_uploads
|
63
|
+
trigger_server_updates
|
64
|
+
rescue DeployError
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
def config_file_load
|
69
|
+
print " `.spout.yml` Check: "
|
70
|
+
@config = Spout::Helpers::ConfigReader.new
|
71
|
+
|
72
|
+
@slug = @config.slug
|
73
|
+
|
74
|
+
if @slug == ''
|
75
|
+
message = "#{INDENT}Please specify a dataset slug in your `.spout.yml` file!".colorize(:red) + " Ex:\n---\nslug: mydataset\n".colorize(:orange)
|
76
|
+
failure(message)
|
77
|
+
end
|
78
|
+
|
79
|
+
if @config.webservers.empty?
|
80
|
+
message = "#{INDENT}Please specify a webserver in your `.spout.yml` file!".colorize(:red) + " Ex:\n---\nwebservers:\n - name: production\n url: https://sleepdata.org\n - name: staging\n url: https://staging.sleepdata.org\n".colorize(:orange)
|
81
|
+
failure(message)
|
82
|
+
end
|
83
|
+
|
84
|
+
matching_webservers = @config.webservers.select{|wh| /^#{@environment}/i =~ wh['name'].to_s.downcase}
|
85
|
+
if matching_webservers.count == 0
|
86
|
+
message = "#{INDENT}0 webservers match '#{@environment}'.".colorize(:red) + " The following webservers exist in your `.spout.yml` file:\n" + "#{INDENT}#{@config.webservers.collect{|wh| wh['name'].to_s.downcase}.join(', ')}".colorize(:white)
|
87
|
+
failure(message)
|
88
|
+
elsif matching_webservers.count > 1
|
89
|
+
message = "#{INDENT}#{matching_webservers.count} webservers match '#{@environment}'.".colorize(:red) + " Did you mean one of the following?\n" + "#{INDENT}#{matching_webservers.collect{|wh| wh['name'].to_s.downcase}.join(', ')}".colorize(:white)
|
90
|
+
failure(message)
|
91
|
+
end
|
92
|
+
|
93
|
+
@url = URI.parse(matching_webservers.first['url'].to_s.strip) rescue @url = nil
|
94
|
+
|
95
|
+
if @url.to_s == ''
|
96
|
+
message = "#{INDENT}Invalid URL format for #{matching_webservers.first['name'].to_s.strip.downcase} webserver: ".colorize(:red) + "'#{matching_webservers.first['url'].to_s.strip}'".colorize(:white)
|
97
|
+
failure(message)
|
98
|
+
end
|
99
|
+
|
100
|
+
puts "PASS".colorize(:green)
|
101
|
+
puts " Target Server: " + "#{@url}".colorize(:white)
|
102
|
+
puts " Target Dataset: " + "#{@slug}".colorize(:white)
|
103
|
+
end
|
104
|
+
|
105
|
+
# - **Version Check**
|
106
|
+
# - Git Repo should have zero uncommitted changes
|
107
|
+
# - `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
108
|
+
# - "v#{VERSION}" matches HEAD git tag annotation
|
109
|
+
def version_check
|
110
|
+
stdout = quietly do
|
111
|
+
`git status --porcelain`
|
112
|
+
end
|
113
|
+
|
114
|
+
print " Git Status Check: "
|
115
|
+
if stdout.to_s.strip == ''
|
116
|
+
puts "PASS".colorize(:green) + " " + "nothing to commit, working directory clean".colorize(:white)
|
117
|
+
else
|
118
|
+
message = "#{INDENT}working directory contains uncomitted changes".colorize(:red)
|
119
|
+
failure message
|
120
|
+
end
|
121
|
+
|
122
|
+
changelog = File.open('CHANGELOG.md', &:readline).strip rescue changelog = ''
|
123
|
+
if changelog.match(/^## #{@version.split('.')[0..2].join('.')}/)
|
124
|
+
puts " CHANGELOG.md: " + "PASS".colorize(:green) + " " + changelog.colorize(:white)
|
125
|
+
else
|
126
|
+
print " CHANGELOG.md: "
|
127
|
+
message = "#{INDENT}Expected: ".colorize(:red) + "## #{@version}".colorize(:white) +
|
128
|
+
"\n#{INDENT} Actual: ".colorize(:red) + changelog.colorize(:white)
|
129
|
+
failure message
|
130
|
+
end
|
131
|
+
|
132
|
+
stdout = quietly do
|
133
|
+
`git describe --exact-match HEAD`
|
134
|
+
end
|
135
|
+
|
136
|
+
print " Version Check: "
|
137
|
+
tag = stdout.to_s.strip
|
138
|
+
if "v#{@version}" != tag
|
139
|
+
message = "#{INDENT}Version specified in `VERSION` file ".colorize(:red) + "'v#{@version}'".colorize(:white) + " does not match git tag on HEAD commit ".colorize(:red) + "'#{tag}'".colorize(:white)
|
140
|
+
failure message
|
141
|
+
else
|
142
|
+
puts "PASS".colorize(:green) + " VERSION " + "'v#{@version}'".colorize(:white) + " matches git tag " + "'#{tag}'".colorize(:white)
|
143
|
+
end
|
144
|
+
|
145
|
+
end
|
146
|
+
|
147
|
+
def test_check
|
148
|
+
print " Spout Tests: "
|
149
|
+
|
150
|
+
stdout = quietly do
|
151
|
+
`spout t`
|
152
|
+
end
|
153
|
+
|
154
|
+
if stdout.match(/[^\d]0 failures, 0 errors,/)
|
155
|
+
puts "PASS".colorize(:green)
|
156
|
+
else
|
157
|
+
message = "#{INDENT}spout t".colorize(:white) + " had errors or failures".colorize(:red) + "\n#{INDENT}Please fix all errors and failures and then run spout deploy again."
|
158
|
+
failure message
|
159
|
+
end
|
160
|
+
|
161
|
+
puts " Spout Coverage: " + "SKIP".colorize(:blue)
|
162
|
+
end
|
163
|
+
|
164
|
+
def user_authorization
|
165
|
+
puts " Get your token here: " + "#{@url}/token".colorize(:blue).on_white.underline
|
166
|
+
print " Enter your token: "
|
167
|
+
@token = STDIN.gets.chomp
|
168
|
+
# failure ''
|
169
|
+
# puts "PASS".colorize(:green)
|
170
|
+
end
|
171
|
+
|
172
|
+
def graph_generation
|
173
|
+
# failure ''
|
174
|
+
require 'spout/commands/graphs'
|
175
|
+
Spout::Commands::Graphs.new([], @version, true, @url, @slug, @token)
|
176
|
+
puts "\r Graph Generation: " + "DONE ".colorize(:green)
|
177
|
+
end
|
178
|
+
|
179
|
+
def image_generation
|
180
|
+
# failure ''
|
181
|
+
require 'spout/commands/images'
|
182
|
+
Spout::Commands::Images.new([], [], [], @version, [], true, @url, @slug, @token)
|
183
|
+
puts "\r Image Generation: " + "DONE ".colorize(:green)
|
184
|
+
end
|
185
|
+
|
186
|
+
def dataset_uploads
|
187
|
+
print " Dataset Uploads: "
|
188
|
+
# failure ''
|
189
|
+
# puts "PASS".colorize(:green)
|
190
|
+
puts "SKIP".colorize(:blue)
|
191
|
+
end
|
192
|
+
|
193
|
+
def trigger_server_updates
|
194
|
+
print "Launch Server Scripts: "
|
195
|
+
# failure ''
|
196
|
+
# puts "PASS".colorize(:green)
|
197
|
+
puts "SKIP".colorize(:blue)
|
198
|
+
end
|
199
|
+
|
200
|
+
def failure(message)
|
201
|
+
puts "FAIL".colorize(:red)
|
202
|
+
puts message
|
203
|
+
raise DeployError
|
204
|
+
end
|
205
|
+
end
|
206
|
+
end
|
207
|
+
end
|
@@ -3,16 +3,22 @@ require 'fileutils'
|
|
3
3
|
require 'rubygems'
|
4
4
|
require 'json'
|
5
5
|
require 'yaml'
|
6
|
+
require 'colorize'
|
6
7
|
|
7
8
|
require 'spout/helpers/subject_loader'
|
8
9
|
require 'spout/helpers/chart_types'
|
9
10
|
require 'spout/helpers/config_reader'
|
11
|
+
require 'spout/helpers/send_file'
|
10
12
|
|
11
13
|
module Spout
|
12
14
|
module Commands
|
13
15
|
class Graphs
|
14
|
-
def initialize(variables, standard_version)
|
16
|
+
def initialize(variables, standard_version, deploy_mode = false, url = '', slug = '', token = '')
|
17
|
+
@deploy_mode = deploy_mode
|
18
|
+
@url = url
|
15
19
|
@standard_version = standard_version
|
20
|
+
@slug = slug
|
21
|
+
@token = token
|
16
22
|
|
17
23
|
@config = Spout::Helpers::ConfigReader.new
|
18
24
|
|
@@ -54,7 +60,7 @@ module Spout
|
|
54
60
|
@subjects = @subject_loader.subjects
|
55
61
|
compute_tables_and_charts
|
56
62
|
|
57
|
-
puts "Took #{Time.now - t} seconds." if @subjects.size > 0
|
63
|
+
puts "Took #{Time.now - t} seconds." if @subjects.size > 0 and not @deploy_mode
|
58
64
|
end
|
59
65
|
|
60
66
|
def compute_tables_and_charts
|
@@ -67,7 +73,12 @@ module Spout
|
|
67
73
|
variable_name = json['id'].to_s.downcase
|
68
74
|
next unless Spout::Models::Subject.method_defined?(variable_name)
|
69
75
|
|
70
|
-
|
76
|
+
if @deploy_mode
|
77
|
+
print "\r Graph Generation: " + "#{"% 3d" % ((file_index+1)*100/variable_files_count)}% Uploaded".colorize(:white)
|
78
|
+
else
|
79
|
+
puts "#{file_index+1} of #{variable_files_count}: #{variable_file.gsub(/(^variables\/|\.json$)/, '').gsub('/', ' / ')}"
|
80
|
+
end
|
81
|
+
|
71
82
|
|
72
83
|
|
73
84
|
stats = {
|
@@ -101,9 +112,17 @@ module Spout
|
|
101
112
|
chart_json_file = File.join('graphs', @standard_version, "#{json['id']}.json")
|
102
113
|
File.open(chart_json_file, 'w') { |file| file.write( JSON.pretty_generate(stats) + "\n" ) }
|
103
114
|
|
115
|
+
if @deploy_mode
|
116
|
+
send_to_server(chart_json_file)
|
117
|
+
end
|
118
|
+
|
104
119
|
end
|
105
120
|
end
|
106
121
|
|
122
|
+
def send_to_server(chart_json_file)
|
123
|
+
response = Spout::Helpers::SendFile.post("#{@url}/datasets/#{@slug}/upload_graph.json", chart_json_file, @standard_version, @token)
|
124
|
+
end
|
125
|
+
|
107
126
|
# [["Visit 1", "1"], ["Visit 2", "2"], ["CVD Outcomes", "3"]]
|
108
127
|
def visits
|
109
128
|
@visits ||= begin
|
@@ -7,12 +7,20 @@ require 'yaml'
|
|
7
7
|
require 'spout/helpers/subject_loader'
|
8
8
|
require 'spout/helpers/chart_types'
|
9
9
|
require 'spout/helpers/config_reader'
|
10
|
+
require 'spout/helpers/send_file'
|
10
11
|
|
11
12
|
module Spout
|
12
13
|
module Commands
|
13
14
|
class Images
|
14
15
|
|
15
|
-
def initialize(types, variable_ids, sizes, standard_version, argv)
|
16
|
+
def initialize(types, variable_ids, sizes, standard_version, argv, deploy_mode = false, url = '', slug = '', token = '')
|
17
|
+
@deploy_mode = deploy_mode
|
18
|
+
@url = url
|
19
|
+
@standard_version = standard_version
|
20
|
+
@slug = slug
|
21
|
+
@token = token
|
22
|
+
|
23
|
+
|
16
24
|
@variable_files = Dir.glob('variables/**/*.json')
|
17
25
|
@standard_version = standard_version
|
18
26
|
@pretend = (argv.delete('--pretend') != nil)
|
@@ -34,7 +42,7 @@ module Spout
|
|
34
42
|
@subjects = @subject_loader.subjects
|
35
43
|
|
36
44
|
compute_images
|
37
|
-
puts "Took #{Time.now - t} seconds." if @subjects.size > 0
|
45
|
+
puts "Took #{Time.now - t} seconds." if @subjects.size > 0 and not @deploy_mode
|
38
46
|
end
|
39
47
|
|
40
48
|
def compute_images
|
@@ -53,7 +61,11 @@ module Spout
|
|
53
61
|
variable_name = json['id'].to_s.downcase
|
54
62
|
next unless Spout::Models::Subject.method_defined?(variable_name)
|
55
63
|
|
56
|
-
|
64
|
+
if @deploy_mode
|
65
|
+
print "\r Image Generation: " + "#{"% 3d" % ((file_index+1)*100/variable_files_count)}% Uploaded".colorize(:white)
|
66
|
+
else
|
67
|
+
puts "#{file_index+1} of #{variable_files_count}: #{variable_file.gsub(/(^variables\/|\.json$)/, '').gsub('/', ' / ')}"
|
68
|
+
end
|
57
69
|
|
58
70
|
filtered_subjects = @subjects.select{ |s| s.send(@config.visit) != nil }
|
59
71
|
|
@@ -115,8 +127,13 @@ module Spout
|
|
115
127
|
puts phantomjs_command
|
116
128
|
else
|
117
129
|
`#{phantomjs_command}`
|
130
|
+
|
131
|
+
send_to_server(graph_path) if @deploy_mode
|
118
132
|
end
|
133
|
+
end
|
119
134
|
|
135
|
+
def send_to_server(file)
|
136
|
+
response = Spout::Helpers::SendFile.post("#{@url}/datasets/#{@slug}/upload_graph.json", file, @standard_version, @token, 'images')
|
120
137
|
end
|
121
138
|
|
122
139
|
end
|
@@ -4,13 +4,13 @@ module Spout
|
|
4
4
|
module Helpers
|
5
5
|
class ConfigReader
|
6
6
|
|
7
|
-
attr_reader :slug, :visit, :charts
|
8
|
-
|
7
|
+
attr_reader :slug, :visit, :charts, :webservers
|
9
8
|
|
10
9
|
def initialize
|
11
10
|
@slug = ''
|
12
11
|
@visit = ''
|
13
12
|
@charts = []
|
13
|
+
@webservers = []
|
14
14
|
parse_yaml_file
|
15
15
|
end
|
16
16
|
|
@@ -26,6 +26,12 @@ module Spout
|
|
26
26
|
else
|
27
27
|
[]
|
28
28
|
end
|
29
|
+
|
30
|
+
@webservers = if spout_config['webservers'].kind_of?(Array)
|
31
|
+
spout_config['webservers'].select{|c| c.kind_of?(Hash)}
|
32
|
+
else
|
33
|
+
[]
|
34
|
+
end
|
29
35
|
else
|
30
36
|
puts "The YAML file needs to be in the following format:"
|
31
37
|
puts "---\nvisit: visit_variable_name\ncharts:\n- chart: age_variable_name\n title: Age\n- chart: gender_variable_name\n title: Gender\n- chart: race_variable_name\n title: Race\n"
|
@@ -0,0 +1,27 @@
|
|
1
|
+
|
2
|
+
module Spout
|
3
|
+
module Helpers
|
4
|
+
module Quietly
|
5
|
+
|
6
|
+
# From Rails: http://apidock.com/rails/v3.2.13/Kernel/silence_stream
|
7
|
+
def silence_stream(stream)
|
8
|
+
old_stream = stream.dup
|
9
|
+
stream.reopen(RbConfig::CONFIG['host_os'] =~ /mswin|mingw/ ? 'NUL:' : '/dev/null')
|
10
|
+
stream.sync = true
|
11
|
+
yield
|
12
|
+
ensure
|
13
|
+
stream.reopen(old_stream)
|
14
|
+
end
|
15
|
+
|
16
|
+
# From Rails: http://apidock.com/rails/v3.2.13/Kernel/quietly
|
17
|
+
def quietly
|
18
|
+
silence_stream(STDOUT) do
|
19
|
+
silence_stream(STDERR) do
|
20
|
+
yield
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,102 @@
|
|
1
|
+
require 'openssl'
|
2
|
+
require 'net/http'
|
3
|
+
require 'json'
|
4
|
+
|
5
|
+
module Spout
|
6
|
+
module Helpers
|
7
|
+
class SendFile
|
8
|
+
class << self
|
9
|
+
def post(*args)
|
10
|
+
new(*args).post
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
attr_reader :url
|
15
|
+
|
16
|
+
def initialize(url, filename, version, token, type = nil)
|
17
|
+
|
18
|
+
@params = {}
|
19
|
+
@params["version"] = version
|
20
|
+
@params["auth_token"] = token if token
|
21
|
+
@params["type"] = type if type
|
22
|
+
begin
|
23
|
+
file = File.open(filename, "rb")
|
24
|
+
@params["file"] = file
|
25
|
+
|
26
|
+
mp = Multipart::MultipartPost.new
|
27
|
+
@query, @headers = mp.prepare_query(@params)
|
28
|
+
ensure
|
29
|
+
file.close if file
|
30
|
+
end
|
31
|
+
|
32
|
+
begin
|
33
|
+
@url = URI.parse(url)
|
34
|
+
|
35
|
+
@http = Net::HTTP.new(@url.host, @url.port)
|
36
|
+
if @url.scheme == 'https'
|
37
|
+
@http.use_ssl = true
|
38
|
+
@http.verify_mode = OpenSSL::SSL::VERIFY_NONE
|
39
|
+
end
|
40
|
+
rescue
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def post
|
45
|
+
begin
|
46
|
+
response = @http.start do |http|
|
47
|
+
http.post(@url.path, @query, @headers)
|
48
|
+
end
|
49
|
+
JSON.parse(response.body)
|
50
|
+
rescue
|
51
|
+
nil
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
|
59
|
+
module Multipart
|
60
|
+
class Param
|
61
|
+
attr_accessor :k, :v
|
62
|
+
def initialize( k, v )
|
63
|
+
@k = k
|
64
|
+
@v = v
|
65
|
+
end
|
66
|
+
|
67
|
+
def to_multipart
|
68
|
+
return "Content-Disposition: form-data; name=\"#{k}\"\r\n\r\n#{v}\r\n"
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
class FileParam
|
73
|
+
attr_accessor :k, :filename, :content
|
74
|
+
def initialize( k, filename, content )
|
75
|
+
@k = k
|
76
|
+
@filename = filename
|
77
|
+
@content = content
|
78
|
+
end
|
79
|
+
|
80
|
+
def to_multipart
|
81
|
+
mime_type = 'application/octet-stream'
|
82
|
+
return "Content-Disposition: form-data; name=\"#{k}\"; filename=\"#{filename}\"\r\n" + "Content-Transfer-Encoding: binary\r\n" + "Content-Type: #{mime_type}\r\n\r\n" + content + "\r\n"
|
83
|
+
end
|
84
|
+
end
|
85
|
+
class MultipartPost
|
86
|
+
BOUNDARY = 'a#41-93r1-^Õ-rule0000'
|
87
|
+
HEADER = {"Content-type" => "multipart/form-data, boundary=" + BOUNDARY + " "}
|
88
|
+
|
89
|
+
def prepare_query (params)
|
90
|
+
fp = []
|
91
|
+
params.each {|k,v|
|
92
|
+
if v.respond_to?(:read)
|
93
|
+
fp.push(FileParam.new(k, v.path, v.read))
|
94
|
+
else
|
95
|
+
fp.push(Param.new(k,v))
|
96
|
+
end
|
97
|
+
}
|
98
|
+
query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
|
99
|
+
return query, HEADER
|
100
|
+
end
|
101
|
+
end
|
102
|
+
end
|
@@ -41,11 +41,11 @@ module Spout
|
|
41
41
|
@csv_files = Dir.glob("csvs/#{@csv_directory}/*.csv")
|
42
42
|
@csv_files.each_with_index do |csv_file, index|
|
43
43
|
count = 0
|
44
|
-
|
44
|
+
print "\nParsing #{csv_file}"
|
45
45
|
CSV.parse( File.open(csv_file, 'r:iso-8859-1:utf-8'){|f| f.read}, headers: true, header_converters: lambda { |h| h.to_s.downcase } ) do |line|
|
46
46
|
row = line.to_hash
|
47
47
|
count += 1
|
48
|
-
print
|
48
|
+
print "\rParsing #{csv_file} - line ##{count}" if (count % 10 == 0)
|
49
49
|
@subjects << Spout::Models::Subject.create do |t|
|
50
50
|
t._visit = row[@visit]
|
51
51
|
|
@@ -68,7 +68,7 @@ module Spout
|
|
68
68
|
# puts "Memory Used: " + (`ps -o rss -p #{$$}`.strip.split.last.to_i / 1024).to_s + " MB" if count % 1000 == 0
|
69
69
|
break if @number_of_rows != nil and count >= @number_of_rows
|
70
70
|
end
|
71
|
-
puts "\n
|
71
|
+
puts "\n"
|
72
72
|
end
|
73
73
|
|
74
74
|
if @csv_directory != @standard_version
|
data/lib/spout/version.rb
CHANGED
data/lib/spout.rb
CHANGED
@@ -11,7 +11,8 @@ Spout::COMMANDS = {
|
|
11
11
|
'c' => :coverage_report,
|
12
12
|
'p' => :generate_images,
|
13
13
|
'g' => :generate_charts_and_tables,
|
14
|
-
'o' => :outliers_report
|
14
|
+
'o' => :outliers_report,
|
15
|
+
'd' => :deploy
|
15
16
|
}
|
16
17
|
|
17
18
|
module Spout
|
@@ -72,6 +73,8 @@ The most common spout commands are:
|
|
72
73
|
[g]raphs Generates JSON graphs for each variable
|
73
74
|
in a dataset and places them
|
74
75
|
in `<project_name>/graphs/<version>/`
|
76
|
+
[d]eploy NAME Push dataset and data dictionary to a
|
77
|
+
webserver specified in `.spout.yml`
|
75
78
|
[v]ersion Returns the version of Spout
|
76
79
|
|
77
80
|
Commands can be referenced by the first letter:
|
@@ -80,6 +83,11 @@ Commands can be referenced by the first letter:
|
|
80
83
|
EOT
|
81
84
|
end
|
82
85
|
|
86
|
+
def self.deploy(argv)
|
87
|
+
require 'spout/commands/deploy'
|
88
|
+
Spout::Commands::Deploy.new(argv, standard_version)
|
89
|
+
end
|
90
|
+
|
83
91
|
def self.importer(argv)
|
84
92
|
require 'spout/commands/importer'
|
85
93
|
Spout::Commands::Importer.new(argv)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: spout
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.10.0.
|
4
|
+
version: 0.10.0.beta6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Remo Mueller
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-10-
|
11
|
+
date: 2014-10-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rake
|
@@ -110,6 +110,7 @@ files:
|
|
110
110
|
- bin/spout
|
111
111
|
- lib/spout.rb
|
112
112
|
- lib/spout/commands/coverage.rb
|
113
|
+
- lib/spout/commands/deploy.rb
|
113
114
|
- lib/spout/commands/exporter.rb
|
114
115
|
- lib/spout/commands/graphs.rb
|
115
116
|
- lib/spout/commands/images.rb
|
@@ -122,7 +123,9 @@ files:
|
|
122
123
|
- lib/spout/helpers/iterators.rb
|
123
124
|
- lib/spout/helpers/json_loader.rb
|
124
125
|
- lib/spout/helpers/number_helper.rb
|
126
|
+
- lib/spout/helpers/quietly.rb
|
125
127
|
- lib/spout/helpers/semantic.rb
|
128
|
+
- lib/spout/helpers/send_file.rb
|
126
129
|
- lib/spout/helpers/subject_loader.rb
|
127
130
|
- lib/spout/helpers/table_formatting.rb
|
128
131
|
- lib/spout/models/coverage_result.rb
|