spout 0.10.0.beta4 → 0.10.0.beta6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +4 -1
- data/README.md +53 -0
- data/lib/spout/commands/deploy.rb +207 -0
- data/lib/spout/commands/graphs.rb +22 -3
- data/lib/spout/commands/images.rb +20 -3
- data/lib/spout/helpers/config_reader.rb +8 -2
- data/lib/spout/helpers/quietly.rb +27 -0
- data/lib/spout/helpers/send_file.rb +102 -0
- data/lib/spout/helpers/subject_loader.rb +3 -3
- data/lib/spout/version.rb +1 -1
- data/lib/spout.rb +9 -1
- metadata +5 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: d2c5a33171e6f9d0ed9ea65595d6fa2cc846c522
|
4
|
+
data.tar.gz: b83e43ccf34a70eac9d4485861a40f4a0f2269dc
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8c6f37201645cd745c38d14ad5599ee702d313e41dc6f27cea28883d9a21c7cdf616287f7340df1e2fd1588c621fdb43f3a0c52b584e5744d60a8fcf277d3ade
|
7
|
+
data.tar.gz: e5d504aa8ab9bf0acdb71e0d0ad5a17fbecf4889b5d577fb1f8ecbca10d21446af6dd0e17ad925db8950af1ea5783a8b8c83b15955f85aeb41c9f19469b4be3a
|
data/CHANGELOG.md
CHANGED
@@ -1,11 +1,14 @@
|
|
1
1
|
## 0.10.0
|
2
2
|
|
3
3
|
### Enhancements
|
4
|
+
- **Deploy Command**
|
5
|
+
- `spout deploy` command added that allows a data dictionary to be deployed to a web URL
|
6
|
+
- The webserver that is specified must respond to the web requests in a similar manner to nsrr/www.sleepdata.org
|
4
7
|
- **Import Command**
|
5
8
|
- `spout import` now changes variables with all caps `display_name` to use title case instead
|
6
9
|
- Display names that use mixed case are unaffected
|
7
10
|
- `spout import --domains` now marks options as `"missing": true` if the option value starts with a dot `.` or a dash `-`
|
8
|
-
- Display names for
|
11
|
+
- Display names for domain options are also changed to title case if they are all caps
|
9
12
|
- **Testing Changes**
|
10
13
|
- Tests now include checks to assure that variable display_name fields don't exceed 255 length requirement
|
11
14
|
- `include Spout::Tests::VariableDisplayNameLength`
|
data/README.md
CHANGED
@@ -327,3 +327,56 @@ This will generate charts and tables for each variable in the dataset plotted ag
|
|
327
327
|
}
|
328
328
|
```
|
329
329
|
|
330
|
+
### Deploy your data dictionary to a staging or production webserver
|
331
|
+
|
332
|
+
```
|
333
|
+
spout deploy NAME
|
334
|
+
```
|
335
|
+
|
336
|
+
This command pushes a tagged version of the data dictionary to a webserver specified in the `.spout.yml` file.
|
337
|
+
|
338
|
+
```
|
339
|
+
webservers:
|
340
|
+
- name: production
|
341
|
+
url: https://sleepdata.org
|
342
|
+
- name: staging
|
343
|
+
url: https://staging.sleepdata.org
|
344
|
+
```
|
345
|
+
|
346
|
+
Shorthand
|
347
|
+
|
348
|
+
**Deploy to Production**
|
349
|
+
```
|
350
|
+
spout d p
|
351
|
+
```
|
352
|
+
|
353
|
+
**Deploy to Staging**
|
354
|
+
```
|
355
|
+
spout d s
|
356
|
+
```
|
357
|
+
|
358
|
+
The following steps are run:
|
359
|
+
|
360
|
+
- **User Authorization**
|
361
|
+
- User authenticates via token, the user must be a dataset editor
|
362
|
+
- **Version Check**
|
363
|
+
- "v#{VERSION}" matches HEAD git tag annotation
|
364
|
+
- `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
365
|
+
- Git Repo should have zero uncommitted changes
|
366
|
+
- **Tests Pass**
|
367
|
+
- `spout t` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
368
|
+
- `spout c` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
369
|
+
- **Graph Generation**
|
370
|
+
- `spout g` is run
|
371
|
+
- Graphs are pushed to server
|
372
|
+
- **Image Generation**
|
373
|
+
- `spout p` is run
|
374
|
+
- `optipng` is run on image then uploaded to server
|
375
|
+
- Images are pushed to server
|
376
|
+
- **Dataset Uploads**
|
377
|
+
- Dataset CSV data dictionary is generated (variables, domains, forms)
|
378
|
+
- Dataset and data dictionary CSVs uploaded to files section of dataset
|
379
|
+
- **Server-Side Updates**
|
380
|
+
- Server checks out branch of specified tag
|
381
|
+
- Server runs `load_data_dictionary!` for specified dataset slug
|
382
|
+
- Server refreshes dataset folder to reflect new dataset and data dictionaries
|
@@ -0,0 +1,207 @@
|
|
1
|
+
require 'colorize'
|
2
|
+
require 'net/http'
|
3
|
+
|
4
|
+
require 'spout/helpers/config_reader'
|
5
|
+
require 'spout/helpers/quietly'
|
6
|
+
|
7
|
+
# - **User Authorization**
|
8
|
+
# - User authenticates via token, the user must be a dataset editor
|
9
|
+
# - **Version Check**
|
10
|
+
# - "v#{VERSION}" matches HEAD git tag annotation
|
11
|
+
# - `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
12
|
+
# - Git Repo should have zero uncommitted changes
|
13
|
+
# - **Tests Pass**
|
14
|
+
# - `spout t` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
15
|
+
# - `spout c` passes for RC and FINAL versions (Include .rc, does not include .beta)
|
16
|
+
# - **Graph Generation**
|
17
|
+
# - `spout g` is run
|
18
|
+
# - Graphs are pushed to server
|
19
|
+
# - **Image Generation**
|
20
|
+
# - `spout p` is run
|
21
|
+
# - `optipng` is run on image then uploaded to server
|
22
|
+
# - Images are pushed to server
|
23
|
+
# - **Dataset Uploads**
|
24
|
+
# - Dataset CSV data dictionary is generated (variables, domains, forms)
|
25
|
+
# - Dataset and data dictionary CSVs uploaded to files section of dataset
|
26
|
+
# - **Server-Side Updates**
|
27
|
+
# - Server checks out branch of specified tag
|
28
|
+
# - Server runs `load_data_dictionary!` for specified dataset slug
|
29
|
+
# - Server refreshes dataset folder to reflect new dataset and data dictionaries
|
30
|
+
|
31
|
+
class DeployError < StandardError
|
32
|
+
end
|
33
|
+
|
34
|
+
module Spout
|
35
|
+
module Commands
|
36
|
+
class Deploy
|
37
|
+
|
38
|
+
include Spout::Helpers::Quietly
|
39
|
+
|
40
|
+
INDENT_LENGTH = 23
|
41
|
+
INDENT = " "*INDENT_LENGTH
|
42
|
+
|
43
|
+
attr_accessor :token, :version, :slug, :url, :config, :environment
|
44
|
+
|
45
|
+
def initialize(argv, version)
|
46
|
+
# puts "CODE GREEN INITIALIZED...".colorize(:green)
|
47
|
+
# puts "Deploying to server...".colorize(:red)
|
48
|
+
@environment = argv[1].to_s
|
49
|
+
@version = version
|
50
|
+
@skip_checks = false
|
51
|
+
run_all
|
52
|
+
end
|
53
|
+
|
54
|
+
def run_all
|
55
|
+
begin
|
56
|
+
config_file_load
|
57
|
+
version_check unless @skip_checks
|
58
|
+
test_check unless @skip_checks
|
59
|
+
user_authorization
|
60
|
+
graph_generation
|
61
|
+
image_generation
|
62
|
+
dataset_uploads
|
63
|
+
trigger_server_updates
|
64
|
+
rescue DeployError
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
def config_file_load
|
69
|
+
print " `.spout.yml` Check: "
|
70
|
+
@config = Spout::Helpers::ConfigReader.new
|
71
|
+
|
72
|
+
@slug = @config.slug
|
73
|
+
|
74
|
+
if @slug == ''
|
75
|
+
message = "#{INDENT}Please specify a dataset slug in your `.spout.yml` file!".colorize(:red) + " Ex:\n---\nslug: mydataset\n".colorize(:orange)
|
76
|
+
failure(message)
|
77
|
+
end
|
78
|
+
|
79
|
+
if @config.webservers.empty?
|
80
|
+
message = "#{INDENT}Please specify a webserver in your `.spout.yml` file!".colorize(:red) + " Ex:\n---\nwebservers:\n - name: production\n url: https://sleepdata.org\n - name: staging\n url: https://staging.sleepdata.org\n".colorize(:orange)
|
81
|
+
failure(message)
|
82
|
+
end
|
83
|
+
|
84
|
+
matching_webservers = @config.webservers.select{|wh| /^#{@environment}/i =~ wh['name'].to_s.downcase}
|
85
|
+
if matching_webservers.count == 0
|
86
|
+
message = "#{INDENT}0 webservers match '#{@environment}'.".colorize(:red) + " The following webservers exist in your `.spout.yml` file:\n" + "#{INDENT}#{@config.webservers.collect{|wh| wh['name'].to_s.downcase}.join(', ')}".colorize(:white)
|
87
|
+
failure(message)
|
88
|
+
elsif matching_webservers.count > 1
|
89
|
+
message = "#{INDENT}#{matching_webservers.count} webservers match '#{@environment}'.".colorize(:red) + " Did you mean one of the following?\n" + "#{INDENT}#{matching_webservers.collect{|wh| wh['name'].to_s.downcase}.join(', ')}".colorize(:white)
|
90
|
+
failure(message)
|
91
|
+
end
|
92
|
+
|
93
|
+
@url = URI.parse(matching_webservers.first['url'].to_s.strip) rescue @url = nil
|
94
|
+
|
95
|
+
if @url.to_s == ''
|
96
|
+
message = "#{INDENT}Invalid URL format for #{matching_webservers.first['name'].to_s.strip.downcase} webserver: ".colorize(:red) + "'#{matching_webservers.first['url'].to_s.strip}'".colorize(:white)
|
97
|
+
failure(message)
|
98
|
+
end
|
99
|
+
|
100
|
+
puts "PASS".colorize(:green)
|
101
|
+
puts " Target Server: " + "#{@url}".colorize(:white)
|
102
|
+
puts " Target Dataset: " + "#{@slug}".colorize(:white)
|
103
|
+
end
|
104
|
+
|
105
|
+
# - **Version Check**
|
106
|
+
# - Git Repo should have zero uncommitted changes
|
107
|
+
# - `CHANGELOG.md` top line should include version, ex: `## 0.1.0`
|
108
|
+
# - "v#{VERSION}" matches HEAD git tag annotation
|
109
|
+
def version_check
|
110
|
+
stdout = quietly do
|
111
|
+
`git status --porcelain`
|
112
|
+
end
|
113
|
+
|
114
|
+
print " Git Status Check: "
|
115
|
+
if stdout.to_s.strip == ''
|
116
|
+
puts "PASS".colorize(:green) + " " + "nothing to commit, working directory clean".colorize(:white)
|
117
|
+
else
|
118
|
+
message = "#{INDENT}working directory contains uncomitted changes".colorize(:red)
|
119
|
+
failure message
|
120
|
+
end
|
121
|
+
|
122
|
+
changelog = File.open('CHANGELOG.md', &:readline).strip rescue changelog = ''
|
123
|
+
if changelog.match(/^## #{@version.split('.')[0..2].join('.')}/)
|
124
|
+
puts " CHANGELOG.md: " + "PASS".colorize(:green) + " " + changelog.colorize(:white)
|
125
|
+
else
|
126
|
+
print " CHANGELOG.md: "
|
127
|
+
message = "#{INDENT}Expected: ".colorize(:red) + "## #{@version}".colorize(:white) +
|
128
|
+
"\n#{INDENT} Actual: ".colorize(:red) + changelog.colorize(:white)
|
129
|
+
failure message
|
130
|
+
end
|
131
|
+
|
132
|
+
stdout = quietly do
|
133
|
+
`git describe --exact-match HEAD`
|
134
|
+
end
|
135
|
+
|
136
|
+
print " Version Check: "
|
137
|
+
tag = stdout.to_s.strip
|
138
|
+
if "v#{@version}" != tag
|
139
|
+
message = "#{INDENT}Version specified in `VERSION` file ".colorize(:red) + "'v#{@version}'".colorize(:white) + " does not match git tag on HEAD commit ".colorize(:red) + "'#{tag}'".colorize(:white)
|
140
|
+
failure message
|
141
|
+
else
|
142
|
+
puts "PASS".colorize(:green) + " VERSION " + "'v#{@version}'".colorize(:white) + " matches git tag " + "'#{tag}'".colorize(:white)
|
143
|
+
end
|
144
|
+
|
145
|
+
end
|
146
|
+
|
147
|
+
def test_check
|
148
|
+
print " Spout Tests: "
|
149
|
+
|
150
|
+
stdout = quietly do
|
151
|
+
`spout t`
|
152
|
+
end
|
153
|
+
|
154
|
+
if stdout.match(/[^\d]0 failures, 0 errors,/)
|
155
|
+
puts "PASS".colorize(:green)
|
156
|
+
else
|
157
|
+
message = "#{INDENT}spout t".colorize(:white) + " had errors or failures".colorize(:red) + "\n#{INDENT}Please fix all errors and failures and then run spout deploy again."
|
158
|
+
failure message
|
159
|
+
end
|
160
|
+
|
161
|
+
puts " Spout Coverage: " + "SKIP".colorize(:blue)
|
162
|
+
end
|
163
|
+
|
164
|
+
def user_authorization
|
165
|
+
puts " Get your token here: " + "#{@url}/token".colorize(:blue).on_white.underline
|
166
|
+
print " Enter your token: "
|
167
|
+
@token = STDIN.gets.chomp
|
168
|
+
# failure ''
|
169
|
+
# puts "PASS".colorize(:green)
|
170
|
+
end
|
171
|
+
|
172
|
+
def graph_generation
|
173
|
+
# failure ''
|
174
|
+
require 'spout/commands/graphs'
|
175
|
+
Spout::Commands::Graphs.new([], @version, true, @url, @slug, @token)
|
176
|
+
puts "\r Graph Generation: " + "DONE ".colorize(:green)
|
177
|
+
end
|
178
|
+
|
179
|
+
def image_generation
|
180
|
+
# failure ''
|
181
|
+
require 'spout/commands/images'
|
182
|
+
Spout::Commands::Images.new([], [], [], @version, [], true, @url, @slug, @token)
|
183
|
+
puts "\r Image Generation: " + "DONE ".colorize(:green)
|
184
|
+
end
|
185
|
+
|
186
|
+
def dataset_uploads
|
187
|
+
print " Dataset Uploads: "
|
188
|
+
# failure ''
|
189
|
+
# puts "PASS".colorize(:green)
|
190
|
+
puts "SKIP".colorize(:blue)
|
191
|
+
end
|
192
|
+
|
193
|
+
def trigger_server_updates
|
194
|
+
print "Launch Server Scripts: "
|
195
|
+
# failure ''
|
196
|
+
# puts "PASS".colorize(:green)
|
197
|
+
puts "SKIP".colorize(:blue)
|
198
|
+
end
|
199
|
+
|
200
|
+
def failure(message)
|
201
|
+
puts "FAIL".colorize(:red)
|
202
|
+
puts message
|
203
|
+
raise DeployError
|
204
|
+
end
|
205
|
+
end
|
206
|
+
end
|
207
|
+
end
|
@@ -3,16 +3,22 @@ require 'fileutils'
|
|
3
3
|
require 'rubygems'
|
4
4
|
require 'json'
|
5
5
|
require 'yaml'
|
6
|
+
require 'colorize'
|
6
7
|
|
7
8
|
require 'spout/helpers/subject_loader'
|
8
9
|
require 'spout/helpers/chart_types'
|
9
10
|
require 'spout/helpers/config_reader'
|
11
|
+
require 'spout/helpers/send_file'
|
10
12
|
|
11
13
|
module Spout
|
12
14
|
module Commands
|
13
15
|
class Graphs
|
14
|
-
def initialize(variables, standard_version)
|
16
|
+
def initialize(variables, standard_version, deploy_mode = false, url = '', slug = '', token = '')
|
17
|
+
@deploy_mode = deploy_mode
|
18
|
+
@url = url
|
15
19
|
@standard_version = standard_version
|
20
|
+
@slug = slug
|
21
|
+
@token = token
|
16
22
|
|
17
23
|
@config = Spout::Helpers::ConfigReader.new
|
18
24
|
|
@@ -54,7 +60,7 @@ module Spout
|
|
54
60
|
@subjects = @subject_loader.subjects
|
55
61
|
compute_tables_and_charts
|
56
62
|
|
57
|
-
puts "Took #{Time.now - t} seconds." if @subjects.size > 0
|
63
|
+
puts "Took #{Time.now - t} seconds." if @subjects.size > 0 and not @deploy_mode
|
58
64
|
end
|
59
65
|
|
60
66
|
def compute_tables_and_charts
|
@@ -67,7 +73,12 @@ module Spout
|
|
67
73
|
variable_name = json['id'].to_s.downcase
|
68
74
|
next unless Spout::Models::Subject.method_defined?(variable_name)
|
69
75
|
|
70
|
-
|
76
|
+
if @deploy_mode
|
77
|
+
print "\r Graph Generation: " + "#{"% 3d" % ((file_index+1)*100/variable_files_count)}% Uploaded".colorize(:white)
|
78
|
+
else
|
79
|
+
puts "#{file_index+1} of #{variable_files_count}: #{variable_file.gsub(/(^variables\/|\.json$)/, '').gsub('/', ' / ')}"
|
80
|
+
end
|
81
|
+
|
71
82
|
|
72
83
|
|
73
84
|
stats = {
|
@@ -101,9 +112,17 @@ module Spout
|
|
101
112
|
chart_json_file = File.join('graphs', @standard_version, "#{json['id']}.json")
|
102
113
|
File.open(chart_json_file, 'w') { |file| file.write( JSON.pretty_generate(stats) + "\n" ) }
|
103
114
|
|
115
|
+
if @deploy_mode
|
116
|
+
send_to_server(chart_json_file)
|
117
|
+
end
|
118
|
+
|
104
119
|
end
|
105
120
|
end
|
106
121
|
|
122
|
+
def send_to_server(chart_json_file)
|
123
|
+
response = Spout::Helpers::SendFile.post("#{@url}/datasets/#{@slug}/upload_graph.json", chart_json_file, @standard_version, @token)
|
124
|
+
end
|
125
|
+
|
107
126
|
# [["Visit 1", "1"], ["Visit 2", "2"], ["CVD Outcomes", "3"]]
|
108
127
|
def visits
|
109
128
|
@visits ||= begin
|
@@ -7,12 +7,20 @@ require 'yaml'
|
|
7
7
|
require 'spout/helpers/subject_loader'
|
8
8
|
require 'spout/helpers/chart_types'
|
9
9
|
require 'spout/helpers/config_reader'
|
10
|
+
require 'spout/helpers/send_file'
|
10
11
|
|
11
12
|
module Spout
|
12
13
|
module Commands
|
13
14
|
class Images
|
14
15
|
|
15
|
-
def initialize(types, variable_ids, sizes, standard_version, argv)
|
16
|
+
def initialize(types, variable_ids, sizes, standard_version, argv, deploy_mode = false, url = '', slug = '', token = '')
|
17
|
+
@deploy_mode = deploy_mode
|
18
|
+
@url = url
|
19
|
+
@standard_version = standard_version
|
20
|
+
@slug = slug
|
21
|
+
@token = token
|
22
|
+
|
23
|
+
|
16
24
|
@variable_files = Dir.glob('variables/**/*.json')
|
17
25
|
@standard_version = standard_version
|
18
26
|
@pretend = (argv.delete('--pretend') != nil)
|
@@ -34,7 +42,7 @@ module Spout
|
|
34
42
|
@subjects = @subject_loader.subjects
|
35
43
|
|
36
44
|
compute_images
|
37
|
-
puts "Took #{Time.now - t} seconds." if @subjects.size > 0
|
45
|
+
puts "Took #{Time.now - t} seconds." if @subjects.size > 0 and not @deploy_mode
|
38
46
|
end
|
39
47
|
|
40
48
|
def compute_images
|
@@ -53,7 +61,11 @@ module Spout
|
|
53
61
|
variable_name = json['id'].to_s.downcase
|
54
62
|
next unless Spout::Models::Subject.method_defined?(variable_name)
|
55
63
|
|
56
|
-
|
64
|
+
if @deploy_mode
|
65
|
+
print "\r Image Generation: " + "#{"% 3d" % ((file_index+1)*100/variable_files_count)}% Uploaded".colorize(:white)
|
66
|
+
else
|
67
|
+
puts "#{file_index+1} of #{variable_files_count}: #{variable_file.gsub(/(^variables\/|\.json$)/, '').gsub('/', ' / ')}"
|
68
|
+
end
|
57
69
|
|
58
70
|
filtered_subjects = @subjects.select{ |s| s.send(@config.visit) != nil }
|
59
71
|
|
@@ -115,8 +127,13 @@ module Spout
|
|
115
127
|
puts phantomjs_command
|
116
128
|
else
|
117
129
|
`#{phantomjs_command}`
|
130
|
+
|
131
|
+
send_to_server(graph_path) if @deploy_mode
|
118
132
|
end
|
133
|
+
end
|
119
134
|
|
135
|
+
def send_to_server(file)
|
136
|
+
response = Spout::Helpers::SendFile.post("#{@url}/datasets/#{@slug}/upload_graph.json", file, @standard_version, @token, 'images')
|
120
137
|
end
|
121
138
|
|
122
139
|
end
|
@@ -4,13 +4,13 @@ module Spout
|
|
4
4
|
module Helpers
|
5
5
|
class ConfigReader
|
6
6
|
|
7
|
-
attr_reader :slug, :visit, :charts
|
8
|
-
|
7
|
+
attr_reader :slug, :visit, :charts, :webservers
|
9
8
|
|
10
9
|
def initialize
|
11
10
|
@slug = ''
|
12
11
|
@visit = ''
|
13
12
|
@charts = []
|
13
|
+
@webservers = []
|
14
14
|
parse_yaml_file
|
15
15
|
end
|
16
16
|
|
@@ -26,6 +26,12 @@ module Spout
|
|
26
26
|
else
|
27
27
|
[]
|
28
28
|
end
|
29
|
+
|
30
|
+
@webservers = if spout_config['webservers'].kind_of?(Array)
|
31
|
+
spout_config['webservers'].select{|c| c.kind_of?(Hash)}
|
32
|
+
else
|
33
|
+
[]
|
34
|
+
end
|
29
35
|
else
|
30
36
|
puts "The YAML file needs to be in the following format:"
|
31
37
|
puts "---\nvisit: visit_variable_name\ncharts:\n- chart: age_variable_name\n title: Age\n- chart: gender_variable_name\n title: Gender\n- chart: race_variable_name\n title: Race\n"
|
@@ -0,0 +1,27 @@
|
|
1
|
+
|
2
|
+
module Spout
|
3
|
+
module Helpers
|
4
|
+
module Quietly
|
5
|
+
|
6
|
+
# From Rails: http://apidock.com/rails/v3.2.13/Kernel/silence_stream
|
7
|
+
def silence_stream(stream)
|
8
|
+
old_stream = stream.dup
|
9
|
+
stream.reopen(RbConfig::CONFIG['host_os'] =~ /mswin|mingw/ ? 'NUL:' : '/dev/null')
|
10
|
+
stream.sync = true
|
11
|
+
yield
|
12
|
+
ensure
|
13
|
+
stream.reopen(old_stream)
|
14
|
+
end
|
15
|
+
|
16
|
+
# From Rails: http://apidock.com/rails/v3.2.13/Kernel/quietly
|
17
|
+
def quietly
|
18
|
+
silence_stream(STDOUT) do
|
19
|
+
silence_stream(STDERR) do
|
20
|
+
yield
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
@@ -0,0 +1,102 @@
|
|
1
|
+
require 'openssl'
|
2
|
+
require 'net/http'
|
3
|
+
require 'json'
|
4
|
+
|
5
|
+
module Spout
|
6
|
+
module Helpers
|
7
|
+
class SendFile
|
8
|
+
class << self
|
9
|
+
def post(*args)
|
10
|
+
new(*args).post
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
attr_reader :url
|
15
|
+
|
16
|
+
def initialize(url, filename, version, token, type = nil)
|
17
|
+
|
18
|
+
@params = {}
|
19
|
+
@params["version"] = version
|
20
|
+
@params["auth_token"] = token if token
|
21
|
+
@params["type"] = type if type
|
22
|
+
begin
|
23
|
+
file = File.open(filename, "rb")
|
24
|
+
@params["file"] = file
|
25
|
+
|
26
|
+
mp = Multipart::MultipartPost.new
|
27
|
+
@query, @headers = mp.prepare_query(@params)
|
28
|
+
ensure
|
29
|
+
file.close if file
|
30
|
+
end
|
31
|
+
|
32
|
+
begin
|
33
|
+
@url = URI.parse(url)
|
34
|
+
|
35
|
+
@http = Net::HTTP.new(@url.host, @url.port)
|
36
|
+
if @url.scheme == 'https'
|
37
|
+
@http.use_ssl = true
|
38
|
+
@http.verify_mode = OpenSSL::SSL::VERIFY_NONE
|
39
|
+
end
|
40
|
+
rescue
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
def post
|
45
|
+
begin
|
46
|
+
response = @http.start do |http|
|
47
|
+
http.post(@url.path, @query, @headers)
|
48
|
+
end
|
49
|
+
JSON.parse(response.body)
|
50
|
+
rescue
|
51
|
+
nil
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
|
59
|
+
module Multipart
|
60
|
+
class Param
|
61
|
+
attr_accessor :k, :v
|
62
|
+
def initialize( k, v )
|
63
|
+
@k = k
|
64
|
+
@v = v
|
65
|
+
end
|
66
|
+
|
67
|
+
def to_multipart
|
68
|
+
return "Content-Disposition: form-data; name=\"#{k}\"\r\n\r\n#{v}\r\n"
|
69
|
+
end
|
70
|
+
end
|
71
|
+
|
72
|
+
class FileParam
|
73
|
+
attr_accessor :k, :filename, :content
|
74
|
+
def initialize( k, filename, content )
|
75
|
+
@k = k
|
76
|
+
@filename = filename
|
77
|
+
@content = content
|
78
|
+
end
|
79
|
+
|
80
|
+
def to_multipart
|
81
|
+
mime_type = 'application/octet-stream'
|
82
|
+
return "Content-Disposition: form-data; name=\"#{k}\"; filename=\"#{filename}\"\r\n" + "Content-Transfer-Encoding: binary\r\n" + "Content-Type: #{mime_type}\r\n\r\n" + content + "\r\n"
|
83
|
+
end
|
84
|
+
end
|
85
|
+
class MultipartPost
|
86
|
+
BOUNDARY = 'a#41-93r1-^Õ-rule0000'
|
87
|
+
HEADER = {"Content-type" => "multipart/form-data, boundary=" + BOUNDARY + " "}
|
88
|
+
|
89
|
+
def prepare_query (params)
|
90
|
+
fp = []
|
91
|
+
params.each {|k,v|
|
92
|
+
if v.respond_to?(:read)
|
93
|
+
fp.push(FileParam.new(k, v.path, v.read))
|
94
|
+
else
|
95
|
+
fp.push(Param.new(k,v))
|
96
|
+
end
|
97
|
+
}
|
98
|
+
query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
|
99
|
+
return query, HEADER
|
100
|
+
end
|
101
|
+
end
|
102
|
+
end
|
@@ -41,11 +41,11 @@ module Spout
|
|
41
41
|
@csv_files = Dir.glob("csvs/#{@csv_directory}/*.csv")
|
42
42
|
@csv_files.each_with_index do |csv_file, index|
|
43
43
|
count = 0
|
44
|
-
|
44
|
+
print "\nParsing #{csv_file}"
|
45
45
|
CSV.parse( File.open(csv_file, 'r:iso-8859-1:utf-8'){|f| f.read}, headers: true, header_converters: lambda { |h| h.to_s.downcase } ) do |line|
|
46
46
|
row = line.to_hash
|
47
47
|
count += 1
|
48
|
-
print
|
48
|
+
print "\rParsing #{csv_file} - line ##{count}" if (count % 10 == 0)
|
49
49
|
@subjects << Spout::Models::Subject.create do |t|
|
50
50
|
t._visit = row[@visit]
|
51
51
|
|
@@ -68,7 +68,7 @@ module Spout
|
|
68
68
|
# puts "Memory Used: " + (`ps -o rss -p #{$$}`.strip.split.last.to_i / 1024).to_s + " MB" if count % 1000 == 0
|
69
69
|
break if @number_of_rows != nil and count >= @number_of_rows
|
70
70
|
end
|
71
|
-
puts "\n
|
71
|
+
puts "\n"
|
72
72
|
end
|
73
73
|
|
74
74
|
if @csv_directory != @standard_version
|
data/lib/spout/version.rb
CHANGED
data/lib/spout.rb
CHANGED
@@ -11,7 +11,8 @@ Spout::COMMANDS = {
|
|
11
11
|
'c' => :coverage_report,
|
12
12
|
'p' => :generate_images,
|
13
13
|
'g' => :generate_charts_and_tables,
|
14
|
-
'o' => :outliers_report
|
14
|
+
'o' => :outliers_report,
|
15
|
+
'd' => :deploy
|
15
16
|
}
|
16
17
|
|
17
18
|
module Spout
|
@@ -72,6 +73,8 @@ The most common spout commands are:
|
|
72
73
|
[g]raphs Generates JSON graphs for each variable
|
73
74
|
in a dataset and places them
|
74
75
|
in `<project_name>/graphs/<version>/`
|
76
|
+
[d]eploy NAME Push dataset and data dictionary to a
|
77
|
+
webserver specified in `.spout.yml`
|
75
78
|
[v]ersion Returns the version of Spout
|
76
79
|
|
77
80
|
Commands can be referenced by the first letter:
|
@@ -80,6 +83,11 @@ Commands can be referenced by the first letter:
|
|
80
83
|
EOT
|
81
84
|
end
|
82
85
|
|
86
|
+
def self.deploy(argv)
|
87
|
+
require 'spout/commands/deploy'
|
88
|
+
Spout::Commands::Deploy.new(argv, standard_version)
|
89
|
+
end
|
90
|
+
|
83
91
|
def self.importer(argv)
|
84
92
|
require 'spout/commands/importer'
|
85
93
|
Spout::Commands::Importer.new(argv)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: spout
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.10.0.
|
4
|
+
version: 0.10.0.beta6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Remo Mueller
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-10-
|
11
|
+
date: 2014-10-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rake
|
@@ -110,6 +110,7 @@ files:
|
|
110
110
|
- bin/spout
|
111
111
|
- lib/spout.rb
|
112
112
|
- lib/spout/commands/coverage.rb
|
113
|
+
- lib/spout/commands/deploy.rb
|
113
114
|
- lib/spout/commands/exporter.rb
|
114
115
|
- lib/spout/commands/graphs.rb
|
115
116
|
- lib/spout/commands/images.rb
|
@@ -122,7 +123,9 @@ files:
|
|
122
123
|
- lib/spout/helpers/iterators.rb
|
123
124
|
- lib/spout/helpers/json_loader.rb
|
124
125
|
- lib/spout/helpers/number_helper.rb
|
126
|
+
- lib/spout/helpers/quietly.rb
|
125
127
|
- lib/spout/helpers/semantic.rb
|
128
|
+
- lib/spout/helpers/send_file.rb
|
126
129
|
- lib/spout/helpers/subject_loader.rb
|
127
130
|
- lib/spout/helpers/table_formatting.rb
|
128
131
|
- lib/spout/models/coverage_result.rb
|