snowly 0.1.6 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +22 -17
- data/lib/snowly/app/collector.rb +49 -15
- data/lib/snowly/app/views/index.erb +2 -2
- data/lib/snowly/extensions/custom_dependencies.rb +1 -1
- data/lib/snowly/multi_validator.rb +21 -0
- data/lib/snowly/protocol_schema_finder.rb +34 -0
- data/lib/snowly/request.rb +7 -5
- data/lib/snowly/validator.rb +7 -27
- data/lib/snowly/version.rb +1 -1
- data/lib/snowly.rb +1 -0
- data/snowly.gemspec +0 -1
- metadata +4 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9254b6f1c821d30a0c34f25cf92c619935f30c8b
|
4
|
+
data.tar.gz: bbbcb42cc46d319243ea3fba626dbc1dafdcd313
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3f6d0d38ce02be8812ec8ef1d45d039802debb0df4b1e0c0ae953586b90bf96383ed6a2e12722dbc8811bcc1ec3c9cbe7a9a381b6d07b456720171123cbd31ba
|
7
|
+
data.tar.gz: 9e63cebeec4c54f429ba8cee65403ad46e36900f7f8638857a11314a8bb14711fc366fa1dfbb6297a7e282fb5ec21d8882d6a8fe74707ebcd18bf12741ffcc48
|
data/README.md
CHANGED
@@ -16,7 +16,7 @@ Snowplow has an excellent toolset, but the first implementation stages can be ha
|
|
16
16
|
|
17
17
|
### Features
|
18
18
|
|
19
|
-
With Snowly you can use [Json Schemas](http://spacetelescope.github.io/understanding-json-schema/) to define more expressive event requirements. Aside from assuring that you're fully compatible with the snowplow protocol, you can go even further and extend it with a set of more specific rules.
|
19
|
+
With Snowly you can use [Json Schemas](http://spacetelescope.github.io/understanding-json-schema/) to define more expressive event requirements. Aside from assuring that you're fully compatible with the snowplow protocol, you can go even further and extend it with a set of more specific rules. Snowly emulates both cloudfront and closure collectors and will handle its differences automatically.
|
20
20
|
|
21
21
|
Use cases:
|
22
22
|
|
@@ -84,7 +84,7 @@ Other options:
|
|
84
84
|
|
85
85
|
### Output
|
86
86
|
|
87
|
-
When Snowly finds something wrong, it renders
|
87
|
+
When Snowly finds something wrong, it renders a parsed array of requests along with its errors.
|
88
88
|
|
89
89
|
If everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode. In debug mode it always renders the parsed contents of your requests.
|
90
90
|
|
@@ -92,25 +92,30 @@ If you can't investigate the request's response, you can start Snowly in the for
|
|
92
92
|
`snowly -d -F`
|
93
93
|
|
94
94
|
Example:
|
95
|
-
`http://0.0.0.0:5678/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0`
|
95
|
+
`http://0.0.0.0:5678/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&eid=ev-id-1`
|
96
96
|
```json
|
97
|
-
|
98
|
-
|
99
|
-
"
|
100
|
-
"
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
"
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
97
|
+
[
|
98
|
+
{
|
99
|
+
"event_id": "ev-id-1",
|
100
|
+
"errors": [
|
101
|
+
"The property '#/platform' value \"i\" did not match one of the following values: web, mob, pc, srv, tv, cnsl, iot in schema snowplow_protocol.json",
|
102
|
+
"The property '#/' did not contain a required property of 'useragent' in schema snowplow_protocol.json"
|
103
|
+
],
|
104
|
+
"content": {
|
105
|
+
"event": "pv",
|
106
|
+
"page_title": "Root README",
|
107
|
+
"page_url": "http://github.com/snowplow/snowplow",
|
108
|
+
"app_id": "snowplow",
|
109
|
+
"platform": "i",
|
110
|
+
"v_tracker": "no-js-0.1.0",
|
111
|
+
"event_id": "ev-id-1"
|
112
|
+
}
|
110
113
|
}
|
111
|
-
|
114
|
+
]
|
112
115
|
```
|
113
116
|
|
117
|
+
If you're using the closure collector and can't see your requests firing up right away, try [manually flushing](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#54-manual-flushing) or change your emitter's buffer_size(number of events before flusing) to a lower value.
|
118
|
+
|
114
119
|
## JSON Schemas
|
115
120
|
|
116
121
|
JSON Schema is a powerful tool for validating the structure of JSON data. I recommend reading this excellent [Guide](http://spacetelescope.github.io/understanding-json-schema/) from Michael Droettboom to understand all of its capabilities, but you can start with the examples bellow.
|
data/lib/snowly/app/collector.rb
CHANGED
@@ -1,34 +1,34 @@
|
|
1
1
|
require 'erb'
|
2
2
|
require 'base64'
|
3
3
|
require 'sinatra'
|
4
|
-
require
|
4
|
+
require 'sinatra/reloader' if development?
|
5
5
|
|
6
6
|
module Snowly
|
7
7
|
module App
|
8
8
|
class Collector < Sinatra::Base
|
9
|
-
GIF = Base64.decode64(
|
9
|
+
GIF = Base64.decode64('R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==')
|
10
10
|
configure :development do
|
11
11
|
register Sinatra::Reloader
|
12
12
|
end
|
13
13
|
|
14
|
-
|
15
|
-
|
16
|
-
@
|
17
|
-
|
14
|
+
def extract_content(validator)
|
15
|
+
multi = validator.respond_to?(:validators)
|
16
|
+
@content ||= if multi
|
17
|
+
validator.validators.each_with_object([]) do |item, memo|
|
18
|
+
item_request = item.request.as_hash
|
19
|
+
memo << { event_id: item_request['event_id'], errors: item.errors, content: item_request }
|
20
|
+
end
|
18
21
|
else
|
19
|
-
|
20
|
-
end
|
21
|
-
erb :index
|
22
|
+
[{ event_id: validator.request.as_hash['event_id'], errors: validator.errors, content: validator.request.as_hash }]
|
23
|
+
end.to_json
|
22
24
|
end
|
23
25
|
|
24
|
-
|
25
|
-
content_type :json
|
26
|
-
validator = Snowly::Validator.new request.query_string
|
26
|
+
def handle_response(validator)
|
27
27
|
if validator.validate
|
28
28
|
status 200
|
29
|
-
content = { content: validator.request.as_hash }.to_json
|
30
|
-
Snowly.logger.info content
|
31
29
|
if params[:debug] || Snowly.debug_mode
|
30
|
+
content = extract_content validator
|
31
|
+
Snowly.logger.info content
|
32
32
|
body(content)
|
33
33
|
else
|
34
34
|
content_type 'image/gif'
|
@@ -36,11 +36,45 @@ module Snowly
|
|
36
36
|
end
|
37
37
|
else
|
38
38
|
status 500
|
39
|
-
content =
|
39
|
+
content = extract_content validator
|
40
40
|
Snowly.logger.error content
|
41
41
|
body (content)
|
42
42
|
end
|
43
43
|
end
|
44
|
+
|
45
|
+
get '/' do
|
46
|
+
@url = request.url.gsub(/(http|https)\:\/\//,'')[0..-2]
|
47
|
+
@resolved_schemas = if resolver = Snowly.development_iglu_resolver_path
|
48
|
+
Dir[File.join(resolver,"/**/*")].select{ |e| File.file? e }
|
49
|
+
else
|
50
|
+
nil
|
51
|
+
end
|
52
|
+
erb :index
|
53
|
+
end
|
54
|
+
|
55
|
+
get '/i' do
|
56
|
+
content_type :json
|
57
|
+
validator = Snowly::Validator.new request.query_string
|
58
|
+
handle_response(validator)
|
59
|
+
end
|
60
|
+
|
61
|
+
post '/com.snowplowanalytics.snowplow/tp2' do
|
62
|
+
response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS'
|
63
|
+
response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept'
|
64
|
+
response.headers['Access-Control-Allow-Credentials'] = 'true'
|
65
|
+
response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*'
|
66
|
+
payload = JSON.parse request.body.read
|
67
|
+
validator = Snowly::MultiValidator.new payload
|
68
|
+
handle_response(validator)
|
69
|
+
end
|
70
|
+
|
71
|
+
options '*' do
|
72
|
+
response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS'
|
73
|
+
response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept'
|
74
|
+
response.headers['Access-Control-Allow-Credentials'] = 'true'
|
75
|
+
response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*'
|
76
|
+
200
|
77
|
+
end
|
44
78
|
end
|
45
79
|
end
|
46
80
|
end
|
@@ -22,10 +22,10 @@
|
|
22
22
|
<p><strong>Snowly</strong> is a minimal collector implementation intended to validate your event tracking requests before emitting them to cloudfront or a closure collector.</p>
|
23
23
|
<p>When <strong>Snowly</strong> finds something wrong, it renders the parsed request along with its errors.</p>
|
24
24
|
<p>If everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode.</p>
|
25
|
-
<p>Point your collector URL to <code><%= url
|
25
|
+
<p>Point your collector URL to <code><%= url %></code> and have fun!</p>
|
26
26
|
<p>
|
27
27
|
<a class="btn btn-lg btn-success" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=web&tv=no-js-0.1.0&ua=firefox&&eid=u2i3&debug=true" role="button">See it working!</a>
|
28
|
-
<a class="btn btn-lg btn-warning" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&debug=true" role="button">Event with errors!</a>
|
28
|
+
<a class="btn btn-lg btn-warning" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&eid=u2i3&debug=true" role="button">Event with errors!</a>
|
29
29
|
</p>
|
30
30
|
<% unless Snowly.development_iglu_resolver_path %>
|
31
31
|
<div class="alert alert-danger" role="alert">The Local Iglu Resolver Path is missing.</div>
|
@@ -32,7 +32,7 @@ class CustomDependenciesAttribute < JSON::Schema::Attribute
|
|
32
32
|
def self.validate_dependency(schema, data, property, dependency_hash, fragments, processor, attribute, options)
|
33
33
|
key, value = Array(dependency_hash).flatten
|
34
34
|
return unless data[key.to_s] == value.to_s
|
35
|
-
return if data.has_key?(property.to_s)
|
35
|
+
return if data.has_key?(property.to_s) && not(data[property.to_s].blank?)
|
36
36
|
message = "The property '#{build_fragment(fragments)}' did not contain a required property of '#{property}' when property '#{key}' is '#{value}'"
|
37
37
|
validation_error(processor, message, fragments, schema, attribute, options[:record_errors])
|
38
38
|
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'snowly/validator'
|
2
|
+
module Snowly
|
3
|
+
class MultiValidator
|
4
|
+
attr_reader :validators
|
5
|
+
|
6
|
+
def initialize(payload)
|
7
|
+
@validators = payload['data'].map do |req|
|
8
|
+
Validator.new(req)
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
def validate
|
13
|
+
validators.each(&:validate)
|
14
|
+
valid?
|
15
|
+
end
|
16
|
+
|
17
|
+
def valid?
|
18
|
+
validators.all? { |v| v.valid? }
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
module Snowly
|
2
|
+
class ProtocolSchemaFinder
|
3
|
+
PROTOCOL_FILE_NAME = 'snowplow_protocol.json'
|
4
|
+
attr_reader :schema
|
5
|
+
|
6
|
+
def initialize(custom_schema = nil)
|
7
|
+
@custom_schema = custom_schema
|
8
|
+
@schema = load_protocol_schema
|
9
|
+
end
|
10
|
+
|
11
|
+
def find_protocol_schema
|
12
|
+
return @custom_schema if @custom_schema
|
13
|
+
if resolver && alternative_protocol_schema
|
14
|
+
alternative_protocol_schema
|
15
|
+
else
|
16
|
+
File.expand_path("../../schemas/#{PROTOCOL_FILE_NAME}", __FILE__)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def resolver
|
21
|
+
Snowly.development_iglu_resolver_path
|
22
|
+
end
|
23
|
+
|
24
|
+
def alternative_protocol_schema
|
25
|
+
Dir[File.join(resolver,"/**/*")].select{ |f| File.basename(f) == PROTOCOL_FILE_NAME }[0]
|
26
|
+
end
|
27
|
+
|
28
|
+
# Loads the protocol schema created to describe snowplow events table attributes
|
29
|
+
# @return [Hash] parsed schema
|
30
|
+
def load_protocol_schema
|
31
|
+
JSON.parse File.read(find_protocol_schema)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
data/lib/snowly/request.rb
CHANGED
@@ -1,9 +1,11 @@
|
|
1
1
|
require 'snowly/transformer'
|
2
2
|
module Snowly
|
3
3
|
class Request
|
4
|
-
|
5
|
-
|
6
|
-
|
4
|
+
|
5
|
+
attr_reader :parsed_payload
|
6
|
+
|
7
|
+
def initialize(payload)
|
8
|
+
@parsed_payload = payload.is_a?(String) ? parse_query(payload) : payload
|
7
9
|
end
|
8
10
|
|
9
11
|
# Retuns request as json, after transforming parameters into column names
|
@@ -15,12 +17,12 @@ module Snowly
|
|
15
17
|
# Retuns request as hash, after transforming parameters into column names
|
16
18
|
# @return [Hash]
|
17
19
|
def as_hash
|
18
|
-
@hash ||= Transformer.transform(
|
20
|
+
@hash ||= Transformer.transform(parsed_payload)
|
19
21
|
end
|
20
22
|
|
21
23
|
# Returns query parameters as hash
|
22
24
|
# @return [Hash]
|
23
|
-
def
|
25
|
+
def parse_query(query_string)
|
24
26
|
@parsed_query ||= Rack::Utils.parse_nested_query(query_string)
|
25
27
|
end
|
26
28
|
|
data/lib/snowly/validator.rb
CHANGED
@@ -1,17 +1,15 @@
|
|
1
1
|
# Performs the validation for the root attributes and associated contexts and unstructured events.
|
2
2
|
require 'snowly/request'
|
3
|
+
require 'snowly/protocol_schema_finder'
|
3
4
|
require 'snowly/extensions/custom_dependencies'
|
4
5
|
|
5
6
|
module Snowly
|
6
7
|
class Validator
|
7
|
-
|
8
|
+
attr_reader :request, :errors
|
8
9
|
|
9
|
-
|
10
|
-
|
11
|
-
def initialize(query_string)
|
12
|
-
@request = Request.new query_string
|
10
|
+
def initialize(payload)
|
11
|
+
@request = Request.new payload
|
13
12
|
@errors = []
|
14
|
-
@protocol_schema = load_protocol_schema
|
15
13
|
end
|
16
14
|
|
17
15
|
# If request is valid
|
@@ -27,29 +25,11 @@ module Snowly
|
|
27
25
|
valid?
|
28
26
|
end
|
29
27
|
|
30
|
-
|
31
|
-
|
32
|
-
def find_protocol_schema
|
33
|
-
if resolver && alternative_protocol_schema
|
34
|
-
alternative_protocol_schema
|
35
|
-
else
|
36
|
-
File.expand_path("../../schemas/#{PROTOCOL_FILE_NAME}", __FILE__)
|
37
|
-
end
|
38
|
-
end
|
39
|
-
|
40
|
-
def resolver
|
41
|
-
Snowly.development_iglu_resolver_path
|
42
|
-
end
|
43
|
-
|
44
|
-
def alternative_protocol_schema
|
45
|
-
Dir[File.join(resolver,"/**/*")].select{ |f| File.basename(f) == PROTOCOL_FILE_NAME }[0]
|
28
|
+
def protocol_schema
|
29
|
+
@protocol_schema ||= ProtocolSchemaFinder.new.schema
|
46
30
|
end
|
47
31
|
|
48
|
-
|
49
|
-
# @return [Hash] parsed schema
|
50
|
-
def load_protocol_schema
|
51
|
-
JSON.parse File.read(find_protocol_schema)
|
52
|
-
end
|
32
|
+
private
|
53
33
|
|
54
34
|
# @return [Hash] all contexts content and schema definitions
|
55
35
|
def associated_contexts
|
data/lib/snowly/version.rb
CHANGED
data/lib/snowly.rb
CHANGED
data/snowly.gemspec
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: snowly
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Alexandre Angelim
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-09-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: json-schema
|
@@ -192,20 +192,6 @@ dependencies:
|
|
192
192
|
- - ~>
|
193
193
|
- !ruby/object:Gem::Version
|
194
194
|
version: '0.9'
|
195
|
-
- !ruby/object:Gem::Dependency
|
196
|
-
name: yard
|
197
|
-
requirement: !ruby/object:Gem::Requirement
|
198
|
-
requirements:
|
199
|
-
- - ~>
|
200
|
-
- !ruby/object:Gem::Version
|
201
|
-
version: '0.8'
|
202
|
-
type: :development
|
203
|
-
prerelease: false
|
204
|
-
version_requirements: !ruby/object:Gem::Requirement
|
205
|
-
requirements:
|
206
|
-
- - ~>
|
207
|
-
- !ruby/object:Gem::Version
|
208
|
-
version: '0.8'
|
209
195
|
description: Snowly is a minimal collector implementation intended to validate your
|
210
196
|
event tracking requests before emitting them to cloudfront or a closure collector.
|
211
197
|
email:
|
@@ -230,6 +216,8 @@ files:
|
|
230
216
|
- lib/snowly/app/collector.rb
|
231
217
|
- lib/snowly/app/views/index.erb
|
232
218
|
- lib/snowly/extensions/custom_dependencies.rb
|
219
|
+
- lib/snowly/multi_validator.rb
|
220
|
+
- lib/snowly/protocol_schema_finder.rb
|
233
221
|
- lib/snowly/request.rb
|
234
222
|
- lib/snowly/schema_cache.rb
|
235
223
|
- lib/snowly/transformer.rb
|