snowly 0.1.6 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +22 -17
- data/lib/snowly/app/collector.rb +49 -15
- data/lib/snowly/app/views/index.erb +2 -2
- data/lib/snowly/extensions/custom_dependencies.rb +1 -1
- data/lib/snowly/multi_validator.rb +21 -0
- data/lib/snowly/protocol_schema_finder.rb +34 -0
- data/lib/snowly/request.rb +7 -5
- data/lib/snowly/validator.rb +7 -27
- data/lib/snowly/version.rb +1 -1
- data/lib/snowly.rb +1 -0
- data/snowly.gemspec +0 -1
- metadata +4 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9254b6f1c821d30a0c34f25cf92c619935f30c8b
|
4
|
+
data.tar.gz: bbbcb42cc46d319243ea3fba626dbc1dafdcd313
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3f6d0d38ce02be8812ec8ef1d45d039802debb0df4b1e0c0ae953586b90bf96383ed6a2e12722dbc8811bcc1ec3c9cbe7a9a381b6d07b456720171123cbd31ba
|
7
|
+
data.tar.gz: 9e63cebeec4c54f429ba8cee65403ad46e36900f7f8638857a11314a8bb14711fc366fa1dfbb6297a7e282fb5ec21d8882d6a8fe74707ebcd18bf12741ffcc48
|
data/README.md
CHANGED
@@ -16,7 +16,7 @@ Snowplow has an excellent toolset, but the first implementation stages can be ha
|
|
16
16
|
|
17
17
|
### Features
|
18
18
|
|
19
|
-
With Snowly you can use [Json Schemas](http://spacetelescope.github.io/understanding-json-schema/) to define more expressive event requirements. Aside from assuring that you're fully compatible with the snowplow protocol, you can go even further and extend it with a set of more specific rules.
|
19
|
+
With Snowly you can use [Json Schemas](http://spacetelescope.github.io/understanding-json-schema/) to define more expressive event requirements. Aside from assuring that you're fully compatible with the snowplow protocol, you can go even further and extend it with a set of more specific rules. Snowly emulates both cloudfront and closure collectors and will handle its differences automatically.
|
20
20
|
|
21
21
|
Use cases:
|
22
22
|
|
@@ -84,7 +84,7 @@ Other options:
|
|
84
84
|
|
85
85
|
### Output
|
86
86
|
|
87
|
-
When Snowly finds something wrong, it renders
|
87
|
+
When Snowly finds something wrong, it renders a parsed array of requests along with its errors.
|
88
88
|
|
89
89
|
If everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode. In debug mode it always renders the parsed contents of your requests.
|
90
90
|
|
@@ -92,25 +92,30 @@ If you can't investigate the request's response, you can start Snowly in the for
|
|
92
92
|
`snowly -d -F`
|
93
93
|
|
94
94
|
Example:
|
95
|
-
`http://0.0.0.0:5678/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0`
|
95
|
+
`http://0.0.0.0:5678/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&eid=ev-id-1`
|
96
96
|
```json
|
97
|
-
|
98
|
-
|
99
|
-
"
|
100
|
-
"
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
"
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
97
|
+
[
|
98
|
+
{
|
99
|
+
"event_id": "ev-id-1",
|
100
|
+
"errors": [
|
101
|
+
"The property '#/platform' value \"i\" did not match one of the following values: web, mob, pc, srv, tv, cnsl, iot in schema snowplow_protocol.json",
|
102
|
+
"The property '#/' did not contain a required property of 'useragent' in schema snowplow_protocol.json"
|
103
|
+
],
|
104
|
+
"content": {
|
105
|
+
"event": "pv",
|
106
|
+
"page_title": "Root README",
|
107
|
+
"page_url": "http://github.com/snowplow/snowplow",
|
108
|
+
"app_id": "snowplow",
|
109
|
+
"platform": "i",
|
110
|
+
"v_tracker": "no-js-0.1.0",
|
111
|
+
"event_id": "ev-id-1"
|
112
|
+
}
|
110
113
|
}
|
111
|
-
|
114
|
+
]
|
112
115
|
```
|
113
116
|
|
117
|
+
If you're using the closure collector and can't see your requests firing up right away, try [manually flushing](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#54-manual-flushing) or change your emitter's buffer_size(number of events before flusing) to a lower value.
|
118
|
+
|
114
119
|
## JSON Schemas
|
115
120
|
|
116
121
|
JSON Schema is a powerful tool for validating the structure of JSON data. I recommend reading this excellent [Guide](http://spacetelescope.github.io/understanding-json-schema/) from Michael Droettboom to understand all of its capabilities, but you can start with the examples bellow.
|
data/lib/snowly/app/collector.rb
CHANGED
@@ -1,34 +1,34 @@
|
|
1
1
|
require 'erb'
|
2
2
|
require 'base64'
|
3
3
|
require 'sinatra'
|
4
|
-
require
|
4
|
+
require 'sinatra/reloader' if development?
|
5
5
|
|
6
6
|
module Snowly
|
7
7
|
module App
|
8
8
|
class Collector < Sinatra::Base
|
9
|
-
GIF = Base64.decode64(
|
9
|
+
GIF = Base64.decode64('R0lGODlhAQABAPAAAAAAAAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==')
|
10
10
|
configure :development do
|
11
11
|
register Sinatra::Reloader
|
12
12
|
end
|
13
13
|
|
14
|
-
|
15
|
-
|
16
|
-
@
|
17
|
-
|
14
|
+
def extract_content(validator)
|
15
|
+
multi = validator.respond_to?(:validators)
|
16
|
+
@content ||= if multi
|
17
|
+
validator.validators.each_with_object([]) do |item, memo|
|
18
|
+
item_request = item.request.as_hash
|
19
|
+
memo << { event_id: item_request['event_id'], errors: item.errors, content: item_request }
|
20
|
+
end
|
18
21
|
else
|
19
|
-
|
20
|
-
end
|
21
|
-
erb :index
|
22
|
+
[{ event_id: validator.request.as_hash['event_id'], errors: validator.errors, content: validator.request.as_hash }]
|
23
|
+
end.to_json
|
22
24
|
end
|
23
25
|
|
24
|
-
|
25
|
-
content_type :json
|
26
|
-
validator = Snowly::Validator.new request.query_string
|
26
|
+
def handle_response(validator)
|
27
27
|
if validator.validate
|
28
28
|
status 200
|
29
|
-
content = { content: validator.request.as_hash }.to_json
|
30
|
-
Snowly.logger.info content
|
31
29
|
if params[:debug] || Snowly.debug_mode
|
30
|
+
content = extract_content validator
|
31
|
+
Snowly.logger.info content
|
32
32
|
body(content)
|
33
33
|
else
|
34
34
|
content_type 'image/gif'
|
@@ -36,11 +36,45 @@ module Snowly
|
|
36
36
|
end
|
37
37
|
else
|
38
38
|
status 500
|
39
|
-
content =
|
39
|
+
content = extract_content validator
|
40
40
|
Snowly.logger.error content
|
41
41
|
body (content)
|
42
42
|
end
|
43
43
|
end
|
44
|
+
|
45
|
+
get '/' do
|
46
|
+
@url = request.url.gsub(/(http|https)\:\/\//,'')[0..-2]
|
47
|
+
@resolved_schemas = if resolver = Snowly.development_iglu_resolver_path
|
48
|
+
Dir[File.join(resolver,"/**/*")].select{ |e| File.file? e }
|
49
|
+
else
|
50
|
+
nil
|
51
|
+
end
|
52
|
+
erb :index
|
53
|
+
end
|
54
|
+
|
55
|
+
get '/i' do
|
56
|
+
content_type :json
|
57
|
+
validator = Snowly::Validator.new request.query_string
|
58
|
+
handle_response(validator)
|
59
|
+
end
|
60
|
+
|
61
|
+
post '/com.snowplowanalytics.snowplow/tp2' do
|
62
|
+
response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS'
|
63
|
+
response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept'
|
64
|
+
response.headers['Access-Control-Allow-Credentials'] = 'true'
|
65
|
+
response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*'
|
66
|
+
payload = JSON.parse request.body.read
|
67
|
+
validator = Snowly::MultiValidator.new payload
|
68
|
+
handle_response(validator)
|
69
|
+
end
|
70
|
+
|
71
|
+
options '*' do
|
72
|
+
response.headers['Allow'] = 'HEAD,GET,PUT,POST,DELETE,OPTIONS'
|
73
|
+
response.headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-HTTP-Method-Override, Content-Type, Cache-Control, Accept'
|
74
|
+
response.headers['Access-Control-Allow-Credentials'] = 'true'
|
75
|
+
response.headers['Access-Control-Allow-Origin'] = env['HTTP_ORIGIN'] || '*'
|
76
|
+
200
|
77
|
+
end
|
44
78
|
end
|
45
79
|
end
|
46
80
|
end
|
@@ -22,10 +22,10 @@
|
|
22
22
|
<p><strong>Snowly</strong> is a minimal collector implementation intended to validate your event tracking requests before emitting them to cloudfront or a closure collector.</p>
|
23
23
|
<p>When <strong>Snowly</strong> finds something wrong, it renders the parsed request along with its errors.</p>
|
24
24
|
<p>If everything is ok, Snowly delivers the default Snowplow pixel, unless you're using the debug mode.</p>
|
25
|
-
<p>Point your collector URL to <code><%= url
|
25
|
+
<p>Point your collector URL to <code><%= url %></code> and have fun!</p>
|
26
26
|
<p>
|
27
27
|
<a class="btn btn-lg btn-success" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=web&tv=no-js-0.1.0&ua=firefox&&eid=u2i3&debug=true" role="button">See it working!</a>
|
28
|
-
<a class="btn btn-lg btn-warning" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&debug=true" role="button">Event with errors!</a>
|
28
|
+
<a class="btn btn-lg btn-warning" href="/i?&e=pv&page=Root%20README&url=http%3A%2F%2Fgithub.com%2Fsnowplow%2Fsnowplow&aid=snowplow&p=i&tv=no-js-0.1.0&eid=u2i3&debug=true" role="button">Event with errors!</a>
|
29
29
|
</p>
|
30
30
|
<% unless Snowly.development_iglu_resolver_path %>
|
31
31
|
<div class="alert alert-danger" role="alert">The Local Iglu Resolver Path is missing.</div>
|
@@ -32,7 +32,7 @@ class CustomDependenciesAttribute < JSON::Schema::Attribute
|
|
32
32
|
def self.validate_dependency(schema, data, property, dependency_hash, fragments, processor, attribute, options)
|
33
33
|
key, value = Array(dependency_hash).flatten
|
34
34
|
return unless data[key.to_s] == value.to_s
|
35
|
-
return if data.has_key?(property.to_s)
|
35
|
+
return if data.has_key?(property.to_s) && not(data[property.to_s].blank?)
|
36
36
|
message = "The property '#{build_fragment(fragments)}' did not contain a required property of '#{property}' when property '#{key}' is '#{value}'"
|
37
37
|
validation_error(processor, message, fragments, schema, attribute, options[:record_errors])
|
38
38
|
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'snowly/validator'
|
2
|
+
module Snowly
|
3
|
+
class MultiValidator
|
4
|
+
attr_reader :validators
|
5
|
+
|
6
|
+
def initialize(payload)
|
7
|
+
@validators = payload['data'].map do |req|
|
8
|
+
Validator.new(req)
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
def validate
|
13
|
+
validators.each(&:validate)
|
14
|
+
valid?
|
15
|
+
end
|
16
|
+
|
17
|
+
def valid?
|
18
|
+
validators.all? { |v| v.valid? }
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
module Snowly
|
2
|
+
class ProtocolSchemaFinder
|
3
|
+
PROTOCOL_FILE_NAME = 'snowplow_protocol.json'
|
4
|
+
attr_reader :schema
|
5
|
+
|
6
|
+
def initialize(custom_schema = nil)
|
7
|
+
@custom_schema = custom_schema
|
8
|
+
@schema = load_protocol_schema
|
9
|
+
end
|
10
|
+
|
11
|
+
def find_protocol_schema
|
12
|
+
return @custom_schema if @custom_schema
|
13
|
+
if resolver && alternative_protocol_schema
|
14
|
+
alternative_protocol_schema
|
15
|
+
else
|
16
|
+
File.expand_path("../../schemas/#{PROTOCOL_FILE_NAME}", __FILE__)
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def resolver
|
21
|
+
Snowly.development_iglu_resolver_path
|
22
|
+
end
|
23
|
+
|
24
|
+
def alternative_protocol_schema
|
25
|
+
Dir[File.join(resolver,"/**/*")].select{ |f| File.basename(f) == PROTOCOL_FILE_NAME }[0]
|
26
|
+
end
|
27
|
+
|
28
|
+
# Loads the protocol schema created to describe snowplow events table attributes
|
29
|
+
# @return [Hash] parsed schema
|
30
|
+
def load_protocol_schema
|
31
|
+
JSON.parse File.read(find_protocol_schema)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
data/lib/snowly/request.rb
CHANGED
@@ -1,9 +1,11 @@
|
|
1
1
|
require 'snowly/transformer'
|
2
2
|
module Snowly
|
3
3
|
class Request
|
4
|
-
|
5
|
-
|
6
|
-
|
4
|
+
|
5
|
+
attr_reader :parsed_payload
|
6
|
+
|
7
|
+
def initialize(payload)
|
8
|
+
@parsed_payload = payload.is_a?(String) ? parse_query(payload) : payload
|
7
9
|
end
|
8
10
|
|
9
11
|
# Retuns request as json, after transforming parameters into column names
|
@@ -15,12 +17,12 @@ module Snowly
|
|
15
17
|
# Retuns request as hash, after transforming parameters into column names
|
16
18
|
# @return [Hash]
|
17
19
|
def as_hash
|
18
|
-
@hash ||= Transformer.transform(
|
20
|
+
@hash ||= Transformer.transform(parsed_payload)
|
19
21
|
end
|
20
22
|
|
21
23
|
# Returns query parameters as hash
|
22
24
|
# @return [Hash]
|
23
|
-
def
|
25
|
+
def parse_query(query_string)
|
24
26
|
@parsed_query ||= Rack::Utils.parse_nested_query(query_string)
|
25
27
|
end
|
26
28
|
|
data/lib/snowly/validator.rb
CHANGED
@@ -1,17 +1,15 @@
|
|
1
1
|
# Performs the validation for the root attributes and associated contexts and unstructured events.
|
2
2
|
require 'snowly/request'
|
3
|
+
require 'snowly/protocol_schema_finder'
|
3
4
|
require 'snowly/extensions/custom_dependencies'
|
4
5
|
|
5
6
|
module Snowly
|
6
7
|
class Validator
|
7
|
-
|
8
|
+
attr_reader :request, :errors
|
8
9
|
|
9
|
-
|
10
|
-
|
11
|
-
def initialize(query_string)
|
12
|
-
@request = Request.new query_string
|
10
|
+
def initialize(payload)
|
11
|
+
@request = Request.new payload
|
13
12
|
@errors = []
|
14
|
-
@protocol_schema = load_protocol_schema
|
15
13
|
end
|
16
14
|
|
17
15
|
# If request is valid
|
@@ -27,29 +25,11 @@ module Snowly
|
|
27
25
|
valid?
|
28
26
|
end
|
29
27
|
|
30
|
-
|
31
|
-
|
32
|
-
def find_protocol_schema
|
33
|
-
if resolver && alternative_protocol_schema
|
34
|
-
alternative_protocol_schema
|
35
|
-
else
|
36
|
-
File.expand_path("../../schemas/#{PROTOCOL_FILE_NAME}", __FILE__)
|
37
|
-
end
|
38
|
-
end
|
39
|
-
|
40
|
-
def resolver
|
41
|
-
Snowly.development_iglu_resolver_path
|
42
|
-
end
|
43
|
-
|
44
|
-
def alternative_protocol_schema
|
45
|
-
Dir[File.join(resolver,"/**/*")].select{ |f| File.basename(f) == PROTOCOL_FILE_NAME }[0]
|
28
|
+
def protocol_schema
|
29
|
+
@protocol_schema ||= ProtocolSchemaFinder.new.schema
|
46
30
|
end
|
47
31
|
|
48
|
-
|
49
|
-
# @return [Hash] parsed schema
|
50
|
-
def load_protocol_schema
|
51
|
-
JSON.parse File.read(find_protocol_schema)
|
52
|
-
end
|
32
|
+
private
|
53
33
|
|
54
34
|
# @return [Hash] all contexts content and schema definitions
|
55
35
|
def associated_contexts
|
data/lib/snowly/version.rb
CHANGED
data/lib/snowly.rb
CHANGED
data/snowly.gemspec
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: snowly
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Alexandre Angelim
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-09-17 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: json-schema
|
@@ -192,20 +192,6 @@ dependencies:
|
|
192
192
|
- - ~>
|
193
193
|
- !ruby/object:Gem::Version
|
194
194
|
version: '0.9'
|
195
|
-
- !ruby/object:Gem::Dependency
|
196
|
-
name: yard
|
197
|
-
requirement: !ruby/object:Gem::Requirement
|
198
|
-
requirements:
|
199
|
-
- - ~>
|
200
|
-
- !ruby/object:Gem::Version
|
201
|
-
version: '0.8'
|
202
|
-
type: :development
|
203
|
-
prerelease: false
|
204
|
-
version_requirements: !ruby/object:Gem::Requirement
|
205
|
-
requirements:
|
206
|
-
- - ~>
|
207
|
-
- !ruby/object:Gem::Version
|
208
|
-
version: '0.8'
|
209
195
|
description: Snowly is a minimal collector implementation intended to validate your
|
210
196
|
event tracking requests before emitting them to cloudfront or a closure collector.
|
211
197
|
email:
|
@@ -230,6 +216,8 @@ files:
|
|
230
216
|
- lib/snowly/app/collector.rb
|
231
217
|
- lib/snowly/app/views/index.erb
|
232
218
|
- lib/snowly/extensions/custom_dependencies.rb
|
219
|
+
- lib/snowly/multi_validator.rb
|
220
|
+
- lib/snowly/protocol_schema_finder.rb
|
233
221
|
- lib/snowly/request.rb
|
234
222
|
- lib/snowly/schema_cache.rb
|
235
223
|
- lib/snowly/transformer.rb
|