ruby-druid 0.9.0 → 0.11.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/README.md +51 -51
- data/lib/druid/aggregation.rb +28 -0
- data/lib/druid/data_source.rb +31 -4
- data/lib/druid/dimension.rb +54 -0
- data/lib/druid/filter.rb +37 -1
- data/lib/druid/query.rb +55 -8
- data/lib/druid/version.rb +1 -1
- data/lib/druid/zk.rb +12 -12
- data/ruby-druid.gemspec +6 -6
- data/spec/lib/data_source_spec.rb +26 -11
- data/spec/lib/query_spec.rb +29 -1
- metadata +42 -30
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 0f5eb5f76213ef418c72e7a23982396bd2191bfcb1c8e45f246c956d2c31736a
|
4
|
+
data.tar.gz: 6d2e0f22836c012242e177d283a6ec846a21bb102c20f5d81680fb9888ea7be7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4da2d77ab40643ed4050ae3536880a38e3d03215c46fee9e67851d04c2618b3f3cc17ada6d8f77d67b13bcac847a3a3bcc0114708eaf3029fda7105060e2d694
|
7
|
+
data.tar.gz: e2c000c181b04bc74bb4a949bbb2dfba189771fd6c2fb4abd1f98bc45cfb588f4e47818ec1aa2b6938154ba9a7c85ded82ab7f8092e0788baf802511aa9c86b1
|
data/README.md
CHANGED
@@ -29,32 +29,22 @@ gem install ruby-druid
|
|
29
29
|
|
30
30
|
## Usage
|
31
31
|
|
32
|
-
|
33
|
-
Druid::Client.new('zk1:2181,zk2:2181/druid').query('service/source')
|
34
|
-
```
|
35
|
-
|
36
|
-
returns a query object on which all other methods can be called to create a full and valid Druid query.
|
37
|
-
|
38
|
-
A query object can be sent like this:
|
32
|
+
A query can be constructed and sent like so:
|
39
33
|
|
40
34
|
```ruby
|
41
|
-
|
42
|
-
query = Druid::Query.new(
|
43
|
-
|
35
|
+
data_source = Druid::Client.new('zk1:2181,zk2:2181/druid').data_source('service/source')
|
36
|
+
query = Druid::Query::Builder.new.long_sum(:aggregate1).last(1.day).granularity(:all)
|
37
|
+
result = data_source.post(query)
|
44
38
|
```
|
45
39
|
|
46
|
-
The `
|
40
|
+
The `post` method on the `DataSource` returns the parsed response from the Druid server as an array.
|
47
41
|
|
48
|
-
|
42
|
+
If you don't want to use ZooKeeper for broker discovery, you can explicitly construct a `DataSource`:
|
49
43
|
|
50
44
|
```ruby
|
51
|
-
|
45
|
+
data_source = Druid::DataSource.new('service/source', 'http://localhost:8080/druid/v2')
|
52
46
|
```
|
53
47
|
|
54
|
-
Supported options are:
|
55
|
-
* `static_setup` to explicitly specify a broker url, e.g. `static_setup: { 'my/source_name' => 'http://1.2.3.4:8080/druid/v2/' }`
|
56
|
-
* `http_timeout` to define a timeout for sending http queries to a broker (in minutes, default value is 2)
|
57
|
-
|
58
48
|
### GroupBy
|
59
49
|
|
60
50
|
A [GroupByQuery](http://druid.io/docs/latest/querying/groupbyquery.html) sets the
|
@@ -63,7 +53,7 @@ dimensions to group the data.
|
|
63
53
|
`queryType` is set automatically to `groupBy`.
|
64
54
|
|
65
55
|
```ruby
|
66
|
-
Druid::Query.new
|
56
|
+
Druid::Query::Builder.new.group_by([:dimension1, :dimension2])
|
67
57
|
```
|
68
58
|
|
69
59
|
### TimeSeries
|
@@ -71,7 +61,7 @@ Druid::Query.new('service/source').group_by([:dimension1, :dimension2])
|
|
71
61
|
A [TimeSeriesQuery](http://druid.io/docs/latest/querying/timeseriesquery.html) returns an array of JSON objects where each object represents a value asked for by the timeseries query.
|
72
62
|
|
73
63
|
```ruby
|
74
|
-
Druid::Query.new
|
64
|
+
Druid::Query::Builder.new.time_series([:aggregate1, :aggregate2])
|
75
65
|
```
|
76
66
|
|
77
67
|
### Aggregations
|
@@ -79,7 +69,7 @@ Druid::Query.new('service/source').time_series([:aggregate1, :aggregate2])
|
|
79
69
|
#### longSum, doubleSum, count, min, max, hyperUnique
|
80
70
|
|
81
71
|
```ruby
|
82
|
-
Druid::Query.new
|
72
|
+
Druid::Query::Builder.new.long_sum([:aggregate1, :aggregate2])
|
83
73
|
```
|
84
74
|
|
85
75
|
In the same way could be used the following methods for [aggregations](http://druid.io/docs/latest/querying/aggregations.html) adding: `double_sum, count, min, max, hyper_unique`
|
@@ -87,7 +77,7 @@ In the same way could be used the following methods for [aggregations](http://dr
|
|
87
77
|
#### cardinality
|
88
78
|
|
89
79
|
```ruby
|
90
|
-
Druid::Query.new
|
80
|
+
Druid::Query::Builder.new.cardinality(:aggregate, [:dimension1, dimension2], <by_row: true | false>)
|
91
81
|
```
|
92
82
|
|
93
83
|
#### javascript
|
@@ -95,19 +85,29 @@ Druid::Query.new('service/source').cardinality(:aggregate, [:dimension1, dimensi
|
|
95
85
|
For example calculation for `sum(log(x)/y) + 10`:
|
96
86
|
|
97
87
|
```ruby
|
98
|
-
Druid::Query.new
|
88
|
+
Druid::Query::Builder.new.js_aggregation(:aggregate, [:x, :y],
|
99
89
|
aggregate: "function(current, a, b) { return current + (Math.log(a) * b); }",
|
100
90
|
combine: "function(partialA, partialB) { return partialA + partialB; }",
|
101
91
|
reset: "function() { return 10; }"
|
102
92
|
)
|
103
93
|
```
|
104
94
|
|
95
|
+
#### filtered aggregation
|
96
|
+
|
97
|
+
A filtered aggregator wraps any given aggregator, but only aggregates the values for which the given dimension filter matches.
|
98
|
+
|
99
|
+
```ruby
|
100
|
+
Druid::Query::Builder.new.filtered_aggregation(:aggregate1, :aggregate_1_name, :longSum) do
|
101
|
+
dimension1.neq 1 & dimension2.neq 2
|
102
|
+
end
|
103
|
+
```
|
104
|
+
|
105
105
|
### Post Aggregations
|
106
106
|
|
107
107
|
A simple syntax for post aggregations with +,-,/,* can be used like:
|
108
108
|
|
109
109
|
```ruby
|
110
|
-
query = Druid::Query.new
|
110
|
+
query = Druid::Query::Builder.new.long_sum([:aggregate1, :aggregate2])
|
111
111
|
query.postagg { (aggregate2 + aggregate2).as output_field_name }
|
112
112
|
```
|
113
113
|
|
@@ -124,7 +124,7 @@ query.postagg { js('function(aggregate1, aggregate2) { return aggregate1 + aggre
|
|
124
124
|
The interval for the query takes a string with date and time or objects that provide an `iso8601` method.
|
125
125
|
|
126
126
|
```ruby
|
127
|
-
query = Druid::Query.new
|
127
|
+
query = Druid::Query::Builder.new.long_sum(:aggregate1)
|
128
128
|
query.interval("2013-01-01T00", Time.now)
|
129
129
|
```
|
130
130
|
|
@@ -139,14 +139,14 @@ The period `'day'` or `:day` will be interpreted as `'P1D'`.
|
|
139
139
|
If a period granularity is specifed, the (optional) second parameter is a time zone. It defaults to the machines local time zone. i.e.
|
140
140
|
|
141
141
|
```ruby
|
142
|
-
query = Druid::Query.new
|
142
|
+
query = Druid::Query::Builder.new.long_sum(:aggregate1)
|
143
143
|
query.granularity(:day)
|
144
144
|
```
|
145
145
|
|
146
146
|
is (on my box) the same as
|
147
147
|
|
148
148
|
```ruby
|
149
|
-
query = Druid::Query.new
|
149
|
+
query = Druid::Query::Builder.new.long_sum(:aggregate1)
|
150
150
|
query.granularity('P1D', 'Europe/Berlin')
|
151
151
|
```
|
152
152
|
|
@@ -154,18 +154,18 @@ query.granularity('P1D', 'Europe/Berlin')
|
|
154
154
|
|
155
155
|
```ruby
|
156
156
|
# equality
|
157
|
-
Druid::Query.new
|
157
|
+
Druid::Query::Builder.new.having { metric == 10 }
|
158
158
|
```
|
159
159
|
|
160
160
|
```ruby
|
161
161
|
# inequality
|
162
|
-
Druid::Query.new
|
162
|
+
Druid::Query::Builder.new.having { metric != 10 }
|
163
163
|
```
|
164
164
|
|
165
165
|
```ruby
|
166
166
|
# greater, less
|
167
|
-
Druid::Query.new
|
168
|
-
Druid::Query.new
|
167
|
+
Druid::Query::Builder.new.having { metric > 10 }
|
168
|
+
Druid::Query::Builder.new.having { metric < 10 }
|
169
169
|
```
|
170
170
|
|
171
171
|
#### Compound having filters
|
@@ -174,17 +174,17 @@ Having filters can be combined with boolean logic.
|
|
174
174
|
|
175
175
|
```ruby
|
176
176
|
# and
|
177
|
-
Druid::Query.new
|
177
|
+
Druid::Query::Builder.new.having { (metric != 1) & (metric2 != 2) }
|
178
178
|
```
|
179
179
|
|
180
180
|
```ruby
|
181
181
|
# or
|
182
|
-
Druid::Query.new
|
182
|
+
Druid::Query::Builder.new.having { (metric == 1) | (metric2 == 2) }
|
183
183
|
```
|
184
184
|
|
185
185
|
```ruby
|
186
186
|
# not
|
187
|
-
Druid::Query.new
|
187
|
+
Druid::Query::Builder.new.having{ !metric.eq(1) }
|
188
188
|
```
|
189
189
|
|
190
190
|
### Filters
|
@@ -197,27 +197,27 @@ Filters can be chained `filter{...}.filter{...}`
|
|
197
197
|
|
198
198
|
```ruby
|
199
199
|
# equality
|
200
|
-
Druid::Query.new
|
201
|
-
Druid::Query.new
|
200
|
+
Druid::Query::Builder.new.filter{dimension.eq 1}
|
201
|
+
Druid::Query::Builder.new.filter{dimension == 1}
|
202
202
|
```
|
203
203
|
|
204
204
|
```ruby
|
205
205
|
# inequality
|
206
|
-
Druid::Query.new
|
207
|
-
Druid::Query.new
|
206
|
+
Druid::Query::Builder.new.filter{dimension.neq 1}
|
207
|
+
Druid::Query::Builder.new.filter{dimension != 1}
|
208
208
|
```
|
209
209
|
|
210
210
|
```ruby
|
211
211
|
# greater, less
|
212
|
-
Druid::Query.new
|
213
|
-
Druid::Query.new
|
214
|
-
Druid::Query.new
|
215
|
-
Druid::Query.new
|
212
|
+
Druid::Query::Builder.new.filter{dimension > 1}
|
213
|
+
Druid::Query::Builder.new.filter{dimension >= 1}
|
214
|
+
Druid::Query::Builder.new.filter{dimension < 1}
|
215
|
+
Druid::Query::Builder.new.filter{dimension <= 1}
|
216
216
|
```
|
217
217
|
|
218
218
|
```ruby
|
219
219
|
# JavaScript
|
220
|
-
Druid::Query.new
|
220
|
+
Druid::Query::Builder.new.filter{a.javascript('dimension >= 1 && dimension < 5')}
|
221
221
|
```
|
222
222
|
|
223
223
|
#### Compound Filters
|
@@ -226,17 +226,17 @@ Filters can be combined with boolean logic.
|
|
226
226
|
|
227
227
|
```ruby
|
228
228
|
# and
|
229
|
-
Druid::Query.new
|
229
|
+
Druid::Query::Builder.new.filter{dimension.neq 1 & dimension2.neq 2}
|
230
230
|
```
|
231
231
|
|
232
232
|
```ruby
|
233
233
|
# or
|
234
|
-
Druid::Query.new
|
234
|
+
Druid::Query::Builder.new.filter{dimension.neq 1 | dimension2.neq 2}
|
235
235
|
```
|
236
236
|
|
237
237
|
```ruby
|
238
238
|
# not
|
239
|
-
Druid::Query.new
|
239
|
+
Druid::Query::Builder.new.filter{!dimension.eq(1)}
|
240
240
|
```
|
241
241
|
|
242
242
|
#### Inclusion Filter
|
@@ -244,18 +244,18 @@ Druid::Query.new('service/source').filter{!dimension.eq(1)}
|
|
244
244
|
This filter creates a set of equals filters in an or filter.
|
245
245
|
|
246
246
|
```ruby
|
247
|
-
Druid::Query.new
|
247
|
+
Druid::Query::Builder.new.filter{dimension.in(1,2,3)}
|
248
248
|
```
|
249
249
|
#### Geographic filter
|
250
250
|
|
251
251
|
These filters have to be combined with time_series and do only work when coordinates is a spatial dimension [GeographicQueries](http://druid.io/docs/latest/development/geo.html)
|
252
252
|
|
253
253
|
```ruby
|
254
|
-
Druid::Query.new
|
254
|
+
Druid::Query::Builder.new.time_series().long_sum([:aggregate1]).filter{coordinates.in_rec [[50.0,13.0],[54.0,15.0]]}
|
255
255
|
```
|
256
256
|
|
257
257
|
```ruby
|
258
|
-
Druid::Query.new
|
258
|
+
Druid::Query::Builder.new.time_series().long_sum([:aggregate1]).filter{coordinates.in_circ [[53.0,13.0], 5.0]}
|
259
259
|
```
|
260
260
|
|
261
261
|
#### Exclusion Filter
|
@@ -263,7 +263,7 @@ Druid::Query.new('service/source').time_series().long_sum([:aggregate1]).filter{
|
|
263
263
|
This filter creates a set of not-equals fitlers in an and filter.
|
264
264
|
|
265
265
|
```ruby
|
266
|
-
Druid::Query.new
|
266
|
+
Druid::Query::Builder.new.filter{dimension.nin(1,2,3)}
|
267
267
|
```
|
268
268
|
|
269
269
|
#### Hash syntax
|
@@ -271,9 +271,9 @@ Druid::Query.new('service/source').filter{dimension.nin(1,2,3)}
|
|
271
271
|
Sometimes it can be useful to use a hash syntax for filtering for example if you already get them from a list or parameter hash.
|
272
272
|
|
273
273
|
```ruby
|
274
|
-
Druid::Query.new
|
274
|
+
Druid::Query::Builder.new.filter{dimension => 1, dimension1 =>2, dimension2 => 3}
|
275
275
|
# which is equivalent to
|
276
|
-
Druid::Query.new
|
276
|
+
Druid::Query::Builder.new.filter{dimension.eq(1) & dimension1.eq(2) & dimension2.eq(3)}
|
277
277
|
```
|
278
278
|
|
279
279
|
## Contributing
|
data/lib/druid/aggregation.rb
CHANGED
@@ -59,6 +59,34 @@ module Druid
|
|
59
59
|
attr_accessor :byRow
|
60
60
|
validates :byRow, allow_nil: true, inclusion: { in: [true, false] }
|
61
61
|
|
62
|
+
class FilterValidator < ActiveModel::EachValidator
|
63
|
+
TYPES = %w[filtered].freeze
|
64
|
+
def validate_each(record, attribute, value)
|
65
|
+
if TYPES.include?(record.type)
|
66
|
+
record.errors.add(attribute, 'may not be blank') if value.blank?
|
67
|
+
else
|
68
|
+
record.errors.add(attribute, "is not supported by type=#{record.type}") if value
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
attr_accessor :filter
|
74
|
+
validates :filter, filter: true
|
75
|
+
|
76
|
+
class AggregatorValidator < ActiveModel::EachValidator
|
77
|
+
TYPES = %w[filtered].freeze
|
78
|
+
def validate_each(record, attribute, value)
|
79
|
+
if TYPES.include?(record.type)
|
80
|
+
record.errors.add(attribute, 'may not be blank') if value.blank?
|
81
|
+
else
|
82
|
+
record.errors.add(attribute, "is not supported by type=#{record.type}") if value
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
86
|
+
|
87
|
+
attr_accessor :aggregator
|
88
|
+
validates :aggregator, aggregator: true
|
89
|
+
|
62
90
|
def as_json(options = {})
|
63
91
|
super(options.merge(except: %w(errors validation_context)))
|
64
92
|
end
|
data/lib/druid/data_source.rb
CHANGED
@@ -74,22 +74,49 @@ module Druid
|
|
74
74
|
return self.post(query)
|
75
75
|
end
|
76
76
|
|
77
|
-
raise Error.new(response)
|
77
|
+
raise Error.new(response)
|
78
78
|
end
|
79
79
|
|
80
80
|
MultiJson.load(response.body)
|
81
81
|
end
|
82
82
|
|
83
83
|
class Error < StandardError
|
84
|
-
attr_reader :response
|
84
|
+
attr_reader :error, :error_message, :error_class, :host, :response
|
85
|
+
|
85
86
|
def initialize(response)
|
86
87
|
@response = response
|
88
|
+
parsed_body = MultiJson.load(response.body)
|
89
|
+
@error, @error_message, @error_class, @host = parsed_body.values_at(*%w(
|
90
|
+
error
|
91
|
+
errorMessage
|
92
|
+
errorClass
|
93
|
+
host
|
94
|
+
))
|
87
95
|
end
|
88
96
|
|
89
97
|
def message
|
90
|
-
|
98
|
+
error
|
99
|
+
end
|
100
|
+
|
101
|
+
def query_timeout?
|
102
|
+
error == 'Query timeout'.freeze
|
103
|
+
end
|
104
|
+
|
105
|
+
def query_interrupted?
|
106
|
+
error == 'Query interrupted'.freeze
|
91
107
|
end
|
92
|
-
end
|
93
108
|
|
109
|
+
def query_cancelled?
|
110
|
+
error == 'Query cancelled'.freeze
|
111
|
+
end
|
112
|
+
|
113
|
+
def resource_limit_exceeded?
|
114
|
+
error == 'Resource limit exceeded'.freeze
|
115
|
+
end
|
116
|
+
|
117
|
+
def unknown_exception?
|
118
|
+
error == 'Unknown exception'.freeze
|
119
|
+
end
|
120
|
+
end
|
94
121
|
end
|
95
122
|
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
module Druid
|
2
|
+
class Dimension
|
3
|
+
include ActiveModel::Model
|
4
|
+
|
5
|
+
attr_accessor :type
|
6
|
+
validates :type, inclusion: { in: %w(default extraction) }
|
7
|
+
|
8
|
+
attr_accessor :dimension
|
9
|
+
validates :dimension, presence: true
|
10
|
+
|
11
|
+
attr_accessor :outputName
|
12
|
+
validates :outputName, presence: true
|
13
|
+
|
14
|
+
class ExtractionFnValidator < ActiveModel::EachValidator
|
15
|
+
TYPES = %w(extraction)
|
16
|
+
def validate_each(record, attribute, value)
|
17
|
+
if TYPES.include?(record.type)
|
18
|
+
record.errors.add(attribute, 'may not be blank') if value.blank?
|
19
|
+
else
|
20
|
+
record.errors.add(attribute, "is not supported by type=#{record.type}") if value
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
attr_accessor :extractionFn
|
26
|
+
validates :extractionFn, extraction_fn: true
|
27
|
+
|
28
|
+
def initialize(params)
|
29
|
+
if params.is_a?(Hash)
|
30
|
+
super
|
31
|
+
else
|
32
|
+
super(type: 'default', dimension: params.to_s, outputName: params.to_s)
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
def as_json(options = {})
|
37
|
+
super(options.merge(except: %w(errors validation_context)))
|
38
|
+
end
|
39
|
+
|
40
|
+
def self.lookup(dimension, namespace, outputName: nil, retain: true, injective: false)
|
41
|
+
new({
|
42
|
+
type: 'extraction',
|
43
|
+
dimension: dimension,
|
44
|
+
outputName: outputName || namespace,
|
45
|
+
extractionFn: {
|
46
|
+
type: 'registeredLookup',
|
47
|
+
lookup: namespace,
|
48
|
+
retainMissingValue: retain,
|
49
|
+
injective: injective,
|
50
|
+
},
|
51
|
+
})
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
data/lib/druid/filter.rb
CHANGED
@@ -166,6 +166,14 @@ module Druid
|
|
166
166
|
CircFilter.new(@dimension, bounds)
|
167
167
|
end
|
168
168
|
|
169
|
+
def bound(params)
|
170
|
+
BoundFilter.new(@dimension, params)
|
171
|
+
end
|
172
|
+
|
173
|
+
def search(params)
|
174
|
+
SearchFilter.new(@dimension, params)
|
175
|
+
end
|
176
|
+
|
169
177
|
def eq(value)
|
170
178
|
case value
|
171
179
|
when ::Array
|
@@ -299,7 +307,35 @@ module Druid
|
|
299
307
|
@bound = {
|
300
308
|
type: 'radius',
|
301
309
|
coords: bounds.first,
|
302
|
-
radius: bounds.last
|
310
|
+
radius: bounds.last
|
311
|
+
}
|
312
|
+
end
|
313
|
+
end
|
314
|
+
|
315
|
+
class BoundFilter < Filter
|
316
|
+
include BooleanOperators
|
317
|
+
|
318
|
+
def initialize(dimension, params)
|
319
|
+
super()
|
320
|
+
@type = 'bound'
|
321
|
+
@dimension = dimension
|
322
|
+
@ordering = params[:ordering]
|
323
|
+
@upper = params[:upper]
|
324
|
+
@upperStrict = params[:upperStrict]
|
325
|
+
end
|
326
|
+
end
|
327
|
+
|
328
|
+
class SearchFilter < Filter
|
329
|
+
include BooleanOperators
|
330
|
+
|
331
|
+
def initialize(dimension, params)
|
332
|
+
super()
|
333
|
+
@type = 'search'
|
334
|
+
@dimension = dimension
|
335
|
+
@query = {
|
336
|
+
type: 'contains',
|
337
|
+
value: params[:value],
|
338
|
+
caseSensitive: params[:case_sensitive] || false
|
303
339
|
}
|
304
340
|
end
|
305
341
|
end
|
data/lib/druid/query.rb
CHANGED
@@ -5,6 +5,7 @@ require 'active_support/all'
|
|
5
5
|
require 'active_model'
|
6
6
|
|
7
7
|
require 'druid/granularity'
|
8
|
+
require 'druid/dimension'
|
8
9
|
require 'druid/aggregation'
|
9
10
|
require 'druid/post_aggregation'
|
10
11
|
require 'druid/filter'
|
@@ -80,7 +81,16 @@ module Druid
|
|
80
81
|
TYPES = %w(groupBy select)
|
81
82
|
def validate_each(record, attribute, value)
|
82
83
|
if TYPES.include?(record.queryType)
|
83
|
-
|
84
|
+
if !value.is_a?(Array) || value.blank?
|
85
|
+
record.errors.add(attribute, 'must be a list with at least one dimension')
|
86
|
+
else
|
87
|
+
value.each(&:valid?) # trigger validation
|
88
|
+
value.each do |avalue|
|
89
|
+
avalue.errors.messages.each do |k, v|
|
90
|
+
record.errors.add(attribute, { k => v })
|
91
|
+
end
|
92
|
+
end
|
93
|
+
end
|
84
94
|
else
|
85
95
|
record.errors.add(attribute, "is not supported by type=#{record.queryType}") if value
|
86
96
|
end
|
@@ -90,14 +100,34 @@ module Druid
|
|
90
100
|
attr_accessor :dimensions
|
91
101
|
validates :dimensions, dimensions: true
|
92
102
|
|
103
|
+
def dimensions
|
104
|
+
@dimensions ||= []
|
105
|
+
end
|
106
|
+
|
107
|
+
def dimensions=(value)
|
108
|
+
if value.is_a?(Array)
|
109
|
+
@dimensions = value.map do |x|
|
110
|
+
x.is_a?(Dimension) ? x : Dimension.new(x)
|
111
|
+
end
|
112
|
+
else
|
113
|
+
@dimensions = [
|
114
|
+
value.is_a?(Dimension) ? value : Dimension.new(value)
|
115
|
+
]
|
116
|
+
end
|
117
|
+
end
|
118
|
+
|
93
119
|
class AggregationsValidator < ActiveModel::EachValidator
|
94
120
|
TYPES = %w(timeseries groupBy topN)
|
95
121
|
def validate_each(record, attribute, value)
|
96
122
|
if TYPES.include?(record.queryType)
|
97
|
-
value.
|
98
|
-
|
99
|
-
|
100
|
-
|
123
|
+
if !value.is_a?(Array) || value.blank?
|
124
|
+
record.errors.add(attribute, 'must be a list with at least one aggregator')
|
125
|
+
else
|
126
|
+
value.each(&:valid?) # trigger validation
|
127
|
+
value.each do |avalue|
|
128
|
+
avalue.errors.messages.each do |k, v|
|
129
|
+
record.errors.add(attribute, { k => v })
|
130
|
+
end
|
101
131
|
end
|
102
132
|
end
|
103
133
|
else
|
@@ -116,10 +146,12 @@ module Druid
|
|
116
146
|
def aggregations=(value)
|
117
147
|
if value.is_a?(Array)
|
118
148
|
@aggregations = value.map do |x|
|
119
|
-
Aggregation.new(x)
|
149
|
+
x.is_a?(Aggregation) ? x : Aggregation.new(x)
|
120
150
|
end
|
121
151
|
else
|
122
|
-
@aggregations = [
|
152
|
+
@aggregations = [
|
153
|
+
value.is_a?(Aggregation) ? value : Aggregation.new(value)
|
154
|
+
]
|
123
155
|
end
|
124
156
|
end
|
125
157
|
|
@@ -333,7 +365,9 @@ module Druid
|
|
333
365
|
|
334
366
|
def group_by(*dimensions)
|
335
367
|
query_type(:groupBy)
|
336
|
-
@query.dimensions = dimensions.flatten
|
368
|
+
@query.dimensions = dimensions.flatten.map do |dimension|
|
369
|
+
dimension.is_a?(Dimension) ? dimension : Dimension.new(dimension)
|
370
|
+
end
|
337
371
|
self
|
338
372
|
end
|
339
373
|
|
@@ -417,6 +451,19 @@ module Druid
|
|
417
451
|
self
|
418
452
|
end
|
419
453
|
|
454
|
+
def filtered_aggregation(metric, name, aggregation_type, &filter)
|
455
|
+
@query.aggregations << Aggregation.new(
|
456
|
+
type: 'filtered',
|
457
|
+
filter: Filter.new.instance_exec(&filter),
|
458
|
+
aggregator: Aggregation.new(
|
459
|
+
type: aggregation_type.to_s.camelize(:lower),
|
460
|
+
name: name,
|
461
|
+
fieldName: metric
|
462
|
+
)
|
463
|
+
) unless @query.contains_aggregation?(name)
|
464
|
+
self
|
465
|
+
end
|
466
|
+
|
420
467
|
## post aggregations
|
421
468
|
|
422
469
|
def postagg(type = :long_sum, &block)
|
data/lib/druid/version.rb
CHANGED
data/lib/druid/zk.rb
CHANGED
@@ -13,37 +13,37 @@ module Druid
|
|
13
13
|
end
|
14
14
|
|
15
15
|
def register
|
16
|
-
$log.
|
16
|
+
$log.debug("druid.zk register discovery path") if $log
|
17
17
|
@zk.on_expired_session { register }
|
18
18
|
@zk.register(@discovery_path, only: :child) do |event|
|
19
|
-
$log.
|
19
|
+
$log.debug("druid.zk got event on discovery path") if $log
|
20
20
|
check_services
|
21
21
|
end
|
22
22
|
check_services
|
23
23
|
end
|
24
24
|
|
25
25
|
def close!
|
26
|
-
$log.
|
26
|
+
$log.debug("druid.zk shutting down") if $log
|
27
27
|
@zk.close!
|
28
28
|
end
|
29
29
|
|
30
30
|
def register_service(service, brokers)
|
31
|
-
$log.
|
31
|
+
$log.debug("druid.zk register", service: service, brokers: brokers) if $log
|
32
32
|
# poor mans load balancing
|
33
33
|
@registry[service] = brokers.shuffle
|
34
34
|
end
|
35
35
|
|
36
36
|
def unregister_service(service)
|
37
|
-
$log.
|
37
|
+
$log.debug("druid.zk unregister", service: service) if $log
|
38
38
|
@registry.delete(service)
|
39
39
|
unwatch_service(service)
|
40
40
|
end
|
41
41
|
|
42
42
|
def watch_service(service)
|
43
43
|
return if @watched_services.include?(service)
|
44
|
-
$log.
|
44
|
+
$log.debug("druid.zk watch", service: service) if $log
|
45
45
|
watch = @zk.register(watch_path(service), only: :child) do |event|
|
46
|
-
$log.
|
46
|
+
$log.debug("druid.zk got event on watch path for", service: service, event: event) if $log
|
47
47
|
unwatch_service(service)
|
48
48
|
check_service(service)
|
49
49
|
end
|
@@ -52,12 +52,12 @@ module Druid
|
|
52
52
|
|
53
53
|
def unwatch_service(service)
|
54
54
|
return unless @watched_services.include?(service)
|
55
|
-
$log.
|
55
|
+
$log.debug("druid.zk unwatch", service: service) if $log
|
56
56
|
@watched_services.delete(service).unregister
|
57
57
|
end
|
58
58
|
|
59
59
|
def check_services
|
60
|
-
$log.
|
60
|
+
$log.debug("druid.zk checking services") if $log
|
61
61
|
zk_services = @zk.children(@discovery_path, watch: true)
|
62
62
|
|
63
63
|
(services - zk_services).each do |service|
|
@@ -70,7 +70,7 @@ module Druid
|
|
70
70
|
end
|
71
71
|
|
72
72
|
def verify_broker(service, name)
|
73
|
-
$log.
|
73
|
+
$log.debug("druid.zk verify", broker: name, service: service) if $log
|
74
74
|
info = @zk.get("#{watch_path(service)}/#{name}")
|
75
75
|
node = MultiJson.load(info[0])
|
76
76
|
uri = "http://#{node['address']}:#{node['port']}/druid/v2/"
|
@@ -78,7 +78,7 @@ module Druid
|
|
78
78
|
method: :get, url: "#{uri}datasources/",
|
79
79
|
timeout: 5, open_timeout: 5
|
80
80
|
})
|
81
|
-
$log.
|
81
|
+
$log.debug("druid.zk verified", uri: uri, sources: check) if $log
|
82
82
|
return [uri, MultiJson.load(check.to_str)] if check.code == 200
|
83
83
|
rescue
|
84
84
|
return false
|
@@ -96,7 +96,7 @@ module Druid
|
|
96
96
|
known = @registry[service].map { |node| node[:name] }
|
97
97
|
live = @zk.children(watch_path(service), watch: true)
|
98
98
|
new_list = @registry[service].select { |node| live.include?(node[:name]) }
|
99
|
-
$log.
|
99
|
+
$log.debug("druid.zk checking", service: service, known: known, live: live, new_list: new_list) if $log
|
100
100
|
|
101
101
|
# verify the new entries to be living brokers
|
102
102
|
(live - known).each do |name|
|
data/ruby-druid.gemspec
CHANGED
@@ -19,13 +19,13 @@ Gem::Specification.new do |spec|
|
|
19
19
|
spec.test_files = Dir["spec/**/*"]
|
20
20
|
spec.require_paths = ["lib"]
|
21
21
|
|
22
|
-
spec.add_dependency "activesupport", "
|
23
|
-
spec.add_dependency "activemodel", "
|
24
|
-
spec.add_dependency "iso8601", "~> 0.
|
25
|
-
spec.add_dependency "multi_json", "~> 1.
|
26
|
-
spec.add_dependency "rest-client", "
|
22
|
+
spec.add_dependency "activesupport", ">= 3.0.0"
|
23
|
+
spec.add_dependency "activemodel", ">= 3.0.0"
|
24
|
+
spec.add_dependency "iso8601", "~> 0.8"
|
25
|
+
spec.add_dependency "multi_json", "~> 1.0"
|
26
|
+
spec.add_dependency "rest-client", ">= 1.8", "< 3.0"
|
27
27
|
spec.add_dependency "zk", "~> 1.9"
|
28
|
-
spec.add_development_dependency "bundler", "
|
28
|
+
spec.add_development_dependency "bundler", ">= 1.3.0", "< 2.2"
|
29
29
|
spec.add_development_dependency "rake", "~> 11.2"
|
30
30
|
spec.add_development_dependency "rspec", "~> 3.4"
|
31
31
|
spec.add_development_dependency "webmock", "~> 2.1"
|
@@ -19,20 +19,35 @@ describe Druid::DataSource do
|
|
19
19
|
end
|
20
20
|
|
21
21
|
it 'raises on request failure' do
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
:headers => {
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
22
|
+
stub_request(:post, 'http://www.example.com/druid/v2')
|
23
|
+
.with(
|
24
|
+
:body => %q({"context":{"queryId":null},"queryType":"timeseries","intervals":["2013-04-04T00:00:00+00:00/2013-04-04T00:00:00+00:00"],"granularity":"all","dataSource":"test"}),
|
25
|
+
:headers => {
|
26
|
+
'Accept' => '*/*',
|
27
|
+
'Accept-Encoding' => 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3',
|
28
|
+
'Content-Type' => 'application/json',
|
29
|
+
'User-Agent' => 'Ruby',
|
30
|
+
}
|
31
|
+
)
|
32
|
+
.to_return(
|
33
|
+
:status => 500,
|
34
|
+
:body => %q({"error":"Unknown exception","errorMessage":"NullPointerException","errorClass":"java.lang.NullPointerException","host":"www.example.com"}),
|
35
|
+
:headers => {},
|
36
|
+
)
|
37
|
+
|
32
38
|
ds = Druid::DataSource.new('test/test', 'http://www.example.com/druid/v2')
|
33
39
|
query = Druid::Query::Builder.new.interval('2013-04-04', '2013-04-04').granularity(:all).query
|
34
40
|
query.context.queryId = nil
|
35
|
-
|
41
|
+
|
42
|
+
expect { ds.post(query) }.to raise_error { |error|
|
43
|
+
expect(error).to be_a(Druid::DataSource::Error)
|
44
|
+
expect(error.message).to eq(error.error)
|
45
|
+
expect(error.error).to eq('Unknown exception')
|
46
|
+
expect(error.error_message).to eq('NullPointerException')
|
47
|
+
expect(error.error_class).to eq('java.lang.NullPointerException')
|
48
|
+
expect(error.host).to eq('www.example.com')
|
49
|
+
expect(error).to be_unknown_exception
|
50
|
+
}
|
36
51
|
end
|
37
52
|
end
|
38
53
|
|
data/spec/lib/query_spec.rb
CHANGED
@@ -21,7 +21,9 @@ describe Druid::Query do
|
|
21
21
|
|
22
22
|
it 'takes dimensions from group_by method' do
|
23
23
|
@query.group_by(:a, :b, :c)
|
24
|
-
expect(JSON.parse(@query.query.to_json)['dimensions']).to eq([
|
24
|
+
expect(JSON.parse(@query.query.to_json)['dimensions']).to eq([{"type"=>"default", "dimension"=>"a", "outputName"=>"a"},
|
25
|
+
{"type"=>"default", "dimension"=>"b", "outputName"=>"b"},
|
26
|
+
{"type"=>"default", "dimension"=>"c", "outputName"=>"c"}])
|
25
27
|
end
|
26
28
|
|
27
29
|
it 'takes dimension, metric and threshold from topn method' do
|
@@ -207,6 +209,32 @@ describe Druid::Query do
|
|
207
209
|
end
|
208
210
|
end
|
209
211
|
|
212
|
+
describe '#filtered_aggregation' do
|
213
|
+
it 'builds filtered aggregations' do
|
214
|
+
@query.filtered_aggregation(:a, :a_filtered, :longSum) do
|
215
|
+
b.eq(2) & c.neq(3)
|
216
|
+
end
|
217
|
+
expect(JSON.parse(@query.query.to_json)['aggregations']).to eq [
|
218
|
+
{
|
219
|
+
'type' => 'filtered',
|
220
|
+
'filter' => {
|
221
|
+
'type' => 'and',
|
222
|
+
'fields' => [
|
223
|
+
{ 'dimension' => 'b', 'type' => 'selector', 'value' => 2 },
|
224
|
+
{
|
225
|
+
'type' => 'not',
|
226
|
+
'field' => {
|
227
|
+
'dimension' => 'c', 'type' => 'selector', 'value' => 3
|
228
|
+
}
|
229
|
+
}
|
230
|
+
]
|
231
|
+
},
|
232
|
+
'aggregator' => { 'type' => 'longSum', 'name' => 'a_filtered', 'fieldName' => 'a' }
|
233
|
+
}
|
234
|
+
]
|
235
|
+
end
|
236
|
+
end
|
237
|
+
|
210
238
|
it 'appends long_sum properties from aggregations on calling long_sum again' do
|
211
239
|
@query.long_sum(:a, :b, :c)
|
212
240
|
@query.double_sum(:x,:y)
|
metadata
CHANGED
@@ -1,85 +1,91 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: ruby-druid
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.11.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ruby Druid Community
|
8
|
-
autorequire:
|
8
|
+
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-11-04 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activesupport
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - "
|
17
|
+
- - ">="
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version:
|
19
|
+
version: 3.0.0
|
20
20
|
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - "
|
24
|
+
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version:
|
26
|
+
version: 3.0.0
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: activemodel
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
|
-
- - "
|
31
|
+
- - ">="
|
32
32
|
- !ruby/object:Gem::Version
|
33
|
-
version:
|
33
|
+
version: 3.0.0
|
34
34
|
type: :runtime
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
|
-
- - "
|
38
|
+
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
|
-
version:
|
40
|
+
version: 3.0.0
|
41
41
|
- !ruby/object:Gem::Dependency
|
42
42
|
name: iso8601
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
44
44
|
requirements:
|
45
45
|
- - "~>"
|
46
46
|
- !ruby/object:Gem::Version
|
47
|
-
version: '0.
|
47
|
+
version: '0.8'
|
48
48
|
type: :runtime
|
49
49
|
prerelease: false
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
51
51
|
requirements:
|
52
52
|
- - "~>"
|
53
53
|
- !ruby/object:Gem::Version
|
54
|
-
version: '0.
|
54
|
+
version: '0.8'
|
55
55
|
- !ruby/object:Gem::Dependency
|
56
56
|
name: multi_json
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
58
58
|
requirements:
|
59
59
|
- - "~>"
|
60
60
|
- !ruby/object:Gem::Version
|
61
|
-
version: '1.
|
61
|
+
version: '1.0'
|
62
62
|
type: :runtime
|
63
63
|
prerelease: false
|
64
64
|
version_requirements: !ruby/object:Gem::Requirement
|
65
65
|
requirements:
|
66
66
|
- - "~>"
|
67
67
|
- !ruby/object:Gem::Version
|
68
|
-
version: '1.
|
68
|
+
version: '1.0'
|
69
69
|
- !ruby/object:Gem::Dependency
|
70
70
|
name: rest-client
|
71
71
|
requirement: !ruby/object:Gem::Requirement
|
72
72
|
requirements:
|
73
|
-
- - "
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '1.8'
|
76
|
+
- - "<"
|
74
77
|
- !ruby/object:Gem::Version
|
75
|
-
version: '
|
78
|
+
version: '3.0'
|
76
79
|
type: :runtime
|
77
80
|
prerelease: false
|
78
81
|
version_requirements: !ruby/object:Gem::Requirement
|
79
82
|
requirements:
|
80
|
-
- - "
|
83
|
+
- - ">="
|
84
|
+
- !ruby/object:Gem::Version
|
85
|
+
version: '1.8'
|
86
|
+
- - "<"
|
81
87
|
- !ruby/object:Gem::Version
|
82
|
-
version: '
|
88
|
+
version: '3.0'
|
83
89
|
- !ruby/object:Gem::Dependency
|
84
90
|
name: zk
|
85
91
|
requirement: !ruby/object:Gem::Requirement
|
@@ -98,16 +104,22 @@ dependencies:
|
|
98
104
|
name: bundler
|
99
105
|
requirement: !ruby/object:Gem::Requirement
|
100
106
|
requirements:
|
101
|
-
- - "
|
107
|
+
- - ">="
|
108
|
+
- !ruby/object:Gem::Version
|
109
|
+
version: 1.3.0
|
110
|
+
- - "<"
|
102
111
|
- !ruby/object:Gem::Version
|
103
|
-
version: '
|
112
|
+
version: '2.2'
|
104
113
|
type: :development
|
105
114
|
prerelease: false
|
106
115
|
version_requirements: !ruby/object:Gem::Requirement
|
107
116
|
requirements:
|
108
|
-
- - "
|
117
|
+
- - ">="
|
118
|
+
- !ruby/object:Gem::Version
|
119
|
+
version: 1.3.0
|
120
|
+
- - "<"
|
109
121
|
- !ruby/object:Gem::Version
|
110
|
-
version: '
|
122
|
+
version: '2.2'
|
111
123
|
- !ruby/object:Gem::Dependency
|
112
124
|
name: rake
|
113
125
|
requirement: !ruby/object:Gem::Requirement
|
@@ -153,7 +165,7 @@ dependencies:
|
|
153
165
|
description: |2
|
154
166
|
ruby-druid is a Ruby client for Druid. It includes a Squeel-like query DSL
|
155
167
|
and generates a JSON query that can be sent to Druid directly.
|
156
|
-
email:
|
168
|
+
email:
|
157
169
|
executables: []
|
158
170
|
extensions: []
|
159
171
|
extra_rdoc_files: []
|
@@ -165,6 +177,7 @@ files:
|
|
165
177
|
- lib/druid/client.rb
|
166
178
|
- lib/druid/context.rb
|
167
179
|
- lib/druid/data_source.rb
|
180
|
+
- lib/druid/dimension.rb
|
168
181
|
- lib/druid/filter.rb
|
169
182
|
- lib/druid/granularity.rb
|
170
183
|
- lib/druid/having.rb
|
@@ -182,7 +195,7 @@ homepage: https://github.com/ruby-druid/ruby-druid
|
|
182
195
|
licenses:
|
183
196
|
- MIT
|
184
197
|
metadata: {}
|
185
|
-
post_install_message:
|
198
|
+
post_install_message:
|
186
199
|
rdoc_options: []
|
187
200
|
require_paths:
|
188
201
|
- lib
|
@@ -197,14 +210,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
197
210
|
- !ruby/object:Gem::Version
|
198
211
|
version: '0'
|
199
212
|
requirements: []
|
200
|
-
|
201
|
-
|
202
|
-
signing_key:
|
213
|
+
rubygems_version: 3.0.3
|
214
|
+
signing_key:
|
203
215
|
specification_version: 4
|
204
216
|
summary: A Ruby client for Druid
|
205
217
|
test_files:
|
218
|
+
- spec/spec_helper.rb
|
206
219
|
- spec/lib/client_spec.rb
|
220
|
+
- spec/lib/zk_spec.rb
|
207
221
|
- spec/lib/data_source_spec.rb
|
208
222
|
- spec/lib/query_spec.rb
|
209
|
-
- spec/lib/zk_spec.rb
|
210
|
-
- spec/spec_helper.rb
|