xcflushd 1.0.0.rc2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.codeclimate.yml +31 -0
- data/.gitignore +8 -0
- data/.rspec +3 -0
- data/.rubocop.yml +1156 -0
- data/.ruby-gemset +1 -0
- data/.ruby-version +1 -0
- data/.simplecov +3 -0
- data/.travis.yml +5 -0
- data/Dockerfile +99 -0
- data/Gemfile +4 -0
- data/Gemfile.lock +78 -0
- data/LICENSE +202 -0
- data/Makefile +17 -0
- data/NOTICE +14 -0
- data/README.md +118 -0
- data/Rakefile +6 -0
- data/bin/console +14 -0
- data/bin/setup +8 -0
- data/docs/design.md +100 -0
- data/exe/xcflushd +114 -0
- data/lib/xcflushd/3scale_client_ext.rb +12 -0
- data/lib/xcflushd/authorization.rb +49 -0
- data/lib/xcflushd/authorizer.rb +122 -0
- data/lib/xcflushd/credentials.rb +66 -0
- data/lib/xcflushd/flusher.rb +146 -0
- data/lib/xcflushd/flusher_error_handler.rb +78 -0
- data/lib/xcflushd/gli_helpers.rb +83 -0
- data/lib/xcflushd/logger.rb +9 -0
- data/lib/xcflushd/priority_auth_renewer.rb +253 -0
- data/lib/xcflushd/reporter.rb +70 -0
- data/lib/xcflushd/runner.rb +165 -0
- data/lib/xcflushd/storage.rb +263 -0
- data/lib/xcflushd/storage_keys.rb +113 -0
- data/lib/xcflushd/threading.rb +12 -0
- data/lib/xcflushd/version.rb +3 -0
- data/lib/xcflushd.rb +11 -0
- data/script/test +10 -0
- data/xcflushd.gemspec +39 -0
- metadata +266 -0
data/docs/design.md
ADDED
@@ -0,0 +1,100 @@
|
|
1
|
+
# DESIGN
|
2
|
+
|
3
|
+
## Description
|
4
|
+
|
5
|
+
xcflushd is a daemon used together with [XC](https://github.com/3scale/apicast-xc),
|
6
|
+
which is a module for [Apicast](https://github.com/3scale/apicast), 3scale's
|
7
|
+
API Gateway.
|
8
|
+
|
9
|
+
If you are not familiar with XC yet, we recommend you to start reading its
|
10
|
+
documentation before reading this document.
|
11
|
+
|
12
|
+
In XC, the xcflushd daemon is responsible for these three things:
|
13
|
+
|
14
|
+
* Reporting to 3scale's backend the reports cached in XC.
|
15
|
+
* Updating the status of the authorizations cached in XC.
|
16
|
+
* Retrieving an authorization status from 3scale's backend when the
|
17
|
+
authorization is not cached in XC.
|
18
|
+
|
19
|
+
It is important to keep in mind that the first 2 always happen in background,
|
20
|
+
while the third happens in request time.
|
21
|
+
|
22
|
+
|
23
|
+
## How xcflushd works
|
24
|
+
|
25
|
+
Once xcflushd starts, it will:
|
26
|
+
|
27
|
+
* Start "flushing" periodically to 3scale.
|
28
|
+
* Start listening in a the Redis pubsub channel that XC uses to ask for
|
29
|
+
authorizations that are not cached.
|
30
|
+
|
31
|
+
Let us explain those 2 operations in more detail.
|
32
|
+
|
33
|
+
|
34
|
+
### Periodic flushing
|
35
|
+
|
36
|
+
The flushing consists of 4 steps:
|
37
|
+
|
38
|
+
1. Retrieve the cached reports from Redis.
|
39
|
+
2. Send those cached reports to 3scale's backend.
|
40
|
+
3. Wait for a few seconds.
|
41
|
+
4. Renew the cached authorizations in Redis of the applications that appear in
|
42
|
+
the reports that have just been sent to 3scale.
|
43
|
+
|
44
|
+
You might be wondering why there's a waiting time between sending the cached
|
45
|
+
reports and renewing the cached authorizations. The reason is that reporting is
|
46
|
+
asynchronous in 3scale's backend API. That means that, when reporting, we'll
|
47
|
+
get an OK http response code if our reports do not contain any format errors
|
48
|
+
and the credentials are OK, but we do not have a way to know when those reports
|
49
|
+
are effective and considered for rate limiting. We know that 3scale is pretty
|
50
|
+
quick doing that, though! This is a trade-off that 3scale makes between
|
51
|
+
request latency and accurateness of rate limits.
|
52
|
+
|
53
|
+
Another important thing is that when renewing cached authorizations, xcflushd
|
54
|
+
does not only renew the ones of the metrics that appear in the cached reports.
|
55
|
+
The call to 3scale backend allows us to ask for the limits of most of the
|
56
|
+
metrics of an application, so we take advantage of that and renew all the ones
|
57
|
+
we can in one network round-trip. More specifically, the ones that are not
|
58
|
+
renewed are the ones that meet these two conditions: 1) have not been included
|
59
|
+
in the reports sent to 3scale's backend, and 2) do not have any limits defined.
|
60
|
+
|
61
|
+
The whole flushing process makes 2 calls to 3scale per application reported.
|
62
|
+
One call to send the report to 3scale, and another one to get the current
|
63
|
+
authorization status. Compare that to making one request for each one that
|
64
|
+
arrives to the proxy as in the case of Apicast. Imagine that you have 1k rps,
|
65
|
+
and 100 apps. If you define a period of 1 minute for the flushing process, you
|
66
|
+
will be making 200 requests to 3scale's backend per minute instead of 60k.
|
67
|
+
|
68
|
+
We use the [3scale client gem](https://github.com/3scale/3scale_ws_api_for_ruby)
|
69
|
+
to make the requests to 3scale's backend.
|
70
|
+
|
71
|
+
|
72
|
+
### Requests via Redis pubsub
|
73
|
+
|
74
|
+
xcflushd offers a mechanism that allows clients to ask for a specific
|
75
|
+
authorization status without having to wait for the next flush cycle. This
|
76
|
+
mechanism is based on Redis pubsub.
|
77
|
+
|
78
|
+
xcflushd subscribes to a channel to which clients publish messages with a
|
79
|
+
`(service, app credentials, metric)` tuple encoded. When xcflushd receives one
|
80
|
+
of those messages, it makes a request to the 3scale backend to check the
|
81
|
+
authorization status of the given `(service, app credentials, metric)` tuple,
|
82
|
+
and it publishes the result to another pubsub channel to which clients need to
|
83
|
+
subscribe. To make the most out of this network round-trip to 3scale's backend,
|
84
|
+
we take the opportunity to store in the Redis cache the authorization statuses
|
85
|
+
that we just retrieved. Although the message only contained one metric, we
|
86
|
+
renew all the ones we can get in a single request to 3scale's backend. Just
|
87
|
+
like we do in the case of the periodic flushing detailed above.
|
88
|
+
|
89
|
+
xcflushd might receive several requests at the same time about the same
|
90
|
+
`(service, app credentials, metric)` tuple. xcflushd takes this into account
|
91
|
+
and does not make extra requests to 3scale's backend for authorizations that
|
92
|
+
are already being checked.
|
93
|
+
|
94
|
+
|
95
|
+
## Redis keys
|
96
|
+
|
97
|
+
The format of the Redis keys used is specified in the `Xcflushd::StorageKeys`
|
98
|
+
class. The keys need to follow the same format as the one defined in the XC
|
99
|
+
lua module. Check the [XC lua module design doc](https://github.com/3scale/apicast-xc/blob/master/doc/design.md#redis-keys-format)
|
100
|
+
for more info about this.
|
data/exe/xcflushd
ADDED
@@ -0,0 +1,114 @@
|
|
1
|
+
#!/usr/bin/env ruby
|
2
|
+
require 'gli'
|
3
|
+
require 'redis'
|
4
|
+
require 'xcflushd/gli_helpers'
|
5
|
+
|
6
|
+
include GLI::App
|
7
|
+
include Xcflushd::GLIHelpers
|
8
|
+
|
9
|
+
program_desc 'XC flush daemon'
|
10
|
+
|
11
|
+
version Xcflushd::VERSION
|
12
|
+
|
13
|
+
subcommand_option_handling :normal
|
14
|
+
arguments :strict
|
15
|
+
|
16
|
+
desc 'Starts the XC flusher'
|
17
|
+
arg_name ' '
|
18
|
+
command :run do |c|
|
19
|
+
c.desc 'Redis URI'
|
20
|
+
c.flag [:r, :redis], required: true, type: RedisURI
|
21
|
+
|
22
|
+
c.desc '3scale backend URI'
|
23
|
+
c.flag [:b, :backend], required: true, type: BackendURI
|
24
|
+
|
25
|
+
c.desc '3scale provider key'
|
26
|
+
c.flag [:k, :'provider-key'], required: true
|
27
|
+
|
28
|
+
c.desc 'Validity of authorizations (in seconds)'
|
29
|
+
c.flag [:a, :'auth-ttl'], required: true, type: Integer, must_match: POSITIVE_N_RE
|
30
|
+
|
31
|
+
c.desc 'Reporting frequency (in seconds)'
|
32
|
+
c.flag [:f, :frequency], required: true, type: Integer, must_match: POSITIVE_N_RE
|
33
|
+
|
34
|
+
c.desc 'Number of threads for the main thread pool (min:max)'
|
35
|
+
c.flag [:t, :threads], default_value: 'auto', type: PositiveMinMaxInt
|
36
|
+
|
37
|
+
c.desc 'Number of threads for the priority auth renewer (min:max)'
|
38
|
+
c.flag [:p, :'prio-threads'], default_value: 'auto', type: PositiveMinMaxInt
|
39
|
+
|
40
|
+
c.desc 'Run this program as a background service'
|
41
|
+
c.switch :daemonize, default_value: false, negatable: true, multiple: true
|
42
|
+
|
43
|
+
c.desc 'Use HTTPS scheme to connect to 3scale backend'
|
44
|
+
c.switch :secure, default_value: true, negatable: true, multiple: true
|
45
|
+
|
46
|
+
c.action do |_global_options, options, _args|
|
47
|
+
# options contains 2 keys for each option. One is a String and the other a
|
48
|
+
# Symbol. That way we can use options['redis'] and options[:redis].
|
49
|
+
# That's fine as long as we only read from the hash. If we need to modify
|
50
|
+
# it, it is problematic because we need to remember to modify two values.
|
51
|
+
# For simplicity, let's just keep the elements that have symbols as keys.
|
52
|
+
options.keep_if { |k, _v| k.is_a?(Symbol) }
|
53
|
+
|
54
|
+
# Reporting frequency should be lower or equal to the authorization TTL.
|
55
|
+
# Otherwise authorizations would be renewed without taking into account the
|
56
|
+
# last available reports. Note: whenever we do reporting we also do renewal
|
57
|
+
# of authorizations; the TTL is important in case of problems renewing
|
58
|
+
# authorizations.
|
59
|
+
if options[:frequency] > options[:'auth-ttl']
|
60
|
+
exit_now!('frequency needs to be <= auth-ttl')
|
61
|
+
end
|
62
|
+
|
63
|
+
if (options[:backend].scheme == 'http' && options[:secure]) ||
|
64
|
+
(options[:backend].scheme == 'https' && !options[:secure])
|
65
|
+
exit_now!("the specified backend scheme does not match the --[no-]secure" \
|
66
|
+
" flag.\nCan't continue with conflicting settings.")
|
67
|
+
end
|
68
|
+
|
69
|
+
if options[:threads] == 'auto'
|
70
|
+
options.delete :threads
|
71
|
+
options.delete :t
|
72
|
+
end
|
73
|
+
|
74
|
+
if options[:'prio-threads'] == 'auto'
|
75
|
+
options.delete :'prio-threads'
|
76
|
+
options.delete :p
|
77
|
+
end
|
78
|
+
|
79
|
+
banner = "xcflushd [#{Xcflushd::VERSION}]"
|
80
|
+
|
81
|
+
if options[:daemonize]
|
82
|
+
require 'daemons'
|
83
|
+
Daemons.run_proc(banner) { start_xcflusher(options) }
|
84
|
+
else
|
85
|
+
set_title(banner)
|
86
|
+
start_xcflusher(options)
|
87
|
+
end
|
88
|
+
end
|
89
|
+
end
|
90
|
+
|
91
|
+
pre do |_global, _command, _options, _args|
|
92
|
+
# Pre logic here
|
93
|
+
# Return true to proceed; false to abort and not call the
|
94
|
+
# chosen command
|
95
|
+
# Use skips_pre before a command to skip this block
|
96
|
+
# on that command only
|
97
|
+
true
|
98
|
+
end
|
99
|
+
|
100
|
+
post do |_global, _command, _options, _args|
|
101
|
+
# Post logic here
|
102
|
+
# Use skips_post before a command to skip this
|
103
|
+
# block on that command only
|
104
|
+
end
|
105
|
+
|
106
|
+
on_error do |exception|
|
107
|
+
# Error logic here
|
108
|
+
# return false to skip default error handling
|
109
|
+
# When forking, the parent executes 'exit', and GLI error handling kicks in.
|
110
|
+
# Let's just skip it when that's the case.
|
111
|
+
not SystemExit === exception
|
112
|
+
end
|
113
|
+
|
114
|
+
exit run(ARGV)
|
@@ -0,0 +1,12 @@
|
|
1
|
+
# We need to configure the ThreeScale::Client::HTTPClient class of the
|
2
|
+
# 3scale_client gem before we use it.
|
3
|
+
# Internally, the 3scale_client uses Net::HTTP with the keep-alive option
|
4
|
+
# enabled when it is available and the net-http-persistent gem
|
5
|
+
# https://github.com/drbrain/net-http-persistent when it is not.
|
6
|
+
# The first option is not thread-safe, and the second one is. This is why we
|
7
|
+
# need to force the second option.
|
8
|
+
|
9
|
+
require 'net/http/persistent'
|
10
|
+
|
11
|
+
ThreeScale::Client::HTTPClient.persistent_backend =
|
12
|
+
ThreeScale::Client::HTTPClient::NetHttpPersistent
|
@@ -0,0 +1,49 @@
|
|
1
|
+
module Xcflushd
|
2
|
+
Authorization = Struct.new(:allowed, :reason) do
|
3
|
+
def initialize(authorized, reason = nil)
|
4
|
+
super(authorized, authorized ? nil : reason)
|
5
|
+
end
|
6
|
+
|
7
|
+
def authorized?
|
8
|
+
allowed
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
class Authorization
|
13
|
+
# This is inevitably tied to the 3scale backend code
|
14
|
+
LIMITS_EXCEEDED_CODE = 'limits_exceeded'.freeze
|
15
|
+
private_constant :LIMITS_EXCEEDED_CODE
|
16
|
+
|
17
|
+
ALLOWED = new(true).freeze
|
18
|
+
private_constant :ALLOWED
|
19
|
+
DENIED = new(false).freeze
|
20
|
+
private_constant :DENIED
|
21
|
+
LIMITS_EXCEEDED = new(false, LIMITS_EXCEEDED_CODE).freeze
|
22
|
+
private_constant :LIMITS_EXCEEDED
|
23
|
+
|
24
|
+
private_class_method :new
|
25
|
+
|
26
|
+
def limits_exceeded?
|
27
|
+
reason == LIMITS_EXCEEDED_CODE
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.allow
|
31
|
+
ALLOWED
|
32
|
+
end
|
33
|
+
|
34
|
+
def self.deny_over_limits
|
35
|
+
LIMITS_EXCEEDED
|
36
|
+
end
|
37
|
+
|
38
|
+
def self.deny(reason = nil)
|
39
|
+
if reason.nil?
|
40
|
+
DENIED
|
41
|
+
# this test has to be done in case the code changes
|
42
|
+
elsif reason == LIMITS_EXCEEDED_CODE
|
43
|
+
LIMITS_EXCEEDED
|
44
|
+
else
|
45
|
+
new(false, reason)
|
46
|
+
end
|
47
|
+
end
|
48
|
+
end
|
49
|
+
end
|
@@ -0,0 +1,122 @@
|
|
1
|
+
require '3scale_client'
|
2
|
+
|
3
|
+
module Xcflushd
|
4
|
+
class Authorizer
|
5
|
+
|
6
|
+
# Exception raised when the 3scale client is called with the right params
|
7
|
+
# but it returns a ServerError. Most of the time this means that 3scale is
|
8
|
+
# unreachable.
|
9
|
+
class ThreeScaleInternalError < Flusher::XcflushdError
|
10
|
+
def initialize(service_id, credentials)
|
11
|
+
super("Error renewing auths of service with ID #{service_id} "\
|
12
|
+
"and credentials #{credentials}. 3scale is unreachable")
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
# Extensions used by the Authorizer
|
17
|
+
# In our case we want 3scale to respond with the hierarchy
|
18
|
+
EXTENSIONS = { hierarchy: 1 }.freeze
|
19
|
+
private_constant :EXTENSIONS
|
20
|
+
|
21
|
+
def initialize(threescale_client)
|
22
|
+
@threescale_client = threescale_client
|
23
|
+
end
|
24
|
+
|
25
|
+
# Returns the authorization status of all the limited metrics of the
|
26
|
+
# application identified by the received (service_id, credentials) pair and
|
27
|
+
# also, the authorization of those metrics passed in reported_metrics that
|
28
|
+
# are not limited.
|
29
|
+
#
|
30
|
+
# @return Array<Authorization>
|
31
|
+
def authorizations(service_id, credentials, reported_metrics)
|
32
|
+
# We can safely assume that reported metrics that do not have an
|
33
|
+
# associated report usage are non-limited metrics.
|
34
|
+
|
35
|
+
# First, let's check if there is a problem that has nothing to do with
|
36
|
+
# limits (disabled application, bad credentials, etc.).
|
37
|
+
auth = with_3scale_error_rescue(service_id, credentials) do
|
38
|
+
auths_params = { service_id: service_id,
|
39
|
+
extensions: EXTENSIONS }.merge!(credentials.creds)
|
40
|
+
|
41
|
+
if credentials.oauth?
|
42
|
+
threescale_client.oauth_authorize(auths_params)
|
43
|
+
else
|
44
|
+
threescale_client.authorize(auths_params)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
if !auth.success? && !auth.limits_exceeded?
|
49
|
+
return reported_metrics.inject({}) do |acc, metric|
|
50
|
+
acc[metric] = Authorization.deny(auth.error_code)
|
51
|
+
acc
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
55
|
+
auths_according_to_limits(auth, reported_metrics)
|
56
|
+
end
|
57
|
+
|
58
|
+
private
|
59
|
+
|
60
|
+
attr_reader :threescale_client
|
61
|
+
|
62
|
+
def next_hit_auth?(usages)
|
63
|
+
usages.all? { |usage| usage.current_value < usage.max_value }
|
64
|
+
end
|
65
|
+
|
66
|
+
def usage_reports(auth, reported_metrics)
|
67
|
+
# We are grouping the reports for clarity. We can change this in the
|
68
|
+
# future if it affects performance.
|
69
|
+
reports = auth.usage_reports.group_by { |report| report.metric }
|
70
|
+
non_limited_metrics = reported_metrics - reports.keys
|
71
|
+
non_limited_metrics.each { |metric| reports[metric] = [] }
|
72
|
+
reports
|
73
|
+
end
|
74
|
+
|
75
|
+
# Returns an array of metric names. The array is guaranteed to have all the
|
76
|
+
# parents first, and then the rest.
|
77
|
+
# In 3scale, metric hierarchies only have 2 levels. In other words, a
|
78
|
+
# metric that has a parent cannot have children.
|
79
|
+
def sorted_metrics(metrics, hierarchy)
|
80
|
+
# 'hierarchy' is a hash where the keys are metric names and the values
|
81
|
+
# are arrays with the names of the children metrics. Only metrics with
|
82
|
+
# children and with at least one usage limit appear as keys.
|
83
|
+
parent_metrics = hierarchy.keys
|
84
|
+
child_metrics = metrics - parent_metrics
|
85
|
+
parent_metrics + child_metrics
|
86
|
+
end
|
87
|
+
|
88
|
+
def auths_according_to_limits(app_auth, reported_metrics)
|
89
|
+
metrics_usage = usage_reports(app_auth, reported_metrics)
|
90
|
+
|
91
|
+
# We need to sort the metrics. When the authorization of a metric is
|
92
|
+
# denied, all its children should be denied too. If we check the parents
|
93
|
+
# first, when they are denied, we know that we do not need to check the
|
94
|
+
# limits for their children. This saves us some work.
|
95
|
+
sorted_metrics(metrics_usage.keys, app_auth.hierarchy).inject({}) do |acc, metric|
|
96
|
+
unless acc[metric]
|
97
|
+
acc[metric] = if next_hit_auth?(metrics_usage[metric])
|
98
|
+
Authorization.allow
|
99
|
+
else
|
100
|
+
auth = Authorization.deny_over_limits
|
101
|
+
children = app_auth.hierarchy[metric]
|
102
|
+
if children
|
103
|
+
children.each do |child_metric|
|
104
|
+
acc[child_metric] = auth
|
105
|
+
end
|
106
|
+
end
|
107
|
+
auth
|
108
|
+
end
|
109
|
+
end
|
110
|
+
|
111
|
+
acc
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
def with_3scale_error_rescue(service_id, credentials)
|
116
|
+
yield
|
117
|
+
rescue ThreeScale::ServerError, SocketError
|
118
|
+
# We'll get a SocketError if there's a timeout when contacting 3scale.
|
119
|
+
raise ThreeScaleInternalError.new(service_id, credentials)
|
120
|
+
end
|
121
|
+
end
|
122
|
+
end
|
@@ -0,0 +1,66 @@
|
|
1
|
+
module Xcflushd
|
2
|
+
|
3
|
+
# Credentials contains all the fields required to authenticate an app.
|
4
|
+
# In 3scale there are 3 authentication modes:
|
5
|
+
# * App ID: app_id (required), app_key, referrer, user_id
|
6
|
+
# * API key: user_key (required), referrer, user_id
|
7
|
+
# * Oauth: access_token (required), app_id, referrer, user_id
|
8
|
+
class Credentials
|
9
|
+
|
10
|
+
FIELDS = [:app_id, :app_key, :referrer, :user_id, :user_key, :access_token].freeze
|
11
|
+
private_constant :FIELDS
|
12
|
+
|
13
|
+
attr_reader :creds
|
14
|
+
|
15
|
+
# Initializes a credentials object from a 'creds' hash.
|
16
|
+
# The accepted fields of the hash are:
|
17
|
+
# app_id, app_key, referrer, user_id, user_key, and access_token.
|
18
|
+
# Extra fields are discarded.
|
19
|
+
def initialize(creds)
|
20
|
+
@creds = creds.select { |k, _| FIELDS.include?(k) }
|
21
|
+
end
|
22
|
+
|
23
|
+
# This method returns all the credentials with this format:
|
24
|
+
# credential1:value1,credential2:value2, etc.
|
25
|
+
# The delimiters used, ',' and ':', are escaped in the values. Also, the
|
26
|
+
# credentials appear in alphabetical order.
|
27
|
+
def to_sorted_escaped_s
|
28
|
+
creds.sort_by { |cred, _| cred }
|
29
|
+
.map { |cred, value| "#{escaped(cred.to_s)}:#{escaped(value)}" }
|
30
|
+
.join(',')
|
31
|
+
end
|
32
|
+
|
33
|
+
def ==(other)
|
34
|
+
self.class == other.class && creds == other.creds
|
35
|
+
end
|
36
|
+
|
37
|
+
def oauth?
|
38
|
+
!creds[:access_token].nil?
|
39
|
+
end
|
40
|
+
|
41
|
+
# Creates a Credentials object from an escaped string. The string has this
|
42
|
+
# format: credential1:value1,credential2:value2, etc. ',' and ':' are
|
43
|
+
# escaped in the values
|
44
|
+
def self.from(escaped_s)
|
45
|
+
creds_hash = escaped_s.split(/(?<!\\),/)
|
46
|
+
.map { |field_value| field_value.split(/(?<!\\):/) }
|
47
|
+
.map { |split| [unescaped(split[0]).to_sym,
|
48
|
+
unescaped(split[1])] }
|
49
|
+
.to_h
|
50
|
+
|
51
|
+
new(creds_hash)
|
52
|
+
end
|
53
|
+
|
54
|
+
private
|
55
|
+
|
56
|
+
def escaped(string)
|
57
|
+
string.gsub(','.freeze, "\\,".freeze)
|
58
|
+
.gsub(':'.freeze, "\\:".freeze)
|
59
|
+
end
|
60
|
+
|
61
|
+
def self.unescaped(string)
|
62
|
+
string.gsub(/\\([,:])/, '\1')
|
63
|
+
end
|
64
|
+
|
65
|
+
end
|
66
|
+
end
|
@@ -0,0 +1,146 @@
|
|
1
|
+
require 'concurrent'
|
2
|
+
require 'xcflushd/threading'
|
3
|
+
|
4
|
+
module Xcflushd
|
5
|
+
class Flusher
|
6
|
+
|
7
|
+
WAIT_TIME_REPORT_AUTH = 5 # in seconds
|
8
|
+
private_constant :WAIT_TIME_REPORT_AUTH
|
9
|
+
|
10
|
+
XcflushdError = Class.new(StandardError)
|
11
|
+
|
12
|
+
def initialize(reporter, authorizer, storage, auth_ttl, error_handler, threads)
|
13
|
+
@reporter = reporter
|
14
|
+
@authorizer = authorizer
|
15
|
+
@storage = storage
|
16
|
+
@auth_ttl = auth_ttl
|
17
|
+
@error_handler = error_handler
|
18
|
+
|
19
|
+
min_threads, max_threads = if threads
|
20
|
+
[threads.min, threads.max]
|
21
|
+
else
|
22
|
+
Threading.default_threads_value
|
23
|
+
end
|
24
|
+
|
25
|
+
@thread_pool = Concurrent::ThreadPoolExecutor.new(
|
26
|
+
min_threads: min_threads, max_threads: max_threads)
|
27
|
+
end
|
28
|
+
|
29
|
+
def shutdown
|
30
|
+
@thread_pool.shutdown
|
31
|
+
end
|
32
|
+
|
33
|
+
def wait_for_termination(secs = nil)
|
34
|
+
@thread_pool.wait_for_termination(secs)
|
35
|
+
end
|
36
|
+
|
37
|
+
def terminate
|
38
|
+
@thread_pool.kill
|
39
|
+
end
|
40
|
+
|
41
|
+
# TODO: decide if we want to renew the authorizations every time.
|
42
|
+
def flush
|
43
|
+
reports_to_flush = reports
|
44
|
+
report(reports_to_flush)
|
45
|
+
|
46
|
+
# Ideally, we would like to ensure that once we start checking
|
47
|
+
# authorizations, they have taken into account the reports that we just
|
48
|
+
# performed. However, in 3scale, reports are asynchronous and the current
|
49
|
+
# API does not provide a way to know whether a report has already been
|
50
|
+
# processed.
|
51
|
+
# For now, let's just wait a few seconds. This will greatly mitigate the
|
52
|
+
# problem.
|
53
|
+
sleep(WAIT_TIME_REPORT_AUTH)
|
54
|
+
|
55
|
+
renew(authorizations(reports_to_flush))
|
56
|
+
end
|
57
|
+
|
58
|
+
private
|
59
|
+
|
60
|
+
attr_reader :reporter, :authorizer, :storage, :auth_ttl,
|
61
|
+
:error_handler, :thread_pool
|
62
|
+
|
63
|
+
def reports
|
64
|
+
storage.reports_to_flush
|
65
|
+
end
|
66
|
+
|
67
|
+
def report(reports)
|
68
|
+
report_tasks = async_report_tasks(reports)
|
69
|
+
report_tasks.values.each(&:execute)
|
70
|
+
report_tasks.values.each(&:value) # blocks until all finish
|
71
|
+
|
72
|
+
failed = report_tasks.select { |_report, task| task.rejected? }
|
73
|
+
.map { |report, task| [report, task.reason] }
|
74
|
+
.to_h
|
75
|
+
|
76
|
+
error_handler.handle_report_errors(failed) unless failed.empty?
|
77
|
+
end
|
78
|
+
|
79
|
+
def authorizations(reports)
|
80
|
+
auth_tasks = async_authorization_tasks(reports)
|
81
|
+
auth_tasks.values.each(&:execute)
|
82
|
+
|
83
|
+
auths = []
|
84
|
+
failed = {}
|
85
|
+
auth_tasks.each do |report, auth_task|
|
86
|
+
auth = auth_task.value # blocks until finished
|
87
|
+
|
88
|
+
if auth_task.fulfilled?
|
89
|
+
auths << { service_id: report[:service_id],
|
90
|
+
credentials: report[:credentials],
|
91
|
+
auths: auth }
|
92
|
+
else
|
93
|
+
failed[report] = auth_task.reason
|
94
|
+
end
|
95
|
+
end
|
96
|
+
|
97
|
+
error_handler.handle_auth_errors(failed) unless failed.empty?
|
98
|
+
|
99
|
+
auths
|
100
|
+
end
|
101
|
+
|
102
|
+
def renew(authorizations)
|
103
|
+
authorizations.each do |authorization|
|
104
|
+
begin
|
105
|
+
storage.renew_auths(authorization[:service_id],
|
106
|
+
authorization[:credentials],
|
107
|
+
authorization[:auths],
|
108
|
+
auth_ttl)
|
109
|
+
rescue Storage::RenewAuthError => e
|
110
|
+
error_handler.handle_renew_auth_error(e)
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
def async_report_tasks(reports)
|
116
|
+
reports.map do |report|
|
117
|
+
task = Concurrent::Future.new(executor: thread_pool) do
|
118
|
+
reporter.report(report[:service_id],
|
119
|
+
report[:credentials],
|
120
|
+
report[:usage])
|
121
|
+
end
|
122
|
+
[report, task]
|
123
|
+
end.to_h
|
124
|
+
end
|
125
|
+
|
126
|
+
# Returns a Hash. The keys are the reports and the values their associated
|
127
|
+
# async authorization tasks.
|
128
|
+
def async_authorization_tasks(reports)
|
129
|
+
# Each call to authorizer.authorizations might need to contact 3scale
|
130
|
+
# several times. The number of calls equals 1 + number of reported
|
131
|
+
# metrics without limits.
|
132
|
+
# This is probably good enough for now, but in the future we might want
|
133
|
+
# to make sure that we perform concurrent calls to 3scale instead of
|
134
|
+
# authorizer.authorizations.
|
135
|
+
reports.map do |report|
|
136
|
+
task = Concurrent::Future.new(executor: thread_pool) do
|
137
|
+
authorizer.authorizations(report[:service_id],
|
138
|
+
report[:credentials],
|
139
|
+
report[:usage].keys)
|
140
|
+
end
|
141
|
+
[report, task]
|
142
|
+
end.to_h
|
143
|
+
end
|
144
|
+
|
145
|
+
end
|
146
|
+
end
|
@@ -0,0 +1,78 @@
|
|
1
|
+
module Xcflushd
|
2
|
+
class FlusherErrorHandler
|
3
|
+
|
4
|
+
REPORTER_ERRORS = { temp: [Reporter::ThreeScaleInternalError].freeze,
|
5
|
+
non_temp: [Reporter::ThreeScaleBadParams,
|
6
|
+
Reporter::ThreeScaleAuthError].freeze }.freeze
|
7
|
+
private_constant :REPORTER_ERRORS
|
8
|
+
|
9
|
+
AUTHORIZER_ERRORS = { temp: [Authorizer::ThreeScaleInternalError].freeze,
|
10
|
+
non_temp: [].freeze }.freeze
|
11
|
+
private_constant :AUTHORIZER_ERRORS
|
12
|
+
|
13
|
+
STORAGE_ERRORS = { temp: [Storage::RenewAuthError].freeze,
|
14
|
+
non_temp: [].freeze}.freeze
|
15
|
+
private_constant :STORAGE_ERRORS
|
16
|
+
|
17
|
+
NON_TEMP_ERRORS = [REPORTER_ERRORS, AUTHORIZER_ERRORS, STORAGE_ERRORS].map do |errors|
|
18
|
+
errors[:non_temp]
|
19
|
+
end.flatten
|
20
|
+
private_constant :NON_TEMP_ERRORS
|
21
|
+
|
22
|
+
TEMP_ERRORS = [REPORTER_ERRORS, AUTHORIZER_ERRORS, STORAGE_ERRORS].map do |errors|
|
23
|
+
errors[:temp]
|
24
|
+
end.flatten
|
25
|
+
private_constant :TEMP_ERRORS
|
26
|
+
|
27
|
+
def initialize(logger, storage)
|
28
|
+
@logger = logger
|
29
|
+
@storage = storage
|
30
|
+
end
|
31
|
+
|
32
|
+
# @param failed_reports [Hash<Report, Exception>]
|
33
|
+
def handle_report_errors(failed_reports)
|
34
|
+
failed_reports.values.each { |exception| log(exception) }
|
35
|
+
storage.report(failed_reports.keys)
|
36
|
+
end
|
37
|
+
|
38
|
+
# @param failed_auths [Hash<Auth, Exception>]
|
39
|
+
def handle_auth_errors(failed_auths)
|
40
|
+
failed_auths.values.each { |exception| log(exception) }
|
41
|
+
end
|
42
|
+
|
43
|
+
# @param exception [Exception]
|
44
|
+
def handle_renew_auth_error(exception)
|
45
|
+
# Failing to renew an authorization in the cache should not be a big
|
46
|
+
# problem. It is probably caused by a temporary issue (like a Redis
|
47
|
+
# connection error) and the auth will probably be successfully renewed
|
48
|
+
# next time. So for now, we just log the error.
|
49
|
+
log(exception)
|
50
|
+
end
|
51
|
+
|
52
|
+
private
|
53
|
+
|
54
|
+
attr_reader :logger, :storage
|
55
|
+
|
56
|
+
# For exceptions that are likely to require the user intervention, we log
|
57
|
+
# errors. For example, when the report could not be made because the 3scale
|
58
|
+
# client received an invalid provider key.
|
59
|
+
# On the other hand, for errors that are likely to be temporary, like when
|
60
|
+
# we could not connect with 3scale, we log a warning.
|
61
|
+
def log(exception)
|
62
|
+
msg = error_msg(exception)
|
63
|
+
case exception
|
64
|
+
when *NON_TEMP_ERRORS
|
65
|
+
logger.error(msg)
|
66
|
+
when *TEMP_ERRORS
|
67
|
+
logger.warn(msg)
|
68
|
+
else
|
69
|
+
logger.error(msg)
|
70
|
+
end
|
71
|
+
end
|
72
|
+
|
73
|
+
def error_msg(exception)
|
74
|
+
"#{exception.message} Cause: #{exception.cause || '-'.freeze}"
|
75
|
+
end
|
76
|
+
|
77
|
+
end
|
78
|
+
end
|