xcflushd 1.0.0.rc2

data/docs/design.md

@@ -0,0 +1,100 @@
+# DESIGN
+
+## Description
+
+xcflushd is a daemon used together with [XC](https://github.com/3scale/apicast-xc),
+which is a module for [Apicast](https://github.com/3scale/apicast), 3scale's
+API Gateway.
+
+If you are not familiar with XC yet, we recommend you to start reading its
+documentation before reading this document.
+
+In XC, the xcflushd daemon is responsible for these three things:
+
+* Reporting to 3scale's backend the reports cached in XC.
+* Updating the status of the authorizations cached in XC.
+* Retrieving an authorization status from 3scale's backend when the
+  authorization is not cached in XC.
+
+It is important to keep in mind that the first 2 always happen in background,
+while the third happens in request time.
+
+
+## How xcflushd works
+
+Once xcflushd starts, it will:
+
+* Start "flushing" periodically to 3scale.
+* Start listening in a the Redis pubsub channel that XC uses to ask for
+  authorizations that are not cached.
+
+Let us explain those 2 operations in more detail.
+
+
+### Periodic flushing
+
+The flushing consists of 4 steps:
+
+1. Retrieve the cached reports from Redis.
+2. Send those cached reports to 3scale's backend.
+3. Wait for a few seconds.
+4. Renew the cached authorizations in Redis of the applications that appear in
+   the reports that have just been sent to 3scale.
+
+You might be wondering why there's a waiting time between sending the cached
+reports and renewing the cached authorizations. The reason is that reporting is
+asynchronous in 3scale's backend API. That means that, when reporting, we'll
+get an OK http response code if our reports do not contain any format errors
+and the credentials are OK, but we do not have a way to know when those reports
+are effective and considered for rate limiting. We know that 3scale is pretty
+quick doing that, though! This is a trade-off that 3scale makes between
+request latency and accurateness of rate limits.
+
+Another important thing is that when renewing cached authorizations, xcflushd
+does not only renew the ones of the metrics that appear in the cached reports.
+The call to 3scale backend allows us to ask for the limits of most of the
+metrics of an application, so we take advantage of that and renew all the ones
+we can in one network round-trip. More specifically, the ones that are not
+renewed are the ones that meet these two conditions: 1) have not been included
+in the reports sent to 3scale's backend, and 2) do not have any limits defined.
+
+The whole flushing process makes 2 calls to 3scale per application reported.
+One call to send the report to 3scale, and another one to get the current
+authorization status. Compare that to making one request for each one that
+arrives to the proxy as in the case of Apicast. Imagine that you have 1k rps,
+and 100 apps. If you define a period of 1 minute for the flushing process, you
+will be making 200 requests to 3scale's backend per minute instead of 60k.
+
+We use the [3scale client gem](https://github.com/3scale/3scale_ws_api_for_ruby)
+to make the requests to 3scale's backend.
+
+
+### Requests via Redis pubsub
+
+xcflushd offers a mechanism that allows clients to ask for a specific
+authorization status without having to wait for the next flush cycle. This
+mechanism is based on Redis pubsub.
+
+xcflushd subscribes to a channel to which clients publish messages with a
+`(service, app credentials, metric)` tuple encoded. When xcflushd receives one
+of those messages, it makes a request to the 3scale backend to check the
+authorization status of the given `(service, app credentials, metric)` tuple,
+and it publishes the result to another pubsub channel to which clients need to
+subscribe. To make the most out of this network round-trip to 3scale's backend,
+we take the opportunity to store in the Redis cache the authorization statuses
+that we just retrieved. Although the message only contained one metric, we
+renew all the ones we can get in a single request to 3scale's backend. Just
+like we do in the case of the periodic flushing detailed above.
+
+xcflushd might receive several requests at the same time about the same
+`(service, app credentials, metric)` tuple. xcflushd takes this into account
+and does not make extra requests to 3scale's backend for authorizations that
+are already being checked.
+
+
+## Redis keys
+
+The format of the Redis keys used is specified in the `Xcflushd::StorageKeys`
+class. The keys need to follow the same format as the one defined in the XC
+lua module. Check the [XC lua module design doc](https://github.com/3scale/apicast-xc/blob/master/doc/design.md#redis-keys-format)
+for more info about this.

data/exe/xcflushd

@@ -0,0 +1,114 @@
+#!/usr/bin/env ruby
+require 'gli'
+require 'redis'
+require 'xcflushd/gli_helpers'
+
+include GLI::App
+include Xcflushd::GLIHelpers
+
+program_desc 'XC flush daemon'
+
+version Xcflushd::VERSION
+
+subcommand_option_handling :normal
+arguments :strict
+
+desc 'Starts the XC flusher'
+arg_name ' '
+command :run do |c|
+  c.desc 'Redis URI'
+  c.flag [:r, :redis], required: true, type: RedisURI
+
+  c.desc '3scale backend URI'
+  c.flag [:b, :backend], required: true, type: BackendURI
+
+  c.desc '3scale provider key'
+  c.flag [:k, :'provider-key'], required: true
+
+  c.desc 'Validity of authorizations (in seconds)'
+  c.flag [:a, :'auth-ttl'], required: true, type: Integer, must_match: POSITIVE_N_RE
+
+  c.desc 'Reporting frequency (in seconds)'
+  c.flag [:f, :frequency], required: true, type: Integer, must_match: POSITIVE_N_RE
+
+  c.desc 'Number of threads for the main thread pool (min:max)'
+  c.flag [:t, :threads], default_value: 'auto', type: PositiveMinMaxInt
+
+  c.desc 'Number of threads for the priority auth renewer (min:max)'
+  c.flag [:p, :'prio-threads'], default_value: 'auto', type: PositiveMinMaxInt
+
+  c.desc 'Run this program as a background service'
+  c.switch :daemonize, default_value: false, negatable: true, multiple: true
+
+  c.desc 'Use HTTPS scheme to connect to 3scale backend'
+  c.switch :secure, default_value: true, negatable: true, multiple: true
+
+  c.action do |_global_options, options, _args|
+    # options contains 2 keys for each option. One is a String and the other a
+    # Symbol. That way we can use options['redis'] and options[:redis].
+    # That's fine as long as we only read from the hash. If we need to modify
+    # it, it is problematic because we need to remember to modify two values.
+    # For simplicity, let's just keep the elements that have symbols as keys.
+    options.keep_if { |k, _v| k.is_a?(Symbol) }
+
+    # Reporting frequency should be lower or equal to the authorization TTL.
+    # Otherwise authorizations would be renewed without taking into account the
+    # last available reports. Note: whenever we do reporting we also do renewal
+    # of authorizations; the TTL is important in case of problems renewing
+    # authorizations.
+    if options[:frequency] > options[:'auth-ttl']
+      exit_now!('frequency needs to be <= auth-ttl')
+    end
+
+    if (options[:backend].scheme == 'http' && options[:secure]) ||
+        (options[:backend].scheme == 'https' && !options[:secure])
+      exit_now!("the specified backend scheme does not match the --[no-]secure" \
+                " flag.\nCan't continue with conflicting settings.")
+    end
+
+    if options[:threads] == 'auto'
+      options.delete :threads
+      options.delete :t
+    end
+
+    if options[:'prio-threads'] == 'auto'
+      options.delete :'prio-threads'
+      options.delete :p
+    end
+
+    banner = "xcflushd [#{Xcflushd::VERSION}]"
+
+    if options[:daemonize]
+      require 'daemons'
+      Daemons.run_proc(banner) { start_xcflusher(options) }
+    else
+      set_title(banner)
+      start_xcflusher(options)
+    end
+  end
+end
+
+pre do |_global, _command, _options, _args|
+  # Pre logic here
+  # Return true to proceed; false to abort and not call the
+  # chosen command
+  # Use skips_pre before a command to skip this block
+  # on that command only
+  true
+end
+
+post do |_global, _command, _options, _args|
+  # Post logic here
+  # Use skips_post before a command to skip this
+  # block on that command only
+end
+
+on_error do |exception|
+  # Error logic here
+  # return false to skip default error handling
+  # When forking, the parent executes 'exit', and GLI error handling kicks in.
+  # Let's just skip it when that's the case.
+  not SystemExit === exception
+end
+
+exit run(ARGV)

data/lib/xcflushd/3scale_client_ext.rb

@@ -0,0 +1,12 @@
+# We need to configure the ThreeScale::Client::HTTPClient class of the
+# 3scale_client gem before we use it.
+# Internally, the 3scale_client uses Net::HTTP with the keep-alive option
+# enabled when it is available and the net-http-persistent gem
+# https://github.com/drbrain/net-http-persistent when it is not.
+# The first option is not thread-safe, and the second one is. This is why we
+# need to force the second option.
+
+require 'net/http/persistent'
+
+ThreeScale::Client::HTTPClient.persistent_backend =
+    ThreeScale::Client::HTTPClient::NetHttpPersistent

data/lib/xcflushd/authorization.rb

@@ -0,0 +1,49 @@
+module Xcflushd
+  Authorization = Struct.new(:allowed, :reason) do
+    def initialize(authorized, reason = nil)
+      super(authorized, authorized ? nil : reason)
+    end
+
+    def authorized?
+      allowed
+    end
+  end
+
+  class Authorization
+    # This is inevitably tied to the 3scale backend code
+    LIMITS_EXCEEDED_CODE = 'limits_exceeded'.freeze
+    private_constant :LIMITS_EXCEEDED_CODE
+
+    ALLOWED = new(true).freeze
+    private_constant :ALLOWED
+    DENIED = new(false).freeze
+    private_constant :DENIED
+    LIMITS_EXCEEDED = new(false, LIMITS_EXCEEDED_CODE).freeze
+    private_constant :LIMITS_EXCEEDED
+
+    private_class_method :new
+
+    def limits_exceeded?
+      reason == LIMITS_EXCEEDED_CODE
+    end
+
+    def self.allow
+      ALLOWED
+    end
+
+    def self.deny_over_limits
+      LIMITS_EXCEEDED
+    end
+
+    def self.deny(reason = nil)
+      if reason.nil?
+        DENIED
+      # this test has to be done in case the code changes
+      elsif reason == LIMITS_EXCEEDED_CODE
+        LIMITS_EXCEEDED
+      else
+        new(false, reason)
+      end
+    end
+  end
+end

data/lib/xcflushd/authorizer.rb

@@ -0,0 +1,122 @@
+require '3scale_client'
+
+module Xcflushd
+  class Authorizer
+
+    # Exception raised when the 3scale client is called with the right params
+    # but it returns a ServerError. Most of the time this means that 3scale is
+    # unreachable.
+    class ThreeScaleInternalError < Flusher::XcflushdError
+      def initialize(service_id, credentials)
+        super("Error renewing auths of service with ID #{service_id} "\
+              "and credentials #{credentials}. 3scale is unreachable")
+      end
+    end
+
+    # Extensions used by the Authorizer
+    # In our case we want 3scale to respond with the hierarchy
+    EXTENSIONS = { hierarchy: 1 }.freeze
+    private_constant :EXTENSIONS
+
+    def initialize(threescale_client)
+      @threescale_client = threescale_client
+    end
+
+    # Returns the authorization status of all the limited metrics of the
+    # application identified by the received (service_id, credentials) pair and
+    # also, the authorization of those metrics passed in reported_metrics that
+    # are not limited.
+    #
+    # @return Array<Authorization>
+    def authorizations(service_id, credentials, reported_metrics)
+      # We can safely assume that reported metrics that do not have an
+      # associated report usage are non-limited metrics.
+
+      # First, let's check if there is a problem that has nothing to do with
+      # limits (disabled application, bad credentials, etc.).
+      auth = with_3scale_error_rescue(service_id, credentials) do
+        auths_params = { service_id: service_id,
+                         extensions: EXTENSIONS }.merge!(credentials.creds)
+
+        if credentials.oauth?
+          threescale_client.oauth_authorize(auths_params)
+        else
+          threescale_client.authorize(auths_params)
+        end
+      end
+
+      if !auth.success? && !auth.limits_exceeded?
+        return reported_metrics.inject({}) do |acc, metric|
+          acc[metric] = Authorization.deny(auth.error_code)
+          acc
+        end
+      end
+
+      auths_according_to_limits(auth, reported_metrics)
+    end
+
+    private
+
+    attr_reader :threescale_client
+
+    def next_hit_auth?(usages)
+      usages.all? { |usage| usage.current_value < usage.max_value }
+    end
+
+    def usage_reports(auth, reported_metrics)
+      # We are grouping the reports for clarity. We can change this in the
+      # future if it affects performance.
+      reports = auth.usage_reports.group_by { |report| report.metric }
+      non_limited_metrics = reported_metrics - reports.keys
+      non_limited_metrics.each { |metric| reports[metric] = [] }
+      reports
+    end
+
+    # Returns an array of metric names. The array is guaranteed to have all the
+    # parents first, and then the rest.
+    # In 3scale, metric hierarchies only have 2 levels. In other words, a
+    # metric that has a parent cannot have children.
+    def sorted_metrics(metrics, hierarchy)
+      # 'hierarchy' is a hash where the keys are metric names and the values
+      # are arrays with the names of the children metrics. Only metrics with
+      # children and with at least one usage limit appear as keys.
+      parent_metrics = hierarchy.keys
+      child_metrics = metrics - parent_metrics
+      parent_metrics + child_metrics
+    end
+
+    def auths_according_to_limits(app_auth, reported_metrics)
+      metrics_usage = usage_reports(app_auth, reported_metrics)
+
+      # We need to sort the metrics. When the authorization of a metric is
+      # denied, all its children should be denied too. If we check the parents
+      # first, when they are denied, we know that we do not need to check the
+      # limits for their children. This saves us some work.
+      sorted_metrics(metrics_usage.keys, app_auth.hierarchy).inject({}) do |acc, metric|
+        unless acc[metric]
+          acc[metric] = if next_hit_auth?(metrics_usage[metric])
+                          Authorization.allow
+                        else
+                          auth = Authorization.deny_over_limits
+                          children = app_auth.hierarchy[metric]
+                          if children
+                            children.each do |child_metric|
+                              acc[child_metric] = auth
+                            end
+                          end
+                          auth
+                        end
+        end
+
+        acc
+      end
+    end
+
+    def with_3scale_error_rescue(service_id, credentials)
+      yield
+    rescue ThreeScale::ServerError, SocketError
+      # We'll get a SocketError if there's a timeout when contacting 3scale.
+      raise ThreeScaleInternalError.new(service_id, credentials)
+    end
+  end
+end

data/lib/xcflushd/credentials.rb

@@ -0,0 +1,66 @@
+module Xcflushd
+
+  # Credentials contains all the fields required to authenticate an app.
+  # In 3scale there are 3 authentication modes:
+  #   * App ID: app_id (required), app_key, referrer, user_id
+  #   * API key: user_key (required), referrer, user_id
+  #   * Oauth: access_token (required), app_id, referrer, user_id
+  class Credentials
+
+    FIELDS = [:app_id, :app_key, :referrer, :user_id, :user_key, :access_token].freeze
+    private_constant :FIELDS
+
+    attr_reader :creds
+
+    # Initializes a credentials object from a 'creds' hash.
+    # The accepted fields of the hash are:
+    #   app_id, app_key, referrer, user_id, user_key, and access_token.
+    # Extra fields are discarded.
+    def initialize(creds)
+      @creds = creds.select { |k, _| FIELDS.include?(k) }
+    end
+
+    # This method returns all the credentials with this format:
+    # credential1:value1,credential2:value2, etc.
+    # The delimiters used, ',' and ':', are escaped in the values. Also, the
+    # credentials appear in alphabetical order.
+    def to_sorted_escaped_s
+      creds.sort_by { |cred, _| cred }
+           .map { |cred, value| "#{escaped(cred.to_s)}:#{escaped(value)}" }
+           .join(',')
+    end
+
+    def ==(other)
+      self.class == other.class && creds == other.creds
+    end
+
+    def oauth?
+      !creds[:access_token].nil?
+    end
+
+    # Creates a Credentials object from an escaped string. The string has this
+    # format: credential1:value1,credential2:value2, etc. ',' and ':' are
+    # escaped in the values
+    def self.from(escaped_s)
+      creds_hash = escaped_s.split(/(?<!\\),/)
+                            .map { |field_value| field_value.split(/(?<!\\):/) }
+                            .map { |split| [unescaped(split[0]).to_sym,
+                                            unescaped(split[1])] }
+                            .to_h
+
+      new(creds_hash)
+    end
+
+    private
+
+    def escaped(string)
+      string.gsub(','.freeze, "\\,".freeze)
+            .gsub(':'.freeze, "\\:".freeze)
+    end
+
+    def self.unescaped(string)
+      string.gsub(/\\([,:])/, '\1')
+    end
+
+  end
+end

data/lib/xcflushd/flusher.rb

@@ -0,0 +1,146 @@
+require 'concurrent'
+require 'xcflushd/threading'
+
+module Xcflushd
+  class Flusher
+
+    WAIT_TIME_REPORT_AUTH = 5 # in seconds
+    private_constant :WAIT_TIME_REPORT_AUTH
+
+    XcflushdError = Class.new(StandardError)
+
+    def initialize(reporter, authorizer, storage, auth_ttl, error_handler, threads)
+      @reporter = reporter
+      @authorizer = authorizer
+      @storage = storage
+      @auth_ttl = auth_ttl
+      @error_handler = error_handler
+
+      min_threads, max_threads = if threads
+                                   [threads.min, threads.max]
+                                 else
+                                   Threading.default_threads_value
+                                 end
+
+      @thread_pool = Concurrent::ThreadPoolExecutor.new(
+        min_threads: min_threads, max_threads: max_threads)
+    end
+
+    def shutdown
+      @thread_pool.shutdown
+    end
+
+    def wait_for_termination(secs = nil)
+      @thread_pool.wait_for_termination(secs)
+    end
+
+    def terminate
+      @thread_pool.kill
+    end
+
+    # TODO: decide if we want to renew the authorizations every time.
+    def flush
+      reports_to_flush = reports
+      report(reports_to_flush)
+
+      # Ideally, we would like to ensure that once we start checking
+      # authorizations, they have taken into account the reports that we just
+      # performed. However, in 3scale, reports are asynchronous and the current
+      # API does not provide a way to know whether a report has already been
+      # processed.
+      # For now, let's just wait a few seconds. This will greatly mitigate the
+      # problem.
+      sleep(WAIT_TIME_REPORT_AUTH)
+
+      renew(authorizations(reports_to_flush))
+    end
+
+    private
+
+    attr_reader :reporter, :authorizer, :storage, :auth_ttl,
+                :error_handler, :thread_pool
+
+    def reports
+      storage.reports_to_flush
+    end
+
+    def report(reports)
+      report_tasks = async_report_tasks(reports)
+      report_tasks.values.each(&:execute)
+      report_tasks.values.each(&:value) # blocks until all finish
+
+      failed = report_tasks.select { |_report, task| task.rejected? }
+                           .map { |report, task| [report, task.reason] }
+                           .to_h
+
+      error_handler.handle_report_errors(failed) unless failed.empty?
+    end
+
+    def authorizations(reports)
+      auth_tasks = async_authorization_tasks(reports)
+      auth_tasks.values.each(&:execute)
+
+      auths = []
+      failed = {}
+      auth_tasks.each do |report, auth_task|
+        auth = auth_task.value # blocks until finished
+
+        if auth_task.fulfilled?
+          auths << { service_id: report[:service_id],
+                     credentials: report[:credentials],
+                     auths: auth }
+        else
+          failed[report] = auth_task.reason
+        end
+      end
+
+      error_handler.handle_auth_errors(failed) unless failed.empty?
+
+      auths
+    end
+
+    def renew(authorizations)
+      authorizations.each do |authorization|
+        begin
+          storage.renew_auths(authorization[:service_id],
+                              authorization[:credentials],
+                              authorization[:auths],
+                              auth_ttl)
+        rescue Storage::RenewAuthError => e
+          error_handler.handle_renew_auth_error(e)
+        end
+      end
+    end
+
+    def async_report_tasks(reports)
+      reports.map do |report|
+        task = Concurrent::Future.new(executor: thread_pool) do
+          reporter.report(report[:service_id],
+                          report[:credentials],
+                          report[:usage])
+        end
+        [report, task]
+      end.to_h
+    end
+
+    # Returns a Hash. The keys are the reports and the values their associated
+    # async authorization tasks.
+    def async_authorization_tasks(reports)
+      # Each call to authorizer.authorizations might need to contact 3scale
+      # several times. The number of calls equals 1 + number of reported
+      # metrics without limits.
+      # This is probably good enough for now, but in the future we might want
+      # to make sure that we perform concurrent calls to 3scale instead of
+      # authorizer.authorizations.
+      reports.map do |report|
+        task = Concurrent::Future.new(executor: thread_pool) do
+          authorizer.authorizations(report[:service_id],
+                                    report[:credentials],
+                                    report[:usage].keys)
+        end
+        [report, task]
+      end.to_h
+    end
+
+  end
+end

data/lib/xcflushd/flusher_error_handler.rb

@@ -0,0 +1,78 @@
+module Xcflushd
+  class FlusherErrorHandler
+
+    REPORTER_ERRORS = { temp: [Reporter::ThreeScaleInternalError].freeze,
+                        non_temp: [Reporter::ThreeScaleBadParams,
+                                   Reporter::ThreeScaleAuthError].freeze }.freeze
+    private_constant :REPORTER_ERRORS
+
+    AUTHORIZER_ERRORS = { temp: [Authorizer::ThreeScaleInternalError].freeze,
+                          non_temp: [].freeze }.freeze
+    private_constant :AUTHORIZER_ERRORS
+
+    STORAGE_ERRORS = { temp: [Storage::RenewAuthError].freeze,
+                       non_temp: [].freeze}.freeze
+    private_constant :STORAGE_ERRORS
+
+    NON_TEMP_ERRORS = [REPORTER_ERRORS, AUTHORIZER_ERRORS, STORAGE_ERRORS].map do |errors|
+      errors[:non_temp]
+    end.flatten
+    private_constant :NON_TEMP_ERRORS
+
+    TEMP_ERRORS = [REPORTER_ERRORS, AUTHORIZER_ERRORS, STORAGE_ERRORS].map do |errors|
+      errors[:temp]
+    end.flatten
+    private_constant :TEMP_ERRORS
+
+    def initialize(logger, storage)
+      @logger = logger
+      @storage = storage
+    end
+
+    # @param failed_reports [Hash<Report, Exception>]
+    def handle_report_errors(failed_reports)
+      failed_reports.values.each { |exception| log(exception) }
+      storage.report(failed_reports.keys)
+    end
+
+    # @param failed_auths [Hash<Auth, Exception>]
+    def handle_auth_errors(failed_auths)
+      failed_auths.values.each { |exception| log(exception) }
+    end
+
+    # @param exception [Exception]
+    def handle_renew_auth_error(exception)
+      # Failing to renew an authorization in the cache should not be a big
+      # problem. It is probably caused by a temporary issue (like a Redis
+      # connection error) and the auth will probably be successfully renewed
+      # next time. So for now, we just log the error.
+      log(exception)
+    end
+
+    private
+
+    attr_reader :logger, :storage
+
+    # For exceptions that are likely to require the user intervention, we log
+    # errors. For example, when the report could not be made because the 3scale
+    # client received an invalid provider key.
+    # On the other hand, for errors that are likely to be temporary, like when
+    # we could not connect with 3scale, we log a warning.
+    def log(exception)
+      msg = error_msg(exception)
+      case exception
+        when *NON_TEMP_ERRORS
+          logger.error(msg)
+        when *TEMP_ERRORS
+          logger.warn(msg)
+        else
+          logger.error(msg)
+      end
+    end
+
+    def error_msg(exception)
+      "#{exception.message} Cause: #{exception.cause || '-'.freeze}"
+    end
+
+  end
+end