apisonator 2.100.0

data/NOTICE

@@ -0,0 +1,15 @@
+   Red Hat 3scale API Management Platform Apisonator
+   Copyright (c) 2010-2016 3scale, Inc
+   Copyright (c) 2016-2018 Red Hat, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,230 @@
+# Apisonator
+
+[![Docker Repository on Quay](https://quay.io/repository/3scale/apisonator/status "Docker Repository on Quay")](https://quay.io/repository/3scale/apisonator)
+[![CircleCI](https://circleci.com/gh/3scale/apisonator.svg?style=shield)](https://circleci.com/gh/3scale/apisonator)
+[![Maintainability](https://api.codeclimate.com/v1/badges/d2cea8016f0089cb2fd6/maintainability)](https://codeclimate.com/github/3scale/apisonator/maintainability)
+
+This software is licensed under the [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).
+
+See the LICENSE and NOTICE files that should have been provided along with this
+software for details.
+
+## Description
+
+This is the Red Hat 3scale API Management backend.
+
+It has the following components:
+
+### Apisonator listener
+
+It provides the point of entry of the API Management Platform's Backend.
+The Service management API (SM API) is provided to authorize and report consumer
+API requests.
+
+Three main operations can be performed with this API:
+
+ * Report: Reports a number of hits to one or more metrics, performing the
+   corresponding metric aggregations
+ * Authorize: Authorize a request. The authorization of a request checks that:
+   * The provided API key to authorize the request is valid
+   * The current usage metrics of the API related to the request are within
+     the specified limits
+ * Authrep: Combination of both the above
+
+Make sure to read the corresponding Swagger-generated documentation of this
+operations, located in [docs/active_docs/Service Management API.json](docs%2Factive_docs%2FService%20Management%20API.json)
+
+It attempts to respond with lowest possible latency, performing minimal work
+and offloading other work to Apisonator workers by enqueuing tasks into job queues.
+
+This component needs access to a Redis database to perform the following actions:
+ * Enqueue reports, which will be processed by the Apisonator worker component
+ * Perform authorization of the requests
+
+These two actions can be configured to be performed in different Redis
+databases if desired (see the [Prerequisites](#prerequisites)
+and [Configuration](#configuration) sections).
+
+Finally, another API named 'Internal API' is provided to configure services
+in Apisonator. This API is intended only for administrative purposes and not
+for general consumption. Therefore, usage of this API should be protected or
+not exposed to untrusted parties. You can also generate its documentation with
+Rake tasks. The [Pisoni](https://github.com/3scale/pisoni) API client can be
+used to interact with the Internal API.
+
+To quickly test Apisonator, random services can be created and configured on it
+via the use of the 'Buddhi' tool located in our performance tests
+toolkit: [3scale perftest-toolkit](https://github.com/3scale/perftest-toolkit/).
+
+This component may also be referred to as '3scale_backend'.
+
+### Apisonator worker
+
+It is responsible for performing background tasks off-loaded from
+Apisonator listeners (enqueued jobs).
+
+The worker component takes care of running these enqueued jobs, mainly related
+to reporting of previous traffic.
+
+Specifically, this component:
+ * Dequeues and runs the report jobs that have been submitted to the Redis
+   background jobs database by the Apisonator listener/s or the
+   Apisonator failed jobs rescheduler
+ * Stores the results of running the report jobs in the Redis Storage database
+
+This component may also be referred to as '3scale_backend_worker'.
+
+### Apisonator failed jobs rescheduler
+
+This is a simple task that acts as a cron scheduler to requeue jobs that failed
+when being processed by an Apisonator worker. The jobs are requeued into
+the Redis background jobs database.
+
+This component may also be referred to as 'backend-cron'.
+
+## Development
+
+See the file [DEVELOPMENT](DEVELOPMENT.md)
+
+## Documentation
+
+You can find documentation about Apisonator (also called referred to as `3scale
+backend`) at the [Red Hat 3scale API Management product pages](https://access.redhat.com/products/red-hat-3scale/).
+
+Documentation about specific parts of Apisonator (APIs, specs, behaviour, etc)
+can be found in the [`docs`](https://github.com/3scale/apisonator/tree/master/docs) folder, though this is mostly meant for development and design purposes rather
+than user documentation.
+
+## How to run
+
+### Prerequisites
+
+* Docker (requires version 1.10.0 or later)
+* A Redis database, used to store API request statistics and services. Also
+  used to perform API requests authorizations. In Apisonator this database
+  is commonly referred to as 'Redis Storage'
+* A Redis database, used to store background jobs. The Redis Resque library
+  is used for this. In Apisonator this database is commonly referred to as
+  'Redis Resque', or as the 'background jobs database'
+
+The two previous Redis databases can be configured in the following ways:
+
+ * In a single machine/vm, using a single Redis process by specifying
+   different database identifiers, which is supported by the Redis URI
+	 specification. i.e. redis://host:port/0, redis://host:port/1
+ * In a single machine/vm, using different Redis processes with different
+   assigned ports
+ * In separate machines/vms
+
+The first thing you will need is cloning the project:
+> `$ git clone git@github.com:3scale/apisonator.git`
+
+Next cd into the directory, `cd apisonator`.
+
+### Apisonator image generation
+
+Go to the `openshift` directory and execute `make build`. This will generate
+a local docker image named `amp:apisonator-<version_number>` based on CentOS 7.
+
+### Configuration
+
+To run any Apisonator component, application-related environment variables must
+be previously set. This can be done by setting them via the `--env` flag in
+Docker or by placing them in a ENV file and setting the ENV file in Docker via
+the `--env-file` flag.
+
+The most important variables to set are:
+
+ * CONFIG_QUEUES_MASTER_NAME: Set this to the [`redis://` URL](http://www.iana.org/assignments/uri-schemes/prov/redis)
+   of where the Redis Storage has been installed
+ * CONFIG_REDIS_PROXY: Set this to the [`redis://` URL](http://www.iana.org/assignments/uri-schemes/prov/redis)
+   of where the Redis Resque has been installed
+ * CONFIG_INTERNAL_API_USER: Set this to an arbitrary username <username>
+   that will be the one used to be able to use the Apisonator internal api
+ * CONFIG_INTERNAL_API_PASSWORD: Set this to an arbitrary
+   password <password> that will be the one used to be able to use the
+   Apisonator internal api
+ * RACK_ENV: Set this to 'production'
+
+A complete list of configuration variables that can be set can be
+found in the file `openshift/3scale_backend.conf`
+
+An example of an ENV file can be found at `openshift/.env.test`
+
+### Automatic execution (with Makefile)
+
+Makefile rules can be run to execute the different Apisonator components
+with some predefined behaviour. To do this a file named `.env` in
+the `openshift` directory must be created before.
+
+Once this has been performed, go to the `openshift` directory and execute
+one of the available Makefile commands to run Apisonator components:
+
+#### Apisonator Listener
+
+Execute the Apisonator Listener, exposing the port 3001:
+
+```
+make listener
+```
+
+#### Apisonator Worker
+
+```
+make worker
+```
+
+#### Apisonator failed jobs rescheduler
+
+Execute the 'cron' Apisonator component:
+
+```
+make cron
+```
+
+#### Apisonator bash shell
+
+Execute a bash shell with the Apisonator source code with all the available
+components:
+
+```
+make bash
+```
+
+### Manual execution
+
+Another way of executing the Apisonator components is by running a container
+using the previously generated Apisonator image:
+
+#### Apisonator Listener
+
+To run an Apisonator listener, the script bin/3scale_backend is used. To
+run it from a previously generated Apisonator docker image:
+
+```
+docker run -p 3001:3001 --env-file <myenv_file> -it amp:apisonator-<version_number> 3scale_backend start -p 3001 -x /dev/stdout
+```
+
+You can see all the available options of the apisonator listener by executing:
+
+```
+docker run -it amp:apisonator-<version_number> 3scale_backend help
+```
+
+#### Apisonator Worker
+
+```
+docker run --env-file <myenv_file> -it amp:apisonator-<version_number> 3scale_backend_worker
+```
+
+#### Apisonator failed jobs rescheduler
+
+```
+docker run --env-file <myenv_file> -it amp:apisonator-<version_number> backend-cron
+```
+
+#### Apisonator bash shell
+
+```
+docker run --env-file <myenv_file> -it amp:apisonator-<version_number> bash
+```

data/Rakefile

@@ -0,0 +1,287 @@
+# encoding: utf-8
+
+require '3scale/backend/configuration'
+require '3scale/backend'
+require '3scale/tasks/helpers'
+
+include ThreeScale::Tasks::Helpers
+
+load 'lib/3scale/tasks/connectivity.rake'
+
+
+if Environment.saas?
+  require '3scale/backend/logging/external'
+
+  ThreeScale::Backend::Logging::External.setup_rake
+
+  load 'lib/3scale/tasks/swagger.rake'
+  load 'lib/3scale/tasks/stats.rake'
+
+  if Environment.testable?
+
+    ENV['RACK_ENV'] = "test"
+    require 'rake/testtask'
+
+    task :default => [:test, :spec]
+
+    test_task_dependencies = ['test:unit', 'test:integration']
+
+    desc 'Run unit and integration tests'
+    task :test => test_task_dependencies
+
+    desc 'Benchmark'
+    task :bench, [:file] do |_, args|
+      require 'benchmark/ips'
+      require 'pathname'
+      require File.dirname(__FILE__) + '/test/test_helpers/configuration'
+
+      filelist = if args[:file]
+                   "#{args[:file].sub(/\Abench\//, '')}"
+                 else
+                   '**/*_bench.rb'
+                 end
+      FileList["#{File.dirname(__FILE__)}/bench/#{filelist}"].each do |f|
+        bench = Pathname.new(f).cleanpath
+        if bench.to_s.start_with?(File.dirname(__FILE__) + File::SEPARATOR)
+          puts "Running benchmark #{bench}"
+          load f
+        else
+          STDERR.puts "Ignoring path #{f} as it points outside the project"
+        end
+      end
+
+      puts "Benchmarks finished"
+    end
+
+    namespace :test do
+      Rake::TestTask.new(:unit) do |task|
+        task.test_files = FileList['test/unit/**/*_test.rb']
+        task.verbose = true
+        task.warning = false
+      end
+
+      Rake::TestTask.new(:integration) do |task|
+        task.test_files = FileList['test/integration/**/*_test.rb']
+        task.verbose = true
+        task.warning = false
+      end
+    end
+
+    require 'rspec/core/rake_task'
+    spec_task_dependencies = [
+      'spec:unit', 'spec:integration', 'spec:acceptance', 'spec:api', 'spec:use_cases', 'spec:server'
+    ]
+
+    desc 'Run RSpec tests'
+    task :spec => spec_task_dependencies
+
+    namespace :spec do
+
+      require 'rspec/core/rake_task'
+      desc 'Run all RSpec tests (unit, integration, acceptance, api, use_cases, server)'
+      task :all => spec_task_dependencies
+
+      desc 'Run RSpec unit tests'
+      RSpec::Core::RakeTask.new(:unit) do |task|
+        task.pattern = 'spec/unit/**/*_spec.rb'
+        #We require spec_helper because some tests
+        #do not include spec_helper by themselves
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run RSpec integration tests'
+      RSpec::Core::RakeTask.new(:integration) do |task|
+        task.pattern = 'spec/integration/**/*_spec.rb'
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run RSpec acceptance tests'
+      RSpec::Core::RakeTask.new(:acceptance) do |task|
+        task.pattern = 'spec/acceptance/**/*_spec.rb'
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run RSpec api tests'
+      RSpec::Core::RakeTask.new(:api) do |task|
+        task.pattern = 'spec/api/**/*_spec.rb'
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run RSpec use_cases tests'
+      RSpec::Core::RakeTask.new(:use_cases) do |task|
+        task.pattern = 'spec/use_cases/**/*_spec.rb'
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run Rspec server tests'
+      RSpec::Core::RakeTask.new(:server) do |task|
+        task.pattern = 'spec/server/**/*_spec.rb'
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+      desc 'Run RSpec test/s specified by input file pattern'
+      RSpec::Core::RakeTask.new(:specific, :test_name) do |task, task_args|
+        task.pattern = "#{task_args[:test_name]}"
+        task.rspec_opts = '--require=spec_helper'
+      end
+
+    end
+
+    desc 'Generate API request documentation from API specs'
+    RSpec::Core::RakeTask.new('docs:generate') do |t|
+      t.pattern = 'spec/acceptance/**/*_spec.rb'
+      t.rspec_opts = ['--format RspecApiDocumentation::ApiFormatter']
+    end
+
+    desc 'Tag and push the current version'
+    task :release => ['release:tag', 'release:push']
+
+    namespace :release do
+      namespace :changelog do
+        task :shortlog do
+          version = `git describe --abbrev=0`.chomp
+          STDOUT.puts "Changes from #{version} to HEAD\n\n"
+          system "git shortlog --no-merges #{version}.."
+        end
+
+        # Link all issues and PRs in the changelog that have no link.
+        #
+        # By default links to PRs (althought GitHub redirects IIRC) but you can
+        # specify an issue link by preceding it with "issue"/"Issue" or specify
+        # a PR link (default) by preceding it with "PR".
+        task :link_prs, [:file] do |_, args|
+          file = args[:file] || File.join(File.dirname(__FILE__), 'CHANGELOG.md')
+
+          File.open file, File::RDWR do |f|
+            contents = f.read
+            contents.
+              # this regexp is not perfect but ok - ie. it would match issue(#5)
+              gsub!(/(\b|[^[:alnum]])([iI]ssue|PR)?([\([:space:]])#(\d+)/) do
+
+              # unfortunately gsub uses globals for groups :(
+              type_prefix, type, separator, number = $1, $2, $3, $4
+
+              link = case type.to_s.upcase
+              when 'ISSUE'
+                # even if quoted like this, remember that \ still escapes
+                %{https://github.com/3scale/apisonator/issues/%s}
+              else
+                # default to PR links
+                %{https://github.com/3scale/apisonator/pull/%s}
+              end
+
+              prefix = if type && separator == ' '
+                         # remove "issue "
+                         type_prefix
+                       else
+                         "#{type_prefix}#{separator}"
+                       end
+
+              prefix << "[##{number}](#{link % number})"
+            end
+
+            # Overwrite the changelog
+            f.seek 0, IO::SEEK_SET
+            f.write contents
+          end
+        end
+      end
+
+      task :tag do
+        require File.dirname(__FILE__) + '/lib/3scale/backend/version'
+        version = "v#{ThreeScale::Backend::VERSION}"
+        STDOUT.puts "Creating tag #{version}"
+        system "git tag -sa #{version} -m \"#{version}\""
+      end
+
+      task :push do
+        system 'git push --tags'
+      end
+    end
+
+    desc 'Start the backend server in development'
+    task :start do
+      system "ruby -Ilib bin/3scale_backend start -p #{ENV['PORT'] || 3001}"
+    end
+
+    desc 'Start a backend_worker in development'
+    task :start_worker do
+      system 'ruby -Ilib bin/3scale_backend_worker --no-daemonize'
+    end
+
+    desc 'Stop a backend_worker in development'
+    task :stop_worker do
+      system 'ruby -Ilib bin/3scale_backend_worker stop'
+    end
+
+    desc 'Restart a backend_worker in development'
+    task :restart_worker do
+      system 'ruby -Ilib bin/3scale_backend_worker restart'
+    end
+
+    desc 'Check license compliance of dependencies'
+    task :license_finder do
+      STDOUT.puts "Checking license compliance\n"
+      unless system("license_finder --decisions-file=#{File.dirname(__FILE__)}" \
+                    "/.dependency_decisions.yml")
+        STDERR.puts "\n*** License compliance test failed  ***\n"
+        exit 1
+      end
+    end
+
+    namespace :license_finder do
+      namespace :report do
+        desc 'Generate an XML report for licenses'
+        task :xml do
+          require 'license_finder'
+          LicenseFinder::CLI::Main.start [
+            'report',
+            "--decisions-file=#{File.dirname(__FILE__)}/.dependency_decisions.yml",
+            '--format=xml',
+            '--quiet'
+          ]
+        end
+      end
+    end
+  end
+end
+
+desc 'Reschedule failed jobs'
+task :reschedule_failed_jobs do
+  reschedule_method = ThreeScale::Backend::FailedJobsScheduler.method(:reschedule_failed_jobs)
+
+  result = if Environment.using_async_redis?
+             Async { reschedule_method.call }.result
+           else
+             reschedule_method.call
+           end
+
+  puts "Rescheduled: #{result[:rescheduled]}. "\
+       "Failed and discarded: #{result[:failed_while_rescheduling]}. "\
+       "Pending failed jobs: #{result[:failed_current]}."
+end
+
+desc 'Delete stats of services marked for deletion'
+namespace :stats do
+  task :cleanup, [:redis_urls, :log_deleted_keys] do |_, args|
+    redis_urls = args[:redis_urls] && args[:redis_urls].split(' ')
+
+    if redis_urls.nil? || redis_urls.empty?
+      puts 'No Redis URLs specified'
+      exit(false)
+    end
+
+    redis_clients = redis_urls.map do |redis_url|
+      parsed_uri = URI.parse(ThreeScale::Backend::Storage::Helpers.send(
+        :to_redis_uri, redis_url)
+      )
+      Redis.new(host: parsed_uri.host, port: parsed_uri.port)
+    end
+
+    log_deleted = args[:log_deleted_keys] == 'true' ? STDOUT : nil
+
+    ThreeScale::Backend::Stats::Cleaner.delete!(
+      redis_clients, log_deleted_keys: log_deleted
+    )
+  end
+end