3scale_toolbox 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1faca82c3c88cb4312fa76704e70d656a400f53c3aecb03c507e8a1aeb374284
-  data.tar.gz: 32c28726d94b60b8238720b1ffe055b06b4ecb1aab3945fd069b899d19f12595
+  metadata.gz: bc76523728a2d5fa59805175676eac19b02a689db6c2b6045d821f8b6d5b8fc3
+  data.tar.gz: 33908dec9c0714102d0899643e40bcc933b90a412b440325ef8476b4427d8596
 SHA512:
-  metadata.gz: 720b320e9e20edecdb93256c2a804ffaff13593f180960e2e10027d5c915c94094a7c2bbf2aca9c9a3b78fc7e9c993bcfc3151d4ac61eab8bf0113a63ed0d5d5
-  data.tar.gz: c27a4f1aea1076ad163437cdf0b0082763a6f49fa10bcd2045fe63047ea08dc3ba0258833f8fcd8015a8de4604909c9ba7cf9b477b55511999a4c3544dcab5a2
+  metadata.gz: ff2630a709bb3af2a8e93cc650359f8d3461444a41438aa261415922a02b9ea9f425dbd027f6d8d46652462544d4e388e9b15ef5066f772bd1ae99457794ead5
+  data.tar.gz: 22c140218478ee375e3762e5cb5ffe674e9d1502e7b702d7303341c98d84ad4680ef3887f6608f9ed42cf7bb584a195a099e33d6dfc4196c11201be21b2ca920

data/README.md

@@ -2,9 +2,18 @@
 
 3scale toolbox is a set of tools to help you manage your 3scale product. Using the [3scale API Ruby Client](https://github.com/3scale/3scale-api-ruby).
 
-## Installation
-
+## Table of contents
+* [Installation](#installation)
+* [Usage](#usage)
+   * [Copy a service](#copy-a-service)
+   * [Update a service](#update-a-service)
+   * [Import from CSV](#import-from-csv)
+* [Development](#development)
+* [Plugins](#plugins)
+* [Troubleshooting](#troubleshooting)
+* [Contributing](#contributing)
 
+## Installation
 Install the CLI:
 
     $ gem install 3scale_toolbox
@@ -12,20 +21,150 @@ Install the CLI:
 ## Usage
 
 ```shell
-3scale help
+$ 3scale help
+NAME
+    3scale - 3scale CLI Toolbox
+
+USAGE
+    3scale <command> [options]
+
+DESCRIPTION
+    3scale CLI tools to manage your API from the terminal.
+
+COMMANDS
+    copy       3scale copy command
+    help       show help
+    import     3scale import command
+    update     3scale update command
+
+OPTIONS
+    -k --insecure      Proceed and operate even for server connections
+                       otherwise considered insecure
+    -v --version       Prints the version of this command
 ```
 
 ### Copy a service
+Will create a new services, copy existing proxy settings, metrics, methods, application plans and mapping rules.
+
+Help message:
 
-Will create a new service, copy existing methods, metrics, application plans and their usage limits.
+```shell
+$ 3scale copy service --help
+NAME
+    service - Copy service
+
+USAGE
+    3scale copy service [opts] -s <src> -d <dst>
+    <service_id>
+
+DESCRIPTION
+    Will create a new services, copy existing proxy settings, metrics,
+    methods, application plans and mapping rules.
+
+OPTIONS
+    -d --destination=<value>             3scale target instance. Format:
+                                         "http[s]://<provider_key>@3scale_url"
+    -s --source=<value>                  3scale source instance. Format:
+                                         "http[s]://<provider_key>@3scale_url"
+    -t --target_system_name=<value>      Target system name
+
+OPTIONS FOR COPY
+    -h --help                            show help for this command
+    -k --insecure                        Proceed and operate even for server
+                                         connections otherwise considered
+                                         insecure
+    -v --version                         Prints the version of this command
+```
 
 ```shell
 3scale copy service NUMBER --source=https://provider_key@foo-admin.3scale.net --destination=https://provider_key@foo2-admin.3scale.net
 ```
 
+### Update a service
+
+Will update existing service, update proxy settings, metrics, methods, application plans and mapping rules.
+
+Help message:
+
+```shell
+NAME
+    service - Update service
+
+USAGE
+    3scale update service [opts] -s <src> -d <dst>
+    <src_service_id> <dst_service_id>
+
+DESCRIPTION
+    Will update existing service, update proxy settings, metrics, methods,
+    application plans and mapping rules.
+
+OPTIONS
+    -d --destination=<value>      3scale target instance. Format:
+                                  "http[s]://<provider_key>@3scale_url"
+    -f --force                    Overwrites the mapping rules by deleting
+                                  all rules from target service first
+    -r --rules-only               Updates only the mapping rules
+    -s --source=<value>           3scale source instance. Format:
+                                  "http[s]://<provider_key>@3scale_url"
+
+OPTIONS FOR UPDATE
+    -h --help                     show help for this command
+    -k --insecure                 Proceed and operate even for server
+                                  connections otherwise considered insecure
+    -v --version                  Prints the version of this command
+```
+
+Example:
+
+```shell
+$ 3scale update service -s https://9874598743@source.example.com -d https://2342342342342@destination.example.com 3 2
+```
+
 ### Import from CSV
 
-Will create a new services, metrics, methods and mapping rules.
+Will create new services, metrics, methods, and mapping rules having as source comma separated values (CSV) formatted file.
+
+CSV header
+
+```csv
+service_name,endpoint_name,endpoint_http_method,endpoint_path,auth_mode,endpoint_system_name,type
+```
+
+File example
+
+```csv
+service_name,endpoint_name,endpoint_http_method,endpoint_path,auth_mode,endpoint_system_name,type
+Movies ,Movies (Biography),GET,/movies/biography/,api_key,movies_biography,metric
+Movies ,Movies (Drama),GET,/movies/drama/,api_key,movies_drama,method
+```
+
+Help message:
+
+```shell
+$ 3scale import csv -h
+NAME
+    csv - Import csv file
+
+USAGE
+    3scale import csv [opts] -d <dst> -f <file>
+
+DESCRIPTION
+    Create new services, metrics, methods and mapping rules from CSV
+    formatted file
+
+OPTIONS
+    -d --destination=<value>      3scale target instance. Format:
+                                  "http[s]://<provider_key>@3scale_url"
+    -f --file=<value>             CSV formatted file
+
+OPTIONS FOR IMPORT
+    -h --help                     show help for this command
+    -k --insecure                 Proceed and operate even for server
+                                  connections otherwise considered insecure
+    -v --version                  Prints the version of this command
+```
+
+Example:
 
 ```shell
 3scale import csv --destination=https://provider_key@user-admin.3scale.net --file=examples/import_example.csv
@@ -37,6 +176,17 @@ After checking out the repo, run `bin/setup` to install dependencies. You can al
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
+## Plugins
+
+As of 3scale Toolbox 0.5.0, 3scale Toolbox will load plugins installed in gems or $LOAD_PATH. Plugins are discovered via Gem::find_files then loaded.
+Install, uninstall and update plugins using tools like [RubyGems](https://guides.rubygems.org/rubygems-basics/) and/or [Bundler](https://bundler.io/).
+
+[Make your own plugin](docs/plugins.md)
+
+## Troubleshooting
+
+* [SSL errors](docs/ssl_errors.md): If you run into SSL issues with the toolbox, you can take actions to resolve them.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/3scale/3scale_toolbox.

data/exe/3scale

@@ -1,14 +1,8 @@
 #!/usr/bin/env ruby
 
+require '3scale_toolbox'
 require '3scale_toolbox/cli'
 
-options, argv = ThreeScaleToolbox::CLI.parse
+args = ARGV.clone
 
-unless options.command
-  puts 'Available subcommands: '
-  puts ThreeScaleToolbox::CLI.subcommands
-  puts
-  ThreeScaleToolbox::CLI.print_help!
-end
-
-exec options.command.full_path, *ARGV
+ThreeScaleToolbox::CLI.run args

data/lib/3scale_toolbox/base_command.rb

@@ -0,0 +1,28 @@
+
+module ThreeScaleToolbox
+  module Command
+    def subcommands
+      @subcommands ||= []
+    end
+
+    def add_subcommand(command)
+      subcommands << command
+    end
+
+    ##
+    # Override to command
+    #
+    def command
+      raise Exception, 'base command has no command definition'
+    end
+
+    ##
+    # Iterate recursively over command tree
+    #
+    def build_command
+      subcommands.each_with_object(command) do |subcommand, root_command|
+        root_command.add_command(subcommand.build_command)
+      end
+    end
+  end
+end

data/lib/3scale_toolbox/cli.rb

@@ -1,78 +1,22 @@
-require 'optparse'
+require '3scale_toolbox'
+require '3scale_toolbox/commands'
 
-module ThreeScaleToolbox
-  module CLI
-    Options = Struct.new(:command)
-
-    class Parser
-      def self.parse(options)
-        args = Options.new(nil)
-
-        opt_parser = OptionParser.new do |opts|
-          opts.banner = "Usage: 3scale <command> [options]"
-
-
-          opts.on("-h", "--help", "Prints this help") do
-            puts opts
-            exit
-          end
-        end
-
-        begin
-          opt_parser.order!(options)
-        rescue OptionParser::InvalidOption => e
-          p e
-        end
-
-        return args
-      end
-    end
-
-    def self.parse(argv = ARGV)
-      options = Parser.parse(argv)
-      options.command = argv.shift
-      options.command = subcommands.find { |subcommand| subcommand.name == options.command }
-
-      [ options, argv ]
-    end
-
-    def self.print_help!
-      Parser.parse %w[--help]
-    end
-
-    def self.plugins
-      Gem.loaded_specs.select{ |name, _| name.start_with?('3scale') }.values
-    end
-
-    def self.current_command
-      File.expand_path($0, Dir.pwd)
-    end
-
-    def self.subcommands
-      plugins
-          .flat_map { |spec| spec.executables.flat_map{ |bin| Subcommand.new(bin, spec) } }
-          .reject { |subcommand| subcommand.full_path == current_command || subcommand.name.nil? }
-    end
-
-    class Subcommand
-      attr_reader :executable, :spec
-
-      def initialize(executable, spec)
-        @executable = executable
-        @spec = spec
-      end
+module ThreeScaleToolbox::CLI
+  def self.root_command
+    ThreeScaleToolbox::Commands::ThreeScaleCommand
+  end
 
-      def to_s
-        name
-      end
+  def self.add_command(command)
+    root_command.add_subcommand(command)
+  end
 
-      def name
-        executable.split('-', 2)[1]
-      end
+  def self.load_builtin_commands
+    ThreeScaleToolbox::Commands::BUILTIN_COMMANDS.each(&method(:add_command))
+  end
 
-      def full_path
-        spec.bin_file(executable)
-      end
-    end
+  def self.run(args)
+    load_builtin_commands
+    ThreeScaleToolbox.load_plugins
+    root_command.build_command.run args
   end
 end

data/lib/3scale_toolbox/commands/3scale_command.rb

@@ -0,0 +1,28 @@
+require 'cri'
+require '3scale_toolbox/version'
+require '3scale_toolbox/base_command'
+
+module ThreeScaleToolbox
+  module Commands
+    module ThreeScaleCommand
+      extend ThreeScaleToolbox::Command
+      def self.command
+        Cri::Command.define do
+          name        '3scale'
+          usage       '3scale <command> [options]'
+          summary     '3scale CLI Toolbox'
+          description '3scale CLI tools to manage your API from the terminal.'
+          flag :v, :version, 'Prints the version of this command' do |_, _|
+            puts ThreeScaleToolbox::VERSION
+            exit
+          end
+          flag :k, :insecure, 'Proceed and operate even for server connections otherwise considered insecure'
+          flag :h, :help, 'show help for this command' do |_, cmd|
+            puts cmd.help
+            exit 0
+          end
+        end
+      end
+    end
+  end
+end

data/lib/3scale_toolbox/commands/copy_command/copy_service.rb

@@ -0,0 +1,218 @@
+require 'cri'
+require '3scale_toolbox/base_command'
+
+module ThreeScaleToolbox
+  module Commands
+    module CopyCommand
+      module CopyServiceSubcommand
+        extend ThreeScaleToolbox::Command
+        def self.command
+          Cri::Command.define do
+            name        'service'
+            usage       'service [opts] -s <src> -d <dst> <service_id>'
+            summary     'Copy service'
+            description 'Will create a new services, copy existing proxy settings, metrics, methods, application plans and mapping rules.'
+
+            required  :s, :source, '3scale source instance. Format: "http[s]://<provider_key>@3scale_url"'
+            required  :d, :destination, '3scale target instance. Format: "http[s]://<provider_key>@3scale_url"'
+            required  :t, 'target_system_name', 'Target system name'
+
+            run do |opts, args, _|
+              CopyServiceSubcommand.run opts, args
+            end
+          end
+        end
+
+        def self.run(opts, args)
+          source      = fetch_required_option(opts, :source)
+          destination = fetch_required_option(opts, :destination)
+          system_name = fetch_required_option(opts, :target_system_name)
+          insecure    = opts[:insecure] || false
+          exit_with_message 'error: missing service_id argument' if args.empty?
+          service_id = args[0]
+          copy_service(service_id, source, destination, system_name, insecure)
+        end
+
+        def self.exit_with_message(message)
+          puts message
+          exit 1
+        end
+
+        def self.fetch_required_option(options, key)
+          options.fetch(key) { exit_with_message "error: Missing argument #{key}" }
+        end
+
+        def self.compare_hashes(first, second, keys)
+          keys.map{ |key| first.fetch(key) } == keys.map{ |key| second.fetch(key) }
+        end
+
+        def self.provider_key_from_url(url)
+          url[/\w*@/][0..-2]
+        end
+
+        def self.endpoint_from_url(url)
+          url.sub /\w*@/, ''
+        end
+
+
+        # Returns new hash object with not nil valid params
+        def self.filter_params(valid_params, source)
+          valid_params.each_with_object({}) do |key, target|
+            target[key] = source[key] unless source[key].nil?
+          end
+        end
+
+        def self.copy_service_params(original, system_name)
+          service_params = filter_params(Commands.service_valid_params, original)
+          service_params.tap do |hash|
+            hash['system_name'] = system_name if system_name
+          end
+        end
+
+        def self.copy_service(service_id, source, destination, system_name, insecure)
+          require '3scale/api'
+
+          source_client = ThreeScale::API.new(
+            endpoint:     endpoint_from_url(source),
+            provider_key: provider_key_from_url(source),
+            verify_ssl: !insecure
+          )
+          client = ThreeScale::API.new(
+            endpoint:     endpoint_from_url(destination),
+            provider_key: provider_key_from_url(destination),
+            verify_ssl: !insecure
+          )
+
+          service = source_client.show_service(service_id)
+          copy    = client.create_service(copy_service_params(service, system_name))
+
+          raise "Service has not been saved. Errors: #{copy['errors']}" unless copy['errors'].nil?
+
+          service_copy_id = copy.fetch('id')
+
+          puts "new service id #{service_copy_id}"
+
+          proxy = source_client.show_proxy(service_id)
+          client.update_proxy(service_copy_id, proxy)
+          puts "updated proxy of #{service_copy_id} to match the original"
+
+          metrics = source_client.list_metrics(service_id)
+          metrics_copies = client.list_metrics(service_copy_id)
+
+          hits = metrics.find{ |metric| metric['system_name'] == 'hits' } or raise 'missing hits metric'
+          hits_copy = metrics_copies.find{ |metric| metric['system_name'] == 'hits' } or raise 'missing hits metric'
+
+          methods = source_client.list_methods(service_id, hits['id'])
+          methods_copies = client.list_methods(service_copy_id, hits_copy['id'])
+
+          puts "original service hits metric #{hits['id']} has #{methods.size} methods"
+          puts "copied service hits metric #{hits_copy['id']} has #{methods_copies.size} methods"
+
+          missing_methods = methods.reject { |method|  methods_copies.find{|copy| compare_hashes(method, copy, ['system_name']) } }
+
+          puts "creating #{missing_methods.size} missing methods on copied service"
+
+          missing_methods.each do |method|
+            copy = { friendly_name: method['friendly_name'], system_name: method['system_name'] }
+            client.create_method(service_copy_id, hits_copy['id'], copy)
+          end
+
+          metrics_copies = client.list_metrics(service_copy_id)
+
+          puts "original service has #{metrics.size} metrics"
+          puts "copied service has #{metrics_copies.size} metrics"
+
+          missing_metrics = metrics.reject { |metric| metrics_copies.find{|copy| compare_hashes(metric, copy, ['system_name']) } }
+
+          missing_metrics.map do |metric|
+            metric.delete('links')
+            client.create_metric(service_copy_id, metric)
+          end
+
+          puts "created #{missing_metrics.size} metrics on the copied service"
+
+          plans = source_client.list_service_application_plans(service_id)
+          plan_copies = client.list_service_application_plans(service_copy_id)
+
+          puts "original service has #{plans.size} application plans "
+          puts "copied service has #{plan_copies.size} application plans"
+
+          missing_application_plans = plans.reject { |plan| plan_copies.find{|copy| plan.fetch('system_name') == copy.fetch('system_name') } }
+
+          puts "copied service missing #{missing_application_plans.size} application plans"
+
+          missing_application_plans.each do |plan|
+            plan.delete('links')
+            plan.delete('default') # TODO: handle default plan
+
+            if plan.delete('custom') # TODO: what to do with custom plans?
+              puts "skipping custom plan #{plan}"
+            else
+              client.create_application_plan(service_copy_id, plan)
+            end
+          end
+
+          application_plan_mapping = client.list_service_application_plans(service_copy_id).map do |plan_copy|
+            plan = plans.find{|plan| plan.fetch('system_name') == plan_copy.fetch('system_name') }
+
+            [plan['id'], plan_copy['id']]
+          end
+
+          metrics_mapping = client.list_metrics(service_copy_id).map do |copy|
+            metric = metrics.find{|metric| metric.fetch('system_name') == copy.fetch('system_name') }
+            metric ||= {}
+
+            [metric['id'], copy['id']]
+          end.to_h
+
+          puts "destroying all mapping rules of the copy which have been created by default"
+          client.list_mapping_rules(service_copy_id).each do |mapping_rule|
+            client.delete_mapping_rule(service_copy_id, mapping_rule['id'])
+          end
+
+          mapping_rules = source_client.list_mapping_rules(service_id)
+          mapping_rules_copy = client.list_mapping_rules(service_copy_id)
+
+          puts "the original service has #{mapping_rules.size} mapping rules"
+          puts "the copy has #{mapping_rules_copy.size} mapping rules"
+
+          unique_mapping_rules_copy = mapping_rules_copy.dup
+
+          missing_mapping_rules = mapping_rules.reject do |mapping_rule|
+            matching_metric = unique_mapping_rules_copy.find do |copy|
+              compare_hashes(mapping_rule, copy, %w(pattern http_method delta)) &&
+                metrics_mapping.fetch(mapping_rule.fetch('metric_id')) == copy.fetch('metric_id')
+            end
+
+            unique_mapping_rules_copy.delete(matching_metric)
+          end
+
+          puts "missing #{missing_mapping_rules.size} mapping rules"
+
+          missing_mapping_rules.each do |mapping_rule|
+            mapping_rule.delete('links')
+            mapping_rule['metric_id'] = metrics_mapping.fetch(mapping_rule.delete('metric_id'))
+            client.create_mapping_rule(service_copy_id, mapping_rule)
+          end
+          puts "created #{missing_mapping_rules.size} mapping rules"
+
+          puts "extra #{unique_mapping_rules_copy.size} mapping rules"
+          puts unique_mapping_rules_copy.each{|rule| rule.delete('links') }
+
+          application_plan_mapping.each do |original_id, copy_id|
+            limits = source_client.list_application_plan_limits(original_id)
+            limits_copy = client.list_application_plan_limits(copy_id)
+
+            missing_limits = limits.reject { |limit| limits_copy.find{|limit_copy| limit.fetch('period') == limit_copy.fetch('period') } }
+
+            missing_limits.each do |limit|
+              limit.delete('links')
+              client.create_application_plan_limit(copy_id, metrics_mapping.fetch(limit.fetch('metric_id')), limit)
+            end
+            puts "copied application plan #{copy_id} is missing #{missing_limits.size} from the original plan #{original_id}"
+          end
+        end
+      end
+    end
+  end
+end