cloudstack_client 1.6.0 → 2.0.0

data/README.md

@@ -1,24 +1,34 @@
 # cloudstack_client
 
-[![Gem Version](https://badge.fury.io/rb/cloudstack_client.png)](http://badge.fury.io/rb/cloudstack_client)
+[![Gem Version](https://img.shields.io/gem/v/cloudstack_client.svg)](https://rubygems.org/gems/cloudstack_client)
+[![CI](https://github.com/niwo/cloudstack_client/actions/workflows/ci.yml/badge.svg)](https://github.com/niwo/cloudstack_client/actions/workflows/ci.yml)
 
 A CloudStack API client written in Ruby.
 
+## Version 2.0.0
+
+Version 2.0.0 requires Ruby 3.0 or newer. It also fixes request dispatch when
+no custom host is configured and improves support for CloudStack commands with
+acronyms, such as `create_ssh_key_pair`.
+
 ## Installation
 
 Install the cloudstack_client gem:
 
 ```bash
-$ gem install cloudstack_client
+gem install cloudstack_client
 ```
 
 ## Features
 
 - Access to the whole CloudStack-API from Ruby
-- Interactive console for playing with the CloudStack API: ```cloudstack_client console```
+- Interactive console for playing with the CloudStack API:
+  `cloudstack_client console`
 - Dynamically builds API methods based on the listApis function of CloudStack
-- Command names are converted to match Ruby naming conventions (i.e. ListVirtualMachines becomes list_virtual_machines)
-- Accepts Ruby Hash arguments passed to commands as options (i.e. list_all: true becomes listall=true)
+- Command names are converted to match Ruby naming conventions (i.e.
+  ListVirtualMachines becomes list_virtual_machines)
+- Accepts Ruby Hash arguments passed to commands as options (i.e. `list_all: true`
+  becomes `listall=true`)
 - Assure all required arguments are passed
 - Removes unsupported arguments and arguments with nil values from commands
 
@@ -69,11 +79,15 @@ cs = CloudstackClient::Client.new(
 
 ### Pagination and Response Options
 
-When working with paginated responses, you can include the total count in the API response:
+When working with paginated responses, you can include the total count in the
+API response:
 
 ```ruby
 # Get paginated results with count information
-vms = cs.list_virtual_machines({ page: 1, pagesize: 10 }, { include_count: true })
+vms = cs.list_virtual_machines(
+  { page: 1, pagesize: 10 },
+  { include_count: true }
+)
 total_count = vms[:count]
 items = vms[:virtualmachine]
 
@@ -84,9 +98,10 @@ vms = cs.list_virtual_machines(page: 1, pagesize: 10)
 
 ### Using the configuration module
 
-The configuration module of CloudstackClient makes it easy to load CloudStack API settings from configuration files.
+The configuration module of CloudstackClient makes it easy to load CloudStack
+API settings from configuration files.
 
-#### Example
+#### Configuration Example
 
 ```ruby
 require "cloudstack_client"
@@ -94,7 +109,11 @@ require "cloudstack_client/configuration"
 
 # looks for ~/.cloudstack.yml per default
 config = CloudstackClient::Configuration.load
-cs = CloudstackClient::Client.new(config[:url], config[:api_key], config[:secret_key])
+cs = CloudstackClient::Client.new(
+  config[:url],
+  config[:api_key],
+  config[:secret_key]
+)
 ```
 
 #### Configuration files
@@ -120,62 +139,117 @@ test:
 
 ### Configuration options
 
-You can pass `options` as 4th argument in `CloudstackClient::Client.new`. All its keys are optional.
+You can pass `options` as 4th argument in `CloudstackClient::Client.new`.
+All its keys are optional.
 
 ```ruby
 options = {
-  symbolize_keys: true, # pass symbolize_names: true in JSON#parse for Cloudstack responses, default: false
-  host: 'localhost', # custom host header to be used in Net::Http. May be useful when Cloudstack is set up locally via docker (i.e. Cloudstack-simulator), default: parsed from config[:url] via Net::Http
-  read_timeout: 10, # timeout in seconds of a connection to the Cloudstack, default: 60
-  request_retries: 3 # number of attempts for HTTP requests before raising a ConnectionError, default: 1 (no retries). Uses incremental back-off between attempts.
+  # Pass symbolize_names: true in JSON#parse for CloudStack responses.
+  # Default: false.
+  symbolize_keys: true,
+  # Custom host header for Net::HTTP. Useful with CloudStack-simulator.
+  # Default: parsed from config[:url] via Net::HTTP.
+  host: 'localhost',
+  # Timeout in seconds for a connection to CloudStack. Default: 60.
+  read_timeout: 10,
+  # HTTP request attempts before raising ConnectionError. Default: 1.
+  # Uses incremental back-off between attempts.
+  request_retries: 3,
+  # Verify HTTPS certificates. Default: true.
+  verify_ssl: true,
+  # Optional path to a custom CA certificate bundle.
+  ca_file: "/path/to/ca-bundle.pem",
+  # Raise ParameterError for unsupported command parameters. Default: false.
+  strict_params: false
 }
-cs = CloudstackClient::Client.new(config[:url], config[:api_key], config[:secret_key], options)
+cs = CloudstackClient::Client.new(
+  config[:url],
+  config[:api_key],
+  config[:secret_key],
+  options
+)
 ```
 
-For a single call you can override defaults on the **second** hash (client options), without changing the client instance:
+HTTPS certificate verification is enabled by default. Set `verify_ssl: false`
+only when connecting to a trusted endpoint with a certificate that cannot be
+validated, such as a local development server.
+
+By default, unsupported command parameters are ignored. Set
+`strict_params: true` to raise `ParameterError` instead.
+
+For a single call you can override defaults on the **second** hash (client
+options), without changing the client instance:
 
 ```ruby
-cs.deploy_virtual_machine({ zoneid: "...", serviceofferingid: "...", templateid: "..." },
-                          async_timeout: 600, async_poll_interval: 5)
+cs.deploy_virtual_machine(
+  { zoneid: "...", serviceofferingid: "...", templateid: "..." },
+  async_timeout: 600,
+  async_poll_interval: 5
+)
 ```
 
 ### Interactive Console
 
 cloudstack_client comes with an interactive console.
 
-#### Example
+#### Console Example
 
 ```bash
-$ cloudstack_client console -e prod
+cloudstack_client console -e prod
 prod >> list_virtual_machines
 ```
 
 ## Development
 
+### Running the tests
+
+```bash
+bundle install
+bundle exec rake          # tests and RuboCop
+bundle exec rake test     # tests only
+bundle exec rake rubocop  # RuboCop only
+bundle exec rake benchmark
+```
+
+Tests use [Minitest](https://github.com/minitest/minitest) in spec style and
+[WebMock](https://github.com/bblimke/webmock). Outbound network access is
+disabled in the test suite, so any request that is not stubbed fails the test
+rather than reaching a real endpoint. Helpers for stubbing CloudStack responses
+live in `test/support/api_stubs.rb`.
+
+`Gemfile.lock` is deliberately not checked in, so each supported Ruby version
+resolves its own compatible dependency set.
+
 ### Generate or update API definitions
 
 New API definitions can be generated using the `list_apis` command.
 
-#### Example
+#### API Definition Example
 
 ```bash
 # running against a CloudStack 4.15 API endpoint:
-$ cloudstack_client list_apis > data/4.15.json
-$ gzip data/4.15.json
+cloudstack_client list_apis > data/4.15.json
+gzip data/4.15.json
 ```
 
 ### GitHub Actions
 
 This repository includes GitHub Actions workflows for:
 
-- Running tests and gem build on every push and pull request (`CI`)
+- Running tests and gem build against every supported Ruby version, plus a
+  RuboCop lint job, on every push and pull request (`CI`)
 - Publishing the gem to RubyGems when a GitHub Release is published (`Release`)
 
-To enable publishing, add this repository secret:
+Dependency and GitHub Actions updates are proposed weekly by Dependabot.
 
-- `RUBYGEMS_AUTH_TOKEN`: your RubyGems API key with push permissions
+To enable publishing, configure a RubyGems trusted publisher for this
+repository, workflow, and `rubygems` environment. The release workflow grants
+the publish job GitHub's OIDC `id-token` permission, so no RubyGems API key
+secret is required.
 
-The release workflow checks that `CloudstackClient::VERSION` is greater than the latest version on RubyGems before building, then uses the `rubygems` environment to publish.
+The release workflow checks that `CloudstackClient::VERSION` is greater than
+the latest version on RubyGems before building, then uses the `rubygems`
+environment to publish.
 
 ## References
 
@@ -191,4 +265,6 @@ The release workflow checks that `CloudstackClient::VERSION` is greater than the
 
 ## License
 
-Released under the MIT License. See the [LICENSE](https://raw.github.com/niwo/cloudstack_client/master/LICENSE.txt) file for further details.
+Released under the MIT License. See the
+[LICENSE](https://raw.github.com/niwo/cloudstack_client/master/LICENSE.txt)
+file for further details.

data/Rakefile

@@ -9,5 +9,20 @@ Rake::TestTask.new do |t|
   t.warning = false
 end
 
-desc "Run Tests"
-task default: :test
+begin
+  require 'rubocop/rake_task'
+  RuboCop::RakeTask.new
+rescue LoadError
+  desc "rubocop is not available"
+  task :rubocop do
+    abort "RuboCop is not available. Run `bundle install` first."
+  end
+end
+
+desc "Run the API benchmark"
+task :benchmark do
+  ruby "-Ilib test/benchmark.rb"
+end
+
+desc "Run tests and RuboCop"
+task default: %i[test rubocop]

data/cloudstack_client.gemspec

@@ -8,20 +8,39 @@ Gem::Specification.new do |gem|
   gem.version       = CloudstackClient::VERSION
   gem.authors       = ["Nik Wolfgramm"]
   gem.email         = ["nik.wolfgramm@gmail.com"]
-  gem.description   = %q{CloudStack API client written in Ruby}
+  gem.description   = %q{A Ruby client for the Apache CloudStack API that builds its
+                          methods dynamically from the CloudStack API definition,
+                          including an interactive console.}.gsub(/\s+/, ' ')
   gem.summary       = %q{CloudStack API client written in Ruby}
   gem.homepage      = "https://github.com/niwo/cloudstack_client"
   gem.license       = 'MIT'
 
-  gem.required_ruby_version = '>= 1.9.3'
+  gem.required_ruby_version = '>= 3.0'
   gem.files         = `git ls-files`.split($/)
   gem.executables   = %w(cloudstack_client)
-  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
   gem.rdoc_options  = %w[--line-numbers --inline-source]
 
+  gem.metadata = {
+    "homepage_uri"      => gem.homepage,
+    "bug_tracker_uri"   => "#{gem.homepage}/issues",
+    "changelog_uri"     => "#{gem.homepage}/releases",
+    "rubygems_mfa_required" => "true"
+  }
+
+  # base64 stopped being a default gem in Ruby 3.4, so `require "base64"`
+  # in connection.rb fails under Bundler unless it is declared here.
+  gem.add_dependency('base64', '>= 0.1')
+
   gem.add_development_dependency('rake', '~> 13.0')
   gem.add_development_dependency('thor', '~> 1.1')
   gem.add_development_dependency('ripl', '~> 0.7')
-  gem.add_development_dependency('minitest', '~> 5.14')
+  # Allow minitest 6 on Ruby >= 3.2 while staying installable on Ruby 3.0/3.1.
+  gem.add_development_dependency('minitest', '>= 5.14', '< 7')
+  gem.add_development_dependency('minitest-reporters', '~> 1.8')
+  gem.add_development_dependency('webmock', '~> 3.26')
+  gem.add_development_dependency('benchmark', '>= 0.3')
+  gem.add_development_dependency('rubocop', '~> 1.90')
+  gem.add_development_dependency('rubocop-minitest', '~> 0.40')
+  gem.add_development_dependency('rubocop-rake', '~> 0.7')
 end

data/lib/cloudstack_client/api.rb

@@ -1,5 +1,6 @@
 require "zlib"
 require "json"
+require "cloudstack_client/error"
 require "cloudstack_client/utils"
 
 module CloudstackClient
@@ -25,14 +26,29 @@ module CloudstackClient
     end
 
     def command_supported?(command)
-      @commands.has_key? underscore_to_camel_case(command)
+      !find_command(command).nil?
     end
 
     def command_supports_param?(command, key)
-      command = underscore_to_camel_case(command)
-      @commands[command]["params"].detect do |params|
-        params["name"] == key.to_s
-      end ? true : false
+      command = find_command(command)
+      return false if command.nil?
+
+      command["params"].any? { |param| param["name"] == key.to_s }
+    end
+
+    # Resolves a command given either its CloudStack name ("createSSHKeyPair")
+    # or its underscored Ruby name ("create_ssh_key_pair").
+    #
+    # underscore_to_camel_case cannot be relied on here: converting to
+    # underscores loses acronym casing, so "create_ssh_key_pair" converts back
+    # to "createSshKeyPair", which is not a CloudStack command. Underscored
+    # names are resolved through an index built in the lossy direction instead.
+    #
+    # The index is built on first use: most lookups arrive as CloudStack names
+    # and hit @commands directly, so instantiating an Api should not pay for it.
+    def find_command(command)
+      command = command.to_s
+      @commands[command] || underscored_commands[command]
     end
 
     def required_params(command)
@@ -68,11 +84,7 @@ module CloudstackClient
     end
 
     def set_api_path(options)
-      @api_path = if options[:api_path]
-        File.expand_path(options[:api_path])
-      else
-        API_PATH
-      end
+      @api_path = options[:api_path] ? File.expand_path(options[:api_path]) : API_PATH
     end
 
     def set_api_version(options)
@@ -89,13 +101,26 @@ module CloudstackClient
       @api_version
     end
 
+    def underscored_commands
+      @underscored_commands ||= @commands.each_with_object({}) do |(name, command), index|
+        index[camel_case_to_underscore(name)] = command
+      end
+    end
+
     def load_commands
       @commands = {}
-      Zlib::GzipReader.open(@api_file) do |gz|
+      @underscored_commands = nil
+      parsed = Zlib::GzipReader.open(@api_file) do |gz|
         JSON.parse(gz.read)
-      end.each {|cmd| @commands[cmd["name"]] = cmd }
-    rescue => e
-      raise "Error: Unable to read file '#{@api_file}': #{e.message}"
+      end
+      unless parsed.is_a?(Array) && parsed.all?(Hash)
+        raise ApiDefinitionError,
+              "Unable to read API definition '#{@api_file}': unexpected format (expected an array of command hashes)"
+      end
+      parsed.each { |cmd| @commands[cmd["name"]] = cmd }
+    rescue Zlib::Error, JSON::ParserError, SystemCallError, EOFError => e
+      raise ApiDefinitionError,
+            "Unable to read API definition '#{@api_file}': #{e.message}"
     end
 
   end

data/lib/cloudstack_client/cli.rb

@@ -98,7 +98,7 @@ module CloudstackClient
 
       ARGV.clear
       Ripl.config[:prompt] = "#{@config[:environment]} >> "
-      Ripl.start binding: cs_client.instance_eval{ binding }
+      Ripl.start binding: cs_client.instance_eval { binding }
     end
 
     no_commands do

data/lib/cloudstack_client/client.rb

@@ -25,8 +25,11 @@ module CloudstackClient
 
           args.each do |k, v|
             k = k.to_s.gsub("_", "")
-            if v && @api.command_supports_param?(command["name"], k)
+            if !v.nil? && @api.command_supports_param?(command["name"], k)
               params[k] = v
+            elsif !v.nil? && options.fetch(:strict_params, @options[:strict_params])
+              raise ParameterError,
+                    "#{command['name']} does not support parameter #{k}"
             end
           end
 

data/lib/cloudstack_client/configuration.rb

@@ -10,7 +10,7 @@ module CloudstackClient
       end
 
       begin
-        config = YAML::load(IO.read file)
+        config = YAML.safe_load(IO.read(file), permitted_classes: [Symbol])
       rescue => e
         message = "Can't load configuration from file '#{file}'."
         if configuration[:debug]
@@ -20,13 +20,25 @@ module CloudstackClient
         raise ConfigurationError, message
       end
 
+      unless config.is_a?(Hash)
+        raise ConfigurationError, "Configuration file '#{file}' must contain a hash."
+      end
+
       if env = configuration[:env] || config[:default]
         unless config = config[env]
           raise ConfigurationError, "Can't find environment #{env}."
         end
       end
 
-      unless config.key?(:url) && config.key?(:api_key) && config.key?(:secret_key)
+      unless config.is_a?(Hash)
+        raise ConfigurationError, "Environment #{env} must contain a hash."
+      end
+
+      required_keys = %i[url api_key secret_key]
+      missing_keys = required_keys.reject do |key|
+        config.key?(key) && !config[key].to_s.strip.empty?
+      end
+      unless missing_keys.empty?
         message = "The environment #{env || '\'-\''} does not contain all required keys."
         raise ConfigurationError, message
       end

data/lib/cloudstack_client/connection.rb

@@ -4,12 +4,16 @@ require "uri"
 require "cgi"
 require "net/http"
 require "json"
+require "cloudstack_client/request_handling"
 
 module CloudstackClient
   class Connection
     include Utils
+    include RequestHandling
 
-    attr_accessor :api_url, :api_key, :secret_key, :verbose, :debug, :symbolize_keys, :host, :read_timeout
+    attr_reader :api_key, :secret_key
+    attr_accessor :api_url, :verbose, :debug, :symbolize_keys, :host, :read_timeout
+    attr_accessor :verify_ssl, :ca_file
     attr_accessor :async_poll_interval, :async_timeout, :request_retries
 
     DEF_POLL_INTERVAL = 2.0
@@ -25,6 +29,8 @@ module CloudstackClient
       @debug = options[:debug] ? true : false
       @symbolize_keys = options[:symbolize_keys] ? true : false
       @host = options[:host]
+      @verify_ssl = options.fetch(:verify_ssl, true)
+      @ca_file = options[:ca_file]
       @read_timeout = options[:read_timeout] || DEF_REQ_TIMEOUT
       @async_poll_interval = options[:async_poll_interval] || DEF_POLL_INTERVAL
       @async_timeout = options[:async_timeout] || DEF_ASYNC_TIMEOUT
@@ -37,60 +43,6 @@ module CloudstackClient
     # Sends a synchronous request to the CloudStack API and returns the response as a Hash.
     #
 
-    def send_request(params, opts = {})
-      params['response'] = 'json'
-      params['apiKey'] = @api_key
-      print_debug_output JSON.pretty_generate(params) if @debug
-
-      data = params_to_data(params)
-      uri = URI.parse "#{@api_url}?#{data}&signature=#{create_signature(data)}"
-
-      http = Net::HTTP.new(uri.host, uri.port)
-      if uri.scheme == 'https'
-        http.use_ssl = true
-        http.verify_mode = OpenSSL::SSL::VERIFY_NONE
-      end
-      http.read_timeout = @read_timeout
-
-      retries = 0
-      begin
-        req = Net::HTTP::Get.new(uri.request_uri)
-        req['Host'] = host if host.present?
-        response = http.request(req)
-      rescue => e
-        retries += 1
-        if retries < @request_retries
-          sleep(retries) # incremental back-off
-          print "." if @verbose
-          retry
-        end
-        raise ConnectionError, "API URL \'#{@api_url}\' is not reachable (after #{retries} attempt#{'s' if retries > 1}): #{e.message}"
-      end
-
-      begin
-        body = JSON.parse(response.body, symbolize_names: @symbolize_keys).values.first
-      rescue JSON::ParserError
-        raise ParseError,
-              "Response from server is not readable. Check if the API endpoint (#{@api_url}) is valid and accessible."
-      end
-
-      if response.is_a?(Net::HTTPOK)
-        return body unless body.respond_to?(:keys)
-        if body.size == 2 && body.key?(k('count'))
-          return opts[:include_count] ? body : body.reject { |key, _| key == k('count') }.values.first
-        elsif body.size == 1 && body.values.first.respond_to?(:keys)
-          item = body.values.first
-          return (item.is_a?(Array) || item.is_a?(Hash)) ? item : []
-        else
-          body.reject! { |key, _| key == k('count') } if body.key?(k('count')) && !opts[:include_count]
-          body.size == 0 ? [] : body
-        end
-      else
-        message = body[k('errortext')] rescue body
-        raise ApiError, "Status #{response.code}: #{message}."
-      end
-    end
-
     ##
     # Sends an asynchronous request and waits for the response.
     #
@@ -116,7 +68,11 @@ module CloudstackClient
         when 1
           return data[k('jobresult')]
         when 2
-          raise JobError, "Request failed (#{data[k('jobresultcode')]}): #{data[k('jobresult')][k('errortext')]}."
+          result = data[k('jobresult')]
+          error_text = result.is_a?(Hash) ? result[k('errortext')] : nil
+          error_text ||= "Unknown error"
+          raise JobError,
+                "Request failed (#{data[k('jobresultcode')]}): #{error_text}."
         end
 
         STDOUT.flush if @verbose

data/lib/cloudstack_client/error.rb

@@ -4,6 +4,7 @@ module CloudstackClient
   class ConnectionError < Error; end
   class ConfigurationError < Error; end
   class ParseError < Error; end
+  class ApiDefinitionError < Error; end
   class ApiError < Error; end
   class JobError < Error; end
   class TimeoutError < Error; end

data/lib/cloudstack_client/request_handling.rb

@@ -0,0 +1,91 @@
+module CloudstackClient
+  module RequestHandling
+    def send_request(params, opts = {})
+      params['response'] = 'json'
+      params['apiKey'] = @api_key
+      print_debug_output JSON.pretty_generate(params) if @debug
+
+      data = params_to_data(params)
+      uri = URI.parse "#{@api_url}?#{data}&signature=#{create_signature(data)}"
+      http = build_http(uri)
+      response = request_with_retries(http, uri)
+
+      handle_response(response, opts)
+    end
+
+    private
+
+    def build_http(uri)
+      http = Net::HTTP.new(uri.host, uri.port)
+      return configure_https(http) if uri.scheme == 'https'
+
+      http.read_timeout = @read_timeout
+      http
+    end
+
+    def configure_https(http)
+      http.use_ssl = true
+      http.verify_mode = @verify_ssl ? OpenSSL::SSL::VERIFY_PEER : OpenSSL::SSL::VERIFY_NONE
+      http.ca_file = @ca_file if @ca_file
+      http.read_timeout = @read_timeout
+      http
+    end
+
+    def request_with_retries(http, uri)
+      retries = 0
+      begin
+        request = Net::HTTP::Get.new(uri.request_uri)
+        request['Host'] = host unless host.to_s.strip.empty?
+        http.request(request)
+      rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::EHOSTUNREACH,
+             Errno::ENETUNREACH, Errno::ETIMEDOUT, Net::OpenTimeout,
+             Net::ReadTimeout, SocketError, EOFError => e
+        retries += 1
+        if retries < @request_retries
+          sleep(retries)
+          print "." if @verbose
+          retry
+        end
+        raise ConnectionError,
+              "API URL '#{@api_url}' is not reachable " \
+              "(after #{retries} attempt#{'s' if retries > 1}): #{e.message}"
+      end
+    end
+
+    def handle_response(response, opts)
+      body = parse_response_body(response)
+      return normalize_success(body, opts) if response.is_a?(Net::HTTPOK)
+
+      message = body.is_a?(Hash) ? body[k('errortext')] : body
+      raise ApiError, "Status #{response.code}: #{message}."
+    end
+
+    def parse_response_body(response)
+      JSON.parse(response.body, symbolize_names: @symbolize_keys).values.first
+    rescue JSON::ParserError
+      raise ParseError,
+            "Response from server is not readable. Check if the API endpoint " \
+            "(#{@api_url}) is valid and accessible."
+    end
+
+    def normalize_success(body, opts)
+      return body unless body.respond_to?(:keys)
+      return normalize_counted_response(body, opts) if body.size == 2 && body.key?(k('count'))
+      return normalize_nested_response(body) if body.size == 1 && body.values.first.respond_to?(:keys)
+
+      body.reject! { |key, _| key == k('count') } if body.key?(k('count')) && !opts[:include_count]
+      body.empty? ? [] : body
+    end
+
+    def normalize_counted_response(body, opts)
+      return body if opts[:include_count]
+
+      body.reject { |key, _| key == k('count') }.values.first
+    end
+
+    def normalize_nested_response(body)
+      item = body.values.first
+      item.is_a?(Array) || item.is_a?(Hash) ? item : []
+    end
+  end
+end

data/lib/cloudstack_client/version.rb

@@ -1,3 +1,3 @@
 module CloudstackClient
-  VERSION = "1.6.0"
+  VERSION = "2.0.0"
 end