kitchen-vmpool 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9e4fff8380ee9f18fec8ef01a1d609b54ebfbfc4
-  data.tar.gz: fdaa0e13db8f1efaeca8c49c0a41382f2a793457
+  metadata.gz: 8e0a7a42b3db814a96216aaa4292bef0fa20b866
+  data.tar.gz: 405f15c69be0901b3fd7d6d3450fa7eeea54ac02
 SHA512:
-  metadata.gz: b3ca03be5000e386e92defb694ac0c9d0efec5778124427c607c190c26b61284e7b33a85447907236955426820059afd313b65cc8b6bcfbf943e1278e56e393b
-  data.tar.gz: 5dc30b54179d657966a1ffdf12b071a58117d25df8533f44e3b72afa631744fa6790a6ab0aef68074a3cb2c3299251d7b6463fb939ccc18ca751476cd776624f
+  metadata.gz: 6a5e6bdc0e6e2235b76b52bb4331a5bc088372c6430e4af37d29f87e0b0aa2d77efe42ced1c061904eafb21642c29afe91d81803d8999e329b3675f51729058b
+  data.tar.gz: b0eec9a6b626562359f18293310938471ddf094ce15009b8a8a8ab6da4ae3d4eefca8255aa295dfff75dac971e2e58dec6dddfadc57ac649a45401fb7ff4316a

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Kitchen-Vmpool
+
+## Unreleased
+
+## v0.3.0
+ * Major refactor of file based stores
+ * changes instances to size
+ * removes separate pool_name in vmpool config
+ * removes reference too payload_file since it is provider specific
+ * Creates base gitlabstore class, moves gitlab monkey patch
+ 
+## v0.2.0
+ * adds a gitlab commit store
+ * Fix accounting bugs with used and pool instances

data/README.md

@@ -1,9 +1,11 @@
 # Kitchen::Vmpool
 
-Provides the ability to select a memeber of a pool to use during a kitchen test.  This allows you to instantly startup
-a test instead of waiting for a machine to provision.
+Ever wished you didn't have to wait for test-kitchen to create a test node?  With kitchen-vmpool you can create your test nodes
+ahead of time and simply populate the pool with the list of hostnames.  During the create step test kitchen will select a member
+of the test pool and you will instantly have a system to work with.
 
-Pluggable backend is used for storing pool information. 
+Kitchen-vmpool allows external scripts or programs to populate the pool so it is not tied to any single vm/container technology.
+Additionaly, kitchen-vmpool contains a pluggable storage backend so that you can store the pool information in just about anything. 
 
 ## Installation
 
@@ -22,12 +24,176 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+To use setup the kitchen driver to use the vmpool store and configure one of the stores if required. 
+
+### Gitlab Commit Store Example
+If you have no central place to store files the gitlab_commit store may be an option for you.
+If you have a simple setup thus may suffice.  If you plan to expand your usage of test kitchen to 
+include multiple parallel test runs, you may run into race conditions due to the lack of a central queue
+to handle reads and writes.  Perfect for simple setups.  Later on you can move to a different state store.
+
+```yaml
+driver:
+  name: vmpool
+  reuse_instances: false
+  state_store: gitlab_commit
+  store_options:
+    pool_file: 'vmpool'
+    project_id: 329302
+    
+platforms:
+    - name: rhel6
+      driver:
+        pool_name: base_rhel6_pool
+    - name: windows10
+      driver:
+        pool_name: windows10_pool  
+    
+```
+
+### Gitlab Snippet Store Example
+I don't recommend using the snippet store because gitlab's permission's model only allows a single
+user to make changes.  Therefore this is a bad central place to keep things if only a single user can update.
+Use this as an example to create your own store providers.
+
+```yaml
+driver:
+  name: vmpool
+  reuse_instances: false
+  state_store: gitlab_snippet
+  store_options:
+    pool_file: 'vmpool'
+    project_id: 329302
+    snippet_id: 49
+    
+platforms:
+    - name: rhel6
+      driver:
+        pool_name: base_rhel6_pool
+    - name: windows10
+      driver:
+        pool_name: windows10_pool    
+```
+
+### Plain old file store
+Probably the easiest to setup, but is useful only for a single person on the same system.
+
+```yaml
+driver:
+  name: vmpool
+  reuse_instances: false
+  state_store: file
+  store_options:
+    pool_file: 'vmpool'
+    
+platforms:
+    - name: rhel6
+      driver:
+        pool_name: base_rhel6_pool
+    - name: windows10
+      driver:
+        pool_name: windows10_pool      
+```
+
+### Pool data structure
+The basic structure of the pool data can be found below.  This is the current format that each state store will follow. 
+
+```yaml
+base_rhel6_pool:
+  payload_file: base_rhel6_payload.json
+  size: 1
+  pool_instances: []
+  requests: []
+  used_instances: []
+  
+windows10_pool:
+  payload_file: windows10_payload.json
+  size: 1
+  pool_instances: []
+  requests: []
+  used_instances: []
+     
+```
+
+The payload_file key is not required and was used for other purposes outside of kitchen-vmpool in order to create the instances.
+It can be expected that some users will throw extra metadata in these pools for their own purposes.  So care must be
+taken to not wipe out this data when creating a store.
+
+### Puppet's VMpooler
+Consider the VMpooler state store to be the ultimate backend for kitchen-vmpool.  While vmpool doesn't currently support vmpooler
+it is on the roadmap to support. 
+
+https://github.com/puppetlabs/vmpooler
+
+Once the vmpooler state store is implemented this kitchen plugin might be pretty popular.
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+This plugin was intended to support multiple ways to populate the pool and multiple ways to store the state of those pools.
+Therefore we leave it up to the user to create the pool instances while kitchen's job is only to interface with the pool information.
+Pool members need to be created outside of kitchen-vmpool.  Additionally pool member information must also be updated outside of kitchen-vmpool.
+
+Puppet's vmpooler will handle the maintenance of the pool state which is probably what you want.
+
 
+### Creating a State Store
+
+In order to create a new state store you must do the following:
+
+1. inherit from the BaseStore or a subclass of the BaseStore
+2. Implement the following methods:
+    * initialize(options = {}) 
+    * take_pool_member
+    * mark_unused
+    
+4. All other methods used with your store must be private 
+
+You must be careful to overwrite the entire pool data.  It is expected that some users
+will put other metadata in the pool file for other purposes.  So when you write your data
+please ensure you merge with the previous data first.
+
+Example:
+
+```ruby
+module Kitchen
+  module Driver
+    module VmpoolStores
+      class FileBaseStore < BaseStore
+        # @return [String] - a random host from the list of systems
+        # mark them used so nobody else can use it
+        def take_pool_member(pool_name)
+          member = pool_hosts(pool_name).sample
+          raise Kitchen::Driver::PoolMemberNotFound.new("No pool members exist for #{pool_name}, please create some pool members") unless member
+          mark_used(member, pool_name)
+          return member
+        end
+
+        # @param name [String] - the hostname to mark not used
+        # @return Array[String] - list of unused instances
+        def mark_unused(name, pool_name, reuse = false)
+          if reuse
+            used_hosts(pool_name).delete(name)
+            pool_hosts(pool_name) << name unless pool_hosts(pool_name).include?(name)
+          end
+          save
+          pool_hosts(pool_name)
+        end
+      end
+    end
+  end    
+end
+
+```
+
+### Adding Configuration for your store
+
+You can pass configuration to your store by setting the `store_options` in the driver section.  This is a simple hash
+that allows the user to pass in required settings if your store requires configuration.  It is up to you
+use these options since not every store will require configuration.
+
+
+### Development Setup
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing

data/exe/vra_create_pool

@@ -20,7 +20,7 @@ def options
 end
 
 def create_pool(pool_data)
-  (1..pool_data['instances']).map do |num| 
+  (1..pool_data['size']).map do |num|
      puts pool_data['payload_file']
      submit_new_request(pool_data['payload_file']).id 
   end
@@ -86,7 +86,7 @@ def create_pools
     # create the pool, and save the request in the requests
     # do not create if the number of systems and requests are more than the requested amount
     current_total = value['pool_instances'].count + pools[key]['requests'].count
-    unless current_total >= value['instances']
+    unless current_total >= value['size']
       reqs = create_pool(value)
       pools[key]['requests'] = reqs
     end

data/lib/kitchen-vmpool/version.rb

@@ -1,3 +1,3 @@
 module KitchenVmpool
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/lib/kitchen/driver/vmpool.rb

@@ -26,8 +26,6 @@ require 'kitchen-vmpool/version'
 module Kitchen
   module Driver
 
-    class PoolMemberNotFound < Exception; end
-
     class Vmpool < Kitchen::Driver::Base
       include Kitchen::Logging
 
@@ -44,78 +42,24 @@ module Kitchen
 
       # (see Base#create)
       def create(state)
-        state[:hostname] = take_pool_member
+        member = store.take_pool_member(config[:pool_name])
+        info("Pool member #{member} was selected")
+        state[:hostname] = member
       end
 
       # (see Base#destroy)
       def destroy(state)
         return if state[:hostname].nil?
-        mark_unused(state[:hostname])
-        state.delete(:hostname)
-      end
-
-      private
-
-      # @return [String] - a random host from the list of systems
-      # mark them used so nobody else can use it
-      def take_pool_member
-        member = pool_hosts.sample
-        raise PoolMemberNotFound.new("No pool members exist for #{config[:pool_name]}, please create some pool members") unless member
-        mark_used(member)
-        info("Pool member #{member} was selected")
-        return member
-      end
-
-      # @return Array[String] - a list of pool names
-      def pool_names
-        store.pool_data.keys
-      end
-
-      # @return [Hash] - a pool hash by the given pool_name from the config
-      def pool
-        name = config[:pool_name]
-        raise ArgumentError.new("Pool #{name} does not exist") unless pool_exists?(name)
-        store.pool_data[name]
-      end
-
-      # @return [Boolean] - true if the pool exists
-      def pool_exists?(name)
-        pool_names.include?(name)
-      end
-
-      # @return Array[String] - a list of host names in the pool
-      def pool_hosts
-        pool['pool_instances'] ||= []
-      end
-
-      # @return Array[String] - a list of used host names in the pool
-      def used_hosts
-        pool['used_instances'] ||= []
-      end
-
-      # @param name [String] - the hostname to mark not used
-      # @return Array[String] - list of unused instances
-      def mark_unused(name)
+        store.mark_unused(state[:hostname], config[:pool_name], config[:reuse_instances])
         if config[:reuse_instances]
           info("Marking pool member #{name} as unused")
-          used_hosts.delete(name)
-          pool_hosts << name unless pool_hosts.include?(name)
+        else
+          info("Marking pool member #{name} as used")
         end
-        store.save
-        pool_hosts
+        state.delete(:hostname)
       end
 
-      # @param name [String] - the hostname to mark used
-      # @return Array[String] - list of used instances
-      def mark_used(name)
-        debug("Marking pool member #{name} as used")
-        # ideally the member should not already be in this array
-        # but just in case we will protect against that
-        pool_hosts.delete(name)
-        used_hosts << name unless used_hosts.include?(name)
-        store.save
-        used_hosts
-      end
+      private
 
       # @return [Hash] - a store hash that contains one or more pools
       def store
@@ -140,23 +84,3 @@ module Kitchen
   end
 end
 
-require 'gitlab'
-# monkey patch error in error code until it is fixed upstream
-module Gitlab
-  module Error
-    class ResponseError
-      # Human friendly message.
-      #
-      # @return [String]
-      private
-      def build_error_message
-        parsed_response = @response.parsed_response
-        message = parsed_response.respond_to?(:message) ? parsed_response.message : parsed_response['message']
-        message = parsed_response.error unless message
-        "Server responded with code #{@response.code}, message: " \
-        "#{handle_message(message)}. " \
-        "Request URI: #{@response.request.base_uri}#{@response.request.path}"
-      end
-    end
-  end
-end

data/lib/kitchen/driver/vmpool_stores/base_store.rb

@@ -1,51 +1,25 @@
 require 'yaml'
-require 'kitchen/logger'
 require 'kitchen'
-require 'kitchen/logging'
 
 module Kitchen
   module Driver
+    class PoolMemberNotFound < Exception; end
+
     module VmpoolStores
       class BaseStore
-        attr_reader :pool_file, :pool_data
-        include Kitchen::Logging
-
-        def update(content = nil)
-          #info("Updating vmpool data")
-          write_content(content)
-          read
-        end
-
-        def create
-          #info("Creating new vmpool data")
-          write_content(base_content)
-          read
-        end
-
-        def read
-          #info("Reading vmpool data")
-          read_content
-        end
-
-        def reread
-          pool_data(true)
-        end
 
-        def save
-          #info("Saving vmpool data")
-          write_content
-          read
+        # @return [String] - a random host from the list of systems
+        # mark them used so nobody else can use it
+        # @param pool_name [String] - the name of the pool to yank the memeber from
+        def take_pool_member(pool_name)
+          raise NotImplemented
         end
 
-        def pool_data(refresh = false)
-          @pool_data = nil if refresh
-          @pool_data ||= YAML.load(read_content)
-        end
-
-        private
-
-        def read_content
-          raise NotImplementedError
+        # @param name [String] - the hostname to mark not used
+        # @param pool_name [String] - the name of the pool to yank the memeber from
+        # @return Array[String] - list of unused instances
+        def mark_unused(name, pool_name, reuse = false)
+          raise NotImplemented
         end
       end
     end

data/lib/kitchen/driver/vmpool_stores/file_base_store.rb

@@ -0,0 +1,113 @@
+require "kitchen/driver/vmpool_stores/base_store"
+
+module Kitchen
+  module Driver
+    class PoolMemberNotFound < Exception; end
+
+    module VmpoolStores
+      class FileBaseStore < BaseStore
+        attr_reader :pool_file, :pool_data
+        include Kitchen::Logging
+
+        # @return [String] - a random host from the list of systems
+        # mark them used so nobody else can use it
+        def take_pool_member(pool_name)
+          member = pool_hosts(pool_name).sample
+          raise Kitchen::Driver::PoolMemberNotFound.new("No pool members exist for #{pool_name}, please create some pool members") unless member
+          mark_used(member, pool_name)
+          return member
+        end
+
+        # @param name [String] - the hostname to mark not used
+        # @return Array[String] - list of unused instances
+        def mark_unused(name, pool_name, reuse = false)
+          if reuse
+            used_hosts(pool_name).delete(name)
+            pool_hosts(pool_name) << name unless pool_hosts(pool_name).include?(name)
+          end
+          save
+          pool_hosts(pool_name)
+        end
+
+        def pool_data(refresh = false)
+          @pool_data = nil if refresh
+          @pool_data ||= YAML.load(read_content)
+        end
+
+        private
+
+        # @return Array[String] - a list of pool names
+        def pool_names
+          pool_data.keys
+        end
+
+        # @return [Hash] - a pool hash by the given pool_name from the config
+        def pool(name)
+          raise ArgumentError.new("Pool #{name} does not exist") unless pool_exists?(name)
+          pool_data[name]
+        end
+
+        # @return [Boolean] - true if the pool exists
+        def pool_exists?(name)
+          pool_names.include?(name)
+        end
+
+        # @return Array[String] - a list of host names in the pool
+        def pool_hosts(pool_name)
+          pool(pool_name)['pool_instances'] ||= []
+        end
+
+        # @return Array[String] - a list of used host names in the pool
+        def used_hosts(pool_name)
+          pool(pool_name)['used_instances'] ||= []
+        end
+
+        # @param name [String] - the hostname to mark used
+        # @return Array[String] - list of used instances
+        def mark_used(name, pool_name)
+          # ideally the member should not already be in this array
+          # but just in case we will protect against that
+          pool_hosts(pool_name).delete(name)
+          used_hosts(pool_name) << name unless used_hosts(pool_name).include?(name)
+          save
+          used_hosts(pool_name)
+        end
+
+        def update(content = nil)
+          #info("Updating vmpool data")
+          write_content(content)
+          read
+        end
+
+        def create
+          #info("Creating new vmpool data")
+          write_content(base_content)
+          read
+        end
+
+        def read
+          #info("Reading vmpool data")
+          read_content
+        end
+
+        def reread
+          pool_data(true)
+        end
+
+        def save
+          #info("Saving vmpool data")
+          write_content
+          read
+        end
+        
+        def read_content
+          raise NotImplementedError
+        end
+
+        def write_content(content = pool_data)
+          raise NotImplementedError
+        end
+      end
+    end
+  end
+end

data/lib/kitchen/driver/vmpool_stores/file_store.rb

@@ -1,9 +1,8 @@
-require 'yaml'
-require "kitchen/driver/vmpool_stores/base_store"
+require "kitchen/driver/vmpool_stores/file_base_store"
 module Kitchen
   module Driver
     module VmpoolStores
-      class FileStore < BaseStore
+      class FileStore < FileBaseStore
 
         # @option pool_file [String] - the file path that holds the pool information
         def initialize(options = nil)
@@ -17,10 +16,9 @@ module Kitchen
         def base_content
           {
             pool1: {
-              pool_name: pool1,
-              payload_file: pool1_payload.yaml,
-              instances: 1,
+              size: 1,
               pool_instances: [],
+              used_instances: [],
               requests: []
             }
           }

data/lib/kitchen/driver/vmpool_stores/gitlab_base_store.rb

@@ -0,0 +1,40 @@
+require 'gitlab'
+require "kitchen/driver/vmpool_stores/file_base_store"
+require 'yaml'
+
+module Kitchen
+  module Driver
+    module VmpoolStores
+      class GitlabBaseStore < FileBaseStore
+
+        private
+
+        def client
+          @client ||= Gitlab.client
+        end
+
+      end
+    end
+  end
+end
+
+# monkey patch error in error code until it is fixed upstream
+module Gitlab
+  module Error
+    class ResponseError
+      # Human friendly message.
+      #
+      # @return [String]
+      private
+      def build_error_message
+        parsed_response = @response.parsed_response
+        message = parsed_response.respond_to?(:message) ? parsed_response.message : parsed_response['message']
+        message = parsed_response.error unless message
+        "Server responded with code #{@response.code}, message: " \
+        "#{handle_message(message)}. " \
+        "Request URI: #{@response.request.base_uri}#{@response.request.path}"
+      end
+    end
+  end
+end
+

data/lib/kitchen/driver/vmpool_stores/gitlab_commit_store.rb

@@ -1,11 +1,9 @@
-require 'gitlab'
-require "kitchen/driver/vmpool_stores/base_store"
-require 'yaml'
+require "kitchen/driver/vmpool_stores/gitlab_base_store"
 
 module Kitchen
   module Driver
     module VmpoolStores
-      class GitlabCommitStore < BaseStore
+      class GitlabCommitStore < GitlabBaseStore
 
         attr_accessor :project_id
         attr_reader :pool_file, :branch
@@ -14,13 +12,17 @@ module Kitchen
         # @option commit_id [Integer] - the snipppet id in the gitlab project
         # @option pool_file [String] - the snipppet file name
         def initialize(options = nil)
-          options ||= { project_id: nil, pool_file: 'vmpool.yaml'}
+          # there is currently some sort of weird bug in gitlab that prevents us from creating files with a yaml extension
+          # thus we have ranmed the default pool file to vmpool
+          options ||= { "project_id" => nil, "pool_file" => 'vmpool'}
           raise ArgumentError.new("You must pass the project_id option") unless options['project_id'].to_i > 0
           @project_id = options['project_id']  #ie. 89
-          @pool_file = options['pool_file'] || 'vmpool.yaml'
+          @pool_file = options['pool_file'] || 'vmpool'
           @branch = 'master'
         end
 
+        private
+
         def update(content = nil)
           #info("Updating vmpool data")
           update_file
@@ -34,7 +36,7 @@ module Kitchen
         end
 
         def save
-         # info("Saving vmpool data")
+          # info("Saving vmpool data")
           update_file
           read
         end
@@ -43,12 +45,6 @@ module Kitchen
           read_content(project, file)
         end
 
-        private
-
-        def client
-          @client ||= Gitlab.client
-        end
-
         def create_file(project = project_id)
           actions = [{
                          "action" => "create",

data/lib/kitchen/driver/vmpool_stores/gitlab_snippet_store.rb

@@ -1,10 +1,8 @@
-require 'gitlab'
-require "kitchen/driver/vmpool_stores/base_store"
-require 'yaml'
+require "kitchen/driver/vmpool_stores/gitlab_base_store"
 module Kitchen
   module Driver
     module VmpoolStores
-      class GitlabSnippetStore < BaseStore
+      class GitlabSnippetStore < GitlabBaseStore
 
         attr_accessor :project_id, :snippet_id
         attr_reader :pool_file
@@ -15,11 +13,13 @@ module Kitchen
         def initialize(options = nil)
           options ||= { project_id: nil, snippet_id: nil, pool_file: 'vmpool'}
           raise ArgumentError.new("You must pass the project_id option") unless options['project_id'].to_i > 0
-          @snippet_id = options['snippet_id']  #ie. 630
+          @snippet_id = options['snippet_id']  #ie. 34422
           @project_id = options['project_id']  #ie. 89
           @pool_file = options['pool_file']
         end
 
+        private
+
         def update(content = nil)
           #info("Updating vmpool data")
           update_snippet
@@ -39,12 +39,6 @@ module Kitchen
           read
         end
 
-        private
-
-        def client
-          @client ||= Gitlab.client
-        end
-
         def snippet_exists?(project = project_id)
           return false unless snippet_id
           client.snippets(project, {

data/vmpool.yaml

@@ -5,3 +5,4 @@ base_rhel6_pool:
   instances: 1
   pool_instances: []
   requests: []
+  used_instances: []

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: kitchen-vmpool
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Corey Osman
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-09-07 00:00:00.000000000 Z
+date: 2017-09-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gitlab
@@ -78,6 +78,7 @@ files:
 - ".gitignore"
 - ".rspec"
 - ".travis.yml"
+- CHANGELOG.md
 - CODE_OF_CONDUCT.md
 - Gemfile
 - LICENSE.txt
@@ -91,7 +92,9 @@ files:
 - lib/kitchen-vmpool/version.rb
 - lib/kitchen/driver/vmpool.rb
 - lib/kitchen/driver/vmpool_stores/base_store.rb
+- lib/kitchen/driver/vmpool_stores/file_base_store.rb
 - lib/kitchen/driver/vmpool_stores/file_store.rb
+- lib/kitchen/driver/vmpool_stores/gitlab_base_store.rb
 - lib/kitchen/driver/vmpool_stores/gitlab_commit_store.rb
 - lib/kitchen/driver/vmpool_stores/gitlab_snippet_store.rb
 - vmpool.yaml