knife-tar 1.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4183e7e044edb84698f6e0a61626f3300f28f213
+  data.tar.gz: a922309881959011c66459e8dc94d9ea05da9594
+SHA512:
+  metadata.gz: 6a93d3e779f0dbb8a75f18a359c4757c0476fd5e5c35de7f55d49476a1e5c5e6098b90f33554adfaabad933eb2be87fbe2a1fdf751e8f838defe0b8afeffeee3
+  data.tar.gz: 2fb7077cd62c278439a56beae3c56d96ac97991f16191b823ee5d5b2b7f86e8b25a9fed338a70c82497b57eac65ba1ea1f768acaa82141f49d2a95c077cf35f1

data/Gemfile

@@ -0,0 +1,5 @@
+# coding: UTF-8
+
+source 'https://rubygems.org/'
+
+gemspec

data/README.md

@@ -0,0 +1,281 @@
+knife-tar
+=========
+
+Description
+-----------
+
+A knife plugin facilitating uploading chef components from a tar.gz file/url to your 
+chef server as well as downloading components from chef-server to tar.gz format.
+
+Why?
+----
+
+This originally started when we were looking into the best way to 'release' our
+cookbooks. Numerous sources online will tell you that the suggested way to 'release'
+a cookbook is to create a tag in your source control manager and upload your cookbook
+from that tag. There are even chef plugins for uploading cookbooks from git as well as
+other tools like [berkshelf](http://berkshelf.com/) that allow you to download cookbooks 
+from github. This idea didn't really mesh well with the way we would release our artifacts 
+which was to push everything into an artifact repository. From there we came up with the idea of 
+dropping all of our cookbook's files into a tar file and push that out to our 
+repositories.
+
+Being beginners at chef we often ran into the issue where we wanted to upload our cookbook 
+to the chef-server but we forgot to include the dependencies. From that we decided our tar 
+file should support multiple cookbooks so we could include cookbook dependencies. At that 
+point others were asking about adding support for roles or environments, that is when we 
+realized we could support all of the chef components and modeled the format after Opscode's 
+[chef-repo](https://github.com/opscode/chef-repo). From there we included the download 
+functionality and supported multiple versions of the same cookbook.
+
+Installation
+------------
+
+Install the chef gem prior to installing knife-tar.
+
+    gem install knife-tar
+
+Requirements
+------------
+
+* Chef >= 0.10.10 (Does not work with Chef 11)
+* `tar` is installed and on your `$PATH`
+
+Conventions
+-----------
+
+### Tar File Structure
+
+The knife-tar plugin focuses to allow a tar file to be used to upload a variety of chef components. Therefore 
+the tar file must be created in a specific way. We tried to follow the chef-repo structure seen here
+(https://github.com/opscode/chef-repo).
+
+We assume that the tar file will look like,
+
+\[tarName\].tar.gz  
+|- cookbooks  
+| |- \[cookbookName\] | \[cookbookName\]-\[cookbookVersion\]  
+|- data_bags  
+| |- \[dataBagName\]  
+| | |- \[valueName\].\[json|js|rb\]  
+|- environments  
+| |- \[environmentName\].\[json|js|rb\]  
+|- roles  
+| |- \[roleName\].\[json|js|rb\]  
+|- web_users  
+| |- \[webUserName\].\[json|js|rb\]  
+|- api_clients  
+| | - \[clientName\].\[json|js|rb\]  
+|- nodes  
+| |- \[nodeName\].\[json|js|rb\]  
+
+OR
+
+\[tarName\].tar.gz  
+|- \[projectName\]   
+| |- cookbooks  
+| | |- \[cookbookName\] | \[cookbookName\]-\[cookbookVersion\]    
+| |- data_bags  
+| | |- \[dataBagName\]  
+| | | |- \[valueName\].\[json|js|rb\]  
+| |- environments  
+| | |- \[environmentName\].\[json|js|rb\]  
+| |- roles  
+| | |- \[roleName\].\[json|js|rb\]  
+| |- web_users  
+| | |- \[webUserName\].\[json|js|rb\]  
+| |- api_clients  
+| | | - \[clientName\].\[json|js|rb\]  
+| |- nodes  
+| | |- \[nodeName\].\[json|js|rb\]  
+
+### Chef Cookbook Names
+
+In order to support uploading multiple versions of the same cookbook the following directory names are valid for cookbooks,
+
+* \[cookbookName\] (i.e. java)
+* \[cookbookName\]-\[cookbookVersion\] (i.e. java-1.0.2)
+
+### Chef Component File Extensions
+
+All chef components files (i.e. all components except cookbooks) must end in the proper extension
+either '.js', '.json' or '.rb' defined by Chef.
+
+### Additional Notes
+
+The root directory (or \[projectName\]) in the second structure can be anything except the names of the chef components 
+(cookbooks, data_bags, environments, roles...)
+
+You do not have to have all of the directories in your tar file in order to use the knife-tar plugin. 
+You only need the directories that have the components you want to use with knife-tar. So if you only want to use 
+it to upload cookbooks a valid tar file could look like,
+
+\[tarName\].tar.gz  
+|- cookbooks  
+| |- \[cookbookName\]  
+
+OR
+
+\[tarName\].tar.gz  
+|- \[projectName\]  
+| |- cookbooks  
+| | |- \[cookbookName\]  
+
+Usage
+-----
+
+In all of the below instances, 'tarPath' can be either the path
+to your tar file on your local filesystem or a url.
+
+### Uploading 
+
+#### Everything
+
+**NOTE**: If you have api clients in your tar file this command requires that you 
+either be running on the chef-server or have configured 'couchdb_url' in your 
+knife.rb to work properly.
+
+If you want to upload all your chef resources from your tar file you can use the
+following command,
+
+Command: 'knife tar upload tarPath (options)'
+
+#### Cookbooks
+
+If you want to upload only your cookbooks from your tar file you can use the
+following command,
+
+Command: 'knife cookbook tar upload tarPath (options)'
+
+#### Roles
+
+If you want to upload only your roles from your tar file you can use the
+following command,
+
+Command: 'knife role tar upload tarPath (options)'
+
+#### Data Bags
+
+If you want to upload only your data bags from your tar file you can use the
+following command,
+
+Command: 'knife data bag tar upload tarPath (options)'
+
+#### Environments
+
+If you want to upload only your environment from your tar file you can use the
+following command,
+
+Command: 'knife environment tar upload tarPath (options)'
+
+#### Nodes
+
+If you want to upload only your nodes from your tar file you can use the
+following command,
+
+Command: 'knife node tar upload tarPath (options)'
+
+#### Web UI Users
+
+If you want to upload only your users from your tar file you can use the
+following command,
+
+Command: 'knife user tar upload tarPath (options)'
+
+
+#### API Clients
+
+**NOTE**: This command requires that you either be running on the chef-server or
+have configured 'couchdb_url' in your knife.rb to work properly.
+
+If you want to upload only your clients from your tar file you can use the
+following command,
+
+Command: 'knife client tar upload tarPath (options)'
+
+### Downloading
+
+#### Everything
+
+If you want to download all of your chef resources from chef-server to a tar
+file you can use the following command,
+
+Command: 'knife tar download tarPath (options)'
+
+#### Cookbooks
+
+If you want to download all your cookbooks from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife cookbook tar download tarPath (options)'
+
+#### Data Bags
+
+If you want to download all your data bags from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife data bag tar download tarPath (options)'
+
+#### Environments
+
+If you want to download all your environments from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife environment tar download tarPath (options)'
+
+#### Nodes
+
+If you want to download all your nodes from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife node tar download tarPath (options)'
+
+#### Roles
+
+If you want to download all your roles from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife role tar download tarPath (options)'
+
+#### Api Clients
+
+If you want to download all your api clients from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife client tar download tarPath (options)'
+
+#### Web UI Users
+
+If you want to download all your web ui users from chef-server to a tar file you
+can use the following command,
+
+Command: 'knife user tar download tarPath (options)'
+
+Author
+------
+
+Bryan Baugher  
+Bryan.Baugher@Cerner.com
+
+Contributing
+------------
+
+This project is licensed under the Apache License, Version 2.0.
+
+When contributing to the project please add your name to the CONTRIBUTORS.txt file. Adding your name to the CONTRIBUTORS.txt file
+signifies agreement to all rights and reservations provided by the License.
+
+To contribute to the project execute a pull request through github. The pull request will be reviewed by the community and merged 
+by the project committers. Please attempt to conform to the test, code conventions, and code formatting standards if any
+are specified by the project before submitting a pull request.
+
+LICENSE
+-------
+
+Copyright 2014 Cerner Innovation, Inc.
+
+Licensed under the Apache License, Version 2.0 (the 'License'); you may not use this file except in compliance with the License. You may obtain a copy of the License at
+
+[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) Unless required by applicable law or agreed to in writing, software distributed 
+under the License is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language 
+governing permissions and limitations under the License.

data/Rakefile

@@ -0,0 +1,60 @@
+# coding: UTF-8
+
+task :default => [:build]
+
+task :build  do
+  puts "Building the gem"
+  runCommand "gem build knife-tar.gemspec"
+end
+
+task :release => [:build] do
+  deployGem
+  release
+end
+
+def deployGem
+  begin
+    puts "Pushing the gem to rubygems.org"
+    runCommand "gem push knife-tar*.gem"
+  ensure
+    system "rm -f knife-tar*.gem"
+  end
+end
+
+def release
+  
+  #Tag the release
+  puts "Tagging the release"
+  runCommand "git tag -a #{rawVersion} -m 'Released #{rawVersion}'"
+  runCommand "git push origin #{rawVersion}"
+  
+  # Update bump VERSION file
+  rawVersion = `cat VERSION`.chomp
+  versions = rawVersion.split "."
+  versions[1] = versions[1].to_i + 1
+  newVersion = versions.join "."
+  
+  puts "Changing version from #{rawVersion} to #{newVersion}"
+  
+  runCommand "echo '#{newVersion}' > VERSION"
+  
+  #Commit the updated VERSION file
+  puts "Commiting the new VERSION file"
+  runCommand "git add VERSION"
+  runCommand "git commit -m 'Released #{rawVersions} and bumped version to #{newVersion}'"
+  runCommand "git push origin master"
+  
+end
+
+def runCommand command
+  output = system command
+  unless output
+    # Removes changes to tracked files
+    system "git reset --hard"
+    
+    # Removes any new un-tracked files
+    system "git clean -f -d"
+    
+    raise "Command : #{command} failed"
+  end
+end

data/lib/chef/knife/client_tar_download.rb

@@ -0,0 +1,40 @@
+require 'chef/knife'
+require 'chef/knife/core/object_loader'
+require 'chef/json_compat'
+require 'chef/api_client'
+
+class Chef
+  class Knife
+    class ClientTarDownload < Chef::Knife
+
+      banner "knife client tar download tarPath [options]"
+      category "client tar"
+
+      def run
+        #Get Arguments
+        if @name_args.size != 1
+          ui.info("Please specify a tar path")
+          show_usage
+          exit 1
+        end
+        
+        tar_file = Chef::TarFile.new(@name_args.first, true)
+        ClientTarDownload.download_clients tar_file
+        tarFile.save
+        
+      end
+      
+      def self.download_clients(tar_file)
+        dir = tar_file.api_clients_path
+        Chef::ApiClient.list.each do |component_name, url|
+          Chef::Log.info("Backing up client #{component_name}")
+          component_obj = Chef::ApiClient.load(component_name)
+          File.open(File.join(dir, "#{component_name}.json"), "w") do |component_file|
+            component_file.print(component_obj.to_json)
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/chef/knife/client_tar_upload.rb

@@ -0,0 +1,49 @@
+require 'chef/knife'
+require 'chef/knife/core/object_loader'
+require 'chef/json_compat'
+
+class Chef
+  class Knife
+    class ClientTarUpload < Chef::Knife
+
+      banner "knife client tar upload tarPath [options]"
+      category "client tar"
+
+      def run
+        #Get Arguments
+        if @name_args.size != 1
+          ui.info("Please specify a tar path")
+          show_usage
+          exit 1
+        end
+        
+        tar_file = Chef::TarFile.new(@name_args.first)
+        ClientTarUpload.upload_clients tar_file
+        
+      end
+      
+      def self.upload_clients(tar_file)
+        ui.confirm "This command will only work when running on chef-server or by updating the couchdb_url in your knife config to point to your couchdb instance. Are you sure you want to continue"
+        
+        client_loader = Chef::Knife::Core::ObjectLoader.new(Chef::ApiClient, ui)
+        current_clients = Chef::ApiClient.list.keys
+        
+        tar_file.api_clients.each do |api_client_path|
+          
+          client = client_loader.load_from("clients", api_client_path)
+          
+          # In order to 'update' a client we have to remove it first, so if the client exists destroy it
+          if current_clients.include? client.name
+            ApiClient.load(client.name).destroy
+          end
+          
+          client.cdb_save
+          
+          ui.info("Updated Client : #{client.name}")
+          
+        end
+      end
+
+    end
+  end
+end

data/lib/chef/knife/cookbook_tar_download.rb

@@ -0,0 +1,49 @@
+require 'chef/knife'
+require 'chef/cookbook/metadata'
+require 'chef/cookbook_uploader'
+require 'chef/tar_file'
+require 'chef/cookbook_version'
+require 'chef/rest'
+
+class Chef
+  class Knife
+    class CookbookTarDownload < Chef::Knife
+      
+      banner "knife cookbook tar download tarPath (options)"
+      category "cookbook tar"
+      
+      # This method will be executed when you run this knife command.
+      def run
+        
+        #Get Arguments
+        if @name_args.size != 1
+          ui.info("Please specify a tarPath")
+          show_usage
+          exit 1
+        end
+        
+        tar_file = Chef::TarFile.new(@name_args.first, true)
+        CookbookTarDownload.download_cookbooks tar_file
+        tar_file.save
+        
+      end
+      
+      def self.download_cookbooks(tar_file)
+        #Gets the list of cookbooks and their versions
+        rest = Chef::REST.new(Chef::Config[:chef_server_url])
+        cookbook_versions = rest.get_rest('cookbooks?num_versions=all')
+        
+        cookbook_versions.each do | cookbook_name, cookbook_hash |
+          cookbook_hash['versions'].each do | version_hash |
+            cookbook_download = Chef::Knife::CookbookDownload.new
+            cookbook_download.config[:download_directory] = tar_file.cookbooks_path
+            cookbook_download.name_args.push cookbook_name
+            cookbook_download.name_args.push version_hash["version"]
+            cookbook_download.run
+          end
+        end
+      end
+      
+    end
+  end
+end

data/lib/chef/knife/cookbook_tar_upload.rb

@@ -0,0 +1,178 @@
+require 'chef/knife'
+require 'chef/cookbook/metadata'
+require 'chef/cookbook_uploader'
+require 'chef/tar_file'
+require 'tmp_directory'
+
+class Chef
+  class Knife
+    class CookbookTarUpload < Chef::Knife
+      
+      # This is the list of CookbookVersion's that are already uploaded to the chef-server
+      @@current_cookbooks = nil
+      
+      banner "knife cookbook tar upload tarPath (options)"
+      category "cookbook tar"
+      
+      # This method will be executed when you run this knife command.
+      def run
+        
+        if @name_args.size != 1
+          ui.info("Please specify a tarPath")
+          show_usage
+          exit 1
+        end
+        
+        tar_file = Chef::TarFile.new(@name_args.first)
+        CookbookTarUpload.upload_cookbooks tar_file
+        
+      end
+      
+      def self.upload_cookbooks(tar_file)
+        # This is the list of cookbooks and their groupings, we upload all the cookbooks in a group at a time
+        cookbook_upload_groups = Array.new
+        
+        cookbook_paths = tar_file.cookbooks
+        
+        cookbook_group_index = 0
+        
+        total_cookbooks = cookbook_paths.length
+        
+        # Since we now allow you to upload multiple versions of the same cookbook we have to be smart about how we upload the cookbooks
+        # If we upload a cookbook without the proper dependency we will get errors
+        
+        # So the strategy is to categorize cookbooks into groups based on their dependencies and what has been uploaded
+        # Then move the cookbooks to a tmp directory (renaming their directories if necessary) and upload each group in order
+        
+        while !cookbook_paths.empty?
+          
+          cookbook_upload_groups.push(Array.new)
+          
+          # An array of current cookbook names we are uploading. This is used to ensure we dont upload the same cookbook 
+          # (different version) in the same group
+          cookbookNamesToUpload = Array.new
+          
+          cookbook_paths.each do |cookbook_path|
+            cookbookName = CookbookTarUpload.get_cookbook_name_from_path cookbook_path
+            
+            # Verify we aren't uploading another version of the cookbook in this group
+            unless cookbookNamesToUpload.index(cookbookName)
+              
+              # Check the cookbook has no dependencies or its dependencies have been uploaded
+              
+              md = Chef::Cookbook::Metadata.new
+              md.from_file File.join(cookbook_path, "metadata.rb")
+              
+              uploadable = true
+              md.dependencies.each do |cookbook_dependency, version_constraint|
+                  
+                # Verify we have uploaded the specific cookbook / version
+                unless CookbookTarUpload.is_dependency_uploaded? cookbook_dependency, version_constraint, cookbook_upload_groups
+                  ui.error "Cookbook #{cookbookName} depends on cookbook '#{cookbook_dependency}' version '#{version_constraint}',"
+                  ui.error "which is not currently being uploaded and cannot be found on the server."
+                  uploadable = false
+                  break
+                end  
+                  
+              end
+              
+              if uploadable
+                # Add the cookbook to the group
+                cookbook_upload_groups[cookbook_group_index].push cookbook_path
+                # Add the cookbook's name to our list
+                cookbookNamesToUpload.push(cookbookName)
+              end
+            end
+          end
+          
+          # If we still have cookbooks that need to find a group but we did not add any to this group
+          # we have an error. This can be caused by a missing dependency or a circular dependency.
+          if cookbookNamesToUpload.empty?
+            raise "Unable to upload cookbooks"
+          end
+          
+          # Remove the cookbooks we have added to this group
+          cookbook_upload_groups[cookbook_group_index].each do |cookbook_path|
+            cookbook_paths.delete cookbook_path
+          end
+          
+          cookbook_group_index+=1
+          
+        end
+        
+        puts "#{total_cookbooks} cookbooks have been grouped into #{cookbook_upload_groups.length} groups"
+        puts "Uploading cookbooks ..."
+        
+        # Upload all cookbooks one group at a time
+        cookbook_upload_groups.each do |cookbookGroup|
+          cookbookTmpPath = ::TmpDirectory.new("cookbooks").path
+          
+          # Move cookbooks to tmp directory and rename if necessary
+          cookbookGroup.each do |cookbook_path|
+            FileUtils.cp_r cookbook_path, File.join(cookbookTmpPath, CookbookTarUpload.get_cookbook_name_from_path(cookbook_path))
+          end
+          
+          #Upload cookbook group
+          cookbookUploader = Chef::Knife::CookbookUpload.new
+          cookbookUploader.config[:cookbook_path] = cookbookTmpPath
+          cookbookUploader.config[:all] = true
+          cookbookUploader.run
+        end
+      end
+      
+      private 
+      
+      # Returns the name of the cookbook from its path which could include its version information
+      # (i.e. /var/chef/cookbooks/myCookbook-1.0.0 would return myCookbook)      
+      def self.get_cookbook_name_from_path path
+        directory = File.basename(path)
+        index = directory =~ /(-[\d]+.[\d]+\.[\d]+\z|-[\d]+\.[\d]+\z)/
+        if index!=nil
+          return directory[0, index]
+        end
+        directory
+      end
+      
+      # Given a cookbook's dependency information, and the current cookbooks that will be uploaded, determine if 
+      # we have uploaded a valid version of that cookbook
+      def self.is_dependency_uploaded? cookbook_dependency, version_constraint, cookbook_upload_groups
+        vc = Chef::VersionConstraint.new(version_constraint)
+        
+        cookbook_upload_groups.each do |cookbook_group|
+          cookbook_group.each do |cookbook_path|
+            if CookbookTarUpload.get_cookbook_name_from_path(cookbook_path)==cookbook_dependency
+              # Get the version of the cookbook that is already in a group
+              md = Chef::Cookbook::Metadata.new
+              md.from_file File.join(cookbook_path, "metadata.rb")
+              if vc.include? md.version
+                return true
+              end
+            end
+          end
+        end
+        
+        # The dependency we are wanting is not part of an upload group so check if it already exists on the chef-server
+        if CookbookTarUpload.current_cookbooks.has_key? cookbook_dependency
+          CookbookTarUpload.current_cookbooks[cookbook_dependency].each do |cookbookVersions|
+            CookbookTarUpload.current_cookbooks[cookbook_dependency]["versions"].each do |cookbookVersion|
+              if vc.include? cookbookVersion["version"]
+                return true
+              end
+            end
+          end
+        end
+        
+        false
+      end
+      
+      def self.current_cookbooks
+        unless @@current_cookbooks
+          rest = Chef::REST.new(Chef::Config[:chef_server_url])
+          @@current_cookbooks = rest.get_rest('cookbooks?num_versions=all')
+        end
+        @@current_cookbooks
+      end
+      
+    end
+  end
+end