chef-dk 0.4.0 → 0.5.0.rc.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dc8cebdbcdaa5a3c3f4e2b3bf18fc1c3caca8843
-  data.tar.gz: 9f57e9e37cf5b66823e9a6e3f6470afb369177fe
+  metadata.gz: ba6c0ab110f7a9fdc6434c4999a965e966a6d764
+  data.tar.gz: 1cdc0b41d6847ad1db3f23a83bcc02eaa33f5c1c
 SHA512:
-  metadata.gz: 0f3578a7438281d5360cc4278e16e424e84c1dcec4f7087a8d9ce9dbbecd8a0e74c3b67d5fde04128bf58ee7815a21f3a48e12d444241928f2f2dd75cc1dc082
-  data.tar.gz: e2e691462cb8febe46e0fec9d07f5f6f21e1bfc73bb89c7bb7556e475253f25ad75f521521cdd523f61c62ceaa3ab7323aead451a568c5339e9bd64c0ca61e21
+  metadata.gz: 74ad30f7c8a851fec368a512fb64c352b5bd9e6275598658680394469933a8b9af12fbdeb510f5d04b358ea21ef97ff8afc1b418cf389034944234b6e87fff6c
+  data.tar.gz: a621d898759ebe1527bf1b4c43e7619bbc2c6a4ca001c0fc625aea937c2b9087d55262f15f8a7e19428c2e57cadbcf0f5bd2f2aa1657e15007a4f8fbd5ca46c4

data/CONTRIBUTING.md

@@ -11,9 +11,7 @@ We utilize **Github Issues** for issue tracking and contributions. You can contr
 
 We have a 3 step process that utilizes **Github Issues**:
 
-1. Sign our
-  [Individual Contributor License Agreement (CLA)](https://secure.echosign.com/public/hostedForm?formid=PJIF5694K6L)
-  or [Corporate CLA](https://secure.echosign.com/public/hostedForm?formid=PIE6C7AX856) online once.
+1. Sign or be added to an existing [Contributor License Agreement (CLA)](https://supermarket.chef.io/become-a-contributor).
 2. Create a Github Pull Request.
 3. Do [Code Review](#cr) with the **Chef Engineering Team** or **Chef Core Committers** on the pull request.
 

data/README.md

@@ -17,7 +17,7 @@ package is built via the 'chefdk' project in
 You can get the [latest release of ChefDK from the downloads page][ChefDK].
 
 On Mac OS X, you can also use [homebrew-cask](http://caskroom.io)
-to install ChefDK.
+to `brew cask install chefdk`.
 
 Once you install the package, the `chef-client` suite, `berks`,
 `kitchen`, and this application (`chef`) will be symlinked into your
@@ -104,8 +104,8 @@ in our build cluster to verify each build).
 
 ### `chef exec`
 `chef exec <command>` runs any arbitrary shell command with the PATH
-environment variable and the ruby environment variables (GEM_HOME,
-GEM_PATH, etc) setup to point at the embedded ChefDK omnibus environment.
+environment variable and the ruby environment variables (`GEM_HOME`,
+`GEM_PATH`, etc.) setup to point at the embedded ChefDK omnibus environment.
 
 ### `chef shell-init`
 `chef shell-init SHELL_NAME` emits shell commands that modify your
@@ -114,11 +114,11 @@ help you decide if this is desirable and instructions, see "Using ChefDK
 as Your Primary Development Environment" below.
 
 ### `chef install`
-`chef install` reads a Policyfile.rb document, which contains a
+`chef install` reads a `Policyfile.rb` document, which contains a
 `run_list` and optional cookbook version constraints, finds a set of
 cookbooks that provide the desired recipes and meet dependency
-constraints, and emits a Policyfile.lock.json describing the expanded
-run list and locked cookbook set. The Policyfile.lock.json can be used
+constraints, and emits a `Policyfile.lock.json` describing the expanded
+run list and locked cookbook set. The `Policyfile.lock.json` can be used
 to install the cookbooks on another machine. The policy lock can be
 uploaded to a Chef Server (via the `chef push` command) to apply the
 expanded run list and locked cookbook set to nodes in your
@@ -152,21 +152,27 @@ ChefDK's environment.
 
 To try it temporarily, in a new terminal session, run:
 
-    eval "$(chef shell-init SHELL_NAME)"
+```sh
+eval "$(chef shell-init SHELL_NAME)"
+```
 
-where `SHELL_NAME` is the name of your shell, (usually bash, but zsh is
+where `SHELL_NAME` is the name of your shell (usually bash, but zsh is
 also common). This modifies your `PATH` and `GEM_*` environment
 variables to include ChefDK's paths (run without the `eval` to see the
 generated code). Now your default `ruby` and associated tools will be
 the ones from ChefDK:
 
-    which ruby
-    # => /opt/chefdk/embedded/bin/ruby
+```sh
+which ruby
+# => /opt/chefdk/embedded/bin/ruby
+```
 
 To add ChefDK to your shell's environment permanently, add the
 initialization step to your shell's profile:
 
-    echo 'eval "$(chef shell-init SHELL_NAME)"' >> ~/.YOUR_SHELL_PROFILE
+```sh
+echo 'eval "$(chef shell-init SHELL_NAME)"' >> ~/.YOUR_SHELL_PROFILE
+```
 
 Where `YOUR_SHELL_PROFILE` is `~/.bash_profile` for most bash users,
 `~/.zshrc` for zsh, and `~/.bashrc` on Ubuntu.
@@ -177,7 +183,7 @@ Where `YOUR_SHELL_PROFILE` is `~/.bash_profile` for most bash users,
 
 You can uninstall Chef Development Kit on Mac using below commands:
 
-```
+```sh
 # Remove the installed files
 sudo rm -rf /opt/chefdk
 
@@ -186,7 +192,6 @@ sudo pkgutil --forget com.getchef.pkg.chefdk
 
 # Remove the symlinks under /usr/bin for Chef Development Kit
 ls -la /usr/bin | egrep '/opt/chefdk' | awk '{ print $9 }' | sudo xargs -I % rm -f /usr/bin/%
-
 ```
 
 ### Windows
@@ -198,7 +203,7 @@ Kit from your system.
 
 You can use `rpm` to uninstall Chef Development Kit on RHEL based systems:
 
-```
+```sh
 rpm -qa *chefdk*
 yum remove <package>
 rm -rf /opt/chefdk
@@ -209,7 +214,7 @@ rm -rf ~/.chefdk
 
 You can use `dpkg` to uninstall Chef Development Kit on Ubuntu based systems:
 
-```
+```sh
 dpkg --list | grep chefdk # or dpkg --status chefdk
 
 # Purge chefdk from the system.

data/lib/chef-dk/cli.rb

@@ -22,11 +22,14 @@ require 'chef-dk/builtin_commands'
 require 'chef-dk/helpers'
 require 'chef-dk/ui'
 require 'chef/util/path_helper'
+require 'chef/mixin/shell_out'
+require 'bundler'
 
 module ChefDK
   class CLI
     include Mixlib::CLI
     include ChefDK::Helpers
+    include Chef::Mixin::ShellOut
 
     banner(<<-BANNER)
 Usage:
@@ -84,13 +87,27 @@ BANNER
     def handle_options
       parse_options(argv)
       if config[:version]
-        msg("Chef Development Kit Version: #{ChefDK::VERSION}")
+        show_version
       else
         show_help
       end
       exit 0
     end
 
+    def show_version
+      msg("Chef Development Kit Version: #{ChefDK::VERSION}")
+
+      ["chef-client", "berks", "kitchen"].each do |component|
+        result = Bundler.with_clean_env { shell_out("#{component} --version") }
+        if result.exitstatus != 0
+          msg("#{component} version: ERROR")
+        else
+          version = result.stdout.scan(/[\d+\.]+\S+/).join
+          msg("#{component} version: #{version}")
+        end
+      end
+    end
+
     def show_help
       msg(banner)
       msg("\nAvailable Commands:")

data/lib/chef-dk/command/verify.rb

@@ -98,7 +98,7 @@ module ChefDK
 
       add_component "chef-client" do |c|
         c.base_dir = "chef"
-        c.unit_test { sh("bundle exec rspec -fp spec/unit") }
+        c.unit_test { sh("bundle exec rspec -fp -t '~volatile_from_verify' spec/unit") }
         c.integration_test { sh("bundle exec rspec -fp spec/integration spec/functional") }
 
         c.smoke_test do

data/lib/chef-dk/policyfile/cookbook_location_specification.rb

@@ -80,7 +80,7 @@ module ChefDK
 
       def errors
         error_messages = []
-        if installer.nil?
+        if source_options_invalid?
           error_messages << "Cookbook `#{name}' has invalid source options `#{source_options.inspect}'"
         end
         error_messages
@@ -124,6 +124,10 @@ module ChefDK
         installer.lock_data
       end
 
+      def source_options_invalid?
+        !source_options.empty? && installer.nil?
+      end
+
     end
   end
 end

data/lib/chef-dk/policyfile/read_cookbook_for_compat_mode_upload.rb

@@ -22,6 +22,48 @@ require 'chef/cookbook/chefignore'
 
 module ChefDK
   module Policyfile
+
+    class CookbookLoaderWithChefignore
+
+      # Convenience method to load a cookbook and return a
+      # Chef::CookbookVersion object.
+      #
+      def self.load(name, directory_path)
+        new(name, directory_path).cookbook_version
+      end
+
+      attr_reader :cookbook_name
+      attr_reader :directory_path
+
+      def initialize(cookbook_name, directory_path)
+        @cookbook_name = cookbook_name
+        @directory_path = directory_path
+
+        @cookbook_version = nil
+        @loader = nil
+      end
+
+      def cookbook_version
+        @cookbook_version ||= loader.cookbook_version
+      end
+
+      def loader
+        @loader ||=
+          begin
+            cbvl = Chef::Cookbook::CookbookVersionLoader.new(directory_path, chefignore)
+            cbvl.load!
+            cbvl
+          end
+      end
+
+      def chefignore
+        @chefignore ||= Chef::Cookbook::Chefignore.new(File.join(directory_path, "chefignore"))
+      end
+
+    end
+
+    # TODO: when compat mode is removed, this class should be removed and the
+    # file should be renamed
     class ReadCookbookForCompatModeUpload
 
       # Convenience method to load a cookbook, set up name and version overrides
@@ -47,7 +89,9 @@ module ChefDK
         @cookbook_version ||=
           begin
             cookbook_version = loader.cookbook_version
+            # TODO: dont do this for non-compat mode
             cookbook_version.version = version_override
+            # TODO: dont do this either
 
             # Fixup manifest.
             # What happens is, the 'manifest' representation of cookbook

data/lib/chef-dk/policyfile/uploader.rb

@@ -49,7 +49,16 @@ module ChefDK
       end
 
       def upload
-        ui.msg("WARN: Uploading policy to policy group #{policy_group} in compatibility mode")
+        ui.msg("Uploading policy to policy group #{policy_group}")
+
+        if using_policy_document_native_api?
+          ui.msg(<<-DRAGONS)
+WARN: Using native policy API preview mode. You may be required to delete and
+re-upload this data when upgrading to the final release version of the feature.
+DRAGONS
+        else
+          ui.msg("WARN: Uploading policy to policy group #{policy_group} in compatibility mode")
+        end
 
         upload_cookbooks
         upload_policy
@@ -65,7 +74,7 @@ module ChefDK
       end
 
       def upload_policy_native
-        http_client.put("/policies/#{policy_group}/#{policy_name}", policyfile_lock.to_lock)
+        http_client.put("/policy_groups/#{policy_group}/policies/#{policy_name}", policyfile_lock.to_lock)
       end
 
       def data_bag_create
@@ -97,12 +106,17 @@ module ChefDK
 
       def uploader
         # TODO: uploader runs cookbook validation; we want to do this at a different time.
-        @uploader ||= Chef::CookbookUploader.new(cookbook_versions_to_upload, :rest => http_client)
+        @uploader ||= Chef::CookbookUploader.new(cookbook_versions_to_upload, rest: http_client, policy_mode: using_policy_document_native_api?)
       end
 
       def cookbook_versions_to_upload
         cookbook_versions_for_policy.inject([]) do |versions_to_upload, cookbook_with_lock|
           cb = cookbook_with_lock.cookbook
+          # When we abandon custom identifier support in favor of the one true
+          # hash, identifier generation code can be moved into chef proper and
+          # this can be removed.
+          cb.identifier = cookbook_with_lock.lock.identifier
+
           versions_to_upload << cb unless remote_already_has_cookbook?(cb)
           versions_to_upload
         end
@@ -111,13 +125,27 @@ module ChefDK
       def remote_already_has_cookbook?(cookbook)
         return false unless existing_cookbook_on_remote.key?(cookbook.name.to_s)
 
+        if using_policy_document_native_api?
+          native_mode_cookbook_exists_on_remote?(cookbook)
+        else
+          compat_mode_cookbook_exists_on_remote?(cookbook)
+        end
+      end
+
+      def native_mode_cookbook_exists_on_remote?(cookbook)
+        existing_cookbook_on_remote[cookbook.name.to_s]["versions"].any? do |cookbook_info|
+          cookbook_info["identifier"] == cookbook.identifier
+        end
+      end
+
+      def compat_mode_cookbook_exists_on_remote?(cookbook)
         existing_cookbook_on_remote[cookbook.name.to_s]["versions"].any? do |cookbook_info|
           cookbook_info["version"] == cookbook.version
         end
       end
 
       def existing_cookbook_on_remote
-        @existing_cookbook_on_remote ||= http_client.get('cookbooks?num_versions=all')
+        @existing_cookbook_on_remote ||= http_client.get(list_cookbooks_url)
       end
 
       # An Array of Chef::CookbookVersion objects representing the full set that
@@ -125,7 +153,23 @@ module ChefDK
       def cookbook_versions_for_policy
         return @cookbook_versions_for_policy if @cookbook_versions_for_policy
         policyfile_lock.validate_cookbooks!
-        @cookbook_versions_for_policy = policyfile_lock.cookbook_locks.map do |name, lock|
+        @cookbook_versions_for_policy =
+          if using_policy_document_native_api?
+            load_cookbooks_in_native_mode
+          else
+            load_cookbooks_in_compat_mode
+          end
+      end
+
+      def load_cookbooks_in_native_mode
+        policyfile_lock.cookbook_locks.map do |name, lock|
+          cb = CookbookLoaderWithChefignore.load(name, lock.cookbook_path)
+          LockedCookbookForUpload.new(cb, lock)
+        end
+      end
+
+      def load_cookbooks_in_compat_mode
+        policyfile_lock.cookbook_locks.map do |name, lock|
           cb = ReadCookbookForCompatModeUpload.load(name, lock.dotted_decimal_identifier, lock.cookbook_path)
           LockedCookbookForUpload.new(cb, lock)
         end
@@ -137,8 +181,16 @@ module ChefDK
 
       private
 
+      def list_cookbooks_url
+        if using_policy_document_native_api?
+          'cookbook_artifacts?num_versions=all'
+        else
+          'cookbooks?num_versions=all'
+        end
+      end
+
       def upload_cookbooks
-        ui.msg("WARN: Uploading cookbooks using semver compat mode")
+        ui.msg("WARN: Uploading cookbooks using semver compat mode") unless using_policy_document_native_api?
 
         uploader.upload_cookbooks unless cookbook_versions_to_upload.empty?
 

data/lib/chef-dk/policyfile_lock.rb

@@ -15,6 +15,8 @@
 # limitations under the License.
 #
 
+require 'digest/sha1'
+
 require 'chef-dk/policyfile/storage_config'
 require 'chef-dk/policyfile/cookbook_locks'
 require 'chef-dk/policyfile/solution_dependencies'
@@ -125,6 +127,7 @@ module ChefDK
 
     def to_lock
       {}.tap do |lock|
+        lock["revision_id"] = revision_id
         lock["name"] = name
         lock["run_list"] = run_list
         lock["named_run_lists"] = named_run_lists unless named_run_lists.empty?
@@ -133,6 +136,45 @@ module ChefDK
       end
     end
 
+    # Returns a fingerprint of the PolicyfileLock by computing the SHA1 hash of
+    # #canonical_revision_string
+    def revision_id
+      Digest::SHA1.new.hexdigest(canonical_revision_string)
+    end
+
+    # Generates a string representation of the lock data in a specialized
+    # format suitable for generating a checksum of the lock itself. Only data
+    # that modifies the behavior of a chef-client using the lockfile is
+    # included in this format; for example, a modification to the source
+    # options in a `Policyfile.rb` that yields identical code (such as
+    # switching to a github fork at the same revision) will not cause a change
+    # in the PolicyfileLock's canonical_revision_string.
+    #
+    # This format is intended to be used only for generating an identifier for
+    # a particular revision of a PolicyfileLock. It should not be used as a
+    # serialization format, and is not guaranteed to be a stable interface.
+    def canonical_revision_string
+      canonical_rev_text = ""
+
+      canonical_rev_text << "name:#{name}\n"
+
+      run_list.each do |item|
+        canonical_rev_text << "run-list-item:#{item}\n"
+      end
+
+      named_run_lists.each do |name, run_list|
+        run_list.each do |item|
+          canonical_rev_text << "named-run-list:#{name};run-list-item:#{item}\n"
+        end
+      end
+
+      cookbook_locks_for_lockfile.each do |name, lock|
+        canonical_rev_text << "cookbook:#{name};id:#{lock["identifier"]}\n"
+      end
+
+      canonical_rev_text
+    end
+
     def cookbook_locks_for_lockfile
       cookbook_locks.inject({}) do |locks_map, (name, location_spec)|
         location_spec.validate!

data/lib/chef-dk/skeletons/code_generator/files/default/{repo/cookbooks → cookbook_readmes}/README.md

@@ -1,4 +1,4 @@
-This directory contains the cookbooks used to configure systems in your infrastructure with Chef.
+This directory contains the cookbooks used to configure systems in your infrastructure with Chef - an example basic cookbook called `example` has been automatically created for you.
 
 Knife needs to be configured to know where the cookbooks are located with the `cookbook_path` setting. If this is not set, then several cookbook operations will fail to work properly.
 

data/lib/chef-dk/skeletons/code_generator/files/default/repo/cookbooks/example/attributes/default.rb

@@ -0,0 +1,7 @@
+# This is a Chef attributes file. It can be used to specify default and override
+# attributes to be applied to nodes that run this cookbook.
+
+# Set a default name
+default["example"]["name"] = "Sam Doe"
+
+# For further information, see the Chef documentation (http://docs.getchef.com/essentials_cookbook_attribute_files.html).