kitchen-scribe 0.1.0 → 0.2.0

data/Gemfile

@@ -2,3 +2,4 @@ source :rubygems
 
 gem "chef"
 gem "rspec"
+gem "kitchen-scribe"

data/Gemfile.lock

@@ -25,6 +25,8 @@ GEM
     highline (1.6.15)
     ipaddress (0.8.0)
     json (1.6.1)
+    kitchen-scribe (0.1.0)
+      chef (>= 0.10.10)
     mime-types (1.19)
     mixlib-authentication (1.3.0)
       mixlib-log
@@ -70,4 +72,5 @@ PLATFORMS
 
 DEPENDENCIES
   chef
+  kitchen-scribe
   rspec

data/README.md

@@ -4,33 +4,42 @@ Kitchen Scribe
 DESCRIPTION
 -----------
 
-This is a small Knife plugin for keeping track of various changes in your Chef environemnt.
+This is a small Knife plugin for making and keeping track of various changes in your Chef environemnt.
 
-In it's core it pulls the configuration of all your environements, roles, and nodes then saves them into json files in the place of your choosing and commits it using a local git repository. It can also pull/push them to a remote repostory for safekeeping.
+It performs two main functions. 
 
-The plugin is in an early alpha (as indicated by the commit dates) so things may change rather drastically in the near future.
+1.  _Documenting changes_. It pulls the configuration of all your environements, roles, and nodes then saves them into json files in the place of your choosing and commits the changes to a local git repository. It can also pull/push them to a remote repostory for safekeeping.
+
+2.  _Making precise changes_. It can perform precise updates on your environments, roles and nodes by using json data structure describing the change. *[This feature is still being tested and it's not available through rubygems yet. If you would like to try it please install it driectly from the github repo]* 
+
+The philosophy behind using scribe to update your environments, roles an nodes is that you may want to make prepare some changes in advance, be able to test them and then have them applied to the final setup. Also it might be important to isolate those changes in a clear way so people who are not familiar with chef don't have to edit a huge json object to get them in. Lastly you can now automate applying your changes as well, and automation is what Chef is all about in the end:). 
+
+The plugin is still in the beta stage, I've tested it manualy to some extent, but I'm sure there are things I missed. Please submit any bugs you find through the github issue system and I promiss to take care of them as soon as possible.
 
 [![Code Climate](https://codeclimate.com/github/khozlov/kitchen-scribe.png)](https://codeclimate.com/github/khozlov/kitchen-scribe)
+[![Gem Version](https://badge.fury.io/rb/kitchen-scribe.png)](http://badge.fury.io/rb/kitchen-scribe)
 
 USAGE
 -----
 
-Currently Kitchen Scribe can perform two actions.
+Install as gem using `gem install kitchen-scribe`.
+
+### Documenting Changes
 
 First you need to hire a scribe using `knife scribe hire`. By default this will initialize a local directory called `.chronicle` where all backups will be performed. This task takes the following parameters:
 
-* `-p` or `--chronicle-path` followed by a path allows you to specify a different location for the backup dir.
-* `-r` or `--remote-url` followed by a git remote url will set up a remote in the backup dir pointing to the specified url. By default the name of the remote will be `origin`
-* `--remote-name` followed by a remote_name allows you to specify a name for the remote repository. If `-r` is not used this has no effect.
+*   `-p` or `--chronicle-path` followed by a path allows you to specify a different location for the backup dir.
+*   `-r` or `--remote-url` followed by a git remote url will set up a remote in the backup dir pointing to the specified url. By default the name of the remote will be `origin`
+*   `--remote-name` followed by a remote_name allows you to specify a name for the remote repository. If `-r` is not used this has no effect.
 
 `hire` action can be performed multiple times to set up additional remotes.
 
-Next you probably want to get your scribe to actually do something for a change. `knife scribe copy` will fully back up your `roles` and `environments` and partially backup your `nodes` (only `name`, `env`, `attributes` and `run_list`). It assumes that you already hired your scribe and by default will look for your chronicle at `.chronicle`, assume that your remote name is `origin`, your default branch name is `master` and you want each of your commit messages to say _"Commiting chef state as of [current date and time]"_. Again you may customize this using:
+Next you probably want to get your scribe to actually do something for a change. `knife scribe copy` will fully back your `roles` and `environments` up and partially back your `nodes` up (only `name`, `env`, `attributes` and `run_list`). It assumes that you already hired your scribe and by default will look for your chronicle at `.chronicle`, assume that your remote name is `origin`, your default branch name is `master` and you want each of your commit messages to say _"Commiting chef state as of [current date and time]"_. Again you may customize this using:
 
-* `-p` or `--chronicle-path` followed by a path to specify a different location for the chronicle.
-* `--remote-name` followed by a remote_name to specify a different remote name.
-* `--branch` followed by a branch name to indicate that you want to use a different branch.
-* `-m` or `--commit-message` followed by a message to, suprise, suprise, specify your own custom message (use `%TIME%` anywhere in the message to get it substitued with current time).
+*   `-p` or `--chronicle-path` followed by a path to specify a different location for the chronicle.
+*   `--remote-name` followed by a remote_name to specify a different remote name.
+*   `--branch` followed by a branch name to indicate that you want to use a different branch.
+*   `-m` or `--commit-message` followed by a message to, suprise, suprise, specify your own custom message (use `%TIME%` anywhere in the message to get it substitued with current time).
 
 You can also specify all the params in your `knife.rb` not to type it in every time by putting a config hash in there:
 
@@ -38,15 +47,76 @@ You can also specify all the params in your `knife.rb` not to type it in every t
                        :remote_name => "your_remote_name",
                        :remote_url => "your_remote_url",
                        :branch => "your_branch",
-                       :commit_message => "your_commit_message"                                                                                                                                                        
-    }
-Have fun!
+                       :commit_message => "your_commit_message"
+                     }
+
+### Making Changes
+
+`adjust` action is your friend here.
+
+I takes any amount of filenames (but at least one) with the changes specified in a JSON object. Apart from `author_name`, `author_email` and `description` which aren't mandatory this object needs to contain a property called `adjustments` that will in turn be an array of JSON objects containing at least the following properties:
+
+*   `action` - The actual action to perform.
+*   `type` - What you are trying to update. It can be either `environment`, `role` or `node`
+*   `search` - the search query that will be used to figure out what to update. If a simple string is given (without a `:` character) scribe will assume it's a name and act accordingly 
+*   `adjustment` - the hash containing the actual changes
+
+The action to perform can be one of the following:
+
+*   `merge` - a deep merge that combines the chef object with the adjustment, adding new entries and updating values. Arrays will be combined.
+*   `hash_only_merge` - same as merge but arrays will be overwritten instead of combined (only in Chef version 11.0+).
+*   `overwrite` - a simple merge on the top level of the chef object. Usefull for overwriting run lists.
+*   `delete` - as the name suggests it can be used to delete parts of the config. The adjustment may be an integer or string in which case scribe will attempt to remove this key from the objec at the top level. It can be hash, which scribe treats as a map to the key that needs to be removed. Finally It can be an array which represents a set of changes that needs to be done on a single level. A quick example:
+
+    With a simple envrionment
 
-WHAT'S THE PLAN?
-----------------
+        { "chef_type": "environment"
+          "cookbook_versions": { "apache2": "<= 1.1.8",
+                                 "apt": "<= 1.4.9"
+                               },
+          "default_attributes": { "env" : "dev",
+                                  "ports" : [ 80, 8080 ],
+                                  "app" : { "storage_method" : "s3",
+                                            "storage_url" : "foo.bar"
+                                          }
+                                }
+        }
 
-- Turn Scribe into a gem
-- Mysterious Next Step ;-)
+    Applying the folowing `delete` adjustment
+
+        { "default_attributes" : [ "env",
+                                   { "app" : "storage_method" },
+                                   { "ports" : 0 }
+			                     ],
+		  "cookbook_versions" : "apt"
+	    }
+
+    Will remove default_attributes/env, default_attributes/app/storage_method, cookbook_versions/apt keys and first port from the default_attributes/ports array. The final product will be:
+
+        { "chef_type": "environment"
+          "cookbook_versions": { "apache2": "<= 1.1.8" }
+          "default_attributes": { "ports" : [ 8080 ],
+                                  "app" : { "storage_url" : "foo.bar" }
+          }
+        }
+
+You can use `adjust` with a `-g` or `--generate` option. It will then fill all the files specified with an adjustment template (overwriting any existing content of the files).
+
+An additional option `-t` or `--type` allows you to decide what adjustment template should be used (possible variants are `environment`, `role` and `node` - `environment` being the default)
+
+Runing `adjust` with a `--dryrun` option won't update any objects on the server, but will allow you to review the result of your adjustments in a diff format.
+
+Lastly you can use the `-d` or `--document` option which will surround your adjustments with a `scribe copy` action (initializing the repo with `scribe hire` just in case). It will use the same configuration form your `knife.rb` file that a standalone call to `hire` or `copy` would. You can also use the same command line prameters (`--chronicle-path`, `--remote-name`, `--remote-url` and `--branch`) to set everything up.
+
+**Important note:** When you're trying to apply a change to all your environemnts, don't use `*:*` as your search term. Chef will then try to apply those changes to the `_default` environemnt which is frozen and can't me modified. Try using `-name:_default` instead. 
+
+Have fun!
+
+WHAT'S NEXT
+-----------
+* Refactoring, refactoring, refactoring
+* Testing
+* Bug fixes
 
 LICENSE
 -------

data/lib/chef/knife/scribe_adjust.rb

@@ -0,0 +1,325 @@
+#
+# Author:: Pawel Kozlowski (<pawel.kozlowski@u2i.com>)
+# Copyright:: Copyright (c) 2013 Pawel Kozlowski
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+require 'chef/mixin/deep_merge'
+require 'chef/mixin/shell_out'
+
+class Chef
+  class Knife
+    class ScribeAdjust < Chef::Knife
+
+      include Chef::Mixin::DeepMerge
+      include Chef::Mixin::ShellOut
+
+      deps do
+        require_relative 'scribe_hire'
+        require_relative 'scribe_copy'
+      end
+
+      TEMPLATE_HASH = { "author_name" => "",
+        "author_email" => "",
+        "description" => "",
+        "adjustments" => []
+      }
+
+      ENVIRONMENT_ADJUSTMENT_TEMPLATE = {
+        "adjustment" => { "default_attributes" => { },
+          "override_attributes" => { },
+          "cookbook_versions" => { }
+        }
+      }
+
+      ROLE_ADJUSTMENT_TEMPLATE = {
+        "adjustment" => { "default_attributes" => { },
+          "override_attributes" => { },
+          "run_list" => [ ]
+        }
+      }
+
+      NODE_ADJUSTMENT_TEMPLATE = {
+        "adjustment" => { "attributes" => { },
+          "run_list" => [ ]
+        }
+      }
+
+      banner "knife scribe adjust FILE [FILE..]"
+
+      option :generate,
+      :short => "-g",
+      :long  => "--generate",
+      :description => "generate adjustment templates"
+
+      option :document,
+      :short => "-d",
+      :long  => "--document",
+      :description => "document with copy copy"
+
+      option :dryrun,
+      :long  => "--dryrun",
+      :description => "do a test run"
+
+
+      option :type,
+      :short => "-t TYPE",
+      :long  => "--type TYPE",
+      :description => "generate adjustment templates [environemnt|node|role]",
+      :default => "environment"
+
+
+      option :chronicle_path,
+      :short => "-p PATH",
+      :long => "--chronicle-path PATH",
+      :description => "Path to the directory where the chronicle should be located",
+      :default => nil
+
+      option :remote_name,
+      :long => "--remote-name REMOTE_NAME",
+      :description => "Name of the remote chronicle repository",
+      :default => nil
+
+      option :remote_url,
+      :short => "-r REMOTE_URL",
+      :long => "--remote-url REMOTE_URL",
+      :description => "Url of the remote chronicle repository",
+      :default => nil
+
+      option :branch,
+      :long => "--branch BRANCH_NAME",
+      :description => "Name of the branch you want to use",
+      :default => nil
+
+
+      alias_method :action_merge, :merge
+      alias_method :action_hash_only_merge, :hash_only_merge if respond_to?(:hash_only_merge)
+
+      def changes
+        @changes ||= {}
+      end
+
+      def descriptions
+        @descriptions ||= []
+      end
+
+      def errors
+        @errors ||= []
+      end
+
+
+      def run
+        if @name_args[0].nil?
+          show_usage
+          ui.fatal("At least one adjustment file needs to be specified!")
+          exit 1
+        end
+        if config[:generate] == true
+          @name_args.each { |filename| generate_template(filename) }
+        else
+          parse_adjustments
+        end
+      end
+
+      def generate_templates
+
+      end
+
+      def parse_adjustments
+        @name_args.each do |filename|
+          errors.push({ "name" => filename, "general" => nil, "adjustments" => {} })
+          parse_adjustment_file(filename)
+        end
+        if errors?
+          print_errors
+          exit 1 unless config[:dryrun]
+        end
+        if config[:dryrun]
+          diff
+        else
+          if config[:document] == true
+            hire
+            record_state
+          end
+          write_adjustments
+          record_state(descriptions.join("\n").strip) if config[:document] == true
+        end
+      end
+
+      def generate_template(filename)
+        unless ["environment", "role", "node"].include?(config[:type])
+          ui.fatal("Incorrect adjustment type! Only 'node', 'environment' or 'role' allowed.")
+          exit 1
+        end
+        TEMPLATE_HASH["adjustments"] = [self.class.class_eval(config[:type].upcase + "_ADJUSTMENT_TEMPLATE")]
+        File.open(filename, "w") { |file| file.write(JSON.pretty_generate(TEMPLATE_HASH)) }
+      end
+
+      def parse_adjustment_file(filename)
+        if !File.exists?(filename)
+          errors.last["general"] = "File does not exist!"
+        else
+          begin
+            adjustment_file = File.open(filename, "r") { |file| JSON.load(file) }
+            if adjustment_file_valid? adjustment_file
+              adjustment_file["adjustments"].each_with_index do |adjustment, index|
+                apply_adjustment(adjustment) if adjustment_valid?(adjustment, index)
+              end
+            end
+            if adjustment_file["adjustments"].length > errors.last["adjustments"].length
+              description = adjustment_file["description"]
+              description += "[with errors]" if errors.last["adjustments"].size > 0
+              descriptions.push(description)
+            end
+          rescue JSON::ParserError
+            errors.last["general"] = "Malformed JSON!"
+          end
+        end
+      end
+
+      def apply_adjustment(adjustment)
+        query = adjustment["search"].include?(":") ? adjustment["search"] : "name:" + adjustment["search"]
+        Chef::Search::Query.new.search(adjustment["type"], query ) do |result|
+          result_hash = result.to_hash
+          key = result_hash["chef_type"] + ":" + result_hash["name"]
+          if changes.has_key? key
+            result_hash = changes[key]["adjusted"]
+          else
+            changes.store(key, { "original" => result_hash })
+          end
+          changes[key].store("adjusted", send(("action_" + adjustment["action"]).to_sym, result_hash, adjustment["adjustment"]))
+        end
+      end
+
+      def write_adjustments
+        changes.values.each do |change|
+          Chef.const_get(change["adjusted"]["chef_type"].capitalize).json_create(change["adjusted"]).save
+        end
+      end
+
+      def adjustment_file_valid? adjustment_file
+        unless adjustment_file.kind_of?(Hash)
+          errors.last["general"] = "Adjustment file must contain a JSON hash!"
+          return false
+        end
+
+        unless adjustment_file["adjustments"].kind_of?(Array)
+          errors.last["general"] = "Adjustment file must contain an array of adjustments!"
+          return false
+        end
+        true
+      end
+
+      def adjustment_valid?(adjustment, index)
+        unless adjustment.kind_of?(Hash)
+          errors.last["adjustments"].store(index,"Adjustment must be a JSON hash!")
+          return false
+        end
+
+        ["action", "type", "search", "adjustment"].each do |required_key|
+          unless adjustment.has_key?(required_key)
+            errors.last["adjustments"].store(index, "Adjustment hash must contain " + required_key + "!")
+            return false
+          end
+        end
+
+        unless respond_to?("action_" + adjustment["action"])
+          errors.last["adjustments"].store(index, "Incorrect action!")
+          return false
+        end
+        true
+      end
+
+      def hire
+        hired_scribe = Chef::Knife::ScribeHire.new
+        [:chronicle_path, :remote_url, :remote_name].each { |key| hired_scribe.config[key] = config[key] }
+        hired_scribe.run
+      end
+
+      def record_state(message = nil)
+        if @copyist.nil?
+          @copyist = Chef::Knife::ScribeCopy.new
+          [:chronicle_path, :remote_name, :branch].each { |key| @copyist.config[key] = config[key] }
+        end
+        @copyist.config[:message] = message
+        @copyist.run
+      end
+
+      def errors?
+        errors.each { |err| return true if !err["general"].nil? || (err["adjustments"].size > 0) }
+        false
+      end
+
+      def print_errors
+        ui.error("ERRORS OCCURED:")
+        errors.each do |err|
+          ui.error(err["name"]) if !err["general"].nil? || (err["adjustments"].size > 0)
+          ui.error("\t" + err["general"]) if !err["general"].nil?
+          err["adjustments"].each { |num, adj_err| ui.error("\t[Adjustment #{num}]: #{adj_err}") }
+        end
+      end
+
+      def action_overwrite(base, overwrite_with)
+        if base.kind_of?(Hash) && overwrite_with.kind_of?(Hash)
+          base.merge(overwrite_with)
+        elsif overwrite_with.nil?
+          base
+        else
+          overwrite_with
+        end
+      end
+    end
+
+    def deep_delete(delete_from, delete_spec)
+      deep_delete!(delete_from.dup, delete_spec.dup)
+    end
+
+    alias_method :action_delete, :deep_delete
+
+    def deep_delete!(delete_from, delete_spec)
+      if delete_from.kind_of?(Hash) || delete_from.kind_of?(Array)
+        if delete_spec.kind_of?(Array)
+          delete_spec.each { |item| deep_delete!(delete_from, item) }
+        elsif delete_spec.kind_of?(Hash)
+          delete_spec.each { |key,item| deep_delete!(delete_from[key], item) }
+        else
+          delete_from.kind_of?(Array) ? delete_from.delete_at(delete_spec) : delete_from.delete(delete_spec)
+        end
+      end
+      delete_from
+    end
+
+    def diff
+      original_file = Tempfile.new("original")
+      adjusted_file = Tempfile.new("adjusted")
+      begin
+        changes.each do |key, change|
+          ui.info("[#{key}]")
+          original_file.write(JSON.pretty_generate(change["original"]))
+          adjusted_file.write(JSON.pretty_generate(change["adjusted"]))
+          original_file.rewind
+          adjusted_file.rewind
+          diff_output = shell_out("diff -L original -L adjusted -u #{original_file.path} #{adjusted_file.path}")
+          ui.info(diff_output.stdout)
+        end
+      ensure
+        original_file.close
+        original_file.unlink
+        adjusted_file.close
+        adjusted_file.unlink
+      end
+    end
+  end
+end