cheftacular 2.11.1 → 2.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bbe129ab859d02919ea7cbd2c10452743690a34c
-  data.tar.gz: 43b46e686688fb8e59903bcd9c1767e3d41dfe80
+  metadata.gz: e2f0c0d7f232e7617f6c0f511806a71d37f7b7fa
+  data.tar.gz: 43aeba1d59eec85f9c5fc7d5f20b168cb8283cf9
 SHA512:
-  metadata.gz: e6233b4d64ea85195bef515458a241440dd97d29a2326bfa3c29f32de05d38889beda87636a84cfea652efafd2937f2165264414f483e430aefad0512993f650
-  data.tar.gz: 69193f44961e51a9aa16a6a5a7c0eafb7494517212be1aa6034089000e85bd5e5383c0246be79adb9cefec4272239a26021cde96a8e854bae03a76ca5bbb1364
+  metadata.gz: 2b2567debc1835e4079e46486301ce31ee675efcc24cb1e89c739178d29a97f33fb237c28800733a6466fd5b8b55dbfb1d1cc821b456e0ef547a802ae59b295e
+  data.tar.gz: f8a09335967bd9bc172e77a00a710708bde5cc1c24a34c99bbcde186aea7d89865b96f796ceeff463af8af8e882207c362702e506bceb8c9075d8dce88a407de

data/lib/cheftacular/README.md

@@ -41,7 +41,73 @@
 
 7.  `-R|--repository NAME` will make the command run against a specific repository or context (automatically set for application mode)
 
-8.  `-v|--verbose` toggles on verbose logging. All commands that write logs will also output to terminal AND write the logs.
+8.  `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.
+
+9.  `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.
+
+10. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment.
+
+11.  `-v|--verbose` toggles on verbose logging. All commands that write logs will also output to terminal AND write the logs.
+
+### Help Related
+
+1. `-h|--help` Displays the full readme and exits.
+
+### Action Flags
+
+1.  `-e|--except-role ROLE_NAME` will *prevent* any server with this role from being *deployed to* for the deploy command. Other commands will ignore this argument.
+
+2.  `-z|--unset-github-deploy-args` will unset a custom revision specified in the arg below and make the codebase utilize the default.
+
+3.  `-Z|--revision REVISION` will force the role you're deploying to to utilize the revision specified here. This can be a specific commit, a branch name or even a tag.
+
+    1. Note: The system does not check if the revision exists, if you pass a non-existent revision no one will be able to deploy to that role until -Z with a correction revision or -z is passed.
+
+4.  The `-O ORGANIZATION` flag can be used with TheCheftacularCookbook to set an *organization* your app can try deploying from, your git user needs access to these forks / organization(s).
+
+    3.  The `-z|--unset-github-deploy-args` option will clear your current `-Z` and `-O` flags.
+
+## Arguments and flags for cheftacular
+
+### Environment flags
+
+1.  `-d|--dev-remote` toggles on dev-remote mode. Commands passed to cft will hit the devremote server(s) instead of the default server(s)
+
+2.  `--env ENV` sets the environment commands hit to one you specify instead of the default one.
+
+3.  `-p|--prod` toggles on production mode. Commands passed to cft will hit the production server(s) instead of the default server(s)
+
+4.  `-Q|--qa` toggles on QA mode. Commands passed to cft will hit the QA server(s) instead of the default server(s)
+
+5.  `-s|--staging` toggles on staging mode. Commands passed to cft will hit the staging server(s) instead of the default server(s)
+
+6.  `--split-env SPLIT_ENV_NAME` sets the sub-environment to SPLIT_ENV_NAME. This only slightly affects certain commands.
+
+7.  `-t|--test` toggles on test mode. Commands passed to cft will hit the test server(s) instead of the default server(s)
+
+### General Flags
+
+1.  `-a|--address ADDRESS` will force the command to only run against the specified address if it belongs to a node
+
+2.  `-D|--debug` toggles on extremely verbose logging. Chef-client runs will generate ~10 times the amounts of logs including any additional effects that the `-v` flag will activate
+
+3. `--no-logs` will make the cft commands not generate log files, you must still specify `-v` if you want output of most verbose commands to your terminal.
+
+4.  `-n|--node-name NODE_NAME` will force the command to only run against the specified name if it belongs to a node
+
+5.  `-q|--quiet` will make the cft commands only output information that is a direct result of the command being run
+
+6.  `-r|--role-name ROLE_NAME` will force the command to only run against the specified role if it exists (this argument is generally not needed though it can be used to deploy a codebase for an application you're not currently cd'd into when running this as a gem)
+
+7.  `-R|--repository NAME` will make the command run against a specific repository or context (automatically set for application mode)
+
+8.  `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.
+
+9.  `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.
+
+10. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment.
+
+11.  `-v|--verbose` toggles on verbose logging. All commands that write logs will also output to terminal AND write the logs.
 
 ### Help Related
 
@@ -64,15 +130,17 @@
 
 ## Commands that can be run in the application context
 
-1. `cft backups [activate|deactivate|load|restore]` this command sets the fetch_backups and restore_backups flags in your config data bag for an environment. These can be used to give application developers a way to trigger / untrigger restores in an environment
+1. `cft backups [activate|deactivate|fetch|load|restore]` this command sets the fetch_backups and restore_backups flags in your config data bag for an environment. These can be used to give application developers a way to trigger / untrigger restores in an environment
 
     1. `activate` will turn on automated backup running (turns on the flag for the env in the config bag).
 
     2. `deactivate` will turn off automated backup running.
 
-    3. `load` will fetch the latest backup from the production primary **if it doesn't already exist on the server** and run the _backup loading command_ to load this backup into the env.
+    3. `fetch` will fetch the latest backup and drop it onto your machine. This argument accepts the --save-to-file LOCATION flag.
+
+    4. `load` will fetch the latest backup from the production primary **if it doesn't already exist on the server** and run the _backup loading command_ to load this backup into the env.
 
-    4. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server.
+    5. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server.
 
 2. `cft check [all]` Checks the commits for all servers for a repository (for an environment) and returns them in a simple chart. Also shows when these commits were deployed to the server.
 
@@ -116,6 +184,12 @@
 
     2. `-W|--with-priv` option will make this command display the server's local (private) ip address. This address is also the server's `local.<SERVER_DNS_NAME>`.
 
+    3. `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.
+
+    4. `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.
+
+    5. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment.
+
 8. `cft console` will create a console session on the first node found for a repository.
 
     1. Attempts to setup a console for the unique stack, stacks currently supported for console is only Rails.
@@ -269,15 +343,17 @@
 
     1. `SPECIFIC_REPOSITORY` is a special argument, if left blank the key will be placed in the authorized_keys array in the bag, otherwise it will be placed in the specific_authorized_keys hash under a key named for the repository that is passed. The script will error if SPECIFIC_REPOSITORY does not exist in the cheftacular.yml respositories hash. You can then use this data to give users selective ssh access to certain servers.
 
-4. `cft backups [activate|deactivate|load|restore]` this command sets the fetch_backups and restore_backups flags in your config data bag for an environment. These can be used to give application developers a way to trigger / untrigger restores in an environment
+4. `cft backups [activate|deactivate|fetch|load|restore]` this command sets the fetch_backups and restore_backups flags in your config data bag for an environment. These can be used to give application developers a way to trigger / untrigger restores in an environment
 
     1. `activate` will turn on automated backup running (turns on the flag for the env in the config bag).
 
     2. `deactivate` will turn off automated backup running.
 
-    3. `load` will fetch the latest backup from the production primary **if it doesn't already exist on the server** and run the _backup loading command_ to load this backup into the env.
+    3. `fetch` will fetch the latest backup and drop it onto your machine. This argument accepts the --save-to-file LOCATION flag.
+
+    4. `load` will fetch the latest backup from the production primary **if it doesn't already exist on the server** and run the _backup loading command_ to load this backup into the env.
 
-    4. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server.
+    5. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server.
 
 5. `cft check_cheftacular_yml_keys` allows you to check to see if your cheftacular yml keys are valid to the current version of cheftacular. It will also set your missing keys to their likely default and let you know to update the cheftacular.yml file.
 
@@ -327,6 +403,12 @@
 
     2. `-W|--with-priv` option will make this command display the server's local (private) ip address. This address is also the server's `local.<SERVER_DNS_NAME>`.
 
+    3. `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.
+
+    4. `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.
+
+    5. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment.
+
 13. `cft cloud <FIRST_LEVEL_ARG> [<SECOND_LEVEL_ARG>[:<SECOND_LEVEL_ARG_QUERY>]*] ` this command handles talking to various cloud APIs. If no args are passed nothing will happen.
 
     1. `domain` first level argument for interacting with cloud domains
@@ -623,7 +705,11 @@
 
     3. The argument `skip_update_tld` will stop the long process of checking and updating all the server domains _before_ cloudflare is updated. Only skip if you believe your domain info on your cloud is accurate.
 
-49. `cft update_split_branches` will perform a series of git commands that will merge all the split branches for your split_branch enabled repositories with what is currently on master and push them.
+49. `cft update_cookbook [COOKBOOK_NAME]` allows you to specifically update a single cookbook
+
+    1. This command passed with no arguments will update TheCheftacularCookbook
+
+50. `cft update_split_branches` will perform a series of git commands that will merge all the split branches for your split_branch enabled repositories with what is currently on master and push them.
 
     1. Repository must be set with `-R REPOSITORY_NAME` for this command to work.
 
@@ -633,8 +719,6 @@
 
     4. This command will return a helpful error statement if you attempt to run the command with changes to your current working directory. You must commit these changes before running this command.
 
-50. `cft update_thecheftacularcookbook` allows you to update ONLY the internal chef-repo's TheCheftacularCookbook.
-
 51. `cft update_tld TLD` command will force a full dns update for a tld in the preferred cloud. It will ensure all the subdomain entries are correct (based on the contents of the addresses data bag) and update them if they are not. It will also create the local subdomain for the entry as well if it does exist and point it to the correct private address.
 
 52. `cft upload_nodes` This command will resync the chef server's nodes with the data in our chef-repo/node_roles. 

data/lib/cheftacular/actions/migrate.rb

@@ -36,7 +36,7 @@ class Cheftacular
 
         puts("Beginning migration run for #{ n.name } (#{ n.public_ipaddress }) on role #{ options['role'] }") unless options['quiet']
 
-        log_data, timestamp, exit_status = start_task( n.name, n.public_ipaddress, n.run_list, "#{ bundle_command } exec rake db:migrate", options, locs, cheftacular)
+        log_data, timestamp, exit_status = start_task( n.name, n.public_ipaddress, n.run_list, "#{ bundle_command } exec rake db:migrate", options, locs, cheftacular, passwords)
 
         logs_bag_hash["#{ n.name }-#{ __method__ }"] = { "text" => log_data.scrub_pretty_text, "timestamp" => timestamp, "exit_status" => exit_status }
       end

data/lib/cheftacular/actions/run.rb

@@ -46,7 +46,7 @@ class Cheftacular
 
         puts("Beginning task run for #{ n.name } (#{ n.public_ipaddress }) on role #{ options['role'] }") unless options['quiet']
 
-        log_data, timestamp, exit_status = start_task( n.name, n.public_ipaddress, n.run_list, "#{ bundle_command } exec #{ command }", options, locs, cheftacular)
+        log_data, timestamp, exit_status = start_task( n.name, n.public_ipaddress, n.run_list, "#{ bundle_command } exec #{ command }", options, locs, cheftacular, passwords)
 
         logs_bag_hash["#{ n.name }-#{ __method__ }"] = { "text" => log_data.scrub_pretty_text, "timestamp" => timestamp, "exit_status" => exit_status }
       end

data/lib/cheftacular/cheftacular.rb

@@ -30,6 +30,7 @@ class Cheftacular
   def initialize options={'env'=>'staging'}, config={}
     @options, @config                = options, config
     SSHKit.config.format             = :blackhole
+    Excon.defaults[:ssl_verify_peer] = false #https://github.com/excon/excon/issues/479 # issues with cert bundle chains
     #Fog::Logger[:warning]            = nil
     @config['start_time']            = Time.now
     @config['helper']                = Helper.new(@options, @config)

data/lib/cheftacular/file_system.rb

@@ -137,7 +137,7 @@ class Cheftacular
       File.open( File.join(@config['locs']['chef-repo'], "config", to_be_created_filename), "w") { |f| f.write(File.read(File.join(@config['locs']['examples'], example_filename))) }
     end
 
-    def parse_latest_berkshelf_cookbook_versions berkshelf_cookbooks={}
+    def parse_berkshelf_cookbook_versions version='latest', berkshelf_cookbooks={}
 
       Dir.foreach(@config['locs']['berks']) do |berkshelf_cookbook|
         next if is_junk_filename?(berkshelf_cookbook)
@@ -148,7 +148,12 @@ class Cheftacular
 
         #get only the latest version, berkshelf pulls in multiple commits from git repos for SOME REASON
         use_current_cookbook = !berkshelf_cookbooks.has_key?(true_cookbook_name)
-        use_current_cookbook = @config['helper'].is_higher_version?(cookbook_version, berkshelf_cookbooks[true_cookbook_name]['version']) if berkshelf_cookbooks.has_key?(true_cookbook_name)
+
+        if version == 'latest'
+          use_current_cookbook = @config['helper'].is_higher_version?(cookbook_version, berkshelf_cookbooks[true_cookbook_name]['version']) if berkshelf_cookbooks.has_key?(true_cookbook_name)
+        else
+          use_current_cookbook = true if berkshelf_cookbooks.has_key?(true_cookbook_name) && cookbook_version == version
+        end
 
         if use_current_cookbook
           berkshelf_cookbooks[true_cookbook_name] ||= {}
@@ -191,7 +196,7 @@ class Cheftacular
 
     def scrub_from_known_hosts target
       puts "Clearing #{ target } from known_hosts file."
-      case CONFIG['host_os']
+      case RbConfig::CONFIG['host_os']
       when /mswin|windows/i
         puts "#{ __method__ } does not support this operating system at this time"
       when /linux|arch/i

data/lib/cheftacular/helper.rb

@@ -248,7 +248,11 @@ class Cheftacular
           if hash['exit_status'] && hash['exit_status'] == 1
             @config['slack_queue'] << { message: hash['text'].prepend('```').insert(-1, '```') }
 
-            @config['error'].exception_output(on_failing_exit_status_message) if !on_failing_exit_status_message.blank?
+            if !on_failing_exit_status_message.blank?
+              @config['queue_master'].work_off_slack_queue
+
+              @config['error'].exception_output(on_failing_exit_status_message)
+            end
           end
         end
       end
@@ -316,6 +320,41 @@ class Cheftacular
         exit
       end
     end
+
+    def display_cheftacular_config_diff
+      diff_hash = @config['initial_cheftacular_yml'].deep_diff(@config['default']['cheftacular_bag_hash'], true).except('mode', 'default_repository').compact
+
+      recursive_hash_scrub(diff_hash)
+      recursive_hash_scrub(diff_hash) #scrub out any leftover empty hashes
+
+      if diff_hash.empty?
+        puts "No difference detected between your cheftacular.yml and the global environment."
+      else
+        puts "Difference detected between local cheftacular.yml and data bag cheftacular.yml! Displaying...\n\n"
+
+        ap diff_hash
+          
+        if @config['helper'].running_in_mode?('application') && @config['default']['cheftacular_bag_hash']['slack']['webhook'] && !diff_hash.empty?
+          @config['slack_queue'] << { message: diff_hash.awesome_inspect({plain: true, indent: 2}).prepend('```').insert(-1, '```'), channel: @config['cheftacular']['slack']['notify_on_yaml_sync'] }
+        end
+
+        puts("If these are your intended changes you want to sync into the environment, you should run `cft cheftacular_config sync`\n\n") if ARGV[0] != 'cheftacular_config' && ARGV[1] != 'sync'
+      end
+    end
+
+    def recursive_hash_scrub hash
+      hash.each_pair do |key, value|
+        if value.nil?
+          hash.delete(key)
+        elsif value.class == Hash && value.empty?
+          hash.delete(key)
+        elsif value.class == Hash && value[value.keys.first].empty?
+          hash.delete(key)
+        elsif value.class == Hash
+          recursive_hash_scrub(hash[key])
+        end
+      end
+    end
   end
 end
 

data/lib/cheftacular/initializer.rb

@@ -12,6 +12,8 @@ class Cheftacular
 
       initialize_locations
 
+      initialize_directories
+
       initialize_data_bag_cheftacular_hash if !@config['helper'].is_initialization_command?(ARGV[0]) && !@config['helper'].running_on_chef_node?
 
       initialize_monkeypatches unless @config['helper'].running_on_chef_node?
@@ -30,8 +32,6 @@ class Cheftacular
 
       initialize_classes
 
-      initialize_directories
-
       unless @config['helper'].is_initialization_command?(ARGV[0])
         initialize_cloud_checks
 
@@ -111,6 +111,10 @@ class Cheftacular
           @options['verbose'] = true
         end
 
+        opts.on('-E', '--search-env-name ENV_NAME', 'For commands that support searching, return results with this environment in them') do |name|
+          @options['search_env_name'] = name
+        end
+
         opts.on('-n', '--node-name NAME', "Run your command against this node_name") do |name|
           @options['node_name'] = name
         end
@@ -127,6 +131,14 @@ class Cheftacular
           @options['repository'] = name
         end
 
+        opts.on('-s', '--search-node-name NODE_NAME', 'For commands that support searching, return results with NODE_NAME in them') do |name|
+          @options['search_node_name'] = name
+        end
+
+        opts.on('-S', '--search-role-name ROLE_NAME', 'For commands that support searching, return results with ROLE_NAME in them') do |name|
+          @options['search_role_name'] = name
+        end
+
         opts.on('-v', '--verbose', "Activates slightly more verbose logging, also causes commands to output to terminal and logs") do
           @options['verbose'] = true
         end
@@ -261,6 +273,16 @@ class Cheftacular
       @config['initial_cheftacular_yml'] = @config['cheftacular'].deep_dup
 
       @config['cheftacular'] = @config['default']['cheftacular_bag_hash'].deep_merge(@config['cheftacular'])
+
+      parsed_cheftacular = Digest::SHA2.hexdigest(@config['helper'].get_cheftacular_yml_as_hash.to_s)
+
+      return true if File.exist?(@config['filesystem'].local_cheftacular_file_cache_path) && File.read(@config['filesystem'].local_cheftacular_file_cache_path) == parsed_cheftacular
+
+      @config['helper'].display_cheftacular_config_diff
+
+      puts "Creating file cache for #{ Time.now.strftime("%Y%m%d") }'s local cheftacular.yml."
+
+      @config['filesystem'].write_local_cheftacular_cache_file parsed_cheftacular
     end
 
     def initialize_default_cheftacular_options

data/lib/cheftacular/queue_master.rb

@@ -22,6 +22,10 @@ class Cheftacular
         found_result = true
       end
 
+      server_hash.each_pair do |key, val|
+        server_hash[key] = '' if val.nil?
+      end
+
       @config[server_queue] << server_hash unless found_result
 
       server_hash

data/lib/cheftacular/stateless_actions/arguments.rb

@@ -36,7 +36,13 @@ class Cheftacular
 
         '7.  `-R|--repository NAME` will make the command run against a specific repository or context (automatically set for application mode)',
 
-        '8.  `-v|--verbose` toggles on verbose logging. All commands that write logs will also output to terminal AND write the logs.',
+        '8.  `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.',
+
+        '9.  `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.',
+
+        '10. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment.',
+
+        '11.  `-v|--verbose` toggles on verbose logging. All commands that write logs will also output to terminal AND write the logs.',
 
         '### Help Related',
 

data/lib/cheftacular/stateless_actions/backups.rb

@@ -4,7 +4,7 @@ class Cheftacular
     def backups
       @config['documentation']['stateless_action'][__method__] ||= {}
       @config['documentation']['stateless_action'][__method__]['long_description'] = [
-        "`cft backups [activate|deactivate|load|restore]` this command " +
+        "`cft backups [activate|deactivate|fetch|load|restore]` this command " +
         "sets the fetch_backups and restore_backups flags in your config data bag for an environment. " +
         "These can be used to give application developers a way to trigger / untrigger restores in an environment",
       
@@ -13,10 +13,12 @@ class Cheftacular
 
           "    2. `deactivate` will turn off automated backup running.",
 
-          "    3. `load` will fetch the latest backup from the production primary **if it doesn't already exist on " +
+          "    3. `fetch` will fetch the latest backup and drop it onto your machine. This argument accepts the --save-to-file LOCATION flag.",
+
+          "    4. `load` will fetch the latest backup from the production primary **if it doesn't already exist on " +
           "the server** and run the _backup loading command_ to load this backup into the env.",
 
-          "    4. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server."
+          "    5. `restore` will simply just run the _backup loading command_ to load the latest backup onto the server."
         ]
       ]
 
@@ -30,7 +32,7 @@ class Cheftacular
     def backups command=''
       command = ARGV[1] if command.blank?
 
-      raise "Unsupported command (#{ command }) for cft backups" unless command =~ /activate|deactivate|load|restore/
+      raise "Unsupported command (#{ command }) for cft backups" unless command =~ /activate|deactivate|fetch|load|restore/
 
       self.send("backups_#{ command }")
     end
@@ -46,44 +48,12 @@ class Cheftacular
     end
 
     def backups_load status_hash={}
-      old_role, old_env, backup_env = @options['role'], @options['env'], @config['cheftacular']['backup_config']['global_backup_environ']
-
-      if backup_env != @options['env']
-        @config['initializer'].initialize_data_bags_for_environment(backup_env, false, ['addresses', 'server_passwords', 'logs'])
-        @config['initializer'].initialize_passwords backup_env
-      end
-
-      @options['role'] = @config['cheftacular']['backup_config']['global_backup_role_name']
-      @options['env']  = backup_env
-
-      puts "Deploying to backup master to force refresh of the ssh keys..." 
-      @config['action'].deploy
+      target_db_primary, nodes, status_hash, backup_master_local_ip = backups_get_status_hash_from_backupmaster
 
-      @options['role'] = old_role
-      @options['env']  = old_env
-
-      target_db_primary, nodes = @config['getter'].get_db_primary_node_and_nodes
-
-      args_config = [
-        { unless: "role[#{ @config['cheftacular']['backup_config']['global_backup_role_name'] }]" },
-        { if: { not_env: backup_env } }
-      ]
-
-      backup_master          = @config['parser'].exclude_nodes( nodes, args_config, true)
-      backup_master_local_ip = @config['getter'].get_address_hash(backup_master.first.name, true)[backup_master.first.name]['priv']
+      return false unless status_hash['latest_backup']['file_check']
 
       options, locs, ridley, logs_bag_hash, pass_bag_hash, bundle_command, cheftacular, passwords = @config['helper'].set_local_instance_vars
 
-      on ( backup_master.map { |n| @config['cheftacular']['deploy_user'] + "@" + n.public_ipaddress } ) do |host|
-        n = get_node_from_address(nodes, host.hostname)
-
-        puts("Beginning latest db_fetch_and_check for #{ n.name } (#{ n.public_ipaddress }) for env #{ backup_env }") unless options['quiet']
-
-        status_hash['latest_backup'] = start_db_check_and_fetch( n.name, n.public_ipaddress, options, locs, cheftacular, passwords)
-      end
-
-      return false unless status_hash['latest_backup']['file_check']
-
       on ( target_db_primary.map { |n| @config['cheftacular']['deploy_user'] + "@" + n.public_ipaddress } ) do |host|
         n = get_node_from_address(nodes, host.hostname)
 
@@ -113,6 +83,17 @@ class Cheftacular
       end
     end
 
+    def backups_fetch
+      target_db_primary, nodes, status_hash, backup_master_local_ip = backups_get_status_hash_from_backupmaster(__method__.to_s)
+
+      full_backup_dir  = File.join(@config['cheftacular']['backup_config']['db_primary_backup_path'], status_hash['latest_backup']['file_dir'])
+      full_backup_path = File.join(full_backup_dir, status_hash['latest_backup']['filename'])
+
+      file_scp_execute(target_db_primary, 'scp', full_backup_dir, status_hash['latest_backup']['filename'])
+    end
+
+    private
+
     def backups_toggle_setting restore_backup, fetch_backup
       initial_fetch_backup   = @config[@options['env']]['config_bag_hash'][@options['sub_env']]['fetch_backups']
       initial_restore_backup = @config[@options['env']]['config_bag_hash'][@options['sub_env']]['restore_backups']
@@ -134,6 +115,52 @@ class Cheftacular
 
       @config['action'].deploy
     end
+
+    def backups_get_status_hash_from_backupmaster mode='backups_load', status_hash={}
+      backup_env = @config['cheftacular']['backup_config']['global_backup_environ']
+
+      if backup_env != @options['env']
+        @config['initializer'].initialize_data_bags_for_environment(backup_env, false, ['addresses', 'server_passwords', 'logs'])
+        @config['initializer'].initialize_passwords backup_env
+      end
+
+      if mode == 'backups_load'
+        old_role, old_env = @options['role'], @options['env']
+
+        @options['role'] = @config['cheftacular']['backup_config']['global_backup_role_name']
+        @options['env']  = backup_env
+
+        puts "Deploying to backup master to force refresh of the ssh keys..." 
+        @config['action'].deploy
+
+        @options['role'] = old_role
+        @options['env']  = old_env
+      end
+
+      target_db_primary, nodes = @config['getter'].get_db_primary_node_and_nodes
+
+      args_config = [
+        { unless: "role[#{ @config['cheftacular']['backup_config']['global_backup_role_name'] }]" },
+        { if: { not_env: backup_env } }
+      ]
+
+      backup_master          = @config['parser'].exclude_nodes( nodes, args_config, true)
+      backup_master_local_ip = @config['getter'].get_address_hash(backup_master.first.name, true)[backup_master.first.name]['priv']
+
+      options, locs, ridley, logs_bag_hash, pass_bag_hash, bundle_command, cheftacular, passwords = @config['helper'].set_local_instance_vars
+
+      on ( backup_master.map { |n| @config['cheftacular']['deploy_user'] + "@" + n.public_ipaddress } ) do |host|
+        n = get_node_from_address(nodes, host.hostname)
+
+        puts("Beginning latest db_fetch_and_check for #{ n.name } (#{ n.public_ipaddress }) for env #{ backup_env }") unless options['quiet']
+
+        status_hash['latest_backup'] = start_db_check_and_fetch( n.name, n.public_ipaddress, options, locs, cheftacular, passwords)
+      end
+
+      return [nil, nil, {}] unless status_hash['latest_backup']['file_check']
+
+      [target_db_primary, nodes, status_hash, backup_master_local_ip]
+    end
   end
 end
 

data/lib/cheftacular/stateless_actions/bootstrappers/ubuntu_bootstrap_from_queue.rb

@@ -16,10 +16,6 @@ class Cheftacular
     def ubuntu_bootstrap_from_queue threads=[], execution_hash_array=[]
       raise "This action is not meant to be called directly!" if !@options['in_scaling'] && !@options['in_single_server_creation']
 
-      if `which sshpass`.empty?
-        raise "sshpass not installed! Please run brew install https://raw.github.com/eugeneoden/homebrew/eca9de1/Library/Formula/sshpass.rb (or get it from your repo for linux)"
-      end
-
       @config['bootstrap_timestamp'] ||= Time.now.strftime("%Y%m%d%H%M%S")
 
       @config['queue_master'].generate_passwords_for_each_server_hash_in_queue
@@ -31,7 +27,7 @@ class Cheftacular
       execution_hash_array  = execution_hash_array.flatten(1)
 
       @config['server_creation_queue'].each do |server_hash|
-        puts("#{ server_name_output(server_hash) }_Starting initial setup for server...") if @options['in_scaling']
+        puts("#{ server_name_output(server_hash) }_Starting initial setup for server...")
 
         threads << Thread.new { execute_execution_hash_array(server_hash, execution_hash_array) }
       end
@@ -83,7 +79,7 @@ class Cheftacular
       deploy_commands = [
         "#{ sudo } apt-get update",
         "#{ sudo } apt-get install curl #{ @config['cheftacular']['pre_install_packages'] } -y",
-        "#{ sudo } apt-get upgrade -y"
+        "#{ sudo } DEBIAN_FRONTEND=noninteractive apt-get upgrade -y -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confold\""
       ]
 
       if @config['cheftacular']['install_rvm_on_boot']

data/lib/cheftacular/stateless_actions/chef_bootstrap_from_queue.rb

@@ -27,7 +27,7 @@ class Cheftacular
       @config['bootstrap_timestamp'] ||= Time.now.strftime("%Y%m%d%H%M%S")
 
       @config['server_creation_queue'].each do |server_hash|
-        puts("(#{ server_hash['node_name'] })[#{ server_hash['address'] }] Starting chef-client installation...") unless @options['quiet']
+        puts("#{ server_name_output(server_hash) } Starting chef-client installation...") unless @options['quiet']
 
         threads << Thread.new { execute_execution_hash_array(server_hash, execution_hash_array) }
       end

data/lib/cheftacular/stateless_actions/cheftacular_config.rb

@@ -39,35 +39,13 @@ class Cheftacular
     end
 
     def cheftacular_config_diff
-      diff_hash = @config['initial_cheftacular_yml'].deep_diff(@config['default']['cheftacular_bag_hash'], true).except('mode', 'default_repository').compact
-
-      diff_hash.each_pair do |key, value|
-        diff_hash.delete(key) if value.empty? || value.nil?
-      end
-
-      if @config['helper'].running_in_mode?('devops') && !diff_hash.empty?
-        puts "Difference detected between local cheftacular.yml and data bag cheftacular.yml! Displaying..."
-
-        ap diff_hash
-      elsif @config['helper'].running_in_mode?('application') && @config['default']['cheftacular_bag_hash']['slack']['webhook'] && !diff_hash.empty?
-        @config['slack_queue'] << diff_hash.awesome_inspect({plain: true, indent: 2}).prepend('```').insert(-1, '```')
-      end
+      @config['helper'].display_cheftacular_config_diff
     end
 
     def cheftacular_config_sync
-      cheftacular_config_diff
-
-      parsed_cheftacular = Digest::SHA2.hexdigest(@config['helper'].get_cheftacular_yml_as_hash.to_s)
-
-      return true if File.exist?(@config['filesystem'].local_cheftacular_file_cache_path) && File.read(@config['filesystem'].local_cheftacular_file_cache_path) == parsed_cheftacular
-
       @config['default']['cheftacular_bag_hash'] = @config['cheftacular'].deep_dup.except('default_repository', 'mode') #the values have already been merged
 
       @config['ChefDataBag'].save_cheftacular_bag
-
-      puts "Creating file cache for #{ Time.now.strftime("%Y%m%d") }'s local cheftacular.yml."
-
-      @config['filesystem'].write_local_cheftacular_cache_file parsed_cheftacular
     end
   end
 end

data/lib/cheftacular/stateless_actions/clean_cookbooks.rb

@@ -38,17 +38,15 @@ class Cheftacular
         out = `berks install`
         puts "#{out}\nFinished... Beginning directory scanning and conflict resolution..."
 
-        berkshelf_cookbooks = @config['filesystem'].parse_latest_berkshelf_cookbook_versions
+        berkshelf_cookbooks = @config['filesystem'].parse_berkshelf_cookbook_versions
 
         chef_repo_cookbooks = @config['filesystem'].parse_chef_repo_cookbook_versions
 
-        berkshelf_cookbooks.each_pair do |berkshelf_cookbook, version|
-          new_name = berkshelf_cookbook.rpartition('-').first
-
-          if chef_repo_cookbooks.has_key?(new_name) || chef_repo_cookbooks.has_key?(berkshelf_cookbook) #don't overwrite cookbooks without user input
+        berkshelf_cookbooks.each_pair do |berkshelf_cookbook, cookbook_hash|
+          if chef_repo_cookbooks.has_key?(berkshelf_cookbook) || chef_repo_cookbooks.has_key?(cookbook_hash['location']) #don't overwrite cookbooks without user input
             if local_options['interactive']
-              puts "COOKBOOK::~~~~#{ new_name }~~~~::VERSION::~~~~~~~~#{ version } VS #{ chef_repo_cookbooks[new_name] }"
-              puts "\nEnter O | o | overwrite to overwrite ~~~~#{ new_name }~~~~ in the chef-repo (THIS SHOULD NOT BE DONE LIGHTLY)"
+              puts "COOKBOOK::~~~~#{ berkshelf_cookbook }~~~~::VERSION::~~~~~~~~#{ cookbook_hash['version'] } VS #{ chef_repo_cookbooks[berkshelf_cookbook] }"
+              puts "\nEnter O | o | overwrite to overwrite ~~~~#{ berkshelf_cookbook }~~~~ in the chef-repo (THIS SHOULD NOT BE DONE LIGHTLY)"
               puts "Enter N | n | no        to skip to the next conflict"
               puts "If you pass force to this script, it will always overwrite."
               #puts "If you pass a STRING of comma delimited cookbooks, it will skip these cookbooks automatically and overwrite others"
@@ -59,12 +57,12 @@ class Cheftacular
               next if (input =~ /N|n|no/) == 0
             end
 
-            next if @config['helper'].is_higher_version?(chef_repo_cookbooks[new_name], version)
+            next if @config['helper'].is_higher_version?(chef_repo_cookbooks[berkshelf_cookbook], cookbook_hash['version'])
           end
 
-          cmnd = "#{ @config['locs']['berks'] }/#{ berkshelf_cookbook } #{ @config['locs']['cookbooks'] }/#{ new_name }"
-          puts "Moving #{ cmnd } (#{ version }:#{ chef_repo_cookbooks[new_name] })" if @options['verbose']
-          `rm -Rf #{ @config['locs']['cookbooks'] }/#{ new_name }`
+          cmnd = "#{ @config['locs']['berks'] }/#{ cookbook_hash['location'] } #{ @config['locs']['cookbooks'] }/#{ berkshelf_cookbook }"
+          puts "Moving #{ cmnd } (#{ cookbook_hash['version'] }:#{ chef_repo_cookbooks[berkshelf_cookbook] })" if @options['verbose']
+          `rm -Rf #{ @config['locs']['cookbooks'] }/#{ berkshelf_cookbook }`
           `cp -Rf #{ cmnd }`
         end
       end

data/lib/cheftacular/stateless_actions/client_list.rb

@@ -12,7 +12,13 @@ class Cheftacular
           "whether its password is stored on the chef server and what that password is.",
 
           "    2. `-W|--with-priv` option will make this command display the server's local (private) ip address. " + 
-          "This address is also the server's `local.<SERVER_DNS_NAME>`."
+          "This address is also the server's `local.<SERVER_DNS_NAME>`.",
+
+          "    3. `-s|--search-node-name NODE_NAME` option will make this command return results that INCLUDE the NODE_NAME.",
+
+          "    4. `-S|--search-role-name ROLE_NAME` option will make this command return results that INCLUDE the ROLE_NAME.",
+
+          "    5. `-E|--search-env-name ENV_NAME` option will make this command return results that have this environment."
         ]
       ]
 
@@ -36,9 +42,11 @@ class Cheftacular
 
       environments.uniq.each do |env|
         next if env == '_default'
+        next if @options['env'] != @config['cheftacular']['default_environment'] && env != @options['env']
+        next if @options['search_env_name'] && env != @options['search_env_name']
 
         env_nodes = @config['parser'].exclude_nodes(nodes, [{ if: { not_env: env } }])
-        puts "\nFound #{ env_nodes.count } #{ env } nodes:"
+        puts "\nFound #{ env_nodes.count } #{ env } total nodes: (If you input search arguments the number returned will be smaller)"
         out = "  #{ 'name'.ljust(22) } #{ 'ip_address'.ljust(21) }"
         out << "#{ 'private_address'.ljust(21) }"                   if @options['with_private']
         out << "#{ 'pass?'.ljust(5) } #{ 'domain'.ljust(41) }"      if @options['verbose']
@@ -52,7 +60,9 @@ class Cheftacular
         addresses_hash = @config['getter'].get_addresses_hash env
 
         env_nodes.each do |node|
-          #client = @ridley.client.find(options['node_name'])
+          next if @options['search_node_name'] && !node.name.include?(@options['search_node_name'])
+          next if @options['search_role_name'] && !node.run_list.join(', ').gsub('role[','').gsub(']','').include?(@options['search_role_name'])
+          
           out = "  #{ node.chef_id.ljust(22,'_') }_#{ node.public_ipaddress.ljust(20,'_') }"
 
           if @options['with_private']
@@ -80,7 +90,7 @@ class Cheftacular
             end
           end
 
-          out << "_#{ node.run_list.join(', ') }"
+          out << "_#{ node.run_list.join(', ').gsub('role[','').gsub(']','') }"
           
           puts out
         end

data/lib/cheftacular/stateless_actions/cloud_bootstrap.rb

@@ -44,10 +44,6 @@ class Cheftacular
 
       puts "Preparing to boot #{ @options['node_name'] }(#{ @options['flavor_name'] })..."
 
-      if `which sshpass`.empty?
-        raise "sshpass not installed! Please run brew install https://raw.github.com/eugeneoden/homebrew/eca9de1/Library/Formula/sshpass.rb (or get it from your repo for linux)"
-      end
-
       real_node_name = @config['getter'].get_current_real_node_name
 
       #the output of the cloud command is a hash, this hash is UPDATED every time a rax command is run so you only need to grab it when you need it

data/lib/cheftacular/stateless_actions/knife_upload.rb

@@ -8,7 +8,9 @@ class Cheftacular
         "This command is analog for `knife upload /`",
 
         [
-          "    1. The force option will add the force option to knife upload."
+          "    1. The force option will add the force option to knife upload.",
+
+          "    2. Utilize `knife cookbook upload -a -V --cookbook-path ./cookbooks` if this command gives you trouble"
         ]
       ]
 

data/lib/cheftacular/stateless_actions/update_cookbook.rb

@@ -0,0 +1,45 @@
+
+class Cheftacular
+  class StatelessActionDocumentation
+    def update_cookbook
+      @config['documentation']['stateless_action'][__method__] ||= {}
+      @config['documentation']['stateless_action'][__method__]['long_description'] = [
+        "`cft update_cookbook [COOKBOOK_NAME] [INSTALL_VERSION]` allows you to specifically update a single cookbook",
+
+        [
+          "    1. This command passed with no arguments will update TheCheftacularCookbook"
+        ]
+      ]
+
+      @config['documentation']['stateless_action'][__method__]['short_description'] = 'Useful for updating specific cookbooks'
+    end
+  end
+
+  class StatelessAction
+    def update_cookbook cookbook='TheCheftacularCookbook', version='latest'
+      raise "This action can only be performed if the mode is set to devops" unless @config['helper'].running_in_mode?('devops')
+
+      cookbook = ARGV[1] if ARGV[1]
+      version  = ARGV[2] if ARGV[2]
+
+      @config['cheftacular']['wrapper_cookbooks'].split(',').each do |wrapper_cookbook|
+        wrapper_cookbook_loc = "#{ @config['locs']['cookbooks'] }/#{ wrapper_cookbook }"
+        FileUtils.rm_rf(File.expand_path("#{ @config['locs']['berks'] }/cookbooks")) if File.exists?(File.expand_path("#{ @config['locs']['berks'] }/cookbooks"))
+        
+        Dir.chdir wrapper_cookbook_loc
+        puts "Installing new cookbooks..."
+        out = `berks install`
+        puts "#{out}\nFinished fetching cookbooks, moving #{ cookbook } into local chef repo"
+
+        specific_cookbook = @config['filesystem'].parse_berkshelf_cookbook_versions(version).select {|key| key.include?(cookbook)}[cookbook]
+
+        puts "Moving #{ cookbook } (#{ specific_cookbook['version'] })[#{ specific_cookbook['mtime'] }] to your chef-repo!"
+
+        `rm -Rf #{ @config['locs']['cookbooks'] }/#{ cookbook }` if File.exists?(File.expand_path("#{ @config['locs']['cookbooks'] }/#{ cookbook }"))
+        `cp -Rf #{ @config['locs']['berks'] }/#{ specific_cookbook['location'] } #{ @config['locs']['cookbooks'] }/#{ cookbook }`
+        
+        break
+      end
+    end
+  end
+end

data/lib/cheftacular/version.rb

@@ -1,5 +1,5 @@
 class Cheftacular
   #major_version.minor_version.bugfixes
-  VERSION      = "2.11.1"
+  VERSION      = "2.12.0"
   RUBY_VERSION = "2.2.2"
 end

data/lib/sshkit/actions/start_task.rb

@@ -1,7 +1,7 @@
 module SSHKit
   module Backend
     class Netssh
-      def start_task name, ip_address, run_list, command, options, locs, cheftacular, out=""
+      def start_task name, ip_address, run_list, command, options, locs, cheftacular, passwords, out=""
         log_loc, timestamp = set_log_loc_and_timestamp(locs)
         true_env = get_true_environment run_list, cheftacular['run_list_environments'][options['env']], options['env']
 
@@ -9,7 +9,7 @@ module SSHKit
 
         target_loc = "#{ cheftacular['base_file_path'] }/#{ options['repository'] }/current"
 
-        if test( target_loc )
+        if !sudo_test( passwords[ip_address], target_loc )
           puts "#{ name } (#{ ip_address }) cannot run #{ command } as there is no directory at #{ target_loc }!"
 
           return ['', timestamp]

data/lib/sshkit/monkeypatches.rb

@@ -9,8 +9,6 @@ module SSHKit
         message = ""
         message += "#{command} exit status: " + exit_status.to_s + "\n"
         message += "#{command} stdout: " + (full_stdout.strip || "Nothing written") + "\n"
-
-        stderr_message = [stderr.strip, full_stderr.strip].delete_if(&:empty?).first
         message += "#{command} stderr: " + (full_stderr.strip || 'Nothing written') + "\n\n"
         raise Failed, message
       end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cheftacular
 version: !ruby/object:Gem::Version
-  version: 2.11.1
+  version: 2.12.0
 platform: ruby
 authors:
 - Louis Alridge
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-12-02 00:00:00.000000000 Z
+date: 2015-12-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: hashie
@@ -72,14 +72,28 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '4.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: berkshelf-api-client
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: highline
   requirement: !ruby/object:Gem::Requirement
@@ -320,8 +334,8 @@ files:
 - lib/cheftacular/stateless_actions/update_chef_client.rb
 - lib/cheftacular/stateless_actions/update_cheftacular.rb
 - lib/cheftacular/stateless_actions/update_cloudflare_dns_from_cloud.rb
+- lib/cheftacular/stateless_actions/update_cookbook.rb
 - lib/cheftacular/stateless_actions/update_split_branches.rb
-- lib/cheftacular/stateless_actions/update_thecheftacularcookbook.rb
 - lib/cheftacular/stateless_actions/update_tld.rb
 - lib/cheftacular/stateless_actions/upload_nodes.rb
 - lib/cheftacular/stateless_actions/upload_roles.rb

data/lib/cheftacular/stateless_actions/update_thecheftacularcookbook.rb

@@ -1,38 +0,0 @@
-
-class Cheftacular
-  class StatelessActionDocumentation
-    def update_thecheftacularcookbook
-      @config['documentation']['stateless_action'][__method__] ||= {}
-      @config['documentation']['stateless_action'][__method__]['long_description'] = [
-        "`cft update_thecheftacularcookbook` allows you to update ONLY the internal chef-repo's TheCheftacularCookbook."
-      ]
-
-      @config['documentation']['stateless_action'][__method__]['short_description'] = 'Useful for ONLY updating TheCheftacularCookbook'
-    end
-  end
-
-  class StatelessAction
-    def update_thecheftacularcookbook local_options={'interactive' => true}
-      raise "This action can only be performed if the mode is set to devops" unless @config['helper'].running_in_mode?('devops')
-
-      @config['cheftacular']['wrapper_cookbooks'].split(',').each do |wrapper_cookbook|
-        wrapper_cookbook_loc = "#{ @config['locs']['cookbooks'] }/#{ wrapper_cookbook }"
-        FileUtils.rm_rf(File.expand_path("#{ @config['locs']['berks'] }/cookbooks")) if File.exists?(File.expand_path("#{ @config['locs']['berks'] }/cookbooks"))
-        
-        Dir.chdir wrapper_cookbook_loc
-        puts "Installing new cookbooks..."
-        out = `berks install`
-        puts "#{out}\nFinished fetching cookbooks, moving TheCheftacularCookbook into local chef repo"
-
-        cheftacular_cookbook = @config['filesystem'].parse_latest_berkshelf_cookbook_versions.select {|key| key.include?('TheCheftacularCookbook')}['TheCheftacularCookbook']
-
-        puts "Moving TheCheftacularCookbook (#{ cheftacular_cookbook['version'] })[#{ cheftacular_cookbook['mtime'] }] to your chef-repo!"
-
-        `rm -Rf #{ @config['locs']['cookbooks'] }/TheCheftacularCookbook` if File.exists?(File.expand_path("#{ @config['locs']['cookbooks'] }/TheCheftacularCookbook"))
-        `cp -Rf #{ @config['locs']['berks'] }/#{ cheftacular_cookbook['location'] } #{ @config['locs']['cookbooks'] }/TheCheftacularCookbook`
-        
-        break
-      end
-    end
-  end
-end