taste_tester 0.0.13 → 0.0.18

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 13e9b1fb00c7044fe801ccb1158bbaacbbe30dd9
-  data.tar.gz: 3fcd362ea2bbdfed358847a485a695c9669d44df
+SHA256:
+  metadata.gz: e8206a39afa26db4e27600fa362c26d33d16fcefe6a5e79a4869fe339b9e1e18
+  data.tar.gz: 2eb1a8bdfd58d5713dc1f5197da4b10cd0167542fefa641bfe3d6863c252fc58
 SHA512:
-  metadata.gz: f4da6aa14ee5b8c3f732eae1480f9793318b377f1574bf1bb12f2ec8b39c91edd8e8635d9ebd6e9042bf62a767ae55655ee447b0b47bfb83fbba4d1b13832e66
-  data.tar.gz: 67db565bdee6e84b0ad23c73183f88af0873ac12a84ae1451c8587038222634e0bf6888ef1e9a30280631586ad7a350087ad7a2f74f249f3a324f208a3a2395a
+  metadata.gz: 40e15c8cf0858f82e65e0b94e4bdcfa44223b0c891ef9242aa0cbb264c36c6931e033b9b10cf70fcc68b74be4ff4628042872d78466f761bfc367de69f116897
+  data.tar.gz: 28dbed4002b508d4eb7222b3007bc6580dd4c881d4894e456a849bc882339d6b97ccec6858682c5bc71b0dbf8aa03d377de47eefa85e1c5cff09601884ca19bf

data/README.md

@@ -1,7 +1,6 @@
 # Taste Tester
 
-[![TravisCI](https://travis-ci.org/facebook/taste-tester.svg)](http://travis-ci.org/facebook/taste-tester)
-[![CircleCI](https://circleci.com/gh/facebook/taste-tester.svg?style=svg)](https://circleci.com/gh/facebook/taste-tester)
+![Continuous Integration](https://github.com/facebook/taste-tester/workflows/Continuous%20Integration/badge.svg?event=push)
 
 ## Intro
 Ohai!
@@ -25,6 +24,7 @@ Typical usage is:
 
 ```text
 vi cookbooks/...             # Make your changes and commit locally
+taste-tester impact          # Check where your changes are used
 taste-tester test -s [host]  # Put host in Taste Tester mode
 ssh root@[host]              # Log in to host
   # Run chef and watch it break
@@ -37,13 +37,13 @@ taste-tester untest [host]   # Put host back in production
                              #   (optional - will revert itself after 1 hour)
 ```
 
-See the help for futher information.
+See the help for further information.
 
 ## Prerequisites
 
-* Taste Tester assumes that /etc/chef/client.rb and /etc/chef/client.pem on your
-servers is a symlink and that your real config is /etc/chef/client-prod.rb and
-/etc/chef/client-prod.pem, respectively.
+* Taste Tester assumes that `/etc/chef/client.rb` and `/etc/chef/client.pem` on your
+servers is a symlink and that your real config is `/etc/chef/client-prod.rb` and
+`/etc/chef/client-prod.pem`, respectively.
 
 * Taste Tester assumes that it's generally safe to "go back" to production. I.e.
 We set things up so you can set a cronjob to un-taste-test a server after the
@@ -61,6 +61,8 @@ you want to test on, i.e. SSH public/private keys, SSH certificates, Kerberos
 * Mixlib::Config
 * Colorize
 * BetweenMeals
+* Minitar
+* Chef
 
 ## Automatic Untesting
 
@@ -79,18 +81,19 @@ is also killing the ssh tunnel whose PID is in `/etc/chef/test_timestamp`
 ## Config file
 
 The default config file is `/etc/taste-tester-config.rb` but you may use -c to
-specify another. The config file works the same as client.rb does for Chef -
-there are a series of keywords that take an arguement and anything else is just
+specify another. The config file works the same as `client.rb` does for Chef -
+there are a series of keywords that take an argument and anything else is just
 standard Ruby.
 
 All command-line options are available in the config file:
 * debug (bool, default: `false`)
 * timestamp (bool, default: `false`)
 * config_file (string, default: `/etc/taste-tester-config.rb`)
-* plugin_path (string, default: `/etc/taste-tester-plugin.rb`)
+* plugin_path (string, no default)
 * repo (string, default: `#{ENV['HOME']}/ops`)
 * testing_time (int, default: `3600`)
-* chef_client_command (strng, default: `chef-client`)
+* chef_client_command (string, default: `chef-client`)
+* json (bool, default: `false`)
 * skip_repo_checks (bool, default: `false`)
 * skip_pre_upload_hook (bool, default: `false`)
 * skip_post_upload_hook (bool, default: `false`)
@@ -102,7 +105,7 @@ The following options are also available:
 * base_dir - The directory in the repo under which to find chef configs.
   Default: `chef`
 * cookbook_dirs - An array of cookbook directories relative to base_dir.
-  Default: `['cookbooks']
+  Default: `['cookbooks']`
 * role_dir - A directory of roles, relative to base_dir. Default: `roles`
 * databag_dir - A directory of databags, relative to base_dir.
   Default: `databags`
@@ -110,6 +113,14 @@ The following options are also available:
   `#{ENV['HOME']}/.chef/taste-tester-ref.txt`
 * checksum_dir - The checksum directory to put in knife.conf for users. Default:
   `#{ENV['HOME']}/.chef/checksums`
+* bundle - use a single tar.gz file for transporting cookbooks, roles and
+  databags to clients. Experimental. Value is tri-state:
+  * `false` - server uses knife upload, client uses `chef_server`
+  * `:compatible` - make server support both methods, client uses tar.gz
+  * `true` - server only creates tar.gz, client uses tar.gz
+  Default: false
+* impact - analyze local changes to determine which hosts/roles to test.
+  Default: false
 
 ## Plugin
 
@@ -150,3 +161,43 @@ Stuff to do after putting all hosts in test mode.
 * self.repo_checks(dryrun, repo)
 
 Additional checks you want to do on the repo as sanity checks.
+
+* self.find_impact(changes)
+
+Custom implementation of impact analysis. Uses `knife deps` by default. May
+return any data structure, provided one or both of `self.post_impact` or
+`self.print_impact` are defined.
+
+* self.post_impact(basic_impact)
+
+Stuff to do after preliminary impact analysis. May be used to extend the
+information generated by `self.find_impact`, reformat the data structure, etc.
+
+* self.print_impact(final_impact)
+
+Custom output of calculated impact, useful if defining either of the other
+impact hooks. Must return a truthy value to prevent the default output from
+printing.
+
+## Plugin example
+
+This is an example `/etc/taste-tester-plugin.rb` to add a user-defined string
+to `client-taste-tester.rb` on the remote system:
+
+```
+Hooks.class_eval do
+  def self.test_remote_client_rb_extra_code(_hostname)
+    %(
+      # This comment gets added to client-taste-tester.rb
+      # This one too.
+      )
+  end
+end
+```
+
+Be sure to pass this plugin file with `-p` on the command line or set it as
+`plugin_path` in your `taste-tester-config.rb` file.
+
+## License
+
+See the `LICENSE` file.

data/bin/taste-tester

@@ -28,27 +28,27 @@ require 'taste_tester/commands'
 require 'taste_tester/hooks'
 require 'taste_tester/exceptions'
 
-include TasteTester::Logging
-
-if ENV['USER'] == 'root'
-  logger.warn('You should not be running as root')
-  exit(1)
-end
-
 # Command line parsing and param descriptions
 module TasteTester
+  extend TasteTester::Logging
+
   verify = 'Verify your changes were actually applied as intended!'.red
 
+  if ENV['USER'] == 'root'
+    logger.warn('You should not be running as root')
+    exit(1)
+  end
+
   # Do an initial read of the config file if it's in the default place, so
   # that if people override chef_client_command the help message is correct.
-  if File.exists?(File.expand_path(TasteTester::Config.config_file))
+  if File.exist?(File.expand_path(TasteTester::Config.config_file))
     TasteTester::Config.from_file(
       File.expand_path(TasteTester::Config.config_file),
     )
   end
 
   cmd = TasteTester::Config.chef_client_command
-  description = <<-EOF
+  description = <<-ENDOFDESCRIPTION
 Welcome to taste-tester!
 
 Usage: taste-tester <mode> [<options>]
@@ -66,7 +66,8 @@ TLDR; Most common usage is:
   taste-tester untest -s [host] # Put host back in production
                                 #   (optional - will revert itself after 1 hour)
 
-And you're done! See the above wiki page for more details.
+And you're done!
+Note: There may be site specific testing instructions, see local documentation for details.
 
 MODES:
   test
@@ -74,6 +75,13 @@ MODES:
     point some production server specified by -s to your virtual chef server for
     testing.  If you have you a plugin that uses the hookpoint, it'll may amend
     your commit message to denote the server you tested.
+    Returns:
+      0 Success.
+      1 Failure, e.g. host not reachable or usage error.
+      2 Partial success, mixture of success and failure due to hosts being taste
+        tested by other users.
+      3 All hosts were not taste-tested because they are being taste-tested by
+        other users.
 
   upload
     Sync your local repo to your virtual Chef server (i.e. just the first step
@@ -83,6 +91,13 @@ MODES:
     cookbooks and roles. It also does a fair amount of sanity checking on
     your repo and you may specify --skip-repo-checks to bypass this.
 
+  impact
+    Examine local repo changes to determine which roles are potentially
+    impacted by the changes. Compares the set of modified cookbooks with the
+    dependency lists of each role and reports any role which depends on a
+    modified cookbook. It is recommended to test your changes on at least one
+    server of each role, potentially also on multiple platforms.
+
   keeptesting
     Extend the testing time on server specified by -s by 1 hour unless
     otherwise specified by -t.
@@ -111,7 +126,7 @@ MODES:
     You probably don't want this. It will restart up the chef-zero server on
     your localhost. taste-tester dynamically starts this if it's down, so there
     should be no need to do this manually.
-  EOF
+  ENDOFDESCRIPTION
 
   mode = ARGV.shift unless !ARGV.empty? && ARGV[0].start_with?('-')
 
@@ -121,7 +136,6 @@ MODES:
   end
 
   options = { :config_file => TasteTester::Config.config_file }
-  # rubocop:disable Metrics/BlockLength
   parser = OptionParser.new do |opts|
     opts.banner = description
 
@@ -129,7 +143,7 @@ MODES:
     opts.separator 'Global options:'.upcase
 
     opts.on('-c', '--config FILE', 'Config file') do |file|
-      unless File.exists?(File.expand_path(file))
+      unless File.exist?(File.expand_path(file))
         logger.error("Sorry, cannot find #{file}")
         exit(1)
       end
@@ -146,7 +160,7 @@ MODES:
     end
 
     opts.on('-p', '--plugin-path FILE', String, 'Plugin file') do |file|
-      unless File.exists?(File.expand_path(file))
+      unless File.exist?(File.expand_path(file))
         logger.error("Sorry, cannot find #{file}")
         exit(1)
       end
@@ -230,6 +244,7 @@ MODES:
       'Until when should the host remain in testing.' +
       ' Anything parsable is ok, such as "5/18 4:35" or "16/9/13".'
     ) do |time|
+      # can make this an implicit rescue after we drop ruby 2.4
       begin
         options[:testing_until] = Time.parse(time)
       rescue StandardError
@@ -241,7 +256,7 @@ MODES:
     opts.on(
       '-t', '--testing-time TIME',
       'How long should the host remain in testing.' +
-      ' Takes a simple relative time string, such as "45m", "4h" or "2d".'
+      ' Takes a simple relative time string, such as "45m", "4h" or "1d".'
     ) do |time|
       m = time.match(/^(\d+)([d|h|m]+)$/)
       if m
@@ -260,7 +275,7 @@ MODES:
 
     opts.on(
       '-r', '--repo DIR',
-      "Custom repo location, current deafult is #{TasteTester::Config.repo}." +
+      "Custom repo location, current default is #{TasteTester::Config.repo}." +
         ' Works on upload and test.'
     ) do |dir|
       options[:repo] = dir
@@ -273,6 +288,16 @@ MODES:
       options[:repo_type] = type
     end
 
+    opts.on(
+      '--clowntown-no-repo', 'This option enables taste-tester to run ' +
+      'without a SCM repo of any sort. This has negative implications like ' +
+      'always doing a full upload. This option is here for weird corner ' +
+      'cases like having to sync out the export of a repo, but is not ' +
+      'generally a good idea or well supported. Use at your own risk.'
+    ) do
+      options[:no_repo] = true
+    end
+
     opts.on(
       '-R', '--roles ROLE[,ROLE]', Array,
       'Specific roles to upload. Intended mostly for debugging,' +
@@ -281,6 +306,17 @@ MODES:
       options[:roles] = roles
     end
 
+    opts.on(
+      '-J', '--jumps JUMP',
+      'Uses ssh\'s `ProxyJump` support to ssh across bastion/jump hosts. ' +
+      'This is particularly useful in tunnel mode to test machines that ' +
+      'your workstatation doesn\'t have direct access to. The format is ' +
+      'the same as `ssh -J`: a comma-separated list of hosts to forward ' +
+      'through.'
+    ) do |jumps|
+      options[:jumps] = jumps
+    end
+
     opts.on('--really', 'Really do link-only. DANGEROUS!') do |r|
       options[:really] = r
     end
@@ -311,6 +347,12 @@ MODES:
       options[:ssh_command] = c
     end
 
+    opts.on(
+      '--ssh-connect-timeout TIMEOUT', 'SSH \'-o ConnectTimeout\' value'
+    ) do |c|
+      options[:ssh_connect_timeout] = c
+    end
+
     opts.on('--skip-repo-checks', 'Skip repository sanity checks') do
       options[:skip_repo_checks] = true
     end
@@ -319,6 +361,22 @@ MODES:
       options[:yes] = true
     end
 
+    opts.on(
+      '--json', 'Format output as JSON for programatic consumption.' +
+      ' Default to false. Works on impact.'
+    ) do
+      options[:json] = true
+    end
+
+    opts.on(
+      '-w', '--windows-target',
+      'The target is a Windows machine. You will likely want to override ' +
+      '`test_timestamp` and `chef_config_path`, but *not* `config_file`. ' +
+      'Requires the target be running PowerShell >= 5.1 as the default shell.'
+    ) do
+      options[:windows_target] = true
+    end
+
     opts.separator ''
     opts.separator 'Control local hook behavior with these options:'
 
@@ -352,7 +410,6 @@ MODES:
       options[:skip_post_test_hook] = true
     end
   end
-  # rubocop:enable Metrics/BlockLength
 
   if mode == 'help'
     puts parser
@@ -361,7 +418,7 @@ MODES:
 
   parser.parse!
 
-  if File.exists?(File.expand_path(options[:config_file]))
+  if File.exist?(File.expand_path(options[:config_file]))
     TasteTester::Config.from_file(File.expand_path(options[:config_file]))
   end
   TasteTester::Config.merge!(options)
@@ -370,7 +427,7 @@ MODES:
 
   if TasteTester::Config.plugin_path
     path = File.expand_path(TasteTester::Config.plugin_path)
-    unless File.exists?(path)
+    unless File.exist?(path)
       logger.error("Plugin not found (#{path})")
       exit(1)
     end
@@ -397,6 +454,8 @@ MODES:
       TasteTester::Commands.runchef
     when :upload
       TasteTester::Commands.upload
+    when :impact
+      TasteTester::Commands.impact
     else
       logger.error("Invalid mode: #{mode}")
       puts parser
@@ -408,5 +467,7 @@ MODES:
 end
 
 if $PROGRAM_NAME == __FILE__
-  include TasteTester
+  module TasteTester
+    include TasteTester
+  end
 end

data/lib/taste_tester/client.rb

@@ -14,10 +14,14 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+require 'minitar'
+require 'find'
 require 'taste_tester/logging'
 require 'between_meals/repo'
 require 'between_meals/knife'
 require 'between_meals/changeset'
+require 'chef/log'
+require 'chef/cookbook/chefignore'
 
 module TasteTester
   # Client side upload functionality
@@ -44,16 +48,22 @@ module TasteTester
         :databag_dir => TasteTester::Config.databags,
         :checksum_dir => TasteTester::Config.checksum_dir,
         :role_type => TasteTester::Config.role_type,
+        :config => TasteTester::Config.knife_config,
       )
       @knife.write_user_config
-      @repo = BetweenMeals::Repo.get(
-        TasteTester::Config.repo_type,
-        TasteTester::Config.repo,
-        logger,
-      )
-      unless @repo.exists?
+      if TasteTester::Config.no_repo
+        @repo = nil
+      else
+        @repo = BetweenMeals::Repo.get(
+          TasteTester::Config.repo_type,
+          TasteTester::Config.repo,
+          logger,
+        )
+      end
+      if @repo && !@repo.exists?
         fail "Could not open repo from #{TasteTester::Config.repo}"
       end
+
       @track_symlinks = TasteTester::Config.track_symlinks
     end
 
@@ -64,26 +74,30 @@ module TasteTester
     end
 
     def upload
-      checks unless @skip_checks
-
-      logger.info("Last commit: #{@repo.head_rev} " +
-        "'#{@repo.last_msg.split("\n").first}'" +
-        " by #{@repo.last_author[:email]}")
+      head_rev = nil
+      if @repo
+        head_rev = @repo.head_rev
+        checks unless @skip_checks
+        logger.info("Last commit: #{head_rev} " +
+          "'#{@repo.last_msg.split("\n").first}'" +
+          " by #{@repo.last_author[:email]}")
+      end
 
-      if @force || !@server.latest_uploaded_ref
+      if @force || !@server.latest_uploaded_ref || !@repo
         logger.info('Full upload forced') if @force
+        logger.info('No repo, doing full upload') unless @repo
         unless TasteTester::Config.skip_pre_upload_hook
           TasteTester::Hooks.pre_upload(TasteTester::Config.dryrun,
                                         @repo,
                                         nil,
-                                        @repo.head_rev)
+                                        head_rev)
         end
         time(logger) { full }
         unless TasteTester::Config.skip_post_upload_hook
           TasteTester::Hooks.post_upload(TasteTester::Config.dryrun,
                                          @repo,
                                          nil,
-                                         @repo.head_rev)
+                                         head_rev)
         end
       else
         # Since we also upload the index, we always need to run the
@@ -93,7 +107,7 @@ module TasteTester
           TasteTester::Hooks.pre_upload(TasteTester::Config.dryrun,
                                         @repo,
                                         @server.latest_uploaded_ref,
-                                        @repo.head_rev)
+                                        head_rev)
         end
         begin
           time(logger) { partial }
@@ -105,23 +119,118 @@ module TasteTester
           TasteTester::Hooks.post_upload(TasteTester::Config.dryrun,
                                          @repo,
                                          @server.latest_uploaded_ref,
-                                         @repo.head_rev)
+                                         head_rev)
         end
       end
 
-      @server.latest_uploaded_ref = @repo.head_rev
+      @server.latest_uploaded_ref = head_rev
+      @server.last_upload_time = Time.new.strftime('%Y-%m-%d %H:%M:%S')
     end
 
     private
 
+    def populate(stream, writer, path, destination)
+      full_path = File.join(File.join(TasteTester::Config.repo, path))
+      return unless File.directory?(full_path)
+      chefignores = Chef::Cookbook::Chefignore.new(full_path)
+      # everything is relative to the repo dir. chdir makes handling all the
+      # paths within this simpler
+      Dir.chdir(full_path) do
+        look_at = ['']
+        while (prefix = look_at.pop)
+          Dir.glob(File.join("#{prefix}**", '*'), File::FNM_DOTMATCH) do |p|
+            minus_first = p.split(
+              File::SEPARATOR,
+            )[1..-1].join(File::SEPARATOR)
+            next if chefignores.ignored?(p) ||
+              chefignores.ignored?(minus_first)
+            name = File.join(destination, p)
+            if File.directory?(p)
+              # we don't store directories in the tar, but we do want to follow
+              # top level symlinked directories as they are used to share
+              # cookbooks between codebases.
+              if minus_first == '' && File.symlink?(p)
+                look_at.push("#{p}#{File::SEPARATOR}")
+              end
+            elsif File.symlink?(p)
+              # tar handling of filenames > 100 characters gets complex. We'd
+              # use split_name from Minitar, but it's a private method. It's
+              # reasonable to assume that all symlink names in the bundle are
+              # less than 100 characters long. Long term, the version of minitar
+              # in chefdk should be upgraded.
+              fail 'Add support for long symlink paths' if name.size > 100
+              # The version of Minitar included in chefdk does not support
+              # symlinks directly. Therefore we use direct writes to the
+              # underlying stream to reproduce the symlinks
+              symlink = {
+                :name => name,
+                :mode => 0644,
+                :typeflag => '2',
+                :size => 0,
+                :linkname => File.readlink(p),
+                :prefix => '',
+              }
+              stream.write(Minitar::PosixHeader.new(symlink))
+            else
+              File.open(p, 'rb') do |r|
+                writer.add_file_simple(
+                  name, :mode => 0644, :size => File.size(r)
+                ) do |d, _opts|
+                  IO.copy_stream(r, d)
+                end
+              end
+            end
+          end
+        end
+      end
+    end
+
+    def bundle_upload
+      dest = File.join(@server.bundle_dir, 'tt.tgz')
+      begin
+        Tempfile.create(['tt', '.tgz'], @server.bundle_dir) do |tempfile|
+          stream = Zlib::GzipWriter.new(tempfile)
+          Minitar::Writer.open(stream) do |writer|
+            TasteTester::Config.relative_cookbook_dirs.each do |cb_dir|
+              populate(stream, writer, cb_dir, 'cookbooks')
+            end
+            populate(
+              stream, writer, TasteTester::Config.relative_role_dir, 'roles'
+            )
+            populate(
+              stream, writer, TasteTester::Config.relative_databag_dir,
+              'databag'
+            )
+          end
+          stream.close
+          File.rename(tempfile.path, dest)
+        end
+      rescue Errno::ENOENT
+        # Normally the temporary file is renamed to the dest name. If this
+        # happens, then the cleanup of of the temporary file doesn't work,
+        # but this is fine and expected.
+        nil
+      end
+    end
+
     def full
       logger.warn('Doing full upload')
+      if TasteTester::Config.bundle
+        bundle_upload
+        # only leave early if true (strictly bundle mode only)
+        return if TasteTester::Config.bundle == true
+      end
       @knife.cookbook_upload_all
       @knife.role_upload_all
       @knife.databag_upload_all
     end
 
     def partial
+      if TasteTester::Config.bundle
+        logger.info('No partial support for bundle mode, doing full upload')
+        bundle_upload
+        return if TasteTester::Config.bundle == true
+      end
       logger.info('Doing differential upload from ' +
                    @server.latest_uploaded_ref)
       changeset = BetweenMeals::Changeset.new(