pdk 1.10.0 → 1.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: c1a405daa77845c20f2b65516c1a43532f4589c0
-  data.tar.gz: 155e5410885b024cbfb942c7b35c281c89a30402
+SHA256:
+  metadata.gz: 5ec16ddd2727410b30d9b0d41457e4c4973f14607af8fffd10cf3ce5f7a5027b
+  data.tar.gz: 5288a7bfae1df2106fc9bd012ae71fd7e61473482404b0ccac0de6375399175e
 SHA512:
-  metadata.gz: 253daf9ea17ec338176dc232ec049abc6fdd88facfe5995fbaba73cb804a30b4d082fa56973fea9b841ad936f86223111939ff474ff810dfddf6ab5beb3ed869
-  data.tar.gz: 45985a21c5d8989d8c7214a0e7fe90df74f022850d9fe1fb429d92ff01642a755a4379adda5eb2f8bf5ee71508b9668e7b164abd17e34b268a7acbcc024c3409
+  metadata.gz: b08bd2bf5c1e54de90c6b1b8f3b91a3569827ddc06a5277b0b2bc5387f9fd21526c23f7a63cc97e7c87eb3c64b859214a94a9c86c79ba9509b3349cd1da39545
+  data.tar.gz: 7d101cca2234748f4edd69e7f5aafab5e99c9161efc08c36af3c8cd2aa4d613bdfbc0b7210ef6c47eee15bfab4332eebb1d2efba9cbe5bb9397454d4f4517057

data/CHANGELOG.md

@@ -4,6 +4,54 @@ All changes to this repo will be documented in this file.
 See the [release notes](https://puppet.com/docs/pdk/latest/release_notes.html) for a high-level summary.
 
 
+## [v1.11.0](https://github.com/puppetlabs/pdk/tree/v1.11.0) (2019-06-27)
+[Full Changelog](https://github.com/puppetlabs/pdk/compare/v1.10.0...v1.11.0)
+
+**Fixed bugs:**
+
+- \(PDK-1348\) remove unused constants throwing warns [\#656](https://github.com/puppetlabs/pdk/pull/656) ([tphoney](https://github.com/tphoney))
+
+**Closed issues:**
+
+- template-ref behaviour in PDK 1.10.0 breaks backwards compatibility [\#661](https://github.com/puppetlabs/pdk/issues/661)
+- pdk validate reports error on is\_to\_s [\#642](https://github.com/puppetlabs/pdk/issues/642)
+- pdk 1.9.1.0 on windows does not set path env variable [\#641](https://github.com/puppetlabs/pdk/issues/641)
+- default\_facts.yml does not override values from facterdb [\#628](https://github.com/puppetlabs/pdk/issues/628)
+- PDK and beaker [\#622](https://github.com/puppetlabs/pdk/issues/622)
+- Configure 'ordering' for rspec-puppet in PDK [\#511](https://github.com/puppetlabs/pdk/issues/511)
+- Cannot override module Hiera 5 config for unit tests [\#487](https://github.com/puppetlabs/pdk/issues/487)
+
+**Merged pull requests:**
+
+- \(FIXUP\) Avoid attempting to append nokogiri pin to nil in package tests [\#686](https://github.com/puppetlabs/pdk/pull/686) ([scotje](https://github.com/scotje))
+- Revert "\(PDK-1366\) Update default operatingsystem versions" [\#685](https://github.com/puppetlabs/pdk/pull/685) ([rodjek](https://github.com/rodjek))
+- \(maint\) Clear Gemfile overrides before pdk update test [\#684](https://github.com/puppetlabs/pdk/pull/684) ([rodjek](https://github.com/rodjek))
+- \(PDK-1366\) Update default operatingsystem versions [\#682](https://github.com/puppetlabs/pdk/pull/682) ([rodjek](https://github.com/rodjek))
+- \(PDK-1362\) Warn user if updating module with older PDK version [\#681](https://github.com/puppetlabs/pdk/pull/681) ([rodjek](https://github.com/rodjek))
+- \(PDK-1365\) Use dynamic ruby detection for default ruby instance [\#678](https://github.com/puppetlabs/pdk/pull/678) ([glennsarti](https://github.com/glennsarti))
+- \(PDK-1354\) Default template ref for custom templates should always be master [\#677](https://github.com/puppetlabs/pdk/pull/677) ([rodjek](https://github.com/rodjek))
+- \(maint\) Pin cri to \<= 2.15.6 [\#675](https://github.com/puppetlabs/pdk/pull/675) ([rodjek](https://github.com/rodjek))
+- \(MAINT\) Fix issues related to Cri behavior change with options hash [\#672](https://github.com/puppetlabs/pdk/pull/672) ([scotje](https://github.com/scotje))
+- \(PDK-1337\) Warn and unset any of the legacy \*\_GEM\_VERSION env vars [\#671](https://github.com/puppetlabs/pdk/pull/671) ([rodjek](https://github.com/rodjek))
+- \(PDK-1345\) Disable analytics during package tests [\#670](https://github.com/puppetlabs/pdk/pull/670) ([rodjek](https://github.com/rodjek))
+- \(MAINT\) Fix MSYS2 update on Appveyor [\#669](https://github.com/puppetlabs/pdk/pull/669) ([rodjek](https://github.com/rodjek))
+- \(PDK-1342\) Submit PDK analytics events [\#668](https://github.com/puppetlabs/pdk/pull/668) ([rodjek](https://github.com/rodjek))
+- \(PDK-1336\) Update rubocop to 0.57.2 [\#667](https://github.com/puppetlabs/pdk/pull/667) ([scotje](https://github.com/scotje))
+- \(PDK-1341\) Hook up PDK analytics to Google Analytics [\#665](https://github.com/puppetlabs/pdk/pull/665) ([rodjek](https://github.com/rodjek))
+- \(MAINT\) Add "needs-triage" default labels to issue templates [\#664](https://github.com/puppetlabs/pdk/pull/664) ([scotje](https://github.com/scotje))
+- \(PDK-1264\) Display a nicer error when tarring long paths [\#663](https://github.com/puppetlabs/pdk/pull/663) ([rodjek](https://github.com/rodjek))
+- \(maint\) Add spawned process stdout & stderr to debug log [\#662](https://github.com/puppetlabs/pdk/pull/662) ([rodjek](https://github.com/rodjek))
+- \(PDK-1300\) Ensure `test unit --list` uses correct Puppet/Ruby env [\#660](https://github.com/puppetlabs/pdk/pull/660) ([scotje](https://github.com/scotje))
+- \(MAINT\) Fixup package acceptance tests for 'pdk-default' template URL [\#658](https://github.com/puppetlabs/pdk/pull/658) ([scotje](https://github.com/scotje))
+- \(PDK-1339\) Read or interview for analytics config [\#657](https://github.com/puppetlabs/pdk/pull/657) ([rodjek](https://github.com/rodjek))
+- \(PDK-1350\) Handle SCP style URLs in metadata.json [\#655](https://github.com/puppetlabs/pdk/pull/655) ([rodjek](https://github.com/rodjek))
+- \(PDK-1338\) Initial import of analytics code from Bolt [\#652](https://github.com/puppetlabs/pdk/pull/652) ([rodjek](https://github.com/rodjek))
+- \(MAINT\) Bump version to 1.11.0.pre [\#651](https://github.com/puppetlabs/pdk/pull/651) ([scotje](https://github.com/scotje))
+- \(PDK-1335\) Add development note when on Windows [\#649](https://github.com/puppetlabs/pdk/pull/649) ([glennsarti](https://github.com/glennsarti))
+- \(maint\) Allow developers to add additional gems [\#648](https://github.com/puppetlabs/pdk/pull/648) ([glennsarti](https://github.com/glennsarti))
+- \(PDK-1167\) Validator should honor case sensitive of the file system [\#646](https://github.com/puppetlabs/pdk/pull/646) ([glennsarti](https://github.com/glennsarti))
+- \(PDK-1193\) Saves packaged template-url in metadata as a keyword [\#639](https://github.com/puppetlabs/pdk/pull/639) ([bmjen](https://github.com/bmjen))
+
 ## [v1.10.0](https://github.com/puppetlabs/pdk/tree/v1.10.0) (2019-04-02)
 [Full Changelog](https://github.com/puppetlabs/pdk/compare/v1.9.1...v1.10.0)
 
@@ -22,6 +70,7 @@ See the [release notes](https://puppet.com/docs/pdk/latest/release_notes.html) f
 
 **Merged pull requests:**
 
+- \(PDK-1324\) Release 1.10.0 [\#650](https://github.com/puppetlabs/pdk/pull/650) ([rodjek](https://github.com/rodjek))
 - \(maint\) Fix package specs for template-ref changes [\#647](https://github.com/puppetlabs/pdk/pull/647) ([rodjek](https://github.com/rodjek))
 - \(maint\) Enforce LF line endings in Rubocop [\#645](https://github.com/puppetlabs/pdk/pull/645) ([glennsarti](https://github.com/glennsarti))
 - \(FM-7579, PDK-1236\) bump the version of CRI used [\#638](https://github.com/puppetlabs/pdk/pull/638) ([tphoney](https://github.com/tphoney))
@@ -896,4 +945,4 @@ See the [release notes](https://puppet.com/docs/pdk/latest/release_notes.html) f
 
 
 
-\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
+\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

data/lib/pdk.rb

@@ -1,4 +1,6 @@
+require 'pdk/analytics'
 require 'pdk/answer_file'
+require 'pdk/config'
 require 'pdk/generate'
 require 'pdk/i18n'
 require 'pdk/logger'
@@ -7,4 +9,17 @@ require 'pdk/template_file'
 require 'pdk/validate'
 require 'pdk/version'
 
-module PDK; end
+module PDK
+  def self.analytics
+    @analytics ||= PDK::Analytics.build_client(
+      logger:        PDK.logger,
+      disabled:      ENV['PDK_DISABLE_ANALYTICS'] || PDK.config.user['analytics']['disabled'],
+      user_id:       PDK.config.user['analytics']['user-id'],
+      app_id:        "UA-139917834-#{PDK::Util.development_mode? ? '2' : '1'}",
+      client:        :google_analytics,
+      app_name:      'pdk',
+      app_version:   PDK::VERSION,
+      app_installer: PDK::Util.package_install? ? 'package' : 'gem',
+    )
+  end
+end

data/lib/pdk/analytics.rb

@@ -0,0 +1,28 @@
+require 'securerandom'
+require 'pdk/analytics/util'
+require 'pdk/analytics/client/google_analytics'
+require 'pdk/analytics/client/noop'
+
+module PDK
+  module Analytics
+    CLIENTS = {
+      noop:             Client::Noop,
+      google_analytics: Client::GoogleAnalytics,
+    }.freeze
+
+    def self.build_client(opts = {})
+      opts[:logger] ||= ::Logger.new(STDERR)
+      opts[:client] ||= :noop
+
+      if opts[:disabled]
+        opts[:logger].debug 'Analytics opt-out is set, analytics will be disabled'
+        CLIENTS[:noop].new(opts)
+      else
+        CLIENTS[opts[:client]].new(opts)
+      end
+    rescue StandardError => e
+      opts[:logger].debug "Failed to initialize analytics client, analytics will be disabled: #{e}"
+      CLIENTS[:noop].new(opts)
+    end
+  end
+end

data/lib/pdk/analytics/client/google_analytics.rb

@@ -0,0 +1,138 @@
+module PDK
+  module Analytics
+    module Client
+      class GoogleAnalytics
+        PROTOCOL_VERSION  = 1
+        TRACKING_URL      = 'https://google-analytics.com/collect'.freeze
+        CUSTOM_DIMENSIONS = {
+          operating_system: :cd1,
+          output_format:    :cd2,
+          ruby_version:     :cd3,
+          cli_options:      :cd4,
+          env_vars:         :cd5,
+        }.freeze
+
+        attr_reader :user_id
+        attr_reader :logger
+        attr_reader :app_name
+        attr_reader :app_id
+        attr_reader :app_version
+        attr_reader :app_installer
+
+        def initialize(opts)
+          # lazy-load expensive gem code
+          require 'concurrent/configuration'
+          require 'concurrent/future'
+          require 'httpclient'
+          require 'locale'
+
+          @http = HTTPClient.new
+          @user_id = opts[:user_id]
+          @executor = Concurrent.global_io_executor
+          @os = PDK::Analytics::Util.fetch_os_async
+          @logger = opts[:logger]
+          @app_name = opts[:app_name]
+          @app_id = opts[:app_id]
+          @app_version = opts[:app_version]
+          @app_installer = opts[:app_installer]
+        end
+
+        def screen_view(screen, **kwargs)
+          custom_dimensions = walk_keys(kwargs) do |k|
+            CUSTOM_DIMENSIONS[k] || raise("Unknown analytics key '#{k}'")
+          end
+
+          screen_view_params = {
+            # Type
+            t:  'screenview',
+            # Screen Name
+            cd: screen,
+          }.merge(custom_dimensions)
+
+          submit(base_params.merge(screen_view_params))
+        end
+
+        def event(category, action, label: nil, value: nil, **kwargs)
+          custom_dimensions = walk_keys(kwargs) do |k|
+            CUSTOM_DIMENSIONS[k] || raise("Unknown analytics key '#{k}'")
+          end
+
+          event_params = {
+            # Type
+            t:  'event',
+            # Event Category
+            ec: category,
+            # Event Action
+            ea: action,
+          }.merge(custom_dimensions)
+
+          # Event Label
+          event_params[:el] = label if label
+          # Event Value
+          event_params[:ev] = value if value
+
+          submit(base_params.merge(event_params))
+        end
+
+        def submit(params)
+          # Handle analytics submission in the background to avoid blocking the
+          # app or polluting the log with errors
+          Concurrent::Future.execute(executor: @executor) do
+            logger.debug "Submitting analytics: #{JSON.pretty_generate(params)}"
+            @http.post(TRACKING_URL, params)
+            logger.debug 'Completed analytics submission'
+          end
+        end
+
+        # These parameters have terrible names. See this page for complete documentation:
+        # https://developers.google.com/analytics/devguides/collection/protocol/v1/parameters
+        def base_params
+          {
+            v:    PROTOCOL_VERSION,
+            # Client ID
+            cid:  user_id,
+            # Tracking ID
+            tid:  app_id,
+            # Application Name
+            an:   app_name,
+            # Application Version
+            av:   app_version,
+            # Application Installer ID
+            aiid: app_installer,
+            # Anonymize IPs
+            aip:  true,
+            # User locale
+            ul:   Locale.current.to_rfc,
+            # Custom Dimension 1 (Operating System)
+            cd1:  @os.value,
+          }
+        end
+
+        # If the user is running a very fast command, there may not be time for
+        # analytics submission to complete before the command is finished. In
+        # that case, we give a little buffer for any stragglers to finish up.
+        # 250ms strikes a balance between accomodating slower networks while not
+        # introducing a noticeable "hang".
+        def finish
+          @executor.shutdown
+          @executor.wait_for_termination(0.25)
+        end
+
+        private
+
+        def walk_keys(data, &block)
+          if data.is_a?(Hash)
+            data.each_with_object({}) do |(k, v), acc|
+              v = walk_keys(v, &block)
+              acc[yield(k)] = v
+            end
+          elsif data.is_a?(Array)
+            data.map { |v| walk_keys(v, &block) }
+          else
+            data
+          end
+        end
+      end
+    end
+  end
+end

data/lib/pdk/analytics/client/noop.rb

@@ -0,0 +1,23 @@
+module PDK
+  module Analytics
+    module Client
+      class Noop
+        attr_reader :logger
+
+        def initialize(opts)
+          @logger = opts[:logger]
+        end
+
+        def screen_view(screen, **_kwargs)
+          logger.debug "Skipping submission of '#{screen}' screenview because analytics is disabled"
+        end
+
+        def event(category, action, **_kwargs)
+          logger.debug "Skipping submission of '#{category} #{action}' event because analytics is disabled"
+        end
+
+        def finish; end
+      end
+    end
+  end
+end

data/lib/pdk/analytics/util.rb

@@ -0,0 +1,17 @@
+module PDK
+  module Analytics
+    module Util
+      def self.fetch_os_async
+        require 'concurrent/configuration'
+        require 'concurrent/future'
+
+        Concurrent::Future.execute(executor: :io) do
+          require 'facter'
+          os = Facter.value('os')
+
+          os.nil? ? 'unknown' : "#{os['name']} #{os.fetch('release', {}).fetch('major', '')}".strip
+        end
+      end
+    end
+  end
+end

data/lib/pdk/cli.rb

@@ -13,8 +13,45 @@ require 'pdk/report'
 require 'pdk/util/version'
 require 'pdk/util/puppet_version'
 
+class Cri::Command::CriExitException
+  def initialize(is_error:)
+    @is_error = is_error
+    PDK.analytics.event('CLI', 'invalid command', label: PDK::CLI.anonymised_args.join(' ')) if error?
+  end
+end
+
 module PDK::CLI
+  # Attempt to anonymise the raw ARGV array if the command parsing failed.
+  #
+  # If an item does not start with '-' but is preceeded by an item that does
+  # start with '-', assume that these items are an option/value pair and redact
+  # the value. Any additional values that do not start with '-' that follow an
+  # option/value pair are assumed to be arguments (rather than subcommand
+  # names) and are also redacted.
+  #
+  # @example
+  #   # Where PDK::CLI.args => ['new', 'plan', '--some', 'value', 'plan_name']
+  #
+  #   PDK::CLI.anonymised_args
+  #     => ['new', 'plan', '--some', 'redacted', 'redacted']
+  #
+  # @return Array[String] the command arguments with any identifying values
+  #   redacted.
+  def self.anonymised_args
+    in_args = false
+    @args.map do |arg|
+      if arg.start_with?('-')
+        in_args = true
+        arg
+      else
+        in_args ? 'redacted' : arg
+      end
+    end
+  end
+
   def self.run(args)
+    @args = args
+    PDK::Config.analytics_config_interview! unless PDK::Config.analytics_config_exist?
     @base_cmd.run(args)
   rescue PDK::CLI::ExitWithError => e
     PDK.logger.send(e.log_level, e.message)

data/lib/pdk/cli/build.rb

@@ -21,6 +21,8 @@ module PDK::CLI
         log_level: :info,
       )
 
+      PDK::CLI::Util.analytics_screen_view('build', opts)
+
       module_metadata = PDK::Module::Metadata.from_file('metadata.json')
 
       # TODO: Ensure forge metadata has been set, or call out to interview

data/lib/pdk/cli/bundle.rb

@@ -1,4 +1,3 @@
-
 module PDK::CLI
   @bundle_cmd = @base_cmd.define_command do
     name 'bundle'
@@ -20,6 +19,8 @@ EOF
 
       PDK::CLI::Util.validate_puppet_version_opts({})
 
+      PDK::CLI::Util.analytics_screen_view('bundle')
+
       # Ensure that the correct Ruby is activated before running commend.
       puppet_env = PDK::CLI::Util.puppet_from_opts_or_env({})
       PDK::Util::RubyVersion.use(puppet_env[:ruby_version])

data/lib/pdk/cli/convert.rb

@@ -28,6 +28,8 @@ module PDK::CLI
         raise PDK::CLI::ExitWithError, _('You can not specify --noop and --force when converting a module')
       end
 
+      PDK::CLI::Util.analytics_screen_view('convert', opts)
+
       if opts[:'skip-interview'] && opts[:'full-interview']
         PDK.logger.info _('Ignoring --full-interview and continuing with --skip-interview.')
         opts[:'full-interview'] = false

data/lib/pdk/cli/exec.rb

@@ -121,6 +121,24 @@ module PDK
           @environment.merge!(additional_env)
         end
 
+        def check_for_legacy_env_vars
+          if ENV['PUPPET_GEM_VERSION']
+            PDK.logger.warn_once _(
+              'PUPPET_GEM_VERSION is not supported by PDK. ' \
+              'Please use the --puppet-version option on your PDK command ' \
+              'or set the PDK_PUPPET_VERSION environment variable instead',
+            )
+            @process.environment['PUPPET_GEM_VERSION'] = nil
+          end
+
+          %w[FACTER HIERA].each do |gem|
+            if ENV["#{gem}_GEM_VERSION"]
+              PDK.logger.warn_once _("#{gem}_GEM_VERSION is not supported by PDK.")
+              @process.environment["#{gem}_GEM_VERSION"] = nil
+            end
+          end
+        end
+
         def execute!
           # Start spinning if configured.
           @spinner.auto_spin if @spinner
@@ -130,6 +148,8 @@ module PDK
             @process.environment[k] = v
           end
 
+          check_for_legacy_env_vars
+
           @process.environment['BUNDLE_IGNORE_CONFIG'] = '1'
 
           if [:module, :pwd].include?(context)
@@ -179,7 +199,14 @@ module PDK
             duration: @duration,
           }
 
-          return process_data
+          PDK.logger.debug _('STDOUT: %{output}') % {
+            output: process_data[:stdout].empty? ? 'N/A' : "\n#{process_data[:stdout]}",
+          }
+          PDK.logger.debug _('STDERR: %{output}') % {
+            output: process_data[:stderr].empty? ? 'N/A' : "\n#{process_data[:stderr]}",
+          }
+
+          process_data
         ensure
           @stdout.close
           @stderr.close