knife-ec-backup 2.4.0 → 2.4.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 97caddffcc68ceb8414efdcb07a06dd9e3bd25255bdec00c989cee252d726bba
-  data.tar.gz: e19bc5a8d5927663439cbe2f30026e3b97cc3023bc8836a3726fd9e8a5b0cf23
+  metadata.gz: ddb6a9c53f5ef60525564d583b9906c746c3af34b8fe25fe92d8e0f362d1f039
+  data.tar.gz: '04987273e967c37af6b3738bac962349f07edc5a4db1a0ec13c0004093adc8af'
 SHA512:
-  metadata.gz: f4148c8d37140bf17728bfe6212ee7136e25c2d8e9f841ce419787be577c9c8cbc2d0650c7a8e28c2bf2b6db4e461f866cb96002942580593297330d7f74b679
-  data.tar.gz: a64790d2b77e791985b439939ec442223e89a7f47b6b2af44c83da323d9da58185f4059074c5c6da3cd82bba2071cd33cedcef0410a599a4fae96797b40aaac3
+  metadata.gz: ab40c44efddf1418bf52ed6d329b4e8d4419c927dbdcbfc758440d53b178183747c333c9019bbaec16b59750ec0432e9bf7397bcd651e978445abfe2f4515b4d
+  data.tar.gz: b5a4137aca897ffcf811e1c96fc87b3ffeb435337ac1aaf1a2301ed9ea0683983da8c1991daa9ce5413d6551453de98e2c99c6f8cb1156420f78a5104d9919e6

data/README.md

@@ -1,8 +1,16 @@
-# knife EC backup
-[![Build Status Master](https://travis-ci.org/chef/knife-ec-backup.svg?branch=master)](https://travis-ci.org/chef/knife-ec-backup)
+# Knife EC Backup
+[![Build status](https://badge.buildkite.com/4bc85427aab66accafbd7abb2932b9dd7f9208162c5be33488.svg?branch=master)](https://buildkite.com/chef-oss/chef-knife-ec-backup-master-verify)
 [![Gem Version](https://badge.fury.io/rb/knife-ec-backup.svg)](https://badge.fury.io/rb/knife-ec-backup)
 
-# Description
+**Umbrella Project**: [Knife](https://github.com/chef/chef-oss-practices/blob/master/projects/knife.md)
+
+**Project State**: [Active](https://github.com/chef/chef-oss-practices/blob/master/repo-management/repo-states.md#active)
+
+**Issues [Response Time Maximum](https://github.com/chef/chef-oss-practices/blob/master/repo-management/repo-states.md)**: 14 days
+
+**Pull Request [Response Time Maximum](https://github.com/chef/chef-oss-practices/blob/master/repo-management/repo-states.md)**: 14 days
+
+## Description
 
 knife-ec-backup can backup and restore the data in an Enterprise Chef
 Server installation, preserving the data in an intermediate, editable
@@ -10,29 +18,24 @@ text format.  It is similar to the `knife download` and `knife upload`
 commands and uses the same underlying libraries, but also includes
 workarounds for objects not yet supported by those tools and various
 Server API deficiencies.  The long-run goal is to improve `knife
-download`, `knife upload` and the Chef Server API and deprecate this
+download`, `knife upload` and the Chef Infra Server API and deprecate this
 tool.
 
-# Requirements
+## Requirements
 
-This knife plugin requires Chef Client 11.8+.
+This knife plugin requires Chef Infra Client 11.8+.
 
-## Chef 10
+### Server Support
 
-Users who are still using Chef 10 can use the most recent 1.x version
-of this gem.  Version 1.x additionally depends on knife-essentials.
-
-## Server Support
-
-This plugin currently supports Enterprise Chef 11 and Chef Server 12.
+This plugin currently supports Enterprise Chef 11 and Chef Infra Server 12+.
 Support for the beta key rotation features is provided via the
 `--with-keys-sql` flag, but users of this feature should note that
-this may change once the Chef Server supports an API-based export of
+this may change once the Chef Infra Server supports an API-based export of
 the key data.
 
-# Installation
+## Installation
 
-## Chef Server Install (Recommended)
+### Chef Infra Server Install (Recommended)
 
 This gem is installed with chef-server-core 12.0.0 and newer.
 
@@ -45,7 +48,7 @@ downloading the .zip file. Once unpacked, run:
 /opt/opscode/embedded/bin/gem install knife-ec-backup*gem --no-ri --no-rdoc -V
 ```
 
-### Note on installing with existing development tools:
+#### Note on installing with existing development tools:
 
 The latest versions of knife-ec-backup require gems with native
 extensions, thus you must install a standard build toolchain.  To
@@ -54,6 +57,24 @@ on your system, try the following:
 
    /opt/opscode/embedded/bin/gem install knife-ec-backup -- --with-pg-config=/opt/opscode/embedded/postgresql/9.2/bin/pg_config
 
+This uses the libpq headers that are included in the Chef Infra Server
+package installed in `/opt/opscode`.
+
+### Chef Workstation Install (Unsupported)
+
+On systems other than the Chef Infra Server, installation of this gem is not
+tested or supported. However, if you attempt to do so you will need the
+postgresql libraries installed.
+
+For example, on macOS:
+
+```
+brew install libpq
+gem install knife-ec-backup -- --with-pg-config=/usr/local/Cellar/libpq/9.2/bin/pg_config
+```
+
+The current location of pg_config can be determined with `brew info libpq`.
+
 ## Running tests
 
 ```
@@ -76,25 +97,29 @@ Clone the git repository and run the following from inside:
     gem build knife-ec-backup.gemspec
     gem install knife-ec-backup*gem
 
-# Configuration
+## Configuration
 
-## Permissions
+### Permissions
 
 Note that most users in an EC installation lack the permissions to pull all of the data from all organizations and other users.
-This plugin **REQUIRES THE PIVOTAL KEY AND WEBUI KEY** from the Chef Server.
-It is recommended that you run this from a frontend Enterprise Chef Server, you can use --user and --key to pass the pivotal information along.
+This plugin **REQUIRES THE PIVOTAL KEY AND WEBUI KEY** from the Chef Infra Server.
+It is recommended that you run this from a frontend Enterprise Chef Infra Server, you can use --user and --key to pass the pivotal information along.
 
-# Subcommands
+## Subcommands
 
-## Common Options
+### Common Options
 
 The following options are supported across all subcommands:
 
   * `--sql-host`:
-    The hostname of the Chef Server's postgresql server. (default: localhost)
+    The hostname of the Chef Infra Server's postgresql server. (default: localhost)
 
   * `--sql-port`:
-    The postgresql listening port on the Chef Server. (default: 5432)
+    The postgresql listening port on the Chef Infra Server. (default: 5432)
+
+  * `--sql-db`:
+    The postgresql Chef Infra Server database name. (default: opscode_chef)
+    Specify 'automate-cs-oc-erchef' when using Automate Chef Infra Server API
 
   * `--sql-user`:
     The username of postgresql user with access to the opscode_chef
@@ -110,7 +135,7 @@ The following options are supported across all subcommands:
   * `--dry-run`:
     Report what actions would be taken without performing any. (default: false)
 
-## knife ec backup DEST_DIR (options)
+### knife ec backup DEST_DIR (options)
 
 *Path*: If you have chef-client installed as well, you may need to invoke this as `/opt/opscode/embedded/bin/knife ec backup backup`
 
@@ -132,8 +157,8 @@ The following options are supported across all subcommands:
 
   * `--with-key-sql`: Whether to backup/restore key data directly
     from the database.  This requires access to the listening
-    postgresql port on the Chef Server.  This is required to correctly
-    handle keys in Chef Servers with multikey support. This option
+    postgresql port on the Chef Infra Server.  This is required to correctly
+    handle keys in Chef Infra Servers with multikey support. This option
     will only work on `restore` if it was also used during the
     `backup`.
 
@@ -144,7 +169,7 @@ The following options are supported across all subcommands:
     Chef objects.
 
   * `--skip-version-check`:
-    Skip Chef Server version check. This will also skip any auto-configured options (default: false)
+    Skip Chef Infra Server version check. This will also skip any auto-configured options (default: false)
 
   * `--only-org ORG`:
     Only donwload/restore objects in the named organization. Global
@@ -202,7 +227,7 @@ This compares very closely with the "knife download /" from an OSC server:
     users
       <name>.json>
 
-## knife ec restore DEST_DIR (options)
+### knife ec restore DEST_DIR (options)
 
 Restores all data from the specified DEST_DIR to an Enterprise Chef /
 Private Chef server. DEST_DIR should be a backup directory created by
@@ -227,11 +252,11 @@ Private Chef server. DEST_DIR should be a backup directory created by
     Server. (default: 10)
 
   * `--skip-version-check`:
-    Skip Chef Server version check. This will
+    Skip Chef Infra Server version check. This will
     also skip any auto-configured options (default: false)
 
   * `--[no-]skip-user-ids`:
-    Reuses user ids from the restore destination when updating existing 
+    Reuses user ids from the restore destination when updating existing
     users to avoid database conflicts (default: true)
 
   * `--with-user-sql`:
@@ -244,8 +269,8 @@ Private Chef server. DEST_DIR should be a backup directory created by
 
   * `--with-key-sql`: Whether to backup/restore key data directly
     from the database.  This requires access to the listening
-    postgresql port on the Chef Server.  This is required to correctly
-    handle keys in Chef Servers with multikey support. This option
+    postgresql port on the Chef Infra Server.  This is required to correctly
+    handle keys in Chef Infra Servers with multikey support. This option
     will only work on `restore` if it was also used during the
     `backup`.
 
@@ -256,35 +281,35 @@ Private Chef server. DEST_DIR should be a backup directory created by
     Chef objects.
 
   * `--only-org ORG`:
-    Only donwload/restore objects in the named organization. Global
+    Only download/restore objects in the named organization. Global
     objects such as users will still be downloaded/restored.
 
-## knife ec key export [FILENAME]
+### knife ec key export [FILENAME]
 
-Create a json representation of the users table from the Chef Server
+Create a json representation of the users table from the Chef Infra Server
 database.  If no argument is given, the name of the backup is
 `key_dump.json`.
 
 Please note, most users should use `knife ec backup` with the
 `--with-user-sql` option rather than this command.
 
-## knife ec key import [FILENAME]
+### knife ec key import [FILENAME]
 
 Import a json representation of the users table from FILENAME to the
-the Chef Server database.  If no argument is given, the filename is
+the Chef Infra Server database.  If no argument is given, the filename is
 assumed to be `key_dump.json`.
 
 Please note, most user should use `knife ec restore` with the
 `--with-user-sql` option rather than this command.
 
-# Known Bugs
+## Known Bugs
 
 - knife-ec-backup cannot be installed in the embedded gemset of Chef
-  Server 12.  This will be resolved in a future Chef Server release.
+  Server 12.  This will be resolved in a future Chef Infra Server release.
 
 - `knife ec restore` can fail to restore cookbooks, failing with an
   internal server error. A common cause of this problem is a
-  concurrency bug in Chef Server. Setting `--concurrency 1` can often
+  concurrency bug in Chef Infra Server. Setting `--concurrency 1` can often
   work around the issue.
 
 - `knife ec restore` can fail if the pool of pre-created organizations

data/Rakefile

@@ -1,5 +1,5 @@
 require 'bundler'
-require 'rubygems'
+require 'rubygems' unless defined?(Gem)
 require 'rubygems/package_task'
 require 'rdoc/task'
 require 'rspec/core/rake_task'

data/lib/chef/automate.rb

@@ -0,0 +1,18 @@
+class Chef
+  class Automate
+    def self.is_installed?
+      File.exists?('/hab/svc/automate-cs-oc-erchef/')
+    end
+
+    def self.config
+      {
+        sql_user: 'automate-cs-oc-erchef',
+        sql_cert: '/hab/svc/automate-cs-oc-erchef/config/service.crt',
+        sql_key: '/hab/svc/automate-cs-oc-erchef/config/service.key',
+        sql_rootcert: '/hab/svc/automate-cs-oc-erchef/config/root_ca.crt',
+        sql_db: 'automate-cs-oc-erchef',
+        webui_key: '/hab/svc/automate-cs-oc-erchef/data/webui_priv.pem'
+      }
+    end
+  end
+end

data/lib/chef/knife/ec_backup.rb

@@ -1,5 +1,5 @@
 require 'chef/knife'
-require 'chef/knife/ec_base'
+require_relative 'ec_base'
 
 class Chef
   class Knife
@@ -14,8 +14,8 @@ class Chef
         require 'chef/chef_fs/file_system'
         require 'chef/chef_fs/file_pattern'
         require 'chef/chef_fs/parallelizer'
-        require 'chef/server'
-        require 'fileutils'
+        require_relative '../server'
+        require 'fileutils' unless defined?(FileUtils)
       end
 
       def run
@@ -96,10 +96,10 @@ class Chef
             knife_ec_error_handler.add(ex)
             next
           end
-          # Enterprise Chef 11 and below uses a pool of precreated
+          # Enterprise Chef 11 and below uses a pool of pre-created
           # organizations to account for slow organization creation
           # using CouchDB. Thus, on server versions < 12 we want to
-          # skip any of these precreated organizations by checking if
+          # skip any of these pre-created organizations by checking if
           # they have been assigned or not.  The Chef 12 API does not
           # return an assigned_at field.
           if org['assigned_at'] || server.version >= Gem::Version.new("12")
@@ -127,12 +127,13 @@ class Chef
       end
 
       def export_from_sql
-        require 'chef/knife/ec_key_export'
+        require_relative 'ec_key_export'
         Chef::Knife::EcKeyExport.deps
         k = Chef::Knife::EcKeyExport.new
         k.name_args = ["#{dest_dir}/key_dump.json", "#{dest_dir}/key_table_dump.json"]
         k.config[:sql_host] = config[:sql_host]
         k.config[:sql_port] = config[:sql_port]
+        k.config[:sql_db] = config[:sql_db]
         k.config[:sql_user] = config[:sql_user]
         k.config[:sql_password] = config[:sql_password]
         k.config[:skip_users_table] = !config[:with_user_sql]

data/lib/chef/knife/ec_base.rb

@@ -18,9 +18,10 @@
 
 require 'chef/knife'
 require 'chef/server_api'
-require 'veil'
-require 'chef/knife/ec_error_handler'
-require 'ffi_yajl'
+require 'veil' unless defined?(Veil)
+require_relative 'ec_error_handler'
+require 'ffi_yajl' unless defined?(FFI_Yajl)
+require_relative '../automate'
 
 class Chef
   class Knife
@@ -31,13 +32,17 @@ class Chef
       def self.included(includer)
         includer.class_eval do
 
+          option :error_log_dir,
+            :long => '--error-log-dir PATH',
+            :description => 'Path to a directory where any errors will be logged'
+
           option :concurrency,
             :long => '--concurrency THREADS',
             :description => 'Maximum number of simultaneous requests to send (default: 10)'
 
           option :webui_key,
             :long => '--webui-key KEYPATH',
-            :description => 'Path to the WebUI Key (default: Read from secrets store or /etc/opscode/webui_priv.pem)'
+            :description => 'Path to the WebUI Key (default: Read from secrets store or /etc/opscode/webui_priv.pem or /hab/svc/automate-cs-oc-erchef/data/webui_priv.pem)'
 
           option :secrets_file_path,
             :long => '--secrets-file PATH',
@@ -70,6 +75,10 @@ class Chef
             :description => 'Postgresql database port (default: 5432)',
             :default => 5432
 
+          option :sql_db,
+            :long => '--sql-db DBNAME',
+            :description => 'Postgresql Chef Server database name (default: opscode_chef or automate-cs-oc-erchef)'
+
           option :sql_user,
             :long => "--sql-user USERNAME",
             :description => 'User used to connect to the postgresql database.'
@@ -78,6 +87,18 @@ class Chef
             :long => "--sql-password PASSWORD",
             :description => 'Password used to connect to the postgresql database'
 
+          option :sql_cert,
+            :long => "--sql-cert ",
+            :description => 'Path to client ssl cert'
+
+          option :sql_key,
+            :long => "--sql-key PATH",
+            :description => 'Path to client ssl key'
+
+          option :sql_rootcert,
+          :long => "--sql-rootcert ",
+          :description => 'Path to root ssl cert'
+
           option :with_user_sql,
             :long => '--with-user-sql',
             :description => 'Try direct data base access for user export/import.  Required to properly handle passwords, keys, and USAGs'
@@ -155,7 +176,8 @@ class Chef
       end
 
       def knife_ec_error_handler
-        @knife_ec_error_handler ||= Chef::Knife::EcErrorHandler.new(dest_dir, self.class)
+        error_dir = config[:error_log_dir] || dest_dir
+        @knife_ec_error_handler ||= Chef::Knife::EcErrorHandler.new(error_dir, self.class)
       end
 
       def user_acl_rest
@@ -190,6 +212,8 @@ class Chef
       def webui_key
         if config[:webui_key]
           config[:webui_key]
+        elsif Chef::Automate.is_installed?
+          config[:webui_key] = Chef::Automate.config[:webui_key]
         elsif veil.exist?("chef-server", "webui_key")
           temporary_webui_key
         else

data/lib/chef/knife/ec_key_base.rb

@@ -17,6 +17,7 @@
 #
 
 require 'chef/knife'
+require 'veil'
 
 class Chef
   class Knife
@@ -27,7 +28,8 @@ class Chef
 
           deps do
             require 'sequel'
-            require 'json'
+            require 'json' unless defined?(JSON)
+            require_relative '../automate'
           end
 
           option :sql_host,
@@ -40,6 +42,10 @@ class Chef
           :description => 'Postgresql database port (default: 5432)',
           :default => 5432
 
+          option :sql_db,
+          :long => '--sql-db DBNAME',
+          :description => 'Postgresql Chef Server database name (default: opscode_chef or automate-cs-oc-erchef)'
+
           option :sql_user,
           :long => "--sql-user USERNAME",
           :description => 'User used to connect to the postgresql database.'
@@ -48,6 +54,18 @@ class Chef
           :long => "--sql-password PASSWORD",
           :description => 'Password used to connect to the postgresql database'
 
+          option :sql_cert,
+          :long => "--sql-cert ",
+          :description => 'Path to client ssl cert'
+
+          option :sql_key,
+          :long => "--sql-key PATH",
+          :description => 'Path to client ssl key'
+
+          option :sql_rootcert,
+          :long => "--sql-rootcert ",
+          :description => 'Path to root ssl cert'
+
           option :secrets_file_path,
           :long => '--secrets-file PATH',
           :description => 'Path to a valid private-chef-secrets.json file (default: /etc/opscode/private-chef-secrets.json)',
@@ -68,27 +86,44 @@ class Chef
       def db
         @db ||= begin
                   require 'sequel'
-                  server_string = "#{config[:sql_user]}:#{config[:sql_password]}@#{config[:sql_host]}:#{config[:sql_port]}/opscode_chef"
-                  ::Sequel.connect("postgres://#{server_string}", :convert_infinite_timestamps => :string)
+                  require 'uri'
+                  server_uri = URI('postgres://')
+                  server_uri.host = config[:sql_host]
+                  server_uri.port = config[:sql_port]
+                  server_uri.user = URI.encode_www_form_component(config[:sql_user]) if config[:sql_user]
+                  server_uri.password = URI.encode_www_form_component(config[:sql_password]) if config[:sql_password]
+                  query_params = []
+                  query_params.push("sslcert=#{config[:sql_cert]}") if config[:sql_cert]
+                  query_params.push("sslkey=#{config[:sql_key]}") if config[:sql_key]
+                  query_params.push("sslrootcert=#{config[:sql_rootcert]}") if config[:sql_rootcert]
+                  server_uri.query = query_params.join("&") if query_params.length > 0
+
+                  ::Sequel.connect(server_uri.to_s, :convert_infinite_timestamps => :string)
                 end
       end
 
       # Loads SQL user and password from running config if not passed
       # as a command line option
       def load_config_from_file!
-        if ! File.exists?("/etc/opscode/chef-server-running.json")
-          ui.fatal "SQL User or Password not provided as option and running config cannot be found!"
-          exit 1
+        if Chef::Automate.is_installed?
+          ui.msg "Automate detected"
+          config.merge! Chef::Automate.config {|key, v1, v2| v1}
         else
-          running_config ||= JSON.parse(File.read("/etc/opscode/chef-server-running.json"))
-          # Latest versions of chef server put the database info under opscode-erchef.sql_user
-          hash_key = if running_config['private_chef']['opscode-erchef'].has_key? 'sql_user'
-                       'opscode-erchef'
-                     else
-                       'postgresql'
-                     end
-          config[:sql_user] ||= running_config['private_chef'][hash_key]['sql_user']
-          config[:sql_password] ||= sql_password
+          if ! File.exists?("/etc/opscode/chef-server-running.json")
+            ui.fatal "SQL User or Password not provided as option and running config cannot be found!"
+            exit 1
+          else
+            running_config ||= JSON.parse(File.read("/etc/opscode/chef-server-running.json"))
+            # Latest versions of chef server put the database info under opscode-erchef.sql_user
+            hash_key = if running_config['private_chef']['opscode-erchef'].has_key? 'sql_user'
+                        'opscode-erchef'
+                      else
+                        'postgresql'
+                      end
+            config[:sql_user] ||= running_config['private_chef'][hash_key]['sql_user']
+            config[:sql_password] ||= (running_config['private_chef'][hash_key]['sql_password'] || sql_password)
+            config[:sql_db] ||= 'opscode_chef'
+          end
         end
       end