knife-ec-backup 3.0.5 → 3.0.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +53 -8
- data/lib/chef/knife/ec_base.rb +7 -1
- data/lib/chef/knife/ec_error_handler.rb +1 -1
- data/lib/chef/knife/ec_import.rb +422 -0
- data/lib/chef/knife/ec_key_base.rb +1 -1
- data/lib/chef/knife/ec_key_export.rb +1 -1
- data/lib/chef/knife/ec_key_import.rb +1 -1
- data/lib/chef/knife/ec_restore.rb +58 -0
- data/lib/chef/org_id_cache.rb +1 -1
- data/lib/knife_ec_backup/version.rb +1 -1
- data/spec/chef/knife/ec_import_spec.rb +798 -0
- data/spec/chef/knife/ec_restore_spec.rb +190 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ce4846181eee9032e990d6cf91f487f330c14008839a356ecaac094d1aba7d51
|
|
4
|
+
data.tar.gz: 92561a6aa7f68c1c05fc05ce9125031ce2baf8f6793bc37507de9fa07ce30026
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: d9c42d8b652b129d1fe87ae7f73346837847f92c9c2c89f8e7e12f3fb6bc0bc5c4ebab5691f799484ae5c0588c5e1da4b426228b495cb8566c59cbcb96cb81a8
|
|
7
|
+
data.tar.gz: fa6b5c5710fda71c349c844caec6bdb0b52bc9a564f5295accf4b85b86128e01d74cb9116b9e318b1ece8addc63cb5f2eb24e2473d142ee923862337f74588ef
|
data/README.md
CHANGED
|
@@ -2,14 +2,6 @@
|
|
|
2
2
|
[](https://buildkite.com/chef-oss/chef-knife-ec-backup-main-verify)
|
|
3
3
|
[](https://badge.fury.io/rb/knife-ec-backup)
|
|
4
4
|
|
|
5
|
-
**Umbrella Project**: [Knife](https://github.com/chef/chef-oss-practices/blob/main/projects/knife.md)
|
|
6
|
-
|
|
7
|
-
**Project State**: [Active](https://github.com/chef/chef-oss-practices/blob/main/repo-management/repo-states.md#active)
|
|
8
|
-
|
|
9
|
-
**Issues [Response Time Maximum](https://github.com/chef/chef-oss-practices/blob/main/repo-management/repo-states.md)**: 14 days
|
|
10
|
-
|
|
11
|
-
**Pull Request [Response Time Maximum](https://github.com/chef/chef-oss-practices/blob/main/repo-management/repo-states.md)**: 14 days
|
|
12
|
-
|
|
13
5
|
## Description
|
|
14
6
|
|
|
15
7
|
knife-ec-backup can backup and restore the data in a Chef Infra
|
|
@@ -124,6 +116,9 @@ The following options are supported across all subcommands:
|
|
|
124
116
|
* `--dry-run`:
|
|
125
117
|
Report what actions would be taken without performing any. (default: false)
|
|
126
118
|
|
|
119
|
+
* `--skip-frozen-cookbook-status`:
|
|
120
|
+
Skip backing up and restoring cookbook frozen status. When this option is used, cookbook frozen status will not be preserved during backup/restore operations. (default: false)
|
|
121
|
+
|
|
127
122
|
### knife ec backup DEST_DIR (options)
|
|
128
123
|
|
|
129
124
|
*Path*: If you have Chef Infra Client installed on this server, you may need to invoke this as `/opt/opscode/bin/knife ec backup BACKUP_DIRECTORY`
|
|
@@ -183,6 +178,7 @@ The format of the repository is based on the `knife-essentials` (`knife download
|
|
|
183
178
|
<name>.json
|
|
184
179
|
cookbooks
|
|
185
180
|
<name>-<version>
|
|
181
|
+
status.json (contains frozen status information)
|
|
186
182
|
data_bags
|
|
187
183
|
<bag name>
|
|
188
184
|
<item name>
|
|
@@ -272,6 +268,37 @@ Restores all data from the specified DEST_DIR to a Chef Infra Server. DEST_DIR s
|
|
|
272
268
|
Only download/restore objects in the named organization. Global
|
|
273
269
|
objects such as users will still be downloaded/restored.
|
|
274
270
|
|
|
271
|
+
* `--skip-frozen-cookbook-status`:
|
|
272
|
+
Skip restoring cookbook frozen status. When this option is used, any `status.json` files in the backup will be ignored and cookbook frozen status will not be applied during restore. (default: false)
|
|
273
|
+
|
|
274
|
+
### knife ec import DIRECTORY (options)
|
|
275
|
+
|
|
276
|
+
Imports Chef objects (cookbooks, roles, nodes, etc.) into **existing** organizations from the specified DIRECTORY. This command is designed for environments where identity (users and organizations) is managed externally (e.g., by IAM in Chef 360 DSM).
|
|
277
|
+
|
|
278
|
+
Unlike `restore`, this command:
|
|
279
|
+
- Does **NOT** create users
|
|
280
|
+
- Does **NOT** create organizations (validates they exist)
|
|
281
|
+
- Does **NOT** import user passwords or keys
|
|
282
|
+
- Does **NOT** support direct database access (`--with-user-sql` / `--with-key-sql`)
|
|
283
|
+
|
|
284
|
+
*Options*
|
|
285
|
+
|
|
286
|
+
* `--concurrency THREAD_COUNT`:
|
|
287
|
+
The maximum number of concurrent requests to make to the Chef
|
|
288
|
+
Server. (default: 10)
|
|
289
|
+
|
|
290
|
+
* `--webui-key`:
|
|
291
|
+
Used to set the path to the WebUI Key (default: /etc/opscode/webui_priv.pem)
|
|
292
|
+
|
|
293
|
+
* `--skip-useracl`:
|
|
294
|
+
Skip restoring User ACLs.
|
|
295
|
+
|
|
296
|
+
* `--only-org ORG`:
|
|
297
|
+
Only import objects in the named organization.
|
|
298
|
+
|
|
299
|
+
* `--skip-frozen-cookbook-status`:
|
|
300
|
+
Skip restoring cookbook frozen status.
|
|
301
|
+
|
|
275
302
|
### knife ec key export [FILENAME]
|
|
276
303
|
|
|
277
304
|
Create a json representation of the users table from the Chef Infra Server
|
|
@@ -290,6 +317,21 @@ assumed to be `key_dump.json`.
|
|
|
290
317
|
Please note, most users should use `knife ec restore` with the
|
|
291
318
|
`--with-user-sql` option rather than this command.
|
|
292
319
|
|
|
320
|
+
## Cookbook Frozen Status
|
|
321
|
+
|
|
322
|
+
This plugin supports backing up and restoring cookbook frozen status. When a cookbook is frozen on the Chef Infra Server, this status is preserved during backup operations by creating a `status.json` file within each cookbook directory that contains the frozen state information.
|
|
323
|
+
|
|
324
|
+
During restore operations, the plugin will automatically restore the frozen status of cookbooks by reading the `status.json` files and applying the frozen state to the Chef Infra Server using the appropriate API calls.
|
|
325
|
+
|
|
326
|
+
### Skipping Frozen Status
|
|
327
|
+
|
|
328
|
+
If you want to skip backing up or restoring cookbook frozen status, you can use the `--skip-frozen-cookbook-status` option with both `knife ec backup` and `knife ec restore` commands. When this option is used:
|
|
329
|
+
|
|
330
|
+
- During backup: `status.json` files will not be created for cookbooks
|
|
331
|
+
- During restore: Any existing `status.json` files will be ignored and cookbook frozen status will not be applied
|
|
332
|
+
|
|
333
|
+
This can be useful when migrating between Chef Infra Server versions that may handle frozen cookbooks differently, or when you specifically want to unfreeze all cookbooks during the restore process.
|
|
334
|
+
|
|
293
335
|
## Known Bugs
|
|
294
336
|
|
|
295
337
|
- `knife ec restore` can fail to restore cookbooks, failing with an
|
|
@@ -297,3 +339,6 @@ Please note, most users should use `knife ec restore` with the
|
|
|
297
339
|
concurrency bug in Chef Infra Server. Setting `--concurrency 1` can often
|
|
298
340
|
work around the issue.
|
|
299
341
|
|
|
342
|
+
# Copyright
|
|
343
|
+
|
|
344
|
+
See [COPYRIGHT.md](./COPYRIGHT.md).
|
data/lib/chef/knife/ec_base.rb
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#
|
|
2
2
|
# Author:: Steven Danna (<steve@getchef.com>)
|
|
3
|
-
# Copyright:: Copyright (c)
|
|
3
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
4
4
|
# License:: Apache License, Version 2.0
|
|
5
5
|
#
|
|
6
6
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
@@ -118,6 +118,12 @@ class Chef
|
|
|
118
118
|
:boolean => true | false,
|
|
119
119
|
:default => false,
|
|
120
120
|
:description => 'Report what actions would be taken without performing any.'
|
|
121
|
+
|
|
122
|
+
option :skip_frozen_cookbook_status,
|
|
123
|
+
:long => '--skip-frozen-cookbook-status',
|
|
124
|
+
:boolean => true | false,
|
|
125
|
+
:default => false,
|
|
126
|
+
:description => "This will skip creating a status.json file for each cookbook, which includes the frozen status. This is useful when you dont want to persist cookbook's frozen status."
|
|
121
127
|
end
|
|
122
128
|
|
|
123
129
|
attr_accessor :dest_dir
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Author:: Jeremy Miller (<jmiller@chef.io>)
|
|
2
|
-
# Copyright:: Copyright (c)
|
|
2
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
3
3
|
# License:: Apache License, Version 2.0
|
|
4
4
|
#
|
|
5
5
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
@@ -0,0 +1,422 @@
|
|
|
1
|
+
require_relative '../tsorter'
|
|
2
|
+
require_relative '../server'
|
|
3
|
+
require 'securerandom' unless defined?(SecureRandom)
|
|
4
|
+
require 'chef/knife'
|
|
5
|
+
require_relative 'ec_base'
|
|
6
|
+
require 'chef/json_compat'
|
|
7
|
+
require 'chef/chef_fs/config'
|
|
8
|
+
require 'chef/chef_fs/file_system'
|
|
9
|
+
require 'chef/chef_fs/file_pattern'
|
|
10
|
+
require 'chef/chef_fs/command_line'
|
|
11
|
+
require 'chef/chef_fs/data_handler/acl_data_handler'
|
|
12
|
+
|
|
13
|
+
begin
|
|
14
|
+
require 'chef/chef_fs/parallelizer'
|
|
15
|
+
rescue LoadError
|
|
16
|
+
require 'chef-utils/parallel_map' unless defined?(ChefUtils::ParallelMap)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
class Chef
|
|
20
|
+
class Knife
|
|
21
|
+
class EcImport < Chef::Knife
|
|
22
|
+
|
|
23
|
+
include Knife::EcBase
|
|
24
|
+
|
|
25
|
+
banner 'knife ec import DIRECTORY'
|
|
26
|
+
|
|
27
|
+
# Constants for duplicated strings
|
|
28
|
+
PUBLIC_KEY_READ_ACCESS_JSON = 'public_key_read_access.json'
|
|
29
|
+
FROZEN_STATUS_KEY = 'frozen?'
|
|
30
|
+
ADMIN_GROUPS = ['admins', 'billing-admins'].freeze
|
|
31
|
+
ADMIN_GROUP_FILES = ['billing-admins.json', 'public_key_read_access.json'].freeze
|
|
32
|
+
NOT_FOUND_STATUS = '404'
|
|
33
|
+
|
|
34
|
+
option :tenant_id_header,
|
|
35
|
+
:long => '--tenant-id TENANT_ID',
|
|
36
|
+
:description => 'Tenant identifier added as X-Tenant-Id header for import requests'
|
|
37
|
+
|
|
38
|
+
deps do
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Helper method to read JSON file from backup directory
|
|
42
|
+
def read_json_file(path)
|
|
43
|
+
JSONCompat.from_json(File.read(path))
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# Helper method to construct organization file path
|
|
47
|
+
def org_file_path(orgname, *path_parts)
|
|
48
|
+
File.join(dest_dir, 'organizations', orgname, *path_parts)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# Helper method to construct organization URL
|
|
52
|
+
def org_url(orgname, *path_parts)
|
|
53
|
+
path = path_parts.join('/')
|
|
54
|
+
path.empty? ? "organizations/#{orgname}" : "organizations/#{orgname}/#{path}"
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Helper method to construct cookbook URL
|
|
58
|
+
def cookbook_url(org_name, cookbook_name, version, params = nil)
|
|
59
|
+
url = org_url(org_name, 'cookbooks', cookbook_name, version)
|
|
60
|
+
params ? "#{url}?#{params}" : url
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Helper method to check if public key read access group exists
|
|
64
|
+
def public_key_read_access_exists?(chef_fs_config, type = 'groups')
|
|
65
|
+
::File.exist?(::File.join(chef_fs_config.local_fs.child_paths[type], 'groups', PUBLIC_KEY_READ_ACCESS_JSON))
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
# Helper method to list ChefFS entries with pattern and filter
|
|
69
|
+
def list_chef_fs_entries(chef_fs_config, pattern, exclude_files = [])
|
|
70
|
+
Chef::ChefFS::FileSystem.list(chef_fs_config.local_fs, Chef::ChefFS::FilePattern.new(pattern))
|
|
71
|
+
.select { |entry| !exclude_files.include?(entry.name) }
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Helper method to get admin groups based on what exists
|
|
75
|
+
def get_admin_groups(chef_fs_config)
|
|
76
|
+
groups = ADMIN_GROUPS.dup
|
|
77
|
+
groups.push('public_key_read_access') if public_key_read_access_exists?(chef_fs_config, 'groups')
|
|
78
|
+
groups
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
# Helper method to get admin group ACL paths
|
|
82
|
+
def get_admin_group_acl_paths(chef_fs_config)
|
|
83
|
+
acl_paths = ['/acls/groups/billing-admins.json']
|
|
84
|
+
acl_paths.push('/acls/groups/public_key_read_access.json') if public_key_read_access_exists?(chef_fs_config, 'acls')
|
|
85
|
+
acl_paths
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
# Override set_client_config! to add Tenant-Id header when provided
|
|
89
|
+
def set_client_config!
|
|
90
|
+
super
|
|
91
|
+
if config[:tenant_id_header]
|
|
92
|
+
Chef::Config.custom_http_headers = (Chef::Config.custom_http_headers || {}).merge({'Tenant-Id' => config[:tenant_id_header]})
|
|
93
|
+
end
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
# Override set_skip_user_acl! to avoid calling server.version
|
|
97
|
+
def set_skip_user_acl!
|
|
98
|
+
# Skip user ACLs only if explicitly requested via --skip-useracl flag
|
|
99
|
+
# Default behavior: assume user ACLs are supported
|
|
100
|
+
config[:skip_useracl] ||= false
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# Override user_acl_rest to avoid calling server.version
|
|
104
|
+
def user_acl_rest
|
|
105
|
+
@user_acl_rest ||= rest
|
|
106
|
+
end
|
|
107
|
+
|
|
108
|
+
def run
|
|
109
|
+
set_dest_dir_from_args!
|
|
110
|
+
set_client_config!
|
|
111
|
+
ensure_webui_key_exists!
|
|
112
|
+
set_skip_user_acl!
|
|
113
|
+
|
|
114
|
+
warn_on_incorrect_clients_group(dest_dir, 'import')
|
|
115
|
+
|
|
116
|
+
# Unlike restore, we do NOT restore users or user SQL data
|
|
117
|
+
# We assume users already exist (managed by IAM)
|
|
118
|
+
|
|
119
|
+
for_each_organization do |orgname|
|
|
120
|
+
ui.msg "Verifying organization[#{orgname}]"
|
|
121
|
+
|
|
122
|
+
# Unlike restore, we do NOT create the organization
|
|
123
|
+
# We validate it exists instead
|
|
124
|
+
unless organization_exists?(orgname)
|
|
125
|
+
ui.error("Organization #{orgname} does not exist. Skipping.")
|
|
126
|
+
next
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Note: We skip importing invitations and adding users to org
|
|
130
|
+
# User org membership is now managed by the platform
|
|
131
|
+
# and invites cannot be acted upon by users
|
|
132
|
+
upload_org_data(orgname)
|
|
133
|
+
end
|
|
134
|
+
|
|
135
|
+
# Unlike restore, we do NOT restore key SQL data
|
|
136
|
+
|
|
137
|
+
if config[:skip_useracl]
|
|
138
|
+
ui.warn('Skipping user ACL update. To update user ACLs, remove --skip-useracl.')
|
|
139
|
+
else
|
|
140
|
+
restore_user_acls
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
completion_banner
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
def organization_exists?(orgname)
|
|
147
|
+
rest.get(org_url(orgname))
|
|
148
|
+
true
|
|
149
|
+
rescue Net::HTTPClientException => ex
|
|
150
|
+
if ex.response.code == NOT_FOUND_STATUS
|
|
151
|
+
false
|
|
152
|
+
else
|
|
153
|
+
knife_ec_error_handler.add(ex)
|
|
154
|
+
false
|
|
155
|
+
end
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def restore_user_acls
|
|
159
|
+
ui.msg 'Restoring user ACLs'
|
|
160
|
+
for_each_user do |name|
|
|
161
|
+
begin
|
|
162
|
+
user_acl = read_json_file(File.join(dest_dir, 'user_acls', "#{name}.json"))
|
|
163
|
+
put_acl(user_acl_rest, "users/#{name}/_acl", user_acl)
|
|
164
|
+
rescue Chef::Exceptions::JSON::ParseError, JSON::ParserError => ex
|
|
165
|
+
ui.warn "Failed to parse user ACL for #{name}: #{ex.message}. Skipping."
|
|
166
|
+
knife_ec_error_handler.add(ex)
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def for_each_user
|
|
172
|
+
Dir.foreach("#{dest_dir}/users") do |filename|
|
|
173
|
+
next if filename !~ /(.+)\.json/
|
|
174
|
+
name = $1
|
|
175
|
+
# We don't have overwrite_pivotal option, but we should probably still skip pivotal if it's in the backup
|
|
176
|
+
# to avoid messing with the system user, although we are only doing ACLs here.
|
|
177
|
+
if name == 'pivotal'
|
|
178
|
+
# In restore, we skip pivotal unless overwrite_pivotal is true.
|
|
179
|
+
# Here we don't have that flag, so we should probably always skip pivotal for safety
|
|
180
|
+
# as we are not managing users.
|
|
181
|
+
next
|
|
182
|
+
end
|
|
183
|
+
yield name
|
|
184
|
+
end
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
def for_each_organization
|
|
188
|
+
Dir.foreach("#{dest_dir}/organizations") do |name|
|
|
189
|
+
next if name == '..' || name == '.' || !File.directory?("#{dest_dir}/organizations/#{name}")
|
|
190
|
+
next unless (config[:org].nil? || config[:org] == name)
|
|
191
|
+
yield name
|
|
192
|
+
end
|
|
193
|
+
end
|
|
194
|
+
|
|
195
|
+
PATHS = %w(chef_repo_path cookbook_path environment_path data_bag_path role_path node_path client_path acl_path group_path container_path)
|
|
196
|
+
def upload_org_data(name)
|
|
197
|
+
old_config = Chef::Config.save
|
|
198
|
+
|
|
199
|
+
begin
|
|
200
|
+
# Clear out paths
|
|
201
|
+
PATHS.each do |path|
|
|
202
|
+
Chef::Config.delete(path.to_sym)
|
|
203
|
+
end
|
|
204
|
+
|
|
205
|
+
Chef::Config.chef_repo_path = "#{dest_dir}/organizations/#{name}"
|
|
206
|
+
Chef::Config.versioned_cookbooks = true
|
|
207
|
+
Chef::Config.chef_server_url = "#{server.root_url}/organizations/#{name}"
|
|
208
|
+
|
|
209
|
+
# Upload the admins, public_key_read_access and billing-admins groups and acls
|
|
210
|
+
ui.msg 'Restoring org admin data'
|
|
211
|
+
chef_fs_config = Chef::ChefFS::Config.new
|
|
212
|
+
|
|
213
|
+
# Handle Admins, Billing Admins and Public Key Read Access separately
|
|
214
|
+
groups = get_admin_groups(chef_fs_config)
|
|
215
|
+
|
|
216
|
+
groups.each do |group|
|
|
217
|
+
restore_group(chef_fs_config, group, :clients => false)
|
|
218
|
+
end
|
|
219
|
+
|
|
220
|
+
acls_groups_paths = get_admin_group_acl_paths(chef_fs_config)
|
|
221
|
+
|
|
222
|
+
acls_groups_paths.each do |acl|
|
|
223
|
+
chef_fs_copy_pattern(acl, chef_fs_config)
|
|
224
|
+
end
|
|
225
|
+
|
|
226
|
+
# For import command, we default to using 'pivotal' as node_name
|
|
227
|
+
# since we assume modern Chef Server (version check is skipped)
|
|
228
|
+
Chef::Config.node_name = config[:skip_version] ? org_admin : 'pivotal'
|
|
229
|
+
|
|
230
|
+
# Restore the entire org skipping the admin data and restoring groups and acls last
|
|
231
|
+
# Also skip:
|
|
232
|
+
# - org.json: organization metadata is not managed by import (org must pre-exist)
|
|
233
|
+
# - members.json, invitations.json: user-org membership is managed by the platform
|
|
234
|
+
# - acls, groups: handled separately below in the correct order
|
|
235
|
+
ui.msg 'Restoring the rest of the org'
|
|
236
|
+
chef_fs_config = Chef::ChefFS::Config.new
|
|
237
|
+
skip_entries = %w(acls groups org.json members.json invitations.json)
|
|
238
|
+
top_level_paths = chef_fs_config.local_fs.children.select { |entry| !skip_entries.include?(entry.name) }.map { |entry| entry.path }
|
|
239
|
+
|
|
240
|
+
# Topologically sort groups for upload
|
|
241
|
+
unsorted_groups = list_chef_fs_entries(chef_fs_config, '/groups/*', ADMIN_GROUP_FILES)
|
|
242
|
+
.filter_map do |entry|
|
|
243
|
+
begin
|
|
244
|
+
JSON.parse(entry.read)
|
|
245
|
+
rescue JSON::ParserError, Chef::Exceptions::JSON::ParseError => e
|
|
246
|
+
ui.warn "Failed to parse group file #{entry.name}: #{e.message}. Skipping."
|
|
247
|
+
knife_ec_error_handler.add(e)
|
|
248
|
+
nil
|
|
249
|
+
end
|
|
250
|
+
end
|
|
251
|
+
group_paths = sort_groups_for_upload(unsorted_groups).map { |group_name| "/groups/#{group_name}.json" }
|
|
252
|
+
|
|
253
|
+
group_acl_paths = list_chef_fs_entries(chef_fs_config, '/acls/groups/*', ADMIN_GROUP_FILES)
|
|
254
|
+
.map { |entry| entry.path }
|
|
255
|
+
acl_paths = Chef::ChefFS::FileSystem.list(chef_fs_config.local_fs, Chef::ChefFS::FilePattern.new('/acls/*'))
|
|
256
|
+
.select { |entry| entry.name != 'groups' }
|
|
257
|
+
.map { |entry| entry.path }
|
|
258
|
+
|
|
259
|
+
# Store organization data in a particular order:
|
|
260
|
+
# - clients must be uploaded before groups (in top_level_paths)
|
|
261
|
+
# - groups must be uploaded before any acl's
|
|
262
|
+
# - groups must be uploaded twice to account for Chef Infra Server versions that don't
|
|
263
|
+
# accept group members on POST
|
|
264
|
+
(top_level_paths + group_paths*2 + group_acl_paths + acl_paths).each do |path|
|
|
265
|
+
chef_fs_copy_pattern(path, chef_fs_config)
|
|
266
|
+
end
|
|
267
|
+
|
|
268
|
+
# Apply frozen status to cookbooks after they are uploaded
|
|
269
|
+
restore_cookbook_frozen_status(name, chef_fs_config)
|
|
270
|
+
|
|
271
|
+
# restore clients to groups, using the pivotal user again
|
|
272
|
+
Chef::Config[:node_name] = 'pivotal'
|
|
273
|
+
groups.each do |group|
|
|
274
|
+
restore_group(Chef::ChefFS::Config.new, group)
|
|
275
|
+
end
|
|
276
|
+
ensure
|
|
277
|
+
Chef::Config.restore(old_config)
|
|
278
|
+
end
|
|
279
|
+
end
|
|
280
|
+
|
|
281
|
+
def chef_fs_copy_pattern(pattern_str, chef_fs_config)
|
|
282
|
+
ui.msg "Copying #{pattern_str}"
|
|
283
|
+
pattern = Chef::ChefFS::FilePattern.new(pattern_str)
|
|
284
|
+
Chef::ChefFS::FileSystem.copy_to(pattern, chef_fs_config.local_fs,
|
|
285
|
+
chef_fs_config.chef_fs, nil,
|
|
286
|
+
config, ui,
|
|
287
|
+
proc { |entry| chef_fs_config.format_path(entry) })
|
|
288
|
+
rescue Net::HTTPClientException,
|
|
289
|
+
Chef::ChefFS::FileSystem::NotFoundError,
|
|
290
|
+
Chef::ChefFS::FileSystem::OperationFailedError,
|
|
291
|
+
Chef::Exceptions::JSON::ParseError,
|
|
292
|
+
JSON::ParserError => ex
|
|
293
|
+
ui.error "#{pattern_str} failed to copy: #{ex.message}"
|
|
294
|
+
knife_ec_error_handler.add(ex)
|
|
295
|
+
end
|
|
296
|
+
|
|
297
|
+
def sort_groups_for_upload(groups)
|
|
298
|
+
Chef::Tsorter.new(group_array_to_sortable_hash(groups)).tsort
|
|
299
|
+
end
|
|
300
|
+
|
|
301
|
+
def group_array_to_sortable_hash(groups)
|
|
302
|
+
ret = {}
|
|
303
|
+
groups.each do |group|
|
|
304
|
+
name = group['name']
|
|
305
|
+
ret[name] = if group.key?('groups')
|
|
306
|
+
group['groups']
|
|
307
|
+
else
|
|
308
|
+
[]
|
|
309
|
+
end
|
|
310
|
+
end
|
|
311
|
+
ret
|
|
312
|
+
end
|
|
313
|
+
|
|
314
|
+
def restore_group(chef_fs_config, group_name, includes = {:users => true, :clients => true})
|
|
315
|
+
includes[:users] = true unless includes.key? :users
|
|
316
|
+
includes[:clients] = true unless includes.key? :clients
|
|
317
|
+
|
|
318
|
+
ui.msg "Copying /groups/#{group_name}.json"
|
|
319
|
+
group = Chef::ChefFS::FileSystem.resolve_path(
|
|
320
|
+
chef_fs_config.chef_fs,
|
|
321
|
+
"/groups/#{group_name}.json"
|
|
322
|
+
)
|
|
323
|
+
|
|
324
|
+
# Will throw NotFoundError if JSON file does not exist on disk. See below.
|
|
325
|
+
members_json = Chef::ChefFS::FileSystem.resolve_path(
|
|
326
|
+
chef_fs_config.local_fs,
|
|
327
|
+
"/groups/#{group_name}.json"
|
|
328
|
+
).read
|
|
329
|
+
|
|
330
|
+
members = JSON.parse(members_json).select do |member|
|
|
331
|
+
if includes[:users] && includes[:clients]
|
|
332
|
+
member
|
|
333
|
+
elsif includes[:users]
|
|
334
|
+
member == 'users'
|
|
335
|
+
elsif includes[:clients]
|
|
336
|
+
member == 'clients'
|
|
337
|
+
end
|
|
338
|
+
end
|
|
339
|
+
|
|
340
|
+
group.write(members.to_json)
|
|
341
|
+
rescue Chef::ChefFS::FileSystem::NotFoundError
|
|
342
|
+
Chef::Log.warn "Could not find #{group.display_path} on disk. Will not restore."
|
|
343
|
+
rescue JSON::ParserError, Chef::Exceptions::JSON::ParseError => ex
|
|
344
|
+
ui.warn "Failed to parse group file #{group_name}.json: #{ex.message}. Skipping group restore."
|
|
345
|
+
knife_ec_error_handler.add(ex)
|
|
346
|
+
end
|
|
347
|
+
|
|
348
|
+
def restore_cookbook_frozen_status(org_name, chef_fs_config)
|
|
349
|
+
return if config[:skip_frozen_cookbook_status]
|
|
350
|
+
|
|
351
|
+
ui.msg 'Restoring cookbook frozen status'
|
|
352
|
+
cookbooks_path = org_file_path(org_name, 'cookbooks')
|
|
353
|
+
|
|
354
|
+
return unless File.directory?(cookbooks_path)
|
|
355
|
+
|
|
356
|
+
Dir.foreach(cookbooks_path) do |cookbook_entry|
|
|
357
|
+
next if cookbook_entry == '.' || cookbook_entry == '..'
|
|
358
|
+
|
|
359
|
+
process_cookbook_frozen_status(cookbooks_path, cookbook_entry, org_name)
|
|
360
|
+
end
|
|
361
|
+
end
|
|
362
|
+
|
|
363
|
+
def process_cookbook_frozen_status(cookbooks_path, cookbook_entry, org_name)
|
|
364
|
+
cookbook_path = File.join(cookbooks_path, cookbook_entry)
|
|
365
|
+
return unless File.directory?(cookbook_path)
|
|
366
|
+
|
|
367
|
+
# cookbook_entry is in format "cookbook_name-version"
|
|
368
|
+
# Extract cookbook name and version
|
|
369
|
+
# Use character class negation to prevent backtracking (ReDoS)
|
|
370
|
+
# Match: name-X.Y.Z or name-X.Y.Z.suffix (e.g., mycb-1.0.0 or mycb-1.0.0.beta1)
|
|
371
|
+
return unless cookbook_entry =~ /^([^-]+(?:-[^-]+)*?)-(\d+\.\d+\.\d+(?:\.[^.]+)*)$/
|
|
372
|
+
|
|
373
|
+
cookbook_name = $1
|
|
374
|
+
version = $2
|
|
375
|
+
|
|
376
|
+
status_file = File.join(cookbook_path, 'status.json')
|
|
377
|
+
return unless File.exist?(status_file)
|
|
378
|
+
|
|
379
|
+
begin
|
|
380
|
+
status_data = JSON.parse(File.read(status_file))
|
|
381
|
+
freeze_cookbook(cookbook_name, version, org_name) if status_data['frozen'] == true
|
|
382
|
+
rescue JSON::ParserError => e
|
|
383
|
+
ui.warn "Failed to parse status.json for #{cookbook_name} #{version}: #{e.message}"
|
|
384
|
+
rescue => e
|
|
385
|
+
ui.warn "Failed to restore frozen status for #{cookbook_name} #{version}: #{e.message}"
|
|
386
|
+
end
|
|
387
|
+
end
|
|
388
|
+
|
|
389
|
+
def freeze_cookbook(cookbook_name, version, org_name)
|
|
390
|
+
ui.msg "Freezing cookbook #{cookbook_name} version #{version}"
|
|
391
|
+
|
|
392
|
+
# Get the current cookbook manifest
|
|
393
|
+
manifest = rest.get(cookbook_url(org_name, cookbook_name, version))
|
|
394
|
+
|
|
395
|
+
if manifest[FROZEN_STATUS_KEY] # Ignore if already frozen
|
|
396
|
+
ui.warn "Freezing cookbook #{cookbook_name} version #{version} skipped since it is already frozen!"
|
|
397
|
+
return
|
|
398
|
+
end
|
|
399
|
+
|
|
400
|
+
rest.put(cookbook_url(org_name, cookbook_name, version, 'freeze=true'),
|
|
401
|
+
manifest.tap { |h| h[FROZEN_STATUS_KEY] = true })
|
|
402
|
+
rescue Net::HTTPClientException => ex
|
|
403
|
+
ui.warn "Failed to freeze cookbook #{cookbook_name} #{version}: #{ex.message}"
|
|
404
|
+
knife_ec_error_handler.add(ex)
|
|
405
|
+
end
|
|
406
|
+
|
|
407
|
+
PERMISSIONS = %w{create read update delete grant}.freeze
|
|
408
|
+
def put_acl(rest, url, acls)
|
|
409
|
+
old_acls = rest.get(url)
|
|
410
|
+
old_acls = Chef::ChefFS::DataHandler::AclDataHandler.new.normalize(old_acls, nil)
|
|
411
|
+
acls = Chef::ChefFS::DataHandler::AclDataHandler.new.normalize(acls, nil)
|
|
412
|
+
if acls != old_acls
|
|
413
|
+
PERMISSIONS.each do |permission|
|
|
414
|
+
rest.put("#{url}/#{permission}", { permission => acls[permission] })
|
|
415
|
+
end
|
|
416
|
+
end
|
|
417
|
+
rescue Net::HTTPClientException => ex
|
|
418
|
+
knife_ec_error_handler.add(ex)
|
|
419
|
+
end
|
|
420
|
+
end
|
|
421
|
+
end
|
|
422
|
+
end
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#
|
|
2
2
|
# Author:: Steven Danna (<steve@getchef.com>)
|
|
3
|
-
# Copyright:: Copyright (c)
|
|
3
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
4
4
|
# License:: Apache License, Version 2.0
|
|
5
5
|
#
|
|
6
6
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#
|
|
2
2
|
# Author:: Steven Danna (<steve@getchef.com>)
|
|
3
|
-
# Copyright:: Copyright (c)
|
|
3
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
4
4
|
# License:: Apache License, Version 2.0
|
|
5
5
|
#
|
|
6
6
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#
|
|
2
2
|
# Author:: Steven Danna (<steve@getchef.com>)
|
|
3
|
-
# Copyright:: Copyright (c)
|
|
3
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
4
4
|
# License:: Apache License, Version 2.0
|
|
5
5
|
#
|
|
6
6
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
@@ -298,6 +298,9 @@ class Chef
|
|
|
298
298
|
chef_fs_copy_pattern(path, chef_fs_config)
|
|
299
299
|
end
|
|
300
300
|
|
|
301
|
+
# Apply frozen status to cookbooks after they are uploaded
|
|
302
|
+
restore_cookbook_frozen_status(name, chef_fs_config)
|
|
303
|
+
|
|
301
304
|
# restore clients to groups, using the pivotal user again
|
|
302
305
|
Chef::Config[:node_name] = 'pivotal'
|
|
303
306
|
groups.each do |group|
|
|
@@ -377,6 +380,61 @@ class Chef
|
|
|
377
380
|
Chef::Log.warn "Could not find #{group.display_path} on disk. Will not restore."
|
|
378
381
|
end
|
|
379
382
|
|
|
383
|
+
# Restore cookbook frozen status from status.json files
|
|
384
|
+
def restore_cookbook_frozen_status(org_name, chef_fs_config)
|
|
385
|
+
return if config[:skip_frozen_cookbook_status]
|
|
386
|
+
|
|
387
|
+
ui.msg "Restoring cookbook frozen status"
|
|
388
|
+
cookbooks_path = "#{dest_dir}/organizations/#{org_name}/cookbooks"
|
|
389
|
+
|
|
390
|
+
return unless File.directory?(cookbooks_path)
|
|
391
|
+
|
|
392
|
+
Dir.foreach(cookbooks_path) do |cookbook_entry|
|
|
393
|
+
next if cookbook_entry == '.' || cookbook_entry == '..'
|
|
394
|
+
cookbook_path = File.join(cookbooks_path, cookbook_entry)
|
|
395
|
+
next unless File.directory?(cookbook_path)
|
|
396
|
+
|
|
397
|
+
# cookbook_entry is in format "cookbook_name-version"
|
|
398
|
+
# Extract cookbook name and version
|
|
399
|
+
if cookbook_entry =~ /^(.+)-(\d+\.\d+\.\d+.*)$/
|
|
400
|
+
cookbook_name = $1
|
|
401
|
+
version = $2
|
|
402
|
+
|
|
403
|
+
status_file = File.join(cookbook_path, 'status.json')
|
|
404
|
+
next unless File.exist?(status_file)
|
|
405
|
+
|
|
406
|
+
begin
|
|
407
|
+
status_data = JSON.parse(File.read(status_file))
|
|
408
|
+
if status_data['frozen'] == true
|
|
409
|
+
freeze_cookbook(cookbook_name, version, org_name)
|
|
410
|
+
end
|
|
411
|
+
rescue JSON::ParserError => e
|
|
412
|
+
ui.warn "Failed to parse status.json for #{cookbook_name} #{version}: #{e.message}"
|
|
413
|
+
rescue => e
|
|
414
|
+
ui.warn "Failed to restore frozen status for #{cookbook_name} #{version}: #{e.message}"
|
|
415
|
+
end
|
|
416
|
+
end
|
|
417
|
+
end
|
|
418
|
+
end
|
|
419
|
+
|
|
420
|
+
# Freeze a cookbook on the Chef Server
|
|
421
|
+
def freeze_cookbook(cookbook_name, version, org_name)
|
|
422
|
+
ui.msg "Freezing cookbook #{cookbook_name} version #{version}"
|
|
423
|
+
|
|
424
|
+
# Get the current cookbook manifest
|
|
425
|
+
manifest = rest.get("organizations/#{org_name}/cookbooks/#{cookbook_name}/#{version}")
|
|
426
|
+
|
|
427
|
+
if manifest['frozen?'] # Ignore if already frozen
|
|
428
|
+
ui.warn "Freezing cookbook #{cookbook_name} version #{version} skipped since it is already frozen!"
|
|
429
|
+
return
|
|
430
|
+
end
|
|
431
|
+
|
|
432
|
+
rest.put("organizations/#{org_name}/cookbooks/#{cookbook_name}/#{version}?freeze=true", manifest.tap { |h| h["frozen?"] = true })
|
|
433
|
+
rescue Net::HTTPClientException => ex
|
|
434
|
+
ui.warn "Failed to freeze cookbook #{cookbook_name} #{version}: #{ex.message}"
|
|
435
|
+
knife_ec_error_handler.add(ex)
|
|
436
|
+
end
|
|
437
|
+
|
|
380
438
|
PERMISSIONS = %w{create read update delete grant}.freeze
|
|
381
439
|
def put_acl(rest, url, acls)
|
|
382
440
|
old_acls = rest.get(url)
|
data/lib/chef/org_id_cache.rb
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#
|
|
2
2
|
# Author:: Steven Danna (<steve@chef.io>)
|
|
3
|
-
# Copyright:: Copyright (c)
|
|
3
|
+
# Copyright:: Copyright (c) 2013-2025 Progress Software Corporation and/or its subsidiaries or affiliates. All Rights Reserved.
|
|
4
4
|
# License:: Apache License, Version 2.0
|
|
5
5
|
#
|
|
6
6
|
# Licensed under the Apache License, Version 2.0 (the "License");
|