winrm-fs 0.4.2 → 0.4.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 851dbc1869fdb3ef1a0a8766879d3f61880fc23e
4
- data.tar.gz: 78887cda8faccff607c3a48e58e3137351cc5966
3
+ metadata.gz: 7cb6a97dc554ab9436a28969f3ea70c8771c0858
4
+ data.tar.gz: 4578906a4b74cffeb2f00bfe362dd059a7b33db6
5
5
  SHA512:
6
- metadata.gz: c80764211e5766b36369a57963148300f457c56a2b6a7ab96cb20aab1a2217414a523a6c05599d46ca19ecc88ac00fb292c32a5ffbf27dd404d844921f087d55
7
- data.tar.gz: 0bfef10bad73e39c84bc550aa83283b93de8366577af84460c4d1ef52bf8ed8c0e9054ae26955e2485fc8c78510c27edac0747eff49927a78fd9fcacaa1723bb
6
+ metadata.gz: babcce6f4e58ad60de51a34844149c551edcc61201b1f358472b2796b2d4a91759ae7d0ae878afb88f20a500f1e4928132dcbf0997ddcadd9299c514ee2a0811
7
+ data.tar.gz: c92f3204daef1eb4d47d56f174651adb1640b57bd17664518f05ffd7d795738942fd8c1f8748d3c1fbb98a3f745dd289844fe8dfeed7d7198696ac3b1f38e84c
data/README.md CHANGED
@@ -1,86 +1,86 @@
1
- # File system operations over Windows Remote Management (WinRM) for Ruby
2
- [![Build Status](https://travis-ci.org/WinRb/winrm-fs.svg?branch=master)](https://travis-ci.org/WinRb/winrm-fs)
3
- [![Gem Version](https://badge.fury.io/rb/winrm-fs.svg)](http://badge.fury.io/rb/winrm-fs)
4
- [![Build status](https://ci.appveyor.com/api/projects/status/wm6apa8ojfhfmwsf?svg=true)](https://ci.appveyor.com/project/winrb/winrm-fs)
5
-
6
- ## Uploading files
7
- Files may be copied from the local machine to the winrm endpoint. Individual files or directories, as well as arrays of files and directories may be specified:
8
- ```ruby
9
- require 'winrm-fs'
10
-
11
- service = WinRM::WinRMWebService.new(...
12
- file_manager = WinRM::FS::FileManager.new(service)
13
-
14
- # upload file.txt from the current working directory
15
- file_manager.upload('file.txt', 'c:/file.txt')
16
-
17
- # upload the my_dir directory to c:/foo/my_dir
18
- file_manager.upload('/Users/sneal/my_dir', 'c:/foo/my_dir')
19
-
20
- # upload multiple directories and a file to c:\programData
21
- file_manager.upload([
22
- '/Users/sneal/foo1',
23
- '/Users/sneal/foo2'
24
- '/Users/sneal/fluffy.txt'
25
- ], '$env:ProgramData')
26
- ```
27
-
28
- ### Handling progress events
29
- If you want to implement your own custom progress handling, you can pass a code
30
- block and use the proggress data that `upload` yields to this block:
31
- ```ruby
32
- file_manager.upload('c:/dev/my_dir', '$env:AppData') do |bytes_copied, total_bytes, local_path, remote_path|
33
- puts "#{bytes_copied}bytes of #{total_bytes}bytes copied"
34
- end
35
- ```
36
-
37
- ## Troubleshooting
38
-
39
- If you're having trouble, first of all its most likely a network or WinRM configuration
40
- issue. Take a look at the [WinRM gem troubleshooting](https://github.com/WinRb/WinRM#troubleshooting)
41
- first.
42
-
43
- The most [common error](https://github.com/WinRb/winrm-fs/issues/1) with this gem is getting a 500 error because your maxConcurrentOperationsPerUser limit has been reached.
44
-
45
- ```
46
- The WS-Management service cannot process the request. This user is allowed a
47
- maximum number of 1500 concurrent operations, which has been exceeded. Close
48
- existing operations for this user, or raise the quota for this user.
49
- ```
50
-
51
- You can workaround this by increasing your operations per user quota.
52
-
53
- ## Contributing
54
-
55
- 1. Fork it.
56
- 2. Create a branch (git checkout -b my_feature_branch)
57
- 3. Run the unit and integration tests (bundle exec rake integration)
58
- 4. Commit your changes (git commit -am "Added a sweet feature")
59
- 5. Push to the branch (git push origin my_feature_branch)
60
- 6. Create a pull requst from your branch into master (Please be sure to provide enough detail for us to cipher what this change is doing)
61
-
62
- ### Running the tests
63
-
64
- We use Bundler to manage dependencies during development.
65
-
66
- ```
67
- $ bundle install
68
- ```
69
-
70
- Once you have the dependencies, you can run the unit tests with `rake`:
71
-
72
- ```
73
- $ bundle exec rake spec
74
- ```
75
-
76
- To run the integration tests you will need a Windows box with the WinRM service properly configured. Its easiest to use the Vagrant Windows box in the Vagrantilfe of this repo.
77
-
78
- 1. Create a Windows VM with WinRM configured (see above).
79
- 2. Copy the config-example.yml to config.yml - edit this file with your WinRM connection details.
80
- 3. Run `bundle exec rake integration`
81
-
82
- ## WinRM-fs Authors
83
- * Shawn Neal (https://github.com/sneal)
84
- * Matt Wrock (https://github.com/mwrock)
85
-
86
- [Contributors](https://github.com/WinRb/winrm-fs/graphs/contributors)
1
+ # File system operations over Windows Remote Management (WinRM) for Ruby
2
+ [![Build Status](https://travis-ci.org/WinRb/winrm-fs.svg?branch=master)](https://travis-ci.org/WinRb/winrm-fs)
3
+ [![Gem Version](https://badge.fury.io/rb/winrm-fs.svg)](http://badge.fury.io/rb/winrm-fs)
4
+ [![Build status](https://ci.appveyor.com/api/projects/status/wm6apa8ojfhfmwsf?svg=true)](https://ci.appveyor.com/project/winrb/winrm-fs)
5
+
6
+ ## Uploading files
7
+ Files may be copied from the local machine to the winrm endpoint. Individual files or directories, as well as arrays of files and directories may be specified:
8
+ ```ruby
9
+ require 'winrm-fs'
10
+
11
+ service = WinRM::WinRMWebService.new(...
12
+ file_manager = WinRM::FS::FileManager.new(service)
13
+
14
+ # upload file.txt from the current working directory
15
+ file_manager.upload('file.txt', 'c:/file.txt')
16
+
17
+ # upload the my_dir directory to c:/foo/my_dir
18
+ file_manager.upload('/Users/sneal/my_dir', 'c:/foo/my_dir')
19
+
20
+ # upload multiple directories and a file to c:\programData
21
+ file_manager.upload([
22
+ '/Users/sneal/foo1',
23
+ '/Users/sneal/foo2'
24
+ '/Users/sneal/fluffy.txt'
25
+ ], '$env:ProgramData')
26
+ ```
27
+
28
+ ### Handling progress events
29
+ If you want to implement your own custom progress handling, you can pass a code
30
+ block and use the proggress data that `upload` yields to this block:
31
+ ```ruby
32
+ file_manager.upload('c:/dev/my_dir', '$env:AppData') do |bytes_copied, total_bytes, local_path, remote_path|
33
+ puts "#{bytes_copied}bytes of #{total_bytes}bytes copied"
34
+ end
35
+ ```
36
+
37
+ ## Troubleshooting
38
+
39
+ If you're having trouble, first of all its most likely a network or WinRM configuration
40
+ issue. Take a look at the [WinRM gem troubleshooting](https://github.com/WinRb/WinRM#troubleshooting)
41
+ first.
42
+
43
+ The most [common error](https://github.com/WinRb/winrm-fs/issues/1) with this gem is getting a 500 error because your maxConcurrentOperationsPerUser limit has been reached.
44
+
45
+ ```
46
+ The WS-Management service cannot process the request. This user is allowed a
47
+ maximum number of 1500 concurrent operations, which has been exceeded. Close
48
+ existing operations for this user, or raise the quota for this user.
49
+ ```
50
+
51
+ You can workaround this by increasing your operations per user quota.
52
+
53
+ ## Contributing
54
+
55
+ 1. Fork it.
56
+ 2. Create a branch (git checkout -b my_feature_branch)
57
+ 3. Run the unit and integration tests (bundle exec rake integration)
58
+ 4. Commit your changes (git commit -am "Added a sweet feature")
59
+ 5. Push to the branch (git push origin my_feature_branch)
60
+ 6. Create a pull requst from your branch into master (Please be sure to provide enough detail for us to cipher what this change is doing)
61
+
62
+ ### Running the tests
63
+
64
+ We use Bundler to manage dependencies during development.
65
+
66
+ ```
67
+ $ bundle install
68
+ ```
69
+
70
+ Once you have the dependencies, you can run the unit tests with `rake`:
71
+
72
+ ```
73
+ $ bundle exec rake spec
74
+ ```
75
+
76
+ To run the integration tests you will need a Windows box with the WinRM service properly configured. Its easiest to use the Vagrant Windows box in the Vagrantilfe of this repo.
77
+
78
+ 1. Create a Windows VM with WinRM configured (see above).
79
+ 2. Copy the config-example.yml to config.yml - edit this file with your WinRM connection details.
80
+ 3. Run `bundle exec rake integration`
81
+
82
+ ## WinRM-fs Authors
83
+ * Shawn Neal (https://github.com/sneal)
84
+ * Matt Wrock (https://github.com/mwrock)
85
+
86
+ [Contributors](https://github.com/WinRb/winrm-fs/graphs/contributors)
data/Rakefile CHANGED
@@ -1,28 +1,28 @@
1
- # encoding: UTF-8
2
- require 'rubygems'
3
- require 'bundler/setup'
4
- require 'rspec/core/rake_task'
5
- require 'rubocop/rake_task'
6
-
7
- # Change to the directory of this file.
8
- Dir.chdir(File.expand_path('../', __FILE__))
9
-
10
- # For gem creation and bundling
11
- require 'bundler/gem_tasks'
12
-
13
- RSpec::Core::RakeTask.new(:spec) do |task|
14
- task.pattern = 'spec/unit/*_spec.rb'
15
- task.rspec_opts = ['--color', '-f documentation']
16
- end
17
-
18
- # Run the integration test suite
19
- RSpec::Core::RakeTask.new(:integration) do |task|
20
- task.pattern = 'spec/integration/*_spec.rb'
21
- task.rspec_opts = ['--color', '-f documentation']
22
- end
23
-
24
- RuboCop::RakeTask.new
25
-
26
- task default: [:spec, :rubocop]
27
-
28
- task all: [:default, :integration]
1
+ # encoding: UTF-8
2
+ require 'rubygems'
3
+ require 'bundler/setup'
4
+ require 'rspec/core/rake_task'
5
+ require 'rubocop/rake_task'
6
+
7
+ # Change to the directory of this file.
8
+ Dir.chdir(File.expand_path('../', __FILE__))
9
+
10
+ # For gem creation and bundling
11
+ require 'bundler/gem_tasks'
12
+
13
+ RSpec::Core::RakeTask.new(:spec) do |task|
14
+ task.pattern = 'spec/unit/*_spec.rb'
15
+ task.rspec_opts = ['--color', '-f documentation']
16
+ end
17
+
18
+ # Run the integration test suite
19
+ RSpec::Core::RakeTask.new(:integration) do |task|
20
+ task.pattern = 'spec/integration/*_spec.rb'
21
+ task.rspec_opts = ['--color', '-f documentation']
22
+ end
23
+
24
+ RuboCop::RakeTask.new
25
+
26
+ task default: [:spec, :rubocop]
27
+
28
+ task all: [:default, :integration]
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.4.2
1
+ 0.4.3
data/appveyor.yml CHANGED
@@ -1,39 +1,39 @@
1
- version: "master-{build}"
2
-
3
- os: Windows Server 2012 R2
4
- platform:
5
- - x64
6
-
7
- environment:
8
- winrm_user: test_user
9
- winrm_pass: Pass@word1
10
-
11
- matrix:
12
- - ruby_version: "21"
13
- winrm_endpoint: http://localhost:5985/wsman
14
-
15
- clone_folder: c:\projects\winrm-fs
16
- clone_depth: 1
17
- branches:
18
- only:
19
- - master
20
-
21
- install:
22
- - ps: net user /add $env:winrm_user $env:winrm_pass
23
- - ps: net localgroup administrators $env:winrm_user /add
24
- - ps: winrm set winrm/config/client/auth '@{Basic="true"}'
25
- - ps: winrm set winrm/config/service/auth '@{Basic="true"}'
26
- - ps: winrm set winrm/config/service '@{AllowUnencrypted="true"}'
27
- - ps: $env:PATH="C:\Ruby$env:ruby_version\bin;$env:PATH"
28
- - ps: Write-Host $env:PATH
29
- - ps: ruby --version
30
- - ps: gem --version
31
- - ps: gem install bundler --quiet --no-ri --no-rdoc
32
- - ps: bundler --version
33
-
34
- build_script:
35
- - bundle install || bundle install || bundle install
36
-
37
- test_script:
38
- - SET SPEC_OPTS=--format progress
39
- - bundle exec rake integration
1
+ version: "master-{build}"
2
+
3
+ os: Windows Server 2012 R2
4
+ platform:
5
+ - x64
6
+
7
+ environment:
8
+ winrm_user: test_user
9
+ winrm_pass: Pass@word1
10
+
11
+ matrix:
12
+ - ruby_version: "21"
13
+ winrm_endpoint: http://localhost:5985/wsman
14
+
15
+ clone_folder: c:\projects\winrm-fs
16
+ clone_depth: 1
17
+ branches:
18
+ only:
19
+ - master
20
+
21
+ install:
22
+ - ps: net user /add $env:winrm_user $env:winrm_pass
23
+ - ps: net localgroup administrators $env:winrm_user /add
24
+ - ps: winrm set winrm/config/client/auth '@{Basic="true"}'
25
+ - ps: winrm set winrm/config/service/auth '@{Basic="true"}'
26
+ - ps: winrm set winrm/config/service '@{AllowUnencrypted="true"}'
27
+ - ps: $env:PATH="C:\Ruby$env:ruby_version\bin;$env:PATH"
28
+ - ps: Write-Host $env:PATH
29
+ - ps: ruby --version
30
+ - ps: gem --version
31
+ - ps: gem install bundler --quiet --no-ri --no-rdoc
32
+ - ps: bundler --version
33
+
34
+ build_script:
35
+ - bundle install || bundle install || bundle install
36
+
37
+ test_script:
38
+ - SET SPEC_OPTS=--format progress
39
+ - bundle exec rake integration
data/changelog.md CHANGED
@@ -1,43 +1,46 @@
1
- # WinRM-fs Gem Changelog
2
-
3
- # 0.4.2
4
- - Improved Powershell error handling in metadata checking.
5
-
6
- # 0.4.1
7
- - Fixes a regression on Windows 2008 R2/Windows 7 and below where the WinRM service corrupts the check files metadata resulting in malformed destination paths.
8
-
9
- # 0.4.0
10
- - Correct the destination path of individual files. Always assume it is the full destination path unless it is an existing directory. This may potentialy break some callers expecting the remote path to be a directory that winrm-fs will create if missing as the destination of the local file. A new directory will not be created and the local file will be uploaded directly to the remote path.
11
-
12
- # 0.3.2
13
- - Fix re-extraction of cached directories from temp folder when there is more than one "clean" directory deleted from destination
14
-
15
- # 0.3.1
16
- - Widen logging version constraints to include 2.0 (matching WinRM core gem)
17
-
18
- # 0.3.0
19
- - Jetisons `CommandExecutor` now living in the core WinRM gem and swaps in implementation currently used in the winrm-transport gem. These changes should have little visible effect on current consumers of the `FileManager` class with these exceptions:
20
- - BREAKING CHANGE: When uploading a directory and the destination directory exists on the endpoint, the source base directory will be created below the destination directory on the endpoint and the source directory contents will be unzipped to that location. Prior to this release, the contents of the source directory would be unzipped to an existing destination directory without creating the source base directory. This new behavior is more consistent with SCP and other well known shell copy commands.
21
- - `Upload` may now receive an array of source files and directories rather than just a single file or directory path.
22
-
23
- # 0.2.4
24
- - Fix issue 21, downloading files is extremely slow.
25
- - Add zip file creation debug logging.
26
-
27
- # 0.2.3
28
- - Fix yielding progress data, issue #23
29
-
30
- # 0.2.2
31
- - Fix powershell streams leaking to standard error breaking Windows 10, issue #18
32
-
33
- # 0.2.1
34
- - Fixed issue 16 creating zip file on Windows
35
-
36
- # 0.2.0
37
- - Redesigned temp zip file creation system
38
- - Fixed lots of small edge case issues especially with directory uploads
39
- - Simplified file manager upload method API to take only a single source file or directory
40
- - Expanded acceptable username and hostnames for rwinrmcp
41
-
42
- # 0.1.0
43
- - Initial alpha quality release
1
+ # WinRM-fs Gem Changelog
2
+
3
+ # 0.4.3
4
+ - Fix error handling with wmf5, filtering out progress output from inspected stderr.
5
+
6
+ # 0.4.2
7
+ - Improved Powershell error handling in metadata checking.
8
+
9
+ # 0.4.1
10
+ - Fixes a regression on Windows 2008 R2/Windows 7 and below where the WinRM service corrupts the check files metadata resulting in malformed destination paths.
11
+
12
+ # 0.4.0
13
+ - Correct the destination path of individual files. Always assume it is the full destination path unless it is an existing directory. This may potentialy break some callers expecting the remote path to be a directory that winrm-fs will create if missing as the destination of the local file. A new directory will not be created and the local file will be uploaded directly to the remote path.
14
+
15
+ # 0.3.2
16
+ - Fix re-extraction of cached directories from temp folder when there is more than one "clean" directory deleted from destination
17
+
18
+ # 0.3.1
19
+ - Widen logging version constraints to include 2.0 (matching WinRM core gem)
20
+
21
+ # 0.3.0
22
+ - Jetisons `CommandExecutor` now living in the core WinRM gem and swaps in implementation currently used in the winrm-transport gem. These changes should have little visible effect on current consumers of the `FileManager` class with these exceptions:
23
+ - BREAKING CHANGE: When uploading a directory and the destination directory exists on the endpoint, the source base directory will be created below the destination directory on the endpoint and the source directory contents will be unzipped to that location. Prior to this release, the contents of the source directory would be unzipped to an existing destination directory without creating the source base directory. This new behavior is more consistent with SCP and other well known shell copy commands.
24
+ - `Upload` may now receive an array of source files and directories rather than just a single file or directory path.
25
+
26
+ # 0.2.4
27
+ - Fix issue 21, downloading files is extremely slow.
28
+ - Add zip file creation debug logging.
29
+
30
+ # 0.2.3
31
+ - Fix yielding progress data, issue #23
32
+
33
+ # 0.2.2
34
+ - Fix powershell streams leaking to standard error breaking Windows 10, issue #18
35
+
36
+ # 0.2.1
37
+ - Fixed issue 16 creating zip file on Windows
38
+
39
+ # 0.2.0
40
+ - Redesigned temp zip file creation system
41
+ - Fixed lots of small edge case issues especially with directory uploads
42
+ - Simplified file manager upload method API to take only a single source file or directory
43
+ - Expanded acceptable username and hostnames for rwinrmcp
44
+
45
+ # 0.1.0
46
+ - Initial alpha quality release
@@ -1,529 +1,529 @@
1
- # -*- encoding: utf-8 -*-
2
- #
3
- # Author:: Fletcher (<fnichol@nichol.ca>)
4
- #
5
- # Copyright (C) 2015, Fletcher Nichol
6
- #
7
- # Licensed under the Apache License, Version 2.0 (the "License");
8
- # you may not use this file except in compliance with the License.
9
- # You may obtain a copy of the License at
10
- #
11
- # http://www.apache.org/licenses/LICENSE-2.0
12
- #
13
- # Unless required by applicable law or agreed to in writing, software
14
- # distributed under the License is distributed on an "AS IS" BASIS,
15
- # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16
- # See the License for the specific language governing permissions and
17
- # limitations under the License.
18
-
19
- require 'benchmark'
20
- require 'csv'
21
- require 'digest'
22
- require 'securerandom'
23
- require 'stringio'
24
-
25
- require 'winrm-fs/core/tmp_zip'
26
-
27
- module WinRM
28
- module FS
29
- module Core
30
- # Wrapped exception for any internally raised WinRM-related errors.
31
- #
32
- # @author Fletcher Nichol <fnichol@nichol.ca>
33
- class FileTransporterFailed < ::WinRM::WinRMError; end
34
- # rubocop:disable MethodLength, AbcSize, ClassLength
35
-
36
- # Object which can upload one or more files or directories to a remote
37
- # host over WinRM using PowerShell scripts and CMD commands. Note that
38
- # this form of file transfer is *not* ideal and extremely costly on both
39
- # the local and remote sides. Great pains are made to minimize round
40
- # trips to the remote host and to minimize the number of PowerShell
41
- # sessions being invoked which can be 2 orders of magnitude more
42
- # expensive than vanilla CMD commands.
43
- #
44
- # This object is supported by either a `CommandExecutor` instance as it
45
- # depends on the `#run_cmd` and `#run_powershell_script` API contracts.
46
- #
47
- # An optional logger can be supplied, assuming it can respond to the
48
- # `#debug` and `#debug?` messages.
49
- #
50
- # @author Fletcher Nichol <fnichol@nichol.ca>
51
- # @author Matt Wrock <matt@mattwrock.com>
52
- class FileTransporter
53
- # Creates a FileTransporter given a CommandExecutor object.
54
- #
55
- # @param executor [CommandExecutor] a winrm CommandExecutor object
56
- def initialize(executor, opts = {})
57
- @executor = executor
58
- @logger = executor.service.logger
59
- @id_generator = opts.fetch(:id_generator) { -> { SecureRandom.uuid } }
60
- end
61
-
62
- # Uploads a collection of files and/or directories to the remote host.
63
- #
64
- # **TODO Notes:**
65
- # * options could specify zip mode, zip options, etc.
66
- # * maybe option to set tmpfile base dir to override $env:PATH?
67
- # * progress yields block like net-scp progress
68
- # * final API: def upload(locals, remote, _options = {}, &_progress)
69
- #
70
- # @param locals [Array<String>,String] one or more local file or
71
- # directory paths
72
- # @param remote [String] the base destination path on the remote host
73
- # @return [Hash] report hash, keyed by the local MD5 digest
74
- def upload(locals, remote)
75
- files = nil
76
- report = nil
77
-
78
- elapsed1 = Benchmark.measure do
79
- files = make_files_hash(Array(locals), remote)
80
- report = check_files(files)
81
- merge_with_report!(files, report)
82
- reconcile_destinations!(files)
83
- end
84
- total_size = total_base64_transfer_size(files)
85
-
86
- elapsed2 = Benchmark.measure do
87
- report = stream_upload_files(files) do |local_path, xfered|
88
- yield xfered, total_size, local_path, remote if block_given?
89
- end
90
- merge_with_report!(files, report)
91
- end
92
-
93
- elapsed3 = Benchmark.measure do
94
- report = decode_files(files)
95
- merge_with_report!(files, report)
96
- cleanup(files)
97
- end
98
-
99
- logger.debug(
100
- "Uploaded #{files.keys.size} items " \
101
- "dirty_check: #{duration(elapsed1.real)} " \
102
- "stream_files: #{duration(elapsed2.real)} " \
103
- "decode: #{duration(elapsed3.real)} " \
104
- )
105
-
106
- [total_size, files]
107
- end
108
-
109
- private
110
-
111
- # @return [Integer] the maximum number of bytes that can be supplied on
112
- # a Windows CMD prompt without exceeded the maximum command line
113
- # length
114
- # @api private
115
- MAX_ENCODED_WRITE = 8000
116
-
117
- # @return [String] the Array pack template for Base64 encoding a stream
118
- # of data
119
- # @api private
120
- BASE64_PACK = 'm0'.freeze
121
-
122
- # @return [String] the directory where temporary upload artifacts are
123
- # persisted
124
- # @api private
125
- TEMP_UPLOAD_DIRECTORY = '$env:TEMP\\winrm-upload'.freeze
126
-
127
- # @return [#debug,#debug?] the logger
128
- # @api private
129
- attr_reader :logger
130
-
131
- # @return [Winrm::CommandExecutor] a WinRM CommandExecutor
132
- # @api private
133
- attr_reader :executor
134
-
135
- # Examines the files and corrects the file destination if it is
136
- # targeting an existing folder. In this case, the destination path
137
- # will have the base name of the source file appended. This only
138
- # applies to file uploads and not to folder uploads.
139
- #
140
- # @param files [Hash] files hash, keyed by the local MD5 digest
141
- # @return [Hash] a report hash, keyed by the local MD5 digest
142
- # @api private
143
- def reconcile_destinations!(files)
144
- files.each do |_, data|
145
- if data['target_is_folder'] == 'True'
146
- data['dst'] = File.join(data['dst'], File.basename(data['src']))
147
- end
148
- end
149
- end
150
-
151
- # Adds an entry to a files Hash (keyed by local MD5 digest) for a
152
- # directory. When a directory is added, a temporary Zip file is created
153
- # containing the contents of the directory and any file-related data
154
- # such as MD5 digest, size, etc. will be referring to the Zip file.
155
- #
156
- # @param hash [Hash] hash to be mutated
157
- # @param dir [String] directory path to be Zipped and added
158
- # @param remote [String] path to destination on remote host
159
- # @api private
160
- def add_directory_hash!(hash, dir, remote)
161
- logger.debug "creating hash for directory #{remote}"
162
- zip_io = TmpZip.new(dir, logger)
163
- zip_md5 = md5sum(zip_io.path)
164
-
165
- hash[zip_md5] = {
166
- 'src' => dir,
167
- 'src_zip' => zip_io.path.to_s,
168
- 'zip_io' => zip_io,
169
- 'tmpzip' => "#{TEMP_UPLOAD_DIRECTORY}\\tmpzip-#{zip_md5}.zip",
170
- 'dst' => "#{remote}\\#{File.basename(dir)}",
171
- 'size' => File.size(zip_io.path)
172
- }
173
- end
174
-
175
- # Adds an entry to a files Hash (keyed by local MD5 digest) for a file.
176
- #
177
- # @param hash [Hash] hash to be mutated
178
- # @param local [String] file path
179
- # @param remote [String] path to destination on remote host
180
- # @api private
181
- def add_file_hash!(hash, local, remote)
182
- logger.debug "creating hash for file #{remote}"
183
-
184
- hash[md5sum(local)] = {
185
- 'src' => local,
186
- 'dst' => remote,
187
- 'size' => File.size(local)
188
- }
189
- end
190
-
191
- # Runs the check_files PowerShell script against a collection of
192
- # destination path/MD5 checksum pairs. The PowerShell script returns
193
- # its results as a CSV-formatted report which is converted into a Ruby
194
- # Hash.
195
- #
196
- # @param files [Hash] files hash, keyed by the local MD5 digest
197
- # @return [Hash] a report hash, keyed by the local MD5 digest
198
- # @api private
199
- def check_files(files)
200
- logger.debug 'Running check_files.ps1'
201
- hash_file = create_remote_hash_file(check_files_ps_hash(files))
202
- script = WinRM::FS::Scripts.render('check_files', hash_file: hash_file)
203
- parse_response(executor.run_powershell_script(script))
204
- end
205
-
206
- # Constructs a collection of destination path/MD5 checksum pairs as a
207
- # String representation of the contents of a PowerShell Hash Table.
208
- #
209
- # @param files [Hash] files hash, keyed by the local MD5 digest
210
- # @return [String] the inner contents of a PowerShell Hash Table
211
- # @api private
212
- def check_files_ps_hash(files)
213
- hash = files.map do |md5, data|
214
- [
215
- md5,
216
- {
217
- 'target' => data.fetch('tmpzip', data['dst']),
218
- 'src_basename' => File.basename(data['src']),
219
- 'dst' => data['dst']
220
- }
221
- ]
222
- end
223
- ps_hash(Hash[hash])
224
- end
225
-
226
- # Performs any final cleanup on the report Hash and removes any
227
- # temporary files/resources used in the upload task.
228
- #
229
- # @param files [Hash] a files hash
230
- # @api private
231
- def cleanup(files)
232
- files.select { |_, data| data.key?('zip_io') }.each do |md5, data|
233
- data.fetch('zip_io').unlink
234
- files.fetch(md5).delete('zip_io')
235
- logger.debug "Cleaned up src_zip #{data['src_zip']}"
236
- end
237
- end
238
-
239
- # Creates a remote Base64-encoded temporary file containing a
240
- # PowerShell hash table.
241
- #
242
- # @param hash [String] a String representation of a PowerShell hash
243
- # table
244
- # @return [String] the remote path to the temporary file
245
- # @api private
246
- def create_remote_hash_file(hash)
247
- hash_file = "$env:TEMP\\hash-#{@id_generator.call}.txt"
248
- hash.lines.each { |line| logger.debug line.chomp }
249
- StringIO.open(hash) { |io| stream_upload(io, hash_file) }
250
- hash_file
251
- end
252
-
253
- # Runs the decode_files PowerShell script against a collection of
254
- # temporary file/destination path pairs. The PowerShell script returns
255
- # its results as a CSV-formatted report which is converted into a Ruby
256
- # Hash. The script will not be invoked if there are no "dirty" files
257
- # present in the incoming files Hash.
258
- #
259
- # @param files [Hash] files hash, keyed by the local MD5 digest
260
- # @return [Hash] a report hash, keyed by the local MD5 digest
261
- # @api private
262
- def decode_files(files)
263
- decoded_files = decode_files_ps_hash(files)
264
-
265
- if decoded_files == ps_hash({})
266
- logger.debug 'No remote files to decode, skipping'
267
- {}
268
- else
269
- logger.debug 'Running decode_files.ps1'
270
- hash_file = create_remote_hash_file(decoded_files)
271
- script = WinRM::FS::Scripts.render('decode_files', hash_file: hash_file)
272
-
273
- parse_response(executor.run_powershell_script(script))
274
- end
275
- end
276
-
277
- # Constructs a collection of temporary file/destination path pairs for
278
- # all "dirty" files as a String representation of the contents of a
279
- # PowerShell Hash Table. A "dirty" file is one which has the
280
- # `"chk_dirty"` option set to `"True"` in the incoming files Hash.
281
- #
282
- # @param files [Hash] files hash, keyed by the local MD5 digest
283
- # @return [String] the inner contents of a PowerShell Hash Table
284
- # @api private
285
- def decode_files_ps_hash(files)
286
- file_data = files.select do |_, data|
287
- data['chk_dirty'] == 'True' || data.key?('tmpzip')
288
- end
289
-
290
- i = 0
291
- result = file_data.map do |_, data|
292
- val = { 'dst' => data['dst'] }
293
- val['tmpzip'] = data['tmpzip'] if data['tmpzip']
294
-
295
- [data['tmpfile'] || "clean#{i += 1}", val]
296
- end
297
-
298
- ps_hash(Hash[result])
299
- end
300
-
301
- # Returns a formatted string representing a duration in seconds.
302
- #
303
- # @param total [Integer] the total number of seconds
304
- # @return [String] a formatted string of the form (XmYY.00s)
305
- def duration(total)
306
- total = 0 if total.nil?
307
- minutes = (total / 60).to_i
308
- seconds = (total - (minutes * 60))
309
- format('(%dm%.2fs)', minutes, seconds)
310
- end
311
-
312
- # Contructs a Hash of files or directories, keyed by the local MD5
313
- # digest. Each file entry has a source and destination set, at a
314
- # minimum.
315
- #
316
- # @param locals [Array<String>] a collection of local files or
317
- # directories
318
- # @param remote [String] the base destination path on the remote host
319
- # @return [Hash] files hash, keyed by the local MD5 digest
320
- # @api private
321
- def make_files_hash(locals, remote)
322
- hash = {}
323
- locals.each do |local|
324
- expanded = File.expand_path(local)
325
- expanded += local[-1] if local.end_with?('/', '\\')
326
-
327
- if File.file?(expanded)
328
- add_file_hash!(hash, expanded, remote)
329
- elsif File.directory?(expanded)
330
- add_directory_hash!(hash, expanded, remote)
331
- else
332
- fail Errno::ENOENT, "No such file or directory #{expanded}"
333
- end
334
- end
335
- hash
336
- end
337
-
338
- # @return [String] the MD5 digest of a local file
339
- # @api private
340
- def md5sum(local)
341
- Digest::MD5.file(local).hexdigest
342
- end
343
-
344
- # Destructively merges a report Hash into an existing files Hash.
345
- # **Note:** this method mutates the files Hash.
346
- #
347
- # @param files [Hash] files hash, keyed by the local MD5 digest
348
- # @param report [Hash] report hash, keyed by the local MD5 digest
349
- # @api private
350
- def merge_with_report!(files, report)
351
- files.merge!(report) { |_, oldval, newval| oldval.merge(newval) }
352
- end
353
-
354
- # @param depth [Integer] number of padding characters (default: `0`)
355
- # @return [String] a whitespace padded string of the given length
356
- # @api private
357
- def pad(depth = 0)
358
- ' ' * depth
359
- end
360
-
361
- # Parses CLIXML String into regular String (without any XML syntax).
362
- # Inspired by https://github.com/WinRb/WinRM/issues/106.
363
- #
364
- # @param clixml [String] clixml text
365
- # @return [String] parsed clixml into String
366
- def clixml_to_s(clixml)
367
- doc = REXML::Document.new(clixml)
368
- text = doc.get_elements('//S').map(&:text).join
369
- text.gsub(/_x(\h\h\h\h)_/) do
370
- code = Regexp.last_match[1]
371
- code.hex.chr
372
- end
373
- end
374
-
375
- # Parses response of a PowerShell script or CMD command which contains
376
- # a CSV-formatted document in the standard output stream.
377
- #
378
- # @param output [WinRM::Output] output object with stdout, stderr, and
379
- # exit code
380
- # @return [Hash] report hash, keyed by the local MD5 digest
381
- # @api private
382
- def parse_response(output)
383
- exitcode = output[:exitcode]
384
- stderr = output.stderr
385
- if stderr.include?('The command line is too long')
386
- # The powershell script which should result in `output` parameter
387
- # is too long, remove some newlines, comments, etc from it.
388
- fail StandardError, 'The command line is too long' \
389
- ' (powershell script is too long)'
390
- end
391
- pretty_stderr = clixml_to_s(stderr)
392
-
393
- if exitcode != 0
394
- fail FileTransporterFailed, "[#{self.class}] Upload failed " \
395
- "(exitcode: #{exitcode})\n#{pretty_stderr}"
396
- elsif stderr != '\r\n' && stderr != ''
397
- fail FileTransporterFailed, "[#{self.class}] Upload failed " \
398
- "(exitcode: 0), but stderr present\n#{pretty_stderr}"
399
- end
400
-
401
- logger.debug 'Parsing CSV Response'
402
- logger.debug output.stdout
403
-
404
- array = CSV.parse(output.stdout, headers: true).map(&:to_hash)
405
- array.each { |h| h.each { |key, value| h[key] = nil if value == '' } }
406
- Hash[array.map { |entry| [entry.fetch('src_md5'), entry] }]
407
- end
408
-
409
- # Converts a Ruby hash into a PowerShell hash table, represented in a
410
- # String.
411
- #
412
- # @param obj [Object] source Hash or object when used in recursive
413
- # calls
414
- # @param depth [Integer] padding depth, used in recursive calls
415
- # (default: `0`)
416
- # @return [String] a PowerShell hash table
417
- # @api private
418
- def ps_hash(obj, depth = 0)
419
- if obj.is_a?(Hash)
420
- obj.map do |k, v|
421
- %(#{pad(depth + 2)}#{ps_hash(k)} = #{ps_hash(v, depth + 2)})
422
- end.join(";\n").insert(0, "@{\n").insert(-1, "\n#{pad(depth)}}")
423
- else
424
- %("#{obj}")
425
- end
426
- end
427
-
428
- # Uploads an IO stream to a Base64-encoded destination file.
429
- #
430
- # **Implementation Note:** Some of the code in this method may appear
431
- # slightly too dense and while adding additional variables would help,
432
- # the code is written very precisely to avoid unwanted allocations
433
- # which will bloat the Ruby VM's object space (and memory footprint).
434
- # The goal here is to stream potentially large files to a remote host
435
- # while not loading the entire file into memory first, then Base64
436
- # encoding it--duplicating the file in memory again.
437
- #
438
- # @param input_io [#read] a readable stream or object to be uploaded
439
- # @param dest [String] path to the destination file on the remote host
440
- # @return [Integer,Integer] the number of resulting upload chunks and
441
- # the number of bytes transferred to the remote host
442
- # @api private
443
- def stream_upload(input_io, dest)
444
- dest_cmd = dest.sub('$env:TEMP', '%TEMP%')
445
- read_size = (MAX_ENCODED_WRITE.to_i / 4) * 3
446
- chunk, bytes = 1, 0
447
- buffer = ''
448
- executor.run_cmd(%(echo|set /p=>"#{dest_cmd}")) # truncate empty file
449
- while input_io.read(read_size, buffer)
450
- bytes += (buffer.bytesize / 3 * 4)
451
- executor.run_cmd([buffer].pack(BASE64_PACK)
452
- .insert(0, 'echo ').concat(%( >> "#{dest_cmd}")))
453
- logger.debug "Wrote chunk #{chunk} for #{dest}" if chunk % 25 == 0
454
- chunk += 1
455
- yield bytes if block_given?
456
- end
457
- buffer = nil # rubocop:disable Lint/UselessAssignment
458
-
459
- [chunk - 1, bytes]
460
- end
461
-
462
- # Uploads a local file to a Base64-encoded temporary file.
463
- #
464
- # @param src [String] path to a local file
465
- # @param tmpfile [String] path to the temporary file on the remote
466
- # host
467
- # @return [Integer,Integer] the number of resulting upload chunks and
468
- # the number of bytes transferred to the remote host
469
- # @api private
470
- def stream_upload_file(src, tmpfile, &block)
471
- logger.debug "Uploading #{src} to encoded tmpfile #{tmpfile}"
472
- chunks, bytes = 0, 0
473
- elapsed = Benchmark.measure do
474
- File.open(src, 'rb') do |io|
475
- chunks, bytes = stream_upload(io, tmpfile, &block)
476
- end
477
- end
478
- logger.debug(
479
- "Finished uploading #{src} to encoded tmpfile #{tmpfile} " \
480
- "(#{bytes.to_f / 1000} KB over #{chunks} chunks) " \
481
- "in #{duration(elapsed.real)}"
482
- )
483
-
484
- [chunks, bytes]
485
- end
486
-
487
- # Uploads a collection of "dirty" files to the remote host as
488
- # Base64-encoded temporary files. A "dirty" file is one which has the
489
- # `"chk_dirty"` option set to `"True"` in the incoming files Hash.
490
- #
491
- # @param files [Hash] files hash, keyed by the local MD5 digest
492
- # @return [Hash] a report hash, keyed by the local MD5 digest
493
- # @api private
494
- def stream_upload_files(files)
495
- response = {}
496
- files.each do |md5, data|
497
- src = data.fetch('src_zip', data['src'])
498
- if data['chk_dirty'] == 'True'
499
- tmpfile = "$env:TEMP\\b64-#{md5}.txt"
500
- response[md5] = { 'tmpfile' => tmpfile }
501
- chunks, bytes = stream_upload_file(src, tmpfile) do |xfered|
502
- yield data['src'], xfered
503
- end
504
- response[md5]['chunks'] = chunks
505
- response[md5]['xfered'] = bytes
506
- else
507
- logger.debug "File #{data['dst']} is up to date, skipping"
508
- end
509
- end
510
- response
511
- end
512
-
513
- # Total by byte count to be transferred.
514
- # Calculates count based on the sum of base64 encoded content size
515
- # of all files base 64 that are dirty.
516
- #
517
- # @param files [Hash] files hash, keyed by the local MD5 digest
518
- # @return [Fixnum] total byte size
519
- # @api private
520
- def total_base64_transfer_size(files)
521
- size = 0
522
- files.values.each { |file| size += file['size'] if file['chk_dirty'] == 'True' }
523
- size / 3 * 4
524
- end
525
- end
526
- # rubocop:enable MethodLength, AbcSize, ClassLength
527
- end
528
- end
529
- end
1
+ # -*- encoding: utf-8 -*-
2
+ #
3
+ # Author:: Fletcher (<fnichol@nichol.ca>)
4
+ #
5
+ # Copyright (C) 2015, Fletcher Nichol
6
+ #
7
+ # Licensed under the Apache License, Version 2.0 (the "License");
8
+ # you may not use this file except in compliance with the License.
9
+ # You may obtain a copy of the License at
10
+ #
11
+ # http://www.apache.org/licenses/LICENSE-2.0
12
+ #
13
+ # Unless required by applicable law or agreed to in writing, software
14
+ # distributed under the License is distributed on an "AS IS" BASIS,
15
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
16
+ # See the License for the specific language governing permissions and
17
+ # limitations under the License.
18
+
19
+ require 'benchmark'
20
+ require 'csv'
21
+ require 'digest'
22
+ require 'securerandom'
23
+ require 'stringio'
24
+
25
+ require 'winrm-fs/core/tmp_zip'
26
+
27
+ module WinRM
28
+ module FS
29
+ module Core
30
+ # Wrapped exception for any internally raised WinRM-related errors.
31
+ #
32
+ # @author Fletcher Nichol <fnichol@nichol.ca>
33
+ class FileTransporterFailed < ::WinRM::WinRMError; end
34
+ # rubocop:disable MethodLength, AbcSize, ClassLength
35
+
36
+ # Object which can upload one or more files or directories to a remote
37
+ # host over WinRM using PowerShell scripts and CMD commands. Note that
38
+ # this form of file transfer is *not* ideal and extremely costly on both
39
+ # the local and remote sides. Great pains are made to minimize round
40
+ # trips to the remote host and to minimize the number of PowerShell
41
+ # sessions being invoked which can be 2 orders of magnitude more
42
+ # expensive than vanilla CMD commands.
43
+ #
44
+ # This object is supported by either a `CommandExecutor` instance as it
45
+ # depends on the `#run_cmd` and `#run_powershell_script` API contracts.
46
+ #
47
+ # An optional logger can be supplied, assuming it can respond to the
48
+ # `#debug` and `#debug?` messages.
49
+ #
50
+ # @author Fletcher Nichol <fnichol@nichol.ca>
51
+ # @author Matt Wrock <matt@mattwrock.com>
52
+ class FileTransporter
53
+ # Creates a FileTransporter given a CommandExecutor object.
54
+ #
55
+ # @param executor [CommandExecutor] a winrm CommandExecutor object
56
+ def initialize(executor, opts = {})
57
+ @executor = executor
58
+ @logger = executor.service.logger
59
+ @id_generator = opts.fetch(:id_generator) { -> { SecureRandom.uuid } }
60
+ end
61
+
62
+ # Uploads a collection of files and/or directories to the remote host.
63
+ #
64
+ # **TODO Notes:**
65
+ # * options could specify zip mode, zip options, etc.
66
+ # * maybe option to set tmpfile base dir to override $env:PATH?
67
+ # * progress yields block like net-scp progress
68
+ # * final API: def upload(locals, remote, _options = {}, &_progress)
69
+ #
70
+ # @param locals [Array<String>,String] one or more local file or
71
+ # directory paths
72
+ # @param remote [String] the base destination path on the remote host
73
+ # @return [Hash] report hash, keyed by the local MD5 digest
74
+ def upload(locals, remote)
75
+ files = nil
76
+ report = nil
77
+
78
+ elapsed1 = Benchmark.measure do
79
+ files = make_files_hash(Array(locals), remote)
80
+ report = check_files(files)
81
+ merge_with_report!(files, report)
82
+ reconcile_destinations!(files)
83
+ end
84
+ total_size = total_base64_transfer_size(files)
85
+
86
+ elapsed2 = Benchmark.measure do
87
+ report = stream_upload_files(files) do |local_path, xfered|
88
+ yield xfered, total_size, local_path, remote if block_given?
89
+ end
90
+ merge_with_report!(files, report)
91
+ end
92
+
93
+ elapsed3 = Benchmark.measure do
94
+ report = decode_files(files)
95
+ merge_with_report!(files, report)
96
+ cleanup(files)
97
+ end
98
+
99
+ logger.debug(
100
+ "Uploaded #{files.keys.size} items " \
101
+ "dirty_check: #{duration(elapsed1.real)} " \
102
+ "stream_files: #{duration(elapsed2.real)} " \
103
+ "decode: #{duration(elapsed3.real)} " \
104
+ )
105
+
106
+ [total_size, files]
107
+ end
108
+
109
+ private
110
+
111
+ # @return [Integer] the maximum number of bytes that can be supplied on
112
+ # a Windows CMD prompt without exceeded the maximum command line
113
+ # length
114
+ # @api private
115
+ MAX_ENCODED_WRITE = 8000
116
+
117
+ # @return [String] the Array pack template for Base64 encoding a stream
118
+ # of data
119
+ # @api private
120
+ BASE64_PACK = 'm0'.freeze
121
+
122
+ # @return [String] the directory where temporary upload artifacts are
123
+ # persisted
124
+ # @api private
125
+ TEMP_UPLOAD_DIRECTORY = '$env:TEMP\\winrm-upload'.freeze
126
+
127
+ # @return [#debug,#debug?] the logger
128
+ # @api private
129
+ attr_reader :logger
130
+
131
+ # @return [Winrm::CommandExecutor] a WinRM CommandExecutor
132
+ # @api private
133
+ attr_reader :executor
134
+
135
+ # Examines the files and corrects the file destination if it is
136
+ # targeting an existing folder. In this case, the destination path
137
+ # will have the base name of the source file appended. This only
138
+ # applies to file uploads and not to folder uploads.
139
+ #
140
+ # @param files [Hash] files hash, keyed by the local MD5 digest
141
+ # @return [Hash] a report hash, keyed by the local MD5 digest
142
+ # @api private
143
+ def reconcile_destinations!(files)
144
+ files.each do |_, data|
145
+ if data['target_is_folder'] == 'True'
146
+ data['dst'] = File.join(data['dst'], File.basename(data['src']))
147
+ end
148
+ end
149
+ end
150
+
151
+ # Adds an entry to a files Hash (keyed by local MD5 digest) for a
152
+ # directory. When a directory is added, a temporary Zip file is created
153
+ # containing the contents of the directory and any file-related data
154
+ # such as MD5 digest, size, etc. will be referring to the Zip file.
155
+ #
156
+ # @param hash [Hash] hash to be mutated
157
+ # @param dir [String] directory path to be Zipped and added
158
+ # @param remote [String] path to destination on remote host
159
+ # @api private
160
+ def add_directory_hash!(hash, dir, remote)
161
+ logger.debug "creating hash for directory #{remote}"
162
+ zip_io = TmpZip.new(dir, logger)
163
+ zip_md5 = md5sum(zip_io.path)
164
+
165
+ hash[zip_md5] = {
166
+ 'src' => dir,
167
+ 'src_zip' => zip_io.path.to_s,
168
+ 'zip_io' => zip_io,
169
+ 'tmpzip' => "#{TEMP_UPLOAD_DIRECTORY}\\tmpzip-#{zip_md5}.zip",
170
+ 'dst' => "#{remote}\\#{File.basename(dir)}",
171
+ 'size' => File.size(zip_io.path)
172
+ }
173
+ end
174
+
175
+ # Adds an entry to a files Hash (keyed by local MD5 digest) for a file.
176
+ #
177
+ # @param hash [Hash] hash to be mutated
178
+ # @param local [String] file path
179
+ # @param remote [String] path to destination on remote host
180
+ # @api private
181
+ def add_file_hash!(hash, local, remote)
182
+ logger.debug "creating hash for file #{remote}"
183
+
184
+ hash[md5sum(local)] = {
185
+ 'src' => local,
186
+ 'dst' => remote,
187
+ 'size' => File.size(local)
188
+ }
189
+ end
190
+
191
+ # Runs the check_files PowerShell script against a collection of
192
+ # destination path/MD5 checksum pairs. The PowerShell script returns
193
+ # its results as a CSV-formatted report which is converted into a Ruby
194
+ # Hash.
195
+ #
196
+ # @param files [Hash] files hash, keyed by the local MD5 digest
197
+ # @return [Hash] a report hash, keyed by the local MD5 digest
198
+ # @api private
199
+ def check_files(files)
200
+ logger.debug 'Running check_files.ps1'
201
+ hash_file = create_remote_hash_file(check_files_ps_hash(files))
202
+ script = WinRM::FS::Scripts.render('check_files', hash_file: hash_file)
203
+ parse_response(executor.run_powershell_script(script))
204
+ end
205
+
206
+ # Constructs a collection of destination path/MD5 checksum pairs as a
207
+ # String representation of the contents of a PowerShell Hash Table.
208
+ #
209
+ # @param files [Hash] files hash, keyed by the local MD5 digest
210
+ # @return [String] the inner contents of a PowerShell Hash Table
211
+ # @api private
212
+ def check_files_ps_hash(files)
213
+ hash = files.map do |md5, data|
214
+ [
215
+ md5,
216
+ {
217
+ 'target' => data.fetch('tmpzip', data['dst']),
218
+ 'src_basename' => File.basename(data['src']),
219
+ 'dst' => data['dst']
220
+ }
221
+ ]
222
+ end
223
+ ps_hash(Hash[hash])
224
+ end
225
+
226
+ # Performs any final cleanup on the report Hash and removes any
227
+ # temporary files/resources used in the upload task.
228
+ #
229
+ # @param files [Hash] a files hash
230
+ # @api private
231
+ def cleanup(files)
232
+ files.select { |_, data| data.key?('zip_io') }.each do |md5, data|
233
+ data.fetch('zip_io').unlink
234
+ files.fetch(md5).delete('zip_io')
235
+ logger.debug "Cleaned up src_zip #{data['src_zip']}"
236
+ end
237
+ end
238
+
239
+ # Creates a remote Base64-encoded temporary file containing a
240
+ # PowerShell hash table.
241
+ #
242
+ # @param hash [String] a String representation of a PowerShell hash
243
+ # table
244
+ # @return [String] the remote path to the temporary file
245
+ # @api private
246
+ def create_remote_hash_file(hash)
247
+ hash_file = "$env:TEMP\\hash-#{@id_generator.call}.txt"
248
+ hash.lines.each { |line| logger.debug line.chomp }
249
+ StringIO.open(hash) { |io| stream_upload(io, hash_file) }
250
+ hash_file
251
+ end
252
+
253
+ # Runs the decode_files PowerShell script against a collection of
254
+ # temporary file/destination path pairs. The PowerShell script returns
255
+ # its results as a CSV-formatted report which is converted into a Ruby
256
+ # Hash. The script will not be invoked if there are no "dirty" files
257
+ # present in the incoming files Hash.
258
+ #
259
+ # @param files [Hash] files hash, keyed by the local MD5 digest
260
+ # @return [Hash] a report hash, keyed by the local MD5 digest
261
+ # @api private
262
+ def decode_files(files)
263
+ decoded_files = decode_files_ps_hash(files)
264
+
265
+ if decoded_files == ps_hash({})
266
+ logger.debug 'No remote files to decode, skipping'
267
+ {}
268
+ else
269
+ logger.debug 'Running decode_files.ps1'
270
+ hash_file = create_remote_hash_file(decoded_files)
271
+ script = WinRM::FS::Scripts.render('decode_files', hash_file: hash_file)
272
+
273
+ parse_response(executor.run_powershell_script(script))
274
+ end
275
+ end
276
+
277
+ # Constructs a collection of temporary file/destination path pairs for
278
+ # all "dirty" files as a String representation of the contents of a
279
+ # PowerShell Hash Table. A "dirty" file is one which has the
280
+ # `"chk_dirty"` option set to `"True"` in the incoming files Hash.
281
+ #
282
+ # @param files [Hash] files hash, keyed by the local MD5 digest
283
+ # @return [String] the inner contents of a PowerShell Hash Table
284
+ # @api private
285
+ def decode_files_ps_hash(files)
286
+ file_data = files.select do |_, data|
287
+ data['chk_dirty'] == 'True' || data.key?('tmpzip')
288
+ end
289
+
290
+ i = 0
291
+ result = file_data.map do |_, data|
292
+ val = { 'dst' => data['dst'] }
293
+ val['tmpzip'] = data['tmpzip'] if data['tmpzip']
294
+
295
+ [data['tmpfile'] || "clean#{i += 1}", val]
296
+ end
297
+
298
+ ps_hash(Hash[result])
299
+ end
300
+
301
+ # Returns a formatted string representing a duration in seconds.
302
+ #
303
+ # @param total [Integer] the total number of seconds
304
+ # @return [String] a formatted string of the form (XmYY.00s)
305
+ def duration(total)
306
+ total = 0 if total.nil?
307
+ minutes = (total / 60).to_i
308
+ seconds = (total - (minutes * 60))
309
+ format('(%dm%.2fs)', minutes, seconds)
310
+ end
311
+
312
+ # Contructs a Hash of files or directories, keyed by the local MD5
313
+ # digest. Each file entry has a source and destination set, at a
314
+ # minimum.
315
+ #
316
+ # @param locals [Array<String>] a collection of local files or
317
+ # directories
318
+ # @param remote [String] the base destination path on the remote host
319
+ # @return [Hash] files hash, keyed by the local MD5 digest
320
+ # @api private
321
+ def make_files_hash(locals, remote)
322
+ hash = {}
323
+ locals.each do |local|
324
+ expanded = File.expand_path(local)
325
+ expanded += local[-1] if local.end_with?('/', '\\')
326
+
327
+ if File.file?(expanded)
328
+ add_file_hash!(hash, expanded, remote)
329
+ elsif File.directory?(expanded)
330
+ add_directory_hash!(hash, expanded, remote)
331
+ else
332
+ fail Errno::ENOENT, "No such file or directory #{expanded}"
333
+ end
334
+ end
335
+ hash
336
+ end
337
+
338
+ # @return [String] the MD5 digest of a local file
339
+ # @api private
340
+ def md5sum(local)
341
+ Digest::MD5.file(local).hexdigest
342
+ end
343
+
344
+ # Destructively merges a report Hash into an existing files Hash.
345
+ # **Note:** this method mutates the files Hash.
346
+ #
347
+ # @param files [Hash] files hash, keyed by the local MD5 digest
348
+ # @param report [Hash] report hash, keyed by the local MD5 digest
349
+ # @api private
350
+ def merge_with_report!(files, report)
351
+ files.merge!(report) { |_, oldval, newval| oldval.merge(newval) }
352
+ end
353
+
354
+ # @param depth [Integer] number of padding characters (default: `0`)
355
+ # @return [String] a whitespace padded string of the given length
356
+ # @api private
357
+ def pad(depth = 0)
358
+ ' ' * depth
359
+ end
360
+
361
+ # Parses CLIXML String into regular String (without any XML syntax).
362
+ # Inspired by https://github.com/WinRb/WinRM/issues/106.
363
+ #
364
+ # @param clixml [String] clixml text
365
+ # @return [String] parsed clixml into String
366
+ def clixml_to_s(clixml)
367
+ doc = REXML::Document.new(clixml)
368
+ text = doc.get_elements('//S').map(&:text).join
369
+ text.gsub(/_x(\h\h\h\h)_/) do
370
+ code = Regexp.last_match[1]
371
+ code.hex.chr
372
+ end
373
+ end
374
+
375
+ # Parses response of a PowerShell script or CMD command which contains
376
+ # a CSV-formatted document in the standard output stream.
377
+ #
378
+ # @param output [WinRM::Output] output object with stdout, stderr, and
379
+ # exit code
380
+ # @return [Hash] report hash, keyed by the local MD5 digest
381
+ # @api private
382
+ def parse_response(output)
383
+ exitcode = output[:exitcode]
384
+ stderr = output.stderr
385
+ if stderr.include?('The command line is too long')
386
+ # The powershell script which should result in `output` parameter
387
+ # is too long, remove some newlines, comments, etc from it.
388
+ fail StandardError, 'The command line is too long' \
389
+ ' (powershell script is too long)'
390
+ end
391
+ pretty_stderr = clixml_to_s(stderr)
392
+
393
+ if exitcode != 0
394
+ fail FileTransporterFailed, "[#{self.class}] Upload failed " \
395
+ "(exitcode: #{exitcode})\n#{pretty_stderr}"
396
+ elsif pretty_stderr != '\r\n' && pretty_stderr != ''
397
+ fail FileTransporterFailed, "[#{self.class}] Upload failed " \
398
+ "(exitcode: 0), but stderr present\n#{pretty_stderr}"
399
+ end
400
+
401
+ logger.debug 'Parsing CSV Response'
402
+ logger.debug output.stdout
403
+
404
+ array = CSV.parse(output.stdout, headers: true).map(&:to_hash)
405
+ array.each { |h| h.each { |key, value| h[key] = nil if value == '' } }
406
+ Hash[array.map { |entry| [entry.fetch('src_md5'), entry] }]
407
+ end
408
+
409
+ # Converts a Ruby hash into a PowerShell hash table, represented in a
410
+ # String.
411
+ #
412
+ # @param obj [Object] source Hash or object when used in recursive
413
+ # calls
414
+ # @param depth [Integer] padding depth, used in recursive calls
415
+ # (default: `0`)
416
+ # @return [String] a PowerShell hash table
417
+ # @api private
418
+ def ps_hash(obj, depth = 0)
419
+ if obj.is_a?(Hash)
420
+ obj.map do |k, v|
421
+ %(#{pad(depth + 2)}#{ps_hash(k)} = #{ps_hash(v, depth + 2)})
422
+ end.join(";\n").insert(0, "@{\n").insert(-1, "\n#{pad(depth)}}")
423
+ else
424
+ %("#{obj}")
425
+ end
426
+ end
427
+
428
+ # Uploads an IO stream to a Base64-encoded destination file.
429
+ #
430
+ # **Implementation Note:** Some of the code in this method may appear
431
+ # slightly too dense and while adding additional variables would help,
432
+ # the code is written very precisely to avoid unwanted allocations
433
+ # which will bloat the Ruby VM's object space (and memory footprint).
434
+ # The goal here is to stream potentially large files to a remote host
435
+ # while not loading the entire file into memory first, then Base64
436
+ # encoding it--duplicating the file in memory again.
437
+ #
438
+ # @param input_io [#read] a readable stream or object to be uploaded
439
+ # @param dest [String] path to the destination file on the remote host
440
+ # @return [Integer,Integer] the number of resulting upload chunks and
441
+ # the number of bytes transferred to the remote host
442
+ # @api private
443
+ def stream_upload(input_io, dest)
444
+ dest_cmd = dest.sub('$env:TEMP', '%TEMP%')
445
+ read_size = (MAX_ENCODED_WRITE.to_i / 4) * 3
446
+ chunk, bytes = 1, 0
447
+ buffer = ''
448
+ executor.run_cmd(%(echo|set /p=>"#{dest_cmd}")) # truncate empty file
449
+ while input_io.read(read_size, buffer)
450
+ bytes += (buffer.bytesize / 3 * 4)
451
+ executor.run_cmd([buffer].pack(BASE64_PACK)
452
+ .insert(0, 'echo ').concat(%( >> "#{dest_cmd}")))
453
+ logger.debug "Wrote chunk #{chunk} for #{dest}" if chunk % 25 == 0
454
+ chunk += 1
455
+ yield bytes if block_given?
456
+ end
457
+ buffer = nil # rubocop:disable Lint/UselessAssignment
458
+
459
+ [chunk - 1, bytes]
460
+ end
461
+
462
+ # Uploads a local file to a Base64-encoded temporary file.
463
+ #
464
+ # @param src [String] path to a local file
465
+ # @param tmpfile [String] path to the temporary file on the remote
466
+ # host
467
+ # @return [Integer,Integer] the number of resulting upload chunks and
468
+ # the number of bytes transferred to the remote host
469
+ # @api private
470
+ def stream_upload_file(src, tmpfile, &block)
471
+ logger.debug "Uploading #{src} to encoded tmpfile #{tmpfile}"
472
+ chunks, bytes = 0, 0
473
+ elapsed = Benchmark.measure do
474
+ File.open(src, 'rb') do |io|
475
+ chunks, bytes = stream_upload(io, tmpfile, &block)
476
+ end
477
+ end
478
+ logger.debug(
479
+ "Finished uploading #{src} to encoded tmpfile #{tmpfile} " \
480
+ "(#{bytes.to_f / 1000} KB over #{chunks} chunks) " \
481
+ "in #{duration(elapsed.real)}"
482
+ )
483
+
484
+ [chunks, bytes]
485
+ end
486
+
487
+ # Uploads a collection of "dirty" files to the remote host as
488
+ # Base64-encoded temporary files. A "dirty" file is one which has the
489
+ # `"chk_dirty"` option set to `"True"` in the incoming files Hash.
490
+ #
491
+ # @param files [Hash] files hash, keyed by the local MD5 digest
492
+ # @return [Hash] a report hash, keyed by the local MD5 digest
493
+ # @api private
494
+ def stream_upload_files(files)
495
+ response = {}
496
+ files.each do |md5, data|
497
+ src = data.fetch('src_zip', data['src'])
498
+ if data['chk_dirty'] == 'True'
499
+ tmpfile = "$env:TEMP\\b64-#{md5}.txt"
500
+ response[md5] = { 'tmpfile' => tmpfile }
501
+ chunks, bytes = stream_upload_file(src, tmpfile) do |xfered|
502
+ yield data['src'], xfered
503
+ end
504
+ response[md5]['chunks'] = chunks
505
+ response[md5]['xfered'] = bytes
506
+ else
507
+ logger.debug "File #{data['dst']} is up to date, skipping"
508
+ end
509
+ end
510
+ response
511
+ end
512
+
513
+ # Total by byte count to be transferred.
514
+ # Calculates count based on the sum of base64 encoded content size
515
+ # of all files base 64 that are dirty.
516
+ #
517
+ # @param files [Hash] files hash, keyed by the local MD5 digest
518
+ # @return [Fixnum] total byte size
519
+ # @api private
520
+ def total_base64_transfer_size(files)
521
+ size = 0
522
+ files.values.each { |file| size += file['size'] if file['chk_dirty'] == 'True' }
523
+ size / 3 * 4
524
+ end
525
+ end
526
+ # rubocop:enable MethodLength, AbcSize, ClassLength
527
+ end
528
+ end
529
+ end