ssh2-sftp-client 5.3.1 → 7.0.0

package/CHANGELOG.org

@@ -1,38 +1,69 @@
 * Change Logging
 
-** V5.3.1 (Prod Version)
+** V7.0.0
+   - New version based on new SSH2 version 1.1.0.
+   - Expand option handling for get() and put() methods *Breaking Change*
+   - Re-factored the retry code in the connect() method
+   - Improve error reporting for adding/removing listeners
+   - Extend localExists() method to also verify read or write access
+       
+** V6.0.1
+   - Fix issue with connect retry not releasing 'ready' listeners
+   - Add finally clauses to all promises to ensure temporary listeners are deleted
+   - Add nyc module to report on code test coverage
+   - Add additional utils tests to increase test coverage
+   - Removed some dead code and unused utility functions to reduce download size
+   - Cleanup tests and reduce inter-test dependencies
+
+** V6.0.0.0
+   - Update connection retry code to use the promise-retry module instead of
+     plain rety module
+   - Added optional filter argument for uploadDir/downlDir to select which files
+     and directories are included
+   - Added an optional boolean argument to delete to turn off raising an error
+     when delete target does not exists
+   - Reduced/simplified argument verification code to reduce package size and
+     increase performance
+   - Refactored handling of events and add default close and error listeners to
+     catch connections closed abruptly without an error being raised.
+
+** V5.3.2
+   - Minor README typo fixes
+   - Fix error in local file path checks (#294)
+
+** V5.3.1
    - Fix bug in handling of relative local paths
    - Change handling of stream closures in ~get()~ and ~put()~ methods
 
-** v5.3.0 
+** v5.3.0
    - Refine event handler management
    - Fix path processing for win32 based sftp servers
-   - Update documentation 
-** v5.2.2 
+   - Update documentation
+** v5.2.2
    - Bug fix release. Add error code 4 check to stat() method.
    - bump Mocha version for tests
 
 ** v5.2.1
    - Move some dependencies into dev-Dependencies
-** v5.2.0 
-   - Add new method posixRename() which uses the openSSH POSIX rename extension. 
+** v5.2.0
+   - Add new method posixRename() which uses the openSSH POSIX rename extension.
 ** v5.1.3
    - Fix bug when writing to root directory and failure due to not being able to
      determine parent
    - Refactor some tests to eliminate need to have artificial delays between
      tests
    - Bumped some dependency versions to latest version
-** v5.1.2 
+** v5.1.2
    - Added back global close handler
    - Added dumpListeners() method
-   
-** v5.1.1 
-   - Added separate close handlers to each method. 
+
+** v5.1.1
+   - Added separate close handlers to each method.
    - Added missing return statement in connect method
-   - Added additional troubleshooting documentation for 
+   - Added additional troubleshooting documentation for
      common errors.
 
-** v5.1.0 
+** v5.1.0
    - Fix bug in checkRemotePath() relating to handling of badly
      specified paths (issue #213)
    - Added additional debugging support

package/README.md

@@ -1,22 +1,7 @@
 - [Overview](#sec-1)
 - [Installation](#sec-2)
 - [Basic Usage](#sec-3)
-- [Version 5.x](#sec-4)
-  - [Breaking Changes in Version 5.x](#sec-4-1)
-    - [Error Event Handling](#sec-4-1-1)
-    - [Technical Details](#sec-4-1-2)
-  - [New Methods](#sec-4-2)
-  - [Version 5.0.1](#sec-4-3)
-  - [Version 5.0.2](#sec-4-4)
-  - [Version 5.1.0](#sec-4-5)
-  - [Version 5.1.1](#sec-4-6)
-  - [Version 5.1.2](#sec-4-7)
-  - [Version 5.1.3](#sec-4-8)
-  - [Version 5.2.0](#sec-4-9)
-  - [Version 5.2.1](#sec-4-10)
-  - [Version 5.2.2](#sec-4-11)
-  - [Version 5.3.0](#sec-4-12)
-  - [Version 5.3.1](#sec-4-13)
+- [Version 7.x Changes](#sec-4)
 - [Documentation](#sec-5)
   - [Specifying Paths](#sec-5-1)
   - [Methods](#sec-5-2)
@@ -32,14 +17,14 @@
     - [append(input, remotePath, options) ==> string](#sec-5-2-10)
     - [mkdir(path, recursive) ==> string](#sec-5-2-11)
     - [rmdir(path, recursive) ==> string](#sec-5-2-12)
-    - [delete(path) ==> string](#sec-5-2-13)
+    - [delete(path, noErrorOK) ==> string](#sec-5-2-13)
     - [rename(fromPath, toPath) ==> string](#sec-5-2-14)
     - [posixRename(fromPath, toPath) ==> string](#sec-5-2-15)
     - [chmod(path, mode) ==> string](#sec-5-2-16)
     - [realPath(path) ===> string](#sec-5-2-17)
     - [cwd() ==> string](#sec-5-2-18)
-    - [uploadDir(srcDir, dstDir) ==> string](#sec-5-2-19)
-    - [downloadDir(srcDir, dstDir) ==> string](#sec-5-2-20)
+    - [uploadDir(srcDir, dstDir, filter) ==> string](#sec-5-2-19)
+    - [downloadDir(srcDir, dstDir, filter) ==> string](#sec-5-2-20)
     - [end() ==> boolean](#sec-5-2-21)
     - [Add and Remove Listeners](#sec-5-2-22)
 - [Platform Quirks & Warnings](#sec-6)
@@ -60,6 +45,8 @@
     - [Not returning the promise in a `then()` block](#sec-9-1-1)
     - [Mixing Promise Chains and Async/Await](#sec-9-1-2)
     - [Try/catch and Error Handlers](#sec-9-1-3)
+    - [Server Differences](#sec-9-1-4)
+    - [Avoid Concurrent Operations](#sec-9-1-5)
   - [Debugging Support](#sec-9-2)
 - [Logging Issues](#sec-10)
 - [Pull Requests](#sec-11)
@@ -70,18 +57,14 @@
 
 an SFTP client for node.js, a wrapper around [SSH2](https://github.com/mscdex/ssh2) which provides a high level convenience abstraction as well as a Promise based API.
 
-Documentation on the methods and available options in the underlying modules can be found on the [SSH2](https://github.com/mscdex/ssh2) and [SSH2-STREAMS](https://github.com/mscdex/ssh2-streams/blob/master/SFTPStream.md) project pages.
+Documentation on the methods and available options in the underlying modules can be found on the [SSH2](https://github.com/mscdex/ssh2) project pages.
 
-Current stable release is **v5.3.1**.
+Current stable release is **v7.0.0**.
 
-Code has been tested against Node versions 12.18.2 and 13.14.0
+Code has been tested against Node versions 12.22.1, 14.17.0 and 16.2.0
 
 Node versions < 10.x are not supported.
 
-<span class="underline">WARNING</span> There is currently a regression error with versions of node later than version 14.0. It appears that when using streams with chunk sizes which exceed the high water mark for the stream, a drain event is no longer emitted. As a result, streams with sufficient data will hang indefinitely. This appears to affect fastput, fastget, put and possibly get operations. Until this issue is resolved and a new version of ssh2/ssh2-streams is released, using node v14 is not recommended.
-
-A bug report hass been logged against the ssh2-streams library as [issue 156](https://github.com/mscdex/ssh2-streams/issues/156).
-
 # Installation<a id="sec-2"></a>
 
 ```shell
@@ -108,110 +91,13 @@ sftp.connect({
 });
 ```
 
-# Version 5.x<a id="sec-4"></a>
-
-## Breaking Changes in Version 5.x<a id="sec-4-1"></a>
-
--   The auxList() method has been removed. This method was flagged as deprecated in version 4.x. The functionality provided by `auxList()` is available in `list()`, making `auxList()` unnecessary.
--   The realPath() method now returns `''` if the path does not exist rather than throwing an exception.
--   Improved error handling. The `ssh2` and `ssh2-streams` libraries use events to signal errors. Providing a clean Promise based API and managing these events can be challenging as an error event can fire at any time (including in-between the resolution of one promise and the commencement of another). As you cannot use `try/catch` blocks to reliably manage error events (for a similar reason - see Node's event documentation for details), a slightly more complex solution was required. See the section below on Error Event Handling for more details. In basic terms, a default handler is now used that will log the error and clear the SFTP connection if no Promise error handler has handled the error. This prevents the uncaughtException error and provides a reasonably clean way to deal with unexpected errors that fire in-between Promise execution activities.
--   Ignore Errors during `end()` processing. At least one SFTP server (Azure SFTP) seems to generate an error in response to the `end()` call. As `end()` has been called, we don't really care if an error occurs provided the connection is closed. Therefore, a new default error listener for the `end()` method has been added that will simply ignore any errors which occur during a call to end the connection.
-
-### Error Event Handling<a id="sec-4-1-1"></a>
-
-Providing a clean Promise API for the SSH2 to manage basic SFTP functionality presents a couple of challenges for managing errors. The `SSH2` module uses events to communicate various state changes and error conditions. These events can fire at any time.
-
-On the client side, we wrap basic SFTP actions in Javascript Promises, allowing clients to use either the standard Promise API or async/await to model SFTP interactions. Creating an SFTP connection returns a promise, which resolves if a connection is successfully established and is rejected otherwise. Downloading a file using `get()` or `fastGet()` generates a new Promise which is either resolved, indicating file has been successfully downloaded or rejected, indicating the download failed. All pretty straight-forward.
-
-When the Promise is created, an error event handler is added to the SFTP object to catch any errors that fire during the execution of the promise. If an error event fires, the Promise is rejected and the error returned to the client as part of the rejection. After the Promise has resolved or rejected, the error listener is removed (the error listener is specific to each promise because it needs to call the reject method associated with that promise). As a promise can only be resolved or rejected once, after the Promise has completed, the error listener is of no further use.
-
-This all works fine when an error event fires during the execution of a Promise. However, what about outside promise execution? Consider the following flow;
-
-1.  You have an active SFTP connection which you use to download a file
-2.  When you make the download request, a new Promise is created which will resolve when the file is downloaded or be rejected if the download fails for some reason. The promise resolves successfully.
-3.  You start processing the data downloaded. At this point, you still have an open connection to the SFTP server, but you are not actively interacting with it. There is no active Promise in play.
-4.  The remote SFTP server resets the connection for some reason, generating a ECONNRESET error that is emitted as an error event.
-
-What happens at this point? There is no active promise executing, so no Promise specific error handler in play. Your script is off processing the data from the previously downloaded file, so there is no currently executing try/catch block around the SFTP client object. Basically, there is nothing listening of any errors at this point. What will happen?
-
-Well, basically, the error event will bubble up to the top level of the node process context and cause an uncaughtException error, display the error and dump a stack trace and cause the node process to exit. In basic terms, your process will crash. Not a great outcome.
-
-There are a number of things we can do to improve the situation. However, nearly all of them have some drawbacks. We could -
-
--   Add our own error handler. The `client.on()` method would allow you to add your own error handler. This would provide a way to manage error events, but you want to make sure you only handle error events not handled already by the Promise error handlers. Worse yet, you cannot know before hand the processing context of your script at the point the error event fires. This means your error handling is likely to be complex and difficult to manage. Worse yet, these types of errors are quite rare in most situations and your now being required to add significant additional complexity to deal with a rare edge case. However, sometimes, you just need to deal with this sort of complexity and the `client.on()` method does give you that option.
--   Another alternative is to just add an uncaughtException handler to your Node process object. This would also prevent node from dumping the error and exiting abruptly. However, now you need to think about ALL the possible uncaughtExceptions which might happen, not just those associated with the SFTP client. Again, things are getting complicated for something which only occurs occasionally. .
-
-What we really want is a solution which will be simple for the majority of clients, but provide additional power when needed. What we have done is add a default error handler which will only take action if no Promise error handler has fired. All the default error handler does is log the error to console.error() and set the SFTP connection to undefined so that any further attempts to use the connection will throw an error inside the Promise which attempts to use it.
-
-The advantage of this approach is that it stops the abrupt exiting of the node script due to an uncaught exception error and provides a reasonable outcome for most use cases. For example, in the scenario outlined above, if an error event fires while your script is processing the data already downloaded, it will not impact on your script immediately. An error will be logged to console.error(), but your script will continue to run. Once you have completed processing your data, if you attempt another SFTP call, it will fail with an error about no available SFTP connections. As this will occur within the context of interacting with the SFTP server, your script can take appropriate action to resolve the issue (such as re-connecting to the server). On the other hand, if after processing the file your done and just want to end, then you can just ignore the error, perform any necessary cleanup work and exit successfully.
-
-### Technical Details<a id="sec-4-1-2"></a>
-
-The event handlers added by each Promise are added using the `prependListener()` function. This ensures the handler is fired before any other error handlers which may be defined. As part of the processing, these error handler set a flag property `this.errorHandled` to true, indicating the error has been handled.
-
-In addition to the Promise error handlers, there is a default error handler which will fire after any Promise error handler. The default error handler looks to see if the `this.errorHandler` flag is true. If it is, it knows the error has been handled and it just resets it to false, taking no other action (so taht we are ready for the next error). If the flag is false, the default handler knows it must handle the error. In this case, the handler will log the error to `console.error()`, will set the SFTP connection to undefined to prevent any further attempts to use it and finally, ensure the `this.errorHandler` flag is reset to false in preparation for the next error.
-
-## New Methods<a id="sec-4-2"></a>
-
--   Added the method uploadDir(). This method will upload a directory (including any subdirectories) to the remote server. Only directories and regular files are uploaded (no symbolic links, FIFOs, socket FDs etc). Will overwrite existing files or directories, but will not delete any remote files or directories.
--   Added the method downloadDir(). This method will download a directory (including any subdirectories) to the local file system. Only directories and regular files are downloaded (no symbolic links, FIFOs, socket FDs etc).. Will overwrite existing files or directories, but will not delete any local files in the directories.
--   Added the method posixRename(). This method will use the POSIX atomic rename openSSH extension. As this is an extension to the SFTP protocol, not all servers will support this operation.
-
-## Version 5.0.1<a id="sec-4-3"></a>
-
--   The error checking was a little too stringent. The use of exist() to test for file types had a problem when the user does not have read/execute rights on the directory. Replaced with stat() method, which should avoid this issue.
-
-## Version 5.0.2<a id="sec-4-4"></a>
-
--   Fix error in local directory tests due to missing await statement.
--   Fix path handling under win32. Paths were not being parsed correctly due to the use of path.posix.parse() instead of path.parse().
-
-## Version 5.1.0<a id="sec-4-5"></a>
-
--   Add missing connection check in end() method
--   Add debugging support. Now adding a debug property to the connection configuration object will enable debugging. The value of the debug property should be a function which accepts a single string argument. Typically, this function will send the value passed in to stderr or a file.
--   Fix bug in checkRemotePath() relating to poor path specifications where you cannot determine parent directory.
-
-## Version 5.1.1<a id="sec-4-6"></a>
-
--   Bug fix for unexpected close of connections. It would seem that a connections can be unexpectedly closed without an accompanying error event. As methods only looked for error events, the method promise wold never fulfil and the method would appear to hang. Have now added close event handlers to each method that will reject the promise if the connection is closed unexpectedly.
--   Missing return statement in connect method would result in the connect method attempting to re-connect again after it had reached maximum connect retries. Added the missing return statement.
--   Added some more troubleshooting documentation. Numerous issues have been raised that turn out to be due to client code failing to return Promises inside promise chains. Common symptom is what appears to be truncated file upload/download. What is really happening is that the end method is being called before the transfer has completed.
+# Version 7.x Changes<a id="sec-4"></a>
 
-## Version 5.1.2<a id="sec-4-7"></a>
+-   This version is based on version 1.1.0 of `ssh2`. This version of `ssh2` is a complete re-write of the `ssh2` library. This re-write addresses issues encountered when using node v14 as well as some design weaknesses in the previous 0.8.x version.
 
--   Mainly a bug fix. We needed to add back a global close listener to ensure the sftp object is unset whenever a close event occurs. As close events can occur outside main method calls, only having method based listeners was not sufficient.
--   Also added a utils.dumpListeners() method, useful when debugging issues with listener 'leakage' due to failure to remove listeners when no longer required.
+-   **Breaking Change** Expanded option handling for `get()` and `put()` methods. A number of use cases were identified where setting specific options on the read and write streams and the pipe operation are necessary. For example, disabling `autoClose` on streams or the `end` event in pipes. The `options` argument for `get()` and `put()` calls now supports properties for `readStreamOptions`, `writeStreamOptions` and `pipeOptions`. Note that options are only applied to streams created by the `get()` and `put()` methods. Streams passed into these methods are under the control of the client code and therefore cannot have options supplied in arguments to those streams (you would apply such options when you create the streams). Options are typically only necessary in special use cases. Most of the time, no options are required. However, if you are currently using options to either `put()` or `get()`, you will need to update your code to map these options to the new structure.
 
-## Version 5.1.3<a id="sec-4-8"></a>
-
--   Fix issue with permissions for writing to root directory
--   Cleanup tests to use less connections and eliminate need for test delays
--   Bumped some dependencies to latest versions
-
-## Version 5.2.0<a id="sec-4-9"></a>
-
--   Add posixRename() method. This is an openssh extension added in openssh v4.8 and will only work on servers which support this extension.conflict
--   Bumped through2 dependency version to 4.0.2
-
-## Version 5.2.1<a id="sec-4-10"></a>
-
--   Move some dev dependencies from dependencies to devDependencies.
-
-## Version 5.2.2<a id="sec-4-11"></a>
-
--   Bug fix. Some servers appear to issue errors with code 4 instead of code 2 for file not found errors. This version adds checks for error code 4 to the stat() method. Thanks to teenangst for the fix.
-
-## Version 5.3.0<a id="sec-4-12"></a>
-
--   Add code to only add connect() and end() event handlers if they are not already active. For connect(), remove event handlers as late as possible to help catch error events raised late on some platforms (like win32). don't remove end() error handler as some platforms, like win32, send an additional error event even after a successful and requested end() call.
--   Fix path handling when connecting to a remoe SFTP server running on win32 platform. Assume server honours 'nix' path convention rather than using native win32 path format.
--   Add additional documentation on events/promises, platform quirks and platform differences.
-
-## Version 5.3.1<a id="sec-4-13"></a>
-
--   Fix bug in handling of relative local paths
--   Modified `get()` and `put()` methods to support special purpose streams which require `autoClose` to be `false`. These methods will now look for the `autoClose: false` property in the options object and if it is false, will issue a `destroy()` on the underlying stream just before the promise is resolved. The default is `autoClose: true` and this default should be used unless there is a known specific reason to change it to false.
+-   Improved event handling. A listener for a global error event is now defined to catch errors which occur in-between method calls i.e. connection lost in-between calls to the library methods. A new mechanism has also been added for removal of listeners when no longer required.
 
 # Documentation<a id="sec-5"></a>
 
@@ -221,7 +107,7 @@ All the methods will return a Promise, except for `on()` and `removeListener()`,
 
 ## Specifying Paths<a id="sec-5-1"></a>
 
-The convention with both FTP and SFTP is that paths are specified using a 'nix' style i.e. use '*' as the path separator. This means that even if your SFTP server is running on a win32 platform, you should use '*' instead of '\\' as the path separator. For example, for a win32 path of 'C:\Users\fred' you would actually use '/C:/Users/fred'. If your win32 server does not support the 'nix' path convention, you can try setting the `remotePathSep` property of the `SftpClient` object to the path separator of your remote server. This **might** work, but has not been tested. Please let me know if you need to do this and provide details of the SFTP server so that I can try to create an appropriate environment and adjust things as necessary. At this point, I'm not aware of any win32 based SFTP servers which do not support the 'nix' path convention.
+The convention with both FTP and SFTP is that paths are specified using a 'nix' style i.e. use `/` as the path separator. This means that even if your SFTP server is running on a win32 platform, you should use `/` instead of `\` as the path separator. For example, for a win32 path of `C:\Users\fred` you would actually use `/C:/Users/fred`. If your win32 server does not support the 'nix' path convention, you can try setting the `remotePathSep` property of the `SftpClient` object to the path separator of your remote server. This **might** work, but has not been tested. Please let me know if you need to do this and provide details of the SFTP server so that I can try to create an appropriate environment and adjust things as necessary. At this point, I'm not aware of any win32 based SFTP servers which do not support the 'nix' path convention.
 
 All remote paths must either be absolute e.g. `/absolute/path/to/file` or they can be relative with a prefix of either `./` (relative to current remote directory) or `../` (relative to parent of current remote directory) e.g. `./relative/path/to/file` or `../relative/to/parent/file`. It is also possible to do things like `../../../file` to specify the parent of the parent of the parent of the current remote directory. The shell tilde (`~`) and common environment variables like `$HOME` are NOT supported.
 
@@ -309,7 +195,7 @@ Connect to an sftp server. Full documentation for connection options is availabl
       password: 'borsch', // string Password for password-based user authentication
       agent: process.env.SSH_AGENT, // string - Path to ssh-agent's UNIX socket
       privateKey: fs.readFileSync('/path/to/key'), // Buffer or string that contains
-      passphrase; 'a pass phrase', // string - For an encrypted private key
+      passphrase: 'a pass phrase', // string - For an encrypted private key
       readyTimeout: 20000, // integer How long (in ms) to wait for the SSH handshake
       strictVendor: true // boolean - Performs a strict server vendor check
       debug: myDebug // function - Set this to a function that receives a single
@@ -510,16 +396,21 @@ In general, if your going to pass in a string as the destination, you are better
 
 1.  Options
 
-    The options object can be used to pass options to the underlying readStream used to read the data from the remote server.
+    The `options` argument can be used to pass options to the underlying streams and pipe call used by this method. The argument is an object with three possible properties, `readStreamOptions`, `writeStreamOptions` and `pipeOptions`. The values for each of these properties should be an object containing the required options. For example, possible read stream and pipe options could be defined as
     
     ```javascript
-    {
-      flags: 'r',
-      encoding: null,
-      handle: null,
-      mode: 0o666,
-      autoClose: true
-    }
+    let options = {
+      readStreamOptions: {
+        flags: 'r',
+        encoding: null,
+        handle: null,
+        mode: 0o666,
+        autoClose: true
+      },
+      pipeOptions: {
+        end: false
+      }};
+    
     ```
     
     Most of the time, you won't want to use any options. Sometimes, it may be useful to set the encoding. For example, to 'utf-8'. However, it is important not to do this for binary files to avoid data corruption.
@@ -592,19 +483,20 @@ Upload data from local system to remote server. If the `src` argument is a strin
 
 -   **src:** string | buffer | readable stream. Data source for data to copy to the remote server.
 -   **remotePath:** string. Path to the remote file to be created on the server.
--   **options:** object. Options which can be passed to adjust the write stream used in sending the data to the remote server (see below).
+-   **options:** object. Options which can be passed to adjust the read and write stream used in sending the data to the remote server or the pipe call used to make the data transfer (see below).
 
 1.  Options
 
-    The following options are supported;
+    The options object supports three properties, `readStreamOptions`, `writeStreamOptions` and `pipeOptions`. The value for each property should be an object with options as properties and their associated values representing the option value. For example, you might use the following to set `writeStream` options.
     
     ```javascript
     {
-      flags: 'w',  // w - write and a - append
-      encoding: null, // use null for binary files
-      mode: 0o666, // mode to use for created file (rwx)
-      autoClose: true // automatically close the write stream when finished
-    }
+      writeStreamOptions: {
+        flags: 'w',  // w - write and a - append
+        encoding: null, // use null for binary files
+        mode: 0o666, // mode to use for created file (rwx)
+        autoClose: true // automatically close the write stream when finished
+    }}
     ```
     
     The most common options to use are mode and encoding. The values shown above are the defaults. You do not have to set encoding to utf-8 for text files, null is fine for all file types. However, using utf-8 encoding for binary files will often result in data corruption.
@@ -765,12 +657,14 @@ Remove a directory. If removing a directory and recursive flag is set to `true`,
       });
     ```
 
-### delete(path) ==> string<a id="sec-5-2-13"></a>
+### delete(path, noErrorOK) ==> string<a id="sec-5-2-13"></a>
 
 Delete a file on the remote server.
 
 -   **path:** string. Path to remote file to be deleted.
 
+-   **noErrorOK:** boolean. If true, no error is raised when you try to delete a non-existent file. Default is false.
+
 1.  Example Use
 
     ```javascript
@@ -850,7 +744,7 @@ Change the mode (read, write or execute permissions) of a remote file or directo
 
     ```javascript
     let path = '/path/to/remote/file.txt';
-    let ndwMode = 0o644;  // rw-r-r
+    let newMode = 0o644;  // rw-r-r
     let client = new Client();
     
     client.connect(config)
@@ -877,14 +771,17 @@ Converts a relative path to an absolute path on the remote server. This method i
 
 Returns what the server believes is the current remote working directory.
 
-### uploadDir(srcDir, dstDir) ==> string<a id="sec-5-2-19"></a>
+### uploadDir(srcDir, dstDir, filter) ==> string<a id="sec-5-2-19"></a>
 
 Upload the directory specified by `srcDir` to the remote directory specified by `dstDir`. The `dstDir` will be created if necessary. Any sub directories within `srcDir` will also be uploaded. Any existing files in the remote path will be overwritten.
 
 The upload process also emits 'upload' events. These events are fired for each successfully uploaded file. The `upload` event calls listeners with 1 argument, an object which has properties source and destination. The source property is the path of the file uploaded and the destination property is the path to where the file was uploaded to. The purpose of this event is to provide some way for client code to get feedback on the upload progress. You can add your own lisener using the `on()` method.
 
+The optionsl *filter* argument is a regular expression which can be used to select which files and directories to include in the upload.
+
 -   **srcDir:** A local file path specified as a string
 -   **dstDir:** A remote file path specified as a string
+-   **filter:** A regular expression used to filter which files and directories to include in the upload
 
 1.  Example
 
@@ -934,14 +831,17 @@ The upload process also emits 'upload' events. These events are fired for each s
     
     ```
 
-### downloadDir(srcDir, dstDir) ==> string<a id="sec-5-2-20"></a>
+### downloadDir(srcDir, dstDir, filter) ==> string<a id="sec-5-2-20"></a>
 
 Download the remote directory specified by `srcDir` to the local file system directory specified by `dstDir`. The `dstDir` directory will be created if required. All sub directories within `srcDir` will also be copied. Any existing files in the local path will be overwritten. No files in the local path will be deleted.
 
 The method also emites `download` events to provide a way to monitor download progress. The download event listener is called with one argument, an object with two properties, source and destination. The source property is the path to the remote file that has been downloaded and the destination is the local path to where the file was downloaded to. You can add a listener for this event using the `on()` method.
 
+The optional *filter* argument is a regular expression which can be used to select which files and directories will be downloaded from the remote server.
+
 -   **srcDir:** A remote file path specified as a string
 -   **dstDir:** A local file path specified as a string
+-   **filter:** A regular expression used to match the files and directories to be downloaded
 
 1.  Example
 
@@ -1032,7 +932,7 @@ Although normally not required, you can add and remove custom listeners on the s
 
 ## Server Capabilities<a id="sec-6-1"></a>
 
-All SFTP servers and platforms are not equal. Some facilities provided by `ssh2-sfto-client` either depend on capabilities of the remote server or the underlying capabilities of the remote server platform. As an example, consider `chmod()`. This command depends on a remote filesystem which implements the 'nix' concept of users and groups. The *win32* platform does not have the same concept of users and groups, so `chmod()` will not behave in the same way.
+All SFTP servers and platforms are not equal. Some facilities provided by `ssh2-sftp-client` either depend on capabilities of the remote server or the underlying capabilities of the remote server platform. As an example, consider `chmod()`. This command depends on a remote filesystem which implements the 'nix' concept of users and groups. The *win32* platform does not have the same concept of users and groups, so `chmod()` will not behave in the same way.
 
 One way to determine whether an issue you are encountering is due to `ssh2-sftp-client` or due to the remote server or server platform is to use a simple CLI sftp program, such as openSSH's sftp command. If you observe the same behaviour using plain `sftp` on the command line, the issue is likely due to server or remote platform limitations. Note that you should not use a GUI sftp client, like `Filezilla` or `winSCP` as such GUI programs often attempt to hide these server and platform incompatibilities and will take additional steps to simulate missing functionality etc. You want to use a CLI program which does as little as possible.
 
@@ -1248,7 +1148,7 @@ If you run into an issue which is not repeatable with just the `ssh2` and `ssh2-
 
 Note also that in the repository there are two useful directories. The first is the examples directory, which contain some examples of using `ssh2-sftp-client` to perform common tasks. A few minutes reviewing these examples can provide that additional bit of detail to help fix any problems you are encountering.
 
-The second directory is the tools directory. I have some very basic simple scripts in this directory which perform basic tasks using only the `ssh2` and `ssh2-streams` modules (no ssh2-sftp-client module). These can be useful when trying to determine if the issue is with the underlying `ssh2` and `ssh2-streams` modules.
+The second directory is the validation directory. I have some very simple scripts in this directory which perform basic tasks using only the `ssh2` modules (no `ssh2-sftp-client` module). These can be useful when trying to determine if the issue is with the underlying `ssh2` module or the `ssh2-sftp-client` wrapper module.
 
 ## Common Errors<a id="sec-9-1"></a>
 
@@ -1256,7 +1156,7 @@ There are some common errors people tend to make when using Promises or Asyc/Awa
 
 ### Not returning the promise in a `then()` block<a id="sec-9-1-1"></a>
 
-All methods in `ssh2-sftp-client` return a Promise. This means methods are executed *asynchrnously*. When you call a method inside the `then()` block of a promise chain, it is critical that you return the Promise that call generates. Failing to do this will result in the `then()` block completing and your code starting execution of the next `then()`, `catch()` or `finally()` block before your promise has been fulfilled. For exmaple, the following will not do what you expect
+All methods in `ssh2-sftp-client` return a Promise. This means methods are executed *asynchrnously*. When you call a method inside the `then()` block of a promise chain, it is critical that you return the Promise that call generates. Failing to do this will result in the `then()` block completing and your code starting execution of the next `then()`, `catch()` or `finally()` block before your promise has been fulfilled. For example, the following will not do what you expect
 
 ```javascript
 sftp.connect(config)
@@ -1270,7 +1170,7 @@ sftp.connect(config)
   });
 ```
 
-In the above code, the `sftp.end()` method will almost certainly be called before `sftp.gastGet()` has been fulfilled (unless the *foo.txt* file is really small!). In fact, the whole promise chain will complete and exit even before the `sftp.end()` call has been fulfilled. The correct code would be something like
+In the above code, the `sftp.end()` method will almost certainly be called before `sftp.fastGet()` has been fulfilled (unless the *foo.txt* file is really small!). In fact, the whole promise chain will complete and exit even before the `sftp.end()` call has been fulfilled. The correct code would be something like
 
 ```javascript
 sftp.connect(config)
@@ -1353,6 +1253,16 @@ The basic problem is that the try/catch block will have completed execution befo
 
 Error events are essentially asynchronous code. You don't know when such events will fire. Therefore, you cannot use a try/catch block to catch such event errors. Even creating an error handler which then throws an exception won't help as the key problem is that your try/catch block has already executed. There are a number of alternative ways to deal with this situation. However, the key symptom is that you see occasional uncaught error exceptions that cause your script to exit abnormally despite having try/catch blocks in your script. What you need to do is look at your code and find where errors are raised asynchronously and use an event handler or some other mechanism to manage any errors raised.
 
+### Server Differences<a id="sec-9-1-4"></a>
+
+Not all SFTP servers are the same. Like most standards, the SFTP protocol has some level of interpretation and allows different levels of compliance. This means there can be differences in behaviour between different servers and code which works with one server will not work the same with another. For example, the value returned by *realpath* for non-existent objects can differ significantly. Some servers will throw an error for a particular operation while others will just return null, some servers support concurrent operations (such as used by fastGet/fastPut) while others will not and of course, the text of error messages can vary significantly. In particular, we have noticed significant differences across different platforms. It is therefore advisable to do comprehensive testing when the SFTP server is moved to a new platform. This includes moving from to a cloud based service even if the underlying platform remains the same. I have noticed that some cloud platforms can generate unexpected events, possibly related to additional functionality or features associated with the cloud implementation. For example, it appears SFTP servers running under Azure will generate an error event when the connection is closed even when the client has requested the connection be terminated. The same SFTP server running natively on Windows does not appear to exhibit such behaviour.
+
+### Avoid Concurrent Operations<a id="sec-9-1-5"></a>
+
+Technically, SFTP should be able to perform multiple operations concurrently. As node is single threaded, what we a really talking about is running multiple execution contexts as a pool where node will switch contexts when each context is blocked due to things like waiting on network data etc. However, I have found this to be extremely unreliable and of very little benefit from a performance perspective. My recommendation is to therefore avoid executing multiple requests over the same connection in parallel (for example, generating multiple `get()` promises and using something like `Promise.all()` to resolve them.
+
+If you are going to try and perform concurrent operations, you need to test extensively and ensure you are using data which is large enough that context switching does occur (i.e. the request is not completed in a single run). Some SFTP servers will handle concurrent operations better than others.
+
 ## Debugging Support<a id="sec-9-2"></a>
 
 You can add a `debug` property to the config object passed in to `connect()` to turn on debugging. This will generate quite a lot of output. The value of the property should be a function which accepts a single string argument. For example;
@@ -1393,7 +1303,7 @@ I am happy to try and help diagnose and fix any issues you encounter while using
 -   Version of ssh2-sftp-client you are using
 -   Platform your client is running on (Linux, macOS, Windows)
 -   Platform and software for the remote SFTP server when possible
--   Example of your code. By far, the most common issue is incorrect use of the module API. Example code can usually result in such issues being resolved very quickly.
+-   Example of your code or a minimal script which reproduces the issue you are encountering. By far, the most common issue is incorrect use of the module API. Example code can usually result in such issues being resolved very quickly.
 
 Perhaps the best assistance is a minimal reproducible example of the issue. Once the issue can be readily reproduced, it can usually be fixed very quickly.
 
@@ -1421,3 +1331,4 @@ Thanks to the following for their contributions -
 -   **jhorbulyk:** Contributed posixRename() functionality
 -   **teenangst:** Contributed fix for error code 4 in stat() method
 -   **kennylbj:** Contributed example of using a throttle stream to limit upload/download bandwidth.
+-   **anton-erofeev:** Documentation fix