ssh2-sftp-client 7.2.3 → 8.0.0

package/README.md

@@ -1,58 +1,58 @@
 - [Overview](#sec-1)
+  - [Version 8.x.x Changes](#sec-1-1)
 - [Installation](#sec-2)
 - [Basic Usage](#sec-3)
-- [Version 7.x Changes](#sec-4)
-- [Documentation](#sec-5)
-  - [Specifying Paths](#sec-5-1)
-  - [Methods](#sec-5-2)
-    - [new SftpClient(name) ===> SFTP client object](#sec-5-2-1)
-    - [connect(config) ===> SFTP object](#sec-5-2-2)
-    - [list(path, pattern) ==> Array[object]](#sec-5-2-3)
-    - [exists(path) ==> boolean](#sec-5-2-4)
-    - [stat(path) ==> object](#sec-5-2-5)
-    - [get(path, dst, options) ==> String|Stream|Buffer](#sec-5-2-6)
-    - [fastGet(remotePath, localPath, options) ===> string](#sec-5-2-7)
-    - [put(src, remotePath, options) ==> string](#sec-5-2-8)
-    - [fastPut(localPath, remotePath, options) ==> string](#sec-5-2-9)
-    - [append(input, remotePath, options) ==> string](#sec-5-2-10)
-    - [mkdir(path, recursive) ==> string](#sec-5-2-11)
-    - [rmdir(path, recursive) ==> string](#sec-5-2-12)
-    - [delete(path, noErrorOK) ==> string](#sec-5-2-13)
-    - [rename(fromPath, toPath) ==> string](#sec-5-2-14)
-    - [posixRename(fromPath, toPath) ==> string](#sec-5-2-15)
-    - [chmod(path, mode) ==> string](#sec-5-2-16)
-    - [realPath(path) ===> string](#sec-5-2-17)
-    - [cwd() ==> string](#sec-5-2-18)
-    - [uploadDir(srcDir, dstDir, filter) ==> string](#sec-5-2-19)
-    - [downloadDir(srcDir, dstDir, filter) ==> string](#sec-5-2-20)
-    - [end() ==> boolean](#sec-5-2-21)
-    - [Add and Remove Listeners](#sec-5-2-22)
-- [Platform Quirks & Warnings](#sec-6)
-  - [Server Capabilities](#sec-6-1)
-  - [Promises, Events & Managing Exceptions](#sec-6-2)
-    - [Adding Custom Handlers](#sec-6-2-1)
-  - [Windows Based Servers](#sec-6-3)
-  - [Don't Re-use SftpClient Objects](#sec-6-4)
-- [FAQ](#sec-7)
-  - [Remote server drops connections with only an end event](#sec-7-1)
-  - [How can I pass writable stream as dst for get method?](#sec-7-2)
-  - [How can I upload files without having to specify a password?](#sec-7-3)
-  - [How can I connect through a Socks Proxy](#sec-7-4)
-  - [Timeout while waiting for handshake or handshake errors](#sec-7-5)
-  - [How can I limit upload/download speed](#sec-7-6)
-  - [Connection hangs or fails for larger files](#sec-7-7)
-- [Examples](#sec-8)
-- [Troubleshooting](#sec-9)
-  - [Common Errors](#sec-9-1)
-    - [Not returning the promise in a `then()` block](#sec-9-1-1)
-    - [Mixing Promise Chains and Async/Await](#sec-9-1-2)
-    - [Try/catch and Error Handlers](#sec-9-1-3)
-    - [Server Differences](#sec-9-1-4)
-    - [Avoid Concurrent Operations](#sec-9-1-5)
-  - [Debugging Support](#sec-9-2)
-- [Logging Issues](#sec-10)
-- [Pull Requests](#sec-11)
-- [Contributors](#sec-12)
+- [Documentation](#sec-4)
+  - [Specifying Paths](#sec-4-1)
+  - [Methods](#sec-4-2)
+    - [new SftpClient(name) ===> SFTP client object](#sec-4-2-1)
+    - [connect(config) ===> SFTP object](#sec-4-2-2)
+    - [list(path, pattern) ==> Array[object]](#sec-4-2-3)
+    - [exists(path) ==> boolean](#sec-4-2-4)
+    - [stat(path) ==> object](#sec-4-2-5)
+    - [get(path, dst, options) ==> String|Stream|Buffer](#sec-4-2-6)
+    - [fastGet(remotePath, localPath, options) ===> string](#sec-4-2-7)
+    - [put(src, remotePath, options) ==> string](#sec-4-2-8)
+    - [fastPut(localPath, remotePath, options) ==> string](#sec-4-2-9)
+    - [append(input, remotePath, options) ==> string](#sec-4-2-10)
+    - [mkdir(path, recursive) ==> string](#sec-4-2-11)
+    - [rmdir(path, recursive) ==> string](#sec-4-2-12)
+    - [delete(path, noErrorOK) ==> string](#sec-4-2-13)
+    - [rename(fromPath, toPath) ==> string](#sec-4-2-14)
+    - [posixRename(fromPath, toPath) ==> string](#sec-4-2-15)
+    - [chmod(path, mode) ==> string](#sec-4-2-16)
+    - [realPath(path) ===> string](#sec-4-2-17)
+    - [cwd() ==> string](#sec-4-2-18)
+    - [uploadDir(srcDir, dstDir, filter) ==> string](#sec-4-2-19)
+    - [downloadDir(srcDir, dstDir, filter) ==> string](#sec-4-2-20)
+    - [end() ==> boolean](#sec-4-2-21)
+    - [Add and Remove Listeners](#sec-4-2-22)
+- [Platform Quirks & Warnings](#sec-5)
+  - [Server Capabilities](#sec-5-1)
+  - [Promises, Events & Managing Exceptions](#sec-5-2)
+    - [Adding Custom Handlers](#sec-5-2-1)
+  - [Windows Based Servers](#sec-5-3)
+  - [Don't Re-use SftpClient Objects](#sec-5-4)
+- [FAQ](#sec-6)
+  - [Remote server drops connections with only an end event](#sec-6-1)
+  - [How can I pass writable stream as dst for get method?](#sec-6-2)
+  - [How can I upload files without having to specify a password?](#sec-6-3)
+  - [How can I connect through a Socks Proxy](#sec-6-4)
+  - [Timeout while waiting for handshake or handshake errors](#sec-6-5)
+  - [How can I limit upload/download speed](#sec-6-6)
+  - [Connection hangs or fails for larger files](#sec-6-7)
+- [Examples](#sec-7)
+- [Troubleshooting](#sec-8)
+  - [Common Errors](#sec-8-1)
+    - [Not returning the promise in a `then()` block](#sec-8-1-1)
+    - [Mixing Promise Chains and Async/Await](#sec-8-1-2)
+    - [Try/catch and Error Handlers](#sec-8-1-3)
+    - [Server Differences](#sec-8-1-4)
+    - [Avoid Concurrent Operations](#sec-8-1-5)
+  - [Debugging Support](#sec-8-2)
+- [Logging Issues](#sec-9)
+- [Pull Requests](#sec-10)
+- [Contributors](#sec-11)
 
 
 # Overview<a id="sec-1"></a>
@@ -61,12 +61,18 @@ an SFTP client for node.js, a wrapper around [SSH2](https://github.com/mscdex/ss
 
 Documentation on the methods and available options in the underlying modules can be found on the [SSH2](https://github.com/mscdex/ssh2) project pages.
 
-Current stable release is **v7.2.3**.
+Current stable release is **v8.0.0**.
 
 Code has been tested against Node versions 14.19.1, 16.14.2 and 17.8.0
 
 Node versions < 12.x are not supported. However, node v10.x should still work, although some tests will fail due to changes in file system functions used in test setup and tear down.
 
+## Version 8.x.x Changes<a id="sec-1-1"></a>
+
+-   **Breaking Change**: The API for `uploadDir()` and `downloadDir()` has been changed. These methods now expect a function as the optional 3rd argument. Previously, the 3rd argument was a regular expression used to filter out which files and directories should be included in the upload or download action. The method now expects a predicate function which will return true if the target is to be included in the upload or download and false if it is to be excluded. The predicate function will be called with two arguments, a full path to the target object and a boolean value which is true when the target is a directory, false otherwise. If no filter predicate is supplied, all files and directories under the initial target directory will be transferred. At this time, asynchronous filter functions are not supported.
+
+-   Internal Change: The `rmdir()` method has been refactored to enable asynchronous deletion of files and sub-directories. This has significantly increased performance when deleting larger directory trees, especially trees which are *broad* with lots of files and directories at the same level. For deep narrow trees, there is less performance benefit because sub-directories must be removed before parents, which imposes synchronous processing. It is likely that for extremely large directory trees, the additional resources required to run large numbers of asynchronous tasks will become problematic. In such situations, it may be necessary to manually break up the deletion process into multiple `rmdir()` calls. However, this is considered a fairly extreme use case which is rare (a use case which wold also have been problematic with the old implementation as performance would have been very poor).
+
 # Installation<a id="sec-2"></a>
 
 ```shell
@@ -93,25 +99,13 @@ sftp.connect({
 });
 ```
 
-# Version 7.x Changes<a id="sec-4"></a>
-
--   This version is based on version 1.x.x of `ssh2`. This version of `ssh2` is a complete re-write of the `ssh2` library. This re-write addresses issues encountered when using node v14 as well as some design weaknesses in the previous 0.8.x version.
-
--   **Breaking Change** Expanded option handling for `get()` and `put()` methods. A number of use cases were identified where setting specific options on the read and write streams and the pipe operation are necessary. For example, disabling `autoClose` on streams or the `end` event in pipes. The `options` argument for `get()` and `put()` calls now supports properties for `readStreamOptions`, `writeStreamOptions` and `pipeOptions`. Note that options are only applied to streams created by the `get()` and `put()` methods. Streams passed into these methods are under the control of the client code and therefore cannot have options supplied in arguments to those streams (you would apply such options when you create the streams). Options are typically only necessary in special use cases. Most of the time, no options are required. However, if you are currently using options to either `put()` or `get()`, you will need to update your code to map these options to the new structure.
-
--   **Breaking Change 7.1.0** A race condition was identified when using a put() call with a writeStream option of `autoClose: false`. In some situations, the promise would be resolved before the final close of the write stream. This could result in errors if you immediately attempt to access the uploaded file. To avoid this situatioin, the promise is now resolved once a `close` event is emitted. This means that setting `autoClose: false` can no longer be supported. The write stream for `put()` will autoClose once data writing has completed.
-
--   Improved event handling. A listener for a global error event is now defined to catch errors which occur in-between method calls i.e. connection lost in-between calls to the library methods. A new mechanism has also been added for removal of listeners when no longer required.
-
--   uploadDir/downloadDir change in 7.2.3. THe uploadDir() and downloadDir() methods previously used fastPut() and fastGet() to transfer data. Unfortunately, not all SFTP servers support the concurrent processing used by these methods. This meant these methods would fail on some platforms. For now, the fastPut() and fastGet() calls have been replaced with plain put() and get() calls. This will mean uploadDir()/downloadDir() will be slower. However, there is other larger changes in the works which should see a significant speed imp[rovement for these (and other methods). We may also add an option which will allow for selection of fastPut()/fastGet().
-
-# Documentation<a id="sec-5"></a>
+# Documentation<a id="sec-4"></a>
 
 The connection options are the same as those offered by the underlying SSH2 module. For full details, please see [SSH2 client methods](https://github.com/mscdex/ssh2#user-content-client-methods)
 
 All the methods will return a Promise, except for `on()` and `removeListener()`, which are typically only used in special use cases.
 
-## Specifying Paths<a id="sec-5-1"></a>
+## Specifying Paths<a id="sec-4-1"></a>
 
 The convention with both FTP and SFTP is that paths are specified using a 'nix' style i.e. use `/` as the path separator. This means that even if your SFTP server is running on a win32 platform, you should use `/` instead of `\` as the path separator. For example, for a win32 path of `C:\Users\fred` you would actually use `/C:/Users/fred`. If your win32 server does not support the 'nix' path convention, you can try setting the `remotePathSep` property of the `SftpClient` object to the path separator of your remote server. This **might** work, but has not been tested. Please let me know if you need to do this and provide details of the SFTP server so that I can try to create an appropriate environment and adjust things as necessary. At this point, I'm not aware of any win32 based SFTP servers which do not support the 'nix' path convention.
 
@@ -141,9 +135,9 @@ client.put('/home/fred/test.txt', '/remote/dir/test-copy.txt');
 
 This will copy the local file `test.txt` to the remote file `test-copy.txt` in the directory `/remote/dir`.
 
-## Methods<a id="sec-5-2"></a>
+## Methods<a id="sec-4-2"></a>
 
-### new SftpClient(name) ===> SFTP client object<a id="sec-5-2-1"></a>
+### new SftpClient(name) ===> SFTP client object<a id="sec-4-2-1"></a>
 
 Constructor to create a new `ssh2-sftp-client` object. An optional `name` string can be provided, which will be used in error messages to help identify which client has thrown the error.
 
@@ -179,7 +173,7 @@ Constructor to create a new `ssh2-sftp-client` object. An optional `name` string
       });
     ```
 
-### connect(config) ===> SFTP object<a id="sec-5-2-2"></a>
+### connect(config) ===> SFTP object<a id="sec-4-2-2"></a>
 
 Connect to an sftp server. Full documentation for connection options is available [here](https://github.com/mscdex/ssh2#user-content-client-methods)
 
@@ -242,7 +236,7 @@ Connect to an sftp server. Full documentation for connection options is availabl
     });
     ```
 
-### list(path, pattern) ==> Array[object]<a id="sec-5-2-3"></a>
+### list(path, pattern) ==> Array[object]<a id="sec-4-2-3"></a>
 
 Retrieves a directory listing. This method returns a Promise, which once realised, returns an array of objects representing items in the remote directory.
 
@@ -309,7 +303,7 @@ Retrieves a directory listing. This method returns a Promise, which once realise
     
     The *glob*-style matching is very simple. In most cases, you are best off using a real regular expression which will allow you to do more powerful matching and anchor matches to the beginning/end of the string etc.
 
-### exists(path) ==> boolean<a id="sec-5-2-4"></a>
+### exists(path) ==> boolean<a id="sec-4-2-4"></a>
 
 Tests to see if remote file or directory exists. Returns type of remote object if it exists or false if it does not.
 
@@ -342,7 +336,7 @@ Tests to see if remote file or directory exists. Returns type of remote object i
       });
     ```
 
-### stat(path) ==> object<a id="sec-5-2-5"></a>
+### stat(path) ==> object<a id="sec-4-2-5"></a>
 
 Returns the attributes associated with the object pointed to by `path`.
 
@@ -390,7 +384,7 @@ Returns the attributes associated with the object pointed to by `path`.
       });
     ```
 
-### get(path, dst, options) ==> String|Stream|Buffer<a id="sec-5-2-6"></a>
+### get(path, dst, options) ==> String|Stream|Buffer<a id="sec-4-2-6"></a>
 
 Retrieve a file from a remote SFTP server. The `dst` argument defines the destination and can be either a string, a stream object or undefined. If it is a string, it is interpreted as the path to a location on the local file system (path should include the file name). If it is a stream object, the remote data is passed to it via a call to pipe(). If `dst` is undefined, the method will put the data into a buffer and return that buffer when the Promise is resolved. If `dst` is defined, it is returned when the Promise is resolved.
 
@@ -443,7 +437,7 @@ In general, if you're going to pass in a string as the destination, you are bett
     
     -   **Tip:** See examples file in the Git repository for more examples. You can pass any writeable stream in as the destination. For example, if you pass in `zlib.createGunzip()` writeable stream, you can both download and decompress a gzip file 'on the fly'.
 
-### fastGet(remotePath, localPath, options) ===> string<a id="sec-5-2-7"></a>
+### fastGet(remotePath, localPath, options) ===> string<a id="sec-4-2-7"></a>
 
 Downloads a file at remotePath to localPath using parallel reads for faster throughput. This is the simplest method if you just want to download a file.
 
@@ -483,7 +477,7 @@ Downloads a file at remotePath to localPath using parallel reads for faster thro
       });
     ```
 
-### put(src, remotePath, options) ==> string<a id="sec-5-2-8"></a>
+### put(src, remotePath, options) ==> string<a id="sec-4-2-8"></a>
 
 Upload data from local system to remote server. If the `src` argument is a string, it is interpreted as a local file path to be used for the data to transfer. If the `src` argument is a buffer, the contents of the buffer are copied to the remote file and if it is a readable stream, the contents of that stream are piped to the `remotePath` on the server.
 
@@ -530,7 +524,7 @@ Upload data from local system to remote server. If the `src` argument is a strin
     
     -   **Tip:** If the src argument is a path string, consider just using `fastPut()`.
 
-### fastPut(localPath, remotePath, options) ==> string<a id="sec-5-2-9"></a>
+### fastPut(localPath, remotePath, options) ==> string<a id="sec-4-2-9"></a>
 
 Uploads the data in file at `localPath` to a new file on remote server at `remotePath` using concurrency. The options object allows tweaking of the fast put process.
 
@@ -571,7 +565,7 @@ Uploads the data in file at `localPath` to a new file on remote server at `remot
       });
     ```
 
-### append(input, remotePath, options) ==> string<a id="sec-5-2-10"></a>
+### append(input, remotePath, options) ==> string<a id="sec-4-2-10"></a>
 
 Append the `input` data to an existing remote file. There is no integrity checking performed apart from normal writeStream checks. This function simply opens a writeStream on the remote file in append mode and writes the data passed in to the file.
 
@@ -612,7 +606,7 @@ Append the `input` data to an existing remote file. There is no integrity checki
       });
     ```
 
-### mkdir(path, recursive) ==> string<a id="sec-5-2-11"></a>
+### mkdir(path, recursive) ==> string<a id="sec-4-2-11"></a>
 
 Create a new directory. If the recursive flag is set to true, the method will create any directories in the path which do not already exist. Recursive flag defaults to false.
 
@@ -637,7 +631,7 @@ Create a new directory. If the recursive flag is set to true, the method will cr
       });
     ```
 
-### rmdir(path, recursive) ==> string<a id="sec-5-2-12"></a>
+### rmdir(path, recursive) ==> string<a id="sec-4-2-12"></a>
 
 Remove a directory. If removing a directory and recursive flag is set to `true`, the specified directory and all sub-directories and files will be deleted. If set to false and the directory has sub-directories or files, the action will fail.
 
@@ -664,7 +658,7 @@ Remove a directory. If removing a directory and recursive flag is set to `true`,
       });
     ```
 
-### delete(path, noErrorOK) ==> string<a id="sec-5-2-13"></a>
+### delete(path, noErrorOK) ==> string<a id="sec-4-2-13"></a>
 
 Delete a file on the remote server.
 
@@ -690,7 +684,7 @@ Delete a file on the remote server.
       });
     ```
 
-### rename(fromPath, toPath) ==> string<a id="sec-5-2-14"></a>
+### rename(fromPath, toPath) ==> string<a id="sec-4-2-14"></a>
 
 Rename a file or directory from `fromPath` to `toPath`. You must have the necessary permissions to modify the remote file.
 
@@ -716,7 +710,7 @@ Rename a file or directory from `fromPath` to `toPath`. You must have the necess
       });
     ```
 
-### posixRename(fromPath, toPath) ==> string<a id="sec-5-2-15"></a>
+### posixRename(fromPath, toPath) ==> string<a id="sec-4-2-15"></a>
 
 This method uses the openssh POSIX rename extension introduced in OpenSSH 4.8. The advantage of this version of rename over standard SFTP rename is that it is an atomic operation and will allow renaming a resource where the destination name exists. The POSIX rename will also work on some filesystems which do not support standard SFTP rename because they don't support the system hardlink() call. The POSIX rename extension is available on all openSSH servers from 4.8 and some other implementations. This is an extension to the standard SFTP protocol and therefore is not supported on all sSFTP servers.
 
@@ -740,7 +734,7 @@ client.connect(config)
   });
 ```
 
-### chmod(path, mode) ==> string<a id="sec-5-2-16"></a>
+### chmod(path, mode) ==> string<a id="sec-4-2-16"></a>
 
 Change the mode (read, write or execute permissions) of a remote file or directory.
 
@@ -766,7 +760,7 @@ Change the mode (read, write or execute permissions) of a remote file or directo
       });
     ```
 
-### realPath(path) ===> string<a id="sec-5-2-17"></a>
+### realPath(path) ===> string<a id="sec-4-2-17"></a>
 
 Converts a relative path to an absolute path on the remote server. This method is mainly used internally to resolve remote path names.
 
@@ -774,81 +768,83 @@ Converts a relative path to an absolute path on the remote server. This method i
 
 -   **path:** A file path, either relative or absolute. Can handle '.' and '..', but does not expand '~'.
 
-### cwd() ==> string<a id="sec-5-2-18"></a>
+### cwd() ==> string<a id="sec-4-2-18"></a>
 
 Returns what the server believes is the current remote working directory.
 
-### uploadDir(srcDir, dstDir, filter) ==> string<a id="sec-5-2-19"></a>
+### uploadDir(srcDir, dstDir, filter) ==> string<a id="sec-4-2-19"></a>
 
 Upload the directory specified by `srcDir` to the remote directory specified by `dstDir`. The `dstDir` will be created if necessary. Any sub directories within `srcDir` will also be uploaded. Any existing files in the remote path will be overwritten.
 
-The upload process also emits 'upload' events. These events are fired for each successfully uploaded file. The `upload` event calls listeners with 1 argument, an object which has properties source and destination. The source property is the path of the file uploaded and the destination property is the path to where the file was uploaded to. The purpose of this event is to provide some way for client code to get feedback on the upload progress. You can add your own lisener using the `on()` method.
+The upload process also emits 'upload' events. These events are fired for each successfully uploaded file. The `upload` event calls listeners with 1 argument, an object which has properties source and destination. The source property is the path of the file uploaded and the destination property is the path to where the file was uploaded. The purpose of this event is to provide some way for client code to get feedback on the upload progress. You can add your own listener using the `on()` method.
 
-The optionsl *filter* argument is a regular expression which can be used to select which files and directories to include in the upload.
+The optionsl *filter* argument is a function which will be called for each item to be uploaded. The function will be called with two arguments. The first argument is the full path of the item to be uploaded and the second argument is a boolean, which will be true if the target path is for a directory. The filter function will be called for each item in the source path. If the function returns true, the item will be uploaded. If it returns false, it will be filtered and not uploaded. The filter function is called via the `Array.filter` method. These array comprehension methods are known to be unsafe for asynchronous functions. Therefore, only synchronous filter functions are supported at this time.
 
 -   **srcDir:** A local file path specified as a string
 -   **dstDir:** A remote file path specified as a string
--   **filter:** A regular expression used to filter which files and directories to include in the upload
+-   **filter:** A filter predicate function which is called for each item in the source path. The argument will receive two arguments. The first is the full path to the item and the second is a boolean which will be true if the item is a directory. If the function returns true, the item will be uploaded, otherwise it will be filtered out and ignored.
 
 1.  Example
 
     ```javascript
-        'use strict';
+    'use strict';
     
-        // Example of using the uploadDir() method to upload a directory
-        // to a remote SFTP server
+    // Example of using the uploadDir() method to upload a directory
+    // to a remote SFTP server
     
-        const path = require('path');
-        const SftpClient = require('../src/index');
+    const path = require('path');
+    const SftpClient = require('../src/index');
     
-        const dotenvPath = path.join(__dirname, '..', '.env');
-        require('dotenv').config({path: dotenvPath});
+    const dotenvPath = path.join(__dirname, '..', '.env');
+    require('dotenv').config({path: dotenvPath});
     
-        const config = {
-    host: process.env.SFTP_SERVER,
-    username: process.env.SFTP_USER,
-    password: process.env.SFTP_PASSWORD,
-    port: process.env.SFTP_PORT || 22
-        };
+    const config = {
+      host: process.env.SFTP_SERVER,
+      username: process.env.SFTP_USER,
+      password: process.env.SFTP_PASSWORD,
+      port: process.env.SFTP_PORT || 22
+    };
     
-        async function main() {
-    const client = new SftpClient('upload-test');
-    const src = path.join(__dirname, '..', 'test', 'testData', 'upload-src');
-    const dst = '/home/tim/upload-test';
+    async function main() {
+      const client = new SftpClient('upload-test');
+      const src = path.join(__dirname, '..', 'test', 'testData', 'upload-src');
+      const dst = '/home/tim/upload-test';
     
-    try {
-      await client.connect(config);
-      client.on('upload', info => {
-        console.log(`Listener: Uploaded ${info.source}`);
-      });
-      let rslt = await client.uploadDir(src, dst);
-      return rslt;
-    } finally {
-      client.end();
+      try {
+        await client.connect(config);
+        client.on('upload', info => {
+          console.log(`Listener: Uploaded ${info.source}`);
+        });
+        let rslt = await client.uploadDir(src, dst);
+        return rslt;
+      } catch (err) {
+        console.error(err);
+      } finally {
+        client.end();
+      }
     }
-        }
     
-        main()
-    .then(msg => {
-      console.log(msg);
-    })
-    .catch(err => {
-      console.log(`main error: ${err.message}`);
-    });
+    main()
+      .then(msg => {
+        console.log(msg);
+      })
+      .catch(err => {
+        console.log(`main error: ${err.message}`);
+      });
     
     ```
 
-### downloadDir(srcDir, dstDir, filter) ==> string<a id="sec-5-2-20"></a>
+### downloadDir(srcDir, dstDir, filter) ==> string<a id="sec-4-2-20"></a>
 
 Download the remote directory specified by `srcDir` to the local file system directory specified by `dstDir`. The `dstDir` directory will be created if required. All sub directories within `srcDir` will also be copied. Any existing files in the local path will be overwritten. No files in the local path will be deleted.
 
 The method also emites `download` events to provide a way to monitor download progress. The download event listener is called with one argument, an object with two properties, source and destination. The source property is the path to the remote file that has been downloaded and the destination is the local path to where the file was downloaded to. You can add a listener for this event using the `on()` method.
 
-The optional *filter* argument is a regular expression which can be used to select which files and directories will be downloaded from the remote server.
+The optional *filter* argument is a predicate function which will be called with two arguments for each potential item to be downloaded. The first argument is the full path of the item and the second argument is a boolean, which will be true if the item is a directory. If the function returns true, the item will be included in the download. If it returns false, it will be filtered and ignored. The filter function is called via the `Array.filter` method. These array comprehension methods are known to be unsafe for asynchronous functions. Therefore, only synchronous filter functions are supported at this time.
 
 -   **srcDir:** A remote file path specified as a string
 -   **dstDir:** A local file path specified as a string
--   **filter:** A regular expression used to match the files and directories to be downloaded
+-   **filter:** A predicate function called with two arguments, the full path to an item and a boolean value which will be true if the item is a directory. The function is called for each item in the download path.
 
 1.  Example
 
@@ -898,7 +894,7 @@ The optional *filter* argument is a regular expression which can be used to sele
     
     ```
 
-### end() ==> boolean<a id="sec-5-2-21"></a>
+### end() ==> boolean<a id="sec-4-2-21"></a>
 
 Ends the current client session, releasing the client socket and associated resources. This function also removes all listeners associated with the client.
 
@@ -919,7 +915,7 @@ Ends the current client session, releasing the client socket and associated reso
       });
     ```
 
-### Add and Remove Listeners<a id="sec-5-2-22"></a>
+### Add and Remove Listeners<a id="sec-4-2-22"></a>
 
 Although normally not required, you can add and remove custom listeners on the ssh2 client object. This object supports a number of events, but only a few of them have any meaning in the context of SFTP. These are
 
@@ -937,15 +933,15 @@ Although normally not required, you can add and remove custom listeners on the s
 
     Removes the specified listener from the event specified in eventType. Note that the `end()` method automatically removes all listeners from the client object.
 
-# Platform Quirks & Warnings<a id="sec-6"></a>
+# Platform Quirks & Warnings<a id="sec-5"></a>
 
-## Server Capabilities<a id="sec-6-1"></a>
+## Server Capabilities<a id="sec-5-1"></a>
 
 All SFTP servers and platforms are not equal. Some facilities provided by `ssh2-sftp-client` either depend on capabilities of the remote server or the underlying capabilities of the remote server platform. As an example, consider `chmod()`. This command depends on a remote filesystem which implements the 'nix' concept of users and groups. The *win32* platform does not have the same concept of users and groups, so `chmod()` will not behave in the same way.
 
 One way to determine whether an issue you are encountering is due to `ssh2-sftp-client` or due to the remote server or server platform is to use a simple CLI sftp program, such as openSSH's sftp command. If you observe the same behaviour using plain `sftp` on the command line, the issue is likely due to server or remote platform limitations. Note that you should not use a GUI sftp client, like `Filezilla` or `winSCP` as such GUI programs often attempt to hide these server and platform incompatibilities and will take additional steps to simulate missing functionality etc. You want to use a CLI program which does as little as possible.
 
-## Promises, Events & Managing Exceptions<a id="sec-6-2"></a>
+## Promises, Events & Managing Exceptions<a id="sec-5-2"></a>
 
 One of the challenges in providing a Promise based API over a module like SSH2, which is event based is how to ensure events are handled appropriately. The challenge is due to the synchronous nature of events. You cannot use `try/catch` for events because you have no way of knowing when the event might fire. For example, it could easily fire after your `try/catch` block as completed execution.
 
@@ -961,25 +957,25 @@ The other area where additional events are fired is during the end() call. To de
 
 In addition to the promise based event handlers, `ssh2-sftp-client` also implements global event handlers which will catch any `error`, `end` or `close` events. Essentially, these global handlers only reset the `sftp` property of the client object, effectively ensuring any subsequent calls are rejected and in the case of an error, send the error to the console.
 
-### Adding Custom Handlers<a id="sec-6-2-1"></a>
+### Adding Custom Handlers<a id="sec-5-2-1"></a>
 
 While the above strategies appear to work for the majority of use cases, there are always going to be edge cases which require more flexible or powerful event handling. To support this, the `on()` and `removeListener()` methods are provided. Any event listener added using the `on()` method will be added at the beginning of the list of handlers for that event, ensuring it will be called before any global or promise local events. See the documentation for the `on()` method for details.
 
-## Windows Based Servers<a id="sec-6-3"></a>
+## Windows Based Servers<a id="sec-5-3"></a>
 
 It appears that when the sftp server is running on Windows, a *ECONNRESET* error signal is raised when the end() method is called. Unfortunately, this signal is raised after a considerable delay. This means we cannot remove the error handler used in the end() promise as otherwise you will get an uncaught exception error. Leaving the handler in place, even though we will ignore this error, solves that issue, but unfortunately introduces a new problem. Because we are not removing the listener, if you re-use the client object for subsequent connections, an additional error handler will be added. If this happens more than 11 times, you will eventually see the Node warning about a possible memory leak. This is because node monitors the number of error handlers and if it sees more than 11 added to an object, it assumes there is a problem and generates the warning.
 
 The best way to avoid this issue is to not re-use client objects. Always generate a new sftp client object for each new connection.
 
-## Don't Re-use SftpClient Objects<a id="sec-6-4"></a>
+## Don't Re-use SftpClient Objects<a id="sec-5-4"></a>
 
 Due to an issue with *ECONNRESET* error signals when connecting to Windows based SFTP servers, it is not possible to remove the error handler in the end() method. This means that if you re-use the SftpClient object for multiple connections e.g. calling connect(), then end(), then connect() etc, you run the risk of multiple error handlers being added to the SftpClient object. After 11 handlers have been added, Node will generate a possible memory leak warning.
 
 To avoid this problem, don't re-use SftpClient objects. Generate a new SftpClient object for each connection. You can perform multiple actions with a single connection e.g. upload multiple files, download multiple files etc, but after you have called end(), you should not try to re-use the object with a further connect() call. Create a new object instead.
 
-# FAQ<a id="sec-7"></a>
+# FAQ<a id="sec-6"></a>
 
-## Remote server drops connections with only an end event<a id="sec-7-1"></a>
+## Remote server drops connections with only an end event<a id="sec-6-1"></a>
 
 Many SFTP servers have rate limiting protection which will drop connections once a limit has been reached. In particular, openSSH has the setting `MaxStartups`, which can be a tuple of the form `max:drop:full` where `max` is the maximum allowed unauthenticated connections, `drop` is a percentage value which specifies percentage of connections to be dropped once `max` connections has been reached and `full` is the number of connections at which point all subsequent connections will be dropped. e.g. `10:30:60` means allow up to 10 unauthenticated connections after which drop 30% of connection attempts until reaching 60 unauthenticated connections, at which time, drop all attempts.
 
@@ -987,7 +983,7 @@ Clients first make an unauthenticated connection to the SFTP server to begin neg
 
 One way to avoid this type of issue is to add a delay between connection attempts. It does not need to be a very long delay - just sufficient to permit the previous connection to be authenticated. In fact, the default setting for openSSH is `10:30:60`, so you really just need to have enough delay to ensure that the 1st connection has completed authentication before the 11th connection is attempted.
 
-## How can I pass writable stream as dst for get method?<a id="sec-7-2"></a>
+## How can I pass writable stream as dst for get method?<a id="sec-6-2"></a>
 
 If the dst argument passed to the get method is a writeable stream, the remote file will be piped into that writeable. If the writeable you pass in is a writeable stream created with `fs.createWriteStream()`, the data will be written to the file specified in the constructor call to `createWriteStream()`.
 
@@ -1043,7 +1039,7 @@ sftp
   });
 ```
 
-## How can I upload files without having to specify a password?<a id="sec-7-3"></a>
+## How can I upload files without having to specify a password?<a id="sec-6-3"></a>
 
 There are a couple of ways to do this. Essentially, you want to setup SSH keys and use these for authentication to the remote server.
 
@@ -1075,7 +1071,7 @@ sftp.connect({
 }
 ```
 
-## How can I connect through a Socks Proxy<a id="sec-7-4"></a>
+## How can I connect through a Socks Proxy<a id="sec-6-4"></a>
 
 This solution was provided by @jmorino.
 
@@ -1108,13 +1104,13 @@ client.connect({
 // client is connected
 ```
 
-## Timeout while waiting for handshake or handshake errors<a id="sec-7-5"></a>
+## Timeout while waiting for handshake or handshake errors<a id="sec-6-5"></a>
 
 Some users have encountered the error 'Timeout while waiting for handshake' or 'Handshake failed, no matching client->server ciphers. This is often due to the client not having the correct configuration for the transport layer algorithms used by ssh2. One of the connect options provided by the ssh2 module is `algorithm`, which is an object that allows you to explicitly set the key exchange, ciphers, hmac and compression algorithms as well as server host key used to establish the initial secure connection. See the SSH2 documentation for details. Getting these parameters correct usually resolves the issue.
 
 When encountering this type of problem, one worthwhile approach is to use openSSH's CLI sftp program with the `-v` switch to raise loggin levels. This will show you what algorithms the CLI is using. You can then use this information to match the names with the accepted algorithm names documented in the `ssh2` README to set the properties in the `algorithms` object.
 
-## How can I limit upload/download speed<a id="sec-7-6"></a>
+## How can I limit upload/download speed<a id="sec-6-6"></a>
 
 If you want to limit the amount of bandwidth used during upload/download of data, you can use a stream to limit throughput. The following example was provided by *kennylbj*. Note that there is a caveat that we must set the `autoClose` flag to false to avoid calling an extra `_read()` on a closed stream that may cause \_get Permission Denied error in ssh2-streams.
 
@@ -1151,7 +1147,7 @@ try {
 }
 ```
 
-## Connection hangs or fails for larger files<a id="sec-7-7"></a>
+## Connection hangs or fails for larger files<a id="sec-6-7"></a>
 
 This was contributed by Ladislav Jacho. Thanks.
 
@@ -1159,11 +1155,11 @@ A symptom of this issue is that you are able to upload small files, but uploadin
 
 For more explanation, see [issue #342](https://github.com/theophilusx/ssh2-sftp-client/issues/342).
 
-# Examples<a id="sec-8"></a>
+# Examples<a id="sec-7"></a>
 
 I have started collecting example scripts in the example directory of the repository. These are mainly scripts I have put together in order to investigate issues or provide samples for users. They are not robust, lack adequate error handling and may contain errors. However, I think they are still useful for helping developers see how the module and API can be used.
 
-# Troubleshooting<a id="sec-9"></a>
+# Troubleshooting<a id="sec-8"></a>
 
 The `ssh2-sftp-client` module is essentially a wrapper around the `ssh2` and `ssh2-streams` modules, providing a higher level `promise` based API. When you run into issues, it is important to try and determine where the issue lies - either in the ssh2-sftp-client module or the underlying `ssh2` and `ssh2-streams` modules. One way to do this is to first identify a minimal reproducible example which reproduces the issue. Once you have that, try to replicate the functionality just using the `ssh2` and `ssh2-streams` modules. If the issue still occurs, then you can be fairly confident it is something related to those later 2 modules and therefore and issue which should be referred to the maintainer of that module.
 
@@ -1175,11 +1171,11 @@ Note also that in the repository there are two useful directories. The first is
 
 The second directory is the validation directory. I have some very simple scripts in this directory which perform basic tasks using only the `ssh2` modules (no `ssh2-sftp-client` module). These can be useful when trying to determine if the issue is with the underlying `ssh2` module or the `ssh2-sftp-client` wrapper module.
 
-## Common Errors<a id="sec-9-1"></a>
+## Common Errors<a id="sec-8-1"></a>
 
 There are some common errors people tend to make when using Promises or Asyc/Await. These are by far the most common problem found in issues logged against this module. Please check for some of these before logging your issue.
 
-### Not returning the promise in a `then()` block<a id="sec-9-1-1"></a>
+### Not returning the promise in a `then()` block<a id="sec-8-1-1"></a>
 
 All methods in `ssh2-sftp-client` return a Promise. This means methods are executed *asynchrnously*. When you call a method inside the `then()` block of a promise chain, it is critical that you return the Promise that call generates. Failing to do this will result in the `then()` block completing and your code starting execution of the next `then()`, `catch()` or `finally()` block before your promise has been fulfilled. For example, the following will not do what you expect
 
@@ -1213,7 +1209,7 @@ Note the `return` statements. These ensure that the Promise returned by the clie
 
 A common symptom of this type of error is for file uploads or download to fail to complete or for data in those files to be truncated. What is happening is that the connection is being ended before the transfer has completed.
 
-### Mixing Promise Chains and Async/Await<a id="sec-9-1-2"></a>
+### Mixing Promise Chains and Async/Await<a id="sec-8-1-2"></a>
 
 Another common error is to mix Promise chains and async/await calls. This is rarely a great idea. While you can do this, it tends to create complicated and difficult to maintain code. Select one approach and stick with it. Both approaches are functionally equivalent, so there is no reason to mix up the two paradigms. My personal preference would be to use async/await as I think that is more *natural* for most developers. For example, the following is more complex and difficult to follow than necessary (and has a bug!)
 
@@ -1270,7 +1266,7 @@ async function doSftp() {
 }
 ```
 
-### Try/catch and Error Handlers<a id="sec-9-1-3"></a>
+### Try/catch and Error Handlers<a id="sec-8-1-3"></a>
 
 Another common error is to try and use a try/catch block to catch event signals, such as an error event. In general, you cannot use try/catch blocks for asynchronous code and expect errors to be caught by the `catch` block. Handling errors in asynchronous code is one of the key reasons we now have the Promise and async/await frameworks.
 
@@ -1278,17 +1274,17 @@ The basic problem is that the try/catch block will have completed execution befo
 
 Error events are essentially asynchronous code. You don't know when such events will fire. Therefore, you cannot use a try/catch block to catch such event errors. Even creating an error handler which then throws an exception won't help as the key problem is that your try/catch block has already executed. There are a number of alternative ways to deal with this situation. However, the key symptom is that you see occasional uncaught error exceptions that cause your script to exit abnormally despite having try/catch blocks in your script. What you need to do is look at your code and find where errors are raised asynchronously and use an event handler or some other mechanism to manage any errors raised.
 
-### Server Differences<a id="sec-9-1-4"></a>
+### Server Differences<a id="sec-8-1-4"></a>
 
 Not all SFTP servers are the same. Like most standards, the SFTP protocol has some level of interpretation and allows different levels of compliance. This means there can be differences in behaviour between different servers and code which works with one server will not work the same with another. For example, the value returned by *realpath* for non-existent objects can differ significantly. Some servers will throw an error for a particular operation while others will just return null, some servers support concurrent operations (such as used by fastGet/fastPut) while others will not and of course, the text of error messages can vary significantly. In particular, we have noticed significant differences across different platforms. It is therefore advisable to do comprehensive testing when the SFTP server is moved to a new platform. This includes moving from to a cloud based service even if the underlying platform remains the same. I have noticed that some cloud platforms can generate unexpected events, possibly related to additional functionality or features associated with the cloud implementation. For example, it appears SFTP servers running under Azure will generate an error event when the connection is closed even when the client has requested the connection be terminated. The same SFTP server running natively on Windows does not appear to exhibit such behaviour.
 
-### Avoid Concurrent Operations<a id="sec-9-1-5"></a>
+### Avoid Concurrent Operations<a id="sec-8-1-5"></a>
 
 Technically, SFTP should be able to perform multiple operations concurrently. As node is single threaded, what we a really talking about is running multiple execution contexts as a pool where node will switch contexts when each context is blocked due to things like waiting on network data etc. However, I have found this to be extremely unreliable and of very little benefit from a performance perspective. My recommendation is to therefore avoid executing multiple requests over the same connection in parallel (for example, generating multiple `get()` promises and using something like `Promise.all()` to resolve them.
 
 If you are going to try and perform concurrent operations, you need to test extensively and ensure you are using data which is large enough that context switching does occur (i.e. the request is not completed in a single run). Some SFTP servers will handle concurrent operations better than others.
 
-## Debugging Support<a id="sec-9-2"></a>
+## Debugging Support<a id="sec-8-2"></a>
 
 You can add a `debug` property to the config object passed in to `connect()` to turn on debugging. This will generate quite a lot of output. The value of the property should be a function which accepts a single string argument. For example;
 
@@ -1318,7 +1314,7 @@ If you just want to see debug messages from `ssh2-sftp-client` and exclude debug
 }
 ```
 
-# Logging Issues<a id="sec-10"></a>
+# Logging Issues<a id="sec-9"></a>
 
 Please log an issue for all bugs, questions, feature and enhancement requests. Please ensure you include the module version, node version and platform.
 
@@ -1332,7 +1328,7 @@ I am happy to try and help diagnose and fix any issues you encounter while using
 
 Perhaps the best assistance is a minimal reproducible example of the issue. Once the issue can be readily reproduced, it can usually be fixed very quickly.
 
-# Pull Requests<a id="sec-11"></a>
+# Pull Requests<a id="sec-10"></a>
 
 Pull requests are always welcomed. However, please ensure your changes pass all tests and if you're adding a new feature, that tests for that feature are included. Likewise, for new features or enhancements, please include any relevant documentation updates.
 
@@ -1344,7 +1340,7 @@ This module will adopt a standard semantic versioning policy. Please indicate in
 -   **Minor:** Minor change, enhancement or new feature which does not change existing API and will not break existing client code.
 -   **Bug Fix:** No change to functionality or features. Simple fix of an existing bug.
 
-# Contributors<a id="sec-12"></a>
+# Contributors<a id="sec-11"></a>
 
 This module was initially written by jyu213. On August 23rd, 2019, theophilusx took over responsibility for maintaining this module. A number of other people have contributed to this module, but until now, this was not tracked. My intention is to credit anyone who contributes going forward.