ssh2-sftp-client 9.0.4 → 10.0.0

package/README.org

@@ -7,83 +7,34 @@ an SFTP client for node.js, a wrapper around [[https://github.com/mscdex/ssh2][S
 convenience abstraction as well as a Promise based API.
 
 Documentation on the methods and available options in the underlying modules can
-be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages.
-
-Current stable release is *v9.0.4*.
-
-Code has been tested against Node versions 14.20.0, 16.17.2 and 18.8.0
-
-Node versions < 14.x are not supported. 
-
-** Version 9.x Changes
-  - Fix bug in ~connect()~ method when private key data was corrupted. The method was not
-    handling errors fro corrupted ssh private keys and would hang indefinitely without
-    reporting any error. Now reports that it was unable to parse the private key. 
-  - Fix bug in ~end()~ method where it was possible for the module to attempt calling
-    the underlying ssh2 ~end()~ method when ssh2 has not been initialised. This could
-    lead to undefined reference errors.
-  - Fix bug in ~get()~ method where supplied destination streams were not close, creating
-    a possible resource leak. If the remote file did not exist, the method would return
-    an error, but failed to close any passed in stream supplied as the destination for
-    the data in the ~get()~ call.    
-  - Change the default end and close handlers not to throw error or reject
-    promises. Previously, an end or close event would cause an error to be raised or a
-    promise to be rejected if the event was deemed to be /unexpected/. However,
-    classification of events as being unexpected was unreliable and didn't add much real
-    value. Both these handlers will now invalidate the SFTP connection object and log that
-    the event fired and nothing else.
-  - Changed when event handled flags are reset. Now they are reset after a new set of
-    temporary listeners are added.
-  - Don't throw an error when calling end() if there is no active SFTP connection. It does
-    no harm to call end() when there is no connection, so no need to raise an error.
-  - Use nullish coalescing when setting retry parameters instead of or'ing with
-    defaults. Allows setting values to 0.  
-  - *Breaking Change*: This version uses syntax not supported in node versions
-    prior to v14. Therefore, node versions less than v14 will not work.
-  - *Breaking Change*: This ~list()~ method no longer accepts a regular expression
-    for filtering the entries to be returned. You can now specify a filter
-    function instead. The function is called for each item in the list of items
-    to be returned, passing in the item object as its only argument.
-    Essentially, this is just a call to ~Array.filter()~, so the filter function
-    should behave in the same way i.e. return true for items to be retained and
-    false for those to be dropped.
-  - *Breaking Change*: The ability to set ~autoClose~ on read and write streams and
-    the ability to set ~end~ on ~pipe~ operations has been removed. These options
-    caused confusion for users and were too easy to get wrong, plus it made the
-    methods overly complicated. For those use-cases where you want to control
-    streams at a low level, two new methods have been added, ~createReadStream()~ and
-    ~createWriteStream()~. However, it should be noted that client code is 100%
-    responsible for managing streams obtained using these methods. Use at your
-    own risk!
-  - *Breaking Change*: The 3rd argument to ~uploadDir()~ and ~downloadDir()~ methods
-    has been change. Previously, the argument was a filter function used to
-    select which directories and files to be transferred. The 3rd argument is
-    now an options object with two supported properties, ~filter~ and ~useFastput~
-    (for ~uploadDir()~) or ~useFastget~ (for ~downloadDir()~). If ~useFastput~ is true,
-    the ~fastPut()~ method will be used to upload files. If ~false~ or missing, the
-    slower, but better supported, ~put()~ method will be used. Likewise, the
-    ~useFastget~ options can be set to ~true~ to use the ~fastGet()~ method for
-    downloading files, otherwise the slower, but more reliable, ~get()~ method
-    will be used.
-  - The ~uploadDir()~ and ~downloadDir()~ methods now use asynchronous processes to
-    upload/download files. This should result in improved performance for these
-    two methods.
-  - New Methods: Two new methods, ~createWriteStream()~ and ~createReadStream()~
-    have been added. These methods will return a stream object connected to a
-    remote file on the ~SFTP~ server. Client code is responsible for managing
-    these stream objects. This includes adding any necessary event listeners and
-    disposing of the objects once finished with them.
-  - Re-factoring of Listeners: The library manages temporary listeners in order
-    to provide a way to catch events and processes them inside a ~Promise~
-    context. Previously, every method added its own set of temporary listeners.
-    However, this could result in multiple sets of listeners being added,
-    especially for methods which call other methods as part of their processing
-    e.g. ~rmdir(),~ ~uploadDir()~ and ~dowqnloadDir()~. To avoid this, /internal only/
-    versions of each method have been created. These internal methods use an
-    /underscore/ ~_~ prefix. Client code should not use these methods directly. 
-  - New method: Added ~rcopy()~ method to perform a remote copy of a file on the remote SFTP server. 
-  - Bumped ssh2 version to 1.11.0
+be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages. As this module is really just a wrapper around the
+~ssh2~ module, you will find lots of useful information, tips and examples in the ~ssh2~
+repository. 
 
+Current stable release is *v10.0.0.
+
+Code has been tested against Node versions 16.20.2, 18.18.2, 20.10.0 and 21.5.0. However,
+only versions from v18 are actively supported. It should also be noted that a significant
+performance improvement has been observed with versions >= 18. Version v16 is
+significantly slower. 
+
+Node versions < 16.x are not supported. 
+
+** Version 10.0.0 Changes
+
+- The main change in this version is adding of limits on the number of promises which can
+  be active at the same time. Version 9.1.0 extended the use of multiple promises to
+  improve performance with downloadDir() and uploadDir(). However, for directories with
+  really large numbers of files, this often resulted in an error because the methods would
+  try to create more concurrent promises than was possible given available resources. This
+  issue has been fixed by adding a new property called ~promiseLimit~, which is limited to
+  10 by default. A new configuration property, ~promiseLimit~, is now available for setting
+  the maximum number of concurrent promises the downloadDir()/uploadDir() methods will
+  create when downloading or uploading directory trees.
+
+ - Various minor documentation fixes and some minor fixes for typos in option property
+   names.
+   
 * Installation
 
 #+begin_src shell
@@ -92,7 +43,7 @@ Node versions < 14.x are not supported.
 
 * Basic Usage
 
-#+begin_src javascript
+#+begin_src js
   let Client = require('ssh2-sftp-client');
   let sftp = new Client();
 
@@ -113,10 +64,20 @@ Node versions < 14.x are not supported.
 * Documentation
 
 The connection options are the same as those offered by the underlying SSH2
-module. For full details, please see [[https://github.com/mscdex/ssh2#user-content-client-methods][SSH2 client methods]]
+module, with just a couple of additional properties added to tweak the ~retry~ parameters,
+add a ~debug~ function and set the ~promiseLimit~ property. For full details on the other
+properties, please see [[https://github.com/mscdex/ssh2#user-content-client-methods][SSH2 client methods]]. In particular, see the ~ssh2~ documentation for
+details relating to setting various key exchange and encryption/signing algorithms used as
+part of the ssh2 protocol.
 
-All the methods will return a Promise, except for ~on()~ and
-~removeListener()~, which are typically only used in special use cases.
+All the methods will return a Promise, except for ~on(), ~removeListener()~, ~createReadStream~
+and ~createWriteStream~, which are typically only used in special use cases.
+
+Note that I don't use Typescript and I don't maintain any typescript definition
+files. There are some typescript type definition files for this module, but they are
+maintained separately and have nothing to do with this project. Therefore, please do not
+log any issues arising from the use of these definition files with this project. Instead,
+refer your issues to the maintainers of those modules.
 
 ** Specifying Paths
 
@@ -177,7 +138,7 @@ All the methods will return a Promise, except for ~on()~ and
 
    This will copy the local file ~test.txt~ to the remote file ~test-copy.txt~
    in the directory ~/remote/dir~.
-
+   
 ** Methods
 
 *** new SftpClient(name) ===> SFTP client object
@@ -236,6 +197,19 @@ SSH2 module. These are part of the configuration for the [[https://www.npmjs.com
 is used to enable retrying of sftp connection attempts. See the documentation
 for that package for an explanation of these values.
 
+The ~promiseLimit~ is another option which is not part of the ~ssh2~ module and is specific to
+~ssh2-sftp-client~. It is a property used to limit the maximum number of concurrent promises
+possible when either downloading or uploading a directory tree using the ~downloadDir()~ or
+~uploadDir()~ methods. The default setting for this property is 10. *NOTE*: bigger doe snot
+mean better. Many factors can affect what is the ideal setting for ~promiseLimit~. If it is
+too large, any benefits are lost while node spends time switching contexts and/or withi
+the overheads associated with creating and cleaning up promises. Lots of factors can
+affect what the setting should be, including size of files, number of files, speed of
+network, version of node, capabilities of remote sftp server etc. A setting of 10 seems to
+be a reasonably good default and should be adequate for most use cases. However, if you
+feel it needs to be changed, I highly recommend that you benchmark different values to
+work out what is the best maximum size before you begin to see a performance drop off.
+
 #+begin_src javascript
   // common options
 
@@ -250,12 +224,13 @@ for that package for an explanation of these values.
     privateKey: fs.readFileSync('/path/to/key'), // Buffer or string that contains
     passphrase: 'a pass phrase', // string - For an encrypted private key
     readyTimeout: 20000, // integer How long (in ms) to wait for the SSH handshake
-    strictVendor: true // boolean - Performs a strict server vendor check
-    debug: myDebug // function - Set this to a function that receives a single
+    strictVendor: true, // boolean - Performs a strict server vendor check
+    debug: myDebug,// function - Set this to a function that receives a single
                   // string argument to get detailed (local) debug information.
-    retries: 2 // integer. Number of times to retry connecting
-    retry_factor: 2 // integer. Time factor used to calculate time between retries
-    retry_minTimeout: 2000 // integer. Minimum timeout between attempts
+    retries: 2, // integer. Number of times to retry connecting
+    retry_factor: 2, // integer. Time factor used to calculate time between retries
+    retry_minTimeout: 2000, // integer. Minimum timeout between attempts
+    promiseLimit: 10, // max concurrent promises for downloadDir/uploadDir
   };
 
   // rarely used options
@@ -282,7 +257,7 @@ for that package for an explanation of these values.
 
 #+begin_src javascript
   sftp.connect({
-    host: example.com,
+    host: 'example.com',
     port: 22,
     username: 'donald',
     password: 'youarefired'
@@ -313,7 +288,7 @@ directory.
     password: 'my-secret'
   };
 
-  let sftp = new Client;
+  let sftp = new Client();
 
   sftp.connect(config)
     .then(() => {
@@ -336,19 +311,19 @@ The objects in the array returned by ~list()~ have the following properties;
 
 #+begin_src javascript
   {
-    type: // file type(-, d, l)
-    name: // file name
-    size: // file size
-    modifyTime: // file timestamp of modified time
-    accessTime: // file timestamp of access time
+    type: '-', // file type(-, d, l)
+    name: 'example.txt', // file name
+    size: 43, // file size
+    modifyTime: 1675645360000, // file timestamp of modified time
+    accessTime: 1675645360000, // file timestamp of access time
     rights: {
-      user:
-      group:
-      other:
+      user: 'rw',
+      group: 'r',
+      other: 'r',
     },
-    owner: // user ID
-    group: // group ID
-    longname: // like ls -l line
+    owner: 1000, // user ID
+    group: 1000, // group ID
+    longname: '-rw-r--r--  1 fred  fred  43 Feb 6 12:02 exaple.txt', // like ls -l line
   }
 #+end_src
 
@@ -369,7 +344,7 @@ if it exists or false if it does not.
     password: 'my-secret'
   };
 
-  let sftp = new Client;
+  let sftp = new Client();
 
   sftp.connect(config)
     .then(() => {
@@ -508,6 +483,12 @@ this for binary files to avoid data corruption.
 
 Downloads a file at remotePath to localPath using parallel reads for faster
 throughput. This is the simplest method if you just want to download a file.
+However, fastGet functionality depends heavily on remote sftp server capabilities and not
+all servers have the concurrency support required. See the Platform Quirks & Warnings
+section of this README.
+
+Bottom line, when it works, it tends to work reliably. However, for many servers, it
+simply won't work or will result in truncated/corrupted data. 
 
 - remotePath :: String. Path to the remote file to download
 - localPath :: String. Path on local file system for the downloaded file. The
@@ -617,7 +598,14 @@ resolved.
 *** fastPut(localPath, remotePath, options) ==> string
 
 Uploads the data in file at ~localPath~ to a new file on remote server at
-~remotePath~ using concurrency. The options object allows tweaking of the fast put process.
+~remotePath~ using concurrency. The options object allows tweaking of the fast put
+process. Note that this functionality is heavily dependent on the capabilities of the
+remote sftp server, which must support the concurrency operations used by this
+method. This is not part of the standard and therefore is not available in all sftp
+servers. See the Platform Quirks & Warnings for more details.
+
+Bottom line, when it works, it tends to work well. However, when it doesn't work, it may
+fail completely or it may result in truncated or corrupted data transfers. 
 
 - localPath :: string. Path to local file to upload
 - remotePath :: string. Path to remote file to create
@@ -1176,6 +1164,7 @@ Removes the specified listener from the event specified in eventType. Note that
 the ~end()~ method automatically removes all listeners from the client object.
 
 * Platform Quirks & Warnings
+
 ** Server Capabilities
 
    All SFTP servers and platforms are not equal. Some facilities provided by
@@ -1456,6 +1445,8 @@ configuration.
 
 This solution was provided by @jmorino.
 
+When a SOCKS 5 client is connected it must be ingested by ssh2-sftp-client immediately, otherwise a timeout occurs.
+
 #+begin_src javascript
   import { SocksClient } from 'socks';
   import SFTPClient from 'ssh2-sftp-client';
@@ -1464,7 +1455,7 @@ This solution was provided by @jmorino.
   const port = 22; // default SSH/SFTP port on remote server
 
   // connect to SOCKS 5 proxy
-  const { socks } = await SocksClient.createConnection({
+  const { socket } = await SocksClient.createConnection({
     proxy: {
       host: 'my.proxy', // proxy hostname
       port: 1080, // proxy port
@@ -1477,9 +1468,8 @@ This solution was provided by @jmorino.
   const client = new SFTPClient();
   client.connect({
     host,
-    sock: socks.socket, // pass the socket to proxy here (see ssh2 doc)
-    username: '.....',
-    privateKey: '.....'
+    sock: socket, // pass the socket to proxy here (see ssh2 doc)
+    // other config options
   })
 
   // client is connected
@@ -1544,6 +1534,7 @@ the =ssh2= README to set the properties in the =algorithms= object.
        await client.end();
      }
    #+end_src
+
 ** Connection hangs or fails for larger files
 
  This was contributed by Ladislav Jacho. Thanks. 
@@ -1557,6 +1548,13 @@ the =ssh2= README to set the properties in the =algorithms= object.
  
  For more explanation, see [[https://github.com/theophilusx/ssh2-sftp-client/issues/342][issue #342]].
  
+** Typescript definition file out of date
+
+This project does not use Typescript. However, typescript definition files are provided by
+other 3rd parties. Sometimes, these definition files have not stayed up-to-date with the
+current version of this module. If you encounter this issue, you need to report it to the
+party responsible for the definition file, not this project.
+
 * Examples
 
 I have started collecting example scripts in the example directory of the
@@ -1730,7 +1728,7 @@ the ~ssh2-sftp-client~ wrapper module.
          await sftp.fastGet(`${d}/foo.txt`, 'bat.txt');
        } catch (e) {
          console.error(e.message);
-       } finally () {
+       } finally {
          await sftp.end();
        }
      }
@@ -1836,6 +1834,7 @@ based on messages which start with 'CLIENT' e.g.
     }
   }
 #+end_src
+
 * Logging Issues
 
 Please log an issue for all bugs, questions, feature and enhancement
@@ -1880,8 +1879,10 @@ your pull request what level of change it represents i.e.
 
 - Major :: Change to API or major change in functionality which will require an
            increase in major version number.
+
 - Minor :: Minor change, enhancement or new feature which does not change
            existing API and will not break existing client code.
+
 - Bug Fix :: No change to functionality or features. Simple fix of an existing
              bug.
 
@@ -1913,3 +1914,4 @@ Thanks to the following for their contributions -
   Also included test to check for this regression in future.
 - cakemasher :: Contributed fix for removeTempListeners().
   
+  

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ssh2-sftp-client",
-  "version": "9.0.4",
+  "version": "10.0.0",
   "description": "ssh2 sftp client for node",
   "main": "src/index.js",
   "repository": {
@@ -18,7 +18,7 @@
     "lint": "eslint \"src/**/*.js\" \"test/**/*.js\""
   },
   "engines": {
-    "node": ">=10.24.1"
+    "node": ">=16.20.2"
   },
   "author": "Tim Cross",
   "email": "theophilusx@gmail.com",
@@ -30,27 +30,27 @@
   ],
   "license": "Apache-2.0",
   "devDependencies": {
-    "chai": "^4.3.6",
+    "chai": "^4.3.10",
     "chai-as-promised": "^7.1.1",
     "chai-subset": "^1.6.0",
     "checksum": "^1.0.0",
     "dotenv": "^16.0.0",
-    "eslint": "^8.17.0",
-    "eslint-config-prettier": "^8.5.0",
-    "eslint-plugin-mocha": "^10.0.3",
+    "eslint": "^8.51.0",
+    "eslint-config-prettier": "^9.0.0",
+    "eslint-plugin-mocha": "^10.2.0",
     "eslint-plugin-node": "^11.1.0",
     "eslint-plugin-promise": "^6.0.0",
-    "eslint-plugin-unicorn": "^43.0.2",
+    "eslint-plugin-unicorn": "^50.0.1",
     "mocha": "^10.0.0",
     "moment": "^2.29.1",
     "nyc": "^15.1.0",
-    "prettier": "^2.6.1",
+    "prettier": "^3.0.3",
     "through2": "^4.0.2",
-    "winston": "^3.6.0"
+    "winston": "^3.11.0"
   },
   "dependencies": {
     "concat-stream": "^2.0.0",
     "promise-retry": "^2.0.1",
-    "ssh2": "^1.11.0"
+    "ssh2": "^1.15.0"
   }
 }