ssh2-sftp-client 10.0.2 → 11.0.0

package/README.org

@@ -3,41 +3,148 @@
 
 * Overview
 
-an SFTP client for node.js, a wrapper around [[https://github.com/mscdex/ssh2][SSH2]]  which provides a high level
-convenience abstraction as well as a Promise based API.
+This packan provides the class SftpClient, an SFTP client for node.js. It is a promise
+based decorator class around the excdellent [[https://github.com/mscdex/ssh2][SSH2]] package, which provides a pure node
+Javascript event based ssh2 implementation. 
 
 Documentation on the methods and available options in the underlying modules can
-be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages. As this module is really just a wrapper around the
+be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages. As the ssh2-sftp-client package just a wrapper around the
 ~ssh2~ module, you will find lots of useful information, tips and examples in the ~ssh2~
 repository. 
 
-Current stable release is *v10.0.2
+Current stable release is *v11.0.0.
 
-Code has been tested against Node versions 16.20.2, 18.18.2, 20.10.0 and 21.5.0. However,
-only versions from v18 are actively supported. It should also be noted that a significant
-performance improvement has been observed with versions >= 18. Version v16 is
-significantly slower. 
-
-Node versions < 16.x are not supported. 
+Code has been tested against Node versions 18.20.4, 20.16.0 and 22.5.1. Node versions
+prior to v18.x are not supported. Also note there is currently a deprecation warning when
+using node v22+. This is from the ~ssh2~ package and outside control of this package. 
 
 If you find this module useful and you would like to support the on-going maintenance and
 support of users, please consider making a small [[https://square.link/u/gB2kSdkY?src=embed][donation]].
 
-** Version 10.0.0 Changes
-
-- The main change in this version is adding of limits on the number of promises which can
-  be active at the same time. Version 9.1.0 extended the use of multiple promises to
-  improve performance with downloadDir() and uploadDir(). However, for directories with
-  really large numbers of files, this often resulted in an error because the methods would
-  try to create more concurrent promises than was possible given available resources. This
-  issue has been fixed by adding a new property called ~promiseLimit~, which is limited to
-  10 by default. A new configuration property, ~promiseLimit~, is now available for setting
-  the maximum number of concurrent promises the downloadDir()/uploadDir() methods will
-  create when downloading or uploading directory trees.
+** Version 11.0.0 Changes
+
+The main change in v11 concerns how the package manages events raised by the =ssh2= packagwe
+it depends on. Managing events within the context of asynchronous code as ocurs when
+using promises is challenging. To understand some of the more subtle issues involved, it
+is recommended you read the [[https://nodejs.org/docs/latest/api/events.html#asynchronous-vs-synchronous][Asynchronous vs. synchronous]] and [[https://nodejs.org/docs/latest/api/events.html#error-events][Error events]] sections of the
+Events chapter from the node documentation.
+
+The previous versions of this package use a global event handler approach to manage events
+raised outside the execution of any promise. However, this approach is problematic as all
+the global listeners can really do is raise an error. Attempting to catch errors raised
+inside event handlers is extremely difficult to manage within client code because those
+errors are raised inside a separate asynchronous execution context. In version 11,
+this approach has been replaced by a mechanism whereby the client code can pass in
+application specific global handlers if desired. If no handlers are defined, default
+handlers will log the event and when necessary, invalidate any existing connection
+objects. See the [[Background]] section for details.
+
+** Background
+
+In basic terms =ssh2-sftp-client= is a simple wrapper around the =ssh2= package which provides
+a promise base API for interacting with a remote SFTP server . The =ssh2= package provides
+an event based API for interacting with the ~ssh~ protcolo. The ~ssh2-sftp-client~ package
+uses the ~sftp~ subsystem of this protocol to implement the basic operations typically
+associated with an ~sftp~ client.
+
+Wrapping an event based API with a promised based API comes with a number of
+challenges. In particular, efficiently and reliably managing events within the context of
+asynchrounous code execution. This package uses the following strategies;
+
+    - All direct interactions with the ~ssh2~ API are wrapped in promise objects. When the
+      API call succeeds, the assoiated promise will be sucessfully resolved. When an error occurs,
+      the promise is rejected.
+
+    - An error can either be due to a low level network error, such as a lost connection
+      to the remote server or due to an operational error, such as a file not
+      existing or not having the appropriate permissions for access.
+
+    - Each of the available ~SftpClient~ methods wrap the method call inside a promise. In
+      crwating eacvh promise, the class adds temporary event listenrs for the error, end
+      and close events and links those liseners to the method's promise via it's ~reject()~
+      method. 
+
+    - If a promise is waiting to be fulfilled when either of the two types of errors
+      occurs, the error will be communicated back to client code via a rejected promise.
+
+    - When the ~ssh2~ emitter raises an event outside the context of any promise, that event
+      will be handled by global event handlers. By default, these event handlers will log
+      the event and will invalidate any existing socket connection objects, preventing any
+      further API calls until a new connection is extablished.
+
+    - The ~SftpClient~ class constructor supports an optoinal second argument which is an
+      objedct whi8ch can have any of three properties representing event callbaqck
+      funct5ions which will be executed for each of the possible events error, end and
+      close.
+      
+The need for both global listeners and temporary promise listeners is because network end,
+close or error events can occur at any time, including in-betwseen API calls. During an
+API call, a promise is active and can be used to communicate event information back to the calling
+code via normal promise communication means i.e. async/await with try/catch or promise chain's then/catch
+mechanism. However, outside API calls, no promise exists and there is no reliable
+mechanism to return error and other event information back to calling code. You cannot
+reliably use try/cdatch to catch errorsx thrown inside event listenrs as you lack control
+over when the listener code runs. Your try/catch block can easily complete before the
+error is raised as there is no equivalent /await/ type functionality in this situation.
+
+As there is no simple default way to return error and other event information back to the
+calling code, ~ssh2-sftp-client~ doesn't try to. Instead, the default action is
+to just log the event information and invalidate any existing sftp connections. This
+strategy is often sufficient for many use cases. For those cases where it isn't, client
+code can pass in end, close and/or error listener functions when instantiating the
+~SftpClient~ object. If provided, these listners will be executed whenever the default
+global listeners are executed, which is whenever the ~ssh2~ event emitter raises an end,
+close or error event which is not handled by one of the temporary promise linked event
+listeners.
+
+Version 11 of ~ssh2-sftp-client~ also changes the behaviour of the temporary promise linked
+/end/ and /close/ listeners. Prior to versioln 11, these listeners did not reject
+promises. They would only invalidate the underlying ~ssh~ connection object. Only the /error/
+listener would actually reject the associated promise. This was done because you cannot
+guarantee the order in which events are responded to.
+
+In most cases, events will occur in the order /error/, /end/ and then /close/. The error event
+would contain details about the cause of the error while the end and close events just
+communicvate that these events have been raised. The normal flow would be
+
+    - Error event occurs including error cause description. Listener catches event,
+      creates an error object and calls the associated promises reject function to reject
+      the promise. The calling process receives a rejected promise object.
+
+    - End event occurs. The end listener catches the event and marks the connection object
+      as invalid as the socket connection has been ended. There is no need to call the
+      reject method of the associated promise as it has already been called by the error
+      listener and you can only call one promise resolution function.
+
+    - Close event occurs. This event means the socket connection has been closed. The
+      listener will ensure any connection information has been invalidated. Aga8in, ther
+      is no need to call the reject method of the associated promise as it has already
+      been called by the error listener.
+
+In some cases, no ende event is raised and you only get an error event followed by a close
+event. In versions of ~ssh2-sftp-client~ prior to version 11, neither the end or the close
+listeners attempted to call the reject method of the associated promise. It was assumed
+that all sftp servers would raise an error event whenever a connection was unexpectedly
+ended or closed. Unfortunately, it turns out some sftp servers are not well behaved and
+will terminate the connection without providing any error information or raising an error
+event. When this occurred in versions prior to version 11, it could result in either an
+API call hanging because its associated promise never gets rejected or resolved or the
+call gets rejected with a timeout error aftrer a significant delay.
+
+In order to handle the possible hanging issue in version 11, the temporary promise linked
+end and close listeners have been updated to always call the promise's reject function if
+they fire. While this works, it can cause a minor issue. As wse cannot gurantee the order
+in which events are resonded to by listeners, it is possible that either the end or close
+listener may be executed before the error listener. When this occurs, the promise is
+rejected, but the only information wse have at that point is that the promise wsas reject
+due to either an end or close event. We don't yet have any details regarding what error
+has caused the unexpected end or close event. Furthermore, because only the first promise
+resolution function call has any effect, calling reject within the error listener
+(assuming an error event does eventually arrive) has no effect and does not communicate
+error information back to the caller. This means that in some circumstances, especially
+when working with some poorly behaved sftp servers, an sftp connection will be lost/closed
+with no indication as to reason. This can make diagnosis and bug tracking frustrating. 
 
- - Various minor documentation fixes and some minor fixes for typos in option property
-   names.
-   
 * Installation
 
 #+begin_src shell
@@ -144,7 +251,7 @@ refer your issues to the maintainers of those modules.
    
 ** Methods
 
-*** new SftpClient(name) ===> SFTP client object
+*** new SftpClient(name, callbacks) ===> SFTP client object
 
 Constructor to create a new ~ssh2-sftp-client~ object. An optional ~name~ string
 can be provided, which will be used in error messages to help identify which
@@ -152,8 +259,14 @@ client has thrown the error.
 
 **** Constructor Arguments
 
-- name :: string. An optional name string used in error messages
-
+- name :: string. An optional name string used in error messages. Defaults to the string
+  'sftp'
+- callbacks :: object. An object with the properties error, end and close. Associted with
+  each property is a function that will be executed whenever the associated global event
+  listener for error, end or close is executed. The error function should accept one
+  argument. The end and close functions have no arguments. Default functions just print a
+  message to ~console.log()~.
+  
 **** Example Use
 
 #+begin_src javascript
@@ -989,7 +1102,7 @@ remote SFTP server will result in failures.
 
 *** downloadDir(srcDir, dstDir, options) ==> string
 
-Download the remote directory specified by ~srcDir~ to the local file system
+Download the content of the remote directory specified by ~srcDir~ to the local file system
 directory specified by ~dstDir~. The ~dstDir~ directory will be created if
 required. All sub directories within ~srcDir~ will also be copied. Any existing
 files in the local path will be overwritten. No files in the local path will be

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ssh2-sftp-client",
-  "version": "10.0.2",
+  "version": "11.0.0",
   "description": "ssh2 sftp client for node",
   "main": "src/index.js",
   "repository": {
@@ -18,7 +18,7 @@
     "lint": "eslint \"src/**/*.js\" \"test/**/*.js\""
   },
   "engines": {
-    "node": ">=16.20.2"
+    "node": ">=18.20.4"
   },
   "author": "Tim Cross",
   "email": "theophilusx@gmail.com",
@@ -34,20 +34,20 @@
   ],
   "license": "Apache-2.0",
   "devDependencies": {
-    "chai": "^4.3.10",
-    "chai-as-promised": "^7.1.1",
+    "chai": "^4.5.0",
+    "chai-as-promised": "^7.1.2",
     "chai-subset": "^1.6.0",
     "checksum": "^1.0.0",
     "dotenv": "^16.0.0",
-    "eslint": "^8.51.0",
+    "eslint": "^9.6.0",
     "eslint-config-prettier": "^9.0.0",
     "eslint-plugin-mocha": "^10.2.0",
     "eslint-plugin-node": "^11.1.0",
     "eslint-plugin-promise": "^6.0.0",
-    "eslint-plugin-unicorn": "^50.0.1",
+    "eslint-plugin-unicorn": "^54.0.0",
     "mocha": "^10.0.0",
     "moment": "^2.29.1",
-    "nyc": "^15.1.0",
+    "nyc": "^17.0.0",
     "prettier": "^3.0.3",
     "through2": "^4.0.2",
     "winston": "^3.11.0"

package/src/index.js

@@ -1,5 +1,4 @@
 'use strict';
-
 const { Client } = require('ssh2');
 const fs = require('node:fs');
 const concat = require('concat-stream');
@@ -19,11 +18,18 @@ const {
 const { errorCode } = require('./constants');
 
 class SftpClient {
-  constructor(clientName) {
-    this.version = '10.0.2';
+  constructor(
+    clientName = 'sftp',
+    callbacks = {
+      error: (err) => console.error(`Global error listener: ${err.message}`),
+      end: () => console.log('Global end listener: end event raised'),
+      close: () => console.log('Global close listener: close event raised'),
+    },
+  ) {
+    this.version = '11.0.0';
     this.client = new Client();
     this.sftp = undefined;
-    this.clientName = clientName || 'sftp';
+    this.clientName = clientName;
     this.endCalled = false;
     this.errorHandled = false;
     this.closeHandled = false;
@@ -31,9 +37,10 @@ class SftpClient {
     this.remotePlatform = 'unix';
     this.debug = undefined;
     this.promiseLimit = 10;
-    this.client.on('close', globalListener(this, 'close'));
-    this.client.on('end', globalListener(this, 'end'));
-    this.client.on('error', globalListener(this, 'error'));
+    this.eventCallbacks = callbacks;
+    this.client.on('close', globalListener(this, 'close', this.eventCallbacks));
+    this.client.on('end', globalListener(this, 'end', this.eventCallbacks));
+    this.client.on('error', globalListener(this, 'error', this.eventCallbacks));
   }
 
   debugMsg(msg, obj) {
@@ -128,14 +135,12 @@ class SftpClient {
     return new Promise((resolve, reject) => {
       listeners = addTempListeners(this, 'getConnection', reject);
       doReady = () => {
-        this.debugMsg('getConnection ready listener: got connection - promise resolved');
         resolve(true);
       };
       this.on('ready', doReady);
       try {
         this.client.connect(config);
       } catch (err) {
-        this.debugMsg(`getConnection: ${err.message}`);
         reject(err);
       }
     }).finally(() => {
@@ -219,6 +224,7 @@ class SftpClient {
         }
       });
       const sftp = await this.getSftpChannel();
+      this.endCalled = false;
       return sftp;
     } catch (err) {
       this.end();
@@ -246,18 +252,14 @@ class SftpClient {
       if (addListeners) {
         listeners = addTempListeners(this, 'realPath', reject);
       }
-      this.debugMsg(`realPath -> ${remotePath}`);
       this.sftp.realpath(remotePath, (err, absPath) => {
         if (err) {
           if (err.code === 2) {
-            this.debugMsg('realPath <- ""');
             resolve('');
           } else {
-            this.debugMsg(`${err.message} ${remotePath}`, 'realPath');
             reject(this.fmtError(`${err.message} ${remotePath}`, 'realPath', err.code));
           }
         }
-        this.debugMsg(`realPath <- ${absPath}`);
         resolve(absPath);
       });
     }).finally(() => {
@@ -313,7 +315,6 @@ class SftpClient {
             isFIFO: stats.isFIFO(),
             isSocket: stats.isSocket(),
           };
-          this.debugMsg('_xstat: result = ', result);
           resolve(result);
         }
       };
@@ -382,13 +383,11 @@ class SftpClient {
    *                   object if it does
    */
   async exists(remotePath) {
-    this.debugMsg(`exists: remotePath = ${remotePath}`);
     try {
       if (remotePath === '.') {
         return 'd';
       }
       const info = await this.lstat(remotePath);
-      this.debugMsg('exists: <- ', info);
       if (info.isDirectory) {
         return 'd';
       } else if (info.isSymbolicLink) {
@@ -501,20 +500,20 @@ class SftpClient {
         };
         rdr = this.sftp.createReadStream(remotePath, options.readStreamOptions);
         rdr.once('error', (err) => {
-          if (dst && typeof dst !== 'string' && !dst.destroyed) {
-            dst.destroy();
+          if (dst && typeof dst === 'string' && wtr && !wtr.destroyed) {
+            wtr.destroy();
           }
           reject(this.fmtError(`${err.message} ${remotePath}`, 'get', err.code));
         });
         if (dst === undefined) {
           // no dst specified, return buffer of data
-          this.debugMsg('get resolving buffer of data');
+          this.debugMsg('get resolving with buffer of data');
           wtr = concat((buff) => {
             resolve(buff);
           });
         } else if (typeof dst === 'string') {
           // dst local file path
-          this.debugMsg('get returning local file');
+          this.debugMsg(`get called with file path destination ${dst}`);
           const localCheck = haveLocalCreate(dst);
           if (localCheck.status) {
             wtr = fs.createWriteStream(dst, options.writeStreamOptions);
@@ -528,7 +527,7 @@ class SftpClient {
             );
           }
         } else {
-          this.debugMsg('get: returning data into supplied stream');
+          this.debugMsg('get called with stream destination');
           wtr = dst;
         }
         wtr.once('error', (err) => {
@@ -542,16 +541,17 @@ class SftpClient {
         });
         rdr.once('end', () => {
           if (typeof dst === 'string') {
-            this.debugMsg('get: resolving with dst filename');
             resolve(dst);
           } else if (dst !== undefined) {
-            this.debugMsg('get: resolving with writer stream object');
             resolve(wtr);
           }
         });
         rdr.pipe(wtr, options.pipeOptions);
       }
     }).finally(() => {
+      if (rdr && !rdr.destroyed) {
+        rdr.destroy();
+      }
       if (addListeners) {
         removeTempListeners(this, listeners, 'get');
       }
@@ -661,7 +661,6 @@ class SftpClient {
 
   async fastPut(localPath, remotePath, options) {
     try {
-      this.debugMsg(`fastPut -> local ${localPath} remote ${remotePath}`);
       const localCheck = haveLocalAccess(localPath);
       if (!localCheck.status) {
         throw this.fmtError(
@@ -713,6 +712,9 @@ class SftpClient {
       if (haveConnection(this, '_put', reject)) {
         wtr = this.sftp.createWriteStream(rPath, opts.writeStreamOptions);
         wtr.once('error', (err) => {
+          if (typeof lPath === 'string' && rdr && !rdr.destroyed) {
+            rdr.destroy();
+          }
           reject(
             this.fmtError(
               `Write stream error: ${err.message} ${rPath}`,
@@ -728,10 +730,13 @@ class SftpClient {
           this.debugMsg('put source is a buffer');
           wtr.end(lPath);
         } else {
-          rdr =
-            typeof lPath === 'string'
-              ? fs.createReadStream(lPath, opts.readStreamOptions)
-              : lPath;
+          if (typeof lPath === 'string') {
+            this.debugMsg('put source is string path');
+            rdr = fs.createReadStream(lPath, opts.readStreamOptions);
+          } else {
+            this.debugMsg('put source is a stream');
+            rdr = lPath;
+          }
           rdr.once('error', (err) => {
             reject(
               this.fmtError(
@@ -747,6 +752,9 @@ class SftpClient {
         }
       }
     }).finally(() => {
+      if (wtr && !wtr.destroyed) {
+        wtr.destroy();
+      }
       if (addListeners) {
         removeTempListeners(this, listeners, '_put');
       }
@@ -786,7 +794,6 @@ class SftpClient {
         listeners = addTempListeners(this, '_append', reject);
       }
       if (haveConnection(this, '_append', reject)) {
-        this.debugMsg(`append -> remote: ${rPath} `, opts);
         opts.flags = 'a';
         const stream = this.sftp.createWriteStream(rPath, opts);
         stream.on('error', (err) => {
@@ -826,7 +833,7 @@ class SftpClient {
           errorCode.badPath,
         );
       }
-      await this._append(input, remotePath, options);
+      return await this._append(input, remotePath, options);
     } catch (e) {
       throw e.custom ? e : this.fmtError(e.message, 'append', e.code);
     }
@@ -941,7 +948,6 @@ class SftpClient {
       let listeners;
       return new Promise((resolve, reject) => {
         listeners = addTempListeners(this, '_rmdir', reject);
-        this.debugMsg(`_rmdir: dir = ${dir}`);
         this.sftp.rmdir(dir, (err) => {
           if (err) {
             reject(this.fmtError(`${err.message} ${dir}`, 'rmdir', err.code));
@@ -957,7 +963,6 @@ class SftpClient {
       let listeners;
       return new Promise((resolve, reject) => {
         listeners = addTempListeners(this, '_delFiles', reject);
-        this.debugMsg(`_delFiles: path = ${path} fileList = ${fileList}`);
         const pList = [];
         for (const f of fileList) {
           pList.push(this.delete(`${path}/${f.name}`, true, false));
@@ -973,10 +978,8 @@ class SftpClient {
     };
 
     try {
-      this.debugMsg(`rmdir: dir = ${remoteDir} recursive = ${recursive}`);
       const absPath = await normalizeRemotePath(this, remoteDir);
       const existStatus = await this.exists(absPath);
-      this.debugMsg(`rmdir: ${absPath} existStatus = ${existStatus}`);
       if (!existStatus) {
         throw this.fmtError(
           `Bad Path: ${remoteDir}: No such directory`,
@@ -992,19 +995,14 @@ class SftpClient {
         );
       }
       if (!recursive) {
-        this.debugMsg('rmdir: non-recursive - just try to remove it');
         return await _rmdir(absPath);
       }
       const listing = await this.list(absPath);
-      this.debugMsg(`rmdir: listing count = ${listing.length}`);
       if (!listing.length) {
-        this.debugMsg('rmdir: No sub dir or files, just rmdir');
         return await _rmdir(absPath);
       }
       const fileList = listing.filter((i) => i.type !== 'd');
-      this.debugMsg(`rmdir: dir content files to remove = ${fileList.length}`);
       const dirList = listing.filter((i) => i.type === 'd');
-      this.debugMsg(`rmdir: sub-directories to remove = ${dirList.length}`);
       await _delFiles(absPath, fileList);
       for (const d of dirList) {
         await this.rmdir(`${absPath}/${d.name}`, true);
@@ -1244,11 +1242,7 @@ class SftpClient {
 
     try {
       haveConnection(this, 'uploadDir');
-      this.debugMsg(
-        `uploadDir: srcDir = ${srcDir} dstDir = ${dstDir} options = ${options}`,
-      );
       const { remoteDir, remoteStatus } = await getRemoteStatus(dstDir);
-      this.debugMsg(`uploadDir: remoteDir = ${remoteDir} remoteStatus = ${remoteStatus}`);
       checkLocalStatus(srcDir);
       if (!remoteStatus) {
         await this._mkdir(remoteDir, true);
@@ -1257,7 +1251,6 @@ class SftpClient {
         encoding: 'utf8',
         withFileTypes: true,
       });
-      this.debugMsg(`uploadDir: dirEntries = ${dirEntries}`);
       if (options?.filter) {
         dirEntries = dirEntries.filter((item) =>
           options.filter(join(srcDir, item.name), item.isDirectory()),
@@ -1265,8 +1258,6 @@ class SftpClient {
       }
       const dirUploads = dirEntries.filter((item) => item.isDirectory());
       const fileUploads = dirEntries.filter((item) => !item.isDirectory());
-      this.debugMsg(`uploadDir: dirUploads = ${dirUploads}`);
-      this.debugMsg(`uploadDir: fileUploads = ${fileUploads}`);
       await uploadFiles(srcDir, remoteDir, fileUploads, options?.useFastput);
       for (const d of dirUploads) {
         const src = join(srcDir, d.name);
@@ -1527,6 +1518,7 @@ class SftpClient {
       };
       this.on('close', endCloseHandler);
       if (this.sftp) {
+        this.debugMsg('end: Ending SFTP connection');
         this.client.end();
       } else {
         // no actual connection exists - just resolve
@@ -1536,7 +1528,6 @@ class SftpClient {
     }).finally(() => {
       removeTempListeners(this, listeners, 'end');
       this.removeListener('close', endCloseHandler);
-      this.endCalled = false;
     });
   }
 }