ssh2-sftp-client 9.0.0 → 9.0.3

package/README.org

@@ -9,16 +9,34 @@ convenience abstraction as well as a Promise based API.
 Documentation on the methods and available options in the underlying modules can
 be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages.
 
-Current stable release is *v9.0.0*.
+Current stable release is *v9.0.2*.
 
 Code has been tested against Node versions 14.19.1, 16.15.0 and 18.1.0
 
-Node versions < 12.x are not supported. However, node v10.x should still work,
-although some tests will fail due to changes in file system functions used in
-test setup and tear down.
+Node versions < 14.x are not supported. 
 
 ** Version 9.x Changes
-
+  - Fix bug in ~end()~ method where it was possible for the module to attempt calling
+    the underlying ssh2 ~end()~ method when ssh2 has not been initialised. This could
+    lead to undefined reference errors.
+  - Fix bug in ~get()~ method where supplied destination streams were not close, creating
+    a possible resource leak. If the remote file did not exist, the method would return
+    an error, but failed to close any passed in stream supplied as the destination for
+    the data in the ~get()~ call.    
+  - Change the default end and close handlers not to throw error or reject
+    promises. Previously, an end or close event would cause an error to be raised or a
+    promise to be rejected if the event was deemed to be /unexpected/. However,
+    classification of events as being unexpected was unreliable and didn't add much real
+    value. Both these handlers will now invalidate the sftp connection object and log that
+    the event fired and nothing else.
+  - Changed when event handled flags are reset. Now they are reset after a new set of
+    temporary listeners are added.
+  - Don't throw an error when calling end() if there is no active sftp connection. It does
+    no harm to call end() when there is no connection, so no need to raise an error.
+  - Use nullish coalescing when setting retry parameters instead of or'ing with
+    defaults. Allows setting values to 0.  
+  - *Breaking Change*: This version uses syntax not supported in node versions
+    prior to v14. Therefore, node versions less than v14 will not work.
   - *Breaking Change*: This ~list()~ method no longer accepts a regular expression
     for filtering the entries to be returned. You can now specify a filter
     function instead. The function is called for each item in the list of items
@@ -331,21 +349,6 @@ The objects in the array returned by ~list()~ have the following properties;
   }
 #+end_src
 
-**** Pattern Filter
-
-The filter options can be a regular expression (most powerful option) or a
-simple /glob/-like string where * will match any number of characters, e.g.
-
-#+begin_example
-  foo* => foo, foobar, foobaz
-  ,*bar => bar, foobar, tabbar
-  ,*oo* => foo, foobar, look, book
-#+end_example
-
-The /glob/-style matching is very simple. In most cases, you are best off using
-a real regular expression which will allow you to do more powerful matching and
-anchor matches to the beginning/end of the string etc.
-
 *** exists(path) ==> boolean
 
 Tests to see if remote file or directory exists. Returns type of remote object
@@ -1078,7 +1081,6 @@ services.
 
 #+end_src
 
-
 *** createReadStream(remotePath, options)) ==> stream object
 
   Returns a read stream object which is attached to the remote file specified by
@@ -1191,6 +1193,25 @@ the ~end()~ method automatically removes all listeners from the client object.
    additional steps to simulate missing functionality etc. You want to use a CLI
    program which does as little as possible.
 
+** Issues with ~fastPut()~ and ~fastGet()~ Methods
+
+   The ~fastPut()~ and ~fastGet()~ methods are known to be somewhat dependent on
+   SFTP server capabilities. Some SFTP servers just do not work correctly with
+   concurrent connections and some are known to have issues with negotiating
+   packet sizes. These issues can sometimes be resolved by tweaking the options
+   supplied to the methods, such as setting number of concurrent connections or
+   a psecific packet size.
+
+   To see an example of the type of issues you can observe with ~fastPut()~ or
+   ~fastGet()~, have a look at [[https://github.com/theophilusx/ssh2-sftp-client/issues/407][issue 407]], which describes the experiences of one
+   user. Bottom line, when it works, it tends to work well and be significantly
+   faster than using just ~get()~ or ~put()~. However, when developing code to
+   run against different SFTP servers, especially where you are unable to test
+   against each server, you are likely better off just using ~get()~ and ~put()~
+   or structuring your code so that users can select which method to use (this
+   is what =ssh2-sftp-client= does - for example, see the ~!downloadDir()~ and
+   ~uploadDir()~ methods.
+   
 ** Promises, Events & Managing Exceptions
 
    One of the challenges in providing a Promise based API over a module like

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ssh2-sftp-client",
-  "version": "9.0.0",
+  "version": "9.0.3",
   "description": "ssh2 sftp client for node",
   "main": "src/index.js",
   "repository": {
@@ -40,7 +40,7 @@
     "eslint-plugin-mocha": "^10.0.3",
     "eslint-plugin-node": "^11.1.0",
     "eslint-plugin-promise": "^6.0.0",
-    "eslint-plugin-unicorn": "^42.0.0",
+    "eslint-plugin-unicorn": "^43.0.2",
     "mocha": "^10.0.0",
     "moment": "^2.29.1",
     "nyc": "^15.1.0",

package/src/index.js

@@ -18,7 +18,7 @@ const { errorCode } = require('./constants');
 
 class SftpClient {
   constructor(clientName) {
-    this.version = '9.0.0';
+    this.version = '9.0.2';
     this.client = new Client();
     this.sftp = undefined;
     this.clientName = clientName ? clientName : 'sftp';
@@ -31,7 +31,7 @@ class SftpClient {
     this.debug = undefined;
 
     this.client.on('close', () => {
-      if (this.endCalled || this.closeHandled) {
+      if (this.endCalled || this.errorHandled || this.closeHandled) {
         // we are processing an expected end event or close event handled elsewhere
         this.debugMsg('Global: Ignoring handled close event');
       } else {
@@ -41,7 +41,7 @@ class SftpClient {
     });
 
     this.client.on('end', () => {
-      if (this.endCalled || this.endHandled) {
+      if (this.endCalled || this.errorHandled || this.endHandled) {
         // end event expected or handled elsewhere
         this.debugMsg('Global: Ignoring hanlded end event');
       } else {
@@ -95,26 +95,20 @@ class SftpClient {
     } else {
       switch (err.code) {
         case 'ENOTFOUND':
-          msg =
-            `${name}: ${err.level} error. ` +
-            `Address lookup failed for host ${err.hostname}${retry}`;
+          msg = `${name}: Address lookup failed for host${retry}`;
           break;
         case 'ECONNREFUSED':
-          msg =
-            `${name}: ${err.level} error. Remote host at ` +
-            `${err.address} refused connection${retry}`;
+          msg = `${name}: Remote host refused connection${retry}`;
           break;
         case 'ECONNRESET':
-          msg =
-            `${name}: Remote host has reset the connection: ` +
-            `${err.message}${retry}`;
+          msg = `${name}: Remote host has reset the connection: ${err.message}${retry}`;
           break;
         default:
           msg = `${name}: ${err.message}${retry}`;
       }
       code = err.code ? err.code : errorCode.generic;
     }
-    let newError = new Error(msg);
+    const newError = new Error(msg);
     newError.code = code;
     newError.custom = true;
     this.debugMsg(`${newError.message} (${newError.code})`);
@@ -159,9 +153,7 @@ class SftpClient {
     return new Promise((resolve, reject) => {
       listeners = addTempListeners(this, 'getConnection', reject);
       doReady = () => {
-        this.debugMsg(
-          'getConnection ready listener: got connection - promise resolved'
-        );
+        this.debugMsg('getConnection ready listener: got connection - promise resolved');
         resolve(true);
       };
       this.on('ready', doReady);
@@ -169,7 +161,6 @@ class SftpClient {
     }).finally(() => {
       this.removeListener('ready', doReady);
       removeTempListeners(this, listeners, 'getConnection');
-      this._resetEventFlags();
     });
   }
 
@@ -177,7 +168,6 @@ class SftpClient {
     return new Promise((resolve, reject) => {
       this.client.sftp((err, sftp) => {
         if (err) {
-          this.client.end();
           reject(this.fmtError(err, 'getSftpChannel', err.code));
         } else {
           this.debugMsg('getSftpChannel: SFTP channel established');
@@ -208,10 +198,7 @@ class SftpClient {
       if (config.debug) {
         this.debug = config.debug;
         this.debugMsg('connect: Debugging turned on');
-        this.debugMsg(
-          `ssh2-sftp-client Version: ${this.version} `,
-          process.versions
-        );
+        this.debugMsg(`ssh2-sftp-client Version: ${this.version} `, process.versions);
       }
       if (this.sftp) {
         throw this.fmtError(
@@ -220,10 +207,10 @@ class SftpClient {
           errorCode.connect
         );
       }
-      let retryOpts = {
-        retries: config.retries || 1,
-        factor: config.factor || 2,
-        minTimeout: config.retry_minTimeout || 25000,
+      const retryOpts = {
+        retries: config.retries ?? 1,
+        factor: config.factor ?? 2,
+        minTimeout: config.retry_minTimeout ?? 25000,
       };
       await promiseRetry(retryOpts, async (retry, attempt) => {
         try {
@@ -236,16 +223,8 @@ class SftpClient {
             case 'ERR_SOCKET_BAD_PORT':
               throw err;
             case undefined: {
-              if (
-                err.message.endsWith(
-                  'All configured authentication methods failed'
-                )
-              ) {
-                throw this.fmtError(
-                  err.message,
-                  'getConnection',
-                  errorCode.badAuth
-                );
+              if (err.message.endsWith('All configured authentication methods failed')) {
+                throw this.fmtError(err.message, 'getConnection', errorCode.badAuth);
               }
               retry(err);
               break;
@@ -255,14 +234,13 @@ class SftpClient {
           }
         }
       });
-      let sftp = await this.getSftpChannel();
+      const sftp = await this.getSftpChannel();
       return sftp;
     } catch (err) {
-      this.end();
+      await this.end();
       throw err.custom ? err : this.fmtError(err, 'connect');
     } finally {
       removeTempListeners(this, listeners, 'connect');
-      this._resetEventFlags();
     }
   }
 
@@ -286,9 +264,7 @@ class SftpClient {
             this.debugMsg('_realPath <- ""');
             resolve('');
           } else {
-            reject(
-              this.fmtError(`${err.message} ${rPath}`, 'realPath', err.code)
-            );
+            reject(this.fmtError(`${err.message} ${rPath}`, 'realPath', err.code));
           }
         }
         this.debugMsg(`_realPath <- ${absPath}`);
@@ -309,7 +285,6 @@ class SftpClient {
         : this.fmtError(`${e.message} ${remotePath}`, 'realPath', e.code);
     } finally {
       removeTempListeners(this, listeners, 'realPath');
-      this._resetEventFlags();
     }
   }
 
@@ -336,13 +311,7 @@ class SftpClient {
       this.sftp.stat(aPath, (err, stats) => {
         if (err) {
           if (err.code === 2 || err.code === 4) {
-            reject(
-              this.fmtError(
-                `No such file: ${aPath}`,
-                '_stat',
-                errorCode.notexist
-              )
-            );
+            reject(this.fmtError(`No such file: ${aPath}`, '_stat', errorCode.notexist));
           } else {
             reject(this.fmtError(`${err.message} ${aPath}`, '_stat', err.code));
           }
@@ -380,7 +349,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err, 'stat', err.code);
     } finally {
       removeTempListeners(this, listeners, 'stat');
-      this._resetEventFlags();
     }
   }
 
@@ -437,7 +405,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err, 'exists', err.code);
     } finally {
       removeTempListeners(this, listeners, 'exists');
-      this._resetEventFlags();
     }
   }
 
@@ -447,7 +414,7 @@ class SftpClient {
    * List contents of a remote directory. If a pattern is provided,
    * filter the results to only include files with names that match
    * the supplied pattern. Return value is an array of file entry
-   * objects that include properties for type, name, size, modifiyTime,
+   * objects that include properties for type, name, size, modifyTime,
    * accessTime, rights {user, group other}, owner and group.
    *
    * @param {String} remotePath - path to remote directory
@@ -458,9 +425,7 @@ class SftpClient {
     return new Promise((resolve, reject) => {
       this.sftp.readdir(remotePath, (err, fileList) => {
         if (err) {
-          reject(
-            this.fmtError(`${err.message} ${remotePath}`, 'list', err.code)
-          );
+          reject(this.fmtError(`${err.message} ${remotePath}`, 'list', err.code));
         } else {
           const reg = /-/gi;
           const newList = fileList.map((item) => {
@@ -497,12 +462,9 @@ class SftpClient {
       haveConnection(this, 'list');
       return await this._list(remotePath, filter);
     } catch (e) {
-      throw e.custom
-        ? e
-        : this.fmtError(`${e.message} ${remotePath}`, 'list', e.code);
+      throw e.custom ? e : this.fmtError(`${e.message} ${remotePath}`, 'list', e.code);
     } finally {
       removeTempListeners(this, listeners, 'list');
-      this._resetEventFlags();
     }
   }
 
@@ -536,11 +498,14 @@ class SftpClient {
       };
       rdr = this.sftp.createReadStream(rPath, opts.readStreamOptions);
       rdr.once('error', (err) => {
+        if (dst && typeof dst !== 'string' && !dst.destroyed) {
+          dst.destroy();
+        }
         reject(this.fmtError(`${err.message} ${rPath}`, '_get', err.code));
       });
       if (dst === undefined) {
         // no dst specified, return buffer of data
-        this.debugMsg('get returning buffer of data');
+        this.debugMsg('get resolving buffer of data');
         wtr = concat((buff) => {
           resolve(buff);
         });
@@ -575,8 +540,10 @@ class SftpClient {
       });
       rdr.once('end', () => {
         if (typeof dst === 'string') {
+          this.debugMsg('get: resolving with dst filename');
           resolve(dst);
-        } else {
+        } else if (dst !== undefined) {
+          this.debugMsg('get: resolving with writer stream object');
           resolve(wtr);
         }
       });
@@ -591,12 +558,9 @@ class SftpClient {
       haveConnection(this, 'get');
       return await this._get(remotePath, dst, options);
     } catch (e) {
-      throw e.custom
-        ? e
-        : this.fmtError(`${e.message} ${remotePath}`, 'get', e.code);
+      throw e.custom ? e : this.fmtError(`${e.message} ${remotePath}`, 'get', e.code);
     } finally {
       removeTempListeners(this, listeners, 'get');
-      this._resetEventFlags();
     }
   }
 
@@ -614,9 +578,7 @@ class SftpClient {
     return new Promise((resolve, reject) => {
       this.sftp.fastGet(rPath, lPath, opts, (err) => {
         if (err) {
-          reject(
-            this.fmtError(`${err.message} Remote: ${rPath} Local: ${lPath}`)
-          );
+          reject(this.fmtError(`${err.message} Remote: ${rPath} Local: ${lPath}`));
         }
         resolve(`${rPath} was successfully download to ${lPath}!`);
       });
@@ -630,9 +592,7 @@ class SftpClient {
       haveConnection(this, 'fastGet');
       const ftype = await this.exists(remotePath);
       if (ftype !== '-') {
-        const msg = `${
-          !ftype ? 'No such file ' : 'Not a regular file'
-        } ${remotePath}`;
+        const msg = `${!ftype ? 'No such file ' : 'Not a regular file'} ${remotePath}`;
         throw this.fmtError(msg, 'fastGet', errorCode.badPath);
       }
       const localCheck = haveLocalCreate(localPath);
@@ -648,7 +608,6 @@ class SftpClient {
       throw this.fmtError(err, 'fastGet');
     } finally {
       removeTempListeners(this, listeners, 'fastGet');
-      this._resetEventFlags();
     }
   }
 
@@ -707,7 +666,6 @@ class SftpClient {
       throw e.custom ? e : this.fmtError(e.message, 'fastPut', e.code);
     } finally {
       removeTempListeners(this, listeners, 'fastPut');
-      this._resetEventFlags();
     }
   }
 
@@ -755,9 +713,7 @@ class SftpClient {
         rdr.once('error', (err) => {
           reject(
             this.fmtError(
-              `${err.message} ${
-                typeof lPath === 'string' ? lPath : '<stream>'
-              }`,
+              `${err.message} ${typeof lPath === 'string' ? lPath : '<stream>'}`,
               '_put',
               err.code
             )
@@ -788,7 +744,6 @@ class SftpClient {
       throw e.custom ? e : this.fmtError(e.message, 'put', e.code);
     } finally {
       removeTempListeners(this, listeners, 'put');
-      this._resetEventFlags();
     }
   }
 
@@ -845,7 +800,6 @@ class SftpClient {
       throw e.custom ? e : this.fmtError(e.message, 'append', e.code);
     } finally {
       removeTempListeners(this, listeners, 'append');
-      this._resetEventFlags();
     }
   }
 
@@ -936,7 +890,6 @@ class SftpClient {
       throw this.fmtError(`${err.message}`, 'mkdir', err.code);
     } finally {
       removeTempListeners(this, listeners, 'append');
-      this._resetEventFlags();
     }
   }
 
@@ -977,9 +930,7 @@ class SftpClient {
             }
             const promiseList = [];
             for (const f of files) {
-              promiseList.push(
-                this._delete(`${p}${this.remotePathSep}${f.name}`)
-              );
+              promiseList.push(this._delete(`${p}${this.remotePathSep}${f.name}`));
             }
             await Promise.all(promiseList);
           }
@@ -1015,7 +966,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err.message, 'rmdir', err.code);
     } finally {
       removeTempListeners(this, listeners, 'rmdir');
-      this._resetEventFlags();
     }
   }
 
@@ -1037,9 +987,7 @@ class SftpClient {
           if (notFoundOK && err.code === 2) {
             resolve(`Successfully deleted ${rPath}`);
           } else {
-            reject(
-              this.fmtError(`${err.message} ${rPath}`, 'delete', err.code)
-            );
+            reject(this.fmtError(`${err.message} ${rPath}`, 'delete', err.code));
           }
         }
         resolve(`Successfully deleted ${rPath}`);
@@ -1057,7 +1005,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err.message, 'delete', err.code);
     } finally {
       removeTempListeners(this, listeners, 'delete');
-      this._resetEventFlags();
     }
   }
 
@@ -1098,14 +1045,9 @@ class SftpClient {
     } catch (err) {
       throw err.custom
         ? err
-        : this.fmtError(
-            `${err.message} ${fromPath} ${toPath}`,
-            'rename',
-            err.code
-          );
+        : this.fmtError(`${err.message} ${fromPath} ${toPath}`, 'rename', err.code);
     } finally {
       removeTempListeners(this, listeners, 'rename');
-      this._resetEventFlags();
     }
   }
 
@@ -1147,14 +1089,9 @@ class SftpClient {
     } catch (err) {
       throw err.custom
         ? err
-        : this.fmtError(
-            `${err.message} ${fromPath} ${toPath}`,
-            'posixRename',
-            err.code
-          );
+        : this.fmtError(`${err.message} ${fromPath} ${toPath}`, 'posixRename', err.code);
     } finally {
       removeTempListeners(this, listeners, 'posixRename');
-      this._resetEventFlags();
     }
   }
 
@@ -1191,7 +1128,6 @@ class SftpClient {
         : this.fmtError(`${err.message} ${remotePath}`, 'chmod', err.code);
     } finally {
       removeTempListeners(this, listeners, 'chmod');
-      this._resetEventFlags();
     }
   }
 
@@ -1266,9 +1202,7 @@ class SftpClient {
           }
           this.client.emit('upload', { source: newSrc, destination: newDst });
         } else {
-          this.debugMsg(
-            `uploadDir: File ignored: ${e.name} not a regular file`
-          );
+          this.debugMsg(`uploadDir: File ignored: ${e.name} not a regular file`);
         }
         await Promise.all(fileUploads);
       }
@@ -1291,7 +1225,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err, 'uploadDir');
     } finally {
       removeTempListeners(this, listeners, 'chmod');
-      this._resetEventFlags();
     }
   }
 
@@ -1353,9 +1286,7 @@ class SftpClient {
           }
           this.client.emit('download', { source: newSrc, destination: newDst });
         } else {
-          this.debugMsg(
-            `downloadDir: File ignored: ${f.name} not regular file`
-          );
+          this.debugMsg(`downloadDir: File ignored: ${f.name} not regular file`);
         }
       }
       await Promise.all(downloadFiles);
@@ -1377,7 +1308,6 @@ class SftpClient {
       throw err.custom ? err : this.fmtError(err, 'downloadDir', err.code);
     } finally {
       removeTempListeners(this, listeners, 'downloadDir');
-      this._resetEventFlags();
     }
   }
 
@@ -1402,12 +1332,9 @@ class SftpClient {
       const stream = this.sftp.createReadStream(remotePath, options);
       return stream;
     } catch (err) {
-      throw err.custom
-        ? err
-        : this.fmtError(err.message, 'createReadStream', err.code);
+      throw err.custom ? err : this.fmtError(err.message, 'createReadStream', err.code);
     } finally {
       removeTempListeners(this, listeners, 'createReadStreame');
-      this._resetEventFlags();
     }
   }
 
@@ -1432,12 +1359,9 @@ class SftpClient {
       const stream = this.sftp.createWriteStream(remotePath, options);
       return stream;
     } catch (err) {
-      throw err.custom
-        ? err
-        : this.fmtError(err.message, 'createWriteStream', err.code);
+      throw err.custom ? err : this.fmtError(err.message, 'createWriteStream', err.code);
     } finally {
       removeTempListeners(this, listeners, 'createWriteStream');
-      this._resetEventFlags();
     }
   }
 
@@ -1486,11 +1410,7 @@ class SftpClient {
         );
       }
       if (srcExists !== '-') {
-        throw this.fmtError(
-          `Source not a file ${srcPath}`,
-          'rcopy',
-          errorCode.badPath
-        );
+        throw this.fmtError(`Source not a file ${srcPath}`, 'rcopy', errorCode.badPath);
       }
       const dstPath = await normalizeRemotePath(this, dst);
       const dstExists = await this.exists(dstPath);
@@ -1501,12 +1421,11 @@ class SftpClient {
           errorCode.badPath
         );
       }
-      return await this._rcopy(srcPath, dstPath);
+      return this._rcopy(srcPath, dstPath);
     } catch (err) {
       throw err.custom ? err : this.fmtError(err, 'rcopy');
     } finally {
       removeTempListeners(this, listeners, 'rcopy');
-      this._resetEventFlags();
     }
   }
   /**
@@ -1527,14 +1446,17 @@ class SftpClient {
         resolve(true);
       };
       this.on('close', endCloseHandler);
-      if (haveConnection(this, 'end', reject)) {
+      if (this.client.sftp) {
         this.client.end();
+      } else {
+        // no actual connection exists - just resolve
+        this.debugMsg('end: Called when no connection active');
+        resolve(true);
       }
     }).finally(() => {
       removeTempListeners(this, listeners, 'end');
       this.removeListener('close', endCloseHandler);
       this.endCalled = false;
-      this._resetEventFlags();
     });
   }
 }