ssh2-sftp-client 7.0.4 → 7.2.2

package/README.md

@@ -61,9 +61,9 @@ an SFTP client for node.js, a wrapper around [SSH2](https://github.com/mscdex/ss
 
 Documentation on the methods and available options in the underlying modules can be found on the [SSH2](https://github.com/mscdex/ssh2) project pages.
 
-Current stable release is **v7.0.4**.
+Current stable release is **v7.2.2**.
 
-Code has been tested against Node versions 12.22.1, 14.17.0 and 16.2.0
+Code has been tested against Node versions 14.18.3, 16.13.2 and 17.4.0
 
 Node versions < 10.x are not supported.
 
@@ -99,6 +99,8 @@ sftp.connect({
 
 -   **Breaking Change** Expanded option handling for `get()` and `put()` methods. A number of use cases were identified where setting specific options on the read and write streams and the pipe operation are necessary. For example, disabling `autoClose` on streams or the `end` event in pipes. The `options` argument for `get()` and `put()` calls now supports properties for `readStreamOptions`, `writeStreamOptions` and `pipeOptions`. Note that options are only applied to streams created by the `get()` and `put()` methods. Streams passed into these methods are under the control of the client code and therefore cannot have options supplied in arguments to those streams (you would apply such options when you create the streams). Options are typically only necessary in special use cases. Most of the time, no options are required. However, if you are currently using options to either `put()` or `get()`, you will need to update your code to map these options to the new structure.
 
+-   **Breaking Change 7.1.0** A race condition was identified when using a put() call with a writeStream option of `autoClose: false`. In some situations, the promise would be resolved before the final close of the write stream. This could result in errors if you immediately attempt to access the uploaded file. To avoid this situatioin, the promise is now resolved once a `close` event is emitted. This means that setting `autoClose: false` can no longer be supported. The write stream for `put()` will autoClose once data writing has completed.
+
 -   Improved event handling. A listener for a global error event is now defined to catch errors which occur in-between method calls i.e. connection lost in-between calls to the library methods. A new mechanism has also been added for removal of listeners when no longer required.
 
 # Documentation<a id="sec-5"></a>
@@ -497,11 +499,12 @@ Upload data from local system to remote server. If the `src` argument is a strin
         flags: 'w',  // w - write and a - append
         encoding: null, // use null for binary files
         mode: 0o666, // mode to use for created file (rwx)
-        autoClose: true // automatically close the write stream when finished
     }}
     ```
     
     The most common options to use are mode and encoding. The values shown above are the defaults. You do not have to set encoding to utf-8 for text files, null is fine for all file types. However, using utf-8 encoding for binary files will often result in data corruption.
+    
+    Note that you cannot set `autoClose: false` for `writeStreamOptions`. If you attempt to set this property to false, it will be ignored. This is necessary to avoid a race condition which may exist when setting `autoClose` to false on the writeStream. As there is no easy way to access the writeStream once the promise has been resolved, setting this to autoClose false is not terribly useful as there is no easy way to manually close the stream after the promise has been resolved.
 
 2.  Example Use
 
@@ -1356,3 +1359,6 @@ Thanks to the following for their contributions -
 -   **anton-erofeev:** Documentation fix
 -   **Ladislav Jacho:** Contributed solution explanation for connections hanging when transferring larger files.
 -   **Emma Milner:** Contributed fix for put() bug
+-   **Witni Davis:** Contributed PR to fix put() RCE when using 'finish' rather than 'close' to resolve promise
+-   **Maik Marschner:** Contributed fix for connect() not returning sftp object. Also included test to check for this regression in future.
+-   **cakemasher:** Contributed fix for removeTempListeners().

package/README.org

@@ -9,9 +9,9 @@ convenience abstraction as well as a Promise based API.
 Documentation on the methods and available options in the underlying modules can
 be found on the [[https://github.com/mscdex/ssh2][SSH2]] project pages.
 
-Current stable release is *v7.0.4*.
+Current stable release is *v7.2.2*.
 
-Code has been tested against Node versions 12.22.1, 14.17.0 and 16.2.0
+Code has been tested against Node versions 14.18.3, 16.13.2 and 17.4.0
 
 Node versions < 10.x are not supported.
 
@@ -62,6 +62,14 @@ npm install ssh2-sftp-client
   currently using options to either ~put()~ or ~get()~, you will need to update
   your code to map these options to the new structure. 
 
+- *Breaking Change 7.1.0* A race condition was identified when using a put()
+  call with a writeStream option of ~autoClose: false~. In some situations, the
+  promise would be resolved before the final close of the write stream. This
+  could result in errors if you immediately attempt to access the uploaded
+  file. To avoid this situatioin, the promise is now resolved once a ~close~
+  event is emitted. This means that setting ~autoClose: false~ can no longer be
+  supported. The write stream for ~put()~ will autoClose once data writing has completed.
+   
 - Improved event handling. A listener for a global error event is now defined to
   catch errors which occur in-between method calls i.e. connection lost
   in-between calls to the library methods. A new mechanism has also been added
@@ -546,7 +554,6 @@ option value. For example, you might use the following to set ~writeStream~ opti
       flags: 'w',  // w - write and a - append
       encoding: null, // use null for binary files
       mode: 0o666, // mode to use for created file (rwx)
-      autoClose: true // automatically close the write stream when finished
   }}
 #+end_src
 
@@ -555,6 +562,14 @@ the defaults. You do not have to set encoding to utf-8 for text files, null is
 fine for all file types. However, using utf-8 encoding for binary files will
 often result in data corruption.
 
+Note that you cannot set ~autoClose: false~ for ~writeStreamOptions~. If you
+attempt to set this property to false, it will be ignored. This is necessary to
+avoid a race condition which may exist when setting ~autoClose~ to false on the
+writeStream. As there is no easy way to access the writeStream once the promise
+has been resolved, setting this to autoClose false is not terribly useful as
+there is no easy way to manually close the stream after the promise has been
+resolved.
+
 **** Example Use
 
 #+begin_src javascript
@@ -1765,3 +1780,9 @@ Thanks to the following for their contributions -
 - Ladislav Jacho :: Contributed solution explanation for connections hanging
   when transferring larger files.
 - Emma Milner :: Contributed fix for put() bug
+- Witni Davis :: Contributed PR to fix put() RCE when using 'finish' rather than
+  'close' to resolve promise
+- Maik Marschner :: Contributed fix for connect() not returning sftp object.
+  Also included test to check for this regression in future.
+- cakemasher :: Contributed fix for removeTempListeners().
+  

package/package.json

@@ -1,17 +1,13 @@
 {
   "name": "ssh2-sftp-client",
-  "version": "7.0.4",
+  "version": "7.2.2",
   "description": "ssh2 sftp client for node",
   "main": "src/index.js",
   "repository": {
     "type": "git",
     "url": "https://github.com/theophilusx/ssh2-sftp-client"
   },
-  "keywords": [
-    "sftp",
-    "nodejs",
-    "promises"
-  ],
+  "keywords": ["sftp", "nodejs", "promises"],
   "scripts": {
     "test": "mocha",
     "coverage": "nyc npm run test",
@@ -32,24 +28,24 @@
   "dependencies": {
     "concat-stream": "^2.0.0",
     "promise-retry": "^2.0.1",
-    "ssh2": "^1.4.0"
+    "ssh2": "^1.6.0"
   },
   "devDependencies": {
-    "chai": "^4.2.0",
+    "chai": "^4.3.4",
     "chai-as-promised": "^7.1.1",
     "chai-subset": "^1.6.0",
     "checksum": "^1.0.0",
-    "dotenv": "^10.0.0",
-    "eslint": "^7.32.0",
+    "dotenv": "^15.0.0",
+    "eslint": "^8.5.0",
     "eslint-config-prettier": "^8.3.0",
-    "eslint-plugin-mocha": "^9.0.0",
+    "eslint-plugin-mocha": "^10.0.3",
     "eslint-plugin-node": "^11.1.0",
-    "eslint-plugin-promise": "^5.1.0",
-    "eslint-plugin-unicorn": "^35.0.0",
-    "mocha": "^9.0.2",
+    "eslint-plugin-promise": "^6.0.0",
+    "eslint-plugin-unicorn": "^40.1.0",
+    "mocha": "^9.1.2",
     "moment": "^2.29.1",
     "nyc": "^15.1.0",
-    "prettier": "^2.3.2",
+    "prettier": "^2.5.0",
     "through2": "^4.0.2",
     "winston": "^3.3.3"
   }

package/src/index.js

@@ -34,6 +34,7 @@ class SftpClient {
     this.remotePathSep = '/';
     this.remotePlatform = 'unix';
     this.debug = undefined;
+    this.tempListeners = {};
 
     this.client.on('close', () => {
       if (this.endCalled || this.closeHandled) {
@@ -114,7 +115,7 @@ class SftpClient {
    *
    * @param {Object} config - an SFTP configuration object
    *
-   * @return {Promise} which will resolve to an sftp client object
+   * @return {Promise<Object>} which will resolve to an sftp client object
    *
    */
   getConnection(config) {
@@ -150,6 +151,7 @@ class SftpClient {
       this.client.sftp((err, sftp) => {
         if (err) {
           this.debugMsg(`getSftpChannel: SFTP Channel Error: ${err.message}`);
+          this.client.end();
           reject(fmtError(err, 'getSftpChannel', err.code));
         } else {
           this.debugMsg('getSftpChannel: SFTP channel established');
@@ -173,7 +175,7 @@ class SftpClient {
    *
    * @param {Object} config - an SFTP configuration object
    *
-   * @return {Promise} which will resolve to an sftp client object
+   * @return {Promise<Object>} which will resolve to an sftp client object
    *
    */
   async connect(config) {
@@ -204,7 +206,7 @@ class SftpClient {
           minTimeout: config.retry_minTimeout || 1000,
         }
       );
-      await this.getSftpChannel();
+      return this.getSftpChannel();
     } catch (err) {
       this.debugMsg(`connect: Error ${err.message}`);
       this._resetEventFlags();
@@ -221,7 +223,7 @@ class SftpClient {
    * Returns undefined if the path does not exists.
    *
    * @param {String} remotePath - remote path, may be relative
-   * @returns {Promise} - remote absolute path or undefined
+   * @returns {Promise<String>} - remote absolute path or ''
    */
   realPath(remotePath) {
     return new Promise((resolve, reject) => {
@@ -249,6 +251,13 @@ class SftpClient {
     });
   }
 
+  /**
+   * @async
+   *
+   * Return the current workding directory path
+   *
+   * @returns {Promise<String>} - current remote working directory
+   */
   cwd() {
     return this.realPath('.');
   }
@@ -257,7 +266,7 @@ class SftpClient {
    * Retrieves attributes for path
    *
    * @param {String} remotePath - a string containing the path to a file
-   * @return {Promise} stats - attributes info
+   * @return {Promise<Object>} stats - attributes info
    */
   async stat(remotePath) {
     const _stat = (aPath) => {
@@ -301,7 +310,7 @@ class SftpClient {
           }
         });
       }).finally(() => {
-        removeTempListeners(this, 'stat');
+        removeTempListeners(this, '_stat');
       });
     };
 
@@ -323,7 +332,7 @@ class SftpClient {
    *
    * @param {string} remotePath - path to the object on the sftp server.
    *
-   * @return {Promise} returns false if object does not exist. Returns type of
+   * @return {Promise<Boolean|String>} returns false if object does not exist. Returns type of
    *                   object if it does
    */
   async exists(remotePath) {
@@ -382,8 +391,7 @@ class SftpClient {
    *
    * @param {String} remotePath - path to remote directory
    * @param {RegExp} pattern - regular expression to match filenames
-   * @returns {Promise} array of file description objects
-   * @throws {Error}
+   * @returns {Promise<Array>} array of file description objects
    */
   list(remotePath, pattern = /.*/) {
     return new Promise((resolve, reject) => {
@@ -449,7 +457,7 @@ class SftpClient {
    * @param {Object} options - options object with supported properties of readStreamOptions,
    *                          writeStreamOptions and pipeOptions.
    *
-   * @return {Promise}
+   * @return {Promise<String|Stream|Buffer>}
    */
   get(
     remotePath,
@@ -507,13 +515,29 @@ class SftpClient {
               )
             );
           });
-          rdr.once('end', () => {
-            if (typeof dst === 'string') {
-              resolve(dst);
-            } else {
-              resolve(wtr);
-            }
-          });
+          if (
+            Object.hasOwnProperty.call(options, 'pipeOptions') &&
+            Object.hasOwnProperty.call(options.pipeOptions, 'end') &&
+            !options.pipeOptions.end
+          ) {
+            rdr.once('end', () => {
+              this.debugMsg('get resolved on reader end event');
+              if (typeof dst === 'string') {
+                resolve(dst);
+              } else {
+                resolve(wtr);
+              }
+            });
+          } else {
+            wtr.once('finish', () => {
+              this.debugMsg('get resolved on writer finish event');
+              if (typeof dst === 'string') {
+                resolve(dst);
+              } else {
+                resolve(wtr);
+              }
+            });
+          }
         }
         rdr.pipe(wtr, options.pipeOptions ? options.pipeOptions : {});
       }
@@ -522,14 +546,16 @@ class SftpClient {
       this._resetEventFlags();
       if (
         rdr &&
-        options.readStreamOptions &&
+        Object.hasOwnProperty.call(options, 'readStreamOptions') &&
+        Object.hasOwnProperty.call(options.readStreamOptions, 'autoClose') &&
         options.readStreamOptions.autoClose === false
       ) {
         rdr.destroy();
       }
       if (
         wtr &&
-        options.writeStreamOptions &&
+        Object.hasOwnProperty.call(options, 'writeStreamOptions') &&
+        Object.hasOwnProperty.call(options.writeStreamOptions, 'autoClose') &&
         options.writeStreamOptions.autoClose === false &&
         typeof dst === 'string'
       ) {
@@ -543,13 +569,10 @@ class SftpClient {
    * Downloads a file at remotePath to localPath using parallel reads
    * for faster throughput.
    *
-   * See 'fastGet' at
-   * https://github.com/mscdex/ssh2-streams/blob/master/SFTPStream.md
-   *
    * @param {String} remotePath
    * @param {String} localPath
    * @param {Object} options
-   * @return {Promise} the result of downloading the file
+   * @return {Promise<String>} the result of downloading the file
    */
   async fastGet(remotePath, localPath, options) {
     try {
@@ -569,7 +592,7 @@ class SftpClient {
         err.code = errorCode.badPath;
         throw err;
       }
-      await new Promise((resolve, reject) => {
+      let rslt = await new Promise((resolve, reject) => {
         if (haveConnection(this, 'fastGet', reject)) {
           this.debugMsg(
             `fastGet -> remote: ${remotePath} local: ${localPath} `,
@@ -587,6 +610,7 @@ class SftpClient {
       }).finally(() => {
         removeTempListeners(this, 'fastGet');
       });
+      return rslt;
     } catch (err) {
       this._resetEventFlags();
       throw fmtError(err, 'fastGet');
@@ -604,7 +628,7 @@ class SftpClient {
    * @param {String} localPath
    * @param {String} remotePath
    * @param {Object} options
-   * @return {Promise} the result of downloading the file
+   * @return {Promise<String>} the result of downloading the file
    */
   fastPut(localPath, remotePath, options) {
     this.debugMsg(`fastPut -> local ${localPath} remote ${remotePath}`);
@@ -664,12 +688,16 @@ class SftpClient {
    * @param  {Object} options - options used for read, write stream and pipe configuration
    *                            value supported by node. Allowed properties are readStreamOptions,
    *                            writeStreamOptions and pipeOptions.
-   * @return {Promise}
+   * @return {Promise<String>}
    */
   put(
     localSrc,
     remotePath,
-    options = { readStreamOptions: {}, writeStreamOptions: {}, pipeOptions: {} }
+    options = {
+      readStreamOptions: {},
+      writeStreamOptions: { autoClose: true },
+      pipeOptions: {},
+    }
   ) {
     let wtr, rdr;
 
@@ -691,13 +719,15 @@ class SftpClient {
         addTempListeners(this, 'put', reject);
         wtr = this.sftp.createWriteStream(
           remotePath,
-          options.writeStreamOptions ? options.writeStreamOptions : {}
+          options.writeStreamOptions
+            ? { ...options.writeStreamOptions, autoClose: true }
+            : {}
         );
         wtr.once('error', (err) => {
           this.debugMsg(`put: write stream error ${err.message}`);
           reject(fmtError(`${err.message} ${remotePath}`, 'put', err.code));
         });
-        wtr.once('finish', () => {
+        wtr.once('close', () => {
           this.debugMsg('put: promise resolved');
           resolve(`Uploaded data stream to ${remotePath}`);
         });
@@ -735,19 +765,13 @@ class SftpClient {
       this._resetEventFlags();
       if (
         rdr &&
-        options.readStreamOptions &&
+        Object.hasOwnProperty.call(options, 'readStreamOptions') &&
+        Object.hasOwnProperty.call(options.readStreamOptions, 'autoClose') &&
         options.readStreamOptions.autoClose === false &&
         typeof localSrc === 'string'
       ) {
         rdr.destroy();
       }
-      if (
-        wtr &&
-        options.writeStreamOptions &&
-        options.writeStreamOptions.autoClose === false
-      ) {
-        wtr.destroy();
-      }
     });
   }
 
@@ -757,9 +781,8 @@ class SftpClient {
    * @param  {Buffer|stream} input
    * @param  {String} remotePath
    * @param  {Object} options
-   * @return {Promise}
+   * @return {Promise<String>}
    */
-
   async append(input, remotePath, options = {}) {
     const fileType = await this.exists(remotePath);
     if (fileType && fileType === 'd') {
@@ -807,7 +830,7 @@ class SftpClient {
    *
    * @param {string} remotePath - remote directory path.
    * @param {boolean} recursive - if true, recursively create directories
-   * @return {Promise}
+   * @return {Promise<String>}
    */
   async mkdir(remotePath, recursive = false) {
     const _mkdir = (p) => {
@@ -845,6 +868,14 @@ class SftpClient {
     try {
       haveConnection(this, 'mkdir');
       let rPath = await normalizeRemotePath(this, remotePath);
+      let targetExists = await this.exists(rPath);
+      if (targetExists && targetExists !== 'd') {
+        let error = new Error(`Bad path: ${rPath} already exists as a file`);
+        error.code = errorCode.badPath;
+        throw error;
+      } else if (targetExists) {
+        return `${rPath} already exists`;
+      }
       if (!recursive) {
         return await _mkdir(rPath);
       }
@@ -873,7 +904,7 @@ class SftpClient {
    * @param {string} remotePath - path to directory to be removed
    * @param {boolean} recursive - if true, remove directories/files in target
    *                             directory
-   * @return {Promise}
+   * @return {Promise<String>}
    */
   async rmdir(remotePath, recursive = false) {
     const _rmdir = (p) => {
@@ -926,7 +957,7 @@ class SftpClient {
    * @param {string} remotePath - path to the file to delete
    * @param {boolean} notFoundOK - if true, ignore errors for missing target.
    *                               Default is false.
-   * @return {Promise} with string 'Successfully deleted file' once resolved
+   * @return {Promise<String>} with string 'Successfully deleted file' once resolved
    *
    */
   delete(remotePath, notFoundOK = false) {
@@ -963,7 +994,7 @@ class SftpClient {
    * @param {string} fromPath - path to the file to be renamed.
    * @param {string} toPath - path to the new name.
    *
-   * @return {Promise}
+   * @return {Promise<String>}
    *
    */
   rename(fromPath, toPath) {
@@ -1000,7 +1031,7 @@ class SftpClient {
    * @param {string} fromPath - path to the file to be renamed.
    * @param {string} toPath - path  the new name.
    *
-   * @return {Promise}
+   * @return {Promise<String>}
    *
    */
   posixRename(fromPath, toPath) {
@@ -1036,7 +1067,7 @@ class SftpClient {
    * @param {string} remotePath - path to the remote target object.
    * @param {number | string} mode - the new octal mode to set
    *
-   * @return {Promise}
+   * @return {Promise<String>}
    */
   chmod(remotePath, mode) {
     return new Promise((resolve, reject) => {
@@ -1064,8 +1095,7 @@ class SftpClient {
    * @param {String} dstDir - remote destination directory
    * @param {RegExp} filter - (Optional) a regular expression used to select
    *                         files and directories to upload
-   * @returns {String}
-   * @throws {Error}
+   * @returns {Promise<String>}
    */
   async uploadDir(srcDir, dstDir, filter = /.*/) {
     try {
@@ -1124,8 +1154,7 @@ class SftpClient {
    * @param {String} dstDir - local destination directory
    * @param {RegExp} filter - (Optional) a regular expression used to select
    *                         files and directories to upload
-   * @returns {Promise}
-   * @throws {Error}
+   * @returns {Promise<String>}
    */
   async downloadDir(srcDir, dstDir, filter = /.*/) {
     try {
@@ -1176,6 +1205,7 @@ class SftpClient {
    *
    * End the SFTP connection
    *
+   * @returns {Promise<Boolean>}
    */
   end() {
     let endCloseHandler;

package/src/utils.js

@@ -58,7 +58,13 @@ function fmtError(err, name = 'sftp', eCode, retryCount) {
   return newError;
 }
 
-let tempListeners = [];
+function addToTempListenerList(obj, name, evt, fn) {
+  if (name in obj.tempListeners) {
+    obj.tempListeners[name].push([evt, fn]);
+  } else {
+    obj.tempListeners[name] = [[evt, fn]];
+  }
+}
 
 /**
  * Simple default error listener. Will reformat the error message and
@@ -83,7 +89,7 @@ function errorListener(client, name, reject) {
       }
     }
   };
-  tempListeners.push(['error', fn]);
+  addToTempListenerList(client, name, 'error', fn);
   return fn;
 }
 
@@ -104,7 +110,7 @@ function endListener(client, name, reject) {
       }
     }
   };
-  tempListeners.push(['end', fn]);
+  addToTempListenerList(client, name, 'end', fn);
   return fn;
 }
 
@@ -125,7 +131,7 @@ function closeListener(client, name, reject) {
       }
     }
   };
-  tempListeners.push(['close', fn]);
+  addToTempListenerList(client, name, 'close', fn);
   return fn;
 }
 
@@ -138,10 +144,12 @@ function addTempListeners(obj, name, reject) {
 
 function removeTempListeners(obj, name) {
   obj.debugMsg(`${name}: Removing temp event listeners`);
-  tempListeners.forEach(([e, fn]) => {
-    obj.client.removeListener(e, fn);
-  });
-  tempListeners = [];
+  if (name in obj.tempListeners) {
+    obj.tempListeners[name].forEach(([e, fn]) => {
+      obj.client.removeListener(e, fn);
+    });
+    obj.tempListeners[name] = [];
+  }
 }
 
 /**
@@ -337,6 +345,7 @@ function sleep(ms) {
 
 module.exports = {
   fmtError,
+  addToTempListenerList,
   errorListener,
   endListener,
   closeListener,