@nodesecure/js-x-ray 6.3.0 → 7.1.0

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2021 NodeSecure
+Copyright (c) 2021-2024 NodeSecure
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -68,22 +68,19 @@ require(Buffer.from("6673", "hex").toString());
 Then use `js-x-ray` to run an analysis of the JavaScript code:
 ```js
 import { runASTAnalysis } from "@nodesecure/js-x-ray";
-import { readFileSync } from "fs";
+import { readFileSync } from "node:fs";
 
-const str = readFileSync("./file.js", "utf-8");
-const { warnings, dependencies } = runASTAnalysis(str);
+const { warnings, dependencies } = runASTAnalysis(
+  readFileSync("./file.js", "utf-8")
+);
 
-const dependenciesName = [...dependencies];
-const inTryDeps = [...dependencies.getDependenciesInTryStatement()];
-
-console.log(dependenciesName);
-console.log(inTryDeps);
-console.log(warnings);
+console.log(dependencies);
+console.dir(warnings, { depth: null });
 ```
 
 The analysis will return: `http` (in try), `crypto`, `util` and `fs`.
 
-> [!NOTE]
+> [!TIP]
 > There is also a lot of suspicious code example in the `./examples` cases directory. Feel free to try the tool on these files.
 
 ## Warnings
@@ -137,16 +134,90 @@ This section describe all the possible warnings returned by JSXRay. Click on the
 | [weak-crypto](./docs/weak-crypto.md) | ✔️ | The code probably contains a weak crypto algorithm (md5, sha1...) |
 | [shady-link](./docs/shady-link.md) | ✔️ | The code contains shady/unsafe link |
 
-## API
+## Custom Probes
+
+You can also create custom probes to detect specific pattern in the code you are analyzing.
+
+A probe is a pair of two functions (`validateNode` and `main`) that will be called on each node of the AST. It will return a warning if the pattern is detected.
+Below a basic probe that detect a string assignation to `danger`:
+
+```ts
+export const customProbes = [
+  {
+    name: "customProbeUnsafeDanger",
+    validateNode: (node, sourceFile) => [
+      node.type === "VariableDeclaration" && node.declarations[0].init.value === "danger"
+    ],
+    main: (node, options) => {
+      const { sourceFile, data: calleeName } = options;
+      if (node.declarations[0].init.value === "danger") {
+        sourceFile.addWarning("unsafe-danger", calleeName, node.loc);
+
+        return ProbeSignals.Skip;
+      }
+
+      return null;
+    }
+  }
+];
+```
+
+You can pass an array of probes to the `runASTAnalysis/runASTAnalysisOnFile` functions as `options`, or directly to the `AstAnalyser` constructor.
+
+| Name             | Type                             | Description                                                           | Default Value   |
+|------------------|----------------------------------|-----------------------------------------------------------------------|-----------------|
+| `customParser`   | `SourceParser \| undefined`      | An optional custom parser to be used for parsing the source code.    | `JsSourceParser` |
+| `customProbes`   | `Probe[] \| undefined`           | An array of custom probes to be used during AST analysis.            | `[]`            |
+| `skipDefaultProbes` | `boolean \| undefined`        | If `true`, default probes will be skipped and only custom probes will be used. | `false`         |
+
 
+Here using the example probe upper:
+
+```ts
+import { runASTAnalysis } from "@nodesecure/js-x-ray";
+
+// add your customProbes here (see example above)
+
+const result = runASTAnalysis("const danger = 'danger';", { customProbes, skipDefaultProbes: true });
+
+console.log(result);
+```
+
+Result:
+
+```sh
+✗ node example.js
+{
+  idsLengthAvg: 0,
+  stringScore: 0,
+  warnings: [ { kind: 'unsafe-danger', location: [Array], source: 'JS-X-Ray' } ],
+  dependencies: Map(0) {},
+  isOneLineRequire: false
+}
+```
+
+Congrats, you have created your first custom probe! 🎉
+
+> [!TIP]
+> Check the types in [index.d.ts](index.d.ts) and [types/api.d.ts](types/api.d.ts) for more details about the `options`
+
+## API
 <details>
-<summary>runASTAnalysis(str: string, options?: RuntimeOptions): Report</summary>
+<summary>runASTAnalysis(str: string, options?: RuntimeOptions & AstAnalyserOptions): Report</summary>
 
 ```ts
 interface RuntimeOptions {
   module?: boolean;
-  isMinified?: boolean;
   removeHTMLComments?: boolean;
+  isMinified?: boolean;
+}
+```
+
+```ts
+interface AstAnalyserOptions {
+  customParser?: SourceParser;
+  customProbes?: Probe[];
+  skipDefaultProbes?: boolean;
 }
 ```
 
@@ -165,13 +236,21 @@ interface Report {
 </details>
 
 <details>
-<summary>runASTAnalysisOnFile(pathToFile: string, options?: RuntimeFileOptions): Promise< ReportOnFile ></summary>
+<summary>runASTAnalysisOnFile(pathToFile: string, options?: RuntimeFileOptions & AstAnalyserOptions): Promise< ReportOnFile ></summary>
 
 ```ts
-interface RuntimeOptions {
+interface RuntimeFileOptions {
   module?: boolean;
-  isMinified?: boolean;
   removeHTMLComments?: boolean;
+  packageName?: string;
+}
+```
+
+```ts
+interface AstAnalyserOptions {
+  customParser?: SourceParser;
+  customProbes?: Probe[];
+  skipDefaultProbes?: boolean;
 }
 ```
 
@@ -197,8 +276,9 @@ Click on one of the links to access the documentation of the workspace:
 
 | name | package and link |
 | --- | --- |
-| estree-ast-util | [@nodesecure/estree-ast-util](./workspaces/estree-ast-util) |
+| estree-ast-utils | [@nodesecure/estree-ast-utils](./workspaces/estree-ast-utils) |
 | sec-literal | [@nodesecure/sec-literal ](./workspaces/sec-literal) |
+| ts-source-parser | [@nodesecure/ts-source-parser ](./workspaces/ts-source-parser) |
 
 These packages are available in the Node Package Repository and can be easily installed with [npm](https://docs.npmjs.com/getting-started/what-is-npm) or [yarn](https://yarnpkg.com).
 ```bash
@@ -210,7 +290,7 @@ $ yarn add @nodesecure/estree-ast-util
 ## Contributors ✨
 
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-11-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-17-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
@@ -234,6 +314,14 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://maji.kiwi"><img src="https://avatars.githubusercontent.com/u/33150916?v=4?s=100" width="100px;" alt="Maji"/><br /><sub><b>Maji</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=M4gie" title="Code">💻</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/targos"><img src="https://avatars.githubusercontent.com/u/2352663?v=4?s=100" width="100px;" alt="Michaël Zasso"/><br /><sub><b>Michaël Zasso</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=targos" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/issues?q=author%3Atargos" title="Bug reports">🐛</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/fabnguess"><img src="https://avatars.githubusercontent.com/u/72697416?v=4?s=100" width="100px;" alt="Kouadio Fabrice Nguessan"/><br /><sub><b>Kouadio Fabrice Nguessan</b></sub></a><br /><a href="#maintenance-fabnguess" title="Maintenance">🚧</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=fabnguess" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/jean-michelet"><img src="https://avatars.githubusercontent.com/u/110341611?v=4?s=100" width="100px;" alt="Jean"/><br /><sub><b>Jean</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Tests">⚠️</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Documentation">📖</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/tchapacan"><img src="https://avatars.githubusercontent.com/u/28821702?v=4?s=100" width="100px;" alt="tchapacan"/><br /><sub><b>tchapacan</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=tchapacan" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=tchapacan" title="Tests">⚠️</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="http://miikkak.dev"><img src="https://avatars.githubusercontent.com/u/65869801?v=4?s=100" width="100px;" alt="mkarkkainen"/><br /><sub><b>mkarkkainen</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=mkarkkainen" title="Code">💻</a></td>
+    </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/FredGuiou"><img src="https://avatars.githubusercontent.com/u/99122562?v=4?s=100" width="100px;" alt="FredGuiou"/><br /><sub><b>FredGuiou</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=FredGuiou" title="Documentation">📖</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=FredGuiou" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/madina0801"><img src="https://avatars.githubusercontent.com/u/101329759?v=4?s=100" width="100px;" alt="Madina"/><br /><sub><b>Madina</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=madina0801" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/sairuss7"><img src="https://avatars.githubusercontent.com/u/87803528?v=4?s=100" width="100px;" alt="SairussDev"/><br /><sub><b>SairussDev</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=sairuss7" title="Code">💻</a></td>
     </tr>
   </tbody>
 </table>

package/index.d.ts

@@ -1,10 +1,19 @@
 import {
+  AstAnalyser,
+  AstAnalyserOptions,
+
+  EntryFilesAnalyser,
+  EntryFilesAnalyserOptions,
+
+  SourceParser,
   runASTAnalysis,
   runASTAnalysisOnFile,
   Report,
   ReportOnFile,
   RuntimeFileOptions,
-  RuntimeOptions
+  RuntimeOptions,
+  SourceLocation,
+  Dependency
 } from "./types/api.js";
 import {
   Warning,
@@ -13,19 +22,23 @@ import {
   WarningName,
   WarningNameWithValue
 } from "./types/warnings.js";
-import { ASTDeps, Dependency } from "./types/astdeps.js";
 
 declare const warnings: Record<WarningName, Pick<WarningDefault, "experimental" | "i18n" | "severity">>;
 
 export {
   warnings,
+  AstAnalyser,
+  AstAnalyserOptions,
+  EntryFilesAnalyser,
+  EntryFilesAnalyserOptions,
+  SourceParser,
   runASTAnalysis,
   runASTAnalysisOnFile,
   Report,
   ReportOnFile,
   RuntimeFileOptions,
   RuntimeOptions,
-  ASTDeps,
+  SourceLocation,
   Dependency,
   Warning,
   WarningDefault,

package/index.js

@@ -1,148 +1,51 @@
-// Import Node.js Dependencies
-import fs from "fs/promises";
-import path from "path";
-
-// Import Third-party Dependencies
-import { walk } from "estree-walker";
-import * as meriyah from "meriyah";
-import isMinified from "is-minified-code";
-
 // Import Internal Dependencies
-import Analysis from "./src/Analysis.js";
 import { warnings } from "./src/warnings.js";
-import * as utils from "./src/utils.js";
-
-// CONSTANTS
-const kMeriyahDefaultOptions = {
-  next: true,
-  loc: true,
-  raw: true,
-  jsx: true
-};
+import { JsSourceParser } from "./src/JsSourceParser.js";
+import { AstAnalyser } from "./src/AstAnalyser.js";
 
-export function runASTAnalysis(str, options = Object.create(null)) {
+function runASTAnalysis(
+  str,
+  options = Object.create(null)
+) {
   const {
-    module = true,
-    isMinified = false,
-    removeHTMLComments = false
+    customParser = new JsSourceParser(),
+    customProbes = [],
+    skipDefaultProbes = false,
+    ...opts
   } = options;
 
-  // Note: if the file start with a shebang then we remove it because 'parseScript' may fail to parse it.
-  // Example: #!/usr/bin/env node
-  const strToAnalyze = str.charAt(0) === "#" ? str.slice(str.indexOf("\n")) : str;
-  const body = parseScriptExtended(strToAnalyze, {
-    isEcmaScriptModule: Boolean(module),
-    removeHTMLComments
+  const analyser = new AstAnalyser({
+    customParser,
+    customProbes,
+    skipDefaultProbes
   });
 
-  const sastAnalysis = new Analysis();
-  sastAnalysis.analyzeSourceString(str);
-
-  // we walk each AST Nodes, this is a purely synchronous I/O
-  walk(body, {
-    enter(node) {
-      // Skip the root of the AST.
-      if (Array.isArray(node)) {
-        return;
-      }
-
-      const action = sastAnalysis.walk(node);
-      if (action === "skip") {
-        this.skip();
-      }
-    }
-  });
-
-  const dependencies = sastAnalysis.dependencies;
-  const { idsLengthAvg, stringScore, warnings } = sastAnalysis.getResult(isMinified);
-  const isOneLineRequire = body.length <= 1 && dependencies.size <= 1;
-
-  return {
-    dependencies, warnings, idsLengthAvg, stringScore, isOneLineRequire
-  };
-}
-
-export async function runASTAnalysisOnFile(pathToFile, options = {}) {
-  try {
-    const {
-      packageName = null,
-      module = true,
-      removeHTMLComments = false
-    } = options;
-
-    const str = await fs.readFile(pathToFile, "utf-8");
-    const filePathString = pathToFile instanceof URL ? pathToFile.href : pathToFile;
-
-    const isMin = filePathString.includes(".min") || isMinified(str);
-    const data = runASTAnalysis(str, {
-      isMinified: isMin,
-      module: path.extname(filePathString) === ".mjs" ? true : module,
-      removeHTMLComments
-    });
-    if (packageName !== null) {
-      data.dependencies.removeByName(packageName);
-    }
-
-    return {
-      ok: true,
-      dependencies: data.dependencies,
-      warnings: data.warnings,
-      isMinified: !data.isOneLineRequire && isMin
-    };
-  }
-  catch (error) {
-    return {
-      ok: false,
-      warnings: [
-        { kind: "parsing-error", value: error.message, location: [[0, 0], [0, 0]] }
-      ]
-    };
-  }
+  return analyser.analyse(str, opts);
 }
 
-function parseScriptExtended(strToAnalyze, options = {}) {
-  const { isEcmaScriptModule, removeHTMLComments } = options;
-
-  /**
-   * @see https://github.com/NodeSecure/js-x-ray/issues/109
-   */
-  const cleanedStrToAnalyze = removeHTMLComments ?
-    utils.removeHTMLComment(strToAnalyze) : strToAnalyze;
-
-  try {
-    const { body } = meriyah.parseScript(
-      cleanedStrToAnalyze,
-      {
-        ...kMeriyahDefaultOptions,
-        module: isEcmaScriptModule,
-        globalReturn: !isEcmaScriptModule
-      }
-    );
-
-    return body;
-  }
-  catch (error) {
-    const isIllegalReturn = error.description.includes("Illegal return statement");
-
-    if (error.name === "SyntaxError" && (
-      error.description.includes("The import keyword") ||
-      error.description.includes("The export keyword") ||
-      isIllegalReturn
-    )) {
-      const { body } = meriyah.parseScript(
-        cleanedStrToAnalyze,
-        {
-          ...kMeriyahDefaultOptions,
-          module: true,
-          globalReturn: isIllegalReturn
-        }
-      );
+async function runASTAnalysisOnFile(
+  pathToFile,
+  options = {}
+) {
+  const {
+    customParser = new JsSourceParser(),
+    customProbes = [],
+    skipDefaultProbes = false,
+    ...opts
+  } = options;
 
-      return body;
-    }
+  const analyser = new AstAnalyser({
+    customParser,
+    customProbes,
+    skipDefaultProbes
+  });
 
-    throw error;
-  }
+  return analyser.analyseFile(pathToFile, opts);
 }
 
-export { warnings };
+export {
+  warnings,
+  AstAnalyser,
+  runASTAnalysis,
+  runASTAnalysisOnFile
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nodesecure/js-x-ray",
-  "version": "6.3.0",
+  "version": "7.1.0",
   "description": "JavaScript AST XRay analysis",
   "type": "module",
   "exports": "./index.js",
@@ -20,7 +20,8 @@
   },
   "workspaces": [
     "workspaces/estree-ast-utils",
-    "workspaces/sec-literal"
+    "workspaces/sec-literal",
+    "workspaces/ts-source-parser"
   ],
   "keywords": [
     "ast",
@@ -46,21 +47,20 @@
     "@nodesecure/estree-ast-utils": "^1.3.1",
     "@nodesecure/sec-literal": "^1.2.0",
     "estree-walker": "^3.0.1",
+    "frequency-set": "^1.0.2",
     "is-minified-code": "^2.0.0",
     "meriyah": "^4.3.3",
-    "safe-regex": "^2.1.1"
+    "safe-regex": "^2.1.1",
+    "ts-pattern": "^5.0.6"
   },
   "devDependencies": {
     "@nodesecure/eslint-config": "^1.6.0",
-    "@small-tech/esm-tape-runner": "^2.0.0",
-    "@small-tech/tap-monkey": "^1.4.0",
     "@types/node": "^20.6.2",
-    "c8": "^8.0.1",
+    "c8": "^9.0.0",
     "cross-env": "^7.0.3",
-    "eslint": "^8.31.0",
+    "eslint": "^9.0.0",
     "glob": "^10.3.4",
     "iterator-matcher": "^2.1.0",
-    "pkg-ok": "^3.0.0",
-    "tape": "^5.7.2"
+    "pkg-ok": "^3.0.0"
   }
 }

package/src/AstAnalyser.js

@@ -0,0 +1,137 @@
+// Import Node.js Dependencies
+import fs from "node:fs/promises";
+import path from "node:path";
+
+// Import Third-party Dependencies
+import { walk } from "estree-walker";
+import isMinified from "is-minified-code";
+
+// Import Internal Dependencies
+import { SourceFile } from "./SourceFile.js";
+import { isOneLineExpressionExport } from "./utils/index.js";
+import { JsSourceParser } from "./JsSourceParser.js";
+
+export class AstAnalyser {
+  /**
+   * @constructor
+   * @param {object} [options={}]
+   * @param {SourceParser} [options.customParser]
+   * @param {Array<object>} [options.customProbes]
+   * @param {boolean} [options.skipDefaultProbes=false]
+   */
+  constructor(options = {}) {
+    this.parser = options.customParser ?? new JsSourceParser();
+    this.probesOptions = {
+      customProbes: options.customProbes ?? [],
+      skipDefaultProbes: options.skipDefaultProbes ?? false
+    };
+  }
+
+  analyse(str, options = Object.create(null)) {
+    const {
+      isMinified = false,
+      module = true,
+      removeHTMLComments = false
+    } = options;
+
+    const body = this.parser.parse(this.prepareSource(str, { removeHTMLComments }), {
+      isEcmaScriptModule: Boolean(module)
+    });
+
+    const source = new SourceFile(str, this.probesOptions);
+
+    // we walk each AST Nodes, this is a purely synchronous I/O
+    walk(body, {
+      enter(node) {
+        // Skip the root of the AST.
+        if (Array.isArray(node)) {
+          return;
+        }
+
+        const action = source.walk(node);
+        if (action === "skip") {
+          this.skip();
+        }
+      }
+    });
+
+    return {
+      ...source.getResult(isMinified),
+      dependencies: source.dependencies,
+      isOneLineRequire: isOneLineExpressionExport(body)
+    };
+  }
+
+  async analyseFile(
+    pathToFile,
+    options = {}
+  ) {
+    try {
+      const {
+        packageName = null,
+        module = true,
+        removeHTMLComments = false
+      } = options;
+
+      const str = await fs.readFile(pathToFile, "utf-8");
+      const filePathString = pathToFile instanceof URL ? pathToFile.href : pathToFile;
+
+      const isMin = filePathString.includes(".min") || isMinified(str);
+      const data = this.analyse(str, {
+        isMinified: isMin,
+        module: path.extname(filePathString) === ".mjs" ? true : module,
+        removeHTMLComments
+      });
+
+      if (packageName !== null) {
+        data.dependencies.delete(packageName);
+      }
+
+      return {
+        ok: true,
+        dependencies: data.dependencies,
+        warnings: data.warnings,
+        isMinified: !data.isOneLineRequire && isMin
+      };
+    }
+    catch (error) {
+      return {
+        ok: false,
+        warnings: [
+          { kind: "parsing-error", value: error.message, location: [[0, 0], [0, 0]] }
+        ]
+      };
+    }
+  }
+
+  /**
+   * @param {!string} source
+   * @param {object} options
+   * @param {boolean} [options.removeHTMLComments=false]
+   */
+  prepareSource(source, options = {}) {
+    if (typeof source !== "string") {
+      throw new TypeError("source must be a string");
+    }
+    const { removeHTMLComments = false } = options;
+
+    /**
+     * if the file start with a shebang then we remove it because meriyah.parseScript fail to parse it.
+     * @example
+     * #!/usr/bin/env node
+     */
+    const rawNoShebang = source.startsWith("#") ?
+      source.slice(source.indexOf("\n") + 1) : source;
+
+    return removeHTMLComments ?
+      this.#removeHTMLComment(rawNoShebang) : rawNoShebang;
+  }
+
+  /**
+   * @param {!string} str
+   * @returns {string}
+   */
+  #removeHTMLComment(str) {
+    return str.replaceAll(/<!--[\s\S]*?(?:-->)/g, "");
+  }
+}