@nodesecure/js-x-ray 6.2.1 → 7.0.0

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2021 NodeSecure
+Copyright (c) 2021-2024 NodeSecure
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -20,12 +20,10 @@
     </a>
 </p>
 
-JavaScript AST analysis. This package has been created to export the [Node-Secure](https://github.com/ES-Community/nsecure) AST Analysis to enable better code evolution and allow better access to developers and researchers.
+JavaScript AST analysis. This package has been created to export the [NodeSecure](https://github.com/NodeSecure/cli) AST Analysis to enable better code evolution and allow better access to developers and researchers.
 
 The goal is to quickly identify dangerous code and patterns for developers and Security researchers. Interpreting the results of this tool will still require you to have a set of security notions.
 
-> **Note** I have no particular background in security. I'm simply becoming more and more interested and passionate about static code analysis. But I would be more than happy to learn that my work can help prevent potential future attacks (or leaks).
-
 ## Goals
 The objective of the project is to successfully detect all potentially suspicious JavaScript codes.. The target is obviously codes that are added or injected for malicious purposes..
 
@@ -70,22 +68,20 @@ require(Buffer.from("6673", "hex").toString());
 Then use `js-x-ray` to run an analysis of the JavaScript code:
 ```js
 import { runASTAnalysis } from "@nodesecure/js-x-ray";
-import { readFileSync } from "fs";
-
-const str = readFileSync("./file.js", "utf-8");
-const { warnings, dependencies } = runASTAnalysis(str);
+import { readFileSync } from "node:fs";
 
-const dependenciesName = [...dependencies];
-const inTryDeps = [...dependencies.getDependenciesInTryStatement()];
+const { warnings, dependencies } = runASTAnalysis(
+  readFileSync("./file.js", "utf-8")
+);
 
-console.log(dependenciesName);
-console.log(inTryDeps);
-console.log(warnings);
+console.log(dependencies);
+console.dir(warnings, { depth: null });
 ```
 
 The analysis will return: `http` (in try), `crypto`, `util` and `fs`.
 
-> **Warning** There is also a lot of suspicious code example in the `./examples` cases directory. Feel free to try the tool on these files.
+> [!TIP]
+> There is also a lot of suspicious code example in the `./examples` cases directory. Feel free to try the tool on these files.
 
 ## Warnings
 
@@ -122,8 +118,6 @@ console.log(i18n.getTokenSync(jsxray.warnings["parsing-error"].i18n));
 
 ## Warnings Legends
 
-> **Warning** versions of NodeSecure greather than v0.7.0 are no longer compatible with the warnings table below.
-
 This section describe all the possible warnings returned by JSXRay. Click on the warning **name** for additional information and examples.
 
 | name | experimental | description |
@@ -140,16 +134,90 @@ This section describe all the possible warnings returned by JSXRay. Click on the
 | [weak-crypto](./docs/weak-crypto.md) | ✔️ | The code probably contains a weak crypto algorithm (md5, sha1...) |
 | [shady-link](./docs/shady-link.md) | ✔️ | The code contains shady/unsafe link |
 
-## API
+## Custom Probes
+
+You can also create custom probes to detect specific pattern in the code you are analyzing.
+
+A probe is a pair of two functions (`validateNode` and `main`) that will be called on each node of the AST. It will return a warning if the pattern is detected.
+Below a basic probe that detect a string assignation to `danger`:
+
+```ts
+export const customProbes = [
+  {
+    name: "customProbeUnsafeDanger",
+    validateNode: (node, sourceFile) => [
+      node.type === "VariableDeclaration" && node.declarations[0].init.value === "danger"
+    ],
+    main: (node, options) => {
+      const { sourceFile, data: calleeName } = options;
+      if (node.declarations[0].init.value === "danger") {
+        sourceFile.addWarning("unsafe-danger", calleeName, node.loc);
+
+        return ProbeSignals.Skip;
+      }
+
+      return null;
+    }
+  }
+];
+```
+
+You can pass an array of probes to the `runASTAnalysis/runASTAnalysisOnFile` functions as `options`, or directly to the `AstAnalyser` constructor.
+
+| Name             | Type                             | Description                                                           | Default Value   |
+|------------------|----------------------------------|-----------------------------------------------------------------------|-----------------|
+| `customParser`   | `SourceParser \| undefined`      | An optional custom parser to be used for parsing the source code.    | `JsSourceParser` |
+| `customProbes`   | `Probe[] \| undefined`           | An array of custom probes to be used during AST analysis.            | `[]`            |
+| `skipDefaultProbes` | `boolean \| undefined`        | If `true`, default probes will be skipped and only custom probes will be used. | `false`         |
+
+
+Here using the example probe upper:
+
+```ts
+import { runASTAnalysis } from "@nodesecure/js-x-ray";
+
+// add your customProbes here (see example above)
+
+const result = runASTAnalysis("const danger = 'danger';", { customProbes, skipDefaultProbes: true });
+
+console.log(result);
+```
+
+Result:
+
+```sh
+✗ node example.js
+{
+  idsLengthAvg: 0,
+  stringScore: 0,
+  warnings: [ { kind: 'unsafe-danger', location: [Array], source: 'JS-X-Ray' } ],
+  dependencies: Map(0) {},
+  isOneLineRequire: false
+}
+```
 
+Congrats, you have created your first custom probe! 🎉
+
+> [!TIP]
+> Check the types in [index.d.ts](index.d.ts) and [types/api.d.ts](types/api.d.ts) for more details about the `options`
+
+## API
 <details>
-<summary>runASTAnalysis(str: string, options?: RuntimeOptions): Report</summary>
+<summary>runASTAnalysis(str: string, options?: RuntimeOptions & AstAnalyserOptions): Report</summary>
 
 ```ts
 interface RuntimeOptions {
   module?: boolean;
-  isMinified?: boolean;
   removeHTMLComments?: boolean;
+  isMinified?: boolean;
+}
+```
+
+```ts
+interface AstAnalyserOptions {
+  customParser?: SourceParser;
+  customProbes?: Probe[];
+  skipDefaultProbes?: boolean;
 }
 ```
 
@@ -168,13 +236,21 @@ interface Report {
 </details>
 
 <details>
-<summary>runASTAnalysisOnFile(pathToFile: string, options?: RuntimeFileOptions): Promise< ReportOnFile ></summary>
+<summary>runASTAnalysisOnFile(pathToFile: string, options?: RuntimeFileOptions & AstAnalyserOptions): Promise< ReportOnFile ></summary>
 
 ```ts
-interface RuntimeOptions {
+interface RuntimeFileOptions {
   module?: boolean;
-  isMinified?: boolean;
   removeHTMLComments?: boolean;
+  packageName?: string;
+}
+```
+
+```ts
+interface AstAnalyserOptions {
+  customParser?: SourceParser;
+  customProbes?: Probe[];
+  skipDefaultProbes?: boolean;
 }
 ```
 
@@ -194,11 +270,27 @@ export type ReportOnFile = {
 
 </details>
 
+## Workspaces
+
+Click on one of the links to access the documentation of the workspace:
+
+| name | package and link |
+| --- | --- |
+| estree-ast-utils | [@nodesecure/estree-ast-utils](./workspaces/estree-ast-utils) |
+| sec-literal | [@nodesecure/sec-literal ](./workspaces/sec-literal) |
+| ts-source-parser | [@nodesecure/ts-source-parser ](./workspaces/ts-source-parser) |
+
+These packages are available in the Node Package Repository and can be easily installed with [npm](https://docs.npmjs.com/getting-started/what-is-npm) or [yarn](https://yarnpkg.com).
+```bash
+$ npm i @nodesecure/estree-ast-util
+# or
+$ yarn add @nodesecure/estree-ast-util
+```
 
 ## Contributors ✨
 
 <!-- ALL-CONTRIBUTORS-BADGE:START - Do not remove or modify this section -->
-[![All Contributors](https://img.shields.io/badge/all_contributors-11-orange.svg?style=flat-square)](#contributors-)
+[![All Contributors](https://img.shields.io/badge/all_contributors-16-orange.svg?style=flat-square)](#contributors-)
 <!-- ALL-CONTRIBUTORS-BADGE:END -->
 
 Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):
@@ -222,6 +314,13 @@ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/d
       <td align="center" valign="top" width="14.28%"><a href="https://maji.kiwi"><img src="https://avatars.githubusercontent.com/u/33150916?v=4?s=100" width="100px;" alt="Maji"/><br /><sub><b>Maji</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=M4gie" title="Code">💻</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/targos"><img src="https://avatars.githubusercontent.com/u/2352663?v=4?s=100" width="100px;" alt="Michaël Zasso"/><br /><sub><b>Michaël Zasso</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=targos" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/issues?q=author%3Atargos" title="Bug reports">🐛</a></td>
       <td align="center" valign="top" width="14.28%"><a href="https://github.com/fabnguess"><img src="https://avatars.githubusercontent.com/u/72697416?v=4?s=100" width="100px;" alt="Kouadio Fabrice Nguessan"/><br /><sub><b>Kouadio Fabrice Nguessan</b></sub></a><br /><a href="#maintenance-fabnguess" title="Maintenance">🚧</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=fabnguess" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/jean-michelet"><img src="https://avatars.githubusercontent.com/u/110341611?v=4?s=100" width="100px;" alt="Jean"/><br /><sub><b>Jean</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Tests">⚠️</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=jean-michelet" title="Documentation">📖</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/tchapacan"><img src="https://avatars.githubusercontent.com/u/28821702?v=4?s=100" width="100px;" alt="tchapacan"/><br /><sub><b>tchapacan</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=tchapacan" title="Code">💻</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=tchapacan" title="Tests">⚠️</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="http://miikkak.dev"><img src="https://avatars.githubusercontent.com/u/65869801?v=4?s=100" width="100px;" alt="mkarkkainen"/><br /><sub><b>mkarkkainen</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=mkarkkainen" title="Code">💻</a></td>
+    </tr>
+    <tr>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/FredGuiou"><img src="https://avatars.githubusercontent.com/u/99122562?v=4?s=100" width="100px;" alt="FredGuiou"/><br /><sub><b>FredGuiou</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=FredGuiou" title="Documentation">📖</a> <a href="https://github.com/NodeSecure/js-x-ray/commits?author=FredGuiou" title="Code">💻</a></td>
+      <td align="center" valign="top" width="14.28%"><a href="https://github.com/madina0801"><img src="https://avatars.githubusercontent.com/u/101329759?v=4?s=100" width="100px;" alt="Madina"/><br /><sub><b>Madina</b></sub></a><br /><a href="https://github.com/NodeSecure/js-x-ray/commits?author=madina0801" title="Code">💻</a></td>
     </tr>
   </tbody>
 </table>

package/index.d.ts

@@ -1,10 +1,14 @@
 import {
+  AstAnalyser,
+  SourceParser,
   runASTAnalysis,
   runASTAnalysisOnFile,
   Report,
   ReportOnFile,
   RuntimeFileOptions,
-  RuntimeOptions
+  RuntimeOptions,
+  SourceLocation,
+  Dependency
 } from "./types/api.js";
 import {
   Warning,
@@ -13,19 +17,20 @@ import {
   WarningName,
   WarningNameWithValue
 } from "./types/warnings.js";
-import { ASTDeps, Dependency } from "./types/astdeps.js";
 
 declare const warnings: Record<WarningName, Pick<WarningDefault, "experimental" | "i18n" | "severity">>;
 
 export {
   warnings,
+  AstAnalyser,
+  SourceParser,
   runASTAnalysis,
   runASTAnalysisOnFile,
   Report,
   ReportOnFile,
   RuntimeFileOptions,
   RuntimeOptions,
-  ASTDeps,
+  SourceLocation,
   Dependency,
   Warning,
   WarningDefault,

package/index.js

@@ -1,148 +1,51 @@
-// Import Node.js Dependencies
-import fs from "fs/promises";
-import path from "path";
-
-// Import Third-party Dependencies
-import { walk } from "estree-walker";
-import * as meriyah from "meriyah";
-import isMinified from "is-minified-code";
-
 // Import Internal Dependencies
-import Analysis from "./src/Analysis.js";
 import { warnings } from "./src/warnings.js";
-import * as utils from "./src/utils.js";
-
-// CONSTANTS
-const kMeriyahDefaultOptions = {
-  next: true,
-  loc: true,
-  raw: true,
-  jsx: true
-};
+import { JsSourceParser } from "./src/JsSourceParser.js";
+import { AstAnalyser } from "./src/AstAnalyser.js";
 
-export function runASTAnalysis(str, options = Object.create(null)) {
+function runASTAnalysis(
+  str,
+  options = Object.create(null)
+) {
   const {
-    module = true,
-    isMinified = false,
-    removeHTMLComments = false
+    customParser = new JsSourceParser(),
+    customProbes = [],
+    skipDefaultProbes = false,
+    ...opts
   } = options;
 
-  // Note: if the file start with a shebang then we remove it because 'parseScript' may fail to parse it.
-  // Example: #!/usr/bin/env node
-  const strToAnalyze = str.charAt(0) === "#" ? str.slice(str.indexOf("\n")) : str;
-  const body = parseScriptExtended(strToAnalyze, {
-    isEcmaScriptModule: Boolean(module),
-    removeHTMLComments
+  const analyser = new AstAnalyser({
+    customParser,
+    customProbes,
+    skipDefaultProbes
   });
 
-  const sastAnalysis = new Analysis();
-  sastAnalysis.analyzeSourceString(str);
-
-  // we walk each AST Nodes, this is a purely synchronous I/O
-  walk(body, {
-    enter(node) {
-      // Skip the root of the AST.
-      if (Array.isArray(node)) {
-        return;
-      }
-
-      const action = sastAnalysis.walk(node);
-      if (action === "skip") {
-        this.skip();
-      }
-    }
-  });
-
-  const dependencies = sastAnalysis.dependencies;
-  const { idsLengthAvg, stringScore, warnings } = sastAnalysis.getResult(isMinified);
-  const isOneLineRequire = body.length <= 1 && dependencies.size <= 1;
-
-  return {
-    dependencies, warnings, idsLengthAvg, stringScore, isOneLineRequire
-  };
-}
-
-export async function runASTAnalysisOnFile(pathToFile, options = {}) {
-  try {
-    const {
-      packageName = null,
-      module = true,
-      removeHTMLComments = false
-    } = options;
-
-    const str = await fs.readFile(pathToFile, "utf-8");
-    const filePathString = pathToFile instanceof URL ? pathToFile.href : pathToFile;
-
-    const isMin = filePathString.includes(".min") || isMinified(str);
-    const data = runASTAnalysis(str, {
-      isMinified: isMin,
-      module: path.extname(filePathString) === ".mjs" ? true : module,
-      removeHTMLComments
-    });
-    if (packageName !== null) {
-      data.dependencies.removeByName(packageName);
-    }
-
-    return {
-      ok: true,
-      dependencies: data.dependencies,
-      warnings: data.warnings,
-      isMinified: !data.isOneLineRequire && isMin
-    };
-  }
-  catch (error) {
-    return {
-      ok: false,
-      warnings: [
-        { kind: "parsing-error", value: error.message, location: [[0, 0], [0, 0]] }
-      ]
-    };
-  }
+  return analyser.analyse(str, opts);
 }
 
-function parseScriptExtended(strToAnalyze, options = {}) {
-  const { isEcmaScriptModule, removeHTMLComments } = options;
-
-  /**
-   * @see https://github.com/NodeSecure/js-x-ray/issues/109
-   */
-  const cleanedStrToAnalyze = removeHTMLComments ?
-    utils.removeHTMLComment(strToAnalyze) : strToAnalyze;
-
-  try {
-    const { body } = meriyah.parseScript(
-      cleanedStrToAnalyze,
-      {
-        ...kMeriyahDefaultOptions,
-        module: isEcmaScriptModule,
-        globalReturn: !isEcmaScriptModule
-      }
-    );
-
-    return body;
-  }
-  catch (error) {
-    const isIllegalReturn = error.description.includes("Illegal return statement");
-
-    if (error.name === "SyntaxError" && (
-      error.description.includes("The import keyword") ||
-      error.description.includes("The export keyword") ||
-      isIllegalReturn
-    )) {
-      const { body } = meriyah.parseScript(
-        cleanedStrToAnalyze,
-        {
-          ...kMeriyahDefaultOptions,
-          module: true,
-          globalReturn: isIllegalReturn
-        }
-      );
+async function runASTAnalysisOnFile(
+  pathToFile,
+  options = {}
+) {
+  const {
+    customParser = new JsSourceParser(),
+    customProbes = [],
+    skipDefaultProbes = false,
+    ...opts
+  } = options;
 
-      return body;
-    }
+  const analyser = new AstAnalyser({
+    customParser,
+    customProbes,
+    skipDefaultProbes
+  });
 
-    throw error;
-  }
+  return analyser.analyseFile(pathToFile, opts);
 }
 
-export { warnings };
+export {
+  warnings,
+  AstAnalyser,
+  runASTAnalysis,
+  runASTAnalysisOnFile
+};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@nodesecure/js-x-ray",
-  "version": "6.2.1",
+  "version": "7.0.0",
   "description": "JavaScript AST XRay analysis",
   "type": "module",
   "exports": "./index.js",
@@ -18,6 +18,11 @@
     "type": "git",
     "url": "git+https://github.com/NodeSecure/js-x-ray.git"
   },
+  "workspaces": [
+    "workspaces/estree-ast-utils",
+    "workspaces/sec-literal",
+    "workspaces/ts-source-parser"
+  ],
   "keywords": [
     "ast",
     "nsecure",
@@ -42,16 +47,20 @@
     "@nodesecure/estree-ast-utils": "^1.3.1",
     "@nodesecure/sec-literal": "^1.2.0",
     "estree-walker": "^3.0.1",
+    "frequency-set": "^1.0.2",
     "is-minified-code": "^2.0.0",
     "meriyah": "^4.3.3",
-    "safe-regex": "^2.1.1"
+    "safe-regex": "^2.1.1",
+    "ts-pattern": "^5.0.6"
   },
   "devDependencies": {
     "@nodesecure/eslint-config": "^1.6.0",
     "@types/node": "^20.6.2",
-    "c8": "^8.0.1",
+    "c8": "^9.0.0",
+    "cross-env": "^7.0.3",
     "eslint": "^8.31.0",
     "glob": "^10.3.4",
+    "iterator-matcher": "^2.1.0",
     "pkg-ok": "^3.0.0"
   }
 }

package/src/AstAnalyser.js

@@ -0,0 +1,137 @@
+// Import Node.js Dependencies
+import fs from "node:fs/promises";
+import path from "node:path";
+
+// Import Third-party Dependencies
+import { walk } from "estree-walker";
+import isMinified from "is-minified-code";
+
+// Import Internal Dependencies
+import { SourceFile } from "./SourceFile.js";
+import { isOneLineExpressionExport } from "./utils/index.js";
+import { JsSourceParser } from "./JsSourceParser.js";
+
+export class AstAnalyser {
+  /**
+   * @constructor
+   * @param {object} [options={}]
+   * @param {SourceParser} [options.customParser]
+   * @param {Array<object>} [options.customProbes]
+   * @param {boolean} [options.skipDefaultProbes=false]
+   */
+  constructor(options = {}) {
+    this.parser = options.customParser ?? new JsSourceParser();
+    this.probesOptions = {
+      customProbes: options.customProbes ?? [],
+      skipDefaultProbes: options.skipDefaultProbes ?? false
+    };
+  }
+
+  analyse(str, options = Object.create(null)) {
+    const {
+      isMinified = false,
+      module = true,
+      removeHTMLComments = false
+    } = options;
+
+    const body = this.parser.parse(this.prepareSource(str, { removeHTMLComments }), {
+      isEcmaScriptModule: Boolean(module)
+    });
+
+    const source = new SourceFile(str, this.probesOptions);
+
+    // we walk each AST Nodes, this is a purely synchronous I/O
+    walk(body, {
+      enter(node) {
+        // Skip the root of the AST.
+        if (Array.isArray(node)) {
+          return;
+        }
+
+        const action = source.walk(node);
+        if (action === "skip") {
+          this.skip();
+        }
+      }
+    });
+
+    return {
+      ...source.getResult(isMinified),
+      dependencies: source.dependencies,
+      isOneLineRequire: isOneLineExpressionExport(body)
+    };
+  }
+
+  async analyseFile(
+    pathToFile,
+    options = {}
+  ) {
+    try {
+      const {
+        packageName = null,
+        module = true,
+        removeHTMLComments = false
+      } = options;
+
+      const str = await fs.readFile(pathToFile, "utf-8");
+      const filePathString = pathToFile instanceof URL ? pathToFile.href : pathToFile;
+
+      const isMin = filePathString.includes(".min") || isMinified(str);
+      const data = this.analyse(str, {
+        isMinified: isMin,
+        module: path.extname(filePathString) === ".mjs" ? true : module,
+        removeHTMLComments
+      });
+
+      if (packageName !== null) {
+        data.dependencies.delete(packageName);
+      }
+
+      return {
+        ok: true,
+        dependencies: data.dependencies,
+        warnings: data.warnings,
+        isMinified: !data.isOneLineRequire && isMin
+      };
+    }
+    catch (error) {
+      return {
+        ok: false,
+        warnings: [
+          { kind: "parsing-error", value: error.message, location: [[0, 0], [0, 0]] }
+        ]
+      };
+    }
+  }
+
+  /**
+   * @param {!string} source
+   * @param {object} options
+   * @param {boolean} [options.removeHTMLComments=false]
+   */
+  prepareSource(source, options = {}) {
+    if (typeof source !== "string") {
+      throw new TypeError("source must be a string");
+    }
+    const { removeHTMLComments = false } = options;
+
+    /**
+     * if the file start with a shebang then we remove it because meriyah.parseScript fail to parse it.
+     * @example
+     * #!/usr/bin/env node
+     */
+    const rawNoShebang = source.startsWith("#") ?
+      source.slice(source.indexOf("\n") + 1) : source;
+
+    return removeHTMLComments ?
+      this.#removeHTMLComment(rawNoShebang) : rawNoShebang;
+  }
+
+  /**
+   * @param {!string} str
+   * @returns {string}
+   */
+  #removeHTMLComment(str) {
+    return str.replaceAll(/<!--[\s\S]*?(?:-->)/g, "");
+  }
+}