@shriyanss/js-recon 1.0.0 → 1.1.0-beta.2

package/research/firewall_bypass.md

@@ -1,38 +0,0 @@
-# Firewall Bypass
-While the development of this tool, the tester came across several websites using firewall, majorly Cloudflare. A technique has to be developed to bypass this blockage, and to download the JS files.
-
-## Cloudflare
-### Detection
-Whenever the client is restricted by Cloudflare, it returns a HTML page with a JS challenge to be solved. However, the JS is can't be executed by Node.JS. It would need a browser environment. Sample HTML page is shown below:
-
-```html
-<meta http-equiv="refresh"
-        content="5; URL='/?bm-verify=--snip--'" />
---snip--
-<script> var i = 1749580415; var j = i + Number("7341" + "47171"); </script>
---snip--
-<script> var xhr = new XMLHttpRequest(); xhr.withCredentials = true; xhr.addEventListener("loadend", function () { try { var data = JSON.parse(xhr.responseText); if (data.hasOwnProperty('reload')) { if (data["reload"] == true) { window.location.replace(window.location.href.replace(/[&?]bm-verify=[^#]*/, "")); if (window.location.hash) { window.location.reload(); } } } else if (data.hasOwnProperty(--snip </script>
-```
-
-### Bypass
-#### #1
-The most common way to bypass this is to use a browser environment, such as a headless browser.
-
-To achive this task, a detection mechanism is added in the custom function to make HTTP requests. If CF is detected, it will load the page in a headless browser, and then return the content.
-
-#### #2
-Utilizing a browser plugin to fetch the contents is a more advanced way to bypass this. The user can install the plugin in their testing browser/proxy, and as they navigate the site, the plugin will get the contents and send then to the server for processing (it would be like a caching system).
-
-#### #3
-The `makeRequest()` can be modified to send headers that a browser would send. These include:
-- `User-Agent` of a browser
-- `Accept`
-- `Accept-Language`
-- `Sec-Fetch-Site: same-origin`
-- `Sec-Fetch-Mode: cors`
-- `Sec-Fetch-Dest: empty`
-- `Referer`
-- `Origin`
-
-#### #4
-Utilizing AWS API Gateway or similar services to rotate IP addresses can help in bypassing the firewall. This, however, could be blocked by the by CF if it is configured to block requests from AWS or similar ISPs.

package/research/next_js.md

@@ -1,116 +0,0 @@
-# Next.js Research
-## Tech Detection
-To detect Next.js, following technique(s) can be used: 
-- Iterate through all the HTML tags, and find `src`, `srcSet` or `imageSrcSet` attribute with value starting with`/_next/` in the page source
-
-## Embedded JS files
-A lot of JS files were identified from the page source by inspecting the value of `src` attribute of `script` tags. These included files like `main.js`, `polyfills.js`, `webpack.js` etc.
-
-## Lazy Loaded Files
-### Analysis of [Vercel Docs](https://vercel.com/docs)
-Upon analysis of page source and HTTP requests, it was found that the webpack filename had a pattern of `webpack-<hash>.js`. So, webpack JS files were identified and fetched.
-
-Upon inspection of the webpack JS file, it was found that code was distributed into several functions. It was observed that the function responsible for returning the JS path ended with `".js"`. This observation was seen multiple times in the Next.js apps by the developer (aka web researcher), hence it bacame a standard method to get the path of the JS files.
-
-### Analysis of [X.ai](https://x.ai)
-It was found that most of the things were done in a similar way, however, some extra chars were present after `".js"` in the function responsible for returning JS path in the webpack JS file.
-
-To handle this, the regex was modified to also include some additional characters at the end.
-```js
-.match(/"\.js".{0,15}$/)
-```
-
-Also, it was found that very less URLs were present in the `webpack.js` file (4 at the time of analysis). Upon further inspection, it was found that the URLs are present in the inline `<script>` tags returned in the page source. For example:
-```
-self.__next_f.push([1,"14:I[41498,[\"3867\",\"static/chunks/b6d67c9f-618ea7c61a79562d.js\",\"5017\",\"static/chunks/622eaf3d-350d2142e11f9e13.js\",\"1889\",\"static/chunks/0fd4459a-4e25f772815e6f19.js\",\"4406\",\"static/chunks/4406-fdbbb31c90e98725.js\",\"8627\",\"static/chunks/8627-54829c23d6b1a53f.js\",\"6520\",\"static/chunks/6520-f51ddcfa4e30ebc8.js\",\"1296\",\"static/chunks/1296-b8c2dd03773a40db.js\",\"8922\",\"static/chunks/8922-f01acc4fc5fecfad.js\",\"6929\",\"static/chunks/6929-a636a59ffdb51b17.js\",\"2601\"......snip....../chunks/b6d67c9f-"])
-```
-
-To handle this, a feature was implemented to get the JS files from the inline `<script>` tags as well.
-
-Upon researching further, it was found that additional JS files were being returned in the subsequent HTTP requests. For example, the file at `https://x.ai/_next/static/chunks/app/careers/page-9e04dce6fed05790.js` was loaded from a request to `https://x.ai/careers?_rsc=jwl50`.
-
-Request to `https://x.ai/careers?_rsc=jwl50` should be made with a special header called `RSC: 1` in order to get access to additional JS file paths. This request can be also sent to `/` with the same header to get more files.
-
-Possible solution: To handle this, the strings can be extracted from the page sources, and then requests to them can be made along with this request header. If it returns a `200 OK`, then it can be also added to the list of JS files. 
-
-### Analysis of [OpenAI](https://openai.com)
-Upon inspection of HTTP requests, a similar behavior of sending the requests with `RSC: 1` header to get additional JS files was found.
-
-However, a slight difference was found. When sending the request to `/business` with the same header, the server returned a different (`308 Permanent Redirect` to `/business/`) response. This can be handled by sending the request to `/business/` with the same header.
-
-Additionally, it was noticed that when the requests were sent without the `RSC: 1` header, the request got blocked by Cloudflare, and it resulted in a `403 Forbidden` response with a message `Just a moment...`. This could indicate a potential firewall bypass
-
-## Client-Side Paths/URLs
-Client-side paths/URLs are web addresses handled by the browser using JavaScript, usually without reloading the page. They are used for navigation, API requests, and loading resources dynamically within the client environment.
-
-### Analysis of [X.ai](https://x.ai)
-Upon inspection of the client side paths, it was found that they are present in `href` across JS chunks.
-
-These are a part of a list, which contains objects with keys like `href` (string), `label` (string), `active` (boolean) and `children` (array of objects of the same type).
-
-For instance, here's a example of such a list:
-```js
-let L = [
-    {
-      href: "/grok",
-      label: "Grok",
-      active: e.startsWith("/grok"),
-      children: [
-        { href: "/grok", label: "For Everyone", active: "/grok" == e },
-        {
-          href: "/grok/business",
-          label: "For Business",
-          active: "/grok/business" == e,
-        },
-      ],
-    },
-    {
-      href: "/api",
-      label: "API",
-      active: e.startsWith("/api"),
-      children: [
-        { href: "/api#capabilities", label: "Overview" },
-        {
-          href: "https://docs.x.ai/docs/models?cluster=us-east-1#detailed-pricing-for-all-grok-models",
-          label: "Pricing",
-          external: !0,
-        },
-        {
-          href: "https://console.x.ai",
-          label: "API Console Login",
-          external: !0,
-        },
-        {
-          href: "https://docs.x.ai",
-          label: "Documentation",
-          external: !0,
-        },
-      ],
-    },
-    { href: "/company", label: "Company", active: "/company" == e },
-    { href: "/colossus", label: "Colossus", active: "/colossus" == e },
-    {
-      href: "/careers",
-      label: "Careers",
-      active: e.startsWith("/careers"),
-    },
-    { href: "/news", label: "News", active: e.startsWith("/news") },
-]
-```
-
-Possible methodology: The tool can iterate over all the JS chunks, and find the list of objects with keys like `href` (string), `label` (string), `active` (boolean) and `children` (array of objects of the same type). Then, it can organize them in a report.
-
-### Analyis of [1Password](https://1password.com)
-It was found that the client-side paths were stored in mostly stored in a way like:
-```js
-let s = JSON.parse(
-          '["/state-of-enterprise-security-report/thank-you/",......"/webinars/1p-quarterly-security-spotlight-and-roadmap-review/thank-you/"]',
-)
-```
-
-Some similar pattern was also observed in [OpenAI](https://openai.com), however, the full analysis of OpenAI's client-side paths is not done at the time of writing this.
-
-It was also found that some paths were stored directly as a list. For example:
-```js
-let n = ["/pricing/xam", "/pricing/password-manager"];
-```

package/research/nuxt_js.md

@@ -1,125 +0,0 @@
-# Nuxt.js Research
-## Tech Detection
-Nuxt.js is a framework based on Vue.js. So, if the Vue.js is detected, then it is essential to detect Nuxt.js for accuracy.
-
-To detect Nuxt.js, following technique(s) can be used: 
-- Search for "/_nuxt" in following attributes
-    - `src`, `href`
-
-## Lazy Loaded Files
-### Analysis of [GitLab's About Site](https://about.gitlab.com/)
-It was found that all of the JS files were loaded from the `/_nuxt-new` directory instead of `/_nuxt` directory. The path to the JS files were located in the `<link>` tags' `href` attribute. These `<link>` tags had the value of `rel` attribute as `modulepreload`, and the values of `as` attribute as `script`.
-
-Additionally, there were some `<script>` tags found with `src` attribute as the path to the JS files, which started with `/_nuxt`.
-
-### Analysis of [Vue Mastery](https://www.vuemastery.com/)
-Similar to Gitlab's About Page, Vue Mastery had `<link>` tags with `rel` attribute as `preload` and `href` attribute as the path to the JS files. These `<link>` tags had the value of `as` attribute as `script`, which is common between two sites.
-
-Additionally, there were some `<script>` tags found with `src` attribute as the path to the JS files, which started with `/_nuxt`.
-
-Upon analysis of JS files that can be downloaded with the above two methods, it was found that some of the JS files contained functions to generate dynamic JS files based on the function. One of such function was found in https://www.vuemastery.com/_nuxt/2551e78.js, which was present in a `<script>` tag on the home page source. The function responsible for generating dynamic JS files is:
-```js
-f.p = "/_nuxt/",
-// --snip--
-script.src = function(e) {
-    return f.p + "" + {
-        0: "dad8928",
-        1: "45a7974",
-        2: "4632233",
-        3: "1b5c8a7",
-        4: "605bec3",
-        5: "b2d6888",
-        6: "6f126a6",
-        7: "0660b70",
-        10: "b2c4841",
-        11: "dd7a220",
-        12: "f9f7852",
-        13: "4204cef",
-        14: "6b95651",
-        15: "d23fd5a",
-        16: "674fd56",
-        17: "e43e7d0",
-        18: "6745258",
-        19: "6ea1659",
-        20: "ebdc343",
-        21: "b82338f",
-        22: "6b6976c",
-        23: "3652e0c",
-        24: "ab7b2c1",
-        25: "4d1b31e",
-        26: "4dadfea",
-        27: "e3dff7a",
-        28: "cc26714",
-        29: "5fda719",
-        30: "cae7d03",
-        31: "23f9ebd",
-        32: "8e6c9c2",
-        33: "fe28953",
-        34: "d31580e",
-        35: "da95621",
-        36: "edc6dfe",
-        37: "5a045a0",
-        38: "50dfb9c",
-        39: "6ef34b4",
-        40: "102f187",
-        41: "c1d8d4f",
-        42: "4a0b48e",
-        43: "635c550",
-        44: "0266231",
-        45: "8fd697f",
-        46: "5107115",
-        47: "608af85",
-        48: "500d25b",
-        49: "da3579b",
-        50: "06ee20b",
-        51: "43e16e2",
-        52: "4c13496",
-        53: "a1eb1a8",
-        54: "141dbb5",
-        55: "c9cbc38",
-        56: "7306da6",
-        57: "415103c",
-        58: "57bbbbd",
-        59: "1963258",
-        62: "4fca0c9",
-        63: "93260d7",
-        64: "55a6cb8",
-        65: "15afb5c"
-    }[e] + ".js"
-}(e);
-```
-
-### Analysis of [Vue.js Amsterdam](https://vuejs.amsterdam/)
-Apart from the methods mentioned above, it was found that a lot of JS paths were present in those which are already loaded. They were simple strings like `./sitemap.d7a8ffd7.js`, so they can be extracted on string analysis.
-
-## Client-Side Paths/URLs
-### Analysis of [GitLab's About Site](https://about.gitlab.com/)
-It was found that the client-side paths/URLs were present in the https://about.gitlab.com/_nuxt-new/builds/meta/2d555104-4dad-4b33-a02f-128a966a0c7b.json file. This file was found in the following tab on the home page's source code:
-```html
-<link rel="preload" as="fetch" fetchpriority="low" crossorigin="anonymous" href="/_nuxt-new/builds/meta/2d555104-4dad-4b33-a02f-128a966a0c7b.json">
-```
-
-Here's a preview of the JSON file:
-```json
-{
-    "id": "2d555104-4dad-4b33-a02f-128a966a0c7b",
-    "timestamp": 1750133987484,
-    "matcher": {
-        "static": {},
-        "wildcard": {},
-        "dynamic": {}
-    },
-    "prerendered": [
-        "/de-de/contact-sales",
-        "/de-de/get-help",
-        --snip--
-        "/contact-sales",
-        "/analysts",
-        --snip
-        "/ja-jp",
-        "/ja-jp/search",
-        "/",
-        "/search"
-    ]
-}
-```

package/research/vue_js.md

@@ -1,9 +0,0 @@
-# Vue.js Research
-## Tech Detection
-To detect Vue.js, following technique(s) can be used: 
-- Load the webpage in the browser, and find the `data-v-*` attribute
-    - Note that this attribute might NOT be present if the webpage is not loaded in the browser, i.e. by directly getting the page source
-- However, it was found that most of the Vue.JS sites were using Nuxt. The research for this can be found at [Nuxt.js Research](./nuxt_js.md)
-
-## Lazy Loaded Files
-### Analysis of [Vue.JS official site](https://vuejs.org/)

package/strings/index.js

@@ -1,145 +0,0 @@
-import chalk from "chalk";
-import fs from "fs";
-import path from "path";
-import parser from "@babel/parser";
-import _traverse from "@babel/traverse";
-import prettier from "prettier";
-
-const traverse = _traverse.default;
-
-/**
- * Extracts all string literals from all .js files in a given directory and its
- * subdirectories and writes them to a JSON file.
- * @param {string} directory - The directory to scan for .js files
- * @param {string} output_file - The file to write the extracted strings to
- */
-const strings = async (
-  directory,
-  output_file,
-  extract_urls,
-  extracted_url_path,
-) => {
-  console.log(chalk.cyan("[i] Loading 'Strings' module"));
-
-  // check if the directory exists
-  if (!fs.existsSync(directory)) {
-    console.log(chalk.red("[!] Directory does not exist"));
-    return;
-  }
-
-  console.log(chalk.cyan(`[i] Scanning ${directory} directory`));
-
-  // get all files in the directory and sub-directories
-  const files = fs.readdirSync(directory, { recursive: true });
-
-  // filter out non JS files
-  const jsFiles = files.filter((file) => file.endsWith(".js"));
-
-  // read all JS files
-  let js_files_path = [];
-  for (const file of jsFiles) {
-    const filePath = path.join(directory, file);
-    if (!fs.lstatSync(filePath).isDirectory()) {
-      js_files_path.push(filePath);
-    }
-  }
-
-  console.log(chalk.cyan(`[i] Found ${js_files_path.length} JS files`));
-
-  // read all JS files
-  let all_strings = {};
-  for (const file of js_files_path) {
-    const fileContent = fs.readFileSync(file, "utf-8");
-
-    // parse the file contents with babel
-    const ast = parser.parse(fileContent, {
-      sourceType: "unambiguous",
-      plugins: ["jsx", "typescript"],
-    });
-
-    let strings = [];
-
-    traverse(ast, {
-      StringLiteral(path) {
-        strings.push(path.node.value);
-      },
-    });
-
-    all_strings[file] = strings;
-  }
-
-  let strings_count = 0;
-  for (const file of Object.keys(all_strings)) {
-    strings_count += all_strings[file].length;
-  }
-
-  console.log(chalk.cyan(`[i] Extracted ${strings_count} strings`));
-
-  // write to a JSON file
-  const formatted = await prettier.format(JSON.stringify(all_strings), {
-    parser: "json",
-    printWidth: 80,
-    singleQuote: true,
-  });
-  fs.writeFileSync(output_file, formatted);
-
-  console.log(chalk.green(`[✓] Extracted strings to ${output_file}`));
-
-  if (extract_urls) {
-    console.log(chalk.cyan("[i] Extracting URLs and paths from strings"));
-
-    let urls = [];
-    let paths = [];
-
-    for (const file of Object.keys(all_strings)) {
-      for (const string of all_strings[file]) {
-        if (string.match(/^https?:\/\/[a-zA-Z0-9\.\-_]+\/?.*$/)) {
-          // like https://site.com
-          urls.push(string);
-        }
-        if (string.match(/^\/.+$/)) {
-          // like /path/resource
-          // make sure that the path doesn't start with two special chars except '/_'
-          if (string.match(/^\/[^a-zA-Z0-9]/) && !string.startsWith("/_")) {
-            // ignore the path
-          } else {
-            paths.push(string);
-          }
-        }
-        if (string.match(/^[a-zA-Z0-9_\-]\/[a-zA-Z0-9_\-].*$/)) {
-          // like path/to/resource
-          paths.push(string);
-        }
-        if (string.startsWith("./") || string.startsWith("../")) {
-          // like "./path/to/resource" or "../path/to/resource"
-          paths.push(string);
-        }
-      }
-    }
-
-    // dedupe the two lists
-    urls = [...new Set(urls)];
-    paths = [...new Set(paths)];
-
-    console.log(
-      chalk.cyan(`[i] Found ${urls.length} URLs and ${paths.length} paths`),
-    );
-
-    // write to a JSON file
-    const formatted_urls = await prettier.format(
-      JSON.stringify({ urls, paths }),
-      {
-        parser: "json",
-        printWidth: 80,
-        singleQuote: true,
-      },
-    );
-    fs.writeFileSync(extracted_url_path, formatted_urls);
-
-    console.log(
-      chalk.green(`[✓] Written URLs and paths to ${extracted_url_path}`),
-    );
-  }
-};
-
-export default strings;

package/techDetect/index.js

@@ -1,156 +0,0 @@
-import chalk from "chalk";
-import * as cheerio from "cheerio";
-import makeRequest from "../utility/makeReq.js";
-import puppeteer from "puppeteer";
-
-/**
- * Detects if a webpage uses Next.js by checking if any HTML tag has a src,
- * srcset, or imageSrcSet attribute that starts with "/_next/".
- * @param {CheerioStatic} $ - The Cheerio object containing the parsed HTML.
- * @returns {Promise<{detected: boolean, evidence: string}>}
- *   A promise that resolves to an object with two properties:
- *   - detected: A boolean indicating whether Next.js was detected.
- *   - evidence: A string with the evidence of the detection, or an empty string
- *     if Next.js was not detected.
- */
-const checkNextJS = async ($) => {
-  let detected = false;
-  let evidence = "";
-  // iterate through each HTML tag, and file tag value that starts with `/_next/`
-  $("*").each((_, el) => {
-    const tag = $(el).get(0).tagName;
-
-    // check the value of three attributes
-    const src = $(el).attr("src");
-    const srcSet = $(el).attr("srcset");
-    const imageSrcSet = $(el).attr("imageSrcSet");
-
-    if (src || srcSet || imageSrcSet) {
-      if (src && src.startsWith("/_next/")) {
-        detected = true;
-        evidence = `${tag} :: ${src}`;
-      } else if (srcSet && srcSet.startsWith("/_next/")) {
-        detected = true;
-        evidence = `${tag} :: ${srcSet}`;
-      } else if (imageSrcSet && imageSrcSet.startsWith("/_next/")) {
-        detected = true;
-        evidence = `${tag} :: ${imageSrcSet}`;
-      }
-    }
-  });
-
-  return { detected, evidence };
-};
-
-/**
- * Detects if a webpage uses Vue.js by checking if any HTML tag has a data-v-* attribute.
- * @param {CheerioStatic} $ - The Cheerio object containing the parsed HTML.
- * @returns {Promise<{detected: boolean, evidence: string}>}
- *   A promise that resolves to an object with two properties:
- *   - detected: A boolean indicating whether Vue.js was detected.
- *   - evidence: A string with the evidence of the detection, or an empty string
- *     if Vue.js was not detected.
- */
-const checkVueJS = async ($) => {
-  let detected = false;
-  let evidence = "";
-
-  $("*").each((_, el) => {
-    const tag = $(el).get(0).tagName;
-    const attribs = el.attribs;
-    if (attribs) {
-      for (const [attrName, attrValue] of Object.entries(attribs)) {
-        if (attrName.startsWith("data-v-")) {
-          detected = true;
-          evidence = `${tag} :: ${attrName}`;
-        }
-      }
-    }
-  });
-
-  return { detected, evidence };
-};
-
-
-const checkNuxtJS = async ($)=>{
-    let detected = false;
-    let evidence = "";
-
-    // go through the page source, and check for "/_nuxt" in the src or href attribute
-    $("*").each((_, el)=>{
-        const tag = $(el).get(0).tagName;
-        const attribs = el.attribs;
-        if (attribs) {
-            for (const [attrName, attrValue] of Object.entries(attribs)) {
-                if (attrName === "src" || attrName === "href") {
-                    if (attrValue.includes("/_nuxt")) {
-                        detected = true;
-                        evidence = `${attrName} :: ${attrValue}`;
-                    }
-                }
-            }
-        }
-    });
-
-    return { detected, evidence };
-}
-
-/**
- * Detects the front-end framework used by a webpage.
- * @param {string} url - The URL of the webpage to be detected.
- * @returns {Promise<{name: string, evidence: string}> | null}
- *   A promise that resolves to an object with two properties:
- *   - name: A string indicating the detected framework, or null if no framework was detected.
- *   - evidence: A string with the evidence of the detection, or an empty string if no framework was detected.
- */
-const frameworkDetect = async (url) => {
-  console.log(chalk.cyan("[i] Detecting front-end framework"));
-
-  // get the page source
-  const res = await makeRequest(url);
-
-  // get the page source in the browser
-  const browser = await puppeteer.launch({
-    headless: true,
-    args: [
-      "--disable-gpu",
-      "--disable-dev-shm-usage",
-      "--disable-setuid-sandbox",
-      "--no-sandbox",
-    ],
-  });
-  const page = await browser.newPage();
-  await page.goto(url);
-  await new Promise((resolve) => setTimeout(resolve, 5000));
-  const pageSource = await page.content();
-  await browser.close();
-
-  // if (res === null || res === undefined) {
-  //   return;
-  // }
-
-  // const pageSource = await res.text();
-
-  // cheerio to parse the page source
-  const $ = cheerio.load(pageSource);
-
-  // check all technologies one by one
-  const result_checkNextJS = await checkNextJS($);
-  const result_checkVueJS = await checkVueJS($);
-
-  if (result_checkNextJS.detected === true) {
-    return { name: "next", evidence: result_checkNextJS.evidence };
-  } else if (result_checkVueJS.detected === true) {
-    console.log(chalk.green("[✓] Vue.js detected"));
-    console.log(chalk.cyan(`[i] Checking Nuxt.JS`), chalk.dim("(Nuxt.JS is built on Vue.js)"));
-    const result_checkNuxtJS = await checkNuxtJS($);
-    if (result_checkNuxtJS.detected === true) {
-      return { name: "nuxt", evidence: result_checkNuxtJS.evidence };
-    }
-    return { name: "vue", evidence: result_checkVueJS.evidence };
-  }
-
-  return null;
-};
-
-export default frameworkDetect;

package/utility/globals.js

@@ -1,6 +0,0 @@
-
-export let apiGatewayConfigFile = "";
-export let useApiGateway = false;
-
-export const setApiGatewayConfigFile = (file) => { apiGatewayConfigFile = file; };
-export const setUseApiGateway = (value) => { useApiGateway = value; };