auto-api-discovery 1.0.0 → 1.0.3

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 Anooj Shete
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 Anooj Shete
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,79 +1,52 @@
-# 🕸️ ApiGen
+# ApiGen (auto-api-discovery)
 
-> **Automated API Discovery System with HITL Authentication**
+[![NPM Version](https://img.shields.io/npm/v/auto-api-discovery)](https://www.npmjs.com/package/auto-api-discovery)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 
-![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)
-![TypeScript](https://img.shields.io/badge/TypeScript-007ACC?logo=typescript&logoColor=white)
-![Node.js](https://img.shields.io/badge/Node.js-43853D?logo=node.js&logoColor=white)
-![Playwright](https://img.shields.io/badge/Playwright-2EAD33?logo=playwright&logoColor=white)
+A CLI tool for automated API discovery. ApiGen intercepts browser network traffic (XHR, Fetch, GraphQL) to automatically build OpenAPI 3.0 specifications.
 
----
+## Installation
 
-## 🚀 What is it?
+Install the package globally via npm:
 
-Modern web applications often rely on undocumented, "hidden" internal APIs. Reverse-engineering these endpoints manually by staring at the network tab in DevTools is a tedious, error-prone, and highly time-consuming process.
-
-**ApiGen** solves this pain point by automating the discovery and documentation of these hidden APIs. It seamlessly intercepts network traffic (XHR, Fetch, GraphQL), gracefully handles complex authentication barriers via a Human-in-the-Loop (HITL) system, recursively maps internal paths automatically via headless spidering, and outputs a strict, structurally sound OpenAPI 3.0 representation.
-
----
-
-## 🏗️ Architecture
-
-ApiGen utilizes a **Hybrid API Discovery Approach** consisting of:
-
-1. **Passive Recording (Capture Mode):** Uses a headful instance of Playwright. You navigate a target web application as a real user—bypassing CAPTCHAs, 2FA, and complex login flows. Under the hood, ApiGen intercepts all network requests/responses and logs them securely into a local SQLite database. Upon closing the browser, it actively exports your authenticated session state into `.apigen-session.json`.
-2. **Auto-Spidering (Crawl Mode):** Takes the previously captured authenticated session state and launches a headless Playwright Breadth-First-Search (BFS) spider. It organically navigates the application just like an authenticated user would, discovering and hitting protected routes, intelligently avoiding generic WAF traps using randomized latency offsets, and feeding new underlying API traffic deeply into the local SQLite store.
-3. **OpenAPI Generation (Export):** A local robust schema inference engine processes thousands of SQLite records. It aggressively deduplicates dynamic URLs (folding Object IDs and UUIDs into neat path parameters), accurately infers nested JSON payload structures recursively, and compiles them to a strict, standard OpenAPI 3.0 json specification document.
-
----
-
-## ⚙️ Tech Stack
-
-- **[Node.js](https://nodejs.org/en/) & [TypeScript](https://www.typescriptlang.org/)** - For robust type-safe runtime execution.
-- **[Playwright](https://playwright.dev/)** - For complete DOM navigation, session persistence, and native underlying browser CDP network interception workflows.
-- **[Better-SQLite3](https://github.com/WiseLibs/better-sqlite3)** - Highly scalable, local WAL-mode database used as the high-throughput primary storage layer.
-- **[Commander.js](https://github.com/tj/commander.js/)** - For building the intuitive Command Line Interface.
-- **[Chalk](https://github.com/chalk/chalk)** - For beautiful and clear terminal logging feedback.
-
----
-
-## 🛠️ Getting Started
+```bash
+npm install -g auto-api-discovery
+```
 
-### Installation
+*(Note: The `postinstall` script will automatically download the Chromium binaries required by Playwright.)*
 
-Clone the repository and install dependencies:
+Alternatively, you can run it directly via `npx`:
 
 ```bash
-git clone https://github.com/anoojshete/auto-api-discovery.git
-cd auto-api-discovery
-npm install
-npx playwright install chromium
-npm run build
+npx auto-api-discovery capture https://example.com
 ```
 
-*(Note: The project leverages `npm run apigen` as the execution entrypoint hooked to `ts-node src/index.ts`)*
+## Usage
 
-### Usage
+Once installed, the `apigen` CLI becomes available.
 
-#### 1. Capture Mode
-Boot into target application interactively, bypass authentications, and capture underlying APIs. When you close the browser, your cookies are saved securely to your local working directory.
+### 1. Capture API Traffic
+Launch an interactive browser. You can manually log in, navigate the app, and bypass captchas. ApiGen intercepts the underlying API requests and saves them to a local SQLite database.
 
 ```bash
-npm run apigen capture https://example.com
+apigen capture https://example.com
 ```
 
-#### 2. Crawl Mode
-Trigger the automated, authenticated headless spider to deeply map internal features and capture the underlying APIs organically mapping to those components.
+### 2. Crawl Connected Pages (Auto-Spidering)
+Use your previously captured authenticated session to automatically run a headless crawler. It discovers and logs additional API endpoints in the background.
 
 ```bash
-# Options: -d / --depth, -p / --pages
-npm run apigen crawl https://example.com --depth 3 --pages 50
+apigen crawl https://example.com --depth 3 --pages 50
 ```
 
-#### 3. Export OpenAPI Schema
-Transform your densely populated local SQLite database into a unified, natively grouped and inferred OpenAPI 3.0 Document.
+### 3. Export OpenAPI Schema
+Convert the recorded API traffic into a unified OpenAPI 3.0 specification document. Dynamic IDs and UUIDs are automatically folded into path parameters.
 
 ```bash
-# Options: -b / --base-url
-npm run apigen export ./openapi.json --base-url https://api.example.com
+apigen export ./openapi.json --base-url https://api.example.com
 ```
+
+## How It Works
+- Uses **Playwright** to proxy network requests and manage sessions.
+- Stores metadata and traffic locally using **Better-SQLite3**.
+- Automatically infers JSON payload structures and route schemas.

package/dist/crawler.js

@@ -51,7 +51,6 @@ async function startCrawler(targetUrl, maxDepth = 2, maxPages = 50) {
     try {
         browser = await playwright_1.chromium.launch({ headless: true });
         const context = await browser.newContext();
-        // Load cookies if available to run fully authenticated
         if (fs.existsSync(SESSION_FILE)) {
             try {
                 const cookies = JSON.parse(fs.readFileSync(SESSION_FILE, 'utf-8'));
@@ -63,7 +62,6 @@ async function startCrawler(targetUrl, maxDepth = 2, maxPages = 50) {
             }
         }
         const page = await context.newPage();
-        // Attach the EXACT same interceptor from Milestone 1
         (0, interceptor_1.attachInterceptor)(page);
         let parsedTargetUrl;
         try {
@@ -84,7 +82,6 @@ async function startCrawler(targetUrl, maxDepth = 2, maxPages = 50) {
             if (!current)
                 break;
             const { url: currentUrl, depth } = current;
-            // Normalize URL (strip pure hash fragments to avoid duplicates)
             let normalizedUrl = currentUrl;
             try {
                 const pureUrl = new URL(currentUrl);
@@ -99,28 +96,24 @@ async function startCrawler(targetUrl, maxDepth = 2, maxPages = 50) {
             try {
                 await page.goto(normalizedUrl, { waitUntil: 'domcontentloaded', timeout: 15000 });
                 pagesProcessed++;
-                // Random delay to avoid aggressive WAF blocks
                 await randomDelay(500, 1500);
                 if (depth < maxDepth) {
-                    // Extract all <a> tags and map their absolute URLs
                     const links = await page.$$eval('a', anchors => anchors.map(a => a.href));
                     for (const link of links) {
                         if (!link)
                             continue;
                         try {
                             const parsedLink = new URL(link);
-                            // Filter out external links strictly based on target domain boundaries
                             if (parsedLink.hostname === domain || parsedLink.hostname.endsWith(`.${domain}`)) {
                                 parsedLink.hash = '';
                                 const nextUrl = parsedLink.toString();
-                                // Add new links to queue
                                 if (!visited.has(nextUrl)) {
+                                    visited.add(nextUrl);
                                     queue.push({ url: nextUrl, depth: depth + 1 });
                                 }
                             }
                         }
                         catch (err) {
-                            // Ignore invalid or weird internal hrefs (`javascript:`, etc)
                         }
                     }
                 }

package/dist/db.js

@@ -7,11 +7,9 @@ exports.insertEndpoint = insertEndpoint;
 exports.getAllEndpoints = getAllEndpoints;
 const better_sqlite3_1 = __importDefault(require("better-sqlite3"));
 const path_1 = __importDefault(require("path"));
-// Initialize db
 const dbPath = path_1.default.resolve(process.cwd(), 'apigen.db');
 const db = new better_sqlite3_1.default(dbPath);
 db.pragma('journal_mode = WAL');
-// Create table
 db.exec(`
   CREATE TABLE IF NOT EXISTS endpoints (
     id TEXT PRIMARY KEY,

package/dist/index.js

@@ -66,7 +66,6 @@ program
         await page.goto(url, { waitUntil: 'domcontentloaded' });
         console.log(chalk_1.default.green('Navigation complete. Intercepting API traffic...'));
         console.log(chalk_1.default.gray('Terminal output shows real-time capture. Close the browser window to exit.'));
-        // Save cookies safely in a loop to guarantee they are captured before exit
         const sessionFile = path.resolve(process.cwd(), '.apigen-session.json');
         let isRunning = true;
         browser.on('disconnected', () => {
@@ -74,14 +73,12 @@ program
             console.log(chalk_1.default.yellow('\nBrowser closed. Session saved. Exiting apigen gracefully...'));
             process.exit(0);
         });
-        // Periodically sync the cookies securely to avoid missing them if context is torn down quickly
         while (isRunning) {
             try {
                 const cookies = await context.cookies();
                 fs.writeFileSync(sessionFile, JSON.stringify(cookies, null, 2), 'utf-8');
             }
             catch (e) {
-                // Might hit target closed error silently during exit
             }
             await new Promise(resolve => setTimeout(resolve, 2000));
         }
@@ -103,7 +100,6 @@ program
         process.exit(0);
     }
     console.log(chalk_1.default.blue(`Found ${endpoints.length} raw endpoints. Generating schema map...`));
-    // Process URLs and inference schemas
     const schemaMap = (0, schema_engine_1.generateSchemaMap)(endpoints);
     const finalMap = Object.values(schemaMap);
     console.log(chalk_1.default.green(`Folded into ${finalMap.length} unique routes.`));

package/dist/interceptor.js

@@ -7,25 +7,21 @@ exports.attachInterceptor = attachInterceptor;
 const crypto_1 = require("crypto");
 const chalk_1 = __importDefault(require("chalk"));
 const db_1 = require("./db");
-const IGNORED_RESOURCE_TYPES = new Set(['image', 'stylesheet', 'font', 'media', 'script', 'document']);
 const TARGET_RESOURCE_TYPES = new Set(['xhr', 'fetch']);
 function attachInterceptor(page) {
     page.on('response', async (response) => {
         const request = response.request();
         const resourceType = request.resourceType();
-        // 1. Crucial Filter: ONLY capture traffic if it is XHR, Fetch etc.
         if (!TARGET_RESOURCE_TYPES.has(resourceType)) {
             return;
         }
         const url = request.url();
-        // Ignore tracking domains / static extensions
         if (url.includes('google-analytics.com') ||
             url.includes('googletagmanager.com') ||
             url.match(/\.(png|jpg|jpeg|gif|css|woff2?|js|ico|svg)$/i)) {
             return;
         }
         const method = request.method();
-        // Ignore preflight requests
         if (method === 'OPTIONS')
             return;
         try {
@@ -33,17 +29,15 @@ function attachInterceptor(page) {
             const headers = request.headers();
             let reqBodyParsed = null;
             let resBodyParsed = null;
-            // Parse request post body
             const postData = request.postData();
             if (postData) {
                 try {
                     reqBodyParsed = JSON.parse(postData);
                 }
                 catch {
-                    reqBodyParsed = postData; // Fallback to raw string
+                    reqBodyParsed = postData;
                 }
             }
-            // Parse response body (JSON, text)
             const contentType = response.headers()['content-type'] || '';
             if (contentType.includes('application/json') || contentType.includes('text/')) {
                 try {
@@ -63,16 +57,20 @@ function attachInterceptor(page) {
             else {
                 resBodyParsed = "[Binary or Unsupported Content]";
             }
-            // Extract basic path pattern
+            let finalUrl = url;
+            if (reqBodyParsed && reqBodyParsed.operationName && finalUrl.includes('/graphql')) {
+                const separator = finalUrl.includes('?') ? '&' : '?';
+                finalUrl = `${finalUrl}${separator}op=${reqBodyParsed.operationName}`;
+            }
             let pathPattern = '/';
             try {
-                pathPattern = new URL(url).pathname;
+                pathPattern = new URL(finalUrl).pathname;
             }
             catch { }
             const data = {
                 id: (0, crypto_1.randomUUID)(),
                 method,
-                url,
+                url: finalUrl,
                 path_pattern: pathPattern,
                 request_headers: headers,
                 request_body: reqBodyParsed,

package/dist/openapi-generator.js

@@ -26,7 +26,7 @@ function mapSchemaToOpenAPI(customSchema) {
         }
         return { type: 'object', properties };
     }
-    return {}; // fallback
+    return {};
 }
 function extractPathParams(path) {
     const matches = path.match(/\{([^}]+)\}/g);
@@ -77,7 +77,6 @@ function generateOpenAPI(customSchema, baseUrl) {
                 }
             };
         }
-        // Default response if empty
         if (Object.keys(operation.responses).length === 0) {
             operation.responses['200'] = { description: 'Success' };
         }

package/dist/schema-engine.js

@@ -44,7 +44,6 @@ function inferSchema(data) {
     if (Array.isArray(data)) {
         if (data.length === 0)
             return ['any'];
-        // Infer schema of the object inside the array
         return [inferSchema(data[0])];
     }
     if (typeof data === 'object') {
@@ -74,7 +73,7 @@ function generateSchemaMap(endpoints) {
                 try {
                     bodyData = JSON.parse(bodyData);
                 }
-                catch { } // Leave as string if parsing fails
+                catch { }
             }
             if (bodyData && typeof bodyData === 'object') {
                 const schema = inferSchema(bodyData);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "auto-api-discovery",
-  "version": "1.0.0",
+  "version": "1.0.3",
   "description": "API discovery automation CLI",
   "main": "dist/index.js",
   "bin": {
@@ -12,8 +12,9 @@
   "scripts": {
     "build": "tsc",
     "dev": "ts-node src/index.ts",
+    "test": "tsc --noEmit",
     "prepublishOnly": "npm run build",
-    "postinstall": "playwright install"
+    "postinstall": "playwright install chromium"
   },
   "keywords": [
     "api",
@@ -36,4 +37,4 @@
     "ts-node": "^10.9.2",
     "typescript": "^5.3.3"
   }
-}
+}