wanas-zcrm-extractor 1.0.0

package/README.md

@@ -0,0 +1,140 @@
+# Zoho CRM V8 Metadata Extractor
+
+A robust, production-quality local web application built using **Node.js** and **Express** that authenticates with Zoho CRM using OAuth2 (V8 APIs) and extracts all CRM metadata into structured JSON schemas stored locally. 
+
+This tool is designed with a clean, modular architecture, enabling future extensions for Zoho Creator and Zoho Projects metadata extraction.
+
+---
+
+## 🌟 Key Features
+
+1. **OAuth2 Connection Dashboard:** A premium, glassmorphic dark-theme user interface with a real-time system status indicator.
+2. **Server-Sent Events (SSE) Progress Terminal:** Displays live step-by-step extraction logs from the backend directly in a console window.
+3. **Consolidated Data Analysis:** Generates:
+   - Module summaries detailing field types, required fields, picklists, formulas, lookup associations, and layout counts.
+   - A unified `field_map.json` linking all module fields to their data types.
+   - A consolidated `complete_metadata.json` merging all API responses into a single large object.
+4. **Interactive Module Explorer:** Browse, search, filter, and inspect extracted modules, fields, related lists, and raw summaries through an interactive modal viewer.
+5. **Consolidated ZIP Downloads:** Automatically packages all exported data in a single zip archive using native compression.
+6. **Robust Error Handling:** 
+   - Automatic token refreshes on expiry.
+   - Retry mechanism for API rate limits (HTTP 429) using exponential backoff.
+   - Detailed failure logging in `storage/logs/errors.log`.
+
+---
+
+## 📂 Project Structure
+
+```text
+project/
+├── public/                 # Glassmorphic Frontend UI Assets
+│   ├── index.html          # Dashboard HTML
+│   ├── style.css           # Premium Dark-Theme Styles
+│   └── app.js              # SSE Event Listeners & Interactive Explorer logic
+│
+├── src/
+│   ├── controllers/        # Handles Route Requests (OAuth, Extraction, Downloads)
+│   │   ├── authController.js
+│   │   └── metadataController.js
+│   │
+│   ├── middleware/         # Security & Validation Middlewares
+│   │   └── authMiddleware.js
+│   │
+│   ├── routes/             # API Endpoints Routing
+│   │   ├── authRoutes.js
+│   │   └── metadataRoutes.js
+│   │
+│   ├── services/           # Business Logic Layer
+│   │   ├── authService.js  # Token Storage, Validation, and Refresh Flow
+│   │   └── metadata/       # Platform-Specific Metadata Extractors
+│   │       ├── baseMetadataService.js     # Abstract Base Extractor
+│   │       ├── crmMetadataService.js      # CRM V8 Extraction Implementation
+│   │       ├── creatorMetadataService.js  # Creator Extractor Stub (Future use)
+│   │       └── projectsMetadataService.js # Projects Extractor Stub (Future use)
+│   │
+│   └── utils/              # Common Utilities
+│       ├── apiClient.js    # Axios wrapper with retry, rate-limiting & refresh
+│       └── logger.js       # Writes errors/logs to storage/logs/errors.log
+│
+├── storage/                # Local Filesystem Output
+│   ├── auth/
+│   │   └── tokens.json     # Saved OAuth2 Access and Refresh Tokens
+│   ├── logs/
+│   │   └── errors.log      # API and Process Error Logs
+│   └── metadata/           # Extracted JSON Metadata Files
+│       ├── index.json      # Master Index file
+│       ├── modules.json    # List of Modules
+│       ├── export/         # Export Files (field_map, complete_metadata, modules)
+│       └── modules/        # Module details, layouts, custom_views, related_lists
+│
+├── .env                    # Local Environment Configuration variables
+├── package.json            # Node.js dependencies
+└── server.js               # Application Express Server Entrypoint
+```
+
+---
+
+## 🛠️ Setup & Installation
+
+### 1. Prerequisite
+Ensure you have [Node.js](https://nodejs.org/) (v18+ LTS recommended) installed.
+
+### 2. Install Dependencies
+Run the following command inside the root folder:
+```bash
+npm install
+```
+
+### 3. Environment Configuration
+Create a `.env` file in the root directory and configure your Zoho client credentials:
+```env
+PORT=3000
+ZOHO_CLIENT_ID=your_client_id_here
+ZOHO_CLIENT_SECRET=your_client_secret_here
+ZOHO_REDIRECT_URI=http://localhost:3000/zoho/callback
+ZOHO_DC=com
+```
+*Note: Make sure your Client ID and Client Secret match the data center domain configured in `ZOHO_DC` (e.g. `com`, `eu`, `in`, `com.au`).*
+
+---
+
+## 🚀 Running the Application
+
+### Development Mode
+Runs the server using `nodemon` to automatically restart on code changes:
+```bash
+npm run dev
+```
+
+### Production Mode
+Runs the server normally:
+```bash
+npm start
+```
+
+Once running, navigate to the local dashboard in your browser:
+**[http://localhost:3000](http://localhost:3000)**
+
+---
+
+## 🔄 How the Authentication & Extraction Works
+
+1. **Authentication:**
+   - Click **Login To Zoho** on the dashboard. You will be redirected to Zoho's secure OAuth2 consent screen.
+   - Upon granting permissions, Zoho redirects back to `http://localhost:3000/zoho/callback`.
+   - The server exchanges the authorization code for `access_token` and `refresh_token`, storing them securely in `storage/auth/tokens.json`.
+
+2. **Metadata Extraction:**
+   - Click **Extract Metadata** to fire the extraction task.
+   - The UI connects to `/extract?stream=true` using standard Server-Sent Events, opening a live logging terminal showing the crawler step-by-step.
+   - The service walks through all modules, queries fields, layouts, related lists, custom views, webhooks, and custom functions.
+   - Summary files are written sequentially on the disk.
+   - The master index and consolidated export files are compiled once the extraction completes.
+
+---
+
+## 🔧 Extensibility (Zoho Creator & Zoho Projects)
+
+To extend the extractor to Zoho Creator or Zoho Projects, simply:
+1. Implement the extraction pipeline inside `src/services/metadata/creatorMetadataService.js` or `projectsMetadataService.js` which already inherit from the abstract `BaseMetadataService` template.
+2. Update the target selector inside `public/index.html` to enable the platform option.

package/banner.txt

@@ -0,0 +1,7 @@
+      ___  
+      \  \       __        __                         _                    
+   ___ \  \      \ \      / /_ _ _ __   __ _ ___     / \   _ __  _ __  ___ 
+  /  / /  /       \ \ /\ / / _` | '_ \ / _` / __|   / _ \ | '_ \| '_ \/ __|
+ /  / /__/         \ V  V / (_| | | | | (_| \__ \  / ___ \| |_) | |_) \__ \
+ \  \               \_/\_/ \__,_|_| |_|\__,_|___/ /_/   \_\ .__/| .__/|___/
+  \__\                                                    |_|   |_|        

package/bin/cli.js

@@ -0,0 +1,428 @@
+#!/usr/bin/env node
+
+// Enable CLI Mode configuration before loading services
+process.env.ZCRM_CLI_MODE = 'true';
+
+const { program } = require('commander');
+const inquirer = require('inquirer').default || require('inquirer');
+const http = require('http');
+const url = require('url');
+const path = require('path');
+const { exec } = require('child_process');
+const configStore = require('../src/utils/configStore');
+const authService = require('../src/services/authService');
+const crmMetadataService = require('../src/services/metadata/crmMetadataService');
+
+// Default client credentials for generating Zoho login links
+const DEFAULT_CLIENT_ID = '1000.M1XE1M7CA4VFDDIRANB8E5AKIBQ72U';
+const DEFAULT_CLIENT_SECRET = 'baae847487d4d1fc762214bad5aeb272dd02f6f991';
+
+// Load and color the ASCII banner
+const getBanner = () => {
+  try {
+    const fs = require('fs');
+    const bannerPath = path.join(__dirname, '../banner.txt');
+    if (fs.existsSync(bannerPath)) {
+      const banner = fs.readFileSync(bannerPath, 'utf8');
+      return `\x1b[34m${banner}\x1b[0m\n`;
+    }
+  } catch (err) {
+    // Fail silently
+  }
+  return '';
+};
+
+/**
+ * Draws a beautiful text progress bar in the console.
+ * @param {number} current - Current completed index
+ * @param {number} total - Total items to complete
+ * @param {string} label - The phase label (e.g., 'Modules', 'Functions')
+ * @param {string} action - The currently executing specific item
+ */
+function drawProgress(current, total, label, action) {
+  const readline = require('readline');
+  const width = 30;
+  const percent = total > 0 ? Math.min(100, Math.max(0, Math.round((current / total) * 100))) : 0;
+  const filled = Math.round((percent / 100) * width);
+  const empty = width - filled;
+  
+  const bar = '█'.repeat(filled) + '░'.repeat(empty);
+  const truncatedAction = action.length > 40 ? action.substring(0, 37) + '...' : action;
+  
+  // Clear the current line and move cursor to start
+  readline.clearLine(process.stdout, 0);
+  readline.cursorTo(process.stdout, 0);
+  
+  // Print progress bar, percentage, and label in beautiful colors
+  process.stdout.write(`\x1b[34m[${bar}]\x1b[0m \x1b[1;32m${percent}%\x1b[0m | \x1b[1m${label}\x1b[0m: ${truncatedAction}`);
+}
+
+// Version and metadata configuration
+program
+  .name('zcrm')
+  .description('Zoho CRM V8 Metadata Extractor CLI Utility')
+  .version('1.0.0')
+  .addHelpText('before', getBanner());
+
+/**
+ * CLI Callback Server validation & token retrieval flow.
+ * Spins up a temporary server to process Zoho redirect callback.
+ * 
+ * @param {string} clientId - Zoho Client ID.
+ * @param {string} clientSecret - Zoho Client Secret.
+ * @param {string} redirectUri - Zoho Redirect URI (configured in developer console).
+ * @param {string} dc - Zoho DC domain.
+ * @returns {Promise<void>}
+ */
+function runOAuthServer(clientId, clientSecret, redirectUri, dc) {
+  return new Promise((resolve, reject) => {
+    let parsedUrl;
+    try {
+      parsedUrl = new URL(redirectUri);
+    } catch (e) {
+      return reject(new Error(`Invalid redirect URI structure: ${redirectUri}`));
+    }
+
+    const port = parsedUrl.port || 80;
+
+    const server = http.createServer(async (req, res) => {
+      const parsedReqUrl = url.parse(req.url, true);
+
+      // Handle the redirect callback path
+      if (parsedReqUrl.pathname === parsedUrl.pathname) {
+        const code = parsedReqUrl.query.code;
+        const error = parsedReqUrl.query.error;
+
+        if (error) {
+          res.writeHead(400, { 'Content-Type': 'text/html' });
+          res.end(`
+            <html>
+              <body style="font-family: Arial, sans-serif; text-align: center; padding-top: 50px; background-color: #0b0f19; color: #f3f4f6;">
+                <h1 style="color: #ef4444;">Authorization Failed</h1>
+                <p>${error}</p>
+              </body>
+            </html>
+          `);
+          server.close();
+          reject(new Error(`Authorization failed: ${error}`));
+          return;
+        }
+
+        if (code) {
+          res.writeHead(200, { 'Content-Type': 'text/html' });
+          res.end(`
+            <html>
+              <body style="font-family: Arial, sans-serif; text-align: center; padding-top: 50px; background-color: #0b0f19; color: #f3f4f6;">
+                <h1 style="color: #10b981;">Authentication Successful!</h1>
+                <p>You may now close this browser window and return to your terminal.</p>
+              </body>
+            </html>
+          `);
+
+          try {
+            console.log('\nCode received. Fetching tokens from Zoho accounts server...');
+            
+            // Save client credentials to configStore so handleCallback has them
+            configStore.save({
+              client_id: clientId,
+              client_secret: clientSecret,
+              redirect_uri: redirectUri,
+              dc: dc
+            });
+
+            await authService.handleCallback(code);
+            server.close();
+            resolve();
+          } catch (exchangeErr) {
+            server.close();
+            reject(exchangeErr);
+          }
+        } else {
+          res.writeHead(400, { 'Content-Type': 'text/plain' });
+          res.end('Missing code parameter.');
+        }
+      } else {
+        res.writeHead(404);
+        res.end();
+      }
+    });
+
+    server.on('error', (err) => {
+      if (err.code === 'EADDRINUSE') {
+        reject(new Error(`Port ${port} is currently in use. Please stop other servers or choose another port.`));
+      } else {
+        reject(err);
+      }
+    });
+
+    server.listen(port, () => {
+      console.log(`Temporary listener started on port ${port}...`);
+
+      // Initialize config details for authorizer URL generation
+      configStore.save({
+        client_id: clientId,
+        client_secret: clientSecret,
+        redirect_uri: redirectUri,
+        dc: dc
+      });
+
+      const authUrl = authService.getAuthorizationUrl();
+
+      console.log('\n========================================================================');
+      console.log('Please complete the authentication process in your browser.');
+      console.log('If the browser does not open automatically, visit this URL:');
+      console.log(authUrl);
+      console.log('========================================================================\n');
+
+      // Attempt to auto-open in default browser
+      const launch = process.platform === 'darwin' ? 'open' : process.platform === 'win32' ? 'start' : 'xdg-open';
+      exec(`${launch} "${authUrl.replace(/"/g, '\\"')}"`, (execErr) => {
+        // Suppress errors (user can manually click printed link if this fails)
+      });
+    });
+  });
+}
+
+// REGISTER COMMAND: login
+program
+  .command('login')
+  .description('Login to Zoho CRM using OAuth2 credentials')
+  .option('--client-id <id>', 'Zoho Client ID')
+  .option('--client-secret <secret>', 'Zoho Client Secret')
+  .option('--dc <dc>', 'Zoho Data Center (com, eu, in, jp, com.au, com.cn)')
+  .option('--redirect-uri <uri>', 'OAuth Redirect Callback URI')
+  .option('--refresh-token <token>', 'Headless Refresh Token bypass')
+  .action(async (options) => {
+    try {
+      let clientId = options.clientId || DEFAULT_CLIENT_ID;
+      let clientSecret = options.clientSecret || DEFAULT_CLIENT_SECRET;
+      let dc = options.dc;
+      let redirectUri = options.redirectUri;
+      const refreshToken = options.refreshToken;
+
+      // Run interactive prompts if parameters are missing
+      const prompts = [];
+      if (!dc) {
+        prompts.push({
+          type: 'select',
+          name: 'dc',
+          message: 'Select Zoho Data Center (DC):',
+          choices: ['com', 'eu', 'in', 'jp', 'com.au', 'com.cn'],
+          default: 'com'
+        });
+      }
+      if (!redirectUri && !refreshToken) {
+        prompts.push({
+          type: 'input',
+          name: 'redirectUri',
+          message: 'Enter your Redirect URI:',
+          default: 'http://localhost:3000/zoho/callback'
+        });
+      }
+
+      if (prompts.length > 0) {
+        const answers = await inquirer.prompt(prompts);
+        dc = dc || answers.dc;
+        redirectUri = redirectUri || answers.redirectUri;
+      }
+
+      // 1. Headless flow using direct refresh token
+      if (refreshToken) {
+        console.log('Bypassing browser callback. Verifying refresh token credentials...');
+        
+        configStore.save({
+          client_id: clientId,
+          client_secret: clientSecret,
+          dc: dc || 'com',
+          redirect_uri: redirectUri || 'http://localhost:3000/zoho/callback',
+          tokens: {
+            refresh_token: refreshToken
+          }
+        });
+
+        // Initialize and refresh to verify key validity
+        authService.loadTokens();
+        await authService.refreshAccessToken();
+
+        console.log('🎉 Credentials and session authenticated successfully via Refresh Token!');
+        process.exit(0);
+      }
+
+      // 2. Interactive Browser OAuth2 flow
+      await runOAuthServer(clientId, clientSecret, redirectUri, dc);
+      console.log('🎉 Successfully logged in! Access configuration saved.');
+      process.exit(0);
+    } catch (err) {
+      console.error('\n❌ Login error:', err.message);
+      process.exit(1);
+    }
+  });
+
+// REGISTER COMMAND: logout
+program
+  .command('logout')
+  .description('Logout of Zoho CRM and clear stored configurations')
+  .action(() => {
+    configStore.clear();
+    console.log('🎉 Stored credentials cleared successfully.');
+    process.exit(0);
+  });
+
+// REGISTER COMMAND: pull
+program
+  .command('pull')
+  .description('Pull Zoho CRM metadata and save as JSON file directory structure')
+  .option('-o, --output <dir>', 'Output folder path', './metadata')
+  .action(async (options) => {
+    console.log(getBanner());
+    // Reload tokens
+    authService.loadTokens();
+
+    if (!authService.isAuthenticated()) {
+      console.error('\n❌ Error: Not authenticated. Please login first using:\n   zcrm login');
+      process.exit(1);
+    }
+
+    const outputDir = path.resolve(options.output);
+    console.log(`\x1b[36mInitializing CRM Metadata Pulling...\x1b[0m`);
+    console.log(`Destination Folder: ${outputDir}\n`);
+
+    let currentModule = 0;
+    let totalModules = 0;
+    let currentFunction = 0;
+    let totalFunctions = 0;
+    let inModulesPhase = false;
+    let inFunctionsPhase = false;
+
+    try {
+      await crmMetadataService.extract((stage, message, details = {}) => {
+        switch (stage) {
+          case 'FETCH_ORG':
+            console.log(`\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_ORG_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'FETCH_MODULES':
+            console.log(`\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_MODULES_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'MODULE_START':
+            if (!inModulesPhase) {
+              console.log(`\n\x1b[1mExtracting Modules Metadata:\x1b[0m`);
+              inModulesPhase = true;
+            }
+            currentModule = details.index;
+            totalModules = details.total;
+            drawProgress(currentModule - 1, totalModules, 'Modules', `Extracting ${details.module}...`);
+            break;
+          case 'MODULE_SUCCESS':
+            if (currentModule === totalModules) {
+              const readline = require('readline');
+              readline.clearLine(process.stdout, 0);
+              readline.cursorTo(process.stdout, 0);
+              console.log(`\x1b[32m✔\x1b[0m All modules metadata extracted successfully.`);
+              inModulesPhase = false;
+            } else {
+              drawProgress(currentModule, totalModules, 'Modules', `Completed ${details.module || ''}`);
+            }
+            break;
+          case 'FETCH_WEBHOOKS':
+            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_WEBHOOKS_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'FETCH_PROFILES':
+            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_PROFILES_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'FETCH_ROLES':
+            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_ROLES_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'FETCH_FUNCTIONS':
+            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
+            break;
+          case 'FETCH_FUNCTIONS_SUCCESS':
+            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+            break;
+          case 'FUNCTION_START':
+            if (!inFunctionsPhase) {
+              console.log(`\n\x1b[1mExtracting Custom Functions Deluge Scripts:\x1b[0m`);
+              inFunctionsPhase = true;
+            }
+            currentFunction = details.index;
+            totalFunctions = details.total;
+            drawProgress(currentFunction - 1, totalFunctions, 'Functions', `Downloading ${details.name}...`);
+            break;
+          case 'FUNCTION_SUCCESS':
+          case 'FUNCTION_WARNING':
+            if (currentFunction === totalFunctions) {
+              const readline = require('readline');
+              readline.clearLine(process.stdout, 0);
+              readline.cursorTo(process.stdout, 0);
+              console.log(`\x1b[32m✔\x1b[0m All custom functions deluge scripts extracted.`);
+              inFunctionsPhase = false;
+            } else {
+              drawProgress(currentFunction, totalFunctions, 'Functions', `Completed ${details.name || ''}`);
+            }
+            break;
+          case 'FAILURE':
+            if (inModulesPhase || inFunctionsPhase) {
+              process.stdout.write('\n');
+            }
+            console.error(`\n❌ Error: ${message}`);
+            break;
+          case 'COMPLETE':
+            console.log('\n\x1b[1;32m====================================================================\x1b[0m');
+            console.log('🎉 \x1b[1;32mZoho CRM Metadata Pull Completed Successfully!\x1b[0m');
+            console.log('\x1b[1;32m====================================================================\x1b[0m');
+            console.log(` \x1b[1mOrganization:\x1b[0m      \x1b[36m${details.crm_org || 'Unknown Org'}\x1b[0m`);
+            console.log(` \x1b[1mModules Processed:\x1b[0m \x1b[32m${details.modulesCount || 0}\x1b[0m`);
+            console.log(` \x1b[1mTotal Fields:\x1b[0m      \x1b[32m${details.totalFields || 0}\x1b[0m`);
+            console.log(` \x1b[1mTotal Layouts:\x1b[0m     \x1b[32m${details.totalLayouts || 0}\x1b[0m`);
+            console.log(` \x1b[1mRelated Lists:\x1b[0m     \x1b[35m${details.totalRelatedLists || 0}\x1b[0m (saved to extra/)`);
+            console.log(` \x1b[1mCustom Views:\x1b[0m      \x1b[35m${details.totalCustomViews || 0}\x1b[0m (saved to extra/)`);
+            console.log(` \x1b[1mOutput Directory:\x1b[0m  \x1b[34m${outputDir}\x1b[0m`);
+            
+            if (details.errors && details.errors.length > 0) {
+              console.log(`\n \x1b[1;33m⚠️  Warnings:\x1b[0m \x1b[33m${details.errors.length} API request(s) failed or were skipped due to permissions.\x1b[0m`);
+            }
+            console.log('\x1b[1;32m====================================================================\x1b[0m\n');
+            break;
+          default:
+            if (message && !message.startsWith('  ->')) {
+              console.log(`\x1b[34mℹ\x1b[0m ${message}`);
+            }
+            break;
+        }
+      }, outputDir);
+
+      process.exit(0);
+    } catch (error) {
+      console.error(`\n❌ Metadata pull failed: ${error.message}`);
+      process.exit(1);
+    }
+  });
+
+// REGISTER COMMAND: dashboard
+program
+  .command('dashboard')
+  .description('Start the local Zoho CRM Extractor web dashboard')
+  .option('-p, --port <port>', 'Port to run the dashboard on', '3000')
+  .action((options) => {
+    process.env.PORT = options.port;
+    process.env.ZCRM_CLI_MODE = 'false';
+    console.log(`\x1b[36mLaunching Extractor Dashboard...\x1b[0m`);
+    require('../server.js');
+  });
+
+// Run parser
+program.parse(process.argv);

package/package.json

@@ -0,0 +1,45 @@
+{
+  "name": "wanas-zcrm-extractor",
+  "version": "1.0.0",
+  "description": "Local Zoho CRM V8 Metadata Extractor CLI and Web Dashboard Utility",
+  "main": "server.js",
+  "bin": {
+    "zcrm": "bin/cli.js"
+  },
+  "scripts": {
+    "start": "node server.js",
+    "dev": "nodemon server.js"
+  },
+  "files": [
+    "bin",
+    "src",
+    "public",
+    "server.js",
+    "banner.txt",
+    "package.json",
+    "README.md"
+  ],
+  "keywords": [
+    "zoho",
+    "crm",
+    "metadata",
+    "extractor",
+    "oauth2",
+    "cli",
+    "deluge",
+    "zdk"
+  ],
+  "author": "Wanas Apps",
+  "license": "ISC",
+  "dependencies": {
+    "axios": "^1.7.2",
+    "commander": "^15.0.0",
+    "dotenv": "^16.4.5",
+    "express": "^4.19.2",
+    "fs-extra": "^11.2.0",
+    "inquirer": "^14.0.2"
+  },
+  "devDependencies": {
+    "nodemon": "^3.1.3"
+  }
+}