wanas-zcrm-extractor 1.0.0 → 1.0.2

package/README.md

@@ -1,140 +1,186 @@
-# Zoho CRM V8 Metadata Extractor
-
-A robust, production-quality local web application built using **Node.js** and **Express** that authenticates with Zoho CRM using OAuth2 (V8 APIs) and extracts all CRM metadata into structured JSON schemas stored locally. 
-
-This tool is designed with a clean, modular architecture, enabling future extensions for Zoho Creator and Zoho Projects metadata extraction.
-
----
-
-## 🌟 Key Features
-
-1. **OAuth2 Connection Dashboard:** A premium, glassmorphic dark-theme user interface with a real-time system status indicator.
-2. **Server-Sent Events (SSE) Progress Terminal:** Displays live step-by-step extraction logs from the backend directly in a console window.
-3. **Consolidated Data Analysis:** Generates:
-   - Module summaries detailing field types, required fields, picklists, formulas, lookup associations, and layout counts.
-   - A unified `field_map.json` linking all module fields to their data types.
-   - A consolidated `complete_metadata.json` merging all API responses into a single large object.
-4. **Interactive Module Explorer:** Browse, search, filter, and inspect extracted modules, fields, related lists, and raw summaries through an interactive modal viewer.
-5. **Consolidated ZIP Downloads:** Automatically packages all exported data in a single zip archive using native compression.
-6. **Robust Error Handling:** 
-   - Automatic token refreshes on expiry.
-   - Retry mechanism for API rate limits (HTTP 429) using exponential backoff.
-   - Detailed failure logging in `storage/logs/errors.log`.
-
----
-
-## 📂 Project Structure
-
-```text
-project/
-├── public/                 # Glassmorphic Frontend UI Assets
-│   ├── index.html          # Dashboard HTML
-│   ├── style.css           # Premium Dark-Theme Styles
-│   └── app.js              # SSE Event Listeners & Interactive Explorer logic
-│
-├── src/
-│   ├── controllers/        # Handles Route Requests (OAuth, Extraction, Downloads)
-│   │   ├── authController.js
-│   │   └── metadataController.js
-│   │
-│   ├── middleware/         # Security & Validation Middlewares
-│   │   └── authMiddleware.js
-│   │
-│   ├── routes/             # API Endpoints Routing
-│   │   ├── authRoutes.js
-│   │   └── metadataRoutes.js
-│   │
-│   ├── services/           # Business Logic Layer
-│   │   ├── authService.js  # Token Storage, Validation, and Refresh Flow
-│   │   └── metadata/       # Platform-Specific Metadata Extractors
-│   │       ├── baseMetadataService.js     # Abstract Base Extractor
-│   │       ├── crmMetadataService.js      # CRM V8 Extraction Implementation
-│   │       ├── creatorMetadataService.js  # Creator Extractor Stub (Future use)
-│   │       └── projectsMetadataService.js # Projects Extractor Stub (Future use)
-│   │
-│   └── utils/              # Common Utilities
-│       ├── apiClient.js    # Axios wrapper with retry, rate-limiting & refresh
-│       └── logger.js       # Writes errors/logs to storage/logs/errors.log
-│
-├── storage/                # Local Filesystem Output
-│   ├── auth/
-│   │   └── tokens.json     # Saved OAuth2 Access and Refresh Tokens
-│   ├── logs/
-│   │   └── errors.log      # API and Process Error Logs
-│   └── metadata/           # Extracted JSON Metadata Files
-│       ├── index.json      # Master Index file
-│       ├── modules.json    # List of Modules
-│       ├── export/         # Export Files (field_map, complete_metadata, modules)
-│       └── modules/        # Module details, layouts, custom_views, related_lists
-│
-├── .env                    # Local Environment Configuration variables
-├── package.json            # Node.js dependencies
-└── server.js               # Application Express Server Entrypoint
-```
-
----
-
-## 🛠️ Setup & Installation
-
-### 1. Prerequisite
-Ensure you have [Node.js](https://nodejs.org/) (v18+ LTS recommended) installed.
-
-### 2. Install Dependencies
-Run the following command inside the root folder:
-```bash
-npm install
-```
-
-### 3. Environment Configuration
-Create a `.env` file in the root directory and configure your Zoho client credentials:
-```env
-PORT=3000
-ZOHO_CLIENT_ID=your_client_id_here
-ZOHO_CLIENT_SECRET=your_client_secret_here
-ZOHO_REDIRECT_URI=http://localhost:3000/zoho/callback
-ZOHO_DC=com
-```
-*Note: Make sure your Client ID and Client Secret match the data center domain configured in `ZOHO_DC` (e.g. `com`, `eu`, `in`, `com.au`).*
-
----
-
-## 🚀 Running the Application
-
-### Development Mode
-Runs the server using `nodemon` to automatically restart on code changes:
-```bash
-npm run dev
-```
-
-### Production Mode
-Runs the server normally:
-```bash
-npm start
-```
-
-Once running, navigate to the local dashboard in your browser:
-**[http://localhost:3000](http://localhost:3000)**
-
----
-
-## 🔄 How the Authentication & Extraction Works
-
-1. **Authentication:**
-   - Click **Login To Zoho** on the dashboard. You will be redirected to Zoho's secure OAuth2 consent screen.
-   - Upon granting permissions, Zoho redirects back to `http://localhost:3000/zoho/callback`.
-   - The server exchanges the authorization code for `access_token` and `refresh_token`, storing them securely in `storage/auth/tokens.json`.
-
-2. **Metadata Extraction:**
-   - Click **Extract Metadata** to fire the extraction task.
-   - The UI connects to `/extract?stream=true` using standard Server-Sent Events, opening a live logging terminal showing the crawler step-by-step.
-   - The service walks through all modules, queries fields, layouts, related lists, custom views, webhooks, and custom functions.
-   - Summary files are written sequentially on the disk.
-   - The master index and consolidated export files are compiled once the extraction completes.
-
----
-
-## 🔧 Extensibility (Zoho Creator & Zoho Projects)
-
-To extend the extractor to Zoho Creator or Zoho Projects, simply:
-1. Implement the extraction pipeline inside `src/services/metadata/creatorMetadataService.js` or `projectsMetadataService.js` which already inherit from the abstract `BaseMetadataService` template.
-2. Update the target selector inside `public/index.html` to enable the platform option.
+# Zoho CRM V8 Metadata Extractor (`zcrm`)
+
+A robust, production-quality CLI utility and local web dashboard designed to authenticate with Zoho CRM (OAuth2 V8 APIs) and extract all metadata, settings, and Deluge scripts into a structured, developer-friendly local directory structure.
+
+---
+
+## 🌟 Key Features
+
+1. **Global CLI Wrapper (`zcrm`):** Command-line tools for authentication, data extraction, and starting the local web explorer dashboard.
+2. **Standard ZDK CLI Directory Layout:** Writes module structures, layouts, fields, profiles, and roles in a format fully compatible with the standard Zoho ZDK CLI tool.
+3. **Deluge Script Downloader:** Pulls custom function source codes, organizes them into folders by namespace, and saves them as local Deluge `.ds` files.
+4. **Interactive Dashboard:** Launch a premium, glassmorphic dark-theme dashboard to review extracted modules, view field properties, search and filter APIs, and download structured ZIP archives of the exports.
+5. **Robust Pipeline & Error Handling:** 
+   - Interactive progress indicators showing modules and deluge functions as they download.
+   - Resilient Axios client with auto-refreshing OAuth tokens.
+   - Exponential backoff retry system to handle API rate limits (HTTP 429).
+   - Rich console results screen showing execution statistics.
+
+---
+
+## 🛠️ Installation
+
+Install the CLI utility globally via npm:
+
+```bash
+npm install -g wanas-zcrm-extractor
+```
+
+*(Ensure you have [Node.js](https://nodejs.org/) v18+ installed on your system).*
+
+---
+
+## 🔑 Setup & Registration
+
+To authenticate, you must register a Zoho API Client in the Zoho Developer Console:
+
+1. Visit the [Zoho Developer Console](https://api-console.zoho.com/).
+2. Click **Add Client** and select **Server-based Applications**.
+3. Configure your application details:
+   - **Client Name:** `Zoho CRM Metadata Extractor`
+   - **Homepage URL:** `http://localhost:3000`
+   - **Authorized Redirect URIs:** `http://localhost:3000/zoho/callback` (or your preferred local port redirect URI)
+4. Click **Create** to receive your **Client ID** and **Client Secret**.
+
+---
+
+## 🚀 CLI Commands Reference
+
+### 1. `zcrm login`
+Authenticates the CLI to access your Zoho CRM account.
+
+```bash
+zcrm login [options]
+```
+
+**Options:**
+* `--client-id <id>` - Zoho OAuth Client ID.
+* `--client-secret <secret>` - Zoho OAuth Client Secret.
+* `--dc <dc>` - Zoho Data Center domain (`com`, `eu`, `in`, `jp`, `com.au`, `com.cn`).
+* `--redirect-uri <uri>` - Zoho OAuth Redirect URI.
+* `--refresh-token <token>` - Headless authentication option (bypasses browser authorization).
+
+**Examples:**
+* *Interactive Mode (Prompts for missing details):*
+  ```bash
+  zcrm login
+  ```
+* *Express Login with Custom Credentials:*
+  ```bash
+  zcrm login --client-id 1000.xxxxxxx --client-secret xxxxxxx --dc com --redirect-uri http://localhost:3000/zoho/callback
+  ```
+* *Headless Login (For CI/CD or Server Scripts):*
+  ```bash
+  zcrm login --client-id 1000.xxxxxxx --client-secret xxxxxxx --dc com --refresh-token 1000.xxxxxxxxxxxxxxxxx
+  ```
+
+---
+
+### 2. `zcrm pull`
+Crawls all Zoho CRM endpoints to extract metadata schemas and custom functions deluge scripts.
+
+```bash
+zcrm pull [options]
+```
+
+**Options:**
+* `-o, --output <dir>` - Destination folder path (defaults to `./metadata` in the current working directory).
+
+**What happens during a pull:**
+1. Verifies authentication and refreshes active OAuth access tokens automatically.
+2. Fetches Zoho Org information and sets up local project mapping configs.
+3. Crawls every module to pull field configurations and page layout schemas (with a beautiful progress indicator).
+4. Fetches Profiles, Roles, and Webhooks settings.
+5. Downloads all custom deluge function scripts.
+6. Compiles a consolidated database of schemas and summaries.
+7. Prints a rich-text CLI banner and statistics summary showing the number of modules, layouts, and fields processed.
+
+---
+
+### 3. `zcrm dashboard`
+Launches the premium local web dashboard to explore the pulled metadata in a browser.
+
+```bash
+zcrm dashboard [options]
+```
+
+**Options:**
+* `-p, --port <port>` - Port to run the dashboard on (defaults to `3000`).
+
+Once launched, navigate to **`http://localhost:3000`** in your browser. From here, you can:
+- View live extraction logs.
+- Browse extracted modules, inspect field tables, view picklist values, and lookup relationships.
+- Search and filter across field names.
+- Download a consolidated `.zip` file of all extracted files.
+
+---
+
+### 4. `zcrm logout`
+Clears stored configuration credentials, OAuth tokens, and active sessions from the workspace.
+
+```bash
+zcrm logout
+```
+
+---
+
+## 📂 Generated Directory Structure
+
+When running `zcrm pull -o ./metadata`, the following layout is compiled inside the output folder:
+
+```text
+metadata/
+├── .zdk/                       # Internal ZDK CLI stores
+│   ├── .logs/                  # Extractor logging files and audit history
+│   ├── .org/
+│   │   └── org.json            # Extracted Zoho CRM Organization metadata
+│   └── .store/                 # Extracted cached index formats
+│       ├── index.json          # Master index of extracted modules
+│       ├── modules_raw.json    # Raw API modules catalog response
+│       ├── field_map.json      # Key-value map of every field's data type
+│       ├── complete_metadata.json # Unified JSON database of all endpoints
+│       ├── webhooks.json       # Extracted webhooks metadata
+│       └── functions.json      # Extracted functions catalog
+│
+├── crm/                        # Standard Zoho ZDK CLI metadata layout
+│   └── meta/
+│       ├── modules/            # Individual Module schemas
+│       │   └── <Module_API_Name>/
+│       │       ├── <Module_API_Name>.modules-meta.json
+│       │       ├── fields/     # Field configurations
+│       │       │   └── <Field_API_Name>.fields-meta.json
+│       │       └── layouts/    # Page Layout configurations
+│       │           └── <Layout_API_Name>.layouts-meta.json
+│       ├── profiles/           # Profile Permissions mapping
+│       │   └── <Profile_Name>.profiles-meta.json
+│       └── roles/              # Role Hierarchy definitions
+│           └── <Role_Name>.roles-meta.json
+│
+├── functions/                  # Custom Deluge function files (.ds)
+│   ├── standalone/             # Deluge scripts sorted by namespace
+│   │   └── standalone.<Function_API_Name>.ds
+│   ├── automation/
+│   ├── button/
+│   ├── related_list/
+│   ├── schedule/
+│   └── salessignals/
+│
+├── extra/                      # Custom analysis exports
+│   └── modules/
+│       └── <Module_API_Name>/
+│           ├── summary.json    # Custom module summary (counts, fields, picklists, formulas)
+│           ├── custom_views/   # Custom View configuration JSONs
+│           │   └── <View_API_Name>.custom_views-meta.json
+│           └── related_lists/  # Related List configurations
+│               └── <List_API_Name>.related_lists-meta.json
+│
+├── zdk-project.json            # ZDK CLI project settings config
+├── meta.json                   # ZDK resource configuration
+└── .zdkignore                  # Standard gitignore rules for ZDK caches
+```
+
+---
+
+## 🤝 Contribution & License
+
+This project is maintained by Wanas Apps and licensed under the **ISC License**. For queries or feedback, feel free to contact us.

package/bin/cli.js

@@ -57,11 +57,13 @@ function drawProgress(current, total, label, action) {
   process.stdout.write(`\x1b[34m[${bar}]\x1b[0m \x1b[1;32m${percent}%\x1b[0m | \x1b[1m${label}\x1b[0m: ${truncatedAction}`);
 }
 
+const pkg = require('../package.json');
+
 // Version and metadata configuration
 program
   .name('zcrm')
   .description('Zoho CRM V8 Metadata Extractor CLI Utility')
-  .version('1.0.0')
+  .version(pkg.version, '-v, --version', 'output the current version')
   .addHelpText('before', getBanner());
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "wanas-zcrm-extractor",
-  "version": "1.0.0",
+  "version": "1.0.2",
   "description": "Local Zoho CRM V8 Metadata Extractor CLI and Web Dashboard Utility",
   "main": "server.js",
   "bin": {