wanas-zcrm-extractor 1.0.7 → 1.2.0

package/README.md

@@ -10,16 +10,32 @@ A robust, production-quality CLI utility and local web dashboard designed to aut
 
 ---
 
+> ### ⚠️ Important: read-only — it never modifies your CRM
+>
+> `zcrm` is strictly **read-only**: it **never creates, updates, or deletes** anything in Zoho
+> CRM. By **default** it pulls **schema/metadata only** — module and field definitions, layouts,
+> picklists, related lists, tags, profiles, roles, webhooks, org-wide settings, and custom
+> function source code — with **no access to your CRM records**.
+>
+> Reading actual record data is always **opt-in and explicit**: per-module record *counts* only
+> with `zcrm pull --with-counts` (~50 API credits/module), and audit-log data only via the
+> `zcrm audit` command. Even then, the tool only ever *reads* — it cannot change anything in your CRM.
+
+---
+
 ## 🌟 Key Features
 
 1. **Global CLI Wrapper (`zcrm`):** Command-line tools for authentication, data extraction, and starting the local web explorer dashboard.
 2. **Standard ZCRM CLI Directory Layout:** Writes module structures, layouts, fields, profiles, and roles in a format fully compatible with standard Zoho CRM developer layouts.
 3. **Deluge Script Downloader:** Pulls custom function source codes, organizes them into folders by namespace, and saves them as local Deluge `.ds` files.
 4. **Interactive Dashboard:** Launch a premium, glassmorphic dark-theme dashboard to review extracted modules, view field properties, search and filter APIs, and download structured ZIP archives of the exports.
-5. **Robust Pipeline & Error Handling:** 
+5. **AI Skill Generator (`zcrm skill`):** Generates ready-to-use skill/rules files for Claude Code, Cursor, Windsurf, GitHub Copilot, Cline, Gemini, Codex/AGENTS.md, and plain Markdown — so your AI assistant can read the extracted schema and write valid Deluge using exact API names.
+6. **Robust Pipeline & Error Handling:** 
+   - Concurrent extraction with a configurable, edition-aware cap on simultaneous API calls for fast pulls.
+   - One-way sync: overwrites changed files and removes local metadata deleted on Zoho — without wiping data on a failed run.
    - Interactive progress indicators showing modules and deluge functions as they download.
    - Resilient Axios client with auto-refreshing OAuth tokens.
-   - Exponential backoff retry system to handle API rate limits (HTTP 429).
+   - Exponential backoff retry system to handle API rate/concurrency limits (HTTP 429).
    - Rich console results screen showing execution statistics.
 
 ---
@@ -101,7 +117,7 @@ zcrm login [options]
 ---
 
 ### 2. `zcrm pull`
-Crawls all Zoho CRM endpoints to extract metadata schemas and custom functions deluge scripts.
+Crawls all Zoho CRM endpoints (read-only) to extract metadata schemas and custom function Deluge scripts, and **syncs** them into your local folder.
 
 ```bash
 zcrm pull [options]
@@ -109,15 +125,36 @@ zcrm pull [options]
 
 **Options:**
 * `-o, --output <dir>` - Destination folder path (defaults to `./metadata` in the current working directory).
+* `-c, --concurrency <n>` - Maximum number of simultaneous Zoho API calls, `1`–`25` (defaults to `5`).
+* `--with-counts` - Also fetch each module's **record count**. This reads record data (not just schema) and costs **~50 API credits per module**, so it is **off by default**. Omit it to keep the pull strictly metadata-only.
 
 **What happens during a pull:**
 1. Verifies authentication and refreshes active OAuth access tokens automatically.
 2. Fetches Zoho Org information and sets up local project mapping configs.
-3. Crawls every module to pull field configurations and page layout schemas (with a beautiful progress indicator).
+3. Crawls every module to pull field configurations and page layout schemas (with a live progress indicator).
 4. Fetches Profiles, Roles, and Webhooks settings.
-5. Downloads all custom deluge function scripts.
+5. Downloads all custom Deluge function scripts.
 6. Compiles a consolidated database of schemas and summaries.
-7. Prints a rich-text CLI banner and statistics summary showing the number of modules, layouts, and fields processed.
+7. Prints a rich-text CLI statistics summary showing modules, layouts, fields, and stale files removed.
+
+#### ⚡ Concurrency (faster pulls, safely)
+A pull makes many independent metadata requests, so they are run **in parallel** with a global cap on simultaneous calls. Zoho enforces a per-org **concurrency limit** based on your edition — **5** (Free), **10** (Standard/Starter), **15** (Professional), **20** (Enterprise/Zoho One), **25** (Ultimate/CRM Plus). The metadata endpoints used here are 1 credit each and are *not* subject to Zoho's stricter sub-concurrency limits.
+
+The default of `5` is safe for **every** edition. If you are on a higher edition and want faster pulls, raise it up to your edition's limit:
+
+```bash
+zcrm pull -o ./metadata --concurrency 15
+```
+
+If the limit is ever exceeded (HTTP 429 `TOO_MANY_REQUESTS`), the client automatically backs off exponentially and retries, so a pull never fails just because it was briefly throttled. Leave headroom if other integrations share the same org.
+
+#### 🔄 Sync behavior (overwrite + remove stale)
+A pull is a **one-way sync from Zoho into your folder**:
+* Existing files are **overwritten in place** with the latest metadata.
+* Files that no longer correspond to anything on Zoho (a deleted module, field, layout, custom view, related list, profile, role, or function) are **removed locally** — but only within scopes whose authoritative fetch succeeded this run.
+* If a fetch **fails** (e.g. a transient network/permission error), the previous local data for that scope is **kept**, never deleted. A partial or interrupted pull therefore never wipes your data.
+
+The summary reports how many stale files were removed. Your own files outside the managed `crm/` and `.zcrm/` trees are never touched.
 
 ---
 
@@ -157,7 +194,60 @@ zcrm llm
 
 ---
 
-### 6. `zcrm logout`
+### 6. `zcrm skill`
+Generates an **AI assistant skill / rules file** for your IDE or AI coding tool, derived from the `zcrm llm` guide. The generated file teaches your assistant how to read the extracted metadata snapshot and produce valid Deluge / REST code using exact API names — and clearly states that the snapshot is read-only schema with no CRM record access.
+
+```bash
+zcrm skill [options]
+```
+
+**Options:**
+* `--ide <ide>` - Target tool (skips the interactive prompt). One of: `claude`, `cursor`, `windsurf`, `copilot`, `cline`, `gemini`, `codex`, `markdown`.
+* `-d, --dir <dir>` - Project directory to write into (defaults to the current directory).
+* `-m, --metadata-dir <dir>` - Path to your extracted metadata, referenced inside the skill (defaults to `./metadata`).
+* `-f, --force` - Overwrite an existing dedicated skill file without prompting.
+
+When run without `--ide`, it prompts you to choose your tool and writes the file to the right place:
+
+| Tool | Generated file |
+| :--- | :--- |
+| **Claude Code** | `.claude/skills/zoho-crm-deluge/SKILL.md` |
+| **Cursor** | `.cursor/rules/zoho-crm-deluge.mdc` |
+| **Windsurf** | `.windsurf/rules/zoho-crm-deluge.md` |
+| **GitHub Copilot** | `.github/copilot-instructions.md` |
+| **Cline / Roo Code** | `.clinerules/zoho-crm-deluge.md` |
+| **Gemini CLI** | `GEMINI.md` |
+| **Codex / generic** | `AGENTS.md` |
+| **Plain Markdown** | `ZOHO_CRM_AI_CONTEXT.md` |
+
+> For shared files (`AGENTS.md`, `GEMINI.md`, `.github/copilot-instructions.md`), the command inserts a clearly-delimited `ZCRM` block and re-generates it in place on subsequent runs, leaving the rest of your file untouched.
+
+**Examples:**
+```bash
+# Interactive — pick your IDE from a menu
+zcrm skill
+
+# Non-interactive — generate a Claude Code skill referencing ./crm-meta
+zcrm skill --ide claude --metadata-dir ./crm-meta
+```
+
+---
+
+### `zcrm audit`
+Triggers an asynchronous export of the Zoho CRM audit log. By default, it exports all audit logs for the last 3 years. It automatically polls the API until the job is complete and downloads the resulting CSV or ZIP file to `storage/audit_logs/`.
+
+```bash
+# Export all audit logs
+zcrm audit
+
+# Export with specific criteria using a JSON file
+zcrm audit --filter input.json
+```
+*(See Zoho CRM documentation for the format of the filter JSON).*
+
+---
+
+### 7. `zcrm logout`
 Clears stored configuration credentials, OAuth tokens, and active sessions from the workspace.
 
 ```bash
@@ -185,31 +275,52 @@ metadata/
 │       └── functions.json      # Extracted functions catalog
 │
 ├── crm/                        # Standard Zoho CRM metadata layout
-│   └── meta/
-│       ├── modules/            # Individual Module schemas
-│       │   └── <Module_API_Name>/
-│       │       ├── <Module_API_Name>.modules-meta.json
-│       │       ├── summary.json # Custom module summary (counts, fields, picklists, formulas)
-│       │       ├── fields/     # Field configurations
-│       │       │   └── <Field_API_Name>.fields-meta.json
-│       │       ├── layouts/    # Page Layout configurations
-│       │       │   └── <Layout_API_Name>.layouts-meta.json
-│       │       ├── custom_views/ # Custom View configurations
-│       │       │   └── <View_API_Name>.custom_views-meta.json
-│       │       └── related_lists/ # Related List configurations
-│       │           └── <List_API_Name>.related_lists-meta.json
-│       ├── profiles/           # Profile Permissions mapping
-│       │   └── <Profile_Name>.profiles-meta.json
-│       ├── roles/              # Role Hierarchy definitions
-│       │   └── <Role_Name>.roles-meta.json
-│       └── functions/          # Custom Deluge function files (.ds)
-│           ├── standalone/     # Deluge scripts sorted by namespace
-│           │   └── standalone.<Function_API_Name>.ds
-│           ├── automation/
-│           ├── button/
-│           ├── related_list/
-│           ├── schedule/
-│           └── salessignals/
+│   ├── meta/
+│   │   ├── custom_links/       # Global Custom Links
+│   │   ├── data_sharing/       # Data Sharing Settings
+│   │   ├── data_sharing_actions_summary/ # Data Sharing Actions Summary
+│   │   ├── data_sharing_rules/ # Data Sharing Rules
+│   │   ├── email_templates/    # Email Templates
+│   │   ├── global_picklists/   # Global Picklists
+│   │   ├── inventory_templates/# Inventory Templates
+│   │   ├── territories/        # Territories
+│   │   ├── variables/          # Global Variables
+│   │   ├── wizards/            # CRM Wizards
+│   │   ├── workflow_rules/     # Global Workflow Rules
+│   │   ├── modules/            # Individual Module schemas
+│   │   │   └── <Module_API_Name>/
+│   │   │       ├── <Module_API_Name>.modules-meta.json
+│   │   │       ├── summary.json # Custom module summary (counts, fields, picklists, formulas, records_count)
+│   │   │       ├── duplicate_check_preference.json # Duplicate Check Preferences
+│   │   │       ├── fields/     # Field configurations
+│   │   │       │   └── <Field_API_Name>.fields-meta.json
+│   │   │       ├── layouts/    # Page Layout configurations
+│   │   │       │   ├── <Layout_API_Name>.layouts-meta.json
+│   │   │       │   └── map_dependencies/
+│   │   │       │       └── <Layout_ID>.json
+│   │   │       ├── custom_views/ # Custom View configurations
+│   │   │       │   └── <View_API_Name>.custom_views-meta.json
+│   │   │       ├── record_locking_configurations/ # Record Locking Rules
+│   │   │       │   └── <Config_ID>.json
+│   │   │       ├── related_lists/ # Related List configurations
+│   │   │       │   └── <List_API_Name>.related_lists-meta.json
+│   │   │       ├── tags/       # Module Tags
+│   │   │       │   └── <Tag_ID>.json
+│   │   │       └── workflow_configurations/ # Module Workflow Configurations
+│   │   │           └── <Config_ID>.json
+│   │   ├── profiles/           # Profile Permissions mapping
+│   │   │   └── <Profile_Name>.profiles-meta.json
+│   │   └── roles/              # Role Hierarchy definitions
+│   │       └── <Role_Name>.roles-meta.json
+│   └── functions/              # Custom Deluge function files (.ds)
+│       ├── functions.json      # Complete list of extracted functions
+│       ├── standalone/         # Deluge scripts sorted by namespace
+│       │   └── standalone.<Function_API_Name>.ds
+│       ├── automation/
+│       ├── button/
+│       ├── related_list/
+│       ├── schedule/
+│       └── salessignals/
 │
 ├── zcrm-project.json           # ZCRM CLI project settings config
 ├── meta.json                   # ZCRM resource configuration
@@ -220,14 +331,20 @@ metadata/
 
 ## 🏢 About Wanas Apps
 
-**[Wanas Apps](https://www.wanasapps.com)** is an elite **Zoho Premium Partner** and consulting firm in the Middle East and North Africa (MENA) region. We specialize in digital transformation, Zoho implementation, custom CRM customization, full-stack application development on Zoho Catalyst, and building certified business compliance tools (such as Egyptian E-Invoice compliance integrations). 
+**[Wanas Apps](https://www.wanasapps.com)** is an elite **Zoho Premium Partner** and advanced technical consulting firm serving the Middle East and North Africa (MENA) region. We bridge the gap between standard business processes and advanced technical architecture, specializing in end-to-end digital transformation.
+
+### 🌟 Our Core Expertise
+- **Zoho Ecosystem Implementation:** Tailored deployments of Zoho CRM, Zoho Books, Zoho Creator, and the wider Zoho suite.
+- **Custom Application Development:** Full-stack, scalable, and serverless solutions built on **Zoho Catalyst**, Node.js, and modern frontends like Angular.
+- **Advanced Integrations & Automations:** We start where off-the-shelf software ends—building robust API gateways, custom webhooks, and complex CRM workflow automations.
+- **Compliance & Localization:** Certified business compliance tools, including seamless native integrations for the Egyptian E-Invoice and E-Receipt systems (ETA).
 
-We start where off-the-shelf software ends, building custom features that fit around your business blueprint.
+*We don't just implement software; we build custom features that fit seamlessly around your unique business blueprint.*
 
 ---
 
 ## ✉️ Support & Feedback
 
-If you encounter any issues, have feature requests, or need custom Zoho integrations, please reach out to our team at **support@wanasapps.com**.
+If you encounter any issues, have feature requests, or need custom Zoho integrations, please reach out to our team at **support@wanasapps.com** or visit **[wanasapps.com](https://www.wanasapps.com)**.
 
 This project is maintained by Wanas Apps and licensed under the **ISC License**.

package/banner.txt

@@ -1,7 +1,8 @@
-      ___  
-      \  \       __        __                         _                    
-   ___ \  \      \ \      / /_ _ _ __   __ _ ___     / \   _ __  _ __  ___ 
-  /  / /  /       \ \ /\ / / _` | '_ \ / _` / __|   / _ \ | '_ \| '_ \/ __|
- /  / /__/         \ V  V / (_| | | | | (_| \__ \  / ___ \| |_) | |_) \__ \
- \  \               \_/\_/ \__,_|_| |_|\__,_|___/ /_/   \_\ .__/| .__/|___/
-  \__\                                                    |_|   |_|        
+ __        __                         _                    
+ \ \      / /_ _ _ __   __ _ ___     / \   _ __  _ __  ___ 
+  \ \ /\ / / _` | '_ \ / _` / __|   / _ \ | '_ \| '_ \/ __|
+   \ V  V / (_| | | | | (_| \__ \  / ___ \| |_) | |_) \__ \
+    \_/\_/ \__,_|_| |_|\__,_|___/ /_/   \_\ .__/| .__/|___/
+                                          |_|   |_|        
+      For Support Contact us on support@wanasapps.com
+                     www.wanasapps.com

package/bin/cli.js

@@ -8,10 +8,44 @@ const inquirer = require('inquirer').default || require('inquirer');
 const http = require('http');
 const url = require('url');
 const path = require('path');
-const { exec } = require('child_process');
+const { exec, spawn } = require('child_process');
 const configStore = require('../src/utils/configStore');
 const authService = require('../src/services/authService');
 const crmMetadataService = require('../src/services/metadata/crmMetadataService');
+const auditLogService = require('../src/services/auditLogService');
+const skillGenerator = require('../src/utils/skillGenerator');
+
+/**
+ * Opens a URL in the user's default browser in a cross-platform-safe way.
+ * On Windows, `start` is a cmd builtin whose first quoted token is the window
+ * title, so an empty "" placeholder is required before the URL. On macOS/Linux
+ * the URL is passed as a separate argv entry (no shell) to avoid metacharacter
+ * issues. All failures are swallowed — the URL is also printed for manual use.
+ * @param {string} targetUrl - The URL to open.
+ */
+function openBrowser(targetUrl) {
+  try {
+    if (process.platform === 'win32') {
+      // Quotes protect the '&' query separators; "" is the required title arg.
+      exec(`start "" "${targetUrl.replace(/"/g, '%22')}"`, () => {});
+    } else {
+      const cmd = process.platform === 'darwin' ? 'open' : 'xdg-open';
+      const child = spawn(cmd, [targetUrl], { stdio: 'ignore', detached: true });
+      child.on('error', () => {});
+      child.unref();
+    }
+  } catch (err) {
+    // User can still click the URL printed to the console.
+  }
+}
+
+// A short, prominent reminder of the core guarantee: the tool is strictly
+// read-only and never modifies Zoho CRM. By default it pulls schema/metadata
+// only; reading record data is opt-in (pull --with-counts) or explicit (audit).
+const READ_ONLY_NOTICE =
+  '\x1b[1;33mℹ Note:\x1b[0m This tool is \x1b[1mread-only\x1b[0m — it \x1b[1mnever creates, updates, or deletes\x1b[0m anything\n' +
+  '  in Zoho CRM. By default it extracts \x1b[1mschema/metadata only\x1b[0m (record counts and audit\n' +
+  '  logs are read only when you explicitly ask via --with-counts or the audit command).';
 
 // Default client credentials for generating Zoho login links
 const DEFAULT_CLIENT_ID = '1000.M1XE1M7CA4VFDDIRANB8E5AKIBQ72U';
@@ -64,7 +98,8 @@ program
   .name('zcrm')
   .description('Zoho CRM V8 Metadata Extractor CLI Utility')
   .version(pkg.version, '-v, --version', 'output the current version')
-  .addHelpText('before', getBanner());
+  .addHelpText('before', getBanner())
+  .addHelpText('after', `\n${READ_ONLY_NOTICE}\n`);
 
 /**
  * CLI Callback Server validation & token retrieval flow.
@@ -176,11 +211,8 @@ function runOAuthServer(clientId, clientSecret, redirectUri, dc) {
       console.log(`\x1b[4;34m${authUrl}\x1b[0m`);
       console.log('\x1b[1;33m========================================================================\x1b[0m\n');
 
-      // Attempt to auto-open in default browser
-      const launch = process.platform === 'darwin' ? 'open' : process.platform === 'win32' ? 'start' : 'xdg-open';
-      exec(`${launch} "${authUrl.replace(/"/g, '\\"')}"`, (execErr) => {
-        // Suppress errors (user can manually click printed link if this fails)
-      });
+      // Attempt to auto-open in default browser (failures are non-fatal)
+      openBrowser(authUrl);
     });
   });
 }
@@ -381,8 +413,10 @@ program
 // REGISTER COMMAND: pull
 program
   .command('pull')
-  .description('Pull Zoho CRM metadata and save as JSON file directory structure')
+  .description('Pull Zoho CRM metadata (read-only) and sync it into a local JSON directory structure')
   .option('-o, --output <dir>', 'Output folder path', './metadata')
+  .option('-c, --concurrency <n>', 'Max concurrent Zoho API calls (1-25; default 5 — safe for all editions)', '5')
+  .option('--with-counts', 'Also fetch per-module record counts (reads record data; ~50 API credits per module)')
   .action(async (options) => {
     console.log(getBanner());
     // Reload tokens
@@ -394,94 +428,79 @@ program
     }
 
     const outputDir = path.resolve(options.output);
+    let concurrency = parseInt(options.concurrency, 10);
+    if (!Number.isFinite(concurrency) || concurrency < 1) concurrency = 5;
+    if (concurrency > 25) concurrency = 25;
+
     console.log(`\x1b[36mInitializing CRM Metadata Pulling...\x1b[0m`);
-    console.log(`Destination Folder: ${outputDir}\n`);
+    console.log(`Destination Folder: ${outputDir}`);
+    console.log(`Concurrency: ${concurrency} simultaneous API call(s)`);
+    if (options.withCounts) {
+      console.log('\x1b[33mRecord counts: ON\x1b[0m (reads record data; ~50 API credits per module)');
+    }
+    console.log('');
 
-    let currentModule = 0;
-    let totalModules = 0;
-    let currentFunction = 0;
-    let totalFunctions = 0;
     let inModulesPhase = false;
     let inFunctionsPhase = false;
 
+    // Finalizes an in-progress progress bar line with a success message.
+    const finishPhaseLine = (text) => {
+      const readline = require('readline');
+      readline.clearLine(process.stdout, 0);
+      readline.cursorTo(process.stdout, 0);
+      console.log(`\x1b[32m✔\x1b[0m ${text}`);
+    };
+
     try {
       await crmMetadataService.extract((stage, message, details = {}) => {
         switch (stage) {
           case 'FETCH_ORG':
-            console.log(`\x1b[34mℹ\x1b[0m ${message}`);
-            break;
-          case 'FETCH_ORG_SUCCESS':
-            console.log(`\x1b[32m✔\x1b[0m ${message}`);
-            break;
           case 'FETCH_MODULES':
             console.log(`\x1b[34mℹ\x1b[0m ${message}`);
             break;
+          case 'FETCH_ORG_SUCCESS':
           case 'FETCH_MODULES_SUCCESS':
-            console.log(`\x1b[32m✔\x1b[0m ${message}`);
-            break;
-          case 'MODULE_START':
-            if (!inModulesPhase) {
-              console.log(`\n\x1b[1mExtracting Modules Metadata:\x1b[0m`);
-              inModulesPhase = true;
-            }
-            currentModule = details.index;
-            totalModules = details.total;
-            drawProgress(currentModule - 1, totalModules, 'Modules', `Extracting ${details.module}...`);
-            break;
-          case 'MODULE_SUCCESS':
-            if (currentModule === totalModules) {
-              const readline = require('readline');
-              readline.clearLine(process.stdout, 0);
-              readline.cursorTo(process.stdout, 0);
-              console.log(`\x1b[32m✔\x1b[0m All modules metadata extracted successfully.`);
-              inModulesPhase = false;
-            } else {
-              drawProgress(currentModule, totalModules, 'Modules', `Completed ${details.module || ''}`);
-            }
-            break;
-          case 'FETCH_WEBHOOKS':
-            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
-            break;
           case 'FETCH_WEBHOOKS_SUCCESS':
-            console.log(`\x1b[32m✔\x1b[0m ${message}`);
-            break;
-          case 'FETCH_PROFILES':
-            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
-            break;
           case 'FETCH_PROFILES_SUCCESS':
-            console.log(`\x1b[32m✔\x1b[0m ${message}`);
-            break;
-          case 'FETCH_ROLES':
-            console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
-            break;
           case 'FETCH_ROLES_SUCCESS':
             console.log(`\x1b[32m✔\x1b[0m ${message}`);
             break;
+          case 'FETCH_WEBHOOKS':
+          case 'FETCH_PROFILES':
+          case 'FETCH_ROLES':
           case 'FETCH_FUNCTIONS':
             console.log(`\n\x1b[34mℹ\x1b[0m ${message}`);
             break;
-          case 'FETCH_FUNCTIONS_SUCCESS':
-            console.log(`\x1b[32m✔\x1b[0m ${message}`);
+          case 'MODULES_START':
+            console.log(`\n\x1b[1mExtracting Modules Metadata:\x1b[0m`);
+            inModulesPhase = true;
+            if (!details.total) {
+              finishPhaseLine('No modules to extract.');
+              inModulesPhase = false;
+            }
             break;
-          case 'FUNCTION_START':
-            if (!inFunctionsPhase) {
-              console.log(`\n\x1b[1mExtracting Custom Functions Deluge Scripts:\x1b[0m`);
-              inFunctionsPhase = true;
+          case 'MODULE_PROGRESS':
+            if (details.completed >= details.total) {
+              finishPhaseLine(`All ${details.total} modules processed${details.removed ? '' : ''}.`);
+              inModulesPhase = false;
+            } else {
+              drawProgress(details.completed, details.total, 'Modules', `${details.module || ''}`);
             }
-            currentFunction = details.index;
-            totalFunctions = details.total;
-            drawProgress(currentFunction - 1, totalFunctions, 'Functions', `Downloading ${details.name}...`);
             break;
-          case 'FUNCTION_SUCCESS':
-          case 'FUNCTION_WARNING':
-            if (currentFunction === totalFunctions) {
-              const readline = require('readline');
-              readline.clearLine(process.stdout, 0);
-              readline.cursorTo(process.stdout, 0);
-              console.log(`\x1b[32m✔\x1b[0m All custom functions deluge scripts extracted.`);
+          case 'FUNCTIONS_START':
+            console.log(`\n\x1b[1mExtracting Custom Functions Deluge Scripts:\x1b[0m`);
+            inFunctionsPhase = true;
+            if (!details.total) {
+              finishPhaseLine('No custom functions found.');
+              inFunctionsPhase = false;
+            }
+            break;
+          case 'FUNCTION_PROGRESS':
+            if (details.completed >= details.total) {
+              finishPhaseLine('All custom function deluge scripts extracted.');
               inFunctionsPhase = false;
             } else {
-              drawProgress(currentFunction, totalFunctions, 'Functions', `Completed ${details.name || ''}`);
+              drawProgress(details.completed, details.total, 'Functions', `${details.name || ''}`);
             }
             break;
           case 'FAILURE':
@@ -500,8 +519,9 @@ program
             console.log(` \x1b[1mTotal Layouts:\x1b[0m     \x1b[32m${details.totalLayouts || 0}\x1b[0m`);
             console.log(` \x1b[1mRelated Lists:\x1b[0m     \x1b[35m${details.totalRelatedLists || 0}\x1b[0m (saved inside crm/meta/modules/)`);
             console.log(` \x1b[1mCustom Views:\x1b[0m      \x1b[35m${details.totalCustomViews || 0}\x1b[0m (saved inside crm/meta/modules/)`);
+            console.log(` \x1b[1mStale Files Removed:\x1b[0m \x1b[33m${details.removedStale || 0}\x1b[0m (deleted on Zoho CRM since last pull)`);
             console.log(` \x1b[1mOutput Directory:\x1b[0m  \x1b[34m${outputDir}\x1b[0m`);
-            
+
             if (details.errors && details.errors.length > 0) {
               console.log(`\n \x1b[1;33m⚠️  Warnings:\x1b[0m \x1b[33m${details.errors.length} API request(s) failed or were skipped due to permissions.\x1b[0m`);
             }
@@ -513,7 +533,7 @@ program
             }
             break;
         }
-      }, outputDir);
+      }, outputDir, { concurrency, withCounts: !!options.withCounts });
 
       process.exit(0);
     } catch (error) {
@@ -522,6 +542,40 @@ program
     }
   });
 
+// REGISTER COMMAND: audit
+program
+  .command('audit')
+  .description('Export the Zoho CRM audit log (asynchronous job)')
+  .option('-f, --filter <path>', 'Path to a JSON file containing the audit_log_export criteria')
+  .action(async (options) => {
+    console.log(getBanner());
+    // This command DOES read/export audit-log data, so it gets its own honest
+    // notice rather than the generic "metadata only / no data access" one.
+    console.log(
+      '\x1b[1;33mℹ Note:\x1b[0m This exports your Zoho CRM \x1b[1maudit-log data\x1b[0m (read-only).\n' +
+      '  It never creates, updates, or deletes anything in Zoho CRM.\n'
+    );
+
+    // Ensure user is authenticated
+    authService.loadTokens();
+    if (!authService.isAuthenticated()) {
+      console.error('\n❌ You are not logged in. Please run `zcrm login` first.');
+      process.exit(1);
+    }
+
+    try {
+      await authService.getAccessToken(); // Refresh if necessary
+      await auditLogService.exportAuditLog(options.filter);
+      console.log('\n\x1b[1;32m====================================================================\x1b[0m');
+      console.log('🎉 \x1b[1;32mAudit Log Export Completed Successfully!\x1b[0m');
+      console.log('\x1b[1;32m====================================================================\x1b[0m\n');
+      process.exit(0);
+    } catch (error) {
+      console.error(`\n❌ Audit log export failed: ${error.message}`);
+      process.exit(1);
+    }
+  });
+
 // REGISTER COMMAND: llm
 program
   .command('llm')
@@ -545,6 +599,96 @@ program
     }
   });
 
+// REGISTER COMMAND: skill
+program
+  .command('skill')
+  .description('Generate an AI assistant skill/rules file (from llm.md) for your IDE')
+  .option('--ide <ide>', 'Target IDE/assistant (claude, cursor, windsurf, copilot, cline, gemini, codex, markdown)')
+  .option('-d, --dir <dir>', 'Project directory to write the skill file into', '.')
+  .option('-m, --metadata-dir <dir>', 'Path to the extracted metadata, referenced inside the skill', './metadata')
+  .option('-f, --force', 'Overwrite an existing skill file without prompting')
+  .action(async (options) => {
+    console.log(getBanner());
+    try {
+      let ide = options.ide;
+
+      // Prompt for the target IDE/assistant when not supplied via --ide.
+      if (!ide) {
+        const { selected } = await inquirer.prompt([
+          {
+            type: 'select',
+            name: 'selected',
+            message: 'Select your IDE / AI assistant:',
+            choices: skillGenerator.getChoices()
+          }
+        ]);
+        ide = selected;
+      }
+
+      if (!skillGenerator.TARGETS[ide]) {
+        console.error(`\n❌ Unknown IDE/AI target: "${ide}".`);
+        console.error(`   Valid values: ${Object.keys(skillGenerator.TARGETS).join(', ')}`);
+        process.exit(1);
+      }
+
+      const projectDir = path.resolve(options.dir);
+      const packageDir = path.join(__dirname, '..');
+
+      let result = skillGenerator.generateSkill({
+        ide,
+        projectDir,
+        metadataDir: options.metadataDir,
+        packageDir,
+        force: options.force
+      });
+
+      // Standalone files that already exist require explicit confirmation.
+      if (result.status === 'exists') {
+        const { overwrite } = await inquirer.prompt([
+          {
+            type: 'confirm',
+            name: 'overwrite',
+            message: `File already exists:\n  ${result.filePath}\nOverwrite it?`,
+            default: false
+          }
+        ]);
+        if (!overwrite) {
+          console.log('\n\x1b[33mℹ Cancelled. No file was changed.\x1b[0m\n');
+          process.exit(0);
+        }
+        result = skillGenerator.generateSkill({
+          ide,
+          projectDir,
+          metadataDir: options.metadataDir,
+          packageDir,
+          force: true
+        });
+      }
+
+      const verb = result.status === 'updated' ? 'Updated' : 'Created';
+      console.log('\n\x1b[1;32m====================================================================\x1b[0m');
+      console.log(`🧩 \x1b[1;32m${verb} ${skillGenerator.TARGETS[ide].label.split('→')[0].trim()} skill file.\x1b[0m`);
+      console.log('\x1b[1;32m====================================================================\x1b[0m');
+      console.log(` \x1b[1mFile:\x1b[0m          \x1b[36m${result.filePath}\x1b[0m`);
+      console.log(` \x1b[1mMetadata root:\x1b[0m \x1b[34m${path.resolve(options.metadataDir)}\x1b[0m`);
+      console.log('--------------------------------------------------------------------');
+      console.log(` ${READ_ONLY_NOTICE}`);
+
+      // Helpful nudge if no metadata has been pulled yet.
+      const metaIndex = path.join(path.resolve(options.metadataDir), '.zcrm', '.store', 'index.json');
+      const fs = require('fs');
+      if (!fs.existsSync(metaIndex)) {
+        console.log('--------------------------------------------------------------------');
+        console.log(` \x1b[33mTip:\x1b[0m No metadata found yet. Run \x1b[36mzcrm pull -o "${options.metadataDir}"\x1b[0m to populate it.`);
+      }
+      console.log('\x1b[1;32m====================================================================\x1b[0m\n');
+      process.exit(0);
+    } catch (err) {
+      console.error(`\n❌ Skill generation failed: ${err.message}`);
+      process.exit(1);
+    }
+  });
+
 // REGISTER COMMAND: dashboard
 program
   .command('dashboard')
@@ -552,7 +696,11 @@ program
   .option('-p, --port <port>', 'Port to run the dashboard on', '3000')
   .action((options) => {
     process.env.PORT = options.port;
-    process.env.ZCRM_CLI_MODE = 'false';
+    process.env.ZCRM_VERBOSE = 'true';
+    // Keep ZCRM_CLI_MODE='true' (set at startup) so the dashboard shares the
+    // same global credential/token store as `zcrm login` and `zcrm pull`.
+    // Flipping it to 'false' here would desync the already-initialized
+    // authService singleton from the web-mode token path.
     console.log(`\x1b[36mLaunching Extractor Dashboard...\x1b[0m`);
     require('../server.js');
   });