langaro-api 1.2.3 → 1.2.4

package/lib/cli/documentation-templates/07-jobs.md

@@ -0,0 +1,361 @@
+# Jobs (BullMQ Async Processing)
+
+## Role
+
+Jobs are **asynchronous task handlers** that run in BullMQ workers. They process work that should not block HTTP requests: sending emails, processing files, calling external APIs, batch operations.
+
+**What jobs do:**
+- Execute long-running operations asynchronously
+- Retry failed operations with configurable backoff
+- Process work with rate limiting
+- Chain to other jobs
+- Emit real-time updates via Socket.io
+
+**What jobs do NOT do:**
+- Handle HTTP requests (that's controllers)
+- Define business logic that needs synchronous responses (that's services)
+- Run on a schedule (that's tasks)
+
+---
+
+## Location & Naming
+
+```
+src/jobs/
+├── index.js                          # loadJobs loader (do not modify)
+├── send-mail.js                      # Job: 'send-mail'
+├── export-invoices.js                # Job: 'export-invoices'
+├── product-invoice/                  # Subdirectory for related jobs
+│   ├── transmit.js                   # Job: 'product-invoice-transmit'
+│   ├── process-response.js           # Job: 'product-invoice-process-response'
+│   └── cancel.js                     # Job: 'product-invoice-cancel'
+├── affiliate/                        # Another subdirectory
+│   ├── click-register.js             # Job: 'affiliate-click-register'
+│   └── commission-calculate.js       # Job: 'affiliate-commission-calculate'
+└── ...
+```
+
+**Naming convention:** Kebab-case filenames. Job name derived from path:
+- `send-mail.js` → job name `send-mail`
+- `product-invoice/transmit.js` → job name `product-invoice-transmit`
+- `affiliate/click-register.js` → job name `affiliate-click-register`
+
+Files named `index.js` or ending with `.spec.js` are ignored.
+
+---
+
+## Standard Structure
+
+```javascript
+/** @param {ServicesMap} services @generated-types */
+module.exports = (services) => ({
+
+  /**
+   * Job handler function
+   * @param {object} data - The payload passed to Queue.add()
+   * @param {object} job - BullMQ Job object
+   * @param {object} queue - Queue interface { add(name, data, options) }
+   * @param {object} io - Socket.io server instance
+   */
+  async handle(data, job, queue, io) {
+    const { recipientId, templateName } = data;
+
+    // Access any service
+    const user = await services.UsersServices.getWhere('id', recipientId, {
+      firstOnly: true,
+    });
+
+    // Do the work
+    await sendEmail(user.email, templateName);
+
+    // Emit real-time update
+    io.sockets.in(recipientId).emit('email:sent', { templateName });
+  },
+
+  // Optional: Override default job options
+  jobOptions: {
+    attempts: 10,
+    backoff: { type: 'fixed', delay: 30000 },
+  },
+
+  // Optional: Override default worker options
+  workerOptions: {
+    limiter: { max: 5, duration: 1000 },  // 5 jobs per second
+  },
+});
+```
+
+---
+
+## Handle Function Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `data` | object | The payload passed to `Queue.add(name, data)`. Contains all job-specific data. |
+| `job` | BullMQ Job | The job object. Has `job.data`, `job.id`, `job.attemptsMade`, `job.discard()`. |
+| `queue` | `{ add }` | Queue interface to chain to other jobs: `await queue.add('other-job', payload)`. |
+| `io` | Socket.io Server | For real-time updates: `io.sockets.in(userId).emit(event, data)`. |
+
+---
+
+## Adding Jobs to the Queue
+
+### From Controllers
+
+```javascript
+await this.Queue.add('send-mail', {
+  templateName: 'welcome',
+  recipientsList: [{ Email: user.email, TemplateModel: { name: user.name } }],
+});
+```
+
+### From Other Jobs
+
+```javascript
+async handle(data, job, queue, io) {
+  // Process this job...
+
+  // Then trigger another job
+  await queue.add('send-notification', {
+    userId: data.userId,
+    message: 'Processing complete',
+  });
+}
+```
+
+### From Tasks
+
+```javascript
+module.exports = (services, Queue) => ({
+  task: async () => {
+    const items = await services.ItemsServices.get({ andWhere: [['status', 'pending']] });
+    for (let i = 0; i < items.data.length; i++) {
+      await Queue.add('process-item', { itemId: items.data[i].id });
+    }
+  },
+  cronTime: '0 * * * *',
+  isActive: true,
+});
+```
+
+### With Options
+
+```javascript
+// Delayed execution
+await this.Queue.add('reminder', data, { delay: 60000 }); // 1 minute delay
+
+// Grouped queue (per-company isolation)
+await this.Queue.add('sync-data', data, { groupId: companyId });
+```
+
+---
+
+## Job Options
+
+Default options (applied to all jobs unless overridden):
+
+```javascript
+{
+  attempts: 100,                          // Max retry attempts
+  backoff: { type: 'fixed', delay: 60000 }, // 1 minute between retries
+  removeOnComplete: { age: 2592000, count: 1000 }, // Keep completed for 30 days, max 1000
+  removeOnFail: { age: 7776000, count: 5000 },     // Keep failed for 90 days, max 5000
+}
+```
+
+Override per job:
+
+```javascript
+jobOptions: {
+  attempts: 1,          // No retries (e.g., for import jobs)
+  backoff: null,
+}
+```
+
+```javascript
+jobOptions: {
+  attempts: 10,
+  backoff: {
+    type: 'exponential',  // 1s, 2s, 4s, 8s, 16s...
+    delay: 1000,
+  },
+}
+```
+
+---
+
+## Worker Options
+
+Default options:
+
+```javascript
+{
+  autorun: false,                        // Workers start manually
+  limiter: { max: 2, duration: 1000 },  // 2 jobs per second
+  maxStalledCount: 5,                    // Kill after 5 stalls
+}
+```
+
+Override per job:
+
+```javascript
+// No rate limiting (e.g., email sending)
+workerOptions: {
+  limiter: null,
+}
+
+// Higher throughput
+workerOptions: {
+  limiter: { max: 10, duration: 1000 },  // 10 jobs per second
+}
+```
+
+---
+
+## Grouped Queues
+
+When `group: true`, each unique `groupId` creates its own queue and worker pair. This provides per-entity isolation.
+
+```javascript
+module.exports = (services) => ({
+  async handle(data, job, queue, io) {
+    // This handler runs in an isolated queue per groupId
+    await processForCompany(data.companyId);
+  },
+  group: true,  // Enable grouped queues
+  workerOptions: {
+    limiter: { max: 1, duration: 1000 },  // 1 job per second per group
+  },
+});
+```
+
+**Adding grouped jobs:**
+```javascript
+await this.Queue.add('sync-data', { companyId: 'abc' }, { groupId: 'abc' });
+await this.Queue.add('sync-data', { companyId: 'xyz' }, { groupId: 'xyz' });
+// Creates 2 separate queues: sync-data-abc and sync-data-xyz
+```
+
+Inactive grouped queues auto-close after 10 minutes.
+
+---
+
+## Error Handling
+
+### Retry (default)
+Throw an error to trigger automatic retry:
+```javascript
+async handle(data, job, queue, io) {
+  const response = await callExternalApi(data);
+  if (!response.ok) {
+    throw new Error(`API failed: ${response.status}`);
+    // Will retry according to jobOptions.attempts and backoff
+  }
+}
+```
+
+### Permanent Failure
+Call `job.discard()` then throw to stop retrying:
+```javascript
+async handle(data, job, queue, io) {
+  const response = await callExternalApi(data);
+  if (response.status === 404) {
+    job.discard();  // Mark as permanently failed
+    throw new Error('Resource not found — not retryable');
+  }
+}
+```
+
+### Partial Retry
+Re-queue only the failed items:
+```javascript
+async handle(data, job, queue, io) {
+  const results = await sendBatchEmails(data.recipients);
+  const failed = results.filter(r => !r.success);
+
+  if (failed.length > 0 && failed.length < data.recipients.length) {
+    // Some succeeded, some failed — retry only failed
+    await queue.add('send-mail', {
+      ...data,
+      recipients: failed.map(f => f.recipient),
+    });
+  } else if (failed.length === data.recipients.length) {
+    // All failed — throw to use standard retry
+    throw new Error('All emails failed');
+  }
+}
+```
+
+All job failures are automatically captured by Sentry.
+
+---
+
+## Real-Time Progress Updates
+
+```javascript
+async handle(data, job, queue, io) {
+  const socket = io.sockets.in(data.userId);
+
+  socket.emit('import:started', { id: data.importId });
+
+  for (let i = 0; i < data.rows.length; i++) {
+    await processRow(data.rows[i]);
+
+    // Emit progress every 10%
+    if (i % Math.ceil(data.rows.length / 10) === 0) {
+      socket.emit('import:progress', {
+        id: data.importId,
+        progress: Math.round((i / data.rows.length) * 100),
+      });
+    }
+  }
+
+  socket.emit('import:completed', { id: data.importId });
+}
+```
+
+---
+
+## Creating from Scratch
+
+1. **Create the file:** `src/jobs/{name}.js` (kebab-case)
+2. **Use the template:**
+   ```javascript
+   /** @param {ServicesMap} services @generated-types */
+   module.exports = (services) => ({
+     async handle(data, job, queue, io) {
+       // Job logic here
+     },
+   });
+   ```
+3. **Add job/worker options** if defaults aren't suitable
+4. **Trigger the job** from a controller or task: `await this.Queue.add('job-name', data)`
+5. **Restart the server** to register the new job
+
+---
+
+## Anti-patterns
+
+- **Do NOT run jobs synchronously in controllers.** If it takes more than a few hundred ms, use a job.
+- **Do NOT forget to handle errors.** Unhandled errors cause infinite retries until max attempts.
+- **Do NOT store large payloads in job data.** Store data in the database, pass only IDs.
+- **Do NOT rely on job execution order.** BullMQ does not guarantee ordering across different job instances.
+- **Do NOT use `job.discard()` without throwing.** The discard only takes effect when followed by a throw.
+- **Do NOT create job files named `index.js`** — they are ignored by the loader.
+
+---
+
+## Checklist
+
+When creating a new job:
+
+- [ ] File is in `src/jobs/` (or subdirectory)
+- [ ] Filename is kebab-case (e.g., `send-mail.js`, not `sendMail.js`)
+- [ ] Exports a factory function `(services) => JobDefinition`
+- [ ] Has `handle` method with `(data, job, queue, io)` signature
+- [ ] Has correct `@generated-types` JSDoc annotation
+- [ ] Error handling: either throw (to retry) or discard + throw (permanent failure)
+- [ ] Job data is minimal (IDs, not full objects)
+- [ ] At least one trigger point exists (controller, task, or another job)
+- [ ] Worker options set appropriately for the workload (rate limiting)
+- [ ] Not named `index.js` or `*.spec.js`

package/lib/cli/documentation-templates/08-tasks.md

@@ -0,0 +1,265 @@
+# Tasks (Cron Scheduled Jobs)
+
+## Role
+
+Tasks are **cron-scheduled functions** that run automatically at defined intervals. They handle recurring operations like data syncing, cache warming, expiration checks, and periodic reports.
+
+**What tasks do:**
+- Run on a cron schedule
+- Poll external services for updates
+- Clean up stale data
+- Trigger batches of background jobs
+- Aggregate or sync data periodically
+
+**What tasks do NOT do:**
+- Handle HTTP requests
+- Process individual items in real-time (that's jobs)
+- Run once on demand (use a job or controller for that)
+
+---
+
+## Location & Naming
+
+```
+src/tasks/
+├── index.js                                    # loadTasks loader (do not modify)
+├── update-currencies-exchange-rates.js         # Runs every 6 hours
+├── product-invoices-status-polling.js          # Runs every 5 minutes
+├── send-certificate-expiration-mail.js         # Runs daily
+└── ...
+```
+
+**Naming convention:** Kebab-case filenames describing what the task does.
+
+Files named `index.js` or ending with `.spec.js` are ignored by the loader.
+
+---
+
+## Standard Structure
+
+```javascript
+/** @param {ServicesMap} services @param {QueueMap} Queue @generated-types */
+module.exports = (services, Queue) => {
+  const task = async () => {
+    // Task logic runs on every cron tick
+
+    // Access any service
+    const { data: companies } = await services.CompaniesServices.get({
+      showOnly: ['id'],
+      perPage: 100000,
+      andWhere: [['needs_sync', 1]],
+    });
+
+    if (!companies.length) return;
+
+    // Option 1: Process directly
+    await services.CompaniesServices.syncExternalData(companies.map(c => c.id));
+
+    // Option 2: Queue individual jobs
+    for (let i = 0; i < companies.length; i++) {
+      await Queue.add('sync-company', { companyId: companies[i].id });
+    }
+  };
+
+  return {
+    task,
+    cronTime: '0 */3 * * *',    // Every 3 hours
+    isActive: true,
+  };
+};
+```
+
+---
+
+## Configuration Properties
+
+| Property | Type | Required | Description |
+|----------|------|----------|-------------|
+| `task` | async function | Yes | The function to execute on each cron tick |
+| `cronTime` | string | Yes | Cron expression (standard format) |
+| `isActive` | boolean | Yes | Whether this task should be started |
+| `environments` | string[] | No | Restrict to specific NODE_ENV values. Default: all environments |
+
+### Cron Expression Format
+
+```
+┌────────── minute (0-59)
+│ ┌──────── hour (0-23)
+│ │ ┌────── day of month (1-31)
+│ │ │ ┌──── month (1-12)
+│ │ │ │ ┌── day of week (0-7, 0 and 7 are Sunday)
+│ │ │ │ │
+* * * * *
+```
+
+**Common patterns:**
+```
+'* * * * *'       — Every minute
+'*/5 * * * *'     — Every 5 minutes
+'0 * * * *'       — Every hour (at minute 0)
+'0 */3 * * *'     — Every 3 hours
+'0 6 * * *'       — Every day at 6:00 AM
+'0 0 * * 1'       — Every Monday at midnight
+'0 0 1 * *'       — First day of every month
+```
+
+**Timezone:** All tasks run in `America/Sao_Paulo` timezone (hardcoded in the loader).
+
+### Environment Filtering
+
+```javascript
+return {
+  task,
+  cronTime: '0 */6 * * *',
+  isActive: true,
+  environments: ['production'],           // Only runs in production
+};
+```
+
+```javascript
+return {
+  task,
+  cronTime: '0 */6 * * *',
+  isActive: true,
+  environments: ['production', 'staging'], // Runs in production and staging
+};
+```
+
+If `environments` is omitted, the task runs in all environments (except test — tasks are never loaded in test).
+
+---
+
+## Factory Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `services` | ServicesMap | All service instances |
+| `Queue` | `{ add(name, data, options) }` | Queue interface for triggering jobs |
+
+---
+
+## Common Patterns
+
+### Data Sync with Staleness Check
+
+```javascript
+module.exports = (services) => {
+  const task = async () => {
+    const sevenDaysAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
+
+    const { data: staleCompanies } = await services.CompaniesServices.get({
+      showOnly: ['id'],
+      perPage: 100000,
+      where: (query) => query.where((q) => {
+        q.where('last_synced_at', '<', sevenDaysAgo.toISOString())
+          .orWhereNull('last_synced_at');
+      }),
+    });
+
+    if (!staleCompanies.length) return;
+
+    await services.CompaniesServices.syncData(staleCompanies.map(c => c.id));
+  };
+
+  return { task, cronTime: '5 * * * *', isActive: true };
+};
+```
+
+### Expiration Processing
+
+```javascript
+module.exports = (services, Queue) => {
+  const task = async () => {
+    const { data: expired } = await services.SubscriptionsServices.get({
+      perPage: 100000,
+      andWhere: [
+        ['status', 'active'],
+        ['expires_at', '<', new Date().toISOString()],
+      ],
+    });
+
+    for (let i = 0; i < expired.length; i++) {
+      await Queue.add('process-expiration', { subscriptionId: expired[i].id });
+    }
+  };
+
+  return { task, cronTime: '0 */6 * * *', isActive: true };
+};
+```
+
+### Periodic Report Generation
+
+```javascript
+module.exports = (services, Queue) => {
+  const task = async () => {
+    const { data: companies } = await services.CompaniesServices.get({
+      showOnly: ['id', 'admin_email'],
+      perPage: 100000,
+      andWhere: [['reports_enabled', true]],
+    });
+
+    for (let i = 0; i < companies.length; i++) {
+      await Queue.add('generate-monthly-report', { companyId: companies[i].id });
+    }
+  };
+
+  return {
+    task,
+    cronTime: '0 8 1 * *',  // 1st of every month at 8 AM
+    isActive: true,
+    environments: ['production'],
+  };
+};
+```
+
+---
+
+## Creating from Scratch
+
+1. **Create the file:** `src/tasks/{task-name}.js` (kebab-case)
+2. **Use the template:**
+   ```javascript
+   /** @param {ServicesMap} services @param {QueueMap} Queue @generated-types */
+   module.exports = (services, Queue) => {
+     const task = async () => {
+       // Task logic
+     };
+
+     return {
+       task,
+       cronTime: '0 * * * *',  // Adjust schedule
+       isActive: true,
+     };
+   };
+   ```
+3. **Choose the right cron expression** for the frequency needed
+4. **Consider environment filtering** — not all tasks should run in development
+5. **Restart the server** to register the new task
+
+---
+
+## Anti-patterns
+
+- **Do NOT run CPU-intensive work directly in tasks.** Queue individual jobs instead — tasks should orchestrate, not execute heavy work.
+- **Do NOT forget `isActive: true`.** An omitted or false `isActive` means the task never runs.
+- **Do NOT assume single-instance execution.** If running multiple server instances, the task runs on ALL of them. Use Redis locks to prevent duplicate execution.
+- **Do NOT use large `perPage` without `showOnly`.** Fetching 100K full records wastes memory. Select only the fields you need.
+- **Do NOT name the file `index.js`** — it's ignored by the loader.
+
+---
+
+## Checklist
+
+When creating a new task:
+
+- [ ] File is in `src/tasks/` (or subdirectory)
+- [ ] Filename is kebab-case
+- [ ] Exports a factory function `(services, Queue) => TaskDefinition`
+- [ ] Has correct `@generated-types` JSDoc annotation
+- [ ] Returns object with `task`, `cronTime`, `isActive`
+- [ ] Cron expression is correct (use crontab.guru to verify)
+- [ ] `isActive` is set to `true`
+- [ ] `environments` is set if task should not run in development
+- [ ] Heavy operations are queued as jobs, not run inline
+- [ ] Queries use `showOnly` and `perPage` limits
+- [ ] Not named `index.js` or `*.spec.js`