langaro-api 1.2.3 → 1.2.5

package/lib/cli/documentation-templates/11-config-and-bootstrap.md

@@ -0,0 +1,352 @@
+# Configuration & Bootstrap
+
+## Role
+
+The config layer handles application startup, middleware wiring, database connections, Redis, queue initialization, and error handling. These files are generated by `langaro-api init` and rarely need modification.
+
+---
+
+## server.js
+
+Entry point for the application. Handles environment setup, server creation, and graceful shutdown.
+
+```
+Location: src/config/server.js
+```
+
+**Responsibilities:**
+1. Load environment variables (`.env` or `.env.test`)
+2. Create Knex database connection
+3. Call `getAppInstance(knex)` to initialize the App
+4. Listen on `PORT`
+5. Handle `SIGINT`/`SIGTERM` for graceful shutdown
+
+**Graceful shutdown sequence:**
+1. Stop accepting new connections
+2. Shut down BullMQ queues (wait for active jobs to complete)
+3. Close HTTP server
+4. Destroy Knex connection
+5. Exit process
+
+**When to modify:** Almost never. Only if you need custom shutdown logic or additional initialization before the app starts.
+
+---
+
+## app.js
+
+The core application class that wires everything together.
+
+```
+Location: src/config/app.js
+```
+
+### App Class Structure
+
+```javascript
+class App {
+  constructor(knexInstance, servicesInstance) {
+    // Initialize Express, HTTP server, Socket.io
+    // Set up readiness promise (blocks requests until boot completes)
+    // Call init()
+  }
+
+  async init() {
+    // 1. Set query parser
+    // 2. Apply readiness middleware
+    // 3. Apply standard middlewares
+    // 4. Wire routes (models → services → queues → controllers → routes → tasks)
+    // 5. Apply error handling
+    // 6. Mark as ready
+  }
+
+  middlewares() {
+    // requestIp, CORS, morgan, bodyParser (50MB limit), multer
+  }
+
+  async routes() {
+    // Socket.io setup
+    // loadModels → loadServices → initializeQueues → loadControllers → attachRouters → loadTasks
+  }
+
+  handleErrors() {
+    // Sentry error handler
+    // Axios network errors → 422
+    // Safe errors → err.code + err.data
+    // Unsafe errors → 500 + Sentry
+  }
+
+  bullBoard() {
+    // Queue dashboard at BULLBOARD_PATH
+  }
+}
+```
+
+### Readiness Middleware
+
+The app blocks all incoming requests until initialization completes:
+```javascript
+async readinessMiddleware(req, res, next) {
+  if (this.isReady) return next();
+  try {
+    await this.readyPromise;  // Wait for init() to resolve
+    next();
+  } catch (error) {
+    res.status(503).json({ error: { message: 'Server is starting up.' } });
+  }
+}
+```
+
+### Error Handling
+
+```javascript
+// Safe errors (created via ApiError) — returned to client
+if (err.safe) {
+  res.status(err.code).json({ error: err.data });
+}
+
+// Unsafe errors (uncaught) — generic response + Sentry
+res.status(500).json({
+  error: { message: 'Unable to process request. Please try again later.' },
+});
+```
+
+**Creating safe errors:**
+```javascript
+const { ApiError, StatusCodes } = require('@/utils');
+
+// In controllers or services:
+ApiError(StatusCodes.NOT_FOUND, 'User not found');
+ApiError(StatusCodes.BAD_REQUEST, 'Invalid email format', 'INVALID_EMAIL');
+ApiError(StatusCodes.CONFLICT, 'Email already registered');
+```
+
+**When to modify:** When adding global middleware, changing body size limits, adding new middleware to the chain, or customizing error handling.
+
+---
+
+## Database Connection
+
+```
+Location: src/database/connection.js
+```
+
+Creates and configures the Knex instance:
+
+```javascript
+module.exports = (env = NODE_ENV, dbName = DB_NAME) => {
+  const knex = require('knex')({
+    client: 'mysql2',
+    connection: {
+      host: DB_HOST,
+      user: DB_USER,
+      port: DB_PORT,
+      password: DB_PASSWORD,
+      database: dbName,
+      typeCast(field, next) {
+        // TINY(1) → boolean conversion
+        if (field.type === 'TINY' && field.length === 1) {
+          const value = field.string();
+          if (value === null) return null;
+          return value === '1';
+        }
+        return next();
+      },
+    },
+    pool: { min: 10, max: 100 },
+  });
+
+  // Attach pagination plugin
+  const { attachPaginate } = require('knex-paginate');
+  if (!knex.paginate) attachPaginate();
+
+  // Attach schema inspector
+  const { SchemaInspector } = require('knex-schema-inspector');
+  knex.inspector = SchemaInspector(knex);
+
+  return knex;
+};
+```
+
+**Key features:**
+- MySQL TINY(1) → JavaScript boolean conversion
+- Connection pooling (min 10, max 100)
+- `knex-paginate` for built-in pagination
+- `knex-schema-inspector` for runtime schema introspection
+
+**Test safety:** Throws if test environment tries to use non-test database (name must contain `_tests`).
+
+---
+
+## Redis
+
+### Redis Config
+
+```
+Location: src/database/redis-config.js
+```
+
+```javascript
+module.exports = {
+  connection: {
+    host: process.env.REDIS_HOST || '127.0.0.1',
+    port: process.env.REDIS_PORT || 6379,
+    password: process.env.REDIS_PASSWORD,
+    username: process.env.REDIS_USERNAME,
+    tls: process.env.REDIS_TLS || false,
+    maxRetriesPerRequest: null,
+  },
+};
+```
+
+### Redis Cache
+
+```
+Location: src/database/redis-cache.js
+```
+
+Singleton class providing simple key-value caching:
+
+```javascript
+class RedisCache {
+  async get(key)                         // Returns parsed JSON or null
+  async set(key, value, expiresInSeconds) // Stores as JSON with TTL
+  async delete(key)                       // Removes key
+}
+
+module.exports = new RedisCache();
+```
+
+**Key format:** `redis-cache:{key}`
+
+**Usage:**
+```javascript
+const RedisCache = require('@/database/redis-cache');
+
+// Cache a value for 1 hour
+await RedisCache.set('exchange-rates', rates, 3600);
+
+// Read from cache
+const cached = await RedisCache.get('exchange-rates');
+if (cached) return cached;
+
+// Invalidate
+await RedisCache.delete('exchange-rates');
+```
+
+---
+
+## Queue System
+
+```
+Location: src/config/queues.js
+```
+
+See `12-queues.md` for detailed queue documentation.
+
+---
+
+## Sentry (Error Tracking)
+
+```
+Location: src/config/instrument.js
+```
+
+```javascript
+const Sentry = require('@sentry/node');
+Sentry.init({
+  dsn: process.env.SENTRY_APP,
+  tracesSampleRate: 0,
+  profilesSampleRate: 0,
+  environment: process.env.NODE_ENV,
+});
+module.exports = Sentry;
+```
+
+**When to modify:** To enable performance monitoring (`tracesSampleRate > 0`) or add custom Sentry configuration.
+
+---
+
+## Environment Variables Reference
+
+### Required
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NODE_ENV` | Environment | `development`, `production`, `test` |
+| `PORT` | Server port | `8282` |
+| `DB_HOST` | MySQL host | `127.0.0.1` |
+| `DB_USER` | MySQL user | `root` |
+| `DB_PASSWORD` | MySQL password | `password` |
+| `DB_NAME` | MySQL database | `my_project` |
+| `JWT_SECRET` | JWT signing key | `auto-generated-uuid` |
+
+### Optional
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `DB_PORT` | MySQL port | `3306` |
+| `REDIS_HOST` | Redis host | `127.0.0.1` |
+| `REDIS_PORT` | Redis port | `6379` |
+| `REDIS_PASSWORD` | Redis password | `password` |
+| `REDIS_USERNAME` | Redis username | — |
+| `REDIS_TLS` | Redis TLS | `true` |
+| `MASTER_PASS` | Dev master password | `auto-generated` |
+| `BULLBOARD_PATH` | Queue dashboard URL | `/admin/queues` |
+| `SENTRY_APP` | Sentry DSN | `https://...@sentry.io/...` |
+| `POSTMARK_API_KEY` | Email service | `key-...` |
+| `AWS_ACCESS_KEY_ID` | AWS credentials | `AKIA...` |
+| `AWS_SECRET_ACCESS_KEY` | AWS credentials | `...` |
+| `AWS_REGION` | AWS region | `us-east-1` |
+| `GOOGLE_CLIENT_ID` | OAuth | `...apps.googleusercontent.com` |
+| `GOOGLE_CLIENT_SECRET` | OAuth | `...` |
+
+---
+
+## Babel Configuration
+
+```
+Location: .babelrc
+```
+
+Enables ES6+ features and the `@` path alias:
+
+```json
+{
+  "presets": [["@babel/preset-env", { "targets": { "node": "current" } }]],
+  "plugins": [
+    "@babel/transform-runtime",
+    "@babel/plugin-proposal-class-properties",
+    ["module-resolver", { "root": ["./"], "alias": { "@": "./src/" } }]
+  ]
+}
+```
+
+The `@` alias lets you use `require('@/database/connection')` instead of relative paths.
+
+---
+
+## When to Modify Bootstrap Files
+
+| Need | File to modify |
+|------|---------------|
+| Add global middleware | `app.js` → `middlewares()` |
+| Change body size limit | `app.js` → `middlewares()` bodyParser config |
+| Add custom error handling | `app.js` → `handleErrors()` |
+| Change DB pool size | `connection.js` → `pool` config |
+| Add new env variable | `.env` + `.env.example` |
+| Change graceful shutdown | `server.js` → SIGTERM handler |
+| Configure Sentry | `instrument.js` |
+| Change Redis config | `redis-config.js` |
+
+---
+
+## Checklist
+
+When modifying configuration:
+
+- [ ] Changes work in both development and production
+- [ ] New env variables added to `.env.example`
+- [ ] No hardcoded credentials or secrets
+- [ ] Error handling maintains safe/unsafe distinction
+- [ ] Middleware order is correct (readiness → CORS → logging → parsing → routes → errors)
+- [ ] Pool sizes are appropriate for the deployment environment

package/lib/cli/documentation-templates/12-queues.md

@@ -0,0 +1,205 @@
+# Queue System (BullMQ)
+
+## Role
+
+The queue system provides **reliable asynchronous job processing** using BullMQ backed by Redis. It handles job creation, worker management, retry logic, rate limiting, and monitoring.
+
+---
+
+## Architecture
+
+```
+┌─────────────────┐     ┌──────────────┐     ┌─────────────────┐
+│   Controllers    │────▶│   Queue.add  │────▶│  Redis (BullMQ) │
+│   Tasks          │     │              │     │                 │
+│   Other Jobs     │     └──────────────┘     └────────┬────────┘
+│                  │                                    │
+└──────────────────┘                                    ▼
+                                              ┌─────────────────┐
+                                              │  Worker (per job)│
+                                              │  job.handle()   │
+                                              └─────────────────┘
+```
+
+- **Single Redis connection** shared by all queues
+- **Lazy creation**: Queues and workers are created on first `add()` call
+- **Bull Board dashboard** at `BULLBOARD_PATH` for monitoring
+- **Automatic cleanup**: Inactive queues close after 10 minutes
+
+---
+
+## Queue Lifecycle
+
+### Initialization (at boot)
+
+```
+1. initializeQueues(services, io, bullBoardSetQueues)
+2. Create Redis connection
+3. loadJobs(services) → scan src/jobs/ → register all job configs in Map
+4. Scan Redis for existing queues with pending jobs
+5. Start workers for any queues that have waiting/active/delayed jobs
+6. Start inactivity cleanup interval (checks every 1 minute)
+7. Return { add } function
+```
+
+### Adding a Job
+
+```
+1. Queue.add('job-name', data, options)
+2. Lookup base config from jobConfigs Map
+3. If grouped: extract groupId, create unique queue name
+4. getOrCreateQueueAndWorker():
+   a. If queue exists → update lastActivity, return
+   b. If new → create Queue + Worker with merged options
+   c. Register worker event handlers (failed → Sentry)
+   d. Worker.run()
+5. queue.add(name, data, options) → Redis
+6. Update Bull Board
+```
+
+### Processing a Job
+
+```
+1. Worker picks up job from Redis
+2. Update lastActivity timestamp
+3. Call job.handle(data, job, { add }, io)
+4. If success → job complete → removed per retention policy
+5. If error → retry according to backoff settings
+6. If error + discard → permanent failure
+```
+
+### Inactivity Cleanup
+
+Every 1 minute, the system checks all active queues:
+- If a queue has been inactive for 10+ minutes
+- AND has 0 active, 0 waiting, 0 delayed jobs
+- → Close worker and queue, free resources
+
+### Graceful Shutdown
+
+```
+1. Pause all workers (stop accepting new jobs)
+2. Wait for active jobs to complete (up to ~20 minutes / 400 retries × 3s)
+3. Close all workers and queues
+4. Clear Redis connection
+```
+
+---
+
+## Default Options
+
+### Job Options (applied to all jobs)
+
+```javascript
+{
+  attempts: 100,                               // Max retry attempts
+  backoff: { type: 'fixed', delay: 60000 },   // 1 minute between retries
+  removeOnComplete: { age: 2592000, count: 1000 }, // Keep 30 days / max 1000
+  removeOnFail: { age: 7776000, count: 5000 },     // Keep 90 days / max 5000
+}
+```
+
+### Worker Options (applied to all workers)
+
+```javascript
+{
+  autorun: false,                    // Workers started manually
+  limiter: { max: 2, duration: 1000 }, // 2 jobs per second
+  maxStalledCount: 5,                // Fail after 5 stalls
+}
+```
+
+---
+
+## Grouped Queues
+
+When a job config has `group: true`, each unique `groupId` creates a separate queue + worker:
+
+```
+Queue.add('sync-data', data, { groupId: 'company-abc' })
+  → Creates queue: sync-data-company-abc
+  → Creates worker for: sync-data-company-abc
+
+Queue.add('sync-data', data, { groupId: 'company-xyz' })
+  → Creates queue: sync-data-company-xyz
+  → Creates worker for: sync-data-company-xyz
+```
+
+**Use case:** Per-company processing isolation. Each company's jobs run independently without blocking other companies.
+
+**Missing groupId:** If `group: true` but no `groupId` in options → throws error.
+
+---
+
+## Test Environment
+
+When `NODE_ENV === 'test'`, `initializeQueues` returns a mock:
+
+```javascript
+return { add: jest.fn() };
+```
+
+No Redis connection, no workers, no queues. Tests can verify `Queue.add` was called without side effects.
+
+---
+
+## Monitoring
+
+### Bull Board Dashboard
+
+Available at `process.env.BULLBOARD_PATH` (default: `/admin/queues`).
+
+Shows:
+- All active queues
+- Job counts (waiting, active, completed, failed, delayed)
+- Individual job details and retry history
+
+### Sentry Integration
+
+All worker failures are automatically captured:
+```javascript
+worker.on('failed', (job, error) => {
+  Sentry.captureException(error, {
+    tags: { queue: queueName },
+    contexts: { data: { 'Job data': job?.data, queue: queueName } },
+  });
+});
+```
+
+---
+
+## Queue API Reference
+
+### `Queue.add(name, data, options)`
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `name` | string | Job name (must match a registered job) |
+| `data` | object | Payload passed to `handle(data, ...)` |
+| `options` | object | Optional: `{ delay, groupId, priority, ... }` |
+
+**Options:**
+- `delay: 5000` — Delay execution by 5 seconds
+- `groupId: 'abc'` — Route to grouped queue (requires `group: true` in job config)
+- Any BullMQ job option
+
+---
+
+## Startup Queue Scanning
+
+On boot, the queue system scans Redis for existing `bull:*:meta` keys to find queues that have pending work from before the restart. It matches queue names against registered job configs and starts workers for any that have waiting/active/delayed jobs.
+
+This ensures jobs added before a deploy or restart are still processed.
+
+---
+
+## When to Modify
+
+The queue configuration (`src/config/queues.js`) rarely needs changes. Common modifications:
+
+| Need | What to change |
+|------|---------------|
+| Different retry defaults | `getDefaultJobOptions()` |
+| Different rate limits | `getDefaultWorkerOptions()` |
+| Longer inactivity timeout | `inactivityTimeLimitMs` |
+| Disable cleanup interval | Set `inactivityCheckIntervalMs` to 0 |