@vida-global/core 2.0.0 → 2.0.3

package/lib/jobQueue/README.md

@@ -1,63 +1,156 @@
 # JobQueue
-The `JobQueue` is a general purpose wrapper around an `BullMQ` job queue.
+`JobQueue` is a Vida wrapper around [BullMQ](https://docs.bullmq.io/). It standardizes how jobs are defined, queued, and worked off so that any Vida service can run background work without re-implementing the plumbing.
 
-                                                                                                        
-## USAGE
-The Node Job Queue has 3 key components: `Queues`, `Workers`, and `Jobs`.
 
-                      
-### QUEUES ###                                                                                          
-A `Queue` object is used by a `Job` to add the `Job` and its associated arguments to the Redis queue for a `Worker` in a different process to dequeue and process. The work of a `Queue` is largely encapsulated by the `Job`, so you will rarely need to interact directly with a `Queue`. The exception is when ending a process. In order for your script to exit, you'll need to close any open `Queues`.
+## Setup
 
-Closing a queue looks like this:
-```
-const queue = getQueue(myQueueName);
-await queue.close();
+`Queues` and `Workers` both connect to Redis through the same environment variables:
+
+| Variable | Purpose |
+|---|---|
+| `REDIS_BULL_HOST` | Redis host. |
+| `REDIS_BULL_PORT` | Redis port. |
+| `REDIS_BULL_PASSWORD` | Redis password. |
+
+Queue names are namespaced by `NODE_ENV` automatically, so a queue declared as `'myQueue'` runs as `myQueue-development` in development and `myQueue-production` in production. This keeps environments isolated when they share a Redis instance.
+
+
+## Core Concepts
+
+There are three components:
+
+- **Queue** — Redis-backed list of pending work. Jobs are added to a queue, workers pull from it.
+- **Worker** — long-running process that listens on a queue and runs the matching job class for each entry.
+- **Job** — a class that extends `AbstractJob` and implements `run`. The worker instantiates it per task.
+
+
+## Defining a job
+
+Subclass `AbstractJob`, implement an async `run` method, and declare a `static queueName(...)` that returns the queue the job belongs to. Place files under `jobs/` at the repo root so the worker picks them up automatically.
+
+```js
+const { JobQueue } = require('@vida-global/core');
+const { AbstractJob } = JobQueue;
+
+class MyJob extends AbstractJob {
+    async run(arg1, arg2) {
+        this.logger.info(`starting work on ${arg1}`);
+        // do work here
+
+        await this.updateProgress(50); // optional — reports to BullMQ
+        // ...
+        await this.updateProgress(100);
+    }
+
+    static queueName(arg1, arg2) {
+        return 'myQueue';
+    }
+}
 ```
 
-Other less commonly needed `Queue` methods include:
+Inside `run()`:
+
+- `this.id` — UUID for the job.
+- `this.logger` — child logger tagged with the job's id.
+- `await this.updateProgress(value)` — pushes progress to BullMQ for observation.
+
+Enqueue a job with `queueJob`:
+
+```js
+await MyJob.queueJob(arg1, arg2);
 ```
-await queue.numQueuedJobs(); // returns the number of jobs in the queue
-await queue.getQueuedJobs(); // returns an array of job data representing jobs in the queue
-await queue.numActiveJobs(); // returns the number of jobs currently being worked on
-await queue.getActiveJobs(); // returns an array of job data representing jobs currently being worked on
-await queue.numFailedJobs(); // returns the failed jobs since the last failed jobs clear
-await queue.getFailedJobs(); // returns an array of job data representing jobs failed since the last failed jobs clear
-await queue.clearFailedJobs(); // removes all stored data about failed jobs. This should be done periodically to clear memory in Redis
-await queue.clearCompletedJobs(); // removes all stored data about completed jobs. This should be done periodically to clear memory in Redis
-await queue.clearQueuedJobs(); // removes all jobs from the queue
+
+
+## Retries and backoff
+
+Failed jobs are retried with exponential backoff by default. Override the static getters on your job class to tune the behavior.
+
+| Override | Default | Meaning |
+|---|---|---|
+| `static get numRetries()` | `3` | Attempts (including the first try). |
+| `static get retryBackoffType()` | `'exponential'` | Backoff strategy passed to BullMQ. |
+| `static get retryBackoffDelay()` | `2000` | Backoff delay in ms. |
+
+```js
+class FlakyJob extends AbstractJob {
+    static get numRetries()        { return 5; }
+    static get retryBackoffDelay() { return 5000; }
+    // ...
+}
 ```
 
-### WORKERS
-A `Worker` is what runs on the server side, listening to Redis for new jobs in its queue. Starting a worker running on a server is simple.
-``` 
+
+## Running a worker
+
+One worker per process; the worker auto-imports every job under `jobs/`.
+
+```js
+const { JobQueue } = require('@vida-global/core');
+const { Worker }   = JobQueue;
+
 const worker = new Worker('myQueue');
 await worker.listen();
 ```
 
-### JOBS
-A `Job` is the object that contains the code that will be run by the worker. All `Jobs` will subclass `AbstractJob` and implement the `run` and `queueName` methods. `Jobs` should be defined in the `jobs` directory at the top level of the repo. This will allow them to be auto registered by a worker.
-```                                                                                                     
-class MyJob extends AbstractJob {                                                                       
-    async run(arg1, arg2...) {                                                                          
-        // do work here                                                                                 
-    }
-                                                                                                     
-    static queueName(arg1, arg2) {                                                                         
-        return 'myQueue';                                                                            
-    }                                                                                                
-}                                                                                                    
-```                                                                                                  
-Once you've defined your job's `run` and `queueName` methods, you can add it to the queue by running `queueJob`
-```                                                                                                  
-await MyJob.queueJob(arg1, arg2...)
+If `WORKER_QUEUE_NAME` is set in the environment, you can omit the argument and the worker will pick its queue up from there.
+
+### Default job fallback
+
+When a queue entry arrives whose name doesn't match any registered job, the worker can fall back to a job that opts in:
+
+```js
+class CatchAllJob extends AbstractJob {
+    static defaultFor = 'myQueue';
+
+    async run(payload) { /* ... */ }
+}
+```
+
+
+## Concurrency
+
+A single worker can process multiple jobs in parallel. Set the `WORKER_CONCURRENCY` environment variable to raise the limit (default `1`).
+
+```sh
+WORKER_CONCURRENCY=10 node bin/worker.js
+```
+
+
+## Queue operations
+
+Most of the time the queue work happens via `MyJob.queueJob(...)` and you never touch the queue directly. When you do — for monitoring, draining, or graceful shutdown — fetch it with `getQueue`.
+
+```js
+const { JobQueue } = require('@vida-global/core');
+const { getQueue } = JobQueue;
+
+const queue = getQueue('myQueue');
+
+await queue.close();          // close the connection (call before process exit)
+await queue.pause();          // stop dispatching work
+await queue.resume();         // resume after pause
+
+await queue.numQueuedJobs();  // number of jobs waiting
+await queue.getQueuedJobs();  // formatted snapshot of waiting jobs
+await queue.numActiveJobs();
+await queue.getActiveJobs();
+await queue.numFailedJobs();
+await queue.getFailedJobs();
+
+await queue.clearFailedJobs();    // drop failed-job data (do this periodically)
+await queue.clearCompletedJobs(); // drop completed-job data
+await queue.clearQueuedJobs();    // drain all queued jobs
+
+await queue.numWorkers();         // number of workers currently attached
+```
+
+Each `get*Jobs` result entry has this shape:
+
+```js
+{ queueName, name, args, id, attemptsMade, attemptsStarted, progress, failedReason }
 ```
 
-## CONCURRENCY
-While there should only be one `Worker` per process, each worker can process multiple jobs in parallel. Concurrency settings are defined through environment variables.                                                                                                     
 
-### SIMPLE CONCURRENCY                                                                          
-In order to limit the number of jobs that a worker can run in parallel, simply set the `WORKER_CONCURRENCY` environment variable. If that variable is not set, concurrency will default to 1.
+## TODO
 
-### RATE LIMITING
-To be added in a future release.
+- **Rate limiting.** Pre-job rate limiting is not yet wired up.

package/lib/logger/README.md

@@ -1,8 +1,41 @@
 # Logger
-A simple logger that support multiple scopes and log levels. Set the `LOG_LEVEL` environment variable to determine what log levels are output.
+A scoped logger with per-request child loggers and environment-driven verbosity. Every log line carries a timestamp, level, the chain of scopes, and (when present) an id, so output stays readable when many requests interleave.
 
-## Scopes
+
+## Setup
+
+Set the `LOG_LEVEL` environment variable to control the minimum level that is printed. Anything at or above the configured level is logged; anything below is suppressed.
+
+```sh
+LOG_LEVEL=debug node server.js
+```
+
+Level priority (lowest → highest): `silly` < `debug` < `verbose` < `info` < `warn` < `error`. The default is `info`.
+
+
+## Core Concepts
+
+The root `logger` is a `Logger` instance. Call `addScope(name)` to attach a named child scope; that scope behaves like a logger of its own (with its own scope tag in the output) and is exposed as a property on its parent. Scopes nest — adding a scope to a scope produces a deeper tag. Scopes and child loggers are cached internally, so repeated `addScope` / `child` calls with the same key return the same instance.
+
+
+## Usage
+
+### Levels
+
+```js
+const { logger } = require('@vida-global/core');
+
+logger.silly('rarely useful detail');
+logger.debug('developer-focused trace');
+logger.verbose('one level above debug');
+logger.info('the default');
+logger.warn('something deserves attention');
+logger.error('something is broken');
 ```
+
+### Scopes
+
+```js
 logger.addScope('foo');
 logger.addScope('bar');
 
@@ -22,12 +55,22 @@ logger.bar.error('my log');
 // [2026-04-30 16:16:12.797 -0700][ERROR][BAR] my log
 ```
 
-## IDs
-```
+### Child loggers with IDs
+
+`child(id)` returns a logger that tags every line with the given id. This is the standard pattern for per-request logging — the server framework already calls `logger.child(requestId)` for every controller instance.
+
+```js
 logger.addScope('http');
 const requestId     = 123;
 const requestLogger = logger.http.child(requestId);
 
 requestLogger.debug('my log');
 // [2026-04-30 16:16:12.797 -0700][DEBUG][HTTP][123] my log
-``
+```
+
+
+## Advanced
+
+- `addScope` is idempotent: calling it with a name that's already attached returns silently.
+- Scoped and child loggers are memoized by `${scopes.join('|')}:${id}`, so repeated lookups return the same instance — there's no cost to calling `logger.http.child(id)` on every log statement.
+- Colorization, formatting, and transports are configured by overriding the corresponding getters on a `Logger` subclass (`formatters`, `level`, `transports`).