@kedem/okdb 1.0.0

package/docs/queue.md

@@ -0,0 +1,637 @@
+# Queue
+
+OKDB has a built-in durable job queue.
+
+Queue data lives in the **current environment**:
+
+- `okdb.queue` uses the **default** environment
+- `okdb.env('myenv').queue` uses that **custom** environment
+
+Jobs survive restart, are stored as normal OKDB records, and can be listed, edited, retried, and processed by workers.
+
+No Redis, no RabbitMQ. Just OKDB.
+
+---
+
+## Quick start
+
+```javascript
+const OKDB = require('./okdb');
+
+async function main() {
+    const okdb = new OKDB('./db');
+    await okdb.open();
+
+    const worker = okdb.queue.worker(
+        'send-email',
+        async (payload, ctx) => {
+            await ctx.markProgress('sending email');
+
+            await sendEmail(payload.to, payload.subject, payload.orderId);
+
+            await ctx.okdb.put('email_log', payload.orderId, {
+                to: payload.to,
+                sentAt: Date.now(),
+                jobId: ctx.jobId,
+            });
+        },
+        {
+            concurrency: 4,
+            pollInterval: 100,
+            ttl: 30_000,
+        },
+    );
+
+    const jobId = await okdb.queue.enqueue(
+        'send-email',
+        {
+            to: 'alice@example.com',
+            subject: 'Your order shipped',
+            orderId: 'o42',
+        },
+        {
+            max_tries: 3,
+            retry_delay: 1000,
+            backoff_multiplier: 2,
+        },
+    );
+
+    console.log('enqueued', jobId);
+
+    // later
+    await worker.stop();
+    await okdb.close();
+}
+```
+
+---
+
+## Environments
+
+Queue data is **not** stored in a special dedicated `queue/` environment.
+
+Instead:
+
+- `okdb.queue.enqueue(...)` stores queue records in `default`
+- `customEnv.queue.enqueue(...)` stores queue records in `customEnv`
+
+Example:
+
+```javascript
+const okdb = new OKDB('./db');
+await okdb.open();
+
+const jobsEnv = await okdb.createEnvironment('jobs');
+
+await okdb.queue.enqueue('default-task', { a: 1 });
+await jobsEnv.queue.enqueue('custom-task', { b: 2 });
+```
+
+This is useful when you want separate queues per environment.
+
+---
+
+## Enqueuing jobs
+
+```javascript
+const id = await okdb.queue.enqueue('process-image', {
+    fileId: 'f1',
+    userId: 'u1',
+});
+
+console.log(id); // UUID string
+```
+
+### Signature
+
+```javascript
+await okdb.queue.enqueue(type, payload, options);
+```
+
+- `type` — non-empty job type string
+- `payload` — any serialisable value; stored separately from job metadata
+- `options` — scheduling / retry / routing metadata
+
+### Enqueue options
+
+```javascript
+await okdb.queue.enqueue(
+    'process-image',
+    { fileId: 'f1' },
+    {
+        when: Date.now() + 5000, // schedule in the future
+        priority: 10,
+        tags: ['images', 'urgent'],
+        bucket: 'gpu-workers',
+        bucket_tokens: 1,
+        max_tries: 3, // null = unlimited
+        retry_delay: 2000,
+        backoff_multiplier: 2.0,
+        cron: '0 * * * *',
+    },
+);
+```
+
+Notes:
+
+- `priority` is an integer
+- `tags` can be a string or array of strings
+- `bucket` and `bucket_tokens` integrate with token buckets
+- `max_tries: 1` means “try once, then fail permanently”
+- `max_tries: null` means unlimited retries
+
+---
+
+## Processing jobs with `worker(...)`
+
+Use a normal in-process worker like this:
+
+```javascript
+const worker = okdb.queue.worker(
+    'send-email',
+    async (payload, ctx) => {
+        await ctx.markProgress('validating');
+
+        if (!payload.to) throw new Error('missing recipient');
+
+        await ctx.markProgress('sending');
+        await sendEmail(payload.to, payload.subject);
+
+        await ctx.okdb.put('email_log', payload.orderId, {
+            to: payload.to,
+            sentAt: Date.now(),
+            jobId: ctx.jobId,
+        });
+    },
+    {
+        concurrency: 2,
+        pollInterval: 250,
+        ttl: 30_000,
+        onPermanentFail: async (job, err) => {
+            console.error('permanently failed', job.id, err.message);
+        },
+    },
+);
+```
+
+### Handler signature
+
+The handler receives:
+
+```javascript
+async (payload, ctx) => { ... }
+```
+
+Where `ctx` contains:
+
+- `ctx.okdb` — the root `OKDB` instance
+- `ctx.jobId` — queue job id
+- `ctx.heartbeat()` — extend the claim TTL for long-running jobs
+- `ctx.markProgress(message)` — update the job's `progress` field
+
+These local worker helpers now map to the same public queue API that remote workers can use:
+
+- `okdb.queue.claim(type, options)`
+- `okdb.queue.markJobHeartbeat(jobId, claimId, ttl?)`
+- `okdb.queue.markJobProgress(jobId, claimId, message)`
+- `okdb.queue.markJobComplete(jobId, claimId, result?)`
+- `okdb.queue.markJobFail(jobId, claimId, error, code?)`
+
+### Important note
+
+`worker(...)` runs in the **same Node process** unless you explicitly use `spawn(...)` or a separate worker process yourself.
+
+That means:
+
+- great for I/O-heavy jobs
+- not isolation from CPU-heavy work
+
+---
+
+## Worker options
+
+### `pollInterval`
+
+How long to sleep when the queue is empty.
+
+```javascript
+{
+    pollInterval: 100;
+}
+```
+
+### `concurrency`
+
+How many jobs to process in parallel **inside the same process**.
+
+```javascript
+{
+    concurrency: 4;
+}
+```
+
+This is implemented as multiple independent claim/handle lanes.
+
+### `ttl`
+
+Claim timeout in ms.
+
+```javascript
+{
+    ttl: 30_000;
+}
+```
+
+If your worker dies or stops heartbeating, expired running jobs are re-queued by reconciliation.
+
+### `onPermanentFail`
+
+Called after a job permanently fails.
+
+---
+
+## Remote / external workers
+
+If your workers run outside the embedded Node.js process, use the public queue lifecycle methods directly or call the equivalent HTTP endpoints.
+
+### Public queue methods
+
+```javascript
+const claimed = await okdb.queue.claim('send-email', { ttl: 30_000 });
+if (!claimed) return;
+
+await okdb.queue.markJobProgress(claimed.id, claimed.claim_id, 'sending');
+
+try {
+    await sendEmail(claimed.payload.to, claimed.payload.subject);
+    await okdb.queue.markJobComplete(claimed.id, claimed.claim_id, { delivered: true });
+} catch (err) {
+    await okdb.queue.markJobFail(claimed.id, claimed.claim_id, err, 'send_failed');
+}
+```
+
+### Claim ownership contract
+
+Remote lifecycle operations require the active `claim_id` returned by `claim(...)`.
+
+- only the current claim holder may heartbeat, write progress, complete, or fail the job
+- stale workers are rejected once the claim changes or expires
+- this prevents duplicate workers from mutating the same running job
+
+### HTTP endpoints for remote workers
+
+The queue HTTP API now exposes the full remote-worker lifecycle:
+
+- `POST /api/queue/claim/:type`
+- `POST /api/queue/job/:id/heartbeat`
+- `POST /api/queue/job/:id/progress`
+- `POST /api/queue/job/:id/complete`
+- `POST /api/queue/job/:id/fail`
+
+Example:
+
+```text
+// claim
+POST /api/queue/claim/send-email
+{ "options": { "ttl": 30000 } }
+
+// progress
+POST /api/queue/job/<id>/progress
+{ "claim_id": "<claim>", "progress": "sending" }
+
+// complete
+POST /api/queue/job/<id>/complete
+{ "claim_id": "<claim>", "result": { "delivered": true } }
+
+// fail
+POST /api/queue/job/<id>/fail
+{ "claim_id": "<claim>", "code": "send_failed", "message": "smtp timeout" }
+```
+
+---
+
+## Spawned workers
+
+If you want queue handling in a **separate Node process**, use `spawn(...)`:
+
+```javascript
+const spawned = await okdb.queue.spawn('send-email', './workers/email.js', {
+    concurrency: 4,
+    pollInterval: 100,
+});
+
+// later
+await spawned.stop();
+```
+
+The module must export a function:
+
+```javascript
+module.exports = async function (payload, ctx) {
+    await ctx.okdb.put('email_log', payload.orderId, {
+        sentAt: Date.now(),
+        jobId: ctx.jobId,
+    });
+};
+```
+
+The child process opens the same OKDB path and receives the same `ctx.okdb` semantics as a normal worker.
+
+---
+
+## Job lifecycle
+
+```text
+pending → running → done
+                └→ pending (retry)
+                └→ failed
+```
+
+In more detail:
+
+- `pending` — ready to be claimed
+- `running` — claimed by a worker and currently in-flight
+- `done` — completed successfully
+- `failed` — permanently failed after all tries are exhausted
+
+When a running job's claim expires, reconciliation moves it back to `pending`.
+
+---
+
+## Heartbeats and reconciliation
+
+For long-running jobs, call `ctx.heartbeat()` periodically:
+
+```javascript
+okdb.queue.worker(
+    'import',
+    async (payload, ctx) => {
+        for (let i = 0; i < 10; i++) {
+            await processChunk(i);
+            await ctx.heartbeat();
+            await ctx.markProgress(`chunk ${i + 1}/10`);
+        }
+    },
+    { ttl: 5000 },
+);
+```
+
+You can also manually trigger reconciliation:
+
+```javascript
+await okdb.queue._reconcile();
+```
+
+The admin UI also exposes reconcile actions.
+
+---
+
+## Listing jobs
+
+```javascript
+const { items, cursor } = await okdb.queue.list({
+    type: 'send-email',
+    status: 'pending',
+    sort: 'when', // 'when' | 'priority'
+    direction: 'asc',
+    limit: 50,
+    cursor: null,
+});
+```
+
+Supported filters/options:
+
+- `type`
+- `status`
+- `bucket`
+- `tag`
+- `sort`
+- `direction`
+- `limit`
+- `cursor`
+
+---
+
+## Getting and updating one job
+
+### Get a job
+
+```javascript
+const job = await okdb.queue.getJob(jobId);
+
+console.log(job.status);
+console.log(job.payload);
+console.log(job.progress);
+```
+
+`getJob()` returns job metadata **plus payload**.
+
+### Update a job
+
+```javascript
+await okdb.queue.updateJob(jobId, {
+    payload: { fileId: 'f1', force: true },
+    priority: 9,
+    tags: ['edited', 'manual'],
+    progress: 'manually adjusted',
+    status_message: 'edited from admin ui',
+    retry_delay: 500,
+});
+```
+
+Editable fields currently include:
+
+- `type`
+- `payload`
+- `when`
+- `priority`
+- `tags`
+- `bucket`
+- `bucket_tokens`
+- `cron`
+- `max_tries`
+- `retry_delay`
+- `backoff_multiplier`
+- `progress`
+- `status_message`
+
+---
+
+## Retrying, removing, and clearing jobs
+
+### Retry one job
+
+```javascript
+await okdb.queue.retryJob(jobId);
+```
+
+This resets the job back to `pending` and resets `tries` to `0`.
+
+### Remove one job
+
+```javascript
+await okdb.queue.removeJob(jobId);
+```
+
+This deletes both the job metadata and its payload.
+
+### Bulk cleanup helpers
+
+```javascript
+await okdb.queue.clearDone();
+await okdb.queue.clearFailed();
+await okdb.queue.clearStuck();
+await okdb.queue.retryFailed();
+```
+
+Each method also accepts optional `type` and `limit` parameters.
+
+---
+
+## Buckets (token-bucket rate limiting)
+
+Create a bucket:
+
+```javascript
+await okdb.queue.addBucket('gpu-workers', {
+    capacity: 100,
+    tokens: 100,
+    refill_amount: 10,
+    refill_every: 1000,
+});
+```
+
+Update a bucket:
+
+```javascript
+await okdb.queue.updateBucket('gpu-workers', {
+    tokens: 50,
+    refill_amount: 20,
+});
+```
+
+Remove a bucket:
+
+```javascript
+await okdb.queue.removeBucket('gpu-workers');
+```
+
+Manually claim tokens:
+
+```javascript
+const ok = await okdb.queue.tryClaimTokens('gpu-workers', 2);
+```
+
+Jobs can reference a bucket via `enqueue(..., { bucket, bucket_tokens })`.
+
+---
+
+## Cron support
+
+If you use `cron`, you must configure `queue.cron_next`:
+
+```javascript
+const Croner = require('croner');
+
+const okdb = new OKDB('./db', {
+    queue: {
+        cron_next: (expr, fromMs) => {
+            return new Croner(expr, { currentDate: new Date(fromMs) }).nextRun().getTime();
+        },
+    },
+});
+```
+
+Then enqueue recurring work:
+
+```javascript
+await okdb.queue.enqueue(
+    'daily-report',
+    { reportType: 'summary' },
+    {
+        cron: '0 8 * * *',
+    },
+);
+```
+
+---
+
+## Events
+
+Queue lifecycle events are emitted on **`okdb.events`**, not on `okdb.queue.events`.
+
+Current low-frequency emitted events include:
+
+- `queue:enqueued`
+- `queue:done`
+- `queue:failed`
+- `queue:retry`
+- `queue:removed`
+- `queue:reconciled`
+
+Example:
+
+```javascript
+okdb.events.on('queue:done', ({ id, type }) => {
+    console.log('done', id, type);
+});
+
+okdb.events.on('queue:failed', ({ id, type, status_code }) => {
+    console.log('failed', id, type, status_code);
+});
+```
+
+### About filtering by type or key
+
+Today, filtering by job type or job id is best done in your handler:
+
+```javascript
+okdb.events.on('queue:done', (ev) => {
+    if (ev.type !== 'send-email') return;
+    console.log('email job done', ev.id);
+});
+```
+
+If a richer queue-specific event subscription API is added later, it will likely build on top of these base events.
+
+---
+
+## Sync behaviour
+
+Queue data lives in the current environment, so queue sync follows that environment's sync settings.
+
+That means:
+
+- a job enqueued in a synced env can replicate to peers
+- a worker on another node/process can consume from the same synced data
+- retry/failure state also follows the same environment data
+
+If you want queue isolation, use a separate environment with the sync behavior you want.
+
+---
+
+## Queue-worker engine
+
+For persistent, self-managing workers that use stored functions, see **queue-worker engines** in [Pipelines](./pipelines.md).
+
+The queue-worker engine is a pipeline-compatible engine type that:
+
+- claims jobs from the queue automatically
+- invokes a stored function for each claimed payload
+- handles completion/failure using normal queue semantics
+
+Worker `concurrency` controls job throughput. The function runner pool scales automatically to support it — no manual pool sizing is needed.
+
+---
+
+## Admin UI
+
+The admin queue UI supports:
+
+- filtering by status and type
+- retry / cancel / delete
+- bulk clear / bulk retry actions
+- viewing progress
+- editing payload and queue settings for a job
+
+Expanded rows show key metadata, and the edit dialog allows modifying payload and most queue settings without editing raw `~queue_jobs` records directly.