nuxt-cf-jobs 0.6.3 → 0.7.0

package/README.md

@@ -25,7 +25,7 @@ Typed Cloudflare Queue jobs for Nuxt, with Laravel-style ergonomics.
 - 🗄️ **Durable D1 jobs**: persist a record before enqueue so work survives restarts and delivery gaps, with retry, release, and DLQ.
 - ⏰ **Scheduled tasks**: co-locate a cron with its handler; `nitro.tasks`, `scheduledTasks`, and Cloudflare `triggers.crons` are derived from it.
 - 🧪 **Laravel-style testing**: run handlers inline, fake the queue, drain the outbox, or drive the whole `queue:work` loop on a virtual clock.
-- 🛠️ **`cf-jobs` CLI**: `artisan queue:*`-style status, retry, flush, and migrate against local or remote D1.
+- 🛠️ **`cf-jobs` CLI**: `artisan queue:*`-style status, retry, flush, and migrate against local or remote D1, plus a `work` dev worker that runs durable jobs out-of-band so WebSockets stream live progress in `nuxt dev`.
 
 ## Install
 
@@ -331,6 +331,32 @@ Every command accepts `--config <wrangler path>`, `--db <binding>`, `--remote`,
 
 `cf-jobs` shells out to `wrangler`, resolving the binary from `node_modules/.bin` and falling back to `wrangler` on `PATH` (override with `CF_JOBS_WRANGLER_BIN`).
 
+### Dev worker (`cf-jobs work`)
+
+In production the queue consumer is a separate Worker invocation, so a running job is naturally decoupled from the request that enqueued it. Under `nuxt dev` everything shares one process, so a durable job dispatched on a request can run to completion inside that same request, before a client has a chance to observe it. That hides the asynchronous behaviour you actually want to test, most painfully a WebSocket streaming live job progress: the job finishes before the socket is even connected.
+
+`cf-jobs work` restores that decoupling. It is a long-running dev worker that drains durable jobs **out-of-band**, on its own clock, by driving the running dev server:
+
+```bash
+# in one terminal
+pnpm nuxt dev
+
+# in another: poll the dev server, run whatever durable jobs are ready
+pnpm cf-jobs work
+pnpm cf-jobs work --queue sync-critical   # only one logical queue
+pnpm cf-jobs work --once                  # drain everything ready now, then exit (handy in scripts/CI)
+pnpm cf-jobs work --interval 1000         # idle poll interval in ms (backs off to 5s); default 500
+```
+
+So a client connects to your WebSocket, enqueues a job (the request returns immediately, job persisted in D1), then `cf-jobs work` picks it up a tick later and runs it. Because the worker drives a dev-only endpoint (`POST /__cf-jobs/work`) that fires your app's registered `cloudflare:queue` consumer **in the dev process**, the job runs with the app's real context and an in-memory WebSocket broadcast reaches the connected client, exactly as it would in production via a Durable Object.
+
+Concurrency matches your queue config: each logical queue drains in batches of its wrangler `max_batch_size`, with up to `max_concurrency` batches in flight (default: serial, batch of 10). A queue declaring `{ maxConcurrency: 4, maxBatchSize: 10 }` drains 10-at-a-time, 4 batches concurrent, like its production consumer.
+
+Two things to know:
+
+- This is a **`nuxt dev` companion, not a production worker.** It assumes one process, so the request that runs the job shares memory (and thus WebSocket maps) with the rest of the dev server. Under multi-isolate setups (e.g. `wrangler dev`) the broadcast can land in a different isolate. The `/__cf-jobs/work` endpoint is registered only in dev and is an unauthenticated job executor, so it is never built into a deployment.
+- Unlike the D1-querying commands above, `work` talks HTTP to the running dev server. It takes `--url` (default `http://localhost:3000`), `--db <binding>` to disambiguate when several D1 bindings exist, and `--json` to emit one machine-readable line per active tick.
+
 ## Runtime Validation
 
 The generated registry validates jobs at startup and fails loudly for invalid definitions, duplicate names, and missing or invalid queue names. Queue binding checks are available from `#cf-jobs/app`:

package/dist/cli/index.mjs

@@ -698,13 +698,104 @@ const tasks = defineCommand({
     });
   }
 });
+const TRAILING_SLASH_RE = /\/+$/;
+function sleep(ms) {
+  return new Promise((resolve) => setTimeout(resolve, ms));
+}
+const work = defineCommand({
+  meta: {
+    name: "work",
+    description: "Dev worker: drive a running `nuxt dev` server to drain durable jobs out-of-band so WebSockets observe live progress (nuxt dev only \u2014 NOT a production worker)"
+  },
+  args: {
+    url: { type: "string", description: "Dev server base URL", default: "http://localhost:3000" },
+    interval: { type: "string", description: "Poll interval (ms) when idle; backs off up to 5s", default: "500" },
+    queue: { type: "string", description: "Only drain this logical queue" },
+    limit: { type: "string", description: "Max jobs claimed per tick", default: "100" },
+    db: { type: "string", description: "D1 binding name (default: auto-detect the only binding)" },
+    once: { type: "boolean", description: "Drain everything ready now, then exit", default: false },
+    json: { type: "boolean", description: "Emit one JSON line per active tick", default: false }
+  },
+  async run({ args }) {
+    const base = args.url.replace(TRAILING_SLASH_RE, "");
+    const params = new URLSearchParams({ limit: String(Number(args.limit) || 100) });
+    if (args.queue)
+      params.set("queue", args.queue);
+    if (args.db)
+      params.set("db", args.db);
+    const endpoint = `${base}/__cf-jobs/work?${params}`;
+    const baseInterval = Math.max(50, Number(args.interval) || 500);
+    const maxInterval = Math.max(baseInterval, 5e3);
+    let stopped = false;
+    const stop = () => {
+      stopped = true;
+    };
+    process$1.on("SIGINT", stop);
+    process$1.on("SIGTERM", stop);
+    if (!args.once)
+      process$1.stderr.write(`${color.dim(`cf-jobs work \u2192 ${endpoint} (Ctrl-C to stop)`)}
+`);
+    let warnedAmbiguous = false;
+    let idle = baseInterval;
+    for (; ; ) {
+      if (stopped)
+        break;
+      const tick = await fetch(endpoint, { method: "POST" }).then((r) => r.json()).catch((error) => {
+        process$1.stderr.write(`${color.yellow("\u2026")} dev server unreachable at ${base} (${error instanceof Error ? error.message : String(error)})
+`);
+        return null;
+      });
+      if (!tick || tick.error) {
+        if (tick?.error)
+          process$1.stderr.write(`${color.red("\u2716")} ${tick.error === "no-d1-binding" ? "no D1 binding found on the dev server env" : tick.error}
+`);
+        if (args.once) {
+          process$1.exitCode = 1;
+          break;
+        }
+        await sleep(maxInterval);
+        continue;
+      }
+      if (tick.ambiguousBindings && !warnedAmbiguous) {
+        warnedAmbiguous = true;
+        process$1.stderr.write(`${color.yellow("!")} multiple D1 bindings (${tick.ambiguousBindings.join(", ")}); using the first. Pass --db to choose.
+`);
+      }
+      if (tick.processed > 0) {
+        idle = baseInterval;
+        if (args.json) {
+          process$1.stdout.write(`${JSON.stringify(tick)}
+`);
+        } else {
+          const detail = tick.byQueue && Object.keys(tick.byQueue).length ? ` ${color.dim(`(${Object.entries(tick.byQueue).map(([q, n]) => `${q}:${n}`).join(", ")})`)}` : "";
+          const waiting = tick.remaining > 0 ? color.dim(` \u2014 ${tick.remaining} waiting`) : "";
+          process$1.stdout.write(`${color.green("\u2713")} ran ${tick.processed}${detail}${waiting}
+`);
+        }
+      }
+      if (args.once) {
+        if (tick.processed === 0 && tick.remaining === 0)
+          break;
+        continue;
+      }
+      if (tick.processed === 0) {
+        await sleep(idle);
+        idle = Math.min(maxInterval, idle * 2);
+      } else {
+        await sleep(baseInterval);
+      }
+    }
+    process$1.off("SIGINT", stop);
+    process$1.off("SIGTERM", stop);
+  }
+});
 const main = defineCommand({
   meta: {
     name: "cf-jobs",
     description: "Inspect and manage nuxt-cf-jobs durable jobs in Cloudflare D1"
   },
   args: sharedArgs,
-  subCommands: { status, jobs, failed, retry, forget, flush, clear, prune, migrate, schedule, tasks },
+  subCommands: { status, jobs, failed, retry, forget, flush, clear, prune, migrate, schedule, tasks, work },
   async run({ args, rawArgs }) {
     if (rawArgs.length === 0 || rawArgs.every((a) => a.startsWith("-")))
       await runStatus(args);

package/dist/module.json

@@ -1,7 +1,7 @@
 {
   "name": "nuxt-cf-jobs",
   "configKey": "cfJobs",
-  "version": "0.6.3",
+  "version": "0.7.0",
   "builder": {
     "@nuxt/module-builder": "1.0.2",
     "unbuild": "3.6.1"

package/dist/module.mjs

@@ -1,7 +1,7 @@
 import { existsSync } from 'node:fs';
 import { readFile } from 'node:fs/promises';
 import { resolve, relative, sep } from 'node:path';
-import { defineNuxtModule, createResolver, addServerImports, addTemplate, addTypeTemplate, updateTemplates, useLogger, addServerPlugin, resolveFiles } from '@nuxt/kit';
+import { defineNuxtModule, createResolver, addServerImports, addTemplate, addTypeTemplate, updateTemplates, useLogger, addServerPlugin, addServerHandler, resolveFiles } from '@nuxt/kit';
 import { parseModule } from 'magicast';
 import { cfJobsAppExportNames } from '../dist/runtime/server/app.js';
 import { c as collectTasks, f as findDuplicateTaskNames, b as buildCronUnion, a as buildScheduledTasks, g as renderSuggestedCronsToml, e as findWranglerConfig, p as parseWranglerConfig, d as crossCheckCrons, r as reconcileQueues } from './shared/nuxt-cf-jobs.BkJA3gwQ.mjs';
@@ -158,8 +158,14 @@ const module$1 = defineNuxtModule({
       queues: nuxt.options.runtimeConfig.cfJobs?.queues ?? options.queues,
       defaultQueue: nuxt.options.runtimeConfig.cfJobs?.defaultQueue ?? options.defaultQueue
     };
-    if (nuxt.options.dev)
+    if (nuxt.options.dev) {
       addServerPlugin(resolver.resolve("./runtime/server/plugins/dev-queues"));
+      addServerHandler({
+        route: "/__cf-jobs/work",
+        method: "post",
+        handler: resolver.resolve("./runtime/server/handlers/dev-work")
+      });
+    }
     if (options.validateWrangler !== false)
       runWranglerCrossCheck(options, nuxt.options.rootDir, resolve(nuxt.options.buildDir, "cf-jobs"), nuxt.options.nitro);
     await wireScheduledTasks(options, nuxt, resolve(nuxt.options.buildDir, "cf-jobs"));

package/dist/runtime/server/dev-worker.d.ts

@@ -0,0 +1,53 @@
+import type { D1DatabaseLike } from './d1.js';
+import type { QueueJobMessage } from './outbox.js';
+export interface DevWorkerQueueConfig {
+    /** Wrangler `max_concurrency`: how many batches of this queue drain at once. */
+    maxConcurrency: number;
+    /** Wrangler `max_batch_size`: messages handed to one consumer invocation. */
+    maxBatchSize: number;
+}
+export interface DevWorkerDeps<Queue extends string = string> {
+    /** Ready (unreserved, available, not terminal) durable jobs, oldest first, up to `limit`. */
+    findDispatchable: (limit: number) => Promise<ReadonlyArray<{
+        id: string;
+        queue: Queue;
+    }>>;
+    /**
+     * Drive one consumer invocation in-process — fires the `cloudflare:queue` hook
+     * and AWAITS it, so the handler runs (and broadcasts) before the tick returns.
+     */
+    dispatchBatch: (queue: Queue, messages: ReadonlyArray<QueueJobMessage<Queue>>) => Promise<void>;
+    /** Per-queue sizing — "match the queue" off the wrangler consumer config. */
+    queueConfig: (queue: Queue) => DevWorkerQueueConfig;
+}
+export interface DevWorkerTickResult {
+    /** Jobs handed to a consumer this tick (≈ processed; the consumer ran them inline). */
+    processed: number;
+    /** Per-logical-queue counts. */
+    byQueue: Record<string, number>;
+    /** Ready jobs still waiting after this tick (lets the CLI decide to keep going). */
+    remaining: number;
+}
+export interface DevWorkerTickOptions {
+    /** Max jobs claimed per tick. */
+    limit: number;
+    /** Restrict the tick to a single logical queue. */
+    queue?: string;
+}
+/** Derive per-queue sizing from a `cfJobs.queues` entry (string binding or options object). */
+export declare function resolveQueueWorkerConfig(entry: string | {
+    maxConcurrency?: number;
+    maxBatchSize?: number;
+} | undefined): DevWorkerQueueConfig;
+export declare function chunk<T>(items: readonly T[], size: number): T[][];
+/** Run `tasks` with at most `concurrency` in flight (a simple slot pool). */
+export declare function runWithConcurrency<T>(tasks: ReadonlyArray<() => Promise<T>>, concurrency: number): Promise<T[]>;
+export declare function runDevWorkerTick<Queue extends string = string>(deps: DevWorkerDeps<Queue>, opts: DevWorkerTickOptions): Promise<DevWorkerTickResult>;
+export interface D1BindingMatch {
+    binding: string;
+    db: D1DatabaseLike;
+    /** Set when more than one D1 binding exists and the first was picked. */
+    ambiguous?: string[];
+}
+/** Resolve the durable-jobs D1 database off the runtime env (auto-detect, or by name). */
+export declare function findD1Binding(env: Record<string, unknown>, preferred?: string): D1BindingMatch | undefined;

package/dist/runtime/server/dev-worker.js

@@ -0,0 +1,76 @@
+const DEFAULT_MAX_BATCH_SIZE = 10;
+const DEFAULT_MAX_CONCURRENCY = 1;
+export function resolveQueueWorkerConfig(entry) {
+  if (!entry || typeof entry === "string")
+    return { maxConcurrency: DEFAULT_MAX_CONCURRENCY, maxBatchSize: DEFAULT_MAX_BATCH_SIZE };
+  return {
+    maxConcurrency: clampPositive(entry.maxConcurrency, DEFAULT_MAX_CONCURRENCY),
+    maxBatchSize: clampPositive(entry.maxBatchSize, DEFAULT_MAX_BATCH_SIZE)
+  };
+}
+function clampPositive(value, fallback) {
+  return typeof value === "number" && Number.isFinite(value) && value >= 1 ? Math.floor(value) : fallback;
+}
+export function chunk(items, size) {
+  const step = Math.max(1, size);
+  const out = [];
+  for (let i = 0; i < items.length; i += step)
+    out.push(items.slice(i, i + step));
+  return out;
+}
+export async function runWithConcurrency(tasks, concurrency) {
+  const limit = Math.max(1, concurrency);
+  const results = Array.from({ length: tasks.length });
+  let cursor = 0;
+  async function worker() {
+    for (; ; ) {
+      const index = cursor++;
+      if (index >= tasks.length)
+        return;
+      results[index] = await tasks[index]();
+    }
+  }
+  await Promise.all(Array.from({ length: Math.min(limit, tasks.length) }, worker));
+  return results;
+}
+export async function runDevWorkerTick(deps, opts) {
+  const ready = await deps.findDispatchable(opts.limit);
+  const filtered = opts.queue ? ready.filter((r) => r.queue === opts.queue) : ready;
+  const byQueue = {};
+  if (filtered.length === 0)
+    return { processed: 0, byQueue, remaining: 0 };
+  const groups = /* @__PURE__ */ new Map();
+  for (const record of filtered) {
+    const messages = groups.get(record.queue) ?? [];
+    messages.push({ jobId: record.id, queue: record.queue });
+    groups.set(record.queue, messages);
+  }
+  let processed = 0;
+  await Promise.all([...groups].map(async ([queue, messages]) => {
+    const { maxConcurrency, maxBatchSize } = deps.queueConfig(queue);
+    const batches = chunk(messages, maxBatchSize);
+    await runWithConcurrency(
+      batches.map((batch) => () => deps.dispatchBatch(queue, batch)),
+      maxConcurrency
+    );
+    byQueue[queue] = messages.length;
+    processed += messages.length;
+  }));
+  const after = await deps.findDispatchable(opts.limit);
+  const remaining = opts.queue ? after.filter((r) => r.queue === opts.queue).length : after.length;
+  return { processed, byQueue, remaining };
+}
+export function findD1Binding(env, preferred) {
+  if (preferred) {
+    const db = env[preferred];
+    return isD1Database(db) ? { binding: preferred, db } : void 0;
+  }
+  const matches = Object.entries(env).filter((entry) => isD1Database(entry[1]));
+  const first = matches[0];
+  if (!first)
+    return void 0;
+  return matches.length > 1 ? { binding: first[0], db: first[1], ambiguous: matches.map(([name]) => name) } : { binding: first[0], db: first[1] };
+}
+function isD1Database(value) {
+  return !!value && typeof value === "object" && typeof value.prepare === "function" && typeof value.exec === "function";
+}

package/dist/runtime/server/handlers/dev-work.d.ts

@@ -0,0 +1,10 @@
+/**
+ * Dev-only worker endpoint. Registered ONLY when `nuxt.options.dev` (see
+ * module.ts) and guarded again here — it is an unauthenticated job executor by
+ * design and must never reach a deployment. Driven by `cf-jobs work`, it finds
+ * ready durable jobs in D1 and runs them through the app's `cloudflare:queue`
+ * consumer IN THIS dev process, so an already-connected WebSocket sees live
+ * progress. See `dev-worker.ts` for the rationale.
+ */
+declare const _default: any;
+export default _default;

package/dist/runtime/server/handlers/dev-work.js

@@ -0,0 +1,54 @@
+import { defineEventHandler, getQuery, useNitroApp, useRuntimeConfig } from "nitropack/runtime";
+import { createD1DurableJobRepository } from "../d1.js";
+import { findD1Binding, resolveQueueWorkerConfig, runDevWorkerTick } from "../dev-worker.js";
+import { findDispatchableDurableJobs } from "../outbox.js";
+export default defineEventHandler(async (event) => {
+  if (!import.meta.dev)
+    return { processed: 0, byQueue: {}, remaining: 0, error: "dev-only" };
+  const query = getQuery(event);
+  const limit = clampInt(query.limit, 100, 1, 1e3);
+  const onlyQueue = pickString(query.queue);
+  const preferredDb = pickString(query.db);
+  const env = resolveEnv(event);
+  const d1 = findD1Binding(env, preferredDb);
+  if (!d1)
+    return { processed: 0, byQueue: {}, remaining: 0, error: "no-d1-binding" };
+  const repo = createD1DurableJobRepository(d1.db);
+  const nitroApp = useNitroApp();
+  const queues = useRuntimeConfig().cfJobs?.queues ?? {};
+  const result = await runDevWorkerTick({
+    findDispatchable: (max) => findDispatchableDurableJobs(repo, { limit: max }),
+    queueConfig: (queue) => resolveQueueWorkerConfig(queues[queue]),
+    async dispatchBatch(queue, messages) {
+      await nitroApp.hooks.callHook("cloudflare:queue", {
+        batch: {
+          queue,
+          messages: messages.map((body) => ({ id: body.jobId, body, attempts: 1, ack() {
+          }, retry() {
+          } })),
+          ackAll() {
+          },
+          retryAll() {
+          }
+        },
+        env
+      });
+    }
+  }, { limit, queue: onlyQueue });
+  return d1.ambiguous ? { ...result, d1Binding: d1.binding, ambiguousBindings: d1.ambiguous } : result;
+});
+function resolveEnv(event) {
+  const fromEvent = event.context?.cloudflare?.env;
+  const fromGlobal = globalThis.__env__;
+  return { ...fromGlobal ?? {}, ...fromEvent ?? {} };
+}
+function pickString(value) {
+  const raw = Array.isArray(value) ? value[0] : value;
+  return typeof raw === "string" && raw.length > 0 ? raw : void 0;
+}
+function clampInt(value, fallback, min, max) {
+  const n = Number(Array.isArray(value) ? value[0] : value);
+  if (!Number.isFinite(n))
+    return fallback;
+  return Math.min(max, Math.max(min, Math.floor(n)));
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "nuxt-cf-jobs",
   "type": "module",
-  "version": "0.6.3",
+  "version": "0.7.0",
   "description": "Nuxt module for typed Cloudflare queue jobs.",
   "author": {
     "name": "Harlan Wilton",