enqiu 0.1.2 → 0.4.0-beta.0

package/README.md

@@ -1,286 +1,194 @@
 # Enqiu
 
-A small, type-safe job queue for Node.js and Bun. Start in memory, move to
-Redis without changing your job API.
+[![npm](https://img.shields.io/npm/v/enqiu/beta?style=flat-square&label=beta)](https://www.npmjs.com/package/enqiu)
+[![status](https://img.shields.io/badge/status-beta-2563eb?style=flat-square)](#status-beta)
+[![built on](https://img.shields.io/badge/built_on-BullMQ-b91c1c?style=flat-square)](https://bullmq.io)
+[![license](https://img.shields.io/npm/l/enqiu?style=flat-square)](LICENSE)
+
+A type-safe job API on top of [BullMQ](https://bullmq.io). Define each job once
+with a schema, then call it like a function — the name, input and result types
+are inferred, so there is no separate registry and no string-keyed dispatch.
+
+BullMQ owns storage, scheduling and execution. Enqiu owns the developer
+experience.
+
+[npm](https://www.npmjs.com/package/enqiu) ·
+[Issues](https://github.com/moji2002/enqiu/issues)
+
+## Status: beta
+
+> [!NOTE]
+> **Enqiu is beta software.** The shape of the API is settled and the hard
+> parts — storage, retries, scheduling and crash recovery — are BullMQ's,
+> which is mature and widely deployed.
+>
+> What is new is the layer in between. Expect edge-case bugs there, and pin an
+> exact version: it is published under the `beta` dist-tag, so a plain
+> `npm install enqiu` will not install it.
+>
+> The layer in between is new. It is covered at 99% of statements and 91% of
+> branches against a real Redis, and the two parts that need no server — the
+> BullMQ vocabulary mapping and the serialization check — are held to their own
+> thresholds in either mode.
 
 ```bash
-pnpm add enqiu
+npm install enqiu@beta bullmq ioredis
 ```
 
-Enqiu has no runtime dependencies. Redis, schema, Hono, and telemetry packages
-remain your choice.
+`bullmq` and `ioredis` are peer dependencies — Enqiu does not pick versions or
+open connections for you.
 
 ## Quick start
 
-Define each job once, then call it like a function. The name, input, and result
-types are inferred.
-
 ```ts
 import { enqiu, job } from "enqiu";
 import { z } from "zod";
 
-const jobs = enqiu({
-  sendEmail: job({
-    input: z.object({
-      to: z.email(),
-      subject: z.string(),
+const { jobs, queue, worker, close } = enqiu(
+  {
+    sendEmail: job({
+      input: z.object({ to: z.string(), subject: z.string() }),
+      retry: { attempts: 3, backoff: { type: "exponential", delay: 500 } },
+      timeout: 30_000,
+      run: async (input, { log }) => {
+        log.info("sending", { to: input.to });
+        return { delivered: true, subject: input.subject };
+      },
     }),
-    run: async (email, { signal, log }) => {
-      log.info("Sending email", { to: email.to });
-
-      const response = await fetch("https://example.com/email", {
-        method: "POST",
-        body: JSON.stringify(email),
-        signal,
-      });
-
-      return { delivered: response.ok };
-    },
-  }),
-});
-
-const delivery = await jobs.sendEmail({
-  to: "hello@example.com",
-  subject: "Welcome",
-});
-
-const result = await delivery.result;
-console.log(result.delivered);
-```
-
-`await jobs.sendEmail(input)` waits until the queue accepts the job and returns
-a handle. It does not wait for the handler. Await `handle.result` only when the
-caller needs the result. Ignoring a handle is safe and does not create an
-unhandled rejected promise.
-
-Schemas are optional. A plain handler also infers its result:
-
-```ts
-const jobs = enqiu({
-  resizeImage: async (input: { key: string; width: number }) => {
-    return { key: input.key, width: input.width };
   },
-});
-```
-
-## Testing in your project
-
-Create a fresh queue for each test so workers and queued state never leak
-between cases. This Vitest example uses the in-memory driver, awaits the public
-job handle, checks the persisted status, and always closes the worker:
-
-```ts
-import { afterEach, describe, expect, it } from "vitest";
-import { createJobs } from "./jobs.js";
-
-let jobs: ReturnType<typeof createJobs> | undefined;
-
-afterEach(async () => {
-  await jobs?.worker.close();
-  jobs = undefined;
-});
-
-describe("email jobs", () => {
-  it("returns and stores the handler result", async () => {
-    jobs = createJobs();
-    const handle = await jobs.sendWelcome({ name: "Ada" });
-
-    await expect(handle.result).resolves.toEqual({ subject: "Welcome, Ada" });
-    expect((await handle.refresh()).status).toBe("succeeded");
-  });
-});
-```
-
-Install and run it with:
+  {
+    name: "notifications",
+    connection: { host: "localhost", port: 6379 },
+    worker: { concurrency: 10 },
+  },
+);
 
-```bash
-pnpm add enqiu
-pnpm add -D vitest
-pnpm vitest run
+const handle = await jobs.sendEmail({ to: "a@b.c", subject: "Welcome" });
+const result = await handle.result;   // { delivered: boolean; subject: string }
 ```
 
-The repository keeps the complete, runnable
-[memory and opt-in Redis examples](https://github.com/moji2002/enqiu/tree/main/examples/testing).
-The Redis test runs only when `ENQIU_TEST_REDIS_URL` is set, uses a unique
-namespace instead of flushing the database, and closes the worker before the
-injected Redis client.
-
-## Redis
+`await jobs.sendEmail(input)` resolves once BullMQ accepts the job and returns a
+handle. It does not wait for the handler. Await `handle.result` only when the
+caller needs the result.
 
-Inject an existing client; Enqiu does not create connections or install a Redis
-library. It accepts Bun's `send(command, args)` client shape and node-redis'
-`sendCommand(args)` shape.
+`jobs` holds your jobs and nothing else, which is why no job name is reserved —
+`jobs.queue` is a job you called `queue`. The queue and worker controls sit
+beside it:
 
 ```ts
-import { createClient } from "redis";
-import { enqiu, redis } from "enqiu";
-
-const client = createClient({ url: process.env.REDIS_URL });
-await client.connect();
-
-const jobs = enqiu(definitions, {
-  name: "notifications",
-  driver: redis(client),
-  worker: { concurrency: 20 },
-});
+await queue.stats();      // counts by status
+await queue.onIdle();     // resolves when nothing is outstanding
+await close();            // queue, worker and event stream
 ```
 
-Use the same definitions in a producer-only process:
+Only what Enqiu types or computes is here. Pausing a queue or a worker, setting
+global concurrency and anything else BullMQ already exposes is `bull.queue.*`
+and `bull.worker.*` — a second name for the same call would be one more thing
+to learn and nothing else.
 
-```ts
-const jobs = enqiu(definitions, {
-  name: "notifications",
-  driver: redis(client),
-  worker: false,
-});
-```
-
-Redis jobs use atomic Lua transitions, visibility leases, and deterministic
-recovery so multiple Node.js or Bun workers can safely share a queue.
-
-## Job policies
-
-Policies live beside the handler and keep call sites clean. Durations are
-numbers in milliseconds, so applications may use plain numbers or a helper
-such as `ms("30s")` without making it an Enqiu dependency.
+A plain handler works too, with input and output still inferred:
 
 ```ts
-const jobs = enqiu({
-  syncAccount: job({
-    input: z.object({
-      tenantId: z.string(),
-      accountId: z.string(),
-    }),
-    retry: {
-      attempts: 5,
-      backoff: { type: "exponential", delay: 250, jitter: 0.2 },
-    },
-    timeout: 30_000,
-    expiresIn: 5 * 60_000,
-    concurrency: {
-      limit: 2,
-      by: (input) => input.tenantId,
-    },
-    throttle: {
-      limit: 100,
-      per: 60_000,
-      burst: 10,
-      by: (input) => input.tenantId,
-    },
-    run: async (input, context) => {
-      return syncAccount(input, { signal: context.signal });
-    },
-  }),
-});
+const { jobs } = enqiu(
+  { resizeImage: async (input: { key: string; width: number }) => input },
+  { connection },
+);
 ```
 
-- `concurrency` limits simultaneous work globally or by a key such as tenant.
-- `throttle` limits starts over time; `burst` allows short spikes.
-- `debounce: { mode: "leading" }` keeps the first call in a window.
-- `debounce: { mode: "trailing" }` keeps the most recent call in a window.
-- `expiresIn` prevents stale jobs from starting.
-- `idempotencyKey` makes repeated submissions return the same job.
-
-Per-call delivery options are available when needed:
+## What Enqiu adds
+
+- **Inferred types end to end.** Job names come from the object keys; input and
+  result types come from the schema and handler. No generics to write.
+- **Standard Schema validation at the boundary.** Zod, Valibot, ArkType or
+  anything else implementing the spec. Invalid input is rejected before a job
+  is queued.
+- **Per-attempt `timeout`,** with an `AbortSignal` handed to the handler. BullMQ
+  has no job timeout; Enqiu enforces this itself.
+- **`expiresIn`,** which fails a job that waited too long without running it.
+  Also enforced by Enqiu.
+- **A serialization guard** that rejects functions, symbols, cycles and sparse
+  arrays with the exact path, instead of failing later inside the queue.
+- **A `cancelled` status,** which BullMQ has no state for: cancelling a job that
+  has not started removes it, so Enqiu records the finished snapshot and
+  `refresh()` can still tell "cancelled" from "never existed".
+- **Failures that survive as classes.** BullMQ hands a failure to another
+  process as one string; a timeout or an expiry writes its kind down, so
+  `handle.result` rejects with `JobTimeoutError` rather than a bare `Error`.
+
+## Escaping the layer
+
+Enqiu models a deliberate subset. Everything else BullMQ can do — flows, Pro
+groups, metrics, raw job options — is one property away, with no wrapper in
+between and no fork required:
 
 ```ts
-const handle = await jobs.syncAccount(input, {
-  idempotencyKey: `sync:${input.accountId}`,
-  idempotencyTtl: 24 * 60 * 60_000,
-  delay: 5_000,
-  priority: "high",
-});
+const { bull } = enqiu(definitions, { connection });
+
+bull.queue    // the real BullMQ Queue
+bull.worker   // the real BullMQ Worker, or undefined for a producer
 ```
 
-## Progress and logs
+Enqiu reads its own state from those objects rather than mirroring it, so
+pausing `bull.worker` or closing `bull.queue` is seen on the Enqiu side too —
+the two cannot drift apart.
 
-Progress uses real units rather than an ambiguous fraction:
+Measured against raw BullMQ on the same Redis — 10,000 jobs, concurrency 32,
+contestants interleaved, median of 7 — the typed path costs about 2%, and Zod
+validation about 3%, varying by a point between runs. Calls through `bull` cost
+nothing, because nothing is in the way.
 
-```ts
-const jobs = enqiu({
-  importRows: async (rows: string[], context) => {
-    for (let index = 0; index < rows.length; index += 1) {
-      await importRow(rows[index]);
-      await context.reportProgress({
-        completed: index + 1,
-        total: rows.length,
-        message: "Importing rows",
-      });
-    }
-
-    context.log.info("Import complete", { rows: rows.length });
-  },
-});
-```
+Reproduce with `pnpm tsx bench/overhead.ts`.
 
-Subscribe to lifecycle events with `jobs.queue.on(...)`. Memory events stay
-inside the process; Redis events are shared between producers and workers.
+## What BullMQ provides
 
-## Cron schedules
+Retries and backoff, priorities, delays, cron schedules, deduplication, bulk
+submission, progress, logs, events and cleanup are BullMQ's, surfaced through
+Enqiu's API.
 
-Schedules use standard five-field cron expressions and IANA time zones:
+## Compatibility notes
 
-```ts
-const schedule = await jobs.sendDigest.schedule({
-  id: "weekday-digest",
-  cron: "0 9 * * 1-5",
-  timezone: "Europe/Nicosia",
-  input: { audience: "daily" },
-  catchUp: true,
-});
-
-await schedule.pause();
-await schedule.resume();
-await schedule.remove();
-```
+Enqiu deliberately does not paper over gaps in BullMQ's open-source tier:
 
-Memory schedules live for the process lifetime. Redis schedules are durable
-and use deterministic occurrence IDs to avoid duplicate runs.
+| Not available | Why |
+| --- | --- |
+| Per-key concurrency (`concurrency: { by }`) | BullMQ groups are a **BullMQ Pro** feature. |
+| Per-key rate limiting (`throttle: { by }`) | The OSS limiter is one global `{ max, duration }` per worker. |
+| Debounce | No open-source equivalent. |
+| In-browser queues | BullMQ requires Redis and Node. |
 
-## Hono
+If you need any of those, use BullMQ Pro directly, or pin Enqiu 0.2.x, which
+shipped first-party memory and Redis drivers that implemented them.
 
-Enqiu uses Standard Schema and exposes each job's input schema, so the same
-schema can validate an HTTP route without redefining a type:
+## Testing
 
-```ts
-import { sValidator } from "@hono/standard-validator";
-
-app.post(
-  "/emails",
-  sValidator("json", jobs.sendEmail.input),
-  async (c) => {
-    const handle = await jobs.sendEmail(c.req.valid("json"));
-    return c.json({ id: handle.id }, 202);
-  },
-);
+```bash
+docker run -d -p 6379:6379 redis:7-alpine
+ENQIU_TEST_REDIS_URL=redis://localhost:6379 pnpm run check
+ENQIU_TEST_REDIS_URL=redis://localhost:6379 pnpm run scenarios
 ```
 
-Hono and `@hono/standard-validator` are optional application dependencies.
+Most tests need a real Redis, because most code paths go through BullMQ. The
+exceptions are the vocabulary mapping and the serialization check, which are
+pure and stay covered without a server — so a run without
+`ENQIU_TEST_REDIS_URL` still verifies something rather than nothing.
 
-## Queue and worker controls
+Five runnable, self-asserting scenarios live in
+[`examples/scenarios/`](examples/scenarios): webhook ingestion, notification
+campaigns, report progress, a transcoding pool, and failure triage. The
+reasoning behind the workload choices is in
+[`docs/use-case-research.md`](docs/use-case-research.md).
 
-```ts
-await jobs.queue.pause();
-await jobs.queue.resume();
-await jobs.queue.setConcurrency(50);
-
-const page = await jobs.queue.list({ status: "failed", limit: 100 });
-const snapshot = await jobs.queue.get(handle.id);
-await jobs.queue.redrive(handle.id);
-await jobs.queue.cleanup({ olderThan: Date.now() - 7 * 24 * 60 * 60_000 });
-
-await jobs.worker.pause();
-await jobs.worker.resume();
-await jobs.worker.onIdle();
-await jobs.worker.close();
-```
+## Releasing
 
-## Runtime support
+```bash
+pnpm run release:beta        # npm publish --tag beta
+```
 
-- Node.js 20 and newer
-- Current stable Bun
-- Memory and Redis drivers
-- ESM and TypeScript declarations
+The tag is in the script rather than in `publishConfig`, because npm 11 does not
+honour `publishConfig.tag` — a plain `npm publish` resolves to `latest` and would
+hand a beta to every `npm install enqiu`. `prepack` runs the full check first,
+and `build` cleans `dist` before compiling, since `tsc` leaves deleted modules
+behind and they would otherwise ship.
 
 ## License