@spooled/sdk 1.2.1 → 1.4.0

package/README.md

@@ -21,6 +21,7 @@ Official Node.js/TypeScript SDK for [Spooled Cloud](https://spooled.cloud) - a m
 - **Organization management** with usage tracking
 - **Webhook management** with automatic retries
 - **Dead Letter Queue (DLQ)** operations
+- **Server-side job search** (status lists, text, job type, creation time), paging with `hasMore`, and **job history**
 - **API Key management** with Redis caching (~28x faster authentication)
 - **Billing integration** via Stripe
 - **Automatic case conversion** between camelCase (SDK) and snake_case (API)
@@ -109,6 +110,24 @@ const { id } = await client.jobs.create({
 });
 ```
 
+Find jobs on the server and page through them (API 0.1.122+):
+
+```typescript
+const page = await client.jobs.listPage({
+  status: ["failed", "deadletter"], // one status or several
+  q: "invoice", // id prefix, or text in the queue name or payload.job_type
+  jobType: "send-invoice", // exact payload.job_type
+  createdAfter: new Date(Date.now() - 86_400_000), // Date or RFC 3339 string
+  limit: 100,
+});
+console.log(page.data.length, page.hasMore); // hasMore comes from X-Has-More
+
+// What happened to a job: claims, completions, retries, dead-lettering
+const history = await client.jobs.history(id, { order: "desc" });
+```
+
+`jobs.list()` takes the same filters and still returns a plain array.
+
 ### Workers
 
 Process jobs with the built-in worker runtime:
@@ -136,10 +155,30 @@ await worker.start();
 process.on("SIGTERM", () => worker.stop());
 ```
 
+`stop()` drains. The worker stops claiming, then lets running handlers finish for up to
+`shutdownTimeout` (default 30 s). A handler that returns is completed and one that throws is
+failed, even though the stop was requested meanwhile; `ctx.signal` is not aborted during that
+grace period. Handlers still running when it ends are aborted and failed with "Worker shutdown
+timeout". The worker deregisters last.
+
+If a job has `timeoutSeconds`, the worker aborts the handler (`ctx.signal`) when it runs longer,
+emits `job:timeout`, and fails the job with "Job timeout exceeded". The error text sent with a
+failure is always made acceptable to the API: empty becomes "Job failed", and text over 2048
+characters is truncated.
+
+`complete`, `fail` and both heartbeats are retried on network errors, timeouts and 5xx, because
+the server makes them idempotent. If an ack still fails after the retries the worker emits `error`.
+
 Give any worker that restarts a stable `workerId` (1-128 chars, `[A-Za-z0-9._-]`). Registration then
 upserts a single row instead of minting a new UUID each start, which otherwise leaves the previous
 registration counting against your plan's worker limit until the stale-worker reaper clears it
-(~2 minutes).
+(~2 minutes). `workers.get()` and `workers.list()` report `jobsProcessed` and `jobsFailed` per worker
+(API 0.1.122+).
+
+If you process jobs without `SpooledWorker` (`jobs.claim()` and then `jobs.complete()`,
+`jobs.fail()` or `jobs.heartbeat()`), pass the claimed job's `leaseId` on every one of those calls.
+The API requires it: a call without it fails with a `ValidationError` (HTTP 422), and a stale lease
+with HTTP 409. `SpooledWorker` does this for you.
 
 ### Workflows (DAGs)
 
@@ -203,9 +242,9 @@ Manage your organization and track usage:
 // Get current usage and limits
 const usage = await client.organizations.getUsage();
 console.log(
-  `Active jobs: ${usage.active_jobs.current}/${usage.active_jobs.limit}`,
+  `Active jobs: ${usage.usage.activeJobs.current}/${usage.usage.activeJobs.limit}`,
 );
-console.log(`Plan: ${usage.plan.tier}`);
+console.log(`Plan: ${usage.plan}`); // e.g. "free"
 
 // Generate a unique slug for a new organization
 const { slug } = await client.organizations.generateSlug("My Company");
@@ -227,6 +266,14 @@ const webhook = await client.webhooks.create({
   secret: "webhook_secret_key",
 });
 
+// Post dead-lettered jobs and workflow outcomes straight to a Slack channel
+await client.webhooks.create({
+  name: "On-call",
+  url: "https://hooks.slack.com/services/...",
+  events: ["job.deadlettered", "workflow.failed"],
+  format: "slack", // "json" (default) | "slack" | "discord" | "teams"
+});
+
 // Retry a failed delivery
 await client.webhooks.retryDelivery(webhookId, deliveryId);
 
@@ -235,6 +282,13 @@ await client.webhooks.retryDelivery(webhookId, deliveryId);
 const deliveries = await client.webhooks.getDeliveries(webhookId);
 ```
 
+API 0.1.122 added the events `job.deadlettered`, `job.retried`, `workflow.completed` and
+`workflow.failed`, and `format`, which sends each delivery as a chat message that a Slack, Discord
+or Teams (legacy Office 365 connector) incoming-webhook URL accepts as it is.
+
+The inbound side has a log too: `client.ingest.listDeliveries({ status, since, queueName })` lists
+the accepted requests to your custom webhook URL (`GET /webhooks/deliveries`, no payloads).
+
 After 20 consecutive failed deliveries a webhook is disabled automatically (`enabled: false`,
 `lastStatus: "auto_disabled"`) and stops receiving events until you turn it back on:
 
@@ -282,7 +336,7 @@ Manage API keys programmatically:
 const apiKey = await client.apiKeys.create({
   name: "Production Worker",
   queues: ["emails", "notifications"],
-  rateLimit: 1000,
+  rateLimit: 1000, // optional, requests per MINUTE for this key (can only lower the plan limit)
 });
 
 console.log(`Save this key: ${apiKey.key}`); // Only shown once!
@@ -337,10 +391,13 @@ All operations automatically enforce tier-based limits:
 
 | Tier           | Active Jobs | Daily Jobs | Queues    | Workers   | Webhooks  |
 | -------------- | ----------- | ---------- | --------- | --------- | --------- |
-| **Free**       | 10          | 1,000      | 5         | 3         | 2         |
-| **Starter**    | 100         | 100,000    | 25        | 25        | 10        |
+| **Free**       | 10          | 1,000      | 2         | 1         | 1         |
+| **Starter**    | 500         | 10,000     | 10        | 5         | 5         |
+| **Pro**        | 5,000       | 100,000    | 50        | 25        | 20        |
 | **Enterprise** | Unlimited   | Unlimited  | Unlimited | Unlimited | Unlimited |
 
+(Source: the backend's `config/plans.rs`; Free has no workflows, Starter 5, Pro 25.)
+
 **Limits are enforced on:**
 
 - ✅ HTTP job creation (`POST /jobs`, `POST /jobs/bulk`)
@@ -375,7 +432,23 @@ try {
 }
 ```
 
-(Per-second rate limiting also returns `429`, as a `RateLimitError` with the default `RATE_LIMIT_EXCEEDED` code.)
+(Request-rate limiting also returns `429`, as a `RateLimitError` with the default `RATE_LIMIT_EXCEEDED` code. The plan limit is per organization in requests per second; an API key's own `rateLimit` is requests per minute.)
+
+Since API 0.1.122, responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset`
+(a Unix time in seconds) and `X-RateLimit-Scope` (`organization` or `api_key`), and a 429 adds
+`Retry-After`. `client.getRateLimitInfo()` returns the values from the most recent response that had
+them, and `RateLimitError.rateLimitInfo` those of a 429. The headers are optional: public routes and
+servers without Redis do not send them.
+
+```typescript
+await client.jobs.getStats();
+const rate = client.getRateLimitInfo();
+// { limit: 200, remaining: 199, reset: Date, scope: "organization" }
+```
+
+A plan quota does not clear for hours, so `error.isQuota` is `true` for it and the SDK never retries it.
+Rate-limit 429s are retried after `Retry-After`. No 429 counts toward the circuit breaker, so a burst
+above the plan limit slows the client down instead of opening the breaker for every call.
 
 ## Error Handling
 
@@ -398,8 +471,9 @@ try {
   } else if (error instanceof AuthorizationError) {
     console.log(`Forbidden: ${error.message}`);
   } else if (error instanceof RateLimitError) {
-    // 429: either a plan quota (code "QUOTA_EXCEEDED") or per-second rate limiting
-    console.log(`Retry after ${error.getRetryAfter()} seconds`);
+    // 429: either a plan quota (error.isQuota, code "QUOTA_EXCEEDED") or per-second rate limiting
+    if (!error.isQuota)
+      console.log(`Retry after ${error.getRetryAfter()} seconds`);
   } else if (isSpooledError(error)) {
     console.log(`API error: ${error.code} - ${error.message}`);
   }
@@ -422,11 +496,12 @@ const grpcClient = new SpooledGrpcClient({
   useTls: true,
 });
 
-// Register worker
+// Register worker (a stable workerId is reused across restarts)
 const { workerId } = await grpcClient.workers.register({
   queueName: "emails",
   hostname: "worker-1",
   maxConcurrency: 10,
+  workerId: "emails-worker-1",
 });
 
 // Enqueue job (enforces plan limits automatically)
@@ -444,18 +519,44 @@ const { jobs } = await grpcClient.queue.dequeue({
   leaseDurationSecs: 300,
 });
 
-// Process and complete. Echo leaseId when present so current servers can fence stale executions; legacy servers may omit it and then use worker-ID-only fencing.
+// Process and settle. leaseId is required: the server rejects a complete/fail/renew
+// without the lease token that dequeue returned. fail() retries by default (until the
+// job's maxRetries run out); pass retry: false to dead-letter on purpose.
 for (const job of jobs) {
-  await processJob(job);
-  await grpcClient.queue.complete({
-    jobId: job.id,
-    workerId,
-    leaseId: job.leaseId ?? undefined,
-    result: { success: true },
-  });
+  const leaseId = job.leaseId as string;
+  try {
+    await processJob(job);
+    await grpcClient.queue.complete({
+      jobId: job.id,
+      workerId,
+      leaseId,
+      result: { success: true },
+    });
+  } catch (error) {
+    await grpcClient.queue.fail({
+      jobId: job.id,
+      workerId,
+      leaseId,
+      error: error instanceof Error ? error.message : String(error),
+    });
+  }
 }
 ```
 
+`fail` sends `retry: true` unless you pass `retry: false`; `false` is sent on the wire (proto field 4
+has presence) and dead-letters the job at once, whatever its `maxRetries`.
+
+Over gRPC `payload` and `result` are `google.protobuf.Struct`, so two things differ from REST: only a
+JSON object travels (a job whose payload is an array, string, number or `null` arrives as an empty
+object), and integers at or above 2^53 lose precision (send them as strings). Use REST for such jobs.
+
+`dequeue` hands out up to `batchSize` jobs, each with a lease of `leaseDurationSecs` (5-3600 s; when
+omitted the server default applies, 300 s on Spooled Cloud). For a job that can run longer than its
+lease, pass a longer lease or renew it before it expires (`renewLease` with `jobId`, `workerId`,
+`leaseId`, `extensionSecs`). A gRPC `StreamJobs` stream delivers **one job at a time**: the next
+job is sent only after the previous one on that stream is completed, failed or released, so open
+one stream per job slot you want to run in parallel.
+
 **gRPC Features:**
 
 - **~28x faster** than HTTP (with Redis cache: ~50ms vs 1400ms)