@ultimat3/jobs 12.0.0 → 13.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +67 -0
- package/README.md +149 -0
- package/package.json +5 -5
- package/src/errors.ts +42 -0
- package/src/export-errors.ts +50 -0
- package/src/export-format.ts +132 -0
- package/src/export-pass.ts +178 -0
- package/src/export-sink.ts +105 -0
- package/src/export.ts +187 -0
- package/src/index.ts +50 -1
- package/src/webhook-errors.ts +170 -0
- package/src/webhook-ledger.ts +102 -0
- package/src/webhook.ts +378 -0
package/CLAUDE.md
CHANGED
|
@@ -388,6 +388,73 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
388
388
|
ABANDONED round does not release**: it is still enqueueing, so handing the lock over is that same
|
|
389
389
|
double-fire delivered by the shutdown, and a lease row expiring on its own is the safe end —
|
|
390
390
|
`jobs.scheduler.drain-abandoned` is the line that says so.
|
|
391
|
+
- **`exportRows()` is a FACTORY over `job()`, and the part KEY is what makes at-least-once safe**
|
|
392
|
+
(`As of 2026-08-24`). Same rule `backfill()`, `purge()` and `webhook()` follow. The paging is
|
|
393
|
+
`backfill-pass.ts`'s and never a second idiom — `inBatches()`, one page per `step.run`, and what
|
|
394
|
+
a step persists is the CURSOR and two counters, never the page.
|
|
395
|
+
|
|
396
|
+
**One object per PAGE, not one per export**, and that is the whole design rather than a
|
|
397
|
+
simplification: `StorageDriver.put()` buffers by construction — `packages/storage/src/driver.ts`
|
|
398
|
+
says the server-side path "is for objects that FIT IN MEMORY" — so a single-object export holds
|
|
399
|
+
the entire dataset, which is the failure this factory exists to prevent. Because a part is named
|
|
400
|
+
by its page INDEX, a page that runs twice rewrites the same object with the same bytes: there is
|
|
401
|
+
no idempotency argument to make about the app's rows, and no append anywhere, because a duplicate
|
|
402
|
+
part cannot be expressed. `export-pass.test.ts` proves it by deleting a mid-run checkpoint and
|
|
403
|
+
re-running, which is exactly the window at-least-once opens.
|
|
404
|
+
|
|
405
|
+
**The interleaving assertion is the memory guard, not `maxPartBytes`.** `maxPartBytes` bounds one
|
|
406
|
+
page; it cannot see a rewrite that accumulates every page and writes at the end. The test records
|
|
407
|
+
how many checkpoints had LANDED at each `put` — `0,1,2,…` for a streaming pass and a flat run of
|
|
408
|
+
the final count for a buffering one — and it is the only thing in the suite that fails on that
|
|
409
|
+
rewrite.
|
|
410
|
+
|
|
411
|
+
**An export gets NO cross-tenant escape, and `backfill()` does.** The asymmetry is deliberate and
|
|
412
|
+
is the direction of the data: a sweep that rewrites every tenant's rows is an operator action
|
|
413
|
+
with an audit trail, while an export READS every tenant's rows into one object somebody can
|
|
414
|
+
download. `tenant: 'none'` therefore fails closed on a tenant-scoped entity
|
|
415
|
+
(`X_TENANCY_ACTOR_ORG_REQUIRED`) and is for tables with no tenant column at all.
|
|
416
|
+
|
|
417
|
+
**The format is the framework's and the columns are the app's.** The csv formula guard (a cell
|
|
418
|
+
leading `=`, `+`, `-`, `@`, TAB or CR is evaluated by every spreadsheet) and RFC-4180 quoting are
|
|
419
|
+
identical for a bank and a blog; which columns leave the building never is. The guard is on
|
|
420
|
+
STRINGS only — prefixing every leading `-` would turn a refund column into text.
|
|
421
|
+
- **`webhook()` is a FACTORY over `job()` and delivers ONE event to ONE endpoint** (`As of
|
|
422
|
+
2026-08-24`). Same rule `backfill()` and `purge()` follow. The unit is the ENDPOINT and not the
|
|
423
|
+
event, because retry, backoff and disable-after-N are all per-endpoint facts: a job that fanned
|
|
424
|
+
out inside one body would retry every subscriber because one of them was down, which is the
|
|
425
|
+
defect `15-adding-a-feature.md` already names for a mail loop under a single step. **Which
|
|
426
|
+
endpoints exist is the app's** (axiom 8) — the fan-out is the app's own loop, one `enqueue` per
|
|
427
|
+
endpoint, and the idempotency key is `<name>:<endpointId>:<eventId>` for the same reason: a key
|
|
428
|
+
on the event alone would dedupe every subscriber's delivery into the first one's row.
|
|
429
|
+
|
|
430
|
+
Four things are load-bearing and none is an implementation detail. **The endpoint is never
|
|
431
|
+
checkpointed**: it carries the secret, and a `step.run` output is written to `x_job_steps` — so
|
|
432
|
+
this body uses NO steps and re-reads both seams per attempt. **The signature's timestamp is SEND
|
|
433
|
+
time**, so a delivery retried three days later is signed again now and a receiver's freshness
|
|
434
|
+
window measures the request rather than the age of the fact behind it. **The endpoint row's own
|
|
435
|
+
`headers` merge UNDER the framework's**, because a row that could set
|
|
436
|
+
`x-ultimate-webhook-signature` is a row that can forge one. **`redirect: 'manual'`**, because
|
|
437
|
+
following a 3xx re-POSTs a body signed for one host to whatever the receiver named, signature
|
|
438
|
+
and all.
|
|
439
|
+
|
|
440
|
+
The ledger is a SEAM (`WebhookLedger`), the shape `PurgeTarget` already has: retention is seven
|
|
441
|
+
years for one business and thirty days for the next, so shipping a schema would be shipping one
|
|
442
|
+
of those answers. `record()` answers the consecutive-failure count rather than that being a
|
|
443
|
+
second method — the number the mechanism disables on must not be readable from before the row it
|
|
444
|
+
is deciding about. Every attempt is recorded BEFORE the throw: a failure the ledger cannot see is
|
|
445
|
+
an endpoint that never gets disabled. Re-enabling is always the app's.
|
|
446
|
+
|
|
447
|
+
**The wire format is `@ultimat3/core`'s, RE-EXPORTED and never re-declared** (`As of
|
|
448
|
+
2026-08-24`). `packages/core/src/webhook-signature.ts` states
|
|
449
|
+
`v1:<timestampSeconds>:<eventId>:<topic>:<body>` once — the canonical string, the mac, the three
|
|
450
|
+
header names and the parse a receiver reads back — because this package signs a delivery,
|
|
451
|
+
`@ultimat3/http` verifies one, and neither may import the other. Exactly the argument
|
|
452
|
+
`timing-safe-equal.ts` makes for itself. It shipped for one release as two implementations held
|
|
453
|
+
together by a hex literal asserted in two test files; that works and is not a single source of
|
|
454
|
+
truth. Core's own suite now pins the thing neither package could: that the SENDING form
|
|
455
|
+
(`body: string`) and the RECEIVING form (`body: Uint8Array`) are the same function over the same
|
|
456
|
+
bytes. **Never re-declare the canonical string here** — a second spelling is what the move
|
|
457
|
+
deleted.
|
|
391
458
|
- **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows
|
|
392
459
|
in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a
|
|
393
460
|
backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and
|
package/README.md
CHANGED
|
@@ -337,6 +337,140 @@ export const hourly = task({
|
|
|
337
337
|
without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
|
|
338
338
|
deployment with neither has no background work at all, and this is one more thing it does not do.
|
|
339
339
|
|
|
340
|
+
## Exporting a large dataset is a job too
|
|
341
|
+
|
|
342
|
+
`exportRows()` is a factory over `job()` that streams a paged read to object storage with a
|
|
343
|
+
resumable cursor. It exists for one reason: `const all = await repo.all(); await disk.put(key,
|
|
344
|
+
csv(all))` works on 200 rows and OOM-kills the pod on two million.
|
|
345
|
+
|
|
346
|
+
```ts
|
|
347
|
+
import type { ReadBuilder } from '@ultimat3/entity';
|
|
348
|
+
import { exportRows, t } from '@ultimat3/jobs';
|
|
349
|
+
import { formatMoney, type Money } from '@ultimat3/money';
|
|
350
|
+
import { disk } from '@ultimat3/storage';
|
|
351
|
+
import { formatDate, instant } from '@ultimat3/time';
|
|
352
|
+
|
|
353
|
+
interface Order {
|
|
354
|
+
readonly id: string;
|
|
355
|
+
readonly placedAt: Date;
|
|
356
|
+
readonly total: Money;
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
// The app's own chain: `source` takes whatever `db.orders.where(…)` answers.
|
|
360
|
+
declare const db: {
|
|
361
|
+
readonly orders: { where(filter: { readonly orgId: string }): ReadBuilder<Order> };
|
|
362
|
+
};
|
|
363
|
+
|
|
364
|
+
export const exportOrders = exportRows({
|
|
365
|
+
name: 'orders.export',
|
|
366
|
+
input: t.object({ orgId: t.string, exportId: t.string }),
|
|
367
|
+
// The security boundary of the whole feature — an export concentrates one tenant's rows into a
|
|
368
|
+
// single downloadable object.
|
|
369
|
+
tenant: ({ orgId }) => orgId,
|
|
370
|
+
prefix: ({ orgId, exportId }) => `exports/${orgId}/${exportId}`,
|
|
371
|
+
source: ({ input }) => db.orders.where({ orgId: input.orgId }),
|
|
372
|
+
format: 'csv',
|
|
373
|
+
columns: ['id', 'placedAt', 'total'],
|
|
374
|
+
row: (order) => ({
|
|
375
|
+
id: order.id,
|
|
376
|
+
// A date gets its zone and a Money its currency HERE, where the app knows which it means.
|
|
377
|
+
// `instant()` is the check that turns a stored `Date` into one nothing can format zone-less.
|
|
378
|
+
placedAt: formatDate(instant(order.placedAt), { locale: 'en', zone: 'UTC' }),
|
|
379
|
+
total: formatMoney(order.total, 'en'),
|
|
380
|
+
}),
|
|
381
|
+
sink: disk('exports'), // a StorageDriver already IS an ExportSink
|
|
382
|
+
});
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
The artifact is one object per page plus a manifest:
|
|
386
|
+
|
|
387
|
+
```
|
|
388
|
+
exports/<orgId>/<exportId>/part-00000.csv
|
|
389
|
+
exports/<orgId>/<exportId>/part-00001.csv
|
|
390
|
+
exports/<orgId>/<exportId>/manifest.json
|
|
391
|
+
```
|
|
392
|
+
|
|
393
|
+
| Rule | Why |
|
|
394
|
+
|---|---|
|
|
395
|
+
| one object per PAGE, never one per export | `put()` buffers by construction — its own header says the server-side path "is for objects that FIT IN MEMORY" — so a single-object export holds the whole dataset, which is the failure this exists to prevent |
|
|
396
|
+
| the part key is the page INDEX | so a page that runs twice REWRITES its part. At-least-once needs no idempotency argument about your rows here: a duplicate part cannot be expressed |
|
|
397
|
+
| a step persists the CURSOR and two counters | never the page — `steps.ts` retains a completed step's output for the whole run, so checkpointing rows keeps every exported row until the job ends |
|
|
398
|
+
| every line ends in a newline, header in part 0 only | `cat part-*` is one valid file, which is what makes the parts an artifact rather than fragments |
|
|
399
|
+
| `maxPartBytes` is a heap bound, not a file-size preference | one page is encoded whole before it is written; `X_EXPORT_PART_TOO_LARGE` names the `batch` to lower |
|
|
400
|
+
| a `row()` key `columns` does not carry is REFUSED | both encoders would drop it in silence, and `row: (r) => ({ ...r })` picks up every column the entity gains from the next migration on |
|
|
401
|
+
| csv cells leading `=`, `+`, `-`, `@`, TAB or CR are neutralised | Excel, Sheets and LibreOffice EVALUATE them, so a user-named record is code execution in the reviewer's spreadsheet. Strings only — a negative number stays a number |
|
|
402
|
+
| the manifest COUNTS parts, never lists them | `exportPartKey(prefix, i, format)` rebuilds every key, and a list is the one thing in the pass that would grow with the export |
|
|
403
|
+
| `tenant: 'none'` gets no cross-tenant escape | `backfill()` does (`backfill-scope.ts`) because its lazy chain leaves an author nothing to wrap. An export does not, and the difference is the direction of the data: a sweep rewrites rows under audit, an export writes every tenant's rows into one downloadable object |
|
|
404
|
+
| `rate` has no default, unlike a backfill's | a backfill competes for WRITE capacity on rows the app is still serving; an export is a bounded read somebody is usually waiting for. Declare it for an export big enough to matter to the pool |
|
|
405
|
+
|
|
406
|
+
`ExportSink` is a seam and not a `@ultimat3/storage` import, for the reason `PurgeTarget` is one:
|
|
407
|
+
this package holds no storage dependency, and taking one so a queue could name a disk would put the
|
|
408
|
+
object store on tier 3's import graph.
|
|
409
|
+
|
|
410
|
+
## Outbound webhooks are jobs too
|
|
411
|
+
|
|
412
|
+
`webhook()` is a factory over `job()` and delivers **one event to one endpoint**. Retry with
|
|
413
|
+
backoff, disable-after-N and the ledger are all per-endpoint facts, so the endpoint is the unit: a
|
|
414
|
+
job that fanned out inside one body would retry every subscriber because one of them was down.
|
|
415
|
+
**Which endpoints exist is the app's** — the fan-out is your own `for` loop over your own
|
|
416
|
+
subscription table, one `enqueue` per endpoint.
|
|
417
|
+
|
|
418
|
+
```ts
|
|
419
|
+
import { memoryWebhookLedger, webhook } from '@ultimat3/jobs';
|
|
420
|
+
|
|
421
|
+
declare const db: {
|
|
422
|
+
endpoints: { byId(id: string): Promise<{ id: string; url: string; secret: string } | null> };
|
|
423
|
+
events: { byId(id: string): Promise<{ topic: string; body: string } | null> };
|
|
424
|
+
};
|
|
425
|
+
|
|
426
|
+
export const deliver = webhook({
|
|
427
|
+
name: 'partner.webhooks',
|
|
428
|
+
tenant: 'none',
|
|
429
|
+
// Read once per ATTEMPT and never checkpointed: the endpoint carries a secret, and a step's
|
|
430
|
+
// output is written to `x_job_steps`.
|
|
431
|
+
endpoint: ({ endpointId }) => db.endpoints.byId(endpointId),
|
|
432
|
+
event: ({ eventId }) => db.events.byId(eventId),
|
|
433
|
+
ledger: memoryWebhookLedger(), // dev only — a bounded ring in one heap
|
|
434
|
+
disableAfter: 10,
|
|
435
|
+
});
|
|
436
|
+
|
|
437
|
+
// The fan-out is yours, because the subscription table is yours.
|
|
438
|
+
declare function subscribersOf(topic: string): Promise<readonly { id: string }[]>;
|
|
439
|
+
declare const event: { readonly id: string };
|
|
440
|
+
|
|
441
|
+
for (const endpoint of await subscribersOf('orders.paid')) {
|
|
442
|
+
await deliver.enqueue({ endpointId: endpoint.id, eventId: event.id });
|
|
443
|
+
}
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
The delivery carries three headers beyond `content-type`, and the signature is over
|
|
447
|
+
`v1:<timestampSeconds>:<eventId>:<topic>:<body>`:
|
|
448
|
+
|
|
449
|
+
```
|
|
450
|
+
x-ultimate-webhook-id: evt_01HZ
|
|
451
|
+
x-ultimate-webhook-topic: orders.paid
|
|
452
|
+
x-ultimate-webhook-signature: t=1700000000,v1=<hex hmac-sha256>
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
The receiving half is `verifyWebhookSignature(request, { secret })` in `@ultimat3/http`. The format
|
|
456
|
+
itself — the canonical string, the mac and the header names — is **one module in `@ultimat3/core`**
|
|
457
|
+
(`webhook-signature.ts`), re-exported by both packages and re-declared by neither: this package
|
|
458
|
+
signs, `http` verifies, and neither may import the other, so the one copy lives at the tier both
|
|
459
|
+
can reach. Same argument `timing-safe-equal.ts` makes for itself.
|
|
460
|
+
|
|
461
|
+
| Rule | Why |
|
|
462
|
+
|---|---|
|
|
463
|
+
| the timestamp is **inside** the mac | a captured delivery re-dated to slip back into a receiver's freshness window no longer verifies |
|
|
464
|
+
| the timestamp is SEND time, not event time | a retry three days later signs again now, so the window measures the request rather than the age of the fact |
|
|
465
|
+
| `:` is refused in an id or a topic | one mac over `v1:t:evt:01HZ:orders.paid:<body>` would otherwise authenticate two different id/topic splits — the same delivery under a label the sender never wrote |
|
|
466
|
+
| the endpoint's own `headers` merge **under** the framework's | an endpoint row that could set `x-ultimate-webhook-signature` is an endpoint row that can forge one |
|
|
467
|
+
| `redirect: 'manual'` | following a 3xx would re-POST a body signed for one host to whatever the receiver named |
|
|
468
|
+
| the endpoint is never checkpointed | a `step.run` output lands in `x_job_steps`, and the endpoint carries the secret |
|
|
469
|
+
| every attempt is recorded **before** the throw | a failure the ledger cannot see is a failure the consecutive count cannot see, which is an endpoint that never gets disabled |
|
|
470
|
+
| a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out (clamped by `retry.maxDelay`) rather than guessing a curve against an answer it already has |
|
|
471
|
+
| re-enabling is always yours | an endpoint the framework un-disabled on its own is a retry loop with no end |
|
|
472
|
+
| `WebhookLedger` is a seam, not a table | retention is seven years for one business and thirty days for the next, so shipping a schema would ship one of those answers |
|
|
473
|
+
|
|
340
474
|
## The deadline cancels
|
|
341
475
|
|
|
342
476
|
A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
|
|
@@ -504,6 +638,17 @@ and never detects the occurrence the pod it replaced dropped (`catchUp` and `max
|
|
|
504
638
|
update runs two leaders.
|
|
505
639
|
|
|
506
640
|
```ts
|
|
641
|
+
import {
|
|
642
|
+
createPgLeaseLeader,
|
|
643
|
+
createScheduler,
|
|
644
|
+
type JobDriver,
|
|
645
|
+
type PgExecutor,
|
|
646
|
+
pgSchedulerState,
|
|
647
|
+
} from '@ultimat3/jobs';
|
|
648
|
+
|
|
649
|
+
declare const driver: JobDriver;
|
|
650
|
+
declare const executor: PgExecutor; // `@ultimat3/cli`'s pgExecutorFor(client)
|
|
651
|
+
|
|
507
652
|
createScheduler({
|
|
508
653
|
driver,
|
|
509
654
|
state: pgSchedulerState(executor),
|
|
@@ -518,6 +663,10 @@ expiry and needs no connection affinity. `acquire()` is also the renewal, called
|
|
|
518
663
|
is how a demoted node finds out.
|
|
519
664
|
|
|
520
665
|
```ts
|
|
666
|
+
import { type AnyJobHandle, task } from '@ultimat3/jobs';
|
|
667
|
+
|
|
668
|
+
declare const sendDigest: AnyJobHandle; // the job this cron puts on the queue
|
|
669
|
+
|
|
521
670
|
export const nightlyDigest = task({
|
|
522
671
|
cron: '0 3 * * *',
|
|
523
672
|
tz: 'UTC', // REQUIRED — a cron without a timezone is a bug
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/jobs",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "13.0.0",
|
|
4
4
|
"description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -32,9 +32,9 @@
|
|
|
32
32
|
"test": "bun test"
|
|
33
33
|
},
|
|
34
34
|
"dependencies": {
|
|
35
|
-
"@ultimat3/core": "
|
|
36
|
-
"@ultimat3/entity": "
|
|
37
|
-
"@ultimat3/schema": "
|
|
38
|
-
"@ultimat3/time": "
|
|
35
|
+
"@ultimat3/core": "13.0.0",
|
|
36
|
+
"@ultimat3/entity": "13.0.0",
|
|
37
|
+
"@ultimat3/schema": "13.0.0",
|
|
38
|
+
"@ultimat3/time": "13.0.0"
|
|
39
39
|
}
|
|
40
40
|
}
|
package/src/errors.ts
CHANGED
|
@@ -26,6 +26,16 @@ export const JOB_OWNED_ERROR_CODES = [
|
|
|
26
26
|
'X_JOB_ROW_STATUS_UNKNOWN',
|
|
27
27
|
'X_ACTION_JOB_UNBRIDGED',
|
|
28
28
|
'X_JOB_CLAIM_QUEUES_EMPTY',
|
|
29
|
+
'X_WEBHOOK_ENDPOINT_UNKNOWN',
|
|
30
|
+
'X_WEBHOOK_ENDPOINT_INVALID',
|
|
31
|
+
'X_WEBHOOK_ENDPOINT_DISABLED',
|
|
32
|
+
'X_WEBHOOK_EVENT_UNKNOWN',
|
|
33
|
+
'X_WEBHOOK_EVENT_INVALID',
|
|
34
|
+
'X_WEBHOOK_DELIVERY_FAILED',
|
|
35
|
+
'X_WEBHOOK_DELIVERY_THROTTLED',
|
|
36
|
+
'X_WEBHOOK_DELIVERY_REJECTED',
|
|
37
|
+
'X_EXPORT_ROW_INVALID',
|
|
38
|
+
'X_EXPORT_PART_TOO_LARGE',
|
|
29
39
|
] as const;
|
|
30
40
|
|
|
31
41
|
/**
|
|
@@ -64,6 +74,16 @@ export const JOB_ERROR_TITLES: Readonly<Record<JobOwnedErrorCode, string>> = {
|
|
|
64
74
|
X_JOB_ROW_STATUS_UNKNOWN: 'a queue row carries a status this build does not know',
|
|
65
75
|
X_ACTION_JOB_UNBRIDGED: 'an action projection was registered as a job',
|
|
66
76
|
X_JOB_CLAIM_QUEUES_EMPTY: 'a claim named no queue, and the drivers do not agree what that means',
|
|
77
|
+
X_WEBHOOK_ENDPOINT_UNKNOWN: 'no endpoint carries this id',
|
|
78
|
+
X_WEBHOOK_ENDPOINT_INVALID: 'the endpoint cannot be delivered to as declared',
|
|
79
|
+
X_WEBHOOK_ENDPOINT_DISABLED: 'the endpoint takes no deliveries',
|
|
80
|
+
X_WEBHOOK_EVENT_UNKNOWN: 'no event carries this id',
|
|
81
|
+
X_WEBHOOK_EVENT_INVALID: 'the event cannot be signed as declared',
|
|
82
|
+
X_WEBHOOK_DELIVERY_FAILED: 'the endpoint did not accept the delivery',
|
|
83
|
+
X_WEBHOOK_DELIVERY_THROTTLED: 'the endpoint asked for the delivery later, and said when',
|
|
84
|
+
X_WEBHOOK_DELIVERY_REJECTED: 'the endpoint refused the delivery in a way a retry cannot change',
|
|
85
|
+
X_EXPORT_ROW_INVALID: 'row() answered something the declared columns do not carry',
|
|
86
|
+
X_EXPORT_PART_TOO_LARGE: 'one page encoded to more bytes than a part may hold',
|
|
67
87
|
};
|
|
68
88
|
|
|
69
89
|
// One unconditional call, so a second package claiming one of jobs' codes throws
|
|
@@ -93,6 +113,28 @@ registerErrorRetry({
|
|
|
93
113
|
X_BACKFILL_STALLED: 'terminal',
|
|
94
114
|
X_BACKFILL_ENVIRONMENT: 'terminal',
|
|
95
115
|
X_BACKFILL_APPLIED: 'terminal',
|
|
116
|
+
// The one retryable webhook code, and the reason the split exists at all: a 5xx, a 429 or a
|
|
117
|
+
// connection that never opened is the same request landing later, which is what the job's retry
|
|
118
|
+
// policy is for.
|
|
119
|
+
X_WEBHOOK_DELIVERY_FAILED: 'retryable',
|
|
120
|
+
// `retry-after` and not `retryable`: the receiver NAMED a delay, so `statedDelayMs` reads
|
|
121
|
+
// `meta.retryAfterSeconds` and the nack waits that long (clamped by the policy's `maxDelay`)
|
|
122
|
+
// instead of guessing a curve against an answer it was already given.
|
|
123
|
+
X_WEBHOOK_DELIVERY_THROTTLED: 'retry-after',
|
|
124
|
+
// The rest are `terminal` and every one is LISTED rather than left to the default, because
|
|
125
|
+
// `classifyThrown` reads an unregistered code carrying `terminal` as UNCLASSIFIED — which spends
|
|
126
|
+
// a delivery's whole retry policy re-proving that a 404 endpoint is still a 404.
|
|
127
|
+
X_WEBHOOK_DELIVERY_REJECTED: 'terminal',
|
|
128
|
+
X_WEBHOOK_ENDPOINT_UNKNOWN: 'terminal',
|
|
129
|
+
X_WEBHOOK_ENDPOINT_INVALID: 'terminal',
|
|
130
|
+
X_WEBHOOK_ENDPOINT_DISABLED: 'terminal',
|
|
131
|
+
X_WEBHOOK_EVENT_UNKNOWN: 'terminal',
|
|
132
|
+
X_WEBHOOK_EVENT_INVALID: 'terminal',
|
|
133
|
+
// Both export codes are refusals of the DECLARATION and not of the data: the same `row()` over
|
|
134
|
+
// the same page answers the same way on every attempt, so an unclassified reading would spend
|
|
135
|
+
// the whole retry policy proving it.
|
|
136
|
+
X_EXPORT_ROW_INVALID: 'terminal',
|
|
137
|
+
X_EXPORT_PART_TOO_LARGE: 'terminal',
|
|
96
138
|
});
|
|
97
139
|
|
|
98
140
|
// No `docs:` on any class below, here or in `backfill-errors.ts`. `UltimateError` fills it from
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
// The two `X_EXPORT_*` codes an export pass can end on, apart from `errors.ts` for the reason
|
|
2
|
+
// `backfill-errors.ts` and `webhook-errors.ts` are: one file, one job, and `errors.ts` holds the
|
|
3
|
+
// registry — the codes, the titles and the single `registerErrorCodes()` call.
|
|
4
|
+
//
|
|
5
|
+
// Both are TERMINAL and both are refusals of the DECLARATION rather than of the data: the same
|
|
6
|
+
// `row()` over the same page answers the same way on every attempt, so retrying spends a policy
|
|
7
|
+
// proving it.
|
|
8
|
+
|
|
9
|
+
import { UltimateError } from '@ultimat3/core';
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* `row()` answered something the declaration did not promise: a column missing, a column nobody
|
|
13
|
+
* declared, or a value no format can carry.
|
|
14
|
+
*
|
|
15
|
+
* The extra-column half is the one worth the code. A `row()` that returns a key `columns` omits is
|
|
16
|
+
* silently DROPPED by both encoders — so an export that was supposed to carry a column carries the
|
|
17
|
+
* rest of the file looking perfectly correct, and the gap is found by whoever consumes it, later.
|
|
18
|
+
* The reverse — a column nobody declared — is how PII leaves through an export nobody reviewed:
|
|
19
|
+
* `row: (r) => ({ ...r })` picks up every column the entity gains from then on.
|
|
20
|
+
*
|
|
21
|
+
* `cause` names the COLUMN and never the value: a cell is user data by definition, and a cause
|
|
22
|
+
* reaches the log store as a field nothing can redact.
|
|
23
|
+
*/
|
|
24
|
+
export class ExportRowInvalidError extends UltimateError {
|
|
25
|
+
constructor(input: { export: string; column: string; reason: string }) {
|
|
26
|
+
super({
|
|
27
|
+
code: 'X_EXPORT_ROW_INVALID',
|
|
28
|
+
cause: `export "${input.export}" row() answered a "${input.column}" that ${input.reason}`,
|
|
29
|
+
fix: `return exactly the declared columns from row() on exportRows("${input.export}"), each one a string, a finite number, a boolean or null — format a date with an explicit timeZone and a Money with its own currency before it gets here`,
|
|
30
|
+
meta: { column: input.column },
|
|
31
|
+
});
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* One page encoded to more bytes than a part may hold. The memory bound made mechanical: this
|
|
37
|
+
* factory exists because accumulating a result set before writing it is the failure it prevents,
|
|
38
|
+
* and "we hold a page, not a dataset" is only true while a page is bounded. A row that is a
|
|
39
|
+
* megabyte of JSON turns `batch: 1_000` into a gigabyte in one buffer.
|
|
40
|
+
*/
|
|
41
|
+
export class ExportPartTooLargeError extends UltimateError {
|
|
42
|
+
constructor(input: { export: string; part: number; bytes: number; maxBytes: number }) {
|
|
43
|
+
super({
|
|
44
|
+
code: 'X_EXPORT_PART_TOO_LARGE',
|
|
45
|
+
cause: `export "${input.export}" part ${input.part} encoded to ${input.bytes} bytes and a part may hold ${input.maxBytes}`,
|
|
46
|
+
fix: `lower batch on exportRows("${input.export}") until a page fits, or raise maxPartBytes if this deployment really can hold that much per part — the number is a heap bound, not a file-size preference`,
|
|
47
|
+
meta: { part: input.part, bytes: input.bytes, maxBytes: input.maxBytes },
|
|
48
|
+
});
|
|
49
|
+
}
|
|
50
|
+
}
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
// One page of rows -> the bytes that page contributes to the artifact. Pure and synchronous, so
|
|
2
|
+
// every hazard in it is testable without a queue, a source or a sink.
|
|
3
|
+
//
|
|
4
|
+
// The FORMAT is the framework's and the COLUMNS are the app's, which is exactly the line
|
|
5
|
+
// `docs/idea/20-large-app-readiness.md` draws for this feature. Quoting and the spreadsheet
|
|
6
|
+
// injection guard are identical for a bank and a blog and are the two things every hand-rolled
|
|
7
|
+
// CSV exporter gets wrong; which columns leave the building never is.
|
|
8
|
+
|
|
9
|
+
import { ExportRowInvalidError } from './export-errors';
|
|
10
|
+
|
|
11
|
+
export const EXPORT_FORMATS = ['ndjson', 'csv'] as const;
|
|
12
|
+
export type ExportFormat = (typeof EXPORT_FORMATS)[number];
|
|
13
|
+
|
|
14
|
+
/** The file extension each format's parts carry. `Object.freeze<T>({…})`, so an extra key fails. */
|
|
15
|
+
export const EXPORT_EXTENSION = Object.freeze<Record<ExportFormat, string>>({
|
|
16
|
+
ndjson: 'ndjson',
|
|
17
|
+
csv: 'csv',
|
|
18
|
+
});
|
|
19
|
+
|
|
20
|
+
/**
|
|
21
|
+
* What a cell may hold. Deliberately no `Date` and no `Money`: a date without an explicit IANA
|
|
22
|
+
* zone and a money as a float are both build errors everywhere else in this framework, and an
|
|
23
|
+
* export is the one surface where the wrong answer is archived rather than re-rendered. The app
|
|
24
|
+
* formats both before they get here, where it can see which zone and which currency it means.
|
|
25
|
+
*/
|
|
26
|
+
export type ExportValue = string | number | boolean | null;
|
|
27
|
+
export type ExportRecord = Readonly<Record<string, ExportValue>>;
|
|
28
|
+
|
|
29
|
+
/** Fields a spreadsheet EVALUATES when they lead a cell. See `guardFormula`. */
|
|
30
|
+
const FORMULA_LEAD = /^[=+\-@\t\r]/;
|
|
31
|
+
/** A cell that would otherwise break the row or the file it sits in. */
|
|
32
|
+
const NEEDS_QUOTE = /[",\r\n]/;
|
|
33
|
+
|
|
34
|
+
export interface EncodePageInput {
|
|
35
|
+
/** The export's name, for a refusal to name. Never rendered into the artifact. */
|
|
36
|
+
readonly subject: string;
|
|
37
|
+
readonly format: ExportFormat;
|
|
38
|
+
/** The declared columns, in order. The header's order and the ndjson key order alike. */
|
|
39
|
+
readonly columns: readonly string[];
|
|
40
|
+
readonly records: readonly ExportRecord[];
|
|
41
|
+
/** True for part 0 only: a csv header belongs in the file once, and parts concatenate. */
|
|
42
|
+
readonly header: boolean;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
/**
|
|
46
|
+
* Refused rather than coerced, and BOTH directions matter. A missing column silently becomes an
|
|
47
|
+
* empty cell; an undeclared one is silently dropped — and `row: (r) => ({ ...r })` picks up every
|
|
48
|
+
* column the entity gains from the next migration on, which is how a column nobody reviewed leaves
|
|
49
|
+
* the building. Neither is visible in the artifact, which is why it has to be a refusal.
|
|
50
|
+
*/
|
|
51
|
+
function readCell(input: EncodePageInput, record: ExportRecord, column: string): ExportValue {
|
|
52
|
+
if (!Object.hasOwn(record, column)) {
|
|
53
|
+
throw new ExportRowInvalidError({
|
|
54
|
+
export: input.subject,
|
|
55
|
+
column,
|
|
56
|
+
reason: 'the declared columns name and row() did not answer',
|
|
57
|
+
});
|
|
58
|
+
}
|
|
59
|
+
const value = record[column];
|
|
60
|
+
if (value === null || typeof value === 'string' || typeof value === 'boolean') return value;
|
|
61
|
+
// FINITE, not merely `typeof 'number'`: `NaN` and `Infinity` land in the file as the words `NaN`
|
|
62
|
+
// and `Infinity`, which no consumer parses back to a number and no reviewer notices in a
|
|
63
|
+
// million-row artifact.
|
|
64
|
+
if (typeof value === 'number' && Number.isFinite(value)) return value;
|
|
65
|
+
throw new ExportRowInvalidError({
|
|
66
|
+
export: input.subject,
|
|
67
|
+
column,
|
|
68
|
+
reason: 'is not a string, a finite number, a boolean or null',
|
|
69
|
+
});
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
/** Every column, checked, plus a refusal for any key `columns` does not carry. */
|
|
73
|
+
function project(input: EncodePageInput, record: ExportRecord): readonly ExportValue[] {
|
|
74
|
+
const cells = input.columns.map((column) => readCell(input, record, column));
|
|
75
|
+
for (const key of Object.keys(record)) {
|
|
76
|
+
if (input.columns.includes(key)) continue;
|
|
77
|
+
throw new ExportRowInvalidError({
|
|
78
|
+
export: input.subject,
|
|
79
|
+
column: key,
|
|
80
|
+
reason: 'row() answered and the declaration does not carry — it would be dropped in silence',
|
|
81
|
+
});
|
|
82
|
+
}
|
|
83
|
+
return cells;
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/**
|
|
87
|
+
* A leading `=`, `+`, `-`, `@`, TAB or CR makes Excel, Sheets and LibreOffice EVALUATE the cell —
|
|
88
|
+
* so a record a user named `=cmd|'/c calc'!A1` is remote code execution in the reviewer's
|
|
89
|
+
* spreadsheet, and nothing in the exporting app looks wrong. A leading apostrophe is the one
|
|
90
|
+
* portable neutraliser.
|
|
91
|
+
*
|
|
92
|
+
* STRINGS only. Prefixing every leading `-` would turn a refund column into text in every
|
|
93
|
+
* spreadsheet that opened it, which is a data defect traded for a security one.
|
|
94
|
+
*/
|
|
95
|
+
const guardFormula = (value: string): string => (FORMULA_LEAD.test(value) ? `'${value}` : value);
|
|
96
|
+
|
|
97
|
+
const csvCell = (value: ExportValue): string => {
|
|
98
|
+
if (value === null) return '';
|
|
99
|
+
if (typeof value !== 'string') return String(value);
|
|
100
|
+
const guarded = guardFormula(value);
|
|
101
|
+
return NEEDS_QUOTE.test(guarded) ? `"${guarded.split('"').join('""')}"` : guarded;
|
|
102
|
+
};
|
|
103
|
+
|
|
104
|
+
/** The csv header line, exported so the manifest and the first part cannot spell it differently. */
|
|
105
|
+
export const csvHeader = (columns: readonly string[]): string =>
|
|
106
|
+
`${columns.map((column) => csvCell(column)).join(',')}\n`;
|
|
107
|
+
|
|
108
|
+
/**
|
|
109
|
+
* The page's bytes. Every line ends in a newline in both formats, so `cat part-0000 part-0001` is
|
|
110
|
+
* a valid file — which is what makes one object per page a legitimate artifact rather than a pile
|
|
111
|
+
* of fragments.
|
|
112
|
+
*/
|
|
113
|
+
export function encodeExportPage(input: EncodePageInput): Uint8Array {
|
|
114
|
+
const lines: string[] = [];
|
|
115
|
+
if (input.format === 'csv' && input.header) lines.push(csvHeader(input.columns));
|
|
116
|
+
for (const record of input.records) {
|
|
117
|
+
const cells = project(input, record);
|
|
118
|
+
if (input.format === 'csv') {
|
|
119
|
+
lines.push(`${cells.map(csvCell).join(',')}\n`);
|
|
120
|
+
continue;
|
|
121
|
+
}
|
|
122
|
+
// Rebuilt in the declared column order rather than stringifying `record`: an object literal's
|
|
123
|
+
// key order is whatever `row()` happened to write, and two attempts of the same export must
|
|
124
|
+
// produce byte-identical parts — that is what makes a replayed page an overwrite.
|
|
125
|
+
const object: Record<string, ExportValue> = {};
|
|
126
|
+
input.columns.forEach((column, at) => {
|
|
127
|
+
object[column] = cells[at] ?? null;
|
|
128
|
+
});
|
|
129
|
+
lines.push(`${JSON.stringify(object)}\n`);
|
|
130
|
+
}
|
|
131
|
+
return new TextEncoder().encode(lines.join(''));
|
|
132
|
+
}
|