express-file-cluster 0.3.14 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +204 -52
- package/dist/auth/index.cjs +12 -2
- package/dist/auth/index.cjs.map +1 -1
- package/dist/auth/index.d.cts +13 -2
- package/dist/auth/index.d.ts +13 -2
- package/dist/auth/index.js +12 -2
- package/dist/chunk-3VWD2ZED.js +60 -0
- package/dist/chunk-3VWD2ZED.js.map +1 -0
- package/dist/chunk-47K2532L.cjs +196 -0
- package/dist/chunk-47K2532L.cjs.map +1 -0
- package/dist/chunk-5PDAWMGF.js +438 -0
- package/dist/chunk-5PDAWMGF.js.map +1 -0
- package/dist/chunk-7YEX4MQV.cjs +38 -0
- package/dist/chunk-7YEX4MQV.cjs.map +1 -0
- package/dist/chunk-ALBDM5R3.cjs +60 -0
- package/dist/chunk-ALBDM5R3.cjs.map +1 -0
- package/dist/chunk-JJJJTACK.js +196 -0
- package/dist/chunk-JJJJTACK.js.map +1 -0
- package/dist/chunk-LJNSJFOL.js +38 -0
- package/dist/chunk-LJNSJFOL.js.map +1 -0
- package/dist/chunk-M7MUPWCE.js +116 -0
- package/dist/chunk-M7MUPWCE.js.map +1 -0
- package/dist/chunk-PMF4GMHD.js +154 -0
- package/dist/chunk-PMF4GMHD.js.map +1 -0
- package/dist/chunk-QTRGJ2EN.cjs +2635 -0
- package/dist/chunk-QTRGJ2EN.cjs.map +1 -0
- package/dist/chunk-RSBW2B3D.cjs +126 -0
- package/dist/chunk-RSBW2B3D.cjs.map +1 -0
- package/dist/chunk-RTFJ32T5.js +2635 -0
- package/dist/chunk-RTFJ32T5.js.map +1 -0
- package/dist/chunk-SFVEQ2T5.cjs +438 -0
- package/dist/chunk-SFVEQ2T5.cjs.map +1 -0
- package/dist/chunk-SZT36SGB.cjs +154 -0
- package/dist/chunk-SZT36SGB.cjs.map +1 -0
- package/dist/chunk-TI24L4XE.cjs.map +1 -1
- package/dist/chunk-ULQXD4YZ.cjs +116 -0
- package/dist/chunk-ULQXD4YZ.cjs.map +1 -0
- package/dist/chunk-VYHJX54R.js +126 -0
- package/dist/chunk-VYHJX54R.js.map +1 -0
- package/dist/cli/db-runner.cjs +497 -0
- package/dist/cli/db-runner.cjs.map +1 -0
- package/dist/cli/db-runner.d.cts +2 -0
- package/dist/cli/db-runner.d.ts +2 -0
- package/dist/cli/db-runner.js +497 -0
- package/dist/cli/db-runner.js.map +1 -0
- package/dist/cli/index.cjs +447 -69
- package/dist/cli/index.cjs.map +1 -1
- package/dist/cli/index.js +473 -95
- package/dist/cli/index.js.map +1 -1
- package/dist/docs/api-reference/auth.md +232 -0
- package/dist/docs/api-reference/compose.md +78 -0
- package/dist/docs/api-reference/db.md +351 -0
- package/dist/docs/api-reference/http-error.md +109 -0
- package/dist/docs/api-reference/ignite.md +323 -0
- package/dist/docs/api-reference/table.md +250 -0
- package/dist/docs/api-reference/tasks.md +140 -0
- package/dist/docs/cli/db.md +38 -0
- package/dist/docs/cli/diagnostics.md +145 -0
- package/dist/docs/cli/generate.md +120 -0
- package/dist/docs/cli/index.md +48 -0
- package/dist/docs/cli/start.md +112 -0
- package/dist/docs/contributing/index.md +139 -0
- package/dist/docs/core-concepts/background-tasks.md +228 -0
- package/dist/docs/core-concepts/clustering.md +114 -0
- package/dist/docs/core-concepts/file-based-routing.md +235 -0
- package/dist/docs/core-concepts/index.md +79 -0
- package/dist/docs/core-concepts/middleware.md +144 -0
- package/dist/docs/getting-started/index.md +91 -0
- package/dist/docs/getting-started/project-structure.md +154 -0
- package/dist/docs/guides/ai-agents.md +48 -0
- package/dist/docs/guides/authentication.md +196 -0
- package/dist/docs/guides/caching.md +124 -0
- package/dist/docs/guides/database.md +217 -0
- package/dist/docs/guides/deployment.md +171 -0
- package/dist/docs/guides/environment-variables.md +106 -0
- package/dist/docs/guides/error-handling.md +172 -0
- package/dist/docs/guides/generated-portals.md +115 -0
- package/dist/docs/guides/logging.md +70 -0
- package/dist/docs/guides/mailer.md +166 -0
- package/dist/docs/guides/migrations.md +106 -0
- package/dist/docs/guides/rbac.md +89 -0
- package/dist/docs/guides/validation.md +95 -0
- package/dist/docs/index.md +93 -0
- package/dist/docs/skills/SKILL.md +44 -0
- package/dist/docs/skills/api-surface.md +308 -0
- package/dist/docs/skills/auth-internals.md +141 -0
- package/dist/docs/skills/pre-flight.md +142 -0
- package/dist/docs/skills/routing-rules.md +114 -0
- package/dist/docs/skills/tasks-internals.md +172 -0
- package/dist/docs/skills/what-not-to-invent.md +251 -0
- package/dist/index.cjs +289 -391
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +472 -7
- package/dist/index.d.ts +472 -7
- package/dist/index.js +283 -385
- package/dist/index.js.map +1 -1
- package/dist/pgboss-backend-3E6ATY42.cjs +14 -0
- package/dist/pgboss-backend-3E6ATY42.cjs.map +1 -0
- package/dist/pgboss-backend-UUJOVEYK.js +14 -0
- package/dist/pgboss-backend-UUJOVEYK.js.map +1 -0
- package/dist/tasks/index.cjs.map +1 -1
- package/dist/tasks/index.d.cts +1 -1
- package/dist/tasks/index.d.ts +1 -1
- package/dist/types-DJCPAsM4.d.cts +541 -0
- package/dist/types-DJCPAsM4.d.ts +541 -0
- package/package.json +28 -5
- package/dist/chunk-6522HS2J.cjs +0 -99
- package/dist/chunk-6522HS2J.cjs.map +0 -1
- package/dist/chunk-DGVOJNR6.cjs +0 -55
- package/dist/chunk-DGVOJNR6.cjs.map +0 -1
- package/dist/chunk-KOHLLUN7.js +0 -55
- package/dist/chunk-KOHLLUN7.js.map +0 -1
- package/dist/chunk-VIP3PFWZ.js +0 -99
- package/dist/chunk-VIP3PFWZ.js.map +0 -1
- package/dist/types-BusbSenn.d.cts +0 -162
- package/dist/types-BusbSenn.d.ts +0 -162
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
## Repository layout
|
|
4
|
+
|
|
5
|
+
```
|
|
6
|
+
packages/
|
|
7
|
+
core/ → express-file-cluster (the framework)
|
|
8
|
+
create-efc-app/ → interactive scaffolder CLI
|
|
9
|
+
client/ → marketing site (static HTML/JS)
|
|
10
|
+
Skills/efc/ → reference cards for the /efc assistant skill
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Getting started
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
git clone https://github.com/your-org/efc.js.git
|
|
19
|
+
cd efc.js
|
|
20
|
+
|
|
21
|
+
# Install all workspace dependencies
|
|
22
|
+
npm install
|
|
23
|
+
|
|
24
|
+
# Build all packages
|
|
25
|
+
npm run build
|
|
26
|
+
|
|
27
|
+
# Run the test suite
|
|
28
|
+
npm test
|
|
29
|
+
|
|
30
|
+
# Type-check
|
|
31
|
+
npm run typecheck
|
|
32
|
+
|
|
33
|
+
# Lint
|
|
34
|
+
npm run lint
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
## Integration and end-to-end tests
|
|
40
|
+
|
|
41
|
+
Integration tests skip themselves when a service isn't reachable, so `npm test` works with nothing running — but for full coverage start MongoDB (plus a single-node replica set for transactions), Redis and PostgreSQL, and point the tests at them:
|
|
42
|
+
|
|
43
|
+
| Variable | Default |
|
|
44
|
+
|---|---|
|
|
45
|
+
| `EFC_TEST_MONGO_URL` | `mongodb://127.0.0.1:27017/...` |
|
|
46
|
+
| `EFC_TEST_MONGO_RS_URL` | `mongodb://127.0.0.1:27018/...?replicaSet=rs0&directConnection=true` |
|
|
47
|
+
| `EFC_TEST_REDIS_URL` | `redis://127.0.0.1:6379/15` |
|
|
48
|
+
| `EFC_TEST_PG_URL` | `postgres://postgres@127.0.0.1:5433/efc_test` |
|
|
49
|
+
|
|
50
|
+
The cluster and CLI tests run the built `dist/`, so `npm run build` first.
|
|
51
|
+
|
|
52
|
+
The scaffolder end-to-end suite generates real projects (MongoDB + BullMQ, PostgreSQL + pg-boss, and a JavaScript project), installs them against a packed tarball of `packages/core`, then migrates, runs the dev server, builds, boots and exercises auth over HTTP:
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
npm run build
|
|
56
|
+
EFC_E2E=1 npx vitest run --project create-efc-app e2e
|
|
57
|
+
EFC_E2E=1 EFC_E2E_KEEP=1 npx vitest run --project create-efc-app e2e # keep the generated apps to inspect
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
---
|
|
61
|
+
|
|
62
|
+
## Branch naming
|
|
63
|
+
|
|
64
|
+
| Type | Pattern | Example |
|
|
65
|
+
|---|---|---|
|
|
66
|
+
| Feature | `feat/<topic>` | `feat/postgresql-adapter` |
|
|
67
|
+
| Bug fix | `fix/<topic>` | `fix/route-sort-order` |
|
|
68
|
+
| Documentation | `docs/<topic>` | `docs/task-guide` |
|
|
69
|
+
| Chore | `chore/<topic>` | `chore/bump-bullmq` |
|
|
70
|
+
|
|
71
|
+
---
|
|
72
|
+
|
|
73
|
+
## Commit messages
|
|
74
|
+
|
|
75
|
+
Follow [Conventional Commits](https://www.conventionalcommits.org/):
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
feat(router): support catch-all wildcard routes
|
|
79
|
+
fix(cluster): prevent infinite respawn loop on startup crash
|
|
80
|
+
docs(auth): add localStorage strategy example
|
|
81
|
+
chore(deps): upgrade jose to v5
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Pull requests
|
|
87
|
+
|
|
88
|
+
Every PR must:
|
|
89
|
+
|
|
90
|
+
- Include tests for changed behaviour (unit or integration via Vitest).
|
|
91
|
+
- Pass `npm test`, `npm run lint`, and `npm run typecheck` in CI.
|
|
92
|
+
- Include a changelog entry under `Unreleased` in `CHANGELOG.md`.
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## Package architecture
|
|
97
|
+
|
|
98
|
+
### `packages/core`
|
|
99
|
+
|
|
100
|
+
The framework package published as `express-file-cluster`. Sub-path exports:
|
|
101
|
+
|
|
102
|
+
| Import | Source |
|
|
103
|
+
|---|---|
|
|
104
|
+
| `express-file-cluster` | `src/index.ts` — `ignite`, `gracefulShutdown`, `HttpError`, `compose`, `db`, `defineModel` |
|
|
105
|
+
| `express-file-cluster/auth` | `src/auth/index.ts` — `issueToken`, `revokeToken`, `revokeSession`, `revokeAllSessions`, `signToken`, `requireAuth`, `getCurrentUser` |
|
|
106
|
+
| `express-file-cluster/tasks` | `src/tasks/index.ts` — `defineTask`, `enqueue` |
|
|
107
|
+
| `express-file-cluster/cli` | `src/cli/index.ts` — the `efc` binary |
|
|
108
|
+
|
|
109
|
+
### `packages/create-efc-app`
|
|
110
|
+
|
|
111
|
+
Interactive scaffolder published as `create-efc-app`. Invoked with `npx create-efc-app <name>`. Prompts the user and writes the project tree using the templates in `src/scaffold.ts`.
|
|
112
|
+
|
|
113
|
+
---
|
|
114
|
+
|
|
115
|
+
## Roadmap
|
|
116
|
+
|
|
117
|
+
| Phase | Target | Focus |
|
|
118
|
+
|---|---|---|
|
|
119
|
+
| **1** | Done | Core MVP — router, clustering, auth, DB, tasks, CLI |
|
|
120
|
+
| **2** | Done | Beta — PostgreSQL, pg-boss, request validation, structured logging, cron tasks, catch-all routes, migrations, `defineModel` upgrades |
|
|
121
|
+
| **3** | Q1 2027 | Stable v1.0 — plugins, WebSockets, OpenAPI, OpenTelemetry, testing utilities |
|
|
122
|
+
| **4** | 2027+ | Edge/serverless, gRPC, GraphQL |
|
|
123
|
+
|
|
124
|
+
### Bug fix SLA
|
|
125
|
+
|
|
126
|
+
| Severity | SLA | Example |
|
|
127
|
+
|---|---|---|
|
|
128
|
+
| **P0 — Critical** | Patch within 48 h | Data loss, auth bypass, crash on startup |
|
|
129
|
+
| **P1 — High** | Patch within 1 week | Route not resolving, worker not respawning, task silently dropped |
|
|
130
|
+
| **P2 — Medium** | Next minor release | Wrong TypeScript types, misleading error message |
|
|
131
|
+
| **P3 — Low** | Best effort | Cosmetic CLI issues, doc typos |
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## Influencing the roadmap
|
|
136
|
+
|
|
137
|
+
1. **Open a GitHub Discussion** — propose a feature or share a use case.
|
|
138
|
+
2. **Upvote issues** — 👍 reactions influence prioritisation.
|
|
139
|
+
3. **Submit a PR** — the fastest path from idea to shipping.
|
|
@@ -0,0 +1,228 @@
|
|
|
1
|
+
# Background Tasks
|
|
2
|
+
|
|
3
|
+
The `src/tasks/` directory holds background jobs — work that should run *after* the HTTP response is sent, with retries, backoff, and concurrency control. The file name (without extension) is the task name used with `enqueue()`.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Why background tasks?
|
|
8
|
+
|
|
9
|
+
Two workloads hide behind "do this work":
|
|
10
|
+
|
|
11
|
+
| Workload | Example | How EFC runs it |
|
|
12
|
+
|---|---|---|
|
|
13
|
+
| **I/O-bound** | Send email, call a webhook, write to S3 | Queue → async handler on the event loop |
|
|
14
|
+
| **CPU-bound** | Resize an image, transcode video, parse a large file | Queue → `worker_threads` thread |
|
|
15
|
+
|
|
16
|
+
EFC does not hand-roll the queue. Persistence, retries, backoff, dead-lettering, scheduling and concurrency are delegated to a proven backend: **BullMQ** (Redis) or **pg-boss** (PostgreSQL — no Redis needed). Task code is identical on both.
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Defining a task
|
|
21
|
+
|
|
22
|
+
### I/O-bound task
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
// src/tasks/SendEmail.ts
|
|
26
|
+
import { defineTask } from 'express-file-cluster/tasks';
|
|
27
|
+
|
|
28
|
+
interface SendEmailPayload {
|
|
29
|
+
to: string;
|
|
30
|
+
subject: string;
|
|
31
|
+
body: string;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
export default defineTask<SendEmailPayload>(async ({ to, subject, body }) => {
|
|
35
|
+
await mailer.send({ to, subject, body });
|
|
36
|
+
});
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
> The `create-efc-app` scaffolder generates a real, working version of this file (using `nodemailer` + your configured SMTP credentials) when you enable the **Mailer** option — see [Mailer](../guides/mailer.md).
|
|
40
|
+
|
|
41
|
+
### CPU-bound task
|
|
42
|
+
|
|
43
|
+
Pass `{ thread: true }` as the first argument. The handler runs in a dedicated `worker_threads` thread, keeping the event loop free.
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
// src/tasks/ResizeImage.ts
|
|
47
|
+
import { defineTask } from 'express-file-cluster/tasks';
|
|
48
|
+
import sharp from 'sharp';
|
|
49
|
+
|
|
50
|
+
interface ResizePayload {
|
|
51
|
+
key: string;
|
|
52
|
+
width: number;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
export default defineTask<ResizePayload>(
|
|
56
|
+
{ thread: true },
|
|
57
|
+
async ({ key, width }) => {
|
|
58
|
+
const input = await downloadFromStorage(key);
|
|
59
|
+
const out = await sharp(input).resize(width).toBuffer();
|
|
60
|
+
await uploadToStorage(`${key}@${width}`, out);
|
|
61
|
+
},
|
|
62
|
+
);
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## Task options
|
|
68
|
+
|
|
69
|
+
| Option | Type | Default | Description |
|
|
70
|
+
|---|---|---|---|
|
|
71
|
+
| `thread` | `boolean` | `false` | Run in a `worker_threads` thread (CPU-bound work) |
|
|
72
|
+
| `retries` | `number` | `3` | Re-runs after a failure (up to `retries + 1` attempts) before dead-lettering |
|
|
73
|
+
| `backoff` | `'fixed' \| 'exponential'` | `'exponential'` | Delay strategy between retries (1 s base) |
|
|
74
|
+
| `concurrency` | `number` | inherits `tasks.concurrency` | Parallel jobs for this specific task |
|
|
75
|
+
| `schedule` | `string` | — | Cron pattern for recurring runs, e.g. `'0 3 * * *'` |
|
|
76
|
+
| `timezone` | `string` | server time | IANA timezone for `schedule`, e.g. `'Europe/Berlin'` |
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## Scheduled (cron) tasks
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
// src/tasks/NightlyReport.ts
|
|
84
|
+
import { defineTask } from 'express-file-cluster/tasks';
|
|
85
|
+
|
|
86
|
+
export default defineTask({ schedule: '0 3 * * *', timezone: 'UTC' }, async () => {
|
|
87
|
+
await buildAndEmailReport();
|
|
88
|
+
});
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Scheduled runs receive `{}` as the payload and get the same retries as enqueued jobs; the task can still be `enqueue()`d manually too. Schedules are reconciled on every boot: each cluster worker upserts them idempotently (no duplicate runs), and schedules for tasks that lost their `schedule` — or were deleted — are removed. BullMQ accepts a 6-field pattern with seconds; pg-boss's cron resolution is one minute.
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## Enqueuing a task from a route
|
|
96
|
+
|
|
97
|
+
```ts
|
|
98
|
+
// src/api/users/index.ts
|
|
99
|
+
import type { Request, Response } from 'express';
|
|
100
|
+
import { enqueue } from 'express-file-cluster/tasks';
|
|
101
|
+
import { User } from '../../model/User.js';
|
|
102
|
+
|
|
103
|
+
export const POST = async (req: Request, res: Response) => {
|
|
104
|
+
const user = await User.create(req.body);
|
|
105
|
+
|
|
106
|
+
// Fire-and-forget — respond immediately, job runs off-path
|
|
107
|
+
await enqueue('SendEmail', {
|
|
108
|
+
to: user.email,
|
|
109
|
+
subject: 'Welcome!',
|
|
110
|
+
body: 'Thanks for signing up.',
|
|
111
|
+
});
|
|
112
|
+
|
|
113
|
+
res.status(202).json({ id: user.id, queued: true });
|
|
114
|
+
};
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
`enqueue(name, payload)` rejects if:
|
|
118
|
+
- The task backend has not been initialised (missing `tasks` config in `ignite()`).
|
|
119
|
+
- The task name was not found in the registry (file not in `src/tasks/`, or bad default export).
|
|
120
|
+
- Redis did not accept the job within `tasks.enqueueTimeout` (default 5000 ms). The request fails fast instead of hanging, and the rejected job is not silently queued later.
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
124
|
+
## Task registration
|
|
125
|
+
|
|
126
|
+
On startup, the task scanner walks `src/tasks/` (non-recursively) and:
|
|
127
|
+
|
|
128
|
+
1. Reads each `.ts` / `.js` file.
|
|
129
|
+
2. Imports the module and reads the `default` export.
|
|
130
|
+
3. Validates that `default.handler` is a function.
|
|
131
|
+
4. Calls `registerTask(basename, definition)` to add it to the in-memory registry.
|
|
132
|
+
|
|
133
|
+
The task's `filePath` is stored so `thread: true` tasks can be loaded by the `worker_threads` runner without re-importing the module in the main thread.
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## BullMQ backend
|
|
138
|
+
|
|
139
|
+
When `tasks.backend === 'bullmq'`, `initBullMQ()` in `packages/core/src/tasks/bullmq-backend.ts`:
|
|
140
|
+
|
|
141
|
+
1. Dynamically imports `bullmq` (peer dependency — not bundled).
|
|
142
|
+
2. Creates a `Queue` named `tasks.queue` (default `'efc'`) connected to Redis.
|
|
143
|
+
3. Creates a `Worker` on that queue with the configured `concurrency`, and reconciles cron schedulers in the background (a Redis outage at boot doesn't block the HTTP server).
|
|
144
|
+
4. The worker's processor looks up the task in the registry and either:
|
|
145
|
+
- Calls `def.handler(job.data)` directly (I/O-bound), or
|
|
146
|
+
- Calls `runInThread(def.filePath, job.data)` (CPU-bound).
|
|
147
|
+
5. Wires `setEnqueueImpl` so `enqueue()` calls `queue.add(name, payload, { attempts: retries + 1, backoff })`, bounded by `enqueueTimeout`.
|
|
148
|
+
|
|
149
|
+
On shutdown, `gracefulShutdown()` closes the HTTP server first, then lets the worker finish in-flight jobs before the process exits (still capped by its timeout).
|
|
150
|
+
|
|
151
|
+
### Redis connection
|
|
152
|
+
|
|
153
|
+
The `REDIS_URL` env var (or `tasks.redisUrl` in `ignite()`) is parsed by the built-in `parseRedisUrl` helper. It supports `redis://[user:password@]host[:port][/db]` and `rediss://` (TLS) URLs, so hosted Redis with ACL users and TLS (Upstash, Redis Cloud, ElastiCache) works as-is.
|
|
154
|
+
|
|
155
|
+
```ts
|
|
156
|
+
ignite({
|
|
157
|
+
tasks: {
|
|
158
|
+
backend: 'bullmq',
|
|
159
|
+
redisUrl: process.env.REDIS_URL, // or omit — defaults to redis://localhost:6379
|
|
160
|
+
concurrency: 5,
|
|
161
|
+
enqueueTimeout: 5000, // ms — enqueue() rejects instead of hanging when Redis is down
|
|
162
|
+
queue: 'efc', // apps sharing one Redis database need distinct names
|
|
163
|
+
},
|
|
164
|
+
});
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
---
|
|
168
|
+
|
|
169
|
+
## pg-boss backend
|
|
170
|
+
|
|
171
|
+
For PostgreSQL stacks that don't want to run Redis:
|
|
172
|
+
|
|
173
|
+
```bash
|
|
174
|
+
npm install pg-boss
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
```ts
|
|
178
|
+
ignite({
|
|
179
|
+
databaseUrl: process.env.DATABASE_URL,
|
|
180
|
+
tasks: {
|
|
181
|
+
backend: 'pg-boss',
|
|
182
|
+
// connectionString: '…', // default: databaseUrl
|
|
183
|
+
// schema: 'pgboss', // where pg-boss keeps its tables
|
|
184
|
+
concurrency: 5,
|
|
185
|
+
},
|
|
186
|
+
});
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
pg-boss stores jobs in your Postgres (its own `pgboss` schema, created on first start). Each task becomes a pg-boss queue named `<tasks.queue>.<TaskName>`; retries, backoff, cron, graceful drain on shutdown and `enqueueTimeout` behave as with BullMQ. Any payload works — pg-boss requires objects, so other values are boxed transparently.
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## Dead letters
|
|
194
|
+
|
|
195
|
+
A job that exhausts its retries stays in the backend's failed set. Inspect and act on it from any shell:
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
efc tasks failed # list (with error, attempts, payload) + queue counts
|
|
199
|
+
efc tasks failed --name SendEmail --limit 50 --json
|
|
200
|
+
efc tasks retry <jobId…> # re-queue specific jobs (one more attempt each)
|
|
201
|
+
efc tasks retry --all
|
|
202
|
+
efc tasks purge --all # delete failed jobs
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
The backend is inferred — BullMQ when `REDIS_URL` is set, pg-boss for a `postgres` `DATABASE_URL` — or pass `--backend`, `--queue`, `--redis-url`, `--database-url`. See [CLI › diagnostics](../cli/diagnostics.md).
|
|
206
|
+
|
|
207
|
+
---
|
|
208
|
+
|
|
209
|
+
## Thread runner
|
|
210
|
+
|
|
211
|
+
`packages/core/src/tasks/thread-runner.ts` is a dual-purpose module:
|
|
212
|
+
|
|
213
|
+
- **Worker-thread side** — when loaded inside a `worker_threads` worker, reads `workerData.handlerPath` and `workerData.payload`, imports the task file, calls `def.handler(payload)`, and posts `{ ok: true }` or `{ ok: false, error }` back to the main thread.
|
|
214
|
+
- **Main-thread side** — exports `runInThread(handlerPath, payload)` which spawns a `new Worker(thread-runner.js, { workerData })` and returns a `Promise<void>` that resolves on `{ ok: true }` or rejects on error.
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
|
|
218
|
+
## Imports
|
|
219
|
+
|
|
220
|
+
```ts
|
|
221
|
+
// Task definition
|
|
222
|
+
import { defineTask } from 'express-file-cluster/tasks';
|
|
223
|
+
|
|
224
|
+
// Enqueue from anywhere (route handlers, other tasks, etc.)
|
|
225
|
+
import { enqueue } from 'express-file-cluster/tasks';
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Both are exported from the `tasks` sub-path export of `express-file-cluster`.
|
|
@@ -0,0 +1,114 @@
|
|
|
1
|
+
# Clustering
|
|
2
|
+
|
|
3
|
+
EFC uses Node's built-in `cluster` module to fork identical worker processes — one per CPU core by default. The OS load-balances incoming connections across them with no application-level proxy needed.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## How it works
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
Master Process (cluster.isPrimary)
|
|
11
|
+
┌────────────────────────────────────┐
|
|
12
|
+
│ fork × N workers │
|
|
13
|
+
│ respawn on 'exit' │
|
|
14
|
+
└──┬──────────┬──────────┬───────────┘
|
|
15
|
+
│ │ │
|
|
16
|
+
Worker 1 Worker 2 Worker N
|
|
17
|
+
Pre-Flight Pre-Flight Pre-Flight
|
|
18
|
+
DB Connect DB Connect DB Connect
|
|
19
|
+
Auth Config Auth Config Auth Config
|
|
20
|
+
Scan Routes Scan Routes Scan Routes
|
|
21
|
+
Reg. Tasks Reg. Tasks Reg. Tasks
|
|
22
|
+
HTTP Listen HTTP Listen HTTP Listen
|
|
23
|
+
↑
|
|
24
|
+
OS round-robins connections
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
When `cluster: true` (or `NODE_ENV === 'production'` with no explicit value), `ignite()` checks `cluster.isPrimary`:
|
|
28
|
+
|
|
29
|
+
- **Primary** — calls `runMaster()`, forks workers, watches for exits, and returns `undefined`.
|
|
30
|
+
- **Worker** — runs the full Pre-Flight lifecycle, starts Express, and returns an `http.Server`.
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Pre-Flight lifecycle (per worker)
|
|
35
|
+
|
|
36
|
+
Every worker runs these steps in order before accepting any connections:
|
|
37
|
+
|
|
38
|
+
| Step | What happens |
|
|
39
|
+
|---|---|
|
|
40
|
+
| 0. Configure logging | Applies `logging` — every later line carries the worker label (`w<N>`). |
|
|
41
|
+
| 1. Connect Database | Establishes this worker's own connection pool to MongoDB or PostgreSQL, and selects what `defineModel` compiles to. |
|
|
42
|
+
| 2. Configure Auth | Wires the JWT secret, strategy, and cookie options into the auth module. |
|
|
43
|
+
| 3. Scan tasks | Walks `src/tasks/` and registers every `defineTask` export in the task registry. |
|
|
44
|
+
| 4. Start task backend | Connects BullMQ (Redis) or pg-boss (PostgreSQL), starts the queue worker and reconciles cron schedules. |
|
|
45
|
+
| 5. Scan routes | Walks `src/api/` and builds the `RouteEntry[]` array. |
|
|
46
|
+
| 6. Mount routes | Dynamically imports each route module and registers handlers on Express. |
|
|
47
|
+
| 7. Listen | Starts the HTTP server and logs `Worker <id> listening on :<port>`. |
|
|
48
|
+
|
|
49
|
+
Each worker owns its own database connection pool. There is no shared mutable state across workers.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Configuration
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
ignite({
|
|
57
|
+
// cluster defaults to NODE_ENV === 'production' — only pass it explicitly to
|
|
58
|
+
// force one way regardless of NODE_ENV
|
|
59
|
+
workers: 4, // defaults to os.cpus().length
|
|
60
|
+
onWorkerReady: (id) => console.log(`Worker ${id} ready`),
|
|
61
|
+
onWorkerCrash: (id, code) => console.error(`Worker ${id} crashed (${code})`),
|
|
62
|
+
});
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
| Option | Type | Default | Description |
|
|
66
|
+
|---|---|---|---|
|
|
67
|
+
| `cluster` | `boolean` | `true` when `NODE_ENV === 'production'` | Fork worker processes |
|
|
68
|
+
| `workers` | `number` | `os.cpus().length` | Number of workers to fork |
|
|
69
|
+
| `onWorkerReady` | `(id: number) => void` | — | Called when a worker comes online |
|
|
70
|
+
| `onWorkerCrash` | `(id: number, code: number) => void` | — | Called before a crashed worker is replaced |
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Self-healing
|
|
75
|
+
|
|
76
|
+
If a worker process exits (crash, unhandled rejection, OOM), the master:
|
|
77
|
+
|
|
78
|
+
1. Fires `onWorkerCrash(id, exitCode)`.
|
|
79
|
+
2. Immediately calls `cluster.fork()` to spawn a replacement.
|
|
80
|
+
3. The new worker runs the full Pre-Flight lifecycle before serving traffic.
|
|
81
|
+
|
|
82
|
+
The replacement is instantaneous from the master's perspective — there is no delay or exponential backoff. If workers crash in a tight loop (bad config, missing env var), the master keeps forking. Use `efc doctor` to diagnose configuration problems before deploying.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Disabling clustering
|
|
87
|
+
|
|
88
|
+
Set `cluster: false` to run a single-process server. EFC forces clustering off in development (`efc start dev`) for faster restarts and cleaner stack traces.
|
|
89
|
+
|
|
90
|
+
```ts
|
|
91
|
+
ignite({ cluster: false }); // single process — same as efc start dev
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## Graceful shutdown
|
|
97
|
+
|
|
98
|
+
`gracefulShutdown(server)` registers `SIGTERM` and `SIGINT` handlers:
|
|
99
|
+
|
|
100
|
+
```ts
|
|
101
|
+
import { ignite, gracefulShutdown } from 'express-file-cluster';
|
|
102
|
+
|
|
103
|
+
ignite(config)
|
|
104
|
+
.then(gracefulShutdown);
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
- On a **worker**: calls `server.closeIdleConnections()`, then `server.close()` (in-flight requests finish), drains running task jobs, and exits `0`. Forces exit after a 10-second timeout. Repeated signals are ignored rather than killing it mid-drain.
|
|
108
|
+
- On the **primary** (when `ignite()` returns `undefined`): stops respawning, **relays `SIGTERM` to every worker** — orchestrators such as `docker stop` and Kubernetes signal only PID 1 — and exits once all workers have drained, or force-exits after the timeout + 2 s.
|
|
109
|
+
|
|
110
|
+
Pass a custom timeout (milliseconds) as the second argument:
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
gracefulShutdown(server, 30_000); // 30-second drain window
|
|
114
|
+
```
|