express-file-cluster 0.3.15 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (116) hide show
  1. package/README.md +71 -33
  2. package/dist/auth/index.cjs +12 -2
  3. package/dist/auth/index.cjs.map +1 -1
  4. package/dist/auth/index.d.cts +13 -2
  5. package/dist/auth/index.d.ts +13 -2
  6. package/dist/auth/index.js +12 -2
  7. package/dist/chunk-277BSYVH.js +127 -0
  8. package/dist/chunk-277BSYVH.js.map +1 -0
  9. package/dist/chunk-6OGOVV57.js +3847 -0
  10. package/dist/chunk-6OGOVV57.js.map +1 -0
  11. package/dist/chunk-7YEX4MQV.cjs +38 -0
  12. package/dist/chunk-7YEX4MQV.cjs.map +1 -0
  13. package/dist/chunk-AZYXQU3Z.cjs +3847 -0
  14. package/dist/chunk-AZYXQU3Z.cjs.map +1 -0
  15. package/dist/chunk-BUBXO7WU.js +170 -0
  16. package/dist/chunk-BUBXO7WU.js.map +1 -0
  17. package/dist/chunk-LJNSJFOL.js +38 -0
  18. package/dist/chunk-LJNSJFOL.js.map +1 -0
  19. package/dist/chunk-MGH4VDV3.cjs +170 -0
  20. package/dist/chunk-MGH4VDV3.cjs.map +1 -0
  21. package/dist/chunk-R34R7QPD.cjs +116 -0
  22. package/dist/chunk-R34R7QPD.cjs.map +1 -0
  23. package/dist/chunk-RS4MXCPW.js +116 -0
  24. package/dist/chunk-RS4MXCPW.js.map +1 -0
  25. package/dist/chunk-T4OHFU5T.cjs +154 -0
  26. package/dist/chunk-T4OHFU5T.cjs.map +1 -0
  27. package/dist/chunk-TI24L4XE.cjs.map +1 -1
  28. package/dist/chunk-TRC4DUZB.js +438 -0
  29. package/dist/chunk-TRC4DUZB.js.map +1 -0
  30. package/dist/chunk-TYR2GZZE.cjs +196 -0
  31. package/dist/chunk-TYR2GZZE.cjs.map +1 -0
  32. package/dist/chunk-UBQYQPBU.js +154 -0
  33. package/dist/chunk-UBQYQPBU.js.map +1 -0
  34. package/dist/chunk-UTM4PEVG.js +196 -0
  35. package/dist/chunk-UTM4PEVG.js.map +1 -0
  36. package/dist/chunk-YL37D6KM.cjs +127 -0
  37. package/dist/chunk-YL37D6KM.cjs.map +1 -0
  38. package/dist/chunk-ZDM5JHVK.cjs +438 -0
  39. package/dist/chunk-ZDM5JHVK.cjs.map +1 -0
  40. package/dist/cli/db-runner.cjs +497 -0
  41. package/dist/cli/db-runner.cjs.map +1 -0
  42. package/dist/cli/db-runner.d.cts +2 -0
  43. package/dist/cli/db-runner.d.ts +2 -0
  44. package/dist/cli/db-runner.js +497 -0
  45. package/dist/cli/db-runner.js.map +1 -0
  46. package/dist/cli/index.cjs +447 -69
  47. package/dist/cli/index.cjs.map +1 -1
  48. package/dist/cli/index.js +473 -95
  49. package/dist/cli/index.js.map +1 -1
  50. package/dist/docs/api-reference/auth.md +232 -0
  51. package/dist/docs/api-reference/compose.md +78 -0
  52. package/dist/docs/api-reference/db.md +351 -0
  53. package/dist/docs/api-reference/http-error.md +109 -0
  54. package/dist/docs/api-reference/ignite.md +326 -0
  55. package/dist/docs/api-reference/table.md +458 -0
  56. package/dist/docs/api-reference/tasks.md +140 -0
  57. package/dist/docs/cli/db.md +38 -0
  58. package/dist/docs/cli/diagnostics.md +145 -0
  59. package/dist/docs/cli/generate.md +120 -0
  60. package/dist/docs/cli/index.md +48 -0
  61. package/dist/docs/cli/start.md +112 -0
  62. package/dist/docs/contributing/index.md +139 -0
  63. package/dist/docs/core-concepts/background-tasks.md +228 -0
  64. package/dist/docs/core-concepts/clustering.md +114 -0
  65. package/dist/docs/core-concepts/file-based-routing.md +235 -0
  66. package/dist/docs/core-concepts/index.md +79 -0
  67. package/dist/docs/core-concepts/middleware.md +144 -0
  68. package/dist/docs/getting-started/index.md +91 -0
  69. package/dist/docs/getting-started/project-structure.md +154 -0
  70. package/dist/docs/guides/ai-agents.md +48 -0
  71. package/dist/docs/guides/authentication.md +196 -0
  72. package/dist/docs/guides/caching.md +124 -0
  73. package/dist/docs/guides/database.md +217 -0
  74. package/dist/docs/guides/deployment.md +171 -0
  75. package/dist/docs/guides/environment-variables.md +106 -0
  76. package/dist/docs/guides/error-handling.md +172 -0
  77. package/dist/docs/guides/generated-portals.md +115 -0
  78. package/dist/docs/guides/logging.md +108 -0
  79. package/dist/docs/guides/mailer.md +166 -0
  80. package/dist/docs/guides/migrations.md +106 -0
  81. package/dist/docs/guides/rbac.md +89 -0
  82. package/dist/docs/guides/validation.md +95 -0
  83. package/dist/docs/index.md +93 -0
  84. package/dist/docs/skills/SKILL.md +44 -0
  85. package/dist/docs/skills/api-surface.md +309 -0
  86. package/dist/docs/skills/auth-internals.md +141 -0
  87. package/dist/docs/skills/pre-flight.md +142 -0
  88. package/dist/docs/skills/routing-rules.md +114 -0
  89. package/dist/docs/skills/tasks-internals.md +172 -0
  90. package/dist/docs/skills/what-not-to-invent.md +251 -0
  91. package/dist/index.cjs +264 -391
  92. package/dist/index.cjs.map +1 -1
  93. package/dist/index.d.cts +699 -7
  94. package/dist/index.d.ts +699 -7
  95. package/dist/index.js +258 -385
  96. package/dist/index.js.map +1 -1
  97. package/dist/pgboss-backend-QKBXVJ6B.js +14 -0
  98. package/dist/pgboss-backend-QKBXVJ6B.js.map +1 -0
  99. package/dist/pgboss-backend-TCHI7CHM.cjs +14 -0
  100. package/dist/pgboss-backend-TCHI7CHM.cjs.map +1 -0
  101. package/dist/tasks/index.cjs.map +1 -1
  102. package/dist/tasks/index.d.cts +1 -1
  103. package/dist/tasks/index.d.ts +1 -1
  104. package/dist/types-B6lVw4bj.d.cts +588 -0
  105. package/dist/types-B6lVw4bj.d.ts +588 -0
  106. package/package.json +28 -5
  107. package/dist/chunk-6522HS2J.cjs +0 -99
  108. package/dist/chunk-6522HS2J.cjs.map +0 -1
  109. package/dist/chunk-DGVOJNR6.cjs +0 -55
  110. package/dist/chunk-DGVOJNR6.cjs.map +0 -1
  111. package/dist/chunk-KOHLLUN7.js +0 -55
  112. package/dist/chunk-KOHLLUN7.js.map +0 -1
  113. package/dist/chunk-VIP3PFWZ.js +0 -99
  114. package/dist/chunk-VIP3PFWZ.js.map +0 -1
  115. package/dist/types-BusbSenn.d.cts +0 -162
  116. package/dist/types-BusbSenn.d.ts +0 -162
@@ -0,0 +1,228 @@
1
+ # Background Tasks
2
+
3
+ The `src/tasks/` directory holds background jobs — work that should run *after* the HTTP response is sent, with retries, backoff, and concurrency control. The file name (without extension) is the task name used with `enqueue()`.
4
+
5
+ ---
6
+
7
+ ## Why background tasks?
8
+
9
+ Two workloads hide behind "do this work":
10
+
11
+ | Workload | Example | How EFC runs it |
12
+ |---|---|---|
13
+ | **I/O-bound** | Send email, call a webhook, write to S3 | Queue → async handler on the event loop |
14
+ | **CPU-bound** | Resize an image, transcode video, parse a large file | Queue → `worker_threads` thread |
15
+
16
+ EFC does not hand-roll the queue. Persistence, retries, backoff, dead-lettering, scheduling and concurrency are delegated to a proven backend: **BullMQ** (Redis) or **pg-boss** (PostgreSQL — no Redis needed). Task code is identical on both.
17
+
18
+ ---
19
+
20
+ ## Defining a task
21
+
22
+ ### I/O-bound task
23
+
24
+ ```ts
25
+ // src/tasks/SendEmail.ts
26
+ import { defineTask } from 'express-file-cluster/tasks';
27
+
28
+ interface SendEmailPayload {
29
+ to: string;
30
+ subject: string;
31
+ body: string;
32
+ }
33
+
34
+ export default defineTask<SendEmailPayload>(async ({ to, subject, body }) => {
35
+ await mailer.send({ to, subject, body });
36
+ });
37
+ ```
38
+
39
+ > The `create-efc-app` scaffolder generates a real, working version of this file (using `nodemailer` + your configured SMTP credentials) when you enable the **Mailer** option — see [Mailer](../guides/mailer.md).
40
+
41
+ ### CPU-bound task
42
+
43
+ Pass `{ thread: true }` as the first argument. The handler runs in a dedicated `worker_threads` thread, keeping the event loop free.
44
+
45
+ ```ts
46
+ // src/tasks/ResizeImage.ts
47
+ import { defineTask } from 'express-file-cluster/tasks';
48
+ import sharp from 'sharp';
49
+
50
+ interface ResizePayload {
51
+ key: string;
52
+ width: number;
53
+ }
54
+
55
+ export default defineTask<ResizePayload>(
56
+ { thread: true },
57
+ async ({ key, width }) => {
58
+ const input = await downloadFromStorage(key);
59
+ const out = await sharp(input).resize(width).toBuffer();
60
+ await uploadToStorage(`${key}@${width}`, out);
61
+ },
62
+ );
63
+ ```
64
+
65
+ ---
66
+
67
+ ## Task options
68
+
69
+ | Option | Type | Default | Description |
70
+ |---|---|---|---|
71
+ | `thread` | `boolean` | `false` | Run in a `worker_threads` thread (CPU-bound work) |
72
+ | `retries` | `number` | `3` | Re-runs after a failure (up to `retries + 1` attempts) before dead-lettering |
73
+ | `backoff` | `'fixed' \| 'exponential'` | `'exponential'` | Delay strategy between retries (1 s base) |
74
+ | `concurrency` | `number` | inherits `tasks.concurrency` | Parallel jobs for this specific task |
75
+ | `schedule` | `string` | — | Cron pattern for recurring runs, e.g. `'0 3 * * *'` |
76
+ | `timezone` | `string` | server time | IANA timezone for `schedule`, e.g. `'Europe/Berlin'` |
77
+
78
+ ---
79
+
80
+ ## Scheduled (cron) tasks
81
+
82
+ ```ts
83
+ // src/tasks/NightlyReport.ts
84
+ import { defineTask } from 'express-file-cluster/tasks';
85
+
86
+ export default defineTask({ schedule: '0 3 * * *', timezone: 'UTC' }, async () => {
87
+ await buildAndEmailReport();
88
+ });
89
+ ```
90
+
91
+ Scheduled runs receive `{}` as the payload and get the same retries as enqueued jobs; the task can still be `enqueue()`d manually too. Schedules are reconciled on every boot: each cluster worker upserts them idempotently (no duplicate runs), and schedules for tasks that lost their `schedule` — or were deleted — are removed. BullMQ accepts a 6-field pattern with seconds; pg-boss's cron resolution is one minute.
92
+
93
+ ---
94
+
95
+ ## Enqueuing a task from a route
96
+
97
+ ```ts
98
+ // src/api/users/index.ts
99
+ import type { Request, Response } from 'express';
100
+ import { enqueue } from 'express-file-cluster/tasks';
101
+ import { User } from '../../model/User.js';
102
+
103
+ export const POST = async (req: Request, res: Response) => {
104
+ const user = await User.create(req.body);
105
+
106
+ // Fire-and-forget — respond immediately, job runs off-path
107
+ await enqueue('SendEmail', {
108
+ to: user.email,
109
+ subject: 'Welcome!',
110
+ body: 'Thanks for signing up.',
111
+ });
112
+
113
+ res.status(202).json({ id: user.id, queued: true });
114
+ };
115
+ ```
116
+
117
+ `enqueue(name, payload)` rejects if:
118
+ - The task backend has not been initialised (missing `tasks` config in `ignite()`).
119
+ - The task name was not found in the registry (file not in `src/tasks/`, or bad default export).
120
+ - Redis did not accept the job within `tasks.enqueueTimeout` (default 5000 ms). The request fails fast instead of hanging, and the rejected job is not silently queued later.
121
+
122
+ ---
123
+
124
+ ## Task registration
125
+
126
+ On startup, the task scanner walks `src/tasks/` (non-recursively) and:
127
+
128
+ 1. Reads each `.ts` / `.js` file.
129
+ 2. Imports the module and reads the `default` export.
130
+ 3. Validates that `default.handler` is a function.
131
+ 4. Calls `registerTask(basename, definition)` to add it to the in-memory registry.
132
+
133
+ The task's `filePath` is stored so `thread: true` tasks can be loaded by the `worker_threads` runner without re-importing the module in the main thread.
134
+
135
+ ---
136
+
137
+ ## BullMQ backend
138
+
139
+ When `tasks.backend === 'bullmq'`, `initBullMQ()` in `packages/core/src/tasks/bullmq-backend.ts`:
140
+
141
+ 1. Dynamically imports `bullmq` (peer dependency — not bundled).
142
+ 2. Creates a `Queue` named `tasks.queue` (default `'efc'`) connected to Redis.
143
+ 3. Creates a `Worker` on that queue with the configured `concurrency`, and reconciles cron schedulers in the background (a Redis outage at boot doesn't block the HTTP server).
144
+ 4. The worker's processor looks up the task in the registry and either:
145
+ - Calls `def.handler(job.data)` directly (I/O-bound), or
146
+ - Calls `runInThread(def.filePath, job.data)` (CPU-bound).
147
+ 5. Wires `setEnqueueImpl` so `enqueue()` calls `queue.add(name, payload, { attempts: retries + 1, backoff })`, bounded by `enqueueTimeout`.
148
+
149
+ On shutdown, `gracefulShutdown()` closes the HTTP server first, then lets the worker finish in-flight jobs before the process exits (still capped by its timeout).
150
+
151
+ ### Redis connection
152
+
153
+ The `REDIS_URL` env var (or `tasks.redisUrl` in `ignite()`) is parsed by the built-in `parseRedisUrl` helper. It supports `redis://[user:password@]host[:port][/db]` and `rediss://` (TLS) URLs, so hosted Redis with ACL users and TLS (Upstash, Redis Cloud, ElastiCache) works as-is.
154
+
155
+ ```ts
156
+ ignite({
157
+ tasks: {
158
+ backend: 'bullmq',
159
+ redisUrl: process.env.REDIS_URL, // or omit — defaults to redis://localhost:6379
160
+ concurrency: 5,
161
+ enqueueTimeout: 5000, // ms — enqueue() rejects instead of hanging when Redis is down
162
+ queue: 'efc', // apps sharing one Redis database need distinct names
163
+ },
164
+ });
165
+ ```
166
+
167
+ ---
168
+
169
+ ## pg-boss backend
170
+
171
+ For PostgreSQL stacks that don't want to run Redis:
172
+
173
+ ```bash
174
+ npm install pg-boss
175
+ ```
176
+
177
+ ```ts
178
+ ignite({
179
+ databaseUrl: process.env.DATABASE_URL,
180
+ tasks: {
181
+ backend: 'pg-boss',
182
+ // connectionString: '…', // default: databaseUrl
183
+ // schema: 'pgboss', // where pg-boss keeps its tables
184
+ concurrency: 5,
185
+ },
186
+ });
187
+ ```
188
+
189
+ pg-boss stores jobs in your Postgres (its own `pgboss` schema, created on first start). Each task becomes a pg-boss queue named `<tasks.queue>.<TaskName>`; retries, backoff, cron, graceful drain on shutdown and `enqueueTimeout` behave as with BullMQ. Any payload works — pg-boss requires objects, so other values are boxed transparently.
190
+
191
+ ---
192
+
193
+ ## Dead letters
194
+
195
+ A job that exhausts its retries stays in the backend's failed set. Inspect and act on it from any shell:
196
+
197
+ ```bash
198
+ efc tasks failed # list (with error, attempts, payload) + queue counts
199
+ efc tasks failed --name SendEmail --limit 50 --json
200
+ efc tasks retry <jobId…> # re-queue specific jobs (one more attempt each)
201
+ efc tasks retry --all
202
+ efc tasks purge --all # delete failed jobs
203
+ ```
204
+
205
+ The backend is inferred — BullMQ when `REDIS_URL` is set, pg-boss for a `postgres` `DATABASE_URL` — or pass `--backend`, `--queue`, `--redis-url`, `--database-url`. See [CLI › diagnostics](../cli/diagnostics.md).
206
+
207
+ ---
208
+
209
+ ## Thread runner
210
+
211
+ `packages/core/src/tasks/thread-runner.ts` is a dual-purpose module:
212
+
213
+ - **Worker-thread side** — when loaded inside a `worker_threads` worker, reads `workerData.handlerPath` and `workerData.payload`, imports the task file, calls `def.handler(payload)`, and posts `{ ok: true }` or `{ ok: false, error }` back to the main thread.
214
+ - **Main-thread side** — exports `runInThread(handlerPath, payload)` which spawns a `new Worker(thread-runner.js, { workerData })` and returns a `Promise<void>` that resolves on `{ ok: true }` or rejects on error.
215
+
216
+ ---
217
+
218
+ ## Imports
219
+
220
+ ```ts
221
+ // Task definition
222
+ import { defineTask } from 'express-file-cluster/tasks';
223
+
224
+ // Enqueue from anywhere (route handlers, other tasks, etc.)
225
+ import { enqueue } from 'express-file-cluster/tasks';
226
+ ```
227
+
228
+ Both are exported from the `tasks` sub-path export of `express-file-cluster`.
@@ -0,0 +1,114 @@
1
+ # Clustering
2
+
3
+ EFC uses Node's built-in `cluster` module to fork identical worker processes — one per CPU core by default. The OS load-balances incoming connections across them with no application-level proxy needed.
4
+
5
+ ---
6
+
7
+ ## How it works
8
+
9
+ ```
10
+ Master Process (cluster.isPrimary)
11
+ ┌────────────────────────────────────┐
12
+ │ fork × N workers │
13
+ │ respawn on 'exit' │
14
+ └──┬──────────┬──────────┬───────────┘
15
+ │ │ │
16
+ Worker 1 Worker 2 Worker N
17
+ Pre-Flight Pre-Flight Pre-Flight
18
+ DB Connect DB Connect DB Connect
19
+ Auth Config Auth Config Auth Config
20
+ Scan Routes Scan Routes Scan Routes
21
+ Reg. Tasks Reg. Tasks Reg. Tasks
22
+ HTTP Listen HTTP Listen HTTP Listen
23
+ ↑
24
+ OS round-robins connections
25
+ ```
26
+
27
+ When `cluster: true` (or `NODE_ENV === 'production'` with no explicit value), `ignite()` checks `cluster.isPrimary`:
28
+
29
+ - **Primary** — calls `runMaster()`, forks workers, watches for exits, and returns `undefined`.
30
+ - **Worker** — runs the full Pre-Flight lifecycle, starts Express, and returns an `http.Server`.
31
+
32
+ ---
33
+
34
+ ## Pre-Flight lifecycle (per worker)
35
+
36
+ Every worker runs these steps in order before accepting any connections:
37
+
38
+ | Step | What happens |
39
+ |---|---|
40
+ | 0. Configure logging | Applies `logging` — every later line carries the worker label (`w<N>`). |
41
+ | 1. Connect Database | Establishes this worker's own connection pool to MongoDB or PostgreSQL, and selects what `defineModel` compiles to. |
42
+ | 2. Configure Auth | Wires the JWT secret, strategy, and cookie options into the auth module. |
43
+ | 3. Scan tasks | Walks `src/tasks/` and registers every `defineTask` export in the task registry. |
44
+ | 4. Start task backend | Connects BullMQ (Redis) or pg-boss (PostgreSQL), starts the queue worker and reconciles cron schedules. |
45
+ | 5. Scan routes | Walks `src/api/` and builds the `RouteEntry[]` array. |
46
+ | 6. Mount routes | Dynamically imports each route module and registers handlers on Express. |
47
+ | 7. Listen | Starts the HTTP server and logs `Worker <id> listening on :<port>`. |
48
+
49
+ Each worker owns its own database connection pool. There is no shared mutable state across workers.
50
+
51
+ ---
52
+
53
+ ## Configuration
54
+
55
+ ```ts
56
+ ignite({
57
+ // cluster defaults to NODE_ENV === 'production' — only pass it explicitly to
58
+ // force one way regardless of NODE_ENV
59
+ workers: 4, // defaults to os.cpus().length
60
+ onWorkerReady: (id) => console.log(`Worker ${id} ready`),
61
+ onWorkerCrash: (id, code) => console.error(`Worker ${id} crashed (${code})`),
62
+ });
63
+ ```
64
+
65
+ | Option | Type | Default | Description |
66
+ |---|---|---|---|
67
+ | `cluster` | `boolean` | `true` when `NODE_ENV === 'production'` | Fork worker processes |
68
+ | `workers` | `number` | `os.cpus().length` | Number of workers to fork |
69
+ | `onWorkerReady` | `(id: number) => void` | — | Called when a worker comes online |
70
+ | `onWorkerCrash` | `(id: number, code: number) => void` | — | Called before a crashed worker is replaced |
71
+
72
+ ---
73
+
74
+ ## Self-healing
75
+
76
+ If a worker process exits (crash, unhandled rejection, OOM), the master:
77
+
78
+ 1. Fires `onWorkerCrash(id, exitCode)`.
79
+ 2. Immediately calls `cluster.fork()` to spawn a replacement.
80
+ 3. The new worker runs the full Pre-Flight lifecycle before serving traffic.
81
+
82
+ The replacement is instantaneous from the master's perspective — there is no delay or exponential backoff. If workers crash in a tight loop (bad config, missing env var), the master keeps forking. Use `efc doctor` to diagnose configuration problems before deploying.
83
+
84
+ ---
85
+
86
+ ## Disabling clustering
87
+
88
+ Set `cluster: false` to run a single-process server. EFC forces clustering off in development (`efc start dev`) for faster restarts and cleaner stack traces.
89
+
90
+ ```ts
91
+ ignite({ cluster: false }); // single process — same as efc start dev
92
+ ```
93
+
94
+ ---
95
+
96
+ ## Graceful shutdown
97
+
98
+ `gracefulShutdown(server)` registers `SIGTERM` and `SIGINT` handlers:
99
+
100
+ ```ts
101
+ import { ignite, gracefulShutdown } from 'express-file-cluster';
102
+
103
+ ignite(config)
104
+ .then(gracefulShutdown);
105
+ ```
106
+
107
+ - On a **worker**: calls `server.closeIdleConnections()`, then `server.close()` (in-flight requests finish), drains running task jobs, and exits `0`. Forces exit after a 10-second timeout. Repeated signals are ignored rather than killing it mid-drain.
108
+ - On the **primary** (when `ignite()` returns `undefined`): stops respawning, **relays `SIGTERM` to every worker** — orchestrators such as `docker stop` and Kubernetes signal only PID 1 — and exits once all workers have drained, or force-exits after the timeout + 2 s.
109
+
110
+ Pass a custom timeout (milliseconds) as the second argument:
111
+
112
+ ```ts
113
+ gracefulShutdown(server, 30_000); // 30-second drain window
114
+ ```
@@ -0,0 +1,235 @@
1
+ # File-Based Routing
2
+
3
+ EFC turns your directory tree into your route tree. The scanner walks `src/api/` recursively (a fixed convention, not a configurable option), converts each file path to a URL, and mounts the handlers on Express — no explicit registration required.
4
+
5
+ ---
6
+
7
+ ## Naming rules
8
+
9
+ | Rule | Example file | Resolved URL |
10
+ |---|---|---|
11
+ | Plain file | `api/health.ts` | `/health` |
12
+ | `index.ts` in a directory | `api/users/index.ts` | `/users` |
13
+ | Nested directory | `api/posts/recent.ts` | `/posts/recent` |
14
+ | Dynamic segment | `api/users/[id].ts` | `/users/:id` |
15
+ | Dynamic directory | `api/posts/[slug]/comments.ts` | `/posts/:slug/comments` |
16
+ | Optional segment | `api/users/[[id]].ts` | `/users` **and** `/users/:id` |
17
+ | Catch-all | `api/docs/[...slug].ts` | `/docs/a`, `/docs/a/b/c`, … (not `/docs`) |
18
+ | Optional catch-all | `api/docs/[[...slug]].ts` | `/docs`, `/docs/a`, `/docs/a/b/c`, … |
19
+
20
+ **`[bracket]` → `:param`** — any path segment wrapped in square brackets becomes an Express route parameter accessible as `req.params.<name>`.
21
+
22
+ **`[[double]]` → optional** — `req.params.id` is `undefined` when the segment is absent.
23
+
24
+ **`[...spread]` → catch-all** — matches one or more remaining segments; `req.params.slug` is a **string array** (`/docs/a/b` → `['a', 'b']`, each segment URI-decoded). With double brackets it also matches zero segments (`[]`). A catch-all must be the last segment of the path — `[...slug]/edit.ts` is rejected at startup.
25
+
26
+ **`index.ts` → parent path** — a file named `index.ts` (or `.js`) inside a directory maps to the directory's URL without a trailing `/index`.
27
+
28
+ ---
29
+
30
+ ## Supported file extensions
31
+
32
+ The scanner recognises `.ts`, `.js`, `.mts`, `.mjs`, `.cts`, and `.cjs`.
33
+
34
+ ---
35
+
36
+ ## Route priority
37
+
38
+ Routes register most-specific first, compared segment by segment: at each position a **static** segment beats `[id]`, which beats `[[id]]`, then `[...slug]`, then `[[...slug]]`. Given:
39
+
40
+ ```
41
+ api/docs/intro.ts → /docs/intro (static)
42
+ api/docs/[id].ts → /docs/:id (dynamic)
43
+ api/docs/[[...slug]].ts → /docs/:slug*? (optional catch-all)
44
+ ```
45
+
46
+ `/docs/intro` hits the static route, `/docs/42` the dynamic one, and `/docs` or `/docs/a/b` fall through to the catch-all. Ties are ordered alphabetically. `efc routes` prints the resolved order.
47
+
48
+ ---
49
+
50
+ ## Exporting handlers
51
+
52
+ Each route file exports one or more uppercase HTTP method names as async functions:
53
+
54
+ ```ts
55
+ // src/api/users/index.ts
56
+ import type { Request, Response } from 'express';
57
+
58
+ export const GET = async (req: Request, res: Response) => {
59
+ res.json({ users: [] });
60
+ };
61
+
62
+ export const POST = async (req: Request, res: Response) => {
63
+ res.status(201).json({ id: 'new-id' });
64
+ };
65
+ ```
66
+
67
+ Supported method names: `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS`.
68
+
69
+ Any method not exported on a route that does export at least one method returns **405 Method Not Allowed** with an `Allow` header listing the implemented methods:
70
+
71
+ ```
72
+ HTTP/1.1 405 Method Not Allowed
73
+ Allow: GET, POST
74
+ Content-Type: application/json
75
+
76
+ { "error": "Method Not Allowed" }
77
+ ```
78
+
79
+ ---
80
+
81
+ ## Dynamic parameters
82
+
83
+ ```ts
84
+ // src/api/users/[id].ts
85
+ import type { Request, Response } from 'express';
86
+ import { HttpError } from 'express-file-cluster';
87
+ import { User } from '../../model/User.js';
88
+
89
+ export const GET = async (req: Request, res: Response) => {
90
+ const user = await User.findById(req.params.id);
91
+ if (!user) throw new HttpError(404, 'User not found');
92
+ res.json(user);
93
+ };
94
+
95
+ export const DELETE = async (req: Request, res: Response) => {
96
+ await User.delete(req.params.id);
97
+ res.status(204).send();
98
+ };
99
+ ```
100
+
101
+ ---
102
+
103
+ ## Route metadata (`meta` export)
104
+
105
+ A route file can export a `meta` object to provide documentation shown in the [dashboard](../api-reference/ignite.md#dashboard). `meta` is keyed by HTTP method — each method implemented in the file gets its own description and request/response example, rendered as a separate documentation block. All fields are optional.
106
+
107
+ ```ts
108
+ // src/api/users/[id].ts
109
+ export const meta = {
110
+ GET: {
111
+ description: 'Fetch a user by ID.',
112
+ request: {
113
+ headers: { Authorization: 'Bearer <token>' },
114
+ params: { id: 'usr_01HXZ' },
115
+ query: { include: 'profile' },
116
+ },
117
+ response: {
118
+ status: 200,
119
+ body: {
120
+ id: 'usr_01HXZ',
121
+ name: 'Ada Lovelace',
122
+ createdAt: '2026-01-01T00:00:00.000Z',
123
+ },
124
+ },
125
+ },
126
+ DELETE: {
127
+ description: 'Delete a user by ID.',
128
+ request: {
129
+ headers: { Authorization: 'Bearer <token>' },
130
+ params: { id: 'usr_01HXZ' },
131
+ },
132
+ response: { status: 204 },
133
+ },
134
+ };
135
+ ```
136
+
137
+ **`RouteMeta` / `RouteMethodMeta` interfaces:**
138
+
139
+ ```ts
140
+ type RouteMeta = Partial<Record<string, RouteMethodMeta>>;
141
+
142
+ interface RouteMethodMeta {
143
+ description?: string;
144
+ request?: {
145
+ headers?: Record<string, string>;
146
+ params?: Record<string, string>;
147
+ query?: Record<string, string>;
148
+ body?: unknown;
149
+ };
150
+ response?: {
151
+ status?: number;
152
+ body?: unknown;
153
+ };
154
+ }
155
+ ```
156
+
157
+ Only add an entry for methods you want documented — a method with no key in `meta` simply shows an undocumented block in the dashboard.
158
+
159
+ The dashboard renders `response.body` values as their **type names** (`String`, `Number`, `Boolean`, `Date`) rather than the literal values you provide. ISO 8601 date strings (e.g., `'2026-01-01T00:00:00.000Z'`) are automatically detected and shown as `Date`.
160
+
161
+ ---
162
+
163
+ ## Request validation (`schema` export)
164
+
165
+ Export `schema` to validate `params`, `query` and `body` before your handler runs. Any [Standard Schema](https://standardschema.dev) validator works — zod (3.24+), valibot, arktype — or anything with a zod-style `safeParse`. EFC never imports a validation library itself.
166
+
167
+ ```ts
168
+ // src/api/users/index.ts
169
+ import { z } from 'zod';
170
+
171
+ export const schema = {
172
+ POST: { body: z.object({ email: z.string().email(), age: z.coerce.number().min(18) }) },
173
+ GET: { query: z.object({ page: z.coerce.number().default(1) }) },
174
+ };
175
+
176
+ export const POST = async (req, res) => {
177
+ // req.body is the parsed output: coerced types, defaults applied
178
+ res.status(201).json(req.body);
179
+ };
180
+ ```
181
+
182
+ Key it by method as above, or pass `{ params, query, body }` directly to apply it to every method in the file. Validation runs **after** the file's `middlewares` (so `requireAuth` still answers 401 first). Failures return:
183
+
184
+ ```json
185
+ {
186
+ "error": "Validation failed",
187
+ "statusCode": 400,
188
+ "issues": [{ "location": "body", "path": "email", "message": "Invalid email address" }]
189
+ }
190
+ ```
191
+
192
+ See the [Validation guide](../guides/validation.md).
193
+
194
+ ---
195
+
196
+ ## Route-level middleware
197
+
198
+ Export a `middlewares` array from a route file. The middlewares apply to every handler in that file and run after global middleware.
199
+
200
+ ```ts
201
+ // src/api/admin/settings.ts
202
+ import { requireAuth } from 'express-file-cluster/auth';
203
+
204
+ export const middlewares = [requireAuth];
205
+
206
+ export const GET = async (req, res) => {
207
+ // Only reached if requireAuth passes
208
+ res.json({ settings: {} });
209
+ };
210
+ ```
211
+
212
+ ---
213
+
214
+ ## How the scanner works internally
215
+
216
+ `scanDir(dir)` in `packages/core/src/router/scan.ts`, called with the resolved `src/api/` path:
217
+
218
+ 1. Walks the directory recursively with `fs.readdirSync`.
219
+ 2. For each file, converts the path relative to `src/api/` to a URL:
220
+ - Strips the extension.
221
+ - Replaces `/index` with `''` (parent path).
222
+ - Replaces `[param]` with `:param`, `[[param]]` with `:param?`, `[...param]` with `:param*` and `[[...param]]` with `:param*?`.
223
+ - Prepends `/` if missing.
224
+ 3. Sorts results by per-segment specificity (see Route priority).
225
+ 4. Returns a `RouteEntry[]` array with `urlPath`, `filePath`, `params`, and — for catch-all routes, which Express 4's string paths can't express — a compiled `pattern` RegExp.
226
+
227
+ `mountRoutes(app, routes)` in `packages/core/src/router/mount.ts`:
228
+
229
+ 1. Iterates `RouteEntry[]`.
230
+ 2. Dynamically `import()`s each route module.
231
+ 3. Reads the optional `middlewares`, `schema` and `meta` exports.
232
+ 4. For each known HTTP method that is exported as a function, registers it with `app.<method>(urlPath, ...middlewares, validate?, asyncWrap(handler))`. Catch-all routes mount on their `pattern` behind a small mapper that turns positional captures back into named params.
233
+ 5. Registers an `app.all()` catch for unimplemented methods that returns 405.
234
+
235
+ `asyncWrap` wraps every handler in `Promise.resolve(...).catch(next)` so that thrown errors and rejected promises are forwarded to the Express error handler automatically.
@@ -0,0 +1,79 @@
1
+ # Core Concepts
2
+
3
+ EFC is built around four interlocking ideas. Understanding each one independently makes the whole system easy to reason about.
4
+
5
+ ---
6
+
7
+ ## 1. File-Based Routing
8
+
9
+ The directory tree under `src/api/` **is** the route tree. No `router.get(...)` registration, no explicit path strings — the file path is the URL path.
10
+
11
+ ```
12
+ src/api/users/[id].ts → GET /users/:id
13
+ ```
14
+
15
+ Each route file exports uppercase HTTP method names as async functions. Methods not exported return **405 Method Not Allowed** automatically.
16
+
17
+ → [File-Based Routing](./file-based-routing.md)
18
+
19
+ ---
20
+
21
+ ## 2. Multi-Core Clustering
22
+
23
+ With `cluster: true`, EFC forks `os.cpus().length` identical worker processes. Each worker runs the full **Pre-Flight lifecycle** independently (connect DB → configure auth → scan routes → register tasks → listen).
24
+
25
+ The OS round-robins incoming connections across workers. If a worker crashes, the master immediately forks a replacement.
26
+
27
+ → [Clustering](./clustering.md)
28
+
29
+ ---
30
+
31
+ ## 3. Background Tasks
32
+
33
+ The `src/tasks/` directory holds background jobs — work that should happen *off* the request/response cycle. Define a task with `defineTask()`, trigger it from any route handler with `enqueue()`, and the queue backend — **BullMQ** (Redis) or **pg-boss** (PostgreSQL) — handles retries, backoff, cron schedules, dead letters and concurrency.
34
+
35
+ CPU-bound tasks can opt into `worker_threads` execution with `{ thread: true }`, keeping the event loop free.
36
+
37
+ → [Background Tasks](./background-tasks.md)
38
+
39
+ ---
40
+
41
+ ## 4. Middleware System
42
+
43
+ EFC supports three tiers of middleware with clear, non-overlapping scopes:
44
+
45
+ - **Global** — configured in `ignite()`, applied to every request.
46
+ - **Route-level** — `export const middlewares = [...]` in a route file, applied only to handlers in that file.
47
+ - **Handler-level** — `compose(mw1, mw2, handler)`, applied to a single handler.
48
+
49
+ → [Middleware](./middleware.md)
50
+
51
+ ---
52
+
53
+ ## How the pieces fit together
54
+
55
+ ```
56
+ HTTP Request
57
+ │
58
+ ▼
59
+ Global Middleware (ignite({ globalMiddlewares }))
60
+ │
61
+ ▼
62
+ Route-Level Middleware (export const middlewares = [...])
63
+ │
64
+ ▼
65
+ Handler-Level Middleware (compose(...))
66
+ │
67
+ ▼
68
+ Route Handler (export const GET = ...)
69
+ │
70
+ ├── sync response → res.json(...)
71
+ │
72
+ └── heavy work → enqueue('TaskName', payload)
73
+ │
74
+ ▼
75
+ Queue Backend (BullMQ / pg-boss)
76
+ │
77
+ ├── I/O-bound → event loop
78
+ └── CPU-bound → worker_threads
79
+ ```