@rune-kit/rune 2.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +357 -0
- package/agents/.gitkeep +0 -0
- package/agents/architect.md +29 -0
- package/agents/asset-creator.md +11 -0
- package/agents/audit.md +11 -0
- package/agents/autopsy.md +11 -0
- package/agents/brainstorm.md +11 -0
- package/agents/browser-pilot.md +11 -0
- package/agents/coder.md +29 -0
- package/agents/completion-gate.md +11 -0
- package/agents/constraint-check.md +11 -0
- package/agents/context-engine.md +11 -0
- package/agents/cook.md +11 -0
- package/agents/db.md +11 -0
- package/agents/debug.md +11 -0
- package/agents/dependency-doctor.md +11 -0
- package/agents/deploy.md +11 -0
- package/agents/design.md +11 -0
- package/agents/docs-seeker.md +11 -0
- package/agents/fix.md +11 -0
- package/agents/hallucination-guard.md +11 -0
- package/agents/incident.md +11 -0
- package/agents/integrity-check.md +11 -0
- package/agents/journal.md +11 -0
- package/agents/launch.md +11 -0
- package/agents/logic-guardian.md +11 -0
- package/agents/marketing.md +11 -0
- package/agents/onboard.md +11 -0
- package/agents/perf.md +11 -0
- package/agents/plan.md +11 -0
- package/agents/preflight.md +11 -0
- package/agents/problem-solver.md +11 -0
- package/agents/rescue.md +11 -0
- package/agents/research.md +11 -0
- package/agents/researcher.md +29 -0
- package/agents/review-intake.md +11 -0
- package/agents/review.md +11 -0
- package/agents/reviewer.md +28 -0
- package/agents/safeguard.md +11 -0
- package/agents/sast.md +11 -0
- package/agents/scanner.md +28 -0
- package/agents/scope-guard.md +11 -0
- package/agents/scout.md +11 -0
- package/agents/sentinel.md +11 -0
- package/agents/sequential-thinking.md +11 -0
- package/agents/session-bridge.md +11 -0
- package/agents/skill-forge.md +11 -0
- package/agents/skill-router.md +11 -0
- package/agents/surgeon.md +11 -0
- package/agents/team.md +11 -0
- package/agents/test.md +11 -0
- package/agents/trend-scout.md +11 -0
- package/agents/verification.md +11 -0
- package/agents/video-creator.md +11 -0
- package/agents/watchdog.md +11 -0
- package/agents/worktree.md +11 -0
- package/commands/.gitkeep +0 -0
- package/commands/rune.md +168 -0
- package/compiler/__tests__/openclaw-adapter.test.js +140 -0
- package/compiler/__tests__/parser.test.js +55 -0
- package/compiler/adapters/antigravity.js +59 -0
- package/compiler/adapters/claude.js +37 -0
- package/compiler/adapters/cursor.js +67 -0
- package/compiler/adapters/generic.js +60 -0
- package/compiler/adapters/index.js +45 -0
- package/compiler/adapters/openclaw.js +150 -0
- package/compiler/adapters/windsurf.js +60 -0
- package/compiler/bin/rune.js +288 -0
- package/compiler/doctor.js +153 -0
- package/compiler/emitter.js +240 -0
- package/compiler/parser.js +208 -0
- package/compiler/transformer.js +69 -0
- package/compiler/transforms/branding.js +27 -0
- package/compiler/transforms/cross-references.js +29 -0
- package/compiler/transforms/frontmatter.js +38 -0
- package/compiler/transforms/hooks.js +68 -0
- package/compiler/transforms/subagents.js +36 -0
- package/compiler/transforms/tool-names.js +60 -0
- package/contexts/dev.md +34 -0
- package/contexts/research.md +43 -0
- package/contexts/review.md +55 -0
- package/extensions/ai-ml/PACK.md +517 -0
- package/extensions/analytics/PACK.md +557 -0
- package/extensions/backend/PACK.md +678 -0
- package/extensions/chrome-ext/PACK.md +995 -0
- package/extensions/content/PACK.md +381 -0
- package/extensions/devops/PACK.md +520 -0
- package/extensions/ecommerce/PACK.md +280 -0
- package/extensions/gamedev/PACK.md +393 -0
- package/extensions/mobile/PACK.md +273 -0
- package/extensions/saas/PACK.md +805 -0
- package/extensions/security/PACK.md +536 -0
- package/extensions/trading/PACK.md +597 -0
- package/extensions/ui/PACK.md +947 -0
- package/package.json +47 -0
- package/skills/.gitkeep +0 -0
- package/skills/adversary/SKILL.md +271 -0
- package/skills/asset-creator/SKILL.md +157 -0
- package/skills/audit/SKILL.md +466 -0
- package/skills/autopsy/SKILL.md +200 -0
- package/skills/ba/SKILL.md +279 -0
- package/skills/brainstorm/SKILL.md +266 -0
- package/skills/browser-pilot/SKILL.md +168 -0
- package/skills/completion-gate/SKILL.md +151 -0
- package/skills/constraint-check/SKILL.md +165 -0
- package/skills/context-engine/SKILL.md +176 -0
- package/skills/cook/SKILL.md +636 -0
- package/skills/db/SKILL.md +256 -0
- package/skills/debug/SKILL.md +240 -0
- package/skills/dependency-doctor/SKILL.md +235 -0
- package/skills/deploy/SKILL.md +174 -0
- package/skills/design/DESIGN-REFERENCE.md +365 -0
- package/skills/design/SKILL.md +462 -0
- package/skills/doc-processor/SKILL.md +254 -0
- package/skills/docs/SKILL.md +336 -0
- package/skills/docs-seeker/SKILL.md +166 -0
- package/skills/fix/SKILL.md +192 -0
- package/skills/git/SKILL.md +285 -0
- package/skills/hallucination-guard/SKILL.md +204 -0
- package/skills/incident/SKILL.md +241 -0
- package/skills/integrity-check/SKILL.md +169 -0
- package/skills/journal/SKILL.md +190 -0
- package/skills/launch/SKILL.md +330 -0
- package/skills/logic-guardian/SKILL.md +240 -0
- package/skills/marketing/SKILL.md +229 -0
- package/skills/mcp-builder/SKILL.md +311 -0
- package/skills/onboard/SKILL.md +298 -0
- package/skills/perf/SKILL.md +297 -0
- package/skills/plan/SKILL.md +520 -0
- package/skills/preflight/SKILL.md +231 -0
- package/skills/problem-solver/SKILL.md +284 -0
- package/skills/rescue/SKILL.md +434 -0
- package/skills/research/SKILL.md +122 -0
- package/skills/review/SKILL.md +354 -0
- package/skills/review-intake/SKILL.md +222 -0
- package/skills/safeguard/SKILL.md +188 -0
- package/skills/sast/SKILL.md +190 -0
- package/skills/scaffold/SKILL.md +276 -0
- package/skills/scope-guard/SKILL.md +150 -0
- package/skills/scout/SKILL.md +232 -0
- package/skills/sentinel/SKILL.md +320 -0
- package/skills/sentinel-env/SKILL.md +226 -0
- package/skills/sequential-thinking/SKILL.md +234 -0
- package/skills/session-bridge/SKILL.md +287 -0
- package/skills/skill-forge/SKILL.md +317 -0
- package/skills/skill-router/SKILL.md +267 -0
- package/skills/surgeon/SKILL.md +203 -0
- package/skills/team/SKILL.md +397 -0
- package/skills/test/SKILL.md +271 -0
- package/skills/trend-scout/SKILL.md +145 -0
- package/skills/verification/SKILL.md +201 -0
- package/skills/video-creator/SKILL.md +201 -0
- package/skills/watchdog/SKILL.md +166 -0
- package/skills/worktree/SKILL.md +140 -0
|
@@ -0,0 +1,678 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: "@rune/backend"
|
|
3
|
+
description: Backend patterns — API design, authentication, database patterns, middleware architecture, caching strategies, and background job processing.
|
|
4
|
+
metadata:
|
|
5
|
+
author: runedev
|
|
6
|
+
version: "0.2.0"
|
|
7
|
+
layer: L4
|
|
8
|
+
price: "free"
|
|
9
|
+
target: Backend developers
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# @rune/backend
|
|
13
|
+
|
|
14
|
+
## Purpose
|
|
15
|
+
|
|
16
|
+
Backend codebases accumulate structural debt across six areas: inconsistent API contracts (mixed naming, missing pagination, vague errors), insecure auth flows (token mismanagement, missing refresh rotation, weak RBAC), database anti-patterns (N+1 queries, missing indexes, unsafe migrations), ad-hoc middleware (duplicated validation, no request tracing, inconsistent error format), missing or naive caching (no invalidation strategy, cache stampede risk, unbounded memory growth), and synchronous processing of inherently async work (blocking request threads on email, PDF, image tasks). This pack addresses each systematically — detect the anti-pattern, emit the fix, verify the result. Skills are independent but compound: clean APIs need solid auth, solid auth needs safe queries, safe queries need proper middleware, and high-traffic APIs need caching and background jobs to stay responsive.
|
|
17
|
+
|
|
18
|
+
## Triggers
|
|
19
|
+
|
|
20
|
+
- Auto-trigger: when `routes/`, `controllers/`, `middleware/`, `*.resolver.ts`, `*.service.ts`, `queues/`, `workers/`, or server framework config detected
|
|
21
|
+
- `/rune api-patterns` — audit and fix API design
|
|
22
|
+
- `/rune auth-patterns` — audit and fix authentication flows
|
|
23
|
+
- `/rune database-patterns` — audit and fix database queries and schema
|
|
24
|
+
- `/rune middleware-patterns` — audit and fix middleware stack
|
|
25
|
+
- `/rune caching-patterns` — audit and implement caching strategy
|
|
26
|
+
- `/rune background-jobs` — identify async operations and implement job queues
|
|
27
|
+
- Called by `cook` (L1) when backend task is detected
|
|
28
|
+
- Called by `review` (L2) when API/backend code is under review
|
|
29
|
+
|
|
30
|
+
## Skills Included
|
|
31
|
+
|
|
32
|
+
### api-patterns
|
|
33
|
+
|
|
34
|
+
RESTful and GraphQL API design patterns — resource naming, pagination, filtering, error responses, versioning, rate limiting, OpenAPI generation.
|
|
35
|
+
|
|
36
|
+
#### Workflow
|
|
37
|
+
|
|
38
|
+
**Step 1 — Detect API surface**
|
|
39
|
+
Use Grep to find route definitions (`app.get`, `app.post`, `router.`, `@Get()`, `@Post()`, `@Query`, `@Mutation`). Read each route file to inventory: endpoint paths, HTTP methods, response shapes, error handling approach.
|
|
40
|
+
|
|
41
|
+
**Step 2 — Audit naming and structure**
|
|
42
|
+
Check each endpoint against REST conventions: plural nouns for collections (`/users` not `/getUsers`), nested resources for relationships (`/users/:id/posts`), query params for filtering (`?status=active`), consistent error envelope. Flag violations with specific fix for each.
|
|
43
|
+
|
|
44
|
+
**Step 3 — Add missing pagination and filtering**
|
|
45
|
+
For list endpoints returning unbounded arrays, emit cursor-based or offset pagination. For endpoints with no filtering, add query param parsing with Zod/Joi validation. Emit the middleware or decorator that enforces the pattern.
|
|
46
|
+
|
|
47
|
+
**Step 4 — API versioning strategy**
|
|
48
|
+
Choose versioning approach based on project context: URL path (`/v2/users`) for public APIs with long deprecation windows; `Accept-Version: 2` header for internal APIs needing cleaner URLs; query param (`?version=2`) for simple cases. Emit version routing middleware and a deprecation warning header (`Deprecation: true, Sunset: <date>`) on v1 routes. Document migration path in the route file as a comment.
|
|
49
|
+
|
|
50
|
+
**Step 5 — OpenAPI/Swagger and GraphQL patterns**
|
|
51
|
+
For REST: emit OpenAPI 3.1 schema from route definitions using tsoa decorators (TypeScript), Fastify's built-in JSON Schema (`schema: { body, querystring, response }`), or NestJS `@ApiProperty`. For GraphQL: if schema-first, validate resolvers match schema types; if code-first (NestJS), check `@ObjectType` / `@Field` decorators. Add DataLoader to any resolver with a per-request DB call to prevent N+1 at the GraphQL layer. Emit subscription pattern (WebSocket transport) for real-time fields.
|
|
52
|
+
|
|
53
|
+
#### Example
|
|
54
|
+
|
|
55
|
+
```typescript
|
|
56
|
+
// BEFORE: inconsistent naming, no pagination, bare error
|
|
57
|
+
app.get('/getUsers', async (req, res) => {
|
|
58
|
+
const users = await db.query('SELECT * FROM users');
|
|
59
|
+
res.json(users);
|
|
60
|
+
});
|
|
61
|
+
|
|
62
|
+
// AFTER: REST naming, cursor pagination, error envelope, Zod validation
|
|
63
|
+
const paginationSchema = z.object({
|
|
64
|
+
query: z.object({
|
|
65
|
+
cursor: z.string().optional(),
|
|
66
|
+
limit: z.coerce.number().int().min(1).max(100).default(20),
|
|
67
|
+
status: z.enum(['active', 'inactive']).optional(),
|
|
68
|
+
}),
|
|
69
|
+
});
|
|
70
|
+
|
|
71
|
+
app.get('/users', validate(paginationSchema), async (req, res) => {
|
|
72
|
+
const { cursor, limit, status } = req.query;
|
|
73
|
+
const users = await userRepo.findMany({ cursor, limit: limit + 1, status });
|
|
74
|
+
const hasNext = users.length > limit;
|
|
75
|
+
res.json({
|
|
76
|
+
data: users.slice(0, limit),
|
|
77
|
+
pagination: { next_cursor: hasNext ? users[limit - 1].id : null, has_more: hasNext },
|
|
78
|
+
});
|
|
79
|
+
});
|
|
80
|
+
|
|
81
|
+
// Rate limiting: sliding window with Redis (atomic, no race condition)
|
|
82
|
+
const rateLimitMiddleware = async (req, res, next) => {
|
|
83
|
+
const key = `rl:${req.ip}:${Math.floor(Date.now() / 60_000)}`; // 1-minute window
|
|
84
|
+
const multi = redis.multi();
|
|
85
|
+
multi.incr(key);
|
|
86
|
+
multi.expire(key, 60);
|
|
87
|
+
const [count] = await multi.exec();
|
|
88
|
+
if (count > 100) return res.status(429).json({ error: { code: 'RATE_LIMITED', message: 'Too many requests' } });
|
|
89
|
+
res.setHeader('X-RateLimit-Remaining', 100 - count);
|
|
90
|
+
next();
|
|
91
|
+
};
|
|
92
|
+
|
|
93
|
+
// Fastify: built-in schema validation + OpenAPI generation
|
|
94
|
+
fastify.get('/users/:id', {
|
|
95
|
+
schema: {
|
|
96
|
+
params: { type: 'object', properties: { id: { type: 'string', format: 'uuid' } }, required: ['id'] },
|
|
97
|
+
response: { 200: UserSchema, 404: ErrorSchema },
|
|
98
|
+
},
|
|
99
|
+
}, async (req, reply) => { /* handler */ });
|
|
100
|
+
|
|
101
|
+
// GraphQL: DataLoader prevents N+1 in resolvers
|
|
102
|
+
const userLoader = new DataLoader(async (userIds: string[]) => {
|
|
103
|
+
const users = await prisma.user.findMany({ where: { id: { in: userIds } } });
|
|
104
|
+
return userIds.map(id => users.find(u => u.id === id) ?? new Error(`User ${id} not found`));
|
|
105
|
+
});
|
|
106
|
+
// In resolver: return userLoader.load(post.authorId) — batches all loads per request
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
### auth-patterns
|
|
112
|
+
|
|
113
|
+
Authentication and authorization patterns — JWT, OAuth 2.0 / OIDC, passkeys/WebAuthn, session management, RBAC, API key management, MFA flows.
|
|
114
|
+
|
|
115
|
+
#### Workflow
|
|
116
|
+
|
|
117
|
+
**Step 1 — Detect auth implementation**
|
|
118
|
+
Use Grep to find auth-related code: `jwt.sign`, `jwt.verify`, `bcrypt`, `passport`, `next-auth`, `lucia`, `cookie`, `session`, `Bearer`, `x-api-key`, `WebAuthn`, `passkey`. Read auth middleware and login/register handlers to understand the current approach.
|
|
119
|
+
|
|
120
|
+
**Step 2 — Audit security posture**
|
|
121
|
+
Check for: tokens stored in localStorage (XSS risk → use httpOnly cookies), missing refresh token rotation, JWT without expiry, password hashing without salt rounds check, missing CSRF protection on cookie-based auth, hardcoded secrets. Flag each with severity and specific fix.
|
|
122
|
+
|
|
123
|
+
**Step 3 — Emit secure auth flow**
|
|
124
|
+
Based on detected framework (Express, Fastify, Next.js, etc.), emit the corrected auth flow: access token (short-lived, 15min) + refresh token (httpOnly cookie, 7d, rotation on use), proper password hashing (bcrypt rounds ≥ 12), RBAC middleware with role hierarchy.
|
|
125
|
+
|
|
126
|
+
**Step 4 — OAuth 2.0 / OIDC integration**
|
|
127
|
+
Emit OAuth 2.0 authorization code flow with PKCE (required for public clients). Support Google, GitHub, or custom OIDC provider. Key points: validate `state` parameter to prevent CSRF, validate `id_token` signature and `aud`/`iss` claims, exchange code server-side (never client-side), store provider `sub` as stable user identifier. Use `openid-client` (Node.js) or `authlib` (Python) — never hand-roll token exchange.
|
|
128
|
+
|
|
129
|
+
**Step 5 — API key management and passkeys**
|
|
130
|
+
For API keys: generate with `crypto.randomBytes(32).toString('base64url')`, store hashed (`sha256` is sufficient — no need for bcrypt, keys are long), never store plaintext after initial display. Add scopes (read-only vs read-write), per-key rate limits, and rotation endpoint. For passkeys/WebAuthn: emit registration and authentication ceremonies using `@simplewebauthn/server`. WebAuthn is the correct long-term replacement for passwords — emit as opt-in upgrade path. Stateless vs stateful tradeoff: JWT = stateless, easy to scale horizontally, hard to revoke; sessions = stateful, easy to revoke, requires sticky sessions or shared store (Redis). Recommend JWT + token blacklist on logout for most cases; sessions for admin panels where immediate revocation matters.
|
|
131
|
+
|
|
132
|
+
#### Example
|
|
133
|
+
|
|
134
|
+
```typescript
|
|
135
|
+
// BEFORE: JWT in localStorage, no refresh, no expiry
|
|
136
|
+
const token = jwt.sign({ userId: user.id }, SECRET);
|
|
137
|
+
res.json({ token });
|
|
138
|
+
|
|
139
|
+
// AFTER: short-lived access + httpOnly refresh cookie with rotation
|
|
140
|
+
const accessToken = jwt.sign(
|
|
141
|
+
{ sub: user.id, role: user.role },
|
|
142
|
+
ACCESS_SECRET,
|
|
143
|
+
{ expiresIn: '15m' }
|
|
144
|
+
);
|
|
145
|
+
const refreshToken = jwt.sign(
|
|
146
|
+
{ sub: user.id, jti: crypto.randomUUID() },
|
|
147
|
+
REFRESH_SECRET,
|
|
148
|
+
{ expiresIn: '7d' }
|
|
149
|
+
);
|
|
150
|
+
await tokenStore.save(refreshToken, user.id); // rotation tracking — invalidate old on reuse
|
|
151
|
+
|
|
152
|
+
res.cookie('refresh_token', refreshToken, {
|
|
153
|
+
httpOnly: true, secure: true, sameSite: 'strict',
|
|
154
|
+
maxAge: 7 * 24 * 60 * 60 * 1000,
|
|
155
|
+
});
|
|
156
|
+
res.json({ access_token: accessToken, expires_in: 900 });
|
|
157
|
+
|
|
158
|
+
// API key management
|
|
159
|
+
const generateApiKey = async (userId: string, scopes: string[]): Promise<{ key: string; keyId: string }> => {
|
|
160
|
+
const rawKey = `rk_${crypto.randomBytes(32).toString('base64url')}`;
|
|
161
|
+
const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex');
|
|
162
|
+
const keyId = crypto.randomUUID();
|
|
163
|
+
await db.apiKey.create({ data: { id: keyId, userId, keyHash, scopes, createdAt: new Date() } });
|
|
164
|
+
return { key: rawKey, keyId }; // rawKey shown ONCE — never stored plaintext
|
|
165
|
+
};
|
|
166
|
+
|
|
167
|
+
const authenticateApiKey = async (req, res, next) => {
|
|
168
|
+
const raw = req.headers['x-api-key'];
|
|
169
|
+
if (!raw) return next(); // fallback to JWT auth
|
|
170
|
+
const hash = crypto.createHash('sha256').update(raw).digest('hex');
|
|
171
|
+
const apiKey = await db.apiKey.findUnique({ where: { keyHash: hash } });
|
|
172
|
+
if (!apiKey || apiKey.revokedAt) return res.status(401).json({ error: { code: 'INVALID_API_KEY' } });
|
|
173
|
+
req.user = { id: apiKey.userId, scopes: apiKey.scopes };
|
|
174
|
+
next();
|
|
175
|
+
};
|
|
176
|
+
|
|
177
|
+
// OAuth 2.0 with PKCE (using openid-client)
|
|
178
|
+
import { generators, Issuer } from 'openid-client';
|
|
179
|
+
|
|
180
|
+
const googleIssuer = await Issuer.discover('https://accounts.google.com');
|
|
181
|
+
const client = new googleIssuer.Client({ client_id: GOOGLE_CLIENT_ID, redirect_uris: [CALLBACK_URL], response_types: ['code'] });
|
|
182
|
+
|
|
183
|
+
app.get('/auth/google', (req, res) => {
|
|
184
|
+
const codeVerifier = generators.codeVerifier();
|
|
185
|
+
const codeChallenge = generators.codeChallenge(codeVerifier);
|
|
186
|
+
const state = generators.state();
|
|
187
|
+
req.session.codeVerifier = codeVerifier;
|
|
188
|
+
req.session.state = state;
|
|
189
|
+
res.redirect(client.authorizationUrl({ scope: 'openid email profile', code_challenge: codeChallenge, code_challenge_method: 'S256', state }));
|
|
190
|
+
});
|
|
191
|
+
|
|
192
|
+
app.get('/auth/google/callback', async (req, res) => {
|
|
193
|
+
const params = client.callbackParams(req);
|
|
194
|
+
const tokens = await client.callback(CALLBACK_URL, params, { code_verifier: req.session.codeVerifier, state: req.session.state });
|
|
195
|
+
const claims = tokens.claims(); // validated: iss, aud, exp
|
|
196
|
+
const user = await userRepo.upsertByProvider('google', claims.sub, claims.email);
|
|
197
|
+
// issue internal JWT...
|
|
198
|
+
});
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
203
|
+
### database-patterns
|
|
204
|
+
|
|
205
|
+
Database design and query patterns — schema design, migrations, indexing strategies, N+1 prevention, soft deletes, read replicas, connection pooling, seeding.
|
|
206
|
+
|
|
207
|
+
#### Workflow
|
|
208
|
+
|
|
209
|
+
**Step 1 — Detect ORM and query patterns**
|
|
210
|
+
Use Grep to find ORM usage (`prisma.`, `knex(`, `sequelize.`, `typeorm`, `drizzle`, `mongoose.`, `db.query`) and raw SQL strings. Read schema files (`schema.prisma`, `migrations/`, `models/`) to understand the data model.
|
|
211
|
+
|
|
212
|
+
**Step 2 — Detect N+1 and missing indexes**
|
|
213
|
+
Scan for loops containing database calls (a query inside `for`, `map`, `forEach` → N+1). Check foreign key columns for missing indexes. Identify queries with `WHERE` clauses on unindexed columns. Flag each with the specific query and fix.
|
|
214
|
+
|
|
215
|
+
**Step 3 — Emit optimized queries**
|
|
216
|
+
For N+1: emit eager loading (`include`, `populate`, `JOIN`). For missing indexes: emit migration files. For unsafe raw SQL: emit parameterized version. For connection pooling: check pool config and recommend sizing based on max connections.
|
|
217
|
+
|
|
218
|
+
**Step 4 — Soft delete and query scoping**
|
|
219
|
+
Emit soft delete pattern: add `deleted_at TIMESTAMPTZ` column, update all `findMany`/`findUnique` calls to include `WHERE deleted_at IS NULL`. Cascade consideration: soft-delete parent should soft-delete children (emit trigger or application-level cascade). For Prisma: emit a custom extension that injects the filter automatically. Warn about index bloat from soft-deleted rows — add partial index `WHERE deleted_at IS NULL` to keep index lean.
|
|
220
|
+
|
|
221
|
+
**Step 5 — Read replicas, connection pooling, and seeding**
|
|
222
|
+
Read replicas: emit query routing — writes to primary, reads to replica. Handle replication lag: do not read from replica immediately after write in the same request (use primary for the read-after-write). For Prisma: emit `$extends` with read/write client split. Connection pooling deep dive: PgBouncer in transaction mode for serverless (each query gets a connection); Prisma's built-in pool for long-running servers. Pool sizing formula: `connections = (core_count * 2) + effective_spindle_count`. Seeding: emit factory functions using `@faker-js/faker` — deterministic seeds via `faker.seed(42)` for reproducible test data.
|
|
223
|
+
|
|
224
|
+
#### Example
|
|
225
|
+
|
|
226
|
+
```typescript
|
|
227
|
+
// BEFORE: N+1 — one query per post to get author
|
|
228
|
+
const posts = await prisma.post.findMany();
|
|
229
|
+
for (const post of posts) {
|
|
230
|
+
post.author = await prisma.user.findUnique({ where: { id: post.authorId } });
|
|
231
|
+
}
|
|
232
|
+
|
|
233
|
+
// AFTER: eager loading, single query with JOIN
|
|
234
|
+
const posts = await prisma.post.findMany({
|
|
235
|
+
include: { author: { select: { id: true, name: true, avatar: true } } },
|
|
236
|
+
});
|
|
237
|
+
|
|
238
|
+
// Migration: missing indexes + soft delete column
|
|
239
|
+
-- Migration: add_indexes_and_soft_delete_to_posts
|
|
240
|
+
ALTER TABLE posts ADD COLUMN deleted_at TIMESTAMPTZ;
|
|
241
|
+
CREATE INDEX idx_posts_author_id ON posts(author_id);
|
|
242
|
+
CREATE INDEX idx_posts_created_at ON posts(created_at DESC);
|
|
243
|
+
CREATE INDEX idx_posts_active ON posts(author_id, created_at DESC) WHERE deleted_at IS NULL;
|
|
244
|
+
|
|
245
|
+
// Prisma soft delete extension (auto-scopes all queries)
|
|
246
|
+
const softDelete = Prisma.defineExtension({
|
|
247
|
+
name: 'softDelete',
|
|
248
|
+
query: {
|
|
249
|
+
$allModels: {
|
|
250
|
+
async findMany({ model, operation, args, query }) {
|
|
251
|
+
args.where = { ...args.where, deletedAt: null };
|
|
252
|
+
return query(args);
|
|
253
|
+
},
|
|
254
|
+
async delete({ model, args, query }) {
|
|
255
|
+
return (query as any)({ ...args, data: { deletedAt: new Date() } } as any);
|
|
256
|
+
},
|
|
257
|
+
},
|
|
258
|
+
},
|
|
259
|
+
});
|
|
260
|
+
const prisma = new PrismaClient().$extends(softDelete);
|
|
261
|
+
|
|
262
|
+
// Read replica routing with Prisma
|
|
263
|
+
const primaryClient = new PrismaClient({ datasources: { db: { url: PRIMARY_URL } } });
|
|
264
|
+
const replicaClient = new PrismaClient({ datasources: { db: { url: REPLICA_URL } } });
|
|
265
|
+
const db = { write: primaryClient, read: replicaClient };
|
|
266
|
+
// Usage: db.write.user.create(...) vs db.read.user.findMany(...)
|
|
267
|
+
|
|
268
|
+
// Factory seeding
|
|
269
|
+
import { faker } from '@faker-js/faker';
|
|
270
|
+
faker.seed(42); // reproducible
|
|
271
|
+
|
|
272
|
+
const createUserFactory = (overrides = {}) => ({
|
|
273
|
+
id: faker.string.uuid(),
|
|
274
|
+
email: faker.internet.email(),
|
|
275
|
+
name: faker.person.fullName(),
|
|
276
|
+
createdAt: faker.date.past(),
|
|
277
|
+
...overrides,
|
|
278
|
+
});
|
|
279
|
+
|
|
280
|
+
await prisma.user.createMany({ data: Array.from({ length: 50 }, () => createUserFactory()) });
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
### middleware-patterns
|
|
286
|
+
|
|
287
|
+
Middleware architecture — request validation, error handling, logging, CORS, compression, graceful shutdown, health checks, request ID tracking.
|
|
288
|
+
|
|
289
|
+
#### Workflow
|
|
290
|
+
|
|
291
|
+
**Step 1 — Audit middleware stack**
|
|
292
|
+
Read the main server file (app.ts, server.ts, index.ts) to inventory all middleware in registration order. Check for: missing request ID generation, missing structured logging, inconsistent error responses, missing input validation, CORS misconfiguration (`*` in production).
|
|
293
|
+
|
|
294
|
+
**Step 2 — Detect error handling gaps**
|
|
295
|
+
Use Grep to find `catch` blocks, error middleware signatures (`err, req, res, next`), and unhandled promise rejections. Check if errors return consistent format (same envelope for 400, 401, 403, 404, 500). Flag any that leak stack traces or internal details in production.
|
|
296
|
+
|
|
297
|
+
**Step 3 — Emit middleware improvements**
|
|
298
|
+
For each gap, emit the middleware function: request ID (`X-Request-Id` header, UUID per request), structured JSON logger (request method, path, status, duration, request ID), global error handler with consistent envelope, Zod-based request validation middleware.
|
|
299
|
+
|
|
300
|
+
**Step 4 — Compression strategy**
|
|
301
|
+
Emit response compression middleware. Use `brotli` for static assets and pre-compressible responses (better ratio than gzip, supported by all modern clients). Use `gzip` as fallback for older clients. Conditional compression: skip for already-compressed content types (`image/*`, `video/*`, `application/zip`) — compressing these wastes CPU. In Express: use `compression` package with a `filter` function. In Fastify: `@fastify/compress` with `encodings: ['br', 'gzip']`. Minimum size threshold: do not compress responses < 1KB (overhead exceeds benefit).
|
|
302
|
+
|
|
303
|
+
**Step 5 — Graceful shutdown and health checks**
|
|
304
|
+
Graceful shutdown: on `SIGTERM`/`SIGINT`, stop accepting new connections, wait for in-flight requests to complete (timeout 30s), then close DB pools and exit. Emit the shutdown handler for Express (`server.close()`), Fastify (`fastify.close()`), and worker processes. Health check endpoints: `/health/live` (liveness — is the process alive? return 200 always unless process is broken), `/health/ready` (readiness — can it serve traffic? check DB connection, Redis connection, return 503 if dependencies are down). In Kubernetes: map liveness to `livenessProbe`, readiness to `readinessProbe`. Do NOT check external third-party APIs in readiness — only your own dependencies.
|
|
305
|
+
|
|
306
|
+
#### Example
|
|
307
|
+
|
|
308
|
+
```typescript
|
|
309
|
+
// Request ID middleware
|
|
310
|
+
const requestId = (req, res, next) => {
|
|
311
|
+
req.id = req.headers['x-request-id'] || crypto.randomUUID();
|
|
312
|
+
res.setHeader('X-Request-Id', req.id);
|
|
313
|
+
next();
|
|
314
|
+
};
|
|
315
|
+
|
|
316
|
+
// Structured error handler — consistent envelope, no stack leak
|
|
317
|
+
const errorHandler = (err, req, res, _next) => {
|
|
318
|
+
const status = err.status || 500;
|
|
319
|
+
const message = status < 500 ? err.message : 'Internal server error';
|
|
320
|
+
logger.error({ err, requestId: req.id, path: req.path });
|
|
321
|
+
res.status(status).json({
|
|
322
|
+
error: { code: err.code || 'INTERNAL_ERROR', message },
|
|
323
|
+
request_id: req.id,
|
|
324
|
+
});
|
|
325
|
+
};
|
|
326
|
+
|
|
327
|
+
// Zod validation middleware
|
|
328
|
+
const validate = (schema: z.ZodSchema) => (req, res, next) => {
|
|
329
|
+
const result = schema.safeParse({ body: req.body, query: req.query, params: req.params });
|
|
330
|
+
if (!result.success) {
|
|
331
|
+
return res.status(400).json({ error: { code: 'VALIDATION_ERROR', message: 'Invalid request', details: result.error.flatten() } });
|
|
332
|
+
}
|
|
333
|
+
Object.assign(req, result.data);
|
|
334
|
+
next();
|
|
335
|
+
};
|
|
336
|
+
|
|
337
|
+
// Compression with conditional skip (Express)
|
|
338
|
+
import compression from 'compression';
|
|
339
|
+
app.use(compression({
|
|
340
|
+
filter: (req, res) => {
|
|
341
|
+
const contentType = res.getHeader('Content-Type') as string || '';
|
|
342
|
+
if (/image|video|audio|zip|gz|br/.test(contentType)) return false;
|
|
343
|
+
return compression.filter(req, res);
|
|
344
|
+
},
|
|
345
|
+
threshold: 1024, // skip responses < 1KB
|
|
346
|
+
}));
|
|
347
|
+
|
|
348
|
+
// Graceful shutdown
|
|
349
|
+
const gracefulShutdown = async (signal: string) => {
|
|
350
|
+
console.log(`Received ${signal}, shutting down gracefully...`);
|
|
351
|
+
server.close(async () => {
|
|
352
|
+
try {
|
|
353
|
+
await prisma.$disconnect();
|
|
354
|
+
await redis.quit();
|
|
355
|
+
console.log('All connections closed. Exiting.');
|
|
356
|
+
process.exit(0);
|
|
357
|
+
} catch (err) {
|
|
358
|
+
console.error('Error during shutdown:', err);
|
|
359
|
+
process.exit(1);
|
|
360
|
+
}
|
|
361
|
+
});
|
|
362
|
+
// Force exit after 30s if still not done
|
|
363
|
+
setTimeout(() => { console.error('Forced shutdown after timeout'); process.exit(1); }, 30_000);
|
|
364
|
+
};
|
|
365
|
+
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
|
|
366
|
+
process.on('SIGINT', () => gracefulShutdown('SIGINT'));
|
|
367
|
+
|
|
368
|
+
// Health check endpoints
|
|
369
|
+
app.get('/health/live', (req, res) => res.json({ status: 'ok' }));
|
|
370
|
+
|
|
371
|
+
app.get('/health/ready', async (req, res) => {
|
|
372
|
+
const checks = await Promise.allSettled([
|
|
373
|
+
prisma.$queryRaw`SELECT 1`, // DB check
|
|
374
|
+
redis.ping(), // Redis check
|
|
375
|
+
]);
|
|
376
|
+
const results = { db: checks[0].status, redis: checks[1].status };
|
|
377
|
+
const allHealthy = checks.every(c => c.status === 'fulfilled');
|
|
378
|
+
res.status(allHealthy ? 200 : 503).json({ status: allHealthy ? 'ready' : 'degraded', checks: results });
|
|
379
|
+
});
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
---
|
|
383
|
+
|
|
384
|
+
### caching-patterns
|
|
385
|
+
|
|
386
|
+
Caching strategies for backend applications — in-memory LRU, Redis distributed cache, CDN/edge cache, browser cache headers, invalidation, and stampede prevention.
|
|
387
|
+
|
|
388
|
+
#### Workflow
|
|
389
|
+
|
|
390
|
+
**Step 1 — Identify cacheable endpoints**
|
|
391
|
+
Scan routes for: (a) read-heavy endpoints called frequently with the same inputs (user profile, product catalog, config lookups), (b) expensive computations (aggregations, report generation), (c) external API calls that are rate-limited or slow. Flag endpoints that mutate state as NOT cacheable at the response level (cache the data layer instead). Output a cacheable/non-cacheable classification per endpoint.
|
|
392
|
+
|
|
393
|
+
**Step 2 — Select cache layer**
|
|
394
|
+
Choose layer based on access pattern: in-memory (node-cache, LRU-cache) for single-process data with sub-millisecond access and low cardinality; Redis for distributed cache shared across multiple server instances or processes; CDN (Cloudflare, Fastly) for public, user-agnostic responses (marketing pages, public API responses); browser cache (`Cache-Control` headers) for static assets and safe GET responses. Hybrid: in-memory L1 + Redis L2 for hot-path data that justifies two-layer lookup.
|
|
395
|
+
|
|
396
|
+
**Step 3 — Implement cache pattern**
|
|
397
|
+
Cache-aside (most common): application checks cache first, on miss fetches from DB, writes to cache. Write-through: write to cache and DB together on every write (cache always warm, higher write latency). Write-behind (write-back): write to cache immediately, flush to DB asynchronously (lowest write latency, risk of data loss on crash). Read-through: cache sits in front of DB, handles miss transparently (simpler app code, less control). For most web APIs: cache-aside for reads + TTL-based expiry is the correct default.
|
|
398
|
+
|
|
399
|
+
**Step 4 — Add invalidation strategy**
|
|
400
|
+
TTL-based: set appropriate TTL per data type (user session: match auth token TTL; product catalog: 5–15min; config: 1hr). Event-driven: on mutation, publish event to Redis pub/sub, cache subscribers delete affected keys. Versioned keys: `cache:user:v3:{id}` — bump version in config to invalidate all users atomically. Tag-based: associate keys with tags (`tag:user:123`), delete all keys for a tag on mutation. Stale-while-revalidate: serve stale data immediately, refresh in background — valid for data where slight staleness is acceptable (leaderboards, stats). Emit invalidation hook alongside every write operation.
|
|
401
|
+
|
|
402
|
+
**Step 5 — Monitor hit/miss ratio**
|
|
403
|
+
Instrument cache calls to emit metrics: hit count, miss count, eviction count, cache size. Redis provides `INFO stats` — parse `keyspace_hits` and `keyspace_misses`. Target hit ratio > 80% for hot-path caches; < 50% indicates wrong key granularity or TTL too short. Alert on sudden hit ratio drop (invalidation bug) or memory > 80% of `maxmemory` (eviction risk).
|
|
404
|
+
|
|
405
|
+
#### Example
|
|
406
|
+
|
|
407
|
+
```typescript
|
|
408
|
+
// Redis cache-aside middleware for Express/Fastify
|
|
409
|
+
import { Redis } from 'ioredis';
|
|
410
|
+
const redis = new Redis(REDIS_URL);
|
|
411
|
+
|
|
412
|
+
const cacheMiddleware = (ttlSeconds: number, keyFn?: (req) => string) =>
|
|
413
|
+
async (req, res, next) => {
|
|
414
|
+
const key = keyFn ? keyFn(req) : `cache:${req.method}:${req.originalUrl}`;
|
|
415
|
+
const cached = await redis.get(key);
|
|
416
|
+
if (cached) {
|
|
417
|
+
res.setHeader('X-Cache', 'HIT');
|
|
418
|
+
return res.json(JSON.parse(cached));
|
|
419
|
+
}
|
|
420
|
+
const originalJson = res.json.bind(res);
|
|
421
|
+
res.json = (data) => {
|
|
422
|
+
// Only cache successful responses
|
|
423
|
+
if (res.statusCode < 400) redis.setex(key, ttlSeconds, JSON.stringify(data));
|
|
424
|
+
res.setHeader('X-Cache', 'MISS');
|
|
425
|
+
return originalJson(data);
|
|
426
|
+
};
|
|
427
|
+
next();
|
|
428
|
+
};
|
|
429
|
+
|
|
430
|
+
// Usage: cache product list for 5 minutes
|
|
431
|
+
app.get('/products', cacheMiddleware(300), async (req, res) => { /* handler */ });
|
|
432
|
+
|
|
433
|
+
// Cache stampede prevention: mutex lock on cache miss
|
|
434
|
+
const getWithLock = async <T>(key: string, fetchFn: () => Promise<T>, ttl: number): Promise<T> => {
|
|
435
|
+
const cached = await redis.get(key);
|
|
436
|
+
if (cached) return JSON.parse(cached);
|
|
437
|
+
|
|
438
|
+
const lockKey = `lock:${key}`;
|
|
439
|
+
const lock = await redis.set(lockKey, '1', 'EX', 10, 'NX'); // 10s lock
|
|
440
|
+
if (!lock) {
|
|
441
|
+
// Another process is fetching — wait briefly and retry
|
|
442
|
+
await new Promise(r => setTimeout(r, 100));
|
|
443
|
+
return getWithLock(key, fetchFn, ttl); // retry (max ~10 cycles within 10s lock)
|
|
444
|
+
}
|
|
445
|
+
|
|
446
|
+
try {
|
|
447
|
+
const data = await fetchFn();
|
|
448
|
+
await redis.setex(key, ttl, JSON.stringify(data));
|
|
449
|
+
return data;
|
|
450
|
+
} finally {
|
|
451
|
+
await redis.del(lockKey);
|
|
452
|
+
}
|
|
453
|
+
};
|
|
454
|
+
|
|
455
|
+
// Event-driven invalidation with Redis pub/sub
|
|
456
|
+
const invalidateOnMutation = async (userId: string) => {
|
|
457
|
+
await redis.del(`cache:user:${userId}`);
|
|
458
|
+
await redis.publish('cache:invalidate', JSON.stringify({ type: 'user', id: userId }));
|
|
459
|
+
};
|
|
460
|
+
|
|
461
|
+
// Cache-Control headers for browser/CDN caching
|
|
462
|
+
app.get('/products', (req, res) => {
|
|
463
|
+
res.setHeader('Cache-Control', 'public, max-age=300, stale-while-revalidate=60');
|
|
464
|
+
// ^ CDN caches 5min, serves stale for extra 60s while revalidating in background
|
|
465
|
+
res.json(products);
|
|
466
|
+
});
|
|
467
|
+
|
|
468
|
+
app.get('/user/profile', authenticate, (req, res) => {
|
|
469
|
+
res.setHeader('Cache-Control', 'private, max-age=60'); // user-specific, browser only
|
|
470
|
+
res.json(profile);
|
|
471
|
+
});
|
|
472
|
+
|
|
473
|
+
// In-memory LRU cache for single-process hot data
|
|
474
|
+
import LRU from 'lru-cache';
|
|
475
|
+
const configCache = new LRU<string, unknown>({ max: 500, ttl: 60_000 }); // 500 entries, 1min TTL
|
|
476
|
+
|
|
477
|
+
const getConfig = async (key: string) => {
|
|
478
|
+
if (configCache.has(key)) return configCache.get(key);
|
|
479
|
+
const value = await db.config.findUnique({ where: { key } });
|
|
480
|
+
configCache.set(key, value);
|
|
481
|
+
return value;
|
|
482
|
+
};
|
|
483
|
+
```
|
|
484
|
+
|
|
485
|
+
---
|
|
486
|
+
|
|
487
|
+
### background-jobs
|
|
488
|
+
|
|
489
|
+
Queue-based async processing — BullMQ (Node.js), job patterns, retry strategies, idempotency, dead letter queues, monitoring.
|
|
490
|
+
|
|
491
|
+
#### Workflow
|
|
492
|
+
|
|
493
|
+
**Step 1 — Identify async operations**
|
|
494
|
+
Scan route handlers and service functions for operations that: (a) take > 200ms (PDF generation, image resizing, report aggregation), (b) are non-user-facing (email sending, webhook delivery, analytics events), (c) can tolerate eventual consistency (data sync, cache warming, notification dispatch). Flag these as candidates for background jobs. Output a classification: fire-and-forget vs delayed vs scheduled (cron) vs fan-out.
|
|
495
|
+
|
|
496
|
+
**Step 2 — Choose queue system**
|
|
497
|
+
Node.js: BullMQ (Redis-backed, TypeScript-native, built-in retry/delay/priority/rate-limiting — recommended). Python: Celery + Redis/RabbitMQ broker (mature, distributed workers, beat scheduler for cron). For very simple use cases (single server, low volume): `node-cron` + in-process worker. Avoid in-process queues in production — they die with the process and lose jobs.
|
|
498
|
+
|
|
499
|
+
**Step 3 — Implement job with retry strategy**
|
|
500
|
+
Emit job producer (enqueue) and worker (processor) as separate files. Retry strategy: exponential backoff with jitter (`attempts: 5, backoff: { type: 'exponential', delay: 1000 }`). Idempotency: every job MUST have an idempotency key — use a deterministic ID from the operation (e.g., `email:welcome:${userId}` not a random UUID). This ensures duplicate enqueues (from retries, double-clicks) process exactly once. Dead letter queue: after max retries, move job to a `{queue-name}:failed` queue for inspection and manual replay — never silently drop.
|
|
501
|
+
|
|
502
|
+
**Step 4 — Add monitoring and alerting**
|
|
503
|
+
BullMQ Board or Bull Dashboard for visual queue monitoring. Emit metrics: queue depth (jobs waiting), processing rate (jobs/sec), failure rate (failed/total). Alert when: queue depth > threshold (workers not keeping up), failure rate > 5% (systematic error in processor), job age > expected TTL (stuck job). Use BullMQ events (`queue.on('failed', ...)`) to push metrics to Prometheus or Datadog.
|
|
504
|
+
|
|
505
|
+
**Step 5 — Handle dead letters**
|
|
506
|
+
Emit dead letter inspection endpoint: list failed jobs with error reason, retry count, and last error. Emit replay endpoint: re-enqueue a specific failed job with a fresh retry budget. Purge endpoint: clear dead letter queue after investigation. Add alerting on dead letter queue depth > 0 for critical job types (payment processing, compliance logging).
|
|
507
|
+
|
|
508
|
+
#### Example
|
|
509
|
+
|
|
510
|
+
```typescript
|
|
511
|
+
// BullMQ setup with TypeScript — producer + worker
|
|
512
|
+
import { Queue, Worker, Job } from 'bullmq';
|
|
513
|
+
|
|
514
|
+
const connection = { host: REDIS_HOST, port: 6379 };
|
|
515
|
+
|
|
516
|
+
// Job type definitions
|
|
517
|
+
interface EmailJob { to: string; template: string; data: Record<string, unknown> }
|
|
518
|
+
interface PdfJob { reportId: string; userId: string; format: 'pdf' | 'xlsx' }
|
|
519
|
+
|
|
520
|
+
// Producers
|
|
521
|
+
export const emailQueue = new Queue<EmailJob>('email', { connection });
|
|
522
|
+
export const pdfQueue = new Queue<PdfJob>('pdf', { connection });
|
|
523
|
+
|
|
524
|
+
// Enqueue with idempotency key (jobId = idempotent identifier)
|
|
525
|
+
export const sendWelcomeEmail = (userId: string, email: string) =>
|
|
526
|
+
emailQueue.add('welcome', { to: email, template: 'welcome', data: { userId } }, {
|
|
527
|
+
jobId: `email:welcome:${userId}`, // prevents duplicate welcome emails
|
|
528
|
+
attempts: 3,
|
|
529
|
+
backoff: { type: 'exponential', delay: 2_000 },
|
|
530
|
+
removeOnComplete: { count: 1000 }, // keep last 1000 completed for audit
|
|
531
|
+
removeOnFail: false, // keep all failed for dead letter review
|
|
532
|
+
});
|
|
533
|
+
|
|
534
|
+
// Scheduled/delayed job
|
|
535
|
+
export const sendReminderEmail = (userId: string, delayMs: number) =>
|
|
536
|
+
emailQueue.add('reminder', { to: userId, template: 'reminder', data: {} }, {
|
|
537
|
+
delay: delayMs,
|
|
538
|
+
attempts: 5,
|
|
539
|
+
backoff: { type: 'exponential', delay: 5_000 },
|
|
540
|
+
});
|
|
541
|
+
|
|
542
|
+
// Worker processor with error handling
|
|
543
|
+
const emailWorker = new Worker<EmailJob>('email', async (job: Job<EmailJob>) => {
|
|
544
|
+
const { to, template, data } = job.data;
|
|
545
|
+
// Validate job data — serialized payload may be stale
|
|
546
|
+
if (!to || !template) throw new Error(`Invalid job payload: ${JSON.stringify(job.data)}`);
|
|
547
|
+
await emailService.send({ to, template, data });
|
|
548
|
+
// Return value is stored in job.returnvalue for audit
|
|
549
|
+
return { sentAt: new Date().toISOString() };
|
|
550
|
+
}, {
|
|
551
|
+
connection,
|
|
552
|
+
concurrency: 10, // process up to 10 emails in parallel
|
|
553
|
+
limiter: { max: 100, duration: 60_000 }, // rate limit: 100/min
|
|
554
|
+
});
|
|
555
|
+
|
|
556
|
+
emailWorker.on('failed', async (job, err) => {
|
|
557
|
+
logger.error({ jobId: job?.id, queue: 'email', error: err.message, attempts: job?.attemptsMade });
|
|
558
|
+
if (job?.attemptsMade >= job?.opts.attempts!) {
|
|
559
|
+
// max retries exhausted → alert
|
|
560
|
+
await alerting.notify(`Dead letter: email job ${job.id} failed after ${job.attemptsMade} attempts`);
|
|
561
|
+
}
|
|
562
|
+
});
|
|
563
|
+
|
|
564
|
+
// Fan-out pattern: one job enqueues many children
|
|
565
|
+
const fanOutNotification = async (eventId: string, userIds: string[]) => {
|
|
566
|
+
const jobs = userIds.map(userId => ({
|
|
567
|
+
name: 'notify',
|
|
568
|
+
data: { userId, eventId },
|
|
569
|
+
opts: {
|
|
570
|
+
jobId: `notify:${eventId}:${userId}`,
|
|
571
|
+
attempts: 3,
|
|
572
|
+
backoff: { type: 'exponential', delay: 1_000 },
|
|
573
|
+
},
|
|
574
|
+
}));
|
|
575
|
+
await notificationQueue.addBulk(jobs);
|
|
576
|
+
};
|
|
577
|
+
|
|
578
|
+
// Dead letter inspection API
|
|
579
|
+
app.get('/admin/jobs/failed', authenticate, authorize('admin'), async (req, res) => {
|
|
580
|
+
const failed = await emailQueue.getFailed(0, 50);
|
|
581
|
+
res.json({ count: failed.length, jobs: failed.map(j => ({ id: j.id, data: j.data, reason: j.failedReason, attempts: j.attemptsMade })) });
|
|
582
|
+
});
|
|
583
|
+
|
|
584
|
+
app.post('/admin/jobs/:id/retry', authenticate, authorize('admin'), async (req, res) => {
|
|
585
|
+
const job = await emailQueue.getJob(req.params.id);
|
|
586
|
+
if (!job) return res.status(404).json({ error: { code: 'NOT_FOUND' } });
|
|
587
|
+
await job.retry();
|
|
588
|
+
res.json({ status: 'retried' });
|
|
589
|
+
});
|
|
590
|
+
|
|
591
|
+
// Celery equivalent (Python) — minimal pattern
|
|
592
|
+
# tasks.py
|
|
593
|
+
from celery import Celery
|
|
594
|
+
from celery.utils.log import get_task_logger
|
|
595
|
+
|
|
596
|
+
app = Celery('tasks', broker=REDIS_URL, backend=REDIS_URL)
|
|
597
|
+
app.conf.task_acks_late = True # at-least-once delivery
|
|
598
|
+
app.conf.task_reject_on_worker_lost = True # requeue on worker crash
|
|
599
|
+
logger = get_task_logger(__name__)
|
|
600
|
+
|
|
601
|
+
@app.task(bind=True, max_retries=5, default_retry_delay=60)
|
|
602
|
+
def send_email(self, to: str, template: str, data: dict) -> dict:
|
|
603
|
+
try:
|
|
604
|
+
result = email_service.send(to=to, template=template, data=data)
|
|
605
|
+
return {'sent_at': result.timestamp.isoformat()}
|
|
606
|
+
except TransientError as exc:
|
|
607
|
+
raise self.retry(exc=exc, countdown=2 ** self.request.retries * 60)
|
|
608
|
+
except PermanentError as exc:
|
|
609
|
+
logger.error(f"Permanent failure for {to}: {exc}")
|
|
610
|
+
raise # no retry — goes to dead letter
|
|
611
|
+
```
|
|
612
|
+
|
|
613
|
+
---
|
|
614
|
+
|
|
615
|
+
## Connections
|
|
616
|
+
|
|
617
|
+
```
|
|
618
|
+
Calls → docs-seeker (L3): lookup API documentation and framework guides
|
|
619
|
+
Calls → sentinel (L2): security audit on auth implementations
|
|
620
|
+
Calls → watchdog (L3): monitor queue depth and cache hit ratios
|
|
621
|
+
Called By ← cook (L1): when backend task detected
|
|
622
|
+
Called By ← review (L2): when API/backend code is being reviewed
|
|
623
|
+
Called By ← audit (L2): backend health dimension
|
|
624
|
+
Called By ← deploy (L2): pre-deploy readiness checks (health endpoints, graceful shutdown)
|
|
625
|
+
```
|
|
626
|
+
|
|
627
|
+
## Tech Stack Support
|
|
628
|
+
|
|
629
|
+
| Framework | ORM | Auth Library | Queue | Cache |
|
|
630
|
+
|-----------|-----|-------------|-------|-------|
|
|
631
|
+
| Express 5 | Prisma | Passport / custom JWT | BullMQ | ioredis |
|
|
632
|
+
| Fastify 5 | Drizzle | @fastify/jwt | BullMQ | ioredis |
|
|
633
|
+
| Next.js 16 (Route Handlers) | Prisma | NextAuth v5 / Lucia | BullMQ | ioredis / Upstash |
|
|
634
|
+
| NestJS 11 | TypeORM / Prisma | @nestjs/passport | @nestjs/bull | @nestjs/cache-manager |
|
|
635
|
+
| FastAPI | SQLAlchemy | python-jose / authlib | Celery | redis-py |
|
|
636
|
+
| Django 5 | Django ORM | django-rest-framework | Celery | django-redis |
|
|
637
|
+
|
|
638
|
+
## Constraints
|
|
639
|
+
|
|
640
|
+
1. MUST use parameterized queries for ALL database operations — never string interpolation in SQL.
|
|
641
|
+
2. MUST NOT store secrets (JWT secret, API keys, DB password) in source code — use environment variables validated at startup.
|
|
642
|
+
3. MUST emit migration files for all schema changes — no direct `ALTER TABLE` in application code.
|
|
643
|
+
4. MUST validate all request input at the boundary (middleware/decorator) — not inside business logic.
|
|
644
|
+
5. MUST return consistent error envelope format across all endpoints — `{ error: { code, message }, request_id }`.
|
|
645
|
+
6. MUST assign idempotency keys to all background jobs — never use random UUID as job ID for domain operations.
|
|
646
|
+
7. MUST emit cache invalidation logic alongside every write operation that affects cached data.
|
|
647
|
+
|
|
648
|
+
## Sharp Edges
|
|
649
|
+
|
|
650
|
+
| Failure Mode | Severity | Mitigation |
|
|
651
|
+
|---|---|---|
|
|
652
|
+
| Auth pattern emits JWT without expiry or with excessively long TTL (>24h for access token) | CRITICAL | Hard-code max 15min access / 7d refresh in emitted code; flag any `expiresIn` > threshold |
|
|
653
|
+
| Cache stampede: many concurrent misses hit DB simultaneously under load | HIGH | Emit mutex lock (Redis `SET NX`) pattern on cache miss; probabilistic early recomputation for hot keys |
|
|
654
|
+
| Job payload contains non-serializable data (functions, class instances, circular refs) | HIGH | Validate payload is plain JSON-serializable object before enqueue; emit `JSON.parse(JSON.stringify(data))` guard |
|
|
655
|
+
| N+1 detection misses ORM lazy-loading (Sequelize, TypeORM default behavior) | HIGH | Check ORM config for `lazy: true`; audit `.then()` chains on relations; enable query logging in dev |
|
|
656
|
+
| Migration emitted without rollback script (ALTER without DOWN migration) | HIGH | Every migration must include both `up()` and `down()` — flag any migration without both |
|
|
657
|
+
| Unbounded in-memory cache grows until OOM (missing `max` option on LRU) | HIGH | Always set `max` entries and `ttl` on LRU caches; emit memory usage metric |
|
|
658
|
+
| CORS middleware set to `origin: '*'` in production | MEDIUM | Check NODE_ENV / deployment target; flag wildcard CORS in production configs |
|
|
659
|
+
| Middleware order wrong (error handler before routes, validation after route handler) | MEDIUM | Emit middleware registration in correct order with comments explaining why |
|
|
660
|
+
| Read replica replication lag causes stale reads immediately after writes | MEDIUM | Route read-after-write in the same request to primary; use replica only for independent reads |
|
|
661
|
+
| Dead letter queue ignored — failed jobs accumulate silently for weeks | MEDIUM | Emit alert on dead letter queue depth > 0 for critical queues; add to health dashboard |
|
|
662
|
+
| Graceful shutdown timeout too short — in-flight requests killed mid-operation | LOW | Default 30s timeout; increase to 60s for jobs with long processing time (PDF, video) |
|
|
663
|
+
| Rate limiting suggested but Redis/store not available in project | LOW | Check for existing Redis/memory store; suggest in-memory rate limiter as fallback |
|
|
664
|
+
|
|
665
|
+
## Done When
|
|
666
|
+
|
|
667
|
+
- API audit report emitted with naming violations, missing pagination, versioning strategy, and fix diffs
|
|
668
|
+
- Auth flow hardened: short-lived access tokens, httpOnly refresh cookies, proper hashing, OAuth/OIDC integration ready
|
|
669
|
+
- N+1 queries detected and replaced with eager loading; soft delete pattern applied; missing indexes migrated
|
|
670
|
+
- Middleware stack has: request ID, structured logging, global error handler, input validation, compression, graceful shutdown, health endpoints
|
|
671
|
+
- Caching strategy implemented: cacheable endpoints identified, cache layer selected, invalidation logic emitted alongside every write
|
|
672
|
+
- Async operations moved to background jobs: idempotency keys assigned, retry strategy configured, dead letter queue wired
|
|
673
|
+
- All emitted code uses project's existing framework and ORM (detected from package.json)
|
|
674
|
+
- Structured report emitted for each skill invoked
|
|
675
|
+
|
|
676
|
+
## Cost Profile
|
|
677
|
+
|
|
678
|
+
~10,000–20,000 tokens per full pack run (all 6 skills). Individual skill: ~2,000–4,000 tokens. Sonnet default for code generation and security audit. Use haiku for detection scans (Step 1 of each skill). Escalate to opus for architecture decisions on caching topology or queue system selection in high-traffic systems.
|