@ai-support-agent/cli 0.1.32 → 0.1.33-beta.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/alert-processor.d.ts +0 -8
- package/dist/alert-processor.d.ts.map +1 -1
- package/dist/alert-processor.js +5 -73
- package/dist/alert-processor.js.map +1 -1
- package/dist/api-client.d.ts +0 -4
- package/dist/api-client.d.ts.map +1 -1
- package/dist/api-client.js +0 -4
- package/dist/api-client.js.map +1 -1
- package/dist/commands/claude-code-args.d.ts +1 -0
- package/dist/commands/claude-code-args.d.ts.map +1 -1
- package/dist/commands/claude-code-args.js +3 -0
- package/dist/commands/claude-code-args.js.map +1 -1
- package/dist/commands/claude-code-runner.d.ts.map +1 -1
- package/dist/commands/claude-code-runner.js +2 -1
- package/dist/commands/claude-code-runner.js.map +1 -1
- package/dist/commands/plugin-dir.d.ts +20 -0
- package/dist/commands/plugin-dir.d.ts.map +1 -0
- package/dist/commands/plugin-dir.js +71 -0
- package/dist/commands/plugin-dir.js.map +1 -0
- package/dist/constants.d.ts +0 -1
- package/dist/constants.d.ts.map +1 -1
- package/dist/constants.js +0 -1
- package/dist/constants.js.map +1 -1
- package/dist/plugin/.claude-plugin/plugin.json +9 -0
- package/dist/plugin/LICENSE +26 -0
- package/dist/plugin/README.md +86 -0
- package/dist/plugin/SYNC.md +113 -0
- package/dist/plugin/agents/build-error-resolver.md +159 -0
- package/dist/plugin/agents/code-reviewer.md +277 -0
- package/dist/plugin/agents/django-reviewer.md +159 -0
- package/dist/plugin/agents/infra-reviewer.md +172 -0
- package/dist/plugin/agents/investigator.md +75 -0
- package/dist/plugin/agents/nextjs-reviewer.md +191 -0
- package/dist/plugin/agents/php-reviewer.md +167 -0
- package/dist/plugin/agents/planner.md +184 -0
- package/dist/plugin/agents/python-reviewer.md +161 -0
- package/dist/plugin/agents/react-reviewer.md +150 -0
- package/dist/plugin/agents/silent-failure-hunter.md +158 -0
- package/dist/plugin/agents/typescript-reviewer.md +179 -0
- package/dist/plugin/agents/ui-reviewer.md +203 -0
- package/dist/plugin/commands/add-feature.md +301 -0
- package/dist/plugin/commands/build-fix.md +47 -0
- package/dist/plugin/commands/code-review.md +228 -0
- package/dist/plugin/commands/fix-defect.md +393 -0
- package/dist/plugin/commands/learn-eval.md +94 -0
- package/dist/plugin/commands/learn.md +84 -0
- package/dist/plugin/commands/plan.md +211 -0
- package/dist/plugin/commands/test-coverage.md +64 -0
- package/dist/plugin/commands/update-docs.md +98 -0
- package/dist/plugin/hooks/hooks.json +59 -0
- package/dist/plugin/hooks/scripts/auto-format.sh +63 -0
- package/dist/plugin/hooks/scripts/check-secrets-before-commit.sh +50 -0
- package/dist/plugin/hooks/scripts/guard-dangerous-commands.sh +55 -0
- package/dist/plugin/hooks/scripts/on-command-resume.sh +112 -0
- package/dist/plugin/hooks/scripts/on-command-stop.sh +200 -0
- package/dist/plugin/hooks/scripts/protect-sensitive-files.sh +58 -0
- package/dist/plugin/rules/common/coding-guidelines.md +73 -0
- package/dist/plugin/rules/documentation/api-docs.md +46 -0
- package/dist/plugin/rules/documentation/docs-site.md +60 -0
- package/dist/plugin/rules/documentation/source-docs.md +89 -0
- package/dist/plugin/rules/documentation/test-docs.md +39 -0
- package/dist/plugin/rules/logging/logging-rules.md +83 -0
- package/dist/plugin/rules/php/coding-rules.md +40 -0
- package/dist/plugin/rules/python/coding-rules.md +40 -0
- package/dist/plugin/rules/typescript/coding-rules.md +45 -0
- package/dist/plugin/skills/api-design/SKILL.md +269 -0
- package/dist/plugin/skills/backend-patterns/SKILL.md +312 -0
- package/dist/plugin/skills/database-migrations/SKILL.md +323 -0
- package/dist/plugin/skills/docker-patterns/SKILL.md +308 -0
- package/dist/plugin/skills/docs-site/SKILL.md +341 -0
- package/dist/plugin/skills/e2e-testing/SKILL.md +334 -0
- package/dist/plugin/skills/frontend-patterns/SKILL.md +318 -0
- package/dist/plugin/skills/integration-testing/SKILL.md +273 -0
- package/package.json +2 -2
|
@@ -0,0 +1,312 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: backend-patterns
|
|
3
|
+
description: A collection of backend design and implementation patterns covering layering, input validation, N+1 queries and transactions, caching, error handling, authentication/authorization, rate limiting, structured logging, and background jobs, for NestJS / Django / Flask / Laravel / CakePHP. Use this when designing, implementing, or reviewing backend APIs, service layers, or jobs.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# backend-patterns: Backend Design & Implementation Patterns
|
|
7
|
+
|
|
8
|
+
Target stacks: NestJS / Django / Flask / Laravel / CakePHP.
|
|
9
|
+
Code examples favor NestJS, Django, and Laravel; Flask and CakePHP equivalents are noted alongside them.
|
|
10
|
+
For interface conventions such as URL design and status codes, see the api-design skill.
|
|
11
|
+
|
|
12
|
+
## Layering
|
|
13
|
+
|
|
14
|
+
- Keep controllers (handlers) thin. Their only responsibilities are: accepting and transforming input, calling the service layer, and shaping the response.
|
|
15
|
+
- Put business logic in the service layer. Conditional branching, calculations, and cross-resource consistency all belong here.
|
|
16
|
+
- Keep data access isolated in a repository/ORM layer. Don't leak raw SQL or query-builder details into the service layer.
|
|
17
|
+
- Structuring things so the same logic can be called from both a controller and a job (CLI or queue worker) is a good self-check for whether your layering is actually sound.
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
// NestJS
|
|
21
|
+
// Bad: business logic and data access live directly in the controller
|
|
22
|
+
@Post()
|
|
23
|
+
async create(@Body() body: any) {
|
|
24
|
+
const user = await this.userRepo.findOne({ where: { email: body.email } });
|
|
25
|
+
if (user) throw new BadRequestException('exists');
|
|
26
|
+
if (body.plan === 'pro' && !body.cardToken) throw new BadRequestException();
|
|
27
|
+
return this.userRepo.save({ ...body });
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
// Good: the controller only passes through; decisions live in the service layer
|
|
31
|
+
@Post()
|
|
32
|
+
async create(@Body() dto: CreateUserDto) {
|
|
33
|
+
return this.usersService.register(dto);
|
|
34
|
+
}
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
In Django, keep views thin and push logic into a service module (e.g. `services.py`) or model methods. In Laravel, delegate from controllers to service/action classes, and push complex Eloquent queries into query scopes or repositories. In CakePHP, preserve the Controller → Table (model layer) separation.
|
|
38
|
+
|
|
39
|
+
## Input Validation
|
|
40
|
+
|
|
41
|
+
Use the framework's built-in validation mechanism. Hand-rolled chains of `if` statements are easy to miss cases with and hard to review.
|
|
42
|
+
|
|
43
|
+
- NestJS: DTOs + class-validator + `ValidationPipe` (always enable `whitelist: true`)
|
|
44
|
+
- Django: Forms / DRF Serializers
|
|
45
|
+
- Flask: schema validation libraries such as marshmallow or pydantic
|
|
46
|
+
- Laravel: FormRequest
|
|
47
|
+
- CakePHP: Validator (the Table's `validationDefault`)
|
|
48
|
+
|
|
49
|
+
```typescript
|
|
50
|
+
// NestJS
|
|
51
|
+
// Good: declare types and constraints via a DTO, and strip unknown fields with whitelist
|
|
52
|
+
export class CreateUserDto {
|
|
53
|
+
@IsEmail()
|
|
54
|
+
email: string;
|
|
55
|
+
|
|
56
|
+
@IsString()
|
|
57
|
+
@Length(8, 72)
|
|
58
|
+
password: string;
|
|
59
|
+
}
|
|
60
|
+
// main.ts: app.useGlobalPipes(new ValidationPipe({ whitelist: true }));
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
```python
|
|
64
|
+
# Django REST Framework
|
|
65
|
+
# Bad: passing request.data straight into the model (mass assignment)
|
|
66
|
+
User.objects.create(**request.data)
|
|
67
|
+
|
|
68
|
+
# Good: use a serializer to explicitly declare which fields are accepted
|
|
69
|
+
class UserSerializer(serializers.ModelSerializer):
|
|
70
|
+
class Meta:
|
|
71
|
+
model = User
|
|
72
|
+
fields = ["email", "password"] # avoid fields = "__all__"
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
```php
|
|
76
|
+
// Laravel
|
|
77
|
+
// Bad: mass-assigning $request->all() (is_admin etc. could be overwritten)
|
|
78
|
+
User::create($request->all());
|
|
79
|
+
|
|
80
|
+
// Good: use only FormRequest's validated() data, and keep $fillable minimal
|
|
81
|
+
public function store(StoreUserRequest $request)
|
|
82
|
+
{
|
|
83
|
+
return User::create($request->validated());
|
|
84
|
+
}
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
The principle behind mass-assignment protection is the same everywhere: explicitly whitelist which fields may be accepted. Attributes tied to permissions or ownership — `is_admin`, `role`, `tenant_id`, and the like — should be set from server-side context (the authenticated identity), never from the request body.
|
|
88
|
+
|
|
89
|
+
## Database
|
|
90
|
+
|
|
91
|
+
### Detecting N+1 Queries and Batching Fetches
|
|
92
|
+
|
|
93
|
+
Always be suspicious of queries issued inside a loop. During development, check actual query counts with a query log (Django Debug Toolbar, Laravel Telescope / `DB::listen`, TypeORM's `logging: true`).
|
|
94
|
+
|
|
95
|
+
```python
|
|
96
|
+
# Django
|
|
97
|
+
# Bad: fetches the author one row at a time per article (1 + N queries)
|
|
98
|
+
for article in Article.objects.all():
|
|
99
|
+
print(article.author.name)
|
|
100
|
+
|
|
101
|
+
# Good: preload via JOIN / batch fetch
|
|
102
|
+
for article in Article.objects.select_related("author"):
|
|
103
|
+
print(article.author.name)
|
|
104
|
+
# Use prefetch_related for many-to-many and reverse relations
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
```php
|
|
108
|
+
// Laravel
|
|
109
|
+
// Bad: accessing $article->author inside a loop triggers N lazy-load queries
|
|
110
|
+
$articles = Article::all();
|
|
111
|
+
|
|
112
|
+
// Good: eager loading fetches everything up front
|
|
113
|
+
$articles = Article::with('author')->get();
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
In NestJS + TypeORM, achieve the same with `relations` or `leftJoinAndSelect` on the QueryBuilder. With Prisma, use `include`.
|
|
117
|
+
|
|
118
|
+
### Transaction Boundaries
|
|
119
|
+
|
|
120
|
+
Identify the unit of work that must be "all succeed or all fail," and wrap exactly that unit in a transaction.
|
|
121
|
+
|
|
122
|
+
- Open transactions in the service layer. Opening them in a controller or in an individual repository call leaves the boundary ambiguous.
|
|
123
|
+
- Never put external API calls, email sending, or other long-running operations inside a transaction. Doing so extends how long locks are held, and if the external call fails and you roll back the DB, the external side effect can't be undone. Perform external effects after commit, or hand them off to a job queue.
|
|
124
|
+
|
|
125
|
+
```python
|
|
126
|
+
# Django
|
|
127
|
+
# Good: confirming an order and decrementing stock happen in one transaction
|
|
128
|
+
from django.db import transaction
|
|
129
|
+
|
|
130
|
+
def confirm_order(order_id: int) -> None:
|
|
131
|
+
with transaction.atomic():
|
|
132
|
+
order = Order.objects.select_for_update().get(pk=order_id)
|
|
133
|
+
order.confirm()
|
|
134
|
+
order.save()
|
|
135
|
+
Stock.objects.filter(product=order.product).update(
|
|
136
|
+
quantity=F("quantity") - order.quantity
|
|
137
|
+
)
|
|
138
|
+
# Good: send the email only after commit
|
|
139
|
+
transaction.on_commit(lambda: send_order_mail.delay(order_id))
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Laravel uses `DB::transaction(fn () => ...)`; NestJS + TypeORM uses `dataSource.transaction()` or a QueryRunner. In systems built around command/event sourcing with separate write and read models, a single command is typically the unit of consistency — don't try to force consistency across multiple aggregates into one transaction; instead, design the cross-aggregate effects as eventually-consistent follow-up processing.
|
|
143
|
+
|
|
144
|
+
## Caching
|
|
145
|
+
|
|
146
|
+
The default pattern is cache-aside: check the cache on read, and on a miss, fetch from the database and populate the cache.
|
|
147
|
+
|
|
148
|
+
```typescript
|
|
149
|
+
// NestJS (cache-aside skeleton)
|
|
150
|
+
async getProduct(id: string): Promise<Product> {
|
|
151
|
+
const cached = await this.cache.get<Product>(`product:${id}`);
|
|
152
|
+
if (cached) return cached;
|
|
153
|
+
const product = await this.productRepo.findOneByOrFail({ id });
|
|
154
|
+
await this.cache.set(`product:${id}`, product, 300); // always set a TTL
|
|
155
|
+
return product;
|
|
156
|
+
}
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
- Design assuming invalidation is hard. Combine explicit deletion on update with a TTL as a second line of defense, and decide per data type how much staleness is acceptable until the TTL expires. Standardize keys as `resource-name:id` so you can always identify what to invalidate.
|
|
160
|
+
- Safe to cache: master/reference data, public content, expensive aggregate results — anything that looks the same to everyone and can tolerate some staleness.
|
|
161
|
+
- Do not cache (or be extremely careful with): authorization decisions, entire responses containing personal information, or values that demand strict accuracy such as balances or inventory counts. Storing user-specific data under a key that doesn't include the user ID is a classic way to leak one user's data to another — avoid it absolutely.
|
|
162
|
+
|
|
163
|
+
## Error Handling
|
|
164
|
+
|
|
165
|
+
### Centralized Handlers
|
|
166
|
+
|
|
167
|
+
Don't scatter exception handling throughout the codebase; consolidate it in the framework's centralized mechanism.
|
|
168
|
+
|
|
169
|
+
- NestJS: Exception Filter (`@Catch()`)
|
|
170
|
+
- Django: middleware / DRF's `EXCEPTION_HANDLER`
|
|
171
|
+
- Flask: `@app.errorhandler`
|
|
172
|
+
- Laravel: exception handling configuration in `bootstrap/app.php` (formerly the Handler class)
|
|
173
|
+
- CakePHP: ErrorController / a custom ExceptionRenderer
|
|
174
|
+
|
|
175
|
+
A centralized handler has three jobs: logging (including the stack trace), producing a safe, well-formed response for the client, and mapping errors to status codes.
|
|
176
|
+
|
|
177
|
+
### Preventing Internal Information Leaks
|
|
178
|
+
|
|
179
|
+
Never include stack traces, SQL, file paths, or environment variables in production error responses. Always verify `DEBUG=False` (Django) and `APP_DEBUG=false` (Laravel) in production. Return only an error code and a correlation ID (request ID) to the client; keep the details in server-side logs.
|
|
180
|
+
|
|
181
|
+
```typescript
|
|
182
|
+
// NestJS
|
|
183
|
+
// Bad: returns the raw exception message (leaks internal details)
|
|
184
|
+
catch (e) { throw new InternalServerErrorException(e.message); }
|
|
185
|
+
|
|
186
|
+
// Good: log the details, return only a fixed code and correlation ID to the client
|
|
187
|
+
catch (e) {
|
|
188
|
+
this.logger.error({ requestId, err: e }, 'order confirmation failed');
|
|
189
|
+
throw new InternalServerErrorException({ code: 'INTERNAL_ERROR', requestId });
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
### Retries with Exponential Backoff (External API Calls)
|
|
194
|
+
|
|
195
|
+
Transient external API failures (timeouts, 429s, 5xx) are worth retrying. 4xx errors (validation failures, etc.) won't succeed on retry, so fail fast instead.
|
|
196
|
+
|
|
197
|
+
```python
|
|
198
|
+
# Python (conceptual skeleton — in practice, use a library like tenacity)
|
|
199
|
+
# Good: exponential backoff + jitter, with a cap on retry attempts
|
|
200
|
+
import random, time
|
|
201
|
+
|
|
202
|
+
def call_with_retry(func, max_attempts=4, base=0.5):
|
|
203
|
+
for attempt in range(1, max_attempts + 1):
|
|
204
|
+
try:
|
|
205
|
+
return func()
|
|
206
|
+
except TransientError:
|
|
207
|
+
if attempt == max_attempts:
|
|
208
|
+
raise
|
|
209
|
+
time.sleep(base * (2 ** (attempt - 1)) + random.uniform(0, 0.1))
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Anything you retry must be idempotent. Don't casually retry a non-idempotent POST — only do so if you can attach an idempotency key. Use the silent-failure-hunter subagent to check for swallowed exceptions (`except: pass`, empty `catch` blocks).
|
|
213
|
+
|
|
214
|
+
## Authentication & Authorization
|
|
215
|
+
|
|
216
|
+
- Always enforce authorization server-side. Hiding a button or a screen on the frontend is a UX affordance, not a security control — design as if the API can be hit directly.
|
|
217
|
+
- Separate authentication (who you are) from authorization (what you're allowed to do). Even after authentication succeeds, check permission on the target resource for every request.
|
|
218
|
+
- Use each framework's declarative mechanism for role-based authorization: NestJS uses Guards + custom decorators, Django uses `permission_classes` / `@permission_required`, Laravel uses Policies/Gates, Flask uses decorators, CakePHP uses the Authorization plugin.
|
|
219
|
+
- Guard against IDOR (Insecure Direct Object Reference — being able to view another user's resource just by guessing its ID) by including "does the logged-in user own this" as part of the query condition, not as an afterthought check.
|
|
220
|
+
|
|
221
|
+
```php
|
|
222
|
+
// Laravel
|
|
223
|
+
// Bad: fetching by ID alone (exposes other users' orders)
|
|
224
|
+
$order = Order::findOrFail($id);
|
|
225
|
+
|
|
226
|
+
// Good: scope the query to the owner, then authorize via a Policy
|
|
227
|
+
$order = $request->user()->orders()->findOrFail($id);
|
|
228
|
+
$this->authorize('view', $order);
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
```typescript
|
|
232
|
+
// NestJS
|
|
233
|
+
// Good: declare the required role via a decorator and enforce it centrally with a Guard
|
|
234
|
+
@Roles('admin')
|
|
235
|
+
@UseGuards(JwtAuthGuard, RolesGuard)
|
|
236
|
+
@Delete(':id')
|
|
237
|
+
remove(@Param('id') id: string) { ... }
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
In multi-tenant setups, never accept the tenant ID from the request body — derive it from the authenticated identity and enforce it on every query.
|
|
241
|
+
|
|
242
|
+
## Rate Limiting
|
|
243
|
+
|
|
244
|
+
Apply rate limiting to publicly exposed endpoints (login, sign-up, password reset, search, public APIs). This defends against both brute-force attacks and general overload.
|
|
245
|
+
|
|
246
|
+
- NestJS: `@nestjs/throttler`
|
|
247
|
+
- Django: DRF throttling (`AnonRateThrottle` / `ScopedRateThrottle`), django-ratelimit
|
|
248
|
+
- Flask: Flask-Limiter
|
|
249
|
+
- Laravel: the `throttle` middleware (RateLimiter)
|
|
250
|
+
- CakePHP: a rate-limiting plugin used as middleware
|
|
251
|
+
- Serverless setups: also apply throttling at the API Gateway / WAF layer
|
|
252
|
+
|
|
253
|
+
In multi-instance deployments, the rate-limit counter must live in a shared store such as Redis — otherwise per-instance counters make the limit effectively meaningless.
|
|
254
|
+
|
|
255
|
+
## Structured Logging
|
|
256
|
+
|
|
257
|
+
- Emit logs in a structured format (e.g. JSON) rather than embedding values into message strings, so they're searchable and aggregable as fields.
|
|
258
|
+
- Attach a request ID (correlation ID) to every log line so you can trace one request's full processing path. In NestJS, use AsyncLocalStorage (e.g. nestjs-cls); in Django/Flask, attach it via middleware.
|
|
259
|
+
- For the full house rules on log levels, debug-mode conventions (recording function entry/exit and inputs/outputs), and X-Ray/Sentry integration, consult your repository's own logging conventions/rules document.
|
|
260
|
+
- Include identifying fields (userId, orderId, etc.) as structured fields. Never log personal data or credentials — passwords, tokens, card numbers — under any circumstances.
|
|
261
|
+
|
|
262
|
+
```typescript
|
|
263
|
+
// NestJS (structured logger such as pino)
|
|
264
|
+
// Bad: no context, not searchable, doesn't say what failed
|
|
265
|
+
this.logger.log('failed');
|
|
266
|
+
|
|
267
|
+
// Good: correlation ID and target ID as structured fields
|
|
268
|
+
this.logger.warn(
|
|
269
|
+
{ requestId, orderId, reason: 'stock_shortage' },
|
|
270
|
+
'order confirmation rejected',
|
|
271
|
+
);
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
How to choose a log level:
|
|
275
|
+
|
|
276
|
+
- `error`: a failure that needs a human to respond. Should trigger alerts.
|
|
277
|
+
- `warn`: something abnormal but self-recovered, or an expected-in-business failure (succeeded after retry, frequent validation rejections, etc.).
|
|
278
|
+
- `info`: business-event milestones (order confirmed, job completed).
|
|
279
|
+
- `debug`: development-time detail. Disabled by default in production.
|
|
280
|
+
|
|
281
|
+
Avoid letting your level scheme collapse — e.g. logging normal operation at `error`, or swallowing an exception while logging it at `info`.
|
|
282
|
+
|
|
283
|
+
## Background Jobs
|
|
284
|
+
|
|
285
|
+
- Use a durable queue backend: BullMQ (Redis), Celery (Redis / RabbitMQ), Laravel Queue (Redis / SQS / database), SQS, and similar. In-process queues — an in-memory array, chained `setTimeout` calls, or Laravel's `sync` driver in production — are not acceptable; jobs vanish on process restart or deploy.
|
|
286
|
+
- Design jobs to be idempotent. Queues generally guarantee at-least-once delivery, so the same job may run twice — make sure that doesn't corrupt results. Absorb duplicate execution with a "already processed" flag, a uniqueness constraint, or an idempotency key.
|
|
287
|
+
|
|
288
|
+
```python
|
|
289
|
+
# Celery
|
|
290
|
+
# Bad: running this twice double-charges the customer
|
|
291
|
+
@shared_task
|
|
292
|
+
def charge(order_id):
|
|
293
|
+
api.charge(amount=get_order(order_id).total)
|
|
294
|
+
|
|
295
|
+
# Good: a processed-check plus an idempotency key make double execution harmless
|
|
296
|
+
@shared_task(bind=True, max_retries=5)
|
|
297
|
+
def charge(self, order_id):
|
|
298
|
+
order = get_order(order_id)
|
|
299
|
+
if order.charged:
|
|
300
|
+
return
|
|
301
|
+
api.charge(amount=order.total, idempotency_key=f"charge-{order_id}")
|
|
302
|
+
order.mark_charged()
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
- Always design for the failure path: cap the retry count, use exponential backoff, and on exhausting retries, route the job to a dead-letter queue (DLQ) or a failure table (Laravel's `failed_jobs`) with a notification. Never let a failure disappear silently.
|
|
306
|
+
- Pass an ID into the job, not the full entity, and re-fetch the latest state at execution time — this avoids acting on stale data that sat in the queue for a while.
|
|
307
|
+
|
|
308
|
+
## Related Commands & Subagents
|
|
309
|
+
|
|
310
|
+
- During the design phase, use `/plan` to nail down the implementation approach before starting.
|
|
311
|
+
- After implementation, request review from `/code-review` and code-reviewer subagents.
|
|
312
|
+
- Use the silent-failure-hunter subagent to check for swallowed exceptions and silently discarded errors.
|
|
@@ -0,0 +1,323 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: database-migrations
|
|
3
|
+
description: A pattern library for running database migrations safely. Covers universal principles such as backward compatibility, the expand/contract pattern, and rollback design; dangerous operations to avoid in PostgreSQL/MySQL; tool-specific notes for Django, Laravel, TypeORM, and Prisma; and lazy migration for DynamoDB. Use this when planning or reviewing schema changes, data migrations, or migration files.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Database Migrations — Patterns for Safe DB Migrations
|
|
7
|
+
|
|
8
|
+
Migrations against a production database tend to be operations you can't undo once they fail.
|
|
9
|
+
This skill collects the principles, dangerous patterns, and tool-specific pointers needed to
|
|
10
|
+
carry out schema changes and data migrations with zero downtime and zero data loss.
|
|
11
|
+
|
|
12
|
+
## 1. Universal Principles
|
|
13
|
+
|
|
14
|
+
### 1.1 Backward compatibility: don't break the old code while it's still running
|
|
15
|
+
|
|
16
|
+
During a deploy there is always a moment where "old code + new schema" or "new code + old schema"
|
|
17
|
+
coexist. With rolling deploys this window can last minutes; even with blue/green deploys there's
|
|
18
|
+
a brief moment of overlap at cutover. A migration must therefore always satisfy both of these:
|
|
19
|
+
|
|
20
|
+
- The new schema must not break the old code (don't drop or rename columns first)
|
|
21
|
+
- The new code must be able to start up against the old schema (don't assume a new column is
|
|
22
|
+
already required)
|
|
23
|
+
|
|
24
|
+
### 1.2 Expand / Contract (the two-phase deploy)
|
|
25
|
+
|
|
26
|
+
Never make a breaking change in one shot. Always break it down into "add → migrate → remove."
|
|
27
|
+
|
|
28
|
+
```text
|
|
29
|
+
Phase 1 (Expand) : Add new columns/tables. Old code simply ignores them, so nothing breaks
|
|
30
|
+
Phase 2 (Migrate) : Deploy new code that writes to both, or treats the new side as authoritative.
|
|
31
|
+
Backfill the data
|
|
32
|
+
Phase 3 (Contract) : Once you've confirmed all code reads only from the new side, drop the old
|
|
33
|
+
columns/tables
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Leave at least one full deploy cycle between Phase 1 and Phase 3.
|
|
37
|
+
There's no need to rush Phase 3 — you can always delete later, but deleted data doesn't come back.
|
|
38
|
+
|
|
39
|
+
### 1.3 Rollback-ability
|
|
40
|
+
|
|
41
|
+
- For every migration, ask: "if the app is rolled back to the previous version right after this
|
|
42
|
+
migration is applied, does it still work?"
|
|
43
|
+
- A rollback mechanism doesn't have to be a "down" migration. If you follow expand/contract,
|
|
44
|
+
rolling back just the application *is* the rollback (the schema can safely stay in its
|
|
45
|
+
forward-compatible state)
|
|
46
|
+
- Destructive operations (DROP, TRUNCATE, narrowing a column's type) can't be undone by a down
|
|
47
|
+
migration. Take a backup or archive the data to a separate table before running them
|
|
48
|
+
|
|
49
|
+
### 1.4 Keep schema changes and data migrations separate
|
|
50
|
+
|
|
51
|
+
Don't mix DDL (schema changes) and DML (data migrations) in the same migration.
|
|
52
|
+
|
|
53
|
+
- DDL finishes quickly and its lock impact is easy to estimate. DML scales with row count and can
|
|
54
|
+
run long
|
|
55
|
+
- Mixing them means that if something fails partway through, you're left in the worst possible
|
|
56
|
+
state: the schema changed but the data is half-migrated
|
|
57
|
+
- Put data migrations in their own migration, or better, a separate batch job or management
|
|
58
|
+
command
|
|
59
|
+
|
|
60
|
+
### 1.5 One migration, one purpose
|
|
61
|
+
|
|
62
|
+
Each file should contain exactly one logical change.
|
|
63
|
+
Cramming "add a column to `users` + add an index to `orders` + drop an unused table" into one file
|
|
64
|
+
makes it hard to reason about state after a partial failure and hard to retry just the failed
|
|
65
|
+
piece. Keeping migrations small makes it easy to pinpoint what failed, retry it, and review it.
|
|
66
|
+
|
|
67
|
+
## 2. Dangerous Patterns in PostgreSQL / MySQL
|
|
68
|
+
|
|
69
|
+
### 2.1 Adding a NOT NULL column to a large table
|
|
70
|
+
|
|
71
|
+
```sql
|
|
72
|
+
-- Dangerous: on PostgreSQL 10 and earlier, or some MySQL configurations, this rewrites
|
|
73
|
+
-- every row and holds a long-lived lock
|
|
74
|
+
ALTER TABLE orders ADD COLUMN status varchar(20) NOT NULL DEFAULT 'pending';
|
|
75
|
+
|
|
76
|
+
-- Safe, staged migration (PostgreSQL)
|
|
77
|
+
ALTER TABLE orders ADD COLUMN status varchar(20); -- 1. Add nullable (instant)
|
|
78
|
+
-- 2. Deploy the app so it writes a value on new rows
|
|
79
|
+
-- 3. Backfill existing rows in batches (see 2.4)
|
|
80
|
+
ALTER TABLE orders ALTER COLUMN status SET DEFAULT 'pending'; -- 4. Set the default
|
|
81
|
+
ALTER TABLE orders ALTER COLUMN status SET NOT NULL; -- 5. Enforce NOT NULL last
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
On PostgreSQL 11+, `ADD COLUMN` with a constant default is a metadata-only change and is fast,
|
|
85
|
+
but `SET NOT NULL` still requires a full table scan. On PostgreSQL 12+, it's safer to first add
|
|
86
|
+
`CHECK (status IS NOT NULL) NOT VALID` and then run `VALIDATE CONSTRAINT`.
|
|
87
|
+
|
|
88
|
+
### 2.2 Locking caused by index creation
|
|
89
|
+
|
|
90
|
+
```sql
|
|
91
|
+
-- Dangerous: a plain CREATE INDEX blocks writes to the table
|
|
92
|
+
CREATE INDEX idx_orders_user_id ON orders (user_id);
|
|
93
|
+
|
|
94
|
+
-- Safe: use CONCURRENTLY on PostgreSQL (must run outside a transaction)
|
|
95
|
+
CREATE INDEX CONCURRENTLY idx_orders_user_id ON orders (user_id);
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
- `CONCURRENTLY` cannot run inside a transaction. In Django this means `atomic = False`; other
|
|
99
|
+
tools have their own way to disable the per-migration transaction
|
|
100
|
+
- If it fails, an INVALID index is left behind. Drop it with `DROP INDEX CONCURRENTLY` and retry
|
|
101
|
+
- MySQL (InnoDB) usually supports online DDL, but be explicit with
|
|
102
|
+
`ALTER TABLE ... ALGORITHM=INPLACE, LOCK=NONE` so you get an error (rather than a silent lock)
|
|
103
|
+
if it's not possible
|
|
104
|
+
- For very large tables on MySQL, consider gh-ost or pt-online-schema-change
|
|
105
|
+
|
|
106
|
+
### 2.3 Renaming a column or changing its type: do it as a staged migration
|
|
107
|
+
|
|
108
|
+
`RENAME COLUMN` itself is fast, but as soon as it runs, any old code still referencing the old
|
|
109
|
+
name breaks immediately. Treat a rename as "add a column under a new name."
|
|
110
|
+
|
|
111
|
+
```text
|
|
112
|
+
1. Add the new-named column (nullable)
|
|
113
|
+
2. Deploy the app so it writes to both columns, but still reads from the old one
|
|
114
|
+
3. Backfill the data by copying old -> new
|
|
115
|
+
4. Deploy the app so it reads from the new column (still writing to both)
|
|
116
|
+
5. Deploy the app so it writes only to the new column
|
|
117
|
+
6. Drop the old column (Contract)
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Type changes (e.g. int -> bigint) follow the same recipe by going through a new column of the
|
|
121
|
+
new type. On PostgreSQL, `ALTER TYPE` often rewrites every row under an ACCESS EXCLUSIVE lock.
|
|
122
|
+
The exception is compatible changes, like widening a varchar, which are metadata-only.
|
|
123
|
+
|
|
124
|
+
### 2.4 Batching backfills
|
|
125
|
+
|
|
126
|
+
A single bulk UPDATE against existing rows causes long-running transactions, replication lag,
|
|
127
|
+
lock contention, and (on PostgreSQL) VACUUM pressure. Always split it up.
|
|
128
|
+
|
|
129
|
+
```sql
|
|
130
|
+
-- Dangerous: a single bulk update (tens of millions of rows will blow up locks and WAL/binlog)
|
|
131
|
+
UPDATE orders SET status = 'pending' WHERE status IS NULL;
|
|
132
|
+
|
|
133
|
+
-- Safe: batch by primary key range
|
|
134
|
+
UPDATE orders SET status = 'pending'
|
|
135
|
+
WHERE id IN (
|
|
136
|
+
SELECT id FROM orders
|
|
137
|
+
WHERE status IS NULL
|
|
138
|
+
ORDER BY id
|
|
139
|
+
LIMIT 5000 -- tune the batch size based on observed load
|
|
140
|
+
);
|
|
141
|
+
-- Loop this from the application until zero rows match
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
- Commit each batch in its own transaction, and add a short pause between batches
|
|
145
|
+
- Write it idempotently (the WHERE clause should make it safe to re-run after a partial failure)
|
|
146
|
+
- Check a replica-lag metric inside the loop and pause if it exceeds a threshold
|
|
147
|
+
|
|
148
|
+
## 3. Tool-Specific Notes
|
|
149
|
+
|
|
150
|
+
### 3.1 Django migrations
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
# Always add this to CI: it catches model/migration drift (a missing makemigrations)
|
|
154
|
+
python manage.py makemigrations --check --dry-run
|
|
155
|
+
|
|
156
|
+
# Get in the habit of reviewing the SQL that will actually run before applying it
|
|
157
|
+
python manage.py sqlmigrate shop 0042
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
```python
|
|
161
|
+
from django.db import migrations
|
|
162
|
+
|
|
163
|
+
def forwards(apps, schema_editor):
|
|
164
|
+
# Always use apps.get_model. Importing the model directly will break this
|
|
165
|
+
# migration when the model changes in the future
|
|
166
|
+
Order = apps.get_model("shop", "Order")
|
|
167
|
+
while True:
|
|
168
|
+
ids = list(
|
|
169
|
+
Order.objects.filter(status__isnull=True).values_list("id", flat=True)[:5000]
|
|
170
|
+
)
|
|
171
|
+
if not ids:
|
|
172
|
+
break
|
|
173
|
+
Order.objects.filter(id__in=ids).update(status="pending")
|
|
174
|
+
|
|
175
|
+
class Migration(migrations.Migration):
|
|
176
|
+
atomic = False # Disable the per-migration transaction for bulk data work or
|
|
177
|
+
# CREATE INDEX CONCURRENTLY
|
|
178
|
+
dependencies = [("shop", "0041_add_status")]
|
|
179
|
+
operations = [
|
|
180
|
+
# Always specify reverse. Even if no reverse action is needed, make the noop
|
|
181
|
+
# explicit rather than leaving the migration irreversible
|
|
182
|
+
migrations.RunPython(forwards, migrations.RunPython.noop),
|
|
183
|
+
]
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
- Leaving `reverse` unspecified on `RunPython` makes that migration irreversible
|
|
187
|
+
- For adding indexes on PostgreSQL, use `AddIndexConcurrently`
|
|
188
|
+
(`django.contrib.postgres.operations`) together with `atomic = False`
|
|
189
|
+
- Squash migrations periodically to keep fresh-environment setup time and the dependency graph
|
|
190
|
+
manageable
|
|
191
|
+
|
|
192
|
+
### 3.2 Laravel migrations
|
|
193
|
+
|
|
194
|
+
```php
|
|
195
|
+
public function up(): void
|
|
196
|
+
{
|
|
197
|
+
Schema::table('orders', function (Blueprint $table) {
|
|
198
|
+
// Add as nullable; enforce NOT NULL in a separate migration
|
|
199
|
+
$table->string('status', 20)->nullable();
|
|
200
|
+
});
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
public function down(): void
|
|
204
|
+
{
|
|
205
|
+
Schema::table('orders', function (Blueprint $table) {
|
|
206
|
+
$table->dropColumn('status');
|
|
207
|
+
});
|
|
208
|
+
}
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
- Always write `down()` for local-dev convenience, but don't rely on it as your production
|
|
212
|
+
rollback mechanism. In production, handle rollback via expand/contract plus rolling back the
|
|
213
|
+
application, not `migrate:rollback`
|
|
214
|
+
- MySQL implicitly commits DDL statements, so packing multiple DDL statements into one file can
|
|
215
|
+
leave things half-applied after a mid-migration failure. Stick to one migration, one purpose
|
|
216
|
+
- `php artisan migrate --pretend` lets you preview the SQL before it runs
|
|
217
|
+
|
|
218
|
+
### 3.3 TypeORM / Prisma
|
|
219
|
+
|
|
220
|
+
- TypeORM: never use `synchronize: true` in production (it rewrites the schema implicitly).
|
|
221
|
+
Always eyeball the SQL that `migration:generate` produces to make sure it doesn't contain an
|
|
222
|
+
unintended DROP. Running migrations explicitly from the deploy pipeline via `migration:run` is
|
|
223
|
+
easier to control than auto-running them at app startup (`migrationsRun`)
|
|
224
|
+
- Prisma: use only `prisma migrate deploy` for production. `migrate dev` is for local development
|
|
225
|
+
only and can create/reset a shadow database. Don't use `prisma db push` in production either,
|
|
226
|
+
since it doesn't leave a migration history. The generated `migration.sql` file can be hand
|
|
227
|
+
edited, so make adjustments like switching `CREATE INDEX` to `CONCURRENTLY` directly in that
|
|
228
|
+
file
|
|
229
|
+
- For both tools, "auto-generated" does not mean "safe." A rename being generated as DROP + ADD,
|
|
230
|
+
losing data in the process, is a classic mistake with either tool
|
|
231
|
+
|
|
232
|
+
## 4. DynamoDB
|
|
233
|
+
|
|
234
|
+
Being schemaless doesn't mean migrations go away — it just means the schema has moved into the
|
|
235
|
+
application code, and the application is now responsible for compatibility.
|
|
236
|
+
|
|
237
|
+
### 4.1 Lazy migration (migrate on read)
|
|
238
|
+
|
|
239
|
+
Rewriting every item in bulk is expensive and risky, so convert items to the new shape lazily,
|
|
240
|
+
as they're read.
|
|
241
|
+
|
|
242
|
+
```typescript
|
|
243
|
+
// Give each item a version attribute and convert it incrementally on read
|
|
244
|
+
function migrateItem(item: Record<string, any>): Order {
|
|
245
|
+
const version = item.schemaVersion ?? 1;
|
|
246
|
+
if (version < 2) {
|
|
247
|
+
// v1 -> v2: split fullName into firstName / lastName
|
|
248
|
+
const [firstName, ...rest] = (item.fullName ?? '').split(' ');
|
|
249
|
+
item.firstName = firstName;
|
|
250
|
+
item.lastName = rest.join(' ');
|
|
251
|
+
item.schemaVersion = 2;
|
|
252
|
+
}
|
|
253
|
+
return item as Order;
|
|
254
|
+
}
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
- Always write items back in the latest shape, along with an updated `schemaVersion`
|
|
258
|
+
- Your read-path conversion function needs to keep supporting conversion from every past version.
|
|
259
|
+
If you want to drop support for old versions, run a one-time background scan-and-migrate job
|
|
260
|
+
first to bring everything current
|
|
261
|
+
- If you do run a bulk migration, use a segmented parallel Scan with rate limiting so you don't
|
|
262
|
+
eat into your write capacity
|
|
263
|
+
|
|
264
|
+
### 4.2 Operating GSI additions
|
|
265
|
+
|
|
266
|
+
- Adding a GSI can be done online, but the backfill takes time, and queries against the index
|
|
267
|
+
before it finishes will return incomplete results. Wait until the index reports ACTIVE before
|
|
268
|
+
cutting the application over to it
|
|
269
|
+
- In other words, a GSI addition needs the same two-phase deploy as an RDBMS expand/contract:
|
|
270
|
+
add -> wait for backfill -> cut over the code
|
|
271
|
+
- A GSI's key attributes can't be changed after creation. To change them, add a new GSI and drop
|
|
272
|
+
the old one once you've cut over
|
|
273
|
+
|
|
274
|
+
### 4.3 Key design changes are hard
|
|
275
|
+
|
|
276
|
+
- Partition keys and sort keys can't be changed once a table is created. Changing them really
|
|
277
|
+
means "create a new table, copy all the data, and cut over"
|
|
278
|
+
- In a CQRS setup, if you design things so the read model (query-side table) can be rebuilt from
|
|
279
|
+
the command-side events, then a key-design change on the query side reduces to a re-projection
|
|
280
|
+
- Where a migration truly is needed, set up dual writes via DynamoDB Streams, and cut reads over
|
|
281
|
+
to the new table once the copy is complete
|
|
282
|
+
|
|
283
|
+
## 5. Pre-Execution Safety Checklist
|
|
284
|
+
|
|
285
|
+
1. Will the previous version of the application still work after this schema change (backward
|
|
286
|
+
compatibility)?
|
|
287
|
+
2. If this includes a breaking change (DROP / RENAME / narrowing a type), has it been broken down
|
|
288
|
+
into expand/contract?
|
|
289
|
+
3. Have you checked the row count and size of the target table, and estimated the runtime against
|
|
290
|
+
a production-scale data volume?
|
|
291
|
+
4. Have you identified what kind of lock will be taken, and how lock waits will affect the
|
|
292
|
+
application?
|
|
293
|
+
5. Have you set `lock_timeout` / `statement_timeout` (or `lock_wait_timeout` on MySQL)?
|
|
294
|
+
6. Is the data migration batched and idempotent, with a documented recovery procedure for a
|
|
295
|
+
partial failure?
|
|
296
|
+
7. Have you documented the rollback procedure (one that doesn't depend on a down migration)?
|
|
297
|
+
8. Have you verified that a backup / snapshot / PITR is in place right before running it?
|
|
298
|
+
9. Have you rehearsed this against production-scale data in staging?
|
|
299
|
+
10. Have you picked a low-traffic execution window and lined up monitoring (replica lag, locks,
|
|
300
|
+
error rate)?
|
|
301
|
+
|
|
302
|
+
## 6. Anti-patterns
|
|
303
|
+
|
|
304
|
+
| Anti-pattern | What happens | Do this instead |
|
|
305
|
+
| --- | --- | --- |
|
|
306
|
+
| Releasing a column drop and the code change together | Old code crashes mid-rollout | Do the Contract step in a separate release, after the rollout is fully complete |
|
|
307
|
+
| Running `RENAME COLUMN` in one shot | Old code errors out immediately | Add a new column, dual-write, then cut over in stages |
|
|
308
|
+
| Bulk UPDATE against a huge table | Long transactions, locking, replica lag | Batch by primary-key range, with pauses |
|
|
309
|
+
| `CREATE INDEX CONCURRENTLY` inside a transaction | Errors out, or silently runs as a regular, locking `CREATE INDEX` | Disable the transaction and run it standalone |
|
|
310
|
+
| Mixing DDL and DML in one migration | A failure leaves things half-applied | Separate schema changes from data migrations |
|
|
311
|
+
| Treating a down migration as your production rollback plan | Data loss, or an unverified down that itself fails | Preserve forward compatibility and roll back via the application instead |
|
|
312
|
+
| Applying ORM-generated SQL without review | A rename becomes DROP + ADD and data is lost | Always eyeball the generated SQL |
|
|
313
|
+
| Using `synchronize: true` / `db push` in production | Implicit schema changes with no migration history | Change the schema only via migration files |
|
|
314
|
+
| Rewriting all DynamoDB items in one pass | Production impact from exhausted capacity | Use lazy migration, or a rate-limited bulk migration |
|
|
315
|
+
| Importing application models directly inside a migration | A future model change breaks past migrations | Use a historical model (e.g. Django's `apps.get_model`) |
|
|
316
|
+
|
|
317
|
+
## 7. Related Workflow
|
|
318
|
+
|
|
319
|
+
- Before starting a large schema change, use `/plan` to work out the breakdown into
|
|
320
|
+
expand/contract steps and the release ordering
|
|
321
|
+
- Send changes that include migrations to `/code-review`
|
|
322
|
+
- For Django-specific migration safety (transaction control, `RunPython` reverse functions,
|
|
323
|
+
lock-holding operations), have the `django-reviewer` subagent take a close look
|