start-vibing-stacks 2.6.0 → 2.7.3

package/stacks/_shared/skills/database-migrations/SKILL.md

@@ -0,0 +1,256 @@
+---
+name: database-migrations
+version: 1.0.0
+description: Safe database migrations under load — backfills, concurrent indexes, lock timeouts, parallel-change pattern, rollback strategy. Invoke before writing or running any schema migration.
+---
+
+# Database Migrations — Safety Patterns
+
+**ALWAYS invoke before writing or executing a schema migration on any environment with data.**
+
+> A migration that locks the table for 90 seconds at 1k writes/sec creates a 90,000-request outage. The only safe migration is one that ships in pieces.
+
+## The Cardinal Rule: Parallel Change
+
+For any non-trivial schema change, deploy in **at least two releases**:
+
+```
+Release N   → expand:  add new column/table, dual-write, leave old
+Release N+1 → migrate: backfill data
+Release N+2 → contract: drop old, switch reads
+```
+
+Never combine expand + contract in one deploy. You will regret it the moment you need to roll back.
+
+---
+
+## 1. Adding a Column
+
+### NOT NULL with default
+
+PostgreSQL ≥ 11 / MySQL 8 / MariaDB 10.5+: `ADD COLUMN ... NOT NULL DEFAULT ...` is fast (metadata-only). On older versions or other engines, it rewrites the whole table.
+
+**Safe pattern (works on any version):**
+```sql
+-- Step 1 (release N): nullable, no default → instant
+ALTER TABLE users ADD COLUMN plan VARCHAR(20);
+
+-- Step 2 (release N): app dual-writes — sets plan on every INSERT/UPDATE
+-- Step 3 (release N+1): backfill in chunks (see §3)
+-- Step 4 (release N+2): set NOT NULL + default
+ALTER TABLE users ALTER COLUMN plan SET NOT NULL;
+ALTER TABLE users ALTER COLUMN plan SET DEFAULT 'free';
+```
+
+### Generated columns
+Prefer over backfilling derived data when possible — DB keeps it consistent.
+
+---
+
+## 2. Adding an Index
+
+Indexes lock the table during build on most engines.
+
+### PostgreSQL
+```sql
+-- WRONG — exclusive lock for the duration
+CREATE INDEX idx_users_email ON users(email);
+
+-- CORRECT — non-blocking
+CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
+-- Caveats: cannot run inside a transaction; on failure leaves invalid index — drop with `DROP INDEX CONCURRENTLY`
+```
+
+### MySQL / MariaDB
+InnoDB ≥ 5.6 supports `ALGORITHM=INPLACE, LOCK=NONE` for most index types:
+```sql
+ALTER TABLE users ADD INDEX idx_email (email), ALGORITHM=INPLACE, LOCK=NONE;
+```
+
+If `LOCK=NONE` fails, the engine tells you why (e.g. fulltext, spatial). Don't override — the lock is real.
+
+### MongoDB
+```js
+db.users.createIndex({ email: 1 }, { background: true })   // 4.2 default; deprecated in 6+ where it's always non-blocking
+```
+
+### Tooling
+- **Postgres**: `pg_repack` for table-rewrite migrations without long locks
+- **MySQL**: `pt-online-schema-change` (Percona) or `gh-ost` (GitHub) — both work via shadow tables + triggers
+- **Mongo**: rolling builds across replica set members
+
+---
+
+## 3. Backfilling — Chunked
+
+Single `UPDATE` over millions of rows = long transaction = long lock = replication lag = page.
+
+### SQL (chunked, app-driven)
+```sql
+-- pseudo: loop in app code
+UPDATE users
+SET plan = COALESCE(plan, 'free')
+WHERE id BETWEEN :start AND :end
+  AND plan IS NULL;
+-- Sleep 50ms between batches; chunk size 1k–10k rows
+```
+
+### Laravel
+```php
+User::whereNull('plan')->chunkById(1000, function ($users) {
+    foreach ($users as $user) {
+        $user->update(['plan' => 'free']);
+    }
+    usleep(50_000);   // 50ms breathing room
+});
+```
+
+### Mongoose
+```ts
+const cursor = User.find({ plan: { $exists: false } }).cursor();
+for await (const user of cursor) {
+  await User.updateOne({ _id: user._id }, { $set: { plan: 'free' } });
+}
+```
+
+### Django
+Use `RunPython` migration with `atomic=False` and chunk:
+```python
+class Migration(migrations.Migration):
+    atomic = False
+    operations = [migrations.RunPython(backfill, reverse_code=migrations.RunPython.noop)]
+
+def backfill(apps, schema_editor):
+    User = apps.get_model("app", "User")
+    qs = User.objects.filter(plan__isnull=True).only("id")
+    for batch in chunked(qs.iterator(), 1000):
+        User.objects.filter(id__in=[u.id for u in batch]).update(plan="free")
+```
+
+---
+
+## 4. Lock Timeouts (Critical)
+
+Set a short timeout so a stuck migration fails fast instead of blocking the app:
+
+```sql
+-- Postgres
+SET LOCAL lock_timeout = '5s';
+SET LOCAL statement_timeout = '15s';
+ALTER TABLE users ADD COLUMN x INT;
+
+-- MySQL / MariaDB
+SET SESSION lock_wait_timeout = 5;
+SET SESSION innodb_lock_wait_timeout = 5;
+```
+
+Without this, an `ALTER` waiting for a long-running `SELECT` will block every subsequent write behind it (Postgres queue).
+
+---
+
+## 5. Renaming — Never Direct in Prod
+
+Never `ALTER TABLE users RENAME COLUMN email TO email_address` in a single deploy. The old code on the previous release will break.
+
+```
+Release N:   add new column, dual-write
+Release N+1: backfill, switch reads to new column
+Release N+2: drop old column
+```
+
+Same for table renames, type changes (often disguised renames + casts), enum value removal.
+
+---
+
+## 6. Foreign Keys
+
+Adding a FK validates every existing row. On large tables this locks.
+
+### Postgres
+```sql
+-- Two-step
+ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id) NOT VALID;  -- fast
+-- Validate when load is low; takes only SHARE UPDATE EXCLUSIVE lock
+ALTER TABLE orders VALIDATE CONSTRAINT fk_user;
+```
+
+### MySQL — no equivalent; use offline tooling (`gh-ost`).
+
+---
+
+## 7. Dropping — Always Soft First
+
+```
+Release N:   stop writing to column (deploy)
+Release N+1: stop reading from column (deploy)  ← observe metrics
+Release N+2: drop column
+```
+
+Add a feature flag for the new code path so you can roll back without a schema change.
+
+---
+
+## 8. Migration Tooling Per Stack
+
+| Stack | Tool | Notes |
+|---|---|---|
+| Node + SQL | Drizzle / Kysely / Knex / TypeORM | Drizzle is the modern choice; uses `meta` snapshots |
+| Node + Mongo | Mongoose schema versioning | No DDL — but evolve with backfills |
+| Python + SQL | Alembic (SQLAlchemy), Django migrations | |
+| PHP | Laravel migrations + `doctrine/dbal` for DDL | |
+
+Rules regardless of tool:
+- Migration files are **append-only** once merged. Edit = rewrite history = production drift.
+- Each migration has both `up` and `down` (or a documented "no rollback" reason).
+- Migrations run idempotently (safe to re-run).
+- Never include data migrations in schema migrations on huge tables — separate scripts.
+
+---
+
+## 9. Rollback Strategy
+
+For every migration ask: **"How do I undo this in 5 minutes if prod breaks?"**
+
+| Operation | Rollback |
+|---|---|
+| Add nullable column | `DROP COLUMN` (instant) |
+| Add NOT NULL column with default | `DROP COLUMN` |
+| Add index | `DROP INDEX [CONCURRENTLY]` |
+| Rename column | None — that's why parallel change |
+| Drop column | Restore from backup. **Don't.** |
+| Backfill | Run inverse update OR accept the data is now correct |
+
+If you can't write a 5-minute rollback, the migration is too risky for one release. Split it.
+
+---
+
+## 10. Pre-Deploy Checklist
+
+- [ ] Migration runs in < 5s OR is concurrent / non-blocking
+- [ ] Lock + statement timeouts set
+- [ ] Tested on a copy of prod-sized data (or a representative subset)
+- [ ] Backfills are chunked with breathing room
+- [ ] No combined expand + contract in same deploy
+- [ ] App code on previous release survives if migration runs first
+- [ ] App code on next release survives if migration is delayed
+- [ ] Rollback plan written (or "no rollback" justified)
+- [ ] DBA / on-call notified for high-risk migrations
+
+## FORBIDDEN
+
+| Pattern | Reason |
+|---|---|
+| `ALTER TABLE` without lock timeout in prod | Single bad query = full outage |
+| `CREATE INDEX` without `CONCURRENTLY` (Postgres) | Locks writes |
+| Renaming a column in one deploy | Breaks rolling deploys |
+| Single huge `UPDATE` | Replication lag, locks, redo bloat |
+| Editing a merged migration file | Prod drift, dev/prod mismatch |
+| Dropping a column without two prior soft-delete deploys | No rollback |
+| Migrations running automatically on every deploy without gates | Surprise outage |
+| Mixing schema and data migrations on hot tables | Single-tx behavior unpredictable |
+
+## See Also
+
+- `observability` — track migration duration and lag during runs
+- `error-handling` — service-side feature flags for parallel change
+- `ci-pipelines` — gate migrations behind manual approval for production

package/stacks/_shared/skills/debugging-patterns/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: debugging-patterns
+version: 1.0.0
+---
+
 # Debugging Patterns
 
 ## Universal Approach

package/stacks/_shared/skills/docker-patterns/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: docker-patterns
+version: 1.0.0
+---
+
 # Docker Patterns
 
 ## PHP (PHP-FPM + Nginx)

package/stacks/_shared/skills/docs-tracker/SKILL.md

@@ -1,3 +1,8 @@
+---
+name: docs-tracker
+version: 1.0.0
+---
+
 # Docs Tracker — Automatic Documentation System
 
 **ALWAYS invoke AFTER implementation completes.**

package/stacks/_shared/skills/error-handling/SKILL.md

@@ -0,0 +1,335 @@
+---
+name: error-handling
+version: 1.0.0
+description: Universal error-handling patterns. Error taxonomy, Result types, error boundaries, structured exceptions, never-swallow rules. Invoke when designing service layers, API responses, or any try/catch.
+---
+
+# Error Handling
+
+**ALWAYS invoke when adding try/catch, designing service layers, API error responses, or handling external API failures.**
+
+> The default state of software is broken. Errors are not exceptional — handling them is the job.
+
+## Core Principles
+
+1. **Errors are values.** Plan for them in the type signature, not as a side channel.
+2. **Throw at boundaries, return at edges.** Internal modules return Results; HTTP layer translates to status codes; user sees a generic message.
+3. **Never swallow.** A `catch` with no log, no rethrow, and no recovery is a bug.
+4. **Fail closed.** On unexpected error, deny — don't continue with degraded state.
+5. **Distinguish operational from programmer errors.** Operational = expected (network blip, validation). Programmer = bug. They get different treatment.
+
+---
+
+## Error Taxonomy (mandatory)
+
+Every error you create classifies into one of these:
+
+| Class | Examples | Strategy |
+|---|---|---|
+| **Validation** | Bad input, schema mismatch | Return 4xx with field details; do not log as error |
+| **Authentication** | Missing/expired/invalid token | 401, log at info |
+| **Authorization** | Authenticated but not allowed | 403, log at warn (could be probing) |
+| **NotFound** | Resource missing | 404, log at info |
+| **Conflict** | Optimistic lock, duplicate key | 409, log at warn |
+| **RateLimit** | Quota exceeded | 429 + `Retry-After`, log at info |
+| **External** | Upstream API failed | Retry+backoff if idempotent; 502/503; log at error |
+| **Programmer** | Null deref, type mismatch, switch missed | 500, log at error, alert, fix |
+
+If the error doesn't fit, you have a new class — add it deliberately.
+
+---
+
+## Pattern 1 — Custom Error Classes (TS / Python / PHP)
+
+### TypeScript
+```ts
+// lib/errors.ts
+export class AppError extends Error {
+  constructor(
+    public readonly code: string,
+    public readonly httpStatus: number,
+    message: string,
+    public readonly cause?: unknown,
+    public readonly meta?: Record<string, unknown>,
+  ) {
+    super(message);
+    this.name = this.constructor.name;
+  }
+}
+
+export class ValidationError extends AppError {
+  constructor(message: string, fields?: Record<string, string[]>) {
+    super('validation_error', 422, message, undefined, { fields });
+  }
+}
+export class NotFoundError extends AppError {
+  constructor(resource: string) { super('not_found', 404, `${resource} not found`); }
+}
+export class UnauthorizedError extends AppError {
+  constructor(message = 'Unauthorized') { super('unauthorized', 401, message); }
+}
+export class ForbiddenError extends AppError {
+  constructor(message = 'Forbidden') { super('forbidden', 403, message); }
+}
+export class ConflictError extends AppError {
+  constructor(message: string) { super('conflict', 409, message); }
+}
+export class ExternalError extends AppError {
+  constructor(service: string, cause: unknown) {
+    super('external_error', 502, `Upstream ${service} failed`, cause);
+  }
+}
+```
+
+### Python
+```python
+class AppError(Exception):
+    code: str = "error"
+    http_status: int = 500
+    def __init__(self, message: str, *, cause: Exception | None = None, meta: dict | None = None):
+        super().__init__(message)
+        self.cause, self.meta = cause, meta or {}
+
+class ValidationError(AppError):  code, http_status = "validation_error", 422
+class NotFoundError(AppError):    code, http_status = "not_found", 404
+class UnauthorizedError(AppError):code, http_status = "unauthorized", 401
+class ForbiddenError(AppError):   code, http_status = "forbidden", 403
+class ConflictError(AppError):    code, http_status = "conflict", 409
+class ExternalError(AppError):    code, http_status = "external_error", 502
+```
+
+### PHP
+Use Laravel's `HttpException` family + custom domain exceptions. Override `Handler::register` to map domain exceptions to HTTP responses.
+
+---
+
+## Pattern 2 — Result Types (for internal layers)
+
+Throwing across many layers is expensive (TypeScript) and opaque (it's not in the signature). For internal service code, use a Result.
+
+### TypeScript
+```ts
+type Result<T, E = AppError> =
+  | { ok: true;  data: T }
+  | { ok: false; error: E };
+
+const ok = <T>(data: T): Result<T> => ({ ok: true, data });
+const err = <E extends AppError>(error: E): Result<never, E> => ({ ok: false, error });
+
+// Service returns Result; caller handles both branches
+async function findUser(id: string): Promise<Result<User>> {
+  const user = await db.user.findById(id);
+  if (!user) return err(new NotFoundError('user'));
+  return ok(user);
+}
+
+// Caller (route handler) — throws to be caught by error middleware
+const result = await findUser(id);
+if (!result.ok) throw result.error;
+return result.data;
+```
+
+### Python
+```python
+from dataclasses import dataclass
+from typing import Generic, TypeVar
+T = TypeVar("T"); E = TypeVar("E")
+
+@dataclass(frozen=True)
+class Ok(Generic[T]):  data: T
+@dataclass(frozen=True)
+class Err(Generic[E]): error: E
+
+Result = Ok[T] | Err[E]
+```
+
+Or use `returns` library: `from returns.result import Result, Success, Failure`.
+
+---
+
+## Pattern 3 — Centralized HTTP Error Handler
+
+### Express
+```ts
+// errors of type AppError → mapped; everything else → 500
+app.use((err: unknown, req: Request, res: Response, _next: NextFunction) => {
+  const requestId = res.getHeader('x-request-id');
+  if (err instanceof AppError) {
+    req.log.warn({ err, code: err.code, status: err.httpStatus }, err.message);
+    return res.status(err.httpStatus).json({
+      error: { code: err.code, message: err.message, requestId, ...err.meta },
+    });
+  }
+  // Unknown — programmer error, never leak details
+  req.log.error({ err }, 'Unhandled exception');
+  res.status(500).json({ error: { code: 'internal_error', message: 'Internal Server Error', requestId } });
+});
+```
+
+### Next.js — App Router
+Use `error.tsx` + `global-error.tsx` for client-rendered errors, and a wrapper for Route Handlers:
+
+```ts
+// app/api/_handler.ts
+export function safeHandler<T extends Request>(fn: (req: T) => Promise<Response>) {
+  return async (req: T): Promise<Response> => {
+    try { return await fn(req); }
+    catch (e) {
+      if (e instanceof AppError) {
+        return Response.json({ error: { code: e.code, message: e.message, ...e.meta } }, { status: e.httpStatus });
+      }
+      logger.error({ err: e }, 'Unhandled');
+      return Response.json({ error: { code: 'internal_error', message: 'Internal Server Error' } }, { status: 500 });
+    }
+  };
+}
+```
+
+### FastAPI
+```python
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse
+
+@app.exception_handler(AppError)
+async def app_error_handler(req: Request, exc: AppError):
+    return JSONResponse(
+        {"error": {"code": exc.code, "message": str(exc), **exc.meta}},
+        status_code=exc.http_status,
+    )
+
+@app.exception_handler(Exception)
+async def unhandled_handler(req: Request, exc: Exception):
+    log.exception("unhandled", request_id=req.headers.get("x-request-id"))
+    return JSONResponse(
+        {"error": {"code": "internal_error", "message": "Internal Server Error"}},
+        status_code=500,
+    )
+```
+
+---
+
+## Pattern 4 — Retry with Backoff (External Calls)
+
+```ts
+// Idempotent operations only. Never retry payments without idempotency keys.
+async function withRetry<T>(
+  fn: () => Promise<T>,
+  { tries = 3, baseMs = 200, factor = 2, jitter = 0.3 } = {},
+): Promise<T> {
+  let lastErr: unknown;
+  for (let i = 0; i < tries; i++) {
+    try { return await fn(); }
+    catch (e) {
+      lastErr = e;
+      if (i === tries - 1) break;
+      const delay = baseMs * Math.pow(factor, i) * (1 + (Math.random() - 0.5) * 2 * jitter);
+      await new Promise(r => setTimeout(r, delay));
+    }
+  }
+  throw new ExternalError('upstream', lastErr);
+}
+```
+
+For HTTP retries, use `axios-retry`, `got` (built-in), or `tenacity` (Python). Don't roll your own unless you must.
+
+**Idempotency keys** for non-idempotent operations (POST that creates / charges):
+```ts
+await stripe.charges.create({ amount, customer }, { idempotencyKey: orderId });
+```
+
+---
+
+## Pattern 5 — Circuit Breaker (External Outages)
+
+When upstream is down, fail fast instead of piling up requests:
+
+```ts
+import CircuitBreaker from 'opossum';
+const breaker = new CircuitBreaker(callUpstream, {
+  timeout: 3000,
+  errorThresholdPercentage: 50,
+  resetTimeout: 30000,
+});
+breaker.fallback(() => ({ data: cached, stale: true }));
+```
+
+Python: `pybreaker`. PHP: `ackintosh/ganesha`.
+
+---
+
+## Pattern 6 — Async / Promise Hygiene
+
+```ts
+// WRONG — unhandled rejection crashes the process (Node 15+)
+somePromise();
+
+// CORRECT — always handle, even if just to log
+somePromise().catch(err => logger.error({ err }, 'background task failed'));
+
+// Promise.all rejects on first failure — for partial success use allSettled
+const results = await Promise.allSettled([fetchA(), fetchB(), fetchC()]);
+const failures = results.filter(r => r.status === 'rejected');
+if (failures.length > 0) logger.warn({ failures }, 'partial failure');
+```
+
+Catch handler at process level (defense in depth):
+```ts
+process.on('unhandledRejection', (reason) => {
+  logger.fatal({ reason }, 'Unhandled rejection');
+  process.exit(1);
+});
+process.on('uncaughtException', (err) => {
+  logger.fatal({ err }, 'Uncaught exception');
+  process.exit(1);
+});
+```
+
+---
+
+## Anti-Patterns
+
+| Anti-pattern | Why it's bad | Fix |
+|---|---|---|
+| `try { ... } catch {}` | Silent failure | Log + rethrow or recover deliberately |
+| `catch (e) { console.log(e) }` | No structured context | Use logger; include `request_id`, `user_id` |
+| `catch (e) { throw new Error(e.message) }` | Loses stack trace | `throw new AppError(..., e)` (preserve cause) |
+| `if (err) return null` | Caller can't distinguish "no result" from "failed" | Result type |
+| Returning `{ error: 'something' }` from a function | Convention drift, untyped | Result type or throw |
+| Catching `Error` to coerce | Hides bugs | Catch specific classes |
+| Generic 500 response with raw stack | Info disclosure | Log full, return generic message + requestId |
+| Retrying non-idempotent calls | Double-charge / double-create | Idempotency keys |
+| Retrying without backoff | DoS your own upstream | Exponential + jitter |
+
+## API Error Response Shape (Standardize)
+
+```json
+{
+  "error": {
+    "code": "validation_error",
+    "message": "Invalid input",
+    "fields": {
+      "email": ["must be a valid email"],
+      "age": ["must be at least 13"]
+    },
+    "requestId": "req_abc123"
+  }
+}
+```
+
+Stable contract: client code can switch on `code`. Always include `requestId` so support can correlate with logs.
+
+## Pre-Commit Checklist
+
+- [ ] No empty `catch` blocks
+- [ ] Every `catch` either logs+rethrows, recovers deliberately, or transforms to `AppError`
+- [ ] Service-layer functions that can fail return `Result` or throw `AppError`
+- [ ] HTTP layer has a single error mapper; routes do not stringify exceptions
+- [ ] External calls have retry + backoff (if idempotent) or idempotency keys
+- [ ] No raw stack traces in API responses
+- [ ] Process-level `unhandledRejection` / `uncaughtException` handlers exist (Node)
+
+## See Also
+
+- `observability` — what to log on errors
+- `security-baseline` — A09 logging, A05 config (don't leak stack traces)
+- Stack `api-security-*` — error mapping in framework handlers