@zigrivers/scaffold 3.6.0 → 3.8.0

package/content/knowledge/backend/backend-requirements.md

@@ -0,0 +1,103 @@
+---
+name: backend-requirements
+description: API-first design principles, SLA requirements (latency p99, uptime, throughput), scalability targets, backwards compatibility commitments, and API versioning strategy
+topics: [backend, requirements, sla, api, versioning, scalability, performance]
+---
+
+Backend requirements are the contract the service makes with its consumers — other teams, external developers, and end users. Setting explicit SLAs, versioning policies, and scalability targets before any code is written eliminates the most expensive class of late-breaking architectural changes. A backend that surprises its callers with latency spikes or breaking changes destroys trust and creates cascading toil.
+
+## Summary
+
+Backend requirements establish the contract the service makes with its consumers. Write the API contract (OpenAPI, GraphQL schema, protobuf) before implementation. Set explicit SLA targets for latency (p99), availability, throughput, and error budgets before sprint one. Define backwards compatibility commitments and API versioning strategy upfront.
+
+## Deep Guidance
+
+### API-First Design
+
+Write the API contract — OpenAPI spec, GraphQL schema, or protobuf definition — before writing implementation code. This forces explicit thinking about the consumer's perspective and produces a concrete artifact reviewable by all stakeholders before a single endpoint exists.
+
+- **Design the interface, not the implementation**: Endpoints should model domain operations, not database rows. Prefer `POST /orders/{id}/cancel` over `PATCH /orders/{id}` with a `status: "cancelled"` payload.
+- **Publish specs early**: Generate mock servers from OpenAPI specs so frontend teams can integrate immediately. Tools: Prism, Mockoon, Stoplight.
+- **Review the contract**: Treat spec changes like code changes — pull requests, approval gates, and a changelog for breaking vs non-breaking modifications.
+
+### SLA Requirements
+
+Establish these as project requirements before sprint one:
+
+- **Latency p99**: Typical web API targets: p99 < 500 ms for user-facing reads; p99 < 1 s for writes. Internal service calls: p99 < 100 ms. Set per-endpoint budgets for expensive operations (search, report generation).
+- **Availability / uptime**: 99.9% (three nines) = 8.7 hours downtime/year. 99.95% = 4.4 hours. Each extra nine requires substantially more operational investment. Match the target to business impact.
+- **Throughput**: Define peak requests per second at both steady state and spike conditions. A service handling 100 req/s steady must survive a 5× traffic spike during a product launch.
+- **Error budget**: Define what percentage of requests may fail without triggering an incident. A 0.1% error rate budget at 10,000 req/s = 10 errors/s. Track this in the SLA dashboard.
+
+### Scalability Targets
+
+State growth expectations explicitly:
+
+- **User growth**: Design for 10× current volume with known architectural changes. Design for 100× only if the business case is concrete — over-engineering for hypothetical scale is expensive.
+- **Data volume**: Specify retention policy and expected row counts at 1 year and 3 years. A billion-row table needs a different indexing strategy than a million-row table.
+- **Horizontal scaling**: Stateless service design is a requirement if horizontal scaling is expected. Document any stateful dependencies (sessions, local caches) that prevent it.
+
+### Backwards Compatibility Commitments
+
+Define the breaking-change policy upfront:
+
+- **Non-breaking changes** (can ship anytime): Adding optional fields to responses, adding optional request parameters, adding new endpoints, expanding enum values in non-exhaustive enums.
+- **Breaking changes** (require versioning or migration period): Removing or renaming fields, changing field types, requiring previously optional fields, changing error response shape.
+- **Deprecation window**: Commit to how long deprecated endpoints remain available — typically 6–12 months for external APIs, 4–8 weeks for internal services.
+
+### API Versioning Strategy
+
+Choose one strategy per API surface and document it:
+
+- **URL path versioning** (`/v1/`, `/v2/`): Most visible, easy to route and document. Preferred for public REST APIs.
+- **Accept/Content-Type header**: `Accept: application/vnd.api.v2+json`. Cleaner URLs, harder to test manually.
+- **Query parameter**: `?version=2`. Easiest to add but pollutes every request.
+- **No versioning (schema evolution)**: Only viable with strict non-breaking-change discipline and GraphQL or Protobuf with field deprecation support.
+
+### Encoding SLAs as Tests
+
+SLA commitments only have teeth if they are automatically verified:
+
+- Add p99 latency assertions to load tests using k6, Gatling, or Locust. Fail the pipeline if p99 exceeds the budget under synthetic load.
+- Instrument production with percentile metrics (not just averages) via Prometheus or Datadog. Average latency hides tail-latency problems that affect the top 1% of users — which at scale is thousands of people.
+- Implement synthetic monitoring: fire real API calls from external locations every 60 seconds and alert on latency or error rate breaches.
+
+### Capacity Planning Requirements
+
+Document capacity expectations before implementation to avoid architectural surprises:
+
+- **Concurrent users**: Define peak concurrent users for the first year and projected growth. A service designed for 100 concurrent users needs fundamentally different connection pooling, caching, and infrastructure than one designed for 100,000.
+- **Data retention**: Define how long each data type is stored. Unbounded retention eventually degrades query performance and storage costs. Set explicit TTLs for logs, events, sessions, and temporary data.
+- **Burst traffic**: Identify expected traffic spikes (marketing campaigns, seasonal events, time-zone-aligned usage). Design rate limiting and auto-scaling to handle 5-10x sustained traffic without degradation.
+- **Geographic distribution**: Define where users are located. A single-region deployment with users on the opposite side of the world will never meet a 200ms p99 latency target regardless of optimization.
+
+### Rate Limiting Requirements
+
+Define rate limits as requirements, not afterthoughts:
+
+- **Public API tiers**: Free tier (100 req/min), standard (1,000 req/min), enterprise (custom). Document limits in API documentation and enforce via API key scoping.
+- **Internal service limits**: Even internal services need rate limits to prevent cascading failures. A misbehaving upstream service should not be able to overwhelm a downstream service.
+- **Auth endpoint limits**: Login, registration, and password reset endpoints need aggressive rate limiting (5-10 attempts per minute per IP) to prevent credential stuffing and brute-force attacks.
+
+### Documentation Requirements
+
+Backend requirements must include documentation standards:
+
+- **OpenAPI / AsyncAPI spec**: Every endpoint documented in a machine-readable spec. Generate client SDKs from the spec. Block PRs that change behavior without updating the spec.
+- **Runbooks**: Every service has a runbook documenting: how to restart it, how to roll back, how to check health, what the common failure modes are, and who owns it.
+- **Architecture Decision Records**: Document every significant technical decision (database choice, caching strategy, auth approach) with context, alternatives considered, and rationale.
+
+### Compliance and Regulatory Requirements
+
+Identify applicable regulations before designing the system:
+
+- **Data residency**: Some jurisdictions require data to be stored within geographic boundaries (GDPR for EU data, data sovereignty laws). This affects database region selection and CDN configuration.
+- **PII handling**: Define which fields are personally identifiable information. PII requires encryption at rest, access logging, right-to-deletion support, and data minimization (collect only what is needed).
+- **Audit logging**: Financial services, healthcare, and government applications require immutable audit logs of all data access and modifications. Design the audit log schema as a first-class requirement, not a bolt-on.
+- **Accessibility**: Backend APIs serving web frontends must support the data structures needed for accessible UIs — alternative text for images, structured content for screen readers, and proper error message formatting.
+
+Identify compliance requirements during project inception, not after launch. Retrofitting data residency or audit logging into an existing system is an order of magnitude more expensive than designing for it from the start. Engage legal and compliance stakeholders during the requirements phase to surface these constraints early.
+
+### Observability Requirements
+
+Define observability requirements alongside functional requirements. Every service should have: structured logging with correlation IDs, RED metrics (Rate, Errors, Duration) for all endpoints, SLO targets documented and enforced via alerting, and distributed tracing via OpenTelemetry. These are not nice-to-haves — they are the tools that make production debugging possible.

package/content/knowledge/backend/backend-security.md

@@ -0,0 +1,104 @@
+---
+name: backend-security
+description: Input validation, SQL injection prevention, rate limiting, OWASP API Security Top 10, secrets management, and dependency auditing
+topics: [backend, security, validation, sql-injection, rate-limiting, owasp, secrets]
+---
+
+Security vulnerabilities in backend services are disproportionately expensive to fix after launch — building in input validation, injection prevention, rate limiting, and secrets hygiene from the start is always cheaper than retrofitting them under pressure after a breach.
+
+## Summary
+
+Backend security starts with input validation at every trust boundary using schema libraries, parameterized queries for all database access, and rate limiting at multiple layers (IP, user, endpoint). The OWASP API Security Top 10 identifies the highest-impact API risks including broken object-level authorization, authentication weaknesses, and unrestricted resource consumption.
+
+Secrets management, dependency auditing, and pre-commit secret scanning are operational requirements, not optional hardening steps.
+
+## Deep Guidance
+
+### Input Validation
+
+Validate all input at every trust boundary using a schema library — Zod (TypeScript), Joi (Node.js), Pydantic (Python), or similar. Never trust data from clients, webhooks, or upstream services without validation.
+
+**Validation layers:**
+- **Type and shape:** Reject requests that don't match the expected schema before any business logic runs.
+- **Range and format:** Validate string lengths, numeric ranges, enum membership, date formats, regex patterns.
+- **Business constraints:** Validate cross-field invariants and domain rules (e.g., `endDate > startDate`).
+
+Parse, don't sanitize. Return explicit error messages that identify which field failed and why — this helps legitimate callers but doesn't expose internals. Strip unknown fields (`stripUnknown: true` in Joi, `.strict()` in Zod) to prevent mass-assignment vulnerabilities.
+
+### SQL Injection Prevention
+
+Use parameterized queries for all database access — no exceptions.
+
+```typescript
+// BAD: string interpolation is vulnerable
+db.query(`SELECT * FROM users WHERE email = '${email}'`);
+
+// GOOD: parameterized
+db.query('SELECT * FROM users WHERE email = $1', [email]);
+
+// GOOD: ORM (Prisma, Drizzle) parameterizes automatically
+db.users.findFirst({ where: { email } });
+```
+
+Never build dynamic SQL from user input. If dynamic column names or table names are required, validate them against a strict allowlist before interpolation. Apply the same rules to NoSQL queries — validate that operator keys like `$where`, `$gt`, and `$ne` cannot be injected by user-controlled input.
+
+### Rate Limiting
+
+**Token bucket:** Smooth bursty traffic. Each caller has a bucket that refills at a fixed rate. Supports short bursts while enforcing a sustained rate. Appropriate for general API endpoints.
+
+**Sliding window:** Count requests in a rolling time window (e.g., 100 requests per 60 seconds). More precise than fixed-window, no edge-case spikes at window boundaries. Use Redis sorted sets or a purpose-built library (rate-limiter-flexible, upstash/ratelimit).
+
+**Limits by layer:**
+- IP-level limits to prevent anonymous abuse.
+- User/API-key level limits for authenticated callers.
+- Endpoint-specific limits for expensive operations (search, export, auth).
+
+Return `429 Too Many Requests` with a `Retry-After` header and `X-RateLimit-Limit` / `X-RateLimit-Remaining` / `X-RateLimit-Reset` headers. Log rate-limit hits for abuse monitoring.
+
+### OWASP API Security Top 10
+
+The OWASP API Security project identifies the most critical API-specific risks. The top concerns for backend APIs:
+
+- **API1 — Broken Object Level Authorization:** Verify ownership on every resource access. An authenticated user must not be able to access another user's records by changing an ID in the URL.
+- **API2 — Broken Authentication:** Use short-lived tokens, enforce MFA for sensitive operations, rate-limit auth endpoints.
+- **API3 — Broken Object Property Level Authorization:** On PATCH/PUT, validate that the caller is allowed to modify each field. Reject attempts to write admin-only fields.
+- **API4 — Unrestricted Resource Consumption:** Enforce pagination limits, maximum query sizes, file upload limits, and request body size limits.
+- **API5 — Broken Function Level Authorization:** Admin and internal endpoints must require role checks, not just authentication.
+- **API8 — Security Misconfiguration:** Disable debug endpoints in production, apply security headers, restrict CORS to known origins.
+
+### Secrets Management
+
+Never hardcode secrets in source code. Store secrets in environment variables for simple deployments, and a vault system (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) for production.
+
+**Rules:**
+- Add `.env` to `.gitignore`. Commit `.env.example` with placeholder values.
+- Run secret-scanning in CI (gitleaks, truffleHog, GitHub secret scanning).
+- Install pre-commit hooks (git-secrets, detect-secrets) to block accidental commits.
+- If a secret is committed, rotate it immediately — git history removal is not sufficient because the repo may have been cloned.
+- Never log secrets. Implement logging middleware that redacts known-sensitive field names.
+
+### Dependency Auditing
+
+Run `npm audit --audit-level=high` (or equivalent) on every CI build. Block merges on critical and high vulnerabilities. Keep a policy: critical fixes same day, high within 24 hours, medium within a sprint.
+
+Use lockfiles (`package-lock.json`, `poetry.lock`) and `npm ci` (not `npm install`) to verify checksums. Pin indirect dependencies that have a history of supply-chain incidents. Subscribe to GitHub security advisories for critical dependencies. Periodically run `npm outdated` and schedule dedicated update sprints rather than letting packages fall behind by major versions.
+
+### Security Headers
+
+Set security headers on all HTTP responses. Configure these at the reverse proxy or middleware level so they apply to every endpoint:
+
+- `Strict-Transport-Security: max-age=31536000; includeSubDomains` — enforce HTTPS
+- `X-Content-Type-Options: nosniff` — prevent MIME-type sniffing
+- `X-Frame-Options: DENY` — prevent clickjacking
+- `Content-Security-Policy` — restrict which resources can be loaded, preventing XSS
+- `Referrer-Policy: strict-origin-when-cross-origin` — control referrer information leakage
+
+Use `helmet` (Node.js/Express) or equivalent middleware to set these headers with sensible defaults. Validate headers with `securityheaders.com` after deployment.
+
+### CORS Configuration
+
+Cross-Origin Resource Sharing must be configured explicitly on backend APIs consumed by browser clients:
+
+- **Origin allowlist**: List specific allowed origins (`https://app.example.com`). Never use `*` in production with credentials. A wildcard origin with `Access-Control-Allow-Credentials: true` is a security vulnerability.
+- **Preflight caching**: Set `Access-Control-Max-Age` to cache preflight OPTIONS responses (3600 seconds is a reasonable default). This reduces the number of preflight requests for repeated API calls.
+- **Expose only needed headers**: Use `Access-Control-Expose-Headers` to explicitly list response headers the browser may access. Do not expose all headers — limit to those the frontend needs (pagination cursors, rate limit headers).

package/content/knowledge/backend/backend-testing.md

@@ -0,0 +1,101 @@
+---
+name: backend-testing
+description: API integration tests, contract testing, database testing patterns, mocking external services, and load testing
+topics: [backend, testing, integration-tests, contract-testing, database-testing, mocking, load-testing]
+---
+
+Backend testing strategy determines how much confidence you have before every deploy — a well-layered test suite catches regressions at the fastest possible feedback loop while still exercising the real data layer and honoring API contracts with consumers.
+
+## Summary
+
+Backend testing strategy layers three concerns: API integration tests that verify the full request-response cycle against a real database, contract tests that ensure API boundary compatibility across services, and database tests using transaction rollback for isolation. Mock only external boundaries you do not own; use real instances for internal services and databases.
+
+Load testing with explicit p99 latency thresholds must run before launching any high-traffic endpoint.
+
+## Deep Guidance
+
+### API Integration Tests
+
+Integration tests verify the full request-response cycle through the application stack, including middleware, routing, validation, business logic, and the database. Unlike unit tests, they catch wiring problems that unit tests miss.
+
+**With supertest (Node.js):** Mount the Express/Fastify app without starting a server. Supertest handles the HTTP layer in-process — fast, no port conflicts.
+
+```typescript
+const app = createApp(); // factory, no app.listen()
+it('POST /orders returns 201 with created order', async () => {
+  const res = await request(app)
+    .post('/orders')
+    .set('Authorization', `Bearer ${token}`)
+    .send({ productId: 'prod_1', quantity: 2 });
+  expect(res.status).toBe(201);
+  expect(res.body.data.id).toBeDefined();
+});
+```
+
+Run integration tests against a real database (see Database Testing below), not mocks. The value of integration tests comes from exercising the real data layer.
+
+### Contract Testing
+
+Contract tests verify that a service honors the API contract its consumers depend on, without requiring consumers and providers to be deployed together.
+
+**Consumer-driven contracts (Pact):** Consumers define their expectations (request shape + response shape) as a contract file. The provider verifies the contract runs against their implementation. Breaks are caught in CI before deployment, not in staging.
+
+**Schema validation:** For APIs consumed by external parties, use JSON Schema or OpenAPI schema validation in tests. Assert that every response matches the documented schema. This catches unintentional breaking changes (removed fields, type changes) before they reach consumers.
+
+**Use contract testing when:** You own both sides of an API boundary, or you maintain a public API with known consumers. Skip it for internal services where integration tests cover the boundary adequately.
+
+### Database Testing
+
+**Use transactions for isolation:** Wrap each test in a database transaction and roll it back after the test. No cleanup code needed; each test starts from a known state.
+
+```typescript
+beforeEach(async () => { await db.transaction.begin(); });
+afterEach(async () => { await db.transaction.rollback(); });
+```
+
+**Test fixtures:** Use factory functions (not fixture files) to create test data. Factories are composable, easier to maintain, and produce minimal records with explicit variation. Libraries: `fishery`, `@anatine/zod-mock`, or plain functions.
+
+**Test database:** Use a separate database for tests (configured via `DATABASE_URL` environment variable in CI). Run migrations before the test suite. Never run tests against the production database.
+
+**Seed data:** Limit seed data to what is required by the test. Broad seed data creates invisible dependencies between tests — a test that only creates one record but relies on seeded data is fragile and hard to understand.
+
+### Mocking External Services
+
+**MSW (Mock Service Worker):** Intercepts HTTP requests at the network layer using a service worker (browser) or Node.js interceptor. Unlike mocking `fetch` or axios, MSW mocks at the protocol level — the same mock works for any HTTP client. Define handlers that return realistic responses; test error cases by returning 5xx responses or network errors.
+
+**nock (Node.js):** Intercepts Node.js `http`/`https` modules. Appropriate for testing code that uses native HTTP or libraries that don't support service workers. Supports response delays, request matching by headers and body, and assertion that expected requests were made.
+
+**Principles:**
+- Mock only external boundaries — services you don't own (Stripe, Twilio, third-party APIs).
+- Don't mock your own database or internal services; use real instances in integration tests.
+- Keep mock responses realistic — use actual API response shapes, not minimal stubs.
+- Test the unhappy path: timeout, 429, 500, malformed response.
+
+### Load Testing
+
+**k6:** JavaScript-based load testing tool with a clean API. Define scenarios with virtual users and ramp profiles. Check assertions (thresholds) on p95 latency and error rate inline with the test script. Outputs structured results; integrates with Grafana for visualization.
+
+**Artillery:** YAML/JSON configuration-driven load testing with a plugin ecosystem. Better for teams that prefer declarative configuration over scripting. Supports WebSocket and gRPC in addition to HTTP.
+
+**When to run:** Load test before launching any endpoint that will receive significant concurrent traffic, before scaling down infrastructure, and after performance-sensitive refactors. Run load tests in a staging environment with production-representative data volume. Define pass/fail thresholds (p99 < 500ms, error rate < 0.1%) and fail CI when thresholds are breached.
+
+### Test Data Management
+
+Production-like test data is essential for meaningful integration and load tests:
+
+- **Factory functions over fixtures**: Factories programmatically generate test records with sensible defaults and explicit overrides. They compose — `createOrderWithItems(3)` creates an order, 3 items, and the necessary customer record. Fixture files are static and rot.
+- **Database seeding for integration tests**: Use a dedicated seed script that creates the minimum dataset for the test suite. Run it once before the suite, not before each test.
+- **Data anonymization**: For staging environments that need production-like volume, use anonymized production data. Tools like `postgresql_anonymizer` or custom scripts replace PII (names, emails, addresses) with realistic fakes while preserving data distribution and relationships.
+
+### Flaky Test Prevention
+
+Flaky tests undermine test suite credibility. Common causes and fixes:
+
+- **Time-dependent tests**: Mock `Date.now()` or use a clock injection. Never assert on real-time values.
+- **Order-dependent tests**: Each test must set up its own state and clean up after itself. Use transaction rollback or database truncation between tests.
+- **Network-dependent tests**: Mock all external HTTP calls in unit/integration tests. Reserve real network calls for E2E tests on a schedule.
+- **Race conditions**: Use `waitFor` patterns instead of fixed `setTimeout` delays. In database tests, ensure writes are committed before reads.
+
+Quarantine known flaky tests in a separate CI job that does not block merges. Fix quarantined tests within one sprint — a test that stays quarantined indefinitely should be deleted.
+
+Track flaky test frequency with test analytics tools (Datadog CI Visibility, BuildPulse, or a simple CSV log). A test that fails intermittently 5% of the time will block CI multiple times per week on an active team.

package/content/knowledge/backend/backend-worker-patterns.md

@@ -0,0 +1,100 @@
+---
+name: backend-worker-patterns
+description: Background job frameworks, cron scheduling, event consumers, dead letter queues, retry strategies, and graceful shutdown for workers
+topics: [backend, workers, bullmq, celery, temporal, cron, background-jobs, dlq]
+---
+
+Background workers offload time-consuming and deferred work from the request path, but they introduce their own failure modes — jobs that silently vanish, duplicate executions, and unclean shutdowns during deploys all require deliberate design to prevent.
+
+## Summary
+
+Background workers offload time-consuming work from the request path. Choose BullMQ (Node.js), Celery (Python), or Temporal (durable workflows) based on the complexity of job orchestration. Define cron schedules in one place with overlap prevention. Design all event consumers to be idempotent since most message systems guarantee at-least-once delivery.
+
+DLQ monitoring, worker health heartbeats, and graceful SIGTERM handling are operational requirements for any production worker system.
+
+## Deep Guidance
+
+### Background Job Frameworks
+
+**BullMQ (Node.js):** Redis-backed job queue with strong TypeScript support, priority queues, delayed jobs, repeatable jobs, flow producers (job hierarchies), and a rich event API. Workers are long-running processes that pull jobs from queues. Multiple workers for the same queue compete for jobs (work queue pattern). Suitable for most Node.js use cases from simple email sending to complex multi-step workflows.
+
+**Celery (Python):** The de facto Python task queue. Supports multiple brokers (Redis, RabbitMQ, SQS) and result backends. Integrates with Django and Flask. Use `@shared_task` with `bind=True` for self-aware tasks that can access retry state. Canvas primitives (`chain`, `group`, `chord`) compose complex workflows.
+
+**Temporal:** Workflow-as-code platform for long-running, durable workflows. Workflows are ordinary functions that survive crashes and restarts because Temporal replays the event history. Use Temporal when: workflows span hours or days, involve human approvals, require complex compensation logic, or must survive process restarts mid-execution. Steeper learning curve but eliminates entire classes of distributed systems bugs.
+
+### Cron Scheduling
+
+Define cron schedules in one place — application code or infrastructure (Kubernetes CronJob, AWS EventBridge Scheduler), not both. Infrastructure-level crons are more reliable (no process needs to be running continuously to trigger them) but harder to test. Application-level crons (BullMQ repeatable jobs, Celery beat) are easier to test and version alongside the code.
+
+**Overlap prevention:** Long-running jobs can overlap if the next run starts before the previous finishes. Prevent overlap by: acquiring a distributed lock before executing (Redis `SET NX EX`), using a queue with `removeOnFail: false` and checking for in-progress jobs before enqueuing, or using a framework like Temporal that manages this natively.
+
+**Missed jobs:** Decide whether a missed job (process was down during the scheduled time) should run on restart. For time-sensitive jobs (send a daily digest), skip the missed run. For idempotent catch-up jobs (sync data), run the missed job. Make this behavior explicit in code.
+
+### Event Consumers
+
+Event consumers are workers that read from a message queue or event stream (Kafka, SQS, RabbitMQ, Redis Streams) and process each message.
+
+**At-least-once delivery:** Most message systems guarantee at-least-once delivery — a message may be delivered more than once if the consumer crashes after processing but before acknowledging. Design all consumer logic to be idempotent: track processed message IDs, use database upserts, check state before acting.
+
+**Acknowledgment:** Acknowledge messages only after successfully processing them. Never auto-ack before processing. If processing fails, either nack (return to queue for retry) or move to a DLQ after the maximum retries are exhausted.
+
+**Concurrency:** Run multiple concurrent consumers for throughput. Set concurrency based on downstream resource limits (database connection pool, external API rate limit) rather than CPU count. In BullMQ, set `concurrency` per worker instance. In Celery, set `--concurrency` per worker process.
+
+### Dead Letter Queue Handling
+
+A DLQ captures messages that failed after the maximum retry attempts. Treat the DLQ as a first-class operational concern:
+
+- Alert when DLQ depth exceeds a threshold.
+- Log the failure reason and original message payload when moving to DLQ.
+- Provide an operational runbook and tooling for replaying DLQ messages after the bug is fixed.
+- Set a retention policy on the DLQ (retain for 14 days) to allow investigation without unbounded growth.
+
+Never replay from the DLQ blindly — diagnose the failure first. A bug that caused the original failures will cause the replayed messages to fail again.
+
+### Worker Health Monitoring
+
+Workers are long-running processes. Monitor their health with:
+
+- **Heartbeat:** Workers publish a heartbeat (timestamp) to a shared store (Redis) on each job completion. An external health checker alerts if a worker's heartbeat is stale (no heartbeat in 2 minutes = likely dead worker).
+- **Queue depth metrics:** Export queue pending count, in-progress count, and DLQ count as metrics. Alert on queue depth growth that indicates workers are falling behind.
+- **Job duration metrics:** Track and alert on unexpectedly long-running jobs (potential deadlocks or infinite loops).
+
+### Graceful Shutdown for Workers
+
+Workers must handle `SIGTERM` cleanly to avoid corrupting in-progress jobs during deployments:
+
+1. Stop polling for new jobs immediately on `SIGTERM`.
+2. Allow in-progress jobs to finish (up to a timeout — typically 30 seconds for most jobs, longer for known slow jobs).
+3. Nack or requeue any jobs that cannot complete within the timeout so they are picked up by another worker.
+4. Close database connections and queue broker connections.
+5. Exit with code 0.
+
+Configure Kubernetes `terminationGracePeriodSeconds` to match the shutdown timeout. Without graceful shutdown, rolling deployments drop in-flight jobs.
+
+### Job Prioritization
+
+When a queue processes both urgent and non-urgent work, prioritization prevents starvation:
+
+- **Priority queues**: BullMQ supports priority levels per job. Higher-priority jobs are dequeued first. Use sparingly — too many priority levels create implicit ordering complexity.
+- **Separate queues**: Dedicate separate queues for different urgency levels (critical, standard, background). Assign more workers to the critical queue. This provides stronger isolation — a flood of background jobs cannot starve critical ones.
+- **Fair scheduling**: For multi-tenant systems, ensure one tenant's burst of jobs does not starve other tenants. Use per-tenant rate limiting or round-robin dequeuing across tenant-specific sub-queues.
+
+### Worker Deployment Patterns
+
+Workers have different deployment considerations than HTTP services:
+
+- **Separate deployment**: Deploy workers as separate processes or containers from the API. This allows independent scaling — add more workers when queues are deep without scaling the API.
+- **Resource sizing**: Workers often need more memory than API servers (processing large files, holding state for long workflows). Size worker containers independently based on the job profile.
+- **Rolling deploys**: During a deployment, old and new worker versions run simultaneously. Ensure job schemas are backwards-compatible — a job enqueued by the old version must be processable by the new version, and vice versa.
+- **Singleton workers**: For jobs that must run on exactly one instance (leader election, exclusive cron), use a distributed lock (Redis `SET NX EX`) or a framework that supports this natively (Temporal).
+
+### Job Observability
+
+Workers are invisible unless explicitly instrumented. Essential metrics for every worker system:
+
+- **Job throughput**: Jobs completed per second, broken down by queue and job type. Track trends to detect throughput degradation before it becomes a queue depth problem.
+- **Job duration**: p50, p95, and p99 execution time per job type. Alert on p99 exceeding the expected duration — long-running jobs may indicate a dependency issue or a bug.
+- **Failure rate**: Percentage of jobs that fail after all retries. Track separately from the retry rate (jobs that failed once but succeeded on retry). A rising failure rate after a deployment signals a bug.
+- **Queue age**: Time the oldest unprocessed message has been waiting. This is the most direct measure of consumer lag. Alert if queue age exceeds the SLO (e.g., "all jobs processed within 5 minutes").
+
+Emit structured logs at job start, completion, and failure with the job ID, queue name, duration, and any relevant business identifiers. This enables tracing a specific job through the entire processing lifecycle.

package/content/knowledge/cli/cli-architecture.md

@@ -0,0 +1,101 @@
+---
+name: cli-architecture
+description: Command router patterns, plugin systems, middleware chains, config resolution order, and lazy loading for CLI tools
+topics: [cli, architecture, command-router, plugins, middleware, config-resolution, lazy-loading]
+---
+
+CLI architecture is simpler than web architecture but has its own failure modes: slow startup, monolithic command files that resist extension, config that behaves unpredictably, and plugin systems that become security liabilities. Decisions made at the architecture level — how commands are registered, how config is resolved, how plugins hook in — determine whether the tool remains maintainable as it grows.
+
+## Summary
+
+CLI architecture centers on the command router (mapping argv to handlers via established frameworks like yargs, clap, click, or cobra), middleware chains for cross-cutting concerns, deterministic config resolution order, and lazy loading to keep startup under 100ms. Plugin systems extend the CLI without modifying its source but require scoped capabilities and trust verification.
+
+## Deep Guidance
+
+### Command Router Patterns
+
+The command router is the core of a CLI: it maps `argv` tokens to handler functions.
+
+**Framework-based routing (recommended for most tools)**
+
+Use an established CLI framework rather than hand-rolling argument parsing:
+
+- **Node.js**: `yargs` (batteries-included, good for complex CLIs), `commander` (minimal, explicit), `oclif` (framework for large CLIs with plugin support)
+- **Rust**: `clap` (derive macros for zero-boilerplate arg parsing), `argh` (minimal)
+- **Python**: `click` (decorator-based, composable), `typer` (type-annotation-based, wraps click), `argparse` (stdlib, verbose)
+- **Go**: `cobra` (git/kubectl pattern, de facto standard)
+
+**Registration pattern**: Each command module registers itself with the router. The router does not know about individual commands — it discovers them. This prevents the router from becoming a long switch statement.
+
+### Plugin and Extension Systems
+
+Plugins extend the CLI without modifying its source. Design the plugin API before building it:
+
+- **Registration hook**: Plugins export a `register(cli)` function; the CLI passes the router instance. Plugins call `cli.addCommand(...)` to register their subcommands
+- **Lifecycle hooks**: `beforeCommand`, `afterCommand`, `onError` — plugins can intercept at these points without modifying core code
+- **Capability scope**: Define what plugins can access. Plugins should not be able to override built-in commands or access internal state not explicitly exposed
+
+Security concern: plugins run arbitrary code. Load only from trusted sources; consider a signature or registry verification step for security-sensitive tools.
+
+### Middleware Chains
+
+Middleware runs before and after every command handler. Common middleware concerns:
+
+- **Auth verification**: Check credentials before any command that requires auth
+- **Config loading**: Resolve and validate config before handler execution
+- **Telemetry**: Record command name, exit code, duration (with user consent)
+- **Update checking**: Run in background after command completes; notify user of available updates without blocking
+
+Implement as a chain: each middleware calls `next()` to proceed, or throws/exits to abort. This is the Express/Koa pattern applied to CLIs.
+
+### Config Resolution Order
+
+Enforce a deterministic precedence (highest to lowest):
+
+1. **CLI flags** — explicit user intent, always wins
+2. **Environment variables** — deployment/CI context, overrides files
+3. **Project-level config file** — `.mytoolrc` or `mytool.config.json` in CWD (walk up to repo root)
+4. **User-level config file** — `~/.config/mytool/config.json`
+5. **System-level config** — `/etc/mytool/config.json` (optional, for managed environments)
+6. **Built-in defaults**
+
+Merging strategy: deep merge for object keys, last-wins for scalar values at higher precedence levels. Log the resolved config origin in debug mode so users can diagnose unexpected values.
+
+### Lazy Loading
+
+Startup time is a first-class metric for CLIs. Every millisecond of startup latency is felt on every invocation:
+
+- **Node.js**: Use dynamic `import()` inside command handlers rather than top-level imports. A `build` command that imports a bundler should not load that bundler when the user runs `my-cli --version`
+- **Rust**: Lazy loading is less critical (compile-time linking), but feature flags (`#[cfg(feature = "...")]`) exclude large optional dependencies from the binary
+- **Python**: Import heavy libraries (`numpy`, `boto3`) inside command functions, not at module level
+
+Target: `my-cli --help` should complete in under 100ms on any machine. Measure startup time with `time my-cli --help` and set it as a CI regression gate.
+
+### Error Handling Architecture
+
+Define a typed error hierarchy at the architecture level:
+
+- `UsageError` (exit 2): Bad arguments, unknown flags — show usage hint
+- `ConfigError` (exit 1): Invalid or missing config — show config file path and what was expected
+- `RuntimeError` (exit 1): Operation failed — show what failed and why
+- `NetworkError` (exit 1): External call failed — show endpoint, status code, retry hint
+
+Catch all errors at the top-level router, format based on type, and emit structured JSON when `--json` is active. Never let uncaught exceptions reach the user as stack traces in production.
+
+### Telemetry and Analytics
+
+If the CLI collects usage telemetry (command frequency, error rates, feature adoption):
+
+- **Opt-in by default**: Never collect telemetry without explicit user consent. Prompt on first run and respect the answer. Provide `my-cli telemetry disable` to opt out at any time.
+- **Transparency**: Document exactly what is collected and where it is sent. Provide `my-cli telemetry status` to show current settings.
+- **Privacy**: Never collect file paths, file contents, environment variable values, or personally identifiable information. Collect only: command name, exit code, duration, OS/version, and tool version.
+- **Non-blocking**: Telemetry must never slow down the tool. Send data asynchronously in a background process after command completion. If the telemetry endpoint is down, silently discard the data.
+
+### Startup Performance Architecture
+
+CLI startup time is felt on every invocation. Architectural decisions that affect startup:
+
+- **Lazy command loading**: Only load the code for the invoked command. A `deploy` command should not load the `build` command's bundler dependency.
+- **Cache config resolution**: If config file discovery involves walking the directory tree, cache the resolved path for the current session.
+- **Avoid top-level I/O**: Do not read files, make network calls, or check for updates during module initialization. Defer all I/O to the command handler or a background process.
+- **Benchmark in CI**: Add `time my-cli --version` as a CI step and fail if it exceeds 100ms. Startup regressions creep in gradually — CI gates catch them early.

package/content/knowledge/cli/cli-conventions.md

@@ -0,0 +1,117 @@
+---
+name: cli-conventions
+description: Flag naming, subcommand patterns, help text standards, --version behavior, and NO_COLOR support for CLI tools
+topics: [cli, conventions, flags, subcommands, help-text, no-color, version]
+---
+
+CLI conventions exist because shell users build mental models across dozens of tools. When your tool follows the same conventions as `git`, `docker`, and `kubectl`, users already know how to use it before reading the documentation. Deviating from convention has a real cost: every exception must be explicitly learned.
+
+## Summary
+
+CLI conventions exist because shell users build mental models across dozens of tools. Use established flag vocabulary (`--verbose`, `--quiet`, `--dry-run`, `--force`, `--output`). Structure subcommands as `<tool> <noun> <verb>`. Print help to stdout with USAGE, FLAGS, and EXAMPLES sections. Follow the `NO_COLOR` standard and detect TTY to disable interactive features in pipes.
+
+## Deep Guidance
+
+### Flag Naming
+
+Flag names communicate intent. Use the established vocabulary:
+
+- `--verbose` / `-v`: Increase output detail. Stackable in some tools (`-vvv`)
+- `--quiet` / `-q`: Suppress non-error output
+- `--dry-run`: Show what would happen without doing it
+- `--force` / `-f`: Skip confirmation prompts, overwrite without asking
+- `--output` / `-o`: Specify output file or format
+- `--config` / `-c`: Path to config file
+- `--yes` / `-y`: Auto-confirm all prompts (useful in CI)
+- `--no-<flag>`: Boolean negation. If `--color` is default-on, `--no-color` disables it
+
+Avoid inventing synonyms for established flags. If the tool ecosystem uses `--output`, do not use `--out`, `--dest`, or `--target` for the same concept.
+
+### Subcommand Patterns
+
+For tools with multiple operations, use the `<tool> <verb> [args]` pattern:
+
+```
+mytool init
+mytool build --watch
+mytool deploy --env production
+mytool config set key value
+```
+
+Noun-verb ordering (`mytool config set`) matches how humans describe actions: "I want to set a config value." Avoid verb-noun (`mytool set-config`) — it does not compose well when subcommands are nested.
+
+**Aliases**: Provide short aliases for common subcommands (`ls` → `list`, `rm` → `remove`, `ps` → `status`). Document aliases in help text.
+
+**Consistent behavior**: All subcommands should support `--help`. The top-level `--help` should list all subcommands with one-line descriptions.
+
+### Help Text Standards
+
+Help text is the primary documentation for most users. Structure it:
+
+```
+USAGE
+  mytool <subcommand> [flags] [args]
+
+SUBCOMMANDS
+  init     Initialize a new project
+  build    Build the project
+  deploy   Deploy to a target environment
+
+FLAGS
+  -v, --verbose   Show detailed output
+  -q, --quiet     Suppress non-error output
+      --help      Show this help text
+      --version   Print version and exit
+
+EXAMPLES
+  mytool init my-project
+  mytool build --watch
+  mytool deploy --env staging --dry-run
+
+Run 'mytool <subcommand> --help' for subcommand-specific flags.
+```
+
+Rules:
+- Lead with USAGE, then FLAGS, then EXAMPLES — in that order
+- Always include at least two concrete examples
+- One-line descriptions must fit in a terminal; keep them under 60 characters
+- Print help to stdout, not stderr (allows `mytool --help | less`)
+
+### --version Behavior
+
+Version output should be machine-parseable:
+
+```
+mytool 2.4.1
+```
+
+Some tools include build metadata:
+
+```
+mytool 2.4.1 (commit abc1234, built 2024-01-15)
+```
+
+Always exit 0 after printing version. Never print to stderr. The version string should be parseable by scripts using `mytool --version | awk '{print $2}'`.
+
+### NO_COLOR and Color Output
+
+Color output must be defeatable. Follow the `NO_COLOR` standard (no-color.org):
+
+- If `NO_COLOR` environment variable is set (any value), disable all color output
+- If `--no-color` flag is passed, disable color output
+- If stdout is not a TTY (`!process.stdout.isTTY`), disable color output automatically
+- `--color` / `FORCE_COLOR` can re-enable color for non-TTY output when explicitly requested (useful for CI that does support color)
+
+This order of precedence: `--no-color` flag > `NO_COLOR` env > TTY detection > default (color on TTY).
+
+### Configuration Precedence Convention
+
+When flags, environment variables, and config files all influence behavior, follow this precedence (highest to lowest):
+
+1. CLI flags (most explicit)
+2. Environment variables
+3. Project-local config file (`.mytoolrc`, `mytool.config.json`)
+4. User config file (`~/.config/mytool/config.json`)
+5. Built-in defaults
+
+Document this precedence in `--help` output or the man page. Users need to know where a value is coming from when debugging unexpected behavior.