universal-dev-standards 5.4.0 → 5.6.0

package/bundled/core/cost-budget-test.md

@@ -0,0 +1,69 @@
+# Cost Budget Test Standards
+
+## Overview
+
+AI agent systems that call LLM APIs can accumulate costs rapidly if a pipeline runs in a runaway loop, encounters an unexpected token explosion, or has misconfigured budget thresholds. Cost budget tests verify that the zone classifier, threshold constants, and pipeline budget guards behave correctly at every boundary.
+
+## Zone Classification
+
+Most AI agent token budget systems divide usage ratios into zones. The boundaries between zones are the highest-risk points for off-by-one errors.
+
+```typescript
+// Vitest boundary tests using TOKEN_BUDGET constants (not magic numbers)
+import { classifyTokenZone, TOKEN_BUDGET } from "../types/index.js"
+import { describe, it, expect } from "vitest"
+
+describe("TokenBudgetZone classification boundaries", () => {
+  it.each([
+    [0.0,                                  "safe",     "zero usage"],
+    [TOKEN_BUDGET.WARNING_THRESHOLD - 0.01, "safe",     "just below WARNING"],
+    [TOKEN_BUDGET.WARNING_THRESHOLD,        "warning",  "exactly at WARNING"],
+    [TOKEN_BUDGET.DANGER_THRESHOLD - 0.01,  "warning",  "just below DANGER"],
+    [TOKEN_BUDGET.DANGER_THRESHOLD,         "danger",   "exactly at DANGER"],
+    [TOKEN_BUDGET.BLOCKING_THRESHOLD - 0.01,"danger",   "just below BLOCKING"],
+    [TOKEN_BUDGET.BLOCKING_THRESHOLD,       "blocking", "exactly at BLOCKING"],
+    [1.0,                                  "blocking", "fully exhausted"],
+  ])("ratio=%f → %s (%s)", (ratio, expected) => {
+    expect(classifyTokenZone(ratio)).toBe(expected)
+  })
+
+  it("returns 'blocking' for ratio > 1.0 (over-budget)", () => {
+    expect(classifyTokenZone(1.5)).toBe("blocking")
+  })
+})
+```
+
+## Pipeline Budget Config Tests
+
+```typescript
+import type { PipelineBudgetConfig } from "../types/index.js"
+
+describe("PipelineBudgetConfig semantics", () => {
+  it("warningThreshold defaults should be 0-1 range", () => {
+    const config: PipelineBudgetConfig = {
+      maxCostPerRun: 1.0,
+      maxCostPerDay: 10.0,
+      warningThreshold: 0.8,
+      autoDowngrade: true,
+    }
+    expect(config.warningThreshold).toBeGreaterThan(0)
+    expect(config.warningThreshold).toBeLessThan(1)
+  })
+})
+```
+
+## What to Test
+
+| Test Category | Why |
+|---------------|-----|
+| Exact boundary values (WARNING/DANGER/BLOCKING) | Off-by-one errors hide here |
+| Below each boundary | Confirm zone below is correct |
+| Zero usage ratio | Clean state |
+| Ratio > 1.0 | Over-budget should still block |
+| All TOKEN_BUDGET constants referenced | Mutation survival prevention |
+
+## Related Standards
+
+- [Mutation Testing Standards](mutation-testing.md) — constants without test coverage survive mutations
+- [Testing Standards](testing.md) — overall test pyramid
+- [LLM Output Validation](llm-output-validation.md) — output-layer budget constraints

package/bundled/core/cross-flow-regression.md

@@ -0,0 +1,190 @@
+# Cross-Flow Regression
+
+> **Language**: English | [繁體中文](../locales/zh-TW/core/cross-flow-regression.md)
+
+**Version**: 1.0.0
+**Last Updated**: 2026-05-05
+**Applicability**: All software projects with multiple user flows or business processes
+**Scope**: universal
+**Industry Standards**: ISTQB Advanced Test Analyst (Regression Test Strategy)
+**References**: `core/flow-based-testing.md`, `core/testing-standards.md`
+
+---
+
+## Purpose
+
+This standard defines cross-flow regression testing — verifying that changes to one flow do not break other flows, and that **combinations of flows** behave correctly when executed in sequence.
+
+### Boundary with `flow-based-testing.md`
+
+| Standard | Scope | What It Catches |
+|----------|-------|----------------|
+| `flow-based-testing.md` (Multi-Gate Model) | Single flow: Decision Points, Terminal States, Decision Table | Intra-flow branch coverage gaps |
+| **This standard** | Multiple flows: interaction, shared state, sequential composition | Inter-flow contamination, accumulated-state bugs, regression across flows |
+
+These are complementary, not overlapping. A project needs both.
+
+---
+
+## Why Cross-Flow Bugs Are Distinct
+
+Intra-flow testing (Multi-Gate) proves that "Login" handles all 7 terminal states. But it cannot detect:
+
+- **State contamination**: after a failed "Create Order" (FAIL_QUOTA_EXCEEDED), the quota counter is corrupted → next "Create Order" attempt fails even after quota resets
+- **Shared resource conflicts**: "Report Generation" and "Data Export" running concurrently corrupt a shared temp directory
+- **Sequential dependency**: "Cancel Subscription" succeeds, but the subsequent "Reactivate" flow assumes subscription still exists → NullPointerException
+
+---
+
+## Cross-Flow Test Suite Definition
+
+### Tier-1: Critical User Journeys (CUJ)
+
+Critical User Journeys are end-to-end sequences spanning ≥ 2 flows that represent core business value paths. Every release must include a CUJ regression suite.
+
+**CUJ identification**:
+1. List all flows (from requirement-template §2.4)
+2. Identify pairs/triples that share state or are commonly executed in sequence
+3. Tag business-critical combinations (purchase, onboarding, authentication + downstream)
+
+**CUJ Coverage Requirement**:
+
+| Metric | Tier-2 Threshold (default) | Tier-1 Critical Path |
+|--------|--------------------------|---------------------|
+| CUJ pass rate | ≥ 95% | 100% |
+| Business-critical flow combos | 100% | 100% |
+
+### Tier-2: Regression on Flow Change
+
+When any flow's §2.4 (Decision Points, Terminal States) is modified, the full CUJ suite must re-run — not just the tests for the changed flow. The triggering logic:
+
+```bash
+# In CI: detect flow spec changes
+changed_flows=$(git diff origin/main... --name-only | grep -E "requirement-template|SPEC.*\.md")
+if [ -n "$changed_flows" ]; then
+  npm run test:cross-flow-regression
+fi
+```
+
+### Tier-3: Concurrency and Shared Resource Tests
+
+For projects with concurrent user operations:
+- Two users executing the same flow simultaneously
+- Flow A and Flow B sharing a write resource
+- Long-running Flow (async) interacting with a short Flow result
+
+---
+
+## Test Structure
+
+Cross-flow regression tests use **sequential state threading** across flows (extending the `ctx` pattern from `flow-based-testing.md`):
+
+```typescript
+describe("CUJ: Register → Verify Email → Create First Order", () => {
+  const ctx: {
+    userId?: string
+    token?: string
+    orderId?: string
+  } = {}
+
+  // Flow 1: Register
+  it("Flow-1 Step 1: Register new user", async () => {
+    ctx.userId = await registerUser({ email: testEmail, plan: "trial" })
+    expect(ctx.userId).toBeDefined()
+  })
+
+  // Flow 2: Email Verification (depends on Flow 1 output)
+  it("Flow-2 Step 1: Verify email token", async () => {
+    const token = await getEmailToken(ctx.userId!)
+    ctx.token = await verifyEmail(token)
+    expect(ctx.token).toBeDefined()
+  })
+
+  // Flow 3: Create Order (depends on Flow 2 auth token)
+  it("Flow-3 Step 1: Create first order", async () => {
+    ctx.orderId = await createOrder(ctx.token!, orderPayload)
+    expect(ctx.orderId).toMatch(/^ord-/)
+  })
+
+  // Cross-flow verification: order state reflects trial plan limits
+  it("Cross-flow: Trial plan quota enforced on first order", async () => {
+    const order = await getOrder(ctx.token!, ctx.orderId!)
+    expect(order.quota_applied).toBe("trial")
+  })
+})
+```
+
+### Cross-Flow Failure Isolation
+
+When a cross-flow test fails, the failure message must identify **which flow** introduced the state corruption:
+
+```typescript
+// BAD: generic assertion
+expect(result).toBe("success")
+
+// GOOD: includes flow context
+expect(result).toBe("success")  // Flow-3 depends on Flow-2 token being valid
+                                 // If this fails, check Flow-2 email verification output
+```
+
+---
+
+## Release Gate Integration
+
+Cross-flow regression is **Dimension 6** in `release-readiness-gate.md` (Tier-2).
+
+### Trigger Conditions
+
+| Trigger | Scope |
+|---------|-------|
+| Every release candidate | Full CUJ suite |
+| PR modifying any flow §2.4 | Full CUJ suite (pre-merge) |
+| Post-deploy to staging | Smoke subset of CUJ |
+| Post-deploy to production | Critical path CUJ only (canary) |
+
+### Evidence for Sign-off
+
+```
+| 6 | Cross-flow Regression | PASS | CUJ suite: 47/47 passed; 0 flow-interaction failures | QA Lead |
+```
+
+### WARN Threshold
+
+- ≥ 95% CUJ pass rate but < 100% → WARN with specific failed CUJ documented and root-caused
+- < 95% CUJ pass rate → FAIL (release blocked)
+- Business-critical combo fails → FAIL regardless of overall rate
+
+---
+
+## Anti-Patterns
+
+- **Running cross-flow tests only when CI is slow** — they must run on every release candidate by definition
+- **Testing each flow in isolation and calling it "regression"** — flow isolation is covered by Multi-Gate; cross-flow must have inter-flow state dependencies
+- **Reusing the same `ctx` object across unrelated `describe` blocks** — each CUJ needs a clean, isolated `ctx`; contamination between CUJs masks bugs
+- **No flow attribution in failure messages** — cross-flow failures are hard to debug; always indicate which upstream flow produced the corrupted state
+- **Treating CUJ failures as flaky** — cross-flow state bugs are deterministic; "flaky" cross-flow tests are almost always a symptom of shared state corruption
+
+---
+
+## Relationship to Other Standards
+
+- **`flow-based-testing.md`** — intra-flow gate (prerequisite for cross-flow)
+- **`testing-standards.md`** — regression layer in the testing pyramid
+- **`release-readiness-gate.md`** — Dimension 6 (Tier-2)
+- **`e2e-testing.md`** — CUJ tests typically run at E2E or System Test level
+
+---
+
+## Version History
+
+| Version | Date | Changes |
+|---------|------|---------|
+| 1.0.0 | 2026-05-05 | Initial release: CUJ definition, sequential state threading, release gate criteria, Tier-1/2/3 classification |
+
+---
+
+## License
+
+This standard is released under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
+
+**Source**: [universal-dev-standards](https://github.com/AsiaOstrich/universal-dev-standards)

package/bundled/core/data-migration-testing.md

@@ -0,0 +1,110 @@
+# Data Migration Testing
+
+## Overview
+
+Database schema migrations are high-risk operations: they transform persistent data in ways that cannot be easily undone without a tested rollback path. A comprehensive migration test suite validates three axes — correctness (up applies cleanly), safety (down restores state), and robustness (applying twice is harmless).
+
+## Requirements Summary
+
+| ID | Rule | Rationale |
+|----|------|-----------|
+| REQ-DMT-001 | Every migration must have an up test | Unverified migrations corrupt production schema |
+| REQ-DMT-002 | Every migration with a down function must have a rollback test | Untested rollbacks fail during incidents |
+| REQ-DMT-003 | Applying the same migration twice must not fail | CI retries can trigger double-apply |
+| REQ-DMT-004 | Migrations that alter data must include a data preservation test | Schema correctness ≠ data correctness |
+| REQ-DMT-005 | Each test must use an isolated database | Shared state causes non-deterministic failures |
+
+## Test Structure
+
+### Isolation
+
+Every migration test runs against an isolated database — either in-memory (SQLite `:memory:`) or a fresh Docker container (PostgreSQL). Never run migration tests against a shared development or staging database.
+
+```typescript
+// Good: isolated in-memory database per test file
+const db = new Database(':memory:')
+await applyBaseline(db)
+
+// Bad: tests share a dev database
+const db = openDatabase(process.env.DATABASE_URL)
+```
+
+### Up Test
+
+Apply the migration to a baseline schema. Assert the expected post-state.
+
+```typescript
+it('adds email column to users table', async () => {
+  await migrate.up(db)
+  const columns = db.prepare("PRAGMA table_info(users)").all()
+  expect(columns.map(c => c.name)).toContain('email')
+})
+```
+
+### Down Test (Rollback)
+
+Apply up, then down. Assert the schema returns to its pre-migration state.
+
+```typescript
+it('rollback removes email column', async () => {
+  await migrate.up(db)
+  await migrate.down(db)
+  const columns = db.prepare("PRAGMA table_info(users)").all()
+  expect(columns.map(c => c.name)).not.toContain('email')
+})
+```
+
+### Idempotency Test
+
+Apply the migration twice. The second apply must not throw.
+
+```typescript
+it('applying migration twice is safe', async () => {
+  await migrate.up(db)
+  await expect(migrate.up(db)).resolves.not.toThrow()
+})
+```
+
+### Data Preservation Test
+
+Seed rows before the migration. Assert data integrity after.
+
+```typescript
+it('preserves existing user rows', async () => {
+  db.prepare("INSERT INTO users (id, name) VALUES (1, 'Alice')").run()
+  await migrate.up(db)
+  const user = db.prepare("SELECT * FROM users WHERE id = 1").get()
+  expect(user.name).toBe('Alice')
+})
+```
+
+## Tooling
+
+### SQLite / Drizzle ORM
+
+```typescript
+import Database from 'better-sqlite3'
+import { drizzle } from 'drizzle-orm/better-sqlite3'
+import { migrate } from 'drizzle-orm/better-sqlite3/migrator'
+
+const sqlite = new Database(':memory:')
+const db = drizzle(sqlite)
+await migrate(db, { migrationsFolder: './drizzle' })
+```
+
+### PostgreSQL
+
+Use `testcontainers` to spin up a fresh PostgreSQL container per test suite. The container is destroyed after the suite, guaranteeing isolation.
+
+## Anti-Patterns
+
+- **Testing against a shared database** — causes cross-test pollution, non-repeatable builds
+- **Skipping down migration tests** — rollbacks fail during production incidents
+- **Writing migration tests after the fact without seeding data** — misses data preservation bugs entirely
+- **Committing a migration without a corresponding test** — the migration is untested until production
+
+## See Also
+
+- `database-standards.ai.yaml` — schema design principles
+- `testing.ai.yaml` — general test structure and pyramid
+- `verification-evidence.ai.yaml` — audit evidence requirements

package/bundled/core/disaster-recovery-drill.md

@@ -0,0 +1,73 @@
+# Disaster Recovery Drill Standards
+
+## Overview
+
+An untested DR plan is a false sense of security. Teams that have never executed their recovery runbook under pressure will discover gaps at the worst possible time. DR drills expose these gaps safely.
+
+## RTO/RPO Targets
+
+Define these before writing the runbook:
+
+| Metric | Definition | VibeOps Commercial Target |
+|--------|-----------|--------------------------|
+| RTO (Recovery Time Objective) | Max acceptable downtime | < 1 hour |
+| RPO (Recovery Point Objective) | Max acceptable data loss | < 24 hours (daily backup) |
+
+## Backup Restore Script
+
+```bash
+#!/usr/bin/env bash
+# scripts/backup-restore.sh — DR drill backup restore verification
+set -euo pipefail
+
+BACKUP_DIR="${BACKUP_DIR:-/var/backups/vibeops}"
+RESTORE_DIR="${RESTORE_DIR:-/tmp/dr-restore}"
+DB_FILE="${DB_FILE:-vibeops.db}"
+
+echo "=== DR Drill: Backup Restore Verification ==="
+echo "Source: ${BACKUP_DIR}/${DB_FILE}.backup"
+echo "Target: ${RESTORE_DIR}/${DB_FILE}"
+
+mkdir -p "$RESTORE_DIR"
+
+# Find latest backup
+LATEST=$(ls -t "${BACKUP_DIR}"/*.backup 2>/dev/null | head -1)
+if [[ -z "$LATEST" ]]; then
+  echo "FAIL: No backup found in ${BACKUP_DIR}"
+  exit 1
+fi
+
+# Restore
+cp "$LATEST" "${RESTORE_DIR}/${DB_FILE}"
+
+# Verify integrity (SQLite)
+if command -v sqlite3 >/dev/null 2>&1; then
+  sqlite3 "${RESTORE_DIR}/${DB_FILE}" "PRAGMA integrity_check;" | grep -q "ok" && \
+    echo "OK: Database integrity check passed" || \
+    { echo "FAIL: Integrity check failed"; exit 1; }
+fi
+
+BACKUP_AGE=$(( ($(date +%s) - $(stat -c %Y "$LATEST" 2>/dev/null || stat -f %m "$LATEST")) / 3600 ))
+echo "OK: Backup age: ${BACKUP_AGE} hours (RPO target: 24h)"
+
+echo "=== PASS: Restore complete ==="
+```
+
+## Game Day Protocol
+
+1. **Announce**: Notify team 1 week in advance, define scope
+2. **Baseline**: Document current system state
+3. **Inject**: Simulate failure (rename/delete DB, kill process, etc.)
+4. **Execute**: Team follows runbook from scratch — no shortcuts
+5. **Measure**: Record RTO, RPO, issues encountered
+6. **Retrospective**: What was unclear? What was missing?
+
+## Runbook Template
+
+See `docs/DR-RUNBOOK.md` for the full runbook template.
+
+## Related Standards
+
+- [Deployment Standards](deployment-standards.md) — deployment pipeline
+- [Chaos Engineering Standards](chaos-engineering-standards.md) — failure injection
+- [Verification Evidence Standards](verification-evidence.md) — drill records

package/bundled/core/flaky-test-management.md

@@ -0,0 +1,73 @@
+# Flaky Test Management Standards
+
+## Overview
+
+A single flaky test in a 3000-test suite can erode CI confidence enough that developers start ignoring failures. Once developers learn to "just re-run CI", real bugs slip through. The cost of eliminating flaky tests is always lower than the cost of the false sense of security they create.
+
+## Definition
+
+A test is **flaky** if it produces different results (pass/fail) on consecutive runs with the same code. The 2% threshold: if a test fails ≥ 2% of runs on `main` without code changes, it is flaky.
+
+## Detection
+
+Most CI systems can detect flakiness automatically:
+
+- **GitHub Actions**: Look for `Flaky tests detected` annotations
+- **Manual**: Run `npx vitest run --reporter=verbose` 5 times, look for non-deterministic results
+- **Vitest**: `vitest run --repeat=5` (runs each test 5 times)
+
+## Quarantine Workflow
+
+```
+Detected → Quarantine (< 48h) → Track → Fix or Delete (< 30 days)
+```
+
+### Quarantine Annotation
+
+```typescript
+// TODO: quarantined 2026-05-05 — flaky race condition, see issue #42
+it.skip("reconnects after WebSocket disconnect", async () => {
+  // ... test body preserved for reference
+})
+```
+
+### Tracking Issue Template
+
+```markdown
+**Flaky Test**: `describe > test name`
+**File**: `src/path/to/test.ts`
+**Quarantined**: 2026-05-05
+**Failure rate**: ~5% on main
+**Known failure mode**: `Cannot read property 'socket' of undefined`
+**Root cause hypothesis**: Race condition in WebSocket teardown
+**Deadline**: 2026-06-05
+```
+
+## Common Root Causes
+
+| Root Cause | Fix |
+|-----------|-----|
+| Race condition | Use `waitFor()`, `vi.waitFor()`, proper async coordination |
+| Shared state | Reset state in `beforeEach`/`afterEach` |
+| External service | Mock the dependency |
+| File system ordering | Use deterministic sort |
+| Random without seed | Set fixed seed in test |
+| Timing-dependent | Fake timers (`vi.useFakeTimers()`) |
+
+## Vitest Configuration
+
+```typescript
+// vitest.config.ts
+export default defineConfig({
+  test: {
+    retry: 2,              // retry failed tests up to 2 times
+    testTimeout: 10000,    // 10s timeout prevents infinite hangs
+    hookTimeout: 5000,     // 5s hook timeout
+  }
+})
+```
+
+## Related Standards
+
+- [Testing Standards](testing.md) — overall test pyramid
+- [Test Governance Standards](test-governance.md) — CI policies