@loomfsm/bundle-code 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. package/LICENSE +201 -0
  2. package/agents/acceptance.md +141 -0
  3. package/agents/api-contract.md +89 -0
  4. package/agents/architect.md +52 -0
  5. package/agents/challenger-reviewer.md +104 -0
  6. package/agents/classifier.md +74 -0
  7. package/agents/code-analyzer.md +43 -0
  8. package/agents/context-doc-verifier.md +94 -0
  9. package/agents/dependency-auditor.md +42 -0
  10. package/agents/implementer.md +135 -0
  11. package/agents/logic-reviewer.md +132 -0
  12. package/agents/migration.md +55 -0
  13. package/agents/performance.md +95 -0
  14. package/agents/plan-conformance.md +127 -0
  15. package/agents/plan-grounding-check.md +106 -0
  16. package/agents/planner.md +143 -0
  17. package/agents/playwright.md +68 -0
  18. package/agents/research.md +52 -0
  19. package/agents/security.md +88 -0
  20. package/agents/style-reviewer.md +85 -0
  21. package/agents/test.md +206 -0
  22. package/agents/ui-consistency.md +75 -0
  23. package/dist/manifest.d.ts +2 -0
  24. package/dist/manifest.js +34 -0
  25. package/dist/manifest.js.map +1 -0
  26. package/dist/src/bundle.d.ts +2 -0
  27. package/dist/src/bundle.js +424 -0
  28. package/dist/src/bundle.js.map +1 -0
  29. package/dist/src/index.d.ts +5 -0
  30. package/dist/src/index.js +14 -0
  31. package/dist/src/index.js.map +1 -0
  32. package/dist/src/invariants.d.ts +10 -0
  33. package/dist/src/invariants.js +208 -0
  34. package/dist/src/invariants.js.map +1 -0
  35. package/dist/src/policy-resolver.d.ts +2 -0
  36. package/dist/src/policy-resolver.js +65 -0
  37. package/dist/src/policy-resolver.js.map +1 -0
  38. package/dist/src/sandbox-rules.d.ts +2 -0
  39. package/dist/src/sandbox-rules.js +40 -0
  40. package/dist/src/sandbox-rules.js.map +1 -0
  41. package/dist/test/bundle.test.d.ts +1 -0
  42. package/dist/test/bundle.test.js +289 -0
  43. package/dist/test/bundle.test.js.map +1 -0
  44. package/dist/test/sandbox-rules.test.d.ts +1 -0
  45. package/dist/test/sandbox-rules.test.js +73 -0
  46. package/dist/test/sandbox-rules.test.js.map +1 -0
  47. package/knowledge/references/api-design.md +188 -0
  48. package/knowledge/references/arch-patterns.md +106 -0
  49. package/knowledge/references/caching.md +190 -0
  50. package/knowledge/references/concurrency.md +195 -0
  51. package/knowledge/references/db-postgres.md +153 -0
  52. package/knowledge/references/e2e-flutter.md +56 -0
  53. package/knowledge/references/e2e-playwright.md +53 -0
  54. package/knowledge/references/error-handling.md +208 -0
  55. package/knowledge/references/next-app-router.md +231 -0
  56. package/knowledge/references/observability.md +169 -0
  57. package/knowledge/references/optimization-strategy.md +197 -0
  58. package/knowledge/references/perf-flutter.md +62 -0
  59. package/knowledge/references/perf-nestjs.md +59 -0
  60. package/knowledge/references/perf-python.md +50 -0
  61. package/knowledge/references/perf-react.md +52 -0
  62. package/knowledge/references/react19.md +176 -0
  63. package/knowledge/references/redis.md +175 -0
  64. package/knowledge/references/security-backend.md +219 -0
  65. package/knowledge/references/test-flutter.md +65 -0
  66. package/knowledge/references/test-nestjs.md +82 -0
  67. package/knowledge/references/test-python.md +76 -0
  68. package/knowledge/references/test-react.md +66 -0
  69. package/knowledge/references/test-strategy.md +175 -0
  70. package/knowledge/references/ui-flutter.md +56 -0
  71. package/knowledge/references/ui-web.md +51 -0
  72. package/package.json +34 -0
  73. package/schemas/agent-feedback.schema.json +80 -0
  74. package/schemas/category-vocab.json +170 -0
  75. package/schemas/classifier-output.schema.json +53 -0
  76. package/schemas/finding.schema.json +92 -0
  77. package/schemas/pipeline-state.schema.json +238 -0
  78. package/schemas/reviewer-output.schema.json +62 -0
  79. package/schemas/state-extension.schema.json +53 -0
  80. package/schemas/validator-output.schema.json +48 -0
  81. package/stack-candidates.yaml +248 -0
@@ -0,0 +1,59 @@
1
+ ---
2
+ tags: [performance, nestjs, nodejs, orm, n-plus-one, backend]
3
+ stack_signals:
4
+ - language: [typescript, javascript]
5
+ - project_type: [backend, monorepo]
6
+ summary: |
7
+ NestJS / Node.js performance checklist — N+1 patterns, missing indexes,
8
+ pagination, eager loading, ORM mistakes, request-handler hot paths.
9
+ when_to_load: |
10
+ Task touches NestJS controllers / services / providers, TypeORM or Prisma
11
+ queries, request-pipeline middleware on a Node.js backend with perf
12
+ concerns or scale targets.
13
+ agent_hints: [performance, logic-reviewer, challenger-reviewer]
14
+ ---
15
+
16
+ # Performance: NestJS / Node.js
17
+
18
+ ## Database & ORM
19
+ - N+1 query patterns (loading relations in loops)
20
+ - Missing database indexes (implied by WHERE/ORDER BY columns)
21
+ - Missing pagination on list endpoints (unbounded queries)
22
+ - Non-parameterized queries prevent database query plan caching (also a security risk)
23
+ - Eager loading too many relations (over-fetching)
24
+ - Missing select() — fetching all columns when only a few are needed
25
+ - Transaction scope too wide (holding locks longer than necessary)
26
+
27
+ ## API & HTTP
28
+ - Synchronous operations blocking the event loop (CPU-heavy in request handler)
29
+ - Missing caching for expensive repeated operations (Redis, in-memory)
30
+ - No rate limiting on public/expensive endpoints
31
+ - Large response payloads without pagination or streaming
32
+ - Missing compression (gzip/brotli)
33
+ - File uploads without size limits
34
+ - Missing timeout on external HTTP calls
35
+
36
+ ## Serialization & Validation
37
+ - ClassSerializerInterceptor on every response — expensive for hot paths; consider manual DTOs
38
+ - class-validator with deeply nested DTOs — use `whitelist: true` and `forbidNonWhitelisted: true`
39
+ - JSON serialization of large objects without streaming
40
+
41
+ ## Architecture
42
+ - Blocking constructor operations (should be in onModuleInit)
43
+ - Synchronous file I/O (readFileSync, writeFileSync)
44
+ - Request-scoped providers where singleton would work (Scope.REQUEST propagates to all consumers)
45
+ - Unused providers still registered (loaded but never called)
46
+ - Missing queue/background job for heavy operations in request path (emails, reports, file processing)
47
+ - Circular dependencies via `forwardRef()` — unexpected init overhead
48
+
49
+ ## Memory & Resources
50
+ - Event listeners or intervals not cleaned up in onModuleDestroy
51
+ - Large objects held in module-scoped variables (memory leak)
52
+ - Missing stream processing for large files (loading entire file to memory)
53
+ - Unbounded caches without TTL or max size
54
+
55
+ ## Node.js Runtime
56
+ - Not utilizing multiple CPU cores — consider cluster mode or worker threads for CPU-bound ops
57
+ - Node.js doesn't cache DNS by default — repeated external HTTP calls can suffer from DNS lookup latency
58
+ - Verbose logging in production (string formatting overhead even when log level disabled)
59
+ - Consider Fastify adapter over Express for high-throughput services (~2x faster)
@@ -0,0 +1,50 @@
1
+ ---
2
+ tags: [performance, python, fastapi, asyncio, backend]
3
+ stack_signals:
4
+ - language: [python]
5
+ - project_type: [backend, monorepo]
6
+ summary: |
7
+ Python / FastAPI / asyncio performance checklist — N+1, pagination, async
8
+ pool sizing, transaction scope, gather vs serial calls.
9
+ when_to_load: |
10
+ Task touches Python backend code (FastAPI, Django, Flask) with perf
11
+ concerns, scale targets, or async/await + DB interaction. Diff in
12
+ *.py with route handlers, async functions, or DB query construction.
13
+ agent_hints: [performance, logic-reviewer, challenger-reviewer]
14
+ ---
15
+
16
+ # Performance: Python / FastAPI / asyncio
17
+
18
+ ## Database
19
+ - N+1 queries (loading in loops instead of batch)
20
+ - Missing database indexes (implied by WHERE/ORDER BY columns)
21
+ - Missing pagination on list endpoints (unbounded queries)
22
+ - Transaction scope too wide (holding DB connections across gRPC/HTTP calls)
23
+ - asyncpg pool exhaustion — missing `min_size`/`max_size` tuning, or not releasing connections promptly
24
+ - Unbounded caches without TTL
25
+
26
+ ## Async Runtime
27
+ - Blocking sync calls in async handlers (sync I/O, CPU-heavy ops without executor)
28
+ - GIL-bound CPU work — `asyncio.to_thread()` only helps I/O; CPU parallelism needs `ProcessPoolExecutor`
29
+ - Missing `asyncio.Semaphore` for concurrent external calls
30
+ - Missing timeout on external HTTP/gRPC calls (`httpx` timeout, `asyncio.wait_for`)
31
+ - Sync file I/O in async context
32
+
33
+ ## API & HTTP
34
+ - Missing connection pool limits on outbound HTTP clients
35
+ - Large response payloads without pagination
36
+ - Missing `StreamingResponse` for large data exports (building entire response in memory)
37
+ - Heavy work in request path that should use `BackgroundTasks` or task queue
38
+ - Missing response class optimization (`ORJSONResponse` vs default `JSONResponse` for large payloads)
39
+
40
+ ## Serialization & Validation
41
+ - Pydantic model validation cost on hot paths — use `model_construct` for trusted internal data
42
+ - Repeated validation of same data across middleware/dependencies
43
+ - `json.dumps` for large payloads — use `orjson` for 3-10x speedup
44
+
45
+ ## FastAPI-Specific
46
+ - Heavy middleware running on every request that could be scoped to specific routes
47
+ - `Depends()` chains re-executing expensive lookups per request without caching
48
+ - Not pre-warming connection pools at startup (lifespan handler)
49
+ - Excessive DEBUG-level logging in production (string formatting even when log level disabled)
50
+ - Import-time side effects slowing startup
@@ -0,0 +1,52 @@
1
+ ---
2
+ tags: [performance, react, nextjs, rendering, bundle, hydration, frontend]
3
+ stack_signals:
4
+ - language: [typescript, javascript]
5
+ - project_type: [frontend-app, monorepo]
6
+ summary: |
7
+ React/Next.js performance checklist — unnecessary re-renders, heavy
8
+ build/render paths, bundle bloat, hydration mismatches, asset
9
+ optimization, App Router server/client boundary choices.
10
+ when_to_load: |
11
+ Task touches React/Next.js components, hooks, rendering perf, bundle size,
12
+ image/font loading, or Next.js-specific perf concerns (revalidate, ISR,
13
+ Server vs Client components). Reviewer fan-out includes performance
14
+ or UI-consistency on a React/Next stack.
15
+ agent_hints: [performance, logic-reviewer, ui-consistency]
16
+ ---
17
+
18
+ # Performance: React / Next.js
19
+
20
+ ## Rendering
21
+ - Unnecessary re-renders (missing memo/useMemo/useCallback where it matters)
22
+ - Heavy computations in render path — move to useMemo or outside component
23
+ - Context provider placed too high — causes cascading re-renders in unrelated subtrees
24
+ - State lifted too high when it could be local (colocate state with consumers)
25
+ - Large inline objects/arrays in JSX — creates new references every render, defeats React.memo
26
+ - Key prop misuse — using array index as key in dynamic lists causes unnecessary unmount/remount
27
+ - useEffect with unstable dependencies (object/array refs) running every render
28
+
29
+ ## Data Fetching
30
+ - Client-side data fetching that could be server-side (SSR/SSG)
31
+ - Missing React Query / SWR deduplication for same endpoint called from multiple components
32
+ - Hydration mismatch causing full client-side re-render
33
+
34
+ ## Bundle & Loading
35
+ - Large new dependencies added to bundle (check with bundlephobia)
36
+ - Missing lazy loading for heavy routes/components (`React.lazy` / `next/dynamic`)
37
+ - Missing virtualization for long lists (50+ items) — use react-window/react-virtual
38
+ - Missing debounce/throttle on frequent events (search input, scroll, resize)
39
+
40
+ ## Assets
41
+ - Unoptimized images (missing `next/image`, no width/height, no srcset)
42
+ - Missing font loading strategy (`next/font` or `font-display: swap`)
43
+ - Third-party scripts loaded synchronously — use `next/script` with `strategy="lazyOnload"`
44
+
45
+ ## Next.js Specific
46
+ - Using client components where server component would suffice
47
+ - Missing `revalidate` on ISR pages (stale data served indefinitely)
48
+ - `getServerSideProps` where `getStaticProps` + revalidate would work
49
+ - Not using Route Handlers / Server Actions for mutations (unnecessary client-side fetch)
50
+
51
+ ## Note
52
+ React 19 compiler (React Forget) auto-memoizes — manual useMemo/useCallback may become less critical. Still flag missing memoization but note this.
@@ -0,0 +1,176 @@
1
+ ---
2
+ tags: [react, react19, rsc, server-components, server-actions, suspense, frontend]
3
+ stack_signals:
4
+ - language: [typescript, javascript]
5
+ - project_type: [frontend-app, monorepo]
6
+ summary: |
7
+ React 19 stance — Server Components, Server Actions, use(), useOptimistic,
8
+ useFormStatus, useActionState, Suspense boundaries, React Compiler.
9
+ Delete code that used to be hand-rolled, but don't over-apply primitives.
10
+ when_to_load: |
11
+ Project uses react@>=19 (per package.json) or Next.js ≥15 (depends on React
12
+ 19). Diff includes Server/Client component boundary changes, Server Actions,
13
+ use() hook, useOptimistic, useFormStatus, useActionState, useTransition,
14
+ Suspense boundaries, or React Compiler annotations.
15
+ agent_hints: [logic-reviewer, performance, ui-consistency, challenger-reviewer]
16
+ ---
17
+
18
+ # React 19 — Senior Stance
19
+
20
+ ## When this applies
21
+ Load when project uses `react@>=19` (check package.json) or Next.js ≥15 (which depends on React 19). Reviewer auto-loads when diff includes Server/Client component boundary changes, Server Actions, `use()` hook, `useOptimistic`, `useFormStatus`, `useActionState`, `useTransition`, Suspense boundaries, or React Compiler annotations.
22
+
23
+ ## Default Stance
24
+ React 19 collapses several layers that React 18 made you build by hand: form actions, optimistic updates, async-aware suspension, ref forwarding. The win is "delete code that used to be hand-rolled". The risk is over-applying the new primitives where simpler local state was already correct. Don't refactor working `useState` to `useOptimistic` without a real concurrent-mutation reason. Rename of `forwardRef` is real cleanup; rewriting all forms to Actions is opt-in based on actual benefit.
25
+
26
+ ## Patterns (use these)
27
+
28
+ ### `ref` is now a regular prop (no more `forwardRef`)
29
+ React 19 lets function components accept `ref` directly:
30
+ ```tsx
31
+ function Button({ ref, ...props }: ButtonProps & { ref?: React.Ref<HTMLButtonElement> }) {
32
+ return <button ref={ref} {...props} />;
33
+ }
34
+ ```
35
+ `forwardRef` still works but is being deprecated. Migrate when touching the file, don't sweep.
36
+
37
+ ### `use()` for reading promises and context conditionally
38
+ - `use(promise)` suspends until resolution. Read inside any component, including conditionally (unlike hooks).
39
+ - `use(context)` works inside conditionals — the only legal hook-like call inside `if`.
40
+ - Cache promises outside render, or use a stable promise source (route loader, parent prop). Creating a new promise inside render = infinite suspend loop.
41
+
42
+ ### `useActionState` for form submissions
43
+ Replaces hand-rolled `useState` + `useTransition` + error tracking + pending tracking around form posts.
44
+ ```tsx
45
+ const [state, action, isPending] = useActionState(submitForm, initialState);
46
+ return <form action={action}>...</form>;
47
+ ```
48
+ Server Action signature: `async function submitForm(prevState, formData) { ... return newState }`.
49
+
50
+ ### `useOptimistic` for rapid optimistic mutation
51
+ Use when **the user makes the same kind of mutation rapidly** (likes, comments, drag-reorder) and visual lag is the dominant UX cost. Returns optimistic state during the action, reverts on failure.
52
+ ```tsx
53
+ const [optimisticItems, addOptimistic] = useOptimistic(items, (state, newItem) => [...state, newItem]);
54
+ ```
55
+ Don't reach for it on rare mutations where a brief loading state is fine.
56
+
57
+ ### `useFormStatus` to read parent form state
58
+ Inside a child component nested under a `<form>` — read pending/data/method without prop drilling.
59
+ ```tsx
60
+ function SubmitButton() {
61
+ const { pending } = useFormStatus();
62
+ return <button disabled={pending}>...</button>;
63
+ }
64
+ ```
65
+
66
+ ### Server Actions
67
+ `'use server'` directive on a function exposes it as an RPC endpoint. Client can call it directly; React handles serialization and form integration.
68
+ - **Validate inputs server-side, always.** `'use server'` is the boundary; client-supplied args are untrusted.
69
+ - Auth check **inside** the action body, not in the calling component. Components can be bypassed.
70
+ - Sensitive data must not be returned through revalidation paths that may surface in cache layers.
71
+
72
+ ### Server Components
73
+ - Server Components run only on server, never ship to client. Default for static and DB-bound content.
74
+ - Cannot use hooks (`useState`, `useEffect`) — they're client-only. If you need state, the component is `'use client'`.
75
+ - Pass server data to client components via props at the boundary. Don't lift `'use client'` higher than necessary.
76
+
77
+ ### Suspense boundary granularity
78
+ Each `<Suspense>` is an independent loading region. Place at the smallest unit where loading has meaning (a single widget, not the whole page). Coarse Suspense boundaries cause whole-screen flashes when one part is slow.
79
+
80
+ ### React Compiler (when adopted)
81
+ - Memoizes automatically. **Stop manually wrapping things in `useMemo` / `useCallback`** when compiler is on.
82
+ - Verify compiler is on with `// eslint-disable-next-line react-compiler/react-compiler` guard rules.
83
+ - Compiler bails out of components with rule violations (impure render, mutating props, etc.). Read compiler logs.
84
+
85
+ ## Anti-Patterns (DO NOT)
86
+
87
+ ### Creating promises inside render
88
+ ```tsx
89
+ // BAD
90
+ function Page() {
91
+ const data = use(fetchData()); // new promise every render → infinite suspend
92
+ }
93
+ ```
94
+ **Rule:** create the promise outside the component (route loader, server fetch returned as prop) or stable-cache it.
95
+
96
+ ### Server Action without auth check
97
+ **Why it bites:** the calling component verifies auth, but the action is callable directly via fetch from anywhere. The component isn't a security boundary.
98
+ **Rule:** every Server Action begins with explicit `requireAuth(...)` or equivalent. No exceptions.
99
+
100
+ ### Adding `'use client'` to a component because "it has a button"
101
+ **Why it bites:** marking a tree boundary as Client ships it all to the browser. Big perf regression on what should be server-rendered.
102
+ **Rule:** isolate the interactive island. Wrap just the button in a Client component, leave the rest Server.
103
+
104
+ ### `useOptimistic` on rare or critical mutations
105
+ **Why it bites:** on failure the UI flips back, looks broken; on a payment / save / publish flow the user thinks the action succeeded.
106
+ **Rule:** optimistic UI for repetitive low-stakes actions (likes, cart count, drag). Pessimistic with explicit success state for high-stakes.
107
+
108
+ ### Mixing `useState` for form data with `useActionState`
109
+ **Why it bites:** two sources of truth, redundant pending tracking, divergent error handling.
110
+ **Rule:** pick one. If using `useActionState`, the `state` from it is the form's truth.
111
+
112
+ ### Putting `'use server'` on shared utility files
113
+ **Why it bites:** every export becomes a public RPC endpoint. Easy to accidentally expose internal logic.
114
+ **Rule:** `'use server'` files contain only intended actions. Internal helpers live in non-`'use server'` files.
115
+
116
+ ### Hand-rolled memoization with React Compiler enabled
117
+ **Why it bites:** compiler does it; manual `useMemo` adds noise, can be inconsistent with compiler's analysis, diverges over time.
118
+ **Rule:** when compiler is on, delete `useMemo`/`useCallback` unless there's a measured benefit it doesn't cover.
119
+
120
+ ### Returning huge data from Server Action
121
+ **Why it bites:** action result is serialized through the wire. 1MB response = slow, may exceed framework limits.
122
+ **Rule:** return what the form needs (success flag, errors, redirect target). Refetch heavy data via the page.
123
+
124
+ ### Fine-grained Suspense everywhere
125
+ **Why it bites:** every Suspense = a network round trip in some setups, plus visual jank as parts pop in independently.
126
+ **Rule:** boundaries match user-meaningful regions. Sidebar, main content, secondary content. Not every component.
127
+
128
+ ### Using async client components
129
+ **Why it bites:** client components can't be async. The boundary is server-only.
130
+ **Rule:** async function = Server Component. If you need a Client Component, it's sync; data comes via props or `use()`-wrapped promise from parent server.
131
+
132
+ ### Mutating props or state inside render
133
+ **Why it bites:** React Compiler bails out → no memoization. React 19 strict mode catches more of these. Was tolerated; now broken.
134
+ **Rule:** treat all render inputs as immutable.
135
+
136
+ ## Decision Framework
137
+
138
+ | Situation | Choice |
139
+ |---|---|
140
+ | Form submission with server-side logic | Server Action + `useActionState` |
141
+ | Optimistic UI for likes/cart-add | `useOptimistic` |
142
+ | Optimistic UI for save/publish/payment | Pessimistic + explicit pending |
143
+ | Reading async data in a component | Server Component (default) or `use()` in client with stable promise |
144
+ | Need state inside what's currently a Server Component | Extract interactive island as Client Component |
145
+ | Forwarding ref to a custom component | Direct `ref` prop (not `forwardRef`) |
146
+ | Heavy compute in render | Server Component, or `useMemo` if React Compiler off |
147
+ | Conditional context read | `use(MyContext)` (legal in conditionals) |
148
+ | Pending UI inside form button | `useFormStatus` (no prop drilling) |
149
+ | Migrating to compiler-managed memo | Run compiler in opt-in mode first, fix bail-outs, then turn on globally |
150
+
151
+ ## Cost Model
152
+
153
+ | Pattern | Cost / Win |
154
+ |---|---|
155
+ | `'use client'` on a leaf | +5-30KB JS bundle for that island |
156
+ | `'use client'` on parent of large tree | +entire-tree JS to bundle |
157
+ | Server Component | 0 client JS for that component |
158
+ | Server Action call | 1 round trip + serialization cost |
159
+ | `use(promise)` with stable promise | suspends, no extra round trip if promise is from server |
160
+ | `useOptimistic` reverting on error | Visual flip 200-1000ms after action |
161
+ | React Compiler enabled | ~5-20% bundle reduction vs hand-memo, fewer re-renders |
162
+
163
+ ## Red Flags in Diff
164
+
165
+ - `'use client'` added to a component without an interactive primitive (no `onClick`, `useState`, `useEffect`) — flag (probably can stay Server).
166
+ - New `forwardRef` in React 19 codebase — flag, prefer direct `ref` prop.
167
+ - Server Action without explicit auth check at top of function body — flag immediately (security).
168
+ - `use(somePromise())` where `somePromise()` is invoked inline in render — flag (infinite suspend).
169
+ - `useOptimistic` on mutation with side-effect failure modes (write to external system, payment) — flag (UX/integrity risk).
170
+ - `useState` + `useTransition` + manual error tracking around a form submit — flag (use `useActionState`).
171
+ - Async function inside `'use client'` file as a default export → flag (client components can't be async).
172
+ - Suspense boundary wrapping the entire page or layout → flag (too coarse).
173
+ - `useMemo` / `useCallback` everywhere AND `react-compiler` is on in config → flag (delete the manual ones).
174
+ - `'use server'` file exporting non-action utilities → flag (accidental RPC exposure).
175
+ - Mutation of props or state object during render → flag (compiler bail-out).
176
+ - Server Action returning > 100KB of data → flag (probably should refetch via page).
@@ -0,0 +1,175 @@
1
+ ---
2
+ tags: [redis, cache, session, queue, rate-limit, pubsub, distributed-lock]
3
+ stack_signals:
4
+ - project_type: [backend, monorepo]
5
+ summary: |
6
+ Redis design — single-threaded per shard, in-memory budget. Keep keys
7
+ small, commands O(1) or bounded, and design every feature to degrade
8
+ gracefully on a Redis blip.
9
+ when_to_load: |
10
+ Task touches cache layer, session store, rate limiter, queue (BullMQ,
11
+ Sidekiq, RQ), pub/sub, distributed lock, ratelimit, or real-time presence.
12
+ Diff including Redis client calls (redis., ioredis, node-redis, redis-py,
13
+ lettuce, Bull, BullMQ, RedisCacheStore) qualifies.
14
+ agent_hints: [logic-reviewer, performance, challenger-reviewer]
15
+ ---
16
+
17
+ # Redis — Senior Stance
18
+
19
+ ## When this applies
20
+ Load when task touches: cache layer, session store, rate limiter, queue (BullMQ, Sidekiq, RQ), pub/sub, distributed lock, ratelimit, real-time presence. Reviewer auto-loads when diff includes Redis client calls (`redis.`, `ioredis`, `node-redis`, `redis-py`, `lettuce`, `Bull`, `BullMQ`, `RedisCacheStore`).
21
+
22
+ ## Default Stance
23
+ Redis is single-threaded per shard and held in memory. Every command competes for the same CPU and the same RAM budget. Treat it as a shared resource with a strict SLA: keep keys small, commands O(1) or bounded, and persistence trade-offs explicit. A Redis outage cascades — design every feature so a 30-second Redis blip degrades, not breaks.
24
+
25
+ ## Patterns (use these)
26
+
27
+ ### Pick the right primitive
28
+ - **String** — value + optional TTL. Default for cached payload, counter (`INCR`), feature flag.
29
+ - **Hash** — object with multiple fields, when you read/write fields independently. Cheaper than JSON-string for partial updates.
30
+ - **Set** — uniqueness. Online users, deduplication, tag membership.
31
+ - **Sorted Set (ZSET)** — leaderboards, time-ordered streams of bounded size, top-N queries. `ZADD` + `ZRANGE` is the workhorse.
32
+ - **List** — simple FIFO queue, recent-N items (`LPUSH` + `LTRIM`).
33
+ - **Stream (XADD/XREADGROUP)** — durable queue with consumer groups, replay. Use this for queue-of-record (not List).
34
+ - **HyperLogLog** — approximate cardinality. 12KB for billions of entries. Don't reach for Set when "approximate count of unique" is enough.
35
+ - **Bitmap** — daily active flags. `SETBIT user_id`, `BITCOUNT`. 1MB = 8M users.
36
+
37
+ ### Persistence — choose deliberately
38
+ - **None (cache only)** — RDB off, AOF off. Acceptable for pure cache. Restart = empty.
39
+ - **RDB snapshots** — periodic dump. Loses last N seconds. Cheap. Default for many setups.
40
+ - **AOF (appendonly yes, fsync everysec)** — fsync each second. Loses ≤1s on crash. The right default for state-bearing Redis.
41
+ - **AOF fsync always** — every write fsync'd. 10x slower writes. Almost never the right choice.
42
+ - Replication is not a backup. Use snapshots for backup.
43
+
44
+ ### Eviction policy
45
+ Set explicit `maxmemory` and `maxmemory-policy`. Default of "no eviction" turns Redis into a wall when full → all writes fail.
46
+ - **allkeys-lru** — pure cache.
47
+ - **volatile-lru** — mixed cache + persistent keys (only TTL'd keys evict).
48
+ - **noeviction** — only when Redis is single-purpose state with strict guarantees.
49
+
50
+ ### TTL discipline
51
+ Every cache key has a TTL. No exceptions. Without TTL, a bug accumulates dead data forever. Set TTL on `SET` directly (`SET k v EX 600`), not as a separate `EXPIRE`.
52
+
53
+ ### Pipelining and MULTI
54
+ - **Pipeline** — batch many commands without atomicity. ~10x throughput on round-trip-bound workloads.
55
+ - **MULTI/EXEC** — atomic multi-command. All-or-nothing. Use when you need atomic compose (e.g. `INCR` + `EXPIRE`). Note: still single-threaded, blocks server briefly for the block duration.
56
+ - **Lua script (EVAL)** — atomic, but server runs it single-threaded. Keep scripts O(1) or bounded.
57
+
58
+ ### Distributed locking
59
+ The naive lock — `SET key value NX EX 10` — is correct for single-node Redis if you also:
60
+ 1. Use a unique random value per acquirer.
61
+ 2. Release via Lua script that checks value before DEL (avoids releasing someone else's lock after expiry).
62
+
63
+ For multi-node Redis cluster: **Redlock has known correctness issues under network partitions.** If you actually need distributed mutual exclusion across replicas, use a real consensus system (Postgres advisory locks, etcd, ZooKeeper). Redlock is "best-effort with a fence token", not "guaranteed mutex".
64
+
65
+ ### Rate limiting
66
+ - **Fixed window** — `INCR key`, `EXPIRE key 60` if first. Simple, has boundary spikes.
67
+ - **Sliding log** — `ZADD`, prune old, `ZCARD`. Memory-heavy.
68
+ - **Token bucket via Lua** — best correctness/cost balance. One Lua script, atomic, bounded.
69
+ - For high-volume — use a pre-built lib (`redis-cell` module, `node-rate-limiter-flexible`) instead of writing your own.
70
+
71
+ ### Pub/Sub vs Streams
72
+ - **Pub/Sub** — fire-and-forget. Subscriber not connected = message lost. No replay. Use for real-time only when loss is acceptable.
73
+ - **Streams** — durable, consumer groups, replay, ack. Use for "queue of work". Default to streams for anything that must not be lost.
74
+
75
+ ### Cache key design
76
+ - Namespace prefix: `app:env:entity:id` — `wandr:prod:user:123`. Makes debugging and bulk-purge easy.
77
+ - Include version in key when shape may change: `wandr:prod:user:123:v2`. Makes safe rollouts trivial — change version, old cache evicts naturally via TTL.
78
+ - Hash large keys: `user:hash(email)` rather than `user:long-email-string`.
79
+
80
+ ## Anti-Patterns (DO NOT)
81
+
82
+ ### `KEYS *` in production
83
+ **Why it bites:** O(N) over the entire keyspace, blocks the single thread for the duration. On a 10M-key Redis = full server stall for seconds.
84
+ **Rule:** use `SCAN` with `MATCH` and `COUNT`. Iterates with bounded work per call.
85
+
86
+ ### Storing 1MB+ values in a single key
87
+ **Why it bites:** `GET` of 1MB blocks the single thread for the network write. Memory fragmentation worsens with large values.
88
+ **Rule:** keep values < 100KB. Split into hash fields if needed. For huge blobs, use object storage and put the URL in Redis.
89
+
90
+ ### Using Redis as primary database
91
+ **Why it bites:** persistence is best-effort (even AOF every-sec loses 1s). No transactions across multiple keys with rich constraints. No queries beyond key lookup.
92
+ **Rule:** Redis = cache + ephemeral state + queue. Source of truth = real DB.
93
+
94
+ ### No TTL on cache keys
95
+ **Why it bites:** memory grows monotonically. Eventually `maxmemory` hit, eviction kicks in, but eviction may evict the wrong things. Bugs in cache invalidation accumulate forever.
96
+ **Rule:** every cache key has TTL at write time. Even "we'll invalidate manually" — set a backup TTL.
97
+
98
+ ### `SUBSCRIBE` for queue work
99
+ **Why it bites:** subscriber dead for 1ms during deploy = message lost forever. No retry. No DLQ.
100
+ **Rule:** Streams (`XADD` + `XREADGROUP`) for any work that matters.
101
+
102
+ ### Long-running Lua scripts
103
+ **Why it bites:** Lua runs on the single thread. A 100ms script = 100ms freeze of the entire Redis instance for every other client.
104
+ **Rule:** Lua must be bounded. No loops over unbounded sets.
105
+
106
+ ### Multiple Redis clients per process without connection pool
107
+ **Why it bites:** each client = TCP connection + command queue. Connection storm under load.
108
+ **Rule:** one shared client (or a small pool) per process. Configure with retry strategy and backoff.
109
+
110
+ ### Storing JSON-stringified objects when you need partial updates
111
+ **Why it bites:** read-modify-write race. Two clients update field A and B → last write wins → one update lost.
112
+ **Rule:** use Hash type with `HSET field value`. Field-level atomic updates. Or use Lua script for compound update.
113
+
114
+ ### Cache-aside without stampede protection
115
+ **Why it bites:** cache miss → 1000 requests all execute the expensive backing query simultaneously. DB overloaded, sometimes outage.
116
+ **Rule:** cache stampede mitigation — see `caching.md` (lock around fill, or probabilistic early refresh, or request coalescing).
117
+
118
+ ### Hot keys (single key getting all the traffic)
119
+ **Why it bites:** Redis Cluster shards by key. One hot key = one hot shard = no parallelism. CPU pinned.
120
+ **Rule:** if a key is read 10k+ times per second, replicate to local in-memory cache (LRU per app instance) with short TTL. Or shard the key (`counter:0`...`counter:9` and pick at random, sum on read).
121
+
122
+ ### Using `MULTI/EXEC` for "transactions" that aren't actually atomic logic
123
+ **Why it bites:** MULTI/EXEC doesn't roll back on error inside the block. Errors in commands inside MULTI return errors, but other commands still execute. People assume rollback semantics — there are none.
124
+ **Rule:** if you need conditional atomic logic, use Lua. MULTI/EXEC is just batching with no interleaved commands.
125
+
126
+ ## Decision Framework
127
+
128
+ | Need | Use | Avoid |
129
+ |---|---|---|
130
+ | Pure cache, restart-OK | String + TTL, RDB or no persist | AOF always-fsync |
131
+ | Session store | Hash + TTL, AOF every-sec | String JSON-blob |
132
+ | Counter | `INCR` (atomic) | GET → +1 → SET |
133
+ | Top-N | ZSET | sorted Set + manual trim |
134
+ | Approximate unique | HyperLogLog | Set with millions of entries |
135
+ | Queue, must not lose | Stream + consumer group | Pub/Sub or List |
136
+ | Rate limit | Token bucket Lua, or `redis-cell` | Hand-rolled INCR/EXPIRE with race |
137
+ | Distributed lock, single node | `SET NX EX` + Lua release | naive `SETNX` then `EXPIRE` (race) |
138
+ | Distributed lock, multi-node strong | Postgres advisory lock / etcd | Redlock for hard mutex |
139
+ | Real-time messaging, loss-tolerable | Pub/Sub | Streams (overkill) |
140
+ | Object with partial updates | Hash | JSON in String |
141
+
142
+ ## Cost Model
143
+
144
+ | Operation | Cost (single-node, cached, no I/O) |
145
+ |---|---|
146
+ | `GET` / `SET` | 50-100µs |
147
+ | `HSET` / `HGET` | 50-100µs |
148
+ | `INCR` | 50-100µs |
149
+ | Pipelined batch of 100 ops | 1-2ms total (~10-20µs each) |
150
+ | `KEYS *` on 1M keys | 100ms+ blocking |
151
+ | Lua script, 100 iterations | 0.5-1ms blocking |
152
+ | `SUBSCRIBE` notification | sub-ms when subscriber alive |
153
+ | `XADD` to stream | 100µs |
154
+
155
+ | Storage | Bytes |
156
+ |---|---|
157
+ | Empty Hash | ~80B |
158
+ | String value, 100B payload | ~150B with metadata |
159
+ | ZSET, 100 members | ~5KB |
160
+ | Stream entry, 100B payload | ~200B |
161
+
162
+ ## Red Flags in Diff
163
+
164
+ - `KEYS` / `FLUSHDB` / `FLUSHALL` — flag immediately, almost always wrong in app code.
165
+ - `SET` without `EX` / TTL — flag (cache key without expiry).
166
+ - `SUBSCRIBE` for work-style messaging where loss matters — flag (use Streams).
167
+ - `MULTI` / `EXEC` block doing logic that only works if all succeed — flag (Lua instead).
168
+ - New Redis client constructed inside a request handler — flag (should be shared).
169
+ - Lua script with unbounded loop or `KEYS *`-style iteration — flag.
170
+ - Distributed lock implemented as `SETNX` + separate `EXPIRE` → race condition window. Flag.
171
+ - Cache fill without stampede protection in hot path → flag.
172
+ - Single key receiving high write rate without sharding plan → flag (hot key).
173
+ - Storing JSON-stringified blob > 50KB in a String key → flag.
174
+ - Lock release `DEL key` without checking the value first → flag (releases someone else's lock).
175
+ - Rate limiter using `INCR` + separate `EXPIRE` first-time-key check → race window allows N+1 requests through.