@coralai/sps-cli 0.42.0 → 0.44.0

package/skills/backend/references/caching.md

@@ -0,0 +1,181 @@
+# Caching
+
+Rules, strategies, pitfalls. Cache-aside covers 90% of cases.
+
+## Cache-aside (lazy loading)
+
+Application checks cache first; on miss, loads from source and populates cache.
+
+```
+get(id):
+    v = cache.get(key(id))
+    if v is not None:
+        return v                      # hit
+    v = source.load(id)               # miss
+    if v is not None:
+        cache.set(key(id), v, ttl=5min)
+    return v
+```
+
+Pros: simple; stale data only appears on cached keys; source is authoritative.
+Cons: first reader after expiry pays full latency; risk of **cache stampede** when many readers miss together.
+
+## Write-through
+
+Application writes to cache AND source atomically (usually: write source first, then cache).
+
+```
+save(entity):
+    source.save(entity)
+    cache.set(key(entity.id), entity, ttl=5min)
+```
+
+Pros: cache is always fresh after a write.
+Cons: writes are slower; if cache write fails, you have stale data (decide: rollback, or fire-and-forget with expiry).
+
+## Write-behind (deferred)
+
+Application writes to cache; a background job flushes to source later. Rare — only for very high write volume and tolerance for delayed durability. Almost always the wrong choice; you're trading data loss risk for write throughput.
+
+## What to cache (and what NOT to)
+
+**Cache-friendly**:
+- Read-heavy, changes rarely (config, product catalog, user profile)
+- Expensive to compute (rendered HTML, aggregations, vector search)
+- Idempotent reads
+
+**Avoid caching**:
+- Per-user personalized data with high cardinality (cache hit rate too low)
+- Rapidly changing data (reconciliation cost > cache benefit)
+- Anything where staleness is a correctness bug (balances, seat availability)
+
+## TTL strategy
+
+Every cache entry must expire. No TTL = memory leak.
+
+| Data type | Starting TTL |
+|---|---|
+| Static config | 1–24 h |
+| User profile | 5–60 min |
+| Hot aggregation | 10 s – 5 min |
+| Computed render | minutes |
+| Feature flag eval | 30–60 s |
+
+Add a small random jitter (±10%) so entries don't all expire at the same instant → stampede.
+
+## Invalidation
+
+The second hardest problem in computing. Three approaches:
+
+1. **TTL only** — simple; tolerate staleness up to TTL. Default choice.
+2. **Explicit invalidation** — on write, delete the cache key. Works if your mutation paths are countable.
+   ```
+   save(user):
+       db.update(user)
+       cache.delete(key(user.id))
+   ```
+3. **Event-driven** — publish `UserUpdated`; subscribers invalidate their caches. Needed when many services cache the same entity.
+
+Don't try to *update* the cache on write in complex systems — delete instead and let the next read repopulate. Updates race; deletes don't.
+
+## Cache key design
+
+Stable, explicit, version-prefixed.
+
+```
+# ✅
+user:v2:{user_id}
+product:v1:{sku}:detail
+list:orders:v1:user={uid}:status=paid:cursor={c}
+
+# ❌
+u_123                  # ambiguous across services
+users:123:details      # no version
+${JSON.stringify(query)}  # fragile; order-dependent
+```
+
+Version prefix lets you deploy a new format without stampeding the old one; old keys simply age out.
+
+## Stampede protection
+
+When a hot key expires, many requests miss at once and pile onto the source. Two fixes:
+
+### Single-flight / coalescing
+
+In-process: at most one loader per key; concurrent callers wait for the same result.
+
+```
+load(key):
+    with singleFlight(key):
+        return source.load(key)
+```
+
+### Probabilistic early expiration (XFetch)
+
+Before the TTL, some fraction of readers voluntarily refresh.
+
+```
+get(key):
+    v, ttl_remaining = cache.get_with_ttl(key)
+    if v is None or should_refresh_early(ttl_remaining):
+        v = source.load(key)
+        cache.set(key, v, ttl=5min)
+    return v
+```
+
+## Negative caching
+
+Cache misses are expensive if they happen constantly (e.g., 404 lookups). Cache the absence too, with a short TTL.
+
+```
+get(id):
+    v = cache.get(key(id))
+    if v is MISSING_SENTINEL:
+        return None                 # known-not-found
+    if v is not None:
+        return v
+    v = source.load(id)
+    cache.set(key(id), v if v else MISSING_SENTINEL, ttl=30s)
+    return v
+```
+
+Short TTL — don't cache `None` for hours; the item may just have been created.
+
+## HTTP-level caching
+
+For public GET endpoints, let the HTTP layer cache. Free, correctly implemented, respected by CDNs.
+
+```
+Cache-Control: public, max-age=300, s-maxage=600, stale-while-revalidate=60
+ETag: "abc123"
+```
+
+- `max-age`: browser/client
+- `s-maxage`: shared caches (CDN)
+- `stale-while-revalidate`: serve stale while refreshing in the background
+- `ETag` + `If-None-Match`: 304 responses save bandwidth
+
+## Local (in-process) cache vs distributed
+
+| | Local (in-process) | Distributed (Redis, Memcached) |
+|---|---|---|
+| Latency | Nanoseconds | ~1 ms |
+| Consistency across instances | No — each pod has its own | Yes |
+| Size | Limited to process memory | Limited to cluster |
+| Eviction | LRU, LFU | LRU, LFU, TTL |
+| Cost | Free | Infra + ops |
+| Invalidation | Hard across pods | One call |
+
+Use local for small hot data; distributed for shared state. Don't mix carelessly — a per-pod cache that's supposed to be consistent will drift.
+
+## Anti-patterns
+
+| Anti-pattern | Why bad |
+|---|---|
+| No TTL anywhere | Memory leak; stale data forever |
+| Caching mutable objects by reference | Next reader mutates the cached copy |
+| Caching per-user data with high cardinality | Low hit rate; wastes memory |
+| Cache key includes a timestamp that changes every request | Every request is a miss |
+| Serializing cache writes into the request path without timeout | Cache outage → requests hang |
+| Reading cache without a fallback path | Cache is a dependency; treat it as optional |
+| Storing secrets in shared cache | Secret sprawl across cluster |

package/skills/backend/references/data-access.md

@@ -0,0 +1,173 @@
+# Data Access
+
+Transactions, queries, migrations, connection pooling. Language-neutral patterns.
+
+## N+1 queries — the universal killer
+
+The single most common backend performance bug.
+
+```
+# ❌ N+1
+orders = orderRepo.findAll()              # 1 query
+for order in orders:
+    order.user = userRepo.find(order.userId)   # N queries
+
+# ✅ Batch fetch
+orders = orderRepo.findAll()
+userIds = unique(o.userId for o in orders)
+users = userRepo.findByIds(userIds)        # 1 query
+userMap = { u.id: u for u in users }
+for o in orders:
+    o.user = userMap[o.userId]
+
+# ✅ Join (if the ORM supports eager loading)
+orders = orderRepo.findAll(include=['user'])
+```
+
+Detect early: log every query in test mode; assert query count on hot paths.
+
+## Select only what you need
+
+Wide `SELECT *` costs bandwidth, memory, and breaks when the schema changes.
+
+```
+# ❌
+SELECT * FROM users WHERE active = true
+
+# ✅
+SELECT id, email, name FROM users WHERE active = true
+```
+
+## Indexes
+
+An index is a write-time tax for a read-time refund. Worth it on columns used in WHERE, JOIN, ORDER BY of hot queries.
+
+```
+# Common first indexes
+CREATE INDEX idx_orders_user_id     ON orders(user_id);
+CREATE INDEX idx_orders_status      ON orders(status) WHERE status = 'pending';  -- partial
+CREATE INDEX idx_users_email_lower  ON users (LOWER(email));                      -- expression
+```
+
+Rules:
+- Read the query plan. Don't guess.
+- Composite index order matters: `(user_id, created_at)` helps `WHERE user_id = ? ORDER BY created_at`, not the reverse.
+- Every index slows writes. More indexes ≠ faster system.
+
+## Transactions
+
+One business operation = one transaction. Cross the boundary at the use case, not inside a repository.
+
+```
+unitOfWork.begin()
+try:
+    order = orderRepo.save(newOrder)
+    inventoryRepo.decrement(order.items)
+    eventBus.publish(OrderPlaced(order.id))
+    unitOfWork.commit()
+except:
+    unitOfWork.rollback()
+    raise
+```
+
+Isolation levels:
+- **READ COMMITTED**: default on most DBs, fine for most workloads
+- **REPEATABLE READ**: if you read the same row twice within a transaction and want consistency
+- **SERIALIZABLE**: correctness over throughput; expect retries
+
+Keep transactions short. Long-running transactions hold locks and block everyone.
+
+## Connection pooling
+
+Every real backend uses a pool, not per-request connections. DBs limit max connections (Postgres default ~100); without pooling, a traffic spike exhausts the DB.
+
+| Pool param | Starting value | Notes |
+|---|---|---|
+| min idle | 2–5 | Warm connections for low traffic |
+| max size | (DB max ÷ replicas) − safety margin | e.g., 100 ÷ 4 = 25 per instance, then leave room |
+| connection timeout | 2–5 s | Fail fast if pool is saturated |
+| idle timeout | 30 s – 5 min | Recycle stale connections |
+| max lifetime | 30 min | Force re-resolve DNS, rotate creds |
+
+Serverless + traditional DB: use a pooler (PgBouncer, RDS Proxy) — each cold lambda can't open its own pool.
+
+## Read replicas
+
+Route reads to replicas, writes to primary. Beware of replication lag:
+
+```
+user.save(newEmail)            # primary
+user = user.reload()           # replica — may still show old email
+```
+
+Common fix: stick to primary for N seconds after a write, or read-your-writes from primary only.
+
+## Migrations
+
+Every schema change is a migration file, checked in, applied in CI/CD, reversible where possible.
+
+Rules:
+- **Never edit a merged migration.** Write a new one.
+- **Additive first, destructive later.** Add the new column → backfill → switch code → drop the old column (separate deploys).
+- **Index creation on a hot table**: use `CREATE INDEX CONCURRENTLY` (Postgres) so you don't lock the table.
+- **Default values**: adding a `NOT NULL` column with a default on a big table can rewrite the whole table. In Postgres 11+, adding `DEFAULT` is metadata-only; in older DBs, do `ADD NULLABLE → backfill → SET NOT NULL`.
+
+## Soft deletes
+
+Don't add `deleted_at` everywhere by default. It creates a silent contract that every query must filter. Use it when:
+- You genuinely need to recover records, and
+- You accept the cognitive tax on every query.
+
+Prefer hard deletes + an `audit_log` / `events` table if you only need history.
+
+## Bulk operations
+
+One round-trip per row kills throughput. Use batch APIs.
+
+```
+# ❌
+for row in 10_000_rows:
+    db.insert(row)
+
+# ✅
+db.bulk_insert(10_000_rows)            # one statement
+# or
+db.copy_from(csv_buffer)               # Postgres COPY, fastest
+```
+
+On upserts, use the DB's native construct (`INSERT ... ON CONFLICT`, `MERGE`, `INSERT ... ON DUPLICATE KEY UPDATE`), not read-then-update in app code.
+
+## Pagination queries
+
+Offset pagination gets slow on large tables because the DB still walks the skipped rows.
+
+```
+# ❌ Slow on page 10 000
+SELECT * FROM events ORDER BY id LIMIT 50 OFFSET 500000
+
+# ✅ Keyset / cursor
+SELECT * FROM events WHERE id > :last_id ORDER BY id LIMIT 50
+```
+
+Keyset pagination is O(log n); offset is O(offset + limit).
+
+## NoSQL quick notes
+
+- **Key-value (Redis, DynamoDB)**: design the key; scan queries are evil.
+- **Document (Mongo)**: embed what you always read together; reference what you sometimes read separately.
+- **Wide column (Cassandra, Bigtable)**: query patterns decide the schema, not the other way around.
+- **Graph (Neo4j)**: use when the traversal depth would be painful in SQL.
+
+Rule: pick the store that matches the access pattern. Don't use Mongo because "it's flexible"; flexibility defers modeling pain, it doesn't erase it.
+
+## Anti-patterns
+
+| Anti-pattern | Why bad | Fix |
+|---|---|---|
+| Queries in a loop | N+1; one slow endpoint tanks the DB | Batch / join / cache |
+| No timeout on DB calls | A single slow query hangs threads / pool | Set statement timeout |
+| `SELECT *` in hot code | Brittle, wasteful | List columns |
+| Business logic in stored procedures "for speed" | Hard to test, version, review | Keep logic in code; use SQL for set operations |
+| Multiple orthogonal indexes on the same table | Slow writes, bloated storage | Review `pg_stat_user_indexes`; drop unused |
+| Editing an applied migration | Divergent envs | New migration |
+| Schema changes without a rollback plan | Stuck deploys | Reversible migrations or documented forward-only fix |

package/skills/backend/references/layering.md

@@ -0,0 +1,181 @@
+# Layering
+
+Split the code so business rules don't depend on the framework, the database, or the network. Hexagonal / clean architecture in practical form.
+
+## The four layers
+
+```
+┌──────────────────────────────────────────────┐
+│  Delivery (HTTP handler, CLI, gRPC, worker)  │  — framework-aware
+├──────────────────────────────────────────────┤
+│  Application (use cases, orchestration)      │  — framework-ignorant
+├──────────────────────────────────────────────┤
+│  Domain (entities, value objects, rules)     │  — pure
+├──────────────────────────────────────────────┤
+│  Infrastructure (DB, cache, HTTP clients)    │  — implements domain ports
+└──────────────────────────────────────────────┘
+```
+
+Dependency direction: **only inward**. Delivery → Application → Domain. Infrastructure implements interfaces owned by the inner layers.
+
+If your domain imports an HTTP framework, a DB driver, or a cache client, the layering is broken.
+
+## Minimal layer roles
+
+| Layer | Contains | Does NOT contain |
+|---|---|---|
+| Delivery | Request parsing, auth check, calls a use case, maps result to response | Business rules, DB queries |
+| Application | Use case orchestration, transaction boundaries, calls repositories and services | SQL, HTTP, JSON parsing |
+| Domain | Entities, value objects, invariants, domain events | I/O, frameworks |
+| Infrastructure | Repository impls, HTTP client impls, message queue impls | Business decisions |
+
+## Ports and adapters
+
+The domain declares a **port** (interface). Infrastructure provides an **adapter** (implementation).
+
+```
+Domain declares (port):
+  interface UserRepository
+      findById(id) -> User | null
+      save(user) -> void
+
+Infrastructure provides (adapter):
+  PostgresUserRepository    implements UserRepository
+  InMemoryUserRepository    implements UserRepository  (for tests)
+  RedisUserRepository       implements UserRepository  (cache-aside)
+```
+
+Rule: the adapter file imports the port. The port file never imports any adapter.
+
+## Use case pattern
+
+A use case is one method, one transaction boundary, one business intent.
+
+```
+class CreateOrder:
+    deps: OrderRepository, UserRepository, PaymentGateway, EventBus
+
+    execute(cmd: CreateOrderCommand) -> OrderId:
+        user = userRepository.findById(cmd.userId)
+        if not user:             raise UserNotFound
+        if not user.canOrder():  raise UserCannotOrder
+
+        order = Order.create(user, cmd.items)     # domain rules
+        paymentRepository.authorize(order)        # infra
+        orderRepository.save(order)               # infra
+        eventBus.publish(OrderCreated(order.id))  # infra
+
+        return order.id
+```
+
+Delivery turns an HTTP request into `CreateOrderCommand`, calls `execute`, turns the result into a response. That's it.
+
+## Repository pattern
+
+Collect the DB operations for one aggregate behind one interface.
+
+```
+interface OrderRepository:
+    findById(id)      -> Order | null
+    findByUser(uid)   -> list[Order]
+    save(order)       -> void
+    delete(id)        -> void
+```
+
+Rules:
+- Repositories return **domain objects**, not DB rows.
+- Queries that cross aggregates (reporting, analytics) do NOT belong in a repository; put them in a dedicated `Queries` / `ReadModel` interface.
+- Avoid growing `findByXAndYAndZ` explosions — those signal you need a query object or a read model.
+
+## Service vs domain vs use case
+
+People confuse these. Rough guide:
+
+| Name | Lives in | Contains |
+|---|---|---|
+| Entity / Aggregate | Domain | State + invariants + rules that depend ONLY on that state |
+| Domain Service | Domain | Rules that span multiple aggregates but are still pure |
+| Use Case / Application Service | Application | Orchestration: load, decide, persist, publish |
+| Gateway / Client | Infrastructure | Talks to the outside world (HTTP, DB, queue) |
+
+If you have a `FooService` that does both business rules and DB calls, split it.
+
+## Dependency injection, without magic
+
+Pass dependencies in as constructor args. Don't pull them from globals.
+
+```
+# Good
+CreateOrder(orderRepo, userRepo, paymentGateway, eventBus)
+
+# Bad
+class CreateOrder:
+    def execute():
+        order_repo = Container.get("OrderRepository")   # hidden dep
+```
+
+Any framework DI container that ends up manipulating constructor signatures reflectively becomes impossible to reason about. Prefer explicit wiring in a composition root.
+
+## Composition root
+
+One file where everything is wired up.
+
+```
+# main / bootstrap
+db        = Postgres(config.url)
+cache     = Redis(config.redis_url)
+eventBus  = Kafka(config.brokers)
+
+userRepo   = PostgresUserRepository(db)
+orderRepo  = CachedOrderRepository(
+                PostgresOrderRepository(db),
+                cache,
+             )
+
+createOrder = CreateOrder(orderRepo, userRepo, PaymentStripe(config.key), eventBus)
+
+app.register("POST /orders", lambda req: http_create_order(req, createOrder))
+```
+
+All layering choices become visible in this one file.
+
+## Transaction boundary
+
+The use case decides where the transaction starts and ends, not the repository.
+
+```
+class TransferMoney:
+    execute(cmd):
+        with unitOfWork.begin():
+            src = accountRepo.findById(cmd.fromId)
+            dst = accountRepo.findById(cmd.toId)
+            src.withdraw(cmd.amount)
+            dst.deposit(cmd.amount)
+            accountRepo.save(src)
+            accountRepo.save(dst)
+        # commit happens here; rollback on exception
+```
+
+One transaction per use case, not per repository call. If a use case needs multiple transactions, it's probably two use cases.
+
+## Anti-patterns
+
+| Anti-pattern | Why bad | Fix |
+|---|---|---|
+| Framework objects (HTTP request/response) inside the domain | Couples domain to HTTP | Parse at delivery, pass plain command |
+| Repository returns a DB row | Leaks schema upward | Map to domain object at the edge |
+| Controller calls the DB directly | Skips domain rules | Every write goes through a use case |
+| ORM entities ARE the domain entities | Can't change storage without rewriting rules | Separate persistence model from domain model |
+| Static "Service" class with 40 unrelated methods | No cohesion; everything imports everything | One use case per class |
+| Domain event published before persistence succeeds | Consumers act on data that doesn't exist | Publish after commit, or use transactional outbox |
+
+## Don't over-engineer
+
+A 500-line CRUD service doesn't need four layers, a DI container, and a port-adapter diagram. Start simple:
+
+```
+# Acceptable for small services
+handler -> repository -> db
+```
+
+Introduce the extra seams **when you feel the pain**: when tests get hard, when the DB needs replacing, when rules start repeating across endpoints. Layering is a response to complexity, not a prerequisite for it.