vanara 0.1.0 → 0.3.0

package/catalog/skills/api-pagination/references/counting-and-totals.md

@@ -0,0 +1,41 @@
+# Counting and Totals
+
+`COUNT(*)` is the hidden cost center of pagination. On a large Postgres table it is a full index
+or heap scan — routinely slower than fetching the page itself — and the number is stale the moment
+it is computed.
+
+## Decision table
+
+| Product need | Implementation |
+|---|---|
+| "Load more" / infinite scroll | No count at all — `has_more` from the +1-row trick |
+| "~12,300 results" | Estimated count (below), clearly rounded in the UI |
+| Exact count on small filtered sets (<10k rows) | Real `COUNT(*)`, acceptable |
+| Exact count on big tables, hard requirement | Maintained counter (trigger or async rollup) |
+
+## Estimated counts (Postgres)
+
+```sql
+-- Table-wide estimate: instant, from the planner's statistics
+SELECT reltuples::bigint FROM pg_class WHERE relname = 'items';
+
+-- Filtered estimate: run the query through EXPLAIN and parse rows=
+EXPLAIN (FORMAT JSON) SELECT 1 FROM items WHERE status = 'active';
+-- -> Plan.["Plan Rows"]
+```
+
+Estimates drift after bulk writes until autovacuum/analyze runs — fine for "~N results", wrong for
+billing or quotas.
+
+## Maintained counters
+
+A trigger-updated counter table gives exact totals at O(1) read cost, but serializes writes on hot
+rows (every insert contends on the same counter row). For high write rates, shard the counter
+(N rows, sum on read) or roll up asynchronously and accept seconds of lag. Never compute quotas
+or invoices from pagination counts — those need their own transactional bookkeeping.
+
+## The `total_pages` trap
+
+Exposing `total_pages` invites clients to build page links (`?page=812`), which drags you back to
+deep offsets. If the UI needs a jump-to-end affordance, offer a reversed sort (`?order=asc`)
+instead — last page becomes first page, at first-page cost.

package/catalog/skills/api-pagination/references/cursor-encoding.md

@@ -0,0 +1,34 @@
+# Cursor Encoding
+
+A cursor must carry enough to rebuild the keyset `WHERE` clause and nothing more: the sort-key
+values, the direction, and a version byte. It must be opaque, tamper-evident (public APIs), and
+survive encoding changes across deploys.
+
+## Layout
+
+```
+v1 . base64url( json({ k: ["2026-07-01T12:00:00.000Z", 4711], d: "desc" }) ) . hmac10
+```
+
+- **`v1` prefix** — bump when the payload shape changes; reject unknown versions with 400
+  `CURSOR_INVALID`, which clients treat as "restart iteration".
+- **base64url**, not base64 — cursors travel in query strings; `+/=` break naive clients.
+- **HMAC (first 10 bytes, hex)** over `version + payload` with a server-side key. Not secrecy —
+  integrity. Without it, clients will reverse-engineer the payload and build their own cursors,
+  and your encoding becomes an accidental public API you can never change.
+
+## Rules
+
+- Timestamps in the payload must be **full precision** (milliseconds or better). Truncated seconds
+  re-introduce the duplicate-key ambiguity the tiebreaker was meant to solve.
+- Never embed user IDs, filters, or auth state in the cursor — re-derive those from the request.
+  A cursor pasted into another user's session must not leak or authorize anything.
+- Cursors may expire (e.g., if the underlying snapshot is gone). Expired cursor → 400 with a
+  distinct `CURSOR_EXPIRED` code so clients restart rather than retry.
+- Treat decode failures as client errors (400), never 500 — fuzzers will send garbage.
+
+## Anti-pattern: the "readable" cursor
+
+`?after_id=4711&after_ts=...` looks friendlier but freezes your sort semantics forever: adding a
+tiebreaker, changing direction defaults, or switching the sort column becomes a breaking change.
+Opaque tokens keep the contract at "pass back what we gave you".

package/catalog/skills/api-pagination/scripts/check-cursor.mjs

@@ -0,0 +1,51 @@
+#!/usr/bin/env node
+// Runnable check for the opaque-cursor contract (encode -> decode -> tamper -> version).
+// Usage: node scripts/check-cursor.mjs   (exits 0 on pass, 1 on first failure)
+import { createHmac } from 'node:crypto';
+
+const KEY = 'demo-key-rotate-me';
+const VERSION = 'v1';
+
+const b64u = (buf) => Buffer.from(buf).toString('base64url');
+const sign = (s) => createHmac('sha256', KEY).update(s).digest('hex').slice(0, 20);
+
+export function encodeCursor(keyValues, direction = 'desc') {
+  const payload = b64u(JSON.stringify({ k: keyValues, d: direction }));
+  return `${VERSION}.${payload}.${sign(VERSION + payload)}`;
+}
+
+export function decodeCursor(token) {
+  const [version, payload, mac] = String(token).split('.');
+  if (version !== VERSION) return { error: 'CURSOR_INVALID' }; // unknown version
+  if (!payload || sign(version + payload) !== mac) return { error: 'CURSOR_INVALID' };
+  try {
+    const { k, d } = JSON.parse(Buffer.from(payload, 'base64url').toString('utf8'));
+    if (!Array.isArray(k) || !['asc', 'desc'].includes(d)) return { error: 'CURSOR_INVALID' };
+    return { keyValues: k, direction: d };
+  } catch {
+    return { error: 'CURSOR_INVALID' };
+  }
+}
+
+let failed = 0;
+const check = (name, ok) => {
+  console.log(`${ok ? 'PASS' : 'FAIL'}  ${name}`);
+  if (!ok) failed = 1;
+};
+
+// Round trip preserves full timestamp precision and the tiebreaker.
+const token = encodeCursor(['2026-07-01T12:00:00.412Z', 4711]);
+const out = decodeCursor(token);
+check('round-trip', out.keyValues?.[1] === 4711 && out.keyValues?.[0].endsWith('.412Z'));
+check('url-safe (no + / =)', !/[+/=]/.test(token));
+
+// Tampered payload must be rejected, not half-parsed.
+const [v, p, m] = token.split('.');
+check('tamper detected', decodeCursor(`${v}.${p.slice(0, -2)}xx.${m}`).error === 'CURSOR_INVALID');
+
+// Unknown version and garbage are client errors, never crashes.
+check('unknown version rejected', decodeCursor(`v9.${p}.${m}`).error === 'CURSOR_INVALID');
+check('garbage rejected', decodeCursor('not-a-cursor').error === 'CURSOR_INVALID');
+check('empty rejected', decodeCursor('').error === 'CURSOR_INVALID');
+
+process.exit(failed);

package/catalog/skills/caching-strategies/SKILL.md

@@ -0,0 +1,180 @@
+---
+name: caching-strategies
+description: Deep reference for caching — what to cache, cache-aside vs read/write-through/write-behind, TTLs with jitter, eviction (LRU/LFU/FIFO), invalidation, and surviving stampedes (thundering herd / dogpile). Worked examples and a runnable jitter check.
+type: skill
+version: 2.0.0
+updated: 2026-06-29
+---
+# Caching Strategies
+
+Caching is the cheapest order-of-magnitude performance win available, and the fastest way to serve
+*confidently wrong* data. The hard part was never reading from a cache — it's **invalidation**,
+**consistency under concurrency**, and **what happens the moment the cache is cold or wrong**. This
+skill is the deep reference: the patterns, the decisions, the trade-offs, and the failure modes that
+page you at 3am. Heavy detail lives in `references/`; copy-paste material in `examples/`; a runnable
+check in `scripts/`.
+
+## Mental model
+
+A cache is a **bet**: you trade memory and a correctness risk for latency and load reduction. Every
+caching decision is answering four questions, in order:
+
+| Question | What it decides |
+|---|---|
+| What is hot and tolerant of staleness? | *whether* to cache at all |
+| How does data get *into* the cache? | the **read/write pattern** (cache-aside, write-through…) |
+| How does stale data get *out*? | **TTL + invalidation** |
+| What happens on a miss storm or a dead cache? | **stampede control + fallback** |
+
+If you can't answer the third and fourth questions before you ship, you don't have a caching
+strategy — you have a future incident. Decide the invalidation plan *first*.
+
+## 1. What to cache
+
+Cache data that is **read-often, expensive to produce, and tolerant of some staleness**. Good
+candidates: rendered product pages, the result of an expensive aggregation, a third-party API
+response, a permission lookup hit on every request. Bad candidates: a user's current account balance,
+a one-time-read report, anything where serving a 30-second-old value is a correctness or compliance
+bug.
+
+Quantify it before caching. The benefit is roughly `hitRate × costPerMiss`. A 50% hit rate on a 200ms
+query is enormous; a 50% hit rate on a 2ms query is noise that you've paid for with a consistency
+risk. Measure hit rate in production — a cache below ~80% hit rate for point lookups usually means the
+key space is too sparse or the TTL is too short.
+
+## 2. Read & write patterns
+
+The pattern is *how data flows between your app, the cache, and the source of truth*. Pick one
+deliberately per dataset.
+
+- **Cache-aside (lazy loading)** — the default. App checks cache; on miss, loads from the source and
+  populates the cache. Simple, resilient (a dead cache just means slow, not broken), but the first
+  read of every key is a miss.
+- **Read-through** — the cache library itself loads from the source on a miss. Same shape as
+  cache-aside but the loading logic lives behind the cache abstraction.
+- **Write-through** — writes go to cache *and* source synchronously. Cache is always fresh; writes
+  are slower.
+- **Write-behind (write-back)** — writes hit the cache and are flushed to the source asynchronously.
+  Fast writes, but you risk data loss on a crash before flush.
+
+Full comparison with sequence diagrams and when each one bites:
+[references/cache-patterns.md](references/cache-patterns.md).
+
+```js
+// Cache-aside — the 90% pattern. Note the explicit TTL and miss-population.
+async function getProduct(id, { cache, db, ttl = 300 }) {
+  const key = `product:${id}`;
+  const hit = await cache.get(key);
+  if (hit !== null && hit !== undefined) return JSON.parse(hit);   // cache hit
+
+  const product = await db.getProduct(id);                          // miss -> source of truth
+  if (product) await cache.set(key, JSON.stringify(product), { ex: ttl });
+  return product;
+}
+```
+
+## 3. TTL, eviction & jitter
+
+A cache is finite, so entries leave in two ways: **TTL expiry** (time-based) and **eviction** (space
+pressure). Tune both.
+
+- **TTL** bounds staleness. Short TTL = fresher + more misses; long TTL = fewer misses + staler data.
+- **Eviction policy** decides who gets dropped when memory is full: **LRU** (evict least recently
+  used — the sane default), **LFU** (least frequently used — better for skewed popularity), **FIFO**
+  (rarely what you want). See `references/eviction-and-ttl.md`.
+- **TTL jitter is mandatory at scale.** If you populate 10,000 keys in a batch with a fixed 300s TTL,
+  they all expire in the same second and stampede the database together. Add randomness so expiries
+  spread out.
+
+```js
+// Add ±10% jitter so a batch of keys never expires in lockstep.
+function jitteredTtl(baseSeconds, jitterRatio = 0.1) {
+  const delta = baseSeconds * jitterRatio;
+  const offset = (Math.random() * 2 - 1) * delta;          // uniform in [-delta, +delta]
+  return Math.max(1, Math.round(baseSeconds + offset));
+}
+// jitteredTtl(300) -> ~270..330, never the same instant across a batch
+```
+
+`scripts/ttl-jitter.mjs` is a runnable, self-testing version of this function.
+
+## 4. Invalidation
+
+> "There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton
+
+TTL alone is *eventual* freshness. When correctness matters, invalidate **on write**:
+
+- **Explicit bust** — on update, delete the key (`cache.del(key)`) so the next read repopulates. With
+  cache-aside this is the standard move. Beware the race: delete *after* the DB commits, not before.
+- **Write-through** — keep the cache correct on the write path so there's nothing to bust.
+- **Versioned keys** — embed a version/etag in the key (`product:42:v7`); bumping the version
+  instantly orphans all old entries without a scan.
+
+The subtle bugs (delete-before-commit races, distributed invalidation lag, cache-vs-DB ordering) are
+covered in `references/invalidation-and-stampede.md`.
+
+## 5. Stampede control (thundering herd / dogpile)
+
+When a hot key expires, every concurrent request misses at once and they *all* hit the source —
+the **thundering herd**. A single popular key can take down a database this way. Mitigations:
+
+- **Per-key locking (single-flight):** the first miss acquires a lock and recomputes; everyone else
+  waits for the result instead of also recomputing.
+- **Stale-while-revalidate:** serve the slightly-stale value while one background task refreshes it.
+- **Early/probabilistic recomputation:** refresh a key *before* it expires, with probability rising
+  as expiry approaches, so the herd never forms.
+- **TTL jitter:** prevents *synchronized* expiry in the first place (see §3).
+
+Detailed algorithms and pseudocode: `references/invalidation-and-stampede.md`.
+
+## 6. HTTP caching
+
+Not all caching is a key-value store. The HTTP layer gives you free caching via `Cache-Control`,
+`ETag`, and `Last-Modified` — push work to the browser and CDN before it ever reaches your app. See
+`examples/http-cache-headers.md` for a `Cache-Control` / `ETag` / `304 Not Modified` walkthrough.
+
+## Common pitfalls (failure modes & edge cases)
+
+- **No invalidation plan** — TTL-only freshness shipped where correctness was required. The classic
+  "I updated it but it's still showing the old price for 5 minutes" bug.
+- **Synchronized expiry** — fixed TTL on a batch of keys → herd at expiry. Always jitter.
+- **Caching nulls badly** — either you don't cache "not found" (so every miss re-queries a row that
+  doesn't exist — **cache penetration**) or you cache it forever (so a later insert is invisible).
+  Cache negatives with a *short* TTL.
+- **Unbounded cache / no eviction** — an in-process map with no size cap is a memory leak that ends in
+  an OOM. Always bound size and set an eviction policy.
+- **Delete-before-commit race** — busting the key before the DB transaction commits lets a concurrent
+  read repopulate the cache with the *old* value. Bust after commit.
+- **Stale-on-error** — refresh logic that overwrites a good cached value with an error/empty result.
+  On a source failure, keep serving stale rather than caching the failure.
+- **Cache key collisions** — forgetting a discriminator (locale, tenant, auth scope) in the key, so
+  user A sees user B's cached personalized data. A security bug, not just a correctness one.
+- **Thundering herd ignored** — works fine until the one hot key expires under load.
+
+## When NOT to cache / trade-offs
+
+Caching is not free — it adds a consistency risk, operational surface, and a second source of truth
+to reason about. Skip or reconsider it when:
+
+- **Strong consistency is required** (balances, inventory counts, anything legally/financially
+  binding) — a stale read is a bug, not a latency win.
+- **Low read-to-write ratio** — if data changes nearly as often as it's read, you'll spend more effort
+  invalidating than you save on reads.
+- **The source is already fast** — caching a 1ms lookup buys nothing and adds a failure mode.
+- **Per-user, read-once data** — the entry is evicted before it's ever reused.
+
+When you *do* cache, name the trade-off explicitly: you are choosing **availability and latency over
+strict consistency** (an AP-leaning choice). Make the maximum staleness window a documented number,
+not an accident of the TTL you happened to pick.
+
+## Files in this package
+
+- `references/cache-patterns.md` — cache-aside / read-through / write-through / write-behind, with flows
+- `references/eviction-and-ttl.md` — LRU / LFU / FIFO, TTL tuning, jitter, negative caching
+- `references/invalidation-and-stampede.md` — invalidation strategies, thundering herd, dogpile locks
+- `examples/redis-cache-aside.js` — runnable cache-aside + single-flight against an in-memory fake
+- `examples/http-cache-headers.md` — `Cache-Control`, `ETag`, and `304` revalidation
+- `scripts/ttl-jitter.mjs` — runnable jittered-TTL helper with `--selftest`
+
+Pairs with the `rest-api-design` skill (HTTP cache headers), the `database-scaling` skill (caching as a
+read-load shield), and the `performance-engineer` agent.

package/catalog/skills/caching-strategies/examples/http-cache-headers.md

@@ -0,0 +1,82 @@
+# HTTP Caching: Cache-Control, ETag, and 304
+
+The cheapest cache is the one you never run: push caching to the browser and CDN with HTTP headers. No
+key-value store, no invalidation code — the protocol does it.
+
+## Cache-Control
+
+The primary knob. Tells browsers and shared caches (CDNs, proxies) how long a response is fresh.
+
+```http
+# Public, cacheable by CDNs, fresh for 1 hour, then revalidate.
+Cache-Control: public, max-age=3600, must-revalidate
+
+# Per-user/private data: only the browser may cache, never a shared CDN.
+Cache-Control: private, max-age=0, no-cache
+
+# Truly sensitive (never store): 
+Cache-Control: no-store
+
+# Immutable, content-hashed asset (app.9f3a2.js) — cache for a year, never revalidate.
+Cache-Control: public, max-age=31536000, immutable
+```
+
+Key directives:
+- `max-age=N` — fresh for N seconds.
+- `s-maxage=N` — overrides `max-age` for *shared* caches (CDN) only.
+- `public` / `private` — may a shared cache store it?
+- `no-cache` — store it, but **revalidate** before each use (not "don't cache").
+- `no-store` — never write it to any cache.
+- `stale-while-revalidate=N` — serve stale up to N seconds while revalidating in the background.
+- `immutable` — the body will never change at this URL; skip revalidation entirely.
+
+## ETag + conditional requests (revalidation)
+
+An `ETag` is an opaque fingerprint of the response body. The client sends it back to ask "still
+valid?", and the server answers `304 Not Modified` with *no body* if unchanged — saving bandwidth.
+
+```http
+# First response
+HTTP/1.1 200 OK
+Cache-Control: no-cache
+ETag: "v7-9f3a2c"
+Content-Type: application/json
+
+{ "id": 42, "name": "Product 42", "price": 1999 }
+```
+
+```http
+# Client revalidates with the stored validator
+GET /products/42 HTTP/1.1
+If-None-Match: "v7-9f3a2c"
+```
+
+```http
+# Server: unchanged -> tiny response, no body
+HTTP/1.1 304 Not Modified
+ETag: "v7-9f3a2c"
+Cache-Control: no-cache
+```
+
+`Last-Modified` + `If-Modified-Since` is the date-based equivalent; `ETag` is preferred because it's
+exact (sub-second changes, content-based) rather than 1-second-granular.
+
+## Picking a strategy
+
+| Asset | Headers |
+|---|---|
+| Content-hashed JS/CSS (`app.9f3a2.js`) | `public, max-age=31536000, immutable` |
+| HTML shell | `no-cache` + `ETag` (always revalidate, cheap when unchanged) |
+| Public API list, slow-changing | `public, max-age=60, stale-while-revalidate=300` |
+| Per-user dashboard JSON | `private, no-cache` + `ETag` |
+| Auth tokens, payment pages | `no-store` |
+
+## Pitfalls
+
+- `no-cache` does **not** mean "don't cache" — it means "revalidate every time". Use `no-store` to
+  forbid storage.
+- Forgetting `private` on personalized responses lets a CDN serve user A's data to user B.
+- Long `max-age` on a URL whose content changes = stale clients with no way to bust. Use content-hashed
+  filenames (`immutable`) so the URL changes when the content does.
+- `ETag` mismatches across a load-balanced fleet (e.g. inode-based ETags) cause needless re-downloads;
+  use a deterministic, content-derived ETag.

package/catalog/skills/caching-strategies/examples/redis-cache-aside.js

@@ -0,0 +1,110 @@
+// Cache-aside with single-flight (stampede protection), demonstrated against an
+// in-memory fake that mimics the slice of the Redis API we use (get/set/del +
+// SET NX for locking). Swap `new FakeRedis()` for a real client (e.g. ioredis)
+// and the logic is unchanged.
+//
+// Run it directly with Node (built-ins only; ES module syntax):
+//   node redis-cache-aside.js
+//
+// It fires 50 concurrent reads of one cold key and shows that the expensive
+// "DB" load runs exactly once — the herd is collapsed by the per-key lock.
+
+import { fileURLToPath } from 'node:url';
+import process from 'node:process';
+
+// --- tiny TTL helper (mirrors scripts/ttl-jitter.mjs) ------------------------
+function jitteredTtl(baseSeconds, jitterRatio = 0.1) {
+  const delta = baseSeconds * jitterRatio;
+  const offset = (Math.random() * 2 - 1) * delta;
+  return Math.max(1, Math.round(baseSeconds + offset));
+}
+
+// --- minimal in-memory stand-in for Redis -----------------------------------
+class FakeRedis {
+  constructor() { this.store = new Map(); } // key -> { value, expiresAt }
+  async get(key) {
+    const e = this.store.get(key);
+    if (!e) return null;
+    if (e.expiresAt && e.expiresAt < Date.now()) { this.store.delete(key); return null; }
+    return e.value;
+  }
+  async set(key, value, { ex } = {}) {
+    this.store.set(key, { value, expiresAt: ex ? Date.now() + ex * 1000 : 0 });
+    return 'OK';
+  }
+  // SET key val NX EX ttl  -> returns 'OK' only if the key did not exist (lock).
+  // Atomic check-and-set: in real Redis this is one command; here we do the
+  // check + write synchronously (no await between) so concurrent callers can't
+  // both win the lock.
+  async setNx(key, value, ttlSeconds) {
+    const e = this.store.get(key);
+    const live = e && (!e.expiresAt || e.expiresAt >= Date.now());
+    if (live) return null;
+    this.store.set(key, { value, expiresAt: ttlSeconds ? Date.now() + ttlSeconds * 1000 : 0 });
+    return 'OK';
+  }
+  async del(key) { return this.store.delete(key) ? 1 : 0; }
+}
+
+const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
+
+// --- the expensive source of truth (counts how often it actually runs) ------
+let dbCalls = 0;
+async function loadProductFromDb(id) {
+  dbCalls += 1;
+  await sleep(50); // pretend this is a slow query
+  return { id, name: `Product ${id}`, price: 1999 };
+}
+
+// --- cache-aside read with single-flight lock -------------------------------
+async function getProduct(id, { cache, ttl = 300, lockTtl = 5, maxWaitMs = 2000 }) {
+  const key = `product:${id}`;
+  const lockKey = `lock:${key}`;
+
+  const hit = await cache.get(key);
+  if (hit !== null) return JSON.parse(hit);
+
+  // Try to become the single loader for this key.
+  const gotLock = await cache.setNx(lockKey, '1', lockTtl);
+  if (gotLock === 'OK') {
+    try {
+      const product = await loadProductFromDb(id);
+      await cache.set(key, JSON.stringify(product), { ex: jitteredTtl(ttl) });
+      return product;
+    } finally {
+      await cache.del(lockKey);
+    }
+  }
+
+  // Follower: wait for the winner to populate the cache, then read it.
+  const deadline = Date.now() + maxWaitMs;
+  while (Date.now() < deadline) {
+    await sleep(10);
+    const v = await cache.get(key);
+    if (v !== null) return JSON.parse(v);
+  }
+  // Last resort if the winner died: load directly (degrade, don't fail).
+  return loadProductFromDb(id);
+}
+
+async function main() {
+  const cache = new FakeRedis();
+  const herd = Array.from({ length: 50 }, () => getProduct(42, { cache }));
+  const results = await Promise.all(herd);
+
+  console.log(`Concurrent reads:     ${results.length}`);
+  console.log(`Distinct DB loads:    ${dbCalls}  (single-flight collapsed the herd)`);
+  console.log(`Sample result:        ${JSON.stringify(results[0])}`);
+
+  // A second read is now a pure cache hit (no new DB load).
+  const before = dbCalls;
+  await getProduct(42, { cache });
+  console.log(`DB loads after warm:  ${dbCalls - before}  (0 == cache hit)`);
+}
+
+const isMain = process.argv[1] === fileURLToPath(import.meta.url);
+if (isMain) {
+  main().catch((e) => { console.error(e); process.exit(1); });
+}
+
+export { getProduct, jitteredTtl, FakeRedis };

package/catalog/skills/caching-strategies/references/cache-patterns.md

@@ -0,0 +1,93 @@
+# Cache Read/Write Patterns
+
+The *pattern* is how data flows between your application, the cache, and the source of truth (usually a
+database). Choose one per dataset based on read/write ratio and consistency needs.
+
+## Cache-aside (lazy loading) — the default
+
+The application owns the cache logic. The cache knows nothing about the database.
+
+```
+READ:
+  v = cache.get(key)
+  if v exists: return v                  # hit
+  v = db.load(key)                       # miss
+  cache.set(key, v, ttl)
+  return v
+
+WRITE:
+  db.write(key, v)
+  cache.del(key)                         # invalidate; next read repopulates
+```
+
+- **Pros:** simple; resilient — a dead cache degrades to "slow", not "broken"; only requested data is
+  cached, so the working set stays small.
+- **Cons:** every key's first read is a miss (cold-start latency); cache and DB can drift if you forget
+  to invalidate on write.
+- **Use when:** read-heavy workloads with tolerable staleness. This is the right answer ~80% of the
+  time.
+
+## Read-through
+
+Same flow as cache-aside, but the *cache layer* loads from the source on a miss via a configured
+loader, instead of the application doing it inline.
+
+```
+READ:
+  return cache.getOrLoad(key, loader=db.load)   # cache calls loader on miss
+```
+
+- **Pros:** loading logic is centralized behind the cache abstraction; application code is cleaner.
+- **Cons:** requires a cache library/provider that supports loaders; still cold on first read.
+- **Use when:** you want one canonical load path and your cache library supports it.
+
+## Write-through
+
+Writes are applied to the cache *and* the source synchronously, as one logical operation.
+
+```
+WRITE:
+  cache.set(key, v)
+  db.write(key, v)        # both, synchronously, before returning
+```
+
+- **Pros:** the cache is always consistent with the DB for written keys; reads after a write are always
+  fresh.
+- **Cons:** every write pays the cache write cost; you can cache data that's never read again (write
+  amplification). Often paired with read-through.
+- **Use when:** read-after-write freshness matters and writes are not the bottleneck.
+
+## Write-behind (write-back)
+
+Writes hit the cache immediately and are flushed to the source asynchronously (batched).
+
+```
+WRITE:
+  cache.set(key, v)
+  queue.push(key)         # background worker flushes to db later
+```
+
+- **Pros:** very fast writes; absorbs write spikes; enables batching/coalescing of DB writes.
+- **Cons:** **data loss risk** if the process crashes before flush; the DB is eventually consistent
+  with the cache; ordering and failure handling get complex.
+- **Use when:** high write throughput where some durability risk is acceptable (metrics, counters,
+  activity feeds). Never for data you can't afford to lose.
+
+## Refresh-ahead
+
+Proactively refresh entries that are *about to* expire and are being actively read, so a hot key never
+serves a miss.
+
+- **Pros:** hides refresh latency for hot keys; helps prevent stampedes.
+- **Cons:** wasted refreshes for keys that won't be read again; needs access-pattern prediction.
+- **Use when:** a small set of very hot, predictable keys.
+
+## Quick selector
+
+| Need | Pattern |
+|---|---|
+| General read-heavy, tolerate staleness | cache-aside |
+| Centralized load path | read-through |
+| Read-after-write must be fresh | write-through |
+| Extreme write throughput, can risk loss | write-behind |
+| Hot keys must never miss | refresh-ahead |

package/catalog/skills/caching-strategies/references/eviction-and-ttl.md

@@ -0,0 +1,67 @@
+# Eviction, TTL & Jitter
+
+A cache is finite. Entries leave by **TTL expiry** (time) or **eviction** (space pressure). Tune both
+deliberately, or the cache tunes itself badly.
+
+## Eviction policies
+
+When memory is full and a new entry arrives, the policy picks a victim.
+
+| Policy | Evicts | Good for | Watch out |
+|---|---|---|---|
+| **LRU** (Least Recently Used) | the entry untouched longest | general purpose, temporal locality | a full scan can flush the whole hot set |
+| **LFU** (Least Frequently Used) | the entry with fewest accesses | skewed popularity (a few very hot keys) | new entries look "cold" and get evicted early; needs aging |
+| **FIFO** (First In First Out) | the oldest inserted, regardless of use | rarely the right choice | evicts hot keys just because they're old |
+| **Random** | a random entry | cheap, surprisingly OK under uniform access | unpredictable tail latency |
+| **TTL-only** | nothing until expiry | when size is genuinely unbounded-safe | memory leak if the key space is large |
+
+**Default to LRU.** Move to LFU (or a hybrid like LRU-K / W-TinyLFU) only when you've measured a
+popularity skew where LRU underperforms. Modern Redis offers `allkeys-lru`, `allkeys-lfu`, and
+`volatile-*` variants (evict only keys with a TTL set).
+
+### Always bound the size
+
+An in-process cache (a plain `Map`) with no max size is a memory leak that ends in an OOM crash. Cap by
+entry count or bytes and attach an eviction policy. A bounded LRU map is the minimum viable in-process
+cache.
+
+## TTL tuning
+
+TTL bounds the maximum staleness window. Pick it from the data's tolerance, not by feel:
+
+- **Volatile, correctness-sensitive** (prices shown at checkout): seconds, plus explicit invalidation.
+- **Slowly changing** (a product description, a config blob): minutes to hours.
+- **Effectively static** (country list, feature flags read constantly): hours, with explicit bust on
+  change.
+
+Document the chosen staleness window as a number. "Up to 5 minutes stale" is a contract; "300" is a
+magic number nobody remembers the reason for.
+
+## TTL jitter (mandatory at scale)
+
+If many keys are populated together (a deploy warm-up, a batch import, a cache flush + refill) with the
+*same* fixed TTL, they all expire in the same instant and stampede the source together. Spread expiry
+by adding randomness:
+
+```js
+function jitteredTtl(baseSeconds, jitterRatio = 0.1) {
+  const delta = baseSeconds * jitterRatio;
+  const offset = (Math.random() * 2 - 1) * delta;   // uniform in [-delta, +delta]
+  return Math.max(1, Math.round(baseSeconds + offset));
+}
+```
+
+With `jitterRatio = 0.1`, a nominal 300s TTL becomes a value in `[270, 330]`, so a batch of 10,000 keys
+expires smeared across a 60-second window instead of all at once. See `scripts/ttl-jitter.mjs` for a
+self-tested version.
+
+## Negative caching (avoid cache penetration)
+
+If a key has no value in the source (a row that doesn't exist), naive cache-aside re-queries the DB on
+*every* request for it — **cache penetration**, which an attacker can weaponize by requesting random
+non-existent IDs.
+
+- Cache the "not found" result too, but with a **short** TTL (e.g. 10-30s) so a later insert becomes
+  visible quickly.
+- For high-cardinality miss attacks, front the cache with a **Bloom filter** of known-existing keys to
+  reject impossible lookups before they hit the DB.