@nklisch/pi-agile-workflow 0.15.3

package/skills/perf-scout/references/approximation-and-precomputation.md

@@ -0,0 +1,136 @@
+# Approximation & Precomputation — Perf-Scout Strategy Lens
+
+> **When to load this lens**: anywhere an *exact* answer is computed where an
+> *approximate* one would do, membership/cardinality/quantile/frequency queries,
+> similarity search, or expensive results that could be computed ahead of time.
+>
+> **Where these ideas come from**: streaming algorithms, big-data analytics
+> (sketches), information retrieval (ANN), and the precomputation school. The core
+> move is radical: *change what counts as a correct answer*. Trading a little
+> accuracy (often <1% error) can buy orders of magnitude — but only where the
+> domain tolerates it, so each candidate must name the accuracy it gives up and
+> who's okay with that. Every entry is a *candidate* hypothesis, not a proven win.
+> Cite `file:line`, attribute the source, give a validation path.
+
+## Detection signals (where this lens might apply — not proof)
+- Exact distinct-count / cardinality over a large stream (`COUNT(DISTINCT ...)`, dedup-to-count).
+- Exact membership tests against a huge set on a slow/large store.
+- Top-k / heavy-hitters / frequency over a stream computed by full tallies.
+- Exact quantiles/percentiles over large data (sort-then-index).
+- Nearest-neighbor / similarity search done by comparing against all items.
+- Expensive deterministic functions over a bounded input domain (table-able).
+- "Good enough" metrics (dashboards, recommendations, monitoring) computed exactly.
+
+## Strategies
+
+### 1. HyperLogLog for cardinality
+- **Borrowed from**: streaming analytics (Flajolet et al. 2007; Redis, Presto, BigQuery `APPROX_COUNT_DISTINCT`).
+- **The idea**: estimate distinct counts in kilobytes of state with a standard error of ~1.04/√m (m = registers — e.g. ~2% at 2048 registers, ~1.5 kB) instead of holding every distinct value to count exactly.
+- **Code signals**: building a big set/map just to take its size; `COUNT(DISTINCT)` over huge data; dedup-for-counting.
+- **Speculative win**: huge memory reduction; mergeable across shards.
+- **Cost / risk**: relative error scales as 1/√m, so halving error costs 4x state; not for exact-count requirements.
+- **Validate by**: error vs. exact on a sample; memory before/after — profile via perf-design.
+
+### 2. Bloom filter for membership
+- **Borrowed from**: databases, caches, networking.
+- **The idea**: a compact probabilistic set answers "definitely not present / probably present" so most true-negatives skip the expensive exact check.
+- **Code signals**: membership tests against a large/slow set; existence checks before costly fetches (LSM SSTable probes, cache miss guards).
+- **Speculative win**: removes most negative lookups from the slow path; ~10 bits/key for ~1% false-positive rate.
+- **Cost / risk**: false positives only (never false negatives), tunable by bits/key and hash count; classic Bloom has no deletes (a counting Bloom adds deletes at 1.5-4x space).
+- **Validate by**: false-positive rate vs. configured target; slow-path hits avoided.
+
+### 3. Cuckoo or ribbon filter as a Bloom alternative
+- **Borrowed from**: networking / databases — *cuckoo filter* (Fan et al. 2014; Redis), a different construction from *ribbon filter* (RocksDB, an XOR/linear-system-based static filter).
+- **The idea**: two distinct space-saving alternatives to Bloom. A **cuckoo filter** stores fingerprints in a cuckoo-hash table and adds native O(1) deletes, beating space-optimized Bloom below ~3% target FPR. A **ribbon filter** solves a small XOR linear system to build a *static, immutable* filter ~30% smaller than Bloom at the same FPR, at higher build CPU — not the same family as cuckoo, grouped here only as a low-FPR Bloom substitute.
+- **Code signals**: a Bloom filter that also needs deletes (currently rebuilt or shadowed by tombstones) → cuckoo; a read-only/static membership set wanting minimum bits/key → ribbon.
+- **Speculative win**: deletes without a rebuild (cuckoo), or tighter space than Bloom at low FPR (both) — *if* the workload matches the filter's mutability model.
+- **Cost / risk**: cuckoo inserts can fail when near-full (needs headroom/resize); ribbon is build-once and not incrementally mutable, with extra build cost.
+- **Validate by**: FPR + space vs. an equivalent Bloom at the same target; delete-path correctness for cuckoo.
+
+### 4. Count-Min Sketch for frequency estimation
+- **Borrowed from**: streaming algorithms (Cormode & Muthukrishnan 2005), network telemetry.
+- **The idea**: estimate the frequency of a *queried* item in sublinear space instead of an exact per-item counter map. Note: CMS answers "how often did I see X?"; it does **not** enumerate the heaviest items on its own — to get top-k you pair it with a heap of candidate keys, or use a counter-based summary like SpaceSaving (next entry).
+- **Code signals**: large frequency maps for rate limiting / trending / per-key counts at scale; needing a point-frequency lookup, not the full ranking.
+- **Speculative win**: bounded memory regardless of key count; mergeable across shards.
+- **Cost / risk**: overestimates only, never underestimates (collisions inflate counts); error bounded by ε·N with probability 1−δ; no top-k without a companion structure.
+- **Validate by**: estimate error vs. exact counts on a sample; memory.
+
+### 5. SpaceSaving for top-k / heavy hitters
+- **Borrowed from**: streaming algorithms (Metwally, Agrawal & El Abbadi 2005; "Stream-Summary").
+- **The idea**: track only k counters, evicting the current minimum, to surface the top-k frequent items directly — a counter-based alternative to a sketch when you want the *items*, not per-key frequency lookups.
+- **Code signals**: full per-key tally sorted at the end to take top-k; trending/leaderboard over an unbounded key space.
+- **Speculative win**: O(k) memory independent of distinct-key count; O(1) update.
+- **Cost / risk**: tracked counts are upper bounds (with a known per-counter error margin); rare items below the k-th counter are invisible.
+- **Validate by**: recovered top-k vs. exact top-k on a sample (precision/recall of the set); memory.
+
+### 6. Approximate quantiles (t-digest / KLL / GK)
+- **Borrowed from**: monitoring/metrics systems, streaming statistics (Greenwald-Khanna 2001; KLL 2016; Dunning's t-digest).
+- **The idea**: estimate percentiles in small mergeable state instead of sorting/holding all samples.
+- **Code signals**: collecting all latencies/values to compute p50/p95/p99; sort-to-percentile over large data.
+- **Speculative win**: bounded memory; streaming and mergeable across nodes.
+- **Cost / risk**: pick the sketch by where you need accuracy — t-digest concentrates accuracy at the tails (good for p99) but has no worst-case bound; KLL/GK give uniform rank-error guarantees instead.
+- **Validate by**: quantile error vs. exact at the percentiles you actually report; memory.
+
+### 7. Approximate nearest neighbor (ANN)
+- **Borrowed from**: vector search / IR (HNSW, IVF, LSH; FAISS, ScaNN, DiskANN).
+- **The idea**: index vectors for approximate similarity search so queries examine a small candidate set instead of comparing against every item; product quantization (IVF+PQ) compresses vectors for memory, DiskANN serves billion-scale from SSD.
+- **Code signals**: brute-force similarity/distance over all items per query; full-scan k-NN.
+- **Speculative win**: sublinear query time at 90%+ recall; scales to millions–billions of vectors.
+- **Cost / risk**: recall is the accuracy traded (tunable via ef/nprobe); PQ compression lowers recall@1 further; index build time and memory.
+- **Validate by**: recall@k vs. exact brute force on a query set; query latency and index memory.
+
+### 8. Sampling instead of full computation
+- **Borrowed from**: approximate query processing (BlinkDB), statistics.
+- **The idea**: compute on a representative sample and extrapolate (with confidence bounds) for metrics that tolerate it.
+- **Code signals**: full-dataset aggregation for a dashboard/estimate that doesn't need exactness.
+- **Speculative win**: cost scales with sample size, not dataset size (BlinkDB reported 2-10% error at 200x speedup).
+- **Cost / risk**: sampling error and bias; rare groups / high-variance aggregates need stratified samples.
+- **Validate by**: estimate vs. full result on history; confidence-interval coverage check.
+
+### 9. Reservoir sampling for a fixed-size stream sample
+- **Borrowed from**: streaming statistics (Vitter's Algorithm R, 1985).
+- **The idea**: maintain a uniform random sample of fixed size k from an unbounded stream in one pass, without knowing the length in advance — feeds any downstream estimate (quantiles, distinct-ish, debugging traces).
+- **Code signals**: buffering an entire stream to later pick a random subset; "keep the last N then sample" approximations that bias toward recent items.
+- **Speculative win**: O(k) memory, single pass, unbiased over the whole stream.
+- **Cost / risk**: a uniform sample under-represents rare events; weighted/time-decayed variants needed if recency matters.
+- **Validate by**: sample-derived estimate vs. exact on a replayed stream; distribution check vs. uniform.
+
+### 10. Precomputed lookup tables
+- **Borrowed from**: classic table-driven optimization (trig tables, CRC tables, color LUTs).
+- **The idea**: precompute results for a bounded input domain into a table and replace runtime computation with an array index.
+- **Code signals**: an expensive deterministic function over a small/bounded input range called hot and often.
+- **Speculative win**: computation → a single memory load.
+- **Cost / risk**: table memory; cache footprint; only for bounded domains.
+- **Validate by**: lookup vs. compute benchmark; table size vs. cache.
+
+### 11. Perfect / minimal hashing for static key sets
+- **Borrowed from**: compilers (keyword lookup, `gperf` for small sets; CMPH/CHD, BBHash for massive sets).
+- **The idea**: for a known static set of keys, build a (minimal) perfect hash so lookups are collision-free O(1) with no probing and minimal memory.
+- **Code signals**: hash lookups against a fixed set of keys (keywords, opcodes, config); startup-built immutable maps.
+- **Speculative win**: collision-free single-probe lookups; the function itself is compact (CHD ~2.07 bits/key, BBHash ~3 bits/key — separate from value storage).
+- **Cost / risk**: only for static key sets; build step; the MPHF returns a slot for any input, so non-keys need a separate verification check.
+- **Validate by**: lookup benchmark vs. general hash map; function memory.
+
+### 12. Incremental view maintenance (precompute + update)
+- **Borrowed from**: materialized views; differential dataflow (Timely/Materialize), DBSP.
+- **The idea**: maintain a precomputed result incrementally as inputs change, rather than recomputing — straddles precomputation and incrementalization for derived data.
+- **Code signals**: recomputing a derived dataset from scratch on each input change.
+- **Speculative win**: update cost proportional to the change, not the dataset.
+- **Cost / risk**: incremental logic complexity; correctness vs. full recompute.
+- **Validate by**: property-test incremental == full recompute; benchmark with realistic change rates.
+
+### 13. Succinct / compressed structures queried in place
+- **Borrowed from**: succinct data structures (wavelet trees, FM-index), bioinformatics, search.
+- **The idea**: store data in a near-information-theoretic-minimum form that still answers queries (rank/select on bit vectors in O(1) with o(n) extra bits, prefix) without decompression, fitting far more in memory/cache.
+- **Code signals**: large indexes/bitmaps/text that must be queried but barely fit in memory.
+- **Speculative win**: dramatic memory reduction while staying queryable; more fits in cache.
+- **Cost / risk**: implementation complexity; constant-factor query overhead vs. a plain structure.
+- **Validate by**: memory footprint; query benchmark vs. plain structure.
+
+### 14. Tolerate staleness (cheaper freshness)
+- **Borrowed from**: caching, eventually-consistent reads, monitoring.
+- **The idea**: serve a slightly stale precomputed answer and refresh in the background where the domain doesn't need up-to-the-instant accuracy.
+- **Code signals**: recomputing fresh values per request for data that changes slowly or tolerates lag.
+- **Speculative win**: removes recomputation from the request path.
+- **Cost / risk**: staleness window must be acceptable to the use case.
+- **Validate by**: request latency; staleness tolerance confirmation.

package/skills/perf-scout/references/caching-and-memoization.md

@@ -0,0 +1,123 @@
+# Caching & Memoization — Perf-Scout Strategy Lens
+
+> **When to load this lens**: repeated identical computation or requests,
+> expensive pure functions, hot read paths, derived values recomputed per call,
+> chatty reads against a slow backing store.
+>
+> **Where these ideas come from**: CPU cache hierarchies, CDNs, database buffer
+> pools and materialized views, and web-scale read paths. Caching is a force
+> multiplier *and* a footgun — it trades freshness and invalidation complexity for
+> speed, so each candidate must name what it caches, how it's keyed, and how it's
+> invalidated. Every entry is a *candidate* hypothesis, not a proven win. Cite
+> `file:line`, attribute the source, give a validation path.
+>
+> **Honest caveat to carry into every idea**: caching can hide an underlying
+> algorithmic problem rather than fix it. Prefer "do less work" (algorithmic lens)
+> when the work is reducible; reach for caching when the work is genuinely
+> irreducible and reused.
+
+## Detection signals (where this lens might apply — not proof)
+- The same expensive call/query/computation repeated with identical inputs.
+- Hot read paths hitting a slow backing store (DB, network, disk) for stable data.
+- Derived/computed values (formatting, parsing, aggregation) recomputed every call.
+- Many concurrent misses for the same key (stampede risk).
+- Read-mostly data with rare writes.
+- Per-request recomputation of values that change far slower than request rate.
+
+## Strategies
+
+### 1. Memoize expensive pure functions
+- **Borrowed from**: dynamic programming, functional caching.
+- **The idea**: wrap a deterministic, expensive function in a keyed cache; return stored results on repeat inputs.
+- **Code signals**: a pure function (parse, compile, format, hash, derive) called repeatedly with a small distinct-input set.
+- **Speculative win**: one computation per distinct input instead of per call.
+- **Cost / risk**: unbounded growth without eviction; key must capture all inputs.
+- **Validate by**: hit rate on real input distribution.
+
+### 2. Tiered cache (L1 in-process → L2 shared → origin)
+- **Borrowed from**: CPU cache hierarchy, CDN edge tiers.
+- **The idea**: front a slow origin with a fast small in-process cache, backed by a larger shared cache (Redis/memcached), backed by the origin.
+- **Code signals**: every read hits the DB/network for data reused across requests.
+- **Speculative win**: most reads served from the fastest tier; origin load drops sharply.
+- **Cost / risk**: coherence across tiers; invalidation fan-out; staleness windows.
+- **Validate by**: per-tier hit rates; origin QPS before/after.
+
+### 3. Request coalescing / single-flight
+- **Borrowed from**: `singleflight` (Go), Caffeine `AsyncLoadingCache`, CDN collapse-forwarding.
+- **The idea**: when many concurrent callers miss the same key, let one compute and have the rest await its result instead of all hitting the origin.
+- **Code signals**: `if miss: load_from_origin()` with no in-flight dedup; thundering-herd risk on cold keys.
+- **Speculative win**: collapses N concurrent identical loads to 1; protects the origin under spikes.
+- **Cost / risk**: shared failure (one error fails all waiters); modest coordination.
+- **Validate by**: origin call count under concurrent load; spike test.
+
+### 4. Materialized views / precomputed aggregates
+- **Borrowed from**: database materialized views, OLAP cubes.
+- **The idea**: precompute and store expensive aggregates/joins; refresh on write or schedule; serve reads from the precomputed form.
+- **Code signals**: dashboards/summaries recomputed from raw rows on every read; heavy group-by per request.
+- **Speculative win**: read cost drops to a lookup; compute moves off the read path.
+- **Cost / risk**: refresh strategy and staleness; storage.
+- **Validate by**: read latency and origin load; staleness tolerance check.
+
+### 5. Right eviction policy for the access pattern
+- **Borrowed from**: OS page replacement and DB buffer pools — LRU, LFU, CLOCK, ARC (Megiddo & Modha, FAST 2003: recency+frequency with ghost lists, scan-resistant, self-tuning), and modern scan-resistant designs: W-TinyLFU (Caffeine — an admission filter over a count-min frequency sketch guarding a Segmented-LRU main region), S3-FIFO (SOSP 2023 — three FIFO queues with quick demotion of one-hit objects), SIEVE (NSDI 2024 — a single FIFO with a lazily-advancing "hand", simpler than LRU).
+- **The idea**: match the eviction policy to the workload — LRU for recency, LFU/W-TinyLFU for skewed popularity, ARC/S3-FIFO/SIEVE for scan resistance — instead of a naive map that grows unbounded or evicts blindly.
+- **Code signals**: an unbounded map used as a cache, or a plain LRU under a scan-heavy/skewed workload with a poor hit rate.
+- **Speculative win**: higher hit rate at the same memory; scan resistance; some FIFO-based policies (S3-FIFO, SIEVE) also report better lock-free throughput than LRU.
+- **Cost / risk**: policy complexity; wrong policy can underperform plain LRU; many libraries only ship LRU.
+- **Validate by**: simulate hit rate on real traces across policies (libCacheSim / Caffeine simulator); compare against current policy.
+
+### 6. Negative caching
+- **Borrowed from**: DNS, web caches.
+- **The idea**: cache "not found"/empty results (with a short TTL) so repeated misses for absent keys don't re-hit the origin.
+- **Code signals**: repeated lookups for keys that frequently don't exist, each hitting the DB.
+- **Speculative win**: removes origin load from miss-storms on absent keys.
+- **Cost / risk**: serving stale "absent" after a create; needs short TTL or invalidation.
+- **Validate by**: miss-rate composition; origin load on absent keys.
+
+### 7. Compute-once / hoist derived values
+- **Borrowed from**: loop-invariant code motion.
+- **The idea**: compute a value once and reuse it instead of recomputing per iteration/request — hoist invariants out of loops, cache per-request derived state.
+- **Code signals**: identical derivation inside a loop or repeated per call within one request.
+- **Speculative win**: removes redundant computation proportional to iteration/call count.
+- **Cost / risk**: minimal; ensure the value is truly invariant for the scope.
+- **Validate by**: benchmark; trace recomputation count.
+
+### 8. Content-addressed reuse
+- **Borrowed from**: content-addressed storage (Git, build caches), deduplication.
+- **The idea**: key results by a hash of their inputs so identical work anywhere reuses the same cached artifact (build outputs, rendered fragments, compiled queries).
+- **Code signals**: the same artifact regenerated from identical inputs across requests/builds.
+- **Speculative win**: global dedup of identical work; cross-request/cross-build reuse.
+- **Cost / risk**: hashing cost; cache storage and GC.
+- **Validate by**: dedup/hit rate on real inputs.
+
+### 9. Write-through vs. write-back choice
+- **Borrowed from**: CPU caches, DB buffer management.
+- **The idea**: pick the write policy deliberately — write-back (buffer and flush in batches) for write-heavy paths that tolerate a durability window; write-through for simplicity/consistency.
+- **Code signals**: every write synchronously hits the slow store; write amplification on a hot path.
+- **Speculative win**: batched/deferred writes cut per-write origin cost.
+- **Cost / risk**: durability/consistency window; crash recovery complexity.
+- **Validate by**: write throughput; correctness under crash scenarios.
+
+### 10. Probabilistic early expiration (anti-stampede TTL)
+- **Borrowed from**: XFetch (Vattani, Chierichetti & Lowenstein, "Optimal Probabilistic Cache Stampede Prevention", VLDB 2015).
+- **The idea**: on each read, recompute early with a probability that rises as TTL nears (scaled by recompute cost), so a popular key doesn't expire for everyone at once — no locks or coordination needed.
+- **Code signals**: synchronized expiry of popular keys causing periodic origin spikes.
+- **Speculative win**: smooths refresh load; avoids cliff-edge stampedes without a single-flight lock.
+- **Cost / risk**: a little early recomputation; the beta/cost factor needs tuning.
+- **Validate by**: origin load over time around expiries; compare against fixed-TTL baseline.
+
+### 11. Stale-while-revalidate (serve stale, refresh in background)
+- **Borrowed from**: HTTP `stale-while-revalidate` (RFC 5861), CDN serve-stale.
+- **The idea**: once an entry passes its freshness window, keep serving the stale value to readers while a single background refresh updates it — readers never block on the origin during revalidation.
+- **Code signals**: reads block synchronously on refresh at expiry; latency spikes correlate with TTL boundaries; data tolerates brief staleness.
+- **Speculative win**: refresh latency disappears from the read path; tail latency at expiry flattens.
+- **Cost / risk**: readers may see stale data within the window; needs dedup so only one refresh runs (pairs with single-flight).
+- **Validate by**: p99 read latency around expiry; staleness-window tolerance check; origin refresh count.
+
+### 12. Cache key normalization
+- **Borrowed from**: CDN cache keys (query-string sort, `No-Vary-Search`), HTTP `Vary` discipline.
+- **The idea**: canonicalize keys before lookup — sort/strip irrelevant query params, lowercase, drop tracking tokens, narrow `Vary` — so semantically identical requests collapse to one entry instead of fragmenting the cache.
+- **Code signals**: cache key built from a raw URL/struct with parameter order or noise fields included; low hit rate despite repeated logical requests; `Vary: Cookie`-style over-keying.
+- **Speculative win**: fragmented entries collapse into shared hits; higher hit rate at the same data, no policy change.
+- **Cost / risk**: over-normalization can collapse requests that should differ (correctness bug — must verify which fields are semantically load-bearing).
+- **Validate by**: hit rate before/after on real key distribution; audit that dropped fields don't change the response.

package/skills/perf-scout/references/compiler-runtime-and-language.md

@@ -0,0 +1,136 @@
+# Compiler, Runtime & Language — Perf-Scout Strategy Lens
+
+> **When to load this lens**: hot inner loops, allocation/GC pressure, dynamic
+> dispatch, boxing/autoboxing, interpreters, string building, reflection, heavy
+> abstraction layers on a hot path.
+>
+> **Where these ideas come from**: compilers, language runtimes (JVM, V8, CLR, Go
+> runtime), and low-level systems programming. These are the smaller-constant-
+> factor wins — they rarely change the complexity class, but on a hot path they
+> compound. Reach here after the algorithmic, locality, and I/O lenses. Every
+> entry is a *candidate* hypothesis, not a proven win. Cite `file:line`, attribute
+> the source, give a validation path.
+
+## Detection signals (where this lens might apply — not proof)
+- Allocation inside a tight loop; high GC frequency or pause time.
+- Virtual/interface/dynamic dispatch in a hot loop (polymorphic `update()`/`next()`).
+- Boxing of primitives into objects (collections of `Integer`/`Object`, autoboxing).
+- String building via repeated concatenation; many short-lived intermediate strings.
+- Reflection / dynamic lookups on a hot path.
+- Heavy generic abstraction (iterators of iterators, deep call chains) per element.
+- An interpreter / dispatch loop over an instruction or AST stream.
+- A CPU-bound binary built with only default optimization (no PGO/LTO); hot calls that cross module/crate boundaries and never inline.
+
+## Strategies
+
+### 1. Cut allocations on the hot path
+- **Borrowed from**: GC-runtime tuning, systems programming.
+- **The idea**: reuse buffers, allocate outside the loop, use value types/stack allocation, and avoid intermediate objects so the allocator/GC isn't on the hot path.
+- **Code signals**: `new`/slice/map allocation per iteration; temporary collections created and discarded.
+- **Speculative win**: lower GC pressure and pause frequency; better locality.
+- **Cost / risk**: lifetime discipline; can hurt readability.
+- **Validate by**: allocations/op and GC metrics — perf-design.
+
+### 2. Devirtualize / monomorphize hot dispatch
+- **Borrowed from**: JIT inline caches (V8 IC sites stay fast while monomorphic, degrade as they become polymorphic and fall off the fast path once megamorphic — V8 caps the polymorphic IC at ~4 shapes), Rust/C++ template monomorphization.
+- **The idea**: keep hot call sites monomorphic, or replace virtual/interface dispatch with concrete types, generics/monomorphization, or sealed-type switches so the call inlines and predicts.
+- **Code signals**: polymorphic method called per element; a hot call site that sees many shapes/types (megamorphic); trait-object/`dyn` loops.
+- **Speculative win**: enables inlining, removes indirect-call and branch-mispredict cost.
+- **Cost / risk**: monomorphization can bloat code and *hurt* via instruction-cache misses; less flexible.
+- **Validate by**: inlining/devirt evidence (e.g. JIT logs, `--emit=asm`); loop microbenchmark.
+
+### 3. Avoid boxing / use primitive specializations
+- **Borrowed from**: type-specific primitive collections (fastutil, Trove, HPPC); Go/Rust value types.
+- **The idea**: store primitives in primitive arrays/specialized collections (e.g. fastutil `IntArrayList`/`Int2IntOpenHashMap`) instead of boxed object collections; avoid autoboxing in hot math.
+- **Code signals**: `List<Integer>`/`Map<Integer,...>`, autoboxing in loops, `interface{}`/`Object` numeric storage.
+- **Speculative win**: removes per-element allocation and pointer indirection; denser, more cache-friendly layout.
+- **Cost / risk**: extra dependency and specialized APIs; less generic.
+- **Validate by**: allocations/op and loop benchmark.
+
+### 4. Efficient string building / avoid intermediate strings
+- **Borrowed from**: runtime string-builder idioms.
+- **The idea**: use a builder/buffer (with capacity reserved) instead of `+=`; avoid creating throwaway substrings; work on byte/char slices; intern repeated strings.
+- **Code signals**: string `+=` in a loop; many `substring`/`split`/`format` temporaries on a hot path.
+- **Speculative win**: removes quadratic concatenation and allocation churn.
+- **Cost / risk**: minimal; readability tradeoff.
+- **Validate by**: allocations and timing on the build path.
+
+### 5. Hoist work out of the loop / strength reduction
+- **Borrowed from**: classic compiler optimizations (LICM, strength reduction).
+- **The idea**: move loop-invariant computation out of the loop; replace expensive ops with cheaper equivalents (multiply→add, divide→shift/mul-by-reciprocal, pow→repeated mul).
+- **Code signals**: invariant expressions recomputed each iteration; expensive arithmetic in the inner loop.
+- **Speculative win**: removes redundant per-iteration work.
+- **Cost / risk**: minimal; watch numeric equivalence.
+- **Validate by**: loop microbenchmark.
+
+### 6. Branchless / predictable control flow
+- **Borrowed from**: HFT, codecs, SIMD programming.
+- **The idea**: remove unpredictable branches in tight loops (use conditional moves, masks, table lookups, or sort to make branches predictable) to avoid misprediction stalls.
+- **Code signals**: data-dependent branches in a hot loop over unsorted data; per-element `if` on random conditions.
+- **Speculative win**: fewer branch mispredictions; enables vectorization.
+- **Cost / risk**: branchless code can be less clear and sometimes slower; measure.
+- **Validate by**: branch-miss counters; benchmark both forms.
+
+### 7. Specialize the hot path (partial evaluation)
+- **Borrowed from**: partial evaluation / Futamura projections (specialize a general routine against its known inputs — the model behind Truffle/GraalVM and PyPy/RPython JITs), template metaprogramming.
+- **The idea**: generate or select a specialized version of a general routine for the common case (fixed config, known sizes, common types) instead of running the fully general path every time.
+- **Code signals**: a general function with config/flags re-interpreted per call; an interpreter re-dispatching the same program.
+- **Speculative win**: removes per-call generality overhead; common case runs lean.
+- **Cost / risk**: code duplication; specialization machinery to build and maintain.
+- **Validate by**: common-case microbenchmark vs. the general path.
+
+### 8. Lazy evaluation / avoid eager materialization
+- **Borrowed from**: lazy languages, iterator/stream pipelines.
+- **The idea**: defer computation until needed and stream through transforms instead of building full intermediate collections.
+- **Code signals**: chained `map/filter` materializing each stage; computing values that may not be used.
+- **Speculative win**: skips unused work; lower peak memory.
+- **Cost / risk**: laziness hides cost and complicates debugging; thunk overhead.
+- **Validate by**: fraction of results consumed; memory and timing.
+
+### 9. Copy-on-write / move semantics / avoid deep copies
+- **Borrowed from**: persistent/COW data structures, Rust ownership/moves, C++ move semantics.
+- **The idea**: pass by reference/move, share immutable data, and copy only on mutation instead of defensive deep copies.
+- **Code signals**: large structures deep-copied when passed; defensive clones of read-only data.
+- **Speculative win**: removes large copies and their allocations.
+- **Cost / risk**: aliasing/ownership bugs if mutation leaks; naive COW can *backfire* under concurrency — the atomic refcount became a bottleneck, which is why C++11 forbids COW `std::string` (GCC moved to small-string optimization in 5.1). Prefer moves/SSO for small/contended data; reserve COW for genuinely large, rarely-mutated shares.
+- **Validate by**: copy count/size; benchmark, including the concurrent case.
+
+### 10. Compile-time / startup precomputation (comptime, const-fold, codegen)
+- **Borrowed from**: `constexpr`/Zig `comptime`, code generation, perfect-hash generators (GNU `gperf` emits a collision-free keyword lookup as C — single probe, one string compare).
+- **The idea**: move work to compile time or startup — const-fold, generate lookup/dispatch tables, build a perfect hash for a fixed keyword set, precompile regexes/queries — so runtime just reads results.
+- **Code signals**: recompiling regexes/parsing config/building dispatch tables on every call; linear keyword scans; constant work done at runtime.
+- **Speculative win**: removes invariant setup from the runtime path.
+- **Cost / risk**: build complexity; codegen maintenance; generated artifacts can drift from source if not regenerated.
+- **Validate by**: runtime cost with precomputed artifacts vs. computed-at-runtime baseline.
+
+### 11. Choose the right runtime construct
+- **Borrowed from**: runtime/stdlib performance lore; cache-friendly hash maps (Abseil Swiss tables — dense open-addressing with a SIMD-scanned control byte array; adopted by Rust `hashbrown`/`std` and Go 1.24 maps).
+- **The idea**: use the faster-by-design construct — preallocated/`reserve`d collections, a flat/Swiss-table map over a node-based one, value-type enums over object hierarchies, native arrays over generic containers on hot paths.
+- **Code signals**: default containers grown incrementally; generic/abstract collections where a concrete one fits; node-pointer maps on a lookup-heavy path.
+- **Speculative win**: fewer reallocations and indirections; better constants and cache behavior.
+- **Cost / risk**: small; profile to confirm the swap helps; iteration order/pointer-stability guarantees may differ.
+- **Validate by**: microbenchmark of the construct in context.
+
+### 12. Let escape analysis stack-allocate short-lived objects
+- **Borrowed from**: HotSpot C2 scalar replacement of aggregates (SRA, `-XX:+EliminateAllocations`); Go's escape analysis (`-gcflags=-m`).
+- **The idea**: keep short-lived objects from escaping (don't store them in fields/globals or leak via interfaces) so the compiler can decompose them into registers/stack slots instead of heap-allocating — no GC involvement. (Note: HotSpot does scalar replacement, not literal stack allocation.)
+- **Code signals**: small temporaries (points, iterators, boxed values) that outlive nothing but escape via a field assignment, returned pointer, or `interface{}`.
+- **Speculative win**: removes heap allocations and the GC work they create on a hot path.
+- **Cost / risk**: brittle — a small code change can make an object escape again and silently re-allocate.
+- **Validate by**: escape/allocation logs (`-gcflags=-m`, JFR allocation profiling) and allocations/op before vs. after.
+
+### 13. Profile-guided optimization (PGO)
+- **Borrowed from**: feedback-directed compilation (Clang/GCC `-fprofile-use`, MSVC PGO, Go `pgo`).
+- **The idea**: feed a representative runtime profile back into the compiler so it inlines hot callees, lays out code for the instruction cache, and separates hot from cold paths using real execution counts instead of static guesses.
+- **Code signals**: a CPU-bound binary with a stable hot path; inlining/layout left entirely to static heuristics; large switch/dispatch where the common arm is known at runtime.
+- **Speculative win**: better inlining and code/branch layout; vendors report single-digit-to-~double-digit-percent wins (workload-dependent — never assume the headline number).
+- **Cost / risk**: build/CI complexity to collect and refresh profiles; a stale profile can mislead layout.
+- **Validate by**: end-to-end benchmark of PGO vs. non-PGO build; i-cache/branch counters via perf-design.
+
+### 14. Link-time / whole-program optimization (LTO)
+- **Borrowed from**: interprocedural optimization (`-flto`, ThinLTO, `lto = true`).
+- **The idea**: defer optimization to link time so the compiler sees across translation-unit boundaries and can inline cross-module calls, propagate constants, and drop dead code it otherwise can't.
+- **Code signals**: hot calls that cross library/crate/module boundaries and never inline; thin wrappers in one TU around work in another.
+- **Speculative win**: cross-module inlining and dead-code elimination not possible per-file; sometimes smaller binaries too.
+- **Cost / risk**: longer link times and higher link-time memory; ThinLTO mitigates but adds build config.
+- **Validate by**: benchmark LTO vs. non-LTO build; inspect whether the boundary call now inlines (`--emit=asm`/disasm).

package/skills/perf-scout/references/database-and-storage-internals.md

@@ -0,0 +1,144 @@
+# Database & Storage Internals — Perf-Scout Strategy Lens
+
+> **When to load this lens**: custom storage/query/index code, in-memory stores,
+> scans/filters/joins/aggregations, write-heavy paths, on-disk/serialized formats,
+> or app code that leans heavily on a database.
+>
+> **Where these ideas come from**: database and storage-engine internals
+> (Postgres, SQLite, RocksDB/LevelDB, DuckDB, ClickHouse, Lucene). These systems
+> spent decades making "find and combine data" fast; their techniques transfer to
+> any code that stores, indexes, scans, or joins. Every entry is a *candidate*
+> hypothesis, not a proven win. Cite `file:line`, attribute the source, give a
+> validation path. (For app-side query *usage* like N+1, see the I/O lens — this
+> lens is about storage/query *internals* and patterns.)
+
+## Detection signals (where this lens might apply — not proof)
+- In-memory collections scanned linearly to filter/find/join on a hot path.
+- Custom on-disk or serialized storage formats; hand-rolled indexes.
+- Write-heavy paths doing in-place random updates.
+- Aggregations/group-bys/joins implemented row-at-a-time in app code.
+- Repeated membership tests against a large set, especially on a slow store.
+- Range/prefix/sorted queries served by full scans.
+- Large datasets where only a few columns/fields are ever touched.
+
+## Strategies
+
+### 1. Add the right index (and covering indexes)
+- **Borrowed from**: B-tree/hash indexing.
+- **The idea**: build an index matching the query's predicate (equality → hash, range/sort → ordered tree), and make it *covering* so the query is answered from the index without touching the base data.
+- **Code signals**: linear scans filtering on a stable key; queries that read a few fields after a lookup.
+- **Speculative win**: a scan could drop toward an O(log n)/O(1) lookup if the predicate is selective; a covering index may avoid the base-row fetch.
+- **Cost / risk**: index memory and write-time maintenance.
+- **Validate by**: query plan / lookup count; benchmark — perf-design.
+
+### 2. Probabilistic membership filter before the expensive lookup
+- **Borrowed from**: LSM-tree SSTables (RocksDB attaches a per-SSTable bloom or, since 6.15, ribbon filter), caches.
+- **The idea**: keep a compact probabilistic membership filter so likely-absent keys skip the expensive disk/network lookup entirely. Bloom is cheapest to build; ribbon/quotient variants trade more build CPU for ~27% less memory at the same false-positive rate.
+- **Code signals**: many lookups that miss on a slow store; checking existence before a costly fetch.
+- **Speculative win**: removes most negative lookups from the slow path.
+- **Cost / risk**: false positives (tunable); filter memory and build/maintenance CPU.
+- **Validate by**: negative-lookup rate; slow-store hits before/after.
+
+### 3. Columnar layout + vectorized execution
+- **Borrowed from**: OLAP engines (DuckDB, ClickHouse, Arrow).
+- **The idea**: store fields column-wise and process them in batches/vectors (DuckDB ~1–2K values, ClickHouse blocks up to 65,536) so scans touch only needed columns and run SIMD-friendly per-column kernels. Vectorized execution amortizes per-row interpreter overhead with precompiled kernels — distinct from JIT-compiling the whole query plan to machine code (an alternative model; ClickHouse adds JIT as an optional optimization, DuckDB stays vectorized).
+- **Code signals**: row-oriented storage scanned for analytics over a few fields; aggregate/filter over many rows row-at-a-time.
+- **Speculative win**: less data scanned, better compression, vectorizable operators.
+- **Cost / risk**: poor fit for row-at-a-time OLTP access; row-reconstruction cost.
+- **Validate by**: bytes/columns scanned; scan/aggregate benchmark vs row-at-a-time.
+
+### 4. LSM-tree write batching for write-heavy paths
+- **Borrowed from**: LevelDB/RocksDB/Cassandra.
+- **The idea**: buffer writes in memory and flush sequentially (append-only), compacting in the background, instead of random in-place updates.
+- **Code signals**: write-heavy workload doing random in-place updates; write amplification.
+- **Speculative win**: could turn random writes into sequential ones and raise write throughput if the path is write-bound and read amplification stays tolerable.
+- **Cost / risk**: read amplification and compaction overhead; tuning.
+- **Validate by**: write throughput; read latency under compaction.
+
+### 5. Predicate / projection pushdown
+- **Borrowed from**: query optimizers, Parquet/columnar readers.
+- **The idea**: apply filters and select columns as early/low as possible (at the storage/scan layer) so less data flows up the pipeline.
+- **Code signals**: load-everything-then-filter; full objects fetched to use a subset.
+- **Speculative win**: less data materialized, transferred, and processed.
+- **Cost / risk**: pushing logic into the storage layer; coverage limits.
+- **Validate by**: rows/bytes after pushdown; pipeline timing.
+
+### 6. Better join strategy
+- **Borrowed from**: relational query execution (hash join, sort-merge join).
+- **The idea**: replace a nested-loop join with a hash join (build a hash table on the smaller side, probe with the larger) or a sort-merge join when inputs are already sorted, for large joins done in app code.
+- **Code signals**: `for a: for b: if a.key == b.key` — nested-loop join over big collections.
+- **Speculative win**: O(n·m) → ~O(n+m) for hash; ~O(n log n + m log m) for sort-merge if sorting is needed, ~O(n+m) if already sorted.
+- **Cost / risk**: build-side memory (hash); sort cost (sort-merge); both degrade with key skew.
+- **Validate by**: benchmark across input sizes.
+
+### 7. Partition pruning
+- **Borrowed from**: partitioned tables, time-series databases.
+- **The idea**: organize data into partitions (by time/key range) and skip whole partitions that can't match a query.
+- **Code signals**: scanning all data when queries are bounded by time/range; growing scan cost over time.
+- **Speculative win**: skips most of the dataset for bounded queries.
+- **Cost / risk**: partition scheme must match query patterns; cross-partition queries.
+- **Validate by**: partitions scanned per query; timing.
+
+### 8. Compression / encoding (dictionary, RLE, delta)
+- **Borrowed from**: columnar stores, time-series compression.
+- **The idea**: encode data compactly (dictionary-encode repeated values, run-length for runs, delta for sequences) to shrink scans and fit more in cache/memory.
+- **Code signals**: large low-cardinality or sequential datasets stored verbosely.
+- **Speculative win**: less data to scan/transfer; sometimes operate directly on encoded form.
+- **Cost / risk**: encode/decode CPU; random access into encoded blocks.
+- **Validate by**: size reduction; scan benchmark on encoded data.
+
+### 9. Append-only / sequential write design
+- **Borrowed from**: write-ahead logs, event logs, journaling.
+- **The idea**: write sequentially (append) and treat the log as the source of truth, deriving state from it — sequential I/O is far faster than random on both SSD and disk.
+- **Code signals**: random in-place file/record updates; scattered small writes.
+- **Speculative win**: sequential throughput; simpler durability/crash recovery.
+- **Cost / risk**: compaction/snapshotting to bound size; read-path derivation.
+- **Validate by**: write throughput; recovery correctness.
+
+### 10. MVCC / snapshot reads instead of read locks
+- **Borrowed from**: snapshot isolation (Postgres keeps new row versions in-table + VACUUM; InnoDB keeps the latest in-table and reconstructs old versions from undo logs).
+- **The idea**: keep versioned data so readers see a consistent snapshot without blocking writers (and vice versa), removing reader-writer lock contention.
+- **Code signals**: read locks blocking writers (or the reverse) on a hot store; lock contention between reads and writes.
+- **Speculative win**: readers and writers might stop blocking each other if contention is genuinely reader-writer (not write-write).
+- **Cost / risk**: version storage and garbage collection (Postgres bloat / InnoDB undo growth); complexity.
+- **Validate by**: contention/off-CPU profiling; concurrency benchmark.
+
+### 11. Precomputed / incremental aggregates (summary tables)
+- **Borrowed from**: materialized aggregates, time-series rollups.
+- **The idea**: maintain rollup/summary tables (counts, sums, histograms per bucket) updated incrementally on write, so aggregate queries read the summary instead of scanning raw rows.
+- **Code signals**: expensive group-by/aggregate over raw rows per query.
+- **Speculative win**: aggregate reads may become small lookups if queries align with the precomputed buckets.
+- **Cost / risk**: summary maintenance on write; staleness if async.
+- **Validate by**: aggregate query latency; rows scanned.
+
+### 12. Zone maps / min-max indexes (data skipping)
+- **Borrowed from**: ClickHouse skip indexes, Snowflake zone maps, Parquet row-group stats.
+- **The idea**: store per-block min/max (and optionally null/count) metadata so a predicate can skip whole blocks whose range can't match — pruning before any data is read or decompressed. Cheap to maintain and most effective when data is sorted/clustered on the filtered column.
+- **Code signals**: range/equality filters over large block-structured or chunked data; blocks read then fully discarded by a `WHERE`-style filter.
+- **Speculative win**: skips most blocks for selective, range-correlated queries.
+- **Cost / risk**: near-useless if values are randomly distributed across blocks (overlapping ranges); small metadata overhead.
+- **Validate by**: blocks/row-groups pruned per query; bytes scanned before/after.
+
+### 13. Late materialization (defer row reconstruction)
+- **Borrowed from**: columnar OLAP execution (Vertica/C-Store lineage, DuckDB, Redshift).
+- **The idea**: carry row IDs / position lists through filters and joins, fetching the non-predicate columns only for rows that survive — instead of assembling full rows up front (early materialization).
+- **Code signals**: full objects/rows hydrated before filtering; expensive fields loaded for rows later discarded.
+- **Speculative win**: avoids materializing columns for filtered-out rows; less memory and I/O on selective queries.
+- **Cost / risk**: random-access tuple reconstruction can hurt for low-selectivity queries; more complex operators.
+- **Validate by**: rows/columns materialized vs rows returned; query benchmark across selectivities.
+
+### 14. Compressed bitmap index (roaring) for filtering
+- **Borrowed from**: OLAP/search bitmap indexes (Druid, Pinot, Lucene; Roaring bitmaps).
+- **The idea**: represent each value's matching row set as a compressed bitmap, then resolve multi-predicate filters with fast bitwise AND/OR/NOT over those bitmaps instead of scanning rows.
+- **Code signals**: multi-attribute equality/set filters combined with AND/OR over a large dataset; repeated tag/category/flag filtering.
+- **Speculative win**: filter resolution could reduce to bitwise set operations, cheaply intersecting many predicates, if cardinality is low-to-medium.
+- **Cost / risk**: best for low-to-medium cardinality; index build/storage and update cost on writes.
+- **Validate by**: rows scanned vs matched; multi-predicate filter benchmark.
+
+### 15. Index intersection / multi-index AND (bitmap)
+- **Borrowed from**: relational query optimizers — Postgres "Combining Multiple Indexes" (BitmapAnd / BitmapOr over per-index bitmaps), Oracle bitmap index AND.
+- **The idea**: rather than one wide composite index per query shape, let the optimizer combine several single-column indexes at query time — scan each into an in-memory bitmap and AND/OR them — so a mix of queries touching different column subsets is served by a few reusable indexes instead of a combinatorial set of composites.
+- **Code signals**: ad-hoc multi-column `WHERE` filters whose columns vary per query; many overlapping composite indexes maintained to cover every combination; one predicate indexed while another forces a residual scan.
+- **Speculative win**: a multi-predicate filter might be answered by intersecting per-column bitmaps if each predicate is individually selective, avoiding both a full scan and a proliferation of composite indexes.
+- **Cost / risk**: each extra index scan adds cost, so the planner may still prefer one composite or a single index; bitmap loses index ordering (an `ORDER BY` then needs a sort); maintaining several indexes costs write time.
+- **Validate by**: query plan shows a BitmapAnd/intersection (not a full scan); rows scanned vs matched; benchmark vs a single composite index — profile via perf-design.