@zigrivers/scaffold 3.22.0 → 3.24.0

package/content/knowledge/data-science/data-science-observability.md

@@ -0,0 +1,161 @@
+---
+name: data-science-observability
+description: Monitoring deployed DS models and pipelines — prediction logging to Parquet, scheduled evaluation, basic drift detection, and Evidently for deeper analysis
+topics: [data-science, observability, monitoring, drift, evidently]
+---
+
+Models don't fail loudly. A scoring job keeps running, rows keep landing in the output table, dashboards stay green — and quietly, the predictions get worse. The world drifts away from whatever snapshot you trained on, and nobody notices until a stakeholder says "these numbers look weird." Observability for a solo DS isn't a platform; it's a small set of habits that give you a chance to catch decay before someone else does.
+
+## Summary
+
+For a solo or small-team data scientist with something deployed (even just a weekly cron), observability boils down to four habits: log every prediction with its inputs to a dated Parquet file, re-run your evaluation script on a schedule and alert on metric drops, check a handful of key features for distributional drift, and reach for `Evidently` when you want a pre-built drift report instead of writing your own. The goal is a tripwire, not a dashboard — you want to get paged when something's wrong, not stare at graphs hoping to spot it.
+
+## Deep Guidance
+
+### Log predictions + inputs
+
+Every time your model scores something, append a row to a dated Parquet log. This is the single most useful thing you can do for future-you — drift analysis, debugging, label backfill, and post-mortems all depend on having this log.
+
+Layout:
+
+```
+data/processed/predictions/
+  2026-04-21/
+    run-20260421T0300-abc123.parquet
+  2026-04-22/
+    run-20260422T0300-def456.parquet
+```
+
+Schema (one row per prediction):
+
+```python
+# src/monitor/prediction_log.py
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+import pandas as pd
+
+def log_predictions(
+    features: pd.DataFrame,
+    predictions: pd.Series,
+    model_version: str,
+    log_root: Path = Path("data/processed/predictions"),
+) -> Path:
+    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+    run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M") + "-" + uuid.uuid4().hex[:6]
+    out_dir = log_root / today
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    df = features.copy()
+    df["prediction"] = predictions.values
+    df["model_version"] = model_version
+    df["logged_at"] = datetime.now(timezone.utc)
+    df["run_id"] = run_id
+    df["ground_truth"] = pd.NA  # Backfilled later when labels arrive
+
+    out = out_dir / f"run-{run_id}.parquet"
+    df.to_parquet(out, index=False)
+    return out
+```
+
+Parquet is right for this: columnar, compressed, fast to scan across dates with `pd.read_parquet("data/processed/predictions/**/*.parquet")`. If inputs have PII, hash or drop those columns before logging — you rarely need raw identifiers to do drift or error analysis.
+
+### Scheduled eval re-runs
+
+Your training-time evaluation script is also your monitoring script. Run it weekly or monthly against recent predictions joined to whatever ground truth has arrived, and alert when the headline metric breaches a threshold.
+
+```python
+# src/monitor/eval.py
+import sys
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+THRESHOLD = 0.80  # Alert if AUC drops below this
+
+def main() -> int:
+    preds = pd.read_parquet("data/processed/predictions/")
+    labels = pd.read_parquet("data/processed/labels/")
+    joined = preds.merge(labels, on="record_id", how="inner")
+    if len(joined) < 500:
+        print("Not enough labeled data yet; skipping.")
+        return 0
+
+    auc = roc_auc_score(joined["actual"], joined["prediction"])
+    print(f"AUC on {len(joined)} labeled rows: {auc:.3f}")
+    if auc < THRESHOLD:
+        # Send email / Slack webhook here
+        print(f"ALERT: AUC {auc:.3f} below threshold {THRESHOLD}", file=sys.stderr)
+        return 1
+    return 0
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+```
+
+Schedule it with whatever you already have — cron, a GitHub Actions `schedule:` workflow, Airflow if you run it, or your platform's scheduled job. Exit code 1 plus a Slack webhook is a perfectly good alerting system at this scale.
+
+### Basic drift detection
+
+Before reaching for a library, do the cheap thing: compare this period's feature distribution to your training distribution. Mean, std, a couple of quantiles, and a KS statistic cover most of what you need.
+
+```python
+# src/monitor/drift.py
+from scipy.stats import ks_2samp
+import pandas as pd
+
+def feature_drift(reference: pd.Series, current: pd.Series) -> dict:
+    stat, p = ks_2samp(reference.dropna(), current.dropna())
+    return {
+        "ref_mean": reference.mean(), "cur_mean": current.mean(),
+        "ref_std": reference.std(),   "cur_std": current.std(),
+        "ks_stat": stat, "ks_p": p,
+        "drifted": p < 0.01,
+    }
+
+train = pd.read_parquet("data/processed/train.parquet")
+recent = pd.read_parquet("data/processed/predictions/2026-04-21/")
+for col in ["amount", "user_tenure_days", "n_items"]:
+    print(col, feature_drift(train[col], recent[col]))
+```
+
+Run this alongside your scheduled eval. You don't need a dashboard — printing to the job log and alerting on `drifted=True` on any monitored feature is enough.
+
+### Evidently for more
+
+When you outgrow ad-hoc KS tests, `Evidently` gives you a pre-built drift report across all features, plus data quality checks and target drift, as an HTML page you can open or ship to S3.
+
+```python
+# src/monitor/evidently_report.py
+import pandas as pd
+from evidently import Report
+from evidently.presets import DataDriftPreset
+
+reference = pd.read_parquet("data/processed/train.parquet")
+current = pd.read_parquet("data/processed/predictions/2026-04-21/")
+
+report = Report([DataDriftPreset()])
+snapshot = report.run(reference_data=reference, current_data=current)
+snapshot.save_html("reports/drift-2026-04-21.html")
+```
+
+This is opt-in. If plain pandas + SciPy is telling you what you need to know, don't add a dependency. Reach for Evidently when you have enough features that per-column code is tedious, or when you want a shareable artifact for a stakeholder.
+
+### The prediction / feedback loop
+
+Ground truth almost never arrives at prediction time. A churn model predicts today who'll cancel next month; a fraud model predicts now whether a transaction is bad, confirmed days later. That delay is why the Parquet log exists — you keep predictions around until labels catch up, then join.
+
+```python
+# src/monitor/backfill_labels.py
+import pandas as pd
+
+preds = pd.read_parquet("data/processed/predictions/")
+labels = pd.read_parquet("data/processed/labels/")  # record_id, actual, label_time
+merged = preds.merge(labels, on="record_id", how="left")
+merged.to_parquet("data/processed/predictions_labeled.parquet", index=False)
+```
+
+Keep at least one full feedback cycle of prediction logs (if labels arrive after 30 days, keep 60-90 days). This join is how you get a real accuracy number on production traffic, not just your static test-set number from training day.
+
+### What NOT to build
+
+Resist the urge to over-engineer. At solo scale you do not need streaming drift detection, a Prometheus/Grafana stack, a model registry with canary deploys, or a dedicated monitoring dashboard. Those are ML-platform-team concerns — build them when there's a team to own them. A dated Parquet log, a scheduled eval script, a handful of drift checks, and an alert that emails you is more than enough to catch the failures that actually happen.

package/content/knowledge/data-science/data-science-project-structure.md

@@ -0,0 +1,178 @@
+---
+name: data-science-project-structure
+description: Opinionated directory layout for solo and small-team data-science projects — notebooks, src, data, models, reports, tests, configs — with a promotion path from exploration to tested modules
+topics: [data-science, project-structure, layout]
+---
+
+A solo data-science project accumulates artifacts faster than most software: half-finished notebooks, CSV dumps, parquet caches, serialized models, PNG charts, and the occasional markdown write-up. Without a deliberate directory structure, the project turns into a folder of 40 loose files within a month and a new contributor — including future-you — cannot tell what is canonical, what is scratch, and what is safe to delete. A clear layout fixes three problems at once: discoverability (where does X live?), git hygiene (what is tracked vs generated?), and the promotion path (how does throwaway notebook code become tested library code?).
+
+## Summary
+
+A solo DS project has six top-level directories that each answer one question: `notebooks/` (exploration), `src/` (importable Python modules), `data/` (split into raw/interim/processed — `data/raw/` is always gitignored; small processed artifacts may be committed or DVC-tracked), `models/` (serialized artifacts, tracked via DVC or git-lfs), `reports/` (rendered outputs — figures, HTML, markdown), and `tests/` (pytest suite mirroring `src/`). `configs/` holds YAML run parameters, and `pyproject.toml` at the root defines the package. The `.gitignore` excludes raw data, most of `models/`, and common binary formats that were not deliberately promoted. Reusable logic follows a strict promotion path: explored in a notebook, extracted into `src/`, unit-tested in `tests/`, then re-imported by notebooks or pipeline scripts.
+
+## Deep Guidance
+
+### Top-level layout
+
+```
+project-root/
+├── notebooks/          # Exploratory notebooks (Marimo preferred; numbered chronologically)
+├── src/                # Importable Python modules — the library
+│   └── <project>/
+│       ├── __init__.py
+│       ├── ingestion.py    # Load raw data from source (CSV, DB, API)
+│       ├── features.py     # Feature engineering / transforms
+│       ├── training.py     # Model fitting routines
+│       ├── evaluation.py   # Metrics, CV loops, slice analysis
+│       └── serving.py      # Inference helpers (load artifact, predict)
+├── data/               # Datasets at every pipeline stage
+│   ├── raw/            # Immutable inputs — GITIGNORED (always)
+│   ├── interim/        # Cached intermediates — small Parquet may be committed
+│   └── processed/      # Analysis-ready — usually DVC-tracked; small files may be committed
+├── models/             # Serialized model artifacts (DVC / git-lfs tracked)
+├── reports/            # Rendered output: figures/, HTML reports, markdown summaries
+│   └── figures/
+├── tests/              # pytest suite — mirrors src/ structure
+├── configs/            # YAML run configs (Hydra-style or plain)
+├── pyproject.toml      # Package metadata, dependencies, tool config
+├── .gitignore
+└── README.md
+```
+
+One-liners per dir:
+- `notebooks/` — exploration, EDA, prototyping; numbered `01-…`, `02-…` so ordering is obvious
+- `src/` — every reusable function that a second notebook or a pipeline script will call
+- `data/` — all datasets at every stage; raw is always gitignored, selected processed artifacts (small Parquet in `data/interim/` or `data/processed/`) may be committed directly or tracked via DVC — see `data-science-data-versioning`
+- `models/` — trained model artifacts; tracked through DVC or git-lfs pointers, never raw binaries
+- `reports/` — things a human reads: charts, HTML reports, markdown summaries
+- `tests/` — pytest tests for code in `src/`
+- `configs/` — experiment parameters (paths, seeds, hyperparams) separate from code
+
+### Data: gitignore raw, deliberately admit small processed artifacts
+
+The single hardest rule in DS project hygiene: **never commit raw datasets under `data/raw/` or raw model binaries under `models/` to git**. A 200 MB parquet file committed to history is permanent — `git filter-repo` is the only cure and it rewrites every commit. Prevent the problem at the `.gitignore` layer before it happens.
+
+Gitignoring the entire `data/` tree is the safest default, but it under-serves a common small-team workflow: a cleaned, analysis-ready Parquet in `data/interim/` that's <10 MB, changes rarely, and is useful to have alongside the code. See `data-science-data-versioning` for the full size-based decision rule. The pattern below gitignores raw data and external copies wholesale, and allows opt-in commits of small processed Parquet through a deliberate un-ignore rule. Anything larger (>50 MB, frequent churn, binary artifacts) goes through DVC or git-lfs instead — never direct git commits.
+
+```gitignore
+# Raw / external data — never committed (bulky, usually not redistributable)
+data/raw/
+data/external/
+
+# Processed / interim data — default: ignore; opt in to specific small artifacts below
+data/interim/*
+data/processed/*
+!data/.gitkeep
+!data/interim/.gitkeep
+!data/processed/.gitkeep
+# Allow small cleaned Parquet to be committed (see data-science-data-versioning
+# for size guidance — under ~10 MB, rare changes). Larger artifacts belong in
+# DVC or git-lfs.
+!data/interim/*.parquet
+!data/processed/*.parquet
+
+# Model artifacts — tracked via DVC or git-lfs, not raw binaries
+models/
+!models/.gitkeep
+!models/**/*.dvc
+
+# Common large binary formats (defense in depth — catch anything dropped elsewhere)
+*.feather
+*.joblib
+*.pt
+*.pth
+*.onnx
+*.h5
+*.hdf5
+*.npy
+*.npz
+
+# Python
+__pycache__/
+*.pyc
+.venv/
+.ruff_cache/
+.pytest_cache/
+*.egg-info/
+
+# Notebook outputs (if not using a tool that strips them)
+.ipynb_checkpoints/
+
+# Environment / secrets
+.env
+.env.*
+!.env.example
+```
+
+Two things are load-bearing in this snippet. First, `*.parquet` is **not** in the blanket block-list — we want `data/interim/*.parquet` to match as "allowed" once the un-ignore rules kick in. Second, the `!data/interim/*.parquet` and `!data/processed/*.parquet` patterns mean processed Parquet is committable **by default** at this layer; the policy choice of whether to actually commit a given file is made at `git add` time, not in `.gitignore`. If your team's policy is DVC-first for every dataset, drop those `!…*.parquet` lines. The `!data/.gitkeep` family keeps the directories present in fresh clones.
+
+For versioned datasets and models, see `data-science-data-versioning` — DVC or git-lfs pointers are committed, the binaries themselves live in remote storage. Prefer `joblib` or framework-native formats (`.pt`, `.onnx`) over stdlib pickle for model artifacts — pickle loads execute arbitrary code, so a model file from an untrusted source becomes an RCE vector.
+
+### Notebooks → src/ promotion
+
+Notebooks are for exploration, not production. The moment a function in a notebook becomes useful to a second notebook — or looks like it will survive longer than the current sitting — it gets promoted:
+
+1. **Identify**: a cell (or few cells) encapsulating reusable logic — a loader, a transform, a metric computation
+2. **Extract**: move the function into the appropriate `src/<project>/` module (`ingestion.py`, `features.py`, etc.) with type hints and a docstring
+3. **Test**: add a pytest case in `tests/` that exercises a representative input → output case
+4. **Re-import**: the notebook now does `from <project>.features import clean_customer_ids` instead of defining the function inline
+
+This discipline keeps notebooks short (exploration, narrative, charts) and concentrates correctness-critical code where it can be reviewed, tested, and reused. See `notebook-discipline` for the mechanics of cell size, output clearing, and `%autoreload` so edits in `src/` are picked up in the notebook without a kernel restart.
+
+### Configs and reproducibility
+
+Hard-coded paths and hyperparameters inside notebook cells are the single biggest reproducibility killer in a DS project. Push them into `configs/` so a run is defined by a config file + a git SHA.
+
+```yaml
+# configs/train_baseline.yaml
+run_name: baseline_v1
+seed: 42
+
+data:
+  raw_path: data/raw/transactions_2024.csv
+  processed_path: data/processed/transactions_clean.parquet
+  target: churned_30d
+  test_size: 0.2
+  split_seed: 42
+
+features:
+  include:
+    - tenure_days
+    - monthly_spend
+    - support_tickets_30d
+  log_transform:
+    - monthly_spend
+
+model:
+  type: gradient_boosting
+  params:
+    n_estimators: 200
+    max_depth: 5
+    learning_rate: 0.05
+
+output:
+  model_path: models/baseline_v1.joblib
+  report_path: reports/baseline_v1.html
+```
+
+Training code reads the config with `yaml.safe_load` (or Hydra / pydantic-settings for richer projects) and a teammate can reproduce the run with `python -m <project>.training --config configs/train_baseline.yaml`. For Hydra specifically, configs split into `configs/data/`, `configs/model/`, `configs/training/` and compose at the command line.
+
+### Tests layout
+
+`tests/` mirrors `src/` one-to-one. If `src/<project>/features.py` defines `clean_customer_ids`, then `tests/test_features.py` contains `test_clean_customer_ids_strips_whitespace` and friends.
+
+```
+tests/
+├── conftest.py             # Shared fixtures (tiny sample dataframes, tmp_path helpers)
+├── test_ingestion.py       # Tests for src/<project>/ingestion.py
+├── test_features.py        # Tests for src/<project>/features.py
+├── test_training.py        # Tests for src/<project>/training.py — usually smoke tests
+└── test_evaluation.py      # Tests for src/<project>/evaluation.py
+```
+
+Naming rules:
+- Test files: `test_<module>.py` — pytest discovers these by default
+- Test functions: `test_<unit>_<behavior>` — e.g. `test_clean_customer_ids_strips_whitespace`, `test_load_transactions_raises_on_missing_file`
+- Fixtures live in `conftest.py` at the `tests/` root when shared across files; local fixtures stay in the file that uses them
+
+Training and evaluation tests are typically **smoke tests** over a 10-row fixture dataframe, not full-dataset runs — the goal is catching shape/dtype/column regressions, not validating model quality (model quality belongs in the evaluation report, not the unit test suite).

package/content/knowledge/data-science/data-science-reproducibility.md

@@ -0,0 +1,164 @@
+---
+name: data-science-reproducibility
+description: Reproducibility for solo/small-team DS — pin deps with uv lock, seed everything, set PYTHONHASHSEED, and reach for Docker only at OS boundaries
+topics: [data-science, reproducibility, determinism, uv, docker]
+---
+
+You show a result in Monday's meeting. Six months later, on a new laptop, you can't reproduce it. Three things usually cause this: dependencies drifted (a minor NumPy release changed a default), randomness wasn't pinned (a shuffle or init picked a different seed), or the data changed underneath you. Reproducibility is the discipline of eliminating all three so the same inputs always produce the same numbers.
+
+## Summary
+
+Pin dependencies with `uv lock` and commit `uv.lock` — `uv sync --frozen` rebuilds the exact environment anywhere. Control randomness with a single `set_seed(seed)` helper that seeds Python `random`, NumPy, PyTorch, and TensorFlow at the top of every script. Export `PYTHONHASHSEED=0` via `.envrc` so hash-order is deterministic across interpreter runs. Log the git SHA and data hash with every run so you can walk back to the exact code + data that produced any number. Reach for Docker only when you're crossing an OS or CUDA boundary — for greenfield solo work, `uv sync` is enough.
+
+## Deep Guidance
+
+### Pinning dependencies with uv
+
+`uv` resolves the full transitive dependency graph into `uv.lock`, which records the exact version and content hash of every package, including transitive deps you never directly imported. Commit it. On a new machine, `uv sync --frozen` reproduces the environment byte-for-byte without re-resolving anything.
+
+```bash
+# First time: declare top-level deps in pyproject.toml, then lock
+uv lock
+
+# On any machine (CI, teammate's laptop, 6 months later):
+uv sync --frozen       # install exactly what's in uv.lock, never re-resolve
+
+# Upgrade a single package intentionally:
+uv lock --upgrade-package numpy
+# Review the lock diff in PR. Re-run your eval suite before merging.
+
+# Add a new dependency:
+uv add pandas          # updates pyproject.toml AND uv.lock atomically
+```
+
+Rules:
+- Commit `uv.lock`. It is not a build artifact; it is a reproducibility contract.
+- Use `--frozen` in CI and release scripts. A silent re-resolve on deploy is the bug you're trying to prevent.
+- Upgrade packages one at a time, with a PR and an eval run. Bulk upgrades hide which bump broke your metrics.
+- Pin the Python version too: add `requires-python = "==3.12.*"` in `pyproject.toml` and let uv install and manage the interpreter. Minor Python versions change float formatting, dict ordering guarantees, and stdlib behavior in ways that can move your numbers.
+
+### Seed management
+
+Every source of randomness in your stack has its own PRNG. Seed all of them from a single call, at the top of every train/eval/predict entry point.
+
+```python
+# src/utils/seed.py
+import os
+import random
+import numpy as np
+
+def set_seed(seed: int = 42) -> None:
+    """Seed every PRNG we might touch. Call at the top of every script."""
+    os.environ["PYTHONHASHSEED"] = str(seed)
+    random.seed(seed)
+    np.random.seed(seed)
+
+    try:
+        import torch
+        torch.manual_seed(seed)
+        if torch.cuda.is_available():
+            torch.cuda.manual_seed_all(seed)
+    except ImportError:
+        pass
+
+    try:
+        import tensorflow as tf
+        tf.random.set_seed(seed)
+    except ImportError:
+        pass
+```
+
+Call `set_seed(42)` before any data split, model init, or sampling. If a library accepts a `random_state` argument (scikit-learn does almost everywhere), pass the seed explicitly — global seeding is a safety net, not a substitute.
+
+```python
+# Explicit is better than implicit:
+from sklearn.model_selection import train_test_split
+from sklearn.ensemble import RandomForestClassifier
+
+set_seed(42)  # global safety net
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y, test_size=0.2, random_state=42      # explicit
+)
+model = RandomForestClassifier(random_state=42)  # explicit
+```
+
+The one gotcha: multi-worker DataLoaders in PyTorch spawn subprocesses that need their own seeding. Pass `worker_init_fn` to seed each worker, or you'll get different augmentation sequences across runs even with `set_seed` called in the main process.
+
+### Hash determinism
+
+Python randomizes the hash seed per interpreter run by default. That means dict iteration order, set iteration order, and anything that depends on `hash()` varies between runs — a subtle reproducibility leak that only shows up when you try to diff two training runs.
+
+```bash
+# .envrc (direnv)
+export PYTHONHASHSEED=0
+```
+
+`set_seed()` sets this too, but exporting it in `.envrc` covers everything in the shell session — notebooks, ad-hoc scripts, the test runner — before any Python code runs.
+
+### GPU determinism (brief)
+
+Full GPU determinism requires cuDNN-level flags and disabling non-deterministic kernels:
+
+```python
+# Only if you actually need this:
+torch.backends.cudnn.deterministic = True
+torch.backends.cudnn.benchmark = False
+```
+
+This has a real performance cost (often 10-30% slower training) and doesn't cover every op. For DS-1, don't chase it. CPU-level determinism from `set_seed()` + pinned deps is enough for 95% of analyses. Reach for GPU determinism only under regulatory requirement, scientific publication, or when debugging a numerics bug that you can't otherwise isolate.
+
+### Git SHA and data versioning
+
+A reproducible run needs four things pinned: code, dependencies, randomness, and data. We've covered three. For code, log the git SHA with every experiment (see `data-science-experiment-tracking.md` for the logging pattern — don't duplicate the plumbing here). For data, hash the input dataset or pin a DVC / lakeFS / Git-LFS reference (see `data-science-data-versioning.md`).
+
+The minimum metadata for any reported result:
+
+```text
+git_sha:     a1b2c3d4
+uv_lock:     sha256:...          # hash of uv.lock
+seed:        42
+data_hash:   sha256:...          # hash of the input dataset(s)
+python:      3.12.1
+platform:    darwin-arm64
+```
+
+If all five match, the numbers should match. If any differ, you know exactly which knob moved.
+
+A working pattern: log these fields into your experiment tracker alongside metrics, and include them in any reported result (paper, slide, dashboard tile). The friction cost is near zero once automated; the debugging cost of a result you can't trace back to its exact code + data is enormous.
+
+### Docker: only at OS boundaries
+
+Docker solves a real problem: "it works on my Mac but not on the Linux GPU box." It does not solve "I forgot to commit `uv.lock`." Reach for containers when you're genuinely crossing a boundary:
+
+- Developing on macOS, deploying on Linux — native wheels differ, BLAS differs, occasionally results differ.
+- CUDA version mismatch between dev and prod GPUs.
+- A team standardizing a shared prod environment where `uv sync` isn't enough because the base OS libs drift.
+
+For a solo greenfield project on one laptop, a Dockerfile is pure overhead. Start with `uv sync --frozen` and add Docker the first time you actually hit a cross-OS reproducibility failure — not before.
+
+When you do reach for it, keep the image minimal and derived from your lockfile:
+
+```dockerfile
+FROM python:3.12-slim
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
+WORKDIR /app
+COPY pyproject.toml uv.lock ./
+RUN uv sync --frozen --no-dev
+COPY src/ ./src/
+ENV PYTHONHASHSEED=0
+CMD ["uv", "run", "python", "-m", "src.train"]
+```
+
+Pin the base image by digest (`python:3.12-slim@sha256:...`) once the project is in prod — floating tags drift and will silently give you a different glibc next month.
+
+### Reproducibility checklist
+
+Before calling any analysis "done":
+
+- `uv.lock` committed and current (`uv sync --frozen` in CI succeeds)
+- `set_seed()` called at the top of every entry point
+- `PYTHONHASHSEED=0` in `.envrc` (and `.envrc` committed, `.env` gitignored)
+- Git SHA + data hash logged with every experiment run
+- Eval suite passes on a clean clone in CI — the real test of reproducibility is a fresh machine, not your own
+

package/content/knowledge/data-science/data-science-requirements.md

@@ -0,0 +1,151 @@
+---
+name: data-science-requirements
+description: Problem framing, success metrics, evaluation-test design, stakeholder contracts, and nonfunctional requirements for solo/small-team data science projects
+topics: [data-science, requirements, evaluation, success-metrics, reproducibility]
+---
+
+As a solo or small-team data scientist without an existing data platform, the single biggest risk to your project is not a bad model — it is ambiguous requirements. Without a tight written spec, a DS project sprawls: the question drifts week to week, the notebook becomes unreproducible, and the stakeholder quietly reinterprets the output. This document defines what "done" looks like for an analytical pipeline, model, or report built from scratch — so you can stop work on time and defend the result.
+
+## Summary
+
+A data-science requirements doc states a single well-framed question, one primary success metric with a numeric acceptance threshold declared before any modeling, an evaluation design using held-out data, a stakeholder contract (who consumes the output, in what format, on what cadence), and a nonfunctional budget (reproducibility, runtime, storage). Write the target threshold into a test before you touch training data. If you cannot name the metric and the number, you are not ready to start.
+
+## Deep Guidance
+
+### Problem framing
+
+Most DS projects fail at step one: the question is fuzzy ("understand churn") rather than decidable ("predict 30-day churn for active paying users, with recall >= 0.6 at precision >= 0.3"). The discipline is to force yourself, in writing, to name the decision the output will drive. If you cannot name that decision, stop and interview the stakeholder until you can.
+
+Use a short, copyable problem-statement block at the top of your project README or PRD. The one below is opinionated — it forces every ambiguous field to get filled in before modeling starts. The tradeoff: for pure exploratory work (e.g. a one-off investigation) this is overkill; a 3-line hypothesis is enough.
+
+```yaml
+# docs/problem-statement.yaml
+question: >
+  For monthly paying users active in the last 30 days, predict whether they
+  will cancel their subscription within the next 30 days.
+decision_driven:
+  who: Growth team
+  action: Enroll top-decile predicted churners in a retention email campaign
+  cadence: Weekly scoring
+unit_of_analysis: user_id x scoring_date
+prediction_target: churn_within_30d (bool)
+out_of_scope:
+  - free-tier users
+  - annual subscribers
+  - users less than 14 days old at scoring time
+known_confounders:
+  - planned price change on 2026-05-01
+  - seasonality around end-of-year
+```
+
+### Success metrics
+
+State the primary success metric and its acceptance threshold in writing before you train anything. The number comes from the stakeholder contract, not from what the model can achieve — otherwise you are reverse-engineering the bar to whatever you got. Pick one primary metric; secondary metrics are tie-breakers, not co-equals.
+
+Typical patterns:
+
+- **Predictive model**: one primary metric tied to the downstream decision. For a ranked retention campaign, `recall@top-10%` or `precision@k` beats accuracy or raw AUC, because the campaign can only email the top decile.
+- **Regression / forecast**: `RMSE` in the target's natural unit, plus a naive baseline (last-value, rolling-mean). Beating the baseline is mandatory; if you cannot, the project is not viable.
+- **Analytical pipeline / ETL**: functional correctness plus a p95 runtime budget (e.g. "daily job must finish in < 20 min on the scheduled box").
+- **Report / dashboard**: domain acceptance threshold — the numbers in the report must match an independently computed source-of-truth query within a stated tolerance (e.g. "<= 0.1% deviation from the finance ledger").
+
+Encode the success metric as a function so it is unambiguous and testable. The expression below is the whole contract — write it the day you start.
+
+```python
+# src/metrics.py
+from sklearn.metrics import precision_recall_curve
+import numpy as np
+
+TARGET_RECALL = 0.60
+MIN_PRECISION = 0.30  # at the threshold that achieves TARGET_RECALL
+
+def primary_metric(y_true: np.ndarray, y_score: np.ndarray) -> dict:
+    """Primary success metric: precision at the threshold that hits target recall."""
+    precision, recall, thresholds = precision_recall_curve(y_true, y_score)
+    # Walk from highest threshold down; stop when recall crosses target.
+    idx = np.searchsorted(recall[::-1], TARGET_RECALL)
+    idx = len(recall) - 1 - idx
+    return {
+        "recall": float(recall[idx]),
+        "precision": float(precision[idx]),
+        "threshold": float(thresholds[min(idx, len(thresholds) - 1)]),
+        "passes": bool(recall[idx] >= TARGET_RECALL and precision[idx] >= MIN_PRECISION),
+    }
+```
+
+### Evaluation-test design
+
+The evaluation test is the single gate between "training run" and "ship it." Its job is to answer one question: does the model hit the stated metric on data it has not seen? Get this wrong — leak the future into the past, evaluate on training rows — and every downstream decision is poisoned.
+
+Opinionated defaults:
+
+- **Temporal target**: split by time, not randomly. Train on `[t0, t1)`, hold out `[t1, t2)`. Random splits with temporal data leak future information and will silently inflate metrics.
+- **Non-temporal target**: stratified split by the label, fixed `random_state`, held-out fraction 15-20%.
+- **Small data (< 10k rows)**: 5-fold cross-validation with the same fold seed every run; report mean plus std of the primary metric.
+- **Never** tune hyperparameters on the holdout. Use a third validation split or inner CV. Tradeoff: if your dataset is tiny you may have to pool — document the risk explicitly.
+
+The evaluation belongs in the test suite, not a notebook. The stakeholder should be able to run `pytest tests/test_model_evaluation.py` and see green before accepting the deliverable.
+
+```python
+# tests/test_model_evaluation.py
+import joblib
+import pandas as pd
+import pytest
+from src.metrics import primary_metric, TARGET_RECALL, MIN_PRECISION
+
+HOLDOUT_PATH = "data/holdout_2026_q1.parquet"
+MODEL_PATH = "artifacts/churn_model.pkl"
+
+@pytest.fixture(scope="module")
+def scored_holdout():
+    df = pd.read_parquet(HOLDOUT_PATH)
+    model = joblib.load(MODEL_PATH)
+    X = df.drop(columns=["churn_within_30d"])
+    y_true = df["churn_within_30d"].to_numpy()
+    y_score = model.predict_proba(X)[:, 1]
+    return y_true, y_score
+
+def test_model_beats_acceptance_threshold(scored_holdout):
+    y_true, y_score = scored_holdout
+    result = primary_metric(y_true, y_score)
+    assert result["passes"], (
+        f"Model failed acceptance: recall={result['recall']:.3f} "
+        f"(target {TARGET_RECALL}), precision={result['precision']:.3f} "
+        f"(min {MIN_PRECISION})"
+    )
+
+def test_model_beats_naive_baseline(scored_holdout):
+    # Baseline: predict global churn rate for everyone. Any real model must beat it.
+    y_true, y_score = scored_holdout
+    baseline_score = pd.Series([y_true.mean()] * len(y_true)).to_numpy()
+    assert primary_metric(y_true, y_score)["precision"] > \
+           primary_metric(y_true, baseline_score)["precision"]
+```
+
+### Stakeholder contract
+
+A stakeholder contract makes the hand-off concrete. Without it, you deliver a notebook and the recipient quietly asks for a PDF, a Slack message, a dashboard, or a CSV — all different artifacts. Write this down the same week you write the problem statement.
+
+Minimum fields, in order of how often they get skipped:
+
+- **Consumer**: named human or team, not "the business."
+- **Artifact format**: one of `csv`, `parquet`, `dashboard (URL)`, `API endpoint`, `PDF report`, `Slack summary`. Pick exactly one primary.
+- **Schema**: column names, types, units, PII flags. Include an example row.
+- **Cadence**: one-shot, daily, weekly, on-demand. If recurring, name the day-of-week and time-of-day.
+- **Freshness SLA**: how stale is the underlying data allowed to be at delivery time.
+- **Failure behavior**: what happens if the pipeline fails — silent retry, page the owner, stale-serve, fail loud.
+- **Sunset criteria**: when does this deliverable stop being needed. If you cannot answer, the project has no natural end.
+
+A one-off analysis can collapse this into a single paragraph; a recurring pipeline needs all seven fields in a short `CONTRACT.md` alongside the code.
+
+### Nonfunctional requirements
+
+Nonfunctional requirements are what separates a notebook from a deliverable. Three to name explicitly:
+
+- **Reproducibility**: the pipeline must produce byte-identical outputs given identical inputs. That means a pinned `requirements.txt` (or `pyproject.toml` + lockfile), explicit `random_state` on every stochastic step (train/test split, model init, shuffling, samplers), a recorded data snapshot (immutable parquet under a dated path, not a mutable SQL query), and an entry-point script that runs end-to-end without manual cells. Test it: delete your local `.venv`, re-clone, run the script, diff the outputs. If they differ, reproducibility is broken. The tradeoff: strict byte-reproducibility is hard on GPU — for deep-learning projects, accept statistical reproducibility (metric within a tolerance) and document the exact hardware/CUDA version.
+- **Runtime budget**: name a wall-clock ceiling for the full pipeline on the hardware you actually have. A useful default for small-team work: "end-to-end run (data pull -> train -> evaluate -> scoring output) must complete in <= 1 hour on a 16GB MacBook Pro." If you blow past it, either simplify or move to a bigger box deliberately — do not let runtime creep silently.
+- **Storage budget**: cap the on-disk footprint of raw data, features, and model artifacts. For laptop-scale work, `< 20 GB` total is a reasonable starting point; over that, you need a deliberate story (external object store, partitioned pulls, sampling). Record the budget in the README and check it in CI with a simple `du -sh` assertion.
+
+Encode these as top-of-project invariants, not aspirations. If the model hits the success metric but the pipeline is unreproducible or blows the runtime budget, the project is not done.
+
+Taken together, these five sections — problem framing, success metric, evaluation test, stakeholder contract, and nonfunctional budget — form the acceptance spec for the project. Write them up front, commit them alongside the code, and treat any drift as a scope change that requires re-agreeing with the stakeholder.