@zigrivers/scaffold 3.22.0 → 3.23.0

package/content/knowledge/data-science/data-science-model-evaluation.md

@@ -0,0 +1,160 @@
+---
+name: data-science-model-evaluation
+description: Honest model evaluation for solo/small-team DS — metric choice, one-shot holdout, cross-validation, calibration, and error slicing with sklearn
+topics: [data-science, evaluation, sklearn, cross-validation, calibration]
+---
+
+Every solo DS project produces a moment where a notebook prints `0.92 accuracy` on a test set and the author quietly believes the model works. Then it ships — and recall on the minority class is 0.12, the probabilities are miscalibrated, and a single region drives half the error. Evaluation discipline is the only thing separating a model that works from a model that looked like it worked on a single split. At solo scale you do not have an ML platform team checking your work, which makes the discipline entirely your responsibility.
+
+## Summary
+
+Match the metric to the business question: do not report accuracy on an imbalanced label, do not report RMSE when one outlier dominates the loss. Split the data once, use cross-validation on the training portion for model selection, and touch the holdout exactly once at the end. If downstream decisions consume probabilities (thresholding, expected value, stacking), check calibration — a 0.9 ROC-AUC model can still output probabilities that are wildly overconfident. Always slice errors by meaningful subgroups (region, bucket, cohort); aggregate metrics hide the failures that matter.
+
+## Deep Guidance
+
+### Picking the right metric
+
+Metric choice is a business decision, not a math decision. The right starting question is: "what does a false positive cost, and what does a false negative cost?" If the two are comparable and the classes are balanced, accuracy is fine. Once the costs diverge — or the base rate is skewed — accuracy becomes actively misleading.
+
+A small rubric for classification:
+
+- **Balanced binary classification**: accuracy is fine.
+- **Imbalanced binary (fraud, churn, rare disease)**: precision / recall / F1, and PR-AUC over ROC-AUC (ROC-AUC flatters models on heavy class imbalance).
+- **Ranking / thresholding later**: `roc_auc_score` measures order, not calibration.
+- **Decisions that consume probabilities**: `log_loss` or Brier score — rewards calibrated confidence, punishes overconfident mistakes.
+- **Multi-class**: `classification_report` for per-class precision/recall, and pick `average="macro"` (equal weight per class) vs `"weighted"` (weight by support) deliberately.
+
+And for regression:
+
+- **Magnitude matters**: RMSE (penalizes large errors quadratically).
+- **Outliers you do not want to chase**: MAE (robust to a few extreme points).
+- **Explained variance / reporting to stakeholders**: R².
+- **Relative error across scales**: MAPE, but guard against zeros in the denominator.
+
+```python
+from sklearn.metrics import classification_report, roc_auc_score, log_loss
+
+y_proba = model.predict_proba(X_test)[:, 1]
+y_pred = (y_proba >= 0.5).astype(int)
+
+print(classification_report(y_test, y_pred, digits=3))
+print(f"ROC-AUC:  {roc_auc_score(y_test, y_proba):.3f}")
+print(f"log-loss: {log_loss(y_test, y_proba):.3f}")
+```
+
+Report at least one threshold-free metric (ROC-AUC or PR-AUC) and one threshold-dependent metric (precision/recall at your operating point). Reporting only accuracy on a 95/5 class split is the canonical way to lie to yourself — the "always predict no" baseline gets 0.95 without a model.
+
+### Holdout discipline
+
+Split once, at the top of the notebook, before any exploration on the target:
+
+```python
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y,
+    test_size=0.2,
+    random_state=42,
+    stratify=y,  # preserve class balance for classification
+)
+```
+
+Rules:
+
+1. The test set is touched exactly once — at the end, for the final number you report.
+2. All model selection, feature engineering decisions, and hyperparameter tuning happen on `X_train` (via cross-validation).
+3. If you peek at test performance and then change the model, the test set is contaminated. Either live with the contamination and note it, or collect a new holdout.
+4. Fit preprocessing (`StandardScaler`, `OneHotEncoder`, imputers) on train only, then apply to test — wrap it in a `Pipeline` so you cannot leak by accident.
+
+### Cross-validation for model selection
+
+Use cross-validation on the training set to compare models and pick hyperparameters. This gives you a mean and standard deviation, so you can see whether model A actually beats model B or is one lucky fold away.
+
+```python
+from sklearn.model_selection import StratifiedKFold, cross_val_score, GridSearchCV
+from sklearn.ensemble import RandomForestClassifier
+
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
+
+scores = cross_val_score(
+    RandomForestClassifier(n_estimators=200, random_state=42),
+    X_train, y_train,
+    cv=cv,
+    scoring="roc_auc",
+)
+print(f"CV ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")
+
+grid = GridSearchCV(
+    RandomForestClassifier(random_state=42),
+    param_grid={"max_depth": [4, 8, None], "min_samples_leaf": [1, 5, 20]},
+    cv=cv,
+    scoring="roc_auc",
+    n_jobs=-1,
+)
+grid.fit(X_train, y_train)
+```
+
+Use `StratifiedKFold` for classification (preserves class balance per fold) and plain `KFold` for regression. **For any data with a time dimension, use `TimeSeriesSplit` instead** — random folds leak future information into the training set and will make your model look dramatically better offline than it is in production.
+
+### Calibration
+
+A model with a great ROC-AUC can still output badly calibrated probabilities — random forests and boosted trees are both notorious for this. If downstream code takes `predict_proba` output and uses it as a probability (expected-value calculations, threshold tuning based on cost, stacking, active learning), calibration matters at least as much as discrimination.
+
+```python
+from sklearn.calibration import calibration_curve, CalibratedClassifierCV
+import matplotlib.pyplot as plt
+
+prob_true, prob_pred = calibration_curve(y_test, y_proba, n_bins=10, strategy="quantile")
+
+plt.plot(prob_pred, prob_true, marker="o")
+plt.plot([0, 1], [0, 1], "--", color="gray")
+plt.xlabel("Predicted probability"); plt.ylabel("Observed frequency")
+```
+
+A well-calibrated model tracks the diagonal. If the curve sags below it you are overconfident; if it bulges above, underconfident. Fix with `CalibratedClassifierCV(method="isotonic")` (flexible, needs more data) or `method="sigmoid"` (Platt scaling, works with ~1k examples). Fit calibration on a held-out slice of the training set — never on the test set.
+
+### Error analysis and slicing
+
+Overall metrics hide systematic failures. A pandas `groupby` on the predictions is usually enough:
+
+```python
+import pandas as pd
+
+eval_df = pd.DataFrame({
+    "y_true": y_test,
+    "y_pred": y_pred,
+    "y_proba": y_proba,
+    "region": X_test["region"].values,
+    "age_bucket": pd.cut(X_test["age"], bins=[0, 25, 45, 65, 120]),
+})
+eval_df["correct"] = eval_df["y_true"] == eval_df["y_pred"]
+
+print(eval_df.groupby("region")["correct"].agg(["mean", "count"]))
+print(eval_df.groupby("age_bucket")["correct"].agg(["mean", "count"]))
+```
+
+Look for slices where the metric is materially worse than overall AND the slice has enough examples to be real (set a floor like n ≥ 50). Those are your debugging targets before shipping.
+
+**Fairness note**: slicing by sensitive attributes (age, gender, region, race where legally permitted) surfaces disparate impact. This is a minimum floor — if you ship models that affect people, read a proper fairness reference (Barocas/Hardt/Narayanan "Fairness and Machine Learning") rather than treating a groupby as the whole story.
+
+### What NOT to do
+
+- **Do not tune on the test set.** Every time you look at a test number and change the model, you are fitting to the test set in slow motion. The `GridSearchCV` call above uses CV on the train set specifically to avoid this.
+- **Do not cherry-pick a random seed.** If the model only wins with `random_state=7`, it does not actually win. Run with 3–5 different seeds and report the spread if you suspect the result is seed-fragile.
+- **Do not report only the best fold.** Report mean and std across CV folds. A model with 0.85 ± 0.12 is not better than 0.82 ± 0.02 — the first one is one unlucky fold away from losing.
+- **Do not ship without a trivial baseline.** Compare against predicting the majority class (classification) or the training mean (regression). If your fancy model cannot beat that, the problem is the data or the label, not the model.
+- **Do not evaluate on preprocessed-then-split data.** Fit the scaler, encoder, or imputer on train only, then transform test. Anything else is leakage and will inflate your offline numbers.
+- **Do not change the metric after seeing the results.** Pick the metric before training, based on the business question, and stick with it. Swapping from precision to ROC-AUC because one looked nicer is a cousin of p-hacking.
+
+## Minimum evaluation checklist
+
+Before calling a model "done" at solo scale, every item below should be true:
+
+1. Metric is chosen to match the business cost of errors, documented in the notebook or readme.
+2. Data was split once with a fixed `random_state`, stratified for classification or temporally for time-series.
+3. All preprocessing lives inside a `Pipeline` and is fit on train only.
+4. Model selection was done with cross-validation on the training set, with mean ± std reported per candidate.
+5. At least one trivial baseline was beaten by a margin larger than the CV standard deviation.
+6. Test set was evaluated exactly once, at the end, and that number is what you report.
+7. If `predict_proba` is consumed downstream, a calibration curve was inspected and recalibrated if needed.
+8. Errors were sliced by at least one meaningful business dimension, and any slice with materially worse metrics is either fixed or explicitly noted as a known limitation.

package/content/knowledge/data-science/data-science-notebook-discipline.md

@@ -0,0 +1,170 @@
+---
+name: data-science-notebook-discipline
+description: Notebook discipline for reproducible data science — Marimo as primary, Jupyter plus jupytext as fallback, promoting working cells to tested modules
+topics: [data-science, notebooks, marimo, jupyter, reproducibility]
+---
+
+Every data scientist has shipped a notebook that "worked for me in a session" and then produced different numbers the next morning — or worse, different numbers in a colleague's environment or a production run. The usual cause is not a bug in the code; it is hidden state. Jupyter cells can be executed in any order, re-run selectively, or silently depend on variables that were defined in a cell that has since been edited or deleted. The kernel's in-memory state becomes the real program, and the `.ipynb` file is just a partial, sometimes misleading, transcript. For solo and small-team DS work, this is the single biggest source of "it worked yesterday" pain, and it is entirely avoidable with the right tooling and habits.
+
+## Summary
+
+Use **Marimo** as your primary notebook tool: the file format is pure `.py` (git-diffable), execution is reactive (editing a cell re-runs its downstream dependents automatically), and there is no hidden-cell-order hazard by construction. When you cannot switch — existing Jupyter investment, team inertia, library widgets that only work in classic Jupyter — pair every `.ipynb` with a `.py` via **jupytext** and commit the `.py`. Either way, the key discipline is promotion: when a cell works, extract it to `src/<module>.py`, write a test, and import it back. Run finished notebooks as pipelines with `marimo run` or `papermill`.
+
+## Deep Guidance
+
+### The hidden-state problem
+
+Classic Jupyter lets you execute cells in any order. Consider this sequence:
+
+1. Cell A defines `df = pd.read_csv("raw.csv")`.
+2. Cell B defines `df = df.dropna()`.
+3. You run A, then B, then A again.
+4. `df` is now the raw frame — but cell B's output cell still shows the cleaned version, and any downstream cell that already ran still has the cleaned `df` cached in its own computation.
+
+Nothing about the notebook on disk reveals this inconsistency. "Restart kernel and run all" is the only way to prove a notebook is reproducible, and most DS workflows skip that step for months at a time. Outputs are cached in the `.ipynb`, so a reader sees plausible numbers and has no signal that the state is corrupt. This is **hidden state** — the kernel's memory diverges from the code as written, and the notebook lies about what it computed.
+
+Second-order effects make it worse: merge conflicts on `.ipynb` JSON are unreadable; diffs show base64 image blobs; collaborators re-run cells in different orders and get different results. The notebook as a unit of collaboration is broken unless you impose discipline from outside.
+
+### Marimo as primary
+
+[Marimo](https://marimo.io) is a reactive Python notebook that solves hidden state at the architecture level. Each notebook is a pure `.py` file; cells form a dependency graph; when you edit a cell, Marimo re-runs all of its dependents automatically. There is no way for the displayed state to diverge from what the code computes, because the runtime enforces topological order on every edit.
+
+A minimal Marimo notebook looks like this — note it is ordinary Python you can read in any editor:
+
+```python
+# notebook.py
+import marimo as mo
+
+app = mo.App()
+
+@app.cell
+def __():
+    import pandas as pd
+    df = pd.read_csv("data/raw.csv")
+    return (df,)
+
+@app.cell
+def __(df):
+    clean = df.dropna()
+    mo.md(f"Rows: **{len(clean)}**")
+    return (clean,)
+```
+
+Key commands:
+
+- `marimo edit notebook.py` — opens the reactive editor in your browser
+- `marimo run notebook.py` — serves the notebook as a read-only web app (great for stakeholders)
+- `marimo export html notebook.py -o out.html` — static HTML snapshot for reports
+
+Because the file is `.py`, `git diff` shows real code changes. Code review on a Marimo notebook works the same as code review on any Python file. There are no output cells to strip, no JSON diffs to parse.
+
+### Jupyter plus jupytext fallback
+
+When Marimo is not an option — you depend on a Jupyter-only widget, you share notebooks with non-Marimo users, or your infrastructure is built around `.ipynb` — use **jupytext** to pair each `.ipynb` with a `.py` representation. Install jupytext, then configure pairing at the repo root:
+
+```toml
+# .jupytext.toml
+formats = "ipynb,py:percent"
+notebook_metadata_filter = "-all"
+cell_metadata_filter = "-all"
+```
+
+Or pair a single notebook explicitly:
+
+```bash
+jupytext --set-formats ipynb,py:percent notebooks/eda.ipynb
+```
+
+The `py:percent` format splits cells with `# %%` markers and produces a clean, diffable Python file. Rule of thumb for the repo:
+
+- **Commit** the `.py` version — it is the source of truth for review and diffs
+- **Gitignore** the `.ipynb` (or commit it with `nbstripout` installed to strip outputs; see the data-science-security doc for the outputs-as-secrets angle)
+- **Do not** try to keep both hand-edited — jupytext's pre-save hook keeps them in sync automatically
+
+This does not fix hidden state (Jupyter still runs cells in click-order), but it does make review and merges sane, and it gives you a textual artifact that survives kernel-state bugs.
+
+### Promotion: notebook to src to test to re-import
+
+The most important habit in any notebook workflow — Marimo or Jupyter — is **promotion**. The moment a cell does real work, extract it to a tested module and import it back.
+
+Before (inline in the notebook, untested, untyped):
+
+```python
+@app.cell
+def __(df):
+    df["hour"] = pd.to_datetime(df["ts"]).dt.hour
+    df["is_weekend"] = pd.to_datetime(df["ts"]).dt.dayofweek >= 5
+    df["log_amount"] = np.log1p(df["amount"])
+    return (df,)
+```
+
+After — extract to `src/features/engineer.py`:
+
+```python
+# src/features/engineer.py
+import numpy as np
+import pandas as pd
+
+def add_time_features(df: pd.DataFrame, ts_col: str = "ts") -> pd.DataFrame:
+    """Add hour and is_weekend columns derived from a timestamp column."""
+    out = df.copy()
+    ts = pd.to_datetime(out[ts_col])
+    out["hour"] = ts.dt.hour
+    out["is_weekend"] = ts.dt.dayofweek >= 5
+    return out
+
+def add_log_amount(df: pd.DataFrame, amount_col: str = "amount") -> pd.DataFrame:
+    out = df.copy()
+    out["log_amount"] = np.log1p(out[amount_col])
+    return out
+```
+
+Write a test — small, fast, no data dependency:
+
+```python
+# tests/features/test_engineer.py
+import pandas as pd
+from src.features.engineer import add_time_features
+
+def test_weekend_flag_friday_vs_saturday():
+    df = pd.DataFrame({"ts": ["2026-04-17 10:00", "2026-04-18 10:00"]})
+    out = add_time_features(df)
+    assert out["is_weekend"].tolist() == [False, True]
+```
+
+Re-import in the notebook:
+
+```python
+@app.cell
+def __(df):
+    from src.features.engineer import add_time_features, add_log_amount
+    df = add_log_amount(add_time_features(df))
+    return (df,)
+```
+
+The notebook becomes a thin orchestration + visualization layer over tested modules. Hidden state matters less because the logic lives in files that are exercised by CI. Pull requests become reviewable — the reviewer reads typed functions with tests, not a wall of chained DataFrame mutations.
+
+### Running notebooks as pipelines
+
+Finished notebooks often need to run on a schedule — daily reports, weekly retraining, monthly audits. Do not copy-paste the code into a script; run the notebook directly.
+
+**Marimo**: because the file is already Python, you can run it as a script or as an app:
+
+```bash
+marimo run notebook.py                    # serve as web app
+python notebook.py                        # execute top-to-bottom as a plain script
+marimo export html notebook.py -o out.html # produce a static report artifact
+```
+
+**Jupyter**: use `papermill` to parameterize and execute an `.ipynb`, producing an executed output notebook:
+
+```bash
+papermill notebooks/weekly_report.ipynb \
+          outputs/report_$(date +%Y%m%d).ipynb \
+          -p start_date 2026-04-14 \
+          -p end_date 2026-04-20
+```
+
+Parameterized cells (tagged `parameters` in Jupyter) are injected by papermill at the top of the run. Use Marimo's `mo.cli_args()` for the equivalent in Marimo. Either way, pair this with a lightweight scheduler (cron, GitHub Actions, Airflow, Prefect) — the notebook is the unit of work, not a script that tries to re-implement it.
+
+A useful rule: if a notebook is scheduled to run unattended, its logic should be ~90% imports from `src/` and ~10% glue. The promotion discipline from the previous section is what makes scheduled notebook runs trustworthy.

package/content/knowledge/data-science/data-science-observability.md

@@ -0,0 +1,161 @@
+---
+name: data-science-observability
+description: Monitoring deployed DS models and pipelines — prediction logging to Parquet, scheduled evaluation, basic drift detection, and Evidently for deeper analysis
+topics: [data-science, observability, monitoring, drift, evidently]
+---
+
+Models don't fail loudly. A scoring job keeps running, rows keep landing in the output table, dashboards stay green — and quietly, the predictions get worse. The world drifts away from whatever snapshot you trained on, and nobody notices until a stakeholder says "these numbers look weird." Observability for a solo DS isn't a platform; it's a small set of habits that give you a chance to catch decay before someone else does.
+
+## Summary
+
+For a solo or small-team data scientist with something deployed (even just a weekly cron), observability boils down to four habits: log every prediction with its inputs to a dated Parquet file, re-run your evaluation script on a schedule and alert on metric drops, check a handful of key features for distributional drift, and reach for `Evidently` when you want a pre-built drift report instead of writing your own. The goal is a tripwire, not a dashboard — you want to get paged when something's wrong, not stare at graphs hoping to spot it.
+
+## Deep Guidance
+
+### Log predictions + inputs
+
+Every time your model scores something, append a row to a dated Parquet log. This is the single most useful thing you can do for future-you — drift analysis, debugging, label backfill, and post-mortems all depend on having this log.
+
+Layout:
+
+```
+data/processed/predictions/
+  2026-04-21/
+    run-20260421T0300-abc123.parquet
+  2026-04-22/
+    run-20260422T0300-def456.parquet
+```
+
+Schema (one row per prediction):
+
+```python
+# src/monitor/prediction_log.py
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+import pandas as pd
+
+def log_predictions(
+    features: pd.DataFrame,
+    predictions: pd.Series,
+    model_version: str,
+    log_root: Path = Path("data/processed/predictions"),
+) -> Path:
+    today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
+    run_id = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M") + "-" + uuid.uuid4().hex[:6]
+    out_dir = log_root / today
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    df = features.copy()
+    df["prediction"] = predictions.values
+    df["model_version"] = model_version
+    df["logged_at"] = datetime.now(timezone.utc)
+    df["run_id"] = run_id
+    df["ground_truth"] = pd.NA  # Backfilled later when labels arrive
+
+    out = out_dir / f"run-{run_id}.parquet"
+    df.to_parquet(out, index=False)
+    return out
+```
+
+Parquet is right for this: columnar, compressed, fast to scan across dates with `pd.read_parquet("data/processed/predictions/**/*.parquet")`. If inputs have PII, hash or drop those columns before logging — you rarely need raw identifiers to do drift or error analysis.
+
+### Scheduled eval re-runs
+
+Your training-time evaluation script is also your monitoring script. Run it weekly or monthly against recent predictions joined to whatever ground truth has arrived, and alert when the headline metric breaches a threshold.
+
+```python
+# src/monitor/eval.py
+import sys
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+THRESHOLD = 0.80  # Alert if AUC drops below this
+
+def main() -> int:
+    preds = pd.read_parquet("data/processed/predictions/")
+    labels = pd.read_parquet("data/processed/labels/")
+    joined = preds.merge(labels, on="record_id", how="inner")
+    if len(joined) < 500:
+        print("Not enough labeled data yet; skipping.")
+        return 0
+
+    auc = roc_auc_score(joined["actual"], joined["prediction"])
+    print(f"AUC on {len(joined)} labeled rows: {auc:.3f}")
+    if auc < THRESHOLD:
+        # Send email / Slack webhook here
+        print(f"ALERT: AUC {auc:.3f} below threshold {THRESHOLD}", file=sys.stderr)
+        return 1
+    return 0
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+```
+
+Schedule it with whatever you already have — cron, a GitHub Actions `schedule:` workflow, Airflow if you run it, or your platform's scheduled job. Exit code 1 plus a Slack webhook is a perfectly good alerting system at this scale.
+
+### Basic drift detection
+
+Before reaching for a library, do the cheap thing: compare this period's feature distribution to your training distribution. Mean, std, a couple of quantiles, and a KS statistic cover most of what you need.
+
+```python
+# src/monitor/drift.py
+from scipy.stats import ks_2samp
+import pandas as pd
+
+def feature_drift(reference: pd.Series, current: pd.Series) -> dict:
+    stat, p = ks_2samp(reference.dropna(), current.dropna())
+    return {
+        "ref_mean": reference.mean(), "cur_mean": current.mean(),
+        "ref_std": reference.std(),   "cur_std": current.std(),
+        "ks_stat": stat, "ks_p": p,
+        "drifted": p < 0.01,
+    }
+
+train = pd.read_parquet("data/processed/train.parquet")
+recent = pd.read_parquet("data/processed/predictions/2026-04-21/")
+for col in ["amount", "user_tenure_days", "n_items"]:
+    print(col, feature_drift(train[col], recent[col]))
+```
+
+Run this alongside your scheduled eval. You don't need a dashboard — printing to the job log and alerting on `drifted=True` on any monitored feature is enough.
+
+### Evidently for more
+
+When you outgrow ad-hoc KS tests, `Evidently` gives you a pre-built drift report across all features, plus data quality checks and target drift, as an HTML page you can open or ship to S3.
+
+```python
+# src/monitor/evidently_report.py
+import pandas as pd
+from evidently import Report
+from evidently.presets import DataDriftPreset
+
+reference = pd.read_parquet("data/processed/train.parquet")
+current = pd.read_parquet("data/processed/predictions/2026-04-21/")
+
+report = Report([DataDriftPreset()])
+snapshot = report.run(reference_data=reference, current_data=current)
+snapshot.save_html("reports/drift-2026-04-21.html")
+```
+
+This is opt-in. If plain pandas + SciPy is telling you what you need to know, don't add a dependency. Reach for Evidently when you have enough features that per-column code is tedious, or when you want a shareable artifact for a stakeholder.
+
+### The prediction / feedback loop
+
+Ground truth almost never arrives at prediction time. A churn model predicts today who'll cancel next month; a fraud model predicts now whether a transaction is bad, confirmed days later. That delay is why the Parquet log exists — you keep predictions around until labels catch up, then join.
+
+```python
+# src/monitor/backfill_labels.py
+import pandas as pd
+
+preds = pd.read_parquet("data/processed/predictions/")
+labels = pd.read_parquet("data/processed/labels/")  # record_id, actual, label_time
+merged = preds.merge(labels, on="record_id", how="left")
+merged.to_parquet("data/processed/predictions_labeled.parquet", index=False)
+```
+
+Keep at least one full feedback cycle of prediction logs (if labels arrive after 30 days, keep 60-90 days). This join is how you get a real accuracy number on production traffic, not just your static test-set number from training day.
+
+### What NOT to build
+
+Resist the urge to over-engineer. At solo scale you do not need streaming drift detection, a Prometheus/Grafana stack, a model registry with canary deploys, or a dedicated monitoring dashboard. Those are ML-platform-team concerns — build them when there's a team to own them. A dated Parquet log, a scheduled eval script, a handful of drift checks, and an alert that emails you is more than enough to catch the failures that actually happen.