@zigrivers/scaffold 3.22.0 → 3.24.0

package/content/knowledge/data-science/data-science-experiment-tracking.md

@@ -0,0 +1,194 @@
+---
+name: data-science-experiment-tracking
+description: Local MLflow setup, run instrumentation, git commit tagging, and run comparison for solo and small-team data science work
+topics: [data-science, experiment-tracking, mlflow, weights-and-biases, reproducibility]
+---
+
+Without experiment tracking, data science becomes archaeology: three weeks after a promising result, a stakeholder asks "which config produced that number?" and answering it turns into a forensic exercise — sifting through notebook history, Slack messages, and commented-out cells. A lightweight experiment tracker fixes this with one discipline: every run logs its hyperparameters, metrics, artifacts, and the git commit SHA that produced it. For a solo DS or small team, you do not need a shared server or a cloud account — a local MLflow instance on SQLite is enough to get the full benefit, and you can graduate to a shared deployment later without changing the instrumentation.
+
+## Summary
+
+Self-host MLflow locally with a SQLite backend and a local artifact directory — it is the minimum setup that still gives you a queryable run history, a browsable UI, and reproducible run IDs. Every run logs the full hyperparameter dict, metrics per epoch (or iteration), the git commit SHA as a tag, dataset version, and any config or report artifacts. Weights & Biases is a reasonable cloud alternative if you value the polished UI and do not mind cloud storage — but for a DS-1 setup it is not the primary recommendation. Never log PII into run metadata or artifacts, and never commit `mlflow.db` or `mlartifacts/` to git.
+
+## Deep Guidance
+
+### What to log per run
+
+Treat every training run, hyperparameter tweak, or evaluation pass as a tracked experiment — even the exploratory ones you think will be throwaway. The cost of logging is trivial; the cost of not logging a run that turns out to matter is measured in hours of re-running and second-guessing. The minimum payload is:
+
+- **Hyperparameters**: the full config dict (learning rate, batch size, seed, feature set, model type, loss weights, regularization). Log it all — future-you does not know which knob will matter and adding knobs retroactively is impossible.
+- **Metrics**: logged with `step=epoch` (or `step=iteration`) so the UI can render a time-series plot. Log train and validation metrics side by side; a single final-value log loses the overfitting story.
+- **Git commit SHA**: a tag pointing to the exact commit that produced the run. Without this, "reproduce run 47" is unanswerable, because the config alone does not capture code changes in the training loop, data loader, or feature engineering.
+- **Dataset version**: a tag or param identifying which dataset snapshot was used — a DVC hash, a filename with a date suffix, or a data commit SHA. Without this, "reproduce run 47" is still unanswerable even if you have the code, because the data moved underneath it.
+- **Run name**: a human-readable name (`baseline-v3-with-dropout`) so the UI list is browsable without clicking every row to read the params.
+- **Artifacts**: the resolved config YAML, the evaluation report JSON, any confusion matrix images, and the final model checkpoint. Small artifacts go inline with the run; large model weights can be stored by reference.
+
+### MLflow self-hosted setup
+
+Run the tracking server locally. SQLite is the right backend for a solo workflow — it gives you the full query API without the ops burden of Postgres, and the `mlflow.db` file is small enough that you can zip and share it with a collaborator if you really need to:
+
+```bash
+mlflow server \
+  --backend-store-uri sqlite:///mlflow.db \
+  --default-artifact-root ./mlartifacts \
+  --host 127.0.0.1 --port 5000
+```
+
+Bind to `127.0.0.1` rather than `0.0.0.0` so you do not accidentally expose an unauthenticated tracking server to your network. Leave it running in a terminal tab, a `tmux` pane, or under `launchd`/`systemd` — whatever keeps it up between sessions.
+
+Point your code at the server via an environment variable. Using `direnv` keeps this per-project and avoids polluting your shell:
+
+```bash
+# .envrc
+export MLFLOW_TRACKING_URI=http://localhost:5000
+export MLFLOW_EXPERIMENT_NAME=churn-baseline
+```
+
+Add the tracking artifacts to `.gitignore` — they are large, local, and not reproducible from source. Committing them bloats the repo and leaks local-path metadata into history:
+
+```gitignore
+# .gitignore
+mlflow.db
+mlflow.db-journal
+mlartifacts/
+mlruns/
+```
+
+When you later graduate to a shared MLflow server (team deployment, S3 artifact store, Postgres backend), the only change is the `MLFLOW_TRACKING_URI` — your instrumentation code stays identical, and historical runs stay on your laptop as a personal archive.
+
+### Instrumenting a training / experiment run
+
+Wrap the training loop in `mlflow.start_run`. The context manager handles start and end timestamps, guarantees the run closes even on exception, and exposes `run.info.run_id` — the stable handle you use later for comparison, export, or model loading:
+
+```python
+import subprocess
+import mlflow
+import yaml
+
+mlflow.set_tracking_uri("http://localhost:5000")
+mlflow.set_experiment("churn-baseline")
+
+def train(cfg: dict) -> dict:
+    with mlflow.start_run(run_name=cfg["experiment"]["name"]) as run:
+        # Log full hyperparameter dict (flatten nested keys to dot-paths)
+        mlflow.log_params(_flatten(cfg))
+
+        # Reproducibility tags — git commit is the single most important one
+        git_sha = subprocess.check_output(
+            ["git", "rev-parse", "HEAD"]
+        ).decode().strip()
+        mlflow.set_tag("git_commit", git_sha)
+        mlflow.set_tag("dataset_version", cfg["data"]["version"])
+        mlflow.set_tag("model_type", cfg["model"]["type"])
+
+        # Per-epoch metrics — step=epoch is what gives you a time-series plot
+        for epoch in range(cfg["training"]["epochs"]):
+            train_metrics = train_epoch(...)
+            val_metrics = evaluate(...)
+            mlflow.log_metrics({
+                "train_loss": train_metrics["loss"],
+                "val_loss": val_metrics["loss"],
+                "val_auc": val_metrics["auc"],
+            }, step=epoch)
+
+        # Artifacts: resolved config + eval report
+        with open("configs/resolved.yaml", "w") as f:
+            yaml.safe_dump(cfg, f)
+        mlflow.log_artifact("configs/resolved.yaml")
+        mlflow.log_artifact("reports/eval_report.json")
+
+        return {"run_id": run.info.run_id, **val_metrics}
+```
+
+A few notes on the shape of this code. `mlflow.log_params` takes a flat dict, so a helper like `_flatten` turns `{"optimizer": {"lr": 1e-3}}` into `{"optimizer.lr": "0.001"}` — values are coerced to strings. Log the **resolved** config after any CLI overrides or hydra composition, not the raw file on disk, so the stored params match what actually ran. If the working tree is dirty at training time, either commit first or log `git status --porcelain` output as a tag so you can tell the logged commit is not the whole story. Keep the returned `run_id` — it is the primary key you will use to find this run in the UI, export its metadata, register its model later, or reference it from a downstream evaluation run via `mlflow.set_tag("parent_run_id", ...)`.
+
+### Run comparison and selection
+
+Open the MLflow UI at `http://localhost:5000`. The three views that earn their keep:
+
+- **Run list** — sort by `metrics.val_auc` or filter by `tags.git_commit = "<sha>"`. Tag filters are the fastest way to find "the runs I launched from this branch." Sort by columns to see the run_id of your best-performing experiment, then click through for the full picture.
+- **Parallel coordinates plot** — select several runs, switch to the parallel coordinates view, and see which hyperparameters correlate with your target metric. This is the view that turns dozens of runs into a readable pattern — hover a line to see the full config, drag axes to filter a band, and the plot re-paints to show only the runs that meet your criterion.
+- **Metric plot** — overlay `val_loss` across selected runs to spot overfitting (train loss drops, val loss rises), bad seeds (wildly different trajectories with the same config), or early-stopping candidates (val metric plateaued ten epochs before training ended).
+
+You can also query programmatically when the UI's filters are not expressive enough:
+
+```python
+from mlflow.tracking import MlflowClient
+import pandas as pd
+
+client = MlflowClient()
+runs = client.search_runs(
+    experiment_ids=[client.get_experiment_by_name("churn-baseline").experiment_id],
+    filter_string="metrics.val_auc > 0.82 and tags.dataset_version = '2026-03'",
+    order_by=["metrics.val_auc DESC"],
+    max_results=20,
+)
+df = pd.DataFrame([{
+    "run_id": r.info.run_id,
+    "name": r.info.run_name,
+    "val_auc": r.data.metrics.get("val_auc"),
+    "lr": r.data.params.get("optimizer.lr"),
+} for r in runs])
+```
+
+When you have a winner, export its config back into the repo for a clean retrain:
+
+```python
+run = client.get_run("<run_id>")
+client.download_artifacts(run.info.run_id, "resolved.yaml", dst_path="configs/")
+```
+
+Commit the exported config so "run 47's exact recipe" becomes a file in git, not a memory and not a database row that lives only on your laptop.
+
+### Weights & Biases as alternative
+
+Weights & Biases is the polished cloud alternative. It has a richer UI, built-in system metric logging (GPU, memory, temperature), gradient histograms via `wandb.watch(model, log="gradients")`, media logging (images, audio, tables, confusion matrices rendered inline), and better collaboration features — named reports, shared dashboards, thread-style comments. For a small team that has already decided it is comfortable with cloud storage, W&B removes the "who runs the server" question entirely, and the onboarding for a new teammate is `pip install wandb && wandb login`.
+
+The instrumentation shape is familiar:
+
+```python
+import wandb
+wandb.init(project="churn-baseline", name=cfg["experiment"]["name"],
+           config=cfg, tags=["baseline", "v2-features"])
+for epoch in range(cfg["training"]["epochs"]):
+    wandb.log({"epoch": epoch, "val/auc": val_auc, "train/loss": loss})
+wandb.finish()
+```
+
+Two things to weigh before picking it over MLflow for a DS-1 setup. First, the free tier has trial limits on private projects, artifact retention, and seats — fine for a solo experimenter, worth pricing out before a team commits. Second, you are shipping your experiment metadata (and potentially artifacts) to a third party, which matters if your dataset values, config parameters, or run names might accidentally encode something sensitive. MLflow self-hosted stays on your laptop; W&B lives in someone else's cloud. If your org has a data-residency or vendor-review process, MLflow skips it entirely.
+
+### Graduating from solo to shared
+
+The path from "SQLite on my laptop" to "shared team tracker" is short and deliberately low-risk, because your instrumentation already speaks the MLflow protocol:
+
+1. **Stand up a shared MLflow server** behind your internal network — Postgres backend, S3 or equivalent object store for artifacts, authentication in front (oauth2-proxy, nginx basic auth, or a cloud load balancer).
+2. **Flip the tracking URI** in each project's `.envrc` to point at the shared server. No code changes.
+3. **Optionally backfill historic runs** using `mlflow artifacts download` plus `search_runs`, then re-log to the shared server — only worth it for runs you want the team to see.
+4. **Keep the local server configured** for offline or air-gapped work — you can still set `MLFLOW_TRACKING_URI=file:./mlruns` for quick local-only iteration when the shared server is down or you are on a plane.
+
+This is why the self-hosted local setup is the right default even if you know you will eventually run a team server: the instrumentation you write today is exactly the instrumentation that will talk to production tomorrow.
+
+### Nested runs for sweeps and evaluations
+
+When you run a small hyperparameter sweep from a laptop — a few learning rates, two or three seeds — use MLflow's nested runs rather than one flat run per trial. A parent run captures the sweep-level config and best-metric summary; each trial becomes a child with its own params and metrics:
+
+```python
+with mlflow.start_run(run_name="lr-sweep") as parent:
+    mlflow.log_param("sweep_type", "grid")
+    best_auc = 0.0
+    for lr in [1e-4, 3e-4, 1e-3, 3e-3]:
+        with mlflow.start_run(run_name=f"lr={lr}", nested=True) as child:
+            mlflow.log_param("lr", lr)
+            val_auc = train_one(lr)
+            mlflow.log_metric("val_auc", val_auc)
+            best_auc = max(best_auc, val_auc)
+    mlflow.log_metric("best_val_auc", best_auc)
+```
+
+In the UI, the parent shows an expandable tree of children, which keeps the run list navigable once you have hundreds of rows. For larger sweeps, pair MLflow with Optuna (`mlflow.start_run(nested=True)` inside an `objective` function) — you get Bayesian search on top of MLflow's persistence.
+
+### Hygiene
+
+**Never log PII.** Experiment metadata and artifacts are easy to share with collaborators, screenshot into a ticket, or accidentally export to a cloud UI when you later migrate to W&B or a shared MLflow. Hyperparameter values, metric names, run names, tags, and artifact contents must all be free of customer identifiers, emails, names, or raw records. If a config carries a data path, keep it at the dataset level (`data/processed/churn_2026_03.parquet`) — never at the row or user level (`data/users/alice@example.com/history.parquet`). Evaluation reports that include example predictions must redact any PII in the input columns before being logged as artifacts. See `data-science-security.md` for the broader no-PII rules that apply across the whole DS workflow.
+
+**Never commit the tracking store.** `mlflow.db`, `mlflow.db-journal`, `mlartifacts/`, and `mlruns/` all belong in `.gitignore`. They are large (easily hundreds of MB once you log a few model checkpoints), machine-local (paths and timestamps are specific to your laptop), and not reproducible from git. If you need a teammate to see a run, export the specific artifacts you want to share via `mlflow artifacts download --run-id <id>` or by running a shared tracking server — do not push the whole store into the repo and do not email `mlflow.db` as an attachment.

package/content/knowledge/data-science/data-science-model-evaluation.md

@@ -0,0 +1,160 @@
+---
+name: data-science-model-evaluation
+description: Honest model evaluation for solo/small-team DS — metric choice, one-shot holdout, cross-validation, calibration, and error slicing with sklearn
+topics: [data-science, evaluation, sklearn, cross-validation, calibration]
+---
+
+Every solo DS project produces a moment where a notebook prints `0.92 accuracy` on a test set and the author quietly believes the model works. Then it ships — and recall on the minority class is 0.12, the probabilities are miscalibrated, and a single region drives half the error. Evaluation discipline is the only thing separating a model that works from a model that looked like it worked on a single split. At solo scale you do not have an ML platform team checking your work, which makes the discipline entirely your responsibility.
+
+## Summary
+
+Match the metric to the business question: do not report accuracy on an imbalanced label, do not report RMSE when one outlier dominates the loss. Split the data once, use cross-validation on the training portion for model selection, and touch the holdout exactly once at the end. If downstream decisions consume probabilities (thresholding, expected value, stacking), check calibration — a 0.9 ROC-AUC model can still output probabilities that are wildly overconfident. Always slice errors by meaningful subgroups (region, bucket, cohort); aggregate metrics hide the failures that matter.
+
+## Deep Guidance
+
+### Picking the right metric
+
+Metric choice is a business decision, not a math decision. The right starting question is: "what does a false positive cost, and what does a false negative cost?" If the two are comparable and the classes are balanced, accuracy is fine. Once the costs diverge — or the base rate is skewed — accuracy becomes actively misleading.
+
+A small rubric for classification:
+
+- **Balanced binary classification**: accuracy is fine.
+- **Imbalanced binary (fraud, churn, rare disease)**: precision / recall / F1, and PR-AUC over ROC-AUC (ROC-AUC flatters models on heavy class imbalance).
+- **Ranking / thresholding later**: `roc_auc_score` measures order, not calibration.
+- **Decisions that consume probabilities**: `log_loss` or Brier score — rewards calibrated confidence, punishes overconfident mistakes.
+- **Multi-class**: `classification_report` for per-class precision/recall, and pick `average="macro"` (equal weight per class) vs `"weighted"` (weight by support) deliberately.
+
+And for regression:
+
+- **Magnitude matters**: RMSE (penalizes large errors quadratically).
+- **Outliers you do not want to chase**: MAE (robust to a few extreme points).
+- **Explained variance / reporting to stakeholders**: R².
+- **Relative error across scales**: MAPE, but guard against zeros in the denominator.
+
+```python
+from sklearn.metrics import classification_report, roc_auc_score, log_loss
+
+y_proba = model.predict_proba(X_test)[:, 1]
+y_pred = (y_proba >= 0.5).astype(int)
+
+print(classification_report(y_test, y_pred, digits=3))
+print(f"ROC-AUC:  {roc_auc_score(y_test, y_proba):.3f}")
+print(f"log-loss: {log_loss(y_test, y_proba):.3f}")
+```
+
+Report at least one threshold-free metric (ROC-AUC or PR-AUC) and one threshold-dependent metric (precision/recall at your operating point). Reporting only accuracy on a 95/5 class split is the canonical way to lie to yourself — the "always predict no" baseline gets 0.95 without a model.
+
+### Holdout discipline
+
+Split once, at the top of the notebook, before any exploration on the target:
+
+```python
+from sklearn.model_selection import train_test_split
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X, y,
+    test_size=0.2,
+    random_state=42,
+    stratify=y,  # preserve class balance for classification
+)
+```
+
+Rules:
+
+1. The test set is touched exactly once — at the end, for the final number you report.
+2. All model selection, feature engineering decisions, and hyperparameter tuning happen on `X_train` (via cross-validation).
+3. If you peek at test performance and then change the model, the test set is contaminated. Either live with the contamination and note it, or collect a new holdout.
+4. Fit preprocessing (`StandardScaler`, `OneHotEncoder`, imputers) on train only, then apply to test — wrap it in a `Pipeline` so you cannot leak by accident.
+
+### Cross-validation for model selection
+
+Use cross-validation on the training set to compare models and pick hyperparameters. This gives you a mean and standard deviation, so you can see whether model A actually beats model B or is one lucky fold away.
+
+```python
+from sklearn.model_selection import StratifiedKFold, cross_val_score, GridSearchCV
+from sklearn.ensemble import RandomForestClassifier
+
+cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
+
+scores = cross_val_score(
+    RandomForestClassifier(n_estimators=200, random_state=42),
+    X_train, y_train,
+    cv=cv,
+    scoring="roc_auc",
+)
+print(f"CV ROC-AUC: {scores.mean():.3f} ± {scores.std():.3f}")
+
+grid = GridSearchCV(
+    RandomForestClassifier(random_state=42),
+    param_grid={"max_depth": [4, 8, None], "min_samples_leaf": [1, 5, 20]},
+    cv=cv,
+    scoring="roc_auc",
+    n_jobs=-1,
+)
+grid.fit(X_train, y_train)
+```
+
+Use `StratifiedKFold` for classification (preserves class balance per fold) and plain `KFold` for regression. **For any data with a time dimension, use `TimeSeriesSplit` instead** — random folds leak future information into the training set and will make your model look dramatically better offline than it is in production.
+
+### Calibration
+
+A model with a great ROC-AUC can still output badly calibrated probabilities — random forests and boosted trees are both notorious for this. If downstream code takes `predict_proba` output and uses it as a probability (expected-value calculations, threshold tuning based on cost, stacking, active learning), calibration matters at least as much as discrimination.
+
+```python
+from sklearn.calibration import calibration_curve, CalibratedClassifierCV
+import matplotlib.pyplot as plt
+
+prob_true, prob_pred = calibration_curve(y_test, y_proba, n_bins=10, strategy="quantile")
+
+plt.plot(prob_pred, prob_true, marker="o")
+plt.plot([0, 1], [0, 1], "--", color="gray")
+plt.xlabel("Predicted probability"); plt.ylabel("Observed frequency")
+```
+
+A well-calibrated model tracks the diagonal. If the curve sags below it you are overconfident; if it bulges above, underconfident. Fix with `CalibratedClassifierCV(method="isotonic")` (flexible, needs more data) or `method="sigmoid"` (Platt scaling, works with ~1k examples). Fit calibration on a held-out slice of the training set — never on the test set.
+
+### Error analysis and slicing
+
+Overall metrics hide systematic failures. A pandas `groupby` on the predictions is usually enough:
+
+```python
+import pandas as pd
+
+eval_df = pd.DataFrame({
+    "y_true": y_test,
+    "y_pred": y_pred,
+    "y_proba": y_proba,
+    "region": X_test["region"].values,
+    "age_bucket": pd.cut(X_test["age"], bins=[0, 25, 45, 65, 120]),
+})
+eval_df["correct"] = eval_df["y_true"] == eval_df["y_pred"]
+
+print(eval_df.groupby("region")["correct"].agg(["mean", "count"]))
+print(eval_df.groupby("age_bucket")["correct"].agg(["mean", "count"]))
+```
+
+Look for slices where the metric is materially worse than overall AND the slice has enough examples to be real (set a floor like n ≥ 50). Those are your debugging targets before shipping.
+
+**Fairness note**: slicing by sensitive attributes (age, gender, region, race where legally permitted) surfaces disparate impact. This is a minimum floor — if you ship models that affect people, read a proper fairness reference (Barocas/Hardt/Narayanan "Fairness and Machine Learning") rather than treating a groupby as the whole story.
+
+### What NOT to do
+
+- **Do not tune on the test set.** Every time you look at a test number and change the model, you are fitting to the test set in slow motion. The `GridSearchCV` call above uses CV on the train set specifically to avoid this.
+- **Do not cherry-pick a random seed.** If the model only wins with `random_state=7`, it does not actually win. Run with 3–5 different seeds and report the spread if you suspect the result is seed-fragile.
+- **Do not report only the best fold.** Report mean and std across CV folds. A model with 0.85 ± 0.12 is not better than 0.82 ± 0.02 — the first one is one unlucky fold away from losing.
+- **Do not ship without a trivial baseline.** Compare against predicting the majority class (classification) or the training mean (regression). If your fancy model cannot beat that, the problem is the data or the label, not the model.
+- **Do not evaluate on preprocessed-then-split data.** Fit the scaler, encoder, or imputer on train only, then transform test. Anything else is leakage and will inflate your offline numbers.
+- **Do not change the metric after seeing the results.** Pick the metric before training, based on the business question, and stick with it. Swapping from precision to ROC-AUC because one looked nicer is a cousin of p-hacking.
+
+## Minimum evaluation checklist
+
+Before calling a model "done" at solo scale, every item below should be true:
+
+1. Metric is chosen to match the business cost of errors, documented in the notebook or readme.
+2. Data was split once with a fixed `random_state`, stratified for classification or temporally for time-series.
+3. All preprocessing lives inside a `Pipeline` and is fit on train only.
+4. Model selection was done with cross-validation on the training set, with mean ± std reported per candidate.
+5. At least one trivial baseline was beaten by a margin larger than the CV standard deviation.
+6. Test set was evaluated exactly once, at the end, and that number is what you report.
+7. If `predict_proba` is consumed downstream, a calibration curve was inspected and recalibrated if needed.
+8. Errors were sliced by at least one meaningful business dimension, and any slice with materially worse metrics is either fixed or explicitly noted as a known limitation.

package/content/knowledge/data-science/data-science-notebook-discipline.md

@@ -0,0 +1,170 @@
+---
+name: data-science-notebook-discipline
+description: Notebook discipline for reproducible data science — Marimo as primary, Jupyter plus jupytext as fallback, promoting working cells to tested modules
+topics: [data-science, notebooks, marimo, jupyter, reproducibility]
+---
+
+Every data scientist has shipped a notebook that "worked for me in a session" and then produced different numbers the next morning — or worse, different numbers in a colleague's environment or a production run. The usual cause is not a bug in the code; it is hidden state. Jupyter cells can be executed in any order, re-run selectively, or silently depend on variables that were defined in a cell that has since been edited or deleted. The kernel's in-memory state becomes the real program, and the `.ipynb` file is just a partial, sometimes misleading, transcript. For solo and small-team DS work, this is the single biggest source of "it worked yesterday" pain, and it is entirely avoidable with the right tooling and habits.
+
+## Summary
+
+Use **Marimo** as your primary notebook tool: the file format is pure `.py` (git-diffable), execution is reactive (editing a cell re-runs its downstream dependents automatically), and there is no hidden-cell-order hazard by construction. When you cannot switch — existing Jupyter investment, team inertia, library widgets that only work in classic Jupyter — pair every `.ipynb` with a `.py` via **jupytext** and commit the `.py`. Either way, the key discipline is promotion: when a cell works, extract it to `src/<module>.py`, write a test, and import it back. Run finished notebooks as pipelines with `marimo run` or `papermill`.
+
+## Deep Guidance
+
+### The hidden-state problem
+
+Classic Jupyter lets you execute cells in any order. Consider this sequence:
+
+1. Cell A defines `df = pd.read_csv("raw.csv")`.
+2. Cell B defines `df = df.dropna()`.
+3. You run A, then B, then A again.
+4. `df` is now the raw frame — but cell B's output cell still shows the cleaned version, and any downstream cell that already ran still has the cleaned `df` cached in its own computation.
+
+Nothing about the notebook on disk reveals this inconsistency. "Restart kernel and run all" is the only way to prove a notebook is reproducible, and most DS workflows skip that step for months at a time. Outputs are cached in the `.ipynb`, so a reader sees plausible numbers and has no signal that the state is corrupt. This is **hidden state** — the kernel's memory diverges from the code as written, and the notebook lies about what it computed.
+
+Second-order effects make it worse: merge conflicts on `.ipynb` JSON are unreadable; diffs show base64 image blobs; collaborators re-run cells in different orders and get different results. The notebook as a unit of collaboration is broken unless you impose discipline from outside.
+
+### Marimo as primary
+
+[Marimo](https://marimo.io) is a reactive Python notebook that solves hidden state at the architecture level. Each notebook is a pure `.py` file; cells form a dependency graph; when you edit a cell, Marimo re-runs all of its dependents automatically. There is no way for the displayed state to diverge from what the code computes, because the runtime enforces topological order on every edit.
+
+A minimal Marimo notebook looks like this — note it is ordinary Python you can read in any editor:
+
+```python
+# notebook.py
+import marimo as mo
+
+app = mo.App()
+
+@app.cell
+def __():
+    import pandas as pd
+    df = pd.read_csv("data/raw.csv")
+    return (df,)
+
+@app.cell
+def __(df):
+    clean = df.dropna()
+    mo.md(f"Rows: **{len(clean)}**")
+    return (clean,)
+```
+
+Key commands:
+
+- `marimo edit notebook.py` — opens the reactive editor in your browser
+- `marimo run notebook.py` — serves the notebook as a read-only web app (great for stakeholders)
+- `marimo export html notebook.py -o out.html` — static HTML snapshot for reports
+
+Because the file is `.py`, `git diff` shows real code changes. Code review on a Marimo notebook works the same as code review on any Python file. There are no output cells to strip, no JSON diffs to parse.
+
+### Jupyter plus jupytext fallback
+
+When Marimo is not an option — you depend on a Jupyter-only widget, you share notebooks with non-Marimo users, or your infrastructure is built around `.ipynb` — use **jupytext** to pair each `.ipynb` with a `.py` representation. Install jupytext, then configure pairing at the repo root:
+
+```toml
+# .jupytext.toml
+formats = "ipynb,py:percent"
+notebook_metadata_filter = "-all"
+cell_metadata_filter = "-all"
+```
+
+Or pair a single notebook explicitly:
+
+```bash
+jupytext --set-formats ipynb,py:percent notebooks/eda.ipynb
+```
+
+The `py:percent` format splits cells with `# %%` markers and produces a clean, diffable Python file. Rule of thumb for the repo:
+
+- **Commit** the `.py` version — it is the source of truth for review and diffs
+- **Gitignore** the `.ipynb` (or commit it with `nbstripout` installed to strip outputs; see the data-science-security doc for the outputs-as-secrets angle)
+- **Do not** try to keep both hand-edited — jupytext's pre-save hook keeps them in sync automatically
+
+This does not fix hidden state (Jupyter still runs cells in click-order), but it does make review and merges sane, and it gives you a textual artifact that survives kernel-state bugs.
+
+### Promotion: notebook to src to test to re-import
+
+The most important habit in any notebook workflow — Marimo or Jupyter — is **promotion**. The moment a cell does real work, extract it to a tested module and import it back.
+
+Before (inline in the notebook, untested, untyped):
+
+```python
+@app.cell
+def __(df):
+    df["hour"] = pd.to_datetime(df["ts"]).dt.hour
+    df["is_weekend"] = pd.to_datetime(df["ts"]).dt.dayofweek >= 5
+    df["log_amount"] = np.log1p(df["amount"])
+    return (df,)
+```
+
+After — extract to `src/features/engineer.py`:
+
+```python
+# src/features/engineer.py
+import numpy as np
+import pandas as pd
+
+def add_time_features(df: pd.DataFrame, ts_col: str = "ts") -> pd.DataFrame:
+    """Add hour and is_weekend columns derived from a timestamp column."""
+    out = df.copy()
+    ts = pd.to_datetime(out[ts_col])
+    out["hour"] = ts.dt.hour
+    out["is_weekend"] = ts.dt.dayofweek >= 5
+    return out
+
+def add_log_amount(df: pd.DataFrame, amount_col: str = "amount") -> pd.DataFrame:
+    out = df.copy()
+    out["log_amount"] = np.log1p(out[amount_col])
+    return out
+```
+
+Write a test — small, fast, no data dependency:
+
+```python
+# tests/features/test_engineer.py
+import pandas as pd
+from src.features.engineer import add_time_features
+
+def test_weekend_flag_friday_vs_saturday():
+    df = pd.DataFrame({"ts": ["2026-04-17 10:00", "2026-04-18 10:00"]})
+    out = add_time_features(df)
+    assert out["is_weekend"].tolist() == [False, True]
+```
+
+Re-import in the notebook:
+
+```python
+@app.cell
+def __(df):
+    from src.features.engineer import add_time_features, add_log_amount
+    df = add_log_amount(add_time_features(df))
+    return (df,)
+```
+
+The notebook becomes a thin orchestration + visualization layer over tested modules. Hidden state matters less because the logic lives in files that are exercised by CI. Pull requests become reviewable — the reviewer reads typed functions with tests, not a wall of chained DataFrame mutations.
+
+### Running notebooks as pipelines
+
+Finished notebooks often need to run on a schedule — daily reports, weekly retraining, monthly audits. Do not copy-paste the code into a script; run the notebook directly.
+
+**Marimo**: because the file is already Python, you can run it as a script or as an app:
+
+```bash
+marimo run notebook.py                    # serve as web app
+python notebook.py                        # execute top-to-bottom as a plain script
+marimo export html notebook.py -o out.html # produce a static report artifact
+```
+
+**Jupyter**: use `papermill` to parameterize and execute an `.ipynb`, producing an executed output notebook:
+
+```bash
+papermill notebooks/weekly_report.ipynb \
+          outputs/report_$(date +%Y%m%d).ipynb \
+          -p start_date 2026-04-14 \
+          -p end_date 2026-04-20
+```
+
+Parameterized cells (tagged `parameters` in Jupyter) are injected by papermill at the top of the run. Use Marimo's `mo.cli_args()` for the equivalent in Marimo. Either way, pair this with a lightweight scheduler (cron, GitHub Actions, Airflow, Prefect) — the notebook is the unit of work, not a script that tries to re-implement it.
+
+A useful rule: if a notebook is scheduled to run unattended, its logic should be ~90% imports from `src/` and ~10% glue. The promotion discipline from the previous section is what makes scheduled notebook runs trustworthy.