open-data-sci 0.1.0__py3-none-any.whl

opendatasci/resources/skills/competitive_data_science.md

@@ -0,0 +1,241 @@
+# Competitive Data Science Skill
+
+A phased playbook for high-stakes data science competitions, built around incremental delivery. Each phase opens with planning so direction is set before work begins, and the high-leverage phases close with a review so signals from the current phase loop back into earlier decisions when warranted. The wiring at the end of the document spells out the most common loop-backs explicitly.
+
+The phases describe what tends to matter at each stage, the knowledge that informs the choices, and the tricks that experienced competitors rely on. They do not prescribe a single path — flexibility to adapt the order, skip a step, or revisit a phase is part of the playbook.
+
+## Phase 0 — Reconnaissance & Harness
+
+**Planning**
+- The shape of the campaign — how many days for baseline, EDA, feature work, model development, ensembling, final-week consolidation — is set in this phase; without an explicit cadence, single phases tend to absorb disproportionate time
+- The deliverable from this phase is a working end-to-end pipeline (load → split → train → predict → submission file) and an experiment log; later phases iterate on individual steps without rebuilding the harness
+- Compute and time budgets influence model choice as much as data does; framing them up front avoids late discovery that the chosen architecture is untrainable in the available window
+- Teaming, when permitted, multiplies compute, diversity of ideas, and the volume of experiments that can be run in parallel — many winning solutions are ensemble products of multiple collaborators' independent pipelines; the decision is best made early because team dynamics and shared infrastructure benefit from time to develop, while solo competition remains viable and sometimes preferable when fast iteration matters more than diversity and coordination overhead would slow decisions down
+
+**Knowledge & Information**
+- The scoring metric is part of the modelling problem rather than its preamble; non-standard metrics (MAP@K, RMSLE, weighted log-loss, F-beta, Quadratic Weighted Kappa, AUC-PR variants) typically reward bespoke loss surrogates, custom objectives, or post-processing rather than naive optimisation of a generic objective
+- Sponsoring organisations and prior editions of the competition often reveal which signal sources the data was constructed to expose and which were intentionally redacted; reading past winners' writeups on the same platform compresses weeks of independent exploration into hours of reading
+- Public discussion forums concentrate the highest-density information in the first days of a competition — data quirks, evaluation edge cases, label issues, leaks — and disproportionately benefit those who read them first
+- Competitions with similar data modality or problem type on the same hosting platform frequently share winning patterns (specific feature families, model choices, post-processing tricks) that transfer with light adaptation
+- The submission format (column order, required precision, header expectations, prediction range) is part of the contract; mismatches cost ranked submissions
+- If the dataset derives from a public source, the original documentation, schema descriptions, and domain context often resolve ambiguities the competition description leaves open
+
+**Tricks**
+- A "v0" submission that returns a constant (target mean, modal class, sample submission unchanged) validates the I/O pipeline end-to-end before any model exists and locks in the public LB anchor for later comparisons
+- A versioned experiment log capturing CV score, public LB score, brief change description, feature set, and model class is the difference between knowing what worked and guessing — its value compounds with every submission
+- Pre-committing to file naming conventions for predictions, OOF arrays, and submission files (e.g. `oof_<model>_<seed>.npy`, `sub_<model>_<date>.csv`) removes friction when many experiments coexist
+- Fixing random seeds for every stochastic component from the start ensures later comparisons reflect genuine improvements rather than variance
+
+## Phase 1 — Exploratory Data Analysis
+
+**Planning**
+- A productive EDA phase has an explicit question list (what one row represents, how train and test differ in distribution, where missingness sits, what natural groupings exist, what the target looks like across subgroups) rather than open-ended browsing
+- Time-boxing prevents the common failure mode of polishing plots while the competition clock runs; the goal is enough understanding to design validation and a first feature set, not exhaustive characterisation
+- The artefacts worth producing from EDA — a data dictionary, a list of suspect columns, a hypothesis list for feature engineering, a clear picture of train/test differences — feed directly into the next two phases
+
+**Knowledge & Information**
+- "What does one row represent?" is one of the most consequential questions to settle early; grain mismatches between datasets or between train and test are a frequent source of silent errors when joining or comparing
+- Profiling shape, dtypes, missing-value rates, cardinality, descriptive statistics, and target distribution before modelling surfaces most data quirks worth knowing
+- Inspecting train and test feature distributions side by side, including missingness patterns and category-level coverage, reveals distribution shift and features whose meaning differs across the split
+- Adversarial validation — training a binary classifier to distinguish train from test — quantifies distribution shift; an AUC well above 0.5 means random CV will overestimate test performance, and the most predictive features in that classifier are the suspect ones
+- For time-indexed data, a chronological view reveals gaps, seasonality, trend breaks, and regime changes before they distort downstream work
+- Duplicate rows, near-constant columns, suspect nulls, and outliers can distort aggregations and model training in ways that are difficult to trace later
+- The target's marginal distribution (class balance, skew, heavy tails, zero inflation) informs both loss choice and metric interpretation; rare positives in particular shape sampling and threshold strategies
+
+**Tricks**
+- Plotting target by every feature (binned for numeric, level-by-level for categorical) is a cheap, high-information scan for non-linearity, monotonicity, and useful interactions
+- Computing target statistics across categorical levels directly identifies the strongest column-level predictors and seeds the first round of target-encoded features
+- Inspecting the most and least frequent values per column quickly surfaces encoded missingness sentinels, unit or currency changes, and high-cardinality leakage indicators
+- A short "data dictionary" file summarising column meaning, grain, and observed quirks pays for itself many times over when feature engineering ramps up
+- Visualising row-level NaN patterns (e.g. as a sorted boolean matrix) often reveals structural missingness tied to entity type, time period, or recording source
+
+## Phase 2 — Validation Strategy
+
+**Planning**
+- The validation scheme is the single most leverage-laden decision in a competition and is best designed before serious modelling begins; a strong local CV that correlates tightly with leaderboard score is more valuable than any single model improvement
+- The plan starts from how the test set was constructed (time cutoff, geographic split, entity holdout, draw from a different distribution) and replicates that structure in CV
+- Persistence of fold assignments across the campaign ensures every model's OOF predictions are directly comparable downstream
+
+**Knowledge & Information**
+- CV splits should mirror the test split: time-ordered for temporal holdouts, group-aware when rows share an identity (user, device, session, location), stratified for rare outcomes, nested combinations when several conditions apply
+- The public leaderboard is a noisy signal on a small sample; private leaderboards routinely reshuffle relative to public — trusting a robust local CV over public-LB chasing is the default of strong competitors, though when the public sample is large, the CV-LB correlation has been demonstrably tight across submissions, or the test split is known to be drawn from the same distribution as train, LB carries genuine information worth weighing alongside CV
+- The gap between local CV and public LB across submissions is itself a diagnostic: a consistent offset is acceptable, an inconsistent one signals that the CV scheme does not reflect the test distribution
+- Repeated CV (multiple seeds, multiple shuffles) reduces noise in the validation estimate at the cost of compute and is worth running for final model selection rather than every iteration
+- Out-of-fold (OOF) predictions are a free byproduct of CV that enables stacking, post-processing calibration, and error analysis — saving them by default removes friction later
+- Adversarial validation results from Phase 1 directly inform CV design: when a feature separates train from test, validating on a fold drawn from the train distribution will not reflect how the model performs on the test distribution
+- The public LB is computed on a small fraction of the test set (often 20–50%); its score variance is large enough that small public-LB movements between submissions often reflect noise rather than improvement
+
+**Tricks**
+- Constructing a fold structure that explicitly mimics observed train/test differences (e.g. using the most recent time slice as the validation fold when the test set is the future) is more reliable than relying on stratification alone
+- A small "blend holdout" — a fold reserved for selecting ensemble weights and never used during base model training or hyperparameter search — preserves the integrity of ensembling decisions
+- Persisting fold assignments to disk and re-using them across every model in the campaign keeps OOF predictions directly comparable and enables clean stacking later
+- Sample-weighted CV, where validation weights reflect the test distribution (e.g. up-weighting recent observations under temporal shift), can produce CV scores that track LB more tightly than equal-weight CV
+
+**Review** (close before moving on)
+- Does the CV scheme replicate the test set construction? If not, redesign before any feature or model work proceeds
+- Is the CV variance across folds small enough that meaningful improvements will be distinguishable from noise?
+- Has at least one baseline submission anchored the CV-LB correspondence? If not, defer further work until it has
+- Have OOF predictions and fold assignments been persisted so they can be reused throughout the campaign?
+
+## Phase 3 — Baseline Model
+
+**Planning**
+- The point of the baseline is to compress the end-to-end pipeline into something runnable — data load, validation split, training, prediction, submission file — before optimising any single step
+- A baseline submitted within the first one or two days establishes the floor and reveals pipeline bugs while they are still cheap to fix
+- The baseline doubles as a measurement device: the value of every later improvement is expressed relative to this anchor
+
+**Knowledge & Information**
+- A naive baseline (target mean, mode, last-value carry-forward, simple rule keyed off the most predictive column) is the floor against which all subsequent complexity is measured — sometimes it is surprisingly hard to beat, which is itself a strong signal about the problem
+- For tabular data, a gradient-boosting model with sensible defaults trained on raw features is the natural first real baseline; for text, TF-IDF with logistic regression or a small distilled transformer; for image, a pre-trained backbone with a linear head; for time series, a seasonal-naive or simple boosted-tree lag model
+- Submitting the baseline locks in the CV-LB correspondence and provides the reference point for every later experiment
+- The baseline's per-fold variance characterises noise floor — improvements smaller than this variance are unlikely to be real
+
+**Tricks**
+- Logging baseline metrics across all CV folds (mean, std, per-fold scores) characterises stability and informs how much variance later improvements need to overcome
+- Storing OOF predictions and feature importances from the baseline produces an immediate map of which features matter and where the model is uncertain — both feed directly into Phase 4
+- A baseline ablation (one feature removed at a time, scored against CV) is cheap and surfaces leakage candidates and dead features before serious feature work begins
+
+## Phase 4 — Feature Engineering
+
+**Planning**
+- A feature plan starts from explicit hypotheses about signal sources (relational structure, temporal context, interactions, domain knowledge) rather than mechanical generation of every possible transformation
+- Features cheap to compute and individually testable (each evaluated against the same CV scheme) make iteration fast and attribution clear
+- A per-iteration feature budget — add N features, evaluate, prune — prevents accumulation of dead weight and keeps the feature set interpretable
+- The feature engineering plan is the longest phase in most competitions; structuring it as a sequence of small, evaluable batches makes progress visible and prevents the search from going stale
+
+**Knowledge & Information**
+- Competition datasets frequently reward entity-level aggregations: group-by statistics (mean, std, min, max, count, nunique, skew, median) computed across users, sessions, locations, or time windows encode relational structure that row-level features miss
+- Target encoding (mean of the target per category level, smoothed against the global mean) is consistently effective for high-cardinality categoricals but must be computed within each CV fold to avoid leakage
+- Lag features, rolling statistics, expanding-window aggregations, exponentially weighted means, and time-since-event features form the core vocabulary for temporal datasets; their window sizes are hyperparameters worth searching
+- Interaction features (products, ratios, differences, polynomial terms) between top-importance columns often outperform any single transformation; the cheap heuristic is to interact the top-K features from the baseline's importance ranking
+- Frequency encoding (count of occurrences of each category level) is a near-free, leakage-safe alternative to one-hot for high-cardinality columns
+- Cyclic encodings (sine/cosine of hour, day-of-week, month) preserve continuity at the boundaries that integer encoding does not natively express; the benefit is most pronounced for linear and neural models, while tree-based models can recover the modular structure through multiple splits on the integer-encoded feature when enough data is available
+- For text, character-level and word-level n-grams, length statistics, sentiment scores, and pre-trained sentence embeddings each capture different aspects of the signal and ensemble well
+- For image, traditional descriptors (HOG, colour histograms, LBP) and pre-trained backbone embeddings complement each other when compute is constrained
+- Categorical features with cardinality in the thousands to millions typically respond better to target encoding, hashing, or learned embeddings than to one-hot; the right choice depends on the model family and the data volume
+- Feature provenance matters: any feature computed using information unavailable at prediction time silently inflates CV and LB scores — tracing the construction of every feature against the data timeline is the only defence
+
+**Tricks**
+- Permutation importance and SHAP values on a trained baseline give a more reliable feature ranking than impurity-based importance, which biases toward high-cardinality features; impurity-based importance remains useful as a near-zero-cost first pass, particularly when many features need a directional ranking quickly or when permutation/SHAP would be prohibitively expensive
+- Target encoding with K-fold nested inside the outer CV is the standard leak-safe pattern; failing to nest is the single most common source of silent CV-LB gaps in competition pipelines
+- Forward feature selection by greedy CV gain is expensive but surfaces the truly load-bearing subset; backward elimination starting from the full feature set is faster and often sufficient
+- A "control" feature — pure random noise added to the feature set — calibrates how much importance is attributable to chance; any real feature ranking below it is a candidate for pruning
+- Re-running adversarial validation after adding new features detects features that encode the train/test split itself
+- Storing the feature engineering as a pure transformation function (fit on train, applied to any split) prevents training/inference skew and makes leak-safety easier to audit
+
+**Review** (close before moving on)
+- Does the CV improvement from new features hold on a freshly seeded split? If not, suspect overfitting to fold structure — return to Phase 2 to evaluate CV variance and possibly redesign
+- Has adversarial validation been re-run after the new feature set? If new features separate train from test more easily than before, those features encode the split — return to Phase 1 to inspect them and consider removal
+- Are feature importances dominated by a single feature with implausibly high signal? Suspect leakage — trace the feature's construction against the data timeline before trusting any downstream metric
+- Is the CV-LB gap consistent with the pre-feature-work baseline? A sudden divergence is a leak signal or a CV scheme problem — if persistent, return to Phase 2
+- Has redundancy been pruned? Highly correlated features add noise without value and slow training; a brief correlation review at the end of the phase pays off in later iteration speed
+
+## Phase 5 — Model Development
+
+**Planning**
+- The model plan covers a portfolio of model families to evaluate (linear, tree-based, neural) with a budget per family, rather than a single architecture to perfect
+- Building several distinct base models — even when individually weaker than the strongest — pays compounding returns at the ensembling phase
+- Iteration speed dominates progress in this phase; running on a representative stratified subsample first and full data once the architecture is settled cuts wall-clock cost substantially without losing directional signal
+
+**Knowledge & Information**
+- Model selection follows problem structure and data characteristics rather than a habitual preference for any one family; the choice is a design decision deserving the same rigour as any other modelling choice
+- For tabular data, gradient-boosted decision trees are the most consistent top performer across small-to-medium datasets; differences across implementations in handling of categoricals, missing values, and split criteria occasionally swing one toward better performance on a given dataset, and at very large scale or in domains with rich high-cardinality interactions (CTR prediction, recommendation, certain industrial datasets) neural approaches can match or surpass them
+- Linear and logistic regression with engineered features can be primary contenders in low-data regimes, when interactions are well-captured by hand-crafted features, or under strict interpretability constraints; they also serve as complementary ensemble components, as a sanity check on whether non-linear models add value, and as the standard meta-learner in stacking
+- Neural architectures on tabular data (MLP, TabNet, FT-Transformer, NODE) are competitive at scale and increasingly close the gap to boosted trees on small-to-medium datasets when learned categorical embeddings or cross-feature interactions carry signal; their primary value in many competitions is diversity contribution to the ensemble, but in data regimes where they match or beat boosted trees they belong as a primary candidate rather than only as an ensemble component
+- For text, fine-tuned transformer checkpoints (BERT-family, DeBERTa, RoBERTa, ELECTRA, distilled variants) lift performance substantially over feature-based baselines at meaningful compute cost on most natural-language tasks; on very short, highly structured, or heavily label-noisy text, TF-IDF or hashed n-grams with a linear classifier can match or outperform transformers at a fraction of the cost
+- For image, pre-trained backbones (EfficientNet, ConvNeXt, ViT, Swin) via transfer learning are the standard entry point; augmentation design, head architecture, and training schedule often move the score more than swapping backbones of similar capacity, while in tasks where the backbone's inductive bias aligns particularly well with the data (fine-grained classification, medical imaging, satellite imagery, dense prediction) the backbone choice itself can be the dominant factor
+- For time series with many parallel series, gradient boosting on lag features competes with and often beats dedicated forecasting architectures (LSTM, Temporal Fusion Transformer); the dedicated architectures win when complex exogenous structure or long-range dependencies dominate
+- Test-time augmentation (TTA) — averaging predictions over augmented copies of each test instance — produces small but consistent gains in image tasks and sometimes in text and tabular
+- Models train on the train split and select hyperparameters on the validation fold; the test set is never seen by any model selection process
+
+**Tricks**
+- Training the same model with several random seeds and averaging predictions is the cheapest, most reliable way to reduce variance and improve score
+- For boosted trees, early stopping against the validation fold within each CV split removes the n_estimators hyperparameter from the search and acts as the primary regulariser
+- Pseudo-labelling — training on high-confidence test predictions, then re-training — can add meaningful gains when the test set is large relative to train, but risks amplifying mistakes if confidence calibration is poor
+- Saving OOF and test predictions from every meaningful model run feeds Phase 7 directly; ensembling later without these arrays forces re-running expensive trainings
+- Knowledge distillation (training a smaller or differently-architected student on the soft predictions of a strong teacher) produces useful diversity for ensembling when the student family is genuinely different from the teacher's
+- For any neural training, starting with a low number of epochs (one or two), monitoring train and validation curves, and continuing only when results warrant it avoids wasted compute on unpromising architectures and surfaces data or pipeline issues early
+
+## Phase 6 — Hyperparameter Tuning
+
+**Planning**
+- A tuning budget — number of trials, wall-clock cap — set before search starts prevents the common failure mode of tuning expanding to fill all available time with diminishing returns past the first 50–100 trials per model
+- The search strategy matches the budget: Bayesian optimisation for large continuous spaces, random search for moderate spaces, grid search for small or discrete spaces where exact coverage and reproducibility of the search matter more than efficiency
+- Running the search on a representative stratified subsample and validating the winner on the full dataset is the standard speed-cost tradeoff
+
+**Knowledge & Information**
+- The same CV scheme used for evaluation should be used for tuning to keep estimates consistent and comparable
+- Reporting the distribution of CV scores across configurations (not only the best) characterises sensitivity and reveals whether the winner is a stable optimum or a lucky tail draw
+- For boosted trees, the high-leverage hyperparameters are typically learning rate, number of leaves / max depth, min child weight / min data in leaf, L1/L2 regularisation, feature/row subsampling fractions, and early-stopping rounds
+- For neural networks, learning rate and learning-rate schedule, batch size, optimiser choice, weight decay, dropout, and augmentation strength dominate; architecture changes often matter less than these
+- Successive halving and Hyperband prune unpromising configurations early and routinely cut search cost by 3–10× while typically preserving the winning configuration; the pruning can occasionally drop late-bloomers whose early loss is high but whose converged optimum is strong, so a full evaluation budget on a small confirmation slate is a reasonable hedge when search outcomes are surprising
+- Winning hyperparameters from past competitions on the same data modality often transfer as strong defaults and reduce the search to narrow refinement
+
+**Tricks**
+- Persisting the full search history (every trial's parameters, score, and intermediate state) enables resuming after interruptions, inspecting parameter importance, and warm-starting future searches in the same competition
+- A coarse-to-fine schedule — a wide search with few trials, then a narrow search around the best region — is more efficient than a single broad search
+- Tuning multiple models in parallel using independent studies, then ensembling, often yields more total signal than exhaustively tuning a single model
+
+## Phase 7 — Ensembling & Stacking
+
+**Planning**
+- The ensembling plan starts from the set of diverse base models built across Phase 5 rather than from squeezing a final percentage point out of any single model
+- A simple weighted average is the natural first ensemble; stacking is the next step when base model errors are uncorrelated enough to support a meta-learner
+- Ensemble selection and weighting are performed on a holdout that no base model has seen — typically a reserved blend fold or out-of-fold predictions — to avoid overfitting the blend
+
+**Knowledge & Information**
+- Diversity of predictions drives ensemble gains; two moderately strong models with uncorrelated errors outperform two strong models that make the same mistakes
+- Ensembling models from the same family (multiple gradient-boosting implementations, multiple variants of the same neural architecture) produces highly correlated predictions, but their differences in growth policy (leaf-wise vs. level-wise), categorical handling, regularisation, or initialisation are large enough that small but consistent gains over the single best member are common — in tight competitions where every fraction of a metric point matters, within-family blends are worth keeping in the ensemble even when their marginal lift is modest
+- The largest gains typically come from combining genuinely different families (gradient boosting + neural network + linear) or models trained on substantially different feature sets or data samples
+- Stacking with a simple meta-learner (logistic regression, ridge, light boosted tree with few leaves) on out-of-fold predictions captures systematic differences between base models with low overfitting risk on the blend; more expressive meta-learners can still be appropriate when base models are many and their interactions are non-trivial, provided the meta-learner is itself validated on a fold disjoint from the one used to train it
+- Weight optimisation via constrained optimisers (Nelder-Mead, simplex methods, gradient-based solvers under a simplex constraint) on held-out CV often improves over uniform averaging when base models have meaningfully different strengths; validating the optimised weights on a separate fold guards against overfitting the blend
+- Geometric mean (averaging in log-space) and rank-averaging are alternatives to arithmetic mean that work better when predictions have heterogeneous scale or are used as rankings rather than probabilities
+
+**Tricks**
+- Adding a poorly tuned, low-capacity model from a different family (a small MLP alongside boosted trees) often improves the ensemble despite being individually weaker
+- Correlation matrices of OOF predictions across base models reveal which models contribute genuine diversity and which are redundant; pruning redundant models stabilises the blend without hurting score
+- Training base models on different feature subsets (full, A, B) and ensembling produces cheap diversity without new architectures
+- Capping prediction range to the empirical target range, or clipping outliers, can yield small but reliable gains under metrics that penalise extreme errors
+- Multi-level stacking (a second-stage stacker over first-stage stacker outputs) has won mature, ensemble-heavy competitions where many strong and architecturally varied base models exist, but the marginal return diminishes quickly with each added stage and the complexity overhead is high — single-level stacking with diverse base models captures most of the available signal in most settings, and the additional level is worth the cost mainly when the ensemble is already large and well-validated
+
+**Review** (close before moving on)
+- Does the ensemble CV exceed the best single-model CV by a margin that survives across seeds? If not, the ensemble is not contributing — return to Phase 5 to build genuinely different models, or to Phase 4 to construct alternative feature subsets
+- Are the base model OOF predictions correlated above ~0.95? Diversity is insufficient — return to Phase 5 (different family) or Phase 4 (different feature subset)
+- Is the public LB improvement consistent with the CV improvement? If not, the blend is overfitting OOF — return to Phase 2 to inspect CV structure
+- Has the meta-learner been validated on a fold disjoint from the one used to fit base models? If not, the stacking estimate is optimistic
+
+## Phase 8 — Final Submission Selection
+
+**Planning**
+- Most platforms allow two final submissions for private leaderboard evaluation; diversification — typically one best-CV submission and one best-LB submission — reduces the chance of a catastrophic private-LB reshuffle
+- Final candidates should be prepared and validated 24–48 hours before the deadline; reserving a tested pipeline for final-day use prevents last-minute breakage from costing the competition
+- The final day is for selection, sanity checks, and re-running the submission pipeline end-to-end — not for new architectures or feature ideas
+
+**Knowledge & Information**
+- The private leaderboard frequently reshuffles relative to public; a position-conservative final selection hedges against either signal being misleading
+- Confidence in CV over LB increases when the CV scheme provably replicates the test split, when CV variance is small, and when the CV-LB gap has been consistent across many submissions
+- Some late-stage strategies — clipping predictions to a tighter range, blending with a constant, applying a learned calibration, threshold optimisation on OOF for classification under non-standard metrics — yield small but reliable gains
+- Threshold optimisation under metrics like F-beta or Quadratic Weighted Kappa is performed on OOF predictions and applied to the final test predictions
+- The shake-up risk between public and private leaderboards is higher in competitions with small test sets, severe class imbalance, or distribution shift between train and test — adjusting final-selection conservatism accordingly is part of the strategy
+
+**Tricks**
+- A "safety" submission constructed as the average of the top-N CV submissions is often more robust than picking any single submission and frequently outperforms it on private LB
+- Comparing the prediction distribution of the final submission against the training target distribution catches obvious calibration mistakes (overly confident, biased toward one class) before they cost the leaderboard
+- A short pre-submission checklist (correct rows, correct column order, no NaN, range plausible, file size sane, file opens in the platform's preview) catches the most embarrassing mistakes
+- Test-time augmentation and multi-seed inference averaging for image and neural pipelines apply naturally at the final stage and tend to nudge the score upward without changing any other component
+
+## Phase Wiring
+
+When signals from a later phase challenge the assumptions of an earlier one, looping back is part of the playbook rather than a sign of failure. The most common patterns:
+
+- **Consistent CV-LB gap that diverges after introducing new features** → return to Phase 4's review; if unresolved, return to Phase 2 to redesign the CV scheme
+- **Adversarial validation easily separates train from test** → return to Phase 1 to inspect the responsible features and to Phase 4 to engineer around them (or remove them)
+- **A single feature dominates importance with implausibly high signal** → return to Phase 4 to trace feature provenance for leakage before trusting any downstream metric
+- **Ensemble does not improve over the best single model** → return to Phase 5 to build genuinely different model families, or to Phase 4 to construct alternative feature subsets
+- **Hyperparameter tuning yields large CV gains that do not appear on LB** → return to Phase 2 (CV scheme may not reflect test) or Phase 4 (features may be CV-specific)
+- **Final-week consolidation reveals brittleness in the pipeline** → reserve compute for stability over additional features and return to Phase 0 to harden the harness
+- **Mid-competition shared insight from the discussion forum changes the picture** (e.g. a documented leak, a previously unknown grouping structure) → revisit Phase 1 to incorporate the new understanding, then re-validate Phase 2 and Phase 4 in light of it
+- **Distribution shift detected when comparing final-submission predictions to the training target distribution** → return to Phase 1 (re-examine differences) and Phase 2 (verify CV scheme reflects them)
+- **CV variance per fold is large enough to obscure improvements** → return to Phase 2 to consider repeated CV, more folds, or a fold structure better aligned with the test distribution
+
+The loops are bounded by time: late in the campaign, the cost of redesigning validation or pruning load-bearing features is high, and Phase 8's diversification across CV and LB is the pragmatic hedge against decisions that can no longer be unwound.

opendatasci/resources/skills/data_science.md

@@ -0,0 +1,55 @@
+# Data Science Skill
+
+**Framing the Problem**
+- Vague requests benefit from being grounded in a concrete hypothesis or target metric before touching any data — the question shapes every downstream choice
+- Understanding what a "good" answer looks like up front (decision to be made, threshold for action, audience) prevents wasted analysis
+- Distinguishing exploratory work (generating hypotheses) from confirmatory work (testing them) matters — mixing the two silently inflates false discovery rates
+
+**Exploratory Analysis**
+- Data rarely arrives in the shape expected; profiling shape, dtypes, missing value rates, cardinality, and basic descriptive stats early tends to surface surprises before they become silent errors
+- Duplicate rows, near-constant columns, and unexpected nulls can distort aggregations and models in ways that are hard to trace later
+- For numeric distributions, tools like histograms and box plots reveal skew, modality, and outlier structure; for categoricals, frequency distributions expose long tails and rare levels worth knowing about before modelling
+- Correlation analysis (linear and monotonic) helps map feature relationships; high pairwise correlation can matter for model interpretability even when it doesn't hurt predictive accuracy
+- Time-indexed data rewards a chronological view before any aggregation — gaps, seasonality, trend breaks, and data collection artefacts tend to show up immediately and change how the data should be handled
+
+**Data Quality & Preparation**
+- Understanding *why* data is missing (structurally absent, randomly missing, or missing in a way correlated with the outcome) shapes the right response — imputation, exclusion, or flagging as a separate signal
+- Outliers deserve investigation before any treatment; distinguishing measurement error from genuinely extreme values is consequential — removing real extremes can mask the most interesting signal
+- Joining datasets is a common source of silent row inflation or key loss; checking counts and cardinality before and after a join is a lightweight step that often catches real problems
+- Encoding choices interact with the model: ordered features carry ordinal meaning, nominal features don't — treating them the same can introduce spurious relationships
+- Scaling matters for methods sensitive to feature magnitude and is irrelevant for others; knowing which is which avoids unnecessary transformation
+
+**Causality & Confounding**
+- Correlation between two variables rarely tells you which causes which, or whether a third variable drives both — most real-world datasets are observational and can't establish causation without additional assumptions or experimental design
+- Confounders — variables that influence both the feature and the outcome — can make a spurious relationship look real or mask a genuine one; identifying and controlling for them is central to any analysis aimed at understanding what to do, not just what happened
+- Selection bias and survivorship bias are pervasive: the data available is often not a random sample of the population of interest (e.g., only active customers, only completed transactions, only surviving products); the conclusions drawn are only as valid as that sample
+- Simpson's paradox is surprisingly common: a trend visible in aggregate can reverse when broken down by a subgroup — always worth checking whether aggregate results hold across meaningful partitions before drawing conclusions
+
+**Granularity & Aggregation**
+- "What does one row represent?" is one of the most important questions to establish early — grain mismatches between datasets are a frequent source of silent errors when joining or comparing
+- Aggregation choices (sum vs. mean vs. median, weekly vs. monthly, per-user vs. per-event) embed analytical decisions that change what the numbers mean; making them explicit prevents misinterpretation
+- Aggregating too early can destroy signal; aggregating at the wrong level can introduce it artificially
+
+**Statistical Testing**
+- Sample size and statistical power deserve attention before interpreting null results — a study that fails to detect an effect may simply be underpowered; estimating the sample size needed to detect a meaningful effect size is a useful sanity check
+- The right test depends on data structure, distribution, and the independence assumptions that can actually be defended — parametric tests have assumptions worth checking before applying
+- Non-parametric alternatives trade statistical power for fewer assumptions; the right tradeoff depends on sample size and how badly assumptions are violated
+- Testing many hypotheses simultaneously inflates false positives in ways that compound quickly; multiple comparison correction adjusts for this but also reduces sensitivity — the right correction depends on whether you're guarding against any false positive or controlling the proportion of false discoveries
+- P-values and effect sizes answer different questions: significance says whether an effect is detectable given sample size; effect size says whether it's large enough to matter — both are needed to judge a result
+- Confidence intervals communicate uncertainty more directly than p-values alone and translate more naturally into business language
+
+**Modeling & Evaluation**
+- A naive baseline (mean/mode predictor, last-value carry-forward, a simple rule) makes the value of a model concrete — sometimes the baseline is surprisingly hard to beat, which is itself informative
+- Model choice should be driven by the problem structure, not by a default preference for a particular algorithm: consider the full spectrum from linear models (interpretable, well-regularised) through tree-based ensembles (robust to feature scale, capture interactions) to neural networks (high capacity, need volume and tuning) — the right family depends on signal strength, data volume, interpretability needs, and the nature of the decision boundary or regression surface
+- Metric choice should reflect the real objective and data distribution; accuracy misleads on imbalanced problems, RMSE penalises large errors disproportionately, percentage-based errors behave badly near zero — each metric embeds assumptions worth making explicit
+- Split strategy encodes assumptions about how the model will be used: stratified splits for class balance, time-ordered splits when temporal structure exists, group-aware splits when rows share an entity; the wrong strategy produces optimistic numbers that don't hold
+- Evaluating on a single held-out set can be noisy; cross-validation spread gives a better picture of how stable performance is across different data slices
+- Slicing metrics by relevant subgroups or prediction ranges often reveals where a model underperforms in ways aggregate numbers conceal
+- When combining multiple models, diversity of predictions is what drives ensemble gains — architecturally similar models (e.g., XGBoost, LightGBM, and CatBoost) are likely to produce highly correlated outputs and make the same mistakes, so blending them yields little improvement; meaningful gains come from ensembling models from different families (tree-based, linear, neural) or models trained on different feature sets or subsets of the data
+
+**Communicating Findings**
+- Estimates with uncertainty (intervals, spread across folds) are more useful than point values — they convey how stable a result is and what confidence is warranted
+- Leading with the key finding in plain language, then following with technical detail, tends to serve mixed audiences better than leading with methodology
+- Caveats, assumptions, and plausible alternative explanations are part of a rigorous analysis, not afterthoughts — surfacing them early builds rather than undermines credibility
+- If the data isn't sufficient for a strong conclusion, saying so clearly — and describing what additional data, sample size, or experimental design would change that — is itself a useful output
+""".strip()

opendatasci/resources/skills/data_science_education.md

@@ -0,0 +1,42 @@
+# Data Science Education Skill
+
+**Diagnosing Understanding**
+- Before explaining, probing what the learner already knows prevents both over-explaining and skipping foundations they're missing; the right entry point depends entirely on their current mental model
+- Misconceptions are more persistent than gaps; identifying what the learner believes and why they believe it is more useful than simply restating the correct answer
+- Concrete questions ("walk me through how you'd approach this problem") reveal understanding better than self-assessments ("do you understand X?")
+
+**Building Intuition First**
+- Intuition precedes formalism; a learner who understands *why* a technique exists will navigate the details better than one who has memorised the procedure without the rationale
+- Analogies are high-leverage teaching tools but carry the risk of breaking down at the edges; making the limits of an analogy explicit prevents learners from over-extending it
+- Visual and geometric explanations often make abstract statistical concepts concrete: the bias-variance tradeoff, gradient descent, covariance, and principal components all have strong geometric interpretations that formulas alone don't convey
+- Motivating examples — showing a real problem the technique solves before introducing the technique — give learners a reason to engage with the material
+
+**Structuring Explanations**
+- Moving from concrete to abstract is more effective than the reverse; a worked example before a general formula is harder to forget
+- Chunking — breaking a complex concept into distinct, independently understandable pieces — reduces cognitive load; presenting everything at once can overwhelm even technically capable learners
+- Explicitly naming the key idea being taught helps learners organise new knowledge into their existing mental model
+- Checking for understanding at intermediate steps is more efficient than waiting for a final question; confusion compounds if uncorrected
+
+**Worked Examples & Code**
+- A minimal, self-contained example isolates the concept being taught from confounding complexity; stripping out everything irrelevant to the point makes the lesson clearer
+- Annotating code at the conceptual level ("this step normalises the features so gradient descent converges more smoothly") is more valuable than line-by-line description of what the syntax does
+- Common mistakes deserve explicit treatment: explaining what goes wrong when a concept is misapplied teaches the boundaries of the idea, not just the centre
+- Showing the same concept in two different representations (mathematical notation and code, or two different coding styles) reinforces understanding and accommodates different learning styles
+
+**Calibrating Depth to the Learner**
+- A practitioner who needs to *use* a technique correctly needs different depth than a researcher who needs to *extend* it; over-explaining implementation details to a practitioner is as unhelpful as under-explaining foundations to a researcher
+- Jargon introduced without definition creates the illusion of understanding; defining terms the first time they appear, even when they seem obvious, avoids confusion later
+- When a learner is ready for more depth, signals include: correct use of the concept in a new context, productive questions about edge cases, and spontaneous generalisation
+- Acknowledging genuine complexity honestly — "this part is actually subtle and here's why" — is more useful than false simplicity that will confuse the learner later
+
+**Feedback & Correction**
+- Correcting a misconception requires more than stating the right answer; explaining *why* the misconception is appealing and where it leads astray makes the correction stick
+- Positive reinforcement for correct reasoning (not just correct answers) shapes better thinking habits
+- Asking the learner to explain a concept back in their own words surfaces misunderstandings that a nodded agreement conceals
+- When a learner is stuck, the most useful intervention is often a hint that narrows the search space rather than a complete solution — preserving the problem-solving experience
+
+**Connecting Concepts**
+- Relating a new concept to one the learner already understands (regularisation as a prior, cross-validation as a generalisation test, attention as a weighted average) accelerates learning by reusing existing structure
+- Knowing where a concept sits in a broader framework — what it assumes, what it generalises, what it's a special case of — gives learners the scaffolding to place new ideas as they encounter them
+- Pointing out when two apparently different ideas are the same thing in disguise (e.g., ridge regression and MAP estimation with a Gaussian prior) builds a richer, more compact mental model
+- Explicitly flagging common confusions between related concepts (precision vs. accuracy, correlation vs. causation, validation set vs. test set) reduces the chance they take root

opendatasci/resources/skills/deep_learning.md

@@ -0,0 +1,205 @@
+# Deep Learning Skill
+
+## Library Stack
+
+**Key documentation references:**
+- JAX fundamentals (jit, grad, vmap, scan): https://docs.jax.dev/en/latest/
+- Flax NNX (modules, state, lifting): https://flax.readthedocs.io/en/latest/nnx_basics.html
+- Optax (optimisers, schedules, losses): https://optax.readthedocs.io/en/latest/
+
+## When to Use Deep Learning
+
+Deep learning is the right choice when:
+- The data has spatial, sequential, or relational structure (images, text, audio, time series, graphs) that classical models cannot capture without extensive manual feature engineering
+- Data volume is large enough that representation learning outperforms hand-crafted features — as a rough heuristic, thousands of samples for simple MLPs, tens of thousands for CNNs and RNNs, hundreds of thousands or more for transformers trained from scratch
+- Transfer learning from a pre-trained checkpoint is available for the domain, substantially reducing the data and compute needed to reach strong performance
+- Tabular data warrants a neural approach (MLP, TabNet, FT-Transformer) after gradient-boosting baselines have been tried and plateaued, or when learned embeddings for high-cardinality categoricals carry signal that encoding schemes miss
+
+Prefer scikit-learn or gradient boosting (LightGBM, CatBoost, XGBoost) over deep learning when data is tabular and moderate in size, when interpretability and iteration speed matter more than squeezing out the last percentage point, or when the data volume is too small to support the capacity of a neural model without severe overfitting.
+
+## scikit-learn MLP
+
+Use `MLPClassifier` / `MLPRegressor` when:
+- The task is classification or regression on tabular features and a shallow network (1–3 hidden layers, hundreds of units) is plausible given data size
+- You need sklearn pipeline compatibility (transformers, cross-validation, grid search) and the overhead of JAX is not justified
+- You want a quick neural-network baseline without leaving the sklearn ecosystem
+
+Monitor training loss via `loss_curve_` after fitting. Use early stopping (`early_stopping=True`) with a validation fraction to avoid overfitting. For anything beyond shallow feedforward networks — convolutional layers, recurrent layers, attention, custom loss functions, fine-grained training control — use the JAX stack instead.
+
+## JAX Fundamentals
+
+JAX programs are built from pure functions transformed by a small set of composable primitives. Understanding these primitives and the constraints they impose is essential before writing any model code.
+
+**Functional purity and side effects**
+- JAX transformations (`jit`, `grad`, `vmap`, `scan`) require pure functions: given the same inputs, the function must return the same outputs with no observable side effects; in-place mutation of arrays, Python-level state changes, and I/O inside transformed functions will silently produce incorrect results or raise errors
+- All randomness flows through explicit PRNG keys (`jax.random.key`); splitting a key into subkeys before each stochastic operation (dropout, initialisation, data augmentation) ensures reproducibility and correct behaviour under `jit` and `vmap`
+- State (model parameters, optimiser state, batch-norm statistics, RNG keys) is passed explicitly as function arguments and returned as outputs rather than mutated in place — this is the central design difference from imperative frameworks
+
+**Core transformations**
+- `jax.jit` compiles a function via XLA for fast execution; the first call traces and compiles, subsequent calls with the same input shapes and dtypes hit the cache — shape-changing inputs trigger recompilation, so avoid variable-length sequences without padding
+- `jax.grad` computes gradients of a scalar-valued function with respect to its first argument (or specified `argnums`); for auxiliary outputs alongside the gradient, use `jax.value_and_grad` with `has_aux=True`
+- `jax.vmap` vectorises a function over a batch dimension, replacing explicit loops with efficient batched operations; use it to write per-example logic and let JAX handle batching
+- `jax.lax.scan` replaces Python for-loops over sequential operations (RNN steps, iterative algorithms) with an XLA-compiled loop that is both faster and memory-efficient through automatic gradient checkpointing
+
+**Array semantics**
+- JAX arrays are immutable; "updates" produce new arrays (e.g. `x.at[i].set(v)` returns a new array rather than modifying `x`)
+- Default dtype promotion in JAX follows its own rules, not NumPy's; float32 is the standard training dtype, and explicit dtype management avoids silent precision loss or promotion to float64
+- JAX's NumPy API (`jax.numpy`) mirrors NumPy closely but not identically — in particular, out-of-bounds indexing clamps rather than raising, and some operations behave differently under `jit` when control flow depends on array values
+
+## Flax NNX: Defining Models
+
+Flax NNX is the module API for defining neural network architectures on JAX. It provides a Pythonic, mutable-object interface that handles the functional-purity requirements of JAX under the hood.
+
+**Module basics**
+- Subclass `nnx.Module` to define layers and models; parameters are declared as `nnx.Param` (or created implicitly by built-in layers like `nnx.Linear`, `nnx.Conv`, `nnx.BatchNorm`) and become part of the module's state
+- Modules are mutable Python objects during construction and outside JIT; inside `nnx.jit`-wrapped functions, Flax NNX manages the functional transformation automatically — you write imperative code and NNX lifts it to pure functions for JAX
+- Use `nnx.Rngs` to manage PRNG keys for initialisation, dropout, and other stochastic layers; pass an `nnx.Rngs` object at module construction and Flax handles key splitting across layers
+
+**Built-in layers**
+- `nnx.Linear` — dense layer; the fundamental building block for MLPs and projection heads
+- `nnx.Conv` — convolution; supports arbitrary dimensionality via `kernel_size` and standard options (strides, padding, dilation, feature groups)
+- `nnx.BatchNorm` — batch normalisation; tracks running statistics via `nnx.BatchStat` and requires `use_running_average` to switch between train and eval modes
+- `nnx.LayerNorm` — layer normalisation; preferred over batch norm for small batches, sequence models, and transformers
+- `nnx.Dropout` — inverted dropout; requires `deterministic=False` during training and an active RNG stream, switches to identity with `deterministic=True` at eval
+- `nnx.Embed` — embedding table; maps integer indices to dense vectors, the entry point for categorical features and token-based inputs
+- `nnx.MultiHeadAttention` — scaled dot-product multi-head attention
+
+**State management**
+- `nnx.state(model, nnx.Param)` extracts all trainable parameters as a nested pytree; `nnx.state(model, nnx.BatchStat)` extracts batch statistics — this separation enables clean update logic where only parameters receive gradients
+- For serialisation, `nnx.state(model)` captures the full state which can be saved and restored
+- When composing models from submodules, parameter namespacing follows the attribute hierarchy automatically
+
+**Train/eval mode**
+- Modules that behave differently during training versus inference (dropout, batch norm) are controlled by flags passed to their `__call__` method (`deterministic`, `use_running_average`) or set globally — always ensure the correct mode is active before each forward pass
+
+## Optax: Optimisation and Schedules
+
+Optax provides the optimiser, learning rate schedule, and gradient transformation pipeline for training JAX models.
+
+**Optimiser selection**
+- `optax.adamw` — Adam with decoupled weight decay; the default starting point for most deep learning tasks; weight decay acts as L2 regularisation without interfering with adaptive moment estimates
+- `optax.adam` — standard Adam; suitable when weight decay is handled separately or not needed
+- `optax.sgd` — SGD with optional momentum and Nesterov acceleration; can outperform Adam on well-tuned image classification and other tasks where the loss landscape is smooth and the learning rate schedule is carefully designed
+- `optax.lamb` — layer-wise adaptive moments; scales to very large batch sizes for distributed training
+- `optax.lion` — evolved optimiser; uses sign-based updates and tends to generalise well with lower memory than Adam
+
+**Learning rate schedules**
+- `optax.warmup_cosine_decay_schedule` — linear warmup followed by cosine decay; the most common schedule for transformer training and a strong default for any architecture
+- `optax.cosine_decay_schedule` — cosine annealing without warmup; suitable when training is long enough that warmup is unnecessary
+- `optax.linear_schedule` — linear interpolation between two values; useful for warmup phases or simple decay
+- `optax.exponential_decay` — step-based exponential decay; commonly used with SGD for image classification
+- `optax.piecewise_constant_schedule` — manual step-function schedule; useful when domain knowledge dictates specific rate changes at known training milestones
+- Schedules are passed as the `learning_rate` argument to the optimiser; they receive the step count and return the current rate
+
+**Gradient transformations**
+- `optax.clip_by_global_norm` — clips gradients to a maximum global norm; essential for training RNNs, transformers, and any architecture prone to gradient explosion; a global norm of 1.0 is a common starting point
+- `optax.chain` — composes multiple gradient transformations sequentially (e.g. clip, then scale by learning rate, then apply Adam); the standard way to build custom optimiser pipelines
+- `optax.apply_every` — accumulates gradients over multiple steps before applying; simulates larger effective batch sizes when memory is constrained
+- `optax.ema` — exponential moving average of parameters; used for maintaining a smoothed copy of weights for evaluation (Polyak averaging)
+
+**Loss functions**
+- `optax.softmax_cross_entropy_with_integer_labels` — classification with integer targets; numerically stable and avoids manual one-hot encoding
+- `optax.softmax_cross_entropy` — classification with one-hot or soft targets
+- `optax.sigmoid_binary_cross_entropy` — binary or multi-label classification
+- `optax.l2_loss`, `optax.huber_loss`, `optax.squared_error` — regression losses with different outlier sensitivity profiles
+
+**Optimiser state management**
+- `optax.inject_hyperparams` wraps an optimiser to make hyperparameters (learning rate, weight decay) accessible and modifiable in the optimiser state — useful for logging the current learning rate or implementing custom schedule logic
+- Optimiser state is a pytree that mirrors the parameter structure; it is initialised with `opt.init(params)` and updated with `opt.update(grads, opt_state, params)`
+
+## Training Loop Design
+
+The training loop ties JAX, Flax, and Optax together. Getting the structure right from the start prevents a class of bugs that are difficult to diagnose later.
+
+**Standard loop structure**
+- A training step function takes the model, optimiser state, a batch of data, and (when needed) an RNG key; it computes the forward pass, loss, and gradients, applies the optimiser update, and returns the updated model, updated optimiser state, and metrics — this function is the natural unit of `jit` compilation
+- Use `jax.value_and_grad` with `has_aux=True` to compute the loss and gradients in a single pass while returning auxiliary outputs (per-example losses, logits, intermediate activations for logging)
+- Apply `nnx.jit` (or `jax.jit`) to the training step function; this compiles it once and reuses the compiled version for every batch — ensure all inputs have static shapes to avoid recompilation
+- An epoch loops over batches from the data loader, calls the compiled training step, and accumulates metrics; an outer loop iterates over epochs
+
+**Data loading**
+- JAX does not include a data loading pipeline; data preparation and batching are handled outside JAX using NumPy, Pandas, or any standard Python tooling
+- Convert data to JAX arrays (`jnp.array`) at the batch level, not the dataset level — loading entire large datasets into device memory is often unnecessary and wasteful
+- For datasets that fit in memory, a simple pattern is: shuffle indices at the start of each epoch, slice into batches, and convert each batch to `jnp.array` as it is consumed
+- Ensure consistent batch sizes (pad the last batch if necessary) to avoid triggering JIT recompilation on the final batch of each epoch
+
+**Logging and monitoring**
+- Track training loss, validation loss, and the primary evaluation metric per epoch at minimum; per-batch training loss reveals learning dynamics (oscillation, divergence, plateaus) that per-epoch averages can conceal
+- When using a learning rate schedule, log the current learning rate alongside loss to diagnose whether decay is too aggressive or too slow
+- A sudden spike in training loss often indicates a learning rate that is too high, a data loading bug (corrupted batch), or numerical instability — investigate immediately rather than hoping the model recovers
+
+**Reproducibility**
+- Fix all random seeds: Python's `random.seed`, NumPy's `np.random.seed`, and the initial JAX PRNG key (`jax.random.key(seed)`)
+- Deterministic data shuffling (seeded permutation of indices) ensures the same batch order across runs
+- JAX's XLA compilation is deterministic given the same inputs and platform, but results may differ across hardware (CPU vs GPU) due to floating-point non-associativity in parallel reductions
+
+## Architecture Selection
+
+Architecture choice should be driven by the data modality, the nature of the prediction task, and the available data volume — not by a default preference for a familiar architecture.
+
+**Tabular data**
+- A 2–4 layer MLP with ReLU (or GELU) activations, batch normalisation or layer normalisation, and dropout is the standard neural baseline for tabular data; hidden dimensions between 64 and 512 depending on feature count and data volume
+- Learned embeddings (`nnx.Embed`) for categorical features, concatenated with normalised continuous features, often outperform one-hot or ordinal encoding for high-cardinality columns
+- For tabular tasks, gradient-boosted trees are usually the stronger baseline; the neural model's value is often as an ensemble component providing prediction diversity rather than as a standalone winner
+
+**Sequences and time series**
+- For short-to-medium sequences, 1D convolutions (`nnx.Conv` with appropriate kernel sizes) with residual connections capture local patterns efficiently and are faster to train than recurrent architectures
+- GRU and LSTM cells process sequences step-by-step and naturally handle variable-length inputs; implement the recurrence with `jax.lax.scan` for efficient compiled execution rather than Python for-loops
+- For long sequences where global context matters, self-attention (transformer) architectures are more expressive but scale quadratically with sequence length; for very long sequences, consider windowed or linear attention variants
+
+**Images**
+- Convolutional architectures (ResNet-style blocks using `nnx.Conv` + batch norm + residual connections) are the standard entry point; depth and width scale with data volume and image resolution
+- When building from scratch with limited data, prefer shallower architectures with aggressive data augmentation over deep networks that overfit
+- For transfer learning scenarios, load pre-trained weights into a Flax model and fine-tune the head (or the full model with a lower learning rate for pre-trained layers)
+
+**Attention and transformers**
+- The transformer block (multi-head self-attention + feedforward + layer norm + residual connections) is the dominant architecture for sequence modelling tasks with sufficient data
+- Pre-norm (layer norm before attention and feedforward) tends to train more stably than post-norm, especially for deeper models
+- Positional encoding (sinusoidal, learned, or rotary) is essential — without it, the attention mechanism is permutation-invariant and cannot distinguish token order
+
+## Regularisation and Overfitting
+
+The gap between training and validation performance is the primary diagnostic for overfitting, and the right regularisation strategy depends on the architecture, data volume, and the specific manifestation of the overfit.
+
+**Core regularisation techniques**
+- Dropout (`nnx.Dropout`) randomly zeros activations during training, forcing the network to learn redundant representations; typical rates range from 0.1 to 0.5, with higher rates for larger models or smaller datasets; remember to disable dropout at evaluation time (`deterministic=True`)
+- Weight decay (via `optax.adamw` or explicit L2 penalty) penalises large weights and acts as a smoothness prior; values between 1e-4 and 1e-1 are typical, with larger values for models that overfit aggressively
+- Early stopping — monitoring validation loss and stopping training when it stops improving — is the simplest and most reliable regulariser; patience (number of epochs without improvement before stopping) should be large enough to survive temporary plateaus
+- Batch normalisation and layer normalisation have an implicit regularising effect through noise injection (batch statistics vary per mini-batch); this effect diminishes with larger batch sizes
+- Label smoothing (replacing hard 0/1 targets with soft targets like 0.05/0.95) prevents the model from becoming overconfident and improves calibration, particularly for classification with noisy labels
+
+**Data augmentation**
+- For images: random crops, horizontal flips, colour jitter, cutout, and mixup are the standard augmentation vocabulary; implement as NumPy/JAX transformations applied per batch during data loading
+- For tabular data: noise injection (Gaussian noise on continuous features), feature masking (randomly zeroing features), and mixup between training examples can regularise when data is scarce
+- For sequences: random masking, token dropping, time warping, and window cropping depending on the modality
+
+**Diagnosing capacity problems**
+- When training loss is high and does not decrease: the model lacks capacity (increase width or depth), the learning rate is too low, or the data preprocessing has a bug — check data first
+- When training loss is low but validation loss is high: overfitting — apply or increase regularisation, add data augmentation, or reduce model capacity
+- When both losses plateau at a mediocre level: the feature representation may be insufficient, the architecture may be mismatched to the data structure, or the learning rate schedule may need adjustment
+- Learning curves (validation performance as a function of training set size) distinguish data-limited regimes from model-limited regimes and guide whether to invest in more data or more architecture
+
+## Hyperparameter Tuning
+
+**High-leverage hyperparameters**
+- Learning rate is almost always the single most impactful hyperparameter; a log-uniform search between 1e-5 and 1e-2 is a reasonable starting range for Adam-family optimisers, wider for SGD
+- Batch size affects both optimisation dynamics (smaller batches add noise that can help generalisation) and computational efficiency (larger batches utilise hardware better); typical values are 32, 64, 128, 256 — the interaction between batch size and learning rate (linear scaling rule) deserves attention
+- Weight decay, dropout rate, and the number of layers/units per layer are the next tier; search these after the learning rate is approximately right
+- Learning rate schedule parameters (warmup steps, decay rate, minimum learning rate) are often set by convention rather than searched: warmup over 5–10% of total training steps, decay to 1e-6 or 1e-7
+
+**Search strategy**
+- Start with a small number of epochs (1–3) to do a coarse learning rate sweep; this surfaces obviously bad configurations without wasting compute
+- Use `optuna` for Bayesian hyperparameter search over continuous and categorical spaces; it prunes unpromising trials early via median stopping or Hyperband and supports parallel trials
+- Run the search on a representative subsample of the data to cut per-trial cost; validate the winning configuration on the full dataset before committing
+- Fix random seeds across trials so differences in score reflect hyperparameter choices, not initialisation variance
+
+## Evaluation
+
+The same evaluation principles from classical ML apply: appropriate split strategies, metric selection aligned with the real objective, and subgroup analysis to surface hidden failure modes.
+
+**Neural-specific evaluation considerations**
+- Ensure the model is in evaluation mode (dropout disabled, batch norm using running statistics) before computing any validation or test metrics — forgetting this is a common source of inconsistent results
+- For classification, predicted probabilities from neural networks are often poorly calibrated; temperature scaling or Platt scaling on a held-out calibration set improves probability estimates when they will be used as actual probabilities rather than just rankings
+- Averaging predictions across multiple random seeds (same architecture, different initialisations) reduces variance and gives a more stable performance estimate; the spread across seeds characterises how sensitive the result is to initialisation
+- Test-time augmentation (averaging predictions over augmented copies of each test input) provides small but consistent gains for image tasks and sometimes for other modalities
+- When ensembling neural models with classical models (gradient boosting, linear), the diversity of the neural model's errors relative to the classical model's errors is what drives ensemble gains — even a weaker neural model can improve an ensemble if its mistakes are uncorrelated