Machine Learning

Machine learning — classic/tabular models, done without lying to yourself

Tabular ML is easy to run and easy to fool yourself with. The deliverable is never "the notebook printed 0.99" — it is an honest estimate of how the model behaves on data it has never seen: a Pipeline that fits every transform on train only, a metric that survives class imbalance, and a DummyClassifier baseline it beats. A number you can't reproduce on a sacred test set you touched exactly once isn't a result — it's a leak you haven't found yet.

Is this the right skill? (decide first)

This skill starts at a clean, validated table (rows × features + a target) and ends at a fitted, honestly-scored model with a test-set number and a baseline it beats. Cleaning is upstream — consume the validated frame data-cleaning produced; don't re-teach it here.

Version reality (verify at author time — this line moves monthly)

Verified 2026-07: scikit-learn current major ~1.9 (1.9.0 shipped 2026-06-02, Python 3.11–3.14) — do NOT pin from memory; check the current stable at scikit-learn.org, the 1.x line ships every few months. GBDTs: XGBoost 3.x and LightGBM 4.x (xgboost 3.3, lightgbm 4.6 current), plus sklearn's own HistGradientBoostingClassifier/...Regressor — a fast native GBDT that eats NaN and (with categorical_features="from_dtype") categoricals with no preprocessing. Pin what you ship (python owns the environment and the pinning); state versions as "~X (verify)", never as frozen fact.

The one rule everything else serves: fit on train only

Every preprocessing step — imputation, scaling, encoding, feature selection, target encoding, resampling — learns parameters from data. Learn them from rows the model is later scored on and the score inflates while production underperforms: that is leakage, the #1 way tabular ML lies. The whole apparatus below — Pipeline, ColumnTransformer, CV, the untouched test set — exists to make "fit on train only" automatic instead of something you remember to do by hand (you won't).

scikit-learn: estimators, Pipeline, ColumnTransformer

Every model is an estimator with the same contract: fit(X, y), then predict(X) / predict_proba(X) (classifiers) / score(X, y). Transformers add transform(X) / fit_transform(X, y). A Pipeline chains transformers + a final estimator into one estimator — so fit fits every step on train, and predict/CV transforms test data with parameters learned on train. That is the leakage guard.

A ColumnTransformer routes different columns down different transformer branches (scale the numerics, encode the categoricals) and stitches the result back together — all still inside the pipeline.

from sklearn.compose import ColumnTransformer, make_column_selector as mcs
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

numeric = Pipeline([("impute", SimpleImputer(strategy="median")),
                    ("scale",  StandardScaler())])           # scaling matters for LINEAR/SVM/KNN
categoric = Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))])  # unseen category -> all-zeros, no crash

pre = ColumnTransformer([
    ("num", numeric,   mcs(dtype_include="number")),
    ("cat", categoric, mcs(dtype_include=["object", "category"])),
], remainder="drop")

model = Pipeline([("pre", pre), ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))])
model.fit(X_train, y_train)          # imputers/scaler/encoder ALL fit on X_train only
model.predict_proba(X_test)          # X_test transformed with train-learned params — no leak

Two load-bearing details: OneHotEncoder(handle_unknown="ignore") so a category unseen in train doesn't crash prediction, and remainder="drop" so unrouted columns don't silently leak through raw; model.set_output(transform="pandas") keeps named-column DataFrames through the pipeline.

Trees skip most of this — scaling is pointless for tree models, and HistGradientBoostingClassifier ingests NaN and categoricals natively, so its pipeline is often just the estimator. Preprocess for the model that needs it, not as a ritual. More patterns (make_column_transformer, FunctionTransformer) → references/pipelines-and-cv.md.

GBDTs are your default on tabular data

Gradient-boosted decision trees are the correct first (and usually last) model for tabular problems. This isn't taste: Grinsztajn, Oyallon & Varoquaux (2022), "Why do tree-based models still outperform deep learning on tabular data?" (arXiv:2207.08815) benchmarked across 45 datasets and found GBDTs beat tuned neural nets on medium-sized tabular data, tracing it to three inductive biases NNs lack: robustness to uninformative features, not being rotationally invariant (so they exploit the meaning of individual columns), and ease of learning irregular / non-smooth target functions. Start with a GBDT; reach for deep-learning on tabular only with a specific reason.

All three expose the sklearn estimator API, so they drop into the pipeline and CV below unchanged. Don't agonize over XGBoost-vs-LightGBM before you have a baseline and a leak-free CV — split discipline dwarfs the library choice. Tuning knobs (learning_rate, num_leaves/max_depth, early stopping, monotonic_cst) and importance/SHAP → references/gbdt-and-tuning.md.

Leakage-safe feature engineering

Feature engineering is where leakage sneaks back in after the pipeline "protected" you. Rules:

• Any transform that learns from data goes INSIDE the pipeline, so CV re-fits it per fold. A scaler fit on the whole dataset, a SelectKBest run before splitting, an imputer using the global mean — each leaks test statistics into train. The common_pitfalls canonical example: SelectKBest(k=25).fit_transform(X, y) before train_test_split produces a beautiful, meaningless score.
• Target/mean encoding of high-cardinality categoricals must cross-fit. sklearn's preprocessing.TargetEncoder does this: its fit_transform(X, y) uses an internal cross-fitting scheme so each row is encoded from other folds' targets — fit(X, y).transform(X) deliberately differs and would leak. Use fit_transform on train (inside the pipeline); never hand-roll a group-mean encoder.
• No target-derived features. A column computed from the label (or a near-proxy: "was_refunded" when predicting "will_refund") is leakage wearing a feature's clothes. If a feature is impossibly predictive, suspect it.
• Respect time. With any temporal structure, a feature may use only information available at prediction time — no future aggregates, no lifetime values that include post-cutoff rows. Split by time (TimeSeriesSplit), not randomly.