Bio Machine Learning Survival Analysis

Version Compatibility

Reference examples tested with: scikit-survival 0.22+, lifelines 0.30+, numpy 1.26+, pandas 2.2+ (pycox 0.3+ for deep models).

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures

scikit-survival requires the target y to be a structured array with a boolean event field and a float time field; its IPCW metrics need the training y first. lifelines concordance_index expects higher-score = longer-survival (negate the partial hazard). If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Predictive Survival Modeling

"Build a validated risk model from time-to-event data" -> Fit a penalized Cox or ensemble survival model, then evaluate with censoring-robust discrimination AND calibration, not the C-index alone.

• Penalized Cox / RSF / boosting: sksurv.linear_model.CoxnetSurvivalAnalysis, sksurv.ensemble.RandomSurvivalForest
• Evaluation: concordance_index_ipcw, cumulative_dynamic_auc, integrated_brier_score
• Deep survival (large n): pycox DeepSurv / DeepHit

The Single Most Important Modern Insight -- C-Index Only Is the Cardinal Sin

The C-index is necessary but radically insufficient, for three reasons most papers miss. It is censoring-distribution-dependent: Harrell's C is biased upward under heavy censoring and gives different values on cohorts that differ only in follow-up -- report Uno's IPCW C with an explicit truncation tau instead (Uno 2011). It is invariant to any monotone transform of the risk score, so a model can have an excellent C and be wildly miscalibrated and clinically harmful -- C measures ranking, not the correctness of the predicted probabilities. And it is insensitive: adding a genuinely useful marker barely moves it, which is exactly why reclassification metrics (NRI/IDI) were invented. Decision-grade evaluation is Uno's C(tau) + time-dependent AUC(t) + integrated Brier vs a Kaplan-Meier baseline + calibration curves, all on honestly held-out or external data.

ML-vs-Confirmatory Scope Boundary

Two survival cultures, two skills; mixing them is the most common authoring error. Resolve every overlap by one question: is the goal to estimate and test a treatment effect in a (pre-specified) study, or to build and validate a risk-prediction model?

PH concepts, censoring definitions, and the Cox partial likelihood are foundational to both -- stated in clinical-biostatistics and referenced here. This skill's value begins at "I want a validated predictor."

Model Taxonomy

Load-bearing empirical fact: on typical clinical-omics n, penalized Cox and RSF are very hard to beat; deep survival models usually only pull ahead at large n and/or with genuine non-PH or competing-risk structure. Start with elastic-net Cox and RSF baselines; escalate to deep models only if they demonstrably beat those on a held-out set.

Decision Tree by Scenario

Fitting Predictive Survival Models

Goal: Fit a penalized Cox and an RSF baseline with the correct target format.

Approach: Build the structured y (boolean event, float time), fit coxnet with an elastic-net mix and a baseline model for survival functions, and an RSF for nonlinear structure.

from sksurv.util import Surv
from sksurv.linear_model import CoxnetSurvivalAnalysis
from sksurv.ensemble import RandomSurvivalForest

# event MUST be bool, time float. Field order in from_arrays is (event, time).
y = Surv.from_arrays(event=df['status'].astype(bool), time=df['time'].astype(float))

# Elastic-net Cox: l1_ratio in (0,1]; fit_baseline_model=True enables predict_survival_function.
coxnet = CoxnetSurvivalAnalysis(l1_ratio=0.9, alpha_min_ratio=0.01, fit_baseline_model=True)
coxnet.fit(X_train, y_train)

rsf = RandomSurvivalForest(n_estimators=500, min_samples_leaf=15, max_features='sqrt', n_jobs=-1)
rsf.fit(X_train, y_train)
risk = coxnet.predict(X_test)              # a risk score (higher = higher risk), NOT a probability

Prediction-Grade Evaluation

Goal: Report discrimination AND calibration on out-of-sample data, not the C-index alone.

Approach: Use Uno's IPCW C (truncated at tau), time-dependent AUC over a horizon grid, integrated Brier vs the KM baseline, and a calibration check at clinical horizons. IPCW metrics take the training y first to estimate the censoring distribution.

import numpy as np
from sksurv.metrics import concordance_index_ipcw, cumulative_dynamic_auc, integrated_brier_score

c_uno = concordance_index_ipcw(y_train, y_test, risk, tau=t_horizon)[0]      # censoring-robust

times = np.percentile(y_test['time'][y_test['event']], np.linspace(10, 80, 15))   # inside follow-up
auc_t, mean_auc = cumulative_dynamic_auc(y_train, y_test, risk, times)

surv_fns = coxnet.predict_survival_function(X_test)              # needs fit_baseline_model=True
surv_prob = np.vstack([[fn(t) for t in times] for fn in surv_fns])
ibs = integrated_brier_score(y_train, y_test, surv_prob, times)  # compare against the KM-only IBS
print(f"Uno C: {c_uno:.3f}  mean AUC(t): {mean_auc:.3f}  IBS: {ibs:.3f}")

Calibration (the most decision-relevant, most-ignored axis): at a clinical horizon, plot predicted P(event by t) against the observed event probability (KM within risk groups, or a smooth calibration curve), and summarize with ICI/E50/E90 (Austin 2020) -- never Hosmer-Lemeshow. Calibration is what usually breaks on external validation even when discrimination is preserved.

Competing Risks

A competing event precludes the event of interest (death from another cause precludes cause-specific death). Treating competing events as ordinary censoring is wrong: Kaplan-Meier overestimates cumulative incidence (1-KM >= the cumulative incidence function), so report the CIF (Aalen-Johansen), not 1-KM. Two hazards answer two questions: the cause-specific hazard (Cox censoring competing events) is the etiologic rate; the Fine-Gray subdistribution hazard maps to the CIF and is the prognostic/absolute-risk target -- but a Fine-Gray sHR is NOT the cause-specific HR and not an effect on the event rate (a covariate can raise a CIF purely by lowering the competing hazard). Since this skill is the prediction regime, the CIF is usually the target; report both models for a complete picture. DeepHit is purpose-built for competing risks at large n; evaluate with competing-risks-aware concordance (Wolbers 2009) and CIF-based Brier.

Censoring, Immortal Time, and Landmarking

• Non-informative censoring is assumed by the Cox likelihood, KM, and all IPCW metrics: censored subjects must be representative of those still at risk. If sicker patients drop out (informative censoring), survival is overestimated and IPCW does not fix it with a misspecified censoring model. Administrative censoring is the benign case; informative censoring is generally untestable and needs sensitivity analysis. Truncate IPCW metrics at tau because tail weights explode.
• Immortal time bias -- defining a group by a post-baseline event ("patients who received treatment") manufactures a survival advantage from guaranteed event-free time (Levesque 2010). Endemic in EHR/omics; fix with time-varying exposure, landmarking, or target-trial emulation.
• Landmarking (van Houwelingen 2007) -- internal time-varying covariates (a biomarker that changes with disease) cannot be plugged into the future; pick a landmark time, restrict to those still event-free, use covariate values as of the landmark, and predict forward. It is the robust route to honest dynamic prediction.