Censoring-aware evaluation, calibration, and model selection

Verified for scikit-survival 0.28.0 on 2026-10-01. API statements below use official scikit-survival documentation; interpretation is anchored to the cited primary methodological literature.

Keep the targets distinct

• Discrimination: can the score order subjects by event risk? Harrell C, Uno C, and cumulative/dynamic AUC.
• Probability prediction error: how close are predicted survival probabilities to observed event-free status after censoring adjustment? Brier score and IBS.
• Calibration: do predicted probabilities agree with observed probabilities at a specified horizon and population? Requires horizon-specific assessment; Brier score is not a pure calibration measure.
• Cause-specific cumulative incidence: what is the absolute probability of a particular competing cause by time \(t\)? Standard survival metrics are not automatically cause-specific CIF metrics.
• Clinical/decision utility: do decisions based on predictions improve outcomes under defined consequences? None of the metrics above establishes this.

Do not label one aggregate number "model accuracy" without its estimand, horizon, censoring estimator, and input type.

Prediction contracts

Higher-is-riskier scalar

Shape (n_test,). Used by:

• concordance_index_censored
• concordance_index_ipcw
• cumulative_dynamic_auc (same score at each requested time)

Typical source:

risk = estimator.predict(X_test)

Confirm direction. Cox and ranking-only SVM predictions are higher-is-riskier. Predicted survival/log-time outputs are the opposite direction and must not be silently treated as risk.

Time-dependent risk

Shape (n_test, n_times). Accepted by cumulative_dynamic_auc, where column j is risk at times[j]. For a random survival forest:

functions = estimator.predict_cumulative_hazard_function(X_test)
risk_by_time = np.vstack([fn(times) for fn in functions])

Cumulative hazard is risk-oriented. Untransformed survival probabilities have the opposite direction and are not valid risk inputs to cumulative_dynamic_auc. Numeric shape validation does not detect that semantic error; check output direction explicitly before using a transformed probability or hazard score.

Survival probability

Shape (n_test, n_times), values in [0, 1], non-increasing across columns. Used by:

• brier_score
• integrated_brier_score

functions = estimator.predict_survival_function(X_test)
survival_probability = np.vstack([fn(times) for fn in functions])

Metric functions expect the numeric matrix, not a list of unevaluated StepFunction objects and not a 1D risk score.

Harrell concordance

from sksurv.metrics import concordance_index_censored

harrell_c = concordance_index_censored(
    y_test["event"],
    y_test["time"],
    risk,
)[0]

Harrell C is the fraction of comparable pairs whose risk ordering is concordant, with handling for tied risk and event times. It measures rank discrimination over the observed follow-up mix.

Important limitations:

• It does not assess probability calibration.
• It does not focus on a prespecified horizon.
• The set of comparable pairs changes under censoring.
• Uno et al. showed the conventional estimator's limiting value can depend on the censoring distribution.

There is no universal censoring-percentage cutoff at which Harrell C suddenly becomes invalid. Report censoring, compare sensitivity to IPCW concordance, and justify the estimand.

Uno IPCW concordance

from sksurv.metrics import concordance_index_ipcw

uno_c = concordance_index_ipcw(
    y_train,
    y_test,
    risk,
    tau=tau,
)[0]

survival_train estimates the censoring distribution. Never pass pooled train+test outcomes or y_test as the training distribution.

tau truncates the concordance target. Choose it before seeing model performance and where the estimated training censoring survival is positive. In 0.28.0 only test event times strictly below tau enter its censoring-weight lookup; later records remain potential comparison partners with zero event weight. A valid tau can therefore avoid a late-event support error for Uno C. This does not make the same test outcomes valid for AUC or Brier APIs.

The implementation uses Kaplan-Meier censoring weights and assumes censoring is random/independent of features. If censoring depends on covariates, this marginal weight model may be inadequate. IPCW is not a generic correction for informative censoring.

Cumulative/dynamic AUC

from sksurv.metrics import cumulative_dynamic_auc

auc, mean_auc = cumulative_dynamic_auc(
    y_train,
    y_test,
    risk,   # (n_test,) or (n_test, n_times)
    times,
)

At each \(t\), cumulative cases have an observed event by \(t\), while dynamic controls remain event-free after \(t\). IPCW handles right censoring.

Requirements:

• times is one-dimensional, unique, and strictly increasing;
• every value lies within test follow-up;
• training follow-up supports test outcomes and the grid;
• the training censoring survival is positive over the grid and at every test event time used by the implementation (AUC weights all test events before applying horizons);
• each horizon has at least one observed case and a later control;
• risk has shape (n_test,) or (n_test, n_times);
• higher values mean higher event risk.

The returned mean_auc is not the arithmetic mean. It integrates \(\widehat{AUC}(t)\) over the time range, weighted by the estimated survival function.

The cumulative/dynamic definition is one time-dependent ROC estimand. State it; other incident/dynamic or competing-risk definitions answer different questions.

Brier score

from sksurv.metrics import brier_score, integrated_brier_score

returned_times, brier = brier_score(
    y_train,
    y_test,
    survival_probability,
    times,
)
ibs = integrated_brier_score(
    y_train,
    y_test,
    survival_probability,
    times,
)

The time-dependent Brier score is an IPC-weighted squared error between predicted survival probability and event-free status at \(t\). Lower is better.

Requirements:

• predictions are survival probabilities, not risk scores;
• prediction shape is (n_test, n_times);
• time and training-support rules match the IPCW setting;
• the marginal Kaplan-Meier censoring estimator's independence assumption is plausible.

IBS in 0.28.0 equals np.trapezoid(brier, times) / (times[-1] - times[0]). It is the normalized trapezoidal integral over the supplied interval. It depends on the chosen interval; IBS values from different time ranges are not directly comparable.

Compare against useful reference predictions such as a training-derived Kaplan-Meier survival curve. Do not estimate the reference curve on test outcomes.

Calibration

Brier score responds to both discrimination and calibration. A good Brier score does not prove that predicted 20% risk corresponds to 20% observed risk in every horizon or subgroup.

For a prespecified horizon:

1. fit and tune the model without the calibration-evaluation rows;
2. obtain event probability 1 - S(t | x) on independent validation rows;
3. compare predictions with a censoring-aware observed probability estimate;
4. inspect calibration-in-the-large, slope/shape, uncertainty, and sample support;
5. repeat only for prespecified horizons/subgroups or account for multiplicity.

scikit-survival 0.28 does not expose a dedicated calibration-curve API. Do not use sklearn.calibration.calibration_curve naively on censored binary labels. If an external method is used, document its censoring assumptions and train/validation separation.

Any recalibration layer is another learned model. Fit it on a calibration split or inner resampling, then assess on independent data.

Safe time-grid construction

A conservative grid for jointly computing AUC, Uno C, and Brier metrics:

import numpy as np
from sksurv.nonparametric import CensoringDistributionEstimator

test_time = y_test["time"]
train_time = y_train["time"]

lower = np.quantile(test_time, 0.10)
upper = min(
    np.quantile(test_time, 0.80),
    np.nextafter(train_time.max(), -np.inf),
)
times = np.linspace(lower, upper, 50)

if not (test_time.min() < times[0] < times[-1] < test_time.max()):
    raise ValueError("grid is outside test follow-up")
if not test_time.max() < train_time.max():
    raise ValueError("test follow-up exceeds training support")

censoring = CensoringDistributionEstimator().fit(y_train)
if np.any(censoring.predict_proba(times) <= 0):
    raise ValueError("training censoring survival reaches zero on grid")
censoring.predict_ipcw(y_test)  # AUC needs positive support at all test event times
if not np.any(y_test["event"] & (test_time <= times[0])):
    raise ValueError("AUC has no observed cases at the first horizon")

This strict maximum-follow-up check is a helper policy, not an exact universal API restriction. Native Brier metrics also query censoring survival at all test times; zero weights may be suppressed by the implementation, so a successful call alone is not evidence of valid support. Never silently drop long-followed test subjects to obtain a score: define administrative truncation and recode late events as censored under a prespecified estimand if that is scientifically intended.

Quantiles are an operational example, not a scientific default. Prefer prespecified meaningful horizons, then verify fold support. Do not select a grid because it maximizes a metric.

The bundled evaluator rejects unsupported grids and shape/type confusion:

python skills/scikit-survival/scripts/evaluate_survival_metrics.py \
  --input predictions.npz \
  --output metrics-summary.json

Its NPZ contract is:

• train_event, train_time
• test_event, test_time
• times
• risk
• optional survival

Archives are loaded with allow_pickle=False, with compressed-file and total expanded-member bounds. The evaluator requires at least two horizons for the combined workflow; native AUC and Brier permit a single horizon, whereas IBS does not.

Scorer wrappers and model selection

Survival estimators' default .score() is Harrell concordance and handles their own prediction direction. Discrimination wrappers below call raw .predict(); they do not reverse time-oriented IPCRidge, SVM-regression, or AFT-boosting output. For other targets, scikit-survival provides estimator wrappers:

• as_concordance_index_ipcw_scorer(estimator, tau=None, tied_tol=...)
• as_cumulative_dynamic_auc_scorer(estimator, times, tied_tol=...)
• as_integrated_brier_score_scorer(estimator, times)

Correct pattern:

from sklearn.model_selection import GridSearchCV
from sksurv.metrics import as_integrated_brier_score_scorer

wrapped = as_integrated_brier_score_scorer(
    estimator,
    times=inner_times,
)
search = GridSearchCV(
    wrapped,
    {"estimator__model__max_depth": [1, 2, 4]},
    cv=inner_splits,
)
search.fit(X_outer_train, y_outer_train)

The wrapper:

• is the estimator supplied to GridSearchCV;
• stores its fitted estimator in estimator_;
• exposes nested parameters under estimator__...;
• fits metric state, including training survival information, from each fit fold;
• negates IBS so larger wrapper score remains better.

Do not write scoring=as_integrated_brier_score_scorer(times); the constructor requires an estimator.

Nested CV protocol

For tuned performance:

for each outer split:
    outer_train, outer_valid
    for each inner split within outer_train:
        fit preprocessing + model + censoring-dependent scorer state
        select hyperparameters
    refit selected pipeline on all outer_train
    evaluate once on outer_valid using outer_train censoring distribution
aggregate outer scores and uncertainty

Every fold needs its own valid tau/time grid. A single global grid derived from all outcome times leaks outer-validation support information and may fail when an outer training fold has shorter follow-up.

After protocol evaluation, tune on all development data and evaluate once on an untouched final test set. Do not report the best inner-CV score as generalization performance.

Competing risks

If events have causes:

• all-event risk combines causes and is not cause-specific discrimination;
• treating other causes as censored defines a cause-specific hazard analysis;
• a cause-specific CIF is an absolute probability accounting for all causes;
• standard Brier/AUC functions here are documented for right-censored single-event outcomes, not a complete competing-risk prediction evaluation.

State event coding and use methods designed for the cause-specific estimand. See competing-risks.md.

Reporting checklist

• split and nesting protocol;
• event/censoring counts per evaluation set;
• metric definition and score direction;
• tau and exact time grid/range;
• source of censoring weights;
• prediction shape and whether values are risks or survival probabilities;
• uncertainty across independent outer folds or bootstrap resamples;
• calibration assessment separate from discrimination;
• handling of competing causes;
• no claim of clinical utility from metrics alone.

Primary methodological literature

• Harrell FE Jr, Califf RM, Pryor DB, Lee KL, Rosati RA. "Evaluating the yield of medical tests." JAMA (1982). doi:10.1001/jama.1982.03320430047030
• Uno H, Cai T, Pencina MJ, D'Agostino RB, Wei LJ. "On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data." Statistics in Medicine (2011). doi:10.1002/sim.4154
• Heagerty PJ, Lumley T, Pepe MS. "Time-dependent ROC curves for censored survival data and a diagnostic marker." Biometrics (2000). doi:10.1111/j.0006-341X.2000.00337.x
• Heagerty PJ, Zheng Y. "Survival model predictive accuracy and ROC curves." Biometrics (2005). doi:10.1111/j.0006-341X.2005.030814.x
• Graf E, Schmoor C, Sauerbrei W, Schumacher M. "Assessment and comparison of prognostic classification schemes for survival data." Statistics in Medicine (1999). doi:10.1002/(SICI)1097-0258(19990915/30)18:17/18%3C2529::AID-SIM274%3E3.0.CO;2-5
• Gerds TA, Schumacher M. "Consistent estimation of the expected Brier score in general survival models with right-censored event times." Biometrical Journal (2006). doi:10.1002/bimj.200610301

Official API sources

Checked 2026-10-01:

• Evaluation user guide
• Metrics API index
• IPCW concordance API
• Cumulative/dynamic AUC API
• Brier score API
• Integrated Brier score API
• IPCW scorer wrapper API