Skill Tracer
Skills › Scientific Agent Skills › scikit-survival › data-handling.md

data-handling.md

290 lines · skills/scikit-survival/references/data-handling.md at 5f2bc03

Open on GitHub

Most marks come from matching words in the file, so treat them as pointers.

1# Data handling and leakage-safe preprocessing
2
3Verified for scikit-survival 0.28.0 on 2026-10-01.
4

Data handling and leakage-safe preprocessing

Verified for scikit-survival 0.28.0 on 2026-10-01.

Raw lines 1 to 4
5## Standard right-censored outcome
6
7scikit-survival estimators expect a one-dimensional NumPy structured array with
8exactly two fields:
9
101. a boolean event indicator (`True`=event observed, `False`=right-censored);
112. a floating-point observed time (event or censoring time).
13Field names are configurable, but field order and meaning are fixed.
15```python
16from sksurv.util import Surv
18y = Surv.from_arrays(
19 event=[True, False, True],
20 time=[2.5, 4.0, 7.25],
21 name_event="event",
22 name_time="time",
24assert y.dtype.names == ("event", "time")
25```
27`Surv.from_arrays()` accepts boolean or strict 0/1 event values.
28`Surv.from_dataframe(event, time, data)` accepts pandas and, in 0.28, Polars
29DataFrames:
31```python
32y = Surv.from_dataframe("event", "time", frame)
33```
35Do not use `astype(bool)` on unvalidated strings: `"False"` is a non-empty string
36and therefore converts to `True`. Validate accepted values explicitly.

Standard right-censored outcome

scikit-survival estimators expect a one-dimensional NumPy structured array with exactly two fields:

  1. a boolean event indicator (True=event observed, False=right-censored);
  2. a floating-point observed time (event or censoring time).

Field names are configurable, but field order and meaning are fixed.

from sksurv.util import Surv

y = Surv.from_arrays(
    event=[True, False, True],
    time=[2.5, 4.0, 7.25],
    name_event="event",
    name_time="time",
)
assert y.dtype.names == ("event", "time")

Surv.from_arrays() accepts boolean or strict 0/1 event values. Surv.from_dataframe(event, time, data) accepts pandas and, in 0.28, Polars DataFrames:

y = Surv.from_dataframe("event", "time", frame)

Do not use astype(bool) on unvalidated strings: "False" is a non-empty string and therefore converts to True. Validate accepted values explicitly.

Raw lines 5 to 37
38## Competing-risk outcome is different
40`Surv` is not the input contract for nonparametric competing-risk cumulative
41incidence. Use two arrays:
43```python
44# 0 = right-censored; 1..K = mutually exclusive causes
45event_code = frame["status"].to_numpy(dtype=int)
46observed_time = frame["time"].to_numpy(dtype=float)
47```
49Validate integer cause codes before the integer cast: `.to_numpy(dtype=int)` can
50silently truncate fractional labels. Positive cause codes must be understood before modeling. Do not collapse them to
51boolean until the estimand explicitly requires all-cause event status or a
52cause-specific hazard outcome. See `competing-risks.md`.

Competing-risk outcome is different

Surv is not the input contract for nonparametric competing-risk cumulative incidence. Use two arrays:

# 0 = right-censored; 1..K = mutually exclusive causes
event_code = frame["status"].to_numpy(dtype=int)
observed_time = frame["time"].to_numpy(dtype=float)

Validate integer cause codes before the integer cast: .to_numpy(dtype=int) can silently truncate fractional labels. Positive cause codes must be understood before modeling. Do not collapse them to boolean until the estimand explicitly requires all-cause event status or a cause-specific hazard outcome. See competing-risks.md.

Raw lines 38 to 53
54## Minimum validation
56Before splitting:
58- event and time lengths match feature rows;
59- standard event values are boolean/0/1;
60- competing-risk codes are non-negative integers and 0 means censoring;
61- time is numeric, finite, and strictly positive;
62- outcomes are not included among predictors;
63- feature names are unique and schema roles are explicit;
64- repeated entities, temporal ordering, or sites are identified for the split;
65- every planned training/CV fold contains events and enough comparable pairs;
66 event-stratified CV in the bundled trainer also requires censored rows;
67- missingness is described, but imputation is not yet fitted.
69The bundled validator checks outcome coding, positive finite times, feature exclusion,
70and the local CSV schema. Group independence, fold support, temporal ordering and
71missingness interpretation require the study-specific review above. Strictly positive
72times are a helper policy; some native nonparametric APIs permit zero.
74Validate bounded local CSV input:
76```bash
77python skills/scikit-survival/scripts/validate_survival_csv.py \
78 --input data.csv \
79 --event-column event \
80 --time-column time \
81 --feature-columns x1,x2,group \
82 --structured-output outcome.npy
83```
85The `.npy` file contains a non-object structured array and can be loaded with
86`numpy.load(path, allow_pickle=False)`.

Minimum validation

Before splitting:

  • event and time lengths match feature rows;
  • standard event values are boolean/0/1;
  • competing-risk codes are non-negative integers and 0 means censoring;
  • time is numeric, finite, and strictly positive;
  • outcomes are not included among predictors;
  • feature names are unique and schema roles are explicit;
  • repeated entities, temporal ordering, or sites are identified for the split;
  • every planned training/CV fold contains events and enough comparable pairs; event-stratified CV in the bundled trainer also requires censored rows;
  • missingness is described, but imputation is not yet fitted.

The bundled validator checks outcome coding, positive finite times, feature exclusion, and the local CSV schema. Group independence, fold support, temporal ordering and missingness interpretation require the study-specific review above. Strictly positive times are a helper policy; some native nonparametric APIs permit zero.

Validate bounded local CSV input:

python skills/scikit-survival/scripts/validate_survival_csv.py \
  --input data.csv \
  --event-column event \
  --time-column time \
  --feature-columns x1,x2,group \
  --structured-output outcome.npy

The .npy file contains a non-object structured array and can be loaded with numpy.load(path, allow_pickle=False).

Raw lines 54 to 87
Runs code python skills/scikit-survival/scripts/validate_survival_csv.py :77
88## Split before learned preprocessing
90This order is mandatory:
921. validate types and outcome coding;
932. split rows;
943. fit imputation, encoding, scaling, and feature selection on training rows;
954. transform validation/test rows with training-fitted state;
965. fit the survival estimator;
976. evaluate once on the held-out rows.
99Computing medians, category levels, scaling moments, univariate scores, or
100regularization paths on all rows leaks validation/test information.
102```python
103from sklearn.model_selection import train_test_split
105X_train, X_test, y_train, y_test = train_test_split(
106 X,
107 y,
108 test_size=0.25,
109 stratify=y["event"],
110 random_state=20260723,
112```
114Event stratification does not guarantee balanced follow-up times. Inspect each
115split. Use group-aware splitting for repeated entities and time-respecting
116splitting for future-deployment questions. A random split is not automatically
117appropriate.

Split before learned preprocessing

This order is mandatory:

  1. validate types and outcome coding;
  2. split rows;
  3. fit imputation, encoding, scaling, and feature selection on training rows;
  4. transform validation/test rows with training-fitted state;
  5. fit the survival estimator;
  6. evaluate once on the held-out rows.

Computing medians, category levels, scaling moments, univariate scores, or regularization paths on all rows leaks validation/test information.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y["event"],
    random_state=20260723,
)

Event stratification does not guarantee balanced follow-up times. Inspect each split. Use group-aware splitting for repeated entities and time-respecting splitting for future-deployment questions. A random split is not automatically appropriate.

Raw lines 88 to 118
119## Explicit heterogeneous pipeline
121Use scikit-learn's `ColumnTransformer` when numeric and categorical columns need
122different imputers or when unseen categories must be handled explicitly.
124```python
125from sklearn.compose import ColumnTransformer
126from sklearn.impute import SimpleImputer
127from sklearn.pipeline import make_pipeline
128from sklearn.preprocessing import OneHotEncoder, StandardScaler
129from sksurv.linear_model import CoxPHSurvivalAnalysis
131numeric_pipe = make_pipeline(
132 SimpleImputer(strategy="median"),
133 StandardScaler(),
135categorical_pipe = make_pipeline(
136 SimpleImputer(strategy="most_frequent"),
137 OneHotEncoder(
138 handle_unknown="ignore",
139 drop="first",
140 sparse_output=False,
141 ),
143preprocess = ColumnTransformer(
144 [
145 ("numeric", numeric_pipe, numeric_columns),
146 ("categorical", categorical_pipe, categorical_columns),
147 ],
148 sparse_threshold=0.0,
150pipeline = make_pipeline(
151 preprocess,
152 CoxPHSurvivalAnalysis(alpha=0.1, ties="efron"),
154pipeline.fit(X_train, y_train)
155```
157Keep the estimator in the same pipeline used by cross-validation so each fold
158fits its own preprocessing state.

Explicit heterogeneous pipeline

Use scikit-learn's ColumnTransformer when numeric and categorical columns need different imputers or when unseen categories must be handled explicitly.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sksurv.linear_model import CoxPHSurvivalAnalysis

numeric_pipe = make_pipeline(
    SimpleImputer(strategy="median"),
    StandardScaler(),
)
categorical_pipe = make_pipeline(
    SimpleImputer(strategy="most_frequent"),
    OneHotEncoder(
        handle_unknown="ignore",
        drop="first",
        sparse_output=False,
    ),
)
preprocess = ColumnTransformer(
    [
        ("numeric", numeric_pipe, numeric_columns),
        ("categorical", categorical_pipe, categorical_columns),
    ],
    sparse_threshold=0.0,
)
pipeline = make_pipeline(
    preprocess,
    CoxPHSurvivalAnalysis(alpha=0.1, ties="efron"),
)
pipeline.fit(X_train, y_train)

Keep the estimator in the same pipeline used by cross-validation so each fold fits its own preprocessing state.

Raw lines 119 to 159
160### scikit-survival encoder
162`sksurv.preprocessing.OneHotEncoder(allow_drop=True)`:
164- treats pandas `category`/`object` and Polars categorical/enum/string columns as
165 categorical;
166- leaves non-categorical column order in place;
167- drops one category per categorical feature;
168- returns the same DataFrame library as its input;
169- requires `fit` and `transform` to use the same DataFrame library;
170- supports `get_feature_names_out()` and pipeline use.
172It is convenient for already clean DataFrames:
174```python
175from sklearn.pipeline import make_pipeline
176from sksurv.preprocessing import OneHotEncoder
178pipeline = make_pipeline(
179 OneHotEncoder(),
180 CoxPHSurvivalAnalysis(alpha=0.1),
182pipeline.fit(X_train, y_train)
183```
185For custom files, an explicit `ColumnTransformer` usually makes missing-value,
186unknown-category, and scaling behavior easier to audit.
188`encode_categorical()` is a one-shot transformation, not a fitted train/test
189transformer. Do not call it separately on all data or independently on train and
190test when category sets can differ.

scikit-survival encoder

sksurv.preprocessing.OneHotEncoder(allow_drop=True):

  • treats pandas category/object and Polars categorical/enum/string columns as categorical;
  • leaves non-categorical column order in place;
  • drops one category per categorical feature;
  • returns the same DataFrame library as its input;
  • requires fit and transform to use the same DataFrame library;
  • supports get_feature_names_out() and pipeline use.

It is convenient for already clean DataFrames:

from sklearn.pipeline import make_pipeline
from sksurv.preprocessing import OneHotEncoder

pipeline = make_pipeline(
    OneHotEncoder(),
    CoxPHSurvivalAnalysis(alpha=0.1),
)
pipeline.fit(X_train, y_train)

For custom files, an explicit ColumnTransformer usually makes missing-value, unknown-category, and scaling behavior easier to audit.

encode_categorical() is a one-shot transformation, not a fitted train/test transformer. Do not call it separately on all data or independently on train and test when category sets can differ.

Raw lines 160 to 191
192## Scaling and missing values
194- Scale Coxnet, survival SVM, IPC ridge, and other coefficient/penalty models.
195- Tree ensembles generally do not require scaling.
196- Impute inside the pipeline unless the selected estimator explicitly supports the
197 observed missing-value pattern.
198- SurvivalTree, RandomSurvivalForest, and ExtraSurvivalTrees support missing-value
199 splitting in current releases, but preprocessing may still be needed for
200 categorical data and operational consistency.
201- Never impute event indicators or event/censoring times as ordinary features.
203Missingness can be informative. A convenient imputer does not justify a
204missing-at-random assumption or transportability claim.

Scaling and missing values

  • Scale Coxnet, survival SVM, IPC ridge, and other coefficient/penalty models.
  • Tree ensembles generally do not require scaling.
  • Impute inside the pipeline unless the selected estimator explicitly supports the observed missing-value pattern.
  • SurvivalTree, RandomSurvivalForest, and ExtraSurvivalTrees support missing-value splitting in current releases, but preprocessing may still be needed for categorical data and operational consistency.
  • Never impute event indicators or event/censoring times as ordinary features.

Missingness can be informative. A convenient imputer does not justify a missing-at-random assumption or transportability claim.

Raw lines 192 to 205
206## Feature selection
208Feature selection is learned preprocessing and belongs inside inner CV:
210```python
211pipeline = make_pipeline(
212 preprocess,
213 selector,
214 estimator,
216```
218Do not use a standard classification `SelectKBest` score with structured survival
219outcomes unless the score function explicitly supports censoring. Coxnet or
220componentwise boosting can perform embedded selection, but regularization strength
221still requires fold-contained tuning.
223Fixed "events per variable" thresholds are not universal guarantees. Consider
224effective degrees of freedom, censoring, shrinkage, separation, stability, and
225external validation instead of declaring a model valid from one ratio.

Feature selection

Feature selection is learned preprocessing and belongs inside inner CV:

pipeline = make_pipeline(
    preprocess,
    selector,
    estimator,
)

Do not use a standard classification SelectKBest score with structured survival outcomes unless the score function explicitly supports censoring. Coxnet or componentwise boosting can perform embedded selection, but regularization strength still requires fold-contained tuning.

Fixed "events per variable" thresholds are not universal guarantees. Consider effective degrees of freedom, censoring, shrinkage, separation, stability, and external validation instead of declaring a model valid from one ratio.

Raw lines 206 to 226
227## Built-in datasets
229Current loaders:
231- `load_aids(endpoint=...)`
232- `load_bmt()`
233- `load_cgvhd()`
234- `load_breast_cancer()`
235- `load_flchain()`
236- `load_gbsg2()`
237- `load_whas500()`
238- `load_veterans_lung_cancer()`
239- `load_arff_files_standardized(...)`
241`load_bmt()` returns event codes 0, 1, 2 and `load_cgvhd()` returns 0, 1, 2, 3;
242these are competing-risk outcomes. The remaining listed study loaders return the
243standard boolean right-censored outcome (subject to endpoint options).
245These packaged datasets are useful for reproducing documentation, but they are
246real study datasets. The bundled CLIs and tests do not use them; they use synthetic
247data only. Do not treat examples as clinical advice or a substitute for data-use
248review.
250`load_arff_files_standardized(..., path_testing=...)` concatenates train/test
251features to standardize and harmonize categories. Do not use its default transforms
252for independent validation. Read raw ARFF tables with `sksurv.io.loadarff`, extract
253outcomes explicitly, and fit preprocessing only on training folds.
255In 0.28, dataset loaders accept an `output_type` option where documented, allowing
256pandas (default) or Polars feature output.

Built-in datasets

Current loaders:

  • load_aids(endpoint=...)
  • load_bmt()
  • load_cgvhd()
  • load_breast_cancer()
  • load_flchain()
  • load_gbsg2()
  • load_whas500()
  • load_veterans_lung_cancer()
  • load_arff_files_standardized(...)

load_bmt() returns event codes 0, 1, 2 and load_cgvhd() returns 0, 1, 2, 3; these are competing-risk outcomes. The remaining listed study loaders return the standard boolean right-censored outcome (subject to endpoint options).

These packaged datasets are useful for reproducing documentation, but they are real study datasets. The bundled CLIs and tests do not use them; they use synthetic data only. Do not treat examples as clinical advice or a substitute for data-use review.

load_arff_files_standardized(..., path_testing=...) concatenates train/test features to standardize and harmonize categories. Do not use its default transforms for independent validation. Read raw ARFF tables with sksurv.io.loadarff, extract outcomes explicitly, and fit preprocessing only on training folds.

In 0.28, dataset loaders accept an output_type option where documented, allowing pandas (default) or Polars feature output.

Raw lines 227 to 257
258## Unsupported or specialized structures
260Standard estimators do not directly encode:
262- interval-censored outcomes;
263- ordinary left-censoring;
264- time-varying covariates in counting-process form;
265- recurrent-event dependence;
266- multi-state transitions;
267- delayed entry in the two-field `Surv` estimator outcome.
269Some nonparametric APIs expose entry-time arguments, but that does not make every
270estimator support left truncation. Choose a method whose likelihood and input
271contract match the observation process.

Unsupported or specialized structures

Standard estimators do not directly encode:

  • interval-censored outcomes;
  • ordinary left-censoring;
  • time-varying covariates in counting-process form;
  • recurrent-event dependence;
  • multi-state transitions;
  • delayed entry in the two-field Surv estimator outcome.

Some nonparametric APIs expose entry-time arguments, but that does not make every estimator support left truncation. Choose a method whose likelihood and input contract match the observation process.

Raw lines 258 to 272
273## Local-data safeguards
275- Use only authorized, appropriately de-identified local data.
276- Do not put row-level data in logs or model reports.
277- Do not load untrusted pickle/joblib model files.
278- Keep feature and row bounds proportionate to available resources.
279- Use descriptive script names; never create files named after packages such as
280 `sklearn.py` or `sksurv.py`.

Local-data safeguards

  • Use only authorized, appropriately de-identified local data.
  • Do not put row-level data in logs or model reports.
  • Do not load untrusted pickle/joblib model files.
  • Keep feature and row bounds proportionate to available resources.
  • Use descriptive script names; never create files named after packages such as sklearn.py or sksurv.py.
Raw lines 273 to 281
282## Sources
284Official sources checked 2026-10-01:
286- [Surv API](https://scikit-survival.readthedocs.io/en/stable/api/generated/sksurv.util.Surv.html)
287- [OneHotEncoder API](https://scikit-survival.readthedocs.io/en/stable/api/generated/sksurv.preprocessing.OneHotEncoder.html)
288- [Dataset API](https://scikit-survival.readthedocs.io/en/stable/api/datasets.html)
289- [0.28 release notes](https://scikit-survival.readthedocs.io/en/stable/release_notes/v0.28.html)
290- [Introduction user guide](https://scikit-survival.readthedocs.io/en/stable/user_guide/00-introduction.html)