Skill Tracer
Skills › Scientific Agent Skills › relsa-severity-assessment › forecasting.md

forecasting.md

102 lines · skills/relsa-severity-assessment/references/forecasting.md at 5f2bc03

Open on GitHub
1# Exploratory ARIMA forecasts of RELSA
2
3The [2026 foRcast paper](https://doi.org/10.3389/fphys.2026.1869563) predicts a RELSA score at
4an already specified future observation time. It does not fit a survival model, estimate a
5probability of death, or establish a validated time-to-humane-endpoint predictor.
6
7The bundled Python helper is an independent nonseasonal approximation of that workflow.
8Current [R forecast::auto.arima documentation](https://pkg.robjhyndman.com/forecast/reference/auto.arima.html)
9includes seasonality, separate mean/drift controls, transformations, alternative unit-root
10tests and approximation policies not replicated here. Do not label results R/foRcast-equivalent.

Exploratory ARIMA forecasts of RELSA

The 2026 foRcast paper predicts a RELSA score at an already specified future observation time. It does not fit a survival model, estimate a probability of death, or establish a validated time-to-humane-endpoint predictor.

The bundled Python helper is an independent nonseasonal approximation of that workflow. Current R forecast::auto.arima documentation includes seasonality, separate mean/drift controls, transformations, alternative unit-root tests and approximation policies not replicated here. Do not label results R/foRcast-equivalent.

Raw lines 1 to 11
12## Selection and time semantics
14`auto_arima()` chooses differencing via KPSS, then searches bounded p/q orders using AICc.
15`stepwise=False` searches the configured p/q grid at the selected d. The SARIMAX likelihood,
16KPSS lag selection and neighbor search differ from R. The helper now uses statsmodels'
17`results.aicc` (effective observations and parameter accounting), rejects unconverged fits,
18and disallows a constant when d > 1. A constant at d=0 is an intercept; at d=1 it is drift.
19A finite criterion and numerical convergence do not establish stationarity, calibration or
20biological adequacy. Inspect residuals and compare against a simple last-observation forecast
21on held-out animals. Do not select a model from its evaluation endpoint errors.
23[statsmodels SARIMAXResults](https://www.statsmodels.org/stable/generated/statsmodels.tsa.statespace.sarimax.SARIMAXResults.html)
24provides `get_forecast(steps=...)`; `conf_int(alpha=...)` is read from that prediction result,
25not from parameter confidence intervals on the fitted results object.
27- Training and requested targets must use one finite time unit and unique observation times.
28- Interpolation defaults to 0.1 **input time units**. All finite observed times and targets
29 must align with that grid. Off-grid requests fail, rather than receive a neighboring
30 horizon's forecast labeled with the requested time.
31- With `interpolate_step=None` (CLI `--interpolate-step 0`), observed times must be regular,
32 and each target must be an integer observation step after the last training observation.
33- Missing measurements are excluded; interpolation bridges only gaps between available
34 training points. Inspect gap length and missingness before fitting. Never interpolate
35 using an observation later than the forecast cutoff.
36- The two-observation minimum under interpolation is a computational allowance, not evidence
37 of forecast reliability. Fewer than four genuine observations generate a warning.
38- `predict_endpoint(..., endpoints={id: time})` uses only times strictly earlier than each
39 recorded endpoint. Omitting `endpoints` evaluates the last available observation and warns
40 that it is not necessarily a humane endpoint. `rolling_forecast` sorts times first.

Selection and time semantics

auto_arima() chooses differencing via KPSS, then searches bounded p/q orders using AICc. stepwise=False searches the configured p/q grid at the selected d. The SARIMAX likelihood, KPSS lag selection and neighbor search differ from R. The helper now uses statsmodels' results.aicc (effective observations and parameter accounting), rejects unconverged fits, and disallows a constant when d > 1. A constant at d=0 is an intercept; at d=1 it is drift. A finite criterion and numerical convergence do not establish stationarity, calibration or biological adequacy. Inspect residuals and compare against a simple last-observation forecast on held-out animals. Do not select a model from its evaluation endpoint errors.

statsmodels SARIMAXResults provides get_forecast(steps=...); conf_int(alpha=...) is read from that prediction result, not from parameter confidence intervals on the fitted results object.

  • Training and requested targets must use one finite time unit and unique observation times.
  • Interpolation defaults to 0.1 input time units. All finite observed times and targets must align with that grid. Off-grid requests fail, rather than receive a neighboring horizon's forecast labeled with the requested time.
  • With interpolate_step=None (CLI --interpolate-step 0), observed times must be regular, and each target must be an integer observation step after the last training observation.
  • Missing measurements are excluded; interpolation bridges only gaps between available training points. Inspect gap length and missingness before fitting. Never interpolate using an observation later than the forecast cutoff.
  • The two-observation minimum under interpolation is a computational allowance, not evidence of forecast reliability. Fewer than four genuine observations generate a warning.
  • predict_endpoint(..., endpoints={id: time}) uses only times strictly earlier than each recorded endpoint. Omitting endpoints evaluates the last available observation and warns that it is not necessarily a humane endpoint. rolling_forecast sorts times first.
Raw lines 12 to 41
42## Uncertainty and validation
44Interpolation adds no independent observations. It imposes smoothness and changes the fitted
45correlation structure, so extra grid points cannot justify stronger confidence. Narrowing an
46interval does not itself improve coverage. Compare results across defensible interpolation
47steps, aggregation intervals and observed-only models where enough observations exist.
49The returned Gaussian prediction interval is conditional on the fitted model, parameters,
50preprocessing and reference. It omits reference-estimation and model-selection uncertainty.
51The default floors means and bounds at zero to match the score's nonnegative support; this
52is clipping, not a fitted truncated probability distribution. Use `clip_at_zero=False` to
53inspect the raw Gaussian output. Reference scores and forecasts are not capped at one.
55Inspect the upper bound and interval width as well as observed welfare signs; no bound is
56an automatic care, euthanasia or wait decision. A smooth trajectory cannot predict an
57unobserved abrupt deterioration. Handling must follow the approved study criteria.
59Before prospective evaluation, freeze reference extrema, measurement encodings, baseline
60rules and candidate KDE thresholds using development animals. Split at animal level, and
61where relevant cage/study level, before tuning. Endpoint-selected calibration on evaluation
62animals is retrospective leakage. Repeated observations do not increase the independent
63animal sample size. Evaluate missed endpoints, false alerts, lead time and failure rates
64under a prespecified monitoring policy before claiming operational utility.
66`forecast_indirect()` separately forecasts measurements then recomputes the composite. It
67accepts exactly one animal, warns on failed component fits and supplies a point forecast only;
68it does not propagate joint uncertainty or covariance. Inspect missing components before
69comparing against a direct forecast. The paper's direct-versus-indirect result is specific
70to its sepsis analysis, not a general dominance theorem.

Uncertainty and validation

Interpolation adds no independent observations. It imposes smoothness and changes the fitted correlation structure, so extra grid points cannot justify stronger confidence. Narrowing an interval does not itself improve coverage. Compare results across defensible interpolation steps, aggregation intervals and observed-only models where enough observations exist.

The returned Gaussian prediction interval is conditional on the fitted model, parameters, preprocessing and reference. It omits reference-estimation and model-selection uncertainty. The default floors means and bounds at zero to match the score's nonnegative support; this is clipping, not a fitted truncated probability distribution. Use clip_at_zero=False to inspect the raw Gaussian output. Reference scores and forecasts are not capped at one.

Inspect the upper bound and interval width as well as observed welfare signs; no bound is an automatic care, euthanasia or wait decision. A smooth trajectory cannot predict an unobserved abrupt deterioration. Handling must follow the approved study criteria.

Before prospective evaluation, freeze reference extrema, measurement encodings, baseline rules and candidate KDE thresholds using development animals. Split at animal level, and where relevant cage/study level, before tuning. Endpoint-selected calibration on evaluation animals is retrospective leakage. Repeated observations do not increase the independent animal sample size. Evaluate missed endpoints, false alerts, lead time and failure rates under a prespecified monitoring policy before claiming operational utility.

forecast_indirect() separately forecasts measurements then recomputes the composite. It accepts exactly one animal, warns on failed component fits and supplies a point forecast only; it does not propagate joint uncertainty or covariance. Inspect missing components before comparing against a direct forecast. The paper's direct-versus-indirect result is specific to its sepsis analysis, not a general dominance theorem.

Raw lines 42 to 71
72## Metrics and the published aggregation caveat
74Report `n` (point-evaluable), `n_interval` (interval-evaluable) and forecast failures with:
76- RMSE: square root of mean squared point error on finite actual/prediction pairs.
77- PICP: percentage of actual values inside finite ordered bounds on evaluable predictions.
78- MPIW: mean bound width on the same interval-evaluable predictions, in RELSA units.
80A single forecast has either 0% or 100% observed coverage; neither establishes calibration.
81Group comparisons must disclose pooled-observation versus equal-animal weighting. Include
82uncertainty using animal-level resampling, not independent resampling of repeated time points.
84The paper's Table 1 contains the following historical results (not reproduced in this review):
86| Model | Animals | RMSE | PICP (%) | MPIW |
87| --- | --- | --- | --- | --- |
88| Sepsis | 2 | 0.009 | 100 | 0.30 |
89| 1.5% DSS + restraint | 2 | 0.007 | 100 | 0.66 |
90| 1% DSS + blood sampling | 4 | 0.046 | 75 | 0.53 |
91| 1.5% DSS + blood sampling | 2 | 0.065 | 100 | 0.84 |
92| 1.5% DSS | 1 | 0.095 | 100 | 1.64 |
93| Pancreatic cancer | 1 | 0.177 | 100 | 7.35 |
94| Neurosurgery | 1 | 0.082 | 100 | 0.54 |
95| Published overall | 13 | 0.069 | 96 | 1.69 |
97The overall values are consistent with unweighted means of model rows: PICP is 96.43%,
98whereas pooled animal coverage is 12/13 = 92.31%. The helper's summary pools predictions and
99will not recreate the published overall row merely by reproducing individual forecasts.
100MPIW 1.69 means 1.69 reference scale units, not 169% of a bounded biological severity range.
101Native Python synthetic tests validate arithmetic and mechanics only. R is not installed in
102this review environment; native R forecasting and full published datasets remain unexecuted.

Metrics and the published aggregation caveat

Report n (point-evaluable), n_interval (interval-evaluable) and forecast failures with:

  • RMSE: square root of mean squared point error on finite actual/prediction pairs.
  • PICP: percentage of actual values inside finite ordered bounds on evaluable predictions.
  • MPIW: mean bound width on the same interval-evaluable predictions, in RELSA units.

A single forecast has either 0% or 100% observed coverage; neither establishes calibration. Group comparisons must disclose pooled-observation versus equal-animal weighting. Include uncertainty using animal-level resampling, not independent resampling of repeated time points.

The paper's Table 1 contains the following historical results (not reproduced in this review):

ModelAnimalsRMSEPICP (%)MPIW
Sepsis20.0091000.30
1.5% DSS + restraint20.0071000.66
1% DSS + blood sampling40.046750.53
1.5% DSS + blood sampling20.0651000.84
1.5% DSS10.0951001.64
Pancreatic cancer10.1771007.35
Neurosurgery10.0821000.54
Published overall130.069961.69

The overall values are consistent with unweighted means of model rows: PICP is 96.43%, whereas pooled animal coverage is 12/13 = 92.31%. The helper's summary pools predictions and will not recreate the published overall row merely by reproducing individual forecasts. MPIW 1.69 means 1.69 reference scale units, not 169% of a bounded biological severity range. Native Python synthetic tests validate arithmetic and mechanics only. R is not installed in this review environment; native R forecasting and full published datasets remain unexecuted.

Raw lines 72 to 102