This article explains the estimators used by the supervised helper
family. It is important to keep them separate:
trainRegressionModel() constructs cell-level OOF and
unseen-row ensembles, oofEnsemble() independently
reconstructs training-row residuals, auditQuantiles()
summarizes those residuals as signed offsets, and the two delivery
helpers apply different selection rules. modelPipeline()
orchestrates these helpers; it is not an additional statistical
estimator.
Let \(r\) identify a response, \(d\) a dataset, \(m\) a learner, and \(i\) a training row. A cell is one response–dataset pair \((r,d)\). For a cell, \(y_i\) is the observed response and \(\hat y_{mi}^{OOF}\) is learner \(m\)’s out-of-fold (OOF) prediction. RMSE and response predictions have the response’s units; squared sample correlation \(R^2\) and normalized weights are dimensionless.
The principal API mapping is:
| Mathematical role | Public helper |
|---|---|
| fit caret learners and retain selected-tuning OOF rows | trainModel(), trainRegressionModel() |
| construct cell-level OOF and unseen-row ensembles | trainRegressionModel() |
| independently reconstruct training-row OOF residuals | oofEnsemble() |
| summarize residuals as signed offsets | auditQuantiles() |
| select a response-level cell | predictModel() |
| select final row-level RMSE candidates | aggregateResponses() |
| compute the package optimism diagnostic | auditOverfit() |
Learners are fitted separately with caret::train() (Kuhn 2008). Through trainModel(),
ssel requests five-fold cross-validation, RMSE tuning, selected-tuning
holdout predictions, and tuneLength = 10. A direct
trainRegressionModel() call instead uses its
.numberCV, .tuneLength, and
.metric arguments. The tuning length requests
method-specific grid granularity; it is not a number of points per
tuning parameter.
Separate learner calls do not receive a common explicit resampling index. Their saved OOF rows therefore support learner-specific score estimation, but ssel does not claim identical folds across learners.
For each retained learner in cell \((r,d)\), evaluation reports
\[ \operatorname{RMSE}_{mrd} =\left\{\frac{1}{n}\sum_i(y_i-\hat y_{mi}^{OOF})^2\right\}^{1/2}. \]
\[ R^2_{mrd}=\operatorname{cor}(y,\hat y_m^{OOF})^2. \]
The package defines two cell-local weighting policies:
\[ w^{R2}_{mrd}=\frac{R^2_{mrd}}{\sum_jR^2_{jrd}}, \qquad w^{RMSE}_{mrd} =\frac{\operatorname{RMSE}_{mrd}^{-1}} {\sum_j\operatorname{RMSE}_{jrd}^{-1}}. \]
The corresponding OOF ensemble is
\[ \hat y_{ird}^{g,OOF}=\sum_m w^g_{mrd}\hat y_{mi}^{OOF}, \qquad g\in\{R2,RMSE\}. \]
These normalized scores are package policies, not estimates of theoretically optimal ensemble weights and not guarantees that an ensemble improves every learner.
During OOF evaluation, trainRegressionModel() stores the
exact named \(R^2\) and RMSE lists for
every cell. Before predicting unseen rows in \((r,d)\), it restores that cell’s lists and
intersects their names with the learners that produced valid predictions
for that same cell. A missing model or a missing prediction removes a
learner only from the current cell.
The unseen-row pass uses the same normalization formulas. Its documented fallback substitutes equal weights when the retained \(R^2\) values, or the reciprocal RMSE values, contain a missing value or sum to zero. A cell without stored OOF score state has no unseen-row ensemble. Thus one cell neither borrows scores from nor prunes methods for another cell.
oofEnsemble() does not reuse the preceding score state.
It recomputes a method-level RMSE from each fitted object’s saved,
projected OOF predictions and defines independent inverse-RMSE weights
\(\tilde w_{mrd}\). Its cast key is
jointly \(h=(i,o)\), where \(i\) is the caret row index and \(o\) the averaged saved observation. With
\(A_h\) denoting learners available at
that key,
\[ \hat y_h^{OOF}=\Pi_{[a,b]}\!\left( \frac{\sum_{m\in A_h}\tilde w_{mrd}\tilde y_{mh}} {\sum_{m\in A_h}\tilde w_{mrd}} \right). \]
\[ e_h=o-\hat y_h^{OOF}. \]
Available learners are renormalized row by row. If saved observations differ between learners for one row index, they form different joint keys; repeated row indices and sample identifiers can therefore be legitimate output rows.
For valid bounds \(a\le b\), the interval projection used above is
\[ \Pi_{[a,b]}(z)=\min\{b,\max\{a,z\}\}. \]
This is the ordinary projection onto a closed interval (Parikh and Boyd 2014). Bounds are not proactively reordered, so this name applies only when \(a\le b\).
auditQuantiles() pools reconstructed residual rows
separately within each cell. For \(p\in\{0.50,0.90,0.95\}\), sort the \(n\) residuals and define
\[ H=(n-1)p+1. \]
\[ j=\lfloor H\rfloor,\qquad \gamma=H-j. \]
R’s type-7 sample quantile is then
\[ q_{p,rd}=(1-\gamma)e_{(j)}+\gamma e_{(j+1)}, \]
with the endpoint convention described by Hyndman and Fan (1996). The values are signed, cell-pooled, and expressed in response units. Delivery helpers form
\[ Q_{p,ird}=\Pi_{[a,b]}\{\hat y_{ird}+q_{p,rd}\}. \]
These are descriptive point-plus-residual-offset estimates. They are not conditional predictive quantiles, calibrated uncertainty intervals, or a conformal coverage procedure; those require different constructions (Barber et al. 2021).
predictModel() first chooses one eligible cell per
response. For ensemble.RMSE it takes the first minimum-RMSE
row; for ensemble.R2 it takes the first maximum \(R^2\) row:
\[ d_r^{RMSE}=\operatorname*{first\,argmin}_{d}\operatorname{RMSE}_{rd}. \]
\[ d_r^{R2}=\operatorname*{first\,argmax}_{d}R^2_{rd}. \]
aggregateResponses() applies a later and different rule.
Its candidates are only unseen ensemble.RMSE points and
reconstructed training OOF points. For each sample \(i\) and response \(r\), it selects
\[ d^*_{ir}=\operatorname*{first\,argmax}_{d}R^2_{rd}. \]
The selected candidate supplies all three signed offsets. Because the
second rule is grouped by sample and excludes ensemble.R2
candidates, the two selectors must not be interpreted as one global
best-dataset operation.
auditOverfit() compares projected in-sample RMSE with
the arithmetic mean of the fitted caret object’s stored resample RMSE
values:
\[ \operatorname{rel\_optimism} =\frac{\overline{\operatorname{RMSE}}_{CV}-\operatorname{RMSE}_{in}} {\max\{\overline{\operatorname{RMSE}}_{CV},10^{-9}\}}. \]
Its labels and thresholds are ssel policy. The value is a diagnostic, not a hypothesis test, a universal measure of overfitting, or an input to final aggregation.
The example below creates one fitted-object fixture with two equal signed OOF residuals. All three type-7 offsets must therefore equal one.
root <- tempfile("ssel-offsets-")
model_dir <- file.path(root, "model")
train_dir <- file.path(root, "train")
summary_dir <- file.path(root, "summary")
dir.create(model_dir, recursive = TRUE)
dir.create(train_dir)
data.table::fwrite(
data.table::data.table(y = c(1, 3)),
file.path(train_dir, "target_demo_TRAIN.csv")
)
data.table::fwrite(
data.table::data.table(SampleID = c("S1", "S2")),
file.path(train_dir, "target_demo_TRAIN_ids.csv")
)
fit <- list(
pred = data.frame(
tune = 1L, pred = c(0, 2), obs = c(1, 3), rowIndex = 1:2
),
bestTune = data.frame(tune = 1L)
)
saveRDS(fit, file.path(model_dir, "lm_target_demo.Rds"))
offsets <- auditQuantiles(
.path.model = model_dir,
.path.train = train_dir,
.path.summary = summary_dir,
.response_pattern = "target",
verbose = FALSE
)
offsets$quantiles
#> response dataset n q_50 q_90 q_95
#> <char> <char> <int> <num> <num> <num>
#> 1: target demo 2 1 1 1
stopifnot(all(offsets$quantiles[, c("q_50", "q_90", "q_95")] == 1))
unlink(root, recursive = TRUE)The complete argument, file, projection, skip, and side-effect contracts live in the linked function reference pages. The iterative multi-response and pseudo-label policies are described in their own articles.