ML Foundations
Why pooled out-of-fold R2 can hide poor fold scores
Concatenating held-out predictions changes the target-mean baseline used by R2. Inspect fold scores and residual sums before treating pooled R2 as mean cross-validation performance.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
Pooling predictions changes the comparison baseline
For nonconstant targets, R2 compares the prediction residual sum of squares with the target variation around the evaluated target mean: one minus residual sum divided by baseline sum. A score below zero means the supplied predictions are worse than that mean baseline. Scikit-learn explicitly allows negative scores and describes the constant target-mean predictor as scoring zero. [1]
When predictions from several held-out folds are concatenated, the evaluated mean becomes the global target mean. It is no longer each fold's own target mean. A large difference between fold target levels can therefore make pooled R2 look strong while every individual fold score is poor. This is a change in the comparison baseline, not a reduction in prediction errors.
References: [1]
Keep residual and baseline sums visible
The fixture supplies two equal-sized folds. Their target pairs are zero and one, then 100 and 101. Prediction pairs are zero and zero, then 100 and 100. Each fold has residual sum one and baseline sum 0.5, giving R2=-1. Pooling keeps the total residual sum at two but changes the baseline sum from a fold-summed one to a global 10001. The pooled score is approximately 0.999800.
Run python examples/pooled_fold_r2.py. Assertions verify both fold scores, their mean, pooled R2 and both sums. A second calculation removes the 100-unit offset from both targets and predictions in the second fold. Residuals stay unchanged, while pooled R2 falls to -1. This transformation is a controlled arithmetic check, not a proposed preprocessing step for improving evaluation.
The example checks perfect predictions and rejects four invalid or unsupported fixtures: fewer than two observations, mismatched lengths, constant targets and non-finite targets. Separate local comparisons with scikit-learn 1.8.0 matched both original fold scores and the pooled score. No model was fitted and no cross-validation splitter was executed.
Executed with Python 3.12.14 and NumPy 2.3.5. Separate scikit-learn 1.8.0 reference comparisons passed. Exact stdout and source-byte SHA-256 are preserved in private evidence.
"""Expose the changed target-mean baseline when pooling fold predictions."""
import numpy as np
def r2_parts(actual, predicted):
actual = np.asarray(actual, dtype=np.float64)
predicted = np.asarray(predicted, dtype=np.float64)
if actual.ndim != 1 or actual.shape != predicted.shape or actual.size < 2:
raise ValueError("aligned vectors with at least two observations required")
if not np.all(np.isfinite(actual)) or not np.all(np.isfinite(predicted)):
raise ValueError("finite observations required")
sse = float(np.sum((actual - predicted) ** 2))
sst = float(np.sum((actual - np.mean(actual)) ** 2))
if sst == 0:
raise ValueError("constant-target policy is outside this helper")
return 1 - sse / sst, sse, sst
targets = [np.array([0.0, 1.0]), np.array([100.0, 101.0])]
predictions = [np.array([0.0, 0.0]), np.array([100.0, 100.0])]
parts = [r2_parts(y, p) for y, p in zip(targets, predictions, strict=True)]
fold_scores = [p[0] for p in parts]
pooled, pooled_sse, pooled_sst = r2_parts(np.concatenate(targets), np.concatenate(predictions))
fold_sse = sum(p[1] for p in parts)
fold_sst = sum(p[2] for p in parts)
assert fold_scores == [-1.0, -1.0]
assert fold_sse == pooled_sse == 2.0
assert fold_sst == 1.0 and pooled_sst == 10001.0
assert np.isclose(pooled, 1 - 2 / 10001)
# Removing the same offset from both target and prediction changes no residual.
aligned_targets = [targets[0], targets[1] - 100]
aligned_predictions = [predictions[0], predictions[1] - 100]
aligned_score = r2_parts(np.concatenate(aligned_targets), np.concatenate(aligned_predictions))[0]
assert aligned_score == -1.0
assert r2_parts([0, 1], [0, 1])[0] == 1.0
rejected = 0
for y, p in (([1], [1]), ([0, 1], [0]), ([1, 1], [1, 1]), ([0, np.nan], [0, 1])):
try:
r2_parts(y, p)
except ValueError:
rejected += 1
assert rejected == 4
print("per-fold R2:", fold_scores)
print("mean fold R2:", float(np.mean(fold_scores)))
print("pooled prediction R2:", format(pooled, ".6f"))
print("fold-summed and pooled residual sums:", fold_sse, pooled_sse)
print("fold-summed and pooled baseline sums:", fold_sst, pooled_sst)
print("pooled R2 after aligning fold offsets:", aligned_score)
print("invalid/unsupported fixtures rejected:", rejected)
Recorded output
per-fold R2: [-1.0, -1.0] mean fold R2: -1.0 pooled prediction R2: 0.999800 fold-summed and pooled residual sums: 2.0 2.0 fold-summed and pooled baseline sums: 1.0 10001.0 pooled R2 after aligning fold offsets: -1.0 invalid/unsupported fixtures rejected: 4
References: [1]
Report the aggregation that answers the evaluation question
Scikit-learn warns that evaluating cross_val_predict outputs can differ from cross_val_score or cross_validate unless test sets are equally sized and the metric decomposes over samples. Equal fold sizes alone do not solve this R2 example, because the baseline depends on which targets are pooled. [2]
Keep per-fold R2 values when reporting mean fold performance. If pooled held-out R2 is also useful, label it separately and inspect residual error and fold target spread alongside it. Neither summary alone proves that the split represents deployment. This unweighted, single-output teaching helper deliberately rejects constant targets rather than reproducing scikit-learn's force_finite convention; extreme-range arithmetic and uncertainty estimates are outside its scope.