Split first, fit second: a preprocessing leakage check

An innocent-looking fit can cross the boundary

A preprocessing step is part of learning when it estimates something from the data. Fitting a scaler on the entire dataset before separating training and test rows lets the held-out rows influence that estimate. You can make this mistake without ever passing their labels to the classifier. [1]

The practical boundary is the information allowed into training. Reserve the evaluation partition before fitting learned transformations. For an ordinary held-out evaluation, training operations should not learn from that partition. This guide isolates one scaler mistake rather than presenting a full model-selection recipe.

References: [1]

Make the learned statistic visible

Our training feature values are 0, 2, 4, and 6. Their mean is 3. The two held-out feature values are deliberately distant: 100 and 102. Including them changes the fitted mean to 35.666667. This artificial gap is chosen to expose the dependency, not to imitate a representative dataset.

StandardScaler stores per-feature statistics from its fit input and reuses them during transformation. For this example, scaling uses the training mean and a population standard deviation of sqrt(5), consistent with its documented variance convention. The large transformed test values are expected under those training statistics. They are not a reason to refit on the test partition. [2]

References: [2]

Run a pipeline whose fit input is explicit

Run examples/leakage.py with Python 3.12.14, NumPy 2.3.5, scikit-learn 1.8.0 and SciPy 1.17.0. The fixed toy partitions remove random splitting from this diagnostic. Logistic regression is present to exercise the preprocessing-to-prediction path; no accuracy comparison is reported.

The assertions check the learned mean, the transformed held-out values, and the absence of a mean change after prediction. They verify this code path, not the quality of a model on a real task.

Executed on CPU · Python 3.12.14 · 21 October 2026

"""Check what a scaler learns, rather than inventing an accuracy gain."""
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

# Fixed, disjoint toy partitions make the boundary easy to inspect.
X_train = np.array([[0.], [2.], [4.], [6.]])
y_train = np.array([0, 0, 1, 1])
X_test = np.array([[100.], [102.]])
leaky = StandardScaler().fit(np.vstack([X_train, X_test]))
pipe = make_pipeline(StandardScaler(), LogisticRegression(random_state=0))
pipe.fit(X_train, y_train)
scaler = pipe.named_steps['standardscaler']
np.testing.assert_allclose(scaler.mean_, [3.])
np.testing.assert_allclose(leaky.mean_, [214 / 6])
before = scaler.mean_.copy()
scaled_test = scaler.transform(X_test)
_ = pipe.predict(X_test)  # Uses fitted preprocessing; does not refit it.
np.testing.assert_array_equal(scaler.mean_, before)
np.testing.assert_allclose(scaled_test[:, 0], [97 / np.sqrt(5), 99 / np.sqrt(5)])
print('training-only mean:', scaler.mean_.tolist())
print('leaked mean:', np.round(leaky.mean_, 6).tolist())
print('held-out scaled:', np.round(scaled_test[:, 0], 6).tolist())
print('mean unchanged after predict:', np.array_equal(scaler.mean_, before))

Recorded output

training-only mean: [3.0]
leaked mean: [35.666667]
held-out scaled: [43.379719, 44.274146]
mean unchanged after predict: True

Keep preprocessing attached to the estimator

A pipeline groups the transformer and predictor. In this example, fitting the pipeline learns the scaler from X_train before fitting logistic regression. Prediction applies that fitted transformation to new inputs. You avoid a separate notebook step that can accidentally fit on a broader array. [3]

When evaluating with cross-validation, pass the whole unfitted pipeline to the evaluation tool so its learned preprocessing can be fitted within each training fold. Fitting preprocessing once before cross-validation reintroduces the boundary problem. This is documented guidance; a cross-validation experiment was not executed for this article. [1, 3]

References: [1], [3]

What this test does not prove

The pipeline cannot repair a feature that already contains future information, a duplicate entity placed on both sides of the split, or a split that fails to reflect the deployment question. These are separate data-design checks. Treat the pipeline as one way to encode a boundary, not evidence that every relevant boundary is correct.

Our intentionally shifted held-out values do not demonstrate improved accuracy, an expected gain from fixing leakage, or a suitable deployment distribution. The measured outcome is narrower: the leaky scaler used information it should not have used, while the pipeline scaler retained training-only statistics.

A useful review question

For each learned transformation, ask which rows were available to its fit call. Check imputation, feature selection, and dimensionality reduction as well as scaling. Then inspect the split and feature construction separately. These checks are easier to review when the fitted pipeline and evaluation procedure are kept together. [1]

If the intended task involves time or repeated entities, choose an evaluation design appropriate to that dependency before running this recipe. This toy partition is a diagnostic example, not a recommendation to split production data by array position.

References: [1]

Sources

1. scikit-learn: common pitfalls and data leakage
2. scikit-learn: StandardScaler
3. scikit-learn: Pipeline