When NumPy broadcasting gives you the wrong loss

The result can be valid and still be wrong

Suppose a regression model returns three predictions, and each prediction exactly matches its target. You expect a mean squared error of zero. Instead, your calculation returns 1.333333. There is no exception to investigate: NumPy has performed the operation you requested. The problem is the comparison you accidentally described.

In this example, predictions have shape (3,), while targets have shape (3, 1). The intended contract is one scalar prediction per row. Those two representations look similar when printed, but their axes carry different instructions.

Work from the rightmost axis

For elementwise arithmetic, broadcasting checks dimensions from the right. A pair of dimensions can align when their sizes match or one is 1; absent leading dimensions behave as singleton dimensions. Here the prediction vector effectively aligns as (1, 3), so subtracting a (3, 1) target produces a (3, 3) result. Each target is compared with every prediction. [1]

This is not the rule for every NumPy operation. Matrix multiplication has its own axis semantics. The diagnostic here applies to the elementwise subtraction and squaring used in this loss calculation.

References: [1]

Reproduce the silent error

Run the following as examples/broadcasting.py. This CPU example was executed with Python 3.12.14 and NumPy 2.3.5. Its assertions verify both the unintended pairwise result and the intended aligned result. The large-case calculation uses shapes and integer arithmetic only; it does not allocate a giant array.

Executed on CPU · Python 3.12.14 · 21 October 2026

"""Reproduce a silent pairwise-error bug; CPU only, no dataset download."""
import numpy as np

prediction = np.array([1., 2., 3.])
target = np.array([[1.], [2.], [3.]])
wrong = (prediction - target) ** 2
assert wrong.shape == (3, 3)
assert np.isclose(wrong.mean(), 4 / 3)
target_vector = target[:, 0]  # Deliberate: one scalar target per row.
assert prediction.shape == target_vector.shape
correct = (prediction - target_vector) ** 2
assert correct.shape == (3,)
assert correct.mean() == 0
print('broadcast shape:', np.broadcast_shapes(prediction.shape, target.shape))
print('wrong MSE:', round(float(wrong.mean()), 6))
print('aligned MSE:', float(correct.mean()))
print('result bytes:', wrong.nbytes, correct.nbytes)

# Shape/byte calculation only: do not allocate this large pairwise result.
n = 100_000
assert np.broadcast_shapes((n,), (n, 1)) == (n, n)
print('large pairwise GiB:', round(n * n * 8 / 2**30, 3))

Recorded output

broadcast shape: (3, 3)
wrong MSE: 1.333333
aligned MSE: 0.0
result bytes: 72 24
large pairwise GiB: 74.506

Read the output as a contract check

The nonzero loss comes from nine pairwise comparisons, including comparisons between different rows. Selecting target[:, 0] is deliberate here because our toy data has exactly one target per row. It leaves two matching one-dimensional arrays and three zero errors. Do not flatten an arbitrary multi-output target just to silence a shape mismatch.

A useful guard is to check the expected semantic shape immediately before the loss operation. Equal element counts are insufficient. In a real pipeline, record which axis means batch and which means target dimension; then assert that representation. This is an engineering recommendation derived from the example, not a claim that broadcasting itself is defective.

Estimate the output before allocating it

The broadcast_shapes helper computes the resulting shape and rejects incompatible dimensions without needing the large numerical result. For a vector of length 100,000 and a column with 100,000 rows, the pairwise result contains 10 billion elements. At eight bytes each, one dense result payload is approximately 74.506 GiB. That number is arithmetic, not a measured process-memory peak. [2, 3]

For the actual small arrays, nbytes reports 72 bytes for the nine-element error array and 24 bytes for the three-element aligned array. Array-object overhead and other temporaries are outside those figures. Avoid assuming that conceptual broadcasting of an input also makes the computed output free.

References: [2], [3]

Before you trust the metric

Write down the intended input and output shapes, inspect the actual shapes, and test a case whose answer is obvious. Perfectly matching predictions should produce zero error under this toy loss. If they do not, investigate axis alignment before changing the model.

This guide tests one-dimensional scalar regression arithmetic. It does not establish the shape contract for a particular framework loss, masked sequence task, or multi-label objective. Those require their own API and data checks.

Sources

1. NumPy: broadcasting rules
2. NumPy: broadcast_shapes
3. NumPy: ndarray.nbytes