ML Foundations
Compute binary loss from logits without overflowing sigmoid
For hard binary targets, logaddexp gives a compact per-example loss that avoids explicitly forming saturated probabilities.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
Keep the loss calculation in logit space
Computing a sigmoid probability and then taking binary cross-entropy logs can encounter floating-point problems at extreme logits. PyTorch documents BCEWithLogitsLoss as a combined sigmoid and loss operation that uses the log-sum-exp trick for better numerical stability. Supply logits to that combined operator when using it; its input is not an already-computed sigmoid probability. [1]
The NumPy example isolates the unweighted per-example arithmetic for hard targets, exactly zero or one. Starting from the binary cross-entropy expression, the loss for target zero is log(1 + exp(z)); for target one it is log(1 + exp(-z)). Both can be written as logaddexp(0, (1 - 2*y)*z). NumPy defines logaddexp as log(exp(a) + exp(b)), which supplies the required operation. [1, 2]
That compact expression relies on the hard-label restriction. It is not the corresponding formula for fractional targets, so the helper explicitly rejects them. This restriction is narrower than the target range supported by the documented PyTorch loss.
Test confident mistakes as well as confident correct predictions
Run python examples/binary_loss_logaddexp.py. The five examples include logits of minus and plus 1000, paired with both possible labels, and a zero logit. Direct sigmoid evaluation raises under the chosen overflow policy. The logit-space calculation returns finite losses: the confidently wrong examples cost 1000, while the zero logit costs log(2). Correct extreme predictions have losses that underflow to zero in this float64 fixture.
The assertions compare against these expected values and against the ordinary probability formula on moderate logits, where that formula is well behaved. A symmetry check negates logits and flips labels; the losses must agree. Additional checks reject empty input, mismatched shapes, an infinite logit, a fractional target and a two-dimensional input.
This execution uses NumPy only. The PyTorch source establishes the combined-loss interface and stability rationale; no PyTorch execution, gradient comparison or claim about its internal kernel is made.
Executed with Python 3.12.14 and NumPy 2.3.5 on bounded float64 fixtures. Exact stdout and source-byte SHA-256 are preserved in private evidence.
"""Unweighted per-example binary loss for hard labels, computed from logits."""
import numpy as np
def binary_loss(logits, labels):
z = np.asarray(logits, dtype=np.float64)
y = np.asarray(labels, dtype=np.float64)
if z.ndim != 1 or z.size == 0 or z.shape != y.shape:
raise ValueError('expected equal nonempty one-dimensional inputs')
if not np.isfinite(z).all() or not np.isfinite(y).all():
raise ValueError('expected finite values')
if not ((y == 0) | (y == 1)).all():
raise ValueError('this example accepts hard binary labels only')
with np.errstate(over='raise', invalid='raise', divide='raise', under='ignore'):
return np.logaddexp(0., (1. - 2. * y) * z)
z = np.array([-1000., 1000., -1000., 1000., 0.])
y = np.array([0., 1., 1., 0., 1.])
try:
with np.errstate(over='raise'):
1. / (1. + np.exp(-z))
except FloatingPointError:
print('direct sigmoid: overflow detected')
else:
raise AssertionError('expected direct sigmoid overflow')
loss = binary_loss(z, y)
assert np.isfinite(loss).all() and (loss >= 0).all()
assert np.allclose(loss, [0., 0., 1000., 1000., np.log(2.)])
assert np.allclose(loss, binary_loss(-z, 1. - y))
moderate = np.array([-2., 0., 2.])
labels = np.array([0., 1., 1.])
p = 1. / (1. + np.exp(-moderate))
reference = -(labels * np.log(p) + (1. - labels) * np.log1p(-p))
assert np.allclose(binary_loss(moderate, labels), reference, rtol=1e-12, atol=1e-12)
print('losses from logits:', loss.round(6).tolist())
print('moderate-input agreement:', True)
print('label/logit flip symmetry:', True)
rejected = 0
for bad_z, bad_y in (([], []), ([0., 1.], [1.]), ([np.inf], [1.]), ([0.], [0.5]), ([[0.]], [[1.]])):
try:
binary_loss(bad_z, bad_y)
except ValueError:
rejected += 1
assert rejected == 5
print('invalid fixtures rejected:', rejected)
Recorded output
direct sigmoid: overflow detected losses from logits: [0.0, 0.0, 1000.0, 1000.0, 0.693147] moderate-input agreement: True label/logit flip symmetry: True invalid fixtures rejected: 5
Preserve the intended loss contract
The helper returns one loss per input element and requires equal one-dimensional shapes. It does not average or sum, apply positive-class weights, support soft targets or build an automatic-differentiation graph. PyTorch exposes reduction and weighting controls for those separate choices. Confirm those settings when comparing implementations. [1]
The small fixture establishes the arithmetic and observed overflow avoidance. Extremely small tail losses can still disappear through underflow, and this example does not establish model accuracy or training convergence. Keep the finite-value and shape checks when adapting the calculation to a bounded numerical test.