ML Foundations
When PyTorch DataLoader completion order can skew batches
The in_order option can trade FIFO delivery for earlier completed batches. Understand the reproducibility and class-order risks before enabling it.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
FIFO is a useful default
With multiple workers, batches can finish loading at different times. PyTorch's DataLoader uses `in_order=True` by default and returns batches in first-in, first-out order; `in_order=False` removes that ordering guarantee and applies only when `num_workers > 0`. The documentation also notes that FIFO can require caching batches that arrive early while the loader waits for the next one. [1, 2]
That wait can matter when one worker regularly takes longer than another. It is tempting to treat disabling order as a free speed setting, but it changes which batch reaches the training loop next, so there is a second question beyond throughput: does arrival order matter to this run?
Completion order can change the early sample mix
PyTorch warns that `in_order=False` can harm reproducibility and may skew the data distribution fed to a trainer when the data are imbalanced. One plausible route is that loading latency is correlated with a class or source, so the faster group appears earlier in the epoch. This is a conditional risk, not a claim that every unordered loader will bias a completed epoch. [1]
The small example below is an intentionally constructed schedule: two rare-class batches take longer than four common-class batches. It compares the first four batches under FIFO and completion order. It uses plain Python rather than PyTorch, and demonstrates how a prefix can differ when latency and labels line up; it does not benchmark DataLoader or estimate model quality.
Executed with Python 3.12.14; this is a deterministic toy schedule, not a PyTorch integration test. Exact stdout and source-byte SHA-256 are preserved in private evidence.
"""Illustrate how worker completion order can change an early batch prefix."""
from collections import Counter
# A toy schedule, not a PyTorch timing measurement. Two slow "rare" batches
# finish after four quicker "common" batches.
batches = [
("rare-0", "rare", 5),
("rare-1", "rare", 6),
("common-0", "common", 1),
("common-1", "common", 2),
("common-2", "common", 3),
("common-3", "common", 4),
]
fifo = [batch_id for batch_id, label, _ in batches]
completion_order = [batch_id for batch_id, label, _ in sorted(batches, key=lambda row: row[2])]
labels = {batch_id: label for batch_id, label, _ in batches}
def rare_share(ids):
counts = Counter(labels[batch_id] for batch_id in ids)
return counts["rare"] / len(ids)
prefix_size = 4
print("FIFO first four:", fifo[:prefix_size])
print("completion-order first four:", completion_order[:prefix_size])
print("rare share, FIFO prefix:", f"{rare_share(fifo[:prefix_size]):.0%}")
print("rare share, completion prefix:", f"{rare_share(completion_order[:prefix_size]):.0%}")
print("all batches retained:", sorted(fifo) == sorted(completion_order))
Recorded output
FIFO first four: ['rare-0', 'rare-1', 'common-0', 'common-1'] completion-order first four: ['common-0', 'common-1', 'common-2', 'common-3'] rare share, FIFO prefix: 50% rare share, completion prefix: 0% all batches retained: True
References: [1]
Treat unordered delivery as an experiment
Keep the default when reproducible batch order matters or when you have not checked the relationship between loading time and labels. If you test `in_order=False`, compare label or source composition over the same early-step windows, not just total epoch counts, and repeat the run with a fixed data order where practical. PyTorch's warning supports checking this risk; the suggested comparison is an engineering check, not a guarantee of statistical fairness. [1]
The optimization tutorial describes cases where unordered delivery may reduce head-of-line waiting, while also warning that it might not improve average throughput. Measure the workload you actually have and keep the option disabled if the gain is unclear. These sources do not promise deterministic training from FIFO alone; other random operations and worker behavior still affect reproducibility. [1, 2]