ML Foundations
Why PyTorch IterableDataset workers can repeat your samples
With multi-process loading, each IterableDataset worker gets a dataset replica. Configure each replica to read a distinct shard and check what happens to the tail.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
Why adding workers can repeat an iterable
A map-style dataset gives the loader keys to fetch. An IterableDataset instead supplies the items itself, which is useful for streams and other sources that do not naturally expose random-access indices. When num_workers is positive, PyTorch gives each worker its own replica of the dataset object. If every replica starts reading the same iterable from the beginning, the workers can return the same samples more than once. PyTorch documents this behavior directly, including an example where two workers yield each item twice. [1]
This is an easy mistake to miss because the training loop still receives batches and the process may look healthy. More workers can then mean more repeated observations rather than more unique data. The first check is therefore about sample identity and coverage, not whether the loader starts successfully.
References: [1]
Give each worker a distinct slice
Inside an iterable dataset, torch.utils.data.get_worker_info() returns the current worker's ID and the total worker count; it returns None when loading happens in the main process. PyTorch's documented example uses those values to assign each worker a non-overlapping interval. [1, 2]
For a finite indexed sequence, a simple mental model is to assign worker i the positions i, i + N, i + 2N, where N is the number of workers. The example below compares identical unconfigured replicas with a strided partition. It is a small Python simulation of that assignment rule, not a PyTorch integration test or a recommendation to materialize a large stream as a list.
Executed with Python 3.12.14; captured output and environment are recorded in private research evidence.
from itertools import chain
items = list(range(3, 11))
worker_count = 3
# Each IterableDataset replica sees the full stream unless it is configured.
naive = [items[:] for _ in range(worker_count)]
# Illustrative finite-sequence shard: worker i receives i, i+N, i+2N, ...
shards = [items[worker_id::worker_count] for worker_id in range(worker_count)]
merged = list(chain.from_iterable(shards))
print(f"naive worker outputs: {naive}")
print(f"sharded worker outputs: {shards}")
print(f"all items retained once: {sorted(merged) == items and len(merged) == len(set(merged))}")
Recorded output
naive worker outputs: [[3, 4, 5, 6, 7, 8, 9, 10], [3, 4, 5, 6, 7, 8, 9, 10], [3, 4, 5, 6, 7, 8, 9, 10]] sharded worker outputs: [[3, 6, 9], [4, 7, 10], [5, 8]] all items retained once: True
Check tails and the real source
A partition can be disjoint and still be wrong if it drops the final records. For a finite source, compare the union of worker outputs with the original sample IDs and confirm there are no duplicates; test sizes that do not divide evenly by the worker count. PyTorch also notes that drop_last on an iterable dataset with multiple workers drops the last incomplete batch of each worker's replica, so tail behavior is per replica in that setting. [1]
The demonstration only covers one finite sequence and a strided split. A file reader, message stream, or remote source needs a partition operation that the source actually supports; restarting the same stream in every process and filtering after the fact may be wasteful or may not be safe. For distributed training, worker-local IDs alone are not a complete global partition plan. Treat the code here as a way to check the basic uniqueness-and-coverage invariant, then validate the source-specific and process-wide assignment in the loader you actually run.
References: [1]