Make file-based ML splits repeatable

A seed cannot repair a changing input order

Python documents that `Path.iterdir()` yields children in arbitrary order and that `Path.glob()` returns matches in no particular order. If a training script takes the first part of that list for training and the remainder for validation, a different enumeration can assign different files to each split even when the split code itself has no randomness. [1]

The same issue can appear when a seeded random split consumes the file list: a seed controls the random choices, but those choices still refer to positions in the list. If the positions now name different files, the resulting membership can change. That is a consequence of applying deterministic choices to an unstable sequence, rather than a claim about a particular ML library.

References: [1]

Sort the same file set before using positions

For a fixed set of paths, sorting gives the split code a defined order instead of inheriting directory enumeration order. The Python `pathlib` documentation itself demonstrates sorting glob results and says to sort when a specific order is needed. [1]

The small executable below uses two synthetic enumeration orders and splits each list at the same position. It shows that the unsorted membership changes while sorting these identical file names produces the same two groups. It does not read a directory or exercise an ML framework; the example isolates the ordering step described by the documentation.

Executed with Python 3.12.14; bounded synthetic list example with no filesystem access. Exact stdout and source-byte SHA-256 are preserved in private evidence.

# Synthetic file identifiers: filesystem iteration order is not demonstrated here.
orders = [
    ["sample-03.npy", "sample-01.npy", "sample-04.npy", "sample-02.npy"],
    ["sample-02.npy", "sample-04.npy", "sample-01.npy", "sample-03.npy"],
]

for index, files in enumerate(orders, start=1):
    unsorted_split = (files[:2], files[2:])
    stable_files = sorted(files)
    stable_split = (stable_files[:2], stable_files[2:])
    print(f"input order {index}: {files}")
    print(f"position split {index}: train={unsorted_split[0]}, validation={unsorted_split[1]}")
    print(f"sorted split {index}: train={stable_split[0]}, validation={stable_split[1]}")

print(f"sorted splits match: {sorted(orders[0])[:2] == sorted(orders[1])[:2] and sorted(orders[0])[2:] == sorted(orders[1])[2:]}")

Recorded output

input order 1: ['sample-03.npy', 'sample-01.npy', 'sample-04.npy', 'sample-02.npy']
position split 1: train=['sample-03.npy', 'sample-01.npy'], validation=['sample-04.npy', 'sample-02.npy']
sorted split 1: train=['sample-01.npy', 'sample-02.npy'], validation=['sample-03.npy', 'sample-04.npy']
input order 2: ['sample-02.npy', 'sample-04.npy', 'sample-01.npy', 'sample-03.npy']
position split 2: train=['sample-02.npy', 'sample-04.npy'], validation=['sample-01.npy', 'sample-03.npy']
sorted split 2: train=['sample-01.npy', 'sample-02.npy'], validation=['sample-03.npy', 'sample-04.npy']
sorted splits match: True

References: [1]

Keep the split stable as the dataset evolves

Sorting is useful when the file set stays the same, but inserting or removing a path can move later entries across a positional split boundary. For recurring experiments, save the file-to-split assignment as a versioned manifest, or derive membership from stable sample identifiers using a documented rule. Those are engineering recommendations built on the ordering behavior above; Python's directory APIs do not define an ML split format. [1, 2]

Also make the path comparison rule explicit if filenames may differ by case or represent the same item through different relative paths. The important check is not only whether the random seed is fixed, but whether the exact input identities and their order are recorded alongside the run. Do not treat sorting alone as protection against dataset changes or duplicate samples.

References: [1], [2]

Sources

1. Python pathlib iteration and glob ordering
2. Python os.listdir ordering