Disjoint train and test rows can still share the same entities

Match the split boundary to the deployment question

When the evaluation question is performance on previously unseen users or devices, separating row indices is not enough. Several rows can belong to one entity. Placing different observations of that entity on both sides permits entity overlap even when no individual row is repeated. The example below audits that condition directly; it does not estimate how much any model score would change.

Use the entity identifier as grouping information when that is the required boundary. scikit-learn provides GroupShuffleSplit for randomized partitions based on supplied groups, and GroupKFold for cross-validation with non-overlapping groups. GroupKFold assigns each group to a test fold exactly once across its folds. [1, 2]

References: [1], [2]

Audit both row coverage and entity overlap

The six synthetic rows belong to groups a, b and c, with three, two and one observations respectively. The initial split has three training indices and three test indices. Those index sets are disjoint and cover every row, yet entities a and b occur in both partitions. An index-only assertion passes while the entity-boundary check fails.

Run python examples/group_holdout_boundary.py. Its corrected partition explicitly holds out group c and places all remaining groups in training. The helper is a small standard-library illustration of the boundary, not an implementation or execution of either scikit-learn splitter. It rejects an empty holdout, unknown groups and a selection leaving no training group.

Assertions cover the exact corrected indices and exercise each of the three possible one-group holdouts. Every case must preserve full row coverage, disjoint row indices and an empty intersection between train and test entity sets. This makes the audit useful even if a different splitter later supplies the indices.

Executed with Python 3.12.14 using only the standard library. Exact stdout and source-byte SHA-256 are preserved in private evidence.

"""Audit the entity boundary of a small, explicitly chosen holdout."""
from fractions import Fraction


def entity_overlap(groups, train, test):
    return {groups[i] for i in train} & {groups[i] for i in test}


def hold_out(groups, test_groups):
    known = set(groups)
    chosen = set(test_groups)
    if not chosen or not chosen < known:
        raise ValueError('test groups must be a nonempty proper subset')
    train = [i for i, group in enumerate(groups) if group not in chosen]
    test = [i for i, group in enumerate(groups) if group in chosen]
    return train, test


groups = ['a', 'a', 'a', 'b', 'b', 'c']
row_train, row_test = [0, 3, 5], [1, 2, 4]
assert set(row_train).isdisjoint(row_test)
assert set(row_train + row_test) == set(range(len(groups)))
assert entity_overlap(groups, row_train, row_test) == {'a', 'b'}
print('row split shares row indices:', False)
print('row split shares entities:', sorted(entity_overlap(groups, row_train, row_test)))

train, test = hold_out(groups, {'c'})
assert train == [0, 1, 2, 3, 4] and test == [5]
assert entity_overlap(groups, train, test) == set()
print('group holdout indices:', train, test)
print('held-out group fraction:', str(Fraction(1, len(set(groups)))))
print('held-out row fraction:', str(Fraction(len(test), len(groups))))

# Every possible one-group holdout keeps its complete entity on one side.
for group in sorted(set(groups)):
    train, test = hold_out(groups, {group})
    assert set(train).isdisjoint(test)
    assert set(train + test) == set(range(len(groups)))
    assert not entity_overlap(groups, train, test)
    assert all(groups[i] == group for i in test)
print('all three one-group holdouts: passed')
rejected = 0
for invalid in (set(), {'unknown'}, set(groups)):
    try:
        hold_out(groups, invalid)
    except ValueError:
        rejected += 1
assert rejected == 3
print('invalid group selections rejected:', rejected)

Recorded output

row split shares row indices: False
row split shares entities: ['a', 'b']
group holdout indices: [0, 1, 2, 3, 4] [5]
held-out group fraction: 1/3
held-out row fraction: 1/6
all three one-group holdouts: passed
invalid group selections rejected: 3

References: [1], [2]

Count groups and rows separately

Holding out c selects one third of the groups but only one sixth of the rows. Unequal group sizes make those fractions different. GroupShuffleSplit documents that its train_size and test_size parameters refer to groups, with fractional test_size rounded up to a whole number of groups. Inspect the resulting row counts instead of assuming they match the requested group fraction. [1]

Also distinguish separation within a split from reuse across repeated splits. GroupShuffleSplit can reuse test samples across its randomized splits; GroupKFold has a different across-fold guarantee. Neither grouping choice automatically enforces chronology or class balance. For this fixture, entity IDs are already correct and complete; defining those IDs and choosing the evaluation target remain data-design decisions. [1, 2]

References: [1], [2]

Sources

1. scikit-learn GroupShuffleSplit: group-based partitions and split sizes
2. scikit-learn GroupKFold: non-overlapping groups across folds