Inference
Dropped one-hot categories can collide with ignored unknowns
A dropped baseline and an unseen category can share an all-zero encoding. Test that collision before relying on inverse transforms or model inputs.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
Two input states can become one feature vector
OneHotEncoder with drop='first' removes the first category's output column. With handle_unknown='ignore', an unseen category contributes zeros to that feature's encoded columns. Combining these settings can make a known baseline and an unknown value indistinguishable in the encoded input. Neither setting alone explains the full collision. [1]
The fixture fixes the known vocabulary as blue followed by red. After dropping blue's column, blue becomes a single zero, red becomes a single one, and unseen green also becomes a single zero. A fixed score function using only this encoding consequently assigns blue and green the same score. This is information loss in the representation, not evidence about a fitted model's accuracy.
References: [1]
Compare dropped, full and flagged representations
Run python examples/onehot_unknown_collision.py. The standalone helper is a small standard-library encoding fixture, not a replacement implementation of scikit-learn. Assertions check all three input rows, their fixed scores, the full two-column representation, an appended unknown flag and two rejection cases.
Keeping both known-category columns distinguishes blue's (1, 0) from green's (0, 0). An alternative fixture keeps the dropped representation and adds an explicit membership flag: blue becomes (0, 0), red becomes (1, 0), and green becomes (0, 1). These alternatives preserve a distinction in the input; they do not establish that a downstream model has learned a useful response to unknowns.
A separate execution fitted actual scikit-learn encoders on blue and red. Their dropped and full transforms matched the fixture exactly. The strict unknown policy raised ValueError for green. The dropped transform emitted one warning, captured in the review; the example does not rely on warnings as its unknown-value detection mechanism.
Executed with Python 3.12.14 using only the standard library. Supplemental scikit-learn 1.8.0 transform and inverse-transform checks are recorded in private evidence. Exact stdout and source-byte SHA-256 are preserved.
"""A dropped baseline and an ignored unknown can share the same vector."""
def encode(values, categories, *, drop_first, ignore_unknown):
if len(categories) < 2 or len(set(categories)) != len(categories):
raise ValueError("at least two distinct category names required")
emitted = categories[1:] if drop_first else categories
rows = []
for value in values:
if value not in categories and not ignore_unknown:
raise ValueError("unknown category")
rows.append([int(value == category) for category in emitted])
return rows
def main():
categories = ("blue", "red")
queries = ("blue", "red", "green")
dropped = encode(queries, categories, drop_first=True, ignore_unknown=True)
assert dropped == [[0], [1], [0]]
# Same encoded input gives the same result for this fixed score function.
scores = [2 + 3 * row[0] for row in dropped]
assert scores == [2, 5, 2]
full = encode(queries, categories, drop_first=False, ignore_unknown=True)
assert full == [[1, 0], [0, 1], [0, 0]]
assert full[0] != full[2]
unknown_flags = [int(value not in categories) for value in queries]
augmented = [row + [flag] for row, flag in zip(dropped, unknown_flags, strict=True)]
assert augmented == [[0, 0], [1, 0], [0, 1]]
rejected = 0
try:
encode(queries, categories, drop_first=True, ignore_unknown=False)
except ValueError:
rejected += 1
try:
encode(queries, ("blue", "blue"), drop_first=True, ignore_unknown=True)
except ValueError:
rejected += 1
assert rejected == 2
print("query order:", list(queries))
print("drop-first, ignore-unknown rows:", dropped)
print("fixed score on encoded rows:", scores)
print("full one-hot, ignore-unknown rows:", full)
print("dropped rows plus unknown flag:", augmented)
print("strict-unknown / duplicate-vocabulary rejections:", rejected)
if __name__ == "__main__":
main()
Recorded output
query order: ['blue', 'red', 'green'] drop-first, ignore-unknown rows: [[0], [1], [0]] fixed score on encoded rows: [2, 5, 2] full one-hot, ignore-unknown rows: [[1, 0], [0, 1], [0, 0]] dropped rows plus unknown flag: [[0, 0], [1, 0], [0, 1]] strict-unknown / duplicate-vocabulary rejections: 2
Do not use an inverse transform to reconstruct discarded identity
Scikit-learn's preprocessing guide documents that an all-zero feature block maps back to the dropped category when a category was dropped, and to None otherwise. In the reference execution, inverse-transforming green's dropped encoding therefore returned blue. The full encoding returned None, not the original name green. Once distinct unknown names share the same vector, the encoded data cannot recover which name occurred. [2]
Record the fitted category order, dropped category and unknown policy together. If unknown counts matter, track membership before discarding raw identity. A flag that is always zero during fitting also provides no demonstrated learned effect for its unseen value. The helper covers one feature with at least two distinct string categories; missing values, infrequent-category grouping, multiple feature blocks and model retraining require their own tests.