Inference
Check category membership before using searchsorted IDs
NumPy searchsorted can turn an unseen label into a plausible but incorrect category ID. Validate membership against the fixed vocabulary before encoding.
Evidence and testing
Review record: Tensor Almanac automated evidence gate (no independent human review). Example versions, recorded outputs and limitations appear below.
An insertion position does not prove a match
A compact categorical encoder may use np.searchsorted to turn string labels into vocabulary positions. That function finds where a value could be inserted while preserving order. It does not require an equal value to be present, and its result can equal the vocabulary length. Treating the result as a known category ID therefore needs an additional check. [1]
The executed vocabulary is ant, cat, dog. The unknown strings aardvark, cow and zebra produce positions zero, two and three. The first two are valid array indices but name ant and dog, not the supplied strings. Only checking that an index is in range misses those two wrong assignments. The third position is beyond the vocabulary and cannot index an existing category.
References: [1]
Validate the fixed vocabulary and reject unknown labels
The helper first requires one-dimensional Unicode string arrays and a nonempty, sorted, unique vocabulary. It then applies np.isin to every input label. NumPy documents this function as returning a Boolean membership array with the same shape as the labels. If any entry is false, the helper raises before returning category IDs. [2]
Run python examples/categorical_lookup_membership.py. Known labels dog, ant, cat and ant encode to IDs two, zero, one and zero. Indexing the vocabulary with these IDs reconstructs the original sequence, including its repeated label. Eight rejection fixtures cover unknown labels before, between and after known categories, plus unsorted, duplicate, empty, two-dimensional and non-string inputs. Empty label arrays and a singleton vocabulary are also checked.
This example defines IDs by the sorted training vocabulary. Preserve that vocabulary with the model and reuse it during inference. Sorting an existing model vocabulary into a different order would change the meaning of its IDs; this helper deliberately rejects unsorted input rather than silently changing the mapping.
Executed with Python 3.12.14 and NumPy 2.3.5 on CPU. Exact stdout and source-byte SHA-256 are preserved in private evidence.
"""Reject unknown string categories before producing vocabulary IDs."""
import numpy as np
def encode_known(labels, vocabulary):
labels = np.asarray(labels)
vocabulary = np.asarray(vocabulary)
if labels.ndim != 1 or vocabulary.ndim != 1:
raise ValueError("one-dimensional inputs required")
if labels.dtype.kind != "U" or vocabulary.dtype.kind != "U":
raise ValueError("Unicode string arrays required")
words = vocabulary.tolist()
if not words or words != sorted(set(words)):
raise ValueError("vocabulary must be nonempty, sorted and unique")
if not np.all(np.isin(labels, vocabulary)):
raise ValueError("unknown category")
return np.searchsorted(vocabulary, labels)
vocabulary = np.array(["ant", "cat", "dog"])
unknown = np.array(["aardvark", "cow", "zebra"])
naive = np.searchsorted(vocabulary, unknown)
assert naive.tolist() == [0, 2, 3]
assert vocabulary[naive[:2]].tolist() == ["ant", "dog"]
known = np.array(["dog", "ant", "cat", "ant"])
encoded = encode_known(known, vocabulary)
assert encoded.tolist() == [2, 0, 1, 0]
assert np.array_equal(vocabulary[encoded], known)
assert encode_known(np.array([], dtype="U"), vocabulary).shape == (0,)
assert encode_known(["cat"], ["cat"]).tolist() == [0]
rejected = 0
for labels, words in (
(["aardvark"], vocabulary),
(["cow"], vocabulary),
(["zebra"], vocabulary),
(["ant"], ["cat", "ant", "dog"]),
(["ant"], ["ant", "ant", "cat"]),
(["ant"], np.array([], dtype="U")),
([["ant"]], vocabulary),
([1], vocabulary),
):
try:
encode_known(labels, words)
except ValueError:
rejected += 1
assert rejected == 8
print("unknown insertion positions:", naive.tolist())
print("in-range positions wrongly name:", vocabulary[naive[:2]].tolist())
print("known category IDs:", encoded.tolist())
print("known labels round-trip:", np.array_equal(vocabulary[encoded], known))
print("unknown/schema fixtures rejected:", rejected)
print("empty labels and singleton vocabulary: passed")
Recorded output
unknown insertion positions: [0, 2, 3] in-range positions wrongly name: ['ant', 'dog'] known category IDs: [2, 0, 1, 0] known labels round-trip: True unknown/schema fixtures rejected: 8 empty labels and singleton vocabulary: passed
Make the unknown-category policy explicit
Rejection is the chosen contract here. Some applications reserve an unknown ID, but that ID needs a defined meaning in the trained system. Clipping insertion positions or adding unseen labels during inference silently changes the encoding contract. A round-trip check on known labels plus explicit unknown-label fixtures makes that difference visible.
The helper performs exact string matching for small in-memory arrays. It does not normalize case, Unicode forms or whitespace, accept arbitrary Python objects, train an encoder or benchmark lookup speed. Membership testing and insertion search are separate operations here; no claim is made that this implementation is the fastest categorical encoder.