Include model revisions in inference cache keys

Hidden model state does not distinguish cached calls

A memoized inference wrapper can return an old result after its active model changes if the cache key contains only the request text. Python lru_cache stores results for function arguments. Reading a mutable model-selection setting inside the function body does not make that setting an argument to the cached call. [1]

The fixture uses a deterministic arithmetic stand-in for two model revisions. Revision r1 multiplies text length by two; r2 multiplies it by three. A text-only cached call for ab first returns four. After the active revision changes to r2, the same call still returns four although the new expected result is six. The backend counter stays at one, confirming that the cached result avoided recomputation.

References: [1]

Pass revision identity into the cached computation

The corrected wrapper receives both revision and text. Calls for r1 and r2 now produce four and six in separate entries. Repeating each call reuses its corresponding result even though the external active setting remains r2. The revision-specific backend uses a fixed mapping, so the revision argument determines which stand-in computation runs.

Run python examples/versioned_inference_cache.py. Assertions verify the stale text-only hit, both revision-specific values and the backend call count. Cache statistics show two hits, two misses and two entries for the corrected wrapper. These are observed single-threaded calls, not latency measurements or a guarantee that concurrent misses execute only once.

Cache arguments must be hashable. The example uses revision and text strings; Python documents mutable lists and dictionaries as unhashable. An attempted list request raises TypeError before the backend counter changes. Hashability makes a key usable, but does not establish that it captures every dependency of a computation. [1, 2]

Executed with Python 3.12.14 using only the standard library. No model or inference service was loaded. Exact stdout and source-byte SHA-256 are preserved in private evidence.

"""Make a fixed model revision part of a memoized inference request."""
from functools import lru_cache


# Deterministic stand-ins for two immutable model artifacts; no model is loaded.
coefficients = {"r1": 2, "r2": 3}
active = {"revision": "r1"}
calls = {"implicit": 0, "explicit": 0}


@lru_cache(maxsize=8)
def implicit_revision(text):
    calls["implicit"] += 1
    return coefficients[active["revision"]] * len(text)


@lru_cache(maxsize=8)
def explicit_revision(revision, text):
    calls["explicit"] += 1
    return coefficients[revision] * len(text)


first = implicit_revision("ab")
assert first == 4
active["revision"] = "r2"
stale = implicit_revision("ab")
expected = coefficients["r2"] * len("ab")
assert stale == 4 and expected == 6 and calls["implicit"] == 1
old = explicit_revision("r1", "ab")
new = explicit_revision("r2", "ab")
assert (old, new) == (4, 6)
assert explicit_revision("r2", "ab") == 6
assert explicit_revision("r1", "ab") == 4
stats = explicit_revision.cache_info()
assert (stats.hits, stats.misses, stats.currsize) == (2, 2, 2)
assert calls["explicit"] == 2
before = calls["explicit"]
try:
    explicit_revision("r2", ["ab"])
except TypeError:
    pass
else:
    raise AssertionError("unhashable request unexpectedly accepted")
assert calls["explicit"] == before
implicit_revision.cache_clear()
refreshed = implicit_revision("ab")
assert refreshed == 6 and calls["implicit"] == 2

print("text-only cache before/after revision change:", first, stale)
print("expected result under new revision:", expected)
print("explicit revision results:", old, new)
print("versioned cache hits/misses/entries:", stats.hits, stats.misses, stats.currsize)
print("versioned backend calls:", calls["explicit"])
print("text-only result after cache_clear:", refreshed)
print("unhashable request rejected before backend: True")

Recorded output

text-only cache before/after revision change: 4 4
expected result under new revision: 6
explicit revision results: 4 6
versioned cache hits/misses/entries: 2 2 2
versioned backend calls: 2
text-only result after cache_clear: 6
unhashable request rejected before backend: True

References: [1], [2]

Treat revision names as stable identities

A revision key must identify fixed model behavior. Reusing the same label after replacing its artifact recreates the original problem. Where they affect results, tokenizer identity, preprocessing version and inference settings also belong in the application's cache contract. Passing a revision string while still consulting an unrelated mutable active model inside the backend would not repair that contract.

The fixture also clears the text-only cache and then obtains six. Python exposes cache_clear for invalidation, but that demonstrates only sequential invalidation here. It does not test coordinated clearing during a concurrent model swap. Both caches are bounded to eight entries. This example tests key completeness with fixed scalar outputs; distributed caches, stochastic generation and model-loading lifecycle remain separate concerns. [1]

References: [1], [2]

Sources

1. Python functools.lru_cache: argument-based caching, statistics and invalidation
2. Python glossary: hashable keys and immutable built-in objects