Isolate Semantic Answer Caches by Permission

A semantic cache can reuse an earlier answer when a new question has a similar meaning. That is useful for repeated public enquiries. It becomes more complicated when an assistant answers from customer records, staff policies or documents with different access rights.

The same words can legitimately require different answers. “What is our renewal price?” depends on the account. “Who can approve this?” depends on the process and its current version. A useful cache design must represent those differences before it compares questions.

Decide what is eligible

Begin with an explicit allowlist of cacheable tasks. Public opening hours from an approved source might qualify. A personalised account balance, current stock promise or answer based on a private document usually needs a narrower design or a complete cache bypass.

Separate answer caching from a provider's prompt caching. This article concerns reusing the answer itself. Saving work on a repeated prompt prefix does not imply that an old answer should be returned to another caller.

Document the conditions under which an answer stays valid: its source revision, relevant date, user entitlement, locale and task configuration. If you cannot express an important condition in the cache design, leave that task outside the cache.

Partition first, compare meaning second

Azure API Management's semantic lookup policy provides vary-by expressions for separating cache entries and a vector proximity threshold for similarity. These are different jobs. Tightening similarity does not establish who may receive an answer.

For an application serving multiple organisations, derive the organisation identifier from validated server-side identity. Add the relevant permission boundary, such as a specific user or a reviewed entitlement group. Do not let a caller choose their tenant partition with an unchecked request field.

Other useful partition inputs can include the source release, prompt version and output format. Keep them in a documented configuration so a deployment cannot accidentally omit one. Avoid assuming a group name is a permanent description of membership: a person's rights can change while an entry remains cached.

The existing connector oversharing audit explains why source permissions matter. An answer cache adds another place where those rights must still hold.

Handle access changes explicitly

Microsoft's Azure AI Search security trimming guidance distinguishes an application-supplied security filter from authentication. Matching a principal string is not proof that the caller owns that identity. A cached response can also bypass the retrieval query where the filter would normally run.

Choose how access changes invalidate existing answers. Options include a short validity period for suitable content, a permission revision in the cache partition, or checking the caller's entitlement against the answer's recorded sources before returning it. The right choice depends on how quickly rights must take effect and what metadata the application retains.

Do not rely solely on a lengthy time to live for sensitive material. If an employee moves teams, yesterday's entitlement should not survive as an unexplained exception inside a response cache. Equally, clearing a cache after every harmless profile change may remove its value. Define the events that affect access to the actual source content.

Treat expiry as an operational decision

The semantic store policy uses a duration in seconds. Its documentation also recommends protecting the backend against excess traffic when cache access fails. Plan both content freshness and the extra model load when the cache is empty.

Use source version changes to retire answers derived from old material. Record enough provenance to identify the affected entries, or switch the namespace for that content release. A timer alone cannot know that a price list was corrected five minutes after publication.

Give operators a bypass switch and a scoped invalidation procedure. Keep these separate from deleting the source data. Disabling a faulty cache should send authorised requests through normal processing, subject to normal capacity controls.

A small test set that exposes the boundary

Use synthetic organisations with deliberately conflicting facts. Give both the same question and different renewal dates. Then repeat these checks:

• Ask the first organisation's question from the second organisation's account.
• Remove a user's access after an answer has been cached.
• Change a source revision while keeping the wording of the question unchanged.
• Ask nearly identical questions whose dates or account identifiers differ.
• Make the cache unavailable and observe backend traffic limits.

Inspect whether the request hit a cache, which partition it used and which source revision supported the answer. Log identifiers and decisions rather than copying private answer bodies into routine telemetry.

Roll out one eligible task first. Compare accepted answers and boundary failures as well as the hit rate. A cache that produces fewer model calls is only useful when the reused answers remain correct for the people receiving them.

References

• Microsoft, LLM semantic cache lookup policy. Accessed 17 October 2026.
• Microsoft, LLM semantic cache store policy. Accessed 17 October 2026.
• Microsoft, Security trimming in Azure AI Search. Accessed 17 October 2026.