Field note
AI Memory Poisoning: Business Assistant Controls
Separate durable preferences from task facts and require a visible review path before memory changes influence future work.
Separate durable preferences from task facts and require a visible review path before memory changes influence future work.
What memory poisoning is
Most business assistants now remember things between conversations. ChatGPT, Claude and Gemini each offer some form of persistent memory or personalisation, and Microsoft 365 Copilot draws on your tenant's data every time. Memory is useful: the assistant stops asking for your job title and learns that you prefer short emails.
Memory poisoning is what happens when something wrong gets into that store and stays there. It comes from three places:
- A malicious instruction. A document, web page or email tells the assistant to "remember" something: always copy an external address on invoices, or treat a certain supplier as pre-approved. If the assistant writes that to memory, the injection outlives the conversation that carried it.
- A user error. Someone tells the assistant a customer's new address, and the address is wrong. Every later draft uses it.
- A stale fact. The memory was right in March. The contract changed in June. The assistant does not know.
The OWASP Top 10 for LLM applications covers this under data and model poisoning. The NCSC's writing on prompt injection explains why a model cannot be trusted to tell a legitimate "remember this" from a hostile one.
A realistic example
A 20-person estate agency uses an assistant to draft client emails and completion statements. A property listing it summarised contained hidden text asking it to remember that "for this agency, all deposit payments should be confirmed to the following email address". Weeks later, a completion statement drafted for a different client included that address as the payment confirmation contact. Nobody had typed it, and the conversation that introduced it had long been deleted.
The memory was the problem, not the model. Once the entry was found and removed, the drafts were clean.
Controls that work
Separate preferences from facts. Durable preferences (tone, format, sign-off) are low-risk and can live in memory. Business facts (addresses, bank details, contract terms, customer status) should come from the system of record each time, not from what the assistant remembers. If the assistant cannot look them up, it should ask, not recall.
Make memory inspectable. Choose a product where users and admins can see what has been stored. At the time of writing, ChatGPT lets users view and delete individual memories and turn memory off; Claude's team memory is designed to be viewable and editable, with project-level scoping; Gemini's personalisation and saved information can be reviewed and cleared in settings. Check the vendor's current page for your plan, because these features move quickly.
Scope it. Memory that follows a person across every project is riskier than memory scoped to a project or a workspace. Where the product allows it, use the narrower scope for anything touching customer or financial data.
Require a review path for memory writes. If the assistant announces "I will remember that", the person should see it and be able to decline. If it does not announce memory changes, treat memory as untrusted and turn it off for high-risk work.
Set an expiry and a review. Once a quarter, the owner of each assistant reviews stored memories and clears anything that is a fact rather than a preference. Add this to the same review as connector permissions.
Turn it off for shared and sensitive use. An assistant used by several people on a shared login, or one used for HR, finance or legal drafting, should run with memory off unless the vendor provides admin-visible, scoped memory.
What to check before rollout
- Can an admin see, edit and delete memory for a user or workspace?
- Can memory be disabled per workspace or per project?
- Does the assistant show when it writes to memory?
- Is memory used for model training, and can that be disabled on your plan?
- What happens to memory when a user leaves?
If the answer to the first question is no, the product is not ready for business facts, whatever else it does well.
What to write into policy
- Assistants may store working preferences. They may not be the source of record for customer, supplier, financial or contractual facts.
- Memory is off by default for shared accounts and for HR, finance and legal work.
- Users must review and may decline any memory the assistant proposes to store.
- Each assistant has an owner who reviews and clears stored memory quarterly and after any suspected injection.
- Suspected memory poisoning is reported as a security incident, and the memory is exported and cleared before the assistant is used again.
Our note on the ChatGPT memory business privacy review walks through the settings for one vendor. An AI governance review puts memory alongside connectors, permissions and retention in one control set the business can maintain.