Tuesday morning at an insurer. Three colleagues in the same department open the same chatbot. The first asks it to reword a clumsy sentence from a newsletter. The second pastes in a complaint email to get a tidy reply drafted; at the bottom of the email are a name, a policy number and a bank account number. The third is on the train and summarises an internal report on her phone, through the free account she also uses at home.
Same tool, three different risks. Not because the chatbot behaves differently for each of them, but because most of the risk does not lie in the tool. It lies in what goes in, which account is used and what happens to the answer afterwards.
What decides the risk when the tool is the same?
An AI tool has properties that are fixed: where the vendor processes data, whether it trains on input, how long it keeps conversations. Those properties decide part of the risk, and the organisation assesses them once for each tool. How that works is explained in our guide to assessing the risk of an AI tool.
The rest is decided when you use it, and there are three variables. What you put in the prompt: a general question or a customer file. Which account you work under: the business account the organisation set up, or a free account that was already there. And what you do with the answer: read and adjust it, or forward it as is. Any of the three can still make a properly assessed tool high risk.
Three colleagues, the same tool
| Who | What goes in | Which account | What happens to the answer | Risk |
|---|---|---|---|---|
| Colleague 1: rewording a sentence | A sentence with no personal data | The department's business licence | Read and adjusted | Low |
| Colleague 2: answering a complaint | A customer's name, policy number and bank account | The department's business licence | Sent to the customer after a quick glance | Personal data processed that the task did not need |
| Colleague 3: summarising a report | An internal report | Free account on a personal phone | Pasted into an email | Confidential text under consumer terms, out of the organisation's sight |
The first colleague uses the tool the way the organisation assessed it. The second uses the same account but gives the tool data it did not need: a reply to a complaint is no better for including the bank account number. The third has in effect swapped the tool for a different one, because a free account on a personal phone is a different processing operation from the licence at the office, even with the same logo at the top.
What is in your own hands?
Two of the three variables, which is more than it often feels like.
What goes in. The task almost never needs the identifying details. A complaint can be answered with "the customer" instead of a name and without the policy number. A report can be summarised without the names of those present. The test to apply: if I would not email this text to a stranger outside the organisation, it does not belong in a chatbot either. The full list of what can and cannot go in is in our checklist of what you can and cannot share with AI.
What you do with the answer. A language model produces plausible text, not checked text. A date that is wrong, a condition that reads differently in the policy, a tone that does not suit the customer: the model does not notice; you do, provided you read it before you send. The answer is a draft. That applies to the first colleague as much as the second; the consequences of not reading are just greater with a letter to a customer.
Then there is the third variable, the account. That one is only half yours. You can choose to use the account the organisation set up instead of the free one that was already open. But whether that business account exists, and for which tool, is the organisation's decision. Why that decision matters more than the choice between brands is explained in our comparison of free and business AI accounts.
What is not in your hands?
The vendor's terms, whether a data processing agreement exists and the decision on which tools the organisation allows, allows with limits or would rather not see. That is the organisation's job, and the third colleague cannot be blamed for using the account she had on the train. If nobody has told her which tool is fine and where to look that up, the free account is the logical choice.
That is also why a list of allowed tools in an email from March achieves little by September. The decision has to appear when you open the tool, and it has to name the alternative. "Not this one, use that one" is an instruction you can act on. "Not this one" on its own ends the conversation, and the work carries on by phone.
What happened next at the insurer?
The department had allowed Copilot through its Microsoft licence. An inventory showed that ChatGPT, Perplexity and three specialist tools were also in use, two of which were not yet in the register. That is not an incident: it shows that a one-off list soon falls behind what people have started using in their work.
The organisation took a decision on each tool and made it visible while people work. Anyone who now opens the free chatbot sees that a decision exists and which tool is the alternative. Anyone who types a bank account number or a name into a text box sees it highlighted before the message goes, and chooses: mask, remove or leave it. The colleague is still the one who acts; the organisation makes sure the decision and the risk are visible at that moment.
