The question “where is our data” usually comes up too late: when a customer sends a security questionnaire, when an auditor asks for the record of processing, or when someone realises the whole department is pasting contracts into a public chat. It is better to answer it before AI settles into the company for good. The answer is not complicated, but it requires decisions.
Three places where data ends up
The first is a public chat an employee uses on a personal account. Whatever they paste leaves the company and falls under the vendor’s terms, not your contract. In many free plans the content of conversations can be used to train models. The company has no processing agreement, no logs and no idea who sent what.
The second is an AI service in the cloud under a company agreement: a paid plan, API access or an assistant built into the office suite. Here there is a processing agreement, a processing region and usually a statement that data does not train models. Data still leaves the company, but in a way that can be described and defended.
The third is your own environment: a model running on a server in the company or in a data centre under your control. Data does not go outside. It costs more to start and needs someone to run it, but for some data it is the only way.
Most companies need the second and the third at the same time, and should ban the first. How to balance them is covered separately in your own AI model or the public cloud.
What GDPR and the AI Act say
GDPR does not forbid using AI. It requires the same as for any other processing: a legal basis, a processing agreement with the vendor, knowledge of where the data is processed, and the ability to show it in writing. A public chat on a private account meets none of these conditions. A service under a company agreement meets them, provided someone has read that agreement and recorded it in the register of processing.
The AI Act adds obligations depending on what AI is used for. For most office uses, such as summarising documents or writing emails, it mainly requires transparency and human oversight. For high-risk uses, such as assessing employees or credit decisions, it requires much more. Regardless of the category, the company must know which AI systems it uses. That cannot be done without written rules, which we describe under AI policy and the AI Act.
A separate matter is professional secrecy and sector requirements. A law firm, an accounting office, a medical practice or an entity covered by NIS2 has data that should not be sent to any cloud, however good the agreement. For them the third place, their own environment, is not an option but a requirement.
Questions worth asking the vendor
Before you sign up for an AI service, ask five things. Where the data is physically processed and whether an EU region can be enforced. Whether data from your prompts is used to train models and whether that can be switched off at the contract level, not just in account settings. How long the vendor keeps the content of prompts and responses. Whether you will get a GDPR-compliant processing agreement and a list of subprocessors. Whether there are logs showing who at your company sent what.
If any of these has no clear answer, it is not a service for customer data. It may be fine for writing job ads.
When a local model makes sense
A local language model is a model running on your server with a GPU card or in a data centre under your control. It is not as good as the largest public models, but the gap has been shrinking for two years, and for tasks such as document search, summarising, classification or answering questions from a knowledge base it is sufficient.
It makes sense in three situations. When data must not leave the company because of professional secrecy or regulation. When volume is high and the cloud bill grows faster than the benefit. When the application has to work without the internet or with very short response times, for example on a production line.
It does not make sense as the only solution in a small company that simply wants to write offers faster. There a cloud service under a company agreement and written rules are enough.
How to set it up in practice
Start with data classification, even the simplest: public, internal, confidential, customer data. Then assign each class an allowed place. Public and internal can go to a cloud service under agreement. Confidential and customer data only to your own environment, or not at all.
Next, take that decision out of employees’ heads and move it into the environment. If the company gives people one place to work with AI, where the rules are built in and prompts on sensitive data automatically go to the local model, nobody has to remember every day what is allowed. That is how we design environments for AI: servers and databases in the EU, access to models in the EU region and locally, logs and access control in one place.
Finally, logs. When a customer sends a questionnaire or an audit comes, the question is not “do you use AI” but “who, on which data and where”. The answer has to be in the system, not in a manager’s memory.
What to do this week
Check which AI tools your people use, including on private accounts. Write down three rules: what must not be pasted in, which tools are approved, who to ask when in doubt. Pick one service under a company agreement with an EU region instead of five private accounts. If you have data covered by secrecy or regulation, work out what your own environment would cost instead of assuming it is out of reach.