The Data-Protection Problems Nobody Warns You About When You Put AI in Front of Personal Data
Most teams shipping AI features today discover their hardest privacy problems after launch — not in the model, but in the plumbing around it.
We spent time turning a detailed data-protection review of an AI platform into a set of lessons, and almost none of them were industry-specific. They’re just the things that go wrong when software touches personal data and an AI agent sits in the middle. Here’s what we’d want any product, engineering, or compliance leader to know before they build.
1. Decide whether you’re a “controller” or a “processor” — and accept you’re probably both
Under GDPR and most modern privacy laws, the company that decides why and how personal data is processed is a controller; the one that processes on someone else’s instructions is a processor. Most platforms wear both hats at once. A SaaS product is a processor for the customer data clients load in — but a controller for its own accounts, its sales contacts, and any independent decision it makes (fraud scoring, risk screening, analytics). The moment your AI does something you decided to do, you’ve likely become a controller for that activity.
Transferable lesson: map every category of personal data you touch to a role before you map it to a database table.
2. The “right to be deleted” is a systems-design problem, not a database flag
Almost everyone implements deletion as a soft delete. That isn’t erasure. Real erasure has to reach every place a person’s data came to rest: primary records and child tables, search indexes and caches, vector databases / embeddings used for retrieval-augmented generation (the most commonly forgotten), audit logs (usually pseudonymize rather than delete), backups, and every sub-processor you forwarded data to. And if the same individual exists under multiple customers, you can only erase them when no one still has a lawful reason to keep their data — so deletion has to be reasoned at the level of the human being, not the account.
Transferable lesson: treat erasure as a first-class service with a real propagation design, not a checkbox bolted on later.
3. Some data you legally cannot delete — and that has to override your delete button
The same regulations that grant a right to erasure also create mandatory retention obligations — financial records, tax documents, regulatory filings. When a deletion request collides with a statutory retention requirement, retention wins for that specific data. Your erasure engine has to be smart enough to delete what it can while holding what the law requires, and explain the distinction to the data subject.
4. Never build a legally-required process on top of consent
Consent is withdrawable — that’s the whole point of it. So any processing you are legally obligated to perform — regulatory reporting, sanctions screening, mandatory filings — must rest on a legal basis that can’t be yanked away (“legal obligation” or “public interest”), not on the user clicking “I agree.” This looks like an engineering choice but is actually a legal one; get it ratified by counsel.
5. Redact personal data before it reaches the model — as a platform guarantee
The single best AI-privacy pattern we saw was a gateway that strips sensitive fields — government IDs, dates of birth, payment numbers — out of prompts before they’re ever sent to a model provider. Done at the gateway, it’s data minimization by design: every feature inherits it, and no individual developer can forget. The same layer is the natural home for data residency (keeping regulated-customer data on in-region endpoints), data classification, and tool filtering (limiting which actions an agent can take with which data). The principle underneath is privacy by design — now codified in ISO 31700-1:2023.
Transferable lesson: the boundary between your system and any third-party model is the most valuable place to put privacy controls. Centralize them there.
6. Encrypt the crown-jewel identifiers at the field level — not just “at rest”
Disk encryption protects you if someone steals a hard drive. It does nothing if someone gets a database connection. Your highest-sensitivity identifiers — passport and national ID numbers, anything that uniquely pins a person — deserve application-level field encryption or tokenization, so they’re unreadable even to someone who can query the table. (A related, embarrassingly common finding: secrets and license keys checked into source control. Move them to a managed secret store.)
7. If an AI makes a decision that affects a person, a human has to be able to step in
Where an automated system produces an outcome with a real effect on someone — approving, denying, flagging, scoring — privacy law gives that person the right not to be subject to a solely automated decision. In practice you need three things: human review of adverse outcomes (a real escalate path, not a rubber stamp), a right to contest the decision, and meaningful information about the logic. If you’ve already built human-in-the-loop review queues and you log your agents’ reasoning, you’re most of the way there.
8. Cross-border data transfer is where AI SaaS quietly accumulates the most risk
The instant data leaves its home jurisdiction — to a US-hosted model provider or cloud region — you’re in transfer territory, among the most legally unsettled areas right now. Defensive posture: keep a transfer register, run transfer impact assessments, keep contractual fallbacks (standard contractual clauses) in place even under a current adequacy framework, and default regulated customers to in-region endpoints — using the residency gateway control above as the technical enforcement.
9. The AI Act is a second regulator on top of privacy law — and its fines are bigger
In the EU, the AI Act layers obligations on top of GDPR, with penalties that can exceed it. The phased timeline is already biting: AI-literacy obligations, transparency duties (people must be told when they’re talking to an AI), and high-risk-system requirements with hard deadlines. The encouraging part: the logging, oversight, and human-review machinery good privacy engineering already demands produces most of the evidence the AI Act wants. Build it once, satisfy both.
10. The rules are still being written — so build on a standard, not a guess
The EU AI Act is only the most visible front. In the US, with no comprehensive federal law, states have become the regulators — a fast-moving patchwork (Texas’s TRAIGA, California’s frontier-model rules, New York’s RAISE Act, and Colorado, which passed, delayed, then repealed-and-replaced its statute before it ever took effect). Which rules reach you are legal determinations, not engineering ones — consult counsel. What you can do while the law settles is build on certification-ready standards that stack: ISO/IEC 27701 for a Privacy Information Management System, and ISO/IEC 42001, the first AI management system standard. The 2025 edition of 27701 is designed to integrate with 42001, so you can run them as one coherent governance system.
11. Write down what you do — the paperwork is part of the product
“Accountability” in privacy law is documentation you can produce on demand: records of processing, a data protection impact assessment for high-risk processing, data-processing agreements with customers and sub-processors plus a public sub-processor list, layered privacy notices (including disclosure that AI is involved), a retention schedule naming statutory minimums, and runbooks for breach notification and data-subject requests. In a regulator’s eyes — and a customer’s security review — the absence of these documents is the finding, regardless of how good your engineering is.
The pattern underneath all of this
The most interesting takeaway from the review we worked through: the AI-specific controls were the strongest part of the system — redaction before model calls, residency enforcement, classification, tool filtering, deterministic quality gates, human review for consequential decisions. The gaps were almost entirely in the unglamorous, data-subject-centric lifecycle: real erasure, retention automation, the ability to assemble everything you hold about one person, field-level protection of crown-jewel IDs, and the governance paperwork.
That inversion is the real lesson for anyone putting AI in front of personal data. The model is the part everyone scrutinizes. The risk lives in the plumbing — in deletion that doesn’t reach the vector store, a passport number sitting in plaintext, not being able to say whether you’re a controller or a processor for a given data flow. Get the boring parts right, and the AI parts turn out to be the easy half.
This article is general information, not legal advice. Which obligations apply to you depends on where your users sit, what your AI decides, and your sector — consult qualified counsel.