LLM Data Governance: The Questions Your Security Team Will Actually Ask

The feature was done. A support-summarization tool that read a customer's ticket history and drafted a reply, and in the demo it was genuinely good. Everyone in the room wanted it. Then it went to security review, and the first question was one sentence long: "So where does the customer data go?"

The engineer who built it started to answer, then stopped, because the honest answer was that a customer's entire support history, names, account numbers, the occasional pasted screenshot of a bank statement, was being packed into a prompt and sent to a third-party model API, logged in three places along the way, and retained for as long as nobody had thought to delete it. None of that was malicious. Nobody had thought about it at all. The feature stalled for a quarter while it was retrofitted for governance it should have been designed with, and the team learned the expensive way that the model was never the risky part. The data flow around it was.

This is the pattern with almost every enterprise LLM feature that dies in review. The model works. The prompt engineering is fine. What kills it is that someone shipped a data-processing system without treating it as one. An LLM feature is a pipeline that takes potentially sensitive data, sends it across a trust boundary, transforms it, logs it, and stores derivatives of it. Every one of those steps is a governance question, and the questions are predictable. If you answer them in the design instead of the review, the feature ships. If you wait to be asked, it stalls. Here are the questions, in the order security tends to ask them, and how to actually answer each.

"Where does the data go?"

This is the first question and the one everything else hangs on, so answer it with a diagram, not a sentence. Security is not asking for reassurance. They are asking you to trace every hop the data makes, and specifically the moment it crosses out of your trust boundary.

For most LLM features that boundary is the model API call. The instant you send a prompt to a third-party model provider, customer data has left your infrastructure and entered someone else's. That is not automatically a problem, but it is automatically a question, and the answer has several parts you need to have ready.

Which provider, and under what terms. There is a real difference between the consumer product and the enterprise API of the same vendor. Enterprise and API tiers typically come with a data processing agreement, a commitment not to train on your inputs, and defined retention, while the consumer tier often reserves the right to do the opposite. "We use the enterprise API with a signed DPA and training disabled" is a real answer. "We call the API" is not.

Where the inference physically runs. Data residency is a hard requirement in a lot of regulated contexts, and "the model is in some US region" is a failing answer for a European or Indian customer with localization obligations. You need to know, and be able to prove, which region serves the request, and route to an in-region endpoint when the customer requires it.

Whether the provider retains the prompt at all. Many providers retain API inputs for a window, for abuse monitoring, even when they do not train on them. For sensitive workloads there are usually zero-retention or short-retention options you have to explicitly opt into. Know your provider's default, and know whether you changed it.

The mistake that makes this whole section hard is discovering the data flow during the review instead of designing it. Before you write the feature, draw the map: where the data originates, every service it passes through, the exact point it crosses to the model provider, and everything derived from it that you keep. If you cannot draw that map, you cannot answer the first question, and you are not ready to ship.

"What are you actually sending the model?"

The second question narrows from the flow to the payload. Even with a spotless provider and a signed DPA, the volume and sensitivity of what you put in the prompt is its own risk, and the principle security applies is the oldest one in privacy: data minimization. Send the model the least it needs to do the job, not everything you happen to have.

In practice this means the prompt is a deliberately constructed, minimized object, not a dump of whatever record you had in hand. If the task is summarizing a support thread, the model needs the text of the conversation. It almost certainly does not need the customer's full account number, home address, or internal risk flags, even though all of those were sitting in the same record you pulled from. Every field you include that the task does not require is pure downside: more sensitive data across the boundary, more in the logs, more to explain in the review.

Where the task genuinely needs an identifier, the strong move is to redact or tokenize before the call and rehydrate after. You replace the real sensitive values with placeholders, send the model the placeholdered text, and swap the real values back into the output on your side. The model does useful work on the structure of the data without ever seeing the raw sensitive values.

import re

# Detect-and-mask before the prompt ever leaves your boundary.
# In production use a real PII detector; this shows the shape of the flow.
PATTERNS = {
    "EMAIL": r"[\w.+-]+@[\w-]+\.[\w.-]+",
    "CARD": r"\b(?:\d[ -]*?){13,16}\b",
    "SSN": r"\b\d{3}-\d{2}-\d{4}\b",
}

def redact(text):
    mapping = {}
    def swap(kind, value):
        token = f"[{kind}_{len(mapping)}]"
        mapping[token] = value
        return token
    for kind, pattern in PATTERNS.items():
        text = re.sub(pattern, lambda m: swap(kind, m.group(0)), text)
    return text, mapping

def rehydrate(text, mapping):
    for token, value in mapping.items():
        text = text.replace(token, value)
    return text

# safe_prompt goes to the model; mapping never leaves your process
safe_prompt, mapping = redact(raw_ticket_text)
model_output = call_model(safe_prompt)
final = rehydrate(model_output, mapping)

Two honest caveats, because security will raise them. Regex-based detection is a starting point, not a guarantee; real deployments pair it with a trained PII detector and treat detection recall as something to measure, not assume. And redaction can hurt output quality when the identifier actually mattered to the task, so it is a trade-off you make per field, not a blanket switch. The point is not that you must redact everything. It is that you have consciously decided, field by field, what the model sees and why, and can defend each choice.

"What are you logging, and who can read it?"

This is the question that catches the most teams by surprise, because the leak is not in the feature. It is in the observability you bolted onto the feature. You will log prompts and responses, because you cannot debug or evaluate an LLM system without seeing what went in and what came out. And the moment you do, every piece of sensitive data in those prompts is now sitting in your logging stack, which usually means a third-party observability vendor, replicated across regions, searchable by a broad slice of engineering, and retained on whatever default your log platform ships with.

I have watched a team pass the model-provider review cleanly and then fail the same review on their own logging, because the prompts they were carefully sending to a zero-retention model endpoint were being logged in full to a general-purpose observability platform with a ninety-day retention and org-wide read access. The sensitive data did not leak to the model provider. It leaked to Datadog, and to every engineer with a login.

So logging is a first-class governance surface, and it needs the same discipline as the model call. Redact before you log, using the same machinery you use before the model call. Separate sensitive telemetry from general telemetry, so prompt and response bodies go to a restricted, short-retention store with tight access control, while your normal metrics and traces stay where the whole team can see them. And make access to the sensitive store auditable, because "who can read the prompt logs" is a question with a real answer only if you built one.

# Two sinks, not one. Prompt/response bodies are sensitive telemetry
# and do not belong in the same place as latency and error-rate metrics.
telemetry:
  general:
    sink: observability-platform     # metrics, traces, error rates
    retention_days: 90
    access: engineering
  sensitive:
    sink: restricted-log-store       # redacted prompt/response bodies only
    retention_days: 14               # short by default
    access: llm-oncall               # tight, and every read is audited
    pii_redaction: enforced

The tell that a team has not thought about this is a single logging path that captures everything at the same retention with the same access. Splitting it is not hard, and it converts "the prompts are somewhere in our logs" into "redacted bodies live in a restricted store for fourteen days and every access is recorded," which is the difference between failing and passing this part of the review.

"How long do you keep it, and can you delete it?"

Retention and deletion are where LLM features run into a problem that ordinary CRUD apps do not, and it is worth understanding precisely, because a vague answer here fails immediately in any GDPR or DPDP context.

The ordinary part is straightforward: define a retention period for prompts, responses, and logs, make it as short as the feature can tolerate, and enforce it automatically rather than by intention. The hard part is the right to erasure. When a user exercises their right to be deleted, you have to delete their personal data everywhere it lives, and an LLM feature scatters derivatives of that data into places a normal delete does not reach.

Think through everywhere a single user's data ends up in a typical retrieval-augmented feature. The raw source records, which your existing deletion already handles. The prompt and response logs, which your new retention has to cover. Any cached model outputs keyed to that user. And, most easily forgotten, the embeddings in your vector store. Embeddings are derived from personal data and, under GDPR and similar regimes, are generally treated as personal data themselves, because they are tied to an individual and can leak information about the source. A vector embedding of someone's medical history is not anonymous just because it is a list of floats. So a deletion request has to reach the vector store too, which means you need to be able to find and remove every vector derived from a given user's data.

The design consequence is that everything derived from user data has to carry enough metadata to be found and deleted later. Every chunk you embed, every cached output, every log line gets tagged with the user or tenant it came from, so that erasure is a query, not an archaeological dig.

# Tag every derived artifact at write time so erasure is a lookup, not a hunt.
vector_store.upsert(
    id=chunk_id,
    embedding=embed(chunk_text),
    metadata={
        "user_id": user_id,        # so we can find and delete it later
        "tenant_id": tenant_id,    # also the isolation key, see next section
        "source_doc": doc_id,
        "created_at": now_iso,
    },
)

# Right-to-erasure becomes a bounded, verifiable operation
def erase_user(user_id):
    vector_store.delete(filter={"user_id": user_id})
    cache.delete_by_prefix(f"llm:{user_id}:")
    sensitive_logs.delete(filter={"user_id": user_id})
    # source-record deletion your existing pipeline already handles

If you cannot produce, on demand, a list of everywhere a given user's data and its derivatives live, you cannot honor a deletion request, and that is not a gap you want to discover when the request arrives with a regulatory deadline attached. Design the tagging in from the first embedding you write.

"Can one customer's data reach another?"

For any multi-tenant system this is the question with the highest blast radius, because the failure mode is not a privacy inconvenience, it is one customer seeing another customer's data, which in an enterprise contract is close to unrecoverable.

There are two places cross-tenant leakage hides in LLM systems, and both are easy to get wrong. The first is retrieval. If your feature retrieves context from a shared vector store, a query has to be constrained to the asking tenant's data at query time, enforced by the system, not by a prompt asking the model nicely to only use the right documents. A vector search that ranks by similarity alone will happily return tenant B's chunk to tenant A if it happens to be the closest match, and no amount of instruction in the prompt reliably prevents it. The enforcement has to be a hard metadata filter on the query.

# The tenant filter is not optional and it is not the model's job.
# It is a hard constraint applied before the search runs.
results = vector_store.query(
    embedding=embed(user_question),
    top_k=8,
    filter={"tenant_id": current_tenant_id},   # enforced, every query, no exceptions
)

The safer version of this is not one shared store with a filter but physical separation per tenant, where each tenant's data lives in its own namespace or index and a query can only ever be issued against the caller's own. A filter you can forget to apply is weaker than an isolation boundary you cannot cross by construction. For highly sensitive multi-tenant systems, per-tenant isolation is worth the overhead precisely because it removes the possibility of the forgotten filter.

The second hiding place is fine-tuning. If you fine-tune a model on customer data and then serve that model to other customers, you have baked one tenant's data into a model everyone shares, and models can and do regurgitate their training data. Fine-tuning on raw customer data across tenants is a contamination risk that no query-time control fixes, because the leak is inside the weights. If you fine-tune at all, do it on data you are cleared to use across all tenants, or isolate the fine-tuned model to the single tenant whose data trained it. This is exactly the kind of thing security will ask about specifically, because it is exactly the kind of thing teams do without thinking through where the data ends up.

"Who did what?"

For regulated use, the last question is accountability. When an LLM feature influences a decision that matters, a loan assessment, a medical summary, a support action taken on a customer's account, someone will eventually need to reconstruct what happened. What did the user ask, what context was retrieved, what did the model produce, and what action followed. In a regulated context that audit trail is not optional, and "the model decided" is not an acceptable account of a decision.

So decisions that carry weight get an immutable audit record: the input, the retrieved context, the model and version, the output, and the human action that resulted. This is distinct from your debugging logs, which are redacted and short-lived. The audit trail is a deliberate, access-controlled, long-retained record of consequential decisions, built to satisfy an auditor rather than an on-call engineer. It also quietly protects you, because when someone challenges a decision the system influenced, the difference between having that record and not having it is the difference between a defensible process and a shrug.

The related design principle security will push on is keeping a human meaningfully in the loop for anything consequential and reversible-only-with-effort. Not a rubber-stamp approval nobody reads, but a real checkpoint where a person owns the decision the model informed. The audit trail and the human checkpoint are the same idea from two directions: consequential automated outputs need an accountable owner and a reconstructable history.

"What comes back out?"

There is a governance surface on the output side too, and it is easy to forget because most of the worry goes into what you send. A model can emit personal data it should not, whether by surfacing something from the retrieved context that the user was not entitled to, or by stitching together an inference about a real person. It can also state something false about a real individual with total confidence, which under the accuracy principle in GDPR and similar regimes is its own kind of violation, not just a quality bug. The mitigations are unglamorous but real: filter outputs for the categories of data that should never reach this user, keep the retrieval constrained so the model cannot cite what it should not have seen in the first place, and for any output that asserts facts about a real person, make correction and challenge a supported path rather than a dead end. You govern the response with the same seriousness as the prompt, because to the person on the other end, the output is the product.

Governance is a design input, not a gate at the end

Every one of these questions has a good technical answer, and none of the answers is exotic. Map the data flow. Minimize and redact the payload. Split and restrict the logs. Tag derivatives so you can delete them. Enforce tenant isolation in code, not in the prompt. Keep an audit trail for decisions that matter. What they have in common is that they are all far cheaper to build in than to retrofit, and retrofitting is what teams do when governance shows up as a review at the end instead of a constraint at the start.

The support-summarization feature that stalled for a quarter did eventually ship, and the retrofitted version was not meaningfully different from what a governance-first design would have produced on the first try. The redaction, the split logging, the tenant filter, the retention policy, none of it made the feature worse. It just made it shippable, and it would have cost a fraction of the effort if the data map had been the first artifact instead of the last.

So before you build the next enterprise LLM feature, draw the map first. Trace where the data comes from, where it crosses your boundary, what the model sees, what you log, what you keep, and what you can delete. If you can answer the six questions above from that map, you will walk into the security review with the answers already built, and the feature will ship. The model is the easy part. It always was. The data around it is the actual work, and the teams that treat it that way are the ones whose AI features make it to production instead of dying in a conference room.