The data was bounded. The inference is not. The privacy governance built for the data does not follow the model into its inference capabilities.
Why This Matters Now
AI inference is the application of a trained model to new inputs to produce predictions, classifications, or outputs. The inference capabilities a model develops frequently extend beyond the data attributes explicitly included in training. A model trained on purchase history learns to infer demographic characteristics. A model trained on text learns to infer emotional state, political orientation, and health conditions.
These inference capabilities are not incidental. They are the source of much of AI's commercial value. They are also a privacy risk that is structurally different from the privacy risk of the training data itself. The inference risk is the risk of the model producing sensitive information about individuals that they never disclosed, did not know could be inferred, and have no right to control once the inference has been made.
The privacy risk of AI inference is the risk of learning about people without asking them. This risk was not created by the data collection. It was created by what the model learned from that collection.
The Governance Problem Beneath the Surface
Privacy impact assessments are well-suited for assessing the risks of data collection and storage. They are poorly suited for assessing the privacy risks of AI inference capabilities, because inference capabilities are emergent properties of model training that cannot be fully specified before the model is trained and deployed.
A DPIA completed before an AI system is deployed can document that the system processes certain data attributes for certain purposes. It cannot fully document what sensitive information the system will infer from inputs that appear non-sensitive, because the inference capabilities are determined by the model's learned representations and may only be discovered through post-deployment behavioral assessment.
What This Actually Means in Enterprise Practice
Special Category Data Can Be Inferred from Non-Special-Category Inputs
Under GDPR, special categories of personal data including health, racial or ethnic origin, political opinions, and sexual orientation require explicit consent or specific legal basis. AI systems that infer these categories from non-special-category inputs create special category processing that the original data collection did not contemplate.
An employer that uses an AI HR system to predict employee performance from communication patterns may inadvertently create inferences about health conditions, political views, or protected characteristics. The inputs are non-sensitive. The inferences are special category data.
Aggregate Data Produces Individual-Level Inferences
Statistical aggregate data about populations can be used to make highly specific inferences about individuals. A model that knows the statistical relationship between a postcode, shopping patterns, and health outcomes can infer with material probability what an individual with those characteristics is likely to be experiencing, without accessing any specific medical data.
Individual privacy is not protected by removing individual identifiers from population data when the model can reconstruct individual attributes from population-level patterns.
Inference Outputs Are Not Subject to Individual Rights in the Same Way
Data subjects have rights over personal data held about them. Inference outputs generated by a model are not stored as traditional personal data in most implementations. Whether inference outputs are subject to access rights, rectification rights, and erasure rights under GDPR is not clearly settled.
Third-Party Data Amplifies Inference Capability
AI models that combine first-party data with third-party data enrichment have inference capabilities that exceed either dataset alone. The privacy risk of the combined inference system is not the sum of the privacy risks of the individual data sources.
How Different Teams See This: Where They All Miss
Inference privacy risk governance requires assessing what the model produces from its inputs, not just what the model receives as inputs. Most privacy programs have built the second assessment and not the first.
Framework Control Reference
The specific control obligations most relevant to this topic. Use in governance discussions, vendor assessments, and audit responses.
These controls share a common requirement: the obligation is active, not declarative. Documenting alignment is not the same as demonstrating it.
The Enterprise Reality Gap
The enterprise AI inference privacy reality gap is between the privacy governance coverage organizations build for their AI systems' inputs and the privacy risks those systems create through their inference outputs. The inputs are assessed. The inference capabilities are not.
The most significant privacy risk in many AI systems is not what they process. It is what they can infer from what they process. Governance that addresses inputs and not outputs has governed half the risk.
Enterprise Scenario
The DPIA assessed the privacy risk of purchase history. It did not assess the privacy risk of health condition inferences the model would develop from purchase history. The inference was emergent. The governance gap was structural: the assessment tool was not designed to surface emergent inference risks.
Industry Signal
ICO enforcement actions and EDPB guidance have addressed inference-based privacy violations, including cases where AI systems inferred protected characteristics from non-protected data and used those inferences in ways that created Article 9 special category processing obligations. The enforcement direction is toward treating inference outputs as processing activities subject to their own privacy obligations.
The regulatory trajectory is toward treating sensitive inferences as sensitive processing, regardless of the sensitivity of the inputs. Organizations that have not assessed their AI systems' inference capabilities for special category output risk are behind on a governance challenge that enforcement is increasingly testing.
Enabling Capabilities
- Inference capability assessment: Technical methodology for assessing what sensitive attributes AI models can infer from their inputs.
- Output privacy auditing: Regular assessment of model output distributions for emergent inference of sensitive attributes.
- Special category inference monitoring: Production monitoring specifically designed to detect when model outputs contain inferences about special category data.
- AI-specific DPIA methodology: Extended DPIA frameworks that include inference risk assessment alongside input data risk assessment.
A Practical Starting Point
For your highest-risk AI systems, conduct an inference capability assessment. What sensitive attributes can the model infer from its inputs? Which inferences produce outputs that may constitute special category processing? Which inference capabilities were not anticipated in the original DPIA?
Assess what the model infers, not just what it receives. The second assessment is where the governance work is.
Questions Leaders Should Be Asking
- For our AI systems, have we assessed what sensitive attributes the models can infer from their inputs beyond the data attributes explicitly included in training?
- Have we assessed whether our AI systems produce special category outputs through inference that requires Article 9 legal basis separate from the basis for the input data?
- Do we monitor production AI system outputs for emergent inference of sensitive attributes that were not anticipated in pre-deployment assessment?
- What is our governance process when a post-deployment assessment discovers that an AI system has developed inference capabilities beyond those documented in the original DPIA?
What to Require From Vendors
Ask directly:
"Has your AI system's inference capability been assessed for outputs that may constitute special category processing under GDPR Article 9, and what monitoring exists for detecting emergent inference of sensitive attributes in production?"
Expect as evidence:
- Inference capability assessment documentation covering special category output risk
- Production monitoring for sensitive attribute inference
- Documentation of the training data attributes and known inference capabilities
A vendor who describes their system's privacy governance through input data handling without addressing inference output privacy risk has governed the simpler half of the problem.
Demonstrating Diligence
- Documentation: Inference capability assessment records; special category inference risk analysis; DPIA extended to include inference risk assessment.
- Process: Pre-deployment inference capability assessment; production monitoring for sensitive attribute inference; governance response for emergent inference capability discoveries.
- Technical evidence: Inference capability testing outputs; production output monitoring records.
AI inference privacy diligence requires demonstrating governance of what the system produces, not just what it receives.
Closing Perspective
AI inference creates privacy risk that the data it learned from did not create. That is the governance challenge. The data was bounded. The model is not.
Governance programs that assess input data and do not assess inference outputs are governing what is visible and not governing what is consequential. The inference is where the privacy risk that individuals would most object to lives.
The model infers what the data implied. Govern the implication.
Enterprise practitioner perspective. Not legal advice. Part of the Deep Trust Governance Series by Verisq. Get the free weekly Breach Digest.