Privacy-Preserving Techniques in Generative AI and Large Language Models: A Narrative Review

Abstract

Generative AI, including large language models (LLMs), has transformed the paradigm of data generation and creative content, but this progress raises critical privacy concerns, especially when models are trained on sensitive data. This review provides a comprehensive overview of privacy-preserving techniques aimed at safeguarding data privacy in generative AI, such as differential privacy (DP), federated learning (FL), homomorphic encryption (HE), and secure multi-party computation (SMPC). These techniques mitigate risks like model inversion, data leakage, and membership inference attacks, which are particularly relevant to LLMs. Additionally, the review explores emerging solutions, including privacy-enhancing technologies and post-quantum cryptography, as future directions for enhancing privacy in generative AI systems. Recognizing that achieving absolute privacy is mathematically impossible, the review emphasizes the necessity of aligning technical safeguards with legal and regulatory frameworks to ensure compliance with data protection laws. By discussing the ethical and legal implications of privacy risks in generative AI, the review underscores the need for a balanced approach that considers performance, scalability, and privacy preservation. The findings highlight the need for ongoing research and innovation to develop privacy-preserving techniques that keep pace with the scaling of generative AI, especially in large language models, while adhering to regulatory and ethical standards.

1. Introduction

In recent years, generative artificial intelligence (AI) has revolutionized the creation of synthetic data and creative content, significantly impacting industries such as healthcare, entertainment, and finance. Key models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and large language models (LLMs) have pushed the boundaries of AI’s potential. Still, this progress has also raised substantial concerns about data privacy. These concerns stem from the fact that generative AI models are often trained on sensitive datasets, which can lead to privacy breaches through model inversion, data leakage, and other forms of malicious attacks.

It is crucial to clarify that the focus of this paper is on protecting the privacy of individuals whose sensitive data are processed or stored by generative AI systems. Specifically, this involves safeguarding personally identifiable information (PII) and other sensitive attributes that, if exposed, could result in serious privacy violations. Protecting privacy involves mitigating various threats, including model inversion attacks, membership inference attacks, and unintended data memorization, which can reveal personal information. Additionally, the threats can come from multiple sources, including malicious actors, unauthorized internal access, or even unintentional data exposure during routine AI operations.

Moreover, privacy risks arise at different stages of the generative AI lifecycle: during training, inference, and fine-tuning. Each stage introduces unique privacy challenges, requiring targeted privacy-preserving techniques. For instance, training on sensitive datasets can expose private data through memorization, while during inference, unintended data leakage could occur through generated outputs. Privacy concerns also persist during model fine-tuning, especially when dealing with domain-specific sensitive data. As such, different techniques are required to address specific privacy concerns at each stage of the AI system’s lifecycle.

One critical dimension of this challenge involves navigating the evolving regulatory landscape. For instance, the European Union’s AI Act proposes a legal framework that regulates AI technologies based on their potential risks to fundamental rights, safety, and privacy. The Act categorizes AI applications into different risk levels, with stringent requirements for high-risk systems, such as those used in healthcare or critical infrastructures. Similar regulations, like the GDPR (General Data Protection Regulation), emphasize data protection and user consent, necessitating AI systems incorporate privacy-preserving mechanisms. These regulations are shaping the trajectory of AI development, urging a balance between innovation and compliance with global privacy standards.

One of the most prominent privacy risks is memorization of sensitive data. For instance, LLMs like GPT can inadvertently reveal personal information from training data when generating content, posing a significant privacy risk. Attackers can also exploit these models through model inversion attacks, which allow them to reconstruct input data from a model’s outputs [1,2]. These privacy risks necessitate the development and adoption of robust privacy-preserving mechanisms.

Several privacy-preserving techniques have been proposed to mitigate these risks. Differential privacy (DP) is one of the most widely implemented methods, introducing noise to the model’s outputs to obscure sensitive data, making it particularly useful in domains like healthcare and finance [1]. Federated learning (FL) has also emerged as a powerful approach, enabling decentralized model training across multiple devices or institutions without sharing raw data, thereby reducing privacy risks [3]. Techniques such as homomorphic encryption (HE) and secure multi-party computation (SMPC) allow computations on encrypted data, offering a higher level of protection by ensuring that sensitive information remains encrypted during processing [3].

As generative AI continues to advance, balancing the performance of these models with privacy concerns remains a pressing challenge. Integrating privacy-preserving techniques often entails trade-offs, such as reduced accuracy or increased computational demands, emphasizing the need for further research to optimize these methods for practical applications [1,4,5]. This review provides a detailed examination of the current landscape of privacy-preserving techniques in generative AI, evaluating their strengths, limitations, and potential for future improvements.

2. Legal and Regulatory Perspectives on Privacy in Generative AI

The intersection of generative AI and privacy is not only a technical challenge but also a legal and ethical one. Understanding what privacy means from a legal standpoint is crucial for developing and implementing effective privacy-preserving techniques in AI systems.

2.1. Legal Definitions of Privacy and Personal Data

Under regulations like the General Data Protection Regulation (GDPR) in the European Union, personal data are defined as any information relating to an identified or identifiable natural person (“data subject”). This includes direct identifiers, such as names and addresses, and indirect identifiers, such as IP addresses and biometric data. The GDPR emphasizes that identifiability can be context-dependent, meaning that even seemingly anonymized data can be considered personal data if re-identification is possible in a given context.

2.2. Anonymization and Its Limitations

Anonymization involves processing personal data to irreversibly prevent identification. However, achieving true anonymization is mathematically impossible due to the potential for data linkage and re-identification with auxiliary information. Therefore, data protection laws often require data controllers to implement appropriate safeguards rather than guaranteeing absolute anonymization.

2.3. Key Regulations Affecting Generative AI

The GDPR, along with other regulations like the Health Insurance Portability and Accountability Act (HIPAA) in the United States and the California Consumer Privacy Act (CCPA), imposes strict requirements on how personal data can be collected, processed, and stored. The EU AI Act further proposes risk-based regulations specific to AI systems, emphasizing the need for transparency, accountability, and human oversight.

2.4. The Role of Privacy-Preserving Techniques in Legal Compliance

Privacy-preserving techniques such as differential privacy, federated learning, homomorphic encryption, and secure multi-party computation can help organizations comply with legal obligations by minimizing the processing of personal data, enhancing data security, and enabling data analysis without exposing individual information. However, these techniques must be carefully implemented to align with legal standards and demonstrate that reasonable measures have been taken to protect data subjects’ rights.

2.5. Balancing Innovation and Legal Obligations

Organizations must balance the innovative potential of generative AI with the need to respect privacy rights and comply with legal requirements. This involves adopting a “privacy by design” approach, conducting Data Protection Impact Assessments (DPIAs), and staying informed about evolving legal standards and best practices.

3. Overview of Privacy Risks in Generative AI

Generative AI has become one of the most transformative technologies, yet its use comes with notable privacy risks. Key models such as GPT (ChatGPT), BERT (bidirectional encoder representations from transformers), LLaMA (Large Language Model Meta AI), Claude by Anthropic, and Bard by Google rely on large datasets for training, making them susceptible to various privacy issues.

3.1. Data Memorization and Model Inversion Attacks

Large models, including GPT-4, BERT, and LLaMA, have been found to memorize specific data points from their training datasets. This memorization can inadvertently lead to the disclosure of private information when the model generates outputs in response to user prompts. For example, model inversion attacks exploit a model’s ability to generate realistic outputs, allowing attackers to infer sensitive attributes from training data. Empirical studies [1,5] have demonstrated that models trained on datasets containing PII are more susceptible to privacy attacks, such as membership inference and model inversion, due to the models’ tendency to memorize unique data points.

However, beyond direct model inversion attacks, a more subtle risk comes in the form of contextual privacy attacks. These attacks leverage the model’s ability to generate contextually relevant information, where attackers craft specific queries or prompt sequences designed to coax sensitive data out of the model. These context-specific prompts can extract details about individuals or events embedded within the training data, even if no direct inversion occurs. For instance, users interacting with a language model could, intentionally or not, query for information that indirectly leads to the exposure of sensitive data memorized by the model. As these prompts are often non-malicious, identifying and mitigating such risks remains a challenge.

Contextual attacks exploit the fact that LLMs are particularly prone to generating outputs that fit the linguistic and contextual patterns present in their training data, which can sometimes lead to over-disclosure. While standard model inversion attacks are a recognized threat, contextual attacks represent an emerging area of concern that requires further exploration.

To counter these risks, several mitigation techniques are being actively researched. Selective forgetting or “scrubbing” methods aim to reduce memorization by enabling models to forget specific data points after training without necessitating full model retraining. These approaches are particularly useful when models need to unlearn sensitive information post hoc, an increasingly relevant technique as regulations like the GDPR allow individuals to request the deletion of their data.

Additionally, differential privacy techniques—already discussed as a foundational approach—can be enhanced to focus specifically on the memorization problem in LLMs. By introducing controlled noise during the training process, the model’s ability to memorize specific data points can be reduced. However, the challenge lies in maintaining the utility of large models, especially in applications like healthcare, where the trade-off between privacy and accuracy is critical.

The need for more sophisticated solutions also extends to fine-tuning methods. Fine-tuning large, pre-trained models like GPT-4 or BERT on smaller, sensitive datasets can increase the risk of memorization and data leakage. Privacy-preserving fine-tuning, which involves techniques such as adding noise to gradient updates or limiting the extent of model adaptation to new data, is being explored as a way to reduce this risk while maintaining the performance gains typically achieved through fine-tuning.

Overall, while data memorization and model inversion attacks represent well-documented risks, the emerging category of contextual privacy attacks and the challenge of mitigating memorization in fine-tuning underscore the need for further research. Developing robust techniques for preventing and auditing these risks is crucial for the safe deployment of large language models in privacy-sensitive domains.

3.2. Membership Inference Attacks

Membership inference attacks allow adversaries to determine whether a specific data point was used during training. This risk applies not only to models like GPT but also to generative models across various fields, such as GANs (generative adversarial networks). Overfitted models are particularly vulnerable to these attacks because they retain too much information from the training data. In sensitive domains such as healthcare, membership inference attacks could expose patients’ personal health records, violating laws like the GDPR and HIPAA [4,6].

Recent research has shown that privacy attacks can happen even when attackers have very limited access to a model’s internal details. For example, Cilloni et al. [7] discovered a new method to perform membership inference attacks on stable diffusion models like Stable Diffusion V2 by StabilityAI.

They found that even if someone can only interact with the model by making queries—without knowing anything about how it works inside—they can still uncover privacy vulnerabilities. By repeatedly querying the model and observing the images it generates at different stages, attackers can figure out whether certain images were part of the model’s original training data.

The researchers achieved this by focusing on the intermediate images the model produces during the image creation process. By analyzing these intermediate outputs across multiple queries, they trained another model to determine whether a sequence of images came from data the model had seen during training or from new, unseen data.

This study highlights the urgent need for stronger defenses against such privacy attacks on stable diffusion models. It emphasizes that developers and users of these advanced AI systems need to be aware of these privacy risks and take steps for protection against them, especially when the models are used with sensitive or proprietary data.

Membership inference attacks (MIAs) aim to determine whether a specific data point was part of the training set used to train a machine learning model. These attacks exploit the overfitting of models to identify if an individual’s data have been used, raising privacy concerns, especially in sensitive applications. For example, in medical datasets, MIAs could potentially reveal whether a person’s health information was included in the model, thereby compromising confidentiality [8,9].

3.3. Model Poisoning and Adversarial Attacks

Generative model examples are DALL-E 2 and Stable Diffusion, which can be poisoned in training by injecting manipulated data into the training pipeline. The attackers may introduce backdoor attacks, among others, where the model produces erroneous or harmful outputs whenever a certain trigger, specific words, or images appear. For instance, various studies have shown that poisoning attacks result in Stable Diffusion producing deformed or completely irrelevant images from tampered training data, thus raising serious privacy and security concerns [7,8]. Data poisoning is a critical issue, as one hardly notices it, and it survives model updates. What is worse, attackers may exploit such poisoned models to generate misleading or malicious outputs of systems, substantially shattering trust in the credibility and reliability of AI systems. A very representative example is the “Nightshade” attack proposed by Shan et al. [9], showing the effectiveness of data poisoning in compromising text-to-image generative models such as Stable Diffusion. The authors have shown here how one can corrupt model responses for given prompts with only a relatively few poisoned samples injected into the training dataset. All these poison samples are indistinguishable from benign ones in any way that would be noticed by a human, yet they are crafted to cause the model to misbehave intentionally.

These “prompt-specific poisoning attacks” exploit the fact that generative models often rely on limited data for certain concepts, making them vulnerable to targeted manipulation. Remarkably, the Nightshade attack can corrupt a specific prompt in the Stable Diffusion SDXL model using fewer than 100 poison samples. The poisoned samples cause the model to generate incorrect or harmful outputs when the targeted prompt is used. Additionally, the effects of such poisoning can “bleed through” to related concepts and prompts, amplifying the attack’s impact. Multiple attacks can be composed together within a single prompt, further destabilizing the model.

Moreover, beyond security concerns, poisoning attacks can also lead to significant privacy breaches. Attackers may inject malicious data into the model’s training process with the intention of manipulating the model to leak sensitive or private information. For example, poisoned models could be exploited to generate outputs containing identifiable or confidential data, which may have been unintentionally incorporated into the training dataset. This creates a severe privacy risk, as private information may be exposed without the user’s knowledge or consent, compromising personal data and violating privacy regulations. Thus, data poisoning not only degrades the security and reliability of AI systems but also poses critical privacy threats by enabling the exposure of sensitive information in model outputs.

3.4. Data Leakage from Fine-Tuning

Fine-tuning a large, pre-trained model on new tasks or smaller datasets, such as GPT-4 or even BERT, may leak significant information in the model. While fine-tuning, a model might “memorize” complex details of sensitive data it has been trained on, thereby leaking them through its generated outputs. Other examples, apart from those indicated, might include healthcare or financial domains where fine-tuning may leak private data, including, for instance, patient records and past transaction histories. Recent works underlined those generative models, and in particular, diffusion models boost these risks. For example, in 2023, Cilloni and colleagues demonstrated how membership inference attacks (MIAs) could exploit the outputs of stable diffusion models to determine whether a specific sample was part of the model’s training dataset [7]. They focused on Stable Diffusion V2, a cutting-edge text-to-image generative model developed by StabilityAI.

Further complicating these risks, Li et al. (2023) [10] introduced the Shake-to-Leak concept, showing that fine-tuning pre-trained diffusion models on manipulated data greatly increases the already significant privacy vulnerabilities. Their results showed, in fact, that the occurrence of S2L is evident under several different fine-tuning strategies in this respect, including the concept-injection ones, such as DreamBooth and Textual Inversion, along with the parameter-efficient ones, referring to LoRA and Hypernetwork [10].

Data leakage from fine-tuning occurs when sensitive or private information unintentionally leaks during the fine-tuning process of pre-trained models. Unlike MIAs, which aim to infer the inclusion of specific data points in the training set, data leakage during fine-tuning involves exposing sensitive information due to improper data handling or vulnerabilities in the fine-tuning process. This can lead to inadvertent disclosure of personal information or proprietary data that should remain confidential. For instance, fine-tuning a model on proprietary datasets without sufficient privacy controls could cause the model to output or regenerate sensitive data inadvertently [7,10].

3.5. Privacy Risks in Real-World Applications

Generative models are increasingly being used in critical domains such as healthcare (e.g., generating synthetic patient data), legal services, and finance (e.g., fraud detection). However, these applications pose significant privacy risks. For example, generative AI models in healthcare may unintentionally expose patient data, leading to violations of regulations like the GDPR and HIPAA [5]. In healthcare, differential privacy has been proposed as a way to generate synthetic data to reduce privacy risks, although challenges remain in balancing privacy protection with data utility. As Templin et al. (2024) noted, generative AI systems are vulnerable to privacy breaches, whether through unintentional leakage or adversarial attacks that retrieve sensitive training data. This risk is heightened with the increasing reliance on third-party digital tools for computation, raising ethical and regulatory concerns around the handling of sensitive health information under frameworks like HIPAA and the GDPR [11].

Other significant risks include “hallucinations”—instances where AI models generate incorrect or fabricated information—particularly in health-related domains where such errors can lead to patient harm or regulatory violations. Templin et al. (2024) [11] identified six core challenges in the application of generative AI for digital health, with privacy being a critical concern. While developments in model architecture and security protocols continue, institutions face tough decisions about whether to rely on third-party models or invest in local infrastructure for more secure data handling [11].

Similarly, legal and financial applications of generative AI may lead to the leakage of confidential information about cases or financial transactions, causing organizations to risk non-compliance with regulations.

4. Privacy-Preserving Techniques for Generative AI

The main techniques are differential privacy, federated learning, and homomorphic encryption. In addition to those foundational techniques, a growing number of open-source tools have emerged to help organizations manage and preserve data privacy.

These tools enable privacy-preserving workflows across industries, supporting data anonymization, masking, and regulatory compliance, such as with the GDPR. This section highlights notable tools such as Microsoft Presidio and others that enhance privacy in generative AI models. This section provides an overview of key privacy-preserving techniques used in generative AI. Figure 1 illustrates the workflow of these techniques, showing how they are integrated at different stages of the AI process, from data collection to model deployment.

4.1. Differential Privacy (DP)

Differential privacy (DP) is a foundational technique for protecting individual data in generative AI models by adding noise to the data or model outputs. Google’s RAPPOR system uses DP to collect aggregated browser data without compromising individual privacy [12]. Applications in healthcare have increasingly employed differential privacy (DP) to generate synthetic patient data that protect sensitive information from unauthorized disclosures and comply with regulations like HIPAA [13,14]. While DP is mainly used in research contexts, its use allows health researchers to create and share privacy-preserving datasets, enabling substantial machine learning and inferential applications without exposing real patient data. Techniques such as differentially private normalizing flows (DP-NFs) have been proposed for generating synthetic data from electronic health records (EHRs), balancing privacy and utility. However, there is often a trade-off between the two. Studies show that DP-NF-generated data retain significant utility in predictive tasks—such as forecasting medical conditions like pulmonary hypertension—making it a promising solution for handling sensitive health data, though challenges remain [13]. PySyft (by OpenMined) is an open-source framework for privacy-preserving machine learning, integrating DP capabilities with deep learning libraries like PyTorch and TensorFlow [14].

Moreover, differential privacy has seen significant advancements through mechanisms like the Gaussian and Laplace mechanisms, which add calibrated noise to data queries, enhancing privacy without severely compromising data utility. Recent research has introduced the concept of privacy budgets, allowing organizations to quantitatively measure and manage the trade-off between privacy and accuracy. By carefully selecting the privacy parameters (epsilon and delta), practitioners can tailor the level of privacy protection to meet specific regulatory requirements such as those stipulated by the GDPR, thereby ensuring that individual data contributions remain indistinguishable within the output of generative AI models.

4.2. Federated Learning (FL) and Privacy-Preserving Federated Learning

Federated learning provides methods for training models in decentralized ways, enabling data to reside within local devices. In the healthcare domain, FL has been applied in the development of diagnostic models, where sensitive information about patients does not have to be shared. This allows MioT (Medical Internet of Things) devices and EHRs to undergo local training. Additionally, FL opens a wide avenue toward collaborative healthcare decisions along with diagnosis, ensuring the confidentiality of patient information.

While FL strengthens the level of privacy by reducing data sharing, it introduces challenges in itself. It is possible to regenerate client-sensitive data using model updates shared between the clients and the central server, which may easily breach requirements of privacy. It is due to such vulnerabilities that PPFL combines FL with techniques like homomorphic encryption and differential privacy for enhanced protection of privacy. For example, encrypted model updates enable secure collaboration among hospitals, thereby making huge gains in privacy during model training [3].

Each of these methods has its relative strengths and weaknesses, and thoughtful consideration will be critical to the evolution of FL frameworks in the healthcare sector. For example, differential privacy injects meaningful noise into model updates such that when aggregated on a server, even individual patient data remain confidential. Similarly, homomorphic encryption enables computational functionalities on encrypted data, hence even model training may be carried out in a secure fashion. Moreover, FL applications go beyond mere protection of privacy. It has been applied to practical scenarios such as developing predictive models for hospital readmissions where patient data would stay locally with various hospitals for privacy while improving predictive accuracy. Equally encouraging is the fact that FL is showing its potential in clinical decision support systems, in drug development, and in collaborative disease diagnosis like diabetic retinopathy detection [15].

Since federated learning is continuously evolving, active research should be performed with the aim of refining these privacy enhancement techniques in light of the presented challenges imposed by the healthcare domain. In the case of strong privacy in FL frameworks, healthcare will confidently use AI while protecting sensitive patient information. This will pave the way for more effective and secure healthcare solutions [15]. TensorFlow Federated is an open-source framework for machine learning and other computations on decentralized data. TFF can be used openly to research and experiment with federated learning. This is helpful in applications where the privacy of data is of the utmost essence, say, in healthcare and financial transactions, and in user-centric applications such as virtual keyboards in mobile devices, where sensitive information does not leave the user’s device [16].

In addition, the implementation of secure aggregation protocols within federated learning has further strengthened privacy guarantees by ensuring that individual client updates are encrypted and can only be decrypted in aggregate form by the central server. This means that even if the server is compromised, individual updates remain confidential. Research efforts are also focusing on addressing vulnerabilities such as poisoning and backdoor attacks in FL by developing robust aggregation rules and anomaly detection methods, which enhance the overall security and trustworthiness of the federated learning process in sensitive applications like personalized healthcare and finance.

4.3. Homomorphic Encryption (HE)

Homomorphic encryption is a powerful cryptographic technique that enables direct computation on encrypted data, preserving the privacy of sensitive information throughout the processing phase. For industries handling sensitive data, this method is particularly beneficial, as it allows for secure data analytics without exposing the underlying data [17]. For example, IBM has pioneered the use of fully homomorphic encryption (FHE) in its cloud services, which enables encrypted data computation without revealing sensitive information. This technology is applied in sectors like finance and healthcare for privacy-preserving analytics and AI model training [17]. Very recently, interest in HE has grown due to the general trend of incorporating privacy in mainstream applications as a means of achieving secure data collaboration and sharing while guaranteeing data integrity. With the rise in data breaches, industries have come to realize the need for additional security measures, and HE has emerged as one of the key solutions to cater to the emergence of not only privacy concerns but also the increased value of data in a data-driven economy. It is a question of balance between privacy and utility since it unleashes full AI/ML potential and cuts efficiently through the ever-denser thicket of regulatory compliance and consumers clamoring for security.

Homomorphic encryption has also been integrated into federated learning environments, allowing secure aggregation of encrypted model updates without exposing sensitive raw data [3]. Despite its strong privacy guarantees, HE is computationally expensive. However, advancements in leveled homomorphic encryption have reduced its computational cost, making it more feasible for large-scale applications [18].

Recent developments in homomorphic encryption have led to the creation of more efficient schemes like the Brakerski/Fan–Vercauteren (BFV) and Cheon–Kim–Kim–Song (CKKS) schemes, which support arithmetic operations on ciphertexts suitable for machine learning tasks. These advancements reduce computational overhead and make HE more practical for real-time applications. Additionally, standardization efforts by organizations such as the HomomorphicEncryption.org consortium are promoting interoperability and encouraging wider adoption of HE in industry, aiding compliance with data protection regulations by enabling secure processing of encrypted data without exposing plaintext information.

4.4. Secure Multi-Party Computation (SMPC)

Secure multi-party computation (SMPC) enables multiple parties to jointly compute functions over their private data without revealing their inputs. This technique is widely used in the financial industry, where companies jointly analyze risk models without sharing customer data [19]. SMPC has been applied to generative AI to allow model training across multiple datasets while ensuring no individual data are disclosed. An open-source tool for SMPC is the MP-SPDZ, a library for secure multi-party computation that supports privacy-preserving machine learning [20].

Furthermore, recent protocols in SMPC, like the SPDZ (pronounced “Speedz”) framework, have improved computational efficiency, making it feasible for practical applications in privacy-preserving machine learning. By using preprocessed data and offline computations, these protocols reduce the online computational burden and latency. SMPC’s ability to function without a trusted third party is particularly valuable in collaborative environments where data confidentiality is crucial and legal agreements may not permit data sharing, thus aligning with compliance requirements in cross-border data collaborations under regulations like the GDPR.

4.5. Privacy-Preserving Synthetic Data Generation

Privacy-preserving synthetic data generation is crucial for creating realistic datasets that protect sensitive information. In network traffic analysis, synthetic traffic traces can be generated for testing models without exposing real user data [21]. Microsoft Presidio is a key tool for data anonymization and PII detection in textual data. In generative AI, Presidio can be integrated into preprocessing pipelines to ensure that sensitive PII is detected and masked before the data are used for model training [22]. This tool is widely used in industries like healthcare and finance, where regulatory compliance is critical. Another open-source tool for data anonymization is the ARX, which supports anonymization methods like k-anonymity and differential privacy, making it useful for anonymizing sensitive data in generative AI models [23].

Advancements in generative adversarial networks (GANs) and variational autoencoders (VAEs) have enhanced the quality of synthetic data, making it increasingly difficult to distinguish from real data while preserving statistical properties. Techniques such as differentially private GANs (DP-GANs) incorporate differential privacy into the data generation process, providing formal privacy guarantees. These methods enable organizations to share and analyze datasets that would otherwise be restricted due to privacy concerns, thereby fostering innovation and collaboration without compromising individual privacy rights.

4.6. Privacy-Enhancing Technologies (PETs)

Privacy-enhancing technologies (PETs) are a combination of the capabilities of differential privacy, homomorphic encryption, and secure multi-party computation to ensure data privacy with utility. In fact, such technologies are even applicable in the utilities industry where they permit energy consumption data analysis without giving away the pattern of individual use [24]. They already play an increasingly important role with respect to both LLMs and generative AI systems, where user data protection is paramount. The use of DP and federated learning (FL) in such models as ChatGPT can indeed prevent models from leaking sensitive information by accident [25]. These methods provide a possibility to mitigate the risk of eventual leaks while still keeping the AI models performing well. In domains like drug discovery, where large datasets are crucial for model training but privacy concerns are paramount, PETs provide a solution by ensuring that sensitive medical data remain secure [17].

However, it is very challenging to integrate different PET mechanisms into one architecture. The challenges involve the complexity of guaranteeing seamless interaction between techniques, including DP and HE, and scaling the solution to large datasets or even multiple stakeholders [15]. Second, some recent developments, like fine-tuning diffusion models, have the potential to increase existing privacy risks, especially when model parameters are updated using sensitive or domain-specific data, making these PET implementations in generative AI even more vulnerable to membership inference and data extraction attacks [25]. However, despite these pressing concerns, research studies continue to enhance PETs, making them more resilient to emerging privacy risks.

The integration of PETs is further supported by emerging frameworks that facilitate their combined implementation. For example, platforms like OpenMined’s PyGrid allow for the orchestration of federated learning (FL) with differential privacy and encrypted computation, simplifying the deployment of complex privacy-preserving workflows. This collaborative approach not only enhances data security but also promotes compliance with international data protection standards by providing configurable privacy controls tailored to specific legal requirements.

4.7. Data Masking and Anonymization

Some effective ways to secure PII in AI training datasets include anonymization and masking. Many generative AI models involve large datasets, which may contain sensitive information. Masking is a technique used to hide sensitive data so that they can still be used for testing or analysis without revealing their original form. This is achieved by substituting real data with fictitious data or removing them entirely. Anonymization goes further by permanently removing or obfuscating personal identifiers, making it difficult to trace data back to individuals. Techniques include generalization, where specific values are replaced with broader categories, and adding noise to data. If these data are not properly masked or anonymized, it can lead to data breaches and violations of regulations such as the GDPR and CCPA.

Microsoft’s Presidio is an open-source tool that detects and anonymizes PII in unstructured text data [22]. It automatically identifies sensitive information like names, addresses, and credit-card numbers and replaces them with synthetic or placeholder values, helping to ensure compliance with regulations such as the GDPR. However, the use of these tools does not guarantee compliance unless they are properly implemented within the framework of legal requirements.

ARX is another tool that offers advanced anonymization techniques, including k-anonymity and l-diversity [23], helping organizations balance privacy and data utility. It also includes risk-analysis tools to assess the likelihood of re-identifying individuals in anonymized datasets.

To strengthen data anonymization, advanced techniques such as k-anonymity, l-diversity, and t-closeness are employed to prevent re-identification attacks. These methods ensure that each individual’s data cannot be distinguished from at least k-1 others, adding a layer of group-based anonymity. Additionally, synthetic data generation and data perturbation methods are being refined to preserve essential data characteristics while eliminating personal identifiers. Implementing these techniques supports adherence to data protection laws by effectively reducing the risk of exposing personal data during the use and sharing of datasets in generative AI.

While these tools can support regulatory compliance efforts, they must be part of a comprehensive strategy that aligns with legal standards. Specific endorsements by legal authorities for these tools are limited, so organizations must ensure that they meet all regulatory requirements for privacy and data protection.

4.8. Techniques for Preventing Unintended Data Memorization

A crucial aspect of protecting privacy in generative AI models, including large language models, is preventing the unintended memorization of sensitive data. During their training on vast datasets, these models may inadvertently memorize and reproduce sensitive information from the training data. Applications utilizing these models risk disclosing sensitive information through user interactions, bringing various privacy concerns to the forefront. While noise injection, a core component of differential privacy (DP), aims to make sensitive data leaks less likely, other novel approaches focus directly on preventing models from retaining sensitive information. These methods either limit the model’s capacity during training or ensure that the model can “forget” such data without relying on noise-injection techniques. In the article by Hans et al. (2024) [26], the authors propose innovative techniques to address the issue of unintended data memorization in LLMs. They introduce the “goldfish loss” method, which involves excluding a randomly sampled subset of tokens from the loss computation during training. This approach aims to reduce the model’s tendency to memorize specific data points without significantly impacting its overall performance. The study demonstrates that this technique can effectively mitigate memorization, thereby enhancing the privacy and security of generative AI applications. The authors also discuss the broader implications of their findings for the development of privacy-preserving AI technologies [26].

In addition to the “goldfish loss” method, other techniques are being explored to address unintended memorization, such as regularization techniques and model architecture adjustments that reduce overfitting—a primary cause of memorization. Techniques like dropout, weight decay, and limiting model capacity help prevent models from capturing and reproducing exact details from the training data. Additionally, dataset shuffling and careful curation can minimize the presence of unique or outlier data points that are more likely to be memorized, thereby enhancing the privacy of model outputs.

By incorporating a range of techniques, researchers are developing more robust methods to mitigate the risks of unintended data memorization in generative AI models, helping to secure sensitive information and improve privacy compliance.

4.8.1. Selective Forgetting and Scrubbing

Selective forgetting, also referred to as “scrubbing”, allows for the targeted removal of specific data points from a trained model without requiring complete retraining of the model. This technique is particularly important in light of data protection regulations like the GDPR, which grants individuals the right to have their data deleted from systems. The growing need for compliance with such regulations has made selective forgetting a vital area of research, especially as machine learning models are increasingly deployed in environments where privacy concerns are paramount.

Traditionally, removing specific data points from a model often required full retraining, as most machine learning models do not natively support the removal of data that have influenced their learned parameters. This process is computationally expensive and impractical, especially for large models and datasets. However, recent advancements in algorithmic methods, such as those explored by Ginart et al. (2019) [27], offer more efficient solutions. In their work, they investigate algorithmic principles that enable the efficient deletion of individual data points from machine learning models. Specifically, for the case of k-means clustering, they propose two provably efficient deletion algorithms that achieve up to 100x improvement in deletion efficiency across several datasets when compared to the traditional k-means++ baseline. These algorithms maintain comparable statistical quality in the resulting clusters, demonstrating that selective forgetting can be both effective and efficient.

In addition to clustering models, ongoing research is exploring how to extend these deletion algorithms to more complex models, such as neural networks and generative models, though challenges remain. The primary difficulty lies in minimizing the degradation of model performance while still ensuring compliance with privacy regulations. Optimizing these selective forgetting algorithms is critical for maintaining model utility, as well as ensuring that sensitive information is properly erased from systems post-training.

As the demand for privacy-preserving technologies continues to grow, selective forgetting provides a flexible and scalable solution for managing data rights in machine learning models. The ability to “scrub” specific data points from models not only enhances compliance with privacy laws but also mitigates risks associated with data memorization and potential misuse. Future research in this area will likely focus on improving the scalability and applicability of these techniques to more complex models and real-world scenarios [27].

4.8.2. Retraining with Privacy Filters

Another approach involves retraining models using privacy filters to ensure sensitive information is not memorized during learning. Privacy filters can be combined with differential privacy during the training phase of the model by dynamically applying constraints on the amount of information memorized.

However, the classic mechanisms of DP can result in degraded utility and disparate impacts on underrepresented subgroups, thus raising concerns about fairness and model performance demands. The authors, Mireshghallah et al. (2023) [28], proposed several methods of privacy-preserving regularization for joint optimization of privacy utility. Their approach involves two methods: (1) using a discriminator to detect and mitigate the memorization of sensitive data and hence try to mitigate that, and (2) a triplet-loss term that encourages the models for better generalization with privacy. These regularization techniques provide a favorable trade-off between utility and privacy while also enabling faster training using existing optimization approaches. What is more, such a technique will ensure that subgroups are being treated uniformly and thus is particularly useful when the aim is to avoid data leakage without performance or fairness compromises. This might give an opportunity for even more considerate and efficient training while avoiding so many ongoing model behavior audits [28].

4.8.3. Privacy-Preserving Fine-Tuning

LLMs often require fine-tuning to achieve good performance in task-oriented applications. However, this process increases the risk of memorization, especially when sensitive datasets are involved. Recently, efforts toward privacy-preserving fine-tuning have emerged. Privacy-preserving fine-tuning introduces noise or constraints during the fine-tuning process. For instance, Chen et al. (2024) [29] proposed a privacy-preserving framework that improves the balance between model generalization and privacy preservation. This approach aims to improve the balance between model accuracy and privacy by making the model less sensitive to individual data points, thereby reducing the risk of memorizing specific private information. Techniques such as perturbation-aware optimization and flatness-guided sparse prefix-tuning enhance model utility while maintaining differential privacy guarantees, even on sensitive datasets like those used for text classification. Moreover, new algorithmic techniques have been developed for fine-tuning neural networks with non-convex objectives under modest privacy budgets. These methods enable efficient fine-tuning while ensuring adherence to privacy standards, preventing models from inadvertently memorizing or leaking sensitive data during training [29,30].

4.8.4. Real-Time Privacy Audits

In addition to training-time methods, real-time privacy auditing mechanisms are emerging as key components in preventing unintended memorization. These auditing systems monitor the outputs of LLMs, identifying when sensitive data are being revealed or when memorization has occurred. By integrating these auditing tools into the deployment pipeline, organizations can ensure that sensitive data are not leaked in real-world use cases. Such audits can detect and flag possible privacy violations, providing an added layer of protection [31].

4.8.5. Use of Synthetic Data

Such techniques have been applied across the board nowadays, enabling privacy-preserving AI. Models are trained on artificial data, which are synthetically generated to mimic all statistical properties of real data, without actually containing sensitive information. The model does not see the real sensitive data during training and hence cannot memorize them. Various techniques for generating synthetic data involve generative models, which come in forms like GANs or VAEs; they learn a model with the ability to generate data much like the original dataset. These have achieved much success in generating high-quality synthetic data for domains such as healthcare and finance, where the protection of data plays a key role [32,33].

However, as highlighted by Song et al. (2017) in their work [33], even models that are not directly trained on sensitive datasets can inadvertently memorize information from the data they process, leading to privacy breaches. Their research demonstrates how machine learning models, even those trained using synthetic data or with added noise, can still retain and leak subsets of the training data under certain conditions. This raises some concerns about whether the generation of synthetic data is per se foolproof as a privacy-preserving technique.

Therefore, the research community has adopted the use of active security countermeasures such as differential privacy and even more sophisticated adversarial training along with the generation of synthetic data, aimed at reducing this tendency for memorization and, hence, any possible leakage. It is a hot challenge to strike an optimal balance in the two components of the desired trade-off, that is, statistical utility in synthetic data and the guarantee of robust protection of privacy, within continued research in this area.

Privacy-preserving techniques in generative AI have evolved to address the growing concerns over data privacy, particularly in sensitive domains like healthcare and finance. Each technique offers unique advantages and challenges when balancing privacy, performance, and scalability.

Table 1. Overview of privacy-preserving techniques in generative AI and their key characteristics.

Technique	Key Strengths	Limitations	Best Suited For	Open-Source Tools
Differential Privacy	Strong privacy guarantees, scalable	Reduced model accuracy due to noise	General-purpose generative models	PySyft [14]
Federated Learning	No data sharing, decentralized training	Communication overhead, vulnerable to inference attacks	Healthcare, finance	TensorFlow Federated [16]
Homomorphic Encryption	Computation on encrypted data, strong privacy	High computational cost, scalability issues	Cloud-based generative AI	HElib [34]
Secure MPC	Joint computation without revealing inputs	Significant computational complexity	Regulated industries (e.g., healthcare, finance)	MP-SPDZ [20]
Adversarial Defense Mechanisms	Defense against privacy attacks, enhanced robustness	Specialized training needed, increased computational cost	High-security environments	Adversarial Defense Mechanisms [4,5]
Synthetic Data Generation	Anonymized, realistic data for model training	Data quality may be lower than real-world data	Data-rich environments requiring privacy	Presidio [22], ARX [23]
Privacy-Enhancing Technologies	Combined methods for stronger privacy–utility trade-offs	Complexity in implementation	Critical sectors (e.g., healthcare, finance)	Privacy-Enhancing Technologies
Blockchain for Privacy	Decentralized, transparent audit trails	Scalability concerns, emerging technology	High-transparency sectors (e.g., healthcare, finance)	Blockchain for Privacy

provides a comprehensive overview of the most prominent privacy-preserving methods used in generative AI, highlighting their core mechanisms, benefits, and potential limitations.

5. Emerging Trends and Future Directions

Generative AI continues to be a rapidly evolving field, with increasing attention on privacy concerns. Several emerging trends are shaping the future of privacy-preserving techniques in generative AI, and further research is essential to address the growing challenges and opportunities. This section highlights key areas for future exploration and suggests directions for enhancing privacy in generative AI systems.

5.1. Blockchain for Privacy in Generative AI

Blockchain has become a key technology in enhancing generative AI with privacy, security, and transparency. Nguyen et al. (2023) [35] discuss how integrating generative AI techniques with blockchain technology addresses challenges related to scalability, security, privacy, and interoperability. Blockchain provides a tamper-proof decentralized system with an immutable audit trail of data usage, ensuring that all transactions are transparent and compliant with privacy regulations. This is particularly crucial in federated learning environments, where blockchain can be used to track data processing and sharing across decentralized nodes, preventing unauthorized access and misuse [35].

The integration of blockchain with generative AI is illustrated in Figure 2, which demonstrates how blockchain networks can benefit from generative AI-enabled privacy-enhancing mechanisms. In this system, generative adversarial networks (GANs), variational autoencoders (VAEs), and LLMs are used for various purposes, such as creating synthetic transactions, enhancing privacy by generating fake identities, and auditing smart contracts automatically.

As shown in Figure 2, the decentralized architecture of blockchain, coupled with generative AI technologies, facilitates privacy enhancement by generating fake transaction data and fake identities. This approach ensures the security and privacy of sensitive information in a decentralized ledger, while also enabling scalable and secure data sharing across networks.

While blockchain offers potential benefits for privacy through decentralization and immutability, it also poses significant challenges in compliance with data protection regulations due to its inherent resistance to data alteration or deletion. The immutable nature of blockchain conflicts with the “right to be forgotten” under regulations like the GDPR. Recent research is investigating the use of permissioned blockchains and smart contracts to enable controlled access and support the enforcement of privacy policies [36]. Techniques such as off-chain storage, data encryption, and the use of zero-knowledge proofs are being explored to address these challenges. These explorations aim to harness blockchain’s transparency and security features while developing mechanisms to accommodate legal requirements like the “right to be forgotten”, indicating a possible path forward for integrating blockchain with privacy-preserving generative AI.

5.2. Advancing the Efficiency of Privacy-Enhancing Technologies (PETs) in AI

One of the most promising trends is the integration of Privacy-Enhancing Technologies (PETs) into the standard frameworks of generative AI. PETs such as differential privacy, homomorphic encryption, and secure multi-party computation are increasingly being integrated into AI workflows, but scalability remains a significant challenge [15]. Research is needed to make PETs more efficient and less resource-intensive, especially for real-time applications and large-scale AI models [25].

Moreover, PETs are expanding beyond traditional sectors like finance and healthcare to areas such as education, entertainment, and smart cities, which brings new challenges in terms of compliance and adaptation to different regulatory environments.

An essential aspect of integrating privacy-enhancing technologies (PETs) into mainstream AI is ensuring that these technologies align with existing legal and regulatory frameworks. By carefully considering data protection laws such as the GDPR and HIPAA during the design and implementation of PETs, organizations can better protect individual privacy rights while leveraging AI capabilities. This involves identifying whose privacy is being protected (e.g., end-users or data subjects), what specific data are being safeguarded (e.g., personal identifiers or sensitive health information), and against whom (e.g., external attackers or unauthorized internal access). Addressing these factors not only enhances the effectiveness of PETs but also helps organizations demonstrate compliance with legal obligations, thereby reducing potential liabilities and fostering trust among users.

5.3. Differential Privacy and Federated Learning in Real-Time Applications

Federated learning (FL) and differential privacy (DP) have proven to be powerful tools for protecting privacy, but their real-time implementation in dynamic environments remains under-researched. Future work should explore adaptive FL systems that can adjust to changing conditions, such as fluctuating data distributions, in real time without compromising privacy.

Key research directions in cross-device federated learning involve finding the optimal balance among communication efficiency, model performance, and privacy guarantees [37]. To address these challenges, state-of-the-art methods have focused on hybrid approaches that combine differential privacy with federated learning to offer higher privacy robustness for decentralized AI models. Li et al. (2023) [37] developed a communication-efficient and privacy-preserving federated learning algorithm called IsmDP-FL, which incorporates a two-stage gradient pruning mechanism with differentiated differential privacy. In the first stage, IsmDP-FL selects important model parameters through gradient pruning and prunes non-important ones. In the second stage, differential privacy is applied to the remaining important parameters so that sensitive information is protected while accuracy is maintained. This approach further prunes gradients during the upload for server aggregation, drastically reducing communication costs. Thus, communication efficiency is maximized with minimal use of the privacy budget while ensuring model performance and privacy in a federated learning environment. In addition to improving communication efficiency, differential privacy has also been applied to enhance LLMs. Behnia et al. (2022) [38] emphasized that fine-tuning pre-trained LLMs on private data is highly vulnerable to privacy risks, as malicious adversaries might extract sensitive training data. They presented a DP-based framework for fine-tuning LLMs called EW-Tune. Unlike many previous differential privacy approaches that assume a large number of training iterations, EW-Tune is designed to adapt to the fine-tuning process, where fewer iterations are common. This approach utilizes the Edgeworth accountant to provide finite-sample privacy guarantees, reducing noise injection by up to 5.6%, which is crucial for retaining model performance. Experiments on NLU tasks showed that EW-Tune improves performance by up to 1.1% while providing strong privacy protections. These trends reflect the growing interest in developing both privacy-preserving and communication-efficient solutions for real-time federated learning applications. Embedding differential privacy into techniques such as gradient pruning and fine-tuning frameworks further strengthens the privacy protection offered, ensuring models can be deployed in decentralized environments without significant performance trade-offs [38].

Implementing differential privacy and federated learning in real-time applications necessitates a clear understanding of the privacy models and the specific threats being addressed. Protecting the privacy of individual data owners during model training and inference involves ensuring that sensitive information is not inadvertently leaked or reconstructed by malicious entities. By focusing on the precise purpose of these methods—safeguarding personal data against inference attacks and unauthorized access—developers can tailor these techniques to meet both technical and legal requirements. Moreover, aligning these privacy-preserving methods with legal standards enhances their effectiveness in practice, ensuring that real-time AI applications comply with regulations while maintaining high performance.

5.4. Privacy-Preserving AI in Synthetic Data Generation

Synthetic data generation has been a cornerstone of privacy-preserving AI due to its ability to create high-quality datasets without exposing real data. However, ensuring that synthetic data are both private and realistic remains a challenge. New research is exploring generative adversarial networks (GANs) and variational autoencoders (VAEs) to generate data that not only mimic real-world distributions but also adhere to privacy standards such as differential privacy [32].

Goyal and Mahmoud (2024) consider synthetic data generation an increasingly urgent solution to various challenges, including data scarcity, privacy concerns, and algorithmic biases in machine learning applications. Their systematic review highlights several key limitations, such as high computational costs, instability during training, and challenges in maintaining privacy when applying current generative AI methods. While synthetic data generation has made significant advancements, further research is needed to address these limitations for wider adoption, particularly in fields like healthcare and finance, where anonymized, realistic data are critical for regulatory compliance [32].

The training and evaluation process of generative models like Tabular GAN (TGAN) provides insight into how synthetic data can be generated while retaining both the privacy and utility of real data. TGAN utilizes neural networks to capture and replicate the statistical properties of real datasets, even across large volumes of data that contain both discrete and continuous variables. This approach has been shown to outperform conventional generative models, particularly in terms of scalability and maintaining inter-column correlations. The workflow of TGAN, as illustrated in Figure 3, demonstrates how synthetic data are produced from real training data and subsequently evaluated with multiple machine learning models. This process ensures that the synthetic data retain utility while potentially enhancing privacy during machine learning workflows. Specifically, validation of the synthetic dataset occurs by ensuring that machine learning models trained on synthetic data produce results consistent with those obtained from models trained on real data, which guarantees that the synthetic data mirror the utility of the real data.

In addition to TGAN, several other synthetic data generation techniques are evaluated across critical dimensions such as data fidelity, diversity, scalability, computational cost, and training complexity. Figure 4 provides a comparative analysis of three prominent methods—GANs, VAEs, and LLMs—across these dimensions, highlighting the strengths and limitations of each. The outermost axes in the radar chart represent optimal performance, while the innermost axes represent the lowest performance in the respective dimension. From this visual comparison, it is evident that LLMs, while resource-intensive, offer high data fidelity and scalability, making them suitable for industrial-scale applications that require automation. On the other hand, GANs and VAEs present more balanced trade-offs between computational cost and performance, making them ideal for smaller-scale applications with limited resources. This flexibility makes GANs and VAEs particularly appealing for start-ups and personal projects, as they can generate useful synthetic data without the high overhead of LLM-based models.

The importance of privacy-preserving synthetic data generation continues to grow in areas like clinical trials, where sensitive data cannot be easily shared, and financial modeling, where realistic, anonymized data are necessary for compliance with regulations. As machine learning models continue to evolve, the need for synthetic data to preserve privacy while maintaining data utility will increase.

Synthetic data generation techniques produce data that retain the fundamental structure and patterns of real data without disclosing private information. This facilitates data sharing, collaboration, and algorithm testing while ensuring privacy. As privacy-preserving AI becomes more critical across industries, synthetic data generation will play a key role in enabling secure and ethical data usage.

In the realm of synthetic data generation, a critical consideration is the extent to which these methods preserve privacy in accordance with legal definitions of personal data and anonymization. Generating synthetic data that are sufficiently dissimilar from real individual records helps in preventing re-identification, thereby protecting the privacy of data subjects. By carefully evaluating the potential for de-anonymization and ensuring that synthetic datasets do not contain identifiable information, organizations can use these techniques to share and analyze data without violating privacy laws. This approach not only mitigates legal risks but also supports ethical standards in data handling and AI development.

5.5. Addressing Privacy Attacks in Large Language Models (LLMs)

As LLMs like GPT-4o, Gemini 1.5 Pro, Llama 3.2, and Claude 3.5 Sonnet grow larger, their potential for privacy vulnerabilities, such as membership inference attacks and model inversion attacks, will increase. Recent studies have highlighted that LLMs, when trained on massive datasets, are at risk of unintentionally leaking sensitive data, particularly when these models memorize specific pieces of information from the training data. Efforts to secure these models from privacy risks are ongoing, but fine-tuning processes remain a significant challenge [33]. In this regard, comprehensive tools like the LLM-PBE toolkit have been developed to address such privacy issues. LLM-PBE provides a structured approach to assessing privacy risks throughout the entire lifecycle of LLMs, offering insights into how factors such as model size, data characteristics, and temporal dimensions may influence the risk of data leakage. Through its analysis of various attack and defense strategies, LLM-PBE serves as a critical resource for understanding and mitigating data privacy issues in LLMs [39].

Adversarial training is a promising methodology for improving the robustness of LLMs against privacy attacks by exposing them to adversarial examples during the training process [40]. These adversarial examples help the model develop resistance to privacy attacks, improving its resilience to privacy breaches without significantly impacting performance. Another area of investigation concerns the vulnerability of fine-tuning methods in LLMs. While fine-tuning is necessary for adapting LLMs to specific tasks, there is a significant risk of data leakage if the fine-tuning process is not adequately secured. Li et al. (2024) [40] classify different privacy attacks on LLMs based on adversary capabilities and provide an overview of defenses developed to counter these attacks. Their work emphasizes the importance of privacy-preserving fine-tuning techniques, such as differential privacy and encrypted model updates, which help protect sensitive information during model adaptation. They also suggest several future research directions, emphasizing the growing need for more robust defenses as LLMs continue to evolve and become more widely deployed [40]. Future research will, therefore, focus on designing LLMs that are resistant to privacy attacks without compromising performance [5]. Additionally, efforts should prioritize fine-tuning methods that prevent data leakage during the adaptation of pre-trained models to specific tasks, ensuring that LLMs remain both effective and secure for real-world applications [40].

Privacy-preserving methods like selective forgetting and model scrubbing have been proposed as potential solutions to mitigate these risks. Additionally, incorporating privacy-preserving mechanisms during fine-tuning is increasingly viewed as critical to addressing vulnerabilities in LLMs. Privacy concerns about sensitive data leakage during training or fine-tuning highlight the need for trustworthy AI frameworks, as explored by Feretzakis and Verykios (2024), who discuss the importance of securing sensitive data within LLMs [41].

To effectively address privacy attacks in LLMs, it is crucial to incorporate privacy-preserving techniques that are designed with a clear understanding of the legal implications of data protection. This includes implementing safeguards against model inversion and membership inference attacks that could compromise the privacy of individuals whose data contribute to the training process. By focusing on protecting users’ personal information from unauthorized extraction and aligning defense strategies with legal requirements, developers can enhance the trustworthiness of LLMs. Additionally, ongoing monitoring and auditing of models for potential privacy vulnerabilities are essential practices that support compliance with data protection regulations and uphold ethical standards.

5.6. Legal and Ethical Frameworks for Privacy in Generative AI

As AI technologies become more integrated into everyday life, ethical concerns about privacy, data protection, and unforeseen side effects of AI outputs have become a central focus. As Al-kfairy et al. (2024) [42] point out, generative AI technologies, such as deep learning models, raise new ethical concerns, including copyright infringement, bias, misinformation, and privacy violations. These technologies—particularly those involved in generating deepfake media—may challenge the very foundations of truth, trust, and democratic values. The call for robust ethical frameworks that enhance equity, transparency, and human-rights protection across sectors is a critical interdisciplinary challenge.

One area that requires rapid attention in generative AI systems is the balance between data protection and innovation. As AI systems become more capable, the risk of data misuse—whether through inadvertent memorization or intentional exploitation—grows. Al-kfairy et al. (2024) emphasize the need to develop legal frameworks and guidelines for generative AI that address its complexities while ensuring compliance with data protection regulations and supporting innovation. Interdisciplinary collaboration between policymakers, technologists, and researchers is essential to establish best practices for the responsible development of AI. Ethical AI systems must proactively consider human rights and avoid reinforcing social inequalities.

Moreover, generative AI technologies must comply with privacy regulations like the GDPR and CCPA, which emphasize data minimization and the right to erasure—requirements that are particularly challenging for AI systems that rely on large datasets. Techniques such as differential privacy, federated learning, and encrypted model updates are increasingly being incorporated into generative AI systems to address these challenges. As AI technologies continue to evolve, legal frameworks must also adapt to emerging privacy concerns, such as the use of synthetic data in healthcare and finance, where regulatory compliance is critical [42].

The ethical development of generative AI must also proactively address privacy-related concerns, such as data protection and privacy violations. While issues like algorithmic bias and data discrimination are equally important ethical concerns, they are orthogonal to privacy and would require separate and more focused discussions. In this section, we remain focused on privacy challenges while recognizing the broader ethical implications of AI development, which deserve independent consideration in other studies. AI systems trained on biased datasets can exacerbate social inequality. According to Al-kfairy et al. (2024), transparency in AI decision-making processes and the implementation of fairness guidelines are essential steps to reduce the risks of bias and ensure that generative AI contributes to socially equitable outcomes. Overall, as generative AI continues to evolve, so too must legal and ethical frameworks, ensuring that the development of these technologies is both socially responsible and ethically sound [42]. Research is needed to harmonize these frameworks with the latest advancements in AI technology.

Developing robust legal and ethical frameworks for privacy in generative AI involves a multidisciplinary approach that bridges technology, law, and ethics. Organizations must proactively engage with legal experts to ensure that their AI systems comply with data protection laws and respect individuals’ rights to privacy. This includes understanding the nuances of consent, data ownership, and the right to be forgotten, and integrating these considerations into the AI development lifecycle. By doing so, generative AI technologies can be advanced in a manner that not only drives innovation but also upholds societal values and legal obligations, thereby fostering public trust and acceptance.

5.7. AI and Quantum Cryptography for Privacy Preservation

With the constant development of AI and quantum cryptography, this interdisciplinary convergence holds great promise for creating powerful privacy-preserving technologies in AI applications. Quantum cryptography is an advanced cryptographic technique that leverages quantum mechanics to secure communication channels. It ensures that any attempt to intercept or eavesdrop on data in transit will result in detectable anomalies, making it inherently secure [43].

Many experts believe that combining AI with quantum cryptography will be essential for addressing future security challenges, especially the “quantum threat” posed by the emergence of quantum computers [43,44]. Their studies explore how AI-enhanced quantum cryptographic techniques could revolutionize security by developing stronger defenses against quantum threats, making these systems more adaptive and resilient. These computers could potentially break most existing cryptographic algorithms in a short time. AI, which excels at processing large volumes of data and recognizing patterns, has the potential to enhance quantum cryptographic procedures by making them stronger and more efficient. Quantum cryptography, in turn, will protect AI systems, ensuring that sensitive data handled by AI remain secure [43].

According to Radanliev (2024) [43], integrating neural networks (NNs) into AI-enhanced quantum cryptography could revolutionize cryptographic systems by increasing their efficiency and resilience. This is especially important, as quantum computers are increasingly perceived as a threat to traditional encryption methods. AI-driven cryptography offers a pathway to developing stronger defenses against emerging quantum threats, making quantum cryptography more adaptive, efficient, and secure, even against quantum computers [43].

In parallel, red teaming methodologies are being developed to make AI and quantum cryptographic systems more resilient to cyber attacks. Radanliev, De Roure, and Santos (2023) [44] conducted an in-depth review of the convergence of AI and quantum computing, focusing specifically on natural language processing (NLP) models and quantum cryptographic protocols, including the BB84 method and NIST-approved quantum-resistant algorithms. Their research simulates various cyber-attacks using Python and C++ to evaluate the strength of quantum security measures through iterative testing and simulations [44]. This red teaming methodology is crucial in preparing for the threats posed by quantum-enhanced AI systems, ensuring future digital operations are secure and resilient against AI-driven cyber-attacks.

As these fields continue to evolve, addressing the challenges of integrating AI with quantum cryptography is essential. Key challenges include overcoming the computational complexity of quantum cryptographic protocols and ensuring that AI systems can operate efficiently within these frameworks. Despite these challenges, the potential benefits of this interdisciplinary approach are immense, with significant implications for the future of digital security and privacy protection [43,44].

As AI systems increasingly integrate with quantum cryptography to enhance privacy preservation, it is imperative to address the legal and ethical considerations associated with these emerging technologies. Organizations must ensure that the deployment of quantum cryptographic methods aligns with data protection regulations and does not inadvertently introduce new privacy risks. This includes evaluating the potential impact on data subjects, ensuring transparency in how data are secured, and considering the implications of quantum-resistant encryption on data accessibility and user rights. By carefully navigating these legal and ethical landscapes, the convergence of AI and quantum cryptography can contribute to stronger privacy protections while maintaining compliance with regulatory standards.

5.8. Emerging Applications in Generative AI-Enabled Networks

As generative AI continues to evolve, its applications extend beyond traditional sectors like healthcare and finance into more complex and critical environments, such as communication networks. One notable area is the integration of generative AI in space–air–ground integrated networks (SAGINs). A SAGIN represents a comprehensive system that combines space, air, and ground components to provide seamless connectivity in future 6G and beyond networks. Generative AI models offer promising solutions for enhancing the performance and security of such networks by enabling more advanced decision-making processes and generating synthetic data to optimize operations. For instance, recent research explores how generative AI can improve channel modeling, resource allocation, and security in SAGINs, leveraging diffusion models to enhance the quality of service and optimize network resources [45].

In addition to enhancing traditional communication networks, generative AI agents equipped with LLMs have demonstrated the ability to address complex problems in satellite communication systems. Satellite networks face significant challenges, including transmission interference and resource management issues, especially when scaling to accommodate a massive number of users. By employing a mixture of experts (MoE) approach combined with LLM-driven retrieval-augmented generation (RAG), generative AI agents can design more efficient transmission strategies and optimize satellite communication processes. This method has proven effective in optimizing variables through specialized training and improving adaptability in customized modeling problems for satellite communications [46].

These applications of generative AI in communication networks underscore the vast potential of AI-driven technologies to not only address privacy concerns but also enhance performance and resource management in highly complex systems. The exploration of generative AI-enabled networks exemplifies the expanding frontier of AI research and highlights the importance of continued innovation in real-world applications.

6. Conclusions

The rapid development of generative AI has transformed many industries with unprecedented advancements in data generation and modeling capabilities, raising serious privacy concerns that require urgent and ongoing scrutiny. These models have become powerful enough to generate coherent, contextually relevant text that feels human-like. Among recent releases are GPT-4o, o1 series by OpenAI, Gemini 1.5 Pro from Google DeepMind, Claude 3.5 Sonnet by Anthropic, and LLaMA 3.2 by Meta AI. Throughout this review, we have explored numerous privacy vulnerabilities associated with generative AI, including data memorization, model inversion attacks, membership inference attacks, and data leakage during fine-tuning. These issues are particularly critical in domains such as healthcare, finance, and legal services, where privacy breaches can have severe consequences.

Several techniques for preserving privacy have been explored and deployed. Differential privacy adds carefully calibrated noise to the outputs of computations or queries on data to protect individual privacy, while federated learning enables decentralized model training without sharing raw data by allowing models to be trained across multiple devices holding local data samples. Homomorphic encryption (HE) allows computations on encrypted data, and secure multi-party computation (SMPC) enables parties to jointly compute functions on private inputs without revealing them. Privacy-enhancing technologies (PETs), along with anonymization and masking methods, play a vital role in protecting sensitive information.

However, these techniques still face challenges in balancing model performance, scalability, and privacy preservation. Differential privacy can degrade model accuracy, while HE and SMPC introduce computational overheads, making large-scale or real-time applications difficult. Integrating PETs into mainstream AI workflows remains a complex task that requires thoughtful system design and resource allocation.

Emerging trends such as blockchain integration and post-quantum cryptography point toward promising avenues for enhancing privacy in generative AI. Blockchain offers a transparent, tamper-proof audit trail, improving data integrity and compliance. Post-quantum cryptography is crucial for protecting AI systems from future threats posed by quantum computing, which could undermine current cryptographic security.

As adversarial attacks on AI systems become more sophisticated, privacy-preserving techniques must evolve to keep pace. This includes addressing privacy attack methods that exploit fine-tuning weaknesses. Future research should focus on selective forgetting, privacy-preserving fine-tuning, and real-time privacy auditing to ensure AI systems remain secure.

In light of the evolving threat landscape, it is also critical to clearly define the privacy models that apply in different generative AI use cases. Brand and Pradel [47] highlight the need for explicit articulation of whose privacy is being protected, what kind of privacy is being secured, and against whom this privacy is being defended. They emphasize that different stages of the AI lifecycle introduce distinct privacy risks, requiring customized privacy-preserving techniques for each phase. By ensuring that these methods are tailored to specific privacy threats—such as protecting training data from inversion attacks or safeguarding user interactions during model inference—organizations can better align privacy strategies with actual risks and legal frameworks.

Legal and ethical considerations are also paramount. The evolving regulatory landscape, particularly with AI-specific regulations such as the GDPR, requires AI systems to go beyond data protection and adhere to principles of fairness, transparency, and accountability. In addition, as Krasadakis et al. (2024) emphasize in their survey on NLP advances [48], legal informatics and low-resource languages face unique privacy challenges that demand targeted privacy-preserving solutions. Organizations must ensure compliance with these regulations to avoid infringing on individual rights or societal values.

While privacy-preserving techniques such as differential privacy, federated learning, homomorphic encryption, and others provide valuable tools for mitigating privacy risks, it is essential to acknowledge that achieving absolute privacy is mathematically impossible. These technical solutions cannot guarantee 100% privacy due to inherent limitations and the evolving nature of threats. Therefore, organizations must not solely rely on technological measures and must also ensure that their practices align with legal and regulatory frameworks. Compliance with data protection laws like the GDPR and HIPAA requires a holistic approach that combines technical safeguards with legal strategies and ethical considerations. By collaborating with legal authorities and adhering to established guidelines and standards, organizations can better navigate the complexities of privacy preservation, reduce liability risks, and foster public trust in generative AI systems.

Generative AI and privacy protection remain a rapidly evolving frontier, demanding continuous research, innovation, and collaboration. The path forward will require a delicate balance between leveraging the powerful capabilities of generative AI and protecting individual privacy. This will be one of the defining challenges of the coming years. With a focus on privacy preservation by design, AI systems can reduce risks, gain public trust, and become valuable tools for positive societal impact.