2.2.1. Structure Enhancement of GANs

Deep Regret Analytic GAN [16] (DRAGAN) proposes viewing GAN training as a regret minimization process rather than divergence minimization between real and fake distributions. Through theoretical analysis of GAN convergence under this regret minimization process, the authors connect common training instabilities like mode collapse to undesirable local equilibria. They identify sharp discriminator gradients around some real data points as a characteristic of these poor local equilibria. To address this, the authors propose DRAGAN, a gradient penalty scheme that avoids degenerating local equilibrium and enables faster, more stable training with fewer mode collapses.

Wasserstein GAN [17] (WGAN) introduces a key change in the training of a discriminator. Instead of classifying inputs as real or fake, WGAN changes the discriminator to a critic that outputs real-valued estimates. This is achieved using the Wasserstein distance, a measure of distance in probability space, which provides a smoother and more meaningful gradient to the generator. Although WGAN shows improved training instability, WGAN uses weight clipping to enforce Lipschitz’s constraint on the critic, which can still lead to poor performance and convergence failures. WGAN with gradient penalty [18] (WGAN-GP) proposes to penalize the gradient norm of the critic’s input rather than clipping weights, which not only enhances the stability of WGANs but also supports stable training across various GAN structures with minimal hyperparameter adjustments.

CVAE-GAN [52] merges a VAE with a GAN to synthesize images in specific, fine-grained categories. It represents images as a combination of label and latent attributes within a probabilistic model, allowing for the generation of category-specific images by altering the category label and varying values in a latent attribute vector.

Progressive growing GAN [20] (PGGAN) was proposed to synthesize high-resolution human faces. The key idea of PGGAN is to improve the generator and the discriminator progressively. In detail, PGGAN starts generation from a model structure that can only generate low-resolution images. Then, PGGAN adds new layers that model increasingly fine details as training progresses, and the details of generated images, as training progresses, are also becoming more and more abundant. By adding details progressively, PGGAN can generate high-resolution images. Compared with low-resolution images, high-resolution images can better persuade recipients of content authenticity and better evade visual detection.

Least square GAN [21] (LSGAN) focuses on the vanishing gradient issues during training. This is because regular GANs model the discriminator as a classifier with sigmoid cross entropy loss. LSGAN proposed a least square loss for the discriminator. The least square loss is applied in an a-b coding scheme for the discriminator, where “a” and “b” represent the labels for fake and real data. LSGAN shows its effectiveness in improving both training stability and generated image quality.

BigGAN [22] added the orthogonal regularization and truncation trick in the large-scale training of GAN to achieve both high-fidelity images and a wide variety of generated samples.

Spectral normalization GAN [23] (SNGAN) stabilizes GAN training by normalizing discriminator weights in each iteration. Spectral normalization constrains the Lipschitz constant and limits the capacity of the discriminator. This prevents unstable oscillatory dynamics between the generator and discriminator during adversarial learning.

Auxiliary classifier GAN [24] (AC-GAN) improved training techniques for GANs to synthesize high-resolution 128 × 128 images with global coherence. The proposed label-conditioned GAN structure generates samples with clearer object details compared to lower-resolution images. Besides, the progressive growing technique also speeds up the training process and enhances its stability.

Adaptive GAN [25] (AdaGAN) aimed at addressing the problem of missing modes in GANs, which refers to the inability of a model to produce examples in certain regions of the data space. AdaGAN is an iterative procedure that enhances the training of GANs by adding a new component to a mixture model at each step. This is achieved by running a GAN algorithm on a reweighted sample, drawing inspiration from boosting algorithms.

Unrolled GAN [26] (UGAN) proposed a novel method to stabilize the training of GANs by redefining the generator’s objective in relation to an unrolled optimization of the discriminator. UGAN addresses a critical challenge in GAN training: balancing between using the optimal discriminator in the generator’s objective, which is theoretically ideal but practically infeasible, and relying on the current value of the discriminator, which often leads to instability and suboptimal solutions.

U-Net GAN [27] proposes a U-Net-based discriminator structure for GANs to improve global and local image coherence, which provides detailed per-pixel feedback in addition to global image assessment. This guides the generator in synthesizing shapes/textures that are indistinguishable from real images. A per-pixel consistency regularization is also introduced using CutMix augmentation on the U-Net input. This focuses the discriminator on semantic/structural changes rather than perceptual differences. The proposed U-Net discriminator and consistency regularization improve GAN performance, enabling the generation of images with shapes, textures, and details highly faithful to the real data distribution.

FastGAN [28] studies few-shot high-resolution image synthesis with GANs using minimal computation and limited a small number of training samples. The proposed FastGAN is a lightweight GAN structure that is capable of producing superior image quality at a resolution of 1024 × 1024, with the training process requiring only a few hours on a single GPU.

2.2.2. Deepfake Synthesized With GANs

Deep convolutional GAN [29] (DCGAN) was proposed to synthesize new faces with a class of CNNs. By applying a set of skills to the structure of networks, such as replacing pooling layers with convolutions, adopting BatchNormalization (BN), and adopting a proper activation function, DCGAN stabilizes the training of the network. Though only able to generate low-resolution human faces, DCGAN is considered an early and classic work for fake face generation.

HistoGAN [30] was proposed to deal with the color inconsistency of GAN-generated images. Inspired by the idea of “style mixing” in StyleGAN [31, 32], HistoGAN first obtains histogram features by converting images into the log-chroma space. Then, a GAN network is established by replacing the last two blocks of StyleGAN a with the color histogram feature adopted to learn a lower dimensional representation. With the output of the network and a histogram-aware color-matching loss, the generated images better match the color histogram of the target images.

CVAE-GAN [19] combines a VAE with a GAN to synthesize images in fine-grained categories, such as faces of a specific person or objects in a category. CVAE-GAN models an image as a composition of labels and latent attributes in a probabilistic model. By varying the fine-grained category label fed into the resulting generative model, CVAE-GAN generates images in a specific category with randomly drawn values on a latent attribute vector.

The StyleGAN series comprises a progression of Deepfake generation models featuring progressively enhanced generative capabilities. StyleGAN [31] was proposed by making improvements to PGGAN. StyleGAN does not feed the learned latent code to the generator. Instead, latent code is applied for style mixing before arriving at the generator. Style mixing aims to discover the position of latent code that controls different styles by a mapping network. The latent code is mapped to style code through the network, and the style code is mixed and then fed to the generator. Style mixing allows StyleGAN to control style at each layer in the generator. Besides, StyleGAN introduced noise at each layer in the generator to improve generation diversity. By style mixing and introduction of noise in the generator, StyleGAN synthesizes high-quality images of high resolution and better diversity. StyleGAN2 [32] focuses on the water droplet disadvantage of StyleGAN. It is pointed out that the water droplet appears due to the AdaIN operation, which normalizes each feature map separately and destroys the information between feature maps. StyleGAN2 adopted weight demodulation to remove the water droplet while retaining full controllability. Noticing that the details of generated images are unnaturally fixed to pixel coordinates rather than object surfaces due to aliasing in the generator, StyleGAN3 [33] analyzed that the root cause of this is traced to careless signal processing. Interpreting all signals as continuous, StyleGAN3 derives small architectural changes to prevent unwanted information from leaking into the hierarchical synthesis. StyleGAN-V [34] focuses on video generation by treating videos as continuous signals. StyleGAN-V is a continuous-time video generator that leverages continuous motion representations through positional embeddings. StyleGAN-V also adopted a holistic discriminator that aggregates temporal information more efficiently by concatenating frame features. By training the model on high-resolution images, StyleGAN-V generates high-resolution long videos at high frame rates. StyleGAN-T [35] is a model designed to meet the specific demands of large-scale text-to-image synthesis. Compared with existing generators that require iterative evaluation to generate a single image, StyleGAN-T offers a much faster alternative as they can generate images in a single forward pass. StyleGAN-T is also designed to meet the specific demands of large-scale text-to-image synthesis, including large capacity, stable training on diverse datasets, strong alignment with text inputs, and a balance between controllable variation and text alignment.

Dynamic aspect-aware GAN [36] (DAE-GAN) is a text-to-image synthesizer representing text at multiple levels such as sentence and word aspect. DAE-GAN mimics human iterative drawing by globally enhancing the image and then locally adjusting details based on text. In detail, DAE-GAN generates an initial image from the sentence embedding and then refines it using a novel aspect-aware dynamic re-drawer (ADR). ADR alternates between global refinement with word vectors (attended global refinement) and local detail refinement with aspect vectors (aspect-aware local refinement).

3D-GAN [37] is capable of generating 3D objects by leveraging volumetric convolutional networks and GANs to create 3D objects from a probabilistic space. 3D-GAN features an adversarial criterion, which enables the generator to implicitly understand and replicate the structure of 3D objects, resulting in high-quality outputs. 3D-GAN can generate 3D objects without reference images or CAD models, exploring the manifold of 3D objects.

VQGAN [38] aims to generate long videos by combining 3D-VQGAN with transformers to generate videos comprising thousands of frames. Trained on various datasets, VQGAN is able to generate high-quality long videos of impressive diversity and coherency. By incorporating temporal information with text and audio, VQGAN can generate meaningful long videos that are contextually rich and aligned with the given inputs.

Semantic-spatial aware GAN [39] (SSA-GAN) is designed to generate photorealistic images that are not only semantically consistent with the overall text description but also align with specific textual details. In detail, SSA-GAN enhances the fusion of text and image features and introduces a semantic mask learned in a weakly supervised manner to guide spatial transformation in the image.

GT-GAN [40] generates time series data by handling both regular and irregular time series data without requiring modifications to the model. GT-GAN integrates a range of related techniques, from neural ordinary/controlled differential equations to continuous time-flow processes, into a cohesive system. This integration is a critical factor in the GT-GAN’s success.

2.2.3. Deepfake Manipulated With GANs

StarGAN [41] enables image-to-image translations across multiple domains with a single model. StarGAN’s unified structure allows for simultaneous training on multiple datasets with different domains within one network. This leads to superior quality in translated images compared to existing models and provides the flexibility to translate any input image into any desired target domain. StarGAN can be used on facial attribute transfer and expression synthesis. StarGANv2 [42] presents a single, scalable framework for diverse high-quality image-to-image translation across multiple domains, unlike existing approaches with limited diversity or separate models per domain. Through experiments on CelebA-HQ and a new animal faces dataset AFHQ, StarGAN v2 demonstrates superior visual quality, diversity, and scalability compared to baselines. The release of AFHQ provides a valuable benchmark for assessing image translation models in terms of large inter- and intradomain variability. Overall, StarGAN v2 tackles the key criteria of diversity and scalability for multidomain image mapping within a single unified generator model.

FSGAN [43] enables face swapping and reenactment between pairs of faces without requiring model training on those faces specifically. Key technical contributions include an RNN-based approach to adjust for varying poses and expressions, continuous view interpolation for video, completion of occluded regions, and a blending network for seamless skin and lighting preservation. A novel Poisson blending loss integrates Poisson optimization and perceptual loss. FSGANv2 [44] is unique in that it is subject agnostic, enabling face swapping on any pair of faces without specific training on those faces. It features an iterative DL-based approach for face reenactment, capable of handling significant pose and expression variations in both single images and video sequences. For videos, it offers continuous interpolation of face views using reenactment, Delaunay triangulation, and barycentric coordinates. A face completion network manages occluded face regions, and a face blending network ensures seamless blending while preserving skin color and lighting conditions, utilizing a novel Poisson blending loss.

AttGAN [45] was proposed to make more accurate facial attribute manipulation. The proposed method is based on the observation that attribute-independent latent representation applied in some attribute manipulation methods restricts the capacity of the latent representation. Thus, the generated human faces are oversmooth and distorted. To make an improvement, AttGAN proposed reconstruction learning and adversarial learning to make the generated images visually more realistic. In detail, the proposed reconstruction learning preserves attribute-excluding details by making attribute editing, and the reconstruction task shares the entire encoder–decoder network. The proposed adversarial learning provides the whole network with the capacity for visually realistic editing. Thus, AttGAN can achieve high-quality facial attribute manipulation by making faces more natural.

CAGAN [46] tackles automatic global image editing from linguistic requests to significantly reduce labor for novices. An editing description network (EDNet) is proposed to predict embeddings for image generator cycles. Despite data imbalance, several augmentation strategies combined with an Image-Request Attention module enable spatially adaptive edits. A new semantic evaluation metric is also introduced as superior to pixel losses. Extensive experiments on two benchmarks demonstrate state-of-the-art performance, with effectiveness stemming from explicitly handling limited language-image examples and selective region-based editing guided by linguistic cues.

CycleGAN [47] is an image-to-image translation GAN, which has been used for face-swapping and expression reenactment manipulation. The basic idea under CycleGAN design is interesting. It assumes that for two domains, the source domain and the target domain, the mapped target domain representation of an input, when mapped in the opposite direction, the backward-mapped result should be close to the input, which also explains its name. Based on this idea, the cycle loss is proposed to optimize the whole network.

SPMPGAN [48] achieves semantic image editing by generating desired content in specified regions based on local semantic maps. SPMPGAN introduces a style-preserved modulation with two steps: fusing contextual style and layout to generate modulation parameters and then employing those to modulate feature maps. This injects a semantic layout while maintaining context style. A progressive structure further enables coarse-to-fine-edited content generation. By preserving style, more consistent results are obtained with less noticeable boundaries between edited regions and known pixels. Overall, explicit style modeling significantly improves semantic editing quality and coherence.

Anyconst GAN [49] enables interactive image editing with GANs by allowing fast, perceptually similar previews at reduced computational costs. Inspired by quick preview features in creative software, Anyconst GAN supports variable resolutions and channels for generation at adjustable speeds. Subset generators serve as proxies for full results, trained using sampling, adaptation, and conditional discrimination to improve visual quality over separately tuned models. Novel projection techniques also encourage output consistency across subsets. Anyconst GAN provides up to 10x cost reduction and 6–12x preview speedups on CPUs and edge devices without perceivable quality loss, crucially enabling interactivity.

T2CI-GAN [50] addresses the challenge of generating visual data from textual descriptions, a task that combines natural language processing (NLP) and computer vision techniques. While current methods use GANs to create uncompressed images from text, T2CI-GAN focuses on generating images directly in a compressed format for enhanced storage and computational efficiency. T2CI-GAN employs DCGAN and proposes two models: One trained with JPEG-compressed DCT images to generate compressed images from text and another trained with RGB images to produce JPEG-compressed DCT representations from text.

StyleCLIP [51] manipulates images using the StyleGAN framework, inspired by its ability to generate highly realistic images. The focus is on utilizing the latent spaces of StyleGAN for image manipulation, which traditionally requires extensive human effort or a large annotated image dataset for each manipulation. StyleCLIP introduces the innovative use of contrastive language-image pretraining (CLIP) models to develop a text-based interface for StyleGAN image manipulation, eliminating the need for manual labor. The approach includes an optimization scheme that modifies an input latent vector based on a text prompt using a CLIP-based loss. A latent mapper is also developed to infer text-guided latent manipulation steps for input images, allowing quicker and more stable text-based manipulation. The paper also presents a method to map text prompts to input-agnostic directions in StyleGAN’s style space, facilitating interactive text-driven image manipulation.

2.3.1. Detection of GAN-Synthesized Deepfake

2.3.2. Detection of GAN-Manipulated Deepfake

Neural ODE [90] was proposed and trained on original videos’ heart rates. Testing on commercial and database Deepfake, the model successfully predicts fake heart rates. By evidencing detectable physiological signals in synthesized video, this exploration of Neural-ODE-based heart rate prediction represents a novel approach to automatically discerning real from forged footage.

Face X-ray [91] reveals blended boundaries in forged faces without relying on manipulation technique specifics. Observing a common blending step across methods, face x-ray exposes composites via blending edges in forged images and absence in real ones. Requiring only blending logic at training, face X-ray generalizes to unseen state-of-the-art techniques, unlike existing detectors reliant on method artifacts.

Matern et al. [55] investigated detecting artifacts in video face editing to expose manipulated footage. Many algorithms exhibit classical vision issues from tracking/editing pipelines. Analyzing current facial editing approaches identifies characteristic flaws like Deepfakes and Face2Face. Simple visual feature techniques can effectively expose such manipulations, using explainable signals accessible even to non-experts. These methods easily adapt to new techniques with little data yet achieve high AUC despite simplicity.

Durall et al. [60] made a theoretical analysis and suggested that upsampling (up-convolutional) operation in generative models leaves frequency artifacts in generated images, which may lead to GAN fingerprints. They further noticed that frequency artifacts exist, especially in the high-frequency spectrum, which is consistent with the observation in [96] that high-frequency components carry more GAN fingerprint information. Thus, the GAN fingerprint can be considered a consequence of frequency distortion in the generative model caused by upsampling operations. The observation of fingerprints makes sense, since it is closely related to frequency distortions and inspired many works to look for Deepfake clues in the frequency domain.

Wang et al. [92] explore a “universal” detector to distinguish real images from various CNN-generated images regardless of structure or dataset. With preprocessing, augmentation, and a classifier trained only on ProGAN, surprising cross-structure and dataset generalization are achieved, even with methods like StyleGAN2. The findings suggest modern CNN-generated images share systematic flaws preventing realistic synthesis, evidenced by the success of the simple universal fake detector.

Dang et al. [93] tackle detecting manipulated faces and localizing edited regions, which is crucial as synthesis methods produce highly realistic forgeries. Simply using multitask learning for classification and segmentation prediction provides limited performance. Instead, an attention mechanism is proposed to process and improve classification-focused embeddings by highlighting informative areas. This simultaneously benefits binary real versus fake classification and visualizes manipulated areas for localization. A large-scale fake face dataset enables thorough analysis, showing attention-enhanced models outperform baselines in detecting facial forgeries and revealing the edits through interpretable attention maps. Overall, selective feature improvement and visualization via attention mechanisms provide superior hybrid detection and localization of modern forged face imagery.

Chai et al. [63] proposed to recognize Deepfake by the analysis of image patches, where the final classification was not made globally on the image. Instead, a shallow network with local receptive fields was applied to first predict whether patches are real or fake. The final decision of image authenticity is made by aggravating the classification results of all patches. In this way, the general classifier achieves both detection and localization tasks.

FakeLocator [64] tackles detecting and locating GAN-based fake faces by tracing upsampling artifacts within generation pipelines. Observing such imperfections provides valuable clues. A technique called FakeLocator is proposed to produce high-accuracy localization maps at full resolution. An attention mechanism guides training for universality across facial attributes. Data augmentation and sample clustering further improve generalization over various Deepfake methods. By capitalizing on tell-tale upsampling artifacts, FakeLocator advances multimedia forensics by exposing manipulation boundaries in addition to classification.

M2TR [71] detects Deepfakes through multiscale transformers capturing subtle manipulations. Most methods map images to binary predictions, missing consistency patterns indicative of forgery. Multi-modal Multi-scale TRansformer (M2TR) applies transformers across patch scales, exposing local inconsistencies. Frequency information further improves detection and compression robustness via cross-modality fusion. A new high-quality Deepfake dataset, SR-DF, addresses the limitations of diversity and artifacts in existing benchmarks.

2.3.3. Detection Performance Comparisons

We also compare the detection performance of some state-of-the-art methods in Table 6. We compared the performance of the detector on the test subset of the training dataset (in-set) and its generalization performance on an unseen dataset (cross-set). It can be observed that the most Deepfake detectors can perform well and achieve over 90% classification accuracy and AUC, some even achieve over 99% prediction accuracy (F³-Net and FakeLocator). However, the generalization ability of these detectors remains a significant challenge, as their performance drops considerably when evaluated on the cross-set. The generalization ability is heavily influenced by the data distribution differences between the training and testing datasets. Detectors generally perform better on datasets with similar data distributions to the training set. For instance, UIA-ViT and Face X-ray demonstrate strong generalization when tested on datasets with similar context, quality, and identities to the training set, achieving over 90% AUC and accuracy, respectively. These generalization challenges highlight the primary difficulty in deploying Deepfake detectors in real-world applications, where the diversity of data (contexts, qualities, and identities) can vary greatly.

3.3.1. Structure Enhancement of DMs

The trailblazing work of DMs is the denoising diffusion probabilistic model (DDPM) [97]. DDPM is a class of latent variable models drawing connections to nonequilibrium thermodynamics. Based on a link between DMs and denoising score matching with Langevin dynamics, a weighted variational training approach is proposed in DDPM. This also enables a progressive lossy decoding scheme generalizing autoregressive generation. By formalizing connections to thermodynamic diffusion and denoising objectives, this work advances DMs to match or surpass GAN performance on image datasets. Theoretical grounding and interpretable decoding are valuable properties alongside the sample quality achievements.

DDIM [98] accelerates sampling for DDPM. While DDPMs generate high-quality images without adversarial training, sampling is slow as it requires simulating a Markov chain. DDIMs instead utilize non-Markovian diffusion processes corresponding to faster deterministic generative models. The same training procedure as DDPMs is used. Experiments show DDIMs produce equal quality samples up to 50x faster than DDPMs in terms of wall-clock time. DDIMs also enable computation-quality trade-offs, semantically meaningful latent space interpolation, and low-error observation reconstruction.

Song et al. [99] present a framework unifying score-based generative modeling and diffusion probabilistic models under continuous stochastic processes. A forward SDE injects noise to perturb data into a known prior, while a reverse-time SDE generates back to the data distribution by removing noise. The key insight is the reverse SDE that only requires estimating the score (gradient) of the perturbed data. Neural networks estimate the scores for accurate generative modeling. Encapsulating prior methods, new sampling procedures and capabilities are introduced—including a predictor–corrector to correct errors and an equivalent neural ODE for sampling and likelihood evaluation. The unified perspective also enables high-fidelity 1024 × 1024 image generation for the first time among score-based models. By bridging score modeling and diffusion through continuous processes, the work significantly improved both sampling techniques and sample quality.

Improved DDPM [100] proposed that simple modifications to the standard DDPM framework allow for high-fidelity generation and density estimation. Learning variances of the reverse diffusion further reduces sampling cost by 10x with negligible quality loss, which is important for practical deployment. Precision–recall metrics show that DDPMs cover the target distribution better than GANs. Crucially, DDPM performance smoothly scales with model capacity and compute, enabling straightforward scalability. By enhancing DMs to match or exceed GANs in sample quality, likelihood evaluation, and efficiency, this work greatly expands their capabilities and viability as generative models for images. The demonstrations of precise control over quality-compute trade-offs and scalability further attest to their value for real-world usage.

Iterative latent variable refinement (ILVR) [101] addresses controllable image generation by guiding the stochastic diffusion process in DDPMs. A method called ILVR leverages the generative reversibility of DDPMs to synthesize high-quality images matching a reference image. By iterative refinement, a single unconditional DDPM can adaptively sample diverse sets specified by references without retraining. Applications in multiscale image generation, multidomain translation, paint-to-image synthesis, and scribble-based editing demonstrate controllable generation using a single model. Enhanced control addresses a key limitation of DDPMs while retaining benefits like efficient sampling and good image fidelity. By steering random diffusion trajectories toward desired targets, ILVR significantly advances both the flexibility and practicality of DMs for computer graphics tasks demanding guided synthesis.

ADM [102] improved DMs for unconditional and conditional image generations through structure enhancements and classifier guidance, a method that effectively trades off diversity for fidelity using gradients from a classifier. ADM reports remarkable results on various benchmarks, achieving lower FID scores on datasets like ImageNet, which indicates higher quality and diversity in generated images. Furthermore, the paper discusses the societal impacts and limitations of DMs, highlighting their potential for wide-ranging applications while also acknowledging the challenges in sample speed and the absence of explicit latent representations.

Ho and Salimans [103] introduce classifier-free guidance for conditional DMs to improve sample quality without requiring a separate classifier. Classifier guidance leverages classifier gradients to trade off diversity and fidelity, needing an additionally trained classifier. Instead, a conditional and unconditional DMs are jointly trained. Combining the conditional and unconditional score estimates creates a similar fidelity–diversity spectrum as classifier guidance. This classifier-free technique avoids the cost and complexity of a second model. By showing the generative model alone that contains all necessary signals for controllable generation post-training, this exploration makes guidance simpler and more efficient for denoising diffusion.

Stable diffusion (SD) [104] formulates diffusion generative modeling in latent spaces of pretrained autoencoders. While pixel-based models achieve a promising synthesis, their optimization is expensive and sampling sequentially. By balancing complexity reduction and detail preservation in latent spaces, SD reaches near-optimal trade-offs, enabling high-fidelity generation with greatly reduced computing. The latent formulation also allows convolutional high-resolution synthesis. With cross-attention layers, SD conditions are flexible on inputs like text and bounding boxes. Experiments across unconditional generation, class conditioning, inpainting, text-to-image synthesis, and super-resolution achieve new state-of-the-art results while significantly lowering cost compared to pixel models. By shifting the modeling domain from pixel to latent space, SD unlocks the practical deployment of DMs through substantial computational savings and conditional generation capabilities.

CycleDiffusion [105] formulates an alternative Gaussian latent space for DMs along with a reconstructing encoder mapping images into this space. While DMs typically utilize a denoising sequence space, unlike GANs and VAEs, CycleDiffusion explicitly models a continuous latent code and confers benefits. One key finding is emergent common latent spaces across DMs trained on related domains. This enables applications like unpaired image translation and text-to-image editing with a single encoder–decoder cycle. Additionally, the latent formulation unlocks unified guidance of DMs and GANs via energy-based control signals. Experiments on conditional image generation guided by CLIP demonstrate DMs better capture distribution modes and outliers than GANs. By exposing and harnessing continuous latent structures amid the sequential sampling process, Gaussian embedding confers new applications to leverage DMs intuitively as latent variable models.

DPM-Solver++ [106] proposed a single approach for both continuous and discrete-time DMs. By introducing more accurate numerical methods for solving the diffusion ODEs, DPM-Solver++ achieves high-quality sampling in significantly fewer steps than previous methods.

3.3.2. Deepfake Synthesized With DMs

GLIDE [107] explores DMs for text-to-image synthesis by comparing classifier-free and CLIP guidance strategies to trade off diversity and fidelity. The research demonstrates that samples from a 3.5 billion parameter text-conditional DM utilizing classifier-free guidance are more favored by humans than those from DALL-E, even when DALL-E employs CLIP re-ranking, a more resource-intensive process. The paper also reveals that these models can be fine-tuned for image inpainting tasks, enabling advanced text-driven image editing capabilities. This signifies an important stride in the field of AI-driven image generation and editing, offering more realistic and accurate outputs with more straightforward guidance techniques.

DALL-E [108] leverages CLIP image representations for controllable text-to-image generation. A two-stage model generates a CLIP embedding from text and then decodes this into an image. Explicitly modeling the intermediate representation improves sample diversity with minimal fidelity loss compared to end-to-end synthesis. The approach also enables intuitive image variations and edits by manipulating the compact encoding. DMs prove most effective for both stages, capturing semantics in the latent space and high-quality decoding. Compared to end-to-end conditional GANs and autoregressive models, explicit modeling and inversion of the robust CLIP space confer superior generation control and efficiency. By factorizing generation through a recognizable visual denoising manifold, the work significantly advances the state-of-the-art in flexible text-driven image synthesis.

Vector-quantized discrete diffusion model (VQ-DDM) [109] integrates vector-quantized variational autoencoders (VQ-VAEs) with DMs for discrete sequence generation. While VQ-VAEs provide rich codebooks and autoregressive decoding offers high fidelity, sequential pixel synthesis lacks global context. Denoising diffusion captures global dependencies but is expensive for images. The proposed VQ-DDM combines the strengths of both VQ-VAEs and denoising diffusions. Unlike autoregressive VQ approaches, the framework generates images with global coherence from compact codes without strict pixel order. VQ-DDM matches state-of-the-art image quality with greater efficiency than pixel diffusion while surpassing vector-quantized autoregressive models on tasks like inpainting needing nonlocal patterns. By synergizing discrete sequential synthesis with continuous diffusion mechanisms, the hybrid technique rectifies key limitations of both families for controllable image generation.

Imagen [111] is a text-to-image DM achieving new levels of realism and language understanding. By scaling up transformer structures for both text encoding and image decoding, Imagen reaches high FID on COCO without domain-specific fine-tuning. Human evaluations also show samples comparable to real data in image-text alignment. To comprehensively assess synthesis quality, the DrawBench benchmark is proposed, comparing Imagen to leading approaches on fidelity and alignment. Raters prefer Imagen over alternatives like VQ-GAN + CLIP and DALL-E 2 in side-by-side comparisons. By synergizing large language and image transformers in a diffusion framework, Imagen represents a major advance in programmatic image generation through intuitive text-based control.

Liu et al. [112] propose structured image generation using compositional DMs to address composition failures in large text-to-image models. By interpreting the diffusion process as energy-based models with combinable data distributions, separate component models specializing in visual primitives like objects and relations can be integrated at test time to render complex scenes. This compositional approach generalizes to intricate configurations unseen during training, accurately binding detailed language descriptions to image elements. Experiments demonstrate photorealistic rendering of sentence relations, facial attributes, and rare interobject arrangements—challenging cases for leading models like DALL-E 2. The framework advances structured conceptualization and binding through modular component assembly rather than using emergent reasoning in end-to-end models. By scaling synthesis in a human-inspired combinatorial manner, the technique makes significant progress on controllable image generation from intricate language specifications.

Continual diffusion [113] addresses catastrophic forgetting when sequentially adapting text-to-image models to new visual concepts with few examples. Previous fine-tuning suffers performance degradation on older concepts as new ones are added. A continually self-regularized adaptation approach called C-LoRA is proposed for stable DMs to enable continual adaptation without storing past data. Using fixed random prompts further improves generalization. Experiments on continual classification and the proposed continual diffusion setting demonstrate C-LoRA matches or outperforms rehearsal-based techniques without the storage overhead. By circumventing forgetting during compositional concept acquisition, C-LoRA advances practical applications for on-device personalization and incremental learning where storing growing user data is infeasible. Its effectiveness underscores the importance of continual adaptation techniques for the real-world deployment of customizable generative models.

Latent flow diffusion model (LFDM) [114] presents conditional image-to-video (I2V) synthesis. Unlike direct generation approaches, LFDM warps a given image over a predicted optical flow sequence in latent space to simultaneously model spatial appearance and temporal dynamics. It comprises an unsupervised latent flow autoencoder for content modeling and a DM that generates compact latent flows rather than high-dimensional videos. By factorizing spatial details from motion, LFDM efficiently models dynamic patterns while preserving the fidelity of provided imagery. Experiments across datasets demonstrate LFDM outperforming prior work in photorealism and motion coherence. Simple decoder fine-tuning further enables domain adaptation leveraging pretrained components. With both solid empirical performance and interpretability, LFDM provides an advance in controllable video generation by separating what changes from what persists.

DreamBooth [115] enables personalized image generation by fine-tuning text-to-image models to embed user-provided subjects using a few example images. A unique identifier bind provides individuals with novel contextual synthesis, preserving their traits. Built on pretrained semantic knowledge, an autogenous class-specific loss allows extrapolation to varied scenes and views beyond the reference set. Applications in recontextualization, text-guided view manipulation, and stylization showcase controllable generation while keeping identity intact. A new dataset and protocol rigorously benchmark such conditional synthesis quality. By synergizing external personal datasets with the rich priors of large generative models, the technique significantly advances the control, customizability, and specificity of text-driven image production. Enabling lay users to adapt such models to familiar domains with minimal data also underscores a move toward accessible and trustworthy AI utilization.

Meng et al. [116] distill classifier-free guided DMs to accelerate inference while retaining sample quality. Despite effectiveness for high-fidelity generation, evaluating conditional and unconditional models is computationally expensive. A distillation pipeline first matches the combined output and then progressively transfers sampling to more efficient diffusion. For pixel models, comparable scores to the original are achieved using as few as four steps on CIFAR-10 and ImageNet 64 × 64, accelerating sampling up to 256×. For latent DMs like SD, 10× speedups are attained with 1–4 steps and no visual decline. The approach also enables efficient guided editing and inpainting, generating quality results in just 2–4 denoising iterations. By drastically reducing sampling costs, the distillation framework unlocks the deployment of large conditional DMs under practical time constraints, advancing their use for interactive applications.

Bansal et al. [117] introduce a universal guidance technique enabling conditional image generation from DMs using arbitrary modalities without retraining. Most conditioned DMs only accept specific forms of input like text. The proposed algorithm leverages gradient signals from any external guidance source to steer image sampling, segmentation maps, face recognition, object detectors, or classifiers. Demonstrations with diverse side information validate the capability to specialize generic generative priors for new domains purely through gradient-based conditioning—no network changes needed. By flexibly binding external recognition networks, the approach significantly advances DM adaptability and specificity without costly dataset-driven fine-tuning.

Dual condition face generator (DCFace) [118] tackles synthetic face dataset generation for recognition via DMs conditioned on subject identities and stylistic factors. Controlling interclass similarity and intraclass variability poses challenges for GANs and 3D approaches. DCFace disentangles identity vectors and patch-based style codes to enable consistent identity synthesis under customizable pose, lighting, and other transformations. An identity loss through sampling stabilizes recognition cues. The interpretable and reliable factor manipulation provides a significant advance in configurable synthetic faces spanning intrapersonal imaging dynamics at scale.

Papa et al. [119] provide critical analysis on generating and detecting realistic fake faces using DMs to inform ethical deployment. By identifying text prompts for photorealistic human synthesis with SD, malicious usage risks are exposed should access go unchecked. The study sounds caution about the advancing creative abilities of generative models and the urgency for tamper detection advances to catch up. By demonstrating the ease of synthesis paired with solutions for exposing fakes, the exploration attempts to steer DM progress responsibly toward transparency and monitoring against potential harms.

Imagen Video [120] is a cascaded text-to-video (T2V) DM generating high-quality HD footage. A base generator coupled with interleaved spatial and temporal super-resolution networks enables controllable video synthesis from language descriptions. Techniques transferred from image diffusions combined with progressive distillation produce high-fidelity and efficient sampling. Experiments demonstrate diverse video generation spanning artistic styles, 3D environments, and complex multistage actions described in input prompts. The model exhibits strong world knowledge and understanding of spatiotemporal dynamics in rendered scenes. By advancing multiscale adversarial learning to sequential data, Imagen Video significantly pushes the state-of-the-art in programmatic video synthesis through intuitive text interfaces.

Diffused Heads [121] tackles talking face video generation from a single image and audio using DMs. A proposed autoregressive diffusion approach synthesizes natural head movements and facial expressions like blinks from only a static face and speech clip. Backgrounds are also preserved within the generated talking head video. Evaluation of two datasets shows the method achieves great realism of motion and expressions. By advancing one-shot controlled generation, the technique reduces data needs for personalized video synthesis down to a selfie-style photo. The solid empirical results validate DMs as a promising paradigm for highly conditional sequential data generation grounded in limited supervision.

Harvey et al. [122] tackle long-duration video modeling using diffusion probabilistic models. The proposed framework can conditionally complete arbitrary frame subsets to enable efficient iterative optimization for coherent event sequencing over 25+ minutes. New datasets and metrics based on autonomous driving scenarios provide semantic grounding. By scaling iterative reversible generation with flexible sparse conditioning, the approach surpasses prior video diffusion techniques on closed-loop coherence over extreme time horizons. The capability to extrapolate realistic event chains from sparse keyframes underscores DMs as uniquely suited for long-range visual prediction. The advance demonstrates video understanding through physics-inspired temporal diffusion rather than data-driven discriminative cues alone.

ControlNet [123] proposed to add spatial conditioning controls to large, pretrained text-to-image DMs. ControlNet enables DM generation under various conditions, such as sketches, depth maps, and human poses. ControlNet greatly facilitates the real-world applications of DM-based image generation. Uni-ControlNet [124] enables the simultaneous use of multiple control modes beyond text, incorporating local controls like edge maps, depth maps, and segmentation masks, as well as global controls such as CLIP image embeddings. Uni-ControlNet utilizes only two additional adapters on top of pretrained text-to-image models, eliminating the need for training from scratch and maintaining constant adapters regardless of the number of controls used. Compared with ControlNet, Uni-ControlNet significantly reduces fine-tuning costs and model size, making it more practical for real-world deployment.

VideoCrafter1 [126] presents high-quality video generation via T2V and I2V generation. The T2V model can generate high-quality video, and the I2V model produces videos that strictly adhere to the condition image. VideoCrafter2 [127] focuses on creating high-quality video models without access to premium video datasets. Instead, Videocrafter2 combines low-quality videos with synthesized high-quality images to train an effective model. Based on the analysis of the interaction between spatial and temporal modules in video models extended from SD, the authors noticed that training all modules together creates a stronger coupling between spatial and temporal components. Leveraging this, they fine-tuned spatial modules with high-quality images, improving overall video quality without degrading motion.

There are also some DMs adopted in 3D objects’ generation:

Latent video diffusion model (LVDM) [110] is a lightweight video DM with a low-dimensional 3D latent space. The long video generation ability of LVDM is based on the proposed hierarchical diffusion structure. Besides, LVDM proposed conditional latent perturbation and unconditional guidance to mitigate performance degradation in long video generation. Extensive experiments demonstrate LVDM’s superiority in generating realistic and longer videos. Additionally, LVDM is also extended to large-scale T2V generation tasks.

GaussianDreamer [125] is a framework generating 3D assets from text prompts. In detail, GaussianDreamer leverages 3D Gaussian splatting representation to bridge the gap between these two types of models. It uses 3D DMs to provide initial priors and then enhances geometry and appearance using 2D DMs. The process incorporates noisy point growing and color perturbation techniques to improve the initialized Gaussians.

3.3.3. Deepfake Manipulated With DMs

Stochastic differential editing (SDEdit) [128] enables easy image synthesis and editing by users through stochastic diffusion processing. Balancing realism and faithfulness to inputs like strokes poses challenges for current GAN approaches. SDEdit leverages DM priors without task-specific training or inversions. Given an input image and edits, SDEdit adds and then denoises noise through the SDE to increase realism while preserving user edits. Human studies show SDEdit outperforms GAN baselines substantially in realism and overall satisfaction on stroke-based synthesis/editing and compositing. By building on diffusion generative modeling, SDEdit advances widely applicable guided synthesis without model retraining or input inversion. The demonstrations also attest to the suitability of iterative denoising for user creativity, interacting with an editor rather than a one-shot synthesis.

DiffFace [129] is the first DM for high-fidelity facial identity swapping. DiffFace adopts an identity-conditional model to handle desired face generation. Flexible guidance during sampling then composites target attributes while translating identity via facial recognition networks. A target-preserving blend further retains background details. Benefits of DiffFace over GANs include training stability, quality, and control. User studies validate that DiffFace matches or exceeds state-of-the-art face-swapping techniques in realism and identity preservation. By harnessing iterative denoising for facial generation and composition, DiffFace sets a new bar for Deepfakes that are perceptually indistinguishable from reality while preventing unwanted use through responsible disclosure and monitoring.

RePaint [130] tackles free-form image inpainting using denoising DMs to enable extreme mask generalization. RePaint leverages an unmodified pretrained unconditional DDPM as a generative prior. Conditioning is achieved by only sampling unmasked pixels in reverse diffusion, allowing high-quality and diverse completions for any inpainting mask. Evaluations on faces and generic images with both standard and extreme hole shapes show RePaint outperforms autoregressive and GAN methods on nearly all mask distributions. By simply harnessing pretrained generative reversibility, RePaint pushes DMs toward unconstrained inpainting without assumptions about missing regions. The extreme generalization demonstrates the suitability of iterative denoising for open-ended conditional image synthesis.

DiffEdit [131] leverages text-conditional DMs for semantic image editing without masks. Previous techniques treat editing as conditional inpainting, needing user input to specify regions to change. Instead, DiffEdit automatically identifies areas to edit by contrasting model predictions under different text prompts. Latent inference preserves unchanged content. Without masks, DiffEdit achieves state-of-the-art generative editing results on ImageNet and robustness on more challenging datasets like COCO and text-to-image samples. The capability to reconcile language-driven enhancements with input visuals moves beyond editing as inpainting toward true spatial reasoning over linguistic concepts and visual regions. By contrasting sampler behavior rather than solely using discriminators, DMs prove uniquely positioned for programmatic yet minimally invasive image manipulation.

UniTune [132] enables text-driven image editing by fine-tuning generative models on individual images. While text-to-image generation has seen great progress, editing capabilities lag behind, typically needing additional signals like masks to specify regions of change. Instead, the proposed UniTune method adapts any baseline generator using just example imagery and text descriptions. Initializing sampling from a base image and interpolating details further improves edit quality. Without masks or other guidance, UniTune performs a wide range of edits while maintaining fidelity, even for significant visual changes previously infeasible. Experiments across use cases demonstrate versatile text-controlled image manipulation via simple domain transfer learning rather than architecting separate editors. By harnessing DM reversibility and adaptability, the technique delivers on the promise of intuitive human–AI creativity through natural language alone.

Hertz et al. [133] tackle intuitive text-driven image editing without masks by analyzing text-image attention correlations in generative models. While text-to-image synthesis has seen great progress, editing capabilities lag behind, even small text changes often lead to completely different results. Existing techniques rely on masks or sketches to spatially constrain edits. Instead, cross-attention layers are identified as critical for binding words to spatial layouts. By monitoring and modifying attention distributions via textual edits alone, targeted yet minimally invasive manipulations are enabled. Demonstrations include localized revisions by replacing words, global style adjustments through additions, and granular attenuation of cue magnitudes without specifying regions. Evaluations across diverse images and prompts validate prompt-to-prompt editing quality and fidelity. By exposing and harnessing an interpretable interface for spatial-semantic alignment, the technique delivers truly programmatic image manipulation directly through natural language.

DiffusionCLIP [134] is a technique for text-guided image manipulation using DMs to address limited GAN inversion capability and artifacts. While GAN inversion with CLIP conditioning enables zero-shot editing through language, reconstructing images with novel compositions or attributes often fails. Instead, leveraging DMs’ full inversion potential and high-fidelity synthesis empowers robust real image editing between unseen domains, which is validated on diverse ImageNet samples. A novel noise combination approach further enables straightforward multi-attribute editing. Human evaluations confirm the superior manipulation fidelity over GAN-based methods, better generalizing to complex real-world distributions. By building on iterative denoising rather than adversarial discrimination, DiffusionCLIP makes significant progress toward broadly applicable programmatic image editing through intuitive textual interfaces.

Blended diffusion [135] enables intuitive local text-driven image editing using CLIP guidance and diffusion blending. While global language-conditioned image generation has seen great progress, region-level editing capabilities lag behind. The proposed approach takes an ROI mask and textual description to steer a denoising DM toward desired edits spatially. Fusing the rewritten region with unaltered areas is achieved by progressively blending latent codes and input image copies under increasing noise. Qualitative and quantitative evaluations validate background preservation and text alignment exceeding related techniques and baselines. Demonstrated applications include object addition/removal/replacement, background substitution, and content extrapolation—all through descriptive language edits. By synergizing language-vision guidance with differentiable generative reconstruction, the technique significantly advances the state-of-the-art in programmatic visual editing toward flexible region-based modifications from natural textual requests.

Palette [135] establishes conditional DMs as a unified framework for advancing image-to-image translation across challenging restoration and synthesis tasks. A simple diffusion implementation outperforms specialized GAN and regression techniques on colorization, inpainting, upsampling, and JPEG artifact removal without structure customization. Studies reveal the impact of loss functions and self-attention on sample quality. A standardized benchmark and protocol leveraging ImageNet, human judgment and automated metrics provide rigorous assessment showing strong generalization of a single model trained over multiple domains. By demonstrating both superior task performance and flexibility from a general-purpose diffusion translator, this exploration signals denoising-based generation as a prime candidate for further developing this practical graphics application space.

Imagic [136] achieves complex single-image manipulation with only text queries. Prior text-based editing techniques remain limited to specific domains like overlays or style transfer and often need multiple inputs. Instead, the proposed Imagic approach performs nonrigid semantic edits like adjusting object pose and scene composition within one natural photo using pretrained generative guidance. For instance, making a dog sit or a bird spread its wings in provided shots. Unlike previous work, this text-controlled editing framework requires no additional masks, segmentation, or alternate views. A new benchmark assesses editing quality, with human studies preferring Imagic over prior techniques. Demonstrations across domains exhibit significant advances in open-ended spatial rewriting grounded in language alone. By harnessing inversion and adaptability unique to DMs, the technique makes strides toward flexible image reconfiguration through intuitive textual description.

DragDiffusion [137] extends interactive image editing techniques based on generative guidance to DMs for improved real-world applicability. While most GAN-based editors constrain capacity, editing via latent code optimization unlocks precise spatial control leveraging versatile diffusion priors. Although DMs generate images iteratively, coherent results are achieved by only optimizing a single step’s latent, which enables efficient high-quality manipulation. Experiments across challenging cases with diverse objects, styles, and configurations validate generalizability exceeding GAN counterparts. By porting emerging neural editing paradigms to powerful denoising generative models, DragDiffusion makes precise semantic image manipulation more tractable for practical workflows. The technique underscores the suitability of decoder-based reversible synthesis for performer-centric creativity, interacting directly in output space rather than constrained latent configurations.

4.1.1. GAN-Based and DM-Based Deepfake Generation

4.1.2. Detection of GAN-Generated and DM-Generated Deepfake

We notice that most Deepfake detectors are designed and trained to recognize GAN-generated Deepfake. However, compared with GAN-generated Deepfake, DM-generated Deepfake contains more realistic textures and fewer visual flaws, making DM-generated Deepfake challenging for GAN-target detectors, especially those designed to capture visual artifacts such as abnormal biometrics, unnatural textures, and blending edges.

In fact, Deepfake detectors encounter generalization challenges even when trained and tested on different datasets generated by the same technique (GANs or DMs). The generalization challenge is essential because fake content generated by different models and datasets naturally has different data distributions. Detectors are trained to capture the potential data distributions of the training dataset and become less effective on unseen testing datasets. While a lot of works have proposed methods such as enlarging training datasets, capturing generalizable semantic representations, or adopting external knowledge (multimodal synchronization) to improve the generalization ability, few are practically usable to detect diverse real-world Deepfake, not to mention the generalization to unseen new-generation techniques. From the perspective of attack and defense, the generalization challenge is tough because it is like in an attack-defend competition, an attacker wins by just successfully cracking one vulnerability, while a defender wins only when they can defend all attacks. When new and more powerful generation models and techniques keep emerging and developing, it is difficult for passive Deepfake detectors (defenders) to cope.

4.2.1. Deepfake Generation

4.2.2. Deepfake Detection

By addressing these areas, the field can move toward more proactive, robust, and adaptable methods for detecting Deepfake, safeguarding individuals and societies against the potential misuse of this powerful technology. The evolution of Deepfake detection strategies will likely be an ongoing process, requiring continuous innovation and collaboration among researchers, technologists, and policymakers.