2507.05146

# VERITAS: Verification and Explanation of Realness in Images for Transparency in AI Systems **Authors**: Aadi Srivastava, Vignesh Natarajkumar, Utkarsh Bheemanaboyna, Devisree Akashapu, Nagraj Gaonkar, Archit Joshi > 1 Department of Computer Science & Engineering, Indian Institute of Technology, Madras 2 Department of Electrical Engineering, Indian Institute of Technology, Madras 3 Department of Mechanical Engineering, Indian Institute of Technology, Madras 4 Department of Civil Engineering, Indian Institute of Technology, Madras ## Abstract The widespread and rapid adoption of AI-generated content, created by models such as Generative Adversarial Networks (GANs) and Diffusion Models, has revolutionized the digital media landscape by allowing efficient and creative content generation. However, these models also blur the difference between real images and AI-generated synthetic, or “fake”, images, raising concerns regarding content authenticity and integrity. While many existing solutions to detect fake images focus solely on classification and higher-resolution images, they often lack transparency in their decision-making, making it difficult for users to understand why an image is classified as fake. In this paper, we present VERITAS, a comprehensive framework that not only accurately detects whether a small (32x32) image is AI-generated but also explains why it was classified that way through artifact localization and semantic reasoning. VERITAS produces human-readable explanations that describe key artifacts in synthetic images. We show that this architecture offers clear explanations of the basis of zero-shot synthetic image detection tasks. Code and relevant prompts can be found at https://github.com/V-i-g-n-e-s-h-N/VERITAS footnotetext: $\dagger$ These authors contributed equally to this work. Correspondence to: Aadi Srivastava (aadisrivastava.iitm@gmail.com), Vignesh Natarajkumar (vigneshn@smail.iitm.ac.in) Keywords Computer Vision $\cdot$ Vision-Language Models $\cdot$ Multimodality $\cdot$ Explainable AI ## 1 Introduction The rapid advancement of generative models such as GANs and Diffusion models has democratized access to image generation. Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), Variational AutoEncoders (VAEs) (Kingma and Welling, 2022), Diffusion models (Ho et al., 2020) and other synthetic image generation techniques have created strikingly realistic images and videos. Another important consideration is the growing multimodality of generative models, indicating the ability of models to synthesize audio waveforms and videos along with images (Liu et al., 2024b), (Zhang et al., 2024a), (Li et al., 2023), (Maaz et al., 2024), (Lu et al., 2023). As the performance of these generative models progresses, the gap between human-created ‘real’ images and synthetic images shrinks. Naturally, the ability of these models to generate novel samples from training data finds uses in several media-intensive fields. While these innovations empower new possibilities, they also introduce significant challenges. There are risks associated with AI (Hendrycks et al., 2023), and AI-generated media is no exception. In fields where transparency and accuracy are very important, the prevalence of AI-generated media creates apprehensions regarding the spread of misinformation and malicious usage of generated media. This raises the concern: what is real and what is not?. While several classification systems have emerged to detect synthetic images, they often operate as black boxes, offering limited explainability and seeking to merely improve the accuracy of inference on synthetic data. They also do not always highlight the artifacts used in the classification process. However, to improve trust in and adoption of such frameworks, and to improve fundamental research and understanding in the field of synthetic media detection, such frameworks should be explainable (Bird and Lotfi, 2023), (Kang et al., 2025), (Wu et al., 2025), (Pinhasov et al., 2024), (Abir et al., 2022), (Malolan et al., 2020). This will also assist in responding to alterations and adversarial attacks that can be performed on digital media. Another consideration is the size of images typically used. Detecting whether small (32x32) images are real or AI-generated is a crucial use case due to the growing prevalence of synthetic media and the unique challenges posed by low-resolution content. Small images are often used as profile pictures, thumbnails, icons, or in messaging apps, making them a common vector for misinformation, identity fraud, and malicious activity. Their limited pixel information makes it harder for both humans and automated systems to distinguish genuine photos from AI-generated fakes, increasing the risk of undetected manipulation. Effective detection in this context helps maintain trust in digital platforms, safeguards user identities, and supports the integrity of online communications. ### 1.1 Our Contributions We propose VERITAS, a multi-stage framework that classifies small images as either real or synthetic and uses techniques to offer human-readable explanations of the anomalies and artifacts found in synthetic images. Our key contributions are: - Identifying and reviewing several classification frameworks, algorithms, and architectures to accurately detect synthetic images. - Introducing a novel framework that generates artifact-based explanations of the synthetic nature of small images, providing users with clear, concrete evidence for why an image is identified as artificial. - Observing the impact of variation in Vision-Language Models (VLMs) and prompting approaches on the produced explanations. The rest of this paper is structured as follows: In Section 2 and 3, we review related works in deepfake classification, artifact detection, and explainable AI. In Section 4, we present our methodology, detailing the VERITAS pipeline. Section 5 describes our experimental setup, including the dataset utilized for evaluations, a description of the reference artifacts, and a discussion of the experimental results. Section 6 concludes the paper with an emphasis on the future directions of work identified during the study. Section 7 conveys our acknowledgments to those who supported this study. <details> <summary>extracted/6602897/image.png Details</summary>  ### Visual Description \n ## Photograph: Man Performing a Handstand ### Overview The image is a photograph of a man performing a handstand in an outdoor urban setting. The background features a cityscape with tall buildings and greenery. The image includes a logo in the bottom-left corner. The photograph appears to be promotional material. ### Components/Axes There are no axes or charts present in this image. The key components are: * **Subject:** A man in a handstand. * **Background:** A cityscape with buildings and trees. * **Logo:** Adobe Firefly logo. ### Detailed Analysis or Content Details The man is wearing dark pants, a striped shirt, a beanie, and sneakers. He is smiling and appears to be in good physical condition. The handstand is performed on a concrete surface. The background buildings are blurred, suggesting a shallow depth of field. The lighting is natural and appears to be from an overcast day. The logo in the bottom-left corner consists of a red "A" shape with a flame-like design, accompanied by the text "Adobe Firefly". ### Key Observations The image is visually striking due to the dynamic pose of the man. The composition places the subject centrally, drawing the viewer's attention. The blurred background helps to isolate the subject and emphasize the action. The Adobe Firefly logo suggests the image is related to Adobe's generative AI platform. ### Interpretation The image likely aims to convey a sense of creativity, energy, and innovation. The handstand symbolizes pushing boundaries and thinking outside the box, which aligns with the capabilities of Adobe Firefly as a creative tool. The urban setting suggests a modern and dynamic environment. The overall message is likely to promote Adobe Firefly as a platform that empowers users to express their creativity in new and exciting ways. The image does not contain any factual data or numerical values; it is purely a visual representation intended to evoke a specific feeling or association. </details> Figure 1: Sample image generated by Adobe Firefly, analyzed by the proposed pipeline. Artifact Description provided by model: The man’s legs are bent at unnatural angles—his left leg bends backward while the right leg bends forward. The limbs appear disconnected from the torso, resulting in a surreal and anatomically implausible configuration. ## 2 Related Work While the first stage of the proposed pipeline involves robust and accurate classification of real and synthetic images, the primary focus of our innovation is on the explainable analysis of images classified as synthetic. Given that humans cannot always visually discriminate between real and synthetic images, even in the case of images with adversarial attacks, the pipeline is made explainable and understandable by humans. This is in alignment with the goal of Explainable AI (XAI) (Gunning et al., 2019): to provide explanations of why a decision was made by a model. The work of (Yu et al., 2019) highlights that GANs leave “fingerprints” in synthetic images, and minor differences in the generative model can lead to differences in the fingerprints present in an image. Similarly, (Geirhos et al., 2019) identify that differences in local texture details in specific patches of the image can lead to different CNN-based classification outcomes. This highlights the importance of local spatial features and artifacts in determining if an image is real or synthetic. These artifacts may be observed via spatial characteristics or frequency-domain features (Li et al., 2018) (Haliassos et al., 2021), (Qian et al., 2020), (Lai et al., 2024), (Jeong et al., 2021). It is observed that some local image patches with artifacts contribute more to the classification of the image than others. Hence, patches are selectively chosen and used to generate explanations of artifacts used in the classification of an image as synthetic. To understand which artifacts fake image detectors look for during classification, (Chai et al., 2020) utilize a fully-convolutional patch-based classifier to focus on local patches of images, and they discovered that using such small receptive fields, such as sliding patches, helps classifiers ignore global differences and focus on local artifacts that are common to several synthetic images. This improves the performance of the pipeline on zero-shot classification with out-of-domain images. To counter the inability of synthetic image detectors to perform well across different generative architectures such as GANs and Diffusion models, (Chen et al., 2024) propose a Single Simple Patch (SSP) network that extracts noise fingerprint information from a single patch of an image. This approach is supported by (Zhong et al., 2024), who observe that texture patches of images reveal more traces of generative models than the global nature of the image. Consequently, global information is erased and texture patches are enhanced for synthetic image detection. These observations motivate our patch-based approach. The detection of artifacts becomes significantly more challenging in the case of the CIFAKE dataset (Bird and Lotfi, 2023) (which has been described in greater detail in later sections) owing to the reduced dimensions of the input image and feature map. Further, according to the Information Bottleneck principle (Tishby and Zaslavsky, 2015), there is a possibility of information loss as the feature map is passed through several layers of a network. To counter this, several super-resolution approaches have been explored (Anwar and Barnes, 2019), (Dong et al., 2015). Several methods proposed earlier can be divided into either interpolation-based approaches (Zhang et al., 2018), (Tong and Leung, 2007), (Sun and Li, 2019) or reconstruction-based approaches (Yu et al., 2017), (Chang et al., 2018). Previous studies have also used Transformers, such as IPT (Chen et al., 2021), SwinIR (Liang et al., 2021), and the UFormer (Wang et al., 2021). (Hsu et al., 2024) propose the Dense-residual-connected Transformer (DRCT) architecture, which utilizes the ability of Swin Transformers (Liu et al., 2021) to capture long-range dependencies and reconstruct spatial information for preventing such information bottlenecks. Such elements are also included in our pipeline to compensate for information restrictions and downstream losses in layers. Vision-Language Models (VLMs) align textual and visual representations in a shared embedding space, allowing multimodal data processing. Previous studies have established VLMs as powerful tools for multimodal understanding (Bommasani et al., 2022), (Liu et al., 2023), (Liu et al., 2024a), (Chen et al., 2023), (Zhu et al., ), (Qin et al., 2023), (Chen et al., 2022), (Li et al., 2025). Given the performance of VLMs in complex vision-language tasks such as object localization, fine-grained object description, image-based text generation and detection of abnormalities and defects (Qin et al., 2023) (Chen et al., 2023) (Zhu et al., ), (Chen et al., 2022), integration of these models in frameworks to detect and describe synthetic images is a promising avenue. CLIP (Radford et al., 2021) is an efficient method of learning image-text relationships and employs an image-text contrastive objective that clusters paired images and texts closer and pushes others away in the embedding space. This allows pre-trained VLMs to capture rich vision-language relationships and is used in our proposed framework for relating patch information with artifacts. CLIP also supports zero-shot predictions based on the developed embedding space, performing very well on the description of unseen images. MOLMO (Deitke et al., 2024) is a state-of-the-art open-source VLM that outperforms other models such as Qwen (Bai et al., 2025) and Pixtral (Agrawal et al., 2024) on tasks such as fine-grained understanding, pointing and counting of features, localization of artifacts and visual skills. This makes it ideally suited for an artifact-based classification objective. Naturally, VLMs have been used in previous works for synthetic image detection. For instance, (Keita et al., 2024a) tune VLMs like BLIP-2 to detect synthetic images generated by unseen models using an innovative method called Bi-LORA that combines VLMs with low-rank adaptation tuning. Similarly, (Keita et al., 2024b) uses the complementary relationship between vision and language to detect and assign synthetic images to their source models, prompting their framework to refine outputs. (Khan and Dang-Nguyen, 2024) uses Prompt Tuning to adapt CLIP for deepfake detection while retaining both the visual and textual information during inference, improving the accuracy of the classification while also reducing the required training data. Gradient-based methods, such as GradCAM (Selvaraju et al., 2020), offer a powerful approach to understanding and explaining the predictions and classifications of CNN-based models, aligning with the principles of Explainable AI (XAI) and interpretability. GradCAM utilizes the gradient of a target concept, such as a classification output, and interprets it along with the feature maps generated by the CNN during training. The result is spatially aligned with the input image, and a visual heatmap that highlights critical regions influencing a model’s predictions is generated. This visual dimension can also be used in the classification process to improve accuracy. Notably, using feature maps without gradients to generalize artifact representations across generative model architectures has been previously explored in (Ojha et al., 2024). (Tan et al., 2023) uses a pre-trained CNN to convert GAN-generated and real images into GradCAM gradients, which are used as generalized representations of artifacts. These representations can then be leveraged to classify images as real or synthetic, enabling cross-model and cross-dataset robust performance. (Bird and Lotfi, 2023) also sought to understand the basis for synthetic image detection using CNNs and discovered that while most regions of a real image are relevant for classification, only a few regions and artifacts of a synthetic image determine its class. This encourages the use of GradCAM to highlight significant patches in images for classification. ## 3 Review of Classification Frameworks Synthetic image classification methods aim to distinguish between real and AI-generated content using binary or multiclass classifiers trained on labeled datasets. Recent advancements in synthetic image detection have focused on leveraging diverse neural architectures and algorithms to distinguish AI-generated images from real ones. This section presents an extensive review of relevant previous research. Classification in the presence of Adversarial perturbations is also considered in the Appendix. ### 3.1 Fundamental Architectures Several deepfake classification models primarily rely on Convolutional Neural Networks (CNNs) like ResNet and EfficientNet. For instance, (Bird and Lotfi, 2023) train a CNN on the CIFAKE dataset and achieve commendable accuracy by detecting subtle background artifacts rather than semantic object features. (Akagic et al., 2024) investigate model behavior on real and synthetic image inputs and demonstrate that diffusion-generated samples could substitute real images for training classifiers such as CNNs. The evolution of deepfake detection has seen significant contributions from Vision Transformers (ViTs), which address limitations in CNNs regarding global context modeling. Convolutional Neural Networks (CNNs), such as ResNet18 and EfficientNet, demonstrate strong performance in capturing local, low-level features like textures and edge patterns that are often indicative of low-level synthetic artifacts. On the other hand, transformer-based architectures like ViT-Tiny prove effective in identifying global context and long-range dependencies. Recent surveys, such as (Wang et al., 2024), highlight that ViT-based models generalize better across deepfake datasets and exhibit robustness under compression and domain shifts. Standalone ViT architectures such as FakeFormer (Fak, ) leverage self-attention to capture long-range dependencies in high-resolution images, though they initially underperformed CNNs due to insufficient local artifact sensitivity. Hybrid models combine ViTs with CNNs or graph networks. For instance, the Self-Supervised Graph Transformer (Khormali and Yuan, 2023) integrates a ViT feature extractor with graph convolution layers to improve performance on the FaceForensics++ dataset (Rössler et al., 2019). (Romeo et al., 2024) introduce a Binary Neural Network (BNN) to classify deepfakes and also proposed augmenting the RGB input images with 2 additional filter outputs: the Fast Fourier Transform (FFT) and Local Binary Pattern (LBP). Fast Fourier Transform is employed to include information on frequency-domain patterns created by generative models and frequency-based adversarial attacks as well, which are not discernible by observing RGB images. Local Binary Pattern (LBP) complements this by focusing on spatial anomalies. It encodes the intensity differences between a pixel and its neighboring pixels into a binary pattern, capturing local texture irregularities such as texture and artifacts. Together, these filters help identify global and local features of real and synthetic images. A potential bottleneck of this approach in the setting of this paper is the lack of spatial and frequency-domain information that can be reliably extracted from 32x32 images obtained from CIFAKE, leading to irregular results. (Emami and Martínez-Muñoz, 2023) propose a Sequential Gradient-based Adversarial Training approach to enhance model robustness against progressively sophisticated adversarial attacks. The training begins with a base ResNet50 model (He et al., 2015), initialized with transfer learning weights, and proceeds iteratively. At each stage, adversarial examples are generated using the preceding model, and a new model is trained to resist these attacks. This sequential setup ensures that each subsequent model inherits the weights of its predecessor, with fine-tuning focused primarily on the final layers. Over multiple iterations, the ensemble learns to recognize and counter increasingly complex adversarial attacks. Following the framework of (Hinton et al., 2015), knowledge distillation can be applied by transferring knowledge from a teacher model to a student model to improve model generalization. In our setup, the teacher is a pretrained ResNet18 model, while the student is a similar but lightweight network. The training objective combines the standard cross-entropy loss with a distillation loss that aligns the student’s output distribution with that of the teacher, to help the student learn both the classification outcomes and the feature representations of the teacher. ### 3.2 Ensembling Approaches As described earlier, CNNs and Transformers exhibit proficiency in learning local feature maps and modeling long-range relationships, respectively. Previous studies on harnessing the strengths of multiple models in complex classification tasks such as deepfake detection frequently utilize ensembling of models, such as CNNs and Transformer models (Zhang et al., 2024b), (Naskar et al., 2024), (Coccomini et al., 2022), (Hooda et al., 2022). Ensembling is a widely adopted strategy in deep learning that leverages the strengths of multiple models to improve classification performance and generalization. Additionally, ensembling provides robustness by reducing the impact of individual model biases and failure cases. These ensembles are often tuned by optimizers to obtain the most optimal relative weighting factors with which outputs of different models in the ensemble are to be combined. Recognizing the complementary strengths of CNN and Transformer architectures, a potential solution for robust classification would be an Optuna-tuned ensemble of pretrained ResNet18, EfficientNet, and ViT-Tiny, which optimizes their contribution weights based on extensive hyperparameter trials, in order to obtain the maximum classification accuracy. The algorithm for such a model is given below Algorithm 1 Ensemble Model for Image Classification Input: Input image $I$ of size $32\times 32$ , training dataset $D$ , number of trials $T$ Output: Trained ensemble model and configuration JSON 1: Initialize models: $M_{\text{ViT}}$ , $M_{\text{EfficientNet}}$ , $M_{\text{ResNet18}}$ 2: for all image $i$ in $D$ do 3: Extract ViT features: $f_{\text{ViT}}\leftarrow M_{\text{ViT}}(i)$ 4: Compute loss: $L_{\text{ViT}}\leftarrow\text{ContrastiveLoss}(f_{\text{ViT}},\text{labels})$ 5: Upsample image: $i_{\text{up}}\leftarrow\text{Upsample}(i,224\times 224)$ 6: Extract EfficientNet features: $f_{\text{Eff}}\leftarrow M_{\text{EfficientNet}}(i_{\text{up}})$ 7: Compute loss: $L_{\text{Eff}}\leftarrow\text{BCELoss}(f_{\text{Eff}},\text{labels})$ 8: Extract ResNet features: $f_{\text{Res}}\leftarrow M_{\text{ResNet18}}(i_{\text{up}})$ 9: Compute loss: $L_{\text{Res}}\leftarrow\text{BCELoss}(f_{\text{Res}},\text{labels})$ 10: Update models using $L_{\text{ViT}},L_{\text{Eff}},L_{\text{Res}}$ 11: end for 12: Initialize Optuna study 13: for trial = 1 to $T$ do 14: Suggest weights: $w_{1},w_{2},w_{3}\leftarrow\text{trial.suggest\_float}(0,1)$ 15: Normalize weights: $w_{i}\leftarrow\frac{w_{i}}{\sum w_{i}}$ 16: Compute ensemble output: $f_{\text{ensemble}}\leftarrow w_{1}f_{\text{ViT}}+w_{2}f_{\text{Eff}}+w_{3}f_{ \text{Res}}$ 17: Evaluate ensemble performance 18: Update best weights based on validation score 19: end for 20: Generate output dictionary with model weights, performance metrics, and model paths ### 3.3 Frequency Domain Analysis Apart from the usage of Fast Fourier Transforms (FFT) as described in the earlier subsection, a frequency domain analysis of synthetic images provides a deeper understanding of the signatures left by generative models in the images. Recent advancements in detecting synthetic images have highlighted the importance of examining frequency domain characteristics to differentiate between real and artificial content. For instance, (Qian et al., 2020) propose F3-Net, which uses local frequency statistics for detection. These detection methods involve transforming images from the spatial domain to the frequency domain using techniques such as the Discrete Fourier Transform (DFT) and the Fast Fourier Transform (FFT). (Durall et al., 2020) and (Frank et al., 2020) identify abnormal spectral peaks and imbalances in GAN-generated images using frequency analysis as described above, which can be used to identify synthetic images from these models. However, the same frequency artifacts may not be found in images generated from diffusion models, since frequency fingerprints are unique to each generative model architecture. ### 3.4 Directional Enhanced Feature Learning Network (Li et al., 2024) introduce the Direction Enhanced Feature Learning (DEFL) Network, a model using Multi-Directional High-Pass Filters (MHFs) and randomly initialized filters to enhance the diversity of features extracted from input images. 8 well-defined high-pass “base” filters that extract features of the image in 8 different directions are initialized. To improve the diversity of the base filters and to sufficiently capture subtle features, the study further derives a set of 246 "composite” filters according to the 8 base filters. In particular, $n\leq 7$ base filters are selected to form a composite filter using the convolution relationship $$ h_{c}=h_{1}\times h_{2}\times\dots\times h_{n} $$ where $n$ is the number of base filters chosen. These together form the set of Multi-Directional High-Pass Filters, 2 of which are randomly repeated to form a total of 256 filters. There are also 256 randomly initialized filters that will be learned during the process of training. The filters are arranged in blocks of 64 MHFs and 64 randomly initialized filters, all of which are learnable and sequentially arranged. The DEFL network produces an output of 256 concatenated feature maps of height and width similar to that of the input image. which can be used for classification, and is rich in extracted features. ### 3.5 Latent Space Optimization Given the constraints of working with 32 × 32 images in the CIFAKE and custom datasets, the feature space for extraction is inherently limited. To tackle this, we propose an approach that converts the input images to a high-dimensional space using the embeddings learnt by ViT-Tiny during the training process, following which the boundary between real and synthetic images could be made clearer. Hence, this approach improves the separability of features between real and synthetic images. We utilize the embeddings from the penultimate layer of ViT-Tiny and train them using a combination of Contrastive loss and Triplet loss. The combined loss function is expressed as: $$ \mathcal{L}=\alpha\cdot\mathcal{L}_{\text{contrastive}}+\beta\cdot\mathcal{L}_ {\text{triplet}}, $$ where $\alpha$ and $\beta$ are weighting coefficients for the respective losses. The respective Loss functions are described using the equations below: $$ \mathcal{L}_{\text{contrastive}}=\frac{1}{N}\sum_{i=1}^{N}\left(y_{i}\cdot||f( x_{i})-f(x_{j})||_{2}^{2}\right.\left.+(1-y_{i})\cdot\max\left(0,m-||f(x_{i})- f(x_{j})||_{2}\right)^{2}\right) $$ where N is the total number of pairs, $y_{i}$ is the binary label being 1 for similar pairs, and 0 for dissimilar pairs. $f(x_{i})$ is the embedding, and m is the margin of dissimilar pairs. $$ \mathcal{L}_{\text{triplet}}=\frac{1}{N}\sum_{i=1}^{N}\max\left(0,||f(x_{i}^{a })-f(x_{i}^{p})||_{2}^{2}-||f(x_{i}^{a})-f(x_{i}^{n})||_{2}^{2})+m\right), $$ $x_{i}^{a}$ , $x_{i}^{p}$ , and $x_{i}^{n}$ are the anchor, positive, and negative examples respectively. Upon visualization using Dimensionality Reduction using t-SNE (van der Maaten and Hinton, 2008), while certain features exhibited improved discrimination, others remained challenging to distinguish between real and fake images. The visualization is given below. <details> <summary>extracted/6602897/representation_embeddings.png Details</summary>  ### Visual Description \n ## Scatter Plot: t-SNE Visualization of Learned Embeddings ### Overview The image presents a two-dimensional scatter plot generated using t-distributed Stochastic Neighbor Embedding (t-SNE). The plot visualizes learned embeddings, likely from a machine learning model, and color-codes the data points based on a "Real" label. The plot aims to show the separation or clustering of data points based on their embeddings. ### Components/Axes * **Title:** "t-SNE visualization of learned embeddings" (top-center) * **X-axis:** "t-SNE dimension 1" (bottom-center), ranging approximately from -75 to 75. * **Y-axis:** "t-SNE dimension 2" (left-center), ranging approximately from -40 to 60. * **Legend:** Located in the top-right corner. * Label: "Real" * Color: Blue * **Colorbar:** Located on the right side of the plot. * Scale: Ranges from 0.0 to 1.0. * Color Gradient: From light blue (0.0) to dark red (1.0). ### Detailed Analysis The scatter plot shows two distinct clusters of data points. * **Blue Cluster:** Located on the left side of the plot, centered around x = -25 and y = 0. The points are densely packed, forming a roughly elliptical shape. The color of these points is consistently blue, indicating a "Real" label. The color intensity is relatively uniform across this cluster, with values generally between 0.0 and 0.2. * **Red Cluster:** Located on the right side of the plot, centered around x = 50 and y = 20. This cluster also has a roughly elliptical shape, but it is more elongated vertically. The points are colored in shades of red, indicating a "Real" label. The color intensity is generally between 0.8 and 1.0. * **Connecting Bridge:** A narrow connection or "bridge" of points exists between the two clusters around x = 0 and y = 20. These points exhibit a gradient of colors, transitioning from blue to red, suggesting intermediate values for the "Real" label. There are approximately 500-700 points in the blue cluster and 500-700 points in the red cluster. The density of points appears higher in the central regions of each cluster. ### Key Observations * The two clusters are well-separated, suggesting that the t-SNE embedding has effectively captured underlying differences in the data. * The color gradient in the connecting bridge indicates that some data points may not clearly belong to either cluster. * The "Real" label appears to be associated with both clusters, but the color intensity suggests different degrees of "Realness" or confidence. ### Interpretation The t-SNE visualization suggests that the learned embeddings have successfully separated the data into two distinct groups based on the "Real" label. The separation indicates that the model has learned meaningful representations that capture the underlying structure of the data. The connecting bridge suggests that there may be some ambiguity or overlap between the two groups. The color intensity associated with the "Real" label could represent a confidence score or probability. The red cluster, with higher color intensity, may represent data points that are more confidently classified as "Real" than those in the blue cluster. The visualization is useful for understanding the relationships between data points and for identifying potential outliers or anomalies. It can also be used to evaluate the quality of the learned embeddings and to guide further model development. The fact that the data separates into two clusters suggests a binary classification problem or a dataset with two distinct modes. </details> Figure 2: Embeddings trained on contrastive and triplet loss To evaluate the utility of these modified embeddings, they can be passed directly to the ViT-Tiny model, bypassing its initial input layer. Further improvements in performance can be obtained with appropriate optimization of the weighting coefficients for the Contrastive and the Triplet Losses, an approach that has not been explored in the current study. Algorithm 2 Contrastive Loss Computation Input: Feature matrix $F$ , label vector $y$ , temperature $\tau$ Output: Contrastive loss $L_{\text{pos}}$ 1: Normalize features: $F\leftarrow\text{normalize}(F)$ 2: Compute similarity matrix: $S\leftarrow\frac{FF^{\top}}{\tau}$ 3: Create masks: $M^{+}\leftarrow(y=y^{\top})$ , $M^{-}\leftarrow(y\neq y^{\top})$ 4: Remove self-similarity: $M^{+}\leftarrow M^{+}-I$ 5: Compute exponentials: $P\leftarrow\exp(S)$ 6: Compute log probabilities: $\text{log\_prob}\leftarrow S-\log(\sum_{j}\exp(S_{ij}))$ 7: Compute positive loss: $L_{\text{pos}}\leftarrow-\frac{\sum(M^{+}\odot\text{log\_prob})}{\sum M^{+}}$ 8: return $L_{\text{pos}}$ ### 3.6 Dual-Domain Strategies (Yan et al., 2021) propose a dual-domain filtering approach that combines spatial-domain processing with transform-domain analysis to remove adversarial noise. The features of the architecture are: - Spatial Domain Filtering - It involves separating the input image into edge (high-frequency) and texture (low-frequency) features. Edge and Texture Decomposition is performed by Bilateral Filtering to decompose the spatial features into 2 components- Filtered (which are rich in high-frequency structural information) and Unfiltered texture features (which contain smoother variations and finer details). - Transform Domain Filtering - The unfiltered texture features are converted into the frequency domain using a Short-Time Fourier Transform (STFT). This facilitates localized frequency analysis, targeting perturbations in specific frequency bands. Wavelet shrinkage is applied in the frequency domain to selectively attenuate high-frequency components associated with adversarial perturbations. - Reconstruction - After the dual-domain filtering, the refined texture features are reconstructed and combined with the preserved edge features to produce a denoised and robust image. ### 3.7 AutoEncoder Architectures AutoEncoders have been extensively used in deepfake and adversarial image classification pipelines for two primary purposes: feature purification and anomaly detection, both in the context of adversarial perturbations. The central idea is that AutoEncoders, when trained only on real images, struggle to accurately reconstruct synthetic or adversarially perturbed images. This property can be exploited to flag anomalous reconstructions as potential fakes. One proposed direction involves training the AutoEncoder to reconstruct perturbation maps from adversarial samples and learning to isolate the added noise and distortions. Perturbation maps illustrate the alterations made to adversarial images in comparison to their original counterparts. This purification process allows any following classifiers to focus on underlying semantic features rather than being misled by adversarial artifacts. Previous studies support this approach and the above observations (Kalaria et al., 2022), (Hwang et al., 2019), (Beggel et al., 2019). (Guo et al., 2018) proposed Variational AutoEncoder-based purification methods that map adversarial examples to the distribution of clean images before reclassification. (Du et al., 2020) proposes a Locality-Aware AutoEncoder that learns intrinsic features of images instead of superficial characteristics of training data, enabling better performance on zero-shot classification of deepfakes. ## 4 Proposed Framework This section presents our core contribution: A five-stage pipeline designed for detecting, interpreting, and describing artifacts in low-resolution images. Our pipeline integrates super-resolution, attention-based heatmapping, patch-level analysis, vision-language modeling, and multimodal language generation to provide comprehensive artifact detection with interpretability. The five steps are: 1. Super-resolve input image using DRCT 1. Localize relevant regions with GradCAM heatmaps 1. Divide image into patches weighted by attention 1. Use CLIP-based similarity for artifact classification 1. Generate textual explanations via MOLMO <details> <summary>extracted/6602897/pipeline.png Details</summary>  ### Visual Description \n ## Diagram: Artifact Analysis Pipeline ### Overview This diagram illustrates a pipeline for analyzing artifacts, starting with an input image and culminating in a JSON output. The pipeline involves image processing, artifact segmentation, and textual description generation. The process appears to be designed for archaeological or historical artifact analysis, leveraging computer vision and natural language processing techniques. ### Components/Axes The diagram consists of several processing blocks connected by arrows indicating the flow of data. Key components include: * **Input image (32x32):** The initial input to the pipeline. * **DRCT:** A processing step following the input image. * **Super resolution images 128x128:** An image upscaling step. * **CLIP:** A component likely related to Contrastive Language-Image Pre-training. * **Artifacts split into biological and mechanical:** A segmentation step categorizing artifacts. * **Clip similarity module by using 3-tuple descriptions:** A module for comparing artifacts using CLIP embeddings. * **MOLMO for generating textual descriptions:** A module for generating textual descriptions of the artifacts. * **Output .JSON:** The final output of the pipeline. * **GradCAM:** A visualization technique for understanding which parts of the image contribute to the model's decision. * **Heatmap:** A visual representation of the GradCAM output. * **Variable Patching Module:** A module for dividing the image into variable-sized patches. * **Interpolated heatmap:** A processed heatmap. * **Weighted patches:** Patches assigned weights based on the heatmap. ### Detailed Analysis or Content Details The pipeline flows as follows: 1. An **Input image (32x32)** is fed into the **DRCT** module. 2. The output of **DRCT** is passed to a **Super resolution images 128x128** module, increasing the image resolution. 3. The upscaled image is then processed by **CLIP**. 4. **CLIP**'s output is used to **Artifacts split into biological and mechanical** categories. 5. The categorized artifacts are then processed by a **Clip similarity module by using 3-tuple descriptions**. 6. The output of the similarity module is fed into **MOLMO for generating textual descriptions**. 7. Finally, **MOLMO** generates a **Output .JSON** file. A parallel branch starts with the **Input image (32x32)**: 1. It is processed by **GradCAM**. 2. **GradCAM** generates a **Heatmap**. 3. The **Heatmap** is processed by a **Variable Patching Module**. 4. The output of the patching module is an **Interpolated heatmap**. 5. The **Interpolated heatmap** is used to generate **Weighted patches**. 6. The **Weighted patches** are then fed into the **Clip similarity module by using 3-tuple descriptions**. ### Key Observations The pipeline utilizes both a primary path for artifact analysis and a parallel path for generating heatmaps and weighted patches, which are then integrated into the main analysis stream. The use of CLIP suggests a focus on semantic understanding of the artifacts. The final JSON output indicates a structured representation of the analysis results. The initial image resolution is very low (32x32), suggesting the DRCT and Super Resolution modules are crucial for improving the quality of the input data. ### Interpretation This diagram represents a sophisticated system for automated artifact analysis. The pipeline combines image processing techniques (DRCT, super-resolution, GradCAM, variable patching) with semantic understanding (CLIP, MOLMO) to generate structured textual descriptions of artifacts. The separation of artifacts into "biological" and "mechanical" categories suggests the system is designed to handle a diverse range of artifact types. The integration of heatmap-based weighting indicates an attempt to focus the analysis on the most salient features of the artifacts. The final JSON output allows for easy integration of the analysis results into other systems or databases. The pipeline appears to be designed for automated archaeological or historical research, potentially enabling large-scale analysis of artifact collections. </details> Figure 3: Pipeline for Detecting Artifacts Using Super-Resolution and GradCAM ### 4.1 Step 1: Super-Resolution of Images Using DRCT Detecting fine-grained artifacts in low-resolution images, such as those sized 32×32, is inherently difficult due to the lack of detailed spatial information. While upscaling such images using standard interpolation methods such as Bilinear, Bicubic, and Lanczos Interpolation (Khaledyan et al., 2020), (Bituin and Antonio, 2024) can improve visibility, these techniques often introduce artificial smoothing or ringing effects that distort the original image structure. These distortions can obscure or mimic true artifacts, making reliable analysis more difficult. To address this issue, we employ deep learning-based super-resolution methods, which reconstruct high-resolution images by learning mappings from low-resolution inputs to high-resolution outputs. In our work, we utilize DRCT (Dense-Residual-Connected Transformer), a recent hierarchical super-resolution model proposed by Hsu et al. (Hsu et al., 2024), which utilizes the ability of Swin Transformers (Liu et al., 2021) to capture long-range dependencies and reconstruct spatial information for preventing information bottlenecks. This architecture allows DRCT to maintain high-frequency visual features and refine textures and edges more effectively than traditional recursive or interpolative approaches. ### Step 2: GradCAM Heatmap Generation To interpret the decisions made by our binary classifier in our classification pipeline, we employ GradCAM (Gradient-weighted Class Activation Mapping) (Selvaraju et al., 2020), a technique that highlights the regions of an image most influential in the model’s prediction. Specifically, GradCAM helps us visualize which areas the model focuses on when classifying an image as Real or Fake. The method computes the gradient of the target class score with respect to the feature maps from the final convolutional layer. These gradients are then used to weight the feature maps, generating a class-specific activation map that highlights the most salient regions contributing to the model’s decision. By applying a ReLU operation, only positive influences are retained, resulting in a heatmap that overlays the input image. This heatmap serves two primary functions: This heatmap serves two key purposes: - Interpretability – It provides a visual explanation of the model’s classification by indicating which image regions were most influential. - Artifact localization – It assists in detecting potential artifacts or unnatural features in fake images by highlighting anomalous image areas. Importantly, we apply GradCAM on the original 32 $\times$ 32 images, not on resized or super-resolved versions. This ensures that the attention corresponds directly to the image used during classification, avoiding the risk of introducing analysis distortions from post-processing. Despite the low resolution, GradCAM still provides meaningful spatial cues, supporting artifact detection. ### Step 3: Patching and Weighing Based on GradCAM Heatmap For images passed through DRCT, where artifacts are influenced or revealed more distinctly through super-resolution, we perform spatial patch-based analysis to localize and quantify regions that are likely to contain visual anomalies. After applying super-resolution, the resulting high-resolution image is divided into variable-sized patches to enable fine-grained analysis of local regions. To determine the importance of each patch, we utilize the GradCAM heatmap computed in Step 2. Since this heatmap was originally generated on the low-resolution (32 $\times$ 32) input image, we interpolate it to match the spatial dimensions of the super-resolved image. Each patch is then assigned a weight that reflects the importance of that patch for artifact detection as indicated by the heatmap. The weight $w_{\text{patch}}$ of a given patch is computed as the summation of heatmap intensities over all pixels within the patch: $$ w_{\text{patch}}=\sum_{i\in\text{patch}}H(i), \tag{1} $$ where $H(i)$ denotes the GradCAM heatmap intensity at pixel $i$ . ### Step 4: CLIP-Based Probability Score for Artifacts To simplify and improve the process of identifying artifacts, we employ a CLIP-based classification strategy (Radford et al., 2021), since CLIP is an efficient method of learning image-text relationships. Initially, we apply a binary classifier using CLIP similarity to distinguish between two broad semantic categories: animals and vehicles. These categories were chosen due to their highly separable embeddings in the CLIP latent space, which allows us to obtain a well-defined decision boundary. For more targeted artifact detection, we curated class-specific artifact descriptors for each category. Each artifact is described using a three-tuple format: - Positive Description: a description indicating that the patch contains the artifact being described. - Negative Description: a description indicating that the patch does not contain the artifact being described, and is realistic. - Neutral Description: a description indicating that the patch layout is not relevant to the artifact being searched for. For instance, an artifact relating to edge layouts cannot be present in a patch that has no edges. The descriptions were constructed keeping in mind that the CLIP score is affected by semantic similarity. For each patch (as defined in Step 3), we compute CLIP similarity scores against the three descriptors. Based on the highest similarity, each patch casts a vote as positive, negative, or neutral. These votes are then aggregated using a weighted average based on patch importance derived from the GradCAM heatmap. The overall artifact score $S$ is computed using: $$ S=\frac{\sum_{k}w_{k}v_{k}}{\sum_{k}w_{k}}, \tag{2} $$ where $w_{k}$ is the weight of patch $k$ and $v_{k}$ is its corresponding vote (numerically encoded). A higher value of $S$ indicates stronger evidence of artifacts within the image. Patches (and thus images) that exceed a predefined threshold are retained for further inspection, and the others are filtered out. ### Step 5: Generating Textual Descriptions Using MOLMO To enhance artifact detection with interpretable, human-readable insights, we integrate MOLMO (Deitke et al., 2024), a state-of-the-art open-source Vision-Language Model (VLM). For images containing patches identified as likely to exhibit artifacts, MOLMO is prompted to generate detailed textual descriptions that articulate the type and characteristics of the detected artifacts. This step enables the system to produce explanations that are both contextually relevant and semantically informative, thereby improving the interpretability of the analysis. The algorithmic flow of the proposed framework is given below. Algorithm 3 Artifact Detection and Explanation Pipeline Input: Low-resolution image $I_{lr}$ (e.g., 32 $\times$ 32) Output: Artifact detection score $S$ and textual explanation $E$ 1: Step 1: Super-Resolution 2: Super-resolve $I_{lr}$ using DRCT to obtain high-resolution image $I_{sr}$ 3: Step 2: GradCAM Heatmap Generation 4: Apply binary classifier on $I_{lr}$ to obtain class prediction 5: Generate GradCAM heatmap $H$ on $I_{lr}$ 6: Step 3: Patching and Weighting 7: Interpolate $H$ to match dimensions of $I_{sr}$ 8: Divide $I_{sr}$ into patches $\{p_{k}\}$ 9: for all patch $p_{k}$ do 10: Compute patch weight $w_{k}$ as sum of heatmap intensities in $p_{k}$ 11: end for 12: Step 4: CLIP-Based Artifact Classification 13: for all patch $p_{k}$ do 14: Compute CLIP similarity with positive, negative, and neutral descriptors 15: Assign vote $v_{k}$ based on highest similarity descriptor 16: end for 17: Compute artifact score: $S=\frac{\sum_{k}w_{k}v_{k}}{\sum_{k}w_{k}}$ 18: Filter images with $S$ below threshold 19: Step 5: Generating Textual Explanation 20: if $S$ exceeds threshold then 21: Generate textual explanation $E$ using MOLMO for detected artifacts 22: end if ## 5 Experimental Results ### 5.1 Evaluation dataset As generative models improve, distinguishing real from synthetic media becomes increasingly challenging. CIFAKE Bird and Lotfi (2023) directly addresses this issue by providing a balanced and labeled open-source dataset for binary classification. CIFAKE is generated using Stable Diffusion 1.4 (SDM), an example of Latent Diffusion Models used to generate images (Rombach et al., 2022). The SDM generates a synthetic equivalent of CIFAR-10, a well-known dataset of natural images across ten classes. Each image in CIFAKE is 32×32 pixels in resolution. The dataset consists of 120,000 labeled images, evenly split between real and synthetic samples. The real images are sourced from the CIFAR-10 dataset, while the fake counterparts are generated using the Stable Diffusion 1.4 model. Unlike many deepfake or forgery detection datasets, such as FaceForensics++ (Rössler et al., 2019), which provide higher-resolution images, CIFAKE generalizes the detection task to broader categories at a much lower resolution. This introduces unique challenges: low-resolution images inherently contain less semantic and structural information, making it more difficult to detect subtle generative artifacts. Many existing detection architectures often require upsampling or significant architectural modification to function effectively on CIFAR-sized inputs. Smaller images also pose the challenges of identifying whether the distortion is due to the small resolution or is an artifact introduced due to AI generation. ### 5.2 A Comparison of VLMs To identify the right Vision-Language Model for our use case, we perform a comparative study of 3 popular VLMs with the same prompt, and evaluate the quality of the responses. These 3 models are: - MOLMO (Deitke et al., 2024) is an open-source family of state-of-the-art vision-language models (VLMs) developed by the Allen Institute for AI. Trained on the highly curated PixMo dataset, Molmo excels at image understanding, pointing, and object localization, outperforming many larger proprietary models while remaining fully open and reproducible - Qwen 2.5 VL (Bai et al., 2025) is the latest vision-language model in the Qwen series by Alibaba Group, offering advanced visual recognition, document parsing, and long-video understanding. - Pixtral 12B (Agrawal et al., 2024) is Pixtral 12B AI’s flagship 12-billion-parameter VLM, designed for robust performance across both text and multimodal tasks. It features a novel architecture with a 400M vision encoder and excels in document comprehension, chart analysis, and instruction following A collection of fake images from CIFAKE is sampled, and the characteristic artifacts for these images are determined in advance. A sample of these images is given below. | <details> <summary>extracted/6602897/images/truck.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | <details> <summary>extracted/6602897/images/ship.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | <details> <summary>extracted/6602897/images/deer.jpg Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | <details> <summary>extracted/6602897/images/frog.jpg Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | | --- | --- | --- | --- | | (a) Misaligned Body Panels | (b) Non-manifold Geometries | (c) Biological Asymmetry | (d) Impossible Joints | | <details> <summary>extracted/6602897/images/car.jpg Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | <details> <summary>extracted/6602897/images/dog.jpg Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> | | --- | --- | | (e) Car Panel Misalignment | (f) Incorrect Paw Structures | Images corresponding to artifact types listed in Table 1. With this ground truth of artifacts in mind, the 3 models are compared for accuracy and descriptiveness of their evaluations. A sample of results on the above images are described below. | Misaligned Body Panels | The truck’s body panels are not properly aligned, creating an uneven and distorted appearance. | Slight misalignment, particularly noticeable at the panel edges. | Misalignment around the front grille and lower edge of the cab. | | --- | --- | --- | --- | | Non-manifold Geometries in Rigid Structures | The ship’s hull shows sharp, unrealistic edges disrupting the solid form. | Non-manifold geometries at sharp edges and corners. | Non-manifold geometry at cargo container intersections. | | Biological Asymmetry Errors | The deer’s eyes are asymmetrical, with one appearing larger than the other. | Antlers show noticeable size and shape differences. | One antler is more developed than the other, creating asymmetry. | | Anatomically Impossible Joint Configurations | Frog joints and limbs are twisted in anatomically impossible ways. | Unnatural leg posture that defies natural joint configuration. | Legs are bent in ways not physically possible for real frogs. | | Misaligned Body Panels | Visible misalignment near the left front wheel arch. | Misalignment near the trunk lid causing visible gaps. | Misalignment near the rear left wheel arch with visible gaps. | | Anatomically Incorrect Paw Structures | Dog’s paw has fused digits and an extra digit on the other paw. | Twisted and fused digits in the front left paw. | Extra fused digit causing asymmetrical paw appearance. | Table 1: Comparison of VLM (Vision-Language Model) responses to different artifact prompts across models: MOLMO, Qwen2.5 VL, and Pixtral 12B. MOLMO has 4 variants: MOLMO-72B, MOLMO-7B-D, MOLMO-7B-O, and MOLMOE-1B. For all our pipeline-based evaluations, we used MOLMO-7B-D to have a moderate dependence on the size of the model. The following is the general summary of the VLM models: - MOLMO: Descriptions are visually expressive, and the model tends to highlight visible distortions directly. It is particularly effective for generating high-level summaries and professional artifact reviews. - Qwen2.5 VL: This model leverages technical and biological terminology, often producing more precise and domain-specific outputs. It is especially suitable for scientific and clinical interpretation tasks. - Pixtral 12B: The generated descriptions are straightforward and easily readable, making this model well-suited for general audiences or educational purposes. However, it may sometimes lack the nuance or specificity offered by the other models. Based on this comparative evaluation, we select MOLMO for our analysis due to its visually expressive and comprehensive descriptions, which effectively highlight visible distortions and support high-level artifact interpretation. ### 5.3 Localization Properties of MOLMO MOLMO is particularly well-suited for artifact localization tasks due to its attention-guided vision-language alignment mechanism, which enables the model to focus on specific regions within an image during caption generation. Unlike general-purpose Vision-Language Models such as Qwen2.5 VL or Pixtral 12B, which tend to produce global or loosely grounded descriptions, MOLMO leverages cross-modal attention maps to align linguistic tokens with spatially relevant visual features. This design allows MOLMO to produce fine-grained descriptions such as “The deer’s eyes are asymmetrical, with one appearing larger than the other” instead of generic outputs like “Antlers show noticeable size and shape differences.” (Hannan et al., 2025) Therefore, MOLMO’s localization-aware architecture provides a distinct advantage in detecting and describing not only what artifact is present in an image, but also where it occurs, making it an ideal choice for fine-grained artifact analysis. ### 5.4 Performance of the proposed pipeline The following example demonstrates the descriptive capabilities of our proposed pipeline. Among the many artifacts identified across the dataset, a selected subset is presented below to compare the descriptions generated by MOLMO and those produced by our method. Table 2: Artifact List – Proposed Pipeline vs MOLMO | Abruptly cut off objects | Inconsistent object boundaries | | --- | --- | | Ghosting effects: Semi-transparent duplicates of elements | Discontinuous surfaces | | Dental anomalies in mammals | Floating or disconnected components | | Anatomically incorrect paw structures | Asymmetric features in naturally symmetric objects | | Unrealistic eye reflections | Misaligned bilateral elements in animal faces | | Misshapen ears or appendages | Irregular proportions in mechanical components | | Unnatural pose artifacts | Texture bleeding between adjacent regions | | Biological asymmetry errors | Texture repetition patterns | | Impossible foreshortening in animal bodies | Over-smoothing of natural textures | | Anatomically impossible joint configurations | Artificial noise patterns in uniform surfaces | | Asymmetric features in naturally symmetric objects | Unrealistic specular highlights | | Misaligned bilateral elements in animal faces | Inconsistent material properties | | Incorrect perspective rendering | Dental anomalies in mammals | | Scale inconsistencies within single objects | Anatomically incorrect paw structures | | Spatial relationship errors | Improper fur direction flows | | Scale inconsistencies within the same object class | Unrealistic eye reflections | | Depth perception anomalies | Misshapen ears or appendages | | Improper fur direction flows | Physically impossible structural elements | | Inconsistent object boundaries | Inconsistent scale of mechanical parts | | Blurred boundaries in fine details | Incorrect perspective rendering | | Over-sharpening artifacts | Scale inconsistencies within single objects | | Excessive sharpness in certain image regions | Spatial relationship errors | | Aliasing along high-contrast edges | Depth perception anomalies | | Jagged edges in curved structures | Over-sharpening artifacts | | Fake depth of field | Aliasing along high-contrast edges | | Artificial depth of field in object presentation | Blurred boundaries in fine details | | Discontinuous surfaces | Loss of fine detail in complex structures | | Texture bleeding between adjacent regions | Artificial smoothness | | Texture repetition patterns | Movie-poster like composition of ordinary scenes | | Over-smoothing of natural textures | Dramatic lighting that defies natural physics | | Regular grid-like artifacts in textures | Artificial depth of field in object presentation | | Artificial noise patterns in uniform surfaces | Unnaturally glossy surfaces | | Random noise patterns in detailed areas | Synthetic material appearance | | Repeated element patterns | Multiple inconsistent shadow sources | | Systematic color distribution anomalies | Exaggerated characteristic features | | Unnatural color transitions | Impossible foreshortening in animal bodies | | Color coherence breaks | Scale inconsistencies within the same object class | | Frequency domain signatures | | | Artificial smoothness | | | Exaggerated characteristic features | | | Synthetic material appearance | | | Inconsistent material properties | | | Loss of fine detail in complex structures | | | Resolution inconsistencies within regions | | | Unrealistic specular highlights | | Clearly, our proposed approach is more effective than MOLMO in detecting and helping users understand spatial and frequency-based artifacts, especially in resource-constrained image dimensions. The following examples demonstrate the performance of our pipeline for a few images and selected artifacts. <details> <summary>extracted/6602897/img1.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> (a) Example 1 <details> <summary>extracted/6602897/img3.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> (b) Example 2 Figure 4: Examples 1 and 2. Table 3: Descriptions by Proposed Pipeline vs MOLMO for a few artifacts (Example 1) | Abruptly cut-off objects | Visible cut-off edge of an object, possibly a car or vehicle, with a noticeable gap between the main body and a protruding element. | Objects that appear to be suddenly and artificially truncated, disrupting the natural composition. | | --- | --- | --- | | Ghosting effects: Semi-transparent duplicates of elements | Visible ghosting effects near the center of the image, showing semi-transparent duplicates of the main elements, creating an illusion of double vision or afterimages. | Semitransparent duplicates of elements, creating a ghostly or ethereal appearance. | | Anatomically incorrect paw structures | Clear misalignment of frog’s front legs, with left leg appearing much larger and distorted compared to the right. This anatomical discrepancy is visible near the frog’s body, creating an unnatural and unrealistic appearance. | Paws that don’t accurately represent the natural anatomy of the animal. | | Unrealistic eye reflections | Clear misalignment of eye reflections, with the right eye appearing significantly lighter than the left. This discrepancy in shading and highlights creates an unrealistic and visually jarring effect on the frog’s face. | Artificial or incorrect reflections in eyes, creating an unnatural look. | Table 4: Descriptions by Proposed Pipeline vs MOLMO for a few artifacts (Example 2) | Artificial smoothness | Visible artificial smoothness artifact in the image, likely caused by a flaw in the image capture process or post-processing technique. This smooth area contrasts with the surrounding texture, creating an unusual visual effect. | Excessive uniformity; Unnatural, overly polished surface lacking texture and detail. | | --- | --- | --- | | Improper fur direction flows | Clear artifact visible near the left front wheel arch. Misaligned fur direction flows diagonally from top-right to bottom-left, creating an unnatural appearance and disrupting the vehicle’s sleek profile. | Unnatural flow of fur or hair-like features, inconsistent with natural behavior. | | Texture repetition patterns | Visible texture repetition patterns in the image, likely indicating a manufacturing defect or material anomaly in the vehicle’s surface. | Repetitive motifs in the object’s surface, lacking the complexity of real-world textures. | | Frequency domain signatures | Visible frequency domain signature artifact near the center of the image, slightly to the left. It appears as a distorted, blurred area with a checkered pattern effect, likely caused by a flaw in the image acquisition or processing. | Distinct patterns in the image’s frequency spectrum, often indicating artificial generation or heavy processing. | The following examples further demonstrate the usefulness of the pipeline, with 3 artifacts sampled instead of 4, from the numerous artifacts identified by our pipeline and MOLMO. <details> <summary>extracted/6602897/img2.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> (a) Example 3 <details> <summary>extracted/6602897/img4.png Details</summary>  ### Visual Description Icon/Small Image (32x32) </details> (b) Example 4 Figure 5: Examples 3 and 4. Table 5: Descriptions by Proposed Pipeline vs MOLMO for a few artifacts (Example 3) | Incorrect reflection mapping | Noticeable mismatch in reflection between the top and bottom sections of the vehicle’s surface, creating an unrealistic or inconsistent appearance. | Mismatched light reflections creating an unrealistic appearance on the aircraft’s surfaces | | --- | --- | --- | | Impossible mechanical joints | Clear misalignment of mechanical components near the vehicle’s center, creating an unnatural gap between parts. This impossible joint appears to defy physics, with misaligned edges and an irregular space between the mechanical elements. | Joints that defy physics and engineering principles, appearing impossibly structured or positioned | | Unnaturally glossy surfaces | Visible in the top right corner, extending towards the center. Appears as unnaturally smooth and reflective areas contrasting with the surrounding matte surfaces. These glossy patches are noticeable on the vehicle’s exterior, likely indicating a manufacturing defect or intentional design element that stands out from the rest of the body. | Excessive shine on certain areas of the plane, giving it an artificial and unrealistic look | Table 6: Descriptions by Proposed Pipeline vs MOLMO for a few artifacts (Example 4) | Floating or disconnected components | Visible disconnect between vehicle body and wheel well, creating an illusion of floating or disconnection near the left front wheel arch. | Front bumper appears to be disconnected from the car body. | | --- | --- | --- | | Unnaturally glossy surfaces | Visible on the vehicle’s exterior, particularly on the door panels and roof. Excessively reflective areas contrast with the matte black paint, creating an unnatural appearance. This glossy effect is noticeable on the left side of the vehicle, where it stands out against the intended matte finish. | The car’s surface has an excessive shine that doesn’t match real-world lighting conditions | | Unrealistic specular highlights | Noticeable unrealistic specular highlights visible on the vehicle’s surface, particularly on the front right quarter panel and side panels. These highlights appear inconsistent with the vehicle’s surroundings and create an artificial, computer-generated look. | There are overly bright, artificial highlights on the car’s surface | From the above experiments, it is clear that the proposed pipeline offers the following advantages over conventional synthetic image detection frameworks - Deeper and more fine-grained artifact-based analysis of synthetic images - Clear explanation of realness in small images of sizes 32x32 - Highly understandable explanations of why an image is considered synthetic. ## 6 Conclusion and Future Work The proposed framework works by combining three powerful techniques to detect and explain visual artifacts in generated images. First, it uses a specialized image upscaling method that focuses on preserving fine details like edges and textures, which are often where subtle artifacts appear. Next, the image is broken into patches and analyzed using attention maps (like Grad-CAM) to focus on the regions most relevant to the model’s classification decision. Finally, for the most important patches, a Vision-Language Model generates human-readable descriptions that explain what kind of artifact is present and where it appears. By linking low-level visual signals with high-level semantic descriptions, this framework provides both precise localization and meaningful interpretation of artifacts, making it a powerful tool for analyzing manipulated or AI-generated images. While the proposed Task 2 pipeline demonstrates a structured and interpretable approach to artifact detection and explanation, several areas offer opportunities for further refinement: ### 6.1 Aggregation of GradCAM Attention Maps The current system relies on a single-model GradCAM heatmap to localize discriminative regions. However, GradCAM is inherently model-dependent, and its saliency maps may reflect biases unique to a specific network’s learned representations. Aggregating GradCAM maps from multiple classifiers could produce a consensus-based attention signal, reinforcing regions consistently identified across models. This ensemble-based strategy may reduce model-specific artifacts and improve the robustness of localization. ### 6.2 Mitigation of Language Model Hallucinations Vision-language models like MOLMO can generate fluent but factually unsupported descriptions, introducing hallucinations that undermine interpretability. To address this, future work will incorporate hallucination correction techniques such as Woodpecker (Yin et al., 2024), designed to filter or re-rank model outputs based on factual consistency. Such methods can improve the reliability of textual explanations and prevent misleading interpretations. ### 6.3 Improved Image Resizing Strategies Interpolation-based upscaling methods may introduce artificial artifacts that mimic or mask real anomalies. Future iterations of the pipeline could adopt more advanced super-resolution techniques, such as edge-preserving or perceptually-aware models, that retain structural fidelity while minimizing introduced distortions. This would further distinguish true generative artifacts from resizing artifacts. ### 6.4 Adversarial Robustness with Certified Defenses The current system does not address adversarial perturbations, which can distort classification outcomes and mislead GradCAM-based localization. Such perturbations may compromise the interpretability of the pipeline by altering attention maps or affecting semantic scoring. To address this, certified robustness techniques like Auto-LiRPA (Xu et al., 2020) can be employed. Auto-LiRPA computes provable activation bounds under input perturbations, enabling the training of models that maintain stable predictions within a defined epsilon-neighborhood. Integrating this into our pipeline would enhance resilience and preserve interpretability even under adversarial conditions ### 6.5 Generalization to Diverse Generative Artifacts Different generative models, such as diffusion models (e.g., Stable Diffusion), transformers (e.g., PixArt), and autoencoders, produce artifacts with distinct visual characteristics. Diffusion models may generate overly smooth textures, while autoencoders often introduce repetitive patterns or edge degradation. These differences arise from architectural and training variations, leading to unique “fingerprints” in generated images. If a classifier is trained on artifacts from a limited set of models, it risks overfitting to those model-specific cues, reducing its ability to generalize to unseen or emerging generators. To address this, future work should explore domain generalization methods that encourage the model to learn invariant features across multiple generative domains. Additionally, prompt-based adaptation, especially with models like CLIP (Radford et al., 2021), can help tailor detection strategies to diverse image types. These approaches would improve robustness and adaptability as generative models continue to evolve. ## 7 Acknowledgment The authors thank Adobe Research for providing the Inter-IIT Tech Meet 13.0 Problem Statement focused on detecting AI-generated images and generating interpretable explanations based on visual artifacts. The dual-task challenge, which was to identify synthetic images and provide explanations for the artifacts used in the identification, offered a valuable research opportunity. The authors also thank the organizers of Inter-IIT Tech Meet 13.0 for fostering innovation through this competition. This paper presents our independent research and solution to the tasks described above. ## References - (1) FakeFormer: Efficient Vulnerability-Driven Transformers for Generalisable Deepfake Detection. https://arxiv.org/html/2410.21964. - Abir et al. (2022) Wahidul Abir, Faria Khanam, Kazi Alam, Myriam Hadjouni, Hela Elmannai, Sami Bourouis, Rajesh Dey, and Mohammad Khan. Detecting Deepfake Images Using Deep Learning Techniques and Explainable AI Methods. Intelligent Automation & Soft Computing, 35(2):2151–2169, 2022. ISSN 1079-8587, 2326-005X. doi: 10.32604/iasc.2023.029653. - Agrawal et al. (2024) Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. Pixtral 12B, October 2024. - Akagic et al. (2024) Amila Akagic, Emir Buza, Medina Kapo, and Mahdi Bohlouli. Exploring the Impact of Real and Synthetic Data in Image Classification: A Comprehensive Investigation Using CIFAKE Dataset. In 2024 10th International Conference on Control, Decision and Information Technologies (CoDIT), pages 1207–1212, July 2024. doi: 10.1109/CoDIT62066.2024.10708200. - Akhtar et al. (2021) Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in Adversarial Attacks and Defenses in Computer Vision: A Survey. IEEE Access, 9:155161–155196, 2021. ISSN 2169-3536. doi: 10.1109/ACCESS.2021.3127960. - Amirian and Schwenker (2020) Mohammadreza Amirian and Friedhelm Schwenker. Radial Basis Function Networks for Convolutional Neural Networks to Learn Similarity Distance Metric and Improve Interpretability. IEEE Access, 8:123087–123097, 2020. ISSN 2169-3536. doi: 10.1109/ACCESS.2020.3007337. - Anwar and Barnes (2019) Saeed Anwar and Nick Barnes. Densely Residual Laplacian Super-Resolution, July 2019. - Ba and Caruana (2014) Lei Jimmy Ba and Rich Caruana. Do Deep Nets Really Need to be Deep?, October 2014. - Bai et al. (2025) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VL Technical Report, February 2025. - Beggel et al. (2019) Laura Beggel, Michael Pfeiffer, and Bernd Bischl. Robust Anomaly Detection in Images using Adversarial Autoencoders, January 2019. - Bird and Lotfi (2023) Jordan J. Bird and Ahmad Lotfi. CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images, March 2023. - Bituin and Antonio (2024) Ronie C Bituin and Ronielle Antonio. Ensemble Model of Lanczos and Bicubic Interpolation with Neural Network and Resampling for Image Enhancement. In Proceedings of the 2024 7th International Conference on Software Engineering and Information Management, pages 110–115, Suva Fiji, January 2024. ACM. doi: 10.1145/3647722.3647739. URL https://dl.acm.org/doi/10.1145/3647722.3647739. - Bommasani et al. (2022) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models, July 2022. - Brendel and Bethge (2019) Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet, March 2019. - Carlini and Wagner (2017) Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks, March 2017. - Chai et al. (2020) Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? Understanding properties that generalize, August 2020. - Chang et al. (2018) Kan Chang, Pak Lun Kevin Ding, and Baoxin Li. Single Image Super Resolution Using Joint Regularization. IEEE Signal Processing Letters, 25(4):596–600, April 2018. ISSN 1558-2361. doi: 10.1109/LSP.2018.2815003. - Chen et al. (2021) Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-Trained Image Processing Transformer, November 2021. - Chen et al. (2024) Jiaxuan Chen, Jieteng Yao, and Li Niu. A Single Simple Patch is All You Need for AI-generated Image Detection, April 2024. - Chen et al. (2022) Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18009–18019, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1-6654-6946-3. doi: 10.1109/CVPR52688.2022.01750. - Chen et al. (2023) Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. MiniGPT-v2: Large language model as a unified interface for vision-language multi-task learning, November 2023. - Coccomini et al. (2022) Davide Coccomini, Nicola Messina, Claudio Gennaro, and Fabrizio Falchi. Combining EfficientNet and Vision Transformers for Video Deepfake Detection. volume 13233, pages 219–229. 2022. doi: 10.1007/978-3-031-06433-3_19. - Croce and Hein (2020) Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks, August 2020. - Deitke et al. (2024) Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models, December 2024. - Dhamija and Bansal (2024) Lovi Dhamija and Urvashi Bansal. How to Defend and Secure Deep Learning Models Against Adversarial Attacks in Computer Vision: A Systematic Review. New Generation Computing, 42(5):1165–1235, December 2024. ISSN 0288-3635, 1882-7055. doi: 10.1007/s00354-024-00283-0. - Dong et al. (2015) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image Super-Resolution Using Deep Convolutional Networks, July 2015. - Dong et al. (2020) Xiaoyi Dong, Dongdong Chen, Jianmin Bao, Chuan Qin, Lu Yuan, Weiming Zhang, Nenghai Yu, and Dong Chen. GreedyFool: Distortion-Aware Sparse Adversarial Attack, October 2020. - Du et al. (2020) Mengnan Du, Shiva Pentyala, Yuening Li, and Xia Hu. Towards Generalizable Deepfake Detection with Locality-aware AutoEncoder. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 325–334, Virtual Event Ireland, October 2020. ACM. ISBN 978-1-4503-6859-9. doi: 10.1145/3340531.3411892. - Durall et al. (2020) Ricard Durall, Margret Keuper, Franz-Josef Pfreundt, and Janis Keuper. Unmasking DeepFakes with simple Features, March 2020. - (30) Alexei A Efros and William T Freeman. Image Quilting for Texture Synthesis and Transfer. - Emami and Martínez-Muñoz (2023) Seyedsaman Emami and Gonzalo Martínez-Muñoz. Sequential Training of Neural Networks With Gradient Boosting. IEEE Access, 11:42738–42750, 2023. ISSN 2169-3536. doi: 10.1109/ACCESS.2023.3271515. - Frank et al. (2020) Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging Frequency Analysis for Deep Fake Image Recognition, June 2020. - Geirhos et al. (2019) Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, January 2019. - Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014. - Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples, March 2015. - Gunning et al. (2019) David Gunning, Mark Stefik, Jaesik Choi, Timothy Miller, Simone Stumpf, and Guang-Zhong Yang. XAI—Explainable artificial intelligence. Science Robotics, 4(37):eaay7120, 2019. doi: 10.1126/scirobotics.aay7120. URL https://www.science.org/doi/abs/10.1126/scirobotics.aay7120. _eprint: https://www.science.org/doi/pdf/10.1126/scirobotics.aay7120. - Guo et al. (2018) Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering Adversarial Images using Input Transformations, January 2018. - Haliassos et al. (2021) Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Lips Don’t Lie: A Generalisable and Robust Approach to Face Forgery Detection. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5037–5047, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-6654-4509-2. doi: 10.1109/CVPR46437.2021.00500. - Hannan et al. (2025) Darryl Hannan, John Cooper, Dylan White, Timothy Doster, Henry Kvinge, and Yijing Watkins. Foundation Models for Remote Sensing: An Analysis of MLLMs for Object Localization, April 2025. URL http://arxiv.org/abs/2504.10727. arXiv:2504.10727 [cs]. - He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, December 2015. - Hendrycks et al. (2023) Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An Overview of Catastrophic AI Risks, October 2023. - Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015. - Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models, December 2020. - Hooda et al. (2022) Ashish Hooda, Neal Mangaokar, Ryan Feng, Kassem Fawaz, Somesh Jha, and Atul Prakash. D4: Detection of Adversarial Diffusion Deepfakes Using Disjoint Ensembles, 2022. - Hsu et al. (2024) Chih-Chung Hsu, Chia-Ming Lee, and Yi-Shiuan Chou. DRCT: Saving Image Super-Resolution away from Information Bottleneck. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 6133–6142, Seattle, WA, USA, June 2024. IEEE. ISBN 979-8-3503-6547-4. doi: 10.1109/CVPRW63382.2024.00618. - (46) Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin I P Rubinstein, and J D Tygar. Adversarial Machine Learning. - Hwang et al. (2019) Uiwon Hwang, Jaewoo Park, Hyemi Jang, Sungroh Yoon, and Nam Ik Cho. PuVAE: A Variational Autoencoder to Purify Adversarial Examples, March 2019. - Jeong et al. (2021) Yonghyun Jeong, Doyeon Kim, Seungjai Min, Seongho Joe, Youngjune Gwon, and Jongwon Choi. BiHPF: Bilateral High-Pass Filters for Robust Deepfake Detection, August 2021. - Jia et al. (2022) Shuai Jia, Chao Ma, Taiping Yao, Bangjie Yin, Shouhong Ding, and Xiaokang Yang. Exploring Frequency Adversarial Attacks for Face Forgery Detection, March 2022. - Joe et al. (2019) Byunggill Joe, Sung Ju Hwang, and Insik Shin. Learning to Disentangle Robust and Vulnerable Features for Adversarial Detection, September 2019. - John (2021) Jacob John. Discrete Cosine Transform in JPEG Compression, February 2021. - Kalaria et al. (2022) Dvij Kalaria, Aritra Hazra, and Partha Pratim Chakrabarti. Towards Adversarial Purification using Denoising AutoEncoders, August 2022. - Kang et al. (2025) Hengrui Kang, Siwei Wen, Zichen Wen, Junyan Ye, Weijia Li, Peilin Feng, Baichuan Zhou, Bin Wang, Dahua Lin, Linfeng Zhang, and Conghui He. LEGION: Learning to Ground and Explain for Synthetic Image Detection, March 2025. - Keita et al. (2024a) Mamadou Keita, Wassim Hamidouche, Hessen Bougueffa Eutamene, Abdenour Hadid, and Abdelmalik Taleb-Ahmed. Bi-LORA: A Vision-Language Approach for Synthetic Image Detection, April 2024a. - Keita et al. (2024b) Mamadou Keita, Wassim Hamidouche, Hessen Bougueffa Eutamene, Abdelmalik Taleb-Ahmed, and Abdenour Hadid. FIDAVL: Fake Image Detection and Attribution using Vision-Language Model, August 2024b. - Khaledyan et al. (2020) Donya Khaledyan, Abdolah Amirany, Kian Jafari, Mohammad Hossein Moaiyeri, Abolfazl Zargari Khuzani, and Najmeh Mashhadi. Low-Cost Implementation of Bilinear and Bicubic Image Interpolation for Real-Time Image Super-Resolution, September 2020. URL http://arxiv.org/abs/2009.09622. arXiv:2009.09622 [eess]. - Khamaiseh et al. (2022) Samer Y. Khamaiseh, Derek Bagagem, Abdullah Al-Alaj, Mathew Mancino, and Hakam W. Alomari. Adversarial Deep Learning: A Survey on Adversarial Attacks and Defense Mechanisms on Image Classification. IEEE Access, 10:102266–102291, 2022. ISSN 2169-3536. doi: 10.1109/ACCESS.2022.3208131. - Khan and Dang-Nguyen (2024) Sohail Ahmed Khan and Duc-Tien Dang-Nguyen. CLIPping the Deception: Adapting Vision-Language Models for Universal Deepfake Detection, February 2024. - Khormali and Yuan (2023) Aminollah Khormali and Jiann-Shiun Yuan. Self-Supervised Graph Transformer for Deepfake Detection, July 2023. - Kingma and Welling (2022) Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, December 2022. - Kos et al. (2018) Jernej Kos, Ian Fischer, and Dawn Song. Adversarial Examples for Generative Models. In 2018 IEEE Security and Privacy Workshops (SPW), pages 36–42, May 2018. doi: 10.1109/SPW.2018.00014. - Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, February 2017. - Lai et al. (2024) Yingxin Lai, Zitong Yu, Jing Yang, Bin Li, Xiangui Kang, and Linlin Shen. GM-DF: Generalized Multi-Scenario Deepfake Detection, June 2024. - Li et al. (2024) Jialiang Li, Haoyue Wang, Sheng Li, Zhenxing Qian, Xinpeng Zhang, and Athanasios V. Vasilakos. Are handcrafted filters helpful for attributing AI-generated images?, July 2024. - Li et al. (2023) Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning, June 2023. - Li et al. (2018) Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking. In 2018 IEEE International Workshop on Information Forensics and Security (WIFS), pages 1–7, Hong Kong, Hong Kong, December 2018. IEEE. ISBN 978-1-5386-6536-7. doi: 10.1109/WIFS.2018.8630787. - Li et al. (2025) Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges, April 2025. - Liang et al. (2021) Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. SwinIR: Image Restoration Using Swin Transformer, August 2021. - Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, December 2023. - Liu et al. (2024a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning, May 2024a. - Liu et al. (2024b) Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models, April 2024b. - Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, August 2021. - Lu et al. (2023) Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action, December 2023. - Maaz et al. (2024) Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models, June 2024. - Madry et al. (2019) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, September 2019. - Malolan et al. (2020) Badhrinarayan Malolan, Ankit Parekh, and Faruk Kazi. Explainable Deep-Fake Detection Using Visual Interpretability Methods. In 2020 3rd International Conference on Information and Computer Technologies (ICICT), pages 289–293, March 2020. doi: 10.1109/ICICT50521.2020.00051. - Menet et al. (2020) François Menet, Paul Berthier, Michel Gagnon, and José M. Fernandez. Spartan Networks: Self-feature-squeezing neural networks for increased robustness in adversarial settings. Computers & Security, 88:101537, January 2020. ISSN 01674048. doi: 10.1016/j.cose.2019.05.014. - Meng and Chen (2017) Dongyu Meng and Hao Chen. MagNet: A Two-Pronged Defense against Adversarial Examples, September 2017. - Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2574–2582, June 2016. doi: 10.1109/CVPR.2016.282. - Naskar et al. (2024) Gourab Naskar, Sk Mohiuddin, Samir Malakar, Erik Cuevas, and Ram Sarkar. Deepfake detection using deep feature stacking and meta-learning. Heliyon, 10(4):e25933, February 2024. ISSN 24058440. doi: 10.1016/j.heliyon.2024.e25933. - Ojha et al. (2024) Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards Universal Fake Image Detectors that Generalize Across Generative Models, April 2024. - Papernot et al. (2016a) Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The Limitations of Deep Learning in Adversarial Settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372–387, March 2016a. doi: 10.1109/EuroSP.2016.36. - Papernot et al. (2016b) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks. In 2016 IEEE Symposium on Security and Privacy (SP), pages 582–597, May 2016b. doi: 10.1109/SP.2016.41. - Pinhasov et al. (2024) Ben Pinhasov, Raz Lapid, Rony Ohayon, Moshe Sipper, and Yehudit Aperstein. XAI-Based Detection of Adversarial Attacks on Deepfake Detectors, August 2024. - Qian et al. (2020) Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, volume 12357, pages 86–103. Springer International Publishing, Cham, 2020. ISBN 978-3-030-58609-6 978-3-030-58610-2. doi: 10.1007/978-3-030-58610-2_6. - Qin et al. (2019) Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial Robustness through Local Linearization, October 2019. - Qin et al. (2023) Haotong Qin, Ge-Peng Ji, Salman Khan, Deng-Ping Fan, Fahad Shahbaz Khan, and Luc Van Gool. How Good is Google Bard’s Visual Understanding? An Empirical Study on Open Challenges. Machine Intelligence Research, 20(5):605–613, October 2023. ISSN 2731-538X, 2731-5398. doi: 10.1007/s11633-023-1469-x. - Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. - Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, April 2022. - Romeo et al. (2024) Lanzino Romeo, Fontana Federico, Diko Anxhelo, Marini Marco Raoul, and Cinque Luigi. Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks, June 2024. - Ross and Doshi-Velez (2018) Andrew Ross and Finale Doshi-Velez. Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018. ISSN 2374-3468, 2159-5399. doi: 10.1609/aaai.v32i1.11504. - Rössler et al. (2019) Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics++: Learning to Detect Manipulated Facial Images, August 2019. - Rudin et al. (1992) Leonid I. Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259–268, November 1992. ISSN 01672789. doi: 10.1016/0167-2789(92)90242-F. - Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. International Journal of Computer Vision, 128(2):336–359, February 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007/s11263-019-01228-7. - Sun et al. (2019) Bo Sun, Nian-Hsuan Tsai, Fangchen Liu, Ronald Yu, and Hao Su. Adversarial Defense by Stratified Convolutional Sparse Coding. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11439–11448, Long Beach, CA, USA, June 2019. IEEE. ISBN 978-1-7281-3293-8. doi: 10.1109/CVPR.2019.01171. - Sun and Li (2019) Na Sun and Huina Li. Super Resolution Reconstruction of Images Based on Interpolation and Full Convolutional Neural Network and Application in Medical Fields. IEEE Access, 7:186470–186479, 2019. ISSN 2169-3536. doi: 10.1109/ACCESS.2019.2960828. - Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks, February 2014. - Tan et al. (2023) Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, and Yunchao Wei. Learning on Gradients: Generalized Artifacts Representation for GAN-Generated Images Detection. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12105–12114, Vancouver, BC, Canada, June 2023. IEEE. ISBN 979-8-3503-0129-8. doi: 10.1109/CVPR52729.2023.01165. - Tishby and Zaslavsky (2015) Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1–5, Jerusalem, Israel, April 2015. IEEE. ISBN 978-1-4799-5524-4 978-1-4799-5526-8. doi: 10.1109/ITW.2015.7133169. - Tong and Leung (2007) C. S. Tong and K. T. Leung. Super-resolution reconstruction based on linear interpolation of wavelet coefficients. Multidimensional Systems and Signal Processing, 18(2-3):153–171, September 2007. ISSN 0923-6082, 1573-0824. doi: 10.1007/s11045-007-0023-2. - van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(86):2579–2605, 2008. - Wang et al. (2021) Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A General U-Shaped Transformer for Image Restoration, November 2021. - Wang et al. (2024) Zhikan Wang, Zhongyao Cheng, Jiajie Xiong, Xun Xu, Tianrui Li, Bharadwaj Veeravalli, and Xulei Yang. A Timely Survey on Vision Transformer for Deepfake Detection, May 2024. - Wu et al. (2025) Yixin Wu, Feiran Zhang, Tianyuan Shi, Ruicheng Yin, Zhenghua Wang, Zhenliang Gan, Xiaohua Wang, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. Explainable Synthetic Image Detection through Diffusion Timestep Ensembling, March 2025. - Xiang et al. (2021) Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal. PatchGuard: A Provably Robust Defense against Adversarial Patches via Small Receptive Fields and Masking, March 2021. - Xu et al. (2020) Kaidi Xu, Zhouxing Shi, Huan Zhang, Yihan Wang, Kai-Wei Chang, Minlie Huang, Bhavya Kailkhura, Xue Lin, and Cho-Jui Hsieh. Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond, October 2020. - Yan et al. (2021) Xin Yan, Yanjie Li, Tao Dai, Yang Bai, and Shu-Tao Xia. D2Defend: Dual-Domain based Defense against Adversarial Examples. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2021. doi: 10.1109/IJCNN52387.2021.9533589. - Yin et al. (2024) Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination Correction for Multimodal Large Language Models. Science China Information Sciences, 67(12):220105, December 2024. ISSN 1674-733X, 1869-1919. doi: 10.1007/s11432-024-4251-x. - Yu et al. (2017) Lejun Yu, Siming Cao, Jun He, Bo Sun, and Feng Dai. Single-image super-resolution based on regularization with stationary gradient fidelity. In 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 1–5, October 2017. doi: 10.1109/CISP-BMEI.2017.8301942. - Yu et al. (2019) Ning Yu, Larry Davis, and Mario Fritz. Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 7555–7565, Seoul, Korea (South), October 2019. IEEE. ISBN 978-1-7281-4803-8. doi: 10.1109/ICCV.2019.00765. - Zhang et al. (2024a) Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. MM-LLMs: Recent Advances in MultiModal Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12401–12430, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.738. - Zhang et al. (2024b) Li Zhang, Dezong Zhao, Chee Peng Lim, Houshyar Asadi, Haoqian Huang, Yonghong Yu, and Rong Gao. Video Deepfake classification using particle swarm optimization-based evolving ensemble models. Knowledge-Based Systems, 289:111461, April 2024b. ISSN 09507051. doi: 10.1016/j.knosys.2024.111461. - Zhang et al. (2018) Yunfeng Zhang, Qinglan Fan, Fangxun Bao, Yifang Liu, and Caiming Zhang. Single-Image Super-Resolution Based on Rational Fractal Interpolation. IEEE Transactions on Image Processing, 27(8):3782–3797, August 2018. ISSN 1941-0042. doi: 10.1109/TIP.2018.2826139. - Zhang et al. (2023) Yutong Zhang, Yao Li, Yin Li, and Zhichang Guo. A Review of Adversarial Attacks in Computer Vision, August 2023. - Zhang et al. (2020) Zhanyuan Zhang, Benson Yuan, Michael McCoyd, and David Wagner. Clipped BagNet: Defending Against Sticker Attacks with Clipped Bag-of-features. In 2020 IEEE Security and Privacy Workshops (SPW), pages 55–61, May 2020. doi: 10.1109/SPW50608.2020.00026. - Zhang et al. (2024c) Zhun Zhang, Yi Zeng, Qihe Liu, and Shijie Zhou. Towards a Novel Perspective on Adversarial Examples Driven by Frequency, April 2024c. - Zhong et al. (2024) Nan Zhong, Yiran Xu, Sheng Li, Zhenxing Qian, and Xinpeng Zhang. PatchCraft: Exploring Texture Patch for Efficient AI-generated Image Detection, March 2024. - Zhong and Deng (2019) Yaoyao Zhong and Weihong Deng. Adversarial Learning With Margin-Based Triplet Embedding Regularization. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6548–6557, Seoul, Korea (South), October 2019. IEEE. ISBN 978-1-7281-4803-8. doi: 10.1109/ICCV.2019.00665. - (119) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MINIGPT-4: ENHANCING VISION-LANGUAGE UNDERSTANDING WITH ADVANCED LARGE LANGUAGE MODELS. ## Appendix Appendix A Artifact Prompting Instruction: You are a helpful assistant that identifies errors and artifacts in images. Given the error code, describe instances in the image where the error occurs. JSON Schema: {“artifact”: “...”, “description”: “...” } Example: {“artifact”: “biological_asymmetry”, “description”: “In the given image, the horse has unsymmetrical eyes” } Guidelines: - Only describe the given artifact. Do not mention unrelated defects. - Limit each response to 1–2 lines. - Use directional or anatomical terms (e.g., “left paw,” “lower trunk”). - Highlight visibility using terms like “noticeable,” “clearly seen,” or “subtle.” - Follow the JSON schema strictly. ## Appendix Appendix B Adversarial Attacks and Defense Mechanisms Adversarial attacks on images involve applying carefully crafted perturbations to mislead image classification models. While some attacks are carried out without significant knowledge of the weights, gradients and other features of the target model, such as generic modifications to the spatial or frequency-domain features of the image (black-box attacks), other attacks are designed to affect the classification outcome of a particular model based upon limited or perfect knowledge of the workings of the target model(gray-box or white-box attacks, respectively) (Khamaiseh et al., 2022) (Zhang et al., 2023), (Dhamija and Bansal, 2024). Such attacks have been shown to exploit various vulnerabilities in models and pose serious security risks in practical settings (Goodfellow et al., 2015), (Huang et al., ), (Szegedy et al., 2014), (Papernot et al., 2016a), (Kurakin et al., 2017). Such attacks help enhance machine learning models by exposing vulnerabilities and, if incorporated in the training process, encouraging reliance on meaningful features instead of secondary details. Certain white-box attacks exploit the gradients of the target model, such as the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015), DeepFool (DF) (Moosavi-Dezfooli et al., 2016) and Jacobian-based Saliency Map Attacks (JSMA) (Papernot et al., 2016a), with more advanced attacks also being proposed such as GreedyFool (Dong et al., 2020). Other white-box attacks convert attacks into optimization problems, such as the Carlini and Wagner Attack (C&W) (Carlini and Wagner, 2017). Grey-box attacks include the usage of generative models such as Variational AutoEncoders to generate targeted adversarial examples (Kos et al., 2018). Black-box attacks are usually executed as query-based attacks or transfer-based attacks, which operate solely based on outputs of the model and no knowledge of its inner workings (Akhtar et al., 2021) (Dhamija and Bansal, 2024). Some attacks are discussed in greater detail below: #### Fast Gradient Sign Method (Goodfellow et al., 2015): It exploits the gradient of the loss function with respect to the input data to craft a perturbation in the direction that maximizes the model’s loss. Algorithm 4 Fast Gradient Sign Method (FGSM) Input: Input $x$ , true label $y$ , model parameters $\theta$ , loss function $J(\theta,x,y)$ , perturbation magnitude $\epsilon$ Output: Adversarial example $x^{\prime}$ Compute gradient: $\mathbf{g}\leftarrow\nabla_{x}J(\theta,x,y)$ Compute perturbation: $\eta\leftarrow\epsilon\cdot\text{sign}(\mathbf{g})$ Generate adversarial example: $x^{\prime}\leftarrow x+\eta$ return $x^{\prime}$ #### Projected Gradient Descent (Madry et al., 2019): It is an iterative adversarial attack that builds upon FGSM by applying multiple small gradient steps, followed by a projection onto the allowed perturbation space to ensure the perturbation remains within a defined ball around the original input. Algorithm 5 Projected Gradient Descent (PGD) Attack Input: Input $x$ , true label $y$ , loss function $J(\theta,x,y)$ , perturbation magnitude $\epsilon$ , step size $\alpha$ , iterations $T$ Output: Adversarial example $x^{\prime}$ 1: Initialize $x_{0}\leftarrow x$ 2: for $t=1$ to $T$ do 3: Compute gradient: $\mathbf{g}_{t}\leftarrow\nabla_{x}J(\theta,x_{t},y)$ 4: Update input: $x_{t+1}\leftarrow x_{t}+\alpha\cdot\text{sign}(\mathbf{g}_{t})$ 5: Project: $x_{t+1}\leftarrow\text{Clip}(x_{t+1},x-\epsilon,x+\epsilon)$ 6: end for 7: return $x^{\prime}\leftarrow x_{T}$ [1] #### Wavelet Packet Decomposition (Zhang et al., 2024c): This method uses a wavelet transform to break down an input image into frequency components. By slightly altering the wavelet coefficients at specific scales and then reconstructing the image, it creates an adversarial example designed to mislead neural networks. Algorithm 6 Wavelet Decomposition Attack Input: Input image $x$ , true label $y$ , loss function $J(\theta,x,y)$ , perturbation magnitude $\epsilon$ , wavelet levels $L$ Output: Adversarial example $x^{\prime}$ 1: Compute wavelet decomposition: $W\leftarrow\text{WaveletDecompose}(x)$ 2: Compute gradient: $\mathbf{g}_{W}\leftarrow\nabla_{W}J(\theta,W,y)$ 3: Select coefficients from first $L$ levels 4: Perturb selected coefficients: $W^{\prime}\leftarrow W+\epsilon\cdot\mathbf{g}_{W}$ 5: Reconstruct image: $x^{\prime}\leftarrow\text{InverseWavelet}(W^{\prime})$ 6: return $x^{\prime}$ #### AutoAttack (Croce and Hein, 2020): AutoAttack is an ensemble-based method designed for untargeted adversarial attacks, aiming to generate adversarial examples that cause a model to misclassify without targeting a specific class. By adaptively combining multiple attack strategies, AutoAttack achieves a high success rate of confusing models with lesser magnitude of applied perturbations. Algorithm 7 AutoAttack Input: Input image $x$ , true label $y$ , model $f$ , perturbation magnitude $\epsilon$ Output: Adversarial example $x^{\prime}$ 1: Initialize $x_{0}\leftarrow x$ 2: Select predefined set of attack methods 3: for each attack method $\mathcal{A}$ in the set do 4: Generate attack: $x^{\prime}\leftarrow\mathcal{A}(x_{0},y,f,\epsilon)$ 5: if $x^{\prime}$ is adversarial then 6: return $x^{\prime}$ 7: end if 8: end for 9: return $x^{\prime}$ (final adversarial example) Traditional spatial domain attacks, such as FGSM and PGD, often result in noticeable artifacts that can be easily detected by both humans and detectors. In contrast, frequency-domain attacks, particularly those manipulating high-frequency bands using Discrete Cosine Transform (DCT), allow for imperceptible perturbations that usually leave the image intact visually, making them difficult to identify. A hybrid attack strategy combining both spatial and frequency domains has shown commendable promise in the disruption of model classification (Jia et al., 2022). Improving robustness to adversarial examples has been an active area of research. Some significant directions of improving robustness are adversarial training, image transformations, and image denoising (Goodfellow et al., 2015), (Szegedy et al., 2014), (Madry et al., 2019), (Guo et al., 2018). Adversarial training involves creating a training dataset that contains a variety of adversarial attacks and enabling the model to recognize and accurately classify such perturbations (Goodfellow et al., 2015), (Szegedy et al., 2014), (Madry et al., 2019), (Kurakin et al., 2017). To tackle the issue of computational costs involved with adversarial training of large models and a wide range of attacks, (Qin et al., 2019) proposed a method that simulates adversarial training on the loss surface using a new regularizer. Another method of input transformation involves Feature Representation learning to differentiate between different features better. (Joe et al., 2019) encourages the model to learn the feature space to identify vulnerable latent features, which form the basis for the effectiveness of adversarial attacks. (Menet et al., 2020) proposed Spartan Networks, which are deep neural networks with a modified activation function that reduces the information obtained from the input, hence forcing only relevant features to be learned. Since reducing the amount of excess information has a positive impact on robustness, BAGNet (Brendel and Bethge, 2019) and its derivatives, such as Clipped BAGNet (Zhang et al., 2020), limit the receptive field of the convolutional layers through dimension reduction of filters and introducing averaging methods instead. PatchGuard (Xiang et al., 2021) also introduces a robust masking scheme to mask attacks on images, apart from reducing the input receptive field. (Goodfellow et al., 2015), (Szegedy et al., 2014), (Madry et al., 2019) proposed stochastic input transformations such as bit-depth reduction and JPEG compression, which can be applied to adversarially attacked images before feeding them into convolutional networks, and demonstrated that total variance minimization (Rudin et al., 1992) and image quilting (Efros and Freeman, ) are effective in tackling many adversarial attacks. Defensive distillation, proposed by (Papernot et al., 2016b) and built upon the idea of distillation ideated by (Ba and Caruana, 2014), uses an innovative method of training a “teacher” model on the input data, following which the output probabilities are learned by a “student” model, which helps the student model make smoother decisions and evade attacks. Transitioning from linear models to nonlinear models such as Radial Basis Function (RBF) Networks (Amirian and Schwenker, 2020) has shown improved resilience against adversarial examples due to their local receptive properties. Since several attacks utilize gradient information from models, and since model sensitivity is related to how small changes in the input can affect the outcome of the classification, (Ross and Doshi-Velez, 2018) performs input regularization, which then reduces the tendency of target models to flip decisions with perturbed inputs. Other forms of regularization are also explored in (Zhong and Deng, 2019), (Qin et al., 2019). To target imperceptible manipulations in the frequency domain, some approaches decompose images into high and low-frequency components, highlighting forgery-prone regions such as edges and textures. This decomposition, along with methods such as Fourier analysis, also helps identify frequency-domain artifacts and signatures of generative models, such as GANs (Durall et al., 2020), (Frank et al., 2020). (Qian et al., 2020) propose F3-Net, combining Frequency-Aware Decomposition (FAD) and Local Frequency Statistics (LFS). FAD uses bandpass filtering to isolate high-frequency components, where manipulations like edges or textures are more evident, while LFS applies a sliding-window Discrete Cosine Transform (DCT) (John, 2021) to detect texture inconsistencies. An alternative approach to adversarial robustness is ensuring that the internal representations and feature maps of the model are modified to be resilient to perturbations. This is done by (Sun et al., 2019), which introduces a sparse transformation layer to convert inputs into a quasi-natural picture space. Other defensive strategies typically focus on adversarial purification or robust feature learning. AutoEncoders reconstruct clean approximations of perturbed inputs, suppressing adversarial noise. MagNet (Meng and Chen, 2017) combines detection and purification AutoEncoders to project adversarial samples back onto the manifold of natural images. D2Defend (Yan et al., 2021) introduces a dual-domain denoising strategy that leverages bilateral filtering in the spatial domain and wavelet-based shrinkage in the frequency domain to isolate and remove adversarial signals.