Signature and Log-Signature for the Study of Empirical Distributions Generated with GANs

Abstract

In this paper, we address the research gap in efficiently assessing Generative Adversarial Network (GAN) convergence and goodness of fit by introducing the application of the Signature Transform to measure similarity between image distributions. Specifically, we propose the novel use of Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) Signature, along with Log-Signature, as alternatives to existing methods such as Fréchet Inception Distance (FID) and Multi-Scale Structural Similarity Index Measure (MS-SSIM). Our approach offers advantages in terms of efficiency and effectiveness, providing a comprehensive understanding and extensive evaluations of GAN convergence and goodness of fit. Furthermore, we present innovative analytical measures based on statistics by means of Kruskal–Wallis to evaluate the goodness of fit of GAN sample distributions. Unlike existing GAN measures, which are based on deep neural networks and require extensive GPU computations, our approach significantly reduces computation time and is performed on the CPU while maintaining the same level of accuracy. Our results demonstrate the effectiveness of the proposed method in capturing the intrinsic structure of the generated samples, providing meaningful insights into GAN performance. Lastly, we evaluate our approach qualitatively using Principal Component Analysis (PCA) and adaptive t-Distributed Stochastic Neighbor Embedding (t-SNE) for data visualization, illustrating the plausibility of our method.

1. Introduction

Generative Adversarial Networks (GANs) [1] have gained significant attention in recent years as a powerful tool for generating realistic synthetic images, with a wide range of applications in computer vision [2,3], graphics [4,5], and Machine Learning (ML) [6,7]. Despite their remarkable successes, assessing the quality of the generated samples and measuring the convergence of GANs remain challenging tasks. Existing metrics, such as Fréchet Inception Distance (FID) [8] and Multi-Scale Structural Similarity Index Measure (MS-SSIM) [9] have been widely used, but they suffer from certain limitations. These limitations include the requirement of substantial computational resources and time, dependence on specific Deep Learning (DL) architectures, and limited interpretability, which restrict their practical applicability and hinder further advancements in the field.

To address these challenges, there is a pressing need for a novel approach that can efficiently and effectively assess GAN-generated images while maintaining the same level of accuracy as existing metrics. Moreover, such an approach should provide a deeper understanding of the underlying distributions of the generated samples and be applicable across different GAN architectures and problem domains.

In this paper, we present a novel approach to study empirical distributions generated with GANs, leveraging the well-established Signature Transform and Log-Signature as powerful mathematical tools [10,11,12,13]. Our work is the first to introduce the use of Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) Signature, along with Log-Signature, as alternatives for measuring GAN convergence. Furthermore, we propose the application of analytical measures based on statistics to study the goodness of fit of the GAN sample distribution, which are both efficient and effective. In contrast to existing GAN metrics that involve considerable GPU-based computation, our approach significantly reduces computation time and resources while maintaining the same level of accuracy.

We propose a two-fold approach. First, we introduce a score function based on the Signature Transform [14] to evaluate image quality in a novel manner, offering reliability, speed, and ease of computation for each epoch. Second, we employ statistical techniques to study the goodness of fit of the generated distribution, providing a standardized pipeline for interpreting the results of the converged sample distribution. A key contribution of this paper is the introduction of Kruskal–Wallis for GAN assessment, which enables a robust comparison of the goodness of fit between the generated and target distributions. These statistical techniques are computationally efficient, requiring minimal overhead and enabling on-the-fly computation. To qualitatively illustrate the good performance of our measure, we also utilize Principal Component Analysis (PCA) and t-Distributed Stochastic Neighbor Embedding (t-SNE) [15] for data visualization, enabling a visual assessment of the effectiveness of our proposed method in capturing the intrinsic structure of the generated samples.

The remainder of this paper is organized as follows: Section 2 provides an overview of the field and reviews related work. Section 3 discusses Generative Adversarial Networks. Section 4 covers non-parametric statistical analysis with a focus on Kruskal–Wallis, whereas Section 5 introduces the Signature Transform. Section 6 presents our methodology, with Section 6.1 and Section 6.2 detailing the introduced techniques for statistical analysis of the generated distribution and the RMSE and MAE Signature and Log-Signature, respectively. Section 7 presents the evaluation of our approach, Section 7.1 presents the computational complexity of the proposed approaches in comparison against other methodologies, and Section 7.2 discusses visualization techniques. Finally, Section 8 concludes the paper and offers suggestions for future work.

2. Overview and Related Work

The advent of DL has revolutionized numerous fields and disciplines, enabling game-changing applications that rely on vast amounts of data [16,17,18]. These advancements have significantly improved accuracy and speed, opening the door for the use of automated learning techniques in critical scenarios, such as safety-critical systems and self-driving cars [6,19,20,21,22,23,24].

Some notable works in this area include the development of object detection and image segmentation algorithms [16,18,25,26], as well as pioneering research in image synthesis and style transfer [27,28,29]. Additionally, breakthroughs in image recognition and classification [17,30], attention mechanisms in natural language processing [31,32], and various other domains [33] exemplify the widespread impact of DL. As DL techniques continues to advance, their influence is becoming more pervasive, pushing the boundaries of what is possible in research and real-world applications.

Generative models, particularly Generative Adversarial Networks (GANs), have emerged as a powerful and influential area of research within the DL domain. These models have shown remarkable success in a wide range of applications, such as image synthesis and style transfer [27,28,29], pushing the boundaries of what is possible in research and real-world applications, whereas DL has also brought advancements in other areas, including object detection and image segmentation [16,18,25,26], image recognition and classification [17,30,33], and attention mechanisms in Natural Language Processing (NLP) [31,32]. The focus of our study is in the realm of generative models and their applications, as they hold great potential for further exploration and innovation [34].

The domain of synthetic image generation has witnessed remarkable advancements in recent years. Driven by the demand for synthetic imagery in various applications, such as simulated environments [35], additional training data [36], and style transfer [27], significant research efforts have been devoted to establishing stable and principled methods for achieving these goals. Prominent approaches like Generative Adversarial Networks (GANs) [1,4,37,38,39,40,41,42,43] and Variational AutoEncoders (VAEs) [44] offer stable training mechanisms for convergence.

However, there is still room for improvement in this field, as the capacity of these networks is often limited by the available GPU memory and training resources [6,20,29,35,45,46]. This limitation can lead to reduced performance, effectiveness, and applicability of GANs in real-world scenarios. Challenges such as mode collapse [47] and gradient explosion [48] persist, and the effectiveness of these methods in handling complex tasks, such as generating additional multi-view frames [49], remains to be validated. Furthermore, the development of more efficient training and optimization algorithms could potentially alleviate resource constraints and unlock the full potential of GANs in various applications.

The work presented in [50] introduced an innovative generative model based on annealed Langevin [51,52], which was further developed in [53] to demonstrate competitive image generation capabilities. Building on the principles derived from diffusion-based methods [54], Diffusion Probabilistic Models [55] attained state-of-the-art results on the CIFAR10 dataset. However, Score-Based Generative Models [56] face similar challenges as GANs, making their real-time implementation unfeasible due to the sampling step that requires the output dimension to match the input dimension. Consequently, these models are heavily reliant on GPU memory resources and demand extensive computing time, which poses significant limitations to their applicability and performance in practical scenarios. As the field continues to advance, addressing these challenges will be crucial for unlocking the full potential of generative models and expanding their use across diverse applications. Supplemental recent approaches [3,5,57] are based on the attention mechanism [31] building mainly on Vision Transformers [58]. Other techniques like NeRF [23] could be essential to add structure to the learning paradigm.

Moreover, Stable Diffusion [59,60] has emerged as a promising direction for generative models, building upon the success of earlier diffusion-based methods [61,62,63]. These models are designed to address some of the limitations and challenges faced by their predecessors, such as training instability and poor sample quality [64]. By refining the diffusion process and optimizing the training procedure, Stable Diffusion has shown significant improvements in terms of sample diversity, fidelity, and overall performance [65,66,67]. More recently, approaches inspired by Reinforcement Learning from Human Feedback (RLHF) have also presented a new autoregressive model for images [68].

In this context, our proposed method offers a computationally efficient and effective alternative for assessing GAN convergence [69] and the goodness of fit of the generated sample distribution. By leveraging the Signature Transform and statistical techniques through the use of a non-parametric test, our approach addresses the limitations of existing methods and provides a more practical solution for real-world applications; whereas our focus is on GAN convergence, it is worth noting that the proposed metrics can also be applied to Stable Diffusion or any other generative models capable of producing high-fidelity imagery.

3. Generative Adversarial Networks

Generative Adversarial Networks (GANs) are a class of DL models introduced in [1]. They consist of two neural networks, a generator, and a discriminator that are trained simultaneously in a game-theoretic framework. The generator creates synthetic samples, whereas the discriminator learns to distinguish between real samples from the training data and fake samples generated by the generator. This competition between the two networks drives the generator to produce more realistic samples over time, eventually leading to the generation of samples that are difficult to distinguish from the true data.

3.1. GAN Architecture

Let $\mathcal{X}$ represent the true data distribution and $\mathcal{Z}$ represent the noise distribution. The generator $G:\mathcal{Z}\to \mathcal{X}$ is a neural network that transforms noise samples $z\sim \mathcal{Z}$ into synthetic samples $x_{fake}=G\left(z\right)$. The discriminator $D:\mathcal{X}\to [0,1]$ is a neural network that takes either real samples $x_{real}\sim \mathcal{X}$ or fake samples $x_{fake}$ and outputs the probability that the given sample is from the true data distribution.

3.2. GAN Training

The training process of GANs involves finding the optimal parameters for the generator and discriminator networks by solving a minimax optimization problem:

$$\underset{G}{min}\underset{D}{max}\mathcal{L}(D,G)={\mathbb{E}}_{x_{real}\sim \mathcal{X}}[logD\left(x_{real}\right)]+{\mathbb{E}}_{z\sim \mathcal{Z}}[log(1-D\left(G\left(z\right)\right))].$$

(1)

The discriminator tries to maximize the objective function $\mathcal{L}(D,G)$ by correctly classifying real and fake samples, whereas the generator tries to minimize it by generating samples that the discriminator misclassifies as real. This is achieved by alternating between updating the weights of the discriminator and the generator using gradient-based optimization methods, such as stochastic gradient descent or Adam.

3.3. GAN Convergence

One of the main challenges in training GANs is the convergence issue. Ideally, the training process should converge when the generator produces samples that are indistinguishable from the true data distribution, and the discriminator is unable to differentiate between real and fake samples. In practice, however, GANs may suffer from various issues, such as mode collapse, where the generator produces only a limited variety of samples, or oscillations, where the generator and discriminator keep outperforming each other without reaching a stable equilibrium.

Several metrics have been proposed to measure GAN convergence and assess the quality of the generated samples, such as the Fréchet Inception Distance (FID) [8], the Inception Score (IS), and the Kullback–Leibler (KL) divergence. In this paper, we introduce the use of Signature Transform and Log-Signature as alternative methods for evaluating GAN convergence, providing a novel perspective on the problem.

Other additional metrics that are relevant to the problem are:

• LPIPS (Learned Perceptual Image Patch Similarity) is a perceptual similarity metric introduced in [69]. It computes the similarity between two images by comparing their feature representations in a deep neural network (typically pretrained on a large-scale image classification task). The metric has been shown to correlate well with human perceptual judgments of image similarity, and it has been used in various image synthesis and image quality assessment tasks.
• PSNR (Peak Signal-to-Noise Ratio) is a widely-used metric for image quality assessment, particularly in the field of image compression. It is a simple, easy-to-compute measure that compares the maximum possible power of a signal (in this case, an image) to the power of the corrupting noise (differences between the reference and distorted images). It is calculated as the logarithmic ratio of the maximum possible pixel value squared to the mean squared error (MSE) between the reference and distorted images. Although PSNR is widely used, it has been criticized for not always correlating well with human perception of image quality, as it is based on pixel-wise differences and does not consider higher-level semantic or structural features.

In our study, we have focused on introducing the Signature Transform as a novel approach for evaluating GAN-generated images and measuring their convergence; whereas LPIPS and PSNR are relevant metrics for image quality assessment, they may not be the most appropriate metrics for our specific context, as our goal is to develop a computationally efficient and reliable measure for GAN convergence.

3.4. Stylegan2-ADA

Stylegan2-ADA is an extension of the StyleGAN2 architecture, which was developed in [70] to generate high-quality synthetic images. StyleGAN2 builds on the original StyleGAN [4] by introducing several improvements to address issues such as artifacts and training stability. The main contribution of Stylegan2-ADA is the use of Adaptive Discriminator Augmentation (ADA) to enhance the performance of GANs with limited training data.

StyleGAN2 consists of a Generator (G) and a Discriminator (D), which are trained adversarially. The Generator creates images, whereas the Discriminator evaluates their authenticity. The objective function for the Generator, G, and the Discriminator, D, can be written as:

$$\begin{array}{c}\hfill \underset{G}{min}\underset{D}{max}{\mathbb{E}}_{x\sim p_{data}}\left[D\left(x\right)\right]-{\mathbb{E}}_{z\sim p_z}\left[D\left(G\left(z\right)\right)\right].\end{array}$$

(2)

The generator in StyleGAN2 consists of a mapping network $f\left(z\right)$ and a synthesis network $g\left(w\right)$. The mapping network $f\left(z\right)$ converts the input latent vector $z\in \mathcal{Z}$ to an intermediate latent space $w\in \mathcal{W}$:

$$\begin{array}{c}\hfill w=f\left(z\right).\end{array}$$

(3)

The synthesis network $g\left(w\right)$ then generates an image x from the intermediate latent space w:

$$\begin{array}{c}\hfill x=g\left(w\right).\end{array}$$

(4)

StyleGAN2 introduces an adaptive instance normalization (AdaIN) operation in the synthesis network, which applies learned style information from w to each feature map:

$$\begin{array}{c}\hfill \mathrm{AdaIN}(y_z,w)=\frac{y_z-\mu \left(y_z\right)}{\sigma \left(y_z\right)}·\sigma \left(w\right)+\mu \left(w\right).\end{array}$$

(5)

Here, $y_z$ is the feature map, $\mu (·)$ and $\sigma (·)$ denote the mean and standard deviation, respectively, and w is the style vector derived from the intermediate latent space.

The main innovation of Stylegan2-ADA is the use of Adaptive Discriminator Augmentation to improve GAN training with limited data. ADA applies random augmentations to the real and generated images before feeding them to the Discriminator. The augmentation strength is controlled by a hyperparameter p, which is adapted during training.

ADA introduces a new objective function for the Discriminator:

$$\begin{array}{c}\hfill \underset{G}{min}\underset{D}{max}{\mathbb{E}}_{x\sim p_{data}}\left[D\left(A_p\left(x\right)\right)\right]-{\mathbb{E}}_{z\sim p_z}\left[D\left(A_p\left(G\left(z\right)\right)\right)\right].\end{array}$$

(6)

Here, $A_p(·)$ represents the augmentation function with probability p. During training, the augmentation probability p is gradually increased if the Discriminator becomes too strong, ensuring that the Discriminator focuses on higher-level features instead of relying on the low-level details introduced by the augmentations. In summary, Stylegan2-ADA combines the advanced architecture of StyleGAN2 with Adaptive Discriminator Augmentation to generate high-quality synthetic images even with limited training data. The use of adaptive augmentations allows the model to maintain a balance between the Generator and Discriminator, improving the stability and performance of the training process.

3.5. Fréchet Inception Distance (FID)

FID measures the similarity between the true data distribution and the generated data distribution by comparing their statistics in a feature space. Given a pre-trained Inception network I, the feature representations for real samples $x_{real}$ and fake samples $x_{fake}$ are obtained as ${\mu }_{real}=I\left(x_{real}\right)$ and ${\mu }_{fake}=I\left(x_{fake}\right)$, respectively. The FID is then defined as:

$$\mathrm{FID}(\mathcal{X},G)=\left|\right|{\mu }_{real}-{\mu }_{fake}{\left|\right|}^2+\mathrm{Tr}({\Sigma }_{real}+{\Sigma }_{fake}-2{\left({\Sigma }_{real}{\Sigma }_{fake}\right)}^{1/2}),$$

(7)

where ${\mu }_{real}$ and ${\mu }_{fake}$ are the mean feature vectors, ${\Sigma }_{real}$ and ${\Sigma }_{fake}$ are the covariance matrices, and Tr denotes the trace of a matrix.

3.6. Inception Score (IS)

The Inception Score is another metric that evaluates the quality of generated samples by measuring both the diversity and realism of the samples. It is computed as:

$$\mathrm{IS}\left(G\right)=exp({\mathbb{E}}_{x_{fake}\sim G}[D_{KL}(p\left(y\right|x_{fake})\left|\right|p\left(y\right)\left)\right]),$$

(8)

where $D_{KL}\left(p\right|\left|q\right)$ denotes the KL divergence between probability distributions p and q, $p\left(y\right|x_{fake})$ represents the conditional class probability given a generated sample, and $p\left(y\right)$ is the marginal class probability.

FID has emerged as one of the most widely used and accepted metrics for evaluating the quality of GAN-generated images. Its extensive application in numerous studies has established its reputation as a reliable and effective metric. However, its computational complexity and time consumption, as studied in Section 7.1, primarily due to the use of the Inception Module as feature extractor, make it less than ideal for real-time assessment. This constraint can be a critical factor in applications where real-time performance is essential. By introducing the Signature Transform and Log-Signature as alternative methods for evaluating GAN convergence, we provide a new perspective on the problem, offering a powerful and efficient approach for capturing and comparing the features of empirical distributions generated by GANs.

4. Non-Parametric Statistical Analysis: Kruskal–Wallis

Kruskal–Wallis is a non-parametric statistical method used for comparing multiple independent samples to determine if they originate from the same population. This test is an extension of the Mann–Whitney U test for more than two groups and is particularly useful when the underlying assumptions of parametric tests, such as normality and homoscedasticity, are not met.

Kruskal–Wallis

In our methodology, we employ Kruskal–Wallis as a crucial component for assessing the goodness of fit of the GAN sample distribution. By comparing the generated samples with real data, we can evaluate the degree to which the generated samples resemble the target distribution. This non-parametric statistical test allows us to determine whether there are significant differences between the generated and real samples without making assumptions about the underlying distribution of the data. Using Kruskal–Wallis in our approach is beneficial because it provides an efficient and effective way to compare the generated samples with the target distribution while maintaining robustness to non-normality and unequal variances.

Given k independent samples with sizes $n_1,n_2,\dots ,n_k$, Kruskal–Wallis is based on the ranks of the combined data across all groups. The null hypothesis $H_0$ states that all samples are drawn from the same population, with the same distribution and median. The alternative hypothesis $H_1$ states that at least one sample is drawn from a different population with a distinct distribution or median. Kruskal–Wallis statistic, denoted as H, is computed as:

$$H=\frac{12}{N(N+1)}\sum _{o=1}^k\frac{R_o^2}{n_o}-3(N+1),$$

(9)

where $N={\sum }_{o=1}^k{n}_o$ is the total number of observations and $R_o$ is the sum of the ranks in the o-th group. Under the null hypothesis, the test statistic H follows a chi-square distribution with $k-1$ degrees of freedom, and the p-value can be computed accordingly. If the p-value is less than a predetermined significance level (e.g., 0.05), the null hypothesis is rejected, indicating that the samples are not from the same population.

Our decision to use this particular statistical test was based on several factors that make it a suitable choice for the analysis of GAN-generated images in the context of our study.

• Non-parametric nature: Kruskal–Wallis is a non-parametric test, meaning it does not rely on any assumptions about the underlying distribution of the data. This is particularly important when dealing with GAN-generated images, as the distributions of the generated samples may not necessarily follow a known parametric form, especially during the early stages of training. The non-parametric nature allows us to compare the goodness of fit between the generated and target distributions without making restrictive assumptions about their forms.
• Robustness: Kruskal–Wallis is robust against outliers and deviations from normality, which can be a common occurrence in the context of GAN-generated images. As the test is based on the ranks of the data rather than the raw values, it is less sensitive to extreme values that may arise from the generative process.
• Multiple group comparison: Kruskal–Wallis allows us to compare more than two groups simultaneously, which is useful when evaluating multiple GAN models or different categories within a dataset. This capability makes the test a versatile choice for our study, as it enables us to compare the performance of various GAN models on different datasets in a single analysis.
• Scalability: Kruskal–Wallis is computationally efficient, making it suitable for the large-scale datasets that are often encountered in GAN research. Its computational efficiency allows for the rapid evaluation of GAN-generated images and their convergence, which is a key advantage of our proposed methodology.

Moreover, an alternative such as the Friedman test could indeed be a suitable choice in cases where the observations are not independent; however, we have reasons to believe that even in these cases the Kruskal–Wallis H-test is still a good fit for our study. In our experiments, we have taken care to ensure that the generated samples from different GAN models are, in fact, independent. We achieve this by using different random seeds when sampling from the latent space of each GAN model, thus generating independent sets of synthetic images. By doing so, we maintain the independence assumption required by the Kruskal–Wallis H-test. Moreover, the Kruskal–Wallis H-test is a non-parametric test that compares the medians of multiple groups without making any distributional assumptions. This feature aligns well with our goal of evaluating GAN-generated samples, which often exhibit complex and unknown distributions. On the other hand, the Friedman test assumes that the observations are structured according to a block design, which may not be an accurate representation of our experimental setup. In summary, whereas the Friedman test could be a suitable alternative in certain scenarios, we believe that the Kruskal–Wallis H-test is more appropriate for our study, given the independence of our observations and the non-parametric nature of the test.

5. The Signature Transform

The Signature Transform [12,13], also known as the path signature, is a mathematical tool used to represent a sequence of data points or a path in a Euclidean space. The signature provides a unique and concise representation of the path while encoding its structural properties, making it suitable for various applications, such as ML and data analysis.

Given a continuous path $X:[0,T]\to {\mathbb{R}}^d$ in the Euclidean space ${\mathbb{R}}^d$, the Signature Transform $S\left(X\right)$ is a collection of iterated integrals of all orders:

$$S\left(X\right)=(1,S^1\left(X\right),S^2\left(X\right),\dots ,S^N\left(X\right)),$$

(10)

where $S^k\left(X\right)$ represents the k-th level of the signature and is a tensor in the tensor product space ${\left({\mathbb{R}}^d\right)}^{\otimes k}$, for $k=1,2,\dots ,N$. Each element of the k-th level tensor is defined as:

$$S_{z_1,\dots ,z_k}^k\left(X\right)={\int }_0^T{\int }_0^{s_1}\cdots {\int }_0^{s_{k-1}}d{X}_{z_1}\left(s_1\right)\dots d{X}_{z_k}\left(s_k\right),$$

(11)

where $s_1,s_2,\dots ,s_k\in [0,T]$ and $z_1,z_2,\dots ,z_k\in \{1,2,\dots ,d\}$.

The Log-Signature is a compressed representation of the signature that can be computed efficiently using Chen’s identity, which relates the Log-Signature to the signature through a shuffle product. The Log-Signature $L\left(X\right)$ is defined as:

$$L\left(X\right)=(L^1\left(X\right),L^2\left(X\right),\dots ,L^N\left(X\right)),$$

(12)

where $L^k\left(X\right)$ represents the k-th level of the Log-Signature and is a tensor in the tensor product space ${\left({\mathbb{R}}^d\right)}^{\otimes k}$, for $k=1,2,\dots ,N$. Each element of the k-th level tensor can be calculated using Chen’s identity:

$$L_{z_1,\dots ,z_k}^k\left(X\right)=S_{z_1,\dots ,z_k}^k\left(X\right)-\sum _{\pi \in P(z_1,\dots ,z_k)}{S}^{|{\pi }_1|}{\pi }_1\left(X\right)\otimes \cdots \otimes S^{|{\pi }_m|}{\pi }_m\left(X\right),$$

(13)

where $P(z_1,\dots ,z_k)$ denotes the set of all partitions of the index sequence $(z_1,\dots ,z_k)$, $|{\pi }_o|$ denotes the length of the o-th partition ${\pi }_o$, and ⊗ represents the tensor product.

The Signature Transform and Log-Signature can be used to capture and compare the features of empirical distributions generated by GANs, offering a powerful alternative to traditional measures of GAN convergence. The mathematical properties of these transforms provide a solid foundation for their use in various applications, such as the study of empirical distributions generated with GANs, as proposed in this paper.

6. Methodology

We focus on the problem of generating synthetic images with a limited amount of data, choosing Stylegan2-ADA [70] as the baseline method for our studies. The motivation behind this choice is twofold. First, Stylegan2-ADA has been specifically designed to address the challenges of data efficiency, providing high-quality image synthesis even with limited training data. This property makes it an ideal candidate for applications where large-scale datasets are not available or impractical to collect. Second, StyleGAN2-ADA demonstrates improved training stability and convergence properties compared to its predecessors, which contributes to reduced training time and computational resources. These factors are critical in real-world scenarios, where rapid model development and deployment are often essential. By using Stylegan2-ADA as our baseline, we aim to showcase the effectiveness of our proposed methods in the context of an advanced and widely-used generative model.

6.1. Statistical Analysis of the Generated Distribution

In this study, we perform a preliminary statistical analysis using Kruskal–Wallis [71] to evaluate the goodness of fit between the original and synthetic samples generated by GANs. We use the mean raster image intensities or gray-scale values as a simple image descriptor to capture rough texture information. Prior to conducting Kruskal–Wallis, we assess homoscedasticity using Levene’s test and normality of the distributions using a normality test, such as the Shapiro–Wilk test.

As a result of this preliminary analysis, we find that the original samples do not follow a normal distribution, whereas the synthetic samples do. This is consistent with the GAN architecture, which initially models the samples as White Gaussian and then modifies them to fit the original distribution. However, Kruskal–Wallis does not support the null hypothesis for goodness of fit, suggesting that a more sophisticated method for measuring sample quality in GANs is necessary. Existing measures such as MS-SSIM [37] and FID [8] are commonly used for this purpose. Despite its simplicity, the proposed non-parametric analysis can serve as a unit test for GANs and other variational methods after the model is trained, providing a quick assessment of the sample quality. This approach depicted in Figure 1 has not been extensively explored in the literature and offers a valuable contribution to the field.

Description and interpretation of statistical measures are provided in

Table 1. Interpretation of statistical measures given the proposed pipeline under study (Figure 1). The symbol ‘√’ means we accept the null hypothesis, while the symbol ‘x’ indicates we reject the null hypothesis.

Test	Population	Result	Interpretation
1	$C_1$ and $C_2$	√	(a)
		x	(b)
2	$C_2$	√	(c)
		x	(d)
3	$C_1$ and $C_2$	√	(e)
		x	(f)

:

(a)

Necessary condition but not sufficient to assert that both populations originate from the same distribution.

(b)

There is not enough statistical evidence to attest both populations’ samples originate from the same distribution.

(c)

With high probability the synthetic distribution generated is still close enough to the initial distribution of noise from the GAN architecture. The samples may not show enough fidelity, and there is probably bad generalization behavior.

(d)

The synthetic distribution is far from the initial distribution of noise and has deviated from the original Normal, and may be close to the target distribution.

(e)

If (a) then there is enough statistical evidence to confirm that both populations originate from the same distribution given this image descriptor. If (a) is not fulfilled, then we can only ascertain that the synthetic population is a good approximation.

(f)

There is not enough statistical evidence to attest both populations are from the same distribution.

In

Table 2. Evaluation of the statistical test measures of homoscedasticity (T1), normality (T2), and goodness of fit (T3) on AFHQ and MetFaces using state-of-the-art pretrained models of Stylegan2-ADA [70] and Stylegan3-ADA [43] and NASA Perseverance. The symbol ‘√’ means we accept the null hypothesis, while the symbol ‘x’ indicates we reject the null hypothesis. The best outcome for the proposed pipeline would be for Test 1 and Test 3 to yield positive results (accepting the null hypothesis), and for Test 2 to yield a negative result (rejecting the null hypothesis). However, an alternate good approximation would be when Test 1 and Test 2 yield negative results (rejecting the null hypothesis) and Test 3 yields a positive result (accepting the null hypothesis).

Model	Dataset		T1	T2	T3
Stylegan2-ADA	NASA Perseverance		x	√	x
	AFHQ	Cat	x	x	√
		Dog	x	√	x
		Wild	x	x	√
	MetFaces		x	x	x
r-Stylegan3-ADA			x	x	x
t-Stylegan3-ADA			x	x	x

, we present the evaluation test measures for homoscedasticity (T1), normality (T2), and goodness of fit (T3) on NASA Perseverance, AFHQ [72], and MetFaces [70] datasets. Based on the interpretation outlined in Table 1 and using the given image descriptor, we deduce that the Stylegan2-ada models trained on AFHQ Cat and Wild datasets provide excellent approximations of the original distributions, as the null hypothesis for goodness of fit is accepted. However, we cannot conclude that the distributions are identical since the equality of variances is not confirmed.

For the AFHQ Dog dataset, additional training is required as the null hypothesis for T2 (normality of the synthetic distribution) is accepted, indicating that the learned distribution is close to the original white noise. A similar conclusion applies to the model trained on the NASA Perseverance dataset, which also needs further training. In the case of MetFaces, the learned distribution is considerably different from the original white noise, but the null hypothesis for goodness of fit is not accepted. This finding suggests several possible interpretations: the model may be overfitting, it might require increased capacity to represent all features of the original distribution, or additional training might be needed.

We have introduced statistical measures and a visualization pipeline to examine and comprehend the data at hand. Nevertheless, the high-dimensional nature of images, coupled with the sequential aspect of video streams, brings forth a sense of time and space that our current analysis does not accommodate. In fact, the data comprise a series of images captured over a linear time span, following a specific trajectory. To address this aspect, we will employ tools from harmonic analysis in the subsequent section to offer a more comprehensive interpretation.

6.2. RMSE and MAE Signature and Log-Signature

The Signature Transform [10,73,74,75,76] is roughly equivalent to Fourier; instead of extracting information about frequency, it extracts information about order and area.

However, the Signature Transform differs from Fourier by the fact that it utilizes a basis of the space of functions of paths, a more general case to the basis of space of paths found in the preceding.

Following [10], the truncated signature of order N of the path $\mathbf{x}$ is defined as a collection of coordinate iterated integrals

$$\begin{array}{c}{\mathrm{S}}^N\left(\mathbf{x}\right)=\hfill \\ & {\left({\left(\underset{0<t_1<\cdots <t_a<1}{\int \cdots \int }\prod _{c=1}^a\frac{\mathrm{d}{f}_{z_c}}{\mathrm{d}t}\left(t_c\right)\mathrm{d}{t}_1\cdots \mathrm{d}{t}_a\right)}_{1\le z_1,\dots ,z_a\le d}\right)}_{1\le a\le N}.\hfill \end{array}$$

(14)

The Signature is a homomorphism from the monoid of paths into the grouplike elements of a closed tensor algebra; see Equation (16). It provides a graduated summary of the path $\mathbf{x}$. These extracted features of a path are at the center of the definition of a rough path [14]; they remove the necessity to take into account the inner detailed structure of the path.

$$\begin{array}{c}\hfill \mathrm{S}:\left\{f\in F|f:[x,y]\to E={\mathbb{R}}^d\right\}⟶T\left(E\right),\end{array}$$

(15)

$$\begin{array}{c}\hfill \mathrm{where}T\left(E\right)=T\left({\mathbb{R}}^d\right)=\prod _{c=0}^{\infty }{\left({\mathbb{R}}^d\right)}^{\otimes c}.\end{array}$$

(16)

It has many advantages over other tools of harmonic analysis for ML. It is a universal non-linearity, which means that every continuous function of the input stream may be approximated arbitrarily by a linear function of its signature. Furthermore, among other properties, it presents outstanding robustness behavior to missing or irregularly sampled data, along with optional invariance in terms of translation and sampling. It has recently been introduced in the context of DL to add some structure to the learning process, and it seems a promising tool in Generative Models and Reinforcement Learning, as well as a good theoretical framework. It mainly works on streams of data which could describe from video sequences to our entire life experiences. That is to say, under the correct assumptions and the right application, it could potentially compress all human experiences into a representation that could be stored and processed efficiently. Here, we propose to conduct a preliminary study in terms of harmonic analysis and understand its properties to compare the original and synthetic samples.

The Signature [11,14,77,78,79,80] of an input data stream encodes the order in which data arrive without being concerned with the precise timing of its arrival. This property, known as invariance to time reparameterizations [81], makes it an ideal candidate for measuring GAN-generated distributions against an original data stream. Notably, when sampling the GAN model, instances of the latent space are retrieved in no specific order, even though the original data are inherently time-dependent, as recorded video streams or images captured by sensors are constrained by the temporal nature of the physical world. However, GANs are not yet capable of generating data linearly in time and space, making comparisons using other methods potentially biased or unable to capture all relevant cues.

Furthermore, it is essential to note that the number of components in the truncated signature does not depend on the number of data samples under consideration. Specifically, it maps the infinite-dimensional space of data streams, $\mathcal{S}\left({\mathbb{R}}^d\right)$, into a finite-dimensional space of dimension $(d^{N+1}-1)/(d-1)$, where N corresponds to the order of the truncated signature. This characteristic makes the Signature Transform highly suitable for processing long sequential data with varying lengths or unevenly sampled data.

At the same time, we can introduce the concept of Log-Signature [75,76], which is a more compact representation than the Signature.

Definition 1.

If ${\gamma }_t\in E$ is a path segment and S is its Signature, then

$$\begin{array}{ccc}\hfill S& =& 1+S^1+S^2+\dots \forall c,S^c\in E^{\otimes c},\hfill \\ \hfill log\left(1+x\right)& =& x-x^2/2+\dots ,\hfill \\ \hfill logS& =& \left(S^1+S^2+\dots \right)-{\left(S^1+S^2+\dots \right)}^2/2+\dots \hfill \end{array}$$

The series $logS=\left(S^1+S^2+\dots \right)-{\left(S^1+S^2+\dots \right)}^2/2+\dots$ which is well-defined, is referred to as the Log-Signature of $\gamma .$

In practice, the Log-Signature calculation involves a series expansion that is typically truncated at a certain level to obtain a finite-dimensional representation. The choice of the truncation level depends on the specific application and the desired trade-off between computational complexity and the level of detail captured by the Log-Signature. In our experiments, we have chosen a truncation level that balances these considerations and yields satisfactory performance for our GAN evaluation task.

Unlike the Signature, the Log-Signature does not guarantee universality [14], and as a result, it needs to be combined with non-linear models for learning. However, it is empirically more robust to sparsely sampled data. There is a one-to-one correspondence between the Signature and the Log-Signature, as the logarithm map is bijective [13,75]. This statement also holds true for the truncated case up to the same degree.

In this study, we perform a comparison of the mean signature and Log-Signature for original and synthetic samples at a size of $64\times 64$. We observe that synthetic samples encompass the most relevant information from the original harmonic distribution. We compare against sets of 1000 and 5000 synthetic samples, with each instance considered a path $\mathbf{x}$ of dimension 64 to which we apply the Signature and Log-Signature transforms.

We propose the use of the element-wise mean of the truncated signatures ${\tilde{S}}^N$, depicted in Figure 2, to analyze the convergence of GAN-learned models by employing RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error). We refer to these measures as RMSE and MAE Signature, and RMSE and MAE Log-Signature. For instance, in Figure 3, we can observe that the model is achieving good convergence, though it is not capturing all the information present in the original distribution.

RMSE and MAE, when understood through the element-wise mean, can be considered as score functions built upon the Signature Transform, capable of measuring the quality of the generated distribution. This perspective on these measures is important for future applications, as it allows for the possibility of generalizing them to other tasks [11] or even applying them to other transforms. RMSE and MAE Signature and Log-Signature can serve multiple purposes, such as comparing models, monitoring performance during training across several epochs, and analytically detecting overfitting, as demonstrated in

Table 3. RMSE and MAE Signature and Log-Signature across several iterations of training of Stylegan2-ADA (lower is better, being the best results highlighted in bold). Our synthetic samples are generated using the model 798 which achieves the highest accuracy on RMSE and MAE Signature and Log-Signature.

Iteration Stylegan2-ADA	193	371	596	798	983
RMSE Signature	15,617	13,336	12,353	11,601	25,699
MAE Signature	11,072	10,686	9801	9086	19,481
RMSE Log-Signature	9882	7563	7354	7397	15,621
MAE Log-Signature	6467	5955	5724	5717	12,063

, whereas all these measures capture information about the visual cues present in the distributions, RMSE and MAE Signature, as well as MAE Log-Signature, prove to be more accurate in tracking the convergence of the GAN training procedure. In contrast, the RMSE Log-Signature exhibits less precision.

In Table 3, we present the RMSE and MAE Signature and Log-Signature values for different iterations of Stylegan2-ADA training. These values are calculated to evaluate the performance of the GAN at various stages of its training process. A closer examination of the table reveals that the 798th iteration of Stylegan2-ADA achieves the lowest RMSE and MAE Signature and Log-Signature values, which indicates the highest accuracy among the listed iterations. This table demonstrates the utility of RMSE and MAE Signature and Log-Signature metrics in tracking the progress of GAN training and identifying the optimal model iteration. By comparing the values across different iterations, we can observe the improvements in GAN performance as it learns to generate more realistic images. Furthermore, the table showcases the effectiveness of our proposed metrics in detecting potential overfitting, as evidenced by the increased RMSE and MAE values in the 983rd iteration. This increase in values suggests a decline in the GAN’s performance, likely due to overfitting the training data. In summary, Table 3 highlights the value of our proposed RMSE and MAE Signature and Log-Signature metrics in evaluating GAN performance, enabling us to monitor progress, compare different models, and detect overfitting during the training process.

To provide a more comprehensive understanding of the concepts presented in this section, we will analytically describe the abstraction of a set of images as an unevenly sampled stream of data, for example, a path, and present the definitions used to measure the similarity between image distributions.

A stream of data, $\mathbf{x}\in \mathcal{S}\left({\mathbb{R}}^d\right)$, can be understood as a discrete representation of a path.

Definition 2.

Let $\mathbf{x}=(x_1,\dots ,x_n)\in \mathcal{S}\left({\mathbb{R}}^d\right)$ be a stream of data. Let X be a linear interpolation of $\mathbf{x}$. Then the signature of $\mathbf{x}$ is defined as

$$\mathrm{S}\left(\mathbf{x}\right)=\mathrm{S}\left(X\right),$$

(17)

and the truncated signature of order N of $\mathbf{x}$ is defined as

$${\mathrm{S}}^N\left(\mathbf{x}\right)={\mathrm{S}}^N\left(X\right).$$

(18)

This definition of the signature of a stream of data is independent of the choice of linear interpolation of X by the invariance to time reparameterizations [10].

Definition 3.

Given a set of truncated signatures of order N, ${\left\{{\mathrm{S}}_c^N\left({\mathbf{x}}_c\right)\right\}}_{c=1}^m$, the element-wise mean is defined by

$${\tilde{\mathrm{S}}}^N\left(x^{\left(z\right)}\right)=\frac{1}{m}\sum _{c=1}^m{\mathrm{S}}_c^N\left(x_c^{\left(z\right)}\right),$$

(19)

where $z\in \{1,\dots ,n\}$ is the specific component index of the given signature.

Then, RMSE and MAE Signature, whose results are presented in Table 3 and

Table 4. RMSE and MAE Signature and Log-Signature evaluation and comparison on AFHQ and MetFaces using state-of-the-art pretrained models of Stylegan2-ADA [70] and Stylegan3-ADA [43]. Lower is better, being the best results highlighted in bold.

Model	Dataset		RMSE ${\tilde{\mathit{S}}}^3$	MAE ${\tilde{\mathit{S}}}^3$	RMSE $log{\tilde{\mathit{S}}}^3$	MAE $log{\tilde{\mathit{S}}}^3$
Stylegan2-ADA	AFHQ	Cat	61,450	45,968	29,201	22,297
		Dog	38,861	30,441	31,686	24,612
		Wild	33,306	25,578	26,622	20,359
	MetFaces		33,247	23,428	25,685	18,071
r-Stylegan3-ADA			34,977	22,799	24,707	16,539
t-Stylegan3-ADA			30,894	19,872	21,560	13,761

, can be defined as follows.

Definition 4.

Given n components of the element-wise mean of the signatures ${\left\{{\tilde{y}}^{\left(c\right)}\right\}}_{c=1}^n\subseteq T\left({\mathbb{R}}^d\right)$ from the model chosen as a source of synthetic samples and the same number of components of the element-wise mean of the signatures ${\left\{{\tilde{x}}^{\left(c\right)}\right\}}_{c=1}^n\subseteq T\left({\mathbb{R}}^d\right)$ from the original distribution, we define the Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) by

$$\mathrm{RMSE}\left({\left\{{\tilde{x}}^{\left(c\right)}\right\}}_{c=1}^n,{\left\{{\tilde{y}}^{\left(c\right)}\right\}}_{c=1}^n\right)=\sqrt{\frac{1}{n}\sum _{c=1}^n{\left({\tilde{y}}^{\left(c\right)}-{\tilde{x}}^{\left(c\right)}\right)}^2},$$

(20)

and

$$\mathrm{MAE}\left({\left\{{\tilde{x}}^{\left(c\right)}\right\}}_{c=1}^n,{\left\{{\tilde{y}}^{\left(c\right)}\right\}}_{c=1}^n\right)=\frac{1}{n}\sum _{c=1}^n|{\tilde{y}}^{\left(c\right)}-{\tilde{x}}^{\left(c\right)}|.$$

(21)

The case for Log-Signature is analogous.

7. Evaluation

We present the results of our proposed measures using several state-of-the-art pretrained models in Table 4. For evaluation and testing, we use the standard AFHQ dataset [72] classes ‘cat’, ‘dog’, and ‘wild’, as well as MetFaces [70], in conjunction with their corresponding pretrained models. To compute the RMSE and MAE for ${\tilde{S}}^N$ and $log{\tilde{S}}^N$, we generate 1000 synthetic samples from each model and compare them against the full original dataset. Prior to the Signature Transform, the samples are converted to grayscale and resized to $64\times 64$. Figure 4 and Figure 5 provide a visual comparison of the spectrum, demonstrating that the trained models effectively learn the empirical distribution of the original data.

The AFHQ dataset comprises high-quality images of animal faces, which are divided into three distinct classes: cats, dogs, and wildlife. This dataset provides a challenging evaluation scenario due to the inherent differences between the classes and the detailed textures present in the animal faces. MetFaces, on the other hand, is a collection of face images derived from various art pieces, including paintings, photographs, and sculptures. It showcases a diverse range of artistic styles, time periods, and image content, making it an ideal dataset to assess the performance of our proposed metrics on more complex and varied data distributions. By evaluating our method on both AFHQ and MetFaces, we aim to demonstrate the adaptability and robustness of our approach across different scenarios and data complexities.

In Table 4, we compare the recently developed models $r,t$-Stylegan3-ADA [43] against Stylegan2-ADA using MetFaces. We observe that t-Stylegan3-ADA significantly outperforms Stylegan2-ADA and r-Stylegan3-ADA, which is consistent with the FID results reported in [43], as shown in

Table 5. Evaluation and comparison of FID (as reported in [43]) and RMSE ${\tilde{S}}^3$ on MetFaces. Lower is better, being the best results highlighted in bold.

Model	FID	RMSE ${\tilde{\mathit{S}}}^3$
Stylegan2-ADA	15.22	33,247
r-Stylegan3-ADA	15.33	34,977
t-Stylegan3-ADA	15.11	30,894

. Here, we can see that FID closely resembles the behavior of RMSE ${\tilde{S}}^3$. Nonetheless, our metrics are both effective and efficient. A visual comparison of the spectrum of the Signatures for the given dataset can be seen in Figure 5. Computation is performed on the CPU in seconds, which is orders of magnitude faster and requires fewer resources than FID or MS-SSIM.

Table 4 presents the RMSE and MAE Signature and Log-Signature evaluation results for different GAN models and datasets, including AFHQ and MetFaces. The table showcases a comparison of state-of-the-art pretrained models: Stylegan2-ADA [70], r-Stylegan3-ADA, and t-Stylegan3-ADA [43]. The goal of this comparison is to highlight the performance differences between these models using the proposed metrics. A close inspection of the table reveals that the t-Stylegan3-ADA model consistently achieves the lowest RMSE and MAE Signature and Log-Signature values across all datasets, indicating superior performance in generating synthetic samples that closely resemble the original distributions. This result demonstrates the effectiveness of the t-Stylegan3-ADA model in learning the intricacies of the underlying data distributions and generating high-quality synthetic samples. Additionally, the table illustrates the performance variations between different categories within the AFHQ dataset, with AFHQ Cat and Wild categories having a closer resemblance to the original distributions than AFHQ Dog. This observation aligns with the qualitative assessment of the generated samples visualized in Figure 6 and Figure 7, providing further evidence of the accuracy of our proposed metrics in capturing the characteristics of the generated samples. That is, Table 4 highlights the utility of the RMSE and MAE Signature and Log-Signature metrics in evaluating and comparing the performance of different GAN models across various datasets. By analyzing these metrics, we can gain insights into the quality of the generated samples and their similarity to the original distributions, as well as assess the effectiveness of the GAN models in capturing the essential features of the data.

Table 5 presents a comparison of the FID and RMSE ${\tilde{S}}^3$ metrics on MetFaces for three GAN models: Stylegan2-ADA, r-Stylegan3-ADA, and t-Stylegan3-ADA. The aim of this comparison is to highlight the relationship between the two evaluation metrics and demonstrate the efficacy of RMSE ${\tilde{S}}^3$ in capturing the performance differences among these models. As observed in the table, the FID scores and RMSE ${\tilde{S}}^3$ values show a similar trend, with t-Stylegan3-ADA achieving the best performance in both metrics. This consistency between the two evaluation metrics suggests that RMSE ${\tilde{S}}^3$ can serve as a reliable alternative to FID in assessing GAN performance. Moreover, the lower RMSE ${\tilde{S}}^3$ values for t-Stylegan3-ADA indicate that the model generates synthetic samples that are closer to the original distribution compared to the other two models. Notably, the proposed RMSE ${\tilde{S}}^3$ metric offers significant advantages over FID in terms of computational efficiency and resource requirements. As mentioned in the text, the RMSE ${\tilde{S}}^3$ computations are performed on the CPU in seconds, making it substantially faster and less resource-intensive than FID or MS-SSIM. This efficiency makes the proposed metric more suitable for practical applications, where rapid evaluation and limited resources may be critical factors. In summary, Table 5 demonstrates the effectiveness of the RMSE ${\tilde{S}}^3$ metric as an alternative to FID for evaluating GAN performance. The strong correlation between the two metrics, coupled with the computational advantages of RMSE ${\tilde{S}}^3$, showcases its potential as a valuable tool for assessing the quality of synthetic samples generated by various GAN models.

7.1. Computational Complexity

In this subsection, we elaborate on the computational complexity and time estimates for the element-wise mean of the Signatures and Kruskal–Wallis in comparison to FID, MS-SSIM, LPIPS, and PSNR.

• Element-wise mean of the Signatures: The computation of the Signature Transform has a time complexity of $\mathcal{O}\left(L{M}^2\right)$, where L is the length of the path and M is the order of the signature. However, since we are computing the element-wise mean of the Signatures, the complexity becomes $\mathcal{O}\left(NL{M}^2\right)$, where N is the number of samples. In practice, the Signature Transform can be efficiently computed using recursive algorithms, which keeps the computational cost low.
• Kruskal–Wallis has a time complexity of $\mathcal{O}(NlogN)$ for sorting the samples, followed by $\mathcal{O}\left(N\right)$ for computing the test statistic, resulting in an overall complexity of $\mathcal{O}(NlogN)$. This complexity is relatively low, especially when compared to more computationally demanding metrics such as FID and MS-SSIM.

In comparison:

• The FID calculation involves computing the Inception features for each sample, which requires a forward pass through a deep neural network, followed by computing the mean and covariance of these features. The complexity of the forward pass depends on the architecture of the Inception network, but it is generally much higher than the complexity of the Signature Transform and the Kruskal–Wallis. Additionally, FID requires GPU resources to perform these calculations efficiently, further increasing its computational cost.
• MS-SSIM involves computing the structural similarity index at multiple scales, which requires computing the mean, variance, and covariance for each scale. The complexity of MS-SSIM is $\mathcal{O}\left(NWH\right)$, where W and H are the width and height of the images, respectively, whereas this complexity is not as high as FID, it is still higher than the complexities of the proposed methods.
• LPIPS metric computes the distance between image features extracted from a pretrained deep neural network (e.g., AlexNet or VGG). The complexity of LPIPS is primarily determined by the forward pass through the chosen deep neural network. The complexity of the forward pass depends on the architecture of the network, but in general, it is higher than the complexity of the Signature Transform and the Kruskal–Wallis. Similar to FID, LPIPS also typically requires GPU resources for efficient computation.
• PSNR is a simple and widely used metric for image quality assessment. It is computed as the ratio between the maximum possible power of a signal and the power of corrupting noise that affects the fidelity of its representation. The complexity of PSNR is $\mathcal{O}\left(NWH\right)$, where W and H are the width and height of the images, respectively. Although the complexity of PSNR is similar to that of MS-SSIM, it is still higher than the complexities of the proposed methods (element-wise mean of the Signatures and Kruskal–Wallis).

To summarize, our proposed methods (element-wise mean of the Signatures and Kruskal–Wallis) have significantly lower computational complexity than FID, MS-SSIM, LPIPS, and PSNR, allowing for faster computation and reduced resource usage. Based on the complexity analysis, we can estimate that our methods can be computed on the CPU in seconds, whereas FID, MS-SSIM, LPIPS, and PSNR require more time and resources, particularly when GPUs are not available.

7.2. Visualization

In our study, we first apply PCA to reduce the dimensionality of the data, which helps us to retain the global structure of the dataset. Then, we use t-SNE to visualize the data in a lower-dimensional space, which emphasizes the local differences between samples. This two-step approach allows us to capture both the global and local structures within the data, providing a richer visualization of the generated GAN images compared to using PCA alone.

In Figure 6 and Figure 7, we visualize the sets of images of AFHQ and MetFaces, both original and synthetic, used in the evaluations in Table 2 and Table 4 using PCA Adaptive t-SNE. The importance here is to observe the overall distribution of the samples, which is well captured by our proposed method. For instance, we can observe that the synthetic samples of AFHQ Cat and Wild closely resemble the original distribution in terms of variability and quality. In contrast, AFHQ Dog demonstrates less variability but still achieves high-quality samples, which aligns with the analytical interpretation of the proposed statistical measures shown in Table 2.

In Figure 7, we can observe that the synthetic samples generated with t-Stylegan3-ADA exhibit better quality than those produced by Stylegan2-ADA and r-Stylegan3-ADA, and the model is evidently learning the original distribution. Nonetheless, there is potential for improvement in terms of variability and scope. These observations are consistent with the RMSE and MAE Signature and Log-Signature results, as shown in Table 4.

That being said, our proposed method relies on the Signature Transform and Log-Signature to evaluate GAN-generated samples, which are independent of PCA and t-SNE. The use of PCA and t-SNE in our study is only to provide a visual representation of the original and synthetic distributions, allowing us to better understand and interpret the quality of the generated images. The sample size and analysis time of the generated GAN images are not affected by the application of PCA and t-SNE for visualization. Our methodology remains efficient and effective in assessing the quality of GAN-generated samples without the need to reduce the dimensionality of the images for the actual evaluation process.

8. Conclusions

GAN evaluation has been one of the central research efforts of the community of computer vision during these last years. The ability of the networks to generate high-fidelity samples has inspired researchers all over the world to work on the topic. However, although many variants of the original successful DCGAN architecture are able to generate very realistic samples, neither the advance in proposing metrics to assess the imagery has been effectual, nor the ability of the metrics to guarantee some level of robustness, and overall description of the resultant distribution. The best effort of them being FID suffers from high-computation time and use of GPU resources; it depends mainly on an inception module that extracts features from lots of samples rather than from analytical measures that quantify properly their characteristics.

We are the first to propose the use of the Signature Transform to assess GAN convergence by introducing RMSE and MAE Signature and Log-Signature. The measures are reliable, consistent, efficient, and easy to compute. Additionally, an effective methodology to test the goodness-of-fit according to the original distribution by the use of simple statistical methods is also proposed, being the first to be able to reduce the amount of computation for accurate GAN Synthetic image quality assessment to the order of seconds. Worth mentioning is the proposal of a taxonomical pipeline to systematically assess the resultant distributions using a non-parametric test. Lastly, we also introduce an adaptive technique based on t-SNE and PCA that, without the need for hyperparameter tuning, puts forward exceptional visualization capabilities.

Future work that could be pursued under these assumptions, among others, is to increase the complexity of the descriptor, extend the proposed score functions on top of the Signature Transform to be used in other tasks or use the metrics inside the training loop to assess convergence and help the networks train faster.

In this study, we presented a novel approach to assess GAN convergence and goodness of fit using the Signature Transform, whereas our methodology provides significant advantages over existing methods, we acknowledge the following limitations:

• The proposed RMSE and MAE Signature and Log-Signature metrics are based on the Signature Transform, which inherently captures information about the underlying distribution. However, these metrics may not be sensitive to certain aspects of the generated images, such as fine-grained details or specific structures, which could be essential for certain applications.
• Although our proposed method significantly reduces computation time and resource usage compared to existing GAN evaluation methods, it might still be computationally expensive for extremely large datasets or high-resolution images. Further optimization of the computation process may be necessary to address these challenges.
• The evaluation of GAN performance based on our proposed metrics assumes that the original and synthetic image distributions are stationary. In cases where the data exhibit non-stationary behavior, the effectiveness of our approach might be compromised, and additional methods or adaptations may be required.
• The goodness-of-fit methodology proposed in this study relies on statistical methods, which might not always provide definitive conclusions on the quality of the generated samples. In some cases, additional qualitative assessments or domain-specific evaluations may be necessary to obtain a comprehensive understanding of the GAN’s performance.

In conclusion, despite these limitations, our study introduces a promising and efficient approach to assess GAN convergence and goodness of fit using the Signature Transform. Future work may involve addressing these limitations and further exploring the potential of our proposed metrics in other applications and tasks.