Prospects of Structural Similarity Index for Medical Image Analysis

Abstract

An image quality matrix provides a significant principle for objectively observing an image based on an alteration between the original and distorted images. During the past two decades, a novel universal image quality assessment has been developed with the ability of adaptation with human visual perception for measuring the difference of a degraded image from the reference image, namely a structural similarity index. Structural similarity has since been widely used in various sectors, including medical image evaluation. Although numerous studies have reported the use of structural similarity as an evaluation strategy for computer-based medical images, reviews on the prospects of using structural similarity for medical imaging applications have been rare. This paper presents previous studies implementing structural similarity in analyzing medical images from various imaging modalities. In addition, this review describes structural similarity from the perspective of a family’s historical background, as well as progress made from the original to the recent structural similarity, and its strengths and drawbacks. Additionally, potential research directions in applying such similarities related to medical image analyses are described. This review will be beneficial in guiding researchers toward the discovery of potential medical image examination methods that can be improved through structural similarity index.

1. Introduction

An image quality assessment (IQA) plays a crucial role in accurately measuring a degraded image from a reference image. In general, there are two types of IQA used to evaluate the quality of an image, i.e., subjective and objective [1,2,3]. A subjective measure [4] involves individuals (mostly groups of experts) inspecting an image, and then conducting an evaluation according to their specialties. This measure is considered the best strategy because it offers consistency when assessing the images. However, to reach a reliable conclusion after image measurements, a subjective measure is often inconvenient, time-consuming, and expensive. This is natural because the involvement of human beings is directly connected to their ability, knowledge, and insight. For instance, to analyze a medical image, a medical doctor specializing in radiology is necessary. Renieblas et al. reported inspecting bone plain films, magnetic resonance, and chest plain films, selecting four medical doctors with diagnostic experiences in measuring such images to participate [5]. By realizing the shortcomings of a subjective method, several attractive measurements related to objectively evaluating such images have been developed during the past few decades. The most popular image quality measures are the mean square error (MSE), signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), and absolute error (AE), as well as a derivation of such measures, such as Laplacian MSE (LMSE), peak MSE (PMSE), normalized MSE (NMSE), PSNR, and NAE [6,7]. Although such measures are accepted as a universal image quality index, they provide less sensitivity compared with a human visual system (HVS) [8]. They are considered common measurements because they can be calculated easily and may interpret the physical meaning of an image [9,10]. In addition, contrast-and-size detail (CSD) was developed to separate the visible and invisible inclusions in mimicking the breast environment through simulations by embedding a numerical tumor. In this case, an inclusion represents a breast tumor [11,12]. Again, this image quality metric (IQM) has issues in terms of inconsistency when increasing the contrast ratio with a raised inclusion size and an ambiguous threshold value. Therefore, it is challenging to implement this method and distinguish seeable and unseeable inclusions inside the breast tissue with a provided threshold value.

In early 2000, Wang et al. developed a new universal IQA and tried to replace conventional methods such as MSE and PSNR for measuring the quality of the images, namely the structural similarity (SSIM) index, and adapt to the HVS [10,13]. Their first attempt regarding the possibility of substituting traditional strategies with an applicable metric to measure various images from numerous sectors was reported in 2002. It was reported that the IQA calculates the distortions in a combination of the loss of correlation, luminance distortion, and contrast distortion [14]. In addition, their results indicate that the novel universal index is more exceptional than MSE because the new index measures the information loss and is not focused on the energy loss. This is reasonable because the MSE values of two different distorted images can be the same, although one image is more flawless than another image. Their follow-up study complementing their previous research was published in 2004, and one of the most popular IQMs in this era, i.e., SSIM, was described [10]. They proposed a novel philosophy by considering that image degradation is the perceived changes in structural information, whereas error sensitivity is an estimation of the perceived errors to assess a noised image in comparison with the original. This new philosophy is easy to understand because the human perceptual measure is comfortable quantifying the changes in structural information when two images are compared, and it is more complicated to indicate the error. Moreover, the novel metric suggests the IQA by considering three factors, i.e., luminance, contrast, and structure comparisons. In addition, they suggested that SSIM may be used for several applications [15] other than image processing because SSIM quantifies two signals and compares them to obtain the similarity score, regardless of the complexity in calculating the SSIM when compared to that of the MSE. Research on a single mean SSIM (MSSIM) motivated several further developments of SSIM, and to date, numerous versions of SSIM have been achieved, for example, multiscale SSIM (MS-SSIM), gradient-based SSIM (GSSIM), a three-component weighting region, a four-component weighting region, a complex-wavelet, and an improved SSIM with a sharpness comparison (ISSIM-S) [16,17,18,19,20,21,22]. The SSIM method has recently become popular as a way to improve the sensitivity according to the measurement scope and goal by applying an image processing procedure [23]. Several publications have even reported SSIM implementation in clinical applications and biomedical fields [24,25,26,27,28].

SSIM has shown signs of progress, not only in digital images for communication, video, monitor, television, and watermark technologies [29,30,31,32,33,34,35] but also in medical image analyses [36,37,38,39,40,41,42,43,44,45] to assist clinicians or physicians in complementing an opinion before making a final decision [46,47,48,49]. SSIM can be considered a “second opinion” in an assessment. By understanding the recent progression of SSIM related to medical image quantification, this study reviewed articles concentrating on the SSIM implementation as an objective measure used to evaluate medical images from several modalities, such as magnetic resonance imaging (MRI), ultrasound (US), computerized tomography (CT) scans, X-rays, and optical imaging, as well as other implementations in the medical field. Moreover, we discuss the history and popular progress of SSIM from its origin to recent structural similarities, its strengths and shortcomings, and its potential future research directions in relation to medical image analyses. This review is expected to be a guide for researchers in identifying the potential application of SSIM when objectively measuring medical images.

The remainder of this study is organized as follows. Section 2 describes the history and basic principles of SSIM, and Section 3 describes the types of improvements made to this index. Section 4 presents the use of SSIM in medical imaging, whereas Section 5 presents some final concluding remarks by providing the future prospects of SSIM for medical image analyses.

2. Historical Review and Basic Principles of SSIM

Quantifying an image objectively to acquire quality statistics is a crucial task in an image processing procedure because it can provide the feature and property information of the image; thus, several attempts at developing a computer-based observer have been conducted by researchers. Nevertheless, creating a reliable algorithm for measuring an image is challenging and is concerned with the HVS because humans are the end-users of the images. For example, in terms of video communication, a perfect IQM can be deployed as a benchmark for measuring other IQMs when assessing a particular task. We can select the best IQM algorithm based on performance [33,35,50,51]. Moreover, in the field of medical image analysis, with the assistance of computer vision, clinicians can improve their confidence when diagnosing patients. This becomes more vital if the task is related to human disease diagnosis [52,53,54,55,56].

Two traditional quality metrics, MSE and PSNR, are widely used to evaluate images because they are able to provide a physical meaning and are relatively simple in terms of their calculation. However, such quality measures are frequently inconsistent with the HVS because they can provide the same value of quality for two completely different distorted images, even when one image is more perceivable than another [6,7]. The performances of common IQAs were shown by Eskicioglu and Fisher [9] in 1995, inspiring Wang and Bovik [14] to develop a novel universal IQM to overcome the MSE and PSNR incompatibility in 2002. At the time, MSE and SNR, along with their differentiations considered, were incompatible with HVS, particularly when employing a specific condition directed at an image with a particular level of degradation.

This first attempt in developing a new universal quality index can be utilized not only in a two-dimensional image processing system but also in other areas, such as speech and pattern recognitions relative to a one-dimensional analysis, because the new universal quality metric offers comparisons between two signals. These two signals refer to one signal as a reference and the other acting as the original signal with implemented noise. Using these two signals, we can calculate the signal quality quantitatively. Therefore, their study was recognized as a full-reference (FR) [57] IQA when considering that the model of the image distortion is influenced by three aspects, i.e., correlation loss, luminance distortion, and contrast distortion. A description of the developed novel universal quality index is as follows:

$$Q=\frac{4{\sigma }_{xy}\overline{x}\overline{y}}{\left({\sigma }_x^2+{\sigma }_y^2\right)+\left({\overline{x}}^2+{\overline{y}}^2\right)} ,$$

(1)

where $x=\left\{x_i|i=1, 2, 3, \dots , N\right\}$ is the original image, and $y=\left\{y_i|i=1, 2, 3, \dots , N\right\}$ denotes the image under test, with $\overline{x}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N$}\right.{\displaystyle {{\displaystyle \sum }}_{i=1}^N}{x}_i$ and $\overline{y}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N$}\right.{\displaystyle {{\displaystyle \sum }}_{i=1}^N}{y}_i$ as the average grayscale level (luminance) for the original and test images, respectively. In addition, ${\sigma }_x^2=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N-1$}\right.{\displaystyle {{\displaystyle \sum }}_{i=1}^N}{\left(x_i-\overline{x}\right)}^2$ and ${\sigma }_y^2=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N-1$}\right.{\displaystyle {{\displaystyle \sum }}_{i=1}^N}{\left(y_i-\overline{y}\right)}^2$ are squares of the standard deviation for the original and test images, and ${\sigma }_{xy}=\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$N-1$}\right.{\displaystyle {{\displaystyle \sum }}_{i=1}^N}\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)$ refers to the covariance between the original and test images. Moreover, the quality score $Q$ is within the range of $-1$ to $1$; however, in most cases, $Q$ is from $0$ to $1$ with $0$ representing a non-similarity and $1$ demonstrating a perfect match between the reference and noised images. To simplify Equation (1) into three important components, $Q$ can be defined as

$$Q=\frac{{\sigma }_{xy}}{{\sigma }_x{\sigma }_y}·\frac{2\overline{x}\overline{y}}{{\overline{x}}^2+{\overline{y}}^2}·\frac{2{\sigma }_x{\sigma }_y}{{\sigma }_x^2+{\sigma }_y^2} ,$$

(2)

Because an image is space-variant, for measuring an image, a local assessment is preferred over a global evaluation that quantifies the image by employing a sliding window. This window slides over the entire image from the left-top to right-bottom corners pixel by pixel both horizontally and vertically. During each stride, we obtain $Q_j$, and thus if the window slides over the image for $M$ strides, we acquire the total quality $Q_{tot}={\displaystyle {{\displaystyle \sum }}_{j=1}^M}{Q}_j$, and $Q$ can be written as

$$Q=\frac{1}{M}{\displaystyle \sum }_{j=1}^M{Q}_j,$$

(3)

indicating the mean quality score.

Furthermore, Wang and Bovik indicated that the new quality metric is superior to the MSE. They employed the same MSE value with various distortions to the “Lena” image. They set the MSE value to approximately $255$; however, the $Q$ score could validate the quality by showing the different scores with respect to the perception of a human observer. However, in this first effort, they did not claim to use any HVS models.

Two years later, in 2004, they published their study on SSIM with the help of two additional co-authors [10]. They reported that the assumption of HVS can be well adapted with the perception of the structural information and that human observers have limitations in recognizing errors. This is reasonable because humans can easily identify changes in physical information while complicatedly detecting the variations of an error in the images. To match with the HVS, SSIM demonstrates comparisons of $l$ luminance, $c$ contrast, and $s$ structure, which are specified as

$$l\left(\mathit{x},\mathit{y}\right)=\frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1} ,$$

(4)

$$c\left(\mathit{x},\mathit{y}\right)=\frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2} ,$$

(5)

$$s\left(\mathit{x},\mathit{y}\right)=\frac{{\sigma }_{xy}+C_3}{{\sigma }_x{\sigma }_y+C_3} ,$$

(6)

where ${\mu }_x$ and ${\mu }_y$ denote the mean intensity for reference image $\mathit{x}$ and distorted image $\mathit{y}$, respectively. Compared with Equation (2), Equations (4) and (5) have the same definitions as luminance and contrast comparisons, and meanwhile, Equation (6) has a different description from the correlation to be applied in a structural comparison. Likewise, SSIM considers constant values to avoid instability when ${\mu }_x^2+{\mu }_y^2$, ${\sigma }_x^2+{\sigma }_y^2$, and ${\sigma }_x{\sigma }_y$ are extremely close to zero. These constants are $C_1={\left(K_{1}L\right)}^2$, $C_2={\left(K_{2}L\right)}^2$, and $C_3=\raisebox{1ex}{$C_2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$. In addition, $K_1$ and $K_2$ should be $\ll 1$ and $L$ is $255$ for an $8$-bit grayscale image or an image in three channels, such as red, green, and blue (RGB). As in Equation (2), the SSIM also can be formulated as

$$SSIM\left(\mathit{x},\mathit{y}\right)={\left[l\left(\mathit{x},\mathit{y}\right)\right]}^{\alpha }·{\left[c\left(\mathit{x},\mathit{y}\right)\right]}^{\beta }·{\left[s\left(\mathit{x},\mathit{y}\right)\right]}^{\gamma },$$

(7)

with $\alpha =\beta =\gamma =1$, and thus a specific form can be defined as

$$SSIM\left(\mathit{x},\mathit{y}\right)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{xy}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_x^2+{\sigma }_y^2+C_2\right)} .$$

(8)

Figure 1 shows a diagram of the SSIM measurement procedure. First, the luminance is calculated over the two images by utilizing Equation (4) when employing a sliding window; here, image $\mathit{x}$ is a reference image, whereas image $\mathit{y}$ denotes a noise-degraded image. The contrast is then measured using Equation (5). To obtain the structure, the covariance between $\mathit{x}$ and $\mathit{y}$ must be computed using Equation (6). Once these three factors have been obtained, the combination of the comparisons, as indicated in Equation (7), shows a quality score within the range of $-1$ to $1$ because of the structural influence. However, in various cases, the score is between $0$ and $1$. Therefore, the SSIM satisfies the following conditions:

• Symmetry: $SSIM\left(\mathit{x},\mathit{y}\right)=SSIM\left(\mathit{y},\mathit{x}\right);$
• Boundedness: $SSIM\left(\mathit{x},\mathit{y}\right)\le 1;$
• Unique maximum: $SSIM\left(\mathit{x},\mathit{y}\right)=1$ if and only if $\mathit{x}=\mathit{y}$.

As in previous studies regarding the universal quality index (UQI) [14], SSIM is also effective in inspecting the image locally by implementing a sliding window. Hence, according to the original article on SSIM [10], the sliding window was $11\times 11$. In addition, there was an improvement in the sliding window by applying a Gaussian weighting function $\mathit{w}=\left\{w_i|i=1, 2, 3, \dots , N\right\}$ with a standard deviation of $1.5$. The value of $\mathit{w}$ should fulfill a unit sum of ${\displaystyle {{\displaystyle \sum }}_{i=1}^N}{w}_i=1$. Because of this Gaussian weighting function with a $11\times 11$ local window, the local statistics, such as ${\mu }_x$, ${\mu }_y$, ${\sigma }_x$, ${\sigma }_y$, and ${\sigma }_{xy}$, have the following adjustments:

$${\mu }_x=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{w}_i{x}_i,$$

(9)

$${\mu }_y=\frac{1}{N}{\displaystyle \sum }_{i=1}^N{w}_i{y}_i,$$

(10)

$${\sigma }_x={\left({\displaystyle \sum }_{i=1}^N{w}_i{\left(x_i-{\mu }_x\right)}^2\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.},$$

(11)

$${\sigma }_y={\left({\displaystyle \sum }_{i=1}^N{w}_i{\left(y_i-{\mu }_y\right)}^2\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.},$$

(12)

$${\sigma }_{xy}={\displaystyle \sum }_{i=1}^N{w}_i\left(x_i-{\mu }_x\right)\left(y_i-{\mu }_y\right),$$

(13)

where $K_1$ and $K_2$ are $0.01$ and $0.03$, respectively. When completing the SSIM computation over the entire image using a local window, the mean SSIM (MSSIM) can be obtained as

$$MSSIM\left(\mathit{X},\mathit{Y}\right)=\frac{1}{M}{\displaystyle \sum }_{j=1}^{M}SSIM\left({\mathit{x}}_j,{\mathit{y}}_j\right),$$

(14)

where $\mathit{X}$ and $\mathit{Y}$ denote the reference image and the image under testing, respectively, whereas ${\mathit{x}}_j$ and ${\mathit{y}}_j$ are the images at the $j$-th window when the local window slides over the original and distorted images, and $M$ is the number of local windows in the image.

MSSIM showed consistency when compared with human observers. In addition, these results were confirmed by implementing the PSNR as an evaluation tool to measure other IQAs, which, in this case, is MSSIM. Further, the previous UQI with $K_1$ and $K_2$is $0$, which presents the smallest correlation with the human observers. These results indicate that MSSIM can improve the UQI ability in avoiding a zero in the denominator by setting $K_1$ and $K_2$ as $\ll 1$. Hence, SSIM has become popular as an objective investigative tool for other fields included in medical image analysis [58,59,60,61,62,63,64,65,66,67].

To overcome the weakness of a single-scale SSIM associated with a limitation of view, a multi-scale SSIM (MS-SSIM) was established [22]. The viewing conditions are incorporated with the display resolution, distance when reading the image, luminance background, and other set environments that can affect the image investigation results. Figure 2 shows the procedure used by MS-SSIM in evaluating an image. First, images $\mathit{x}$ and $\mathit{y}$ are processed inputs, as in a single-scale SSIM for a scale of $1$. In this process, we only store the contrast $c$ and structure $s$ for such a scale, namely, $c_1$ and $s_1$. Then, the reference and noised images are filtered using an LPF followed by downsampling by $2$. In this step, again, the downsampled images are computed using single-scale SSIM formulas to obtain $c_2$ and $s_2$. This procedure is repeated until $K$ iterations. Once iteration $K$ is completed, we save all three parameters, $c_K$, $s_K$, and luminance $l_K$. Thus, the MS-SSIM is formulated as

$$MS-SSIM\left(\mathit{x},\mathit{y}\right)={\left[l_K\left(\mathit{x},\mathit{y}\right)\right]}^{{\alpha }_K}{\displaystyle \prod }_{k=1}^K{\left[c_k\left(\mathit{x},\mathit{y}\right)\right]}^{{\beta }_k}·{\left[s_k\left(\mathit{x},\mathit{y}\right)\right]}^{{\gamma }_k}.$$

(15)

Here, ${\alpha }_K$, ${\beta }_k$, and ${\gamma }_k$ accommodate the comparative importance of the three components. In addition, for simplification, because ${\alpha }_k={\beta }_k={\gamma }_k$, thus ${\displaystyle {{\displaystyle \sum }}_{k=1}^K}{\alpha }_k={\displaystyle {{\displaystyle \sum }}_{k=1}^K}{\beta }_k={\displaystyle {{\displaystyle \sum }}_{k=1}^K}{\gamma }_k=1$ when the normalization of the cross-scale setting is established. The genuine MS-SSIM sets $K=5$ with ${\beta }_1=0.0448$, ${\beta }_2=0.2856$, ${\beta }_3=0.3001$, ${\beta }_4=0.2363$, and ${\beta }_5=0.1333$ [20,22].

MS-SSIM has shown promising results in comparison with a PSNR, single-scale SSIM, and Sarnoff. MS-SSIM outperformed when evaluated by human observers’ perception. Its correlation presented the highest.

We have reviewed the historical background of SSIM and several of its basic principles, and have found that UQI, MSSIM, and MS-SSIM are triggers acting as the foundations for all types of SSIM. Developed some years later, they have tried to complement and improve on the original SSIM in terms of image processing when applied to a specific area. In the next section, we described the types of SSIM developed from 2006 to 2021.

3. Current Improvement in SSIM

This section describes several improvements in SSIM since it first emerged. The objective of this section is to provide an adequate understanding of the development of SSIM, thus allowing researchers to select the appropriate SSIM type for comparison when applying a specific SSIM for medical image analysis.

3.1. Gradient-Based SSIM

Gradient-based SSIM (GSSIM) was the upgraded version of SSIM after realizing that the original SSIM has a defect in evaluating badly blurred images. The concept was derived by Chen et al. in 2006 when comparing the similarity values between the “Cameraman” image with Gaussian white noise and a blurred image [16]. The original SSIM showed a similarity score contrary to human perception by presenting a low MSSIM for a Gaussian white-noise-contaminated image while exhibiting a high similarity score for a blurred image. The image with Gaussian noise was perceived more subjectively than the blurred image, and by identifying this flaw, a GSSIM attempts to resolve this discrepancy.

The background of GSSIM emphasizes the sensitivity of the human eye in detecting the edge and contour information. From these two pieces of information, a human can capture the image structure from the scene. Therefore, to modify the original SSIM into GSSIM, the essential image processing insight is highlighting the edge of the images. In the original article on GSSIM, Chen et al. utilized a Sobel operator to spot the edges in the images because it simply generates masks and implements them over the entire image. The Sobel masks consist of two $3\times 3$ windows as filters, namely vertical and horizontal edge masks. Figure 3 shows the Sobel operator masks.

The vertical mask $G_x$ exposes the vertical edges, whereas the horizontal mask $G_y$ discovers the horizontal edges in the images. The magnitude of the gradient, otherwise known as a gradient vector, can be calculated by

$$G=\sqrt{{\left(G_x\right)}^2+{\left(G_y\right)}^2}=\left|G_x\right|+\left|G_y\right|,$$

(16)

and the edge angles can be formulated as

$${\theta }_G=ta{n}^{-1}\frac{G_y}{G_x} .$$

(17)

By applying the masks over the entire image and using Equation (16), we can obtain a gradient map indicating the edge and contour information [68].

Once we acquire the reference image gradient map ${\mathit{X}}^{\prime }$ and noise-contaminated image gradient map ${\mathit{Y}}^{\prime }$, the calculation technique for GSSIM is similar to that of SSIM by changing the contrast comparison $c_g$ and structure comparison $s_g$ as follows:

$$c_g\left(\mathit{x},\mathit{y}\right)=\frac{2{\sigma }_{x^{\prime }}{\sigma }_{y^{\prime }}+C_2}{{\sigma }_{x^{\prime }}^2+{\sigma }_{y^{\prime }}^2+C_2} ,$$

(18)

$$s_g\left(\mathit{x},\mathit{y}\right)=\frac{{\sigma }_{x^{\prime }{y}^{\prime }}+C_3}{{\sigma }_{x^{\prime }}{\sigma }_{y^{\prime }}+C_3} ,$$

(19)

where ${\sigma }_{x^{\prime }}$ is the standard deviation of vector ${\mathit{x}}^{\prime }$, ${\sigma }_{y^{\prime }}$ denotes the standard deviation of vector ${\mathit{y}}^{\prime }$, and ${\sigma }_{x^{\prime }{y}^{\prime }}$ is the covariance for vectors ${\mathit{x}}^{\prime }$ and ${\mathit{y}}^{\prime }$. Hence, GSSIM can be written as

$$GSSIM\left(\mathit{x},\mathit{y}\right)={\left[l\left(\mathit{x},\mathit{y}\right)\right]}^{\alpha }·{\left[c_g\left(\mathit{x},\mathit{y}\right)\right]}^{\beta }·{\left[s_g\left(\mathit{x},\mathit{y}\right)\right]}^{\gamma },$$

(20)

or

$$GSSIM\left(\mathit{x},\mathit{y}\right)=\frac{\left(2{\mu }_x{\mu }_y+C_1\right)\left(2{\sigma }_{x^{\prime }{y}^{\prime }}+C_2\right)}{\left({\mu }_x^2+{\mu }_y^2+C_1\right)\left({\sigma }_{x^{\prime }}^2+{\sigma }_{y^{\prime }}^2+C_2\right)} .$$

(21)

Using the same steps as formulated in Equations (9)–(13) and Figure 4, MGSSIM is as follows

$$MGSSIM\left(\mathit{X},\mathit{Y}\right)=\frac{1}{M}{\displaystyle \sum }_{j=1}^{M}GSSIM\left({\mathit{x}}_j,{\mathit{y}}_j\right).$$

(22)

Figure 4 depicts the GSSIM approach used to obtain the similarity score. The entire process is similar to that shown in Figure 1 in terms of the original SSIM; however, GSSIM implements images $\mathit{x}$ and $\mathit{y}$ only to calculate the luminance. For contrast and structure measures, however, they use the gradient images ${\mathit{X}}^{\prime }$ and ${\mathit{Y}}^{\prime }$. Then, by utilizing Equations (20)–(22), the similarity score can be obtained.

Although GSSIM shows promising results when comparing two different images, it is only effective when the badly blurred image is contaminated with Gaussian blur. In addition, SSIM evolving into GSSIM was a breakthrough to the extent of MGSSIM evolving into MS-GSSIM, using the procedure shown in Figure 5.

Images $\mathit{x}$ and $\mathit{y}$ are inputs treated as in a single-scale GSSIM for a scale of $1$. Then, $\mathit{x}$ and $\mathit{y}$ are processed by applying a Sobel operator to obtain the gradient maps of the reference and noise-contaminated images. In this step, we calculate the contrast $c_{g1}$ and structure $s_{g1}$ on a scale of $1$. Next, the reference and distorted gradient map images are filtered by LPF proceeded by downsampling by $2$. The downsampled gradient map images are calculated using single-scale GSSIM formulas to obtain $c_{g2}$ and $s_{g2}$. This method is repeated until $K$ iterations. When iteration $K$ is complete, we collect three parameters, $c_{gK}$, $s_{gK}$, and luminance $l_K$. For MS-GSSIM, $l_K$ is obtained using the input images after filtering and downsampling by $2$ in the last iteration. Thus, MS-GSSIM can be written as follows:

$$MS-GSSIM\left(\mathit{x},\mathit{y}\right)={\left[l_K\left(\mathit{x},\mathit{y}\right)\right]}^{{\alpha }_K}{\displaystyle \prod }_{k=1}^K{\left[c_{gk}\left(\mathit{x},\mathit{y}\right)\right]}^{{\beta }_k}·{\left[s_{gk}\left(\mathit{x},\mathit{y}\right)\right]}^{{\gamma }_k}.$$

(23)

3.2. Three-Component Weighted SSIM

A three-component weighted SSIM or three-component SSIM (3-SSIM) is an upgraded SSIM used to overcome the issues in blurred and noisy images. This 3-SSIM has the same objective as GSSIM, and thus 3-SSIM uses a similar step in the gradient map comparison of reference and distorted images. In other words, 3-SSIM was inspired by GSSIM [16] and three-component image comparisons segmenting the images into three components, namely edge, smooth, and texture regions [19]. The similarity score can then be calculated utilizing the assigned weights for each region.

A 3-SSIM was first introduced in 2009 and was the initial research direction for developing the SSIM into a four-component weighted SSIM (4-SSIM), as described later in Section 3.3. There are four steps to compute 3-SSIM. First, the SSIM index is calculated. Second, the images are divided into three regions: edge, smooth, and texture regions. An edge region can be found when the gradient is large, whereas a smooth region is relative to a small gradient. In addition, the texture region is extracted from the two thresholds. Third, non-uniform weights are implemented into the SSIM from the three regions. Fourth, the weighted SSIM is united by extracting the weighted average; hence, a similarity score can be obtained [19]. Figure 6 shows the 3-SSIM procedures used to yield a similarity score.

Procedures for 3-SSIM also exist for a multiscale method, called 3-MS-SSIM. In general, the 3-MS-SSIM approach is the same as the procedure depicted in Figure 2 but follows a quantifying procedure, as shown in Figure 6. The steps in Figure 6 can be described as follows:

• Step 1: Compute the SSIM map using Equation (14). Using this SSIM map, we can call up the structure information.
• Step 2: Calculate the gradient magnitude utilizing a Sobel operator over the reference and noised images.
• Step 3: Define the threshold value $T{H}_1=0.12\times g_{max}$ and $T{H}_2=0.06\times g_{max}$, where $g_{max}$ denotes a maximum grayscale level of gradient magnitude when computed over the original image.
• Step 4: Based on step 3, partition the images into edge, smooth, and texture regions using the following rules:
  If $p_o\left(i,j\right)>T{H}_1$ or $p_d\left(i,j\right)>T{H}_1$, it is an edge region;
  if $p_o\left(i,j\right)<T{H}_2$ and $p_d\left(i,j\right)\le T{H}_1$, it is a smooth region; and
  otherwise, if the pixels belong to a texture region but are not edge pixels, it is a texture region.
  Here, $\left(i,j\right)$ denotes the gradient coordinate, $p_o$ is the original image pixel, and $p_d$ denotes a degraded image pixel.

Figure 7 shows the images for every process in a specific region. Figure 7a depicts the original “Lena” image, whereas Figure 7b shows its blurred image, Figure 7c depicts the edge region image, Figure 7d shows the smooth region image, and Figure 7e depicts the texture region image.

3.3. Four-Component Weighted SSIM

The concept of 4-SSIM was derived from the same authors as in 3-SSIM to overcome the SSIM issue related to blurred image measures. However, 4-SSIM in the original article is fractioned into 4-SSIM itself, 4-MS-SSIM, 4-GSSIM, and 4-MS-GSSIM. Therefore, using the four-component model, it is possible to compute four types of SSIM by partitioning the original images into four regions, namely a changed-edge region, preserved-edge region, smooth region, and texture region. The difference between 3-SSIM and 4-SSIM is in the edge region: whereas 3-SSIM has only one edge region, 4-SSIM has two edge regions. In addition, the condition used to determine which pixel belongs to which region is also different [18,19].

Figure 8 shows the procedures used to yield the similarities of 4-SSIM, 4-MS-SSIM, 4-GSSIM, and 4-MS-GSSIM. The steps in Figure 8 can be defined as follows:

• Step 1: Calculate the SSIM map. This SSIM map is called the structure information.
• Step 2: Compute the gradient magnitude applying the Sobel operator for the reference and distorted images.
• Step 3: Define the threshold value $T{H}_1=0.12\times g_{max}$ and $T{H}_2=0.06\times g_{max}$, where $g_{max}$ denotes a maximum grayscale level of the gradient magnitude when computed over the original image. Here, $T{H}_1$ and $T{H}_2$ have an effect on the component regions under these situations, i.e., the smaller the first value, the more “edgey” the region. Furthermore, the smaller the second value, the less smooth the region is.
• Step 4: Based on step 3, the images are segmented into the changed edge, preserved edge, smooth, and texture regions using the following rules:
  If $p_o\left(i,j\right)>T{H}_1$ and $p_d\left(i,j\right)>T{H}_1$, the edge region is preserved;
  If ($p_o\left(i,j\right)>T{H}_1$ and $p_d\left(i,j\right)\le T{H}_1$) or ($p_o\left(i,j\right)\le T{H}_1$ and $p_d\left(i,j\right)>T{H}_1$), edge region is changed; and
  If $p_o\left(i,j\right)<T{H}_2$ and $p_d\left(i,j\right)>T{H}_1$, it is a smooth region.
  Otherwise, the pixels belong to a texture region if they are not part of the edge pixels.
  Here, $\left(i,j\right)$ denotes the gradient coordinate, $p_o$ is original image pixel, and $p_d$ denotes a degraded image pixel.

Figure 9 shows the images for every process within a specific region. Figure 9a depicts a preserved edge image of “Lena,” whereas Figure 9b shows a changed edge region image. In addition, Figure 9c displays a smooth region image, and Figure 9d depicts a texture region image.

3.4. Complex-Wavelet SSIM

After dealing with the SSIM issues in overestimating the similarity of the blurred images, a complex-wavelet SSIM was developed in 2009 to overcome the drawbacks of the original SSIM in the spatial domain related to a high sensitivity when measuring images with a few rotations, translations, and scaling in comparison with the original image [21]. The fundamental aspect behind CW-SSIM is using the wavelet coefficients inspired from [69] over the images to extract the similarity value. Similarly, the main objective of CW-SSIM is to develop an insensitive IQA in evaluating the images under a nonstructured geometric image distortion. CW-SSIM is emphasized more to resolve the drawbacks of the spatial domain by converting the images into a complex wavelet domain and acquiring the wavelet coefficients for determining the similarity score.

CW-SSIM attempts to model the complex wavelet domain IQA with the ability to separate the magnitude and phase distortion assessment. Moreover, it was developed to be more sensitive to phase distortions than magnitude distortions and insensitive to consistent relative phase distortions. The symmetric complex wavelet can be formulated as $w\left(u\right)=g\left(u\right)e^{j{\omega }_{c}u}$ for a low-pass-filter modulation, where ${\omega }_c$ denotes the modulated band-pass filter center frequency and $g\left(u\right)$ is a slowly varying and symmetric function. The dilated and translated version of $w\left(u\right)$ can be written as follows:

$$w_{s,p}\left(u\right)=\frac{1}{\sqrt{s}}w\left(\frac{u-p}{s}\right)=\frac{1}{\sqrt{s}}g\left(\frac{u-p}{s}\right)e^{j{\omega }_c\left(u-p\right)/s},$$

(24)

where $s\in {ℛ}^{+}$ denotes the scale factor, and $p\in ℛ$ is the translation factor. In addition, the continuous wavelet transformation of the real signal $x\left(u\right)$ is expressed as

$$X\left(s,p\right)=\frac{1}{2\pi }{{\displaystyle \int }}_{-\infty }^{\infty }X\left(\omega \right)\sqrt{s}G\left(s\omega -{\omega }_c\right)e^{j\omega p}d\omega ,$$

(25)

where $X\left(\omega \right)$ and $G\left(\omega \right)$ are the Fourier transform (FT) of $x\left(u\right)$ and $g\left(u\right)$, respectively. In a complex wavelet domain, we can determine the coefficients within the same spatial domain location represented in the same wavelet subbands of two images under comparison as ${\mathit{c}}_x\left\{c_{x,i}|i=1,2,3\dots ,N\right\}$ and ${\mathit{c}}_y\left\{c_{y,i}|i=1,2,3\dots ,N\right\}$, respectively. Then, CW-SSIM can be formulated as

$$\tilde{S}\left({\mathit{c}}_x,{\mathit{c}}_y\right)=\frac{2\left|{{\displaystyle \sum }}_{i=1}^N{c}_{x,i}{c}_{y,i}^{*}\right|+K}{{{\displaystyle \sum }}_{i=1}^N{\left|c_{x,i}\right|}^2+{{\displaystyle \sum }}_{i=1}^N{\left|c_{y,i}\right|}^2+K} ,$$

(26)

or can be written as a product of two components:

$$\tilde{S}\left({\mathit{c}}_x,{\mathit{c}}_y\right)=\frac{2{{\displaystyle \sum }}_{i=1}^N\left|c_{x,i}\right|\left|c_{y,i}\right|+K}{{{\displaystyle \sum }}_{i=1}^N{\left|c_{x,i}\right|}^2+{{\displaystyle \sum }}_{i=1}^N{\left|c_{y,i}\right|}^2+K}·\frac{2\left|{{\displaystyle \sum }}_{i=1}^N{c}_{x,i}{c}_{y,i}^{*}\right|+K}{2{{\displaystyle \sum }}_{i=1}^N\left|c_{x,i}{c}_{y,i}^{*}\right|+K} ,$$

(27)

where $c^{*}$ is the complex conjugate of $c$, and $K$ denotes the small positive constant to improve the CW-SSIM robustness at an extremely small local SNR. As in the original SSIM, $\tilde{S}\left({\mathit{c}}_x,{\mathit{c}}_y\right)$ is altered from $0$ to $1$ depending on the degree of similarity. The first component in Equation (27) is a maximum of $1$ if $\left|c_{x,i}\right|=\left|c_{y,i}\right|$. The first component is related to the magnitude. By contrast, the second coefficient is relative to the phase changes.

3.5. Improved SSIM with Sharpness Comparison

In 2016, Lee and Lim developed an improved version of SSIM, called improved SSIM with the sharpness comparison (ISSIM-S) [17]. The idea behind this improvement is to anticipate the shortcomings of SSIM in overestimating when quantifying blurred images but underestimating when measuring the images with a spatial translation; however, the spatial translated images are more visible from a human perspective than the blurred images. ISSIM-S uses the spatial domain with an improved assessment comparison in terms of sharpness. Although the focus of ISSIM-S is to accommodate the feasible IQA in measuring the images contaminating the rotation, translation, and scaling, ISSIM-S shows promising results in the assessment of images through several acquisitions, such as histogram equalization, mean luminance shifting, median filtering, impulsive noise, JPEG compression, and mean filtering.

The main shortcomings of SSIM are in the structural comparison, as in Equation (6). When the calculated SSIM does not include a structural comparison and only consists of the luminance and contrast, as in Equations (4) and (5), respectively, the similarity score of the spatial translation is not predicted to be low and has a consistent measurement with respect to HVS. However, if the three components of the SSIM are integrated into the SSIM calculation, the measure index evaluates the blurred image with a high SSIM value. By contrast, a slight vertical translated image has a low SSIM score, whereas the blurred image is noisier than the translated image.

Figure 10 shows a demonstration of using the “Lena” image to present the original, spatial translated, and JPEG compressed images with a green line in the vertical center of the images. As compared, the JPEG compression image is less perceptible than the image with the spatial translation, as shown in Figure 10b,c when they are compared with the reference image, as shown in Figure 10a.

According to the original article on ISSIM-S, the evaluation of the images shown in Figure 10 offers an overestimation of the SSIM score for Figure 10c; however, there is an underestimation of Figure 10b. Under normal circumstances, Figure 10b is more perceptible than Figure 10c. This small SSIM score is obtained because of the drawback in the structural comparison. Therefore, to fix this disadvantage, ISSIM-S defines a new structure factor to be

$$\tilde{s}\left(\mathit{x},\mathit{y}\right)=\frac{\left(2{\sigma }_{x-}{\sigma }_{y-}+C_2\right)\left(2{\sigma }_{x+}{\sigma }_{y+}+C_2\right)}{\left({\sigma }_{x-}^2+{\sigma }_{y-}^2+C_2\right)\left({\sigma }_{x+}^2+{\sigma }_{y+}^2+C_2\right)} ,$$

(28)

where ${\sigma }_{x-}$ denotes a standard deviation for image $\mathit{x}$ smaller than ${\mu }_x$, whereas ${\sigma }_{x+}$ is the standard deviation for image $\mathit{x}$ larger than ${\mu }_x$. In addition, ${\sigma }_{y-}$ denotes a standard deviation for image $\mathit{y}$ smaller than ${\mu }_y$, and meanwhile, ${\sigma }_{y+}$ is the standard deviation for image $\mathit{y}$ larger than ${\mu }_y$. The definition of $\tilde{s}\left(\mathit{x},\mathit{y}\right)$ is the correlation of the standard deviation when having positive or negative scores because ${\sigma }_{x-}$ and ${\sigma }_{x+}$ (or ${\sigma }_{y-}$ and ${\sigma }_{y+}$) can correspond to the object structure by fractioning into brighter and darker regions locally. Nevertheless, by improving only $s\left(\mathit{x},\mathit{y}\right)$ into $\tilde{s}\left(\mathit{x},\mathit{y}\right)$, the presence of an overestimation even exists in the JPEG compressed image. Therefore, a new comparison is necessary to be added, namely a sharpness comparison $h\left(\mathit{x},\mathit{y}\right)$. With the added $h\left(\mathit{x},\mathit{y}\right)$, ISSIM-S has confidence in the improvement by two novel upgraded parameters. Here, $h\left(\mathit{x},\mathit{y}\right)$ is the correspondence to the normalized digital Laplacian, which is formulated as

$$h\left(\mathit{x},\mathit{y}\right)=\frac{2\left|{\nabla }^2\mathit{x}\right|\left|{\nabla }^2\mathit{y}\right|+C_2}{{\left|{\nabla }^2\mathit{x}\right|}^2+{\left|{\nabla }^2\mathit{y}\right|}^2+C_2} ,$$

(29)

where ${\nabla }^2\mathit{x}$ and ${\nabla }^2\mathit{y}$ are the normalized digital Laplacian determined by

$${\nabla }^2\mathit{x}=\mathit{x}-{\mu }_x,$$

(30)

$${\nabla }^2\mathit{y}=\mathit{y}-{\mu }_y.$$

(31)

Therefore, ISSIM-S is

$$ISSIM-S\left(\mathit{x},\mathit{y}\right)=l\left(\mathit{x},\mathit{y}\right)·c\left(\mathit{x},\mathit{y}\right)·\tilde{s}\left(\mathit{x},\mathit{y}\right)·h\left(\mathit{x},\mathit{y}\right).$$

(32)

Figure 11 shows the ISSIM-S measure used to inspect the images. The luminance and contrast are compared as in the original SSIM, whereas the structure is calculated using the improved version. In addition, the sharpness calculation completes this new IQM. The final step is to combine all comparisons and then obtain the dot product. As described previously, SSIM is better in a local pixel utilizing a sliding window. Therefore, ISSIM-S applies the same method to yield the mean of ISSIM-S (MISSIM-S):

$$MISSIM-S\left(\mathit{X},\mathit{Y}\right)=\frac{1}{M}{\displaystyle \sum }_{j=1}^{M}ISSIM-S\left({\mathit{x}}_j,{\mathit{y}}_j\right).$$

(33)

The similarity score is also a variant from $0$ to $1$, with $1$ if $\mathit{x}=\mathit{y}$.

Several comparisons of SSIM have been conducted [70]. In 2021, Mudeng et al. attempted to use the benefit of MISSIM-S for the first time to assess the reconstructed images from simulated images of diffuse optical tomography (DOT) [71]. They compared four types of SSIMs, i.e., MSSIM, MS-SSIM, MISSIM-S, and MS-ISSIM-S. MS-ISSIM-S can be developed using MS-SSIM and MISSIM-S. Figure 12 shows the measurement processes of MS-ISSIM-S.

The procedure used to calculate MS-ISSIM-S is the same as MS-SSIM, but $h\left(\mathit{x},\mathit{y}\right)$ is computed independently. When $h\left(\mathit{x},\mathit{y}\right)$ is yielded at each $k$-scale, the mean value is employed to complete the procedure, and thus MS-ISSIM-S can be written as follows:

$$MS-SSIM\left(\mathit{x},\mathit{y}\right)=\left(\frac{1}{K}{\displaystyle \sum }_{k=1}^K{h}_k\left(\mathit{x},\mathit{y}\right)\right)\left({\left[l_K\left(\mathit{x},\mathit{y}\right)\right]}^{{\alpha }_K}{\displaystyle \prod }_{k=1}^K{\left[c_k\left(\mathit{x},\mathit{y}\right)\right]}^{{\beta }_k}·{\left[{\tilde{s}}_k\left(\mathit{x},\mathit{y}\right)\right]}^{{\gamma }_k}\right).$$

(34)

3.6. Other SSIM Types

Since SSIM has emerged by bringing about a new IQM model to quantify the images, it has had the potential to substitute for the MSE or PSNR, and thus, to the best of our knowledge, there are more than 20 different types of SSIM that have been developed to solve the issues of the original SSIM or according to the main objective of a specific task. In addition, SSIM IQA can be modified by combining the information content weighting with an MS-SSIM [72]. This paper merely reviews the popular SSIMs suitable for the goal of this paper. Subsequently, this subsection provides a brief description of the improved SSIM developed from 2015–2020.

By splitting the images into two regions according to their inter-patch and intra-patch, a novel SSIM scheme was introduced [73]. The two-component indices complement each other. The first index has a goal to inspect the inter-patch feature by examining the disparities on the center patch and the spatial neighborhoods. By contrast, the second component makes an effort to evaluate the intra-patch feature by measuring the similarity in the curvature and gradient. To obtain a single similarity score, an integration approach is executed. If the quality of the image is low, the assigned weight is higher for the first component.

The SSIM can be expanded to examine not only the image but also the other scopes, such as three-dimensional (3D) video displays [74]. The prime major is focused on the availability of a source image or reference image, and thus SSIM is called FR IQM. In [74], the authors proposed using the SSIM as a perceptual IQA to measure the 3D video visual quality by combining it with a joint bit, which they called the SSIM-based joint bit allocation approach. This scheme promises an enhanced 3D video visual quality. Surprisingly, in 2018, Zhang et al. offered an upgraded SSIM version by providing a no-reference SSIM (NSSIM) [75]. They improved the SSIM from an FR to no-reference (NR) IQA. This scheme offers a solution in a practical environment because the original image is not always available, and even distorted images are widely accessible. Using the re-blur theory by utilizing Gaussian blur, NSSIM may extract the features of the images and then use them to obtain the information related to the blurriness. In addition, the blurriness can define a novel model of NSSIM and can be written as

$$b\left(\mathit{x},\mathit{y}\right)=\frac{2d_x{d}_y+C_4}{d_x^2+d_y^2+C_4} ,$$

(35)

where $d_x$ and $d_y$ are the distorted image and its re-blurred image, respectively. In addition, $C_4$ is a small constant to avoid instability when $d_x^2+d_y^2$ is close to zero. A new metric of SSIM is formulated as

$$SSI{M}_r\left(\mathit{x},\mathit{y}\right)={\left[l\left(\mathit{x},\mathit{y}\right)\right]}^{\alpha }·{\left[c\left(\mathit{x},\mathit{y}\right)\right]}^{\beta }·{\left[s\left(\mathit{x},\mathit{y}\right)\right]}^{\gamma }·{\left[b\left(\mathit{x},\mathit{y}\right)\right]}^{\lambda },$$

(36)

with $\lambda$ as the exponent coefficient for $b\left(\mathit{x},\mathit{y}\right)$. As with the original SSIM, this new metric applies a sliding window to measure the local similarity, and thus the mean score is

$$MSSI{M}_r\left(\mathit{X},\mathit{Y}\right)=\frac{1}{M}{\displaystyle \sum }_{j=1}^{M}SSI{M}_r\left({\mathit{x}}_j,{\mathit{y}}_j\right).$$

(37)

Finally, the NSSIM can be determined as

$$NSSIM\left(\mathit{x},\mathit{y}\right)=1-MSSI{M}_r\left(\mathit{X},\mathit{Y}\right).$$

(38)

Another type of SSIM is the contrast sensitivity function SSIM (CSF + SSIM) [76]. This CSF + SSIM combines the non-linear characteristics of the luminance perception with the contrast sensitivity characteristics from the HVS for a contrast-distorted image evaluation. However, CSF + SSIM is deemed complex in terms of its computations because it separates the images in the color space transform into the luminance, red–green channel, and blue–yellow channel to yield their perceptions. Moreover, CSF + SSIM employs a discrete cosine transform (DCT) to acquire the weights corresponding to the CSF. It then deploys an inverse DCT (IDCT) to obtain the color space for the perceived images, and as the last step, it applies the SSIM to obtain the similarity score. In addition, a spherical SSIM is used to objectively inspect the video quality of omnidirectional video [33]. A spherical uniform SSIM for assessing panoramic video has also been established [77], and a multi-exposure image fusion (MEF) approach by optimizing the SSIM, which is called the color MEF structural similarity (MEF-SSIM_c), has been presented [78]. Finally, a topological SSIM (T-SSIM) was introduced for a specific task to identify a nearby organ populated with tumor-organ distances and volumes for two compared patients [79].

4. SSIM in Medical Imaging

This section discusses the implementation of SSIM, particularly for imaging techniques such as MRI, ultrasonography, CT scan, X-rays, and optical imaging. This section aims to emphasize reviews of SSIM applied to the measurement of medical images. We reviewed SSIM in medical imaging based on the published year of the articles.

To identify the relevant studies, a systematic methods overview [80] along with several major databases were used to search the matched keywords, such as “SSIM” AND “Magnetic Resonance Imaging” OR “Computed Tomography” OR “Ultrasonography” OR “Ultrasound” OR “X-ray” OR “Optical Imaging” OR “Medical Images” OR “Medical Imaging”. The main databases included Google Scholar, PubMed, IEEE, MDPI, Springer, Elsevier, and others. There were 125 identified articles related to the keywords including journals, conference proceedings, and book chapters, consisting of the original articles on UQI, SSIM, MS-SSIM, three- and four-component weighted SSIMs, CW-SSIM, ISSIM-S, and other SSIM families, as well as the SSIM implementation for MRI, CT, ultrasonography, X-ray, and optical imaging. Overall, $72$ articles relevant to the goal of this review paper regarding SSIM applied in medical imaging were reviewed. We did not exclude the same SSIM implementation in medical imaging as with an IQA, and instead, we mentioned, classified, and briefly reviewed them in Section 4.1 and Section 4.5 according to the medical imaging technique used. We provided this method because we prefer to offer a wide range of SSIM implementations and fulfill the objective of this review paper of providing the readers or researchers with potential medical image examination research methods that can be improved using SSIM. Additionally, Section 4.6 provided a thorough review of $4$ articles related to SSIM application in medical imaging for loss function in convolutional neural network (CNN), reducing metal artifact, contour extractor, and IQA. Thus, we described in detail the image acquisition method, filtering, or the other approaches used to acquire the distorted images for comparing with the original image from the medical modality imaging scheme. Additionally, this study’s limitation was stated in Section 4.7.

To avoid misleading and maintain the objective of this review, we emphasized that the SSIM measure has an original goal to substitute the common measures, such as MSE and PSNR in measuring any signals in 1D, 2D, and 3D, as long as there is a reference signal. Thus, the SSIM measure can assess digital images including medical images. To offer a better representation of the methodology when using SSIM to evaluate medical images, we provided Figure 13, Figure 14 and Figure 15 to depict medical images when they were processed using three- and four-component SSIMs, as well as ISSIM-S. However, since this study’s goal is to provide the SSIM prospect in medical images, we did not measure each SSIM type (three-component SSIM, four-component SSIM, and ISSIM-S) similarity score. We used a digital database for screening mammography (DDSM) [81] containing $2620$ cases of normal, benign, and malignant breast cancers extracted from calcification and breast masses abnormalities. Figure 13 shows the images for every process in a specific region for three-component SSIM. Figure 13a depicts the original DDSM right breast masses benign cancer with craniocaudal (CC) view image, Figure 13b shows its blurred image, Figure 13c depicts the edge region image, Figure 13d shows the smooth region image, and Figure 13e depicts the texture region image. Figure 14 shows the images for every process within a specific region for four-component SSIM. Figure 14a depicts a preserved edge image of DDSM right breast masses benign cancer with CC, whereas Figure 14b shows a changed edge region image. In addition, Figure 14c displays a smooth region image, and Figure 14d depicts a texture region image. Figure 15 shows a demonstration of using DDSM right breast masses benign cancer with CC image to present the original, spatial translated, and JPEG compressed images with a green line in the vertical center of the images. As compared, the JPEG compression image is less perceptible than the image with the spatial translation, as shown in Figure 15b,c when they are compared with the reference image, as shown in Figure 15a. As depicted in Figure 13, Figure 14 and Figure 15, using the image processing steps for medical images, we may compute the similarity score to obtain the image quality. These motivated us to review articles related to SSIM in the medical field.

4.1. Magnetic Resonance Imaging

To the best of our knowledge, the first SSIM implementation for evaluating the images of MR was in 2005 [82,83]. The attempt in [83] used distorted MR images and then compared them with the original MR image. Three modules creating a closed-loop system containing de-noising filters, evaluations method, and adjustment rules were used. The corrupted images were examined using SSIM along with the mean absolute error (MAE), root-mean-square error (RMSE), SNR, and PSNR. Their results indicate that SSIM shows a high similarity score when the distorted images are close to the reference image. This is the first time an exploration of the original SSIM was started for use in other applications, such as MR medical images. As we mentioned previously, the original SSIM may interpret the blurred images with a high similarity score owing to the effect of the structural comparison. However, their results indicate that SSIM has the potential to assess medical images. In addition, in [82], the SSIM showed a relatively decent performance to quantify the head MR images. However, there were inconsistencies in the $40\%$ and $70\%$ quality factors. Various quality factors were deployed to acquire the similarity value. Nonetheless, with the higher quality factor of compression, the SSIM predicted the similarity with a low score and was deemed unsuitable for the quality metric in [82].

In 2007, a group of researchers applied an SSIM application for security in the MR and computed tomography (CT) images related to the watermarked medical images [31]. Medical image watermarks are crucial because they may comprise the medical information of the patients, proving ownership, and alternating location on the images. Moreover, watermarked medical images frequently store hidden messages for later extraction to obtain the reports. Their results showed that SSIM is less capable of measuring the degradation of medical images when the images are embedded with a watermark. They speculated that the best metric for a watermarking measure is a steerable visual difference predictor (SVDP). By contrast, SSIM and a quality index based on local variance (QILV) have been exploited for the quality measure of estimating the magnitude of MR based on the linear minimum mean squared error (LMMSE) [84]. The SSIM in this task performed competitively in measuring images fused with an LMMSE estimator.

The compression of medical images is challenging in teleradiology because teleradiology requires transmission to transfer the images [85]. In the transferring process, the image quality may be reduced owing to the limited communication, and thus the image fidelity can be decreased. The quality measure is necessary to determine the threshold value to compress these medical images; hence, in the transmission, the important information of the images is not diminished. A partitioning in hierarchical tree (SPIHT) compression algorithm has also been used to determine the maximum threshold standard for compression. The compressed images were compared with their original to assess their similarity. In this specific task, SSIM and PSNR showed agreeable results with the mean opinion score (MOS), and thus for this case, SSIM is considered appropriate to cut off the threshold value when the medical images are compressed with a defined bit rate. Subsequently, medical image fusion to improve the confidence of radiologists in diagnosing a specific disease was accomplished in 2009 by Zhang and Zheng [86]. The objective of this research is to decrease the inconsistency of a diagnosis when subjective observers read the medical images. In addition, SSIM contributed significantly to the fusion approach. A unique SSIM implementation in this article was shown because SSIM was not used as the IQA metric; instead, it was utilized as the image fusion itself. Based on an understanding of the image fusion from several imaging modalities, the decision confidence may be improved, and the images combined from MRI and CT were employed to generate more perceivable images. Because CT can offer better information in denser tissue with less distortion and MRI provides adequate imaging for soft tissue, a fusion was executed to reduce the workload of the radiologists. Their results indicated that image fusion using SSIM is remarkable compared with existing image fusion methods, such as a Laplacian pyramid (LP), gradient pyramid (GP), contrast pyramid (CP), steerable pyramid (StrP), and discrete wavelet transform (DWT).

For a denoising method to remove Rician noise from MR images, the SSIM and MSE as objective metrics were compared with the MOS to measure the denoising methodology using the discretized total variation [87]. In this study, SSIM performed suitably with the MOS, which is related to HVS. With the highest standard deviation, the SSIM scores diminished following the subjective measurement scores. In addition, in 2013, to reduce the presence of aliasing, an improve compressed sensing technique was introduced, whereas the SSIM and PSNR were the objective measures used to assess the effectiveness of this approach [37]. Their results showed that SSIM is suitable for use with the proposed method as an objective IQM. Subsequently, in 2015 and 2017, to restore the MR images from the existence of noise during the acquisition steps, the SSIM along with SNR, PSNR, MSE, and RMSE was used to evaluate novel denoising algorithms [40,88,89,90]. With the computational improvements, an artificial intelligence (AI) method including deep learning has been proposed to solve the medical image analysis issues, and the results were compared using SSIM [38,39,91,92,93,94]. SSIM was also compared with other objective IQAs to obtain comprehensive insight related to an effective IQM for diagnosis by five radiologists [36] and was employed in measuring an image acquisition [95].

To conclude this subsection,

Table 1. Summary of published articles encompassing SSIM for MRI. MRI, magnetic resonance imaging; CT, computed tomography; MAE, mean-absolute error; RMSE, root-mean-square error; SNR, signal-to-noise ratio; PSNR, peak-signal-to-noise ratio; SVDP, steerable visual difference predictor; QILV, quality index based on local variance; LN, level on noise; LP, Laplacian pyramid; GP, gradient pyramid; CP, contrast pyramid; StrP, steerable pyramid; DWT, discrete wavelet transform; $Q^{AB/F}$, visual information quality; MOS, mean opinion score; SI, similarity index; NRMSE, normalized-root-mean-square error; VIF, visual information fidelity; FSIM, feature similarity index; NQM, noise quality metric; GMSD, gradient magnitude similarity deviation; HDRVDP, high dynamic range visible difference predictor; CNN, convolutional neural network.

Study	Year	Modality	SSIM Implementation	Compared Matrix	Results
Castellanos et al. [83]	2005	MRI	IQA	MAE, RMSE, SNR, and PSNR	SSIM is suitable for this task
Rajagopalan and Robb [82]	2005	MRI	IQA	Subjective measure	The subjective measure is superior
Dowling et al. [31]	2007	MRI and CT	IQA	PSNR and SVDP	SVDP is suitable for this research
Aja-Fernández et al. [84]	2007	MRI	IQA	MSE, QILV, and LN	SSIM has competitive results compared to QILV
Kumar et al. [85]	2009	MRI and CT	IQA	PSNR	SSIM is suitable for this task
Xiao and Zheng [86]	2009	MRI and CT	Images fusion	LP, GP, CP, StrP, and DWT	$Q^{AB/F}=0.62$
Varghess et al. [87]	2012	MRI	IQA	MSE and MOS	SSIM is suitable with MOS
Zhu et al. [37]	2013	MRI	IQA	PSNR	SSIM is suitable for this task
Srivastava et al. [88]	2015	MRI	IQA	PSNR	SSIM is suitable for this task
Srivastava et al. [89]	2015	MRI	IQA	PSNR	SSIM is suitable for this task
Saladi and Prabha [40]	2017	MRI	IQA	SNR, PSNR, MSE, RMSE	SSIM is suitable for this task
Chandrashekar and Sreedevi [90]	2017	MRI	IQA	PSNR, entropy, and MSE	SSIM is suitable for this task
Mostafa et al. [92]	2017	MRI	Segmentation	SI	$\mathrm{Accuracy}=97.5\%$
Duan et al. [39]	2019	MRI	IQA	MAE	SSIM is suitable for this task
Pawar et al. [93]	2019	MRI	IQA	NRMSE	SSIM is suitable for this task
Krohn et al. [95]	2019	MRI	Clustering	-	$SSIM=0.7$$\mathrm{to}>0.9$
Wang et al. [94]	2020	MRI	IQA	PSNR and NRMSE	SSIM and NRMSE performances are decent
Mason et al. [36]	2020	MRI	IQA	MOS, VIF, FSIM, NQM, GMSD, HDRVDP, PSNR, and RMSE	VIF shows the decent results
Nirmalraj and Nagarajan [91]	2020	MRI	IQA	PSNR, MSE, and entropy	SSIM is suitable for this task
Jaubert et al. [38] *	2021	MRI	Loss function	MAE	$p$$\mathrm{value}<0.05$

* A detailed review is provided in Section 4.6.

briefly presents the SSIM applications for MR medical images. In Table 1, SSIM is adopted not only for IQA but also for other purposes, such as image fusion, segmentation, clustering, and loss function. In addition, compared to other metrics, SSIM performances are competitively adequate according to the applied task. Most of the SSIM implementations as an IQA showed that SSIM is suitable for an assigned task, indicating that it offers reliable performance for measuring an enhancement of the image quality in comparison to the other traditional measures.

4.2. Computed Tomography

To the best of our knowledge, SSIM was used for the first time to evaluate CT images in 2007 and was assigned as the IQM for image fusion between CT and MR images [96], as well as for CT, US, and X-ray medical image compression using DCT [97]. According to [94], SSIM was able to quantify the improvement of fused images by showing the tumor location, whereas in [97], the SSIM aided in assessing the compressed medical images to determine the optimal block size for use in a compression method. SSIM showed a decreasing similarity score when the compressed block size was increased. This situation states that the SSIM can measure well the quality of the images according to the compression level. More compression makes more blurriness. With a certain blurriness, the medical images may not be easily readable, thus leading to the wrong diagnosis in telemedicine. The compression level is crucial because, with abundant compression, the important structural information of the images may be lost. By implementing the objective IQM, the threshold value of the compression level can be identified. Herein, we imply implicitly that SSIM has the potential to develop into the direction of a computer-aided diagnosis (CAD) method.

In 2016, studies related to SSIM for CT images were identified for investigating the ability of a denoised method utilizing a Wiener filter and the threshold in the wavelet domain [62]. In addition, a two-dimensional (2D) filter to enhance the CT image resolution by maximizing the SSIM similarity value [98], a low-dose CT denoising algorithm using locally consistent non-local means (LC-NLM) [63], and fusion between CT and MR images implementing a principal component analysis (PCA) and the maximum selection [99] were developed. These four studies presented SSIM as a reliable quality measure to assess the improvement of the images. In 2017–2019, SSIM was found to be a quality metric for measuring the results from 3D printed lung vessels [100], an approach to reducing the metal artifact in CT images by excluding the luminance comparison [101], an IQM for deep learning [41,43,102], and an alternative random forest (ARF) regression tool [103]. It was also used for CT tooth images extracted from denoised images filtered using a wavelet and bilateral filter [65], the removal of Gaussian noise [42], and image restoration and reconstruction [64]. In [101], the role of SSIM was distinguished from IQA as a method to reduce the artifacts caused by metal. In addition, a modified SSIM was utilized to construct this task by ignoring the luminance factor but maintaining the contrast and structural comparisons. This modified SSIM should be completed because the metal artifacts and a superposition map may vary substantially, whereas the structural or edge information can be indistinguishable. With the role of SSIM, correlated images can be obtained, and two correlation maps can then be compared to acquire reduced metal artifact images. In 2020–2021, SSIM was used as an evaluation metric for ovarian cancer [104], a generative adversarial network (GAN) [105], and Franken-CT [67].

To conclude this subsection,

Table 2. Summary of the published articles encompassing SSIM for CT. CT, computed tomography; MRI, magnetic resonance imaging; RMSE, root-mean-square error; MSE, mean-square error; PSNR, peak-signal-to-noise ratio; PRD, percent rate of distortion; CC, correlation coefficient; FF, fusion factor; LI-MAR, linear interpolation metal artifact reduction; NMAR, normalized metal artifact reduction; RMAR, refined metal artifact reduction; IQI, image quality index; VIF, visual information fidelity; ZNCC, zero-normalized cross-correlation; MAE, mean-absolute error.

Study	Year	Modality	SSIM Implementation	Compared Matrix	Results
Senthilkumar and Muttan [96]	2007	CT and MRI	IQA	RMSE	SSIM is suitable for this task
Singh et al. [97]	2007	CT, US, and X-ray	IQA	MSE, PSNR, PRD, and CC	SSIM has the highest score
Diwakar and Kumar [62]	2016	CT	IQA	PSNR	SSIM is suitable for this task
Mahmoud et al. [98]	2016	CT	IQA	PSNR	SSIM is suitable for this task
Green [63]	2016	CT	IQA	-	SSIM is suitable for this task
Himanshi et al. [99]	2016	CT and MRI	IQA	FF	SSIM is suitable for this task
Joemai and Geleijns [100]	2017	CT	IQA	-	SSIM is suitable for this task
Zhang et al. [102]	2018	CT	IQA	RMSE	SSIM is suitable for this task
Kim and Byun [43]	2018	CT	IQA	-	SSIM is suitable for this task
Hu and Zhang [103]	2018	CT and MRI	IQA	PSNR	SSIM is suitable for this task
Wang et al. [65]	2018	CT	IQA	PSNR	SSIM is suitable for this task
Kuanar et al. [41]	2019	CT	IQA	PSNR	SSIM is suitable for this task
Kim et al. [99] *	2019	CT	Reducing metal artifact	LI-MAR, NMAR, and RMAR	$MAE=8.52$$\mathrm{and}10.12$
Elaiyaraja et al. [42]	2019	CT and MRI	IQA	PNSR, IQI, and VIF	SSIM is suitable for this task
Sun et al. [64]	2019	CT	IQA	PSNR	SSIM is suitable for this task
Urase et al. [104]	2020	CT	IQA	PSNR	SSIM is suitable for this task
Gajera et al. [105]	2021	CT	IQA	PSNR	SSIM is suitable for this task
Martinez-Girones et al. [67]	2021	CT and MRI	IQA	ZNCC, MAE, and Dice coefficient for bone class	SSIM is suitable for this task

* A detailed review is provided in Section 4.6.

describes SSIM used for CT medical images. As indicated in Table 2, SSIM has been embraced not only for IQA but also for other objectives, such as noise reduction. In addition, compared to other metrics, SSIM performs competitively well according to the specialized task. In practical terms, all SSIM roles listed in Table 2 are for image quality measures. They indicate that SSIM has the potential to become a favorable IQA.

4.3. Ultrasonography

We review SSIM applications for ultrasonography in this section. We found that the first implemented SSIM for US was in 2007. SSIM along with the SNR, coefficient of correlation (CoC), edge preservation index (EPI), and QI were used to quantify the image enhancement when a versatile wavelet domain algorithm was utilized [106]. In the same year, US with two other medical images, CT and X-ray images, were measured using SSIM, MSE, PSNR, CC, and PRD to identify the effectiveness of a novel algorithm for compressing images in the field of teleradiology using an adaptive threshold value of variance [107]. In 2008, an algorithm was developed to reduce the effects of speckle, and the developed algorithm was measured with the Michelson contrast measure (CM), the Beghdadi and Le Négrate contrast measure (CBN), PSNR, and SSIM [108]. Their results showed that SSIM has potential effectiveness as an IQM for the US, although, at the time, SSIM had existed for only 3 or 4 years.

The reduction in speckle in US images has brought several types of studies to this issue. In 2016, the least-squares Bayesian [60], adaptive non-local means [109], local statistic, and non-local mean filter [110] algorithm estimations were established to reduce the speckle in US images, and several IQAs including SSIM, SNR, MSE, and a sum of the variance (SV) were designated to evaluate the improved algorithms. An uncommon SSIM implementation was conducted using CW-SSIM as a contour extractor of a tongue by Xu et al. [61,111]. They compared the performance of MSSIM and CW-SSIM with the normalized PSNR (NPSNR). MSSIM and the NPSNR demonstrated similar results, whereas CW-SSIM presented superior results by showing the tongue position with the peak of CW-SSIM.

Apparently, more attention from researchers was drawn in 2017–2021 for creating an efficient algorithm to eliminate speckle in US images [58,59,66,112,113,114,115,116,117,118]. In addition, US images can be analyzed using deep learning to conduct breast tumor segmentation and reconstruct images from raw channel data [53,119]. In [53], the combination between SSIM and L1-norm in the loss function was applied to capture the local context information from the surrounding tumor area, whereas in [119], MS-SSIM and PSNR were utilized as the loss function.

To conclude this subsection,

Table 3. Summary of the published articles encompassing SSIM for the US. US, ultrasound; CoC, coefficient of correlation; EPI, edge preservation index; QI, quality index; MSE, mean-square error; PSNR, peak-signal-to-noise ratio; CC, correlation coefficient; PRD, percent rate of distortion; CM, Michelson contrast measure; CBN, Beghdadi and Le Négrate contrast measure; SNR, signal-to-noise ratio; IQI, image quality index; SV, a sum of the variance; MSD, mean sum of distances; NPSNR, normalized peak-signal-to-noise ratio; MAE, mean-absolute error, UQI, universal quality index; ENL, equivalent number of looks; CNR, contrast-to-noise ratio; C, variation of conductance; SNR, signal-to-noise ratio; RMSE, root-mean-square error; IoU, intersection over union; SD, standard deviation.

Study	Year	Modality	SSIM Implementation	Compared Matrix	Results
Gupta et al. [106]	2007	US	IQA	CoC, EPI, and QI	SSIM is suitable for this task
Singh et al. [107]	2007	US and X-ray	IQA	MSE, PSNR, CC, and PRD	SSIM is suitable for this task
Munteanu et al. [108]	2008	US	IQA	CM, CBN, and PSNR	SSIM is suitable for this task
Nagaraj et al. [60]	2016	US	IQA	SNR, PSNR, CoC, and IQI	SSIM is suitable for this task
Ai et al. [109]	2016	US	IQA	PSNR	SSIM is suitable for this task
Yang et al. [110]	2016	US	IQA	SNR, MSE, and SV	SSIM is suitable for this task
Xu et al. [61] *	2016	US	Tongue contour extractor	No similarity constraint and similarity constraint	$MSD=2.96\pm 0.95$$\mathrm{to}3.65\pm 1.02$
Xu et al. [111]	2016	US	Tongue contour extractor	NPSNR	CW-SSIM has the best performance
Sagheer and George [59]	2017	US	IQA	PSNR and EPI	SSIM is suitable for this task
Javed et al. [112]	2018	US	IQA	PSNR	SSIM is suitable for this task
Gupta et al. [113]	2018	US	IQA	PSNR, MSE, and MAE	SSIM is suitable for this task
Ahmed [58]	2018	US	IQA	MSE, SNR, and PNSR	SSIM is suitable for this task
Gupta et al. [114]	2019	US	IQA	PSNR	SSIM is suitable for this task
Nadeem et al. [115]	2019	US	IQA	SNR	SSIM is suitable for this task
Balamurugan et al. [118]	2020	US	IQA	PSNR	SSIM is suitable for this task
Lan and Zhang [116]	2020	US	IQA	PSNR, ENL, and CNR	SSIM is suitable for this task
Singh et al. [66]	2020	US	IQA	EPI and UQI	SSIM is suitable for this task
Singh et al. [53]	2020	US	Loss function	-	$Dice coefficient=0.8682\mathrm{to}0.9376,IoU=0.8037\mathrm{to}0.8882,Sensitivity=0.9011\mathrm{to}0.9155,\mathrm{and}Specificity=0.9949\mathrm{to}0.9973$
Strohm et al. [119]	2020	US	Loss function	MAE	$mean and SD=0.91\pm 0.04$$\mathrm{and}0.90\pm 0.01$
Bharadwaj [117]	2021	US	IQA	C, SNR, PSNR, MSE, and RMSE	SSIM is suitable for this task

* A detailed review is provided in Section 4.6.

illustrates the purposes of using SSIM for US medical images. As indicated in Table 3, SSIM is accepted not only for IQA but also for other purposes, such as a contour extractor and loss function. In addition, compared to other metrics, SSIM performances are competitively appropriate according to their dedicated assignment.

4.4. X-ray

The fourth discussion on medical images is focused on X-ray images. Unlike MRI, CT, and US images, SSIM implementation for medical X-ray images was introduced in 2012 for the first time to the best of our knowledge. As is typical, SSIM was constructed to measure a designated algorithm to estimate the similarity score between the ground-truth image and the denoised images. In 2012, SSIM, PSNR, SNR, and MSE were appointed to evaluate an improved algorithm [120]. Similarly, in 2016, a method to eliminate Gaussian noise was established using an edge preserved wavelet packet transformation [121], and the denoising of a contrast-enhanced X-ray image was accomplished [122]. In addition, SSIM was allocated together with the MSE, RMSE, SNR, PSNR, recall, accuracy, precision, and error rate as an objective IQA. Subsequently, in 2017, a security watermark algorithm was developed to secure the diagnostic information of a patient, as well as the ownership and authentication [123] in X-ray and MR images. Again, SSIM has a role as an IQM along with wavelet domain SNR (WSNR) and PSNR. The feasibility of SSIM was not only shown for a numerical analysis but also in regard to deep learning, such as a CNN. The improved algorithm with deep learning may be measured using SSIM related to compression and noise reduction [124,125,126,127]. Moreover, comparisons of filter techniques used to reduce the noise and image reconstruction, as well as apply image compression, were conducted using X-ray images [44,128,129,130].

To conclude this subsection,

Table 4. Summary of the published articles encompassing SSIM for X-ray. PSNR, peak-signal-to-noise ratio; CT, computed tomography; MRI, magnetic resonance imaging; SNR, signal-to-noise ratio; MSE, mean-square error; RMSE, root-mean-square error; WSNR, wavelet domain signal-to-noise ratio; RMSE, root-mean-square error; FSIM, feature similarity index.

Study	Year	Modality	SSIM Implementation	Compared Matrix	Results
Cerciello et al. [120]	2012	X-ray	IQA	PSNR, SNR, and MSE	SSIM is suitable for this task
Rajith et al. [121]	2016	X-ray	IQA	MSE, RMSE, SNR, PSNR, recall, accuracy, precision, and error rate	SSIM is suitable for this task
Jeon [122]	2016	X-Ray	IQA	PSNR	SSIM is suitable for this task
Kunhu et al. [123]	2017	Xray and MRI	IQA	PSNR and WSNR	SSIM is suitable for this task
Zhang and Yu [124]	2018	X-ray CT	IQA	RMSE	SSIM is suitable for this task
Sushmit et al. [125]	2019	X-ray	IQA	PSNR	SSIM is suitable for this task
Islam et al. [126]	2019	X-ray	IQA	PSNR	SSIM is suitable for this task
Haiderbhai et al. [127]	2020	X-ray	IQA	RMSE and PSNR	SSIM is suitable for this task
Roy and Maity [128]	2020	X-ray	IQA	MSE, PSNR, and SNR	SSIM is suitable for this task
Saeed et al. [129]	2020	X-ray	IQA	RMSE, PSNR, and FSIM	SSIM is suitable for this task
Villarraga-Gómez and Smith [130]	2020	X-ray CT	IQA	RMSE and PSNR	SSIM is suitable for this task
Pourasad and Cavallaro [44] *	2021	X-ray	IQA	PSNR and MSE	SSIM is suitable for this task

* A detailed review is provided in Section 4.6.

exemplifies the purposes of using SSIM for X-ray medical images. As in Table 4, we describe the whole article using X-ray images conveying the SSIM metric as an IQM.

4.5. Optical Imaging

To complete the reviews of SSIM implementation in medical imaging systems, we presented a novel modality using optical imaging. Because optical imaging is a relatively new modality, acquiring SSIM applications in optical imaging is considered rare. Despite our best efforts, we found only two articles comprising SSIM as an IQM to measure the distorted medical images related to the reference images. We excluded optical imaging using SSIM for a scope other than the medical field, such as optical imaging for communication and simulations to predict the optical properties of a reconstruction. In 2020, SSIM along with MAE were utilized to examine images from a single snapshot of optical properties (SSOP) and the ground-truth images [131].

In 2021, SSIM was used for comparing the optical images of a mouse with an embedded tumor [132]. The article reported the application of machine learning, such as K-means and fuzzy c-means, to automatically cluster the tumor. They compared the results between manual clustering and automatic clustering with the aid of SSIM and the Dice coefficient to evaluate the similarity of two images. Here, SSIM performed satisfactorily in objectively examining the segmentation results through a manual approach, showing a similarity score of $0.9168$. In addition, when Gaussian noise was added to the background, SSIM demonstrated a robust performance by showing a stable similarity score. With the assistance of SSIM, the article reported that their clustering algorithms are robust to noise.

To conclude this subsection,

Table 5. Summary of the published articles encompassing SSIM for optical imaging. MAE, mean-absolute-error.

Study	Year	Modality	SSIM Implementation	Compared Matrix	Results
Aguénounon et al. [131]	2020	Optical imaging	IQA	MAE	SSIM is suitable for this task
Ren et al. [132]	2021	Optical imaging	IQA	Dice coefficient	SSIM is suitable for this task

represents the purposes of using SSIM for optical medical images. As in Table 5, we describe two articles using optical images assigning the SSIM metric as an IQM.

4.6. Current Status of SSIM Research in Medical Imaging

We identified that the SSIM applications evolve from only for IQA to be able to implement as a loss function in CNN, reducing metal artifact, and contour extractor. Therefore, this section describes the articles using SSIM for such goals with a “*” mark, as shown in Table 1, Table 2, Table 3 and Table 4.

4.6.1. Loss Function

The loss function along with the activation function, optimizer, and other hyperparameters in CNN is an essential parameter to predict the model architecture ability. A common loss function is MSE since it can provide simple mathematical expressions. In 2021, Jaubert et al. attempted to develop a CNN-based algorithm to suppress the artifact of phase-contrast cardiac MR images in real time. Here, they assigned MAE and SSIM as the loss function when training the model architecture. They called their CNN model U-Net MAE and U-Net SSIM [38].

To begin their study, they prepared the synthetic dataset using $520$ breath-hold, retrospectively cardiac gated, uniform density phase-contrast MR in the aortic position. The dataset was comprised of $40$ combinations between magnitude and phase subtracted images. They split the dataset into $490$ training images and 15 images were for validation and test dataset each. The ground-truth training dataset was obtained by interpolating the phase-contrast MR images to the target temporal resolution. To generate the corrupted images, the original dataset was processed using Fourier transform and undersampled utilizing the chosen trajectory. Then, with the inverse Fourier transform, they acquired the aliased data. The ground-truth and noise-contaminated images were normalized and cropped into $192\times 192\times 40$.

After the dataset preprocessing, they fed the dataset into the U-Net architecture for training. U-Net model consisted of max-pooling, transpose three-dimensional (3D) convolution, 3D convolution, and 1D convolution layers. They trained the model by implementing an adaptive moment estimation algorithm (Adam) as the optimizer, a batch size of $2$, an epoch of $100$, as well as with an initial learning rate of $0.0005$. To complete the configuration, they assigned two loss functions, namely MAE and SSIM (AvgSSIM). Especially for SSIM, the loss can be computed with the complex images over the real (real) and imaginary (imag) elements, as follows

$$AvgSSIM=\frac{SSIM\left(\frac{real+1}{2}\right)+SSIM\left(\frac{imag+1}{2}\right)}{2} ,$$

(39)

These loss functions were utilized to obtain the selected best performance of the U-Net with the most minimum loss calculated from the validation dataset. To measure the effectivity of their U-Net, MAE, PSNR, average SSIM, magnitude SSIM, and phase SSIM were applied.

According to their results, U-Net MAE and U-Net SSIM have a significant improvement to suppress the artifact when compared to the gridded images. Additionally, U-Net SSIM enhanced the images’ sharpness, whereas U-Net MAE was deemed blurrier. U-Net MAE was superior when the metric evaluations were MAE and PSNR; meanwhile, U-Net SSIM was exceptional in the average SSIM, magnitude SSIM, and phase SSIM. Moreover, they compared the U-Net MAE and U-Net SSIM using flow compensated, flow encoded, and combined images to obtain the denoised images. Both U-Net performed sufficiently with the magnitude images. However, U-Net SSIM was remarkable in denoising the phase images compared to U-Net MAE. With these results, U-Net SSIM outperformed U-Net MAE for the $20$ pediatric patients with congenital heart disease.

4.6.2. Reducing Metal Artifact

A metal artifact can exist due to the presence of high-density objects, i.e., dental fillings and surgical clips. In CT imaging, the presence of metal artifacts causes issue of complications in imaging the region of interest (ROI) and may reduce the dose calculation accuracy. In order to suppress the metal artifact, a robust metal artifact reduction (MAR) algorithm is vital to be established. Therefore, in 2019, Kim et al. tried to develop a MAR algorithm based on the tilted CT scan images. Their idea began from the understanding of the normal CT scan position may produce metal artifact contaminated images, and the tilted CT scan can complement the original CT scan position by offering reconstructed images with less metal artifact. Additionally, to obtain the less metal artifact images, they implemented modified SSIM by neglecting the luminance factor and only considering the contrast and structure components in the calculation. They claimed that the luminance factor containing intensity in the areas with the metal artifact between CT images and the superposition map is different completely, whereas the edge information is fairly identical. Thus, the correlation map calculated using the original SSIM may not accurately predict the metal artifact similarity score [99].

The algorithm steps are described in Algorithm 1 for MAR with SSIM.

Algorithm 1. MAR with SSIM

Input: Reconstructed original CT and tilted CT images Output: Reconstructed image with the smallest SSIM for reconstructed images of original CT and tilted CT do Subtract original CT and tilted CT images Obtain the artifact map $\mathrm{Modified}\mathrm{SSIM}\mathrm{calculation}\mathrm{with}\mathbf{5}\mathbf{\times }\mathbf{5}$ windows Obtain the original CT and tilted CT correlation maps Compare original CT and tilted CT correlation maps that contain SSIM score if original CT SSIM < tilted CT SSIM do Choose the intensity of the original CT image as the best reconstruction image Otherwise do Choose the intensity of the tilted CT image as the best reconstruction image End if End for

To show the superiority of their proposed MAR, they compared the proposed method with well-known linear interpolation metal artifact reduction (LI-MAR), normalized metal artifact reduction (NMAR), and refined metal artifact reduction (RMAR). They obtained an exceptional result when using MAR with SSIM by presenting a mean absolute percentage error (MAPE) of $8.52\%$, whereas the original CT scan, tilted CT scan, LI-MAR, NMAR, and RMAR exhibited $62.98\%$, $9.27\%$, $19.97\%$, $14.12\%$, and $11.79\%$, respectively, in the simulation study. Moreover, in the experiment study, they extended the proposed method by combining it with RMAR and generated two new methods, namely tilted CT-based MAR (T-MAR) and augmented tilted CT-based MAR (AT-MAR). Particularly when employing AT-MAR, their results indicated that the metal artifact suppression algorithm can reduce the existence of streaks and bright/dark band artifacts near metallic objects by demonstrating MAPE of $10.12\%$, whereas original CT scan, tilted CT scan, T-MAR, LI-MAR, NMAR, and RMAR had MAPE of $92.12\%$, $60.62\%$, $46.37\%$, $45.84\%$, $34.62\%$, and $29.93\%$, respectively.

4.6.3. Contour Extractor

Contour extraction for tongue image sequences may be beneficial for speech recognition, speech production, and swallowing study. The tongue images can be obtained utilizing the US modality. In 2016, Xu et al. compensated the manual tongue contour tracking by employing CW-SSIM for re-initializing the tongue contour position automatically [61]. They considered CW-SSIM due to its invariant with a few rotations, translations, and scaling distortions. They started the procedure by computing the similarity score between the reference image (the first frame image in this case) and the current frame image. In total, they can have a thousand images depending on the number of frames. Then, they set a threshold value of CW-SSIM for the purpose of re-initializing the contour position to the first frame if the similarity score was greater than the designed threshold value ($0.8$). Their results indicated that the CW-SSIM along with similarity constraint achieved the best performance by showing the less mean sum of distances (MSD) for three subjects, i.e., $3.36\pm 0.86$, $3.65\pm 1.02$, and $2.96\pm 0.95$.

4.6.4. Image Quality Assessment

Most of the SSIM implementation is for IQA/IQM since the initial objective of SSIM is to substitute the traditional measures, such as MSE and PSNR. Thus, in this subsubsection, we reviewed one article allocating SSIM as an IQA to complete the comprehensive insight regarding the SSIM prospect for medical image analysis.

The image compression and quality enhancement in medical image processing are substantial factors in reducing the content redundancy to attain the image optimal form when storing and transferring the images. Thus, Pourasad and Cavallaro in 2021 demonstrated two novel algorithms, namely lossless and lossy compression algorithms [44]. After they compressed the images, they applied several enhancement techniques to acquire the leverage image quality from the compressed images. To assess the algorithms’ performance, they analyzed them with MSE, PSNR, and SSIM. Moreover, lossy compression was used to execute DCT and DWT. Likewise, the lossless method with run-length encoding (RLE) and block truncation coding (BTC) was operated for the study. To achieve a satisfactory medical image quality, two spatial domain enhancement techniques have been used, such as adaptive histogram equalization (AHE) and morphological operations (MO).

Their study employed the medical images from MedPix^®, which is a free and open access medical images, teaching cases, and clinical topics database. This database has approximately $59,000$ medical images with $12,000$ patients case scenarios and $9000$ topics. Their findings indicated that neither AHE nor MO is not suitable for image quality enhancement because the SSIM and PSNR score is decreased after processing the image quality enhancement algorithms. However, their image compression methods are outstanding compared to the other studies, such as the joint photographic experts group (JPEG) and JPEG2000. The best compression can be achieved using DCT with an SSIM of $0.9761$.

4.7. Limitation

We attempted to collect the SSIM-related articles in the medical image analysis to provide a comprehensive review for researchers in guiding the SSIM application for the medical field. However, we limited our reviews by merely describing more detailed information about SSIM implementation as a loss function, MAR, contour extractor, and IQA. We did not describe the SSIM implementation as the segmentation and clustering algorithms as in [90,93] in detail. Additionally, for effectiveness, we reviewed one article on the IQA SSIM because SSIM-related articles are usually relative to IQA; thus, it is reasonable to review only one article representing the other IQA SSIM studies. Furthermore, by providing a brief review in Section 4.1, Section 4.2, Section 4.3, Section 4.4 and Section 4.5, we may hope this review is valuable for readers.

5. Future Potential of SSIM in Medical Image Analyses and Conclusions

This review offers comprehensive insight related to state-of-the-art SSIM approaches for medical imaging, as well as the potential medical image examination research methods that can be improved using SSIM. It is likely that, in the future, SSIM may be used not only for IQM but also for segmentation, clustering, classification, image fusion, and object detection in regard to image processing in medical imaging, and as a feature extractor and loss function relative to the deep learning method [133,134,135]. Furthermore, SSIM can be extended to CAD for assisting physicians and clinicians in their predictions and for providing conclusions when diagnosing a specific disease based on the work of radiologists. Here, SSIM can act as a second opinion to improve the confidence of radiologists. In addition, using an objective measure, SSIM can be efficient in terms of cost and time consumption. However, we must give more attention to the drawbacks of SSIM. As reviewed, the original SSIM showed some shortcomings when evaluating badly blurred images, scaling, rotation, and spatial translation images. Nevertheless, these issues can be overcome by implementing an improved SSIM, such as MS-SSIM, three- and four-component weighted SSIMs, GSSIM, CW-SSIM, ISSIM-S, and other types of SSIM. SSIM has been identified as easy to use and improve because, based on image processing knowledge, it can be upgraded based on the goal of a particular study. Therefore, future studies should address the SSIM implementation by focusing on the improved SSIM for obtaining the optimal SSIM model when assigning it as an IQM or other image processing approach in medical imaging.