CycleDiffusion: Voice Conversion Using Cycle-Consistent Diffusion Models

Abstract

Voice conversion (VC) refers to the technique of modifying one speaker’s voice to mimic another’s while retaining the original linguistic content. This technology finds its applications in fields such as speech synthesis, accent modification, medicine, security, privacy, and entertainment. Among the various deep generative models used for voice conversion, including variational autoencoders (VAEs) and generative adversarial networks (GANs), diffusion models (DMs) have recently gained attention as promising methods due to their training stability and strong performance in data generation. Nevertheless, traditional DMs focus mainly on learning reconstruction paths like VAEs, rather than conversion paths as GANs do, thereby restricting the quality of the converted speech. To overcome this limitation and enhance voice conversion performance, we propose a cycle-consistent diffusion (CycleDiffusion) model, which comprises two DMs: one for converting the source speaker’s voice to the target speaker’s voice and the other for converting it back to the source speaker’s voice. By employing two DMs and enforcing a cycle consistency loss, the CycleDiffusion model effectively learns both reconstruction and conversion paths, producing high-quality converted speech. The effectiveness of the proposed model in voice conversion is validated through experiments using the VCTK (Voice Cloning Toolkit) dataset.

1. Introduction

Voice conversion (VC) is the process of converting the speech of one speaker into the speech of another speaker, while retaining the linguistic information of the utterance [1,2]. Application areas for voice conversion include speech synthesis, accent conversion, medicine, security, privacy, and entertainment. VC typically consists of three steps: feature extraction, feature transformation, and waveform reconstruction. Since spectrograms are typically used as the extracted speech features and neural vocoders are utilized for waveform reconstruction, most recent research efforts focus on feature transformation through deep neural networks (DNNs), which convert the spectrogram of a source speaker’s utterance into that of a target speaker’s utterance.

The variational autoencoder (VAE) [3] is one of the first deep learning models to be applied to VC [4,5,6]. A VAE consists of an encoder and a decoder. The encoder of the VAE is trained to generate the low-dimensional latent vectors that capture only the linguistic information from the input speech feature vectors. The decoder of the VAE is trained to reconstruct the original input by incorporating the input speaker information into the latent vectors. Voice conversion is accomplished by providing the desired target speaker information rather than the input speaker information to the latent vectors. Although it enables many-to-many voice conversion using non-parallel training data, which does not require the same linguistic content spoken by different speakers, the quality of the converted speech is somewhat lower compared to the methods based on generative adversarial networks (GANs) [7].

Since the cycle-consistent adversarial network (CycleGAN) [8] was introduced for image-to-image translation, GAN-based voice conversion has been extensively studied [9,10,11,12,13,14,15,16]. A CycleGAN comprises two GANs. One GAN is trained to transform the voice of the source speaker into that of the target speaker, while the other GAN is trained to revert the target speaker’s voice back to the original source speaker’s voice. This approach utilizes non-parallel training data and, unlike VAEs, it shows superior quality in converted speech by learning voice conversion paths explicitly, instead of just input reconstruction paths. Nonetheless, GANs suffer from training instability and mode collapse, and building many-to-many voice conversion models with CycleGANs involves significant computational complexity.

Recently, diffusion models (DMs) [17,18,19,20,21,22,23,24] have emerged as promising generative models. A DM is trained to reconstruct the original input data by progressively adding noise to the input data through a forward diffusion process, resulting in a noise distribution, and then progressively removing the noise through a reverse diffusion process. DMs are characterized by stability during training and demonstrate excellent performance in generating high-quality data via reverse diffusion. DMs have been applied not only to image generation but also to voice conversion [25,26,27]. However, DMs fundamentally learn only the reconstruction paths, similar to VAEs, rather than the conversion paths as GANs do, leaving much to be improved in the quality of the converted speech. To address this limitation of conventional DMs and enhance the quality of converted speech, we propose a cycle-consistent diffusion (CycleDiffusion) model, which consists of two DMs: one to convert the source speaker’s voice to the target speaker’s voice, and the other to revert the target speaker’s voice back to the source speaker’s voice. By employing two DMs and enforcing a cycle consistency loss, the proposed CycleDiffusion model can effectively learn not only the reconstruction paths but also the conversion paths, leading to high-quality converted speech. The effectiveness of the proposed model for voice conversion is experimentally validated using VCTK (Voice Cloning Toolkit) data.

The contributions of this paper are as follows:

• We propose a novel CycleDiffusion model that can learn both the reconstruction and conversion paths.
• We provide an efficient training algorithm so that the proposed CycleDiffusion model can be reliably trained.
• The proposed CycleDiffusion model is applied to voice conversion without parallel training data.
• We demonstrate the usefulness of the proposed method using VCTK data.

The rest of the paper is structured as follows. Section 2 reviews related previous studies. Section 3 introduces the proposed CycleDiffusion model. Section 4 analyzes the experimental results. Finally, Section 5 concludes the paper. Nomenclature summarizes the notations used throughout the paper.

2. Related Works

In this section, we review the DM-based voice conversion method [23] and the CycleGAN-based voice conversion method [12].

2.1. Diffusion Model-Based Voice Conversion

A DM involves two processes: forward diffusion and reverse diffusion. The forward diffusion process progressively introduces Gaussian noise into the input data, whereas the reverse diffusion process attempts to eliminate this noise. In the context of voice conversion, these forward and reverse diffusions can be represented by the following stochastic differential equations (SDEs):

$$\mathrm{d}{\mathbf{x}}_t={\displaystyle \frac{1}{2}}{\beta }_t\left(\overline{\mathbf{x}}-{\mathbf{x}}_t\right)\mathrm{d}t+\sqrt{{\beta }_t}\mathrm{d}{\mathbf{w}}_t,\mathrm{forward}\mathrm{diffusion}$$

(1)

$$\mathrm{d}{\tilde{\mathbf{x}}}_t=\left({\displaystyle \frac{1}{2}}\left(\overline{\mathbf{x}}-{\tilde{\mathbf{x}}}_t\right)-\nabla \log p\left({\tilde{\mathbf{x}}}_t\right)\right){\beta }_t\mathrm{d}t+\sqrt{{\beta }_t}\mathrm{d}{\tilde{\mathbf{w}}}_t,\mathrm{reverse}\mathrm{diffusion}$$

(2)

where ${\mathbf{x}}_t$ is a forward-time stochastic process at time $t$ with ${\mathbf{x}}_0$ being an input speech feature vector, $\overline{\mathbf{x}}$ is the linguistic information contained in ${\mathbf{x}}_0$, ${\tilde{\mathbf{x}}}_t$ is a reverse-time stochastic process at time $t$ with ${\tilde{\mathbf{x}}}_0$ being a reconstructed speech feature vector, $\mathbf{w}$ and $\tilde{\mathbf{w}}$ are the forward- and reverse-time Wiener processes, respectively; ${\beta }_t$ is a noise schedule, and $t\in \left[0,T\right]$ is a continuous time variable ($T$ is generally set to 1). When ${\tilde{\mathbf{x}}}_T$ is the linguistic information contained in ${\mathbf{x}}_0$ without any speaker details, the reverse-time stochastic process, ${\tilde{\mathbf{x}}}_t$, gradually reconstructs the input speech feature vector, incorporating the input speaker characteristics. The reverse-time Wiener process, $\tilde{\mathbf{w}}$, mirrors the forward-time Wiener process, except that it operates with time flowing backward, from $T$ to 0.

A DM is trained to minimize the discrepancy between the trajectories of the forward and reverse diffusion processes, enforcing ${\tilde{\mathbf{x}}}_0$ similar to ${\mathbf{x}}_0$, as an autoencoder. This can be accomplished as follows. First, note that the forward diffusion SDE, Equation (1), yields the following derivation:

$$p\left({\mathbf{x}}_t|{\mathbf{x}}_0\right)=\mathcal{N}\left({\mathbf{x}}_t;{\alpha }_t{\mathbf{x}}_0+\left(1-{\alpha }_t\right)\overline{\mathbf{x}},\left(1-{\alpha }_t^2\right)\mathbf{I}\right),$$

(3)

where ${\alpha }_t$ is dependent on the noise schedule. Consequently, $p\left({\mathbf{x}}_t|{\mathbf{x}}_0\right)$ evolves into $\mathcal{N}\left({\mathbf{x}}_t;\overline{\mathbf{x}},\mathbf{I}\right)$ for a noise schedule where ${\alpha }_t$ approaches $-\infty$, meaning that the final latent vector ${\mathbf{x}}_T$ contains only linguistic information without any speaker information. Since $p\left({\mathbf{x}}_t|{\mathbf{x}}_0\right)$ is a Gaussian distribution, the gradient of the log probability density with respect to the data (known as the score) can be easily computed. Then, $\nabla \log p\left({\tilde{\mathbf{x}}}_t\right)$ in the reverse diffusion SDE, Equation (2), is replaced with an estimation obtained by minimizing the following loss function:

$${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\left({\mathbf{x}}_0^{\zeta },\zeta \right)={\mathbb{E}}_t\left[\left(1-{\alpha }_t^2\right){\mathbb{E}}_{{\mathbf{x}}_0^{\zeta }}\left[{\mathbb{E}}_{{\mathbf{x}}_t^{\zeta }|{\mathbf{x}}_0^{\zeta }}\left[{‖{\mathcal{S}}_{\theta }\left({\mathbf{x}}_t^{\zeta },\overline{\mathbf{x}},\zeta ,t\right)-\nabla \log p\left({\mathbf{x}}_t^{\zeta }|{\mathbf{x}}_0^{\zeta }\right)‖}_2^2\right]\right]\right],$$

(4)

where ${\mathbf{x}}_0^{\zeta }$ represents an input speech feature vector from speaker $\zeta$, ${\mathcal{S}}_{\theta }$ indicates the estimated score function implemented using a DNN with parameters $\theta$, and ${‖·‖}_2$ denotes the ℓ ${\mathcal{l}}^2$ norm. An encoder is separately trained to extract speaker-independent linguistic information $\overline{\mathbf{x}}$ from the input speech feature vector ${\mathbf{x}}_0^{\zeta }$.

Once the score function ${\mathcal{S}}_{\theta }$ is well trained, voice conversion from speaker $\zeta$ to speaker $\xi$, for the input speech feature vector ${\mathbf{x}}_0^{\zeta }$ from speaker $\zeta$, is achieved by sampling ${\mathbf{x}}_T^{\zeta }$ via Equation (3),

$${\mathbf{x}}_T^{\zeta }=\mathcal{F}\left({\mathbf{x}}_0^{\zeta }\right) ~ p\left({\mathbf{x}}_T^{\zeta }|{\mathbf{x}}_0^{\zeta }\right),$$

(5)

where $\mathcal{F}\left({\mathbf{x}}_0^{\zeta }\right)$ represents the result of forward diffusion process at time $T$, and solving the reverse diffusion SDE to generate the converted speech feature vector ${\tilde{\mathbf{x}}}_0^{\zeta \to \xi }$ of speaker $\xi$ as follows:

$${\tilde{\mathbf{x}}}_0^{\zeta \to \xi }=\mathcal{R}\left({\mathbf{x}}_T^{\zeta },\xi \right),$$

(6)

where $\mathcal{R}\left({\mathbf{x}}_T^{\zeta },\xi \right)$ denotes the solution to Equation (2) at time $t=0$ with the score function replaced by ${\mathcal{S}}_{\theta }\left({\tilde{\mathbf{x}}}_t^{\xi },\overline{\mathbf{x}},\xi ,t\right)$ and ${\mathbf{x}}_T^{\zeta }$ used instead of ${\mathbf{x}}_T^{\xi }$. Note that the speaker embedding for the source speaker $\zeta$ is replaced with the target speaker $\xi$ during inference.

One limitation of DM-based voice conversion is that the score function ${\mathcal{S}}_{\theta }$ is trained to reconstruct the input, as shown in Equation (4), while it is utilized in Equation (6) to convert the input speech from a source speaker to a target speaker. Therefore, it is preferable to train ${\mathcal{S}}_{\theta }$ for conversion rather than just reconstruction.

2.2. CycleGAN-Based Voice Conversion

The CycleGAN-based voice conversion technique employs two GANs; i.e., two sets of generator and discriminator pairs. One generator transforms the speech feature vectors from the source speaker to those of the target speaker, while the other does the reverse. During training, the adversarial loss from the discriminators ensures the correct conversion of speaker characteristics in the feature vectors, while the cycle consistency loss between the input to the first generator and the output of the second generator ensures that the linguistic content of the speech feature vectors remains unchanged.

For instance, an input speech feature vector ${\mathbf{x}}^{\zeta }$ (omitting the time index subscript hereafter for brevity) from speaker $\zeta$ is transformed into ${\mathbf{x}}^{\zeta \to \xi }$ using the first generator ${\mathcal{G}}_1$:

$${\mathbf{x}}^{\zeta \to \xi }={\mathcal{G}}_1\left({\mathbf{x}}^{\zeta }\right).$$

(7)

The first discriminator ${\mathcal{D}}_1\left({\mathbf{x}}^{\zeta \to \xi }\right)$ provides the adversarial loss for ${\mathbf{x}}^{\zeta \to \xi }$ so that the speaker characteristic in the feature vector changes from speaker $\zeta$ to speaker $\xi$. The converted feature vector ${\mathbf{x}}^{\zeta \to \xi }$ is then transformed back to ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ by the second generator ${\mathcal{G}}_2$:

$${\mathbf{x}}^{\zeta \to \xi \to \zeta }={\mathcal{G}}_2\left({\mathbf{x}}^{\zeta \to \xi }\right),$$

(8)

which reverts the speaker characteristic in the feature vector from speaker $\xi$ back to speaker $\zeta$. The second discriminator ${\mathcal{D}}_2\left({\mathbf{x}}^{\zeta \to \xi \to \zeta }\right)$ provides the adversarial loss for ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$. The following cycle consistency loss between the original input vector ${\mathbf{x}}^{\zeta }$ and the reconstructed vector ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ maintains the preservation of linguistic content:

$${\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}\left({\mathbf{x}}^{\zeta },{\mathbf{x}}^{\zeta \to \xi \to \zeta }\right)={‖{\mathbf{x}}^{\zeta }-{\mathbf{x}}^{\zeta \to \xi \to \zeta }‖}_1,$$

(9)

where ${‖·‖}_1$ represents the ${\mathcal{l}}^1$ norm.

Similarly, an input speech feature vector ${\mathbf{x}}^{\xi }$ from speaker $\xi$ is converted to ${\mathbf{x}}^{\xi \to \zeta }$ and then back again to ${\mathbf{x}}^{\xi \to \zeta \to \xi }$ by the generators ${\mathcal{G}}_2$ and ${\mathcal{G}}_1$, respectively, providing another cycle consistency loss. The discriminators ${\mathcal{D}}_2$ and ${\mathcal{D}}_1$ provide the adversarial losses for ${\mathbf{x}}^{\xi \to \zeta }$ and ${\mathbf{x}}^{\xi \to \zeta \to \xi }$, respectively. By using both adversarial loss and cycle consistency loss together, a CycleGAN can directly learn the conversion path without the need for parallel training data.

The recently proposed MaskCycleGAN-VC [12], which is a variant of CycleGAN-based voice conversion, and wav2wav [16], which utilizes cycle consistency for wave-to-wave voice conversion, have demonstrated high performance. However, GANs are prone to training instability and mode collapse, and developing many-to-many voice conversion models with CycleGANs entails substantial computational complexity.

3. Proposed Method

3.1. Cycle-Consistent Diffusion (CycleDiffusion)

As detailed in Section 2.1, conventional DM-based voice conversion learns only the reconstruction path as a VAE does and does not learn the conversion path, resulting in significant room for improvement in the quality of the converted speech. Therefore, we propose to utilize cycle consistency loss for training diffusion models, referred to as cycle-consistent diffusion (or CycleDiffusion for short). The proposed method aims to learn both the reconstruction path and the conversion path while enforcing that linguistic information remains unchanged.

Figure 1 depicts the proposed CycleDiffusion training for a two-speaker scenario. Initially, the DM is trained to reconstruct the input. The input ${\mathbf{x}}^{\zeta }$ from speaker $\zeta$ is reconstructed to ${\tilde{\mathbf{x}}}^{\zeta }$, and the input ${\mathbf{x}}^{\xi }$ from speaker $\xi$ is reconstructed to ${\tilde{\mathbf{x}}}^{\xi }$, which is typical DM training using ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\left({\mathbf{x}}^{\zeta },\zeta \right)$ and ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\left({\mathbf{x}}^{\xi },\xi \right)$, respectively. As the usual reconstruction path training stabilizes, the conversion path training commences as follows. The input ${\mathbf{x}}^{\zeta }$ is converted to ${\mathbf{x}}^{\zeta \to \xi }$ through Equations (5) and (6):

$${\mathbf{x}}^{\zeta \to \xi }=\mathcal{R}\left(\mathcal{F}\left({\mathbf{x}}^{\zeta }\right),\xi \right).$$

(10)

Subsequently, the converted speech vector ${\mathbf{x}}^{\zeta \to \xi }$ is reverted to ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ again via Equations (5) and (6):

$${\mathbf{x}}^{\zeta \to \xi \to \zeta }=\mathcal{R}\left(\mathcal{F}\left({\mathbf{x}}^{\zeta \to \xi }\right),\zeta \right).$$

(11)

Finally, the cycle consistency loss between ${\mathbf{x}}^{\zeta }$ and ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ is minimized using ${\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}\left({\mathbf{x}}^{\zeta },{\mathbf{x}}^{\zeta \to \xi \to \zeta }\right)$. Similarly, the cycle consistency loss between ${\mathbf{x}}^{\xi }$ and ${\mathbf{x}}^{\xi \to \zeta \to \xi }$ is minimized as well.

This cycle consistency loss minimization helps improve voice conversion performance in two aspects. First, the conversion path from speaker $\xi$ to speaker $\zeta$ is explicitly trained by reducing the difference between the original target speaker’s speech ${\mathbf{x}}^{\zeta }$ and the converted speech ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$, which originates from the source speaker’s speech ${\mathbf{x}}^{\zeta \to \xi }$. That is, the input ${\mathbf{x}}^{\zeta \to \xi }$ comes from speaker $\xi$ and the output ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ belongs to speaker $\zeta$. Similarly, the conversion path from speaker $\zeta$ to speaker $\xi$ is explicitly trained as well. Second, since both ${\mathbf{x}}^{\zeta }$ and ${\mathbf{x}}^{\zeta \to \xi \to \zeta }$ (as well as ${\mathbf{x}}^{\xi }$ and ${\mathbf{x}}^{\xi \to \zeta \to \xi }$) should represent the same utterance, minimizing their differences ensures that the linguistic information remains unchanged.

The overall loss function for CycleDiffusion training is as follows:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\left({\mathbf{x}}^{\zeta },\zeta \right)+{\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{f}\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}}\left({\mathbf{x}}^{\xi },\xi \right)+{\lambda }_i\left({\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}\left({\mathbf{x}}^{\zeta },{\mathbf{x}}^{\zeta \to \xi \to \zeta }\right)+{\mathcal{L}}_{\mathrm{c}\mathrm{y}\mathrm{c}\mathrm{l}\mathrm{e}}\left({\mathbf{x}}^{\xi },{\mathbf{x}}^{\xi \to \zeta \to \xi }\right)\right),$$

(12)

where ${\lambda }_i$ is a weight schedule for the cycle consistency loss, starting at 0 and gradually increasing to ${\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}$ as the training progresses. This approach can be easily extended to accommodate multiple speakers.

3.2. Training Algorithm

A pseudo-code of CycleDiffusion training for many-to-many voice conversion is shown in Algorithm 1, where $\eta$ denotes the learning rate. The decrease in loss values may serve as a termination criterion for the algorithm.

3.3. Comparison to Similar Works

The concept of cycle consistency for DMs was presented in a document enhancement task [28]. Nevertheless, it did not employ the cycle consistency loss for DM training. Rather, two independently trained DMs—one in the noisy document domain and the other in the clean document domain—were linked at the latent space during inference, enabling cyclic conversion. The cycle consistency loss for DM training was initially explored in [29] and applied to image-to-image translation tasks. CycleDiffusion differs from [29] in that it uses score-based diffusion models, which are a continuous-time version of DMs, and is applied to voice conversion tasks; whereas [29] used denoising diffusion probabilistic models, which are a discrete-time version of DMs. To the best of the authors’ knowledge, CycleDiffusion is the first work employing cycle consistency loss for score-based DM training in voice conversion tasks.

4. Experiments

To evaluate the effectiveness of the proposed method, we used a subset of the VCTK dataset [30], which is commonly employed in voice conversion research. This subset comprised two female speakers (P236 and P239) and two male speakers (P259 and P263), who served as both source and target speakers. For simplicity, we will refer to these speakers as F1, F2, M1, and M2, respectively, in the subsequent discussion. Each speaker contributed 461 training utterances and 10 test utterances. With 12 conversion directions and 9 target speaker embedding vectors utilized for each speaker, the overall total of testing samples amounted to 1080 utterances.

We utilized 80-dimensional mel-spectrograms as the speech feature vectors, which were extracted from the downsampled speech waveforms at 22.05 kHz. These spectrograms were computed every 256 samples using a Hanning window of 1024 samples in length. For the mel-spectrogram inversion to waveforms, we employed a pre-trained universal HiFi-GAN vocoder [31].

An adam optimizer with a learning rate of 0.00003 was used for the stochastic gradient descent training in lines 6, 12, and 15 of Algorithm 1. The total number of training epochs was set to 270 after monitoring the reduction of loss values. Following [25], a noise schedule of ${\beta }_t={\beta }_0+\left({\beta }_T-{\beta }_0\right)t$ with ${\beta }_0=0.05$ and ${\beta }_T=20.0$ was employed.

In a preliminary experiment, we compared two cycle consistency training schemes: the first scheme updated all parameters in the dotted and dashed lines in Figure 1, and the second scheme updated the parameters in the dashed lines only. We found that both schemes yielded almost similar performance, but the second method was faster. Therefore, during the training of parameters via gradient descent of the cycle consistency loss, the parameters associated with the dotted lines are frozen, and those involved in the dashed lines are adjusted for faster training. To determine when the conversion path training should become effective in Algorithm 1, we conducted multiple experiments with varying starting points. It was generally observed that initiating the conversion path training yields better results when the reconstruction path training has converged to a certain extent. Therefore, ${\lambda }_i$ was set 0 for $i\in \left[\mathrm{1,50}\right]$ and 1 for $i\in \left[\mathrm{51,270}\right]$.

DiffVC [23], recognized as the leading VC method using DMs, served as a baseline for comparison. To ensure fair comparisons, the same pretrained encoder was used for both DiffVC and CycleDiffusion.

4.1. Objective Evaluation

4.1.1. Speaker Similarity Test

One of the most widely employed methods to quantify the similarity between two speakers’ voices used in the field of speaker recognition is cosine similarity. This metric is defined as the inner product of two vectors divided by the product of their magnitudes; i.e., the cosine of the angle between them. In speaker recognition, i-vectors [32] or x-vectors [33] are typically utilized to characterize a speaker. We used the Kaldi toolkit [34] to extract both i-vectors and x-vectors, and calculated the cosine similarities between the converted and target speeches. A higher cosine similarity value indicates better voice conversion performance.

Table 1. Average cosine similarity values along with the 95% confidence intervals for the speech samples converted by DiffVC (baseline) and CycleDiffusion (proposed).

Method	DiffVC	CycleDiffusion
i-vector	0.4850 ± 0.0049	0.5376 ± 0.0047
x-vector	0.8909 ± 0.0033	0.9070 ± 0.0020
Average	0.6880 ± 0.0041	0.7223 ± 0.0034

summarizes the average cosine similarity values along with the corresponding 95% confidence intervals for the converted speeches generated by DiffVC and CycleDiffusion.

The cosine similarity scores for the i-vectors demonstrated a modest enhancement of 10.8% (from 0.4850 to 0.5376). The cosine similarity using the x-vectors showed only marginal improvement. However, the confidence interval contracted by 39.4% (from 0.0033 to 0.0020), indicating a sharper distribution and therefore diminished variance across speakers. On average, the proposed method outperformed the baseline system in the speaker similarity tests using both i-vectors and x-vectors. This suggests that incorporating cycle consistency loss into the diffusion model-based voice conversion training process is an effective strategy for enhancing speaker similarity and achieving more consistent results.

4.1.2. Linguistic Information Preservation Test

Effective voice conversion requires not only a change in speaker characteristics but also the preservation of linguist information. Automatic speech recognition (ASR) can be utilized to quantify how well the linguistic information is maintained in the converted speech by measuring the speech recognition accuracy of the converted speech. We used a publicly available ASR system—Whisper [35]—to measure the speech recognition accuracies. To account for the imperfections of ASR technologies, the speech recognition accuracy of the converted speech is calculated by comparing it against the recognized text of the original input speech, which serves as the reference transcription. An accuracy of 100% signifies complete preservation of the linguistic information, whereas an accuracy of 0% indicates that all linguistic information has been lost during the voice conversion process.

Table 2. Average speech recognition accuracies along with the 95% confidence intervals for the speech samples converted by DiffVC (baseline) and CycleDiffusion (proposed).

Method	DiffVC	CycleDiffusion
ASR	71.3 ± 1.4	74.4 ± 1.4

summarizes the average speech recognition accuracies, along with 95% confidence intervals for the converted speeches generated by DiffVC and CycleDiffusion.

The ASR system achieved a recognition rate of 71.3% on the speech transformed by DiffVC, while the speech converted through CycleDiffusion yielded a superior recognition rate of 74.4%, resulting in a statistically significant relative error reduction rate of 10.8%. The higher speech recognition accuracy achieved by CycleDiffusion can be credited to enhanced intelligibility and better preservation of linguistic information, facilitated by the integration of cycle consistency loss during the training process.

4.1.3. Mel-Cepstral Distance Test

For the last metric of objective evaluation, we used the mel-cepstral distance (MCD) [36] as a quantification of the dissimilarity between the converted speech and the target speech. To compensate for the length discrepancies between the converted and target speeches, dynamic time warping (DTW) was applied before calculating the MCD. The cosine similarity serves as a measure of similarity, while the speech recognition accuracy assesses the quality of linguistic content preservation, and MCD measures overall similarity. A lower MCD value between the converted and the target speeches indicates better voice conversion performance.

Table 3. Average MCDs along with the 95% confidence intervals for the speech samples converted by DiffVC (baseline) and CycleDiffusion (proposed).

Conversion Direction		DiffVC	CycleDiffusion
Intra-gender	F1$\to$F2	5.10 ± 0.24	4.18 ± 0.13
	F2$\to$F1	7.18 ± 0.46	6.28 ± 0.38
	M1$\to$M2	5.25 ± 0.20	4.52 ± 0.17
	M2$\to$M1	6.28 ± 0.28	5.41 ± 0.17
	Average	5.95 ± 0.29	5.10 ± 0.22
Inter-gender	$\mathrm{F}1\to$M1	6.12 ± 0.21	4.98 ± 0.15
	$\mathrm{F}1\to$M2	5.16 ± 0.20	4.32 ± 0.13
	$\mathrm{F}2\to$M1	5.87 ± 0.26	4.85 ± 0.17
	$\mathrm{F}2\to$M2	4.88 ± 0.20	4.33 ± 0.18
	$\mathrm{M}1\to$F1	7.10 ± 0.38	6.49 ± 0.42
	$\mathrm{M}1\to$F2	5.07 ± 0.20	4.36 ± 0.13
	$\mathrm{M}2\to$F1	7.16 ± 0.38	6.58 ± 0.37
	$\mathrm{M}2\to$F2	5.65 ± 0.26	4.80 ± 0.17
	Average	5.88 ± 0.26	5.09 ± 0.21
Average		5.90 ± 0.27	5.09 ± 0.22

summarizes the average MCDs along with the corresponding 95% confidence intervals for the converted speeches generated by DiffVC and CycleDiffusion.

The experimental findings outlined in Table 3 reveal that the proposed CycleDiffusion model surpasses the baseline DiffVC in both intra-gender and inter-gender voice conversion tasks, making a 15.9% increase in MCD on average. The CycleDiffusion model consistently achieves lower MCD values regardless of conversion directions, indicating improved quality in the converted speech. This decrease in the MCD values highlights the essential role of incorporating cycle consistency loss into the diffusion-based voice conversion framework, significantly improving conversion performance compared to the baseline.

4.2. Subjective Evaluation

Subjective evaluations of sound quality and similarity for the baseline DiffVC and the proposed CycleDiffusion were carried out using the pretrained MOSNet [37], which predicts the human mean opinion score (MOS) rating. The MOS scale ranges from 0 to 5, with 5 indicating the most natural speech and 0 indicating the least natural speech. MOSNet is designed to simulate a human evaluator conducting MOS assessments on input speech samples.

Table 4. Average MOS values, along with the 95% confidence intervals, produced by MOSNet for the speech samples converted by DiffVC (baseline) and CycleDiffusion (proposed).

Conversion Direction		DiffVC	CycleDiffusion
Intra-gender	F1$\to$F2	3.90 ± 0.07	4.16 ± 0.03
	F2$\to$F1	3.26 ± 0.06	3.19 ± 0.03
	M1$\to$M2	3.75 ± 0.07	4.22 ± 0.03
	M2$\to$M1	3.28 ± 0.05	3.41 ± 0.07
	Average	3.55 ± 0.06	3.74 ± 0.04
Inter-gender	$\mathrm{F}1\to$M1	3.13 ± 0.03	3.22 ± 0.04
	$\mathrm{F}1\to$M2	3.76 ± 0.07	4.25 ± 0.03
	$\mathrm{F}2\to$M1	3.09 ± 0.03	3.21 ± 0.04
	$\mathrm{F}2\to$M2	3.71 ± 0.08	4.18 ± 0.04
	$\mathrm{M}1\to$F1	3.18 ± 0.04	3.13 ± 0.03
	$\mathrm{M}1\to$F2	3.91 ± 0.07	4.10 ± 0.04
	$\mathrm{M}2\to$F1	3.13 ± 0.04	3.12 ± 0.04
	$\mathrm{M}2\to$F2	3.93 ± 0.08	4.20 ± 0.04
	Average	3.48 ± 0.06	3.68 ± 0.04
Average		3.50 ± 0.06	3.70 ± 0.04

summarizes the mean MOS values and the corresponding 95% confidence intervals produced by MOSNet for the converted speech outputs generated by DiffVC and CycleDiffusion.

The proposed method surpasses the baseline system in both intra-gender and inter-gender conversion scenarios. It is especially effective in enhancing the quality of voices that already perform well with the existing method (MOS of 3.5 or higher). For example, in the case of F1$\to$M2 conversion, the MOS value increased by approximately 0.49 (from 3.76 to 4.25), while in the case of M2$\to$F2 conversion, it improved by 0.27 (from 3.93 to 4.20). In some cases where the existing method struggled, that is, with MOS values of 3.2 or lower, such as in the case of M1$\to$F1 conversion, it decreased slightly by 0.05 (from 3.18 to 3.13). Despite these few less favorable cases of minor decreases, the average MOS values showed a significant improvement of approximately 5.7% (from 3.50 to 3.70), demonstrating that the voices generated by the proposed method sound more natural. It is worth mentioning that, in addition to the improvement in the average MOS values, the confidence interval decreased by 32.1% on average (from 0.06 to 0.04), indicating less variance in voice conversion performance among different speakers.

4.3. Spectrogrmas

Figure 2 shows the sample spectrograms of the utterances converted by DiffVC and CycleDiffusion, respectively. As demonstrated in the figure, the spectrograms of the utterances processed by CycleDiffusion show more distinct and well-defined formant structures in comparison to those generated by DiffVC.

5. Conclusions

DMs have emerged as a promising approach in the field of voice conversion, leveraging their ability to generate high-quality, diverse data through a two-phase process of forward diffusion and reverse denoising. They ensure stability and quality in data generation compared to traditional methods such as VAEs and GANs. Although DMs for voice conversion have shown substantial potential, conventional DM-based voice conversion methods only learn the reconstruction path, similar to VAEs, and fail to learn the conversion path, leaving much room for improvement in the converted speech quality. To address this, we have proposed CycleDiffusion, which leverages cycle consistency loss for training DMs. The proposed approach aims to learn both the reconstruction and conversion paths, while ensuring that linguistic information remains unchanged.

Both objective and subjective evaluations confirmed that the proposed method significantly enhanced the quality of the transformed speech. In the objective assessment of the speaker similarity, CycleDiffusion achieved an improvement ranging from 10.8% to 38.7%. In the objective evaluation of linguistic information retention, it realized a 10.8% error reduction in speech recognition. During the traditional MCD assessment, it noted a 15.9% enhancement. Finally, in the subjective assessment conducted using MOSNet, the proposed method demonstrated a 5.7% increase in the average MOS value and a 32.1% reduction in the confidence interval.

The time complexity of training the proposed method grows quadratically with respect to the number of training speakers. To reduce the computational burden, we used a subset of each batch when training the conversion paths. In future research, we plan to analyze the trade-off between the amount of data used for conversion path training and the resulting voice conversion performance, aiming to develop an efficient conversion path training strategy. Additionally, another future direction could be integrating cycle consistency loss into a hybrid model that combines DM and GAN.