Dual-Channel Cosine Function Based ITD Estimation for Robust Speech Separation

Abstract

In speech separation tasks, many separation methods have the limitation that the microphones are closely spaced, which means that these methods are unprevailing for phase wrap-around. In this paper, we present a novel speech separation scheme by using two microphones that does not have this restriction. The technique utilizes the estimation of interaural time difference (ITD) statistics and binary time-frequency mask for the separation of mixed speech sources. The novelties of the paper consist in: (1) the extended application of delay-and-sum beamforming (DSB) and cosine function for ITD calculation; and (2) the clarification of the connection between ideal binary mask and DSB amplitude ratio. Our objective quality evaluation experiments demonstrate the effectiveness of the proposed method.

1. Introduction

A common example of the well-known ‘cocktail party’ problem is the situation in which the voices of two speakers overlap. How to solve the ‘cocktail party’ problem and obtain an enhanced voice of a particular speaker in machines have grabbed serious attention of researchers.

As for single-channel speech separations, independent component analysis (ICA) [1] and nonnegative-matrix factorization (NMF) [2] are the conventional methods. However, the assumption that signals are statistically independent in ICA and the model in NMF is linear limit their applications. Moreover, NMF generally requires a large amount of computation to determine the speaker independent basis. Recently, in [3], the authors proposed an online adaptive process independent of parameter initialization, with noise reduction as a pre-processing step. Using adaptive parameters computed frame-by-frame, this article constructs a Time Frequency (TF) mask for the separation process. In [4], the authors proposed a pseudo-stereo mixture model by reformulating the binaural blind speech separation algorithm for the monaural speech separation problem. The algorithm estimates the source characteristics and constructs the masks with the parameters estimated through a weighted complex 2D histogram.

Normally, multiple channel sources are separated by measuring the differences of arrival time and sound intensity between microphones [5,6], which are also referred to as the interaural time differences (ITD) and the interaural intensity differences (IID). Interaural phase differences (IPD) have been used in [7,8]. The authors proposed a speech enhancement algorithm that utilizes phase-error based filters that depend only on the phase of the signals. Performances of the above systems depend on how the ITD (or IPD) threshold is selected. Instead of a fixed threshold, in [9], the authors employed a statistical modeling of angle distributions together with a channel weighting to determine which signal components belong to the target signal and which components are part of the background. In [10], the authors proposed a method based on a prediction of the coherence function and then estimated the signal to noise ratio (SNR) to generate Wiener filter. In [11], the author presented a method based on independent component analysis (ICA) and binary time-frequency masking. In [12], the authors proposed that a rough estimate of channel level difference (CLD) threshold yielding the best Signal-to-Distortion Ratio (SDR) could be obtained by cross-correlating the separated sounds. In addition, a combination of negative matrix factorization (NMF) with spatial localization via the generalized cross correlation (GCC) is applied for two-channel speech separation in [13]. For two-channel convolutive source separation, as the number of parameters in the NMF2D grows exponentially and the number of frequency basis increases linearly, the issues of model-order fitness, initialization and parameters estimation become even more critical. In [14], the authors proposed a Gaussian Expectation Maximization and Multiplicative Update (GEM-MU) algorithm to calculate the NMF2D with adaptive sparsity model and to utilize a Gamma-Exponential process in order to estimate the number of components and number of convolutive parameters in NMF2D.

The goal of this paper is to cope with competing-talker scenarios by dual-channel mixtures. In this study, we use DSB to generate the cosine function that evaluates ITD by using several frames of the short-time Fourier transform (STFT) and makes target and competing signals have the same characteristics. Then, we utilize the binary time-frequency mask to obtain the target source. There are two contributions in this paper:

(1)

we novelly upgrade delay-and-sum beamforming (DSB) [15] for estimating the ITD; and

(2)

for the first time, we clarify the connections between ideal binary mask and DSB amplitude ratio. The framework of our approach is illustrated in Figure 1. Moreover, our proposed algorithm can handle the problem of phase wrap-around.

The remainder of this paper is organized as follows: Section 2 provides an overview of time difference model. Our proposed approach including system overview and algorithm will be discussed in Section 3. In Section 4, we will introduce source separation. Then, Section 5 shows our evaluations of the system. Finally, Section 6 puts forward the main conclusions of the work.

2. Time Difference Model

We suppose that there are I ($I=2)$ sources (subscript 1 to represent the target and subscript 2 to represent the noise) in a sonic environment. The signals from two different microphones are defined, respectively, as:

$$\begin{array}{cc}\hfill x_L\left(t\right)& =\sum _{i=1}^I{a}_i^L{s}_i\left(t\right),\hfill \\ \hfill x_R\left(t\right)& =\sum _{i=1}^I{a}_i^R{s}_i(t-{\tau }_i),\hfill \end{array}$$

(1)

where $a_i^L$ and $a_i^R$ denote the weighted coefficients of the recordings of the left and right microphone from the i-th source separately. ${\tau }_i$ is the time delay of arrival (TDOA) of the i-th source between two microphones. Equation (1) can be simplified as:

$$\begin{array}{cc}\hfill x_L\left(t\right)& =\sum _{i=1}^I{s}_i\left(t\right),\hfill \\ \hfill x_R\left(t\right)& =\sum _{i=1}^I{b}_i{s}_i(t-{\tau }_i),\hfill \end{array}$$

(2)

where $b_i$ is the ratio of $a_i^L$ and $a_i^R$. By the short-time Fourier transform (STFT), the signals can be expressed as:

$$\begin{array}{cc}\hfill X_L[m,k]& =\sum _{i=1}^I{S}_i[m,k],\hfill \\ \hfill X_R[m,k]& =\sum _{i=1}^I{b}_i{S}_i[m,k]\times e^{-j{\omega }_k{\tau }_i},\hfill \end{array}$$

(3)

where m is the frame index and ${\omega }_k=2\pi k/K$. k and K are the frequency index and total window length, respectively. Under the assumption of Wdisjoint orthogonal [16], Equation (3) can be rewritten as:

$$\begin{array}{cc}\hfill X_L[m,k]& \approx S_i[m,k],\hfill \\ \hfill X_R[m,k]& \approx b_i{S}_i[m,k]\times e^{-j{\omega }_k{\tau }_i}.\hfill \end{array}$$

(4)

Thus, once the TDOA is obtained, we can make a simple binary decision concerning whether the time-frequency bin $[m,k]$ is likely to belong to the target speaker or not.

3. Proposed Approach

Delay-and-sum (DSB) is an effective means for speech enhancement. Our method is based on DSB under the anechoic condition in the time-frequency domain. In DSB, the enhanced speeches in the time-frequency domain are modeled as:

$$\begin{array}{cc}\hfill Y_1[m,k]=& \frac{X_L[m,k]+X_R[m,k]\times e^{j{\omega }_k{\widehat{\tau }}_1}}{2},\hfill \\ \hfill Y_2[m,k]=& \frac{X_L[m,k]+X_R[m,k]\times e^{j{\omega }_k{\widehat{\tau }}_2}}{2},\hfill \end{array}$$

(5)

where $Y_1[m,k]$ and $Y_2[m,k]$ are the enhanced speech of target and interferer, respectively.

Theoretically, once the correct estimations of ${\tau }_1$ and ${\tau }_2$ are obtained, Equation (5) is written as:

$$\begin{array}{c}\hfill \frac{Y_1[m,k]}{Y_2[m,k]}=\left\{\begin{array}{cc}\hfill \frac{1+b_1}{1+b_1\times e^{j{\omega }_k({\tau }_2-{\tau }_1)}},& \mathrm{if}[m,k]\in s_1,\hfill \\ \hfill \frac{1+b_2\times e^{j{\omega }_k({\tau }_1-{\tau }_2)}}{1+b_2},& \mathrm{if}[m,k]\in s_2.\hfill \end{array}\right.\end{array}$$

(6)

We define $g\left[k\right]$ as:

$$g\left[k\right]=\frac{1}{M}\sum _{m=1}^M{\left|\frac{Y_1[m,k]}{Y_2[m,k]}\right|}^{\mathrm{sgn}(\left|1-\frac{{\mathrm{Y}}_1[\mathrm{m},\mathrm{k}]}{{\mathrm{Y}}_2[\mathrm{m},\mathrm{k}]}\right|)},$$

(7)

where

$$\mathrm{sgn}\left(\mathrm{x}\right)=\left\{\begin{array}{cc}1,\hfill & x\ge 0,\hfill \\ -1,\hfill & x<0.\hfill \end{array}\right.$$

(8)

According to Equations (6) and (7), we treat $g_{The}\left[k\right]$ as the theoretical result of $g\left[k\right]$. Under the assumption of far-field ($b_1$ ≈ $b_2$), $g_{The}\left[k\right]$ is simplified to

$$\begin{array}{c}\hfill g_{The}\left[k\right]\approx \left|\frac{1+b_1\times e^{j{\omega }_k({\tau }_2-{\tau }_1)}}{1+b_1}\right|.\end{array}$$

(9)

We may obtain

$$\begin{array}{c}\hfill g_{The}\left[k\right]\approx \sqrt{1-\frac{2b_1(1-cos({\omega }_k\times ({\tau }_2-{\tau }_1)))}{{(1+b_1)}^2}},\end{array}$$

(10)

where $g_{The}\left[k\right]$ is the cosine function. Specially, if $b_1$ equals 1, we have

$$\begin{array}{c}\hfill g_{The}\left[k\right]\approx \left|cos\left(\frac{{\omega }_k\times ({\tau }_2-{\tau }_1)}{2}\right)\right|.\end{array}$$

(11)

Obviously, the maximum of $g_{The}\left[k\right]$ is 1. Furthermore, we let $g_{real}\left[k\right]$ be the real data of $g\left[k\right]$ according to Equation (6). To ensure that the maximum of $g_{real}\left[k\right]$ is 1, we rectify $g_{real}\left[k\right]$ as:

$$\begin{array}{c}\hfill g_{real\_r}\left[k\right]=g_{real}\left[k\right]+1-max{g}_{real}\left[k\right].\end{array}$$

(12)

We define the minimum of $g_{real}\left[k\right]$ as $g_{min}\left[k\right]$. Under the correct estimations of ${\tau }_1$ and ${\tau }_2$, $g_{real}\left[k\right]$ approximately equals $g_{The}\left[k\right]$. According to Equation (10), $b_1$ can be estimated as:

$$\begin{array}{c}\hfill {\widehat{b}}_1=\frac{1-g_{min}\left[k\right]}{1+g_{min}\left[k\right]}.\end{array}$$

(13)

Figure 2 demonstrates the process of ITD estimation. Figure 3 gives an example about the cosine functions with different estimations of ITD.

We define the criterion function as:

$$\begin{array}{c}\hfill J=\frac{1}{{\Sigma }_{k=1}^K\left|g_{real\_r}\left[k\right]-g_{The}\left[k\right]\right|}.\end{array}$$

(14)

Because of the periodicity of Trigonometric function, we fix $|{\omega }_k({\tau }_1-{\tau }_2)|<\pi$. We use the summation on all frequency bands to avoid phase wrap-around problem. Then, we have

$$\begin{array}{c}\hfill {\widehat{\tau }}_{1_{opt}},{\widehat{\tau }}_{2_{opt}}={argmax}_{{\widehat{\tau }}_1,{\widehat{\tau }}_2}J.\end{array}$$

(15)

4. Source Separation

After obtaining the ITD and attenuation coefficients (namely $b_1$ and $b_2$), we adopt the masking method to separate the target and competing sources. Firstly, we illustrate the effects of attenuation coefficients. Then, we utilize the time-frequency mask based on the DSB ratio.

4.1. The Effects of Weighted Coefficients

In Equation (10), we assume $b_1$ ≈ $b_2$, but sometimes experiment settings can not meet this hypothesis strictly. In this section, we set different values of $b_1$ and $b_2$ artificially to demonstrate the effectiveness of the criterion function in Equation (14). We verify the effects of $b_1$ and $b_2$ with a simple example. Assume that

$$\begin{array}{cc}\hfill x_l\left(t\right)=& s_1\left(t\right)+s_2\left(t\right),\hfill \\ \hfill x_2\left(t\right)=& b_1\times s_1(t-6.1)+b_2\times s_2(t-1.9).\hfill \end{array}$$

(16)

The details are shown in Figure 4. We can observe that even experiment settings do not meet the assumption that $b_1$ ≈ $b_2$ strictly, and the ITD still can be estimated accurately. Moreover, though the values of ${\widehat{b}}_1$ and ${\widehat{b}}_2$ are rough, the binary mask is free from attenuation coefficients since the DSB based mask only relies on ITD information.

4.2. Mask Based on DSB Ratio

Under the assumption of Wdisjoint orthogonal, the ideal ratio mask is defined using a priori energy ratio $R_{SNR}[m,k]$ [17]:

$$\begin{array}{c}\hfill R_{SNR}[m,k]=\frac{{\left|Y_1[m,k]\right|}^2}{{\left|Y_1[m,k]\right|}^2+{\left|Y_2[m,k]\right|}^2}.\end{array}$$

(17)

In addition, the ideal binary is of the form:

$$\mathrm{B}[\mathrm{m},\mathrm{k}]=\left\{\begin{array}{cc}1,\hfill & R_{SNR}[m,k]\ge \lambda ,\hfill \\ 0,\hfill & R_{SNR}[m,k]<\lambda ,\hfill \end{array}\right.$$

(18)

where $\lambda$ is set to be a value in $0.2$–$0.8$.

In our theoretical framework, $\left|\frac{1+b_1}{1+b_1\times e^{j{\omega }_k({\tau }_2-{\tau }_1)}}\right|$ is greater than 1 according to Equation (6), while $\left|\frac{1+b_2\times e^{j{\omega }_k({\tau }_2-{\tau }_1)}}{1+b_2}\right|$ is always less than 1. Then, the DSB ratio is of the form:

$$\begin{array}{c}\hfill R_{DSB}[m,k]=\left\{\begin{array}{cc}\hfill |\frac{Y_1[m,k]}{Y_2[m,k]}|\ge 1,& \mathrm{if}[m,k]\in s_1,\\ \hfill |\frac{Y_1[m,k]}{Y_2[m,k]}|<1,& \mathrm{if}[m,k]\in s_2.\end{array}\right.\end{array}$$

(19)

Comparing $R_{DSB}[m,k]$ to 1, the binary time-frequency mask is obtained as:

$$\begin{array}{c}\hfill M[m,k]=\left\{\begin{array}{cc}\hfill 1,& \mathrm{if}{R}_{DSB}[m,k]\ge 1,\hfill \\ \hfill 0,& \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(20)

It is easy to find that when $\lambda$ is set to 0.5, $B[m,k]$ is equivalent to $M[m,k]$. Equations (6) and (20) demonstrate the essence that $\lambda =0.5$ provides the best performance under the assumption of Wdisjoint orthogonal. Then, the speech can be separated as:

$$\begin{array}{cc}\hfill {\widehat{S}}_1[m,k]& =M[m,k]X_1[m,k],\hfill \\ \hfill {\widehat{S}}_2[m,k]& =(1-M[m,k])X_2[m,k],\hfill \end{array}$$

(21)

where $X[m,k]$ is defined as:

$$\begin{array}{c}\hfill X_i[m,k]=\frac{1}{2}[\mathrm{DFT}\left(x_L\left(t\right)\right)+\mathrm{DFT}\left(x_R(t-t_i)\right)].\end{array}$$

(22)

Finally, we can obtain the separated speech waveforms using the Inverse Fast Fourier Transform (IFFT) and OverLapping and Adding (OLA).

5. Experimental Evaluations

In this section, we first describe the experimental data and evaluation criteria that we used, and then present experimental results.

5.1. Experimental Setup

Figure 5 depicts the simulated experimental set-up. The sources are selected from the TIMIT database [18]. The sample rate of these audio files is 16,000 Hz. For simulated data, we evaluate the target speech separation performance using Perceptual Evaluation of Speech Quality (PESQ), $C_{sig}$, $C_{bak}$ and $C_{ovl}$ [19]. These new composite measures show moderate advantages over the existing objective measures [19]. To meet the SiSEC 2010 campaign’s evaluation criteria, we adopt the standard Source-to-Interference Ratio (SIR) [20] for SiSEC 2010 test data. For these objective measures, the higher values mean better performance.

The window length is 1024 samples with an overlap of 75%. We can calculate the voiced frames detected by Voice Active Detector (VAD) [21] to avoid the situation that $Y_2[m,k]=0$. Actually, $Y_2[m,k]=0$ hardly occurs and we do not have this operation in our experiment. Once the amplitude of $Y_2[m,k]$ is nonzero, we treat $Y_2[m,k]$ as one of the speakers.

5.2. Simulated Data

We generate data for the setup in Figure 5 with source signals of duration 2 s. Reverberation simulations are accomplished using the Room Impulse Response (RIR) open source software package [22] based on the image method. We generate 100 mixed sentences for each experimental set.

Table 1. ITD estimation on $S_1{S}_2^1$.

Anechoic			${\mathit{RT}}_{60}$ = 150 ms
Method	$S_1$	$S_2^1$	Method	$S_1$	$S_2^1$
Real ITD	0.000	2.373	Real ITD	0.000	2.373
DUET	0.058	2.370	DUET	0.520	2.560
Phat	0.017	2.502	Phat	0.217	2.500
Izumi	0.093	2.502	Izumi	0.337	2.946
Proposed	0.024	2.402	Proposed	0.179	2.428

and

Table 2. Interaural Time Difference (ITD) estimation on $S_1{S}_2^2$.

Anechoic			${\mathit{RT}}_{60}$ = 150 ms
Method	$S_1$	$S_2^2$	Method	$S_1$	$S_2^2$
Real ITD	0.000	4.060	Real ITD	0.000	4.060
DUET	0.020	3.963	DUET	1.844	3.448
Phat	0.055	4.009	Phat	0.117	4.122
Izumi	0.045	4.018	Izumi	0.043	4.067
Proposed	0.012	4.039	Proposed	0.042	4.045

show the ITD estimated results in terms of mean square errors. In our experiment, the units of ITD are represented by $\tau \times fs$. We compare our approach with other existing DUET [23], Messl [24], and Izumi [25] methods. Unlike the algorithms based on coherence, our method consolidates the estimation of ${\tau }_1$ and ${\tau }_2$ into one cosine function. Our method acquires better ITD estimation.

Table 3. ITD estimation on $R{T}_{60}$ = 150 ms with different microphone distances.

Mic-Distance	5 cm		10 cm		15 cm
Method	$S_1$	$S_2^1$	$S_1$	$S_2^1$	$S_1$	$S_2^1$
Real ITD	0.000	1.187	0.000	2.373	0.000	3.560
DUET	0.271	1.069	0.520	2.560	1.678	3.135
PHAT	0.163	1.296	0.217	2.500	0.126	3.652
Izumi	0.234	1.334	0.337	2.946	0.031	3.891
Proposed	0.112	1.125	0.179	2.428	0.041	3.527

shows the relations between microphone distances with ITD estimated results. The real ITD is proportional to the distances. The estimated ITDs calculated by our method meet this rule. For all of the distances in our experiment, the proposed method provides better ITD estimations that influence the separation results. Figure 6 shows the details with ITD estimation. Though our method does not take reverberation into consideration, the results demonstrate that our method is effective for low reverberation ($R{T}_{60}$ = 150 ms) conditions. Figure 7 shows the target source separation performance and illustrates that our method has comparable performance. Figure 8 shows the target source separation performance for different microphone distances. For different microphone distances, the source separation performances are effective. Compared with other methods, the proposed method yields better results for all of the microphone distances.

5.3. SiSEC 2010 Test Data

The data of D2-2 sets of the Signal Separation Evaluation Campaign (SiSEC) [26] consists of two-microphone real world recordings. We applied the proposed method to set1 for both room1 and room2. We only compare our method with the classical Fast-ICA [27], since the results with other methods can be found online. Figure 9 shows ITD estimation details. Table 1 and Table 2 illustrate that our method can achieve competitive results.

In Figure 10, we demonstrate the trends between $\lambda$ and mean SIR for room1 and room2. Mean SIR is symmetrical to $\lambda =0.5$, where mean SIR achieves the best performance. These characteristics are consistent with our method.

Table 4. Signal-to-Interference Ratio (SIR) evaluations based on room1 and room2.

Room1		x1	x2	x3	x4	x5	x6
Proposed	$S_1$	11.8	7.8	14.7	26.4	4.9	$-0.9$
	$S_2$	10.5	12.2	$-9.2$	2.7	14.0	21.2
ICA	$S_1$	0.3	$-1.3$	10.2	18.6	$-2.6$	$-7.8$
	$S_2$	3.3	4.8	$-8.34$	$-7.6$	10.0	18.3
Room2		x1	x2	x3	x4	x5	x6
Proposed	$S_1$	3.3	6.2	12.3	27.5	3.2	1.0
	$S_2$	12.8	11.1	$-10.0$	$-1.3$	15.8	22.5
ICA	$S_1$	$-3.2$	$-1.3$	6.6	19.6	$-4.3$	$-9.1$
	$S_2$	6.2	4.8	$-7.3$	$-8.5$	12.0	19.4

¹ The definition of ICA is “Independent Component Analysis”.

shows the separation performance for both room1 and room2.

6. Conclusions

In this paper, we have proposed a novel method based on DSB for dual-channel sources separation. Our method, for the first time, employs the extension of DSB for estimating interaural time difference (ITD) and illustrates the connection between ideal binary mask and DSB amplitude ratio. Our method is valid for phase wrap-around. Although our method is based on the assumption of an anechoic environment, the results illustrate the effectiveness for low reverberation environment ($R{T}_{60}$ = 150 ms). Objective evaluations demonstrate the effectiveness of our proposed methods.

In this paper, we focus on the estimation of the interaural time differences (ITD). In fact, the construction of an effective masking model is also very critical. We could attempt to replace our Time-Frequency Masking with an NMF2D model as proposed in [14], and adopt the GEM-MU and Gamma-Exponential process to separate sound sources. Moreover, in the presence of background noise, the idea of noise reduction in [3] is also valuable for our dual-channel speech separation.