Multiuser Equalizer for Hybrid Massive MIMO mmWave CE-OFDM Systems

Abstract

This paper considers a multiuser broadband uplink massive multiple input multiple output (MIMO) millimeter-wave (mmWave) system. The constant envelope orthogonal frequency division multiplexing (CE-OFDM) is adopted as a modulation technique to allow an efficient power amplification, fundamental for mmWave based systems. Furthermore, a hybrid architecture is considered at the user terminals (UTs) and base station (BS) to reduce the high cost and power consumption required by a full-digital architecture, which has a radio frequency (RF) chain per antenna. Both the design of the UT’s precoder and base station equalizer are considered in this work. With the aim of maximizing the beamforming gain between each UT and the BS, the precoder analog coefficients are computed as a function of the average angles of departure (AoD), which are assumed to be known at the UTs. At the BS, the analog part is derived by assuming a system with no multi-user interference. Then, a per carrier basis nonlinear/iterative multi-user equalizer, based on the iterative block decision feedback equalization (IB-DFE) principle is designed, to explicitly remove both the multi-user and residual inter carrier interferences, not tackled in the analog part. The equalizer design metric is the sum of the mean square error (MSE) of all subcarriers, whose minimization is shown to be equivalent to the minimization of a weighted error between the hybrid and the full digital equalizer matrices. The results show that the proposed hybrid multi-user equalizer has a performance close to the fully digital counterpart.

1. Introduction

Most communication systems use a frequency spectrum in the 300 MHz–3 GHz band [1]. This has led to a high level of congestion in this frequency band and the inability to respond to increased demands on new communication systems, particularly in terms of new services and data rates. The solution is the exploitation of the millimeter wave (mmWave) spectrum band, where the bandwidths can reach 2 GHz or even more [2]. A major obstacle to wireless mmWave communications is the propagation difficulties, namely the severe path loss, penetration losses and fading effects [3]. It is for this reason that mmWave based systems are typically associated to massive MIMO (mMIMO) technology, because with mMIMO it is possible to create narrow beams that direct much of the signal power in a specific direction, compensating the high attenuation of mmWave bands. Traditional sub-6 GHz mMIMO with hundreds of antenna elements appears to be a hard task to implement, however for the mmWave case, due to the small wavelengths inherent to these bands, we can pack these hundreds of antennas in the same volume of sub-6 GHz systems [4].

Although, mMIMO mmWave systems are very promising, they still face some difficulties such as: the mmWave mMIMO channels tend to be more correlated [5,6,7]; the power consumption of some hardware components of radio frequency (RF) chains, such as analog-to-digital converters (ADCs), digital-to-analog converters (DACs), mixers, power amplifiers, is higher; and the cost of some hardware components of the referred RF chains is also higher. Therefore, new techniques that consider the specific characteristics of the channel and hardware constraints should be explored. There are three major architectures that can be explored. The first one, it is the use of low-resolution DACs/ADCs. However, the performance degradation at high signal-to-noise ratio (SNR) regions is substantial [8]. The second one, it is the use of only one RF chain, i.e., a fully analog architecture. However, this limits the achievable performance, and therefore it is usually employed in single-stream transmissions [9]. Finally, the other technique is the use of hybrid analog-digital architectures. In these systems, the number of RF chains is lower than the number of antennas employed at both the user terminals (UTs) and base station (BS), reducing the cost and the power consumption. This is the architecture followed in this paper and in the last years several transmit and receive hybrid beamforming approaches have been proposed [10,11,12,13,14,15,16,17,18], as discussed in Section 1.1.

The use of mmWave bands allows larger bandwidths, but considering orthogonal frequency division multiplexing (OFDM) modulation, this leads to a higher number of subcarriers. Therefore, the use of OFDM, which is very efficient to mitigate the effects of inter-symbol interference (ISI) in frequency selective channels, results in a signal with high amplitude fluctuations, making the large peak-to-average power ratio (PAPR) a problem, which should be taken into account for mmWave mMIMO systems [19]. A high PAPR results in strong nonlinear distortions caused by the power amplifier [20], degrading the system performance. To solve the PAPR problem, constant envelope modulations, with a PAPR of 0 dB, have been proposed, such as the constant envelop OFDM (CE-OFDM) [21,22]. The CE-OFDM presents better performance than OFDM for realistic systems [23]. Some works that explore the CE-OFDM have been addressed in [24,25,26,27,28,29,30], as presented in Section 1.2.

It is well known that CE-OFDM based system require efficient equalizers (usually based on nonlinear techniques) due to residual inter-carrier-interference (ICI) [31]. Therefore, there is a significant interest in the design of iterative/nonlinear equalizers that have been considered to efficiently separate the spatial streams and mitigate the ICI problem in the current MIMO systems [32], and that can be extremely efficient in a mMIMO scenario. Iterative block decision feedback equalization (IB-DFE) approach is one of the most promising nonlinear equalization schemes [32], and it was originally proposed in [33]. It does not need the feedback loop of the channel decoder output, and it can be considered as a low complexity turbo equalizer. IB-DFE has been extended to several scenarios, as diversity scenarios, conventional and cooperative MIMO systems, among others [34,35,36,37,38].

1.1. Previous Work on Hybrid Architectures

In the recent years, some works based on hybrid architectures have been proposed for narrowband single-user systems [10,11]. In [10], it is proposed a solution based on the concept of orthogonal matching pursuit (OMP), where the sparse-scattering structure of mmWave channels is exploited to derive both the precoder and combiner. Basically, a number of dominant propagation paths are selected and then they are digitally combined at baseband. To avoid the complexity of the OMP, the authors of [11] proposed a precoder, where the singular value decomposition (SVD) is used to obtain the phase information of the channel.

Some designs for narrowband multi-user systems were addressed in [12,13]. In [12], an iterative hybrid equalizer was designed to efficiently remove the multiuser interference. The analog and digital parts of the hybrid equalizer are jointly optimized using the average bit-error-rate as metric. To compute the analog part, a set of vectors are selected from a dictionary based on the array response vectors of the channel. The authors of [13], proposed a precoder only using switches and inverters, which is an alternative to the traditional phase shifters. First, several precoder candidates are randomly generated, based on the initial probability distribution. Then, they are weighted based on their achievable relative sum-rate, and optimized by minimizing the cross entropy.

Because mmWave channels are expected to be wideband, solutions for frequency-selective channels have also been proposed [14,15,16,17]. In [14], it is proposed a precoder for a single-user system, where first the optimal hybrid precoding design for a given RF codebook is derived, then the hybrid codebooks are designed, and finally, a near-optimal greedy frequency selective hybrid precoding algorithm is proposed, based on Gram–Schmidt orthogonalization. In [15], hybrid precoder and equalizer designs are proposed for single-user and multiuser OFDM systems. The analog beamformer, based on the average of the covariance matrices of the frequency domain channels, is fixed for the entire band, while the digital beamformers are computed on per subcarrier basis. The authors of [16] proposed a hybrid precoder for downlink space-division multi-access and OFDM systems. The total transmit power of the BS is minimized, considering the coverage constraint of signaling and data rate requirements of users. In [17], the beamformers are computed only with the knowledge of the statistics of the channel. Since the dominant eigenvector of each covariance multi-cluster channel matrix is approximately a linear combination of the dominant eigenvectors of the single-cluster subchannels, the multi-cluster channel was decomposed into multiple single-cluster subchannels. Then, these dominant eigenvectors are used to design the beamformers, forming beams in the statistically more important directions.

All approaches referred to above are related to hybrid systems, where each RF chain is connected to all antennas, named hybrid full-connected architectures. This is the approach adopted in this paper. However, a hybrid subconnected architecture, where each RF chain is connected only to a subset of antennas, is addressed in [18,39], where two variants are presented: In the first one, each RF chain is connected to a fixed subset of antennas, while in the second one, these connections can change dynamically. In the hybrid subconnected architectures, a lower number of phase shifter and connections are required, however, a better performance can be achieved with the hybrid full-connected architectures.

1.2. Previous Work on Constant Envelope Orthogonal Frequency Division Multiplexing (CE-OFDM)

Regarding CE-OFDM, some designs were addressed in [24,25,26] for sub-6 GHz systems, and specifically for mmWave in [27,28,29,30]. The authors of [24], proposed to incorporate a spectral precoder in the CE-OFDM block. The goal is to suppress the side lobe powers, and then to obtain a more efficient system, preserving the advantages of CE-OFDM. In [25], it is proposed a receiver using amplitude-phase demodulator with an iterative detection, for the demodulation of the dual-stream CE-OFDM. In [26], it is proposed a data-aided carrier frequency offset estimation strategy based on the frequency domain pilot symbols.

Regarding the works on mmWave, the authors of [27] designed an iterative space-frequency domain equalizer for a full-digital system, by minimizing the overall mean square error (MSE) of all data streams at each subcarrier. The aim was to exploit the high space-frequency diversity order inherent to CE-OFDM. In [28], it is proposed a substrate integrated waveguide slotted array with a squared cosecant pattern at 73 GHz, where it can be seen that for low modulation indexes, the use of trellis coding and interleaving greatly increases the system performance. The authors of [29], proposed to integrate space-time shift keying with CE-OFDM. The aim was to transmit streams without nonlinear amplitude distortion and to exploit the flexible diversity/multiplexing tradeoff that is inherent to space-time shift keying. Finally, in [30], it is done a study and comparison, for downlink transmission in outdoor picocells, between the CE-OFDM and constant-envelope single-carrier OFDM (CE-SC-OFDM). It was demonstrated that constant envelop multicarrier waveforms enhance robustness and increase coverage and capacity in the proposed scenario, as compared to conventional OFDM and SC-OFDM counterparts. However, to the best of our knowledge, CE-OFDM based systems employing hybrid analog-digital architectures for broadband mmWave mMIMO systems, were not yet addressed in the literature.

1.3. Main Contributions

In this paper, we design a solution for the uplink multi-user broadband mmWave mMIMO systems, where the hardware constraints inherent to these systems are considered. Therefore, our design options are the following:

• The use of CE-OFDM, which has the benefits of OFDM schemes to deal with multi-path effects, but has a PAPR equal to 0 dB, which solve the problem of nonlinear distortions caused by the power amplifiers;
• The use of low-complexity UTs, using an analog-only precoder, to reduce the hardware costs and power consumption. The precoder is based on partial channel state information (CSI) knowledge, shifting the complexity to the BS;
• The use of hybrid analog-digital architecture at BS side, to reduce the power consumption and the hardware costs. The full-digital systems use a dedicated RF chain per antenna, which is impractical for mMIMO;
• The use of iterative equalizer based on IB-DFE principles, to efficiently remove the inter-user and inter-carrier interferences.

Therefore, the aim of this paper is to design an iterative/nonlinear multi-user equalizer for the uplink broadband mmWave mMIMO CE-OFDM systems. At the UTs we consider a low-complex, yet efficient, only analog precoder, with the goal of maximizing the beamforming gain between each UT and the BS and implicitly remove some multi-user interference. At the BS, the analog part of the equalizer is derived by assuming a free multi-user interference system. Then, a per carrier basis nonlinear/iterative multi-user equalizer, based on the IB-DFE principle, is designed, to explicitly remove both the multi-user and residual inter carrier interferences, not tackled in the analog part. The design metric is the sum of the mean square error (MSE) of all subcarriers, whose minimization is shown to be equivalent to the minimization of a weighted error between the hybrid and the full digital equalizer matrices. We also propose a simple, yet accurate, semi-analytical approach for obtaining the performance of the proposed hybrid scheme for CE-OFDM systems.

The remainder of this paper is structured as follows: In Section 2, the system model adopted in the paper is presented. In Section 3, we describe the proposed analog precoder and the iterative hybrid analog-digital multi-user equalizer. The main performance results are shown in Section 4, and finally, in Section 5, the main conclusions of the paper are discussed.

1.4. Notations

Capital boldface letters denote matrices, and lower boldface letters denote column vectors. The operators $\mathrm{tr}(.)$, ${(.)}^{*}$, ${(.)}^T$, ${(.)}^H$, and ${\Vert .\Vert }_F$ are the trace, the conjugate, the transpose, the Hermitian, and the Frobenius norm of a matrix, respectively. ${\left\{{\alpha }_k\right\}}_{k=1}^S$ represents a $S$-length sequence. The operator $\mathrm{diag}(\mathbf{a})$ is the diagonal matrix, where the diagonal entries are equal to vector $\mathbf{a}$, while $\mathrm{diag}(\mathbf{A})$ is a vector equal the diagonal entries of the matrix $\mathbf{A}$. $\mathbf{A}(n,m)$ represents the element of the nth row and mth column of $\mathbf{A}$, while ${\left[\mathbf{B}\right]}_{n,n}$ represents the nth element of diagonal of the square matrix $\mathbf{B}$. The identity matrix of size $N\times N$ is denoted by ${\mathbf{I}}_N$, while ${\mathbf{e}}_u\in {ℂ}^U$ is a $U$-length vector of zeros with the uth entry equal to one. The operator $\arg (c)$ gives the phase of $c.$ Finally, the indexes t, $k$ and $u$ represent the time, subcarrier and user terminal indexes, respectively.

2. System Characterization

In this section we describe the transmitter, the channel model adopted, and the receiver. We consider an uplink mmWave mMIMO system whose modulation technique is the CE-OFDM. The system has $N_c$ available subcarriers. It is assumed L-length data symbol blocks, and $U$ UTs sharing the same radio resources. Each UT employs one RF chain, has $N_{tx}$ transmit antennas and sends one data stream per subcarrier. At the receiver side, the BS employs $N_{rx}^{RF}$ RF chains and has $N_{rx}$ receive antennas, with $U\le N_{rx}^{RF}\le N_{rx}$.

2.1. Transmitter Characterization

In this subsection, the uth UT model with analog precoding is presented. The corresponding block diagram is shown in Figure 1. First, the L-length data symbol block is obtained ${\left\{s_{u,k}\right\}}_{k=1}^L$, where $s_{u,k}$ are symbols selected from a M - quadrature amplitude modulation (M-QAM) constellation, such that $\mathbb{E}[|s_{u,k}{|}^2]={\sigma }_s^2.$ Then, it is made a Hermitian symmetric zero-padded data sequence, ${\left\{z_{u,k}\right\}}_{k=1}^{N_c}\in {ℂ}^{N_c}$ given by:

$${\left\{z_{u,k}\right\}}_{k=1}^{N_c}=\{0,s_{u,1},\dots ,s_{u,L},\underset{N_{zp}}{\underset{⏝}{0,\dots ,0}},0,s_{u,L}^{*},\dots ,s_{u,1}^{*}\},$$

(1)

where the $N_{zp}$ zeros are used to achieve the effect of oversampling of the time domain sequence, whose oversampling factor is given by $N_{os}=N_c/\left(N_c-N_{zp}\right)$. An inverse fast Fourier transform (IFFT) of size $N_c=2L+N_{zp}+2$, is applied, and due to the structure of (1), the obtained sequence, denominated by ${\left\{z_{u,t}\right\}}_{t=1}^{N_c}$ in the following, is only composed by real values, i.e., $z_{u,t}\in ℝ$. After that, the phase modulator is applied, whose output, $c_{u,t}$, is given by:

$$c_{u,t}=A\exp \left(j2\pi h{z}_{u,t}\right),$$

(2)

where $A$ is the signal amplitude of $c_{u,t}$, and $h$ is the modulation index. To keep the signal power, i.e., $\mathbb{E}[|s_{u,k}{|}^2]=\mathbb{E}[|c_{u,k}{|}^2]={\sigma }_s^2$, we set the signal amplitude as $A={\sigma }_u\sqrt{L{N}_c^{-1}}.$ The variance of sequence ${\left\{z_{u,t}\right\}}_{t=1}^{N_c}$ is normalized to 1, which means that the variance of the phase of ${\left\{c_{u,t}\right\}}_{t=1}^{N_c}$ is ${\left(2\pi h\right)}^2.$ To conclude, a cyclic prefix (CP) is added and the analog precoder, ${\mathbf{f}}_{a,u}\in {ℂ}^{N_{tx}}$, based only on analog phase shifters is applied, with $|{\mathbf{f}}_{a,u}(n){|}^2=N_{tx}^{-1}$.

2.2. Channel Model

A clustered channel with $N_{cl}$ clusters, and $N_{ray}$ rays per cluster is considered [14]. For the uth UT, at the kth subcarrier, the channel matrix ${\mathbf{H}}_{u,k}\in {ℂ}^{N_{rx}\times N_{tx}},k=1,\dots ,N_c$ is given in the frequency domain by:

$${\mathbf{H}}_{u,k}={\displaystyle \sum _{d=0}^{D-1}{\mathbf{H}}_{u,d}{e}^{-j\frac{2\pi k}{N_c}d}},$$

(3)

With

$${\mathbf{H}}_{u,d}=\gamma {\displaystyle \sum _{q=1}^{N_{cl}}{\displaystyle \sum _{l=1}^{N_{ray}}{\alpha }_{q,l}^{}{p}_{rc}(d{T}_s-{\tau }_q^{}-{\tau }_{q,l}^{})}}{\mathbf{a}}_{rx,u}\left({\varphi }_{q,l}^{rx},{\theta }_{q,l}^{rx}\right){\mathbf{a}}_{tx,u}{\left({\varphi }_{q,l}^{tx},{\theta }_{q,l}^{tx}\right)}^H,$$

(4)

where $\gamma =\sqrt{N_{rx}{N}_{tx}/{\rho }_{PL}}$ is the normalization factor, which ensures that $\mathbb{E}\left[{\Vert {\mathbf{H}}_{u,d}\Vert }_F^2\right]=N_{rx}{N}_{tx}$. The path-loss between the uth UT and the BS is ${\rho }_{PL}$. The function $p_{rc}(.)$, is the pulse shaping function, being adopted the raised-cosine filter. For the lth ray in the qth scattering cluster: ${\alpha }_{q,l}^{}\sim \mathcal{C}\mathcal{N}\left(0,{\sigma }_{q,l}^2\right)$ is the complex path gain, ${\tau }_q^{}$ is the cluster time delay, while ${\tau }_{q,l}^{}$ is the relative time delay, and finally the angles ${\varphi }_{q,l}^{rx}$,${\theta }_{q,l}^{rx}$,${\varphi }_{q,l}^{tx}$ and ${\theta }_{q,l}^{tx}$, are azimuth and elevation angles of arrival and departure, respectively.

The value ${\sigma }_{q,l}^2$ is the power of the lth ray in the qth scattering cluster, with a decay equal to ${\beta }_{q_1,q_2}$ from the $q_1$th cluster to the $q_2$th cluster, and a decay equal to ${\beta }_{q,l_1,l_2}$ from the $l_1$th ray to the $l_2$th cluster of $q$th cluster. The path delays are uniformly distributed in $\left[0,D{T}_s\right],$ where $T_S$ is the sampling interval and $D$ is the cyclic prefix length. The angles ${\varphi }_{q,l}^{rx}$,${\theta }_{q,l}^{rx}$,${\varphi }_{q,l}^{tx}$ and ${\theta }_{q,l}^{tx}$ have a Laplacian distribution, where for instance, ${\varphi }_{q,l}^{rx}$ has a mean ${\varphi }_q^{rx}$ uniformly distributed in $[0,2\pi ]$ and variance ${\sigma }_{{\varphi }_q^{rx}}^2$.

Finally, the vectors ${\mathbf{a}}_{rx}$ and ${\mathbf{a}}_{tx}$ are the normalized receive and transmit array response vectors, respectively. For a uniform linear array (ULA), the normalized array response vector is:

$${\mathbf{a}}_{ULA}\left(\varphi \right)=\frac{1}{\sqrt{N}}{\left[1,e^{j1\frac{2\pi }{\lambda }d\sin \left(\varphi \right)},e^{j2\frac{2\pi }{\lambda }d\sin \left(\varphi \right)},\dots ,e^{j\left(N-1\right)\frac{2\pi }{\lambda }d\sin \left(\varphi \right)}\right]}^T,$$

(5)

where $N$ is the number of elements of array antennas, $\lambda$ is the wavelength, and finally $d$ is the inter-element spacing.

2.3. Receiver Characterization

The considered receiver structure is presented in Figure 2. The received signal, ${\mathbf{y}}_k$, at the kth subcarrier, is given by:

$${\mathbf{y}}_k={\displaystyle \sum _{u=1}^U{\mathbf{H}}_{u,k}{\mathbf{x}}_{u,k}}+{\mathbf{n}}_k,$$

(6)

where the vector ${\mathbf{n}}_k\in {ℂ}^{N_{rx}}$ is the zero mean Gaussian noise with variance ${\sigma }_n^2$. First, the analog part of the equalizer, ${\mathbf{W}}_a^{}\in {ℂ}^{N_{rx}\times N_{rx}^{RF}}$, with $|{\mathbf{W}}_a(n,l){|}^2=N_{rx}^{-1},$ is applied to the received signal. Then, it is performed the baseband processing, composed by $N_{rx}^{RF}$ RF chains and the iterative digital multi-user equalizer. In its turn, a forward path, and a feedback path compose the iterative digital equalizer. In the feedfoward path, it is applied the digital linear filter ${\mathbf{W}}_{d,k}^{(i)}\in {ℂ}^{U\times N_{rx}^{RF}}$ on each subcarrier k, such that, ${\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{y}}_k$, where $i=0,1,\dots ,i_{\max },$ and $i_{\max }$ is the maximum iteration number. Then, the samples ${\left\{{\tilde{c}}_{u,k}^{(i)}\right\}}_{k=1}^{N_c}$ are moved to the time domain, where it is applied the phase demodulator obtaining the samples ${\left\{{\tilde{z}}_{u,k}^{(i)}\right\}}_{t=1}^{Nc}$. Finally, a $N_c$ - point fast Fourier transform (FFT) and a selection operation are performed to obtain the soft estimates ${\left\{{\tilde{s}}_{u,k}^{(i)}\right\}}_{k=1}^L\in {ℂ}^L,$ $u=1,\dots ,U,$ and the corresponding hard estimates, ${\left\{{\widehat{s}}_{u,k}^{(i)}\right\}}_{k=1}^L\in {ℂ}^L,$.

In the feedback path, the inverse processing is done, i.e., the data recovered from the forward path is again modulated in the symbols ${\left\{{\widehat{s}}_{u,k}^{(i-1)}\right\}}_{k=1}^L\in {ℂ}^L,$ $u=1,\dots ,U,$ the Hermitian symmetric zero-padded data vector is made using the structure of (1), and it is applied a $N_c$ - point IFFT and the phase modulator. Finally, the resulting signal, ${\widehat{\mathbf{c}}}_k^{(i)}={[{\widehat{c}}_{1,k}^{(i-1)},\dots ,{\widehat{c}}_{U,k}^{(i-1)}]}^T\in {ℂ}^U$, is filtered by the feedback equalizer matrix, ${\mathbf{B}}_{d,k}^{(i)}\in {ℂ}^{U\times U}$, at kth subcarrier.

After all this processing, we obtain the processed received signal for iteration i, ${\tilde{\mathbf{c}}}_k^{(i)}={[{\tilde{c}}_{1,k}^{(i)},\dots ,{\tilde{c}}_{U,k}^{(i)}]}^T\in {ℂ}^U$, expressed by:

$${\tilde{\mathbf{c}}}_k^{(i)}={\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{y}}_k-{\mathbf{B}}_{d,k}^{(i)}{\widehat{\mathbf{c}}}_k^{(i-1)}.$$

(7)

3. Precoder and Equalizer Design

In this section, a low-complexity analog precoder, based only on average angles of departure (AoD), is presented. Then, the iterative hybrid analog-digital multi-user equalizer for CE-OFDM systems is derived. A decoupled transmitter-receiver optimization problem is assumed in this paper because the resulting joint optimization problem will be nonconvex, even without considering the analog part constraints, and is therefore, mathematically intractable [40].

3.1. Analog Precoder

The aim of the analog precoder is to maximize the beamforming gain between each UT and the BS. To compute the analog precoder, let us consider:

$${\mathbf{A}}_{tx,u}=[{\mathbf{a}}_{tx,u}({\varphi }_{u,1}^{tx}),\dots ,{\mathbf{a}}_{tx,u}({\varphi }_{u,N_{cl}}^{tx})]\in {ℂ}^{N_{tx}\times N_{cl}},$$

(8)

where ${\mathbf{a}}_{tx,u},u=1,\dots ,U$ is the normalized array response vector, which for ULA case is given by (5), and ${\varphi }_{u,q}^{tx},u=1,\dots ,U,$$q=1,\dots ,N_{cl}$, is the average AoD for qth cluster of uth user channel. Notice that by assuming only the knowledge of the average AoD we reduce the feedback information from the receiver to the transmitter, making our design more useful for practical applications. The analog precoder is obtained from the eigenvalue decomposition of correlation matrix ${\mathbf{A}}_{tx,u}{\mathbf{A}}_{tx,u}^H$, i.e., we compute ${\mathbf{A}}_{tx,u}{\mathbf{A}}_{tx,u}^H={\mathbf{\Lambda }}_{tx,u}{\mathbf{\Sigma }}_{tx,u}{\mathbf{\Lambda }}_{tx,u}^H$. Therefore, the analog precoder vector of the uth user is given by:

$${\mathbf{f}}_{a,u}\left(n\right)=\frac{1}{\sqrt{N_{tx}}}\exp \left\{j\arg \left({\mathbf{\Lambda }}_{tx,u}\left(n,1\right)\right)\right\},n=1,\dots ,N_{tx}.$$

(9)

From Equation (9), we can see that the analog precoder selects the vector of ${\mathbf{\Lambda }}_{tx,u}$ that corresponds to the largest eigenvalue of ${\mathbf{A}}_{tx,u}{\mathbf{A}}_{tx,u}^H$ and then projects its elements along the unit magnitude set; therefore, it points the beam along the dominant channel direction. The discrete transmitted baseband signal of the uth user, at kth subcarrier, ${\mathbf{x}}_{u,k}\in {ℂ}^{N_{tx}}$, is given by:

$${\mathbf{x}}_{u,k}={\mathbf{f}}_{a,u}{c}_{u,k}.$$

(10)

3.2. Hybrid Equalizer

In this section, we design the proposed analog-digital multi-user equalizer to efficiently remove both the multi-user and residual inter carrier interferences. In the following derivation, the closed-form iterative digital equalizer is firstly obtained as a function of the analog part of the equalizer, then, the analog part of the equalizer, which cannot be obtained iteratively, is derived assuming that the digital part will fully remove the interference. The analog part is composed by a set of vectors selected from a dictionary based on the receive array response matrix. Finally, the digital equalizer is iteratively computed using the analog fixed coefficients.

3.2.1. Design of Iterative Digital Equalizer

In this section we derive the digital part of the equalizer, which is composed by the feedforward, ${\mathbf{W}}_{d,k}^{(i)}$, and feedback, ${\mathbf{B}}_{d,k}^{(i)}$, coefficients. The hybrid equalizer is designed by minimizing the sum of the ${\mathrm{MSE}}_k^{(i)}=\mathbb{E}[||{\tilde{\mathbf{c}}}_k^{(i)}-{\mathbf{c}}_k|{|}^2]$ of all subcarriers. Mathematically, the optimization problem may be formulated as:

$$\left({\mathbf{W}}_a^{},{\mathbf{W}}_{d,k}^{(i)},{\mathbf{B}}_{d,k}^{(i)}\right)=\arg \min {\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_k^{(i)}},s.\mathrm{t}.{\displaystyle \sum _{k=1}^{N_c}\mathrm{diag}({\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k)}=N_c{\mathbf{I}}_U,{\mathbf{W}}_a\in {\mathcal{W}}_a,$$

(11)

where ${\mathbf{H}}_k={[{\mathbf{H}}_{1,k}{\mathbf{f}}_{a,1},\dots ,{\mathbf{H}}_{1,k}{\mathbf{f}}_{a,1}]}^T\in {ℂ}^{N_{rx}\times U}$ is the equivalent channel, and ${\mathcal{W}}_a$ denotes the set of feasible analog coefficients. Considering that the entries of ${\mathbf{c}}_k={[c_{1,k}^{},\dots ,c_{U,k}^{}]}^T\in {ℂ}^U$ are approximately Gaussian distributed, it can be proven that the input-output relationship between ${\mathbf{c}}_k$ and ${\widehat{\mathbf{c}}}_k^{(i)}$, $k\in \{1,\dots ,N_c\}$ is [41]:

$${\widehat{\mathbf{c}}}_k^{(i)}\approx {\mathbf{\Psi }}^{(i)}{\mathbf{c}}_k+{\widehat{ϵ}}_k^{(i)},k\in \{1,\dots ,N_c\},$$

(12)

where ${\widehat{ϵ}}_k^{(i)}\in {ℂ}^U$ is the zero mean error, uncorrelated with ${\mathbf{c}}_k.$ Considering (7) and (12) the ${\mathrm{MSE}}_k^{(i)}$ can be expressed as:

$$\begin{array}{ll}{\mathrm{MSE}}_k^{(i)}=& {\Vert {\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k-{\mathbf{B}}_{d,k}^{(i)}{\mathbf{\Psi }}^{(i-1)}-{\mathbf{I}}_U\Vert }_F^2{\sigma }_u^2\\ & +{\Vert {\mathbf{B}}_{d,k}^{(i)}{({\mathbf{I}}_U-|{\mathbf{\Psi }}^{(i-1)}{|}^2)}^{1/2}\Vert }_F^2{\sigma }_u^2+{\Vert {\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a)}^H\Vert }_F^2{\sigma }_n^2.\end{array}$$

(13)

where ${\sigma }_u^2={(2\pi h)}^2{\sigma }_s^2$ and ${\mathbf{\Psi }}^{(i)}\in {ℂ}^{U\times U}$ is a diagonal matrix whose uth element gives a blockwise reliability measure of the estimates of uth block, associated to the ith iteration [32]. The bit-error-rate (BER) for iteration i, at receiver side, can be estimated by the semi-analytic BER approximation [42], given by:

$${\mathrm{BER}}^{\left(i\right)}=\frac{\alpha }{U{N}_c}{\displaystyle \sum _{u=1}^U{\displaystyle \sum _{k=1}^{N_c}Q\left(\sqrt{\beta {\left({\mathrm{MSE}}_{k,u}^{(i)}\right)}^{-1}}\right)}},$$

(14)

where $\alpha =4(1-1/\sqrt{M})/{\log }_2[M]$, $\beta =3/(M-1)$, for a M-QAM constellation, and $Q(.)$ denotes the Q-function. ${\mathrm{MSE}}_{k,u}^{(i)}$ is the mean square error on samples ${\tilde{c}}_{k,u}^{(i)}$, with ${\mathrm{MSE}}_k^{(i)}={\displaystyle {\sum }_{u=1}^U{\mathrm{MSE}}_{k,u}^{(i)}}$, at iteration i.

First, let us calculate the digital feedback matrix, ${\mathbf{B}}_{d,k}^{(i)}$, through the following optimization problem:

$${\mathbf{B}}_{d,k}^{(i)}\left[{\mathbf{W}}_a^{},{\mathbf{W}}_{d,k}^{(i)}\right]=\arg \min {\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_k^{(i)}},$$

(15)

where ${\mathbf{B}}_{d,k}^{(i)}$ appears as a function of ${\mathbf{W}}_a$ and ${\mathbf{W}}_{d,k}^{(i)}$. The problem (15) does not have any constraint (and it is convex), since ${\mathbf{B}}_{d,k}^{(i)}$ is independent of the constraints of (11). From Equation (13) and from the Karush–Kuhn–Tucker (KKT) conditions, i.e., $\partial \left({\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_k^{(i)}}\right)/\partial \left({\mathbf{B}}_{d,k}^{(i)}\right)=\mathbf{0}$, it can be proven that the digital feedback matrix is given by:

$${\mathbf{B}}_{d,k}^{(i)}\left[{\mathbf{W}}_a^{},{\mathbf{W}}_{d,k}^{(i)}\right]=\left({\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k-{\mathbf{I}}_U\right){\left({\mathbf{\Psi }}^{(i-1)}\right)}^H.$$

(16)

To compute the digital feedforward filter, we start by replacing Equation (16) in Equation (13), and obtain:

$${\mathrm{MSE}}_k^{(i)}={\Vert \left({\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a)}^H-{\overline{\mathbf{W}}}_{fd,k}^{(i)}\right){\left({\tilde{\mathbf{R}}}_k^{(i-1)}\right)}^{1/2}\Vert }_F^2,$$

(17)

where ${\overline{\mathbf{W}}}_{fd,k}^{(i)}$ denotes a non-normalized version of full digital equalizer, i.e., ${\mathbf{W}}_{fd,k}^{(i)}=\mathbf{\Omega }{({\mathbf{I}}_U-|{\mathbf{\Psi }}^{(i-1)}{|}^2)}^{-1}{\overline{\mathbf{W}}}_{fd,k}^{(i)},$ with ${\mathbf{W}}_{fd,k}^{(i)}$ given by Equation (A2). The formulation of problem and corresponding solution for the fully digital case is presented in the Appendix A. As can be shown by Equation (17), the MSE can also be expressed as the difference between the hybrid and the full digital equalizer matrices.

Therefore, the optimization problem to compute the iterative digital feedforward equalizer is given by:

$$\begin{array}{l}{\mathbf{W}}_{d,k}^{(i)}[{\mathbf{W}}_a^{}]=\arg \min {\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_k^{(i)}}\\ s.\mathrm{t}.{\displaystyle \sum _{k=1}^{N_c}\mathrm{diag}({\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k)}=N_c{\mathbf{I}}_U,\end{array}$$

(18)

where Equation (18) is also a convex problem, since ${\mathbf{W}}_{d,k}^{(i)}$ is independent of constraint ${\mathbf{W}}_a\in {\mathcal{W}}_a$. To calculate ${\mathbf{W}}_{d,k}^{(i)}$ as function of ${\mathbf{W}}_a$, we can compute the associated Lagrangian of the problem (18) given by:

$$\begin{array}{l}ℒ\left({\mu }_u,{\mathbf{W}}_{d,k}^{(i)}\right)=\\ {\displaystyle \sum _{k=1}^{N_c}{\Vert \left({\mathbf{W}}_{d,k}^{(i)}{\left({\mathbf{W}}_a^{(i)}\right)}^H-{({\overline{\mathbf{W}}}_{fd,k}^{(i)})}_{opt}^{}\right){\left({\tilde{\mathbf{R}}}_k^{(i-1)}\right)}^{1/2}\Vert }_F^2}\\ +{\displaystyle \sum _{u=1}^U{\mu }_u\mathrm{tr}\left({\mathbf{W}}_{d,k}^{(i)}{\left({\mathbf{W}}_a^{(i)}\right)}^H{\mathbf{H}}_k{\mathbf{e}}_u{\mathbf{e}}_u^H\right)}\\ +{\displaystyle \sum _{u=1}^U{\mu }_u\left({\displaystyle \sum _{k^{\prime }=1,k^{\prime }\ne k}^{N_c}\mathrm{tr}\left({\mathbf{W}}_{d,k^{\prime }}^{(i)}{\left({\mathbf{W}}_a^{(i)}\right)}^H{\mathbf{H}}_{k^{\prime }}{\mathbf{e}}_u{\mathbf{e}}_u^H\right)}-N_c\right)},\end{array}$$

(19)

where the coefficients ${\mu }_u,u\in \{1,\dots ,U\}$ are the Lagrange multipliers, and ${\mathbf{e}}_u\in {ℂ}^U$, and make $\partial ℒ\left({\mu }_u,{\mathbf{W}}_{d,k}^{(i)}\right)/\partial \left({\mathbf{W}}_{d,k}^{(i)}\right)=\mathbf{0}.$ Therefore, the solution of (18) is given by:

$${\mathbf{W}}_{d,k}^{(i)}[{\mathbf{W}}_a]={\mathbf{\Omega }}_d{\mathbf{H}}_k^H{\mathbf{W}}_a{\left({({\mathbf{W}}_a)}^H{\tilde{\mathbf{R}}}_k^{(i-1)}{\mathbf{W}}_a^{}\right)}^{-1},$$

(20)

where ${\mathbf{\Omega }}_d$ ensures the power constraint of (18) and it is given by:

$${\mathbf{\Omega }}_d=N_c{\left({\displaystyle \sum _{k=1}^{N_c}\mathrm{diag}\left({\mathbf{H}}_k^H{\mathbf{W}}_a{\left({({\mathbf{W}}_a^{})}^H{\tilde{\mathbf{R}}}_k^{(i-1)}{\mathbf{W}}_a^{}\right)}^{-1}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k\right)}\right)}^{-1}$$

(21)

The pseudo-code to compute the digital part is presented in the Algorithm 1. This pseudo-code uses the analog precoder matrix, ${\mathbf{W}}_a$, as input, which is designed in the next section.

Algorithm 1 Digital part of the equalizer

Input: ${\mathbf{W}}_a^{}$

1: ${\mathbf{\Psi }}^{(0)}={\mathbf{0}}_U$

2: ${\tilde{\mathbf{R}}}_k^{(0)}={\mathbf{H}}_k({\mathbf{I}}_U-|{\mathbf{\Psi }}^{(0)}{|}^2){\mathbf{H}}_k^H+{\sigma }_n^2{\sigma }_u^{-2}{\mathbf{I}}_{N_{rx}}$

3: for $i=1,\dots ,i_{\max }$ do

4:   ${\mathbf{W}}_{d,k}^{(i)}={\mathbf{\Omega }}_d{\mathbf{H}}_k^H{\mathbf{W}}_a^{}{\left({({\mathbf{W}}_a^{})}^H{\tilde{\mathbf{R}}}_k^{(i-1)}{\mathbf{W}}_a^{}\right)}^{-1}$

5:   ${\mathbf{B}}_{d,k}^{(i)}=\left({\mathbf{W}}_{d,k}^{(i)}{({\mathbf{W}}_a^{})}^H{\mathbf{H}}_k-{\mathbf{I}}_U\right){\left({\mathbf{\Psi }}^{(i-1)}\right)}^H$

6:   Compute ${\mathbf{\Psi }}^{(i)}$

7:   ${\tilde{\mathbf{R}}}_k^{(i)}={\mathbf{H}}_k({\mathbf{I}}_U-|{\mathbf{\Psi }}^{(i)}{|}^2){\mathbf{H}}_k^H+{\sigma }_n^2{\sigma }_u^{-2}{\mathbf{I}}_{N_{rx}}$

8: end for

9: return ${\mathbf{W}}_{d,k}^{(i)}$, ${\mathbf{B}}_{d,k}^{(i)}$

3.2.2. Design of Fixed Analog Feedforward Equalizer

In this section, the analog part of the hybrid equalizer is designed. From (11) and after obtaining the solution for ${\mathbf{B}}_{d,k}^{(i)}$ and ${\mathbf{W}}_{d,k}^{(i)}$, we have the following optimization problem:

$$\begin{array}{cc}\hfill {\mathbf{W}}_a^{}=& \arg \min {\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_k^{}}\hfill \\ & \mathrm{s}.\mathrm{t}.{\mathbf{W}}_a^{}\in {\mathcal{W}}_a,\hfill \end{array}$$

(22)

where the power constraint of (11) or (18) is not considered, since it is assured by ${\mathbf{\Omega }}_d$, given by Equation (21), computed in the digital feedforward equalizer. Since the analog part is computed in the first iteration, for simplicity, we removed the index i, and then ${\mathrm{MSE}}_k$ corresponds to ${\mathrm{MSE}}_k^{(1)}$.

The problem (22) is nonconvex because of the constraint ${\mathbf{W}}_a\in {\mathcal{W}}_a$. Then, we propose a solution based on OMP algorithm, where a set of vectors is selected from ${\mathcal{W}}_a$. We assume in this paper that ${\mathcal{W}}_a$ is composed by the set of receive array response vectors, ${\mathbf{a}}_{rx,u}\left({\varphi }_{q,l}^{rx},{\theta }_{q,l}^{rx}\right),$ $q=1,\dots ,N_{cl},$ $l=1,\dots ,N_{ray}$, presented in the channel model (4). The concatenated matrix with this set of vectors is ${\mathbf{A}}_{rx}=[{\mathbf{A}}_{rx,1},\dots ,{\mathbf{A}}_{rx,U}]\in {ℂ}^{N_{rx}\times N_{cl}{N}_{ray}U}$, where:

$${\mathbf{A}}_{rx,u}=\left[{\mathbf{a}}_{rx,u}\left({\varphi }_{1,1}^{rx},{\theta }_{1,1}^{rx}\right),\dots ,{\mathbf{a}}_{rx,u}\left({\varphi }_{N_{cl},N_{ray}}^{rx},{\theta }_{N_{cl},N_{ray}}^{rx}\right)\right].$$

(23)

Let ${\mathbf{w}}_{a,r}\in {ℂ}^{N_{rx}}$ be the equalizer vector of the rth RF chain such that ${\mathbf{W}}_{a,r}=[{\mathbf{w}}_{a,1},\dots ,{\mathbf{w}}_{a,r}]\in {ℂ}^{N_{rx}\times r}$ and define ${\mathbf{W}}_{ad,k,r}={\mathbf{W}}_{d,k,r}{({\mathbf{W}}_{a,r})}^H$, where ${\mathbf{W}}_{d,k,r}=[{\mathbf{w}}_{d,k,1},\dots ,{\mathbf{w}}_{d,k,r}]\in {ℂ}^{U\times r}$. Then, ${\mathbf{W}}_{ad,k,r}$ is given by:

$${\mathbf{W}}_{ad,k,r}={\mathbf{W}}_{ad,k,r-1}+{\mathbf{w}}_{d,k,r}{\left({\mathbf{w}}_{a,r}\right)}^H,$$

(24)

for $r=1,\dots ,N_{rx}^{RF}$. Replacing (24) in (17), the problem (22) simplifies to:

$$\begin{array}{cc}\hfill {\mathbf{w}}_{a,r}^{}=& \arg \min {\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_{k,r}^{}}\hfill \\ & \mathrm{s}.\mathrm{t}.{\mathbf{w}}_{a,r}^{}\in {ℱ}_a,\hfill \end{array}$$

(25)

where ${ℱ}_a$ represents the dictionary defined by columns of ${\mathbf{A}}_{rx},$ ${\mathbf{W}}_{res,k,r-1}^{}={\overline{\mathbf{W}}}_{fd,k}^{(1)}-{\mathbf{W}}_{ad,k,r-1}$ is the residue matrix, and

$${\mathrm{MSE}}_{k,r}={\Vert \left({\mathbf{w}}_{d,k,r}^{}{\left({\mathbf{w}}_{a,r}^{}\right)}^H-{\mathbf{W}}_{res,k,r-1}^{}\right){\left({\tilde{\mathbf{R}}}_k\right)}^{1/2}\Vert }_F^2,$$

(26)

is the MSE using r RF chains. Making $\partial \left({\displaystyle \sum _{k=1}^{N_c}{\mathrm{MSE}}_{k,r}}\right)/\partial \left({\mathbf{w}}_{d,k,r}\right)=\mathbf{0}$, we obtain:

$${\mathbf{w}}_{d,k,r}={\mathbf{W}}_{res,k,r-1}{\tilde{\mathbf{R}}}_k{\mathbf{w}}_{a,r}^{}{\left({\left({\mathbf{w}}_{a,r}\right)}^H{\tilde{\mathbf{R}}}_k^{}{\mathbf{w}}_{a,r}^{}\right)}^{-1}.$$

(27)

Replacing Equation (27) in Equation (26), we obtain

$$\begin{array}{ll}{\mathrm{MSE}}_{k,r}^{}& ={\Vert {\overline{\mathbf{W}}}_{res,k,r-1}^{}{\left({\tilde{\mathbf{R}}}_k^{}\right)}^{-1/2}\Vert }_F^2\\ & +{\Vert {\overline{\mathbf{W}}}_{res,k,r-1}^{}{\mathbf{w}}_{a,r}^{}{\left({\mathbf{w}}_{a,r}^H{\tilde{\mathbf{R}}}_k^{}{\mathbf{w}}_{a,r}^{}\right)}^{-1/2}\Vert }_F^2,\end{array}$$

(28)

where ${\overline{\mathbf{W}}}_{res,k,r-1}^{}={\mathbf{W}}_{res,k,r-1}^{}{\tilde{\mathbf{R}}}_k^{}$. As mentioned, the analog part of equalizer should remain constant due to hardware constraints, thus, we need to fix the coefficients of matrix ${\mathbf{\Psi }}^{(i)}$ to calculate ${\mathbf{W}}_a^{}$. As the number of iterations increases, $i=0,1,\dots ,i_{\max },$ the estimates are more reliable, and the coefficients tend toward 1. Then, we fixed $\mathbf{\Psi }={\mathbf{I}}_U$ for the analog part calculations. Therefore, from Equation (A5) we have ${\tilde{\mathbf{R}}}_k={\sigma }_n^2{\sigma }_u^{-2}{\mathbf{I}}_U$, and as ${\mathbf{w}}_{a,r}$ has magnitude $N_{rx}^{-1/2}$ then we have ${\left({\mathbf{w}}_{a,r}\right)}^H{\tilde{\mathbf{R}}}_k{\mathbf{w}}_{a,r}={\sigma }_n^2{\sigma }_u^{-2}.$

The first norm of Equation (28) does not depend on ${\mathbf{w}}_{a,r}$ while the second one is a correlation involving ${\mathbf{w}}_{a,r}$. Therefore, we may use the second norm of Equation (28) as a metric to select the best entry of dictionary, i.e., Equation (25) is equivalent to the following optimization problem:

$$\begin{array}{cc}\hfill {\mathbf{w}}_{a,r}=& \arg \max {\displaystyle \sum _{k=1}^{N_c}{\Vert {\overline{\mathbf{W}}}_{res,k,r-1}{\mathbf{w}}_{a,r}\Vert }_F^2}\hfill \\ & \mathrm{s}.\mathrm{t}.{\mathbf{w}}_{a,r}^{}\in {ℱ}_a.\hfill \end{array}$$

(29)

From Equation (29), the index vector, $n_{opt,r}$, selected from ${\mathbf{A}}_{rx}$, is given by:

$$n_{opt,r}=\arg {\max }_{l=1,\dots ,N_{cl}{N}_{ray}U}{\displaystyle \sum _{k=1}^{N_c}{\left[{\mathbf{\Pi }}_{k,r}^H{\mathbf{\Pi }}_{k,r}\right]}_{l,l}},$$

(30)

where ${\mathbf{\Pi }}_{k,r}={\overline{\mathbf{W}}}_{res,k,r-1}{\mathbf{A}}_{rx}$. The solution of Equation (29) is presented in the pseudo-code of Algorithm 2, where from Equation (A2) and $\mathbf{\Psi }={\mathbf{I}}_U$, we assume a non-normalized version of fully digital equalizer given by ${\overline{\mathbf{W}}}_{fd,k}^{(1)}={\mathbf{H}}_k^H.$ We start by setting the analog equalizer as an empty matrix, because any vector was still selected. We also set the residue matrix as the trivial value, ${\overline{\mathbf{W}}}_{res,k,0}^{}={\mathbf{H}}_k^H$. Then, the best column from the dictionary ${\mathbf{A}}_{rx}$ is selected for the rth RF chain, and the residue matrix is updated. The previous steps are repeated until $N_{rx}^{RF}$ be selected, and the analog equalizer matrix is obtained.

Algorithm 2 Analog part of equalizer

1: ${\mathbf{W}}_{a,0}^{}$ = Empty Matrix

2: ${\overline{\mathbf{W}}}_{res,k,0}^{}={\mathbf{H}}_k^H$

3: for $r=1,\dots ,N_{rx}^{RF}$ do

4:  ${\mathbf{\Pi }}_{k,r}={\overline{\mathbf{W}}}_{res,k,r-1}^{}{\mathbf{A}}_{rx}$

5:  $n_{opt,r}=\arg {\max }_{l=1,\dots ,N_{cl}{N}_{ray}U}{\displaystyle \sum _{k=1}^{N_c}{\left[{\mathbf{\Pi }}_{k,r}^H{\mathbf{\Pi }}_{k,r}\right]}_{l,l}}$

6:  ${\mathbf{W}}_{a,r}^{}=[{\mathbf{W}}_{a,r-1}^{}|{\mathbf{A}}_{rx}^{(n_{opt,r})}]$

7:  ${\overline{\mathbf{W}}}_{res,k,r-1}^{}=\left({\mathbf{H}}_k^H-{\mathbf{W}}_{ad,k,r}^{}\right){\tilde{\mathbf{R}}}_k^{}$

8: end for

9: return ${\mathbf{W}}_a^{}={\mathbf{W}}_{a,N_{rx}^{RF}}^{}$

3.3. Complexity Analysis

In this section, the complexity of the proposed algorithms are evaluated. In Algorithm 1, the digital equalizer is computed using the inversion of a $U\times U$ matrix, which results in a complexity $O(U^3)$. This inversion is repeated $i_{\max }$ times, and then the total complexity of Algorithm 1 is $O(i_{\max }{U}^3).$

In the Algorithm 2, the metric of Equation (29) is tested for all elements of the codebook ${ℱ}_a$, composed by the vectors of ${\mathbf{A}}_{rx}\in {ℂ}^{N_{rx}\times N_{cl}{N}_{ray}U}$. The product complexity of the matrix of size $U\times N_{rx}$ by a vector of size $N_{rx}$ is $O(N_{rx}U)$. Then, the complexity of the metric evaluation is $O(N_{cl}{N}_{ray}{N}_{rx}{U}^2).$ As these operations are repeated $N_{rx}^{RF}$ times, the complexity of the Algorithm 2 is $O(N_{cl}{N}_{ray}{N}_{rx}{N}_{rx}^{RF}{U}^2).$ Therefore, the total complexity computation of the proposed equalizer is $O(i_{\max }{U}^3+N_{cl}{N}_{ray}{N}_{rx}{N}_{rx}^{RF}{U}^2).$

4. Performance Results

The main performance results of the proposed hybrid analog-digital multiuser equalizer, designed for CE-OFDM based systems, are shown in this section. The performance metric is the average $\mathrm{BER}$ as a function of $E_b/N_0$, where $E_b$ is the average bit energy, and $N_0$ is the one-sided noise power spectral density. We assume that the signal transmitted power of all users is the same, such that, ${\sigma }_s^2=1$, and then $E_b/N_0={\sigma }_s^2/(2{\sigma }_n^2)={\sigma }_n^{-2}/2$. The quadrature phase shift keying (QPSK) modulation is adopted. For each UT, the wideband mmWave channel model defined in Equation (3) is considered and the parameters presented in

Table 1. Simulation parameters.

Parameter	Value
Carrier frequency	72 GHz
Antenna element spacing	Half-wavelength
Array configuration	Uniform Linear Array (ULA)
Decay from the first to the last channel cluster $(N_{cl}{\beta }_{q_1,q_2})$	10 dB
Decay from the first to the last ray of each channel cluster $({\beta }_{q,l_1,l_2})$	10 dB
$N_{cl}$	4
$N_{ray}$	5
${\sigma }_{{\varphi }_q^{rx}}^{},{\sigma }_{{\varphi }_q^{tx}}^{}$	10 degrees
$N_{os}$	2
$N_c$	512
$D$	128
$N_{tx}$	16

. At the receiver side, it is assumed a perfect synchronization and a perfect CSI knowledge. The main fixed simulation parameters are also presented in Table 1, and it is assumed the worst case, i.e., $U=N_{rx}^{RF},$ since for $U\ll N_{rx}^{RF}$. The performance results were obtained by varying the modulation index, $h$, the number of users and of RF chains, of receive antennas, $N_{rx}.$ The specific values for these parameters are indicated in the figures.

First, let us evaluate the impact of modulation index on the performance of proposed hybrid system. As we can see in Figure 3, several values for $2\pi h$ were tested, and the results for 1st and 4th iterations were compared. Cleary, there is an optimal value, which is approximately $2\pi h=0.9$. A deeper discussion about the impact of the modulation index in the system performance can be found in [19]. Hereinafter, the results are obtained for $2\pi h=0.9$.

In Figure 4, we compare the simulation results with the semi-analytic BER approximation in (14). We can see that the results are very close, then we conclude that ${\mathbf{c}}_k^{}$ is approximately Gaussian distributed, and we confirm that the assumption made in Equation (12) was correct. We may also see that the performance gap from iteration 1 to iteration 2 is higher than from iteration 2 to iteration 4. This happens because most of the residual ISI and multiuser interference are removed from the 1st to the 2nd iteration. From the 2nd to the 4th iteration, the gain is smaller because most of the interference was already removed.

In Figure 5 and Figure 6, a comparison between the proposed iterative hybrid equalizer and the iterative fully digital one, for $N_{rx}=32$ and $N_{rx}=64$ was performed, respectively. All expressions of the hybrid case are valid for the full-digital case, assuming the number of RF chains are equal to the number of receive antennas. Therefore, for the fully digital case, only Algorithm 1 is used, assuming ${\mathbf{W}}_a={\mathbf{I}}_{N_{rx}}$. For $N_{rx}=32$ (Figure 5), the gain of the proposed hybrid equalizer, from the 1st to 4th iteration, for a BER target of ${10}^{-3}$, is 7.2 dB, and the penalties from hybrid to fully digital case are 7.1 dB and 2.7 dB, for iterations 1 and 4, respectively. For $N_{rx}=64$ (Figure 6), the gain of the proposed hybrid equalizer, from the 1st to 4th iteration, for a BER target of ${10}^{-3}$, is 4.4 dB, and the penalties from hybrid to fully digital case are 6.6 dB and 3.6 dB, for iterations 1 and 4, respectively. For both cases, the performances are closer for iteration 4 than for iteration 1, which means that the iterative approach is very interesting for hybrid systems.

A performance comparison for a different number of users (and of RF chains) can be seen in the Figure 7. For iteration 1, we can see that the performance gets worse when the number of users increases. This happens because when the number of users increases, the interference level also increases, and the receiver does not have the ability to remove this interference only with a linear filter, i.e., with only one iteration. However, when the number of iteration increases, this interference is removed. Then, at iteration 4, the performance can even improve when the number of users increases, because we have more degrees of freedom at the digital part, since the number of RF chains also increases.

Finally, in the Figure 8, we can see the performance for a different number of receive antennas. This figure proves that there is a gain when we only increase the complexity of analog part, in terms of number of antennas and phase shifters, keeping the same complexity for digital processing. Therefore, the analog part of proposed equalizer is very efficient to improve the system performance, proving that the hybrid analog-digital architecture is a very promising approach. The gain of proposed hybrid equalizer, from 1st to 4th iteration, for a BER target of ${10}^{-3}$, decreases when the number of receive antennas increases. This happens because a higher level of interference is removed at the analog part, when we increase the number of antennas and phase shifters. At iteration 4, the gaps from $N_{rx}=128$ to $N_{rx}=64$, and from $N_{rx}=64$ to $N_{rx}=32$ are 2.2 dB and 2.5 dB, respectively, which is a little less than the expected 3 dB, which is the expected for a typical full-digital system.

5. Conclusions

In this paper, we proposed an iterative hybrid multi-user equalizer for the uplink of broadband mmWave mMIMO CE-OFDM systems. Due to hardware constraints, the analog part of the equalizer is fixed over the iterations and subcarriers, while the digital equalizers are iterative and computed on a per subcarrier basis. These ones were designed by minimizing the sum of the MSE of all subcarriers. The user terminals employ a low complex analog precoder based on AoD.

The results showed that the proposed iterative multi-user equalizer is quite efficient to mitigate the ISI and the multi-user interference, achieving a BER performance closer to the fully digital counterpart. The results also show that by increasing the number of users (and of RF chains), the performance for iteration 4 also increases, which is a promising result for a realistic scenario with several users. Finally, when we increase the analog processing, keeping the same complexity for digital processing, the performance is improved, which proves that the analog part of proposed equalizer is very efficient to improve the overall system performance. Therefore, the proposed equalizer can be a good choice for practical broadband mmWave CE-OFDM systems employing mMIMO terminals.