Securing Relay Networks with Artificial Noise: An Error Performance-Based Approach ^†

Abstract

We apply the concept of artificial and controlled interference in a two-hop relay network with an untrusted relay, aiming at enhancing the wireless communication secrecy between the source and the destination node. In order to shield the square quadrature amplitude-modulated (QAM) signals transmitted from the source node to the relay, the destination node designs and transmits artificial noise (AN) symbols to jam the relay reception. The objective of our considered AN design is to degrade the error probability performance at the untrusted relay, for different types of channel state information (CSI) at the destination. By considering perfect knowledge of the instantaneous CSI of the source-to-relay and relay-to-destination links, we first present an analytical expression for the symbol error rate (SER) performance at the relay. Based on the assumption of an average power constraint at the destination node, we then derive the optimal phase and power distribution of the AN that maximizes the SER at the relay. Furthermore, we obtain the optimal AN design for the case where only statistical CSI is available at the destination node. For both cases, our study reveals that the Gaussian distribution is generally not optimal to generate AN symbols. The presented AN design takes into account practical parameters for the communication links, such as QAM signaling and maximum likelihood decoding.

1. Introduction

Securing data communications relies heavily on modern cryptography. Since Diffie and Hellman [1] first proposed a key exchange protocol based on computational intractability, computational cryptography approaches have been extensively studied, such as the popular one based on the integer factorization problem [2]. However, the emergence of high-performance computers may challenge the existing cryptographic algorithms relying on computational hardness. As a complementary strategy to provide secure data communications, physical layer secrecy has recently drawn considerable attention. In particular, physical layer secrecy is viewed as a promising solution to provide wireless secrecy in 5G, since it does not depend on computational complexity, and has a high scalability to allow the coexistence of communication terminals with different levels of hierarchical architectures [3]. In addition, physical layer secrecy can either provide direct secure data communication or assist the distribution of cryptographic keys, which makes it particularly favorable in 5G networks [3]. Since preliminary works [4,5] characterized the secrecy capacity for wiretap channels, secrecy communication has been extensively studied for various channel models and network setups, such as single-hop wiretap channels [6,7,8,9,10,11], multi-user networks [12,13], and relay networks [14,15,16,17,18]. In relay networks, secrecy is an important issue even without the presence of external eavesdroppers. Despite the fact that the relay accedes to the request from the source to assist in delivering the message to the destination, it is often desirable to keep the message confidential from the relay [19,20,21,22,23]. First proposed in [19], the untrusted relay scenario has drawn considerable attention, since it finds diverse and important applications in modern communication systems. For example, in heterogeneous networks, the relay terminal may have a lower secrecy clearance than the source and destination pair [21,22]. Therefore, the relay can be partly malicious in the sense that it still functions in compliance with the relaying protocol, whilst it leaks secret information. Another application is the multiple-level access control in wireless sensor networks, where sensors have different authorizations and sensitivities depending on their roles (e.g., master and slave sensors) and type of collected data [24,25,26,27]. In such setups, the relaying node might be only allowed to help forwarding messages from one terminal to another, as not all terminals have direct connections to all nodes in the sensor network. For instance, in a secure sensor network at an airport, terminals with low secrecy levels must have restricted data access, whereas a few terminals at a much higher secrecy level are allowed to access all the data [25].

To shield the messages from the untrusted relay and enhance the secrecy of the wireless communication between the source and destination, one popular scheme at the physical layer is to introduce controlled artificial noise (AN) to efficiently jam the signal reception at the relay. This technique has recently been studied from an information-theoretic perspective, such as secrecy rate and secrecy outage probability, see e.g., [22,28,29,30,31,32]. However, such metrics are in general valid for ideal communication assumptions of continuous input messages and random encoding schemes with asymptotically large block lengths. In order to take discrete modulation alphabets and finite block lengths into consideration, other secrecy performance metrics have also been proposed, such as bit error rate [33,34]. The observation that in modern wireless communication systems square quadrature amplitude modulation (QAM) is widely used motivates us to address the problem of how to optimally apply AN to enhance the physical layer secrecy in an untrusted relay network. The symbol error rate (SER) of the demodulated signal at the relay is used as a performance metric in this paper.

In this work, we consider the communication between a pair of source and destination terminals, where a direct link is absent, and therefore, a half-duplex relay terminal is utilized to assist the communication [35,36]. To forward the signal from the source to the destination, the relay employs the non-coherent amplify-and-forward (AF) protocol [37], which is considered as the most promising solution for current and future communication systems, since it offers a reasonable tradeoff between the benefits and implementation costs [38]. As a result, these relays have already been incorporated in the Universal Mobile Telecommunications System (UMTS) and Release 8 of Long-Term Evolution (LTE) in the form of repeaters [35]. For the AF protocol, we make the common assumption that the destination has the instantaneous channel state information (CSI) of the relay-to-destination link and the aggregate source-to-relay-to-destination link [39,40]. Consequently, the destination can obtain the CSI of the relay-to-destination link, which is however not available at the relay. To secure messages against the relay, we propose a novel AN scheme, where the destination designs and broadcasts AN symbols to the relay simultaneously with the transmission of the symbols from the source. It has been shown that a positive secrecy rate can be achievable in this AN-assisted untrusted relay network [41]. For this setup and with the knowledge of instantaneous CSI as assumed in Section 3, we first investigate the problem of how to optimally design the AN to maximize the SER of QAM signals at the relay. Note that the requirement of the instantaneous CSI might be strong in some practical systems. To reduce the requirement, we then investigate the AN design based on statistical CSI in Section 4, i.e., the channel variances. For fading channels, an important performance metric is the average SER (ASER), which quantifies the average decoding error performance over fading channels. Interestingly, for both CSI schemes, our study shows that it is not optimal to generate AN from a Gaussian distribution. For example, based on instantaneous CSI, we note that a QAM or a rotated QAM AN generating method maximizes the SER at the relay. The results in this paper can be used as benchmarks for future analyses of AN-based techniques. The main contributions of this work can be summarized as follows:

• By assuming that instantaneous CSI is available at the untrusted relay and the destination, upon receiving both of the QAM symbols from the source and the AN from the destination, an exact expression for the SER at the relay when decoding the QAM symbols is first derived. Under an average power constraint, this expression is then utilized to obtain the phase and power distribution of the AN symbols to maximize the SER performance at the relay.
• For the case where only statistical CSI is available, we first derive an exact expression for the ASER performance at the relay. Next, the optimal power distribution of the AN symbols that maximizes the ASER performance at the relay is determined. Numerical and simulation results demonstrate that the proposed optimal AN designs guarantee significant error rate performance enhancement compared with conventional AN designs, such as Gaussian distribution.

Furthermore, the AN-based scheme studied in this paper is formulated according to the framework of current cellular standards such as LTE/LTE-Advanced of 4G and the next major phase of mobile telecommunications standard 5G [3,36]. By applying an additional processing unit to generate AN, the studied scheme can be easily embedded in a practical system to secure wireless communications, e.g., key transmission or control signaling. Moreover, our study provides exact SER expressions of the QAM signals at the relay, which can be used as benchmarks for future extensions, e.g., deriving SER expressions for other modulation schemes, and designing AN signaling in other scenarios.

Notation: Throughout this paper, we use $\mathcal{CN}(b_1,b_2)$ to represent a complex circularly symmetric Gaussian distribution with mean $b_1$ and variance $b_2$. We use $\mathcal{E}\left\{·\right\}$ and $Pr(·)$ to denote statistical expectation and probability. Moreover, $\left|b\right|$ and $b^{*}$ represent absolute value and complex conjugate of b, respectively.

2. System Model and Problem Formulation

Let us consider a relay communication channel as shown in Figure 1. A legitimate transmitter (the source) sends information symbols to a legitimate receiver (the destination) assisted by an untrusted relay. We assume that a direct link between the legitimate terminals is not available due to, for example, high path loss. All the terminals are configured with a single antenna (Note that this work can be generalized to the scenario where at least one of the source and destination has multiple antennas. Whilst beyond the scope of this work, the corresponding precoding design at each terminal remains an interesting topic for future extension). We assume quasi-static Rayleigh fading channels and the channel coefficients for the source-to-relay and relay-to-destination links are respectively denoted by h and g, where $h\sim \mathcal{CN}\left(0,{\sigma }_h^2\right)$ and $g\sim \mathcal{CN}\left(0,{\sigma }_g^2\right)$. All channels are assumed to be reciprocal [42] and constant within the transmission duration from the source to the destination. The transmission from the source to destination can be partitioned into two time slots. During the first time slot, the source broadcasts the symbol m. Meanwhile, to prevent the relay from deciphering the information, the destination sends the AN symbol z to increase the noise level at the relay. For this purpose, all the terminals are assumed to be perfectly synchronized. Similar to physical-layer network coding [43], the synchronization is an important issue, which requires further investigation. At the end of the first time slot, the received baseband signal at the relay can be mathematically expressed as,

$$y=hm+gz+n$$

(1)

where the additive white Gaussian noise (AWGN) symbol n is assumed to be zero mean and have the one-sided power spectral density $N_0$ in Watts/Hz. During the second time slot, the source is muted, and the relay applies the amplify-and-forward protocol [37] to forward a scaled version of its received signal to the destination. Upon receiving this signal, the destination removes the AN z prior to decoding m. For the afore-mentioned communication protocol, the destination is required to have the perfect CSI of h and g in order to maximize the system’s transmission rate [44]. To be more specific, the destination performs the channel estimation of the relay-to-destination link and the compound source-to-relay-to-destination link, and the channel coefficient of the source-to-relay link is then obtained [40].

While exploiting the untrusted relay to help the data transmission, the system is designed to ensure that the untrusted relay cannot decipher the source’s symbol m. In addition, we assume that the relay can obtain knowledge about the modulation scheme of the source, e.g., by tracking the common control channel of the network. Moreover, it is important to note that the AN z in Equation (1) is designed and known solely by the destination so that the relay is not able to distinguish between the AN and the regular channel noise n. Therefore, when decoding the symbols transmitted by the source, we assume that the relay treats the AN from the destination as noise. Furthermore, we assume that the source transmits demodulation reference signals (DM-RS) [36] so that the relay can perfectly estimate h to perform ideal coherent demodulation as,

$$\begin{array}{c}\hfill \tilde{y}=\frac{h^{*}}{\left|h\right|}y=\left|h\right|m+\frac{h^{*}g}{\left|h\right|}z+\frac{h^{*}}{\left|h\right|}n.\end{array}$$

(2)

Forwarded by the relay, the destination also has access to h and the DM-RS of the source in order to demodulate the signal m. For the source’s symbol m, we consider square M-QAM modulation types with $M=4^k$ and $k=1,2,\cdots$, which are frequently used in the current and upcoming communication standards [36]. We also make the common assumption that constellation points are uniformly distributed [45]. In addition, we use $E_m$ and $E_z$ to denote the average energy per symbol for the source’s signal m and the AN z sent by the destination, respectively. Finally, $E_m$ is assumed to be known at the destination for the AN design.

Since the destination has perfect knowledge of the instantaneous channel coefficients h and g, this knowledge can be efficiently used to design the AN symbol z. The relay uses the ideal coherent demodulation with minimum distance detection to recover the source’s signal m, and the corresponding SER is the performance metric used in this paper. The objective is to find out the AN symbol z at the destination in order to maximize the SER performance of the square M-QAM modulation at the relay, which we address in Section 3.

3. AN Maximizing Relay’s SER Performance

The SER analysis in this section consists of three parts. In Section 3.1, for given instantaneous channel realizations h and g, we derive the SER expression at the relay for the given AN symbol z of square M-QAM modulation. Based on this expression, in Section 3.2, we further study the problem of how to select the phase of z given its amplitude $\left|z\right|$. The problem of assigning power to the AN symbol z is considered in Section 3.3, where we derive the optimal distribution of the power of z, i.e., ${\left|z\right|}^2$, based on the average power constraint $E_z$.

3.1. SER Expression for a Given z

The SER performance of the square M-QAM modulated signaling over AWGN [45] and over fading channels [46] has been widely studied. To provide a comprehensive study on the SER performance at the relay, we first investigate the AN design using the instantaneous CSI in this section, and our SER analysis is based on the procedure presented in ([45], Chapter 5).

As an example, the constellation diagram for the square 16-QAM modulation is illustrated in Figure 2. Denote the minimum distance between two constellation points as $2a$ ($a>0$), then for a general M-QAM constellation, the average energy per signal symbol $E_m$ can be expressed as ([45], Equation (5-2-76))

$$E_m=\frac{2}{3}{a}^2{T}_m\left(M-1\right)$$

(3)

where $T_m$ denotes the symbol duration.

Since the AN design in this section is based on h and g, for simplicity of notation, we define $s:=\frac{h^{*}g}{\left|h\right|}z$ and $\tilde{n}:=\frac{h^{*}}{\left|h\right|}n$ in Equation (2), and denote the real and imaginary parts of s and $\tilde{n}$ as $s_r$, $s_i$ and ${\tilde{n}}_r$, ${\tilde{n}}_i$, respectively. Thus, we have that $s=s_r+j{s}_i$ and $\tilde{n}={\tilde{n}}_r+j{\tilde{n}}_i$ with $j=\sqrt{-1}$. Note that the equivalent noise $\tilde{n}$ is identically distributed with n, and thus $\tilde{n}\sim \mathcal{CN}(0,2{\sigma }^2)$, where ${\sigma }^2=\frac{N_0}{2T_m}$. For a vertex such as the point “0” in Figure 2, the received symbol $\tilde{y}$ in Equation (2) lies outside the decision region of the point “0” if $s_r+{\tilde{n}}_r\ge \left|h\right|a$ or $s_i+{\tilde{n}}_i\le -\left|h\right|a$. Thus, given s, the error probability of the point “0” is given by $P_{e,0}=1-\left(1-Pr\left(s_r+{\tilde{n}}_r\ge \left|h\right|a\right)\right)\left(1-Pr\left(s_i+{\tilde{n}}_i\le -\left|h\right|a\right)\right)$, which can be computed as,

$$\begin{array}{c}\hfill P_{e,0}=1-\left(1-Q\left(\frac{\left|h\right|a-s_r}{\sigma }\right)\right)\left(1-Q\left(\frac{\left|h\right|a+s_i}{\sigma }\right)\right)\end{array}$$

(4)

where $Q\left(·\right)$ is the Gaussian Q-function ([46], Equation (4.1)). Similarly, the error probability for the ith constellation point $P_{e,l}$ ($l=0,1,\dots ,M-1$) is obtained by computing the probability that $\tilde{y}$ in Equation (2) lies outside its decision region. Based on the assumption that the constellation points are uniformly distributed, for a given z, or equivalently for a given s, the SER of the square M-QAM signal at the relay can be derived by averaging the respective error probability expressions for all symbols under the assumption of a uniform symbol distribution, resulting in,

$$\begin{array}{cc}\hfill \mathrm{SER}\left(s\right)=& c\left[Q\left(\frac{\left|h\right|a-s_r}{\sigma }\right)+Q\left(\frac{\left|h\right|a+s_r}{\sigma }\right)+Q\left(\frac{\left|h\right|a-s_i}{\sigma }\right)+Q\left(\frac{\left|h\right|a+s_i}{\sigma }\right)\right]\hfill \\ & -c^2\left[Q\left(\frac{\left|h\right|a-s_r}{\sigma }\right)+Q\left(\frac{\left|h\right|a+s_r}{\sigma }\right)\right]\left[Q\left(\frac{\left|h\right|a-s_i}{\sigma }\right)+Q\left(\frac{\left|h\right|a+s_i}{\sigma }\right)\right]\hfill \end{array}$$

(5)

where $c=\frac{\sqrt{M}-1}{\sqrt{M}}$. Using Equation (3), we have that,

$$\frac{a}{\sigma }=\sqrt{\frac{3E_m}{N_0\left(M-1\right)}}.$$

(6)

Note that in the case of $z=0$, the SER expression in Equation (5) coincides with that of the AWGN channel ([45], Equation (5-2-79)).

Before proceeding, it is interesting to note that the SER expression in Equation (5) depends on the real and imaginary parts of s, which motivates us to express s as,

$$\begin{array}{c}\hfill s=\left|g\right|\left|z\right|exp\left(j\theta \right)\end{array}$$

(7)

where we have expressed z, g and h as $z=\left|z\right|exp\left(j{\theta }_z\right)$, $g=\left|g\right|exp\left(j{\theta }_g\right)$, $h=\left|h\right|exp\left(j{\theta }_h\right)$, respectively, and $\theta ={\theta }_g-{\theta }_h+{\theta }_z$. Now, inserting Equation (7) into Equation (5), the SER expression can be rewritten as a function of $\theta$ and $\left|z\right|$ as,

$$\begin{array}{cc}\hfill \mathrm{SER}\left(\theta ,\left|z\right|\right)=& c[Q\left(\frac{\left|h\right|a}{\sigma }-\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)+Q\left(\frac{\left|h\right|a}{\sigma }+\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)\hfill \\ & +Q\left(\frac{\left|h\right|a}{\sigma }-\frac{\left|g\right|\left|z\right|}{\sigma }sin\theta \right)+Q\left(\frac{\left|h\right|a}{\sigma }+\frac{\left|g\right|\left|z\right|}{\sigma }sin\theta \right)]\hfill \\ & -c^2\left[Q\left(\frac{\left|h\right|a}{\sigma }-\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)+Q\left(\frac{\left|h\right|a}{\sigma }+\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)\right]\hfill \\ & \times \left[Q\left(\frac{\left|h\right|a}{\sigma }-\frac{\left|g\right|\left|z\right|}{\sigma }sin\theta \right)+Q\left(\frac{\left|h\right|a}{\sigma }+\frac{\left|g\right|\left|z\right|}{\sigma }sin\theta \right)\right].\hfill \end{array}$$

(8)

From Equation (8), we can observe that the channel gains $\left|h\right|$ and $\left|g\right|$ play an important role in the SER performance, which we summarize in the following propositions.

Proposition 1 (SER decreases in $\left|h\right|$).

The SER in Equation (8) is a monotonically decreasing function of $\left|h\right|$.

Proposition 2 (SER increases in $\left|g\right|$).

The SER in Equation (8) is a monotonically increasing function of $\left|g\right|$.

To prove these propositions, we denote,

$$\begin{array}{cc}\hfill \xi \left(\right|h|,|g\left|\right)=& \sum _{l=0}^{1}Q\left(\frac{\left|h\right|a}{\sigma }+{(-1)}^l\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)\hfill \end{array}$$

(9)

and,

$$\begin{array}{cc}\hfill \eta \left(\right|h|,|g\left|\right)=& \sum _{l=0}^{1}Q\left(\frac{\left|h\right|a}{\sigma }+{(-1)}^l\frac{\left|g\right|\left|z\right|}{\sigma }sin\theta \right).\hfill \end{array}$$

(10)

We first focus on Proposition 1. Since $\xi \left(\right|h|,|g\left|\right)$ is a decreasing function of $\left|h\right|$, the first derivative of $\xi \left(\right|h|,|g\left|\right)$ with respect to $\left|h\right|$ satisfies $\frac{\mathrm{d}\xi \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|h\right|}\le 0$, and for a given $\left|g\right|$, $\xi \left(\right|h|,|g\left|\right)\le \xi (0,|g\left|\right)=1$. Similarly, $\frac{\mathrm{d}\eta \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|h\right|}\le 0$ and for a given $\left|g\right|$, $\eta \left(\right|h|,|g\left|\right)\le 1$. In addition, since $0<c<1$, the first derivative of the SER expression of Equation (8) with respect to $\left|h\right|$ can be written as,

$$\begin{array}{c}\hfill c\left(1-c\xi \left(\right|h|,|g\left|\right)\right)\frac{\mathrm{d}\eta \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|h\right|}+c\left(1-c\eta \left(\right|h|,|g\left|\right)\right)\frac{\mathrm{d}\xi \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|h\right|}\le 0\end{array}$$

(11)

which proves Proposition 1. This proposition agrees with the intuition that a stronger source-to-relay link improves the decoding performance at the relay.

Regarding Proposition 2, for a given $\left|h\right|$, we use the first derivative of the Gaussian Q-function

$$\begin{array}{c}\hfill \frac{\mathrm{d}Q\left(x\right)}{\mathrm{d}x}=\frac{1}{\sqrt{2\pi }}exp\left(-\frac{x^2}{2}\right)\end{array}$$

(12)

to obtain,

$$\begin{array}{cc}\hfill \frac{\mathrm{d}\xi \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|g\right|}& =\frac{\frac{\left|z\right|}{\sigma }cos\theta }{\sqrt{2\pi }}\left[exp\left(-\frac{1}{2}{\left(\frac{\left|h\right|a}{\sigma }-\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)}^2\right)-exp\left(-\frac{1}{2}{\left(\frac{\left|h\right|a}{\sigma }+\frac{\left|g\right|\left|z\right|}{\sigma }cos\theta \right)}^2\right)\right]\hfill \\ & \ge 0\hfill \end{array}$$

(13)

which indicates that $\xi \left(\right|h|,|g\left|\right)$ is an increasing function of $\left|g\right|$. Thus, we have,

$$\begin{array}{c}\hfill \xi \left(\right|h|,|g\left|\right)\le \underset{\left|g\right|\to \infty }{lim}\xi \left(\right|h|,|g\left|\right)=1.\end{array}$$

(14)

Therefore, since $0<c<1$, the first derivative of the SER expression of Equation (8) with respect to $\left|g\right|$ can be derived as,

$$\begin{array}{c}\hfill c\left(1-c\xi \left(\right|h|,|g\left|\right)\right)\frac{\mathrm{d}\eta \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|g\right|}+c\left(1-c\eta \left(\right|h|,|g\left|\right)\right)\frac{\mathrm{d}\xi \left(\right|h|,|g\left|\right)}{\mathrm{d}\left|g\right|}\ge 0\end{array}$$

(15)

which proves Proposition 2. This proposition suggests that a stronger relay-to-destination link deteriorates the SER performance at the relay.

A careful observation of Equation (8) shows that the AN design can be decoupled into the optimal design of the phase and the amplitude, respectively. In the following subsection, we first derive the optimal rotation angle $\theta$ that maximizes the SER given by Equation (8). The value of ${\theta }_z$, which denotes the corresponding phase of the AN defined following Equation (7), is then determined accordingly.

3.2. Selecting ${\theta }_z$ for a Fixed $\left|z\right|$

The focus of this subsection is to determine the optimal phase distribution of the AN symbol z. In this case, the power ${\left|z\right|}^2$ of the AN is assumed to be fixed. Then, for simplicity, we can omit $\left|z\right|$ in the argument of $\mathrm{SER}(\theta ,|z\left|\right)$ in Equation (8) and denote the SER expression as $\mathrm{SER}\left(\theta \right)$ in the remainder of this subsection.

Due to the $\frac{\pi }{2}$-periodicity of the function $\mathrm{SER}\left(\theta \right)$ and due to its symmetry, we only consider the interval $\theta \in \left[0,\frac{\pi }{4}\right]$ to determine the maximum of $\mathrm{SER}\left(\theta \right)$. We first numerically verify the following result.

Result 1: $\mathrm{SER}\left(\theta \right)$ is a quasi-convex function of $\theta$ for all $\theta \in \left[0,\frac{\pi }{4}\right]$, and either $\theta =0$ or $\theta =\frac{\pi }{4}$ maximizes the value of $\mathrm{SER}\left(\theta \right)$.

To illustrate this result, consider the constellation point “5” in Figure 2 as an example. Given the power of the AN symbol z, in order to maximize the expected SER with square 16–QAM modulation at the relay according to Result 1, the destination can either allocate all the power ${\left|z\right|}^2$ to the direction of “12” or “13”, or equally distribute this power between the directions of “12” and “13”. Due to the similarity of this AN phase design to the constellation diagram, we refer to the case of $\theta =0$ as the rotated QAM phase selection and the latter case of $\theta =\frac{\pi }{4}$ as the QAM phase selection. Accordingly, the optimal phase selection of the AN z is computed as ${\theta }_z={\theta }_h-{\theta }_g$ or ${\theta }_z=\frac{\pi }{4}+{\theta }_h-{\theta }_g$.

Next we address the question as to in which case using the rotated QAM constellation for the AN maximizes the SER at the relay and in which case the M-QAM constellation is optimal. Towards this aim, we obtain Figure 3, which shows $\left|z\right|/\sigma$ as a function of $a/\sigma$. Note that $\left|z\right|/\sigma$ and $a/\sigma$ correspond to the levels of AN-to-natural noise ratio (ANR) and signal-to-noise ratio (SNR), respectively. Each curve in this figure is plotted by numerically solving the equation $\mathrm{SER}\left(\pi /4\right)=\mathrm{SER}\left(0\right)$ for $\left|z\right|/\sigma$ for given values of $a/\sigma$. The QAM region and the rotated QAM region denote the regions of $\mathrm{SER}\left(\pi /4\right)>\mathrm{SER}\left(0\right)$ and $\mathrm{SER}\left(\pi /4\right)<\mathrm{SER}\left(0\right)$, respectively. Therefore, Figure 3 displays phase selection thresholds for different system parameters. In this figure, $a/\sigma$ and $\left|z\right|/\sigma$ indicate the energy of the source signal m and the energy of the AN z, respectively. We focus on the relation between the signal energy and the AN energy, and thus ignore the effect of channel gains by assuming $h=1$ and $g=1$.

The curves in Figure 3 show that for a given signal power, there exists a threshold for the AN power, under which the optimal phase selection is $\theta =0$, and above which the optimal phase selection is $\theta =\pi /4$. To explain the intuition behind this, consider the constellation point “5" in Figure 2. With the purpose of maximizing the SER performance at the relay, if the power level of the AN $\left|z\right|/\sigma$ is low (as compared to a given signal power $E_m$), the AN transmitted from the destination may deviate the equalized information signals received at the relay towards the adjacent constellation points “1”, “4”, “13”, and “7”, whilst if the power level of the AN $\left|z\right|/\sigma$ is sufficiently high, the AN may shift the receive symbol towards the farther constellation points “0”, “12”, “3”, and “15”. From Figure 3, one can also observe that for small values of $a/\sigma$, the values of the threshold are very close to zero. This fact indicates that QAM phase selection is the preferable scheme if the received SNR at the relay is low.

In particular, from Equation (8), we have,

$$\begin{array}{cc}\hfill \underset{\left|z\right|\to \infty }{lim}\mathrm{SER}\left(0,\left|z\right|\right)=& c+2c(1-c)Q\left(\frac{\left|h\right|a}{\sigma }\right)\le \frac{\sqrt{M}-1}{\sqrt{M}}+\frac{2(\sqrt{M}-1)}{M}\times \frac{1}{2}=\frac{M-1}{M}\hfill \end{array}$$

(16)

and,

$$\begin{array}{c}\hfill \underset{\left|z\right|\to \infty }{lim}\mathrm{SER}\left(\pi /4,\left|z\right|\right)=2c-c^2=\frac{M-1}{M}\end{array}$$

(17)

where the equality in Equation (16) holds only when $a=0$, i.e., when no signal is transmitted. This indicates that when the signal is present, given sufficiently high AN noise power, the QAM phase selection yields a higher SER. This analysis is illustrated by the plot in Figure 3. Also note that $\frac{M-1}{M}$ is the SER when the relay does not have any prior information and randomly guesses the value of signal m for decoding, which serves as an upper bound on the SER of any AN scheme. In this work, we term this as non-informative error performance. Moreover, Equations (16) and (17) reveal that when the AN power is sufficiently high, the M-QAM selection asymptotically achieves the non-informative error performance.

3.3. Assigning $\left|z\right|$ for a Given $E_z$

In Section 3.2, we have shown how to select the phase ${\theta }_z$ of z if the amplitude $\left|z\right|$ is a fixed value. In general, however, it may be optimal to assign different powers to different AN symbols for a given average AN power. In this subsection, we assume that the average energy per AN symbol is limited to an average symbol energy $\overline{E}$, i.e.,

$$E_z=T_m\mathcal{E}\left\{{\left|z\right|}^2\right\}\le \overline{E}.$$

(18)

Furthermore, we denote $\overline{P}=\overline{E}/T_m$ as the average power of z. Based on the results in Section 3.2, the expected SER obtained from optimal phase selection is given by,

$$\widetilde{\mathrm{SER}}\left(\left|z\right|\right)=max\left\{\mathrm{SER}\left(\pi /4,\left|z\right|\right),\mathrm{SER}\left(0,\left|z\right|\right)\right\}$$

(19)

where the $\mathrm{SER}\left(\theta ,\left|z\right|\right)$ is defined according to Equation (8). Following the proof of Proposition 2, one can show that the first derivative of $\mathrm{SER}\left(\theta ,\left|z\right|\right)$ in Equation (8) with respect to $\left|z\right|$ is non-negative, and thus $\mathrm{SER}\left(\theta ,\left|z\right|\right)$ is a monotonically increasing function of $\left|z\right|$ for a given $\theta$. Accordingly, $\widetilde{\mathrm{SER}}\left(\left|z\right|\right)$ in Equation (19) is also a monotonically increasing function of $\left|z\right|$. Figure 4 displays the values of $\widetilde{\mathrm{SER}}\left(\left|z\right|\right)$ for 16-QAM and $a/\sigma =4$, where both the analytical and the simulated results are plotted. From the figure, we can see that as $\left|z\right|/\sigma$ increases, the rotated QAM and the QAM phase selections alternatively achieve a better SER performance. In this figure, note that ${\left|z\right|}^2=\overline{P}$ can be viewed as a deterministic power usage.

The objective of this subsection is to derive the optimal distribution of ${\left|z\right|}^2$ to maximize the expected value of the SER in Equation (19), given the instantaneous CSI h and g. Alternatively, the power needs to be smartly allocated to yield an upper bound on the SER performance with deterministic power usage. Denoting the probability density function (PDF) of ${\left|z\right|}^2$ as $f(·)$, by taking into account the power constraint in Equation (18), the following optimization problem can be formulated:

$$\begin{array}{cc}\hfill \underset{f\left(x\right)}{max}& {\int }_0^{\infty }\widetilde{\mathrm{SER}}\left(\sqrt{x}\right)f\left(x\right)\mathrm{d}x\hfill \end{array}$$

(20a)

$$\begin{array}{cc}\hfill \mathrm{subject}\mathrm{to}& {\int }_0^{\infty }xf\left(x\right)\mathrm{d}x\le \overline{P}\left(\mathrm{average}\mathrm{power}\right)\hfill \end{array}$$

(20b)

$$\begin{array}{c}{\int }_0^{\infty }f\left(x\right)\mathrm{d}x=1\left(\mathrm{total}\mathrm{probability}\right)\hfill \end{array}$$

(20c)

$$\begin{array}{c}f\left(x\right)\ge 0,\mathrm{for}x\ge 0(\mathrm{non}-\mathrm{negativity})\hfill \end{array}$$

(20d)

where the average power constraint Equation (20b) follows from Equation (18). For computational tractability, we only consider PDFs $f\left(x\right)$ for which all integrals in Equation (20) exist.

The following theorem provides interesting insights into the power allocation problem and meanwhile can greatly simplify the computation of the PDF of $\left|z\right|$.

Theorem 1.

(SER-maximizing power distribution): Any PDF $f\left(x\right)$ solving the problem in Equation (20) has the form,

$$f\left(x\right)=\left(1-p\right)\delta \left(x-x_1\right)+p\delta \left(x-x_2\right)$$

(21)

where $\delta \left(x\right)$ is the Dirac delta function defined by ([47], Chapter 2),

$$\begin{array}{c}\hfill {\int }_{-\infty }^{\infty }\delta (t-\tau )g\left(t\right)\mathrm{d}t=g\left(\tau \right)\end{array}$$

(22)

for any continuous function g and any value of $\tau$, and,

$$\begin{array}{c}\hfill p=\frac{\overline{P}-x_1}{x_2-x_1}\end{array}$$

(23)

with $0\le x_1\le \overline{P}\le x_2$.

Proof. 

See Appendix.

☐

This theorem means that only two types of AN symbols are generated from the destination, one with probability $(1-p)$ and one with probability p, and the corresponding powers are $x_1$ and $x_2$, respectively. It is important to note that following Appendix, Theorem 1 is valid for any $\widetilde{\mathrm{SER}}\left(\sqrt{x}\right)$ as long as $\widetilde{\mathrm{SER}}\left(\sqrt{x}\right)$ is a monotonic increasing function. Inserting Equation (21) into Equation (20), the expected SER in Equation (20a) can be computed as,

$$\begin{array}{cc}\hfill \overline{\mathrm{SER}}(x_1,x_2)=& {\int }_0^{\infty }\widetilde{\mathrm{SER}}\left(\sqrt{x}\right)f\left(x\right)\mathrm{d}x\hfill \\ \hfill =& (1-p)\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)+p\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right)\hfill \\ \hfill =& \frac{x_2-\overline{P}}{x_2-x_1}\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)+\frac{\overline{P}-x_1}{x_2-x_1}\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right).\hfill \end{array}$$

(24)

From Equation (24), at high average AN power, i.e., $\overline{P}\to \infty$, we observe that the expected SER in Equation (24) approaches $\frac{M-1}{M}$, which corresponds to the non-informative error performance. The proof can be sketched as follows. From Equations (16) and (17), we know that ${lim}_{\left|z\right|\to \infty }\widetilde{\mathrm{SER}}\left(\left|z\right|\right)=\frac{M-1}{M}$. Given $\overline{P}\to \infty$, by applying the Optimal Power Allocation Theorem, we have $x_2\to \infty$ due to $x_2\ge \overline{P}$. In this case, if $x_1\to \infty$, we have ${lim}_{x_1\to \infty }\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)=\frac{M-1}{M}$ and ${lim}_{x_2\to \infty }\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right)=\frac{M-1}{M}$, thereby $\overline{\mathrm{SER}}(x_1,x_2)$ in Equation (24) approaches $\frac{M-1}{M}$. If $x_1$ is a finite value, $x_2\approx \overline{P}$, then the term $\frac{x_2-\overline{P}}{x_2-x_1}\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)$ in Equation (21) approaches zero, and the term $\frac{\overline{P}-x_1}{x_2-x_1}\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right)$ in Equation (21) converges to $\frac{M-1}{M}$, which explains the above observation.

Another interesting observation is that the maximum $\overline{\mathrm{SER}}(x_1,x_2)$ is a monotonically decreasing function in $\left|h\right|$. This is because $\overline{\mathrm{SER}}(x_1,x_2)$ in Equation (24) is a linear combination of $\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)$ and $\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right)$, each of which is a monotonically decreasing function of $\left|h\right|$ as previously shown. Furthermore, maximizing $\overline{\mathrm{SER}}(x_1,x_2)$ with respect to $x_1$ and $x_2$ preserves the monotonicity. Similarly, it can be shown that $\overline{\mathrm{SER}}(x_1,x_2)$ in Equation (24) is a monotonically increasing function of $\left|g\right|$. Therefore, by applying the optimal AN design, the SER at the relay increases with the relay-to-destination link quality and decreases with the source-to-relay link quality.

The optimal values of $x_1$ and $x_2$ maximizing $\overline{\mathrm{SER}}(x_1,x_2)$ in Equation (24), denoted by $x_1^{*}$ and $x_2^{*}$, can be computed numerically based on h, g, $\overline{P}$, M, a, and $\sigma$. Substituting $x_1^{*}$ and $x_2^{*}$ back into Equation (24) yields the maximum expected SER, which is denoted as ${\overline{\mathrm{SER}}}_{max}$. For example, in the case of $h=1$, $g=1$, $\overline{P}=3.9811$, $M=16$, $a=\sqrt{10}$, and $\sigma =1/\sqrt{2}$, the optimal values can be computed as $x_1=0$ and $x_2=13.7098$, yielding $\widetilde{\mathrm{SER}}\left(\sqrt{x_1}\right)=0$ and $\widetilde{\mathrm{SER}}\left(\sqrt{x_2}\right)=0.5832$. The corresponding maximum expected SER is ${\overline{\mathrm{SER}}}_{max}=0.1694$. Interestingly, if we use a deterministic power usage $\overline{P}$, i.e., the AN PDF written as $f\left(x\right)=\delta (x-\overline{P})$, the corresponding expected SER is 0.0371. Therefore, applying Theorem 1 at the destination yields a SER increment of $357\%$ at the relay compared with the deterministic power usage.

Here, we summarize the procedure to design and generate AN symbols as follows:

(1)

compute the amplitudes of the AN symbols by numerically maximizing Equation (24),

(2)

determine the corresponding phase value for each amplitude using Equation (19),

(3)

generate the AN symbols by the obtained amplitudes and phases.

From the procedure, we can see that the computational complexity of the AN generation mainly stems from the optimal amplitude computation in step (1). There exist numerous non-linear optimization algorithms to solve this optimization problem, such as the Newton–Raphson method, and the Nelder–Mead method [48], which is known as “fminsearch” implemented in MATLAB. For example, using “fminsearch” with the default termination tolerance, i.e., ${10}^{-4}$, the iteration number to solve the problem is generally under 100 for the numerical examples in Section 5. Moreover, note that the computation in steps (1) and (2) is required after the change of the channel coefficients h or g.

3.4. Relation between Phase Selection and Power Allocation

Having determined the optimal phase selection and the SER-maximizing power distribution, it is interesting to examine their interplay in the SER performance, which we illustrate in Figure 5, Figure 6 and Figure 7. Figure 5 depicts the SER performance using a deterministic power level ${\left|z\right|}^2=\overline{P}$ and the optimal power allocation. The line denoted as “non-informative” represents the non-informative SER performance. In the case of 16-QAM, the non-informative SER is equal to $15/16$. Comparing Figure 4 and Figure 5, we observe that for a given SNR $E_m/N_0$, the derived Theorem 1 yields an upper bound on the SER with the deterministic power usage, i.e., the curves $\mathrm{SER}\left(0,\sqrt{\overline{P}}\right)$ and $\mathrm{SER}\left(\frac{\pi }{4},\sqrt{\overline{P}}\right)$. To take a deeper look, we depict in Figure 6 the relation between the optimal power allocation $x_1^{*}$ and $x_2^{*}$. The corresponding phase selection ${\theta }_{x_1^{*}}$ and ${\theta }_{x_2^{*}}$ are obtained from evaluating $\widetilde{\mathrm{SER}}\left(\sqrt{x_1^{*}}\right)$ and $\widetilde{\mathrm{SER}}\left(\sqrt{x_2^{*}}\right)$ in Equation (19), respectively. Figure 7 plots the probability to transmit $x_2^{*}$, i.e., p given in Equation (23), noting that the probability to transmit $x_1^{*}$ is given by $(1-p)$.

In correspondence to Equation (21), we observe from Figure 6 and Figure 7 that as the power of the AN $\overline{P}$ increases, either the probability p remains constant and the powers $x_1^{*}$ or, $x_2^{*}$ increase or the values of $x_1^{*}$ and $x_2^{*}$ remain constant (with $x_2^{*}>x_1^{*}$) and the probability p of transmission with the larger power $x_2^{*}$ increases. To simplify the discussion of Figure 5, Figure 6 and Figure 7, we partition the values of the AN-to-natural noise ratio (ANR) per symbol into multiple regions by introducing the transition points A–D. From these figures, we obtain the following observations. Region I ($0\le E_z/N_0<A$): In this region we observe that when the ANR is low, the optimal AN design is to transmit a noise symbol with power $x_2^{*}$ and rotated QAM phase $\theta =0$ at a constant probability $p_0$, and to transmit no AN ($x_1^{*}=0$) with probability $(1-p_0)$. Thus, as $\overline{P}$ increases, the power $x_2^{*}$ increases while the probability of transmitting AN remains constant.

4. AN Maximizing Relay’s ASER Performance

Region II ($A\le E_z/N_0<B$): When the ANR is medium, the power to transmit rotated QAM AN symbols $x_2^{*}$ reaches a constant, and the probability to transmit the AN symbols increases as $E_z/N_0$ increases. The corresponding SER in Figure 5 is higher than those using the deterministic power $\overline{P}$.

Region III ($B\le E_z/N_0<C$): In this region, the probability to transmit AN symbols with the rotated QAM phase selection reaches 1, and the corresponding transmitting power reaches the highest $x_2^{*}=\overline{P}$. In other words, for the AN symbols, the deterministic power usage and the rotated QAM phase selection achieve the maximum SER, which can also be observed in Figure 5.

Region IV ($C\le E_z/N_0<D$): The ANR arrives at a threshold $E_z/N_0=C$, above which the optimal AN design is to use both of the two kinds of AN symbols: the rotated QAM phase selection with power $x_1^{*}$ and the QAM phase selection with power $x_2^{*}$. The probability to transmit the QAM phase selection symbols increases as the ANR increases.

Region V ($E_z/N_0\ge D$): When the ANR is large, the QAM phase selection with a deterministic power $\overline{P}$ achieves the maximum SER. In particular, when $E_z/N_0$ is sufficiently large, i.e., $E_z/N_0\ge 17$ dB in Figure 5, the SER converges to the non-informative SER, which was mathematically proved in Equation (17).

To intuitively understand the above observations, again take the constellation point “5” in Figure 2 as an example. When the AN power is low, the optimal AN generation scheme is to burst the limited power to move “5” towards the adjacent points “1”, “4”, “13” and “7” in order to introduce decoding errors. As the AN power becomes sufficiently large, some of the power can be used to move “5” towards the points “0”, “12”, “3” and “15” in order to induce more decoding errors. Until the AN power becomes significantly large, beaming all the power to the directions of points “0”, “12”, “3” and “15” yields the maximum SER.

In Section 3, we have obtained the optimal phase and amplitude distribution of the AN based on the instantaneous channel knowledge of the source-to-relay and relay-to-destination links. In this section, we consider the optimal distribution of the AN symbol power${\left|z\right|}^2$ that maximizes the ASER performance of the M-QAM modulation at the relay provided that the long-term statistical CSI is known to the destination. Following a similar approach to Section 3, we first present an analytical expression for the aforementioned performance metric over Rayleigh fading channels. The optimal phase and power distributions of the AN that maximizes this performance are then determined.

4.1. ASER Expression for a Given z

In this section, the channel coefficients h and g are assumed to be random variables. Moreover, the envelope of the relay-to-destination channel coefficient g is assumed to be Rayleigh-distributed. For a given AN symbol z, the received AN at the relay $gz$ can be easily shown to be distributed as $gz\sim \mathcal{CN}\left(0,{\sigma }_g^2{\left|z\right|}^2\right)$. This indicates that the received AN at the relay is an extra source of the AWGN n. Hence, as can be observed from the signal model in Equation (1), the overall received noise at the relay is a superposition of the AN and the natural AWGN. The relay-to-destination channel coefficient g and the AWGN symbol n are independent, and it can be easily shown that the received noise $gz+n$ is distributed as $gz+n\sim \mathcal{CN}\left(0,{\sigma }_g^2{\left|z\right|}^2+2{\sigma }^2\right)$, where $2{\sigma }^2$ is the variance of the AWGN n. For Rayleigh faded source-to-relay channel gain $\left|h\right|$ and the overall noise symbol $gz+n$, the performance of the M-QAM modulation at the relay can be obtained using ([46], Equation (8.106)) as,

$$\begin{array}{cc}\hfill \mathrm{ASER}\left(z\right)=2c\left(1-\sqrt{\frac{1.5{\overline{\gamma }}_s\left(z\right)}{M-1+1.5{\overline{\gamma }}_s\left(z\right)}}\right)-c^2[1-& \sqrt{\frac{1.5{\overline{\gamma }}_s\left(z\right)}{M-1+1.5{\overline{\gamma }}_s\left(z\right)}}\hfill \\ & \times \left(\frac{4}{\pi }{tan}^{-1}\sqrt{\frac{M-1+1.5{\overline{\gamma }}_s\left(z\right)}{1.5{\overline{\gamma }}_s\left(z\right)}}\right)]\hfill \end{array}$$

(25)

where ${\overline{\gamma }}_s\left(z\right)=\frac{{\sigma }_h^2{E}_m}{{\sigma }_g^2{\left|z\right|}^2+2{\sigma }^2}$ represents the average received SNR per symbol.

Similar to Propositions 1 and 2 made on Equation (8), the effects of channel statistics on the ASER given by Equation (25) can be summarized as follows.

Proposition 3 (ASER monotonicity).

The ASER in Equation (25) is a monotonically increasing function of ${\sigma }_g^2$, as well as a monotonically decreasing function of ${\sigma }_h^2$.

This proposition is intuitive and is similar to Propositions 1 and 2 in Section 3: Stronger relay-to-destination channels help to deteriorate the ASER performance at the relay, whereas stronger source-to-relay channels improve the average decoding performance at the relay.

Proposition 4 (high signal and AN power performance).

At high SNR and high ANR, i.e., $E_m\gg {\sigma }^2$ and ${\left|z\right|}^2\gg {\sigma }^2$, ${\overline{\gamma }}_s\left(z\right)\approx \frac{{\sigma }_h^2}{{\sigma }_g^2}\frac{E_m}{{\left|z\right|}^2}$.

Proposition 4 reveals that provided an adequately high signal and AN power is available, the relative strength of the source-to-relay and relay-to-destination links affects the ASER performance of the QAM signals at the relay.

Proposition 5 (high AN power performance).

In the high AN power regime, i.e., for ${\left|z\right|}^2\to \infty$, ${\overline{\gamma }}_s\to 0$, we have ${lim}_{\left|z\right|\to \infty }\mathrm{ASER}\left(z\right)=2c-c^2=\frac{M-1}{M}$.

This proposition shows that given sufficiently high AN power, the ASER at the relay is close to the non-informative error performance.

Furthermore, we observe that the ASER expression in Equation (25) is independent of the phase of the AN, which is different from the AN design in Section 3 in the case of instantaneous CSI. Following a similar approach as in Section 3.3, our objective in the following subsection is to derive the SER-maximizing power distribution for the AN under an average power constraint.

4.2. Assigning $\left|z\right|$ for a Given $E_z$

We rewrite the function $\mathrm{ASER}\left(z\right)$ in Equation (25) as $\mathrm{ASER}\left(\sqrt{x}\right)$ to reflect the effect of AN power, where $x={\left|z\right|}^2$. Similar to Subsection 3.3, the optimal power assignment problem is given by replacing $\widetilde{\mathrm{SER}}\left(\sqrt{x}\right)$ in Equation (20) by $\mathrm{ASER}\left(\sqrt{x}\right)$, and now $f\left(x\right)$ represents the PDF of x for Rayleigh fading channels. Since one can see that $\mathrm{ASER}\left(\sqrt{x}\right)$ in Equation (25) is a monotonically increasing function of x, the distribution of the optimal AN directly follows from the PDF expression derived in Equation (21). Therefore, substituting Equation (21) into Equation (25) yields the expected ASER,

$$\overline{\mathrm{ASER}}(x_1,x_2)=\frac{x_2-\overline{P}}{x_2-x_1}\mathrm{ASER}\left(\sqrt{x_1}\right)+\frac{\overline{P}-x_1}{x_2-x_1}\mathrm{ASER}\left(\sqrt{x_2}\right)$$

(26)

where $\overline{P}=\frac{E_z}{T_0}$ is the average power constraint on the AN, and $0\le x_1\le \overline{P}\le x_2$. The optimal values of $x_1$ and $x_2$ maximizing $\overline{\mathrm{ASER}}(x_1,x_2)$ in Equation (26) are then derived numerically based on $\overline{P}$, M, $E_m$, $\sigma$, ${\sigma }_h^2$, and ${\sigma }_g^2$, where ${\sigma }_h^2$ and ${\sigma }_g^2$ represent the statistical/long-term CSI. Note that the computational complexity of the derivation has been investigated in Section 3.3.

Similar to the analysis following Equation (24), we can also show that as the AN power increases, the expected ASER in Equation (26) converges to the non-informative SER performance.

5. Numerical and Simulation Results

In this section, we present simulation results to validate the obtained analytical results derived in the previous sections. In all figures, the average SNR to the transmitted information symbol is chosen as $E_m/N_0=10$ dB.

In order to depict the SER performance at the relay using the optimal AN design in Section 3, we plot Figure 8, where the signal symbol m is randomly selected from the square 4-QAM constellation, and the channel gains g and h are simply assigned as 1. We observe that all analytical results coincide very well with the corresponding simulation results. For the sake of comparison, the expected SER values for Gaussian AN are also plotted in Figure 8. In the Gaussian case, we assume that the AN symbol z is generated according to a complex Gaussian distribution with the same average symbol energy $E_z$. The line denoted as “non-informative” represents the non-informative SER performance, which has been defined after Equation (17). The curve “w/o AN” in the figure depicts the SERs at the relay without AN, which lie at a value of 0.0016 for the given parameters. Figure 8 shows that the SER at the relay can be significantly increased by applying our AN design. Even when the ANR $E_z/N_0$ is small, e.g., $E_z/N_0=2$ dB, the SER is increased from 0.0016 to 0.05, i.e., an increment of $303\%$. Moreover, Figure 8 clearly demonstrates that the Gaussian distribution is not optimal for the AN symbol generation, since our scheme described in Section 3 can induce much larger SER at the relay. For the case shown in Figure 8, if the ANR $E_z/N_0$ is above 16 dB, the maximum SER, i.e., ${\overline{\mathrm{SER}}}_{max}$, achieves $3/4$. This fact indicates that if the AN z is properly generated and $E_z$ is above a certain threshold, applying ideal coherent demodulation at the relay does not lead to better performance than the non-informative case, and secure data transmission can therefore be ensured by the proposed AN design. Figure 9 shows the SER performance for square 16-QAM constellations. Comparing Figure 9 with Figure 8, similar observations can be made as for the case of 4-QAM.

Now, we investigate the ASER performance at the relay using the optimal AN design in Section 4. Figure 10 depicts the ASER performance for square 4-QAM and 16-QAM signals. In this figure, for each QAM constellation, the curve “${\overline{\mathrm{ASER}}}_{max}$ Ana”. is computed by numerically maximizing $\overline{\mathrm{ASER}}(x_1,x_2)$ in Equation (26) with respect to $x_1$ and $x_2$. The maximization revealed that the optimal power allocation is $x_2^{*}=\overline{P}$ with probability $p=1$ for both of 4-QAM and 16-QAM signals. In other words, different from the AN design based on the instantaneous CSI in Section 3, constant power usage is optimal to design the AN based on the statistical CSI. Intuitively, due to the averaging effect over several channel realizations, the ASER performance is not sensitive to the different instantaneous power allocations. For comparison, the ASER curves (${\overline{\mathrm{ASER}}}_{max}$ Ana./${\overline{\mathrm{ASER}}}_{max}$ Sim.) for various power distributions are plotted, i.e., the uniform and the exponential power distribution. For each curve, the numerical results are obtained from computing ${\int }_0^{\infty }\mathrm{ASER}\left(\sqrt{x}\right)f\left(x\right)\mathrm{d}x$, where,

$$\begin{array}{c}\hfill f\left(x\right)=\frac{1}{2\overline{P}},\mathrm{for}0\le x\le 2\overline{P}\end{array}$$

(27)

and,

$$\begin{array}{c}\hfill f\left(x\right)=\frac{1}{\overline{P}}exp\left(-\frac{x}{\overline{P}}\right),\mathrm{for}x\ge 0\end{array}$$

(28)

are the PDFs of the AN power with uniform and exponential distributions, respectively. Note that there exist numerous AN designs with exponentially distributed power, such as the Gaussian distributed AN. Figure 10 shows that our proposed scheme with the SER-maximizing power distribution yields the largest ASER at the relay. Given sufficiently high ANR, all the ASER curves achieve the non-informative error performance, which confirms Proposition 5. However, the performance difference between different AN designs is not significant. This can be explained by the fact that the ASER expression in Equation (25) does not consider the phase of the AN, and the degree of freedom of the phase design for the AN is not utilized.

Furthermore, it is interesting to compare the SER performance of the optimal AN designs in Section 3 and Section 4. Figure 11 compares the ASER performance of various AN designs. In the figure, the “Instantaneous CSI-Based Sim.” curve is plotted by computing ${\mathcal{E}}_{h,g}\left[{\overline{\mathrm{SER}}}_{max}\right]$ using the Monte Carlo method, and the curve “Statistical CSI-Based Ana.” is plotted using the same method as in Figure 10. The curve “Gaussian Sim.” is plotted by numerically computing ${\mathcal{E}}_{h,g}\left[\mathrm{SER}\left(z\right)\right]$, with Gaussian-distributed AN. From Figure 11, we observe that the instantaneous CSI-based AN design yields a higher ASER than the statistical CSI-based AN design, which is consistent with our intuition. However, the performance difference between these two designs is not quite significant. Without optimal phase and power designs, the Gaussian distribution performs worse than both CSI-based AN designs.

6. Conclusions

In this paper, we have investigated physical layer secrecy for a two-hop single-antenna relay channel, where one source aims to transmit to one destination assisted by one untrusted relay. In particular, the source adopts a squared M-square quadrature amplitude modulated (M-QAM) modulation scheme, which is widely used in current cellular standards. A novel artificial noise (AN) scheme was proposed to achieve physical layer secrecy to protect the M-QAM signals from the source to the destination. The objective was to design AN symbols generated by the destination node that degrades the error probability performance at the untrusted relay by maximizing its achieved symbol error rate (SER) performance. For the case where perfect instantaneous channel state information (CSI) of the source-to-relay and relay-to-destination links is available at the destination, we have derived exact analytical SER expressions for the relay and studied the optimal design of the AN signal to maximize the corresponding SER. It was shown that the Gaussian distribution, which is frequently used in the context of AN [7,49,50], is not optimal in general. The optimal AN for the considered relay channel was found to be QAM or rotated QAM phase selection. Moreover, compared with the Gaussian AN, our optimal AN distribution can yield remarkably higher SER at the relay. For the case where the AN design is based on the long-term CSI of the source-to-relay and relay-to-destination links, we presented the corresponding average SER (ASER) expression at the relay, and the optimal AN distribution to maximize the ASER performance was determined accordingly. Interestingly, it was shown that the phase of the AN does not affect the ASER performance at the relay. Regarding the design of the power distribution of the AN, the obtained power distribution was shown to deliver improved ASER performance compared to various other power distributions, such as the uniform and exponential distributions. Finally, we mention that there are many directions for further extensions of this work. For example, when perfect CSI of the source-to-relay and relay-to-destination links is not available, opportunistic scheduling with low-rate CSI feedback might be applied [51]. Other interesting directions include the AN design for imperfect CSI knowledge as well as for the multiple-antenna relay channel and two-way communication networks.