Secrecy-Constrained UAV-Mounted RIS-Assisted ISAC Networks: Position Optimization and Power Beamforming

Abstract

This paper investigates secrecy solutions for integrated sensing and communication (ISAC) systems, leveraging the combination of a reflecting intelligent surface (RIS) and an unmanned aerial vehicle (UAV) to introduce new degrees of freedom for enhanced system performance. Specifically, we propose a secure ISAC system supported by a UAV-mounted RIS, where an ISAC base station (BS) facilitates secure multi-user communication while simultaneously detecting potentially malicious radar targets. Our goal is to improve parameter estimation performance, measured by the Cramér–Rao bound (CRB), by jointly optimizing the UAV position, transmit beamforming, and RIS beamforming, subject to constraints including the UAV flight area, communication users’ quality of service (QoS) requirements, secure transmission demands, power budget, and RIS reflecting coefficient limits. To address this non-convex, multivariate, and coupled problem, we decompose it into three subproblems, which are solved iteratively using particle swarm optimization (PSO), semi-definite relaxation (SDR), majorization–minimization (MM), and alternating direction method of multipliers (ADMM) algorithms. Our numerical results validate the effectiveness of the proposed scheme and demonstrate the potential of employing UAV-mounted RIS in ISAC systems to enhance radar sensing capabilities.

1. Introduction

Integrated sensing and communication (ISAC), a core feature of next-generation wireless networks, fuses high-quality communication and high-precision sensing functionalities [1,2,3]. Compared to systems dedicated solely to communication and sensing, ISAC can effectively mitigate spectrum resource conflicts. Specifically, by sharing spectrum resources, hardware platforms, and signal processing frameworks, ISAC enables the joint design of communication and sensing functionalities, thereby reducing hardware costs and signaling overhead [4]. To achieve the joint design of dual functionalities, academia and industry have focused on signal processing techniques for ISAC, including waveform design and beamforming, aiming to collaboratively optimize sensing performance and communication performance [5,6,7]. Typical communication performance metrics include the achievable sum rate and signal-to-interference-plus-noise ratio (SINR), while sensing performance is often evaluated using metrics such as the signal-to-noise ratio (SNR), sensing mutual information (MI), beampattern mean square error (MSE), and Cramér–Rao bound (CRB). However, due to the broadcast nature of wireless channels, the communication information embedded in radar detection signals poses a high-security risk of interception and exploitation when the sensing target is malicious (i.e., a potential eavesdropper) [8]. As a result, information security issues in ISAC systems need to be considered and solved.

Reconfigurable intelligent surface (RIS) is expected to serve as a key enabler for ISAC by reconfiguring signal propagation environments. RIS consists of a two-dimensional (2D) metasurface with numerous independently tunable passive reflecting elements [9,10]. By precisely tuning the phase shifts of these elements, the signals re-radiated by the RIS can be coherently overlapped with the direct link signals, thereby improving the SNR for ISAC signals and disrupting eavesdropping [11]. This capability greatly improves sensing accuracy and communication security. However, traditional RIS is typically fixed on building surfaces, and its limited coverage can restrict the performance enhancement of RIS. Benefiting from the high mobility of unmanned aerial vehicles (UAVs), the position of UAV-mounted RIS can be flexibly adjusted, thus significantly improving the probability of establishing line-of-sight (LoS) links [12,13].

Although UAV-mounted RIS provides new possibilities for the implementation of secure ISAC networks, its practical application still faces numerous technical challenges that need to be tackled. First, sensing waveforms can expose private information to malicious sensing targets, making it highly challenging to achieve information eavesdropping prevention while satisfying sensing performance. Second, while the high mobility of UAVs provides greater flexibility in RIS deployment, it also raises complex issues related to optimizing UAV position and RIS beamforming, which require effective algorithmic solutions. Finally, the design of dual-function beamforming must not only satisfy the performance requirements of both communication and sensing but also achieve efficient energy allocation and control under limited transmission power budget to further enhance overall system performance.

Motivated by the aforementioned discussions, we investigate a secure ISAC system assisted by a UAV-mounted RIS, focusing on the parameter estimation performance of the radar sensing component. The primary contributions are summarized as follows:

• First, we introduce a novel UAV-mounted RIS-assisted secure ISAC system, where an ISAC base station (BS) transmits signals for both secure multi-user communication and the detection of a potentially malicious radar target. We develop signal models for reception for both the communication users and the ISAC BS and then derive the SINR-based communication performance metric as well as the CRB-based radar sensing performance metric.
• Second, we formulate the joint UAV position, BS transmit beamforming, and the RIS reflection beamforming design problem that aims to minimize the CRB while satisfying the UAV flight area, communication users’ quality of service (QoS) requirements, secure transmission demands, transmit power budget, and RIS reflection coefficient constraints. To tackle this non-convex, multivariate, and coupled problem, we decompose the original problem into three subproblems that are solved using an alternating iterative approach. Subsequently, we propose efficient algorithms based on particle swarm optimization (PSO), semi-definite relaxation (SDR), majorization–minimization (MM), and alternative direction method of multipliers (ADMM) to address these challenging subproblems.
• Finally, we present extensive simulation results demonstrating that the deployment of UAV-mounted RIS can significantly enhance radar sensing performance in secure ISAC systems. Moreover, the simulation results show that the proposed optimization framework outperforms the state-of-the-art work [14] and the baseline schemes in terms of the CRB.

The rest of this manuscript is outlined as follows: Related work is surveyed and discussed in Section 2. The system model, including the communication signal model, the radar signal model, and the security model, as well as the CRB minimization problem for target DoA estimation, is formulated in Section 3. In Section 4, the joint optimization problem for CRB minimization is reformulated and decomposed into three subproblems, which are solved iteratively using PSO, SDR, MM, and ADMM algorithms. Finally, the simulation results are presented in Section 5, and the conclusions are given in Section 6. For clarity, the main symbols and corresponding physical meanings are summarized in

Table 1. List of main symbols and the physical meanings.

M	Number of the RIS elements	N	Number of transmit/receive antennas at the BS
K	Number of legitimate users	L	Number of samples
${\gamma }_k$	QoS requirement for the legitimate user k	${\gamma }_{e,k}$	Eavesdropping SINR threshold
$\theta$	DoA of the target with respect to the RIS	$P_{\mathrm{BS}}$	Power budget at the BS
$\mathbf{W}$	Communication–radar transmit beamforming matrix	$\mathbf{x}[l]$	Transmitted communication–radar signal at the BS in time slot l
${\mathbf{h}}_{\mathrm{d},\mathrm{k}}$	Channel from the BS to the k-th user	$\mathbf{G}$	Channel from the BS to RIS
${\mathbf{h}}_{\mathrm{r},\mathrm{k}}$	Channel from the RIS to the k-th user	${\mathbf{h}}_{\mathrm{r},\mathrm{t}}$	Channel from the RIS to target
$\mathbf{\Theta }$	Reflection beamforming matrix of the RIS	${\alpha }_{\mathrm{t}}$	Radar cross-section of the target
$\mathbf{q}$	Horizontal position of the UAV-mounted RIS	$h_{\mathrm{U}}$	Specific altitude of the UAV-mounted RIS

.

2. Related Work

2.1. RIS-Assisted ISAC System

There have been abundant works investigating the coordination gain of ISAC and RIS. The authors of [15,16] investigated multiple-input–multiple-out (MIMO) integration in RIS-assisted ISAC systems, focusing on maximizing the radar SINR through the joint design of ISAC waveform and RIS beamforming. The authors of [17,18] proposed deploying RIS to overcome blockages by establishing additional cascaded links for target detection and assisting multi-user communication, aiming to maximize communication performance while ensuring radar detection performance. Additionally, the authors of [19] focused on maximizing the joint utility of the data transmission rate and sensing MI in RIS-assisted ISAC systems. Beyond the aforementioned performance metrics, the CRB is often used as a lower bound for any unbiased estimators to characterize sensing performance. For instance, the authors in [20] minimized the CRB for estimating the direction of arrival (DoA) of one target by jointly designing transmit beamforming and RIS reflection beamforming while satisfying the communication users’ SINR requirements. The authors in [21] studied maximizing the multi-user achievable sum rate under the SNR constraint for target detection or the CRB constraint for parameter estimation in RIS-assisted ISAC systems.

2.2. RIS-Assisted Secure Transmission System

Considering the risk of radar–communication information leakage, existing studies have proposed leveraging RIS to enhance physical layer security (PLS) by exploiting the differences between legitimate and eavesdropping channels [22,23,24,25]. In addition, RIS can also improve security performance by creating a favorable wireless propagation environment by leveraging artificial noise (AN) or jamming signals. In [26], the authors considered a scenario where the sensing target acted as a potential eavesdropper, maximizing the radar SINR under security constraints through the joint design of transmit and receive beamforming. The authors of [27,28] jointly optimized RIS and transmit beamforming to focus signal energy on legitimate users while disrupting the eavesdropper’s channel. In [29], the BS was used to perform beamforming on private information for legitimate users while employing AN to jam eavesdroppers, thereby ensuring the transmission security of ISAC systems assisted by aerial RIS. The authors of [30,31] demonstrated that deploying RIS on UAVs can further expand transmission coverage and enhance system performance, revealing new degrees of freedom (DoFs) for ISAC systems.

2.3. Synthesis

Different from the above works, this paper investigates security solutions for UAV-mounted RIS-assisted ISAC systems. This is because existing works [15,16,17,18,19,20,21] primarily focus on the design and optimization of ground-based RIS-assisted ISAC networks while neglecting potential security issues. Moreover, most works [15,16,17,18,19,20] evaluate ISAC performance from a communication perspective, whereas the authors of [21] analyzed the sensing performance using CRB for target DoA estimation but did not consider security constraints and UAV position optimization.

Motivated by this, we propose deploying RIS on a UAV, leveraging the high mobility of the UAV to expand the deployment flexibility of RIS. Additionally, we conduct an in-depth analysis of the sensing performance using CRB and consider the potential security issues in ISAC systems. In this situation, the position of the UAV-mounted RIS can be dynamically adjusted according to the channel conditions. Moreover, we aim to minimize the CRB under the QoS requirements of communication users and secure transmission demands, thereby achieving an effective trade-off between sensing performance and ensuring secure transmission. To the best of our knowledge, this is the first study on the parameter estimation performance of a secure ISAC system with the assistance of the UAV-mounted RIS.

3. System Model and Problem Formulation

3.1. System Model

We consider a secure ISAC system supported by a UAV-mounted RIS, in which an ISAC BS communicates with K single-antenna legitimate users while sensing a potential malicious target, as illustrated in Figure 1. The ISAC BS is equipped with N transmit/receive antennas arranged as a uniform linear array (ULA) to transmit communication–radar signals and receive echo signals. The UAV-mounted RIS with the M element is located at a specific altitude $h_{\mathrm{U}}$, and $\mathbf{q}=\left(x_{\mathrm{U}},y_{\mathrm{U}}\right)$ denotes its horizontal coordinate. The target, acting as a potential single-antenna eavesdropper, is located in the non-line-of-sight (NLoS) region of the BS, as the direct LoS link between the BS and the target is obstructed by obstacles such as trees and buildings.

The communication–radar signal transmitted during the l-th time slot can be expressed as

$$\begin{array}{cc}\hfill \mathbf{x}[l]& ={\mathbf{W}}_{\mathrm{c}}{\mathbf{s}}_{\mathrm{c}}[l]+{\mathbf{W}}_{\mathrm{r}}{\mathbf{s}}_{\mathrm{r}}[l]\hfill \\ \hfill & =[{\mathbf{W}}_{\mathrm{c}}{\mathbf{W}}_{\mathrm{r}}]{[{\mathbf{s}}_{\mathrm{c}}{[l]}^T{\mathbf{s}}_{\mathrm{r}}{[l]}^T]}^T=\mathbf{Ws}[l],\hfill \end{array}$$

(1)

where ${\mathbf{W}}_{\mathrm{c}}\in {\mathbb{C}}^{N\times K}$ and ${\mathbf{W}}_{\mathrm{r}}\in {\mathbb{C}}^{N\times N}$ denote the communication/radar beamforming matrices. $\mathbf{W}\in {\mathbb{C}}^{N\times (K+N)}$ denotes the equivalent dual-function communication–radar transmit beamforming matrix. ${\mathbf{s}}_{\mathrm{c}}[l]\in {\mathbb{C}}^{K\times 1}$ represents the communication signals’ vector, satisfying $\mathbb{E}\{{\mathbf{s}}_{\mathrm{c}}[l]{\mathbf{s}}_{\mathrm{c}}{[l]}^H\}={\mathbf{I}}_K$. ${\mathbf{s}}_{\mathrm{r}}[l]\in {\mathbb{C}}^{N\times 1}$ represents the radar probing signals’ vector, satisfying $\mathbb{E}\{{\mathbf{s}}_{\mathrm{r}}[l]{\mathbf{s}}_{\mathrm{r}}{[l]}^H\}={\mathbf{I}}_N$ and $\mathbb{E}\{{\mathbf{s}}_{\mathrm{c}}[l]{\mathbf{s}}_{\mathrm{r}}{[l]}^H\}=0$.

(1) Communication Signal Model: The compound received signal for the k-th communication user can be expressed as

$$y_k[l]=({\mathbf{h}}_{\mathrm{d},k}^T+{\mathbf{h}}_{\mathrm{r},k}^T\mathbf{\Theta }\mathbf{G})\mathbf{x}[l]+n_k[l],$$

(2)

where $\mathbf{\Theta }=\mathrm{diag}(\mathit{\psi }), \mathit{\psi }={[{\psi }_1,{\psi }_2,\dots ,{\psi }_M]}^T, \left\{{\psi }_{\mathrm{i}}|{\psi }_{\mathrm{i}}\in \mathbb{C}, \left|{\psi }_i\right|=1, \forall i=1,2,\dots ,M\right\}$ represents the RIS reflection beamforming matrix. ${\mathbf{h}}_{\mathrm{r},k}\in {\mathbb{C}}^{M\times 1}$, ${\mathbf{h}}_{\mathrm{d},k}\in {\mathbb{C}}^{N\times 1}$, and $\mathbf{G}\in {\mathbb{C}}^{M\times N}$ denote the channel matrices from the RIS/BS to the k-th users, and the BS to the RIS, respectively. In addition, the scalar term $n_k[l]$ is modeled as additive white Gaussian noise (AWGN), which follows a complex Gaussian distribution $\mathcal{C}\mathcal{N}(0,{\sigma }_k^2)$. Accordingly, the SINR of the k-th legitimate user can be calculated as

$$\mathrm{SIN}{\mathrm{R}}_k={\displaystyle \frac{|{\mathbf{h}}_k^T{\mathbf{w}}_k{|}^2}{{\sum }_{i=1,i\ne k}^{K+N}|{\mathbf{h}}_k^T{\mathbf{w}}_i{|}^2+{\sigma }_k^2}},$$

(3)

where for brevity the equivalent composite channel ${\mathbf{h}}_k^T\in {\mathbb{C}}^{1\times N}$ from the BS to the k-th user is defined as ${\mathbf{h}}_k^T\triangleq {\mathbf{h}}_{\mathrm{d},k}^T+{\mathbf{h}}_{\mathrm{r},k}^T\mathbf{\Theta }\mathbf{G}$, and ${\mathbf{w}}_i$ represents the i-th column of $\mathbf{W}$.

(2) Radar Signal Model: The transmitted signal can only reach the target via the reflected paths and then back to the BS along the same paths, as the direct BS–target link is obstructed by obstacles. Consequently, the echo signal received at the BS can be expressed as

$${\mathbf{y}}_{\mathrm{r}}[l]={\alpha }_{\mathrm{t}}{\mathbf{G}}^T\mathbf{\Theta }{\mathbf{h}}_{\mathrm{r},\mathrm{t}}{\mathbf{h}}_{\mathrm{r},\mathrm{t}}^T\mathbf{\Theta }\mathbf{GWs}[l]+{\mathbf{n}}_{\mathrm{r}}[l],$$

(4)

where ${\alpha }_{\mathrm{t}}\sim \mathcal{C}\mathcal{N}(0,{\sigma }_{\mathrm{t}}^2)$ is the target’s radar cross-section (RCS). ${\mathbf{h}}_{\mathrm{r},\mathrm{t}}\in {\mathbb{C}}^{M\times 1}$ represents the RIS target’s LoS channel, and ${\mathbf{n}}_{\mathrm{r}}[l]\sim \mathcal{C}\mathcal{N}(\mathbf{0},{\sigma }_{\mathrm{r}}^2{\mathbf{I}}_N)$ is AWGN at the BS. Specifically, ${\mathbf{h}}_{\mathrm{r},\mathrm{t}}\triangleq {\alpha }_{\mathrm{r},\mathrm{t}}{\mathbf{a}}_M(\theta )$, where the steering vector is ${\mathbf{a}}_M(\theta )\triangleq {[1,e^{j\pi sin\theta },\dots ,e^{j(M-1)\pi sin\theta }]}^T$, $\theta$ represents the DoA of the target relative to the RIS, and ${\alpha }_{\mathrm{r},\mathrm{t}}$ is the path loss. We assume that all channels are perfectly known by the BS via conventional uplink training methods.

Radar sensing involves the analysis of the received echo signals over L samples, which are stacked together to represent the combined received echo signals as

$${\mathbf{Y}}_{\mathrm{r}}={\alpha }_{\mathrm{t}}{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}+{\mathbf{N}}_{\mathrm{r}},$$

(5)

where we define ${\mathbf{Q}}_{\mathrm{t}}\triangleq {\mathbf{G}}^T\mathbf{\Theta }{\mathbf{h}}_{\mathrm{r},\mathrm{t}}{\mathbf{h}}_{\mathrm{r},\mathrm{t}}^T\mathbf{\Theta }\mathbf{G}$. Note that the received signal and noise are stacked as $\mathbf{S}\triangleq [\mathbf{s}[1],\dots ,\mathbf{s}[L]]$ and ${\mathbf{N}}_{\mathrm{r}}\triangleq [{\mathbf{n}}_{\mathrm{r}}[1],\dots ,{\mathbf{n}}_{\mathrm{r}}[L]]$, respectively. Then, we vectorize (5) as follows:

$$\tilde{\mathbf{y}}=\mathrm{vec}\{{\mathbf{Y}}_{\mathrm{r}}\}=\mathbf{y}+\mathbf{n},$$

(6)

in which we define $\mathbf{y}\triangleq {\alpha }_{\mathrm{t}}\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}$ and $\mathbf{n}\triangleq \mathrm{vec}\{{\mathbf{N}}_{\mathrm{r}}\}$.

For radar sensing, the accuracy of parameter estimation is typically assessed using the CRB, which represents the theoretical lower bound on the variance of estimation errors under optimal unbiased estimation methods. In our considered system, the BS aims to estimate the unknown target parameters $\mathit{\xi }\triangleq {[\theta ,{\mathsf{\alpha }}^T]}^T$, where $\mathsf{\alpha }\triangleq {[\Re \{{\alpha }_{\mathrm{t}}\},\Im \{{\alpha }_{\mathrm{t}}\}]}^T$. Since extracting detailed target information from $\mathsf{\alpha }$ is challenging, we focus primarily on the estimation of the DoA parameter $\theta$. The CRB matrix for estimating $\mathit{\xi }$ is shown as

$$\begin{array}{c}\hfill {\mathbf{CRB}}_{\mathit{\xi }}={\mathbf{D}}^{-1}(\mathit{\xi }),\end{array}$$

(7)

where $\mathbf{D}(\mathit{\xi })$ represents the Fisher Information Matrix (FIM). According to [32], based on the complex observation $\tilde{\mathbf{y}}\sim \mathcal{C}\mathcal{N}(\mathbf{y}(\mathit{\xi }),{\mathbf{R}}_{\mathrm{n}}(\mathit{\xi }))$ with ${\mathbf{R}}_{\mathrm{n}}\triangleq {\sigma }_{\mathrm{r}}^2{\mathbf{I}}_{NL}$, each element of the FIM ${\mathbf{D}}_{\mathit{\xi }}\in {\mathbb{C}}^{3\times 3}$ used to estimate the $\mathit{\xi }$ can be calculated as follows:

$$\begin{array}{c}\hfill {\mathbf{D}}_{\mathit{\xi }}(i,j)={\displaystyle \frac{2}{{\sigma }_{\mathrm{r}}^2}}\Re \{{\displaystyle \frac{\partial {\mathbf{y}}^H}{\partial {\mathit{\xi }}_i}}{\displaystyle \frac{\partial \mathbf{y}}{\partial {\mathit{\xi }}_j}}\}.\end{array}$$

(8)

The detailed derivation of (8) is provided in Appendix A.

According to the definition of $\mathbf{y}\triangleq {\alpha }_{\mathrm{t}}\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}$, the derivatives of $\mathbf{y}$ with respect to each parameter are calculated as

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial \mathbf{y}}{\partial \theta }}={\alpha }_{\mathrm{t}}\mathrm{vec}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{WS}\},\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\partial \mathbf{y}}{\partial \mathsf{\alpha }}}=\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}[1,j],\hfill \end{array}$$

(9b)

where

$$\begin{array}{c}\hfill {\dot{\mathbf{Q}}}_{\mathrm{t}}={\displaystyle \frac{\partial {\mathbf{Q}}_{\mathrm{t}}}{\partial \theta }}=c_0{\mathbf{G}}^T\mathbf{A}(\mathbf{L}\mathit{\psi }{\mathit{\psi }}^T+\mathit{\psi }{\mathit{\psi }}^{\mathrm{T}}\mathbf{L})\mathbf{A}\mathbf{G},\end{array}$$

(10)

with $\mathbf{A}\triangleq \mathrm{diag}\{{\mathbf{a}}_M(\theta )\}$, $\mathbf{L}\triangleq diag\{0,1,\dots ,M-1\}$ and $c_0\triangleq {\alpha }_{\mathrm{r},\mathrm{t}}^{2}j\pi cos\theta$. To facilitate subsequent handling, we partition ${\mathbf{D}}_{\mathit{\xi }}$ into $2\times 2$ blocks as follows:

$$\begin{array}{c}\hfill {\mathbf{D}}_{\mathit{\xi }}=\left[\begin{array}{cc}{\mathbf{D}}_{\theta \theta }& {\mathbf{D}}_{\theta \mathsf{\alpha }}\\ {\mathbf{D}}_{\theta \mathsf{\alpha }}^T& {\mathbf{D}}_{\mathsf{\alpha }\mathsf{\alpha }}\end{array}\right].\end{array}$$

(11)

Then, by utilizing the transformations $\mathrm{Tr}\{\mathbf{ABCD}\}=\mathrm{ve}{\mathrm{c}}^H\{{\mathbf{D}}^H\}({\mathbf{C}}^T\otimes \mathbf{A})\mathrm{vec}\{\mathbf{B}\}$ and $\mathrm{Tr}\{\mathbf{AB}\}=\mathrm{Tr}\{\mathbf{BA}\}$, the elements of ${\mathbf{D}}_{\mathit{\xi }}$ can be calculated as

$$\begin{array}{cc}\hfill & {\mathbf{D}}_{\theta \theta }={\displaystyle \frac{2}{{\sigma }_{\mathrm{r}}^2}}\Re \{{\alpha }_t^{*}\mathrm{ve}{\mathrm{c}}^H\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{WS}\}{\alpha }_{\mathrm{t}}\mathrm{vec}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{WS}\}\}={\displaystyle \frac{2L|{\alpha }_{\mathrm{t}}{|}^2}{{\sigma }_{\mathrm{r}}^2}}\mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_t^H\},\hfill \end{array}$$

(12a)

$$\begin{array}{cc}\hfill & {\mathbf{D}}_{\theta \mathsf{\alpha }}={\displaystyle \frac{2}{{\sigma }_{\mathrm{r}}^2}}\Re \{{\alpha }_t^{*}\mathrm{ve}{\mathrm{c}}^H\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{WS}\}(\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}[1,j])\}={\displaystyle \frac{2L}{{\sigma }_{\mathrm{r}}^2}}\Re \{{\alpha }_t^{*}\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}[1,j]\},\hfill \end{array}$$

(12b)

$$\begin{array}{cc}\hfill & {\mathbf{D}}_{\mathsf{\alpha }\mathsf{\alpha }}={\displaystyle \frac{2}{{\sigma }_{\mathrm{r}}^2}}\Re \{{(\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}[1,j])}^H(\mathrm{vec}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{WS}\}[1,j])\}={\displaystyle \frac{2L}{{\sigma }_{\mathrm{r}}^2}}\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}{\mathbf{I}}_2.\hfill \end{array}$$

(12c)

It is worth noting that we assume that $\mathbf{S}{\mathbf{S}}^H=L{\mathbf{I}}_{N+K}$ in  (12a)–(12c) due to the fact that $\mathbb{E}\{\mathbf{S}{\mathbf{S}}^H\}=L{\mathbf{I}}_{N+K}$, and sufficient samples are usually collected for parameter estimation. Hence, the CRB for estimating $\theta$ can be denoted as [33]

$$\begin{array}{cc}\hfill {\mathbf{CRB}}_{\theta }& ={[{\mathbf{D}}_{\theta \theta }-{\mathbf{D}}_{\theta \mathsf{\alpha }}{\mathbf{D}}_{\mathsf{\alpha }\mathsf{\alpha }}^{-1}{\mathbf{D}}_{\theta \mathsf{\alpha }}^T]}^{-1}\hfill \\ \hfill & ={\displaystyle \frac{1}{\frac{2L|{\alpha }_{\mathrm{t}}{|}^2}{{\sigma }_{\mathrm{r}}^2}(\mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}-\frac{|\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}{|}^2}{\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}})}}.\hfill \end{array}$$

(13)

(3) Security Model: The received signal at the eavesdropper can be written as

$$\begin{array}{c}\hfill y_e[l]={\mathbf{h}}_{\mathrm{r},\mathrm{t}}^T\mathbf{\Theta }\mathbf{Gx}[l]+n_e[l],\end{array}$$

(14)

where $n_e[l]\sim \mathcal{C}\mathcal{N}(0,{\sigma }_e^2)$ denotes the AWGN with the variance of ${\sigma }_e^2$.The eavesdropping SINR of the k-th legitimate user can be calculated as follows:

$$\begin{array}{c}\hfill \mathrm{SIN}{\mathrm{R}}_{e,k}={\displaystyle \frac{|{\mathbf{h}}_e^T{\mathbf{w}}_k{|}^2}{{\sum }_{i=1,i\ne k}^{K+N}|{\mathbf{h}}_e^T{\mathbf{w}}_i{|}^2+{\sigma }_e^2}},\end{array}$$

(15)

where we define ${\mathbf{h}}_e^T={\mathbf{h}}_{\mathrm{r},\mathrm{t}}^T\mathbf{\Theta }\mathbf{G}$. The eavesdropping SINR is a crucial metric for measuring security performance. To ensure secure transmission, the eavesdropping SINR for legitimate users must remain below a specified threshold.

3.2. Problem Formulation

In this paper, we focus on the minimization of CRB to estimate the target’s DoA $\theta$ by jointly designing the UAV position $\mathbf{q}$, the transmit beamforming matrix $\mathbf{W}$, and the RIS reflection beamforming $\mathit{\psi }$, subject to constraints, including the UAV flight area, the communication users’ QoS requirements, the secure transmission demands, the transmit power budget, and the reflecting coefficient limits. The optimization problem can be formulated as

$$\begin{array}{cc}\hfill \underset{\mathbf{q},\mathbf{W},\mathit{\psi }}{min}& {\mathbf{CRB}}_{\theta }\hfill \end{array}$$

(16a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathbf{q}\in X\times Y,\hfill \end{array}$$

(16b)

$$\begin{array}{c}{\mathrm{SIN}\mathrm{R}}_k\ge {\gamma }_k, \forall k,\hfill \end{array}$$

(16c)

$$\begin{array}{c}{\mathrm{SIN}\mathrm{R}}_{e,k}\le {\gamma }_{e,k}, \forall k,\hfill \end{array}$$

(16d)

$$\begin{array}{c}{\Vert \mathbf{W}\Vert }_F^2\le P_{\mathrm{BS}},\hfill \end{array}$$

(16e)

$$\begin{array}{c}|{\psi }_m| =1, \forall m.\hfill \end{array}$$

(16f)

where $\mathbf{q}$ represents the 2D position coordinate of the UAV-mounted RIS, while $X$ × Y denotes the area where the UAV-mounted RIS can be deployed. $P_{\mathrm{BS}}$ represents the BS’s power budget, and ${\gamma }_k$ and ${\gamma }_{e,k}$ denote the predefined QoS requirement for the legitimate user k and the eavesdropping SINR threshold for ensuring secure transmission, respectively. It is clear that the non-convex problem (16) is challenging to solve due to the complicated expressions (16a), (16c), (16d) with fractional terms, the coupled of optimization variables in both the objective function (16a) and constraints (16c) and (16d), and the unit-modulus constraint (16f). To tackle these challenges, we propose an alternating optimization approach that leverages PSO, SDR, MM, and ADMM to individually solve the subproblems associated with each variable.

4. Joint Optimization for CRB Minimization

4.1. Objective Transformation

According to (13), minimizing the ${\mathbf{CRB}}_{\theta }$ is equivalent to maximizing its denominator so that the original objective function in (16) can be reasonably transformed into

$$\begin{array}{c}\hfill \underset{\mathbf{q},\mathbf{W},\mathit{\psi }}{max}f(\mathbf{q},\mathbf{W},\mathit{\psi })\triangleq \mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}-{\displaystyle \frac{|\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}{|}^2}{\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}}}.\end{array}$$

(17)

Next, we alternately design the UAV position $\mathbf{q}$, the transmit beamforming matrix $\mathbf{W}$, and the RIS reflection beamforming $\mathit{\psi }$ to maximize the objective function $f(\mathbf{W},\mathit{\psi },\mathbf{q})$, thereby enhancing the performance of CRB.

4.2. Optimization of UAV Position

The optimization problem with regard to UAV position $\mathbf{q}$ can be reformulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{q}}{max}& \mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}-{\displaystyle \frac{|\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}{|}^2}{\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}}},\hfill \end{array}$$

(18a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \mathbf{q}\in X\times Y.\hfill \end{array}$$

(18b)

Since the position of the UAV-mounted RIS affects the communication channel, the optimization problem (18) is non-convex and thus unsuitable for traditional convex optimization methods. Therefore, we utilize the heuristic PSO algorithm to solve the subproblem.

The steps involved in PSO are as follows:

(1) Initialize the Population: Randomly generate a population of Z particles.

(2) Update Local and Global Best Solutions: For the z-th particle, if the current solution in the $(i+1)$-th iteration is better than its solution in the i-th iteration, update the particle’s local best solution ${\mathbf{p}}_z^{(i)}$. Similarly, the global best solution ${\mathbf{p}}_g^{(i)}$ is the best among all particle solutions.

(3) Velocity and Position Update: The rules for updating the velocity and position of each particle in the next generation are expressed as

$$\begin{array}{c}\hfill {\mathbf{v}}_z^{(i+1)}=\omega {\mathbf{v}}_z^i+a_1{r}_1({\mathbf{p}}_z^i-{\mathbf{x}}_z^i)+a_2{r}_2({\mathbf{p}}_g^i-{\mathbf{x}}_z^i),\end{array}$$

(19)

and

$$\begin{array}{c}\hfill {\mathbf{x}}_z^{(i+1)}={\mathbf{x}}_z^i+{\mathbf{v}}_z^{(i+1)},\end{array}$$

(20)

where $\omega$ is an inertia factor and $\omega {\mathbf{v}}_z^i$ is known as the momentum component. $a_1$ and $a_2$ are acceleration constants, and $r_1$ and $r_2$ are random numbers within [0, 1]. The term $a_1{r}_1({\mathbf{p}}_z^i-{\mathbf{x}}_z^i)$ is the cognitive part, which reflects the memory of its own best position. In contrast, $a_2{r}_2({\mathbf{p}}_g^i-{\mathbf{x}}_z^i)$ is the social part, quantifying performance relative to neighbors. Obviously, the velocity and position of each particle are adjusted based on its current velocity, local best position, and global best position.

(4) Evaluate Fitness: Now, evaluate the fitness of the new population and then return to step (1) for the next generation.

(5) Termination: The algorithm terminates when there is no improvement in fitness, indicating convergence.

4.3. Optimization of Transmit Beamforming

With fixed $\mathbf{q}$ and $\mathit{\psi }$, the subproblem used to optimize $\mathbf{W}$ can be formulated as

$$\begin{array}{cc}\hfill \underset{\mathbf{W}}{\max }& f(\mathbf{W})\hfill \end{array}$$

(21a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& (1+{\gamma }_k^{-1}){\mathbf{h}}_k^T{\mathbf{w}}_k{\mathbf{w}}_k^H{\mathbf{h}}_k^{*}\ge {\sum }_{i=1}^{K+N}{\mathbf{h}}_k^T{\mathbf{w}}_i{\mathbf{w}}_i^H{\mathbf{h}}_k^{*}+{\sigma }_k^2,\forall k,\hfill \end{array}$$

(21b)

$$\begin{array}{cc}\hfill & (1+{\gamma }_{e,k}^{-1}){\mathbf{h}}_e^T{\mathbf{w}}_k{\mathbf{w}}_k^H{\mathbf{h}}_e^{*}\le {\sum }_{i=1}^{K+N}{\mathbf{h}}_e^T{\mathbf{w}}_i{\mathbf{w}}_i^H{\mathbf{h}}_e^{*}+{\sigma }_e^2,\forall k,\hfill \end{array}$$

(21c)

$$\begin{array}{cc}\hfill & {\sum }_{i=1}^{K+N}\Vert {\mathbf{w}}_i{\Vert }_2^2 \le P_{\mathrm{BS}}.\hfill \end{array}$$

(21d)

Particularly, $f(\mathbf{W})$ is a complicated expression containing both fractional and higher-order terms with regard to $\mathbf{W}$. To tackle the aforementioned problem, we propose introducing a lower bound for $f(\mathbf{W})$ instead of optimizing it directly. Subsequently, the subproblem is efficiently solved using the Schur complement and SDR approach. By introducing an auxiliary variable $t_{\mathrm{W}}$, the problem (21) is reformulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathbf{W},t_{\mathrm{W}}}{\max }& t_{\mathrm{W}}\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& f(\mathbf{W})\ge t_{\mathrm{W}},\hfill \end{array}$$

(22b)

$$\begin{array}{cc}\hfill & (1+{\gamma }_k^{-1}){\mathbf{h}}_k^T{\mathbf{w}}_k{\mathbf{w}}_k^H{\mathbf{h}}_k^{*}\ge {\sum }_{i=1}^{K+N}{\mathbf{h}}_k^T{\mathbf{w}}_i{\mathbf{w}}_i^H{\mathbf{h}}_k^{*}+{\sigma }_k^2,\forall k,\hfill \end{array}$$

(22c)

$$\begin{array}{cc}\hfill & (1+{\gamma }_{e,k}^{-1}){\mathbf{h}}_e^T{\mathbf{w}}_k{\mathbf{w}}_k^H{\mathbf{h}}_e^{*}\le {\sum }_{i=1}^{K+N}{\mathbf{h}}_e^T{\mathbf{w}}_i{\mathbf{w}}_i^H{\mathbf{h}}_e^{*}+{\sigma }_e^2,\forall k,\hfill \end{array}$$

(22d)

$$\begin{array}{cc}\hfill & {\sum }_{i=1}^{K+N}\Vert {\mathbf{w}}_i{\Vert }_2^2 \le P_{\mathrm{BS}}.\hfill \end{array}$$

(22e)

The constraint in (22b) can be further transformed into a semi-definite form by applying the Schur complement:

$$\begin{array}{c}\hfill \left[\begin{array}{cc}\mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}-t_{\mathrm{W}}& \mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}\\ \mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}& \mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}\mathbf{W}{\mathbf{W}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}\end{array}\right]⪰\mathbf{0}.\end{array}$$

(23)

The constraints in (22c), (22d), and (23) remain non-convex with regard to $\mathbf{W}$ and are challenging to deal with. Hence, we propose to convert the optimization variable and leverage the SDR algorithm to solve the problem more easily. Specifically, we define ${\mathbf{W}}_i\triangleq {\mathbf{w}}_i{\mathbf{w}}_i^H,i=1,\dots ,K$, which should satisfy ${\mathbf{W}}_i={\mathbf{W}}_i^H, {\mathbf{W}}_i⪰\mathbf{0},$ and $\mathrm{rank}\{{\mathbf{W}}_i\}=1$. Additionally, to formulate the constraints more concisely, we define ${\mathbf{R}}_{\mathrm{W}}\triangleq \mathbf{W}{\mathbf{W}}^H$. Thus, the only non-convexity in solving (21) stems from rank-one constraints. To address this issue, we apply the SDR method to omit these rank-one constraints, thereby obtaining a relaxed version of problem (21).

$$\begin{array}{cc}\hfill & \underset{{\mathbf{W}}_1,\dots ,{\mathbf{W}}_K,{\mathbf{R}}_{\mathrm{W}},t_{\mathrm{W}}}{max}{t}_{\mathrm{W}}\hfill \end{array}$$

(24a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \left[\begin{array}{cc}\mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}{\mathbf{R}}_{\mathrm{W}}{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}-t_{\mathrm{W}}& \mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}{\mathbf{R}}_{\mathrm{W}}{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}\\ \mathrm{Tr}\{{\dot{\mathbf{Q}}}_{\mathrm{t}}{\mathbf{R}}_{\mathrm{W}}{\mathbf{Q}}_{\mathrm{t}}^H\}& \mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}{\mathbf{R}}_{\mathrm{W}}{\mathbf{Q}}_{\mathrm{t}}^H\}\end{array}\right]⪰\mathbf{0},\hfill \end{array}$$

(24b)

$$\begin{array}{cc}\hfill & (1+{\gamma }_k^{-1}){\mathbf{h}}_k^T{\mathbf{W}}_k{\mathbf{h}}_k^{*}\ge {\mathbf{h}}_k^T{\mathbf{R}}_{\mathrm{W}}{\mathbf{h}}_k^{*}+{\sigma }_k^2, \forall k,\hfill \end{array}$$

(24c)

$$\begin{array}{cc}\hfill & (1+{\gamma }_{e,k}^{-1}){\mathbf{h}}_e^T{\mathbf{W}}_k{\mathbf{h}}_e^{*}\le {\mathbf{h}}_e^T{\mathbf{R}}_{\mathrm{w}}{\mathbf{h}}_e^{*}+{\sigma }_e^2,\forall k,\hfill \end{array}$$

(24d)

$$\begin{array}{cc}\hfill & \mathrm{Tr}\{{\mathbf{R}}_{\mathrm{W}}\}\le P_{\mathrm{BS}},\hfill \end{array}$$

(24e)

$$\begin{array}{cc}\hfill & {\mathbf{R}}_{\mathrm{w}}\in {\mathbb{S}}_N, {\mathbf{W}}_i\in {\mathbb{S}}_N,i=1,\dots ,K,\hfill \end{array}$$

(24f)

$$\begin{array}{cc}\hfill & {\mathbf{R}}_{\mathrm{W}}-{\sum }_{i=1}^K{\mathbf{W}}_i\in {\mathbb{S}}_N.\hfill \end{array}$$

(24g)

in which we define the set of all $N\times N$ Hermitian positive semi-definite matrices as ${\mathbb{S}}_N\triangleq \{\mathbf{S}|\mathbf{S}={\mathbf{S}}^H,\mathbf{S}⪰0\}$. Note that problem (24) is a quadratic semi-definite programming (QSDP) problem, which can be easily solved in polynomial time using standard convex optimization algorithms. Once the optimal solution ${\mathbf{W}}_1,\dots ,{\mathbf{W}}_K,{\mathbf{R}}_{\mathrm{W}}$ of (24) is obtained, the corresponding optimal communication beamforming vectors ${\mathbf{w}}_1...,{\mathbf{w}}_K$ can be recovered as [6]

$$\begin{array}{c}\hfill {\mathbf{w}}_i={({\mathbf{h}}_i^T{\mathbf{W}}_i{\mathbf{h}}_i^{*})}^{-1/2}{\mathbf{W}}_i{\mathbf{h}}_i^{*},i=1,\dots ,K.\end{array}$$

(25)

The radar beamforming vectors ${\mathbf{w}}_i$ for $i>K$ can be calculated by the Cholesky decomposition, i.e.,

$$\begin{array}{c}\hfill {\mathbf{W}}_{\mathrm{r}}{\mathbf{W}}_{\mathrm{r}}^H={\mathbf{R}}_{\mathrm{W}}-{\sum }_{i=1}^K{\mathbf{W}}_i,\end{array}$$

(26)

where ${\mathbf{W}}_{\mathrm{r}}=[{\mathbf{w}}_{K+1},\dots ,{\mathbf{w}}_{K+N}]$ is a lower triangular matrix. Finally, an optimal solution $\mathbf{W}=[{\mathbf{w}}_1,\dots ,{\mathbf{w}}_{K+N}]$ can be achieved.

4.4. Optimization of RIS Beamforming

When $\mathbf{q}$ and $\mathbf{W}$ are given, the optimization for $\mathit{\psi }$ is formulated as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{\psi }}{\max }& f(\mathit{\psi })\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathrm{SIN}\mathrm{R}}_k\ge {\gamma }_k, \forall k,\hfill \end{array}$$

(27b)

$$\begin{array}{cc}\hfill & {\mathrm{SIN}\mathrm{R}}_{e,k}\le {\gamma }_{e,k}, \forall k,\hfill \end{array}$$

(27c)

$$\begin{array}{c}|{\psi }_m| =1, \forall m.\hfill \end{array}$$

(27d)

We first reformulate each term of $f(\mathit{\psi })$ explicitly with regard to $\mathit{\psi }$, i.e.,

$$\begin{array}{ccc}\hfill f_1(\mathit{\psi })& =& \mathrm{Tr}\{\dot{{\mathbf{Q}}_{\mathrm{t}}}\mathbf{W}{\mathbf{W}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}\hfill \\ \hfill & =& |c_0{|}^2({\mathit{\psi }}^H{\mathbf{LR}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathbf{L}\mathit{\psi }+{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{LR}}_2\mathbf{L}\mathit{\psi }\hfill \\ & & +{\mathit{\psi }}^H{\mathbf{LR}}_1\mathbf{L}\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }+{\mathit{\psi }}^H{\mathbf{R}}_1\mathbf{L}\mathit{\psi }{\mathit{\psi }}^H{\mathbf{LR}}_2\mathit{\psi }),\hfill \end{array}$$

(28a)

$$\begin{array}{ccc}\hfill f_2(\mathit{\psi })& =& |\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}{\mathbf{WW}}^H{\dot{\mathbf{Q}}}_{\mathrm{t}}^H\}{|}^2\hfill \\ \hfill & =& {\alpha }_{\mathrm{r},\mathrm{t}}^4|c_0{|}^2(|{\mathit{\psi }}^H{\mathbf{LR}}_1{\mathit{\psi }\mathit{\psi }}^H{\mathbf{R}}_2{\mathit{\psi }|}^2+{\left|{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H\mathbf{L}{\mathbf{R}}_2\mathit{\psi }\right|}^2\hfill \\ & & +{\mathit{\psi }}^H{\mathbf{LR}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathbf{L}\mathit{\psi }+{\mathit{\psi }}^H{\mathbf{R}}_1\mathbf{L}\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }{\mathit{\psi }}^H\mathbf{L}{\mathbf{R}}_2\mathit{\psi }),\hfill \end{array}$$

(28b)

$$\begin{array}{cc}\hfill f_3(\mathit{\psi })& =\mathrm{Tr}\{{\mathbf{Q}}_{\mathrm{t}}{\mathbf{WW}}^H{\mathbf{Q}}_{\mathrm{t}}^H\}={\alpha }_{\mathrm{r},\mathrm{t}}^4{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi },\hfill \end{array}$$

(28c)

where we define ${\mathbf{R}}_1\triangleq {\mathbf{A}}^H{\mathbf{G}}^{*}{\mathbf{W}}^{*}{\mathbf{W}}^T{\mathbf{G}}^T\mathbf{A}$, ${\mathbf{R}}_2\triangleq {\mathbf{A}}^H{\mathbf{G}}^{*}{\mathbf{G}}^T\mathbf{A}$. Then, the objective function $f(\mathit{\psi })$ can be reformulated as an explicit form of the variable $\mathit{\psi }$ as follows:

$$\begin{array}{ccc}\hfill f(\mathit{\psi })& =& f_1(\mathit{\psi })-{\displaystyle \frac{f_2(\mathit{\psi })}{f_3(\mathit{\psi })}}=|c_0{|}^2({\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{\mathit{\psi }}^H{\mathbf{LR}}_2\mathbf{L}\mathit{\psi }+{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }{\mathit{\psi }}^H{\mathbf{LR}}_1\mathbf{L}\mathit{\psi }\hfill \\ \hfill & -& {\displaystyle \frac{{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }{|{\mathit{\psi }}^H{\mathbf{LR}}_1\mathit{\psi }|}^2}{{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }}}-{\displaystyle \frac{{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }{|{\mathit{\psi }}^H{\mathbf{LR}}_2\mathit{\psi }|}^2}{{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }}}),\hfill \end{array}$$

(29)

where

$$\begin{array}{cc}\hfill {\mathit{\psi }}^H{\mathbf{R}}_i\mathit{\psi }& =\mathrm{Tr}\{\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_i\}={\mathrm{vec}}^H\{{\mathbf{R}}_i^H\}\mathrm{vec}\{\mathit{\psi }{\mathit{\psi }}^H\}={\mathsf{\zeta }}_i^H\mathbf{v},\hfill \end{array}$$

(30a)

$$\begin{array}{ccc}\hfill |{\mathit{\psi }}^H{\mathbf{LR}}_i{\mathit{\psi }|}^2& =& \mathrm{Tr}\{\mathbf{L}{\mathbf{R}}_i\mathit{\psi }{\mathit{\psi }}^H{\mathbf{R}}_i\mathbf{L}\mathit{\psi }\}\hfill \\ \hfill & =& {\mathrm{vec}}^H\{\mathit{\psi }{\mathit{\psi }}^H\}({\mathbf{LR}}_i^T\otimes {\mathbf{LR}}_i)\mathrm{vec}\{\mathit{\psi }{\mathit{\psi }}^H\}={\mathbf{v}}^H{\mathsf{\Xi }}_{\widehat{i}}\mathbf{v},\hfill \end{array}$$

(30b)

$$\begin{array}{cc}\hfill {\mathit{\psi }}^H{\mathbf{R}}_i\mathit{\psi }{\mathit{\psi }}^H{\mathbf{LR}}_{\widehat{i}}\mathbf{L}\mathit{\psi }& =\mathrm{Tr}\{{\mathbf{R}}_i\mathit{\psi }{\mathit{\psi }}^H\mathbf{L}{\mathbf{R}}_{\widehat{i}}\mathbf{L}\mathit{\psi }{\mathit{\psi }}^H\}\hfill \\ \hfill & ={\mathrm{vec}}^H\{\mathit{\psi }{\mathit{\psi }}^H\}({\mathbf{LR}}_{\widehat{i}}^T\mathbf{L}\otimes {\mathbf{R}}_i)\mathrm{vec}\{\mathit{\psi }{\mathit{\psi }}^H\}={\mathbf{v}}^H{\mathbf{D}}_i\mathbf{v},\hfill \end{array}$$

(30c)

in which for simplicity, we define

$$\begin{array}{cc}\hfill & \mathbf{v}\triangleq \mathrm{vec}\{\mathit{\psi }{\mathit{\psi }}^H\}={\mathit{\psi }}^{*}\otimes \mathit{\psi },{\mathsf{\zeta }}_i\triangleq \mathrm{vec}\{{\mathbf{R}}_i^H\},i=1,2,\hfill \end{array}$$

(31a)

$$\begin{array}{cc}\hfill & {\mathsf{\Xi }}_i\triangleq {\mathbf{LR}}_{\widehat{i}}^T\otimes {\mathbf{LR}}_{\widehat{i}},{\mathbf{D}}_i\triangleq \mathbf{L}{\mathbf{R}}_{\widehat{i}}^T\mathbf{L}\otimes {\mathbf{R}}_i, \forall i,\widehat{i}\ne i,\hfill \end{array}$$

(31b)

and $\widehat{i}$ expresses the element in the set $\{1,2\}$ that is not equal to i. Then, plugging the transformations in (30) into (29) and defining $\mathbf{D}\triangleq {\mathbf{D}}_1+{\mathbf{D}}_2$, the objective of the optimization problem (27) can be equivalently reformulated in a concise form as

$$\begin{array}{cc}\hfill \underset{\mathit{\psi }}{min}& {\displaystyle \frac{{\mathsf{\zeta }}_1^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_1\mathbf{v}}{{\mathit{\psi }}^H{\mathbf{R}}_2\mathit{\psi }}}+{\displaystyle \frac{{\mathsf{\zeta }}_2^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_2\mathbf{v}}{{\mathit{\psi }}^H{\mathbf{R}}_1\mathit{\psi }}}-{\mathbf{v}}^H\mathbf{D}\mathbf{v}\hfill \end{array}$$

(32a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\mathbf{d}}_k^H\mathit{\psi }\}+c_1-(1+{\gamma }_k^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(32b)

$$\begin{array}{cc}\hfill & -{\mathit{\psi }}^H{\mathbf{C}}_t\mathit{\psi }-c_2+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(32c)

$$\begin{array}{cc}\hfill & |{\psi }_m| =1, \forall m.\hfill \end{array}$$

(32d)

For ease of notation, in (32), we define

$$\begin{array}{cc}\hfill & a_{k,i}\triangleq {\mathbf{h}}_{\mathrm{d},k}^T{\mathbf{w}}_i,{\mathbf{b}}_{k,i}\triangleq \mathrm{diag}\{\mathbf{G}{\mathbf{w}}_i\}{\mathbf{h}}_{\mathrm{r},k},{\mathbf{b}}_{t,i}\triangleq \mathrm{diag}\{\mathbf{G}{\mathbf{w}}_i\}{\mathbf{h}}_{\mathrm{r},t},c_2\triangleq {\sigma }_e^2,\hfill \end{array}$$

(33a)

$$\begin{array}{cc}\hfill & c_1\triangleq {\sum }_{i=1}^{K+N}|a_{k,i}{|}^2-(1+{\gamma }_k^{-1}){|a_{k,k}|}^2+{\sigma }_k^2,{\mathbf{C}}_k\triangleq {\sum }_{i=1}^{K+N}{\mathbf{b}}_{k,i}^{*}{\mathbf{b}}_{k,i}^T,\hfill \end{array}$$

(33b)

$$\begin{array}{cc}\hfill & {\mathbf{C}}_t\triangleq {\sum }_{i=1}^{K+N}{\mathbf{b}}_{t,i}^{*}{\mathbf{b}}_{t,i}^T,{\mathbf{d}}_k\triangleq {\sum }_{i=1}^{K+N}2a_{k,i}{\mathbf{b}}_{k,i}^{*}-2(1+{\gamma }_k^{-1})a_{k,k}{\mathbf{b}}_{k,k}^{*}.\hfill \end{array}$$

(33c)

To deal with the non-convex fractional terms in (32a), two auxiliary variables $t_1$ and $t_2$ are introduced, reformulating them into a more tractable form as

$$\begin{array}{cc}\hfill \underset{\mathit{\psi },t_1,t_2}{\min }& t_1+t_2-{\mathbf{v}}^H\mathbf{D}\mathbf{v}\hfill \end{array}$$

(34a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& t_i\ge {\displaystyle \frac{{\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}}{{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathit{\psi }}}, \forall i,\widehat{i}\ne i,\hfill \end{array}$$

(34b)

$$\begin{array}{cc}\hfill & {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\mathbf{d}}_k^H\mathit{\psi }\}+c_1-(1+{\gamma }_k^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(34c)

$$\begin{array}{cc}\hfill & -{\mathit{\psi }}^H{\mathbf{C}}_t\mathit{\psi }-c_2+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(34d)

$$\begin{array}{cc}\hfill & |{\psi }_m| =1, \forall m.\hfill \end{array}$$

(34e)

Obviously, when $\mathit{\psi }$ is given, the optimal solutions for each iteration of $t_1$ and $t_2$ can be obtained as

$$\begin{array}{c}\hfill t_i^{★}={\displaystyle \frac{{\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}}{{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathit{\psi }}},\forall i,\widehat{i}\ne i.\end{array}$$

(35)

Furthermore, with optimal $t_1$ and $t_2$, problem (34) can be transformed into

$$\begin{array}{cc}\hfill \underset{\mathit{\psi }}{\min }& -{\mathbf{v}}^H\mathbf{D}\mathbf{v}\hfill \end{array}$$

(36a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}-t_i{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathit{\psi }\le 0, \forall i,\widehat{i}\ne i,\hfill \end{array}$$

(36b)

$$\begin{array}{cc}\hfill & {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\mathbf{d}}_k^H\mathit{\psi }\}+c_1-(1+{\gamma }_k^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(36c)

$$\begin{array}{cc}\hfill & -{\mathit{\psi }}^H{\mathbf{C}}_t\mathit{\psi }-c_2+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }\le 0, \forall k,\hfill \end{array}$$

(36d)

$$\begin{array}{cc}\hfill & |{\psi }_m| =1, \forall m.\hfill \end{array}$$

(36e)

Now, it is evident that the presence of the quartic term in (36a) with regard to $\mathit{\psi }$ (i.e., ${\mathbf{v}}^H\mathbf{D}\mathbf{v}$) and the sextic term in (36b) with regard to $\mathit{\psi }$ (i.e., ${\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}$) make problem (36) challenging to handle. To address this difficulty, the MM algorithm is employed, using the first-order and second-order Taylor expansions to seek a sequence of favorable surrogate functions for these non-convex terms, as detailed below.

Transformation for Objective (36a): Specifically, by utilizing the solution ${\mathit{\psi }}_s$ obtained in the previous iteration and applying first-order Taylor expansion, an approximately upper-bound for $-{\mathbf{v}}^H\mathbf{D}\mathbf{v}$ can be constructed as

$$\begin{array}{c}\hfill -{\mathbf{v}}^H\mathbf{D}\mathbf{v}\le -{\mathbf{v}}_s^H\mathbf{D}{\mathbf{v}}_s-2\Re \{{\mathbf{v}}_s^H\mathbf{D}(\mathbf{v}-{\mathbf{v}}_s)\}=\Re \{{\mathbf{v}}^H\mathbf{f}\}+c_1=\Re \{{\mathit{\psi }}^H\tilde{\mathbf{D}}\mathit{\psi }\}+c_1,\end{array}$$

(37)

with $\mathbf{f}\triangleq -2{\mathbf{D}}^H{\mathbf{v}}_s$ and $c_1\triangleq {\mathbf{v}}_s^H\mathbf{D}{\mathbf{v}}_s$. Moreover, $\tilde{\mathbf{D}}$ is a reshaped matrix of $\mathbf{f}$, i.e., $\mathbf{f}=$ $\mathrm{vec}(\tilde{\mathbf{D}})$. However, the real-valued term $\Re \{{\mathit{\psi }}^H\tilde{\mathbf{D}}\mathit{\psi }\}$ remains non-convex. We rewrite it in the form of real-valued variables and further obtain a series of tractable approximate upper bounds for it by the second-order Taylor expansion. Specifically, by defining  $\overline{\mathit{\psi }}\triangleq {\left[\begin{array}{cc}\Re \{{\mathit{\psi }}^T\}& \Im \{{\mathit{\psi }}^T\}\end{array}\right]}^T$ and $\overline{\mathbf{D}}\triangleq \left|\begin{array}{cc}\Re \{\tilde{\mathbf{D}}\}& -\Im \{\tilde{\mathbf{D}}\}\\ \Im \{\tilde{\mathbf{D}}\}& \Re \{\tilde{\mathbf{D}}\}\end{array}\right|$, we have

$$\begin{array}{cc}\hfill \Re \{{\mathit{\psi }}^H\tilde{\mathbf{D}}\mathit{\psi }\}={\overline{\mathit{\psi }}}^T\overline{\mathbf{D}}\overline{\mathit{\psi }}& \le {\overline{\mathit{\psi }}}_s^T\overline{\mathbf{D}}{\overline{\mathit{\psi }}}_s+{\overline{\mathit{\psi }}}_s^T(\overline{\mathbf{D}}+{\overline{\mathbf{D}}}^T)(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)+{\displaystyle \frac{\tilde{\lambda }}{2}}{(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)}^T(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)\hfill \\ \hfill & ={\displaystyle \frac{\tilde{\lambda }}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H\tilde{\mathbf{f}}\}-{\overline{\mathit{\psi }}}_s^T{\overline{\mathbf{D}}}^T{\overline{\mathit{\psi }}}_s+{\displaystyle \frac{\tilde{\lambda }}{2}}{\overline{\mathit{\psi }}}_s^T{\overline{\mathit{\psi }}}_s,\hfill \end{array}$$

(38)

in which $\tilde{\lambda }$ is the maximum eigenvalue of the Hessian matrix $(\overline{\mathbf{D}}+{\overline{\mathbf{D}}}^T)$, $\tilde{\mathbf{f}}\triangleq \mathbf{S}(\overline{\mathbf{D}}+{\overline{\mathbf{D}}}^T-\tilde{\lambda }{\mathbf{I}}_{2M}){\overline{\mathit{\psi }}}_s$ and $\mathbf{S}\triangleq \left[{\mathbf{I}}_M\mathrm{j}{\mathbf{I}}_M\right]$. Thus, the convex surrogate function of $-{\mathbf{v}}^H\mathbf{D}\mathbf{v}$ can be formulated as

$$\begin{array}{c}\hfill -{\mathbf{v}}^H\mathbf{D}\mathbf{v}\le {\displaystyle \frac{\tilde{\lambda }}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H\tilde{\mathbf{f}}\}+c_2,\end{array}$$

(39)

where $c_2\triangleq -{\overline{\mathit{\psi }}}_s^T{\overline{\mathbf{D}}}^T{\overline{\mathit{\psi }}}_s+{\displaystyle \frac{\tilde{\lambda }}{2}}{\overline{\mathit{\psi }}}_s^T{\overline{\mathit{\psi }}}_s+c_1$.

Transformation for Constraint (36b): Review the definition $\overline{\mathit{\psi }}\triangleq {\left[\begin{array}{cc}\Re \{{\mathit{\psi }}^T\}& \Im \{{\mathit{\psi }}^T\}\end{array}\right]}^T$ and define the following simplified notations:

$$\begin{array}{c}\hfill {\overline{\mathsf{\zeta }}}_i\triangleq {\left[\Re \{{\mathsf{\zeta }}_i^T\}\Im \{{\mathsf{\zeta }}_i^T\}\right]}^T,\overline{\mathbf{v}}\triangleq {\left[\Re \{{\mathbf{v}}^T\}\Im \{{\mathbf{v}}^T\}\right]}^T,{\overline{\mathsf{\Xi }}}_i\triangleq \left[\begin{array}{cc}\Re \{{\mathsf{\Xi }}_i\}& -\Im \{{\mathsf{\Xi }}_i\}\\ \Im \{{\mathsf{\Xi }}_i\}& \Re \{{\mathsf{\Xi }}_i\}\end{array}\right].\end{array}$$

(40)

The equivalent real-valued form of the term ${\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}$ in (36b) can be denoted as

$$\begin{array}{c}\hfill g_i(\overline{\mathbf{v}})={\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}={\overline{\mathsf{\zeta }}}_i^T\overline{\mathbf{v}}{\overline{\mathbf{v}}}^T{\overline{\mathsf{\Xi }}}_i\overline{\mathbf{v}},\end{array}$$

(41)

whose first- and second-order derivatives can be derived as

$$\begin{array}{cc}\hfill \nabla g_i(\overline{\mathbf{v}})& ={\overline{\mathsf{\zeta }}}_i^T\overline{\mathbf{v}}({\overline{\mathsf{\Xi }}}_i+{\overline{\mathsf{\Xi }}}_i^T)\overline{\mathbf{v}}+{\overline{\mathbf{v}}}^T{\overline{\mathsf{\Xi }}}_i\overline{\mathbf{v}}{\overline{\mathsf{\zeta }}}_i,\hfill \end{array}$$

(42a)

$$\begin{array}{cc}\hfill {\nabla }^2{g}_i(\overline{\mathbf{v}})& =({\overline{\mathsf{\Xi }}}_i+{\overline{\mathsf{\Xi }}}_i^T)\overline{\mathbf{v}}{\overline{\mathsf{\zeta }}}_i^T+({\overline{\mathsf{\zeta }}}_i{\overline{\mathbf{v}}}^T+{\overline{\mathbf{v}}}^T{\overline{\mathsf{\zeta }}}_i{\mathbf{I}}_{2M^2})({\overline{\mathsf{\Xi }}}_i+{\overline{\mathsf{\Xi }}}_i^T).\hfill \end{array}$$

(42b)

Then, the upper-bounded surrogate function of $g_i(\overline{\mathbf{v}})$ can be derived as

$$\begin{array}{cc}\hfill g_i(\overline{\mathbf{v}})& \begin{array}{c}\hfill \le g_i({\overline{\mathbf{v}}}_s)+{(\nabla g_i({\overline{\mathbf{v}}}_s))}^T(\overline{\mathbf{v}}-{\overline{\mathbf{v}}}_s)\end{array}+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{(\overline{\mathbf{v}}-{\overline{\mathbf{v}}}_s)}^T(\overline{\mathbf{v}}-{\overline{\mathbf{v}}}_s)\hfill \\ \hfill & ={\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{\overline{\mathbf{v}}}^T\overline{\mathbf{v}}+{\overline{\mathbf{v}}}^T{\overline{\ell }}_i+x_i={\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{\mathbf{v}}^H\mathbf{v}+\Re \{{\mathbf{v}}^H{\ell }_i\}+x_i\hfill \\ \hfill & \le \Re \{{\mathit{\psi }}^H{\mathsf{\Omega }}_i\mathit{\psi }\}+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{M}^2{a}_{\max }^4+x_i={\overline{\mathit{\psi }}}^T{\overline{\mathsf{\Omega }}}_i\overline{\mathit{\psi }}+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{M}^2{a}_{\max }^4+x_i\hfill \\ \hfill & \le {\overline{\mathit{\psi }}}_s^T{\overline{\mathsf{\Omega }}}_i{\overline{\mathit{\psi }}}_s+{\overline{\mathit{\psi }}}_s^T({\overline{\mathsf{\Omega }}}_i+{\overline{\mathsf{\Omega }}}_i^T)(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)+{\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)}^T(\overline{\mathit{\psi }}-{\overline{\mathit{\psi }}}_s)+{\displaystyle \frac{{\lambda }_{\mathrm{y},i}}{2}}{M}^2{a}_{\max }^4+x_i\hfill \\ \hfill & ={\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H{\tilde{\ell }}_i\}+{\tilde{x}}_i,\hfill \end{array}$$

(43)

with

$$\begin{array}{cc}\hfill & {\overline{\ell }}_i\triangleq \nabla g_i({\overline{\mathbf{v}}}_s)-{\lambda }_{\mathrm{g},i}{\overline{\mathbf{v}}}_s,x_i\triangleq g_i({\overline{\mathbf{v}}}_s)-{(\nabla g_i({\overline{\mathbf{v}}}_s))}^T{\overline{\mathbf{v}}}_s+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{\overline{\mathbf{v}}}_s^T{\overline{\mathbf{v}}}_s,{\ell }_i\triangleq {\mathbf{S}}_{\mathrm{v}}{\overline{\ell }}_i=\mathrm{vec}\{{\mathsf{\Omega }}_i\},\hfill \\ \hfill & {\mathbf{S}}_{\mathrm{v}}\triangleq [{\mathbf{I}}_{M^2}j{\mathbf{I}}_{M^2}],{\overline{\mathsf{\Omega }}}_i\triangleq \left[\begin{array}{cc}\Re \{{\mathsf{\Omega }}_i\}& -\Im \{{\mathsf{\Omega }}_i\}\\ \Im \{{\mathsf{\Omega }}_i\}& \Re \{{\mathsf{\Omega }}_i\}\end{array}\right],{\tilde{\ell }}_i\triangleq \mathbf{S}({\overline{\mathsf{\Omega }}}_i+{\overline{\mathsf{\Omega }}}_i^T-{\tilde{\lambda }}_{\mathrm{g},i}{\mathbf{I}}_{2M}){\overline{\mathit{\psi }}}_s,\hfill \\ \hfill & {\tilde{x}}_i\triangleq -{\overline{\mathit{\psi }}}_s^T{\overline{\mathsf{\Omega }}}_i^T{\overline{\mathit{\psi }}}_s+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{\overline{\mathit{\psi }}}_s^T{\overline{\mathit{\psi }}}_s+{\displaystyle \frac{{\lambda }_{\mathrm{g},i}}{2}}{M}^2{a}_{\max }^4+x_i.\hfill \end{array}$$

(44)

In (43), ${\lambda }_{\mathrm{g},i}$ and ${\tilde{\lambda }}_{\mathrm{g},i}$ are the maximum eigenvalues of the Hessian matrix ${\nabla }^2{g}_i({\overline{\mathbf{v}}}_s)$ and $({\overline{\mathsf{\Omega }}}_i+{\overline{\mathsf{\Omega }}}_i^T)$, respectively. The first inequality holds since ${\mathbf{v}}^H\mathbf{v}={({\mathit{\psi }}^{*}\otimes \mathit{\psi })}^H({\mathit{\psi }}^{*}\otimes \mathit{\psi })={({\mathit{\psi }}^H\mathit{\psi })}^2\le M^2{a}_{\max }^4$.

Moreover, the linear surrogate function for $-{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathit{\psi }$ can be expressed as

$$\begin{array}{c}\hfill -{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathit{\psi }\le -{\mathit{\psi }}_s^H{\mathbf{R}}_{\widehat{i}}{\mathit{\psi }}_s-2\Re \{{\mathit{\psi }}_s^H{\mathbf{R}}_{\widehat{i}}(\mathit{\psi }-{\mathit{\psi }}_s)\}=\Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_i\}+{\kappa }_i,\end{array}$$

(45)

where ${\mathit{\varrho }}_i\triangleq -2{\mathbf{R}}_{\widehat{i}}^H{\mathit{\psi }}_s$ and ${\kappa }_i\triangleq {\mathit{\psi }}_s^H{\mathbf{R}}_{\widehat{i}}{\mathit{\psi }}_s$. To summarize, the convex surrogate function for constraint (36b) can be expressed as

$$\begin{array}{c}\hfill {\mathsf{\zeta }}_i^H\mathbf{v}{\mathbf{v}}^H{\mathsf{\Xi }}_i\mathbf{v}-t_i{\mathit{\psi }}^H{\mathbf{R}}_{\widehat{i}}\mathbf{\Theta }\le {\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H{\tilde{\mathit{\varrho }}}_i\}+{\tilde{\kappa }}_i,\end{array}$$

(46)

with ${\tilde{\mathit{\varrho }}}_i\triangleq {\tilde{\ell }}_i+t_i{\mathit{\varrho }}_i$ and ${\tilde{\kappa }}_i\triangleq {\tilde{x}}_i+t_i{\kappa }_i$.

Transformation for Constraint (36c): Noticing that the concave term $-(1+{\gamma }_k^{-1}){\mathit{\psi }}^H\times {\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T\mathit{\psi }$ makes constraint (36c) non-convex. Specifically, its linear surrogate function can be derived as follows:

$$\begin{array}{cc}\hfill -{\mathit{\psi }}^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T\mathit{\psi }& \le -{\mathit{\psi }}_s^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T{\mathit{\psi }}_s-2\Re \{{\mathit{\psi }}_s^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T(\mathit{\psi }-{\mathit{\psi }}_s)\}.\hfill \end{array}$$

(47)

Then, we can re-arrange the constraint (36c) as

$$\begin{array}{c}\hfill {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\tilde{\mathbf{d}}}_k^H\mathit{\psi }\}+{\tilde{c}}_1\le 0, \forall k,\end{array}$$

(48)

where we define ${\tilde{\mathbf{d}}}_k^H\triangleq {\mathbf{d}}_k^H-2(1+{\gamma }_k^{-1}){\mathit{\psi }}_s^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T$ and ${\tilde{c}}_1\triangleq c_1+(1+{\gamma }_k^{-1}){\mathit{\psi }}_s^H{\mathbf{b}}_{k,k}^{*}{\mathbf{b}}_{k,k}^T{\mathit{\psi }}_s$ for brevity.

Transformation for Constraint (36d): Similarly to (45), we can derive an upper bound of the concave term $-{\mathit{\psi }}^H{\mathbf{C}}_t\mathit{\psi }$ for constraint (36d) as

$$\begin{array}{c}\hfill -{\mathit{\psi }}^H{\mathbf{C}}_t\mathit{\psi }\le -{\mathit{\psi }}_s^H{\mathbf{C}}_t{\mathit{\psi }}_s-2\Re \{{\mathit{\psi }}_s^H{\mathbf{C}}_t(\mathit{\psi }-{\mathit{\psi }}_s)\}=\Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_t\}+{\kappa }_t,\end{array}$$

(49)

with ${\mathit{\varrho }}_t\triangleq -2{\mathbf{C}}_t^H{\mathit{\psi }}_s$ and ${\kappa }_t\triangleq {\mathit{\psi }}_s^H{\mathbf{C}}_t{\mathit{\psi }}_s$. Thus, (36d) can be reformulated as

$$\begin{array}{c}\hfill \Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_t\}+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }+{\tilde{c}}_2\le 0,\end{array}$$

(50)

where ${\tilde{c}}_2=-c_2+{\kappa }_t$.

Thus, we can formulate the optimization problem (36) with regard to $\mathit{\psi }$ on the $(s+1)$-th iteration as

$$\begin{array}{cc}\hfill \underset{\mathit{\psi }}{\min }& {\displaystyle \frac{\tilde{\lambda }}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H\tilde{\mathbf{f}}\}\hfill \end{array}$$

(51a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H{\tilde{\mathit{\varrho }}}_i\}+{\tilde{\kappa }}_i\le 0, \forall i,\hfill \end{array}$$

(51b)

$$\begin{array}{cc}\hfill & {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\tilde{\mathbf{d}}}_k^H\mathit{\psi }\}+{\tilde{c}}_1\le 0, \forall k,\hfill \end{array}$$

(51c)

$$\begin{array}{cc}\hfill & \Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_t\}+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }+{\tilde{c}}_2\le 0, \forall k,\hfill \end{array}$$

(51d)

$$\begin{array}{cc}\hfill & |{\psi }_m| =1, \forall m.\hfill \end{array}$$

(51e)

To address problem (51) under the unit-modulus constraint, we propose utilizing the ADMM method. Specifically, we first introduce an auxiliary variable $\mathit{\varphi }\triangleq {[{\varphi }_1,\dots ,{\varphi }_M]}^T$ to reformulate problem (51) as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{\psi },\mathit{\varphi }}{\min }& {\displaystyle \frac{\tilde{\lambda }}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H\tilde{\mathbf{f}}\}\hfill \end{array}$$

(52a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H{\tilde{\mathit{\varrho }}}_i\}+{\tilde{\kappa }}_i\le 0, \forall i,\hfill \end{array}$$

(52b)

$$\begin{array}{cc}\hfill & {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\tilde{\mathbf{d}}}_k^H\mathit{\psi }\}+{\tilde{c}}_1\le 0, \forall k,\hfill \end{array}$$

(52c)

$$\begin{array}{cc}\hfill & \Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_t\}+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }+{\tilde{c}}_2\le 0, \forall k,\hfill \end{array}$$

(52d)

$$\begin{array}{cc}\hfill & |{\psi }_m| \le 1, \forall m,\hfill \end{array}$$

(52e)

$$\begin{array}{cc}\hfill & |{\mathit{\varphi }}_m| =1, \forall m,\hfill \end{array}$$

(52f)

$$\begin{array}{cc}\hfill & \mathit{\psi }=\mathit{\varphi }.\hfill \end{array}$$

(52g)

Based on the ADMM algorithm, we turn to optimizing its augmented Lagrangian function, obtaining the solution to (52) as follows:

$$\begin{array}{cc}\hfill \underset{\mathit{\psi },\mathit{\varphi }}{\min }& {\displaystyle \frac{\tilde{\lambda }}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H\tilde{\mathbf{f}}\}+{\displaystyle \frac{1}{2\rho }}{\parallel \mathit{\psi }-\mathit{\varphi }+\rho \mathit{\nu }\parallel }^2\hfill \end{array}$$

(53a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{{\tilde{\lambda }}_{\mathrm{g},i}}{2}}{\mathit{\psi }}^H\mathit{\psi }+\Re \{{\mathit{\psi }}^H{\tilde{\mathit{\varrho }}}_i\}+{\tilde{\kappa }}_i\le 0, \forall i,\hfill \end{array}$$

(53b)

$$\begin{array}{cc}\hfill & {\mathit{\psi }}^H{\mathbf{C}}_k\mathit{\psi }+\Re \{{\tilde{\mathbf{d}}}_k^H\mathit{\psi }\}+{\tilde{c}}_1\le 0, \forall k,\hfill \end{array}$$

(53c)

$$\begin{array}{cc}\hfill & \Re \{{\mathit{\psi }}^H{\mathit{\varrho }}_t\}+(1+{\gamma }_{e,k}^{-1}){\mathit{\psi }}^H{\mathbf{b}}_{t,k}^{*}{\mathbf{b}}_{t,k}^T\mathit{\psi }+{\tilde{c}}_2\le 0, \forall k,\hfill \end{array}$$

(53d)

$$\begin{array}{cc}\hfill & |{\psi }_m| \le 1, \forall m,\hfill \end{array}$$

(53e)

$$\begin{array}{cc}\hfill & |{\varphi }_m| =1, \forall m,\hfill \end{array}$$

(53f)

where $\rho$ is the penalty coefficient, and $\mathit{\nu }\in {\mathbb{C}}^{M\times 1}$ is the Lagrangian dual variable. Then, problem (53) can be solved by alternately updating individual variables while fixing the other variables.

Update $\mathit{\psi }$: Clearly, when $\mathit{\varphi }$ and $\mathit{\nu }$ are fixed, the optimization problem for updating $\mathit{\psi }$ becomes convex and can be efficiently solved in polynomial time using standard convex optimization algorithms.

Update $\mathit{\varphi }$: With $\mathit{\psi }$ and $\mathit{\nu }$ fixed, the optimal ${\mathit{\varphi }}^{★}$ can be determined through phase alignment as follows:

$$\begin{array}{c}\hfill {\mathit{\varphi }}^{★}=e^{j\angle (\mathit{\psi }+\rho \mathit{\nu })}.\end{array}$$

(54)

Update $\mathit{\nu }$: When $\mathit{\psi }$ and $\mathit{\varphi }$ are fixed, the Lagrangian dual variable $\mathit{\nu }$ can be updated as follows:

$$\begin{array}{c}\hfill \mathit{\nu }:=\mathit{\nu }+(\mathit{\psi }-\mathit{\varphi })/\rho .\end{array}$$

(55)

4.5. Overall Algorithm

Given the above derivations, the proposed algorithm for jointly optimizing the UAV position, transmit beamforming, and RIS beamforming is summarized in Algorithm 1. With a suitable initialization, we alternately update the variables $\mathbf{q}$, $\mathbf{W}$, and $\mathit{\psi }$ while keeping the other variables fixed until convergence. The detailed steps for optimizing $\mathbf{q}$ using the PSO algorithm are presented in Section 4.2. It is worth noting that as the penalty factor $\rho$ decreases in each iteration, i.e., $\rho \to 0$, the solution to problem (53) is guaranteed to satisfy the unit-modulus constraint.

Next, we analyze the complexity of the proposed algorithm, assuming that the popular interior-point method is used for solving the convex optimization problems in this paper. Firstly, the complexity of optimizing $\mathbf{q}$ requires approximately $\mathcal{O}(2CZ)$ operations, where C and Z represent the maximum iterations and the number of particles, respectively. Additionally, obtaining the optimal ${\mathbf{W}}_i$, $i=1,\cdots ,K$ and ${\mathbf{R}}_{\mathrm{W}}$ has a computational complexity of $\mathcal{O}(K^{6.5}{N}^{6.5})$. Constructing ${\mathbf{w}}_i$ for $i>K$ requires approximately $\mathcal{O}((K+N)N^3)$. The complexity of calculating $t_1$ and $t_2$ is $\mathcal{O}(M^2)$. The computational complexity of solving the subproblem with regard to $\mathit{\psi }$ has the order of $\mathcal{O}(M^{4.5})$. As a result, the total computational complexity of Algorithm 1 can approximate $\mathcal{O}(2CZ+K^{6.5}{N}^{6.5}+M^{4.5})$.

Algorithm 1 Joint UAV Position, transmit beamforming, and RIS beamforming Design Algorithm for Solving (16).

Require:  ${\mathbf{h}}_{\mathrm{d},k}$, ${\mathbf{h}}_{\mathrm{r},k}$, ${\mathbf{h}}_{\mathrm{r},t}$, $\mathbf{G}$, $P_{\mathrm{BS}}$, ${\gamma }_k$, ${\gamma }_{e,k}$, ${\sigma }_k^2$, ${\sigma }_{\mathrm{r}}^2$, ${\sigma }_{\mathrm{t}}^2$, ${\sigma }_e^2$, N, M, K, L, $\rho$, $\forall k$. Ensure:  ${\mathbf{q}}^{★}$, ${\mathbf{W}}^{★}$, and ${\mathit{\psi }}^{★}$. 1: Initialize feasible $\mathbf{q}$, $\mathbf{W}$, and $\mathit{\psi }$. 2: while no convergence do 3:    Update $\mathbf{q}$ by PSO algorithm. 4:    Obtain ${\mathbf{W}}_i$, $i=1,\cdots ,K$ and ${\mathbf{R}}_{\mathrm{W}}$ by (24); 5:    Construct ${\mathbf{w}}_i$, $i=1,\cdots ,K$ by (25); 6:    Construct ${\mathbf{w}}_i$, $i>K$ by Cholesky decomposition; 7:    Combine $\mathbf{W}=[{\mathbf{w}}_1,\cdots ,{\mathbf{w}}_{K+N}]$. 8:    while no convergence do 9:      Calculate $t_1$ and $t_2$ by (35); 10:      Update $\mathit{\psi }$ by solving (53) giving other variables; 11:      Update $\mathit{\varphi }$ by (54); 12:      Update $\mathit{\nu }$ by (55); 13:      $\rho :=0.7\rho$. 14:    end while 15: end while 16: Return ${\mathbf{q}}^{★}=\mathbf{q}$, ${\mathbf{W}}^{★}=\mathbf{W}$, and ${\mathit{\psi }}^{★}=\mathit{\psi }$.

5. Simulation Results

In this section, we present the results of comprehensive numerical simulations to validate the superiority and effectiveness of the designed joint optimization algorithm. The simulations were conducted using the MATLAB software (2019), and the results were averaged over 100 independent runs. We assumed that an ISAC BS equipped with $N=8$ transmit/receive antennas communicated with $K=4$ legitimate users and sensed a target, aided by a UAV-mounted RIS with $M=12$ elements. The UAV-mounted RIS was located at a specific altitude of $h_{\mathrm{U}}=50\mathrm{m}$, with its initial horizontal coordinates given by $\mathbf{q}=\left(0\mathrm{m},50\mathrm{m}\right)$. The BS was located at $(0\mathrm{m},0\mathrm{m})$. The legitimate users were distributed uniformly in a circle with $\mathbf{q}$ as the center and $5\mathrm{m}$ as the radius. Meanwhile, a potential target was located in $(3\mathrm{m},47\mathrm{m})$. The noise powers were set as ${\sigma }_k^2={\sigma }_{\mathrm{r}}^2={\sigma }_e^2=-90\mathrm{dBm},\forall k$; the RCS was ${\sigma }_{\mathrm{t}}^2=1$, and the number of samples was $L=1024$. The DoA of the target with regard to the RIS was set as $\theta ={\displaystyle \frac{\pi }{4}}$, and the power budget was set as $P_{\mathrm{BS}}=30\mathrm{dBm}$. We adopted a typical path-loss model [34], with path-loss exponents for the BS-RIS, BS/RIS-user, and RIS-target links set to 2.2, 3.5, 2.3, and 2.2, respectively. For simplicity, the minimum QoS requirements were assumed identical for all legitimate users, i.e., ${\gamma }_k=\gamma ,\forall k$. The eavesdropping SINR threshold was set as ${\gamma }_{e,k}={\gamma }_e=-20\mathrm{dB},\forall k$.

Starting from the frontiers and classic works in existing related research, the latest state-of-the-art work [14] and four baseline schemes were compared with the proposed UAV-mounted RIS-assisted secure ISAC scheme (denoted as “proposed”) in terms of CRB to validate its superiority and effectiveness.

•

Joint Transmit and RIS Beamforming (JTRB): This scheme is proposed in [14], which optimizes the joint design of transmit beamforming and RIS beamforming to minimize the CRB using alternating optimization, SDR, and successive convex approximation. The “JTRB” scheme serves as a benchmark to verify the superiority of the “proposed” scheme.

•

Fixed Initial UAV Position (FIUP): With a fixed UAV position $\mathbf{q}$, the beamformers $\mathbf{W}$ and $\mathit{\psi }$ were iteratively solved using Algorithm 1. The comparison with the “proposed” scheme demonstrates the benefits of optimizing the RIS deployment position in improving sensing performance.

•

Random Phase Shifts of RIS (RPSR): The phase shifts of the RIS were randomized to feasible values and remained unchanged. Specifically, the “RPSR” scheme could be achieved by skipping lines 8–14 of Algorithm 1. This scheme was used as a baseline to evaluate the performance of the MM and ADMM algorithms proposed in Section 4.4 for optimizing RIS beamforming.

•

Random Transmit Beamforming (RTB): Compared to the “proposed” scheme, the “RTB” scheme omits the optimization process of transmit beamforming, and all users are operated with the initial $\mathbf{W}$. By comparing with it, the necessity of optimizing BS transmit beamforming is clearly demonstrated.

•

Radar-Only: Only the radar sensing function was optimized in the considered system. The proposed joint UAV position, transmit beamforming, and RIS beamforming design algorithm was utilized without considering the communication users’ QoS requirements and secure transmission constraints. The comparison demonstrates that the “proposed” scheme maintains strong sensing performance while ensuring communication security.

Figure 2 presents the CRB versus the number of RIS reflecting the M element. The “proposed” scheme demonstrates performance nearly equivalent to the “radar-only” scheme while dramatically outperforming the remaining four schemes. In particular, the “proposed” scheme achieves up to an 8.65 dB improvement in radar sensing performance compared to the “RPSR” scheme when M = 12. This is because the “proposed” scheme leverages the high mobility and flexibility of the UAV-mounted RIS, optimizing its configuration to enhance sensing performance in the BS’s non-visible region. Notably, the “FIUP” scheme outperforms the “JTRB” scheme. The reason is that the “FIUP” scheme maintains a strategically chosen UAV position, which ensures effective coverage and performance enhancement, even without UAV position optimization. Additionally, the “FIUP” scheme achieves a lower CRB than both the “RPSR” and “RTB” schemes, confirming the effectiveness of the proposed joint optimization algorithm for transmit and RIS beamforming. These results highlight the effectiveness of the “proposed” scheme. Unlike the “FIUP” scheme and the “JTRB” scheme, the “proposed” scheme integrates UAV mobility with advanced optimization techniques, ensuring both communication quality and sensing accuracy.

Figure 3 illustrates the CRB performance versus the communication QoS requirement $\gamma$. It can be observed that the CRB for all the schemes increases with the communication QoS requirements, meaning a degradation in radar sensing performance. Notably, the performance gap between the “proposed” scheme and the “radar-only” scheme becomes increasingly pronounced at higher communication QoS requirements. This phenomenon occurs because, with a fixed power budget, tighter QoS constraints require more power to be allocated to communication beamforming ${\mathbf{W}}_{\mathrm{c}}$, thus reducing the resources available for radar beamforming ${\mathbf{W}}_{\mathrm{r}}$. This reveals the inherent trade-off between communication and sensing performance in ISAC systems. Furthermore, the “proposed” scheme consistently outperforms the “FIUP”, “JTRB”, “RTB”, and “RPSR” schemes, even under strict communication QoS requirements. This robustness highlights the effectiveness of joint optimization in balancing the competing demands of communication and sensing.

Next, the effect of the transmit power budget $P_{\mathrm{BS}}$ in improving radar sensing performance is unveiled in Figure 4. It is observed that the “proposed” scheme experiences a sensing performance degradation of approximately 2.23 dB compared to the “radar-only” scheme when $P_{\mathrm{BS}}=40$ dBm while maintaining high communication quality and secure transmission. Compared to the “FIUP” scheme, the “JTRB” scheme, the “RTB” scheme, and the “RPSR” scheme, our “proposed” scheme consistently achieves better CRB performance regardless of $P_{\mathrm{BS}}$. Specifically, the CRB is improved by 2.69%, 3.42%, 7.22%, and 7.72%, respectively. This improvement is attributed to the effective integration of UAV position optimization, transmit beamforming, and RIS beamforming, enabling dynamic adjustments to system configurations to enhance sensing performance across different power budgets.

In Figure 5, we present the CRB for target DoA estimation as a function of the number of antennas N. It can be observed that the “proposed” scheme achieves performance very close to that of the “radar-only” scheme while consistently outperforming the “FIUP”, “JTRB”, “RTB”, and “RPSR” schemes across all values of N. As N increases, all schemes exhibit improved CRB performance. The reason is that a larger number of antennas provide greater spatial diversity and higher beamforming gains. This improvement highlights the critical role of antenna configuration in reducing the estimation error and enhancing the radar sensing accuracy.

6. Conclusions

This study focused on the integrated design of UAV position, transmit beamforming, and RIS beamforming within a UAV-mounted RIS-aided secure ISAC system. The CRB minimization problem of target DoA estimation was formulated, considering constraints such as the UAV flight area, communication QoS, secure transmission requirements, power limitations, and RIS reflection coefficients. To address the non-convex optimization problem, an alternating optimization strategy was employed, utilizing methods such as PSO, SDR, MM, and ADMM. Simulation results highlighted the notable benefits of employing UAV-mounted RIS in secure ISAC systems, as well as the effectiveness of the proposed algorithm. The investigation of UAV trajectory optimization will be considered in our future work. Additionally, there are many methods, such as the manifold approach, heuristic methods, gradient descent, and deep learning, to handle the unit-modulus constraint. We intend to explore these approaches in our future work.