A Performance Analysis Model of TCP over Multiple Heterogeneous Paths for 5G Mobile Services

Abstract

Driven by the primary requirement of emerging 5G mobile services, the demand for concurrent multipath transfer (CMT) is still prominent. Yet, multipath transport protocols are not widely adopted and CMT schemes based on Transport Control Protocol (TCP) will still be in dominant position in 5G. However, the performance of TCP flow transferred over multiple heterogeneous paths is prone to the link quality asymmetry, the extent of which was revealed to be significant by our field investigation. In this paper, we present a performance analysis model for TCP over multiple heterogeneous paths in 5G scenarios, where both bandwidth and delay asymmetry are taken into consideration. The evaluation using large-scale simulation and field experiment shows that the proposed model can achieve high accuracy in practical environments. Some interesting inferences can be drawn from the proposed model, such as the dominant factors that affect the performance of TCP over heterogeneous networks, and the criteria of determining the appropriate number of links to be used under different circumstances of path heterogeneity. Thus, the proposed model can provide a guidance to the design of TCP-based CMT solutions for 5G mobile services.

1. Introduction

For emerging and promising 5G mobile services, despite their diverse application scenarios, it is widely agreed that they share a common primary requirement: either high data rate or high reliability. To meet such requirement, evolving wireless techniques and novel network infrastructures for 5G are no doubt necessary. However, we believe that the existing Concurrent Multipath Transfer (CMT) [1] technology could also contribute to the fulfillment of needs of 5G mobile services. CMT can not only improve communication throughput, but also provide communication reliability. CMT in 5G scenarios will pool multiple heterogeneous wireless resources by employing a variety of Radio Access Technologies (RATs) concurrently. Thus, the bandwidth of every RAT will be aggregated [2], achieving higher throughput. Also, thanks to diversity gain of heterogeneous RATs, the communication reliability can be improved [3]. Meanwhile, it is potentially more viable to adopt CMT for mobile services in 5G, since 5G is envisioned to consist of various types of RATs (such as millimeter wave communication, LTE-A and Wi-Fi). Meanwhile, more and more mobile devices have been equipped with multiple wireless interfaces [4].

Multipath techniques that can achieve CMT are still in development, while TCP-based CMT solutions will be in the dominant position. There are many reasons why multipath is not widely used. First, they cannot be widely applied to a variety of network environments. For example, the performance of Multipath TCP (MPTCP) [5], the most popular multipath protocol working at a transport layer, will be severely degraded in some cases [6,7]. Second, the vast majority of operating systems, such as Windows, Linux, and MacOS, do not support multipath protocols well. Since most mobile services will still use TCP for now and for the foreseeable future, feasible CMT solutions for 5G services will be based on TCP. These solutions can be viewed as a middleware between the transport layer and network layer, which is transparent to the existing operating systems. Also, the interoperability between existing TCP based network infrastructure will not be compromised.

However, the performance of TCP flow transferred over multiple heterogeneous wireless networks would be adversely affected by path heterogeneity. This will be a critical feature of the highly integrative 5G system [8]. Briefly, such performance degradation is due to the packet reordering issue [9] caused by the different link quality of employed heterogeneous wireless networks. The inherent re-sequencing mechanism of TCP can correct the problem when the packet reordering is no more than two positions [10]. However, the throughput may drop drastically due to the reduction of the TCP transmission window caused by more serious packet reordering [4]. Some contributions were proposed to solve the problem. Earliest Delivery Path First (EDPF) [11] schedules packets over different links based on their estimated delivery time. Delay Aware Packet Scheduling (DAPS) [12] distributes packets over different links depending on the ratio between slowest and fastest Round-Trip Time (RTT) as well as congestion window. Yet, without the thorough understanding of TCP performance in the given situation, these contributions only provide limited improvement.

If we can analyze how heterogeneous networks affect the performance of TCP flow concurrently transferred over them, more efficient and elegant CMT schemes for 5G mobile services can be developed based on TCP and TCP-like congestion control protocols. Such TCP-based CMT schemes would be more deployable in 5G heterogeneous wireless networks since they are compatible with the current Internet infrastructure.

In this paper, a performance analysis model for TCP over multiple heterogeneous wireless networks is presented. To the best of our knowledge, no similar model has been reported in the literature. The proposed model can provide guidance to the design of novel CMT solutions for 5G mobile services. The main contributions of this paper are as follows:

(1)

We have taken field investigation on present heterogeneous wireless networks to reveal the severe extent of link quality asymmetry in terms of delay and bandwidth. This proves that the impact of network heterogeneity in future 5G is anything but empty talk.

(2)

A performance analysis model is derived based on the careful analysis of segments transmission and acknowledgement response over multiple heterogeneous paths. Both bandwidth asymmetry and delay asymmetry are taken into consideration in the proposed model.

(3)

High analytical accuracy is validated by comparison to the simulation study and field experiment. It proves that our model can be applied in practical environments. Simulation of TCP over multiple heterogeneous paths is created in NS3, while the field measurement is performed in a real-world mobile scenario. The predicted throughput using the proposed model can fit the simulation results and the field measurement results with high accuracy.

(4)

Some interesting inferences are drawn from the proposed model. First, compared to bandwidth asymmetry, delay asymmetry between multiple links is the dominant factor that affects the performance of TCP over heterogeneous paths. Second, the criteria of determining the appropriate number of links to be employed to optimize the TCP multipath performance is discussed.

The remainder of this paper is organized as follows. Some related work is introduced in Section 2. Section 3 details the issue of link quality asymmetry based on the results of field investigation. In Section 4, the performance analysis model for TCP over heterogeneous paths is derived. The accuracy of the proposed model is validated in Section 5. In Section 6 we investigate the effect of path heterogeneity based on the proposed model. Section 7 provides some discussion and design guidance. Section 8 concludes the paper.

2. Related Work

To meet the requirement for high data rate and reliability, some contributions were proposed to try to achieve stable and high-quality communication based on multipath transmission. Stream Control Transmission Protocol (SCTP) [13,14] and its extensions [15,16] try to aggregate the bandwidth of multiple paths. MPTCP, a multipath extension to TCP, has also been standardized to transmit data over multiple paths simultaneously to improve reliability and throughput. IETF Multiple Interfaces (MIF) working group is developing the standards [17] for nodes with multiple interfaces. Recently, the cellular-based multipath solutions are generating more interest with the rapid development of 5G heterogeneous networks. For example, femocells-based schemes [18,19] were proposed to support seamless mobility and maximize the network recourse utilization using multiple interfaces. Device to Device (D2D) communication using multiple access technologies were deployed to support massive connectivity [20,21].

However, apart from the practical deployment challenges, such as the existence of various types of middle boxes [6], the main difficulty is that the performance of multipath solutions may decrease significantly under the circumstances of path heterogeneity, especially when there are some bottleneck paths [7,22,23,24].

Packet reordering is considered the dominant challenge for multipath transmission because it leads to an undesirable reduction in throughput [1]. RFC5236 [25] introduces a metric named reorder density to show how far packets are displaced from their original position. Therefore, an efficient multipath solution must reduce the impact of packet reordering to alleviate its effects. Multipath forwarding is the main reason of packet out-of-order [26]. Different technologies and different paths can lead to significant differences in delay and bandwidth. When packets are forwarded over paths with different characteristics, they are likely to arrive at the receiver out of order.

Some state-of-the-art research [27,28] has measured the characteristics of heterogeneous paths in terms of delays. However, their main purpose is to analyze the performance of different scheduling algorithms in heterogeneous networks, rather than theoretically analyze the relationship between path diversity and TCP performance.

The research of TCP performance analysis, especially in terms of throughput, is still making progress, as TCP is one of most widely deployed transport protocols in today’s Internet. The research can be categorized into two kinds: one aims at improving the accuracy of prior model by novel methods [29,30,31], the other focuses on the performance of TCP applied in emerging scenarios [32,33]. However, the proposed models in these papers only analyze the situation where single path is used for transmitting TCP segments.

Overall, to the best of our knowledge, no one has given a performance analysis model to analyze TCP performance over multiple paths with different link quality in heterogeneous networks, although there are many schemes [34] working at different protocol layers that are proposed to try to improve the performance over multiple paths. We believe that this model can help us design more practical multipath schemes in the future wireless networks.

3. Problem Description and Network Model

Network heterogeneity will become a concrete issue in 5G with the popularity of multi-access devices and deployment of emerging heterogeneous RATs. Multi-access devices that can connect to more than one wireless networks are gaining bigger market share, such as smart phones supporting dual-SIM dual stand-by mode. These devices can concurrently use up to three interfaces, including Wi-Fi, for data transmission. For such a device, the connected multiple wireless networks may share heterogeneous access technologies (e.g., WLAN vs. cellular network), heterogeneous standards or heterogeneous service providers [35]. Even if two interfaces are connected to an identical wireless network, the wireless signals are likely to experience heterogeneous pass loss due to small scale fading. Considering that in 5G more heterogeneous RATs will be deployed and utilized by multi-access devices, the network heterogeneity issue will become more severe than in previous four generations.

Network heterogeneity of multi-access devices is intuitively revealed by the difference in network link quality. For two heterogeneous wireless networks, their network link quality is normally different from each other, to which we refer as network link quality asymmetry. Generally, Data Rate (DR) and RTT are used to describe the network link quality. DR reveals the capacity of a network link, while RTT directly reflects the transmission delay. Accordingly, the network link quality asymmetry can be indicated by DR asymmetry and RTT asymmetry.

Intuitively, the performance of TCP transmission would be prone to network link quality asymmetry, if multiple heterogeneous wireless networks are concurrently employed for delivering segments. Consequently, the performance of TCP-based CMT in 5G will also be degraded by link quality asymmetry. This is because the transmitted segments would suffer different transmission delays due to the dissimilar network link quality of employed wireless networks. This results in segments reaching the receiver out-of-order. This segment reordering issue is widely regarded as the major challenge that undermines the performance of concurrent multipath transmission, as it causes unnecessary retransmission, prevents the congestion window from growing and disrupt ACK-clocking. The higher network link quality asymmetry becomes, the more negative impact it has on TCP performance. The analytical discussion of the relationship between the performance of TCP over multiple wireless networks and the link quality asymmetry will be detailed in Section 4.

To investigate the extent of network link quality asymmetry in a real-world situation, we have taken a filed measurement on a group of heterogeneous wireless networks, and found that their link quality deviated significantly from each other. The measurement was carried out in a test train running on a newly constructed high-speed railway before its service, where few passengers were on board. Thus the interference from other wireless devices can be eliminated. Inside the test train, a dedicated box PC with our proprietary measuring program was deployed to automatically measure and store the download DR and RTT of a certain wireless network. Incorporating different kinds of wireless modems, this device can simultaneously access multiple heterogeneous mobile networks. In the measurement, up to eight modems were adopted, including three FDD-LTE (Frequency Division Duplexing-Long Term Evolution) modems of China Telecom (CT), three FDD-LTE modems of China Unicom (CU) and two TD-LTE (Time Division-Long Term Evolution) modems of China Mobile (CM). After the measurement, a group of RTT dataset and two download DR values (average and maximum) were collected on each modem. We need to clarify that the modem is a factor that can influence the link quality of wireless networks. Different modems connecting to the same wireless network may also cause link quality asymmetry. To minimize this influence, we used identical modems to access wireless networks with same communication standards.

The statistics from the measurement result is shown in Figure 1. Regarding RTT, a boxplot diagram is depicted based on collected dataset of each modem. The blue rectangle in a boxplot diagram represents the interquartile ranges (IQR) of the variation, while the band inside the rectangle represents the median. The red crosses depict the outliers. By visually comparing the two boxplot diagrams, statistical inference can be made about the difference of two dataset. If the median of one dataset does not overlap the IQR of the other dataset, it can be inferred that difference exists between two datasets. Further, if two IQRs don’t overlap, the difference is significant. Applying this criterion to Figure 1, we can infer that the RTT of CT1, CU3, CM1 and CM2 are significantly higher than those of CT2, CU1, CU2. Meanwhile, the RTT of CT1, CU3, CM1 and CM2 are different from the others. These conclusions can reveal the dispersion of RTT among eight modems.

Regarding download DR, the average and maximum values are shown using bar graphs. For maximum download DR, the ratio between the highest (CT1) and the lowest (CU3) is 8.2. As for average download DR, this ratio is even more pronounced, reaching 15.1. This means that notable deviation exists in download DR among eight modems.

To sum up, the field measurement results allow to conclude that the network link quality asymmetry in a real-world situation is truly significant. Besides, it is revealed that the link quality asymmetry not only exists between two heterogeneous networks, but also between two modems using access technology operated by same telecommunication company. According to the above conclusions, we can infer that the network heterogeneity in future 5G will be more severe and become a concrete threat, since the wireless networks in 5G will become more diverse than nowadays with the deployment of emerging RATs.

As we have demonstrated, the network heterogeneity will affect the performance of TCP-based CMT solutions for 5G mobile services. Thus, it is very essential to create a quantitative performance analysis model regarding the relationship between the link quality asymmetry and TCP multipath performance. To build such a performance analysis model, we first present the network model of TCP flow transferred over multiple heterogeneous links, as shown in Figure 2. In this network model, the segments of single TCP connection are concurrently distributed over multiple paths between two endpoints. We use $L=\left\{l_1,l_2,\dots ,l_n\right\}$ to denote the set of all available heterogeneous links, the number of elements in $L$ is $n$. Let $D=\left\{d_1,d_2,\dots ,d_n\right\}$ denote the set of round-trip propagation delay, and $B=\left\{b_1,b_2,\dots ,b_n\right\}$ denote the set of bandwidth. The number of elements in $D$ and $B$ are both $n$. The bandwidth and round-trip propagation delay of link $l$ is $b_l$ and $d_l$ To simplify the analysis, we assume that the propagation delay from the receiver to sender is zero. Round Robin (RR) is used to dispatch packets in the given network mode, which let multiple paths take turns in transferring data packets in a periodically repeated order. We choose NewReno [36] as the congestion control algorithm since it is still the widely deployed variant of TCP.

4. Performance Analysis Model

In this section, the performance analysis model of TCP over multiple heterogeneous paths is built by analyzing the average throughput. We divide the TCP flow into consecutive transmission round. The duration time as well as the number of segments transmitted at each round are first analyzed. Then, the average throughput is derived using an iteration model. At last, the effect of link quality asymmetry on average throughput is discussed.

Table 1. Notations.

Parameter Description
L	The set of available links
n	The number of elements in L
D	The set of round-trip propagation delay of available links
B	The set of bandwidth of available link
s	The size of a segment
mACK	Receiver reply an ACK after receiving mACK consecutive segments
SGMi,j	The j-th segment that sender transmits at i-th round
wi	The congestion window of i-th round
${\delta }_w^i$	The increment of congestion window at i-th round
Ai	The number of segments acknowledged by i-th non-duplicate ACK
Ci	The number of segments that can be transmitted at i-th round
Ti	The time between the i-th round and (i + 1)-th round
ηi,j	The number of the link used to transmit the j-th segment of Ci at i-th round
Di,j	The propagation delay and queuing delay of j-th segment of Ci at i-th round
Is	The number of rounds that the slow start phase ends
Ws	The slow start threshold of congestion window
WI	The initial value of congestion window

summarizes important parameters used in this paper.

4.1. Analysis of i-th Transmission Round

First, we focus on the transmission of segments at sender side. Let $i$ denote the number of transmission round from the beginning of the transmission. At $i$-th round, sender transmits a certain number of unsent segments and waits for the acknowledgements. Since in most TCP implementations (such as NS3) only non-duplicate ACK triggers the transmission of previously unsent data, we can conclude that the $i$-th round begins with the arrival of $\left(i-1\right)$-th non-duplicate ACK.

Let $C_i$ denote the total number of segments transmitted at $i$-th round. $C_i$ equals the free space in the congestion window, which is composed of two parts: the increment in size of congestion window and the decrement in number of outstanding segments. We define $w_i$ as the size of the congestion window of $i$-th round, and ${\delta }_w^i=w_i-w_{i-1}$ as the increment of the congestion window. Let $A_i$ denote the number of segments newly acknowledged by $i$-th non-duplicate ACK, then $C_i$ can be expressed as:

$$C_i=(A_i+{\delta }_w^i).$$

(1)

The $j$-th segment of $C_i$ is defined as ${\mathrm{SGM}}_{\mathrm{i},\mathrm{j}}$. Let ${\eta }_{i,j}$ denote the number of the link used to send the ${\mathrm{SGM}}_{\mathrm{i},\mathrm{j}}$, where $l_{{\eta }_{i,j}}\in L$ and ${\eta }_{i,j}\in \left\{1, 2,\dots ,n\right\}$. Supposing segments are scheduled over $n$ links in a round-robin manner, and the first one travels over link $l_1$. Hence ${\eta }_{i,j}$ can expressed as:

$${\eta }_{i,j}=\left[\left(j-1+{\displaystyle \sum }_{k=1}^{i-1}{C}_k\right) mod n\right]+1.$$

(2)

The round-trip propagation delay as well as the bandwidth of link $l_{{\eta }_{i,j}}$ are $d_{{\eta }_{i,j}}$ and $b_{{\eta }_{i,j}}$ respectively. Let $D_{i,j}$ be the time elapsed between the beginning of $i$-th round and when ${\mathrm{SGM}}_{\mathrm{i},\mathrm{j}}$ reaches the receiver, which is the sum of queuing delay and propagation delay experienced by ${\mathrm{SGM}}_{\mathrm{i},\mathrm{j}}$. Thus,

$$D_{i,j}=\frac{\left(\lfloor \frac{j}{n}\rfloor +1\right)s}{b_{{\eta }_{i,j}}}+d_{{\eta }_{i,j}}.$$

(3)

In Equation (3), $\lfloor \frac{j}{n}\rfloor$ is the quotient of j and n, while $S$ is the average size of segments. The queuing delay is represented by $\left[\left(\lfloor \frac{j}{n}\rfloor +1\right)s\right]/b_{{\eta }_{i,j}}$, while $d_{{\eta }_{i,j}}$ represents the propagation delay.

Then we discuss the arrival of segments and the response of ACKs at receiver side. We define $T_i$ as the latency between the beginning of $i$-th round and the time when sender receives the first non-duplicate ACK that starts the $\left(i+1\right)$-th round from the receiver. The number of segments the first non-duplicate acknowledges is exactly $A_{i+1}$. A non-duplicate ACK will be fired by the receiver only if: (1) an expected number of consecutive segments are received; (2) the first out-of-order segments arrives after some consecutive segments or (3) a segment that fills the gap in the receiver’s buffer arrives. The satisfaction of these criteria highly associates with the arrival order of the first segment transmitted at $i$-th round, which is ${\mathrm{SGM}}_{\mathrm{i},1}$. Hence, based on whether ${\mathrm{SGM}}_{\mathrm{i},1}$ is the first to reach the receiver, we respectively calculate $T_i$ and $A_{i+1}$.

4.1.1. Case I: ${\mathrm{SGM}}_{\mathrm{i},1}$ is the First to Reach the Receiver

We define $P\left(F\right)$ as the probability that ${\mathrm{SGM}}_{\mathrm{i},1}$ arrives at the receiver first, which can be presented as:

$$P\left(F\right)=P\left(D_{i,1}=\underset{j\in \left\{1,2,\dots ,C_i\right\}}{\min }\left\{D_{i,j}\right\}\right).$$

(4)

The segments are scheduled over the links in a round-robin manner, thus $D_{i,j}$ follows a uniform distribution after a large amount of transmission rounds. Hence $P\left(F\right)$ approximately equals $1/C_i$.

Most TCP implementations (such as NS3) utilize a counter to delay replying cumulative ACK. Let $m_{ACK}$ denote this counter, after receiving $m_{ACK}$ consecutive segments the receiver will reply an ACK. In this case, since the receiver receives ${\mathrm{SGM}}_{\mathrm{i},1}$ first, it will wait for the following $m_{ACK}-1$ segments before replying an ACK until the arrival of first out-of-order segment, as shown in Figure 3. Let $m$ be the number of consecutive segments received before the arrival of first out of order segments. In other words, ${\mathrm{SGM}}_{\mathrm{i},2}$ to ${\mathrm{SGM}}_{\mathrm{i},\mathrm{m}}$ arrive consecutive and ${\mathrm{SGM}}_{\mathrm{i},\mathrm{m}+1}$ is out-of-order. Thus, the receiver will reply the first non-duplicate ACK acknowledging $m$ segments approximately after the arrival of ${\mathrm{SGM}}_{\mathrm{i},\mathrm{m}}$.

If $m$ is smaller than $m_{ACK}$, we have $T_i\cong D_{i,m}$ and $A_{i+1}=m,$ where $D_{i,m}$ is the time between the beginning of $i$-th round and arrival of ${\mathrm{SGM}}_{\mathrm{i},\mathrm{m}}$. The probability $P(m<m_{ACK }| F)$ can be calculated as:

$$P(m<m_{ACK }| F)={\displaystyle \sum }_{k=1}^{m_{ACK}-1}\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}.$$

(5)

If $m$ is equal to or larger than $m_{ACK}$, $T_i\cong D_{i,m_{ACK}}$ and $A_{i+1}=m_{ACK}$. The corresponding probability $P(m\ge m_{ACK }| F)$ can be calculated as:

$$P(m\ge m_{ACK }| F)=\frac{1}{\left(C_i-1\right)!}+{\displaystyle \sum }_{k=m_{ACK}}^{C_i-2}\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}.$$

(6)

Let $E^{\prime }\left(T_i\right)$ and $E^{\prime }\left(A_{i+1}\right)$ denote the expected value of $T_i$ and $A_{i+1}$ under the condition that ${\mathrm{SGM}}_{\mathrm{i},1}$ is the first to reach the receiver. Based on the probabilities calculated in Equations (5) and (6), and the corresponding $T_i$ and $A_{i+1}$, $E^{\prime }\left(T_i\right)$ and $E^{\prime }\left(A_{i+1}\right)$ can be derived as:

$$\begin{array}{ccc}\hfill E^{\prime }\left(T_i\right)={\displaystyle \sum }_{k=1}^{m_{ACK}-1}& D_{i,k}\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}\hfill & \\ & +D_{i,m_{ACK}}{\displaystyle \sum }_{k=m_{ACK}}^{C_i-2}& \frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}+\frac{D_{i,m_{ACK}}}{\left(C_i-1\right)!},\end{array}$$

(7)

$$\begin{array}{ccc}\hfill E^{\prime }\left(A_{i+1}\right)={\displaystyle \sum }_{k=1}^{m_{ACK}-1}& k\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}\hfill & \\ & +m_{ACK}{\displaystyle \sum }_{k=m_{ACK}}^{C_i-2}& \frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{\left(C_i-1\right)!}+\frac{m_{ACK}}{\left(C_i-1\right)!}.\end{array}$$

(8)

4.1.2. Case II: ${\mathrm{SGM}}_{\mathrm{i},1}$ is not the First to Reach the Receiver

$P\left(\overline{F}\right)$ is defined as the probability of the ${\mathrm{SGM}}_{\mathrm{i},1}$, where it is not the first to reach the receiver, which approximately equals $\left(1-1/C_i\right)$. In this case, the receiver will not reply any non-duplicate ACK before the arrival of ${\mathrm{SGM}}_{\mathrm{i},1}$. Moreover, since the segments transmitted subsequent to ${\mathrm{SGM}}_{\mathrm{i},1}$ arrives at the receiver earlier than it, there must be gaps in the receiver’s buffer before the arrival of ${\mathrm{SGM}}_{\mathrm{i},1}$. As shown in Figure 4, the receiver will immediately reply a non-duplicate ACK after receiving ${\mathrm{SGM}}_{\mathrm{i},1}$, since the ${\mathrm{SGM}}_{\mathrm{i},1}$ will fill part of the existing gap. Hence, $T_i$ equals $D_{i,1}$.

Let $\left(q-1\right)$ denote the number of received consecutive segments counting from ${\mathrm{SGM}}_{\mathrm{i},2}$ before the arrival of ${\mathrm{SGM}}_{\mathrm{i},1}$. Hence, the number of segments the non-duplicate ACK can acknowledge equals $q$, consequently we have $A_{i+1}=q$. The probability of $P(q=k | \overline{F})$ is derived as follows:

$$P(q=k | \overline{F})=\{\begin{array}{cc}\frac{\left(C_i-2\right)\left(C_i-1\right)!}{2C_i!-2\left(C_i-1\right)!},\hfill & \hfill k=1\\ \frac{(C_i-1)!}{C_i!-(C_i-1)!},\hfill & \hfill k=C_i\\ \frac{\left(C_i-k-1\right)\left[{{\displaystyle \sum }}_{l=k+1}^{C_i}\frac{\left(l-2\right)!}{\left(l-k-1\right)!}\left(C_i-l+1\right)\right]}{C_i!-\left(C_i-1\right)!},\hfill & \hfill 1<k<C_i.\end{array}$$

(9)

Let $E^{″}\left(T_i\right)$ and $E^{″}\left(A_{i+1}\right)$ denote the expected value of $T_i$ and $A_{i+1}$ under the condition that ${\mathrm{SGM}}_{\mathrm{i},1}$ is not the first to reach the receiver, which can be derived as:

$$E^{″}\left(T_i\right)=D_{i,1},$$

(10)

$$E^{″}\left(A_{i+1}\right)=\frac{\left(3C_i-2\right)\left(C_i-1\right)!}{2C_i!-2\left(C_i-1\right)!}+{\displaystyle \sum }_{k=2}^{C_i-1}k\frac{\left(C_i-k-1\right)\left[{{\displaystyle \sum }}_{l=k+1}^{C_i}\frac{\left(l-2\right)!}{\left(l-k-1\right)!}\left(C_i-l+1\right)\right]}{C_i!-\left(C_i-1\right)!}.$$

(11)

Based on case I and case II, we can derive the expected value of $T_i$ and $A_{i+1}$ as $E\left(T_i\right)=E^{\prime }\left(T_i\right)P\left(F\right)+E^{″}\left(T_i\right)P\left(\overline{F}\right)$ and $E\left(A_{i+1}\right)=E^{\prime }\left(A_{i+1}\right)P\left(F\right)+E^{″}\left(A_{i+1}\right)P\left(\overline{F}\right)$. Given that $P\left(F\right)=1/C_i$ and $P\left(\overline{F}\right)=\left(1-1/C_i\right)$, $E\left(T_i\right)$ and $E\left(A_{i+1}\right)$ can be calculated as follows:

$$E\left(T_i\right)=E^{″}\left(T_i\right)+\frac{1}{C_i}\left(E^{\prime }\left(T_i\right)-E^{″}\left(T_i\right)\right),$$

(12)

$$E\left(A_{i+1}\right)=E^{″}\left(A_{i+1}\right)+\frac{1}{C_i}\left(E^{\prime }\left(A_{i+1}\right)-E^{″}\left(A_{i+1}\right)\right).$$

(13)

From Equations (8) and (11) we can find that the expected value of $A_{i+1}$ depends on $C_i$, which means $A_{i+1}$ is a function of $C_i$. For simplicity, we define $A_{i+1}=F\left(C_i\right)$. Since $C_{i+1}$ equals the sum of $A_{i+1}$ and ${\delta }_w^{i+1}$, the increment of the congestion window needs to be discussed.

In the slow start phase, the congestion window is incremented by one segment for each ACK, thus ${\delta }_w^i$ equals 1. Let $W_s$ denote the slow start threshold of congestion window, and $W_I$ the initial size of congestion window. Let $I_s$ be the number of rounds that the slow start phase ends. Since the congestion window is increased by one every round, thus:

$$I_s=W_s-W_I.$$

(14)

In the congestion avoidance phase, the congestion is increased by $1/w$ on every incoming ACK that acknowledges new data. Thus, we have ${\delta }_w^i=1/w_{i-1}$. The congestion window at $\left(i+1\right)$-th round can be expressed as:

$$w_{i+1}=\{\begin{array}{c}{w}_i+1, i<I_s\\ w_i+\frac{1}{w_i},i\ge I_s\end{array}.$$

(15)

Based on the above analysis, the relationship between $C_{i+1}$ and $C_i$ can be derived as Equation (15), where function $F\left(�\right)$ is defined in Equation (13):

$$C_{i+1}=\{\begin{array}{c}F\left(C_i\right)+1, i<I_s\\ F\left(C_i\right)+\frac{1}{w_i},i\ge I_s\end{array}.$$

(16)

4.2. Iteration for Average Throughput

According to Equation (16), the number of segments transmitted at next round can be derived based on that at the current round. Thus, the total segments transmitted from the beginning to current round of transmission can be calculated by iteration from the first round. The total time spent on transmitting can also be obtained by summing up the duration time of each transmission round. Consequently, the average throughput can be derived.

Therefore, we formulate the performing process of the model iteration as follows:

Step 1

Supposing $E$ bytes of data are expected to be received by the receiver. At the first round of transmission, $C_1$ segments are sent within $E(T_1)$ seconds, where $C_1$ equals the initial size of the congestion window, which is $W_I$. $E(T_1)$ can be calculated according to (12).

Step 2

At $i$-th round ($i\ge 2$), substituting $w_{i-1}$ into (15), we can get $w_i$.

Step 3

At $i$-th round ($i\ge 2$), substituting $C_{i-1}$ and $w_{i-1}$ into (16), we can get $C_i$. Further, according to (12), $E\left(T_i\right)$ is computed.

Step 4

Compute total transmitted bytes from beginning to $i$-th round, which is:

$$\mathrm{Total}\mathrm{Transmitted}\mathrm{Bytes}=s{\displaystyle \sum }_{k=1}^i{C}_k.$$

(17)

Step 5

Let $\widehat{T}$ denote total transmission time from beginning to $i$-th round, which can be computed as:

$$\widehat{T}={\displaystyle \sum }_{k=1}^{i}E\left(T_k\right).$$

(18)

Step 6

If total transmitted bytes is smaller than $E$, which is the number of bytes expected by the receiver, repeat Steps 2–5. Otherwise, the iteration stops, and the average throughput can be calculated as:

$$\mathrm{Average}\mathrm{Throughput}=\frac{E}{\widehat{T}}.$$

(19)

4.3. Discussion of Link Quality Asymmetry

Here we discuss how link quality asymmetry affects the throughput of TCP transferred over multiple heterogeneous links. According to the proposed model, when transmitting a certain number of bytes, the average throughput is inversely proportional to the total transmission time $\widehat{T}$. Since $\widehat{T}$ is the sum of $E\left(T_i\right)$ defined in Equation (12), the average throughput decreases with increasing $E\left(T_i\right)$. Substituting $E^{\prime }\left(T_i\right)$ defined in (7) and $E^{″}\left(T_i\right)$ defined in (10) into (12), $E\left(T_i\right)$ can be evaluated as:

$$\begin{array}{c}E\left(T_i\right)={\displaystyle \sum }_{k=m_{ACK}}^{C_i-2}(D_{i,m_{ACK}}-D_{i,1})\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{C_i!}+\frac{(D_{i,m_{ACK}}-D_{i,1})}{C_i!}+D_{i,1}\\ +{\displaystyle \sum }_{k=1}^{m_{ACK}-1}(D_{i,k}-D_{i,1})\frac{\left(C_i-k-1\right)\left(C_i-k-1\right)!}{C_i!}.\end{array}$$

(20)

Apart from parameter $D_{i,1}$, $E\left(T_i\right)$ is primarily associated with $(D_{i,k}-D_{i,1})$, where $k\in \left[1,m_{ACK}\right]$. $(D_{i,k}-D_{i,1})$ represents the time difference between the arrival of the first transmitted segment ${\mathrm{SGM}}_{\mathrm{i},1}$ and the $k$-th transmitted segment ${\mathrm{SGM}}_{\mathrm{i},\mathrm{k}}$ at the receiver side. Longer time difference increments the overall $E\left(T_i\right)$. Substituting (3), $(D_{i,k}-D_{i,1})$ can be evaluated as:

$$D_{i,k}-D_{i,1}=\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)+\frac{\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)s}{b_{{\eta }_{i,k}}{b}_{{\eta }_{i,1}}}+\frac{\lfloor \frac{k}{n}\rfloor s}{b_{{\eta }_{i,k}}}.$$

(21)

As demonstrated in Equation (18), $(D_{i,k}-D_{i,1})$ is mainly dominated by two elements, $\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)$ and $\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)$. $\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)$ and $\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)$ are respectively the delay difference and bandwidth difference between two links that transmit ${\mathrm{SGM}}_{\mathrm{i},1}$ and ${\mathrm{SGM}}_{\mathrm{i},1}$, i.e., $l_{{\eta }_{i,1}}$ and $l_{{\eta }_{i,k}}$. Increasing $\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)$ and $\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)$ leads to greater $(D_{i,k}-D_{i,1})$, and consequently causes larger $E\left(T_i\right)$, which eventually results in a decrease in average throughput.

As mentioned earlier, when scheduled in round-robin manner, the possibility of selecting one of $n$ available links to transmit a certain segment follows a uniform distribution after a large amount of transmission rounds. Thus, $l_{{\eta }_{i,1}}$ and $l_{{\eta }_{i,k}}$ can represent any two links of set $\left\{l_1,l_2,\dots ,l_n\right\}$. Note that $l_{{\eta }_{i,1}}$ is not the first link of $n$ available links, but the link used to transmit ${\mathrm{SGM}}_{\mathrm{i},1}$. Equally, $\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)$ and $\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)$ can be the delay difference and bandwidth difference between any two links. From this point of view, $\left(d_{{\eta }_{i,k}}-d_{{\eta }_{i,1}}\right)$ and $\left(b_{{\eta }_{i,1}}-b_{{\eta }_{i,k}}\right)$ reflect the extent of deviation in link quality of all links. We refer to such delay difference and bandwidth difference between any two links as delay asymmetry and bandwidth asymmetry. Therefore, it can be concluded that the average throughput is subject to delay asymmetry and bandwidth asymmetry. The more significant these two parameters become, the lower average throughput will be.

To quantify delay asymmetry, we introduce Average Delay Asymmetry, which is defined as the average absolute delay difference between any two links of $n$ available links. Average Delay Asymmetry can be calculated as:

$$\mathrm{Average}\mathrm{Delay}\mathrm{Asymmetry}=\frac{2{{\displaystyle \sum }}_{q=2}^n{{\displaystyle \sum }}_{r=1}^{q-1}\left|d_r-d_q\right|}{n\left(n-1\right)}, d_q,d_r\in D.$$

(22)

Similarly, Average Bandwidth Asymmetry can be defined as:

$$\mathrm{Average}\mathrm{Bandwidth}\mathrm{Asymmetry}= \frac{2{{\displaystyle \sum }}_{q=2}^n{{\displaystyle \sum }}_{r=1}^{q-1}\left|b_r-b_q\right|}{n\left(n-1\right)},b_q,b_r\in B.$$

(23)

Average Delay Asymmetry and Average Bandwidth Asymmetry can both affect the performance of TCP transferred over multiple heterogeneous links. Comparison of extent of these two parameters on TCP performance will be presented in Section 6.

5. Model Evaluation

The proposed model in Section 4 is first carefully evaluated through a large scale simulation study. The parameters in both model prediction and simulation are taken from the datasets collected in the measurement discussed in Section 3. The simulation of TCP over multiple heterogeneous links is implemented in Network Simulator 3 (NS3) [37]. To verify that our model can be used in practical environments, the comparison between the model and the field experiment results is further presented.

5.1. Evaluation Using Simulation Study

5.1.1. Simulation Implementation

Figure 5 depicts the simulation topology. Two endpoints are connected by multiple Point-to-Point Protocol (PPP) links. At each point, apart from two PPP network adapters, a virtual network device (VND) that works at the network layer was added. An IP address is assigned to VND. Between two endpoints, a TCP connection binding to the IP addresses of two VNDs is established. At both endpoints, TCP NewReno is used. When a TCP segment of the established connection is pushed down to the network layer, the corresponding IP packet will be forwarded to VND. VND then passes the IP packet to a dedicated packet-processing program attached to it. The IP packet will be encapsulated into a UDP datagram and then sent to the peer from one of the PPP network adapters. A Round-Robin scheduling algorithm is employed to decide which network adapter will be used to transmit the subsequent encapsulated packets. Thus, the segments of the single TCP connection established between two endpoints will be concurrently transmitted from all the available PPP network adapters.

To measure the throughput of simulated TCP over multiple heterogeneous links in NS3, a sending application is installed on endpoint A, and a receiving application is installed on endpoint B. A then sends $C$ bytes data to B, and B records the transfer finish time as $T$ seconds. Thus, the throughput can be calculated as $C/T$ bytes per second.

5.1.2. Evaluation Methodology

Using the proposed model and the simulation respectively, two sets of throughputs of TCP over multiple heterogeneous links are obtained for comparison. For each case, the derivation of throughput is performed under different number of heterogeneous links employed for concurrent transmission. When utilizing a certain number of links, the bandwidth of a link remains constant but different from those of the other links. The value of delay of a link is fetched from an individual dataset associated with that link. For example, if $m$ links are employed for a concurrent transmission, and the delay dataset of each link contains $n$ values, then there will be $n^m$ combinations of delay values. The Average Delay Asymmetry of $n^m$ groups of delay will be calculated and sorted, from which 36 groups of delay will be evenly selected. For selected groups of delay values, the derivation of throughput is repeated using simulation and proposed model correspondingly.

5.1.3. Parameter Settings

The parameters for model prediction or simulation are taken from the measurement results of field investigation towards the wireless network heterogeneity, as described in Section 3. Since eight modems were measured during the investigation, up to eight links can be employed for concurrent transmission in model prediction or simulation, namely link I to link VIII. For example, if our links are needed, Link I, II, III and IV will be utilized. Link I, II and III represents FDD-LTE of China Telecom, Link IV, V and VI represents FDD-LTE of China Unicom, link VII and VIII represents TD-LTE of China Mobile. The bandwidth of link I to link VII are set as the maximum measured download data rates shown in Figure 1b, which are respectively 35.9 Mbps, 18.4 Mbps, 33.3 Mbps, 14.7 Mbps, 14.8 Mbps, 4.4 Mbps, 22.5 Mbps and 12.5 Mbps.

The field measurement results of RTT of a modem are directly adopted as the delay dataset of corresponding link in the simulation.

The other parameters used in proposed model are set according to

Table 2. Evaluation Parameters.

Parameter	Value	Parameter	Value
${\mathrm{W}}_{\mathrm{I}}$	536 bytes	${\mathrm{W}}_{\mathrm{S}}$	65,535 bytes
$\mathrm{S}$	536 bytes	${\mathrm{m}}_{\mathrm{ACK}}$	2 segments

.

5.1.4. Evaluation Results

We introduce prediction accuracy to evaluate the proposed model’s consistency to simulation results. Supposing the predicted throughput using the proposed model is $T_M$, the derived throughput using simulation under same circumstance is $T_S$, then prediction accuracy is defined as:

$$\mathrm{Prediction}\mathrm{Accuracy}=1-\frac{\left|T_S-T_M\right|}{T_S}.$$

(24)

The evaluation results with number of links varying from 2 to 8 are depicted in Figure 6, where throughput is plotted against the cyan circles, which represent the simulation results, and the red crosses indicate the predicted values using the proposed model. It can be observed that there is a good match between the model prediction and the simulation results in all cases. With the number of links employed for concurrent transmission varying from 2 to 8, the prediction accuracies are 89.68%, 83.14%, 79.26%, 75.99%, 73.24%, 71.06% and 69.50%. The prediction accuracies slightly drop as the number of utilized links increases. This is due to that the error introduced by the randomness becomes larger in the proposed mode when the number of links available for transmission grows. In the model derivation, we assumed that the probability with which ${\mathrm{SGM}}_{\mathrm{i},1}$ arrives at the receiver first follows an uniform distribution after a large number of transmission round. This means ${\mathrm{SGM}}_{\mathrm{i},1}$ could be transferred through any link with the same probability. This assumption greatly simplifies the modeling effort, and holds true for the situation when the number of links is not very large. However, if the number of available links becomes large, the number of rounds needed to complete the transmission reduces. Thus the corresponding probability may not be uniformly distributed any more. Even so, the average prediction accuracy can reach 77.41%. Besides, it can be observed that the throughput decreases as the number of links increases when average delay asymmetry is large. This is because a large number of links multiplies the effect of delay asymmetry on throughput.

5.2. Evaluation Using Field Experiment

To further prove the accuracy of the proposed model, we measured the throughput of multipath TCP in real-world heterogeneous mobile scenarios and compared the results to our model. Like the field investigation for link quality asymmetry described in Section 3, this field measurement for multipath TCP throughput was also conducted on a high-seed train. As shown in Figure 7, a client on the train tried to download a large file from the server through the wireless router that can access to 6 heterogeneous wireless networks. The hardware of wireless router was the same as the measurement device described in Section 3, while the forwarding scheme was designed using the same the methodology as in simulation implementation. The wireless router ran Linux 3.13.11.11 and adopted its default TCP congestion control algorithm. The throughput of client was recorded by a bandwidth monitoring software running on it. During the measurement for client throughput, the RTT and data rate of each wireless network were measured at wireless router using the same measuring programs described in Section 3. The collected dataset of RTT and data rate served as the input to our proposed model, which generated the corresponding throughput results.

The comparison between the results of the field measurement and the proposed model is shown in Figure 8. It can be observed that the derived throughput of the proposed agree well with the results of field measurement. Compared to the measured throughput, the accuracy of proposed model can reach 71.25%, which is similar to the result of simulation evaluation using 6 links in Section 5.1.4. Thus, we can conclude that the proposed model is also accurate for TCP over multiple heterogeneous links in practical environment.

6. Analysis Based on the Proposed Model

In this section, the effect of path heterogeneity on performance of TCP flow transferred over multiple heterogeneous paths is analyzed based the proposed model. Firstly, the influence of Average Delay Asymmetry as well as Average Bandwidth Asymmetry on the throughput is investigated. Then we discuss the policy of determining appropriate number of links to transmit the segments of TCP flow over multiple heterogeneous paths.

6.1. The Influence of Delay and Bandwidth Asymmetry

In Section 4 we concluded that the performance of TCP over multiple heterogeneous links is subject to link quality asymmetry. But, to what extent do Average Delay Asymmetry and Average Bandwidth Asymmetry affect the performance? Which is the main factor that affects the TCP performance, Average Delay Asymmetry or Average Bandwidth Asymmetry? It is an interesting issue to study these subjects.

To answer these questions, we use the proposed performance analysis model to evaluate the TCP throughput as a function of both Average Delay Asymmetry and Average Bandwidth Asymmetry. The minimum delay and bandwidth are 5 ms and 100 kbps. The Average Delay Asymmetry and Average Bandwidth Asymmetry are set to vary from 0 ms to 35 ms and from 0 kbps to 700 kbps. In this case, the maximum of Average Delay Asymmetry and Average Bandwidth Asymmetry are both seven times of minimum delay and bandwidth. The number of links utilized for concurrently transmitting data varies from 1 to 4. The evaluation results are shown in Figure 9.

Figure 9 shows that the average throughput drops significantly to the axis of Average Delay Asymmetry but decreases at a much slower pace to the axis of Average Bandwidth Asymmetry. This phenomenon is particularly obvious when four links are used to concurrently transfer the TCP flow. In this case, under highest level of Average Delay Asymmetry, the average throughput decreases by 1.8 times as the Average Bandwidth Asymmetry varies from zero to maximum. In contrast, when Average Bandwidth Asymmetry remains at highest level, and the Average Delay Asymmetry varies from zero to maximum, the average throughput is reduced by 2.8 times.

Based on the above analysis, we can conduct that the Average Delay Asymmetry is the main factor that affects the throughput performance of TCP flow over multiple heterogeneous paths. Equations (20) and (21) can explain this phenomenon. From Equation (20) it is observed that the transmission time is subject to $(D_{i,k}-D_{i,1})$, and $(D_{i,k}-D_{i,1})$ is comprised of three factors. According to Equation (21), the average delay asymmetry contributes more to the value of $(D_{i,k}-D_{i,1})$ than average bandwidth asymmetry. Thus, the throughput performance is more affected by average delay asymmetry. This inference can guide the design of multipath transmission mechanism in heterogeneous networks.

6.2. Relationship between the Throughput Performance and the Number of Links

Knowing that Average Delay Asymmetry is the dominant factor that affects the TCP throughput transferred over multiple heterogeneous paths, we can now investigate the relationship between the TCP performance and the number of links employed for transmission under different level of Average Delay Asymmetry. Further, the optimal number of links should be used to achieve optimized performance is discussed.

Under four groups of minimum delay, we evaluate the throughput of TCP flows employing different number of links as a function of Average Delay Asymmetry. During the evaluation, up to 4 links are utilized and the bandwidth of each link is 100 kbps. Along with the Average Delay Asymmetry varying from 10 to 90 ms, the throughput is derived using the proposed model under the minimum delay of 5 ms, 20 ms, 35 ms and 50 ms respectively. The results of the evaluation are depicted in Figure 10.

According to Figure 10, we can find that the Average Delay Asymmetry of heterogeneous networks compromised the benefits of aggregating bandwidth by utilizing multiple links. Meanwhile, large minimum delay exacerbates the effect of Average Delay Asymmetry on throughput performance using multiple links. Under the minimum delay of 5 ms, when the Average Delay Asymmetry increases to 35.6 ms, the throughput of TCP concurrently transferred over four links decreases to that of TCP using only one link. When minimum delay increases from 5 to 50 ms, such threshold of Average Delay Asymmetry at which the throughput of four links equals to that of one link decreases from 35.6 to 30.6 ms. Also, it can be observed that the performance of TCP degrades as the number of links increases when the average delay asymmetry is at high level. This is because the effect of link quality asymmetry on throughput is magnified by large number of utilized links, especially when the extent of link quality asymmetry is severe. In this situation, the more links used for transmitting a TCP flow, the lower throughput will be. Based on the above evaluation results, we can roughly derive a criterion of determining the number of links to optimize the throughput performance. For example, when the minimum delay is more than 5 ms and the Average Delay Asymmetry is more than 20 ms, utilizing two links to transfer the TCP flow will achieve maximum throughput.

7. Discussion

The proposed model reveals how network heterogeneity can affect the performance of TCP over multiple heterogeneous paths for 5G mobile services. In a nutshell, link quality asymmetry of heterogeneous transmission paths could lead to packet reordering issue, which can deteriorate the throughput of TCP due to the inherent inefficiency of Additive Increase Multiplicative Decrease (AIMD) congestion control. The proposed model and some inferences discussed in Section 6 shows that the performance of TCP over multiple heterogeneous paths can be improved in two directions. One way is to directly reduce the occurrence of packet reordering. For example, one can design a novel scheduling algorithm that stripes packets over multiple paths as a function of link quality asymmetry. Another way is to improve the TCP itself to better tolerate packet reordering. The key is to prevent congestion control from reacting to packet reordering as if real congestion happens. Some similar efforts have been made to design congestion control mechanisms that can differentiate real congestion from other events, such as random loss. Among these efforts, TCP-Friendly Rate Control (TFRC) is a successful attempt [38]. It used a control equation to determine how fast the sender can transmit based on recent loss event rate. Another good source of inspiration is Opportunistic Hybrid Transport Protocol (OHTP), which proposed a congestion control mechanism for cognitive radio sensor networks that can differentiate interruption loss from real congestion [39].

8. Conclusions

In this paper, the severe extent of link quality asymmetry in real world situations is revealed based on field measurement, and then a performance analysis model for TCP over multiple heterogeneous paths for 5G services is derived regarding average throughput. Taking into the consideration of both bandwidth and delay asymmetry, we carefully investigated the transmission of TCP segments over multiple heterogeneous links and derived the corresponding performance analysis model. The proposed model is validated by comparison with the simulation experiment as well as the field experiment. The results prove that the proposed performance analysis model can achieve high analytical accuracy in practical environment. Further analysis based on the proposed model reveals some interesting inferences. First, compared to bandwidth asymmetry, delay asymmetry is the dominant factor that affects the performance of TCP over heterogeneous networks. Second, the criteria of determining appropriate number of links to be used to optimize the TCP multipath performance is discussed. The proposed model can provide a guidance to the design of CMT solutions for 5G mobile services.