Advanced DSP for Coherent Optical Fiber Communication

Abstract

In this paper, we provide an overview of recent progress on advanced digital signal processing (DSP) techniques for high-capacity long-haul coherent optical fiber transmission systems. Not only the linear impairments existing in optical transmission links need to be compensated, but also, the nonlinear impairments require proper algorithms for mitigation because they become major limiting factors for long-haul large-capacity optical transmission systems. Besides the time domain equalization (TDE), the frequency domain equalization (FDE) DSP also provides a similar performance, with a much-reduced computational complexity. Advanced DSP also plays an important role for the realization of space division multiplexing (SDM). SDM techniques have been developed recently to enhance the system capacity by at least one order of magnitude. Some impressive results have been reported and have outperformed the nonlinear Shannon limit of the single-mode fiber (SMF). SDM introduces the space dimension to the optical fiber communication. The few-mode fiber (FMF) and multi-core fiber (MCF) have been manufactured for novel multiplexing techniques such as mode-division multiplexing (MDM) and multi-core multiplexing (MCM). Each mode or core can be considered as an independent degree of freedom, but unfortunately, signals will suffer serious coupling during the propagation. Multi-input–multi-output (MIMO) DSP can equalize the signal coupling and makes SDM transmission feasible. The machine learning (ML) technique has attracted worldwide attention and has been explored for advanced DSP. In this paper, we firstly introduce the principle and scheme of coherent detection to explain why the DSP techniques can compensate for transmission impairments. Then corresponding technologies related to the DSP, such as nonlinearity compensation, FDE, SDM and ML will be discussed. Relevant techniques will be analyzed, and representational results and experimental verifications will be demonstrated. In the end, a brief conclusion and perspective will be provided.

1. Introduction

With the development of Erbium doped fiber amplifier (EDFA), wavelength division multiplexing (WDM), dispersion management and optical fiber nonlinearity compensation technologies, optical fiber communication capacity has been rapidly improved over the past few decades. In addition, the research on high-order modulation formats has also increased the transmission capacity and spectral efficiency (SE) of optical fibers. Due to a smaller bandwidth occupied (for the same bit rate), systems with higher SE are usually more tolerant to chromatic dispersion (CD) and polarization mode dispersion (PMD). The fault tolerance to CD and PMD are particularly crucial in high bit-rate transmission systems. In order to achieve a high SE, early works used direct detection (incoherent detection) and amplitude-based modulation (1-dimension). Actually, higher SE can be obtained using coherent detection and 2-dimension modulation formats.

Coherent detection attracted intensive research in the 1980s due to its high receiver sensitivity. With the invention of EDFA and the development of WDM systems, research in coherent optical communication ceased in early 1990s, because of the difficulty and complexity of system implementation, especially the complicated realization of optical phase locked loop. Phase and polarization management turned out to be the major obstacles for the practical implementation of conventional coherent receivers. Fortunately, both phase and polarization management can be realized in the electrical domain using DSP. With the increasing demand on transmission capacity, coherent optical communication has attracted widespread attention again in recent years, and has become an unprecedented and promising approach for realizing high-capacity long-haul optical communication systems [1,2,3,4,5].

From the beginning of this century, coherent detection combined with large-bandwidth analog-to-digital converter (ADC), digital-to-analog converter (DAC) and DSP has increased the achievable capacity of optical fiber communication systems [6,7,8,9,10]. In coherent optical communication, information is encoded onto optical carrier waves in the complex domain, and the optical signal carrying all information is transmitted to the front end, after undergoing a series of linear and nonlinear impairments. In order to recover the transmitted signal and obtain the original information, it is necessary to measure the full complex electric field of the light wave, which means that the phase and the magnitude of optical carrier field both need to be detected. Using coherent detection, the complex field of the received signal can be completely acquired, and the linear transmission impairments, such as CD and PMD, can be fully compensated by using static and adaptive DSP.

One of the main benefits of coherent optical communication is the possibility of compensation of transmission impairments using DSP. Compensation of linear impairments such as CD and PMD are now implemented using DSP. Compensation of nonlinear impairments brings about an increase in the fundamental capacity limit for fiber transmission, which is an important research area for coherent optical communications. Advanced DSP has also been developed in the frequency domain to reduce the computational complexity and maintain all of the advantages of time domain equalization (TDE) based adaptive digital filters. Machine learning (ML) techniques have become one of the most promising disciplines and is beneficial to optical fiber communication applications such as nonlinearity mitigation, optical performance monitoring (OPM), carrier recovery, in-band optical signal-to-noise ratio (OSNR) estimation and modulation format classification, and especially, advanced DSP.

In this paper, the latest development and progress of DSP approaches for coherent optical communications are reviewed. The paper is organized as follows: Section 2 describes the principle and schematic of the coherent detection as well as linear impairments equalization. From Section 3 to Section 5, nonlinearity compensation, SDM applications and FDE approaches are presented in detail, respectively. Section 6 will explore the application of recently proposed and promising ML technologies in optical communications. A brief conclusion with our perspectives is provided in the last section.

2. Principle of Coherent Detection and Linear Impairments Equalization

The coherent receiver based on an intradyne system is shown in Figure 1a [11]. The input signal interferes with the local oscillator (LO) laser, which usually has the same frequency as the transmitter laser in the 90-degree optical hybrid device. Balanced detectors (BDs) are often used to reject the common mode noise. In order to detect both real and imaginary parts, the input signal is mixed with the real part of the LO in one arm and the imaginary part in the other arm through the 90-degree phase delay between the signal and the LO introduced by the 90-degree hybrid. Polarization diversity is introduced to ensure the detection of both polarizations of the signal. It is noted that the LO does not require the phase and polarization locking for the input signal. Then the electrical signal is digitalized using the ADCs with two samples per symbol, and then the DSP is further applied. Adaptive filtering is a subject of extensive research and many results have been reported [12]. Essentially, an equalizer of butterfly structure is preferred since adaptive algorithms must be applied to recover the polarization-division multiplexed (PDM) signals. The butterfly structured equalizer is shown in Figure 1b. Here we will first focus on the physical background and discuss the principle of coherent DSP for recovering the transmitted information. The details of algorithms will be explored after that.

Why can the electrical field of the detected signal be recovered, and what are the physical reasons behind this? The answer lies in the model of the optical fiber transmission system. In order to simplify the discussion, we firstly consider the SMF system and linear impairments, which can be solved based on linear fiber optics.

The SMF supports the propagation of two polarized lightwaves. Optical waves usually do not remain in the principle orientation when they are propagating through the optical fiber. In the long-haul SMF system, the polarization modes are strongly coupled. Signal propagation in each section can be modeled as a 2 × 2 matrix. When the fiber is longer than the correlation length, it can be modeled as a concatenation of multiple sections with independent characteristics as shown in Figure 2. The overall transmission matrix $H$ is the product of all independent matrices. No matter what the orientation of polarization beam splitters is, signals in two polarizations (EX_out and EY_out) cannot be distinguished due to the serious coupling between two degenerate modes. Therefore, the necessary condition for recovering the transmitted signals is that the transmission matrix has to be reversible (or unitary for the best case). The output electrical field can then be linked to the input electrical field by the matrix. The polarization state variation during transmission can be deduced through a unitary Jones matrix J. Regardless of the polarization dependent loss or signal attenuation (fully compensated through amplification), the overall transmission matrix H is definitely unitary and of course reversible. Compared with the CD, which can be regarded as constant, the channel transfer matrix varies with time, due to the rapid and random polarization coupling, so the adaptive DSP scheme has to be utilized. The equalizer with butterfly structure has been widely utilized. This type of equalizer can demultiplex the PDM signals with significant crosstalk and can also equalize linear impairments well. The reverse transfer matrix H⁻¹ is comprised of four elements. Each element includes a tap-weight vector based finite impulse response (FIR) filter. The length of adaptive digital filter should be equal to, or a bit longer than, the impulse response spread of the distorted signal. A variety of polarization demultiplexing algorithms have been proposed, such as constant modulus algorithm (CMA), least mean square (LMS), recursive least square (RLS), radius directed equalization (RDE) and so on [13,14]. The details of the CMA algorithm are described below, and the tap weights vector of the equalizer is adapted by the following:

$$\begin{array}{l}{h}_{xx}=h_{xx}+\mu {\epsilon }_{CMA,X}E{X}_{out}\cdot E{X}_{in}^{\ast }\\ h_{xy}=h_{xy}+\mu {\epsilon }_{CMA,X}E{X}_{out}\cdot E{Y}_{in}^{\ast }\\ h_{yx}=h_{yx}+\mu {\epsilon }_{CMA,Y}E{Y}_{out}\cdot E{X}_{in}^{\ast }\\ h_{yy}=h_{yy}+\mu {\epsilon }_{CMA,Y}E{Y}_{out}\cdot E{Y}_{in}^{\ast }\end{array}$$

(1)

$$\begin{array}{l}{\epsilon }_{CMA,X}=1-{\left|E{X}_{out}\right|}^2\\ {\epsilon }_{CMA,Y}=1-{\left|E{Y}_{out}\right|}^2\end{array}$$

(2)

where $\mu$ is the iteration factor, $\epsilon$ is the result of the error function. The “⋅” denotes the vector dot product. $E^{*}$ denotes for the complex conjugate form. All tap weights are typically set to zero initially, except for the central taps of $h_{xx}$ and $h_{yy}$ which are set to unity. The sample rate is twice the symbol rate while the filter tap weights are updated every two samples. The equalization outputs for the two polarizations are:

$$\begin{array}{l}E{X}_{out}=h_{xx}\cdot E{X}_{in}+h_{xy}\cdot E{Y}_{in}\\ E{Y}_{out}=h_{yx}\cdot E{X}_{in}+h_{yy}\cdot E{Y}_{in}\end{array}$$

(3)

Another widely employed algorithm is called LMS. LMS is a type of the stochastic gradient algorithms. Tap weights are determined by the update scheme as follows.

$$\begin{array}{l}{h}_{xx}=h_{xx}+\mu {\epsilon }_{LMS,X}E{X}_{in}^{\ast }\\ h_{xy}=h_{xy}+\mu {\epsilon }_{LMS,X}E{Y}_{in}^{\ast }\\ h_{yx}=h_{yx}+\mu {\epsilon }_{LMS,Y}E{X}_{in}^{\ast }\\ h_{yy}=h_{yy}+\mu {\epsilon }_{LMS,Y}E{Y}_{in}^{\ast }\end{array}$$

(4)

$$\begin{array}{l}{\epsilon }_{LMS,X}=R_X-E{X}_{out}\\ {\epsilon }_{LMS,Y}=R_Y-E{Y}_{out}\end{array}$$

(5)

$$\begin{array}{l}{R}_X=\exp (j{\theta }_x)d_x\\ R_Y=\exp (j{\theta }_y)d_y\end{array}$$

(6)

where $E{X}_{in}^{\ast }$, $E{Y}_{in}^{\ast }$ represent the complex conjugate of the sampled input signal vectors and $E{X}_{out}$, $E{Y}_{out}$ is the output of the equalizer. Its equalization error is defined by the difference between the reference signal R and the output signal, which includes both the amplitude and the phase information. R is the reference signal, $d_x$, $d_y$ are training signals or decision signals, ${\theta }_x$, ${\theta }_y$ are the estimated symbol phase after frequency offset compensation and carrier phase estimation (CPE). At the beginning, the equalizer works in the training symbol (TS) mode. Once the equalizer has converged, it moves into a decision-directed (DD) mode. This working scheme is defined as DD-LMS.

Before the final stage of restoring the data and applying forward error correction (FEC), specific algorithms are employed to execute the frequency offset compensation and CPE [15,16,17,18,19,20,21]. In theory, coherent detection requires the frequency and the phase of the LO wave to be exactly the same as those of the signal carrier. However, due to the influence of fabrication imperfection of optical devices and environment variation, the frequencies of the transmitter and the LO lasers would not be completely consistent. Besides, the linewidth of the lasers will also introduce the corresponding phase noise, which is always considered as a Wiener process. The large amount of additional phase noise is quite harmful to the phase modulated signal. The purpose of the carrier recovery algorithm in the coherent receiver is to remove the impairments of carrier frequency offset and phase noise by processing a discrete data sample sequence. The principle of the frequency offset estimation is shown in Figure 3. In the case of considering the symbol phase only, it is assumed that the sampling value of the k_th symbol received is:

$$S(k)=\exp \{j({\theta }_s(k)+\Delta \omega kT+{\theta }_L(k)+{\theta }_{ASE}(k))\}$$

(7)

where ${\theta }_s(k)$ represents the modulated phase, $\Delta \omega kT$ is additional phase caused by frequency offset, ${\theta }_L(k)$ is the phase noise from the laser linewidth, ${\theta }_{ASE}(k)$ is related to the amplified spontaneous emission (ASE) noise, T is the symbol period. In high-speed optical transmission systems, ${\theta }_L$ varies slowly relative to the symbol rate. By calculating the phase difference between the adjacent symbols, the ${\theta }_L$ can be removed.

$$S(k)S^{\ast }(k-1)=\exp \{j(\Delta {\theta }_s+\Delta \omega T+{\theta }_{ASE})\}$$

(8)

Take the quadrature-phase-shift-keying (QPSK) signal as an example. The QPSK signal has two types of modulated phase which are $(0,\frac{\pi }{2},\pi ,\frac{3\pi }{2})$ and $(\frac{\pi }{4},\frac{3\pi }{4},\frac{5\pi }{4},\frac{7\pi }{4})$, but the $\Delta {\theta }_s$ will always be $(0,\frac{\pi }{2},\pi ,\frac{3\pi }{2})$. The modulated phase will disappear by a power of four calculation:

$${(S(k)S^{\ast }(k-1))}^4=\exp \{j(4\Delta \omega T+4{\theta }_{ASE})\}$$

(9)

In high-speed optical transmission systems, the frequency shift is also a slow variation process relative to the symbol rate, so that the frequency offset corresponding to multiple consecutive symbols can be regarded as the same. In this case, a series of consecutive symbols in a block can be processed together to estimate the frequency offset. The effect of ${\theta }_{ASE}$ is quite small after the average operation and can be neglected at the optical SNR scenario. Then the frequency offset is obtained through:

$${\displaystyle \sum _N{(S(k)S^{\ast }(k-1))}^4}/N=\exp \{j(4\Delta \widehat{\omega }T)\}$$

(10)

$$\arg \{{\displaystyle \sum _N{(S(k)S^{\ast }(k-1))}^4}/N\}/4=\Delta \widehat{\omega }T$$

(11)

After the frequency offset estimation and compensation, carrier phase recovery is used to remove the phase noise component ${\theta }_L$ and $\Delta {\omega }^{\prime }KT$ caused by the linewidth and residual frequency offset of the lasers at the transceiver $\Delta {\omega }^{\prime }=\Delta \omega -\Delta \widehat{\omega }$. Viterbi–Viterbi phase estimation algorithm is a commonly used feed-forward digital CPE algorithm [22]. Figure 4 shows the principle diagram of the Viterbi–Viterbi CPE. The processing flow of the algorithm is similar to the aforementioned frequency offset estimation algorithm. The main steps are shown as follows:

$$\begin{array}{c}{S}^{\prime }(k)=\exp \{j({\theta }_s(k)+{\theta }_L(k)+\Delta {\omega }^{\prime }kT+{\theta }_{ASE})\}\\ =\exp \{j({\theta }_s(k)+{\theta }_L{}^{\prime }(k)+{\theta }_{ASE})\}\end{array}$$

(12)

where ${\theta }_L{}^{\prime }(k)$ is the carrier phase noise of the symbol k, ${\theta }_L{}^{\prime }(k)={\theta }_L(k)+\Delta {\omega }^{\prime }kT$

$${(S^{\prime }(k))}^4=\exp \{j(4{\theta }_L{}^{\prime }(k)+4{\theta }_{ASE})\}$$

(13)

${\theta }_L$ caused by the laser linewidth is the main part of the ${\theta }_L{}^{\prime }$. It changes slowly, in contrast to the high-speed symbol stream and can basically be regarded as stable in a continuous symbol block. M symbols are considered to eliminate the effect of ${\theta }_{ASE}$ to increase the accuracy of estimation by means of averaging symbols.

$${\displaystyle \sum _M{(S^{\prime }(k))}^4}/M=\exp \{j(4{\widehat{\theta }}_L{}^{\prime })\}$$

(14)

$$\arg ({\displaystyle \sum _M{(S^{\prime }(k))}^4}/M)/4={\widehat{\theta }}_L{}^{\prime }$$

(15)

Equation (15) is suitable for the $(0,\frac{\pi }{2},\pi ,\frac{3\pi }{2})$ QPSK modulated signal. The residual value of modulated phase $\pi$ will exist in the symbolic phase after the power of four operations if the $(\frac{\pi }{4},\frac{3\pi }{4},\frac{5\pi }{4},\frac{7\pi }{4})$ modulation is applied. Then Equation (15) will be expressed as:

$$(\arg ({\displaystyle \sum _M{(S^{\prime }(k))}^4}/M)-\pi )/4={\widehat{\theta }}_L{}^{\prime }$$

(16)

The carrier phase error calculated by M symbols will be shared with M symbols for phase compensation:

$$S^{″}(k)=S^{\prime }(k)\exp (-j{\widehat{\theta }}_L{}^{\prime })$$

(17)

Some typical achievements have been reported recently. Savory et al. demonstrated a 10 Gbaud PDM-QPSK experiment over 6400 km with DSP equalization [23]. The experimental results of equalized constellation diagrams of the two polarizations are shown in Figure 5. Zhou et al. proposed and demonstrated 400 Gbit/s experiments on a 50 GHz grid and successfully achieved 1200 km and 4000 km transmission [10]. Winzer et al. reported a 56 Gbaud, 224 Gbit/s PDM-QPSK 2500 km transmission experiment assisted by coherent detection with all linear impairments compensation. The experiment results of BER and transmission performance are shown in Figure 6a,b [24]. The authors of [25] developed 10-channel 28 Gbaud PDM 16-ary quadrature amplitude modulation (PDM-16QAM) signals transmitted over 1200 km.

3. Nonlinearity Compensation

Essentially, optical signals will suffer linear and nonlinear impairments during propagation along optical fibers, which limit the transmission capacity. When WDM is introduced to increase the capacity of optical fibers, the nonlinearity and dispersion can significantly affect the signal quality. Therefore, mitigating or compensating these impairments is an important part of optical fiber communication research. At present, the method of compensating linear impairments (including CD and PMD) is almost mature, so the nonlinear impairment becomes the limiting factor of increasing optical fiber system capacity. Enlarging the number of WDM channels or reducing the channel spacing will increase the nonlinear impairment. In addition, in order to use the higher-order modulation format to improve the SE and increase the transmission distance, it is necessary to improve the OSNR. Increasing the OSNR to a given level requires the increase of signal power, which in turn, results in more serious nonlinear impairments. Therefore, the reduction or compensation of nonlinear impairments can significantly increase the capacity of optical fiber channels.

Recent developments in coherent optical communication have enabled the complex electric field of signals to be obtained in the digital domain. Thus, it is possible to compensate for nonlinear impairments in the signal waveforms propagating along the optical fiber. Furthermore, as long as adjacent channels are received, compensation for nonlinear impairments among channels, such as cross-phase modulation (XPM) and four-wave mixing (FWM), is also possible. Among the proposed nonlinearity compensation technologies, the digital backward propagation (DBP) has proved to be the most promising approach. The recent proposed investigation has demonstrated that the nonlinear Schrödinger equation (NLSE) can be solved in the digital domain to compensate all deterministic fiber transmission impairments in a PDM WDM environment. The DBP method is based on solving the nonlinear propagation equation along the backward direction. However, the computational load of DBP is heavy. It is still very important to investigate computational efficiently nonlinearity compensation algorithms for dispersion-unmanaged coherent transmission systems. Li et al. have demonstrated approaches to reduce computational load including implementing DBP using coupled NLSEs [26,27], XPM walk-off factorization [28] and folded DBP for dispersion-managed systems, which have the potential to reduce computational load to the same order of magnitude for dispersion compensation for a dispersion unmanaged system [29].

The schematic of DBP method is shown in Figure 7. A is the complex electric field of the received signal, $\widehat{D}$ and $\widehat{N}$ are the linear and the nonlinear operators, respectively. The signal is distorted in the real transmission fiber and is compensated through the virtual fiber. The received signal is processed by a digital model with opposite propagation parameters after it is detected at the receiver after transmission. Backward propagated NLSE can be expressed as [1]:

$$\frac{\partial A}{\partial (-z)}=(\widehat{D}+\widehat{N})A$$

(18)

$$\widehat{D}=-j\frac{{\beta }_2}{2}\frac{{\partial }^2}{\partial t^2}+\frac{{\beta }_3}{6}\frac{{\partial }^3}{\partial t^3}-\frac{\alpha }{2}$$

(19)

$$\widehat{N}=j\gamma {\left|A\right|}^2$$

(20)

where $\alpha$, ${\beta }_2$, ${\beta }_3$ and $\gamma$ are the attenuation, first- and second-order group velocity dispersion and fiber nonlinear coefficients, respectively.

The performance of DBP algorithm mainly depends on the accurate estimation of NLSE propagation parameters. NLSE is usually solved by split-step Fourier method (SSFM). The optical fiber is divided into N segments, and each section has to be small enough to guarantee the optical power not to change much within the segment, since the nonlinear effects are power dependent. Another role is to ensure that the accumulation of dispersion and nonlinear effects is not so strong so as to cause serious interactions. Then the dispersion and the nonlinear effects can be considered independently in each small section, and can be executed successively (as an approximation to the simultaneous effect). In SSFM, the linear part and the nonlinear part are solved separately. The linear part is solved in frequency domain while the nonlinear part is solved in the time domain. Using backward propagated NLSE equation, all deterministic impairments (linear and nonlinear) in optical fiber transmission can be compensated. Compensation can be performed at the transmitter before signal transmission (pre-compensation) or at the receiver after signal detection (post-compensation). In most cases, DBP is implemented in conjunction with coherent receivers as a post-compensation. In the absence of noise in the transmission link, the two DBP schemes are equivalent. Since the backward propagation operator acts on the complex envelope of the electrical field, the DBP algorithm can be applied to any modulation format in principle. The performance of DBP is limited by ASE noise, because it is a stochastic noise source and cannot be compensated. DBP can only compensate for deterministic signal distortions.

The team from the University of Central Florida has studied the issue of nonlinearity compensation in WDM systems [26,27,28,30,31]. Nonlinear effects in optical fibers include self-phase modulation (SPM), XPM and FWM. Three nonlinear effects in optical fibers are separated using the coupling equation, and the system performance can be optimized by neglecting the items that have little influence on equalization results, but that require significant computation load. Compared with SPM and XPM, the computation load of FWM effect is much heavier. However, FWM needs phase matching to have a significant impact and the FWM effect can be neglected in non-zero dispersion shift optical fiber transmission system because it is far from the phase matching point. In this type of system, FWM items can be omitted and only the effects of SPM and XPM need to be compensated and the computational burden is reduced [26]. The simulation results are shown in Figure 8, Figure 9 and Figure 10. Figure 9 and Figure 10 show that XPM and FWM compensation produces the same Q value in 100 GHz channel spacing, and the influence of FWM is negligible. Larger channel spacing leads to a higher phase mismatch, and the FWM will contribute even less.

But in the low dispersion region, FWM with phase matching condition will affect the performance of the transmitted signal significantly. There is a trade-off between the computational cost and the compensation effect for the FWM mitigation. The influence of the nearest channel and the second nearest channel has to be considered, while the influence of the far channel could be neglected. Using this approximation, the Q value of the observed channel can be increased with a small amount of computation [27]. The interchannel nonlinear effect XPM can be reduced by the walk-off effect caused by dispersion. The step size of SSFM can be greatly increased and the computation cost can be reduced by separating walk-off effect [28]. As shown in Figure 11, the performance of DSP improves dramatically for the XPM compensation. The step size requirements for XPM compensation using the conventional approach, and the approach by the authors of [28], are shown in Figure 12. It indicates that the step size can be increased substantially based on the SSFM. For the PDM WDM system, it can be treated by solving coupled Manakov equations, ignoring the FWM item with high computational requirements, and only compensating the XPM effect [30,31].

DBP has been experimentally studied in a series of recent works from University College London. Makovejs et al. have studied the performance improvement of single channel PDM-16QAM 112 Gbit/s system after nonlinear compensation. It is found that the maximum transmission distance can be increased from 1440 km (EDC only) to 2400 km (DBP) [32]. Savory et al. experimentally studied 112 Gbit/s non-return to zero (NRZ) PDM-QPSK WDM system. It was found that using intrachannel nonlinear compensation, the maximum transmission distance of a single channel system was increased by 46% while that of a WDM system with 100 GHz channel spacing was increased by 23% respectively [33]. Millar et al. studied 10.7 Gbaud PDM-QPSK and PDM-16QAM systems and reported that the transmission reaches have been extended to 7780 km and 1600 km, respectively [34]. Behrens et al. studied a 224 Gbit/s PDM-16QAM system and proved that the maximum transmission distance can be increased by 18–25% with the DBP implemented [35].

A 112 Gbit/s PDM-QPSK system has been studied and proposed by University College Cork in 2011. The simplification method revealed the correlation effect of adjacent symbols and realized a remarkable reduction of computation cost [36]. Similarly, the use of a correlated backpropagation (CBP) for nonlinearity compensation has been reported in a 112 Gbit/s PDM-QPSK system [37], and the step size of DBP can be increased by 70%. Dou et al. have proposed a low-complexity predistortion method for intrachannel nonlinearity compensation and have experimentally verified this approach in a 43 Gbit/s PDM-QPSK coherent system [38]. Other groups also made impressive contributions to this area, and more details can be found in Refs. [39,40,41,42].

4. SDM and Frequency Domain Equalization (FDE)

SDM can increase the system capacity and exceed the nonlinear Shannon limit per fiber. Many efforts have been made in this area and important results have been presented. The maximum number of multiplexing has reached 72 and the capacity has reached beyond 10 Pbit/s. The authors of [43] reported a 10 × 112 Gb/s PDM-QPSK transmission in FMF over 5032 km as quasi-signal mode (QSM) application, after which the reach is extended to 7326 km [44]. Zhao et al. deduced the theoretical analysis result about the minimum number of spans required for a 3000 km QSM system [45]. More demonstrations such as transmission distance reach 14,350 km by using 12 core fiber were also reported [46]. E. Ip et al. demonstrated a WDM and MDM transmission with 146 channels and 3 linear polarized (LP) modes over 500 km. The experimental setup is shown in Figure 13 [47]. Another experiment by Ryf et al. demonstrated a 12 × 12 MIMO transmission over 130 km FMF with 8 wavelengths [48]. Fontaine et al. demonstrated a 30 × 30 MIMO transmission over 15 spatial modes [49] and Ryf et al. demonstrated a 72 × 72 MIMO DSP experimental result which is the current record of multiplexing numbers [50]. The SDM system capacity reached 1 Pbit/s in 2013 [51], and exceeded 2 Pbit/s two years later [52]. In 2017, an over 10 Pbit/s SDM transmission was demonstrated by Soma et al. which made a new milestone to fiber transmission field [53].

The mode coupling and core coupling cause serious signal distortions so the MIMO DSP also needed to be applied to recover the information on each mode or fiber core. In the SDM system, the inverse transmission matrix is extended from 2 × 2 to D × D dimension where D is the overall coupling degree of freedom in the SDM system. The computational complexity of FIR filters increases with the number of delay taps. The transmitted signal suffers a large amount of DMGD and will continuously accumulate with the transmission distance. Large DMGD requires the digital filter with a greater number of taps, so the computational complexity will also increase dramatically. To reduce the complexity of the equalization algorithm, FDE is applied by employing discrete Fourier transform (DFT) and complex multiplications in a block-by-block way instead of convolutions in the time-domain filters [54,55]. Theoretically, the change rate of channel parameters should be much slower than the packet rate to ensure the equalization performance. Optical fiber communication systems are generally slow-varying so this condition can be easily satisfied.

The evaluation criterion for the equalization algorithm is the number of complex multiplications per symbol per mode. The computational cost of TDE scales linearly with the number of taps, while the FDE scales logarithmically with the filter length. The complexity excluding carrier recovery for these two types of algorithms can be expressed as:

$$C_{TDE}=3m\Delta \tau L{R}_s$$

(21)

$$C_{FDE}=\left(4+4m\right){\log }_2\left(2\Delta \tau L{R}_s\right)+8m$$

(22)

where $\Delta \tau$ is the DMGD of the fiber, L is the link distance, $R_s$ is the symbol rate, m can be considered as the number of degree of freedoms utilized for transmission. The algorithmic complexities of both FDE and TDE as a function of filter length are shown in Figure 14. More detailed explanations can be found in Ref. [56].

As mentioned above, the computational complexity is determined by the degree of freedom and the differential mode group delay (DMGD). It is necessary to study and reduce the computational complexity of SDM applications. Several works have been reported to address this issue [57,58,59,60]. Compared with the TDE method, the FDE method can significantly reduce the computational complexity while maintaining the same performance. Some experimental results have been presented for SDM transmission, achieved by FDE. We have experimentally demonstrated an RLS FDE approach for MDM with good performance. The block diagram of the proposed algorithm and experimental results are shown in Figure 15 and Figure 16 [60].

5. AI for Optical Fiber Communication

Optical communication networks and systems have started to use artificial intelligence (AI) technologies to improve the performance, from equipment to control and management. Machine learning (ML), as a main branch of AI, provides many powerful models for optical communication. One of the main applications of ML technology is to perform optimal classification [61,62,63]. Demodulation of signals in point-to-point optical communication systems is crucial for achieving high-capacity transmission performance and is normally affected by nonlinear impairments and their interactions along the propagation. Classic demodulation techniques (e.g., maximum likelihood) cannot cope with high dimensional nonlinearities and channel dynamicity. Therefore, recent advances in ML and deep learning (ANNs, DNNs and CNNs) gives the potential to solve these challenges. The optimal classification can thus be directly applied for an optimal symbol detection in optical communication systems. For a memoryless nonlinearity, such as nonlinear phase noise, I/Q modulator and drive electronics nonlinearity, Euclidean distance measurement results in that the linear decision boundary is no longer optimal. For these special cases, ML techniques can be used to obtain the optimal symbol detection and decision boundary. Then DSP algorithm is used to detect the optimal symbols. ML has also been applied in many other applications such as optical performance monitoring, parameters estimation and nonlinearity mitigation [64,65,66,67,68]. A typical block diagram of ML for symbol classification in optical communication systems is illustrated in Figure 17.

An example is implemented here to show the performance of DNN on mitigating the inter-carrier interference (ICI) in the non-orthogonal multicarrier optical fiber transmission systems with sub-carrier spacing below the symbol rate. The spectrally efficient frequency division multiplexing (SEFDM) technique is employed in such non-orthogonal multicarrier transmission systems [69,70]. The SEFDM transmission occupies a sub-carrier spacing below the symbol rate at the expense of a loss of orthogonality to enhance the achievable spectral efficiency. The challenge in the SEFDM system is the inter-carrier interference (ICI) during the bandwidth compression [71]. As illustrated in Figure 18, five types of neural networks are designed for suppressing the ICI of non-orthogonal signals [72], including no connection-neural network (N-NN), partial connection-neural network (P-NN), hybrid1-neural network (H1-NN), hybrid2-neural network (H2-NN), full connection-neural network (F-NN). These NN architectures represent different connectivity between the input, output and hidden cells in the neural networks. The complexity also depends on the NN architecture. The N-NN has the lowest computational complexity since all the input cells are independent and can be operated in parallel. The F-NN has the highest computational complexity since all neurons are connected and each connection link requires arithmetical computation.

The employed deep neural networks follow the architectures in Figure 18, and the Sigmoid function is applied as the activation function at each layer. The neural network is trained off-line using a large amount of simulated random symbols (~40,000 QPSK) in the optical SEFDM system. The number of sub-carriers in this case is set to 4 and the bandwidth compression factor is set to α = 0.8. Please see Ref. [72] for more detailed information regarding the transmission system.

Simulation results are shown in Figure 19 to evaluate the effectiveness of NNs on SEFDM signal recovery. Their performance is compared with traditional SEFDM signal detection (hard decision). Figure 19a indicates the B2B scenario and Figure 19b indicates the scheme of 80 km transmission. Similar behaviors can be found in both scenarios. The N-DNN shows a performance close to that of the hard-decision detection, as each input cell is independently processed. Although there are sufficient neurons and hidden layers to improve the training accuracy, the independent processing architecture results in similar performance as the hard decision. The P-DNN provide a better performance since two input cells are connected and jointly trained, where more accurate interference features can be extracted from the network. The H1-DNN network shows further improved performance since the middle input cells are connected to neighboring input cells and therefore accurate interference emulation for the middle cells can be realized. As an evolved version, H2-DNN slightly outperforms the H1-DNN, where the interference to the edge input cell can be properly modelled through the neighboring neuron connection. The F-DNN provides the best performance the since the complete interference to each sub-carrier is estimated with additional neuron connections. Therefore, the complete interference information can be extracted from the SEFDM signal and an accurate model can be trained.

6. Conclusions

Coherent optical fiber communication has become an important field with the development of DSP techniques in both hardware and software. Fiber linear impairments can be compensated using static and adaptive filters while fiber nonlinearities can be compensated using the DBP method. SDM is considered as one of the most effective approaches to overcome the capacity crunch. The capacity of SDM system has been increased significantly with the improvement of novel DSP technologies. With the continuous reduction of the system cost and power consumption, DSP based coherent optical communication systems have been deployed widely. AI has been employed as an effective DSP tool in the coherent detection, however it still stays in the initial phase. Future research is greatly required to investigate the true capability and powerful function of ML in optical communications and other areas of optoelectronics. This paper has been focused on the applications of DSP in coherent optical systems. However, there are also applications of advanced DSP algorithms in other promising systems such as direct-detection systems which could offer lower-cost implementations of optical communications. This will be investigated in our future work.