Image Processing Approach for Grading IVF Blastocyst: A State-of-the-Art Review and Future Perspective of Deep Learning-Based Models

Abstract

The development of intelligence-based methods and application systems has expanded for the use of quality blastocyst selection in in vitro fertilization (IVF). Significant models on assisted reproductive technology (ART) have been discovered, including ones that process morphological image approaches and extract attributes of blastocyst quality. In this study, (1) the state-of-the-art in ART is established using an automated deep learning approach, applications for grading blastocysts in IVF, and related image processing techniques. (2) Thirty final publications in IVF and deep learning were found by an extensive literature search from databases using several relevant sets of keywords based on papers published in full-text English articles between 2012 and 2022. This scoping review sparks fresh thought in deep learning-based automated blastocyst grading. (3) This scoping review introduces a novel notion in the realm of automated blastocyst grading utilizing deep learning applications, showing that these automated methods can frequently match or even outperform skilled embryologists in particular deep learning tasks. This review adds to our understanding of the procedure for selecting embryos that are suitable for implantation and offers important data for the creation of an automated computer-based system for grading blastocysts that applies deep learning.

1. Introduction

Infertility is defined as a failure to achieve a pregnancy within a year or more of regular unprotected sexual intercourse [1] due to several reasons, including male or female reproductive systems [2,3,4] and other related disease [5]. Infertility is a global issue, especially in high sociodemographic countries [6], and therefore the use of assisted reproductive technology (ART) by infertile couples has increased by 5% to 10% per year [3]. The most common infertility treatment is in vitro fertilization (IVF) as an alternative to human reproduction that has extensively evolved into the cultivation of human embryos used in embryological laboratories.

IVF is a fertilization procedure in which a mature egg (oocyte) and sperm are combined in vitro (in glass) in a specialized laboratory. The fertilized egg (embryo) is allowed to grow in a protected environment up to seven days [7] before being transferred into the woman’s uterus, hence increasing the chance that a pregnancy will occur. However, different IVF techniques may result in different types of embryo selection that are of questionable quality, particularly at the blastocyst phase of development.

Several articles have been published that propose computer-based methods for IVF applications such as computer vision technology [8], image processing techniques [9,10], and emerging technologies [11]; however, a scoping review with a specific focus on blastocyst-stage quality grading has yet to be conducted. This article aims to provide a state-of-the-art review of the current techniques for blastocyst assessment, with a focus on intelligence-based approaches of distinguishing high-quality embryos based on recent advances in CMA and TL imaging of IVF technology. This review is not limited to human blastocysts, but also includes bovine and murine blastocysts for extensive comparisons. As the selection of good-quality embryos has undoubtedly improved the successful outcomes in IVF, intelligence-based technologies and methods are providing new insights into the traditional approach of morphometric assessment, especially at the blastocyst stage.

2. IVF Culture, Embryo Development, and Selection

Currently, the most commonly used methods of IVF fertilization are conventional morphological assessment (CMA) and embryonic morpho-kinetics using time-lapse (TL) platforms [12]. Both methods similarly make use of embryo assessment and selection, although the TL tool differs in that the embryos are cultured in incubators with built-in microscopes to automatically obtain images at a time interval at a certain focus and magnification, whilst CMA visually monitors on a daily basis. In advanced stages, TL technology provides continuous monitoring of the dynamic development event without disturbing the culture environment and provides reviewable and stable video data for embryo selection [13,14]. Although TL has several potential benefits compared with standard CMA [15], it remains to be elucidated whether the increased precision of embryo evaluation by TL monitoring improves pregnancy rates, since larger randomized clinical studies are needed in future comparative studies [14].

2.1. Stages of Embryo Development in IVF

In the IVF process, the quality of fertilized eggs is highly important as the selection of a good fertilized egg could ensure a successful pregnancy rate [9,16] before the embryo is transferred into the uterus. The fertilized egg develops into a human embryo (zygote) and can be presented in four stages, namely, a pronucleate oocyte (pronuclear), cleavage stage, morula, and blastocyst, as presented in

Table 1. Embryo stages and morphological features in IVF process from pronuclear to blastocyst stage of development at each time point.

Stage	Timing (h) [17]	Expected Features Developed [17]	Ideal Morphology Features/Visibility [18]
Pronuclear (Day 0)	17 ± 1	Pronucleate oocyte	(i) NPB 1 < 3 (ii) NPB always polarized
Cleavage (Day 1)	23 ± 1 to 26 ± 1 (post ICSI 2), 28 ± 1 (post IVF)	Up to 20% may be at or reach the two-cell stage	(i) Mononucleated blastomeres (ii) Equal cell sizes (iii) <20% fragmentation
Cleavage (Day 2)	44 ± 1	Four-cell stage	(i) Mononucleated blastomeres (ii) Equal cell sizes (iii) <20% fragmentation
Cleavage (Day 3)	68 ± 1	Eight-cell stage	(i) Mononucleated blastomeres (ii) Equal cell sizes (iii) <20% fragmentation iv) at least seven blastomeres
Morula (Day 4)	92 ± 2	Compaction volume	(i) Compacted cells (by increase in embryo and ZP space) (ii) Lack of fragments
Blastocyst (Day 5/6/7)	116 ± 2	Fully expanded, through-to-hatched	(i) Expanded blastocoel cavity (ii) Composed of many inner cells mass (iii) Cohesive epithelium cells at TE (trophectoderm) (iv) Zona Pellucida (ZP) thinning

¹ NPB = nucleolar precursor bodies, ² ICSI = intracytoplasmic sperm injection.

, with respect to the hours post insemination and its expected features of each stage of development.

Beside the developed embryo of day 0 to day 3, the morula has been least characterized for embryo grading due to the difficulty in assessing the evidence of compaction in the fertilized embryo and the less well-defined blastomere boundaries [17,19]. Due to the ideal morphological features, blastocysts at day 5 are reported to have higher implantation potential compared to other embryo stages [17]. Hence, morphology-based embryo selection to quantify a good and high-quality blastocyst prior to embryo grading has become more critical to improve the IVF success rate. Meanwhile, delayed blastocyst development at day 7 or 8 is considered a poor prognosis for implantation [20] and they are routinely discarded, since the proportion of top-quality blastocysts are lower at day 7 [7]. Consequently, there are many detailed assessment and grading schemes have been devised for grading the quality of embryos from the pronuclear stage [21], cleavage stage [22], morula stage [19], and blastocyst stage [23,24].

2.2. Embryo Selection and Implantation

In order to achieve successful pregnancy after IVF, the selection of embryos for uterus implantation has been crucially investigated from the earliest stage of development to distinguish the quality of the fertilized embryos [9,16], particularly at the cleavage and blastocyst stages [25]. However, in common practice, blastocysts at day 5 are preferred among embryologists, since this development stage was assessed for evidence of compaction and there was a good prognosis for blastocyst development [26,27,28]. Although there is consensus about the type of embryo transferred, i.e., embryos in which more than 50% of the blastomeres have large vacuoles should not be transferred or frozen as their implantation rate is practically zero [29], the selection method of the developing embryo in either invasive or non-invasive IVF is significantly related to the embryos’ quality. Non-invasive techniques such as time-lapse microscopy represent new methodologies to analyze embryos in stable culture conditions without the risk of sample contamination or sample error [30]. Additionally, non-invasive selection methods appear to be a better strategy for the identification of potential embryos without the risk of possible impacts due to the investigation itself [31].

While not all embryos can develop to the level of a blastocyst prior to insemination, the embryos tend to have a higher success rate if the blastocyst grade is good quality [32]. In fact, the ability of the embryo to become a blastocyst is determined by the blastulation rate, in which a higher rate is found for the blastocyst stage than the cleavage stage [17] and is significantly lower for the pronuclear stage [21]. Therefore, if an embryo’s blastulation rate is low and it is progressing slowly, it may be an early indication that something is amiss with the environment in which the embryo is growing. Hence, the embryologist may require a postmortem or “troubleshoot” to identify the cause.

3. Method

3.1. Search Strategy

Since the grading of embryo quality in IVF technology has significantly developed over the past ten years, this review is conducted by searching articles published between 2012 and 2022. Using a set of keywords, the study searched the research area of automated embryo segmentation and classification in three academic journal databases: Web of Sciences (WoS), Scopus, and PubMed. In order to find related contributions, search engines were queried for papers published on or after 2012 that contained specific key-phrases (“automated classification and segmentation” OR “blastocyst classification and segmentation”) with or without (“IVF”) in their titles or abstracts. Articles that do not primarily focus on the segmentation of blastocyst or IVF-related medical anatomy were excluded. Each paper was reviewed and agreed upon by at least one researcher before inclusion. After searching, there were 518 articles found from the search engines, and we shortlisted 368 of them following the criteria as presented in Figure 1:

3.2. Study Selection

After full-text screening for their relevance to the topic, this review included 77 of them into this study. The distribution of recent studies from 2012 to 2022 related to the use of machine learning techniques in the study of IVF grading quality is shown in Figure 2. The last update to the included papers was on August 2022. All accepted review articles were categorized based on their grading approaches, namely as classic grading, semi-automated grading, and fully-automated grading. Finally, there were 30 full articles screening that included deep learning approaches and its future perspective towards automatized grading in the IVF blastocyst embryo.

4. Morphology of the Blastocyst and Quality Assessment

The post insemination of fertilized oocyte after Day 5 is known as a blastocyst, and its structure is shown in Figure 3. A blastocyst is characterized by the formation of a fluid-filled cavity (blastocoel) that occupies the center of the embryo, which is surrounded by a single layer of cells (the trophectoderm (TE)) which will form most extraembryonic tissues such as the placenta [33]. Additionally, a small protuberance of cells known as embryoblast or inner cell mass (ICM, from which the fetal tissues develop, may also be visible. The TE and ICM morphological structure differentiation was primarily agreed amongst embryologist to represent the progression of the developed blastocyst [25,34].

Generally, the blastocyst expansion can be developed into many different shapes and patterns towards the sizes of expansion, ICM cells and TE development [35]. Occasionally, the features such as the zona pellucida (ZP) thickness [36] and blastocoel [35] were mostly being considered in grading the quality of blastocyst. Specifically, the embryos with blastocoel cavity formation were considered as blastocysts; otherwise, embryos were regarded as non−blastocysts (undeveloped blastocysts) [28]. Currently, most of the embryo grading systems for assessing the viability of IVF embryos are very subjective and fully rely on visual inspection of morphological characteristics of the embryos, i.e., the appearance of embryo by qualitative evaluation [10,34]. In extension, the key morphological features of relevance to embryo viability have been highlighted as [10]:

• Cell number and degree of symmetry: if all cells are similar in size and an appropriate number of cells are present, this indicates that the embryo has a good chance of being viable.
• Fragmentation of cells: a low proportion of embryo volume composed of cell fragments is an indicator of high viability, while an embryo containing many fragmented cells is considered to have reduced potential.
• Characteristics of the zona pellucida (ZP): embryos with a thinner ZP and higher variation in ZP thickness have a greater likelihood of producing a pregnancy.

4.1. Blastocyst Visual Assessment

Generally, the degree of blastocyst expansion in embryo development stage can be characterized into several categories, and this depends on the grading scheme. For example, the numerical scoring system presented by the Istanbul consensus [20], shows many studies have assessed the blastocyst stage according to the morphological appearance of blastocoel cavity, ZP thickness, ICM, and TE cells [35]. However, Gardner et al. [37] (who created Gardner’s grading system) have suggested that the grading of blastocysts mainly focuses on three morphological appearances of the embryo, i.e., the rate of embryo size expansion, the analysis of ICM, and the analysis of TE cells. In assessing the fertilized embryo, the degree of blastocyst expansion presents the capability of the cultured embryo to progressively enlarge the volume up to the hatching process [35]. Meanwhile, the ICM cell will turn to be a zygote or fetus [16,26,35] and therefore it is crucial to carefully grading the quality of ICM cell throughout the IVF selection procedure. In addition, the TE score is also one of the most important parameters to assess concerning embryo selection especially among older women (≥35 years of age) [30] to ensure high successful rate in live birth.

The Gardner’s grading system [38,39] evaluated blastocysts based on their morphology, i.e., the shape, size, and appearance, and has been used as a reference by many IVF laboratories as a tool to help embryologists measuring blastocyst growth and the quality of developed embryo. The Gardner grading system sequentially assigns three different quality scores of each embryo based on blastocyst development stage (expansion and hatching status), ICM score, and TE score, or quality. Based on these three components (degree of expansion, ICM, and TE), blastocysts are given a quality grade of each, and the score is expressed with the expansion grade listed first, the ICM grade listed second and the TE grade third. As an example, the grading of blastocyst quality based on Gardner scoring system has widely been used by many researchers and practitioners as presented in Figure 4. The embryo scoring, being performed mainly via visual evaluation based on embryologist experience, may often result in different interpretations of embryo quality [34]. The ability of embryologists to score embryo morphology correctly with minimum subjectivity and with high intra and inter-observer agreement is dependent upon competence (i.e., skills and training), accuracy and consistency. The interpretation skills could be achieved through constant education, training and validation of operator competency, which should therefore be a priority during continuous training of embryologists [40]. Therefore, a standardized scoring for embryo quality is important for classification and selection task prior to embryo implantation process. As an example, the addition of viability markers has been proposed to increase the possibility of standardized measurements in embryo morphology scoring [40], and further evaluation on blastocyst quality using quantitative methods has been suggested to quantify morphologic parameters [27].

Traditionally, morphology-based grading has been approached in attempt to evaluate and select good quality embryos for transfer. Other approaches of preimplantation genetic testing ((PGT)-based method [42]) have been proposed and are becoming popular as principal modalities for embryo selection. Furthermore, the technological advancements in biomedical and bioinformatics fields sees a significant effort focused into developing techniques for analyzing microscopic embryo images particularly at the blastocyst stage.

4.2. Blastocyst Grading Approaches in IVF

Since a decade ago, a variety of blastocyst grading systems have been introduced to help embryologists select the best quality embryo for implantation [34]. As presented in Table 2, a manual grading approach is exclusively based on morphological parameters by inter and intra-observer [29,34] and may require additional tests to validate the embryo grades for implantation [21,25]. Meanwhile, the semi-automated approach adopted more implementation on image analysis based on image algorithms and statistical methods [27,43] for measuring the blastocyst features of ICM, TE, and ZP. As opposed to manual and semi-automated approaches, the fully automated grading systems based on AI technologies have increased. This phenomenon demonstrated how significant AI-based automated methods are to streamlining and improving the IVF process: they are reducing costs while also increasing embryologists’ capability to analyze the embryo most likely to develop and result in a live birth. [10].

4.2.1. Manual/Traditional

Manual grading was first initiated using visual evaluation [37] by inter and intra-observation based on blastocyst morphological appearance. Prospectively, manual annotation has been found effective for certain measures, i.e., cell numbers, fragmentation, and symmetry in a non-invasive embryo selection process [44]. However, there is a tendency to biased results in manual grading system amongst inter and intra-rater decision [12], and this method is highly prone to subjectivity [31] in judgement of embryo quality. To overcome this problem, more than one rater may need to evaluate a single embryo as proven by [45]. The study has shown that high degree of inter-embryologist and intra-embryologist variability in scoring embryos, likely due to the subjective nature of traditional morphology grading, and this may ultimately lead to less precise disposition decisions and the discarding of viable embryos.

4.2.2. Semi-Automated

Alternatively, a semi-automated system has been practically integrated to the manual grading system to generalize the final decision. Likely, semiautomated grading systems usually focused on extracting specific morphological features of the blastocyst, i.e., segmentation, feature extraction (FE), or classification of blastocyst expansion size, ICM, and TE [27,43] to standardize the grading methods. In addition, the evaluation of blastocyst morphology with digital image systems has been found useful particularly for the embryo selection task since the quantitative parameters’ evaluation is more reliable. As an example, the findings of a semi-automated system suggested that the blastocyst with area ≥ 18,500 μm², with an elevated number of TE cells (25.6 ± 11.3 μm²), and a large well-defined ICM area (3122.7 ± 739.0 μm²) has been highly considered for the transferring task [27].

In fact, blastocyst grading of TE and ICM are two important features that are highly evaluated because the metabolism of the blastocyst occurs in these two different places, i.e., in TE cells where glucose consumption occurs and half is converted to lactate, whereas glycolysis occurs in ICM [46]. Hence, these ICM and TE will turn into a fetus and a placenta, respectively. However, several limitations have been highlighted in semi-automated approaches for blastocyst grading, mainly on the need of user intervention [43] and sample size issue [27,47] i.e., a larger training set may require providing a more robust classifier.

4.2.3. Fully-Automated

To reduce the subjectivity involved in the embryo grading process and to make a more objective classification, the use of digital image processing and artificial intelligence (AI) techniques have seen significant. Therefore, the use of fully automated systems would be desirable to the grading process and efficient for large data sets collection in a fast and precise approach. Moreover, quantitative measurement of biomarkers in automated-based system will improve objective scoring of viability, make it easier to standardize protocols, and reduce intra and inter-clinic variation [40]. On the other hand, an automated-based system should be applicable as a standardized ART tool that generalizes different medical images, while at the same time being robust enough for automated classification tasks as proposed by [48]. Furthermore, the extension of a fully automated system with new emerging technologies such as aneuploid screening [42], metabolomic profiling, and time-lapse imaging analysis provides promising results in embryo assessment [11].

4.2.4. Machine Learning-Based Methods

Advances in image processing not only help experts classifying blastocyst quality accurately [49], but also help in other applications such as segregating overlapping instances in microscopy images [50], developing annotation software [51], validating image analysis systems for blastocyst selection [52], standardizing embryo quality [40], predicting blastocyst development [53], etc. Furthermore, the emerging in AI with the application of image processing has seen several ART tools that have been developed to assist an embryologist in grading quality embryos such as ERICA [54], iDAScore v1.0 [55], CellProfiler 3.0 [56] and Blast-Net [57]. The capability of image processing methods to extract useful information from microscopic images such as texture, gray levels, and light levels [58] of blastocyst morphological parameters, are extensively significant for classifying specific blastocyst morphological features such as ZP [59,60], ICM [61,62] and TE [63]. In fact, automatic segmentation and quantification of various components of a blastocyst image can potentially offer automation in morphology assessment as well as embryo selection with improved implantation and live birth outcomes [49]. As an example, image processing protocols have been used to automatically extract different variables (up to 33 variables) that represent different aspect of the embryo and its expanding blastocyst characteristics [58] to assist embryologists in embryo selection for ART. Moreover, newly developed image processing algorithms such as 3D [56] and 4D [64] views will add more features to the embryo analysis tool.

From the review, the blastocyst morphological segmentation and classification domains have been mostly dominated by semi-automated and fully-automated approaches. Most commonly, the segmentation and classification algorithms were embedded/fused in a deep learning-based model since the model architecture are containing features learning and classification stages [65]. Important information can be extracted from the regions of interest (ROI) in microscopic images by using a segmentation algorithm such as texture analysis [43,50,66,67]. Among the most efficient texture analysis is grey level co-occurrence matrix (GLCM) [58], watershed transform [67] and k-means [67]. With the use of texture descriptor, each pixel of microscopic image can be characterized accordingly by assigning levels of each pixel in the blastocyst components as depicted in Figure 5. This map helps machine learning on identifying a few threshold values in finding the ROI of each pixel. The outmost ellipsoidal boundary represents ZP’s inner boundary while R_i is the average of half major and minor axes of the inner ZP’s calculated for each embryo individually. The outer most ellipsoidal tube (width of 0.1 Ri, Ω₃) represents the region that most definitely includes TE cells, whilst the inner most ellipse (Ω₁) shows the region where ICM cells fully or partially reside in. The region that could include both TE and ICM is denoted as Ω₂.

The application of image processing that uses mathematical techniques or clustering structures [68] to detect morphological texture for classification of TE, ICM and ZP components has been proved by [43,59,67,69,70]. In order to distinguish different textures, statistical measures of pixel values based on local standard deviation, σ and Entropy are computed as expressed in Equations (1) and (2), respectively. The $\overline{x}$ refers to the mean pixel intensity over a neighborhood window of N (3 × 3 pixels) and P_i is the probability of the i-th pixel value in the image.

$$\sigma =\sqrt{\frac{{\displaystyle \sum _i{(x_i-\overline{x})}^2}}{(N-1)}}$$

(1)

$$Entropy=-{\displaystyle \sum _i{p}_i.{\log }_2(p_i)}$$

(2)

In addition, the single detection of a blastocyst component such as ZP [59,60], ICM [61,62], or TE [63] is easier to segment as compared to the detection of all-combined blastocyst components in an image that may need a deep learning model in the network [70,71]. As suggested by [30], the blastocyst grading for patients aged 35 years or above shall be performed using a strict grading policy in terms of TE, ICM and expansion grades together rather than using a single blastocyst parameter to choose the best combined-score blastocyst.

The adaptations of image processing algorithms in machine learning-based models for segmenting and classifying the blastocyst components in fertilized embryo have influence and impacts in grading the embryo quality. As an example, semantic segmentation of images provides an important cornerstone for subsequent tasks of image analysis and understanding by showing better comprehension of multiscale structural features of different blastocyst components [48]. Other novel automatic-based area segmentation by using lightness value was proposed [72] to enhance the quality of input images by deemphasizing blastocyst cavity regions and those areas outside the embryo. The work employed Retinex algorithm to remove small gradients inside and outside the blastocyst region so that cavity region is smoothed.

Meanwhile, the rapid progress in various image processing algorithms, and the associated developments in computer-aided systems classifier such as the decision tree [50,73], CNN [74], random forest [75,76], SVM [43,52], Naive Bayes classifier [52], neural network [53,69,77,78,79,80,81] and deep learning has propelled IVF grading into one of the most important sub-fields in assisted reproduction for humans, and for animals as well. Moreover, the combination of image processing methods and deep learning techniques has seen the development of a robust model such as Life Whisperer AI [82], which has been used as a clinical decision support tool for the prediction of embryo viability during IVF procedures. However, there exist several limitations of different machine learning algorithms in automated analysis based on computer vision technology as reviewed by [8,10,83]. Several advantages have been highlighted in the review, particularly in fully-automated machine learning (Table 2); however, limitations in the form of requirement for a massive dataset for training and high-end computing power are the main issues of those areas.

Table 2. This is a table. Tables should be placed in the main text near to the first time they are cited.

Method	Approach	Description
Conventional/manual	Simplified blastocyst grading system [30]	Inter and intra-observer (5 embryologists). Three classes of A, B, C, Cavitating and Compacting (based on blastocyst expansion, ICM and TE). Implantation rates 79.1% (grade A) and 13.25 (cavitating).
	Morphological embryo selection [21]	Used Gardner’s grading. Implemented chi-square test on embryo grades. Implantation rate 55.9%.
	Embryo scoring based on classification tree model [47]	Subdivided embryos into 4 categories (A, B, C, D). Combining proteomics analysis and TL to improve embryo selection.
	Addition of PN scoring [15]	Additional criterion to improve prediction of embryo implantation potential. Evaluated by chi-square test or Kruskal–Wallis test. Implantation success 95%.
	ASEBIR [10]	Scoring system for early embryos, morulae, and blastocysts. Exclusively based on morphological parameters.
Semi-Automated	Blastocyst segmentation with FE and Classification [40]	Using image algorithm of ZP, TE and ICM Classifier-statistical analysis. Accuracy 67%.
	Blastocyst classification [23]	Measure blastocyst internal area, ICM area, and TE cells number. Statistical analysis—Sigmaplot.
	Embryo growth classification [58]	Image intensity variance + circular Hough Transform. Embryo compaction and cavitation stage.
	Embryo vitrification [43]	Automation instrument (Gavi system).
	Embryo development stages [84]	Data extraction using localize variance based on image distribution.
	Vitrification procedure [85]	Using GAVITM system.
Fully Automated	Prediction of human embryo fitness [58]	Image processing and Segmentation of TE, blastocoel, and ICM. Accuracy 95–96%.
	Bovine blastocyst quality classification [80]	Combined GA and ANN. Classifying embryo quality grades (3 classes). Accuracy 76.4%.
	Blastocyst segmentation [78]	DCT transform and ANN. Precision 80% (ZP), 69% (TE) & 76% (ICM).
	Segmentation and measurement of human blastocyst TE region [72]	Level Set segmentation algorithm. Accuracy 87.8%.
	Segmentation of bovine embryos [86]	Unsupervised segmentation. Accuracy 96%.
	Localization of cleaving embryo (Day 2) [80]	Texture and geometrical features extraction.
	Grading quality of bovine blastocyst [79]	ANN + GA. 77.8% (smartphone) and 85.7% (Blasto3Q).
	Bovine quantitative variable determination using image processing [52]	Extraction of embryonic morphological information. Segmentation and texture analysis.
	Human blastocyst segmentation [67]	Segmentation and texture analysis. Accuracy 86.6% (TE) and 91.3% (ICM).
	Predict blastocyst development [53]	Based on cytoplasm movement velocity K-NN, LSTM-NN, hybrid ensemble classifier Accuracy 82.6%
	Improve blastocyst morphology [77]	ANNs. Mouse and bovine blastocyst.
	Predict implantation after blastocyst transfer [75]	Random Forest model. AUC 0.74.
	Analysis embryo quality: Preliminary study [64]	4D segmentation mouse embryo. HI value of healthy embryo.

Table 2. This is a table. Tables should be placed in the main text near to the first time they are cited.

Method	Approach	Description
Conventional/manual	Simplified blastocyst grading system [30]	Inter and intra-observer (5 embryologists). Three classes of A, B, C, Cavitating and Compacting (based on blastocyst expansion, ICM and TE). Implantation rates 79.1% (grade A) and 13.25 (cavitating).
	Morphological embryo selection [21]	Used Gardner’s grading. Implemented chi-square test on embryo grades. Implantation rate 55.9%.
	Embryo scoring based on classification tree model [47]	Subdivided embryos into 4 categories (A, B, C, D). Combining proteomics analysis and TL to improve embryo selection.
	Addition of PN scoring [15]	Additional criterion to improve prediction of embryo implantation potential. Evaluated by chi-square test or Kruskal–Wallis test. Implantation success 95%.
	ASEBIR [10]	Scoring system for early embryos, morulae, and blastocysts. Exclusively based on morphological parameters.
Semi-Automated	Blastocyst segmentation with FE and Classification [40]	Using image algorithm of ZP, TE and ICM Classifier-statistical analysis. Accuracy 67%.
	Blastocyst classification [23]	Measure blastocyst internal area, ICM area, and TE cells number. Statistical analysis—Sigmaplot.
	Embryo growth classification [58]	Image intensity variance + circular Hough Transform. Embryo compaction and cavitation stage.
	Embryo vitrification [43]	Automation instrument (Gavi system).
	Embryo development stages [84]	Data extraction using localize variance based on image distribution.
	Vitrification procedure [85]	Using GAVITM system.
Fully Automated	Prediction of human embryo fitness [58]	Image processing and Segmentation of TE, blastocoel, and ICM. Accuracy 95–96%.
	Bovine blastocyst quality classification [80]	Combined GA and ANN. Classifying embryo quality grades (3 classes). Accuracy 76.4%.
	Blastocyst segmentation [78]	DCT transform and ANN. Precision 80% (ZP), 69% (TE) & 76% (ICM).
	Segmentation and measurement of human blastocyst TE region [72]	Level Set segmentation algorithm. Accuracy 87.8%.
	Segmentation of bovine embryos [86]	Unsupervised segmentation. Accuracy 96%.
	Localization of cleaving embryo (Day 2) [80]	Texture and geometrical features extraction.
	Grading quality of bovine blastocyst [79]	ANN + GA. 77.8% (smartphone) and 85.7% (Blasto3Q).
	Bovine quantitative variable determination using image processing [52]	Extraction of embryonic morphological information. Segmentation and texture analysis.
	Human blastocyst segmentation [67]	Segmentation and texture analysis. Accuracy 86.6% (TE) and 91.3% (ICM).
	Predict blastocyst development [53]	Based on cytoplasm movement velocity K-NN, LSTM-NN, hybrid ensemble classifier Accuracy 82.6%
	Improve blastocyst morphology [77]	ANNs. Mouse and bovine blastocyst.
	Predict implantation after blastocyst transfer [75]	Random Forest model. AUC 0.74.
	Analysis embryo quality: Preliminary study [64]	4D segmentation mouse embryo. HI value of healthy embryo.

5. Blastocyst Grading Using Deep Learning-Based Methods

The application of deep learning approaches such as convolutional neural networks (DCNNs) can improve reliability and high consistency during the process of embryo selection and disposition, by potentially improving outcomes in an embryology laboratory [45,87]. The deep learning-based methods are constructed from convolutional neural networks (CNNs), and have gone a step further by automatically defining its feature extractor [8] as depicted in Figure 6, in which also is the input for classification task. Furthermore, the CNN model can be joined to other general deep networks as a multimodal discriminative model for learning non-image data as part of the automated embryo grading process.

5.1. Implementation of Deep Learning Models

One recent work implemented a deep learning-based system that was trained with a genetic algorithm (CNNg) and showed capability to differentiate between euploid blastocysts based on their capacity for implantation [88]. The research indicates that the artificial intelligence-based approaches have a high potential to improve success rates of IVF with a significantly better agreement with the actual implantation outcome for embryos with higher implantation scores. In addition, the less complex deep learning model such as Resnet was found feasible for grading cleavage embryos (at day 3) to achieve better performance accuracy [89]. However, this study implemented a relatively simpler dataset and had limited resource constrained settings, i.e., an image capturing process that resulted in different shades of colors and obstruction.

In more advanced models, the integration of deep learning models and other tools like Raman spectroscopy is applicable for predicting whether the cleavage stage embryo at day 3 has potential to develop into a blastocyst or not [89]. This combination in biological detection could help embryologists to provide a target research direction of RNA nuclei acid-related molecules for a follow-up biomarker study. Recent advances in IVF technology demands expensive equipment technologies such as intracytoplasmic sperm injection (ICSI), time-lapse imaging, and pre-implantation genetic screening to help fertility centers in improving clinical outcomes. Due to the need to set up laboratories and handle IVF processes, these technologies might not be available in the majority of underdeveloped nations, hence the suggestion to use portable and inexpensive optical systems employing smartphones, which gave adequate-quality images as a stand-alone imaging system [89]. Although the proposed work was potentially performed with high detection accuracy, the platform will require multiple imaging stations to provide morphological features as a TL imaging system.

To highlight the effectiveness of the deep learning-based methods, some of the state-of-the-art technology in segmentation and classification architectures were deployed for semantic segmentation of human blastocyst images [49,57,62,90]. These models were reimplemented, trained and tested on the various benchmark datasets and comparison results were summarized in Table 3. The current focus of DL applications in embryology can be categorized into the following groups: classification, prediction, segmentation, localization, and detection. Most DL models employed for classification, prediction and segmentation of blastocyst quality based on human blastocyst images were integrated with additional features such as the Grad-CAM algorithm for visualization purposes [91], the image refinement algorithm [59], discriminative features based on low-resolution images [92], and Raman spectroscopy to detect the metabolic spectrum of spent day 3 (D3) embryo culture medium [93].

In addition, the statistical models were also significant in the DL model integration as used in [32,54,73] for predicting the classes of blastocyst quality. However, most studies mainly highlighted the limitation of data resources and applications which contribute to the suitability and generosity for the model developments, i.e., specific only TL images, which are applicable only for ICM or TE and disregard the blastocyst expansion stage, particularly in blastocyst segmentation. Furthermore, a lack of image annotation has constrained technicians to predict the progress stage of blastocyst expansion [94] due to manual processes, and since then, the suggestion to automate the annotation of embryo development based on AI with high accuracy and reliability are essential [95,96]. Technically, the selection of TL and conventional blastocyst images are seen as highly important since none of the studies employed both imaging in a DL model training. Overall, the only study that employed a non-image signal for embryo risk assessment were using fetal heart rate (FHR) and uterine contraction (UC) [97].

Refer to Table 3: most of the studies were recommended to further improve the consistency by considering additional parameters in DL models or through additional training and tested efficacy in clinical setting. However, the embryo deselection or selection reliant on algorithms based solely on kinetic data, or the exclusive use of biomarkers, the morphological data is also crucial in the selection process [15,41]. For example, morpho-kinetic embryo analysis, which is the change of embryo morphology over time, is by far the most important noninvasive embryo selection tool today. It already can be applied to identify embryos likely to develop to the blastocyst stage and it may also have great potential when deciding which embryo to transfer in elective single-embryo transfer. If combined with developing technologies to assess physiological embryo properties such as metabolomics, these may constitute the backbone of every embryology laboratory’s workflow in the future [31].

Table 3. Comparisons studies associated with the blastocyst using deep learning models.

CNN Function/DL Model		Additional Feature/Model	Input/Training Data	Output Class	Limitations
Classification	VGG-16 [91]	Grad-CAM algorithm (for visualization)	Human blastocyst	Two blastocyst quality	Only suitable for TL systems (due to multiple focal depths as input).
	CNN [27,41,50,88,90,92,98]	- Image pre-processing - Xception model - ImageNet - GA - LTSM	Human blastocyst	Two, three and five blastocyst quality	- Performance and data dependency. - Omission of clinical KPIs (pregnancy rates) and analysis of other laboratory KPIs (fertilization rates). - Do not have features for monitoring as TL). - Required randomized control trials before routine use in clinical practice.
	ResNet [90,97]	- GA model	Human embryo	5 class (Grade 1 to 5)/9 risk factors	Issue in image capturing process and limited resources.
	DL (MLP classifier) [94]	Raman spectroscopy	Human embryo (Day 3 hpi)	2 class (blastula and non-blastula)	NA
Prediction	DNN [13,49,63]	- Random forest and logistic regression classifier - Google’s Inception V1) + decision tree	Human blastocyst—TL	2, 3 classes	- Low prediction accuracy. - Low sensitivity and system developed only for blastocysts (Day 5). - Trained algorithm cannot identify positive and negative life birth successfully using embryo morphology.
	CNN [99]	RNN (prediction)	Human blastocyst	Three blastocyst classes	Disregard blastocyst expansion stage (due to annotation).
	ResNet and DenseNet [24,75,83]	NA	Human embryo (Day 5)	2, 3, 5, classes	- Not incorporate information of different days of embryo development. - Not applicable on other TL/data. No clinical characteristics included in the study.
Segmentation	U-Net [55,57,59,61]	Semantic segmentation	Human embryo/blastocyst Day 5	Classify blastocyst phenotypes (1 class)	Only applicable for specific regions (ICM, TE, or ZP).
	SA-Net [98]	NA	Medical images (include human blastocyst	5 classes	NA
	Blast-Net [52]	NA	Human blastocyst	5 classes	NA
	CNN/FCN [56]	NA	Human blastocyst	ICM region only	applicable only for ICM.
	HiNN [54]	Self-supervised Image Specific Refinement	human blastocyst (Day 5)	ZP region only	Applicable only for ZP.
	SSS-Net [71]	NA	Human blastocyst	5 classes	Less availability of medical images.
Localization	VGG16 [100]	NA	Human blastocyst	5 stages	Problem with 3-cell stage.
Detection	DCNN (ResNet backbone) [82]	NA	Mammalian embryo—Day 3, human embryo—Day 4	2 classes	NA

Table 3. Comparisons studies associated with the blastocyst using deep learning models.

CNN Function/DL Model		Additional Feature/Model	Input/Training Data	Output Class	Limitations
Classification	VGG-16 [91]	Grad-CAM algorithm (for visualization)	Human blastocyst	Two blastocyst quality	Only suitable for TL systems (due to multiple focal depths as input).
	CNN [27,41,50,88,90,92,98]	- Image pre-processing - Xception model - ImageNet - GA - LTSM	Human blastocyst	Two, three and five blastocyst quality	- Performance and data dependency. - Omission of clinical KPIs (pregnancy rates) and analysis of other laboratory KPIs (fertilization rates). - Do not have features for monitoring as TL). - Required randomized control trials before routine use in clinical practice.
	ResNet [90,97]	- GA model	Human embryo	5 class (Grade 1 to 5)/9 risk factors	Issue in image capturing process and limited resources.
	DL (MLP classifier) [94]	Raman spectroscopy	Human embryo (Day 3 hpi)	2 class (blastula and non-blastula)	NA
Prediction	DNN [13,49,63]	- Random forest and logistic regression classifier - Google’s Inception V1) + decision tree	Human blastocyst—TL	2, 3 classes	- Low prediction accuracy. - Low sensitivity and system developed only for blastocysts (Day 5). - Trained algorithm cannot identify positive and negative life birth successfully using embryo morphology.
	CNN [99]	RNN (prediction)	Human blastocyst	Three blastocyst classes	Disregard blastocyst expansion stage (due to annotation).
	ResNet and DenseNet [24,75,83]	NA	Human embryo (Day 5)	2, 3, 5, classes	- Not incorporate information of different days of embryo development. - Not applicable on other TL/data. No clinical characteristics included in the study.
Segmentation	U-Net [55,57,59,61]	Semantic segmentation	Human embryo/blastocyst Day 5	Classify blastocyst phenotypes (1 class)	Only applicable for specific regions (ICM, TE, or ZP).
	SA-Net [98]	NA	Medical images (include human blastocyst	5 classes	NA
	Blast-Net [52]	NA	Human blastocyst	5 classes	NA
	CNN/FCN [56]	NA	Human blastocyst	ICM region only	applicable only for ICM.
	HiNN [54]	Self-supervised Image Specific Refinement	human blastocyst (Day 5)	ZP region only	Applicable only for ZP.
	SSS-Net [71]	NA	Human blastocyst	5 classes	Less availability of medical images.
Localization	VGG16 [100]	NA	Human blastocyst	5 stages	Problem with 3-cell stage.
Detection	DCNN (ResNet backbone) [82]	NA	Mammalian embryo—Day 3, human embryo—Day 4	2 classes	NA

As a summary, the evaluation of embryo morphologies at the blastocyst stage is just one of the factors that dictate the clinical outcome in patients and other factors such as the male factor, medication, and patient prognosis and history also need to be taken into consideration by the deep learning model to potentially improve clinical outcomes directly [89]. In addition, Figure 7 illustrates the importance of deep learning models having been used on existing studies in reproductive study, particularly for grading the quality of the blastocyst at developmental stage using a score of numbers of application.

5.2. Challenges in Deep Learning Approaches

The main challenge in deep learning approaches to achieve better accuracy is about requirement of large dataset for training the automated grading model. Although there were many datasets available for medical image analysis [98], a very limited datasets are available in ART. The only available public dataset of human embryology specifically in blastocyst stage was provided by [67] and has been used in many studies as ground truth (GT). Other publicly available dataset of human embryology (day 3–5) is accessible at https://www.kaggle.com/ditzsins/grades-embryo/discussion (accessed on 1 August 2022), but is limited in the number of images and by its unknown source. Other than that, datasets of mouse embryos are publicly available for machine learning training provided by [100] and https://bbbc.broadinstitute.org/BBBC003 (accessed on 2 August 2022) but may not suit the detection model for human embryos due to their different physical appearance. So far, the only study that used deep learning with a large dataset of microscopic embryo images was conducted by [101] to develop a CNN-based prediction model. With more than 170,000 embryo raw images of the Asian population and three classification categories of blastocyst (ICM, and TE), the study obtained promising results of 75.36% predictive accuracy.

Vision processing results using deep learning are also dependent on image resolution. Achieving adequate performance in object classification, for example, requires high-resolution images or video—with the consequent increase in the amount of data that needs to be processed, stored, and transferred. Image resolution is especially important for applications in which it is necessary to detect and classify objects in the distance, e.g., in security camera footage. The frame reduction techniques discussed previously such as using SIFT features [22,36] or optical flow for moving objects [23] to first identify a region of interest are useful with respect to image resolution and also with respect to reducing the time and data required for training.

Among all reviewed papers, the study that most focused on blastocyst development was conducted by [54] to analyze the expanding, hatching, or hatched embryos and its related morphological features. However, it is important to consider the morpho-kinetic parameters and morphological features since these two are complementary markers for embryo selection and blastocyst prediction. In addition, the integration of spatial and temporal information is crucial to accurately predict the blastocyst formation for potential embryo growth [28]. As proven by many studies, the use of temporal information from time-lapse imaging can indeed improve model performance [18,32,91,94] as achieved by [94], in which the accuracy improvements have been increased to 7.2% and 5.1% for ICM and TE grading, respectively.

A data-driven nature plays a crucial role in terms of the implementation of deep leaning model applications in an automated, precise, and robust method for application to explore and delineate specific features of an embryo development at blastocyst. In many applications, all images used to train the system were static, however in the real scenario, the embryo development is a dynamic process. Therefore, the static images cannot reflect the entire status of an embryo and may affected the learning algorithms in machine learning models [74]. In addition, the image inputs for prediction models may vary in terms of 2D or 3D grayscale images obtained either from the image captured or time-lapse videos. Although a 3D camera is expected to improve the quality of the embryo assessment [51] and would be well suited to perform such an analysis, the modification and application of the proposed models on actual 3D imagery is expected to be more complicated and consume training times. In addition, 3D embryo images that were captured as 2D images by the TL machine with only one focal depth made it difficult for the AI model to track the distorted or overlapped cells and then make a correct recognition or classification [28]. Thus, it is important to note that the selection of correct data-driven inputs for machine learning models, particularly the deep learning, could give effect to model performance as well as the precision of grading the blastocyst quality.

Moreover, the clinical characteristics (e.g., parents’ age and infertility reason) of data should also be taken into account in developing the automated models. The influences of these factors on embryo development should not be ignored since a more accurate prediction model that incorporates clinical characteristics with video analysis should be established [40]. As an example, a study by Khosravi et al. [73] explored the possibility of directly predicting the likelihood of pregnancy based on only embryo images that are labeled as “positive live birth” or “negative live birth”. However, the result showed that the trained algorithm cannot identify positive live birth and negative live birth successfully using embryo morphology alone. This indicates that biased selection from sample collection could result in poor performance of ML models in a clinical setting [102].

The greatest challenge of implementing of AI models in an IVF grading system is currently on its stands, which does not incorporate additional information from different days of embryo development [83]. As proposed by Kevin et al. [103], characterizing the AI model for embryo assessment could improve the prediction of clinical pregnancy in terms of image quality, bias, and granularity of scores. However, this approach is only applicable on static IVF images at blastocyst stage. In fact, morphology and morphokinetics of blastocyst images has moderate agreement among embryologists on blastocyst implantation.

A combination of deep learning models with computational algorithms can yield a continuous score that represents the quality of the embryo. As an example, the network that was trained to classify embryos based on their quality has performed well even in differentiating between embryos of the same class when combined with a genetic algorithm (GA) [88]. In addition, several multiple independent studies [78,104] that focused on deep learning-based applications in embryo scoring have been suggested to improve the objectivity and consistency the of model but are limited in terms of generalization of the model. This is because if the networks model is overfit (i.e., memorize) during training, the model tends to perform poorly in terms of consistency. Furthermore, most neural networks classifier in deep learning models do not adapt well to different imaging systems and are limited to systems that were used in gathering the training data. Although CNN-based approaches provide an alternative to current methods of embryo scoring, limitations to their performance, among other factors, are dependent on the dataset used in training and the methodology involved in training such systems. In addition, the application of RNNs and GANs is still limited and further opening research is required in employing transfer learning systems, especially on un-supervised medical datasets.

6. Future Perspective in Automated Blastocyst Grading Using Deep Learning Approaches

Typically, in this kind of study, small sample sizes and semi-automated image processing operation are the main limitations [27], and fully automated systems would be desirable to collect large data numbers in a fast and confident manner. In addition, the use of automated systems advantageously allows IVF procedures to be conducted in a more flexible manner rather than at specific time points defined by the need to conduct observations a set number of hours later [10]. Furthermore, intelligence-based research has yielded tremendous benefits from the development of vast open datasets in reproductive study that provide high-quality training data. Intelligence models such as Deep CNNs are suitable for automatic blastocyst grading from microscopic embryo images and can help embryologists accurately and efficiently choose the best blastocysts for implantation in IVF treatment [23]. As example, by using deep learning models of DenseNet201 and focal loss, long short-term memory (LSTM) network and gradient boosting classifier can accurately predict blastocyst formation and usable blastocysts [28]. In addition, embryo image analysis based on Google’s Inception architecture offered effective runtime and reduced computational cost [73].

The capability of extraction and selection of image features on deep learning model is a significant development direction in automated reproductive technology. It is able to process numbers of features from blastocyst images and select only useful features for the classification task in model construction through dynamic programming and reinforcement learning techniques, hence assisting the embryologist in making better decisions based on patient clinical data. However, a large scale of publicly open embryo dataset, especially at the blastocyst stage is highly suggested between clinical intra-observer to ensure a robust and generalized features selection in deep learning models. In addition, combining medical data from the electronic medical records (EMRs), medical image, laboratory examinations, genetic information and health records with advanced AI methods can potentially change the way in which medicine is practiced [83]. Alternatively, image augmentation [61,81] or generative adversarial networks (GANs) [105] could be useful to encounter the issue of image labeling and annotation for deep learning.

Furthermore, an automatized image processing algorithms has been used widely in deep learning models particularly for multi-label classification and segmentation of blastocyst images [23,28,73]. The implementation of various image processing techniques could extract mathematical variables from either conventional microscopic or TL embryo images [59]. As example, TE and ICM have distinctively rougher texture compared to the blastocoel cavity [67], therefore automated algorithms of image texture, and mathematical and statistical models were considerable to characterize blastocyst regions.

Several recommendations have been expressed in deep learning development such as upgrading existing prediction models on efficient datasets from various infertility clinics [106], improving computational-based embryo evaluation [10], improvement on the machine learning model [107] and wide applications of machine learning models [83]. Although the deep learning model is just a black box, the development and extensive research in IVF are still actively conducted to achieve variable degrees of success. Moreover, stochastic modelling in deep learning applications has been employed in IVF applications for decision making tools [108] and patterning the cleavage-stage embryo [109]. However, these applications were specifically employed for animals’ embryo development and have not yet been applied in human IVF. In the future, it is hoped that a deep learning-based model could assist embryo evaluation and selection as well as streamline the IVF procedure, reducing costs and increasing the ability of embryologists to identify the best quality embryo at blastocyst stage for transfer.

7. Conclusions

This study presents the recent developments in AI-based methods for grading embryo quality, specifically at blastocyst stage, covering a wide range from conventional to deep learning techniques. In addition, this study includes the discussion of recent applications in reproductive technology especially in blastocyst development, and the limitations, challenges, and future trends of deep learning applications in reproductive research. With the increasing availability of various deep learning models and advanced system development, embryo assessment technologies of new markers in blastocyst grading are highly possible for improving the embryo selection method. In this review, we identified 30 deep learning models in the classification of blastocyst quality. However, cross-sectional validation of the models with pregnancy successful rate was the missing part of all studies. Thus, the impacts of them have not yet been analyzed for any applications. Future study can be made to further validate the prediction model using a wider population pool. Moreover, the integration of the clinical information such as patients’ clinical information along with IVF outcome into the prediction model could identify the scenarios associated with increased or decreased successful pregnancy.