Predicting the Young’s Modulus of Rock Material Based on Petrographic and Rock Index Tests Using Boosting and Bagging Intelligence Techniques

Abstract

Rock deformation is considered one of the essential rock properties used in designing and constructing rock-based structures, such as tunnels and slopes. This study applied two well-established ensemble techniques, including boosting and bagging, to the artificial neural networks and decision tree methods for predicting the Young’s modulus of rock material. These techniques were applied to a dataset comprising 45 data samples from a mountain range in Malaysia. The final input variables of these models, including p-wave velocity, interlocking coarse-grained crystals of quartz, dry density, and Mica, were selected through a likelihood ratio test. In total, six models were developed: standard artificial neural networks, boosted artificial neural networks, bagged artificial neural networks, classification and regression trees, extreme gradient boosting trees (as a boosted decision tree), and random forest (as a bagging decision tree). The performance of these models was appraised utilizing correlation coefficient (R), mean absolute error (MAE), and lift chart. The findings of this study showed that, firstly, extreme gradient boosting trees outperformed all models developed in this study; secondly, boosting models outperformed the bagging models.

1. Introduction

Generally, supplying a dependable evaluation of rock mass behaviors (i.e., deformation and strength) is primarily significant in designing and investigating rock engineering implementations, including underground excavation, slope, and foundation. To delve more into the depth of this issue, to analyze the characteristic of rock deformation, two major inputs should be considered for which the elasticity constants of rock are mass (E, v). Generally, it is feasible to measure the parameters of rock mass deformability, including Young’s modulus, by field tests essentially in situ moduli (signified by E_rm) directly and through research lab examinations as intact modulus (signified by E) indirectly [1,2]. Furthermore, considering true deformation analysis regarding rock needs to be conducted due to location circumstances; it is significant to assess the parameters of rock deformation by utilizing both field and lab conditions. Therefore, the intact material modulus resulting from lab practices must be associated with the rock mass modulus utilizing a suitable classified scheme.

The most broadly applied schemes for discovering Young’s modulus contain plate loading and flat jack examination for determinations in the field [3], along with uniaxial compressive strength (UCS) examination in the lab. Nevertheless, the consequences of these experiments may be vulnerable to be ambiguous according to the discontinuity and anisotropic characteristics of rock mass forced to various field pressures [1,4]. However, direct measurements of Young’s modulus are extensively slow and demand expensive facilities, especially for field experiments [1,5].

Various polynomial regression patterns utilizing field along with lab analysis data have been offered to overcome these restrictions. These regression models employed associations with data from field and classification methods of rock mass, consisting of rock mass rating [6], tunnelling quality index [7], and geological strength index [8], to measure the values of deformation modulus, Erm, for an isotropic rock mass. The results from the majority of these associations present a quite reliable fit to the field data, although the modulus-based equations and the exponential relationships, as recommended by [9,10], address poor efficiency in rock mass deformation moduli’s prediction.

Moreover, several regression equations were developed using data from lab analyses. The applied method of developing these regressions was relating a specified spectrum of data from simple index tests, including Schmidt hammer [11,12], ultrasonic velocity [2,13], point load strength [14,15], and porosity [1,16,17], to Young’s modulus, E. While some previous studies stated that the traditional statistical methods lack the power of generalizability [17,18], this present study endeavors to examine this controversy rigorously.

Based on the latest research, the number of claimed successful applications of intelligent methodologies has noticeably increased in the area of science and engineering since almost 20 years ago [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42]. In this regard, artificial intelligence and machine learning methods, such as artificial neural networks (ANNs), have been used in the initial design of rock-based structures [5,43,44,45]. To elaborate, in 2018, [46] put into perspective the consequences of a fuzzy inference system (FIS) model in forecasting the UCS and E of greywacke samples. The author used 54 fuzzy rules in the suggested fuzzy model to outline inputs, rock index properties, output variable, E, and USC. To conclude, the author examined the results of the fuzzy model by analyzing its performance based on multiple regression. In so doing, they observed that the predicted results in comparison with a statistical model were in better agreement with the lab test results. In another investigation, Kahraman, et al. [47] provided an ANN model to forecast the E and UCS levels for Misis fault breccia. Having compared their forecast outcomes with the regression models, they observed a high degree of precision in their designed ANN model. In another comparative study to predict Young’s modulus of gypsum, Yılmaz and Yuksek [15] developed both an ANN and multiple regression, and they concluded that the ANN model could predict the rock modulus more accurately. Mohamad, et al. [48] and Momeni, et al. [49], to show the prosperous application of intelligent methodologies in predicting the UCS of rocks, carried out studies using the hybrid particle swarm optimization technique (PSO) and ANN model.

Table 1. Recent studies on the use of machine learning techniques to predict the E.

Study	Input Variable	Method	R2
Armaghani, Mohamad, Momeni, Monjezi and Narayanasamy [1]	Is(50), n, RN, Vp	ICA-ANN	0.71
Armaghani, et al. [54]	p; Vp, Qtz, Kpr, Plg, Chl, Mica	ANFIS	0.99
Beiki, Majdi and Givshad [17]	p; n; Vp	GA	0.67
Bejarbaneh, Bejarbaneh, Fahimifar, Armaghani and Abd Majid [2]	Rn, Vp, Is(50)	FIS and ANN	0.79 and 0.82
Dehghan, et al. [55]	Vp; Is(50); Rn; n	ANN	0.77
Gokceoglu and Zorlu [56]	Is(50), BPI; Vp, BTS	FIS	0.79
Majdi and Beiki [57]	p; RQD; n; NJ; GSI	GA-ANN	0.89
Singh, et al. [58]	p; Is(50), WA	ANFIS	0.66
Yılmaz and Yuksek [15]	ne, Is(50), Rn; Id	ANN	0.91
Yilmaz and Yuksek [14]	Vp; Is(50); Rn; WC	ANFIS	0.95

BPI = block punch index, GSI = geological strength index, I_d = slake durability index, n_e = effective porosity, NJ = number of joints per meter, RQD = rock quality designation, WA = water absorption, WC = water content, n = porosity, p = density, ICA = imperialism competitive algorithm, quartz = Qtz, alkali feldspar = Kpr, plagioclase = Plg, chlorite = Chl, Schmidt hammer rebound number = R_n, ultrasonic velocity = V_p, point load strength = I_s(50).

clearly shows some newly proposed models for forecasting the rock modulus, E, using soft computation methods. Bejarbaneh, Bejarbaneh, Fahimifar, Armaghani and Abd Majid [2] also developed the multiple regression (MR), ANN, and FIS models to predict the E using data from a dam positioned in Malaysia. Their findings showed that the ANN model outperformed the FIS and MR models in terms of accuracy. It should be mentioned that the successful applications of intelligent methodologies have been reported by many researchers in solving civil and mining engineering problems [50,51,52,53].

While the application of the supervised machine learning (ML) techniques, particularly ANN, was vast in predicting the E, to the authors’ best knowledge, a few studies investigated the application of ensembled ANN for predicting the elasticity modulus. In addition, no study applied DTs and their ensembled variants for E prediction. Bagging and boosting are two well-established ensemble approaches for solving both regression and classification problems. However, rare studies employed these techniques for predicting the E. Thus, this study will apply the ensemble variants of ANN and DTs to investigate the performance of these techniques for predicting the E.

2. Methods

The dataset under study consists of 45 samples of eight variables, i.e., dry density (DD), p-wave velocity (Vp), interlocking coarse-grained crystals of quartz (Qtz), plagioclase (Plg), alkali feldspar (Kpr), chlorite (Chl), Mica, and young’s modulus (E). Note that DD, Vp, and E were obtained using relevant laboratory testing procedure [52] and mineral properties of the samples (i.e., Qtz, Plg, Kpr, Chl, and Mica) were obtained using petrographic analyses. The goal is to investigate the most appropriate ensemble approach for predicting E. The dependent variable that should be expressed in terms of other variables is E.

Two ensemble approaches were applied to ANN and DTs for E prediction. These approaches included bagging and boosting. The performance of these models was assessed utilizing linear correlation (R), mean absolute error (MAE), and lift. A flow diagram of this research is indicated in Figure 1.

2.1. Data Gathering and Case Study

The authors of this present study gathered data from the surface of the Pahang–Selangor raw water transfer (PSRWT) tunnel in Malaysia. This project was constructed to transport the Selangor’s water demand and the state of Kuala Lumpur to Selangor. It is worth noting that the tunnel was passed under the major mountain chain separating Selangor and Pahang states. This mountain chain has an elevation varying from 100 to 1400 m and shaping the backbone of Peninsular Malaysia. Granite is the major rock variety in this tunnel, with a regular rock strength of 150–200 MPa. The excavators unearthed three tunnel parts using three tunnel boring machines (TBMs). The length of these tunnels was 11.7 km, 11.7 km, and 11.3 km.

For lab tests, the research team prepared 45 rock samples. They also examined the samples for any factor that may lead to an unwanted change in qualities and unanticipated breakdowns. Thus, the samples were inspected for any cracks or tiny-scale discontinuities. Dry density (DD) and Vp were among the rock physical tests. In addition, in order to determine the E, the research team conducted uniaxial compression tests. The rock samples’ E were calculated based on the axial strain outcomes and the stress estimated by significant transducers. ISRM [59] standards were implemented during the sample preparation and sample testing.

The data of this research also were used by Armaghani, Mohamad, Momeni and Narayanasamy [54]. The research team also conducted petrographic analyses on the granite samples, employing a polarizing petrological microscope. In order to determine the portion of various minerals in the samples, small parts of the samples were prepared. The specimens present nonporphyritic and holocrystalline mineral composition and usually chiefly involve alkali feldspar (Kpr), interlocking coarse-grained crystals of quartz (Qtz), biotite (Bi), and plagioclase (Plg). Note that micas are factored by three minerals: sericite, muscovite, and biotite.

2.2. Data Analysis

In this research, multiple statistical and simulation procedures were applied to examine the outcomes of lab tests. The consequent parts describe the utilization of the techniques discussed before to gauge the E of granite samples. Eventually, the E values gained from lab experiments were compared with those forecasted.

2.3. Regression Analysis

This research produced various experimental equations employing simple and multiple regression methods. While the simple regression is represented by the general form of Y = A₁X₁ + M, the multiple regression is denoted by the generic structure of Y = A₁X₁ + A₂X₂ + .... + A_nB_n + M, where {A₁, A₂, …, A_n} are the regression’s coefficients and M is Y’s fixed value when every input parameter is equal to zero. Figure 2 displays the simple equations, plots, and linear correlation (R). In addition, two equations were developed using multiple regression techniques (

Table 2. Equations using various multiple regression techniques.

Method	Equation	R	Std. Error of the Estimate
Enter	E = 25.89 × DD + 0.005 × Vp + 1.50 × Qtz − 1.22 × Kpr − 0.10 × Plg − 2.18 × Chl − 1.29 × Mica − 36.16	0.6	26.620
Stepwise/Backwards/Forwards	E = 0.0009094 × Vp + 1.644 × Qtz − 43.93	0.553	26.018

). Four common approaches, including “enter”, “stepwise”, “backwards”, and “forwards”, were applied to develop these equations. The simple regression equations showed that Vp was the most correlated factor with E, followed by Qtz. In addition, three methods of multiple regression techniques, including stepwise, backwards, and forwards, yielded identical equations for estimating “E”. The performances of these equations are not encouraging. Both simple and multiple regression models yielded low accuracy. Moreover, relying only on one predictor in the simple regression technique is insufficient for analyzing and predicting the “E”. Yilmaz and Yuksek [14] stated that the primary technical paucity of regression techniques is that they simply ascertain relations and do not learn the underlying causal structure.

The machine learning methods can overwhelm the abovementioned concerns using their assumption-free qualities. Various ML techniques can explain E-related problems. While some studies considered well-known ML techniques, such as ANN and DT, for E prediction, no study employed these methods’ boosted and bagged variants. Therefore, this research will apply these two advanced techniques to ANN and DT to predict the E values of Main Range granite in Malaysia.

2.4. ANN and DT Models

2.4.1. ANN

Various ML techniques are available for solving regression problems. Two of the most commonly used ML techniques are ANN and DT. An ANN is a modeling procedure stimulated by the human nervous system that enables learning through representative data instances, which explains a physical aspect or a choice process. One different characteristic of ANN is that they can ascertain practical associations between inputs and target variables and also obtain exact knowledge and complicated information from representational datasets. Associative relationships between inputs and target variables are ascertained without presumptions on the analytical description of the phenomes. This technique presents destined benefits over regression-based techniques, including its capability to handle noisy data. NNs comprise a layer of input nodes and a layer of output nodes joined by one or more layers of hidden nodes.

An ANN ensemble is a learning model where several NNs are trained for a similar assignment [60]. It was shown that the generalization capability of the NN model was remarkably enhanced by ensembling various NNs [61]. Typically, an NN ensemble is built in a couple of steps: (1) training several elements of NNs and (2) consolidating the element forecasts.

There are two common approaches for training component NNs, including boosting and bagging. The process of boosting and bagging is shown in Figure 3. Despite the fact that both bagging and boosting involve N students, they are fundamentally distinct. Unlike the bagging technique, which combines predictions of the same type, the boosting method incorporates predictions of various kinds. In bagging, each model is constructed independently of the others, whereas, in boosting, the outcomes of earlier constructed models are impacted. In the bagging technique, every model is given equal weight, while, in the boosting technique, the new ones are weighted according to their performance. In boosting, new components of training data contain observations that the old model incorrectly classified. Bagging utilizes randomly generated subsets of training data.

2.4.2. DTs

Boosting technique was introduced by Schapire [62] and updated by Freund [63], Freund and Schapire [64]. This technique produces a set of element NNs whose training datasets are ascertained by the performance of previous ones. Training examples that are incorrectly forecasted by previous networks will perform more vital tasks in the training of succeeding networks. Bagging is introduced by Breiman [65] following the bootstrap sampling procedure [66]. This technique produces various training datasets from the initial training dataset and later trains an element NN from every training set.

DT is a kind of supervised learning algorithm suitable for classification and regression problems. With a tree-like formation, it builds the training data from the top down by choosing the most suitable decision node to split first. The subsequent nodes are then produced using entropy and information gain. For every chance node, weights are estimated by conditional (joint) probabilities. Various DTs are successfully implemented in the engineering field, including classification and regression decision tree (CART), Chi-square automatic interaction detector (CHAID), Quick, unbiased and efficient statistical tree (QUEST), and C5. However, these DTs are assumed as weak learners because they are vulnerable to changes in data as well as overfit.

To remedy the issues with DTs stated above, ensemble approaches can be employed. The same as the application of the ensemble approach in ANN, the ensemble approach integrates various DTs to create a more suitable predictive performance compared to a single DT. Random forest (RF) [67] and extreme gradient boosting tree (XGBT) [68] are well-known bagging and boosting techniques, which use a CART model for creating a stronger learner.

As earlier discussed, RF is a bagging algorithm that reduces variance. DTs are unstable with regard to the tiny fluctuations in data. RF can offer a strong model, which lessens the diversity caused by the “bagging” technique. Bagging is an ensemble method in which several predictors are made and consolidated by applying some averaging systems, including normal average, majority vote, and weighted average.

Gradient boosting (GB) is an effective technique for boosting. GB applies a gradient descent algorithm that can optimize any differentiable loss function. A combination of DTs is created one by one, and single DTs are summed sequentially. The next DT attempts to improve the loss (variation between real and forecasted values). Extreme gradient boosting tree (XGBT), which enjoys definite assessments to determine the most suitable tree model, is an effective extension of GB. XGBT implements its tree creation process where the similarity score (SS) and gain discover the fittest node divisions. Later, the node division with the most excellent gain is chosen as the most suitable division for the tree.

3. Models’ Development and Results

3.1. Data Preparation

For developing the ML models, it was required to normalize data, since the data were unbalanced. Thus, the input variables were transformed using min/max transformation techniques wherein the minimum value was zero and the maximum was one. Details of inputs’ distribution before and after the transformation are displayed in

Table 3. Details of inputs distribution before and after the transformation.

Input	Before Transformation		After Transformation
	Skewness	SD	Skewness	SD
Vp	0.36	1137.62	0.36	0.19
Qtz	0.21	5.68	0.21	0.17
DD	2.75	0.13	2.75	0.18
Mica	0.43	3.56	0.43	0.21
Chl	0.85	1.65	0.85	0.33
Plg	0.20	7.86	0.20	0.15
Kpr	−1.05	5.36	−1.05	0.17

.

3.2. Input Selection

Before applying boosting and bagging techniques to data, an input selection has been conducted. As previously mentioned, this study adopted a likelihood ratio test for selecting the most relevant inputs for predicting the E. The candidate inputs included DD, Vp, Qtz, Mica, Chl, Plg, and Kpr. This test then selected Vp, Qtz, DD, and Mica, respectively, as the most relevant E prediction inputs. Figure 4 provides a series of 3D density diagrams of the selected variables and shows the frequency distribution of pairs of each input variable and the target variable (E). The inputs selected in this step will be used to develop the standard, boosting, and bagging models.

3.3. Bagging and Boosting Models

This study developed six models, including standard ANN, standard DT (CART), bagged ANN, boosted ANN, bagged DT (random forest), and boosted DT (extreme gradient boosting tree). These models are developed using the transformed and selected inputs in the previous steps. For developing these models, a 10-fold cross-validation technique was employed to avoid overfitting. In k-fold cross-validation, smaller collections are created by dividing the training dataset into k folds. Then, the training data, including k-1 folds, are trained. The unused part of the data is employed to validate the resulting model. Next, the mean of the values calculated in the circle is adopted to report the k-fold cross-validation performance. Figure 5 shows the process of a 10-fold cross-validation used in this study.

The team behind this research employed various hyperparameters controlling the learning process to develop these models. They also used a grid-search strategy to optimize these parameters. The process of a grid-search in this study is shown in Figure 6. As previously mentioned, the single DT developed in this study is CART. Its optimized hyperparameters included levels under root = 5; highest replacements = 5; lowest variation in impurity = 0.0; lowest records in parent branch = 2%; and smallest records in child branch = 1%. Concerning XGBT model, the optimized hyperparameters were: boost rounds = 10; maximum tree depth = 6; minimum child weight = 1.0; objective = reg:linear; sub sample = 1.0; Eta = 0.3; Colsample by tree = 1.0; Colsample be level = and 1.0; Lambda. For RF, the best hyperparameters comprised: number of trees to build = 10 and minimum leaf node size = 1. Concerning the standard ANN model, the hyperparameters were: NN model = MLP; hidden layer 1 = 1; hidden layer 2: 0; and overfit prevention set = 30%. For boosted and bagged models: number of component models for boosting = 10 and combining rule = mean.

Once the data were prepared and models’ hyperparameters were tuned, the models were executed. Figure 7 shows a comparison between the actual and predicted values of “E” by the models developed in this study. The performance of the models was evaluated using two well-known criteria, including linear correlation (R) and mean absolute error (MAE). The calculation formulas of these criteria are shown by Equations (1) and (2).

$$R=\frac{{{\displaystyle \sum }}_{i=1}^n\left(e_i-{\overline{e}}_i\right)\left(m_i-{\overline{m}}_i\right)}{\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left(e_i-{\overline{e}}_i\right)}^2{\left(m_i-{\overline{m}}_i\right)}^2}}$$

(1)

$$MAE=\frac{{{\displaystyle \sum }}_{i=1}^n\left|e_i-m_i\right|}{n}$$

(2)

where, e_i and m_i denote nth real and predicted values, respectively; ē_i and ${\overline{m}}_i$ signify the average values of real and predicted values, respectively; and n stands for the number of samples in the dataset.

Table 4. Models’ performance.

	DT		ANN
	R	MAE	R	MAE
Standard model	0.476	21.111	0.566	18.928
Bagged model	0.91	9.726	0.849	12.551
Boosted model	0.994	3.836	0.972	5.343

and

Table 5. Details of models’ error.

	Minimum Error	Maximum Error	Mean Error	Standard Deviation
Standard DT	−44.944	51.957	0.081	26.845
Bagged DT	−39.5	32.02	−1.205	13.486
Boosted DT	−4.638	21.466	3.231	4.872
Standard ANN	−60.37	72.076	−1.552	25.35
Bagged ANN	−46.864	42.601	−3.742	16.251
Boosted ANN	−15.577	17.198	0.314	7.332

show the performance of each model developed in this study. Concerning standard models, the ANN possessed a better R and MAE than the CART model. RF outperformed its peer, ANN, when the bagged models are concerned. For boosted models, XGBT had a better performance than the boosted ANN. Among all models developed in this study, boosted models outperformed the bagged and standard models. Finally, the boosted DT, which, in fact, was an XGBT, outperformed other models developed in this study.

In addition to R and MAE, a lift chart also was used for models’ evaluation. The lift chart is from the family of cumulative charts. In this chart, the fraction of records in every quantile that is hit with the overall percentage of hits in the training data is compared. Equation (3) shows its calculation formula. Hits refer to the values higher than the midpoint of the values’ range.

$$Lift=\frac{\left(Hq/Rq\right)}{\left(Th/Tr\right)}$$

(3)

where Hq denotes hits in quantile; Rq refers to the records in quantile; Th signifies the total hits; and Tr is the total records.

Figure 8 shows the lift chart of the models developed in this investigation. In this chart, elevated lines show better models, especially on the left edge of the graph. As can be seen, boosted DT (XGBT) line was higher than the others. This line also was more stable compared to other lines.

Great performance of the XGBT model compared to CART and RF models was expected because of the various capabilities of XGBT that help it to produce a reliable and accurate predictive model. XGBT enjoys the similarity score for a straightaway prune tree, beginning the real modeling goals. In addition, XGBT is a reliable choice for unbalanced datasets, but RF is not trustable for these circumstances. One of the most important differences between XGBT and RF is that the XGBT regularly provides added significance to functional space when decreasing the cost of a model, while RF seeks to provide more inclinations to hyperparameters to optimize the model. XGBT also outperformed all forms of ANN, which can add knowledge and experiences in the field of civil engineering, particularly mining.

In comparison with the previous studies, the accuracy of XGBT model (R = 0.994) was more than that of Armaghani et al. [54] (R = 0.992), who applied an ANFIS model to the same database. They used DD, ultrasonic velocity, quartz content, and plagioclase to develop their predictive model. They selected these inputs using simple and multiple regression models. In addition, the accuracy achieved in this study was higher than in other studies reported in Table 1.

4. Conclusions

This study applied two well-established ensemble techniques, including boosting and bagging to the ANN and DT methods, for predicting the Young’s modulus. These techniques were applied to a dataset comprising 45 data samples from a mountain range in Malaysia. The final input variables of these models, including Vp, Qtz, DD, and Mica, were selected through a likelihood ratio test. In total, six models were developed: standard ANN, boosted ANN, bagged ANN, CART, XGBT, and RF. The performance of these models was evaluated using R, MAE, and lift charts. The findings of this study showed that, firstly, XGBT outperformed all models developed in this study; secondly, boosting models outperformed the bagging models.

One constraint of this analysis was its small sample size. Other experiments can utilize these predictive techniques for E prediction using larger sample sizes to obtain higher accuracies. However, the predictive techniques stated above can be implemented for forecasting the E depending on the circumstances. Regarding the performance, it was further remarked that applying the bagging and boosting techniques builds a functional approach to reducing ambiguities during planning rock engineering projects. Moreover, these methods can be employed for predicting many other mining problems, including unconfined compressive strength. XGBT is an effective, reliable, and functional algorithm. XGBT holds both tree learning algorithms and linear model solvers. Therefore, what makes it quick is its ability to administer parallel calculations at the same time.