Transfer Learning in Inorganic Compounds’ Crystal Structure Classification

Abstract

Deep learning consists of deep convolutional layers and an unsupervised feature selection phase. The feature selection of deep learning on a large size dataset can be employed in correlated prediction models with small size datasets. This methodology is titled deep transfer learning model and enhances prediction model generalization. In this research, we proposed a prediction model for the crystal structure classification of inorganic compounds. Deep learning models in structure classification are usually trained using a large size dataset of 300 K compounds from different quantum compounds dataset (DS1). The feature selection of the deep learning models is reused for selecting features in a small size dataset (with 30 K inorganic compounds and containing 150 different crystal structures) and three alloy classes. The selected features are then fed into a random decision forest prediction model as input. The proposed convolutional neural network (CNN) with transfer learning realizes an accuracy of 98.5%. The experiment results display the CPU time consumed by our model, comparing the time required by similar models. The CPU classification time of the proposed model is 21 s on average.

1. Introduction

1.1. Deep Learning and Transfer Learning

The fourth paradigm of science has emerged, and it is data-driven [1]. In research, deep learning has grown to be a crucial addition to experiments, theory, and simulation [2,3,4,5]. Deep learning has recently become a popular tool in substance research and development for forecasting the intricate connections between a crystal’s substances, processes, structure, and features [6,7,8,9].

Typically, structured data is dealt with using conventional deep learning, such as support map deeps, random decision forests, and neural networks. It is usual for the effectiveness of traditional deep learning models to be influenced by the caliber of manually created features (descriptors) and the size of datasets [10,11,12,13,14]. When creating new substances, nevertheless, it can be difficult to find sufficient data and features that are well-built [15,16]. As a result, there are many circumstances where standard deep learning cannot be applied.

Unstructured data (such as images and sounds) can be immediately entered into a deep learning model without the need for manual feature engineering [17,18]. Deep learning combines an automatic selection feature and has been extensively employed in image-related applications, such as defect detection and microstructure recognition [19,20,21].

Transfer learning could be a valuable deep learning procedure that empowers knowledge sharing between models from related spaces [22,23,24,25,26]. Deep learning has special interest in realizing transfer learning because of its chain of command structure: convolution layers (i.e., programmed, including the selection phase) extricate highlights from the low to the high level in the arrangement. A portion of extricated highlights is common and transferable and has 15 million pictures and over 20 thousand categorical names), as opposed to building a modern model from scratch utilizing an irregular start. Moreover, the selection of a well-trained highlight from an expansive dataset empowers the exactness of forecast from a small-size dataset that would otherwise be troublesome to realize [27,28,29,30].

In any case, deep learning models coupled with exchange learning have not been sufficiently used in fabric investigations. Routine deep learning models are still the most used instruments in understanding fabric-related issues. A critical step in applying customary deep learning to substance investigation is to construct a set of manually highlighted features. This is because substance inquiry involves information on crystals, forms, and fabric features, which comprise large, organized information, and datasets in substance inquiry are usually small-sized (deep learning commonly require large size dataset). However, when we outline organized information to 2D pseudo-images, deep learning can be used to extricate highlights [31,32,33,34]. Hence, the extreme and dubious highlight designing can be dodged with deep learning, and the troubles with restricted-size benchmarks might be somewhat overcome by exchange learning.

1.2. Classification of Inorganic Crystal Substances

Crystal patterns in any substance can identify the features of any substance. Classification of crystals and phase formations of any substance are fundamental issues facing substance detection. Substance density theory and force field computation are employed to identify and classify crystals. Nevertheless, their utilizations are hindered by CPU time and computational load [35]. Classification of the crystal substance from the first principle requires high-accuracy computations of entropies for thousands of putative crystals [33,34,35,36]. Many problems still occur in substance density theory, such as computation for composite compounds with high entropy and principal compounds [35,36,37]. Deep learning models are training-driven models which are highly effective compared with statistical methods. To speed up new structures’ classification, they can be employed as another model for simulation [35,36,37,38,39,40]. Deep learning methods [40,41,42,43,44] can predict crystal substance structures. Deep models are used to predict entropy in substances [41,42,43,44,45,46].

Different deep learning models for crystal classification are depicted in

Table 1. Summary of different machine learning and deep learning models to detect the crystal structures in different datasets.

Reference	Model	Dataset	Implementation	Training Time	Classification Time (Seconds)	Accuracy %	Limitation
[15]	Crystal classification using support vector machines	Crystal images	Support vector machines	32 h	127 s	77.5%	Low precision images yielded a high false positive rate
[16]	Crystal structure classification using neural learning	Infrared crystal photos	Convolution network trained with	56 h	212 s	84.4%	Lengthy training time
[17]	Classification of crystal structure using region of interest	Inorganic image dataset	Statistical study	No training process; it is not machine learning	Stochastic process	81%	Small dataset
[18]	Feature fusion	75,000 incidences of inorganic compound crystallization and organic crystals	Decision tree	120 h	220 s	88.2%	Binary classification (inorganic crystal or organic crystal)
[19]	Deep learning CNN model	Inorganic crystal structure data	Deep CNN	79 h	170 s	87.3%	Unbalanced dataset
[20]	3D CNN	3D crystal images	3D deep learning model	240 h	90 s	90.9%	Long training time
[21]	A crystal structure classification intelligent model	Induced dataset	Artificial intelligence method	25 h (small-size datasets)	78 s	77.4%	Low accuracy because of the size of the dataset
[22]	Crystal classification using image segmentation	Inorganic crystal dataset	Encoder-decoder model	120 h	160 s	90.6%	Training time increases with the data size
[23]	Crystal detection in videos	Crystal in video frames	Object recognition model	Feature mining	400 s	89.4%	Lengthy recognition time

.

In this research, we propose a crystal structure classifier that employs transfer learning to predict different compounds of inorganic substances without any prior knowledge.

2. Materials and Methods

2.1. Crystal Structure Mapping

Deep learning models such as CNN require one or two-dimensional data maps as the training phase input. They are fed 2D images for training and prediction. Visual features and associations among various 1 or 2 values are computed via the feature selection process, which comprises several convolutional and pooling layers followed by the rectified linear activation function (ReLU). The authors in [30,31,32,33,34,35] established crystal substance mapping to two-dimensional maps, using 2D maps and atom representation, which allows the CNN to learn from composite data. In the proposed research, we represent the crystal structure of inorganic substances in 2D maps with cation and anion structures. Figure 1a depicts the proposed mapping procedure. A crystal structure is uniquely mapped to a 2D matrix. Each cell of the matrix corresponds to a cation or anion (the cation is 1, and the anion is denoted by 2). The information on the crystal is denoted by a value 1 or 2 located at the same site of the crystal cation or anion in the 2D map (e.g., the information on an example crystal is defined by the 1 or 2 value in its corresponding cell in the matrix. Other cells are occupied by zeros. Figure 1a shows the 2D representation of the crystal compound. Zero values represent the empty areas in the 2D map of a crystal.

2.2. Deep Learning Models

Many CNN networks, such as AlexNet and GoogLeNet, are well-established. Nevertheless, those networks have to go through layer reduction. This reduction decreases the overfitting problem (the DS1 [38] set utilized in this research is 2% size of ImageNet). A CNN network is utilized in this research because of its feature extraction capability. The used CNN has three modules. The first is a transferable feature selection module using the first five convolutional layers, and the second module involves the ReLU activation function, comprising a nonlinear function f(x) = max(0,x) and subsequent max pooling. The third module is the regression process with a fully connected (FC) and a ReLU. The CNN has a 5 * 5 convolutional kernel and 3 * 3, as depicted in detail in

Table 2. Proposed CNN model layers and hyperparameters.

Layer #	Layer Name	Filter	Activation Nonlinear Function
1	Input	128 × 128 × 3	-
2	CL	256/5 × 5	-
3	Max pooling	3 × 3	ReLU Activation function
4	CL	64/3 × 3	-
5	Max Pooling	2 × 2	ReLU Activation function
6	Dropout layer	0.45	-
8	Parameter	32	ReLU Activation function
9	CL	32/3 × 3	-
10	Dropout layer	0.3	-
11	Classifier	-	Softmax
12	Output	-	Crystal class

.

Zero-padding is utilized for the data compounds in the CL by embedding zeros at the boundaries to attain as much data as possible. Filters are induced during the learning phase. The proposed CNN has 17,281 parameters, only a fraction of the other neural networks, such as VGG-16, which possesses 120 million parameters. This can hinder the overfitting problem.

2.3. Transfer Learning

Figure 1b depicts the flow diagram of transfer learning. Two input datasets are utilized, i.e., the large-sized input dataset DS1 [16] and the small-sized target dataset (DS2 dataset in our research). The DS1 dataset is utilized for training the CNN to attain the transfer learning features. The transfer learning features are produced feature maps employed for the DS2 dataset. Prediction is performed on the produced maps by a shallow, random decision forest technique. The feature selection process employs the target DS2. The training process of the new neural model from scratch using new target datasets is not compulsory. The new classifiers alone need to be constructed and trained.

2.4. Training Process

CNNs are constructed and tested by applying the open source Python Keras with Tensorflow libraries for employing neural machine learning and execution of deep learning as the backend [39]. Here, 70% of the dataset is utilized for training the CNN, 15% for validation, and 15% for testing. In the training process, the final output of the CNN is compared to the ground truth. The mean square error (MAE) is used as the mean error displacement to compute the fitness. The number of epochs is fixed to 1800 (where loss function values converge).

Random decision forest is embedded in the Python and mat lab Scikit library [39]. Random decision forest is utilized in transfer learning for the classification of the DS2 dataset because of its hyperparameter minimization to the minimal. We controlled the highest number of decision trees in the random decision forest (200 decision trees in our research) with a depth of a decision tree equal to that in our research. Random data splitting in a circular way is employed in the used dataset to guarantee that training and validation subsets have comparable distribution.

The proposed model is trained on a 3.6 GHz i7 CPU and 32 G RAM. The CNN is trained employing random initial parameters on the DS1 dataset with 300 K compounds for 13 h, with batch size equal to 128 with 2200 epochs. The training process of a random decision forest on our DS2 dataset, which has 30 K compounds and 150 classes, takes 0.5 h. The transfer learning technique gains substantial speed over training deep learning models from scratch.

3. Experimental Results

3.1. Datasets Description

3.1.1. DS1

Material science research and practice have made the highest effort in collecting large-sized datasets of substance features, such as the open quantum substances dataset (DS1) [38]. In addition, other datasets include the automatic flow for substances dataset (AFLOW) [40], the substances project [41], and the crystallography dataset (COD) [42]. These large-sized databases can be used as input for transfer learning to produce general, reusable features. In this article, we utilize a dataset of 30 K crystal compounds from DS1. The DS1 dataset comprises compound crystals and their physical features computed by substance density theory. The CNN is trained on the DS1 training subset to identify the formation entropy (EE) and volume (VOL) of these crystals.

In crystals, atoms are circled by several atoms. Coordination is defined by atomic size and bonding axis. Several coordination systems with ionic radius and bonding orbital are depicted in

Table 3. Configuration of ionic radius.

Coordination	Cation/Anion Radius Ratio (Range)	Configuration
3	0.143–0.223	Tri
4	0.224–0.414	Tetra
6	0.414–0.852	Octal
8	0.852–1.000	Cubic

. Coordination numbers are tri (3), tetra (4), octal (6), and cubic (8). The number 12 defines the close packing arrangements.

In crystals, atoms are bounded by atoms in a coordination arrangement which is identified by ionic radius and bonding link. Coordination parameters with ionic radius and surrounding orbit are depicted in Figure 2 and comprise tri, tetra, octal, and cube configuration.

3.1.2. DS2 Dataset

The DS2 dataset of crystal structures is defined in three features:

• Compounds such as Cu and NaCl;
• Pearson numbers, such as cF4 and cF8;
• Space group symbol.

For compounds, the phase sample of solid alloys, such as Fe_0.4Co_0.2Ni_0.4, is Cu, cF4, 225. We chose a dataset with 30 K inorganic substances. Phase samples with more than 30 items are included. The DS2 dataset has 30,000 inorganic materials and comprises 150 phases. The crystal distribution is depicted in Figure 2. Most crystals of inorganic substances exist in the DS2 dataset, and the distribution of crystal existence is nonuniform. Figure 3 depicts the distribution of the compounds in each phase in the dataset. Of 150 compounds, 80 compounds have a frequency of less than 56, and only a few compounds have a frequency higher than 900 (including MgCu₂, NaCl, and CeAl₂Ga₂). A sample from the 150 compounds is depicted in Figure 3.

There are 14 infrequent compounds with a restricted count (20–40) in our data.

Table 4. Frequency of different inorganic compounds.

Compound	Frequency
MgCu2, cF24	227–229
Ca2Nb2O7, cF88	226–231
Cu, cF4	225
ZnS, cF8	216
CaB6, cP7	221
YbFe2Al10, oS52	63
Pr3WCl3O6, hP26	176
Y4PdGa12, cI34	229
YCo5P3, oP36	62
K3Nb8O21, hP64	193
Y6RuI10, aP17	22
KGdNb6Cl18, hR81	148
KAsF6, hR24	148
Cs3Tl2Cl9, hR84	25
Ba2Cu4YO8, oS30	23
Hf9Mo4B, hP28	32
Er3CrB7, oS44	37
CsMn2P6O18, mS56	12
U3Ni4Si4, oI22	37

describes the count of the instances of different compounds.

3.1.3. The Used Datasets

The public datasets (DS1 and DS2) used in this research can be found in [46,47]. DS1 has 345 compounds (binary): 61 single BCC compounds, 31 single FCC compounds, 15 single HCP compounds, 61 amorphous, and the rest are multiphase compounds. The compounds contain seven to nine crystals. DS1 has 49 crystals and their counts. DS2 dataset has 2335 quandary compounds: 600 single BCC compounds, 534 FCC phase compounds, 222 amorphous compounds, and 1254 multiphase compounds. The dataset has only eight crystals.

3.2. Experimental Results

Figure 4 depicts the CNN accuracy of Pearson correlation value between the classified values and ground truth values in both datasets (we used 20% of the dataset for testing). Using compound structures, the CNN network realizes an accuracy of 98.7% in the classification of energy entropy (EE) and 98.3% in crystal volume (VOL). The experiments depict that the CNN regression does not utilize manual feature computation. Moreover, it enhances testing precision. CNNs can select better features (which are employed for transfer learning). CNNs are pretrained on a large-sized data of 231 K compounds and 92 crystals.

3.3. Experimental Results on Phase Compounds

The feature selection in the DS1 dataset is reused in differentiating between 180 phase compounds. In the deep learning model, the crystal structure is represented by maps and used as input for the feature selection process (EE and VOL), and two feature maps of 188 feature dimensions are generated. These feature maps are fed to the training phase classification scheme (forest trees in our research). We employed three random forest tree models. The first classifier utilizes the feature map produced by the EE feature selection phase (Map^EE) as input. The second classifier utilizes the feature map produced by the VOL feature selection phase (Map^VOL). The third classifier utilizes both Map^EE and Map^VOL (Map^EE&VOL). The comparative study also utilizes a random forest tree model on a structure map of a 112-dimension map representing 112 crystals and counts the percentages in an inorganic compound (Map^comp).

Figure 5 depicts the compared models’ accuracy versus the ratio of the test dataset to the whole dataset. The performance metrics utilized are recall, precision, and F-score, and, compared to the test ratio, depict trends similar to the classification accuracy. All models are constructed on crystal structures. Nevertheless, the accuracy of the transfer learning models is higher than the model without transfer learning. With 90% of the dataset used for training and 10% for testing, the model using Map^EE&VOL can attain an accuracy of 90%, while the model without transfer learning using Map^comp only attains an accuracy of 55%. The accuracy percentage of the transfer learning is unaffected by the training/testing ratio. When the test data rise from 10% to 50%, the accuracy declines from 90% to 86%. The model using Map^EE&VOL depicts the highest accuracy percentage because it uses features from both selection phases.

3.4. Testing Performance Metrics on High-Entropy Compounds

Table 5. Performance metrics of transfer learning models on high-entropy compounds using eightfold validation.

	Accuracy	Precision	Recall	F-Score
EE metric	0.940 ± 0.024	0.945 ± 0.025	0.940 ± 0.024	0.929 ± 0.024
VOL metric	0.949 ± 0.009	0.940 ± 0.008	0.940 ± 0.007	0.949 ± 0.006
EE&VOL	0.986 ± 0.007	0.980 ± 0.004	0.970 ± 0.008	0.986 ± 0.003

depicts the mean performance metrics and mean square errors of the proposed models on high-entropy benchmarks using an eightfold testing process. The transfer learning models can differentiate between BCC, amorphous, multiple-phase mixture, FCC, and HCP, with an eightfold validation process using accuracy, recall, precision, and F-score on the testing datasets over 94%.

3.5. Comparative Study

We performed a comparative study comparing our proposed model with transfer learning against the state-of-the-art models. We utilized the same dataset as depicted in

Table 6. Comparative study on the same datasets with transfer learning.

	Accuracy	Precision	Recall	F-Score
Model1 in [21]	89.12%	88.1%	90.1%	88.1%
Model2 in [34]	92.5%	94.7%	93.55	91.2%
Our proposed model with EE&VOL	98.6%	98.0	97.0	98.6

and Figure 6.

4. Discussions

Difficulties in the phase compounds’ classification using structure alone involve small-sized datasets for the compounds in the dataset, imbalanced data, and a large number of classes (180 classes in our research). The utilized dataset in this research has several inorganic substances such as solid solution compounds, metals, halides, etc. It is problematic in terms of time and accuracy to select features manually. In addition, some physical measures are unobtainable for many crystals in the utilized dataset. For these problems, we used transfer learning.

When using deep learning for a novel compound classification, feature selection engineering transfers the source domain into the model. This decides the performance scores of the deep learning models with insufficient data. Collecting proper descriptors involves deep knowledge of the tools, which is very difficult in classifying new substances. The transfer learning model can be utilized to construct a high-quality standard. As we show, the proposed models in this research can attain high performance using structure without any manual engineering.

Four experiments are performed using 85% of the dataset DS2 (8500 instances) for training and 15% for testing (1500). The subsets are designated randomly. Performance is shown in the confusion matrix for Experiment 1 using the proposed CNN with and without transfer learning, as shown in

Table 7. Experiment 1 confusion matrix for the proposed CNN without the transfer learning using DS1 eightfold testing and (Map^EE) as input for training.

Experiment 1 Testing Phase 8 × 1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic compound	5780	6220	12,000

,

Table 8. Experiment 1 confusion matrix for the proposed CNN with the transfer learning using DS1 eightfold testing and (Map^EE) as input for training.

Experiment 1 Testing Phase 8 × 1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic Compound Value	11,743	257	12,000

,

Table 9. Experiment 2 confusion matrix for the proposed CNN without the transfer learning using DS1 eightfold testing and (Map^VOL) as input for training.

Experiment 1 Testing phase 8 × 1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic compound	6231	5769	12,000

,

Table 10. Experiment 2 confusion matrix for the proposed CNN with the transfer learning using DS1 eightfold testing and (Map^VOL) as input for training.

Experiment 1 Testing Phase 8 × 1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic Compound Value	11,661	339	12,000

and

Table 11. Experiment 3 confusion matrix for the proposed CNN without the transfer learning using DS1 eightfold testing and (Map^EE&VOL) as input for training.

Experiment 1 Testing Phase 8×1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic compound	7023	4977	12,000

. The accuracy and recall are depicted in

Table 12. Experiment 3 confusion matrix for the proposed CNN with the transfer learning using DS1 eightfold testing and (Map^EE&VOL) as input for training.

Experiment 1 Testing Phase 8 × 1500 = 12,000	Actual Compound		Total Cases
	Correct	Incorrect
Predicted inorganic Compound Value	11,922	78	12,000

and

Table 13. Experimental results for the three experiments with transfer learning.

Average Results
Model	Accuracy %	Sensitivity %	Specificity %	Error Rate
Experiment 1	95.7	95.9	95.1	0.0321
Experiment 2	97.73	96.7	96.79	0.0219
Experiment 3	98.93	98.8	98.77	0.0019

. It is proven from these metrics that transfer learning increased performance by 40%.

The correct and incorrect prediction cases of the third experiment using transfer learning and (Map^EE&VOL) have the highest count of correctly predicted compounds.

5. Conclusions

CNN with transfer learning is highly successful in classifying phase compounds and accurately classifies 170 phase inorganic compounds based on crystal structures. Representing the crystal structure of inorganic compounds with 2D images allows CNN to learn from structured data. Transfer learning using feature extraction phases of CNN uses transfer learning on large datasets, such as DS1, and learning outcomes can then be reapplied in new learning processes to produce rich features. Transfer learning reduces the training time for new classification models but also enhances the performance and the process generalization for other models with small-sized datasets.