Applying a Deep-Learning-Based Keypoint Detection in Analyzing Surface Nanostructures

Abstract

Scanning tunneling microscopy (STM) imaging has been routinely applied in studying surface nanostructures owing to its capability of acquiring high-resolution molecule-level images of surface nanostructures. However, the image analysis still heavily relies on manual analysis, which is often laborious and lacks uniform criteria. Recently, machine learning has emerged as a powerful tool in material science research for the automatic analysis and processing of image data. In this paper, we propose a method for analyzing molecular STM images using computer vision techniques. We develop a lightweight deep learning framework based on the YOLO algorithm by labeling molecules with its keypoints. Our framework achieves high efficiency while maintaining accuracy, enabling the recognitions of molecules and further statistical analysis. In addition, the usefulness of this model is exemplified by exploring the length of polyphenylene chains fabricated from on-surface synthesis. We foresee that computer vision methods will be frequently used in analyzing image data in the field of surface chemistry.

1. Introduction

The recent developments in on-surface reactions have enabled the fabrications of complex nanostructures down to atomic precision [1,2,3,4,5]. Notably, scanning tunneling microscopy (STM) imaging has proven to be one of the most powerful tools for investigating the on-surface reactions, providing high-resolution, molecule-level insights [4,6,7,8]. It offers the capability of identifying reactants, products, and important intermediates and thus unraveling the underlying reaction mechanisms of the atomic-scale processes occurring on surfaces. STM works by scanning a sharp tip over the surface of a material while applying a bias voltage. By measuring the tunneling current between the tip and the surface, STM can create topographic maps that reveal the atomic and molecular species on the surface. The resulting images offer fertile information about the size, shape, orientation, and distribution of molecules or atomic structures, enabling researchers to study molecular adsorption, self-assembly, and reactions [9,10,11,12,13,14,15,16].

Despite advances in STM, the analysis of STM images still heavily relies on manual inspection and interpretation, which is labor intensive, time consuming, and often subjective [17]. Experienced researchers rely on their expertise to identify individual molecules or atomic structures in the images. This manual approach becomes increasingly challenging as the complexity and size of the datasets grow, limiting the scalability and efficiency of the analysis process. Additionally, the lack of standardized criteria for image analysis inevitably introduces variability and inconsistency in analyzing similar experiments conducted by different researchers, or even in different researchers analyzing the same data.

Recently, machine learning techniques, particularly computer vision, have emerged as promising tools in material science research for automating the analysis and processing of image data [18,19,20,21,22,23,24,25]. Machine learning has achieved remarkable advancements even in the field of medical and biological sciences [26,27,28]. By leveraging deep learning algorithms and their image recognition capabilities, computer vision techniques offer the potential to overcome the limitations of manual analysis and provide accurate results with much improved efficiency and consistency [29,30]. Computer vision techniques applied to STM images involve training deep learning models on labeled datasets to recognize and classify individual molecules or atomic structures [17]. The deep learning models can then be used to automatically identify and analyze properties of interest in STM images. By applying computer vision techniques in the high-resolution STM imaging, we can extract valuable information of nanoscale phenomena, such as molecular size, shape, orientation, or patterns, with significantly enhanced efficiency and accuracy of image analysis.

In this work, we propose a method for analyzing molecular images using computer vision techniques. Our approach aims to automate the analysis of STM images and alleviate the efforts associated with manual inspection by developing a lightweight deep learning framework based on the YOLO (You Only Look Once) algorithm. The YOLO algorithm [31] is a one-stage method that stands for You Only Look Once. It is a neural network that can output results by looking at an image only once. YOLO has released seven versions so far, with YOLOv1 laying the foundation for the entire YOLO series and subsequent YOLO algorithms constantly improving and innovating on it. The YOLO algorithm uses a single CNN model to achieve end-to-end object detection. The core idea is to use the entire image as the input of the network and directly regress the position of bounding boxes and their categories at the output layer. We can accurately label each molecule in the STM images with its keypoints, enabling molecule classification and counting and statistical analysis of molecular properties in an instant manner (less than one second for one image with hundreds of molecules). In addition, the usefulness of this framework is exemplified by its application in exploring the length of polyphenylene chains fabricated from on-surface synthesis.

2. Discussion and Results

The workflow of our deep learning framework consists of five modules: standard STM image acquisition, bounding box and keypoint labeling, dataset preparation, model training, and model output. The final outputs contain the information of keypoints on each recognized molecule which are used for further statistical analysis. The overall workflow is illustrated in Figure 1.

STM is a powerful technique for imaging molecules on surfaces. To acquire molecular resolved images, we operated STM under ultra-high vacuum (UHV) conditions for minimizing influences such as contaminations. Due to the principle of STM, the obtained images contain abundant molecular and atomic information including molecular electronic properties and the apparent heights of molecules above the surface. Currently, the application of machine learning models in the keypoint recognition of STM molecular images is scarce [32]. Therefore, we harnessed the YOLOv7 model to incorporate the functionality of keypoint recognition.

We opted for YOLOv7 because of its impressive accuracy and superb capability to handle various input images, in addition to its versatility to be deployed on different platforms [33,34]. We have also compared three different deep learning models and confirmed the outstanding performance of YOLO at a reasonably low cost. (See more details of the comparison in Table S3). The architecture of YOLOv7 is composed of four main components: Backbone, Neck, Head, and Loss [35]. The Backbone is primarily responsible for feature extractions and uses an E-ELAN computational block. This block has been designed to constantly improve the network’s learning ability without disrupting the original gradient path. It typically employs CSPDarknet53, which leverages the Cross Stage Partial Network (CSP) structure to enhance efficiency and accuracy in feature extractions. CSPDarknet53 comprises multiple convolutional layers and residual blocks, enabling multi-scale feature extraction. The Neck serves as a feature fusion module that employs both the PANet (Path Aggregation Network) and the BiFPN (Bidirectional Feature Pyramid Network). The PANet combines fine-grained features from lower levels with semantic information from higher levels through up-sampling and down-sampling operations, thereby enhancing the detection performance of small objects and the perception of image details. The BiFPN introduces bidirectional pathways for inter-layer feature interaction, balancing the contributions of different levels through adaptive weight adjustment, which results in improved accuracy and robustness in object detection. The Head component in YOLOv7 includes an auxiliary head in addition to the primary one, which aids in better training and supervision of detection tasks. The Head is tasked with making predictions about object class, position, and keypoints. The Loss function includes classification loss, box regression loss, and object presence loss, all of which are used during the training process. YOLOv7 also implements a compound model scaling approach, where the network width and depth are scaled in coherence for concatenation-based models, facilitating optimization for different sizes and applications.

While there are various publicly available annotated datasets, there is no molecular dataset specifically designed for training nano-objects of STM imaging. To train our model effectively and achieve good results, we need to require a large amount of labeled data as inputs into the ML models. However, manually annotating thousands of STM images as a dataset is time consuming and impractical. We employed data augmentation techniques after manually labeling a small number of STM images of single molecules. To enrich our dataset, we generated a training dataset by maintaining the labels using data augmentation techniques, resulting in thousands of generated images. More detailed explanations of the data augmentation methods are provided in the supporting Table S1.

As a prototypical example, we selected a self-assembled molecular nanostructure composed of two molecules, one with a triangular and the other with a rectangular symmetry, the chemical structures of which are shown in Figure 2 [36]. For simplicity, we refer to these two molecules as molecule 1 (2,4,6-tri(Pyridin-4-yl)-1,3,5-triazine) and molecule 2 (4,4′-Di(Pyridin-4-Yl)-1,1′-Biphenyl). Figure 2b shows a typical STM image of the nanostructure where we can recognize the two molecules by their distinct shapes. Both of the molecules can be represented by four keypoints as marked by the blue dots in Figure 2c,d, where the blue dots mark the three-fold symmetry of 1 and the two-fold symmetry of 2. In addition, a bounding box is defined to label the molecule. The complete dataset was generated by maintaining the labels of the molecules and applying different combinations of the augmentation techniques. Overall, we annotated only a dozen of molecules manually.

mAP (mean Average Precision) is commonly used to assess the performance of the object detection algorithms. [email protected] represents the average precision when the overlap between the predicted bounding boxes and ground truth boxes (usually measured by Intersection over Union, IOU) reaches 0.5:

$$\mathrm{mAP}@0.5=\frac{1}{N}\sum _{i=1}^N{\mathrm{AP}}_{0.5}^{\left(i\right)}$$

where N represents the total number of classes and ${\mathrm{AP}}_{0.5}^{\left(i\right)}$ denotes the average precision for class i at an IOU threshold of 0.5. In object detection tasks, an overlap of 0.5 is considered as a standard threshold, and [email protected] evaluates the performance for most practical applications. In addition, [email protected]:0.95 represents the average precision when the overlap ranges from 0.5 to 0.95, that is,

$$\mathrm{mAP}@0.5:0.95=\frac{1}{N}\sum _{i=1}^N{\mathrm{AP}}_{0.5:0.95}^{\left(i\right)}$$

where N represents the total number of classes and ${\mathrm{AP}}_{0.5:0.95}^{\left(i\right)}$ represents the average precision for class i with the IOU threshold ranging from 0.5 to 0.95. This metric provides a more comprehensive evaluation, as it considers the precision variation under different overlap requirements. In this case, the [email protected]:0.95 provides a more complete assessment, as it considers precision changes at different overlap levels.

To achieve better model performance, we applied and tested different data augmentation techniques and their combinations [37,38]. First, we selected a representative subset of molecules from the original dataset and rotated each molecule by 15 degrees until a full rotation is completed. Then, we performed vertical, horizontal, and diagonal flips on each image. This approach allowed us to capture different orientations of the molecules, thus enriching the dataset. We refer to the data augmentation of image rotation and flipping as S1. Next, we applied image blurring, dropout, and elastic transformations as S2, and operations of image hue, saturation, and brightness as S3 (see more details of the augmentation in Table S2).

We then created different datasets by combining these three data augmentation levels and trained the ML models accordingly. The results obtained are shown in Figure 3, from which we realize that that using only S1 for data augmentation results in the [email protected] and [email protected]:0.95 scores of 19.8% and 5.2%, respectively. However, when taking S1 and S3 for data augmentation, the [email protected] and [email protected]:0.95 scores increase to 48% and 24.2%, respectively. Remarkably, the combination of S1 and S2 results in a significant improvement in the [email protected], but the [email protected]:0.95 score is not dramatic. However, when all the three data augmentation levels are applied, we achieve excellent model performance with [email protected] and [email protected]:0.95 scores of 99.6% and 91.1%, respectively.

We also tested different configuration files for training under the same dataset to make the entire model more compatible with our work. The yolov7-tiny-pose.yaml and yolov7-w6-pose.yaml are two different configuration files, and their difference lies in the depth and width of the model. The file yolov7-tiny-pose.yaml is a relatively small model that is suitable for use when computing resources are limited, while yolov7-w6-pose.yaml is a larger model that is suitable for scenarios that require higher accuracy. As displayed in Figure 3b, the configuration file of yolov7-tiny-pose.yaml does not perform as well as yolov7-w6-pose.yaml. Therefore, we used the yolov7-w6-pose.yaml configuration file to build the ML model. Eventually, our model achieves a maximum [email protected] of 99.7% and [email protected]:0.95 of 99.1%, reflecting a high performance for the molecular imaging recognition (Figure 3c).

In addition to achieving the object detection tasks, our model was used to output keypoints of objects [39]. As demonstrated in Figure 4, the keypoints are predicted for each molecule, from which the triangle-shaped molecule 1 and the rectangle-shaped molecule 2 can be easily recognized through drawing the connection between the keypoints to their nearest neighbors. Notably, the entire deep learning process is run directly on a personal computer, and only requires 2–3 h for model training. Furthermore, predicting the STM image and outputting the results takes only a fraction of a second (see more recognition cases in Supplementary Figures S2–S4).

With the predicted keypoints, we can obtain a wealth of information regarding the investigated molecules. For example, we can analyze the symmetry of the molecules. From our prior knowledge, we know that molecule 1 has a three-fold symmetry while molecule 2 is two-fold symmetric. For 1, the connected edges of the predicted keypoints form triangles, whose side lengths can be determined as well as the average and the computed standard deviation. The smaller the standard deviation, the more it indicates that the molecule possesses three-fold symmetry. The results are shown in Figure 5a. For 2, the connected edges using the predicted keypoints establish quadrilaterals, from which the degrees of each interior angle are calculated and their standard deviations from the theoretical orthogonal angles are determined. Again, the smaller the standard deviation, the more it indicates that the molecules have a rectangular shape (shown in Figure 5b). In addition, with the predicted keypoints, we can also determine the actual area of each molecule. Based on the scale in the STM image, the real size of each pixel in the image can be obtained, allowing the calculation of the actual size of the molecules by multiplying it with the occupied pixels surrounded by the keypoints. The results are shown in Figure 5c.

Lastly, we showcase the usefulness of our deep learning framework by analyzing the surface nanostructures of on-surface synthesis [16]. Figure 6a shows a schematic representation of a well-studied polymerized reaction of a bromine-functionalized polycyclic aromatic hydrocarbon on the surface of Au(111), which generates a polyphenylene chain. In Figure 6b, we show a typical STM image after the reaction. This image can be analyzed using two algorithms.

First, since the sample contains molecules before and after the reaction as reflected by their different lengths, we annotated both the reactants and product molecules. After training the model and outputting the predicted labels, we can determine the reaction yield from the image. As indicated in Figure 6c, the red lines suggest molecules without C-C coupling, and the blue lines suggest the polymerized molecules. We can thus calculate the reaction yield simply by dividing the total sizes of the blue and red lines by the size of the blue ones.

The second method involves determining the length of the polymerized chain by annotating the endpoints of the molecules as indicated in Figure 6d. The polymers after the on-surface reaction are commonly not of uniform length, which can be dimers, trimers, or even longer. Therefore, annotating a large number of oligomers is impractical. We annotated the characteristic gaps between the oligomers, allowing our model to predict the endpoints of the polymerized chain. This allows us to determine the number of molecules in a polymer simply by dividing the length of the chain by the length of a single molecule. The potential of this deep learning framework lies in the prediction and identification of reaction products in on-surface synthesis, particularly in complex systems with different molecules. With the detection of the keypoints of target molecules, we can quickly and accurately determine the composition and structure of reaction products, providing valuable information for further analyses.

3. Conclusions

In summary, we developed a deep learning framework based on the YOLOv7 algorithm for the automated analysis of surface nanostructures in STM images. Our computer vision framework introduces keypoints into STM image recognition. By carefully selecting image augmentation techniques, we only need to include limited STM images containing a dozen of molecules for training. In the testing case of a standard STM image, our model achieves an excellent performance of an [email protected] of 99.52% and an [email protected]:0.95 of 99.93%. We demonstrate that this framework not only recognizes different types of molecules but also provides keypoints for each molecule, which can be used for further data analysis. Moreover, this model is lightweight, easy to operate, and highly accurate, making it a potential general solution for the automated detection and analysis of molecular STM images.