LEM-Detector: An Efficient Detector for Photovoltaic Panel Defect Detection

Abstract

Photovoltaic panel defect detection presents significant challenges due to the wide range of defect scales, diverse defect types, and severe background interference, often leading to a high rate of false positives and missed detections. To address these challenges, this paper proposes the LEM-Detector, an efficient end-to-end photovoltaic panel defect detector based on the transformer architecture. To address the low detection accuracy for Crack and Star crack defects and the imbalanced dataset, a novel data augmentation method, the Linear Feature Augmentation (LFA) module, specifically designed for linear features, is introduced. LFA effectively improves model training performance and robustness. Furthermore, the Efficient Feature Enhancement Module (EFEM) is presented to enhance the receptive field, suppress redundant information, and emphasize meaningful features. To handle defects of varying scales, complementary semantic information from different feature layers is leveraged for enhanced feature fusion. A Multi-Scale Multi-Feature Pyramid Network (MMFPN) is employed to selectively aggregate boundary and category information, thereby improving the accuracy of multi-scale target recognition. Experimental results on a large-scale photovoltaic panel dataset demonstrate that the LEM-Detector achieves a detection accuracy of 94.7% for multi-scale defects, outperforming several state-of-the-art methods. This approach effectively addresses the challenges of photovoltaic panel defect detection, paving the way for more reliable and accurate defect identification systems. This research will contribute to the automatic detection of surface defects in industrial production, ultimately enhancing production efficiency.

1. Introduction

Defects in photovoltaic panels pose a significant challenge, not only reducing power generation efficiency but also presenting a fire hazard. Early detection of these defects during the manufacturing process is crucial to prevent defective panels from advancing to subsequent production stages [1]. Photovoltaic panel defects appear as non-luminous dark areas in electroluminescence (EL) imaging, making it possible to detect defects through electroluminescence images of photovoltaic panels.

These defects exhibit wide variation in scale and diversity, ranging from minute cracks to extensive structural damage. Each type of defect presents unique characteristics, further complicating the detection process. Moreover, manufacturing processes and raw materials often introduce a significant number of randomly distributed, variably shaped grain patterns in photovoltaic panel images. These patterns closely resemble defects, creating complex backgrounds that interfere with defect detection. This interference increases the likelihood of false positives and missed detections, posing substantial challenges to the detection task. Figure 1 illustrates common defects found in photovoltaic panels.

Traditional computer vision-based methods have been widely employed for automated defect identification in photovoltaic panel images. These approaches typically extract features from texture, color, shape, and spectral cues using handcrafted feature descriptors, followed by utilizing classifiers for defect detection [2,3]. However, the feature extraction process heavily relies on handcrafted descriptors, which often involves complex parameter configurations and requires significant domain expertise. Moreover, these techniques are typically developed for specific applications, leading to limited generalization and robustness across diverse defect types and scenarios.

Object detection models are broadly categorized into one-stage and two-stage approaches. One-stage models, such as SSD [4] and YOLO [5], prioritize speed and real-time detection. Two-stage models, like R-CNN [6] and Faster R-CNN [7], emphasize accuracy but often come at the cost of computational efficiency. While R-CNN variants excel in precision, they are resource-intensive, requiring significant memory and processing power. SSD, although faster, struggles with detecting smaller objects and lags behind YOLO in real-time performance.

The YOLO family of models demonstrates a compelling balance between accuracy and speed. However, their reliance on Non-Maximum Suppression [8] for eliminating redundant detections can negatively impact both speed and accuracy. The DETR series [9], on the other hand, eliminates Non-Maximum Suppression altogether, improving both efficiency and accuracy. However, DETR models necessitate substantial computational resources.

RT-DETR [10] addresses these challenges by efficiently processing multi-scale features and introducing IoU-aware query selection, which provides the decoder with more accurate initial object queries. This approach surpasses existing real-time detectors in both accuracy and speed, while eliminating the need for post-processing, leading to a stable and delay-free inference process.

This paper presents an efficient end-to-end detector for photovoltaic panel defect detection, the LEM-Detector, drawing inspiration from the advancements of RT-DETR. The proposed approach addresses the challenges of imbalanced training data and effective feature extraction for multi-scale defect detection.

To tackle the issue of imbalanced training samples, a novel Linear Feature Augmentation (LFA) module is introduced. LFA utilizes the cosine similarity function as a triggering threshold to augment linear defect samples, thereby enhancing the model’s accuracy and robustness.

Furthermore, an Efficient Feature Enhancement Module (EFEM) is proposed to effectively capture subtle texture information associated with defects. The EFEM utilizes dynamic convolutional kernels based on feature weight analysis to improve extraction efficiency. Long-range spatial dependencies are captured by the module, enabling the selection of relevant edge information while suppressing redundant features. Group Normalization and one-dimensional large kernel convolutions further enhance the generalization capability of the EFEM, enabling accurate feature capture.

To address the issue of information loss during feature fusion, a Multi-Scale Multi-Feature Fusion Pyramid Network (MMFPN) is employed. The MMFPN leverages a Complementary Information Aggregation module (CIA), focusing on clearer boundary information in shallow features and richer category information in deep features. This bidirectional fusion mechanism between high- and low-resolution features improves information transfer, enhancing multi-scale feature fusion and boosting the accuracy of multi-scale object recognition.

The contributions of this paper are summarized as follows:

(1)

Novel LFA module: Addresses imbalanced training data through cosine similarity-based augmentation, improving detection accuracy and robustness.

(2)

Efficient EFEM module: Captures subtle texture information and enhances feature representation through dynamic convolutions, long-range dependency analysis, GN and large kernel convolutions.

(3)

MMFPN architecture: Enhances multi-scale feature fusion by combining complementary information from different feature levels, improving information transfer and accuracy.

The remainder of this paper is structured as follows. Section 2 provides a review of existing works related to photovoltaic panel defect detection. Section 3 delves into the methodology employed in this study, outlining the proposed model and its components in detail. Section 4 presents a comprehensive analysis of the proposed model through comparative and ablation experiments, demonstrating its effectiveness and performance. Section 5 concludes the paper, summarizing the key findings and contributions of the research.

2. Related Work

2.1. Data Augmentation

Data augmentation is a widely used technique in computer vision [11], employed to generate synthetic training samples from existing data. By applying various transformations and processing, data augmentation increases the diversity and quantity of datasets, ultimately enhancing the model’s generalization ability and robustness [12,13].

Data augmentation methods encompass a broad spectrum, ranging from traditional image processing techniques to model-based approaches. Traditional methods include transformations such as flipping, rotation, scaling, translation, cropping, color manipulation, noise addition, blurring, and affine transformations. Model-based approaches, on the other hand, leverage techniques like Generative Adversarial Networks [14], Variational Autoencoders (VAEs) [15], and adversarial sample enhancement [16].

Prior research has explored the application of data augmentation in photovoltaic panel defect detection. Tang et al. [17] proposed an automatic classification method for monocrystalline silicon solar cell defects that utilizes geometric morphological features for data augmentation, requiring only a limited number of training samples. Jain et al. [18] employed Generative Adversarial Networks to generate synthetic images of surface defects, training the generator to synthesize new defect images based on random noise. Wei et al. [19] utilized a linear weighted fusion method for fabric defect image reconstruction, producing a residual map that highlights defects and facilitates pixel-level detection of various defect types. Wang et al. [20] addressed the challenges of limited training data and data imbalance in photovoltaic panel detection models through data augmentation and weighted classification methods, demonstrating significant performance improvements.

These studies collectively highlight the significant impact of data augmentation methods in enhancing the detection performance of photovoltaic panels, emphasizing the importance of diversifying and expanding training data for robust and accurate defect detection.

2.2. Feature Fusion Methods

Deep learning approaches for photovoltaic panel defect detection face challenges stemming from the wide variation in defect scales. Single-layer features often lack sufficient multi-scale information, hindering accurate detection of defects across different sizes. Consequently, enhancing model robustness to scale variations and accurately detecting multi-scale defects is crucial.

Feature Pyramid Networks (FPNs) [21] have gained widespread adoption in object detection networks due to their multi-layer feature fusion mechanism, which improves multi-scale representation capabilities [22,23]. However, the reduction in feature channels and layer-by-layer transmission methods within FPNs can lead to information loss, particularly affecting the detection of small targets and limiting accuracy for multi-scale object detection [24].

Various approaches have been proposed to address these challenges. Liu et al. [25] combined CNNs with a feature pyramid structure in a Faster R-CNN-based solar cell defect detection method, enhancing robustness to defect scales by capturing and fusing multi-scale feature information. Su et al. [26] integrated attention mechanisms with bidirectional feature pyramids to improve detection accuracy and model robustness. Tang et al. [27] focused on local and edge information during feature fusion, improving accuracy in medical image segmentation. Zhou et al. [28] proposed a dual pyramid network that further integrates multi-scale features, allowing different feature layers to share similar semantic characteristics, enabling precise localization and classification. Zhao et al. [10] focused on fusing higher-level features with richer semantic concepts.

RT-DETR, specifically, utilizes an Attention-based Intra-scale Feature Interaction (AIFI) module for intra-scale interactions in high-level features, reducing computational redundancy. Feature fusion is achieved through a Cross-scale Feature Fusion (CCFF). While this approach reduces computational complexity and memory usage, it can result in some information loss. This is particularly detrimental for photovoltaic panel defect detection, as these images often contain many small-scale defects that occupy few pixels and carry limited information. This fusion method leads to poor feature representation for small-scale defects, negatively impacting model accuracy.

To address this limitation, we propose a multi-scale multi-feature fusion approach during the feature fusion process. This approach maximizes the retention of small-scale information, effectively enhancing the multi-scale detection performance of photovoltaic panels.

2.3. Detection Methods Incorporating Attention Mechanisms

Attention mechanisms are widely employed in computer vision as a means of adaptively learning weight coefficients that emphasize regions of interest while suppressing irrelevant background information. This selective focus enhances model performance and efficiency. In computer vision, common attention mechanisms include Channel Attention [29], Spatial Attention [30], and Self-Attention [31]. These mechanisms selectively focus on specific areas across different dimensions, effectively highlighting target regions and improving the model’s ability to extract relevant features.

Attention mechanisms operate by adjusting weights across different dimensions to highlight relevant information. Channel Attention mechanisms focus on feature dimensions, emphasizing channels that contribute most to object classification by adjusting their weights. Spatial Attention mechanisms focus on spatial dimensions, learning weights for each position to highlight areas crucial for object classification. Self-Attention mechanisms, designed for sequential data, consider the relationships between elements and adjust their weights to emphasize those of significant importance. While these mechanisms differ in their implementation, their common goal is to enhance model performance.

Hou et al. [32] proposed a novel attention mechanism called Coordinate Attention (CA). CA integrates both channel information and directional positional information, establishing global connections to facilitate improved defect detection. Xu et al. [33], recognizing limitations in CA, identified the detrimental effects of dimensionality reduction on channel attention and the complexity of the attention generation process. To address these challenges, they proposed a combination of one-dimensional convolution and Group Normalization feature enhancement techniques, demonstrating enhanced performance and generalization capabilities.

Photovoltaic panel images often contain a large number of grains, a consequence of manufacturing processes and raw materials. These grains are randomly distributed and exhibit variations in scale and shape, presenting significant challenges for defect detection. Su et al. [34] introduced a Region Proposal Attention Network to address this issue, employing a novel approach that combines channel and spatial attention. This network refines the feature maps extracted by Convolutional Neural Networks (CNNs), enhancing features within defect areas while suppressing background noise. This strategy significantly improves classification and detection performance, demonstrating the effectiveness of attention-based feature processing.

However, the close resemblance between some grains and defects can lead to misclassification and a high false detection rate. Additionally, information related to small-scale defects may be lost during feature extraction, impacting detection effectiveness. Precise feature extraction amidst complex background interference remains a crucial challenge.

Zhang et al. [35] addressed this challenge by proposing receptive field attention. This approach not only focuses on spatial features within the receptive field but also provides effective attention weights for convolutional kernels, effectively addressing the issue of parameter sharing in standard convolutions. By selectively enhancing key features during the feature extraction phase, rather than optimizing features after extraction, this method improves model performance and accuracy.

In summary, the primary challenges faced in photovoltaic panel defect detection include the following: (1) limited and imbalanced defect samples in the dataset, (2) strong background interference, making it easy to overlook small-scale features during feature extraction, leading to false positives and missed detections, and (3) poor defect recognition capabilities with insufficient localization accuracy. These issues affect the stability of automated defect detection in industrial products and hold value for further research.

3. Methods

The LEM-Detector comprises four key modules: an LFA, a Backbone with an integrated EFEM, an MMFPN, and a transformer decoder with auxiliary prediction heads. Figure 2 provides a visual representation of the LEM-Detector framework.

The input image first passes through the LFA module before entering the Backbone, which incorporates the EFEM. The Backbone sequentially processes the image through four blocks, extracting features denoted as $S_2$, $S_3$, $S_4$, and $S_5$. These extracted features are then input into the encoder.

The encoder begins by processing $S_5$ through an AIFI module. Subsequently, $S_5$ is fused with the other feature layers ($S_2$, $S_3$, and $S_4$) in a multi-scale multi-feature fusion process. A minimum uncertainty query selection method is then applied to identify a fixed number of encoder features as the initial object queries for the decoder.

Finally, the decoder iteratively refines these object queries using auxiliary prediction heads to generate class labels and bounding box predictions. This iterative optimization process enables the model to accurately identify and locate objects within the image.

This section presents a detailed exploration of the LEM-Detector’s key components, beginning with the LFA module. We then delve into the EFEM, followed by a comprehensive discussion of the MMFPN architecture. This discussion highlights how these modules work together to facilitate effective multi-scale detection.

3.1. Linear Feature Augmentation (LFA)

The photovoltaic panel dataset encompasses both large-scale global defects and small-scale local defects with linear texture features. Large-scale defects, such as Black core, Horizontal dislocation, Vertical dislocation, and Short circuit, exhibit distinct and easily recognizable features, leading to higher detection accuracy. In contrast, small-scale defects, including Crack, Star crack, and Finger, pose significant challenges due to their high similarity to the background, complex and variable shapes, and scattered nature with strong background interference. These factors contribute to lower detection accuracy and hinder model performance improvement.

Furthermore, the dataset exhibits an imbalance in the distribution of samples, with a relative scarcity of small-scale linear texture defects. While data augmentation can modestly enhance the detection accuracy of Crack and Star crack defects, random processing of darker regions may negatively impact the accuracy of other easily recognizable defect types. Additionally, improper data augmentation can lead to overfitting, reducing model accuracy and consuming substantial computational resources, thereby lowering efficiency.

To address these challenges, a targeted approach is employed, leveraging the distribution characteristics of defect samples. A cosine similarity function is utilized to filter linear texture training samples with similar features. Subsequently, targeted data augmentation is applied specifically to these similar linear texture features, effectively addressing the issue of data imbalance and enhancing the model’s ability to detect small-scale defects while minimizing negative impacts on the detection of other defect types.

The cosine similarity function is a commonly used method for measuring the angle between two vectors. This similarity metric is achieved by calculating the cosine of the angle between the two vectors, which can be expressed mathematically as follows:

$$C={\displaystyle \frac{A\cdot B}{\parallel A\parallel \parallel B\parallel }}={\displaystyle \frac{\sum _{i=1}^n A_i{B}_i}{\sqrt{\sum _{i=1}^n A_i^2}\sqrt{\sum _{i=1}^n B_i^2}}}$$

(1)

The cosine similarity $C$ is defined as the cosine of the angle between two vectors, where $A$ and $B$ denote the dot product of the vectors, and $A$·$B$ are the magnitudes of the vectors ‖$A$‖ and ‖$B$‖, respectively. The advantage of cosine similarity is that it considers only the direction of the vectors, without being affected by vector length.

During data augmentation, cosine similarity is calculated between training samples to select suitable candidates for augmentation. Samples with a cosine similarity exceeding a threshold δ, indicating similar features, are selected for augmentation. This approach preserves the crucial characteristics of linear defects while increasing the diversity of the training set, without significantly altering the original features. As depicted in Figure 3, if a training sample has at least one remaining sample with a cosine similarity exceeding δ, it proceeds to the next step of data augmentation; otherwise, no augmentation is performed.

The LFA incorporates a range of contemporary data augmentation techniques, including Affine, BboxSafeRandomCrop, Mosaic, Erase, and Mixup. These methods enhance the diversity of linear features in the photovoltaic panel dataset, simulating the impact of dark spot regions and increasing the complexity of the dataset. This enhanced data diversity improves the robustness of the detection model.

The LFA was applied to the PVEL-AD [36] dataset, ensuring that the generated samples remain sufficiently close to the real samples within the vector space. This approach enhances the diversity of training samples while preserving the actual data distribution, enabling the selection of appropriate linear feature training samples for targeted data augmentation. By focusing augmentation on specific, relevant samples, computational resources are utilized more efficiently, leading to improved training performance and model robustness. The impact of this data augmentation strategy is visually illustrated in Figure 4.

3.2. Efficient Feature Enhancement Module (EFEM)

The EFEM comprises four structurally similar blocks. Each block employs Group Convolution with a 7 × 7 kernel to process the primary feature information extracted from the image samples, optimizing computational efficiency and reducing memory overhead. Normalization and a ReLU activation function are then applied, enabling the model to capture more complex features and improve convergence speed.

To capture global information and establish long-range dependencies, the dimensions of the sample images are adjusted, and global average pooling is applied separately in both the height and width directions. To avoid information mixing, the height and width features are not directly concatenated. Instead, 1D convolutional kernels of 5 × 5 are used to extract richer feature information in both directions.

Given the limited training samples and potential memory access issues, Group Normalization is employed to normalize feature maps. Group Normalization divides channels into multiple groups, computing the mean and variance within each group. This approach mitigates high false detection rates when using small mini-batches, enhancing model stability.

Finally, the processed features are passed through a sigmoid function to select and amplify important features, allowing the model to focus on critical areas and assign appropriate weights to the feature maps. The specific content is shown in Figure 5.

The input feature information is represented as $X\in R^{C\times H\times W}$, where $C$, $H$, and $W$ denote the channel, height, and width dimensions. The input $X$ undergoes Group Convolution, which groups the different feature maps from the input layer. Different convolution kernels are then applied to each group, thereby reducing the computational load of the convolution process. After processing, K groups of convolution information are obtained as $X_1\in R^{C{K}^2\times H\times W}$, where $K$ represents the number of grouped convolution kernels. After group normalization and ReLU activation, normalizing the input data alleviates the variations in inputs between layers in deep neural networks, resulting in more stable training. The activation function enables the neural network to fit and express more complex features.

Next, we adjust the size of $X_2\in R^{C\times KH\times KW}$ for subsequent feature extraction. $z_c^h$ and $z_c^w$ represent the global average pooling in the vertical and horizontal directions, respectively. Two pooling kernels, sized $KH\times 1$ and $1\times KW$, are used to perform one-dimensional feature encoding on the height and width of the input feature map, yielding feature outputs in both directions. The specifics are as follows:

$$z_c^h\left(h\right)={\displaystyle \frac{1}{H}}{\sum }_{0\le i<H} x_c\left(h,i\right)$$

(2)

$$z_c^w\left(w\right)={\displaystyle \frac{1}{W}}{\sum }_{0\le j<W} x_c\left(j,w\right)$$

(3)

To better capture the global information in the images and enhance the model’s ability to focus on critical regions, we designed a more efficient processing method. This approach preserves more small-scale defect information, preventing the mixing of irrelevant information. We processed the height and width positional information extracted from $z_c^h$ and $z_c^w$ separately. We employed a 5 × 5 one-dimensional convolution to enhance the positional information in both the vertical and horizontal directions. This large kernel convolution effectively processes the height and width directions, allowing for a more detailed capture of spatial information and enhancing feature expression capabilities, while also capturing long-distance spatial dependencies. This approach achieves a balance between detection accuracy and parameter count. Subsequently, we utilized Group Normalization, which is less affected by mini-batch size, making it more suitable for small sample training, to process the enhanced positional information. The representations of position attention in the height and width directions were obtained as follows:

$$y^h=\sigma \left(G_n\left(F_h\left(z_h\right)\right)\right)$$

(4)

$$y^w=\sigma \left(G_n\left(F_w\left(z_w\right)\right)\right)$$

(5)

Let $y^h$ and $y^w$ represent the feature weight maps in the vertical and horizontal directions, respectively. $\sigma$ denotes the sigmoid activation function, and the one-dimensional convolutions are represented as $F_h$ and $F_w$. We set the convolution kernels for $F_h$ and $F_w$ to 5. Finally, we multiply the original input feature map $X$ by the two weight maps to obtain the final output, denoted as $Y$:

$$Y=x_c\times y^h\times y^w$$

(6)

The output $Y$ is capable of more accurately capturing the positional relationships in both vertical and horizontal directions, thereby enhancing meaningful features while suppressing redundant or irrelevant ones. This allows the network to effectively capture precise local information within the data, improving its ability to represent complex patterns and making it more conducive to the detection of small-scale defects.

3.3. Multi-Scale Multi-Feature Fusion Pyramid Network (MMFPN)

As the network depth increases, the features corresponding to small-scale defects often become diminished due to the limited number of pixels they occupy in the image, leading to a loss of information and potential neglect during detection, which adversely affects detection accuracy. To address this issue and re-correct the selection of easily overlooked small-scale information, we propose the MMFPN to integrate the feature information $S_2$, $S_3$, $S_4$, and $S_5$ extracted by the Backbone.

Initially, the high-level feature $S_5$ undergoes intra-scale feature interaction via the AIFI module, and then it is combined with the features $S_2$, $S_3$, and $S_4$ in the MMFPN. The process begins with top-down feature fusion, wherein the high-level feature maps, which contain rich global semantic information but have low spatial resolution, are upsampled and fused with the low-level feature maps. This process enriches the spatial information of high-level feature maps, significantly improving the detection performance of multi-scale targets, especially small objects. The generated fused features $P_1$, $P_2$, and $P_3$ are obtained through the fusion process as described in Equation (7), where $P_i$ represents the fused feature information, and $P_i=CIA(X_i,X_j)$ denotes the Complementary Information Aggregation module (CIA), with $X_i,$ and $X_j$ being the input feature information. Subsequently, three additional bottom-up feature fusions occur, wherein the high spatial resolution but semantically sparse information from the low-level feature maps is downsampled and fused with the high-level feature maps. This enhances the semantic information of the low-level feature maps, contributing to more accurate classification and generating the fused features $P_4$, $P_5$, and $P_6$. The fusion process is described in Equation (8). During feature fusion, the CIA module captures easily overlooked detailed information by receiving adjacent high-level semantic and low-level semantic information. It introduces a bidirectional fusion mechanism between high-resolution and low-resolution features, ensuring more effective information transfer between features, which further enhances the results of multi-scale feature fusion. Finally, the features $P_3$, $P_4$, $P_5$, and $P_6$ are input into the Decoder for precise classification and localization, as shown in Equation (9).

$$P_1=CIA(S_5,S_4) P_2=CIA(P_1,S_3) P_3=CIA(P_2,S_2)$$

(7)

$$P_4=CIA(P_3,P_2) P_5=CIA(P_4,P_1) P_6=CIA(P_5,S_5)$$

(8)

$$P_{out}=(P_3,P_4,P_5,P_6)$$

(9)

In terms of the fusion method, our proposed CIA module focuses on the complementary semantic information within feature layers. By facilitating cross-layer interactions, it learns the semantic dependencies between multi-scale channel feature mappings, effectively integrating clearer boundary information from shallow features with richer categorical information from deep features. This approach preserves the complementary information from different levels of semantic data, thereby achieving more accurate classification and localization. The specific structure of the CIA is illustrated in Figure 6.

To avoid redundancy and inconsistencies that may arise from directly merging semantic information from different feature levels, the CIA selectively aggregates boundary and semantic information. Shallow and deep feature representations, denoted as $f_{\mathrm{L}}$ and $f_{\mathrm{H}}$, are input into the module and are adjusted to a unified size using 1 × 1 convolution.

$$F_1=f_{\mathrm{L}}+f_{\mathrm{L}}\cdot \sigma \left(f_{\mathrm{L}}\right)+(1-\sigma (f_{\mathrm{L}}\left)\right)\cdot \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(\sigma \right(f_{\mathrm{H}})\cdot f_{\mathrm{H}})$$

(10)

$$F_2=f_{\mathrm{H}}+f_{\mathrm{H}}\cdot \sigma \left(f_{\mathrm{H}}\right)+(1-\sigma (f_{\mathrm{H}}\left)\right)\cdot \mathrm{U}\mathrm{p}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\left(\sigma \right(f_{\mathrm{L}})\cdot f_{\mathrm{L}})$$

(11)

$F_1$ and $F_2$ represent the complementary feature information after aggregating shallow and deep features, respectively, while $\sigma$ denotes the sigmoid function. The processed feature information is then fused to compensate for the loss of spatial boundary information in the high-level features and the lack of global category information in the low-level features. Finally, after a 3 × 3 convolution, the two recalibrated feature representations are concatenated for output, as shown in Equation (12):

$$F(F_1, F_2)=\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(GN\right(\mathrm{c}\mathrm{a}\mathrm{t}(F_1,F_2\left)\right))$$

(12)

Here, $\mathrm{cat}(·)$ denotes the concatenation operation along the channel dimension, and $\mathrm{F}\left(F_1, F_2\right)$ represents the fused feature information after applying Group Normalization and the ReLU activation function. This process achieves a robust combination of different features while refining coarse features, ultimately enabling accurate and efficient identification of defects across varying scales.

4. Experiments

This section details the experimental procedures employed in this study, encompassing dataset construction, evaluation metrics, implementation details, experimental assessment, interpretability analysis, and ablation studies. To enhance the clarity and intuitive understanding of our findings, feature map visualization is employed as a supplementary tool to support the experimental results.

4.1. Dataset

The dataset used in this study is the PVEL-AD dataset, collected and created by Hebei University of Technology. This dataset comprises 4500 images representing twelve common defect types in photovoltaic panels: Crack, Finger, Black core, Thick line, Star crack, Corner, Fragment, Scratch, Horizontal dislocation, Vertical dislocation, Printing error, and Short circuit. Each image is annotated with bounding boxes and class labels, specifying the class of the defect and the coordinates of the bounding box’s top-left and bottom-right corners. All images have a resolution of 1024 × 1024 pixels.

Due to the insufficient number of defect instances for the classes Corner, Fragment, Scratch, and Printing error (each with fewer than 32 instances), these four classes were deemed insignificant for experimentation and were removed from the dataset. The remaining eight defect types were retained for further analysis. The dataset was then split into training and testing sets at a ratio of 7:3, resulting in 3155 images for training and 1345 images for testing.

Table 1. Number and distribution of defect instances.

	Ck	Fr	Bc	Tl	Sc	Hd	Vd	St
Training	876	2041	731	695	95	555	97	350
Testing	384	917	297	286	40	243	40	142
Total	1260	2958	1028	981	135	795	137	492

Note: Crack (Ck), Finger (Fr), Black core (Bc), Thick line (Tl), Star crack (Sc), Horizontal dislocation (Hd), Vertical dislocation (Vd), and Short circuit (St).

provides a detailed breakdown of the number and distribution of defect instances for the eight retained defect types in both the training and testing sets.

Following the application of the LFA, an additional 8761 training samples were generated. This augmentation strategy aimed to enhance the representation of linear texture features within the training data, further improving the model’s ability to detect small-scale defects.

4.2. Evaluation Metrics

In region-level defect detection tasks, an objective evaluation of model performance requires considering both classification and localization accuracy. Typically, Precision, Recall, and mean Average Precision (mAP) are employed to assess the accuracy of object detection. mAP represents the mean across all categories, while Precision is defined as the proportion of true defect samples among all detected defect samples. Recall, on the other hand, indicates the proportion of true defect samples among all actual defect samples. The formulas for each metric are provided as Equations (13)–(16):

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(13)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(14)

$$AP={\int }_0^1 PdR$$

(15)

$$mAP={\displaystyle \frac{\sum _{i=1}^N {\mathrm{A}\mathrm{P}}_i}{N}}$$

(16)

Additionally, Frames Per Second (FPS), Giga Floating-point Operations Per Second (GFLOPS), and the number of parameters are utilized to evaluate the overall performance of the model, where TP, FP, and FN represent the number of correctly identified targets (true positives), the number of incorrectly identified targets (false positives), and the number of actual targets that were not detected (false negatives), respectively. Average Precision (AP) refers to the mean precision of the detection algorithm at different recall levels. The P-R curve is typically used for this calculation, where $S_{P\text{-}R}$ denotes the area enclosed by the P-R curve and the axes. The computation method is illustrated by Equation (17), as shown below:

$${AP=S}_{P\text{-}R}$$

(17)

In addition, we introduce the TIDE [37] metric as an additional means of analyzing model performance. In object detection, the primary metric used to assess performance is the mean Average Precision (mAP). However, since a false positive (FP) can arise from various issues such as duplicate detection, misclassification, incorrect localization, confusion with the background, or a combination of these factors, it is not straightforward to analyze detection errors solely through mAP. Similarly, a false negative (FN) may represent a completely missed ground truth (GT), or a potentially correct prediction that is merely misclassified or mislocalized. These error types can significantly impact mAP in different ways. Therefore, we utilize the TIDE evaluation metric to provide insight into our model’s superior performance. TIDE categorizes error types into six categories: classification errors (Cls), localization errors (Loc), both classification and localization errors (Both), duplicate detection errors (Dupe), background errors (Bkg), and missed GT errors (Miss), along with the counts of FPs and FNs.

4.3. Implementation Details

The experimental platform runs on the Ubuntu 20.04.6 LTS operating system, featuring an Intel Xeon CPU (Intel, Santa Clara, CA, USA) and four Tesla V100 GPUs (Tesla, Austin, TX, USA) (32 GB Memory each). The software environment includes CUDA 11.4 and CUDNN 8.0.5. All code is developed using the PyTorch 1.10.0 deep learning framework. Both training and testing are conducted on a single GPU. Images are uniformly resized to 640 × 640 pixels. During training, a stochastic gradient descent optimizer is utilized for parameter optimization, with a fixed weight decay of 0.0001 and a momentum of 0.9. The initial learning rate is set to 0.0001. The number of epochs and batch size are set to 150 and 8, respectively. The trigger threshold δ for LFA is set to 0.7. The trained models do not use pre-trained weights.

4.4. The Result of Defect Detection

The performance of the proposed LEM-Detector is compared to related methods in Figure 7. The LEM-Detector achieves a mean Average Precision (mAP) of 94.7%, representing a 3.4 percentage point improvement over RT-DETR. Compared to Faster R-CNN and RetinaNet [38], our model achieves an accuracy improvement of over 8 percentage points while utilizing fewer parameters. When compared to the latest YOLOv10 [39], which has a similar number of parameters, our model surpasses YOLOv10 by 6% in terms of mAP. These results highlight the significant performance gains achieved by the LEM-Detector, demonstrating its effectiveness in addressing the challenges of multi-scale defect detection in photovoltaic panels.

Table 2. Comparison with advanced object detection detectors (all object detectors were trained starting from the smallest model until they were able to detect all defect types.).

Method	[email protected] (%)	Parameter (M)	GFLOPs	FPS
Faster-RCNN	85.8	261.4	137.0	28
RetinaNet	86.5	190.9	98.0	32
Deformable-DETR	90.1	40.1	126.0	37
DAB-DETR	90.6	44.0	94.0	31
YOLOv8n	89.6	2.9	8.1	88
RT-DETR	91.3	18.9	57.0	260
YOLOv9c	88.7	24.7	102.1	61
YOLOv10x	89.9	28.0	160.0	54
LEM-Detector (ours)	94.7	33.8	67.2	60

provides a detailed view of the superior performance of the LEM-Detector. Compared to existing methods, the LEM-Detector, utilizing LFA, EFEM, and MMFPN, achieves an optimal balance among model size, detection speed, and GFLOPs while ensuring top detection accuracy, making it especially suitable for industrial inspection applications.

Table 3. [email protected] for different defect types.

Method	Ck	Fr	Bc	Tl	Sc	Hd	Vd	St	[email protected]
Faster-RCNN	59.8	87.1	95.5	78.7	71.0	99.5	95.5	99.5	85.8
RetinaNet	59.3	88.8	97.7	81.2	68.8	99.5	97.5	99.5	86.5
Deformable-DETR	73.2	93.6	99.1	89.8	81.1	96.9	97.7	99.5	90.1
DAB-DETR	70.7	87.3	98.5	89.6	82.7	99.5	97.6	99.5	90.6
YOLOv8n	72.8	93.8	99.2	84.1	83.0	98.0	87.2	99.2	89.6
RE-DETR	72.6	94.5	98.9	90.0	82.7	99.5	92.7	99.5	91.3
YOLOv9c	68.0	93.1	98.4	83.1	76.3	98.4	92.7	99.5	88.7
YOLOv10x	72.0	91.5	98.5	84.4	79.1	99.5	95.8	98.6	89.9
LEM-Detector (ours)	84.7	94.3	98.8	91.1	92.0	99.5	97.7	99.5	94.7

Note: Crack (Ck), Finger (Fr), Black core (Bc), Thick line (Tl), Star crack (Sc), Horizontal dislocation (Hd), Vertical dislocation (Vd), and Short circuit (St).

further reveals significant performance gains for the Crack and Star crack defect types, which typically exhibit lower detection accuracy. This improvement is attributed to the integration of the LFA specifically designed for linear crack data. The LFA creates a more realistic dataset with more evenly distributed training samples, enhancing the model’s ability to accurately identify these challenging defects.

The combination of the EFEM and the Multi-Scale Multi-Feature Fusion Pyramid Network MMFPN ensures that each feature layer retains rich scale information while preserving defect feature information outside the candidate boxes. This approach reduces information coupling and enhances edge information, leading to improved detection accuracy. The LEM-Detector’s high detection accuracy further validates the effectiveness of the LFA, EFEM, and MMFPN in multi-scale defect detection of photovoltaic panel images. Notably, the LEM-Detector surpasses Faster R-CNN by 8.9 percentage points in terms of mAP, showcasing the most significant improvement compared to other methods.

The P-R curve in Figure 8 provides a visual representation of the LEM-Detector’s detection performance for each defect type. The area enclosed by each curve and the coordinate axes represents the Average Precision (AP) value for the corresponding defect type.

The P-R curves for Crack and Star crack defects demonstrate a significant performance gap compared to other defect types, highlighting these defects as challenging samples that hinder model performance improvement. This challenge stems from the high similarity of these defects to the background, their complex and variable shapes, and their dispersed nature with substantial background interference, all contributing to increased detection difficulty.

In contrast, the detection performance for Black core and Horizontal dislocation defects is superior. These defects are often large-scale global defects with distinct features, making them relatively easier to detect. This observation further underscores the challenges presented by small-scale, linear defects like Crack and Star crack, and the need for specialized approaches to enhance their detection accuracy.

Figure 9 presents several visual examples of the LEM-Detector’s performance in detecting defects. The figure showcases the model’s effectiveness in identifying defects across various scales. The high threshold for detected defects demonstrates the method’s sensitivity to defect presence. Even in images containing multiple defect types, the LEM-Detector maintains its effectiveness. For example, in the second column of the first row, the model successfully identifies overlapping defects, demonstrating its ability to distinguish individual defects within complex scenarios. These results provide strong evidence of the LEM-Detector’s exceptional performance in detecting multi-scale defects in photovoltaic panels.

4.5. Ablation Studies

We perform ablation experiments on different components of the proposed LEM-Detector, LFA, EFEM, and MMFPN, to verify the effectiveness of those components.

From

Table 4. The results of the LEM-Detector with different components on the PVEL-AD dataset.

LFA	EFEM	MMFPN	[email protected] (%)	Ck	Fr	Bc	Tl	Sc	Hd	Vd	St
			91.3	72.6	94.5	98.9	90.0	82.7	99.5	92.7	99.5
√			92.5	78.2	95.0	99.3	91.5	86.4	99.5	90.8	99.5
	√		92.2	77.9	94.2	98.8	89.3	85.2	99.4	93.0	99.5
		√	92.6	76.4	93.9	99.1	90.5	84.7	99.5	98.4	98.6
√	√		93.1	79.6	94.5	99.4	90.3	86.2	99.5	96.1	99.5
√		√	92.9	78.7	94.4	99.3	90.9	86.9	99.4	94.5	99.5
	√	√	92.8	76.9	94.3	98.3	88.9	88.5	99.5	97.1	99.5
√	√	√	94.7	84.7	94.3	98.8	91.1	92.0	99.5	97.7	99.5

Note: Crack (Ck), Finger (Fr), Black core (Bc), Thick line (Tl), Star crack (Sc), Horizontal dislocation (Hd), Vertical dislocation (Vd), and Short circuit (St).

, it is evident that the use of LFA for enhancing linear features, the EFEM for accurately extracting multi-scale feature information, and the MMFPN module for capturing often-overlooked edge information significantly improves detection accuracy. The overall mAP reaches 94.7%, with each module contributing to enhanced experimental precision. Notably, the detection results for Crack and Star crack types show substantial improvements, with accuracy increases of 12.1 and 9.3 percentage points, respectively, compared to RT-DETR, effectively demonstrating the detection efficacy of the LEM-Detector.

Using the TIDE evaluation metric, as shown in

Table 5. Distribution of error types.

Method	[email protected]	Cls	Loc	Both	Dupe	Bkg	Miss	FP	FN
RT-DETR	91.30	0.48	1.67	0.15	0.43	2.56	0.14	6.84	1.29
LEM-Detector	94.70	0.55	1.02	0.18	0.13	1.99	0.00	4.40	0.66
Improvement	+3.40	−0.07	+0.65	−0.03	+0.30	+0.57	+0.14	+2.44	+0.63

, it is evident that our detection model exhibits significant enhancements in various aspects, particularly in localization errors (Loc), duplicate detection errors (Dupe), background errors (Bkg), and missed ground truths (Miss). The model’s performance in terms of false positives (FPs) and false negatives (FNs) is also markedly superior to that of the RT-DETR model, with only minor deficiencies in classification errors (Cls) and errors involving both classification and localization (Both). Notably, substantial improvements are observed in localization (Loc) and background (Bkg) errors, attributed to our precise retention of defect boundary information during feature fusion. The MMFPN effectively reduces the risk of overfitting by integrating complementary information from different feature layers, enhancing the model’s generalization capabilities. Importantly, our model did not exhibit any missed GT errors (Miss), indicating that the EFEM effectively captures small-scale defect information without omissions, thereby improving the model’s performance in multi-scale defect detection.

4.5.1. The Effect of LFA

Table 6. Ablation experiment of LFA (DA denotes the use of the same data augmentation methods as in LFA, but without processing through the cosine similarity threshold.).

Method	[email protected]	Sample Size	Ck	Fr	Bc	Tl	Sc	Hd	Vd	St
RT-DETR	91.3	3151	72.6	94.5	98.9	90.0	82.7	99.5	92.7	99.5
RT-DETR + DA	91.7	12,470	77.6	93.6	98.4	91.3	86.5	99.5	87.4	99.5
RT-DETR + LFA	92.5	8761	78.2	95.0	99.3	91.5	86.4	99.5	90.8	99.5

Note: Crack (Ck), Finger (Fr), Black core (Bc), Thick line (Tl), Star crack (Sc), Horizontal dislocation (Hd), Vertical dislocation (Vd), and Short circuit (St).

presents the results of the ablation experiments, which clearly support our analysis of the dataset. From the second row of Table 6, it can be seen that using the same data augmentation techniques without the LFA, the mean Average Precision (mAP) is 91.7%, showing a slight improvement over RT-DETR’s mAP of 91.3%. However, there is no significant enhancement in detection accuracy for the following defect types: Finger, Black core, Horizontal dislocation, Vertical dislocation, and Short circuit. In fact, the detection precision for Finger and Vertical dislocation decreases. This is primarily because, apart from the Crack and Star crack defect types, the other six types have easily recognizable features and larger scales, achieving accuracy levels of 90% or higher. Moreover, due to the inherent challenges in achieving uniformity in manual annotations during dataset creation, the potential for improvement is quite limited. Our proposed LFA effectively addresses this issue by focusing on targeted data augmentation for linear features, resulting in an improved mAP of 92.5%. Additionally, compared to the traditional data augmentation approach that generated 12,470 training images, our LFA method utilizes only 8761 training images, thereby enhancing training speed and minimizing resource wastage.

4.5.2. The Effect of EFEM

In contrast to the BasicBlock module used for feature extraction in ResNet, which can be insensitive to information variations due to shared parameters in convolutional operations, the proposed EFEM effectively addresses this limitation. By emphasizing the importance of different features within the receptive field and prioritizing spatial characteristics, EFEM enriches the diversity of feature expression. It selectively enhances meaningful features while suppressing redundant or irrelevant ones, enabling the network to capture precise local information and represent complex features more effectively. This approach incurs negligible computational costs and parameter increments, significantly enhancing network performance. The improved results are presented in

Table 7. Ablation experimental results of EFEM.

Backbone	[email protected] (%)	Parameter (M)	GFLOPs	FPS
BasicBlock	91.3	10.6	20.1	271
EFEM	92.2	10.9	20.9	264

.

To visualize the effectiveness of the proposed EFEM compared to the BasicBlock module used in RT-DETR, GradCAM++ was employed to extract heatmaps from the outputs of the feature extraction modules at the same layer. Figure 10 presents these heatmaps, with the first column displaying the original images, the second column showing the features extracted using the BasicBlock module, and the third column illustrating the features extracted by the EFEM.

The first and second rows of Figure 10 demonstrate that for the challenging cases of Crack and Star crack defects (which typically exhibit lower detection accuracy), the EFEM is more effective at capturing precise information regarding the dispersion of cracks compared to the features extracted by BasicBlock. Furthermore, in the third and fourth rows, even under challenging conditions such as the low brightness of defect samples and defects crossing over busbars, the EFEM demonstrates accurate identification capabilities. These visual comparisons highlight the ability of the EFEM to capture more meaningful and precise features, contributing to the enhanced performance of the LEM-Detector.

4.5.3. The Effect of MMFPN

To visualize the effectiveness of the proposed MMFPN compared to the CCFF used in RT-DETR, GradCAM++ was employed to extract heatmaps from the outputs of the feature fusion modules at the same layer. Figure 11 presents these heatmaps, with the first column displaying the original images, the second column showing the features extracted using CCFF, the third column illustrating the feature maps extracted using EFEM (demonstrating the effectiveness of MMFPN in merging edge information), and the fourth column showcasing our proposed MMFPN.

In the first row of Figure 11, it is evident that for defects not recognized by RT-DETR and for defects that cross busbars (which are difficult to identify in the second row), MMFPN selectively filters and integrates the discontinuous and noisy precise information extracted by the EFEM. This process merges more edge information, enabling accurate defect information acquisition for classification and regression.

In the third row, RT-DETR fails to identify all defect types in the case of stacked defects. In contrast, the MMFPN demonstrates a superior ability to focus on these easily overlooked edge details, allowing for accurate identification and localization of the two defect types. These visual comparisons highlight the ability of MMFPN to effectively fuse multi-scale features and enhance edge information, contributing to the improved performance of the LEM-Detector.

5. Conclusions

This paper presents the LEM-Detector, an efficient end-to-end detector for photovoltaic panel defect detection. The proposed method addresses several challenges in this domain, including the wide range of defect scales, diverse defect types, and severe background interference. Our approach aims to enhance detection accuracy and robustness by introducing several novel components: the LFA, the EFEM, and the MMFPN.

The LFA module addresses the issue of imbalanced training data by focusing on augmenting linear feature samples using a cosine similarity function. This targeted augmentation strategy effectively improves the model’s ability to detect small-scale defects like cracks and star cracks. The EFEM module is designed to enhance the receptive field and capture subtle texture information, which is crucial for identifying defects amidst complex backgrounds. The MMFPN facilitates the aggregation of complementary information from different feature layers, ensuring that both high-level category features and edge information are effectively utilized.

The experimental results demonstrate the effectiveness of the proposed LEM-Detector. On the PVEL-AD dataset, our model achieves an mAP of 94.7%, outperforming several state-of-the-art methods. The improvements are particularly significant for the detection of Crack and Star crack defects, highlighting the effectiveness of the LFA module. Additionally, the EFEM and MMFPN contribute to the overall enhancement of multi-scale defect detection by preserving detailed information and improving feature fusion.

The aim and assumptions of this study have been achieved through the introduction of these novel components, which together address the challenges in photovoltaic panel defect detection. The results of this research are expected to be useful for industrial applications in the photovoltaic sector, enabling more reliable and accurate defect identification systems.

For future research directions, we plan to explore methods for automating the threshold generation in the LFA module to further streamline the training process. Additionally, we aim to develop a more lightweight model to enhance the practicality and deployability of the LEM-Detector in real-world scenarios. Overall, this work provides valuable technical references for visual inspection tasks in similar industrial applications.